Alex, I have commented in the ticket. Please take a look. Thanks! -- Yakov Zhdanov, Director R&D *GridGain Systems* www.gridgain.com
2017-06-29 17:27 GMT+03:00 ALEKSEY KUZNETSOV <alkuznetsov...@gmail.com>: > I've attached HangTest. I suppose it should not hang, am i right ? > > чт, 29 июн. 2017 г. в 14:54, ALEKSEY KUZNETSOV <alkuznetsov...@gmail.com>: > > > Igntrs. > > Im rewieving all usages of threadId of > > transaction.(IgniteTxAdapter#threadID). What is the point of usage > threadId > > in mvcc entry ? > > > > пн, 3 апр. 2017 г. в 9:47, ALEKSEY KUZNETSOV <alkuznetsov...@gmail.com>: > > > >> so what do u think on my idea? > >> > >> пт, 31 Мар 2017 г., 11:05 ALEKSEY KUZNETSOV <alkuznetsov...@gmail.com>: > >> > >>> sorry for misleading you. We planned to support multi-node > transactions, > >>> but failed. > >>> > >>> пт, 31 мар. 2017 г. в 10:51, Alexey Goncharuk < > >>> alexey.goncha...@gmail.com>: > >>> > >>> Well, now the scenario is more clear, but it has nothing to do with > >>> multiple coordinators :) Let me think a little bit about it. > >>> > >>> 2017-03-31 9:53 GMT+03:00 ALEKSEY KUZNETSOV <alkuznetsov...@gmail.com > >: > >>> > >>> > so what do u think on the issue ? > >>> > > >>> > чт, 30 Мар 2017 г., 17:49 ALEKSEY KUZNETSOV < > alkuznetsov...@gmail.com > >>> >: > >>> > > >>> > > Hi ! Thanks for help. I've created ticket : > >>> > > https://issues.apache.org/jira/browse/IGNITE-4887 > >>> > > and a commit : > >>> > > > >>> https://github.com/voipp/ignite/commit/aa3487bd9c203394f534c605f84e06 > >>> > 436b638e5c > >>> > > We really need this feature > >>> > > > >>> > > чт, 30 мар. 2017 г. в 11:31, Alexey Goncharuk < > >>> > alexey.goncha...@gmail.com > >>> > > >: > >>> > > > >>> > > Aleksey, > >>> > > > >>> > > I doubt your approach works as expected. Current transaction > recovery > >>> > > protocol heavily relies on the originating node ID in its internal > >>> logic. > >>> > > For example, currently a transaction will be rolled back if you > want > >>> to > >>> > > transfer a transaction ownership to another node and original tx > >>> owner > >>> > > fails. An attempt to commit such a transaction on another node may > >>> fail > >>> > > with all sorts of assertions. After transaction ownership changed, > >>> you > >>> > need > >>> > > to notify all current transaction participants about this change, > >>> and it > >>> > > should also be done failover-safe, let alone that you did not add > any > >>> > tests > >>> > > for these cases. > >>> > > > >>> > > I back Denis here. Please create a ticket first and come up with > >>> clear > >>> > > use-cases, API and protocol changes design. It is hard to reason > >>> about > >>> > the > >>> > > changes you've made when we do not even understand why you are > making > >>> > these > >>> > > changes and how they are supposed to work. > >>> > > > >>> > > --AG > >>> > > > >>> > > 2017-03-30 10:43 GMT+03:00 ALEKSEY KUZNETSOV < > >>> alkuznetsov...@gmail.com>: > >>> > > > >>> > > > So, what do u think on my idea ? > >>> > > > > >>> > > > ср, 29 мар. 2017 г. в 10:35, ALEKSEY KUZNETSOV < > >>> > alkuznetsov...@gmail.com > >>> > > >: > >>> > > > > >>> > > > > Hi! No, i dont have ticket for this. > >>> > > > > In the ticket i have implemented methods that change > transaction > >>> > status > >>> > > > to > >>> > > > > STOP, thus letting it to commit transaction in another thread. > In > >>> > > another > >>> > > > > thread you r going to restart transaction in order to commit > it. > >>> > > > > The mechanism behind it is obvious : we change thread id to > >>> newer one > >>> > > in > >>> > > > > ThreadMap, and make use of serialization of txState, > transactions > >>> > > itself > >>> > > > to > >>> > > > > transfer them into another thread. > >>> > > > > > >>> > > > > > >>> > > > > вт, 28 мар. 2017 г. в 20:15, Denis Magda <dma...@apache.org>: > >>> > > > > > >>> > > > > Aleksey, > >>> > > > > > >>> > > > > Do you have a ticket for this? Could you briefly list what > >>> exactly > >>> > was > >>> > > > > done and how the things work. > >>> > > > > > >>> > > > > — > >>> > > > > Denis > >>> > > > > > >>> > > > > > On Mar 28, 2017, at 8:32 AM, ALEKSEY KUZNETSOV < > >>> > > > alkuznetsov...@gmail.com> > >>> > > > > wrote: > >>> > > > > > > >>> > > > > > Hi, Igniters! I 've made implementation of transactions of > >>> > non-single > >>> > > > > > coordinator. Here you can start transaction in one thread and > >>> > commit > >>> > > it > >>> > > > > in > >>> > > > > > another thread. > >>> > > > > > Take a look on it. Give your thoughts on it. > >>> > > > > > > >>> > > > > > > >>> > > > > https://github.com/voipp/ignite/pull/10/commits/ > >>> > > > 3a3d90aa6ac84f125e4c3ce4ced4f269a695ef45 > >>> > > > > > > >>> > > > > > пт, 17 мар. 2017 г. в 19:26, Sergi Vladykin < > >>> > > sergi.vlady...@gmail.com > >>> > > > >: > >>> > > > > > > >>> > > > > >> You know better, go ahead! :) > >>> > > > > >> > >>> > > > > >> Sergi > >>> > > > > >> > >>> > > > > >> 2017-03-17 16:16 GMT+03:00 ALEKSEY KUZNETSOV < > >>> > > > alkuznetsov...@gmail.com > >>> > > > > >: > >>> > > > > >> > >>> > > > > >>> we've discovered several problems regarding your > >>> "accumulation" > >>> > > > > >>> approach.These are > >>> > > > > >>> > >>> > > > > >>> 1. perfomance issues when transfering data from temporary > >>> cache > >>> > > to > >>> > > > > >>> permanent one. Keep in mind big deal of concurent > >>> transactions > >>> > in > >>> > > > > >>> Service > >>> > > > > >>> commiter > >>> > > > > >>> 2. extreme memory load when keeping temporary cache in > >>> memory > >>> > > > > >>> 3. As long as user is not acquainted with ignite, working > >>> with > >>> > > > cache > >>> > > > > >>> must be transparent for him. Keep this in mind. User's > >>> node can > >>> > > > > >> evaluate > >>> > > > > >>> logic with no transaction at all, so we should deal with > >>> both > >>> > > types > >>> > > > > of > >>> > > > > >>> execution flow : transactional and > >>> non-transactional.Another > >>> > one > >>> > > > > >>> problem is > >>> > > > > >>> transaction id support at the user node. We would have > >>> handled > >>> > > all > >>> > > > > >> this > >>> > > > > >>> issues and many more. > >>> > > > > >>> 4. we cannot pessimistically lock entity. > >>> > > > > >>> > >>> > > > > >>> As a result, we decided to move on building distributed > >>> > > transaction. > >>> > > > We > >>> > > > > >> put > >>> > > > > >>> aside your "accumulation" approach until we realize how to > >>> solve > >>> > > > > >>> difficulties above . > >>> > > > > >>> > >>> > > > > >>> чт, 16 мар. 2017 г. в 16:56, Sergi Vladykin < > >>> > > > sergi.vlady...@gmail.com > >>> > > > > >: > >>> > > > > >>> > >>> > > > > >>>> The problem "How to run millions of entities, and millions > >>> of > >>> > > > > >> operations > >>> > > > > >>> on > >>> > > > > >>>> a single Pentium3" is out of scope here. Do the math, plan > >>> > > capacity > >>> > > > > >>>> reasonably. > >>> > > > > >>>> > >>> > > > > >>>> Sergi > >>> > > > > >>>> > >>> > > > > >>>> 2017-03-16 15:54 GMT+03:00 ALEKSEY KUZNETSOV < > >>> > > > > alkuznetsov...@gmail.com > >>> > > > > >>> : > >>> > > > > >>>> > >>> > > > > >>>>> hmm, If we have millions of entities, and millions of > >>> > operations, > >>> > > > > >> would > >>> > > > > >>>> not > >>> > > > > >>>>> this approache lead to memory overflow and perfomance > >>> > degradation > >>> > > > > >>>>> > >>> > > > > >>>>> чт, 16 мар. 2017 г. в 15:42, Sergi Vladykin < > >>> > > > > >> sergi.vlady...@gmail.com > >>> > > > > >>>> : > >>> > > > > >>>>> > >>> > > > > >>>>>> 1. Actually you have to check versions on all the values > >>> you > >>> > > have > >>> > > > > >>> read > >>> > > > > >>>>>> during the tx. > >>> > > > > >>>>>> > >>> > > > > >>>>>> For example if we have [k1 => v1, k2 => v2] and do: > >>> > > > > >>>>>> > >>> > > > > >>>>>> put(k1, get(k2) + 5) > >>> > > > > >>>>>> > >>> > > > > >>>>>> We have to remember the version for k2. This logic can > be > >>> > > > > >> relatively > >>> > > > > >>>>> easily > >>> > > > > >>>>>> encapsulated in a framework atop of Ignite. You need to > >>> > > implement > >>> > > > > >> one > >>> > > > > >>>> to > >>> > > > > >>>>>> make all this stuff usable. > >>> > > > > >>>>>> > >>> > > > > >>>>>> 2. I suggest to avoid any locking here, because you > easily > >>> > will > >>> > > > end > >>> > > > > >>> up > >>> > > > > >>>>> with > >>> > > > > >>>>>> deadlocks. If you do not have too frequent updates for > >>> your > >>> > > keys, > >>> > > > > >>>>>> optimistic approach will work just fine. > >>> > > > > >>>>>> > >>> > > > > >>>>>> Theoretically in the Committer Service you can start a > >>> thread > >>> > > for > >>> > > > > >> the > >>> > > > > >>>>>> lifetime of the whole distributed transaction, take a > >>> lock on > >>> > > the > >>> > > > > >> key > >>> > > > > >>>>> using > >>> > > > > >>>>>> IgniteCache.lock(K key) before executing any Services, > >>> wait > >>> > for > >>> > > > all > >>> > > > > >>> the > >>> > > > > >>>>>> services to complete, execute optimistic commit in the > >>> same > >>> > > thread > >>> > > > > >>>> while > >>> > > > > >>>>>> keeping this lock and then release it. Notice that all > the > >>> > > Ignite > >>> > > > > >>>>>> transactions inside of all Services must be optimistic > >>> here to > >>> > > be > >>> > > > > >>> able > >>> > > > > >>>> to > >>> > > > > >>>>>> read this locked key. > >>> > > > > >>>>>> > >>> > > > > >>>>>> But again I do not recommend you using this approach > >>> until you > >>> > > > > >> have a > >>> > > > > >>>>>> reliable deadlock avoidance scheme. > >>> > > > > >>>>>> > >>> > > > > >>>>>> Sergi > >>> > > > > >>>>>> > >>> > > > > >>>>>> > >>> > > > > >>>>>> > >>> > > > > >>>>>> > >>> > > > > >>>>>> > >>> > > > > >>>>>> > >>> > > > > >>>>>> > >>> > > > > >>>>>> 2017-03-16 12:53 GMT+03:00 ALEKSEY KUZNETSOV < > >>> > > > > >>> alkuznetsov...@gmail.com > >>> > > > > >>>>> : > >>> > > > > >>>>>> > >>> > > > > >>>>>>> Yeah, now i got it. > >>> > > > > >>>>>>> There are some doubts on this approach > >>> > > > > >>>>>>> 1) During optimistic commit phase, when you assure no > one > >>> > > altered > >>> > > > > >>> the > >>> > > > > >>>>>>> original values, you must check versions of other > >>> dependent > >>> > > keys. > >>> > > > > >>> How > >>> > > > > >>>>>> could > >>> > > > > >>>>>>> we obtain those keys(in an automative manner, of > course) > >>> ? > >>> > > > > >>>>>>> 2) How could we lock a key before some Service A > >>> introduce > >>> > > > > >> changes? > >>> > > > > >>>> So > >>> > > > > >>>>> no > >>> > > > > >>>>>>> other service is allowed to change this key-value?(sort > >>> of > >>> > > > > >>>> pessimistic > >>> > > > > >>>>>>> blocking) > >>> > > > > >>>>>>> May be you know some implementations of such approach ? > >>> > > > > >>>>>>> > >>> > > > > >>>>>>> ср, 15 мар. 2017 г. в 17:54, ALEKSEY KUZNETSOV < > >>> > > > > >>>>> alkuznetsov...@gmail.com > >>> > > > > >>>>>>> : > >>> > > > > >>>>>>> > >>> > > > > >>>>>>>> Thank you very much for help. I will answer later. > >>> > > > > >>>>>>>> > >>> > > > > >>>>>>>> ср, 15 мар. 2017 г. в 17:39, Sergi Vladykin < > >>> > > > > >>>>> sergi.vlady...@gmail.com > >>> > > > > >>>>>>> : > >>> > > > > >>>>>>>> > >>> > > > > >>>>>>>> All the services do not update key in place, but only > >>> > generate > >>> > > > > >>> new > >>> > > > > >>>>> keys > >>> > > > > >>>>>>>> augmented by otx and store the updated value in the > same > >>> > cache > >>> > > > > >> + > >>> > > > > >>>>>> remember > >>> > > > > >>>>>>>> the keys and versions participating in the transaction > >>> in > >>> > some > >>> > > > > >>>>> separate > >>> > > > > >>>>>>>> atomic cache. > >>> > > > > >>>>>>>> > >>> > > > > >>>>>>>> Follow this sequence of changes applied to cache > >>> contents by > >>> > > > > >> each > >>> > > > > >>>>>>> Service: > >>> > > > > >>>>>>>> > >>> > > > > >>>>>>>> Initial cache contents: > >>> > > > > >>>>>>>> [k1 => v1] > >>> > > > > >>>>>>>> [k2 => v2] > >>> > > > > >>>>>>>> [k3 => v3] > >>> > > > > >>>>>>>> > >>> > > > > >>>>>>>> Cache contents after Service A: > >>> > > > > >>>>>>>> [k1 => v1] > >>> > > > > >>>>>>>> [k2 => v2] > >>> > > > > >>>>>>>> [k3 => v3] > >>> > > > > >>>>>>>> [k1x => v1a] > >>> > > > > >>>>>>>> [k2x => v2a] > >>> > > > > >>>>>>>> > >>> > > > > >>>>>>>> + [x => (k1 -> ver1, k2 -> ver2)] in some > >>> separate > >>> > > > > >>> atomic > >>> > > > > >>>>>> cache > >>> > > > > >>>>>>>> > >>> > > > > >>>>>>>> Cache contents after Service B: > >>> > > > > >>>>>>>> [k1 => v1] > >>> > > > > >>>>>>>> [k2 => v2] > >>> > > > > >>>>>>>> [k3 => v3] > >>> > > > > >>>>>>>> [k1x => v1a] > >>> > > > > >>>>>>>> [k2x => v2ab] > >>> > > > > >>>>>>>> [k3x => v3b] > >>> > > > > >>>>>>>> > >>> > > > > >>>>>>>> + [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] > in > >>> some > >>> > > > > >>>>> separate > >>> > > > > >>>>>>>> atomic cache > >>> > > > > >>>>>>>> > >>> > > > > >>>>>>>> Finally the Committer Service takes this map of > updated > >>> keys > >>> > > > > >> and > >>> > > > > >>>>> their > >>> > > > > >>>>>>>> versions from some separate atomic cache, starts > Ignite > >>> > > > > >>> transaction > >>> > > > > >>>>> and > >>> > > > > >>>>>>>> replaces all the values for k* keys to values taken > >>> from k*x > >>> > > > > >>> keys. > >>> > > > > >>>>> The > >>> > > > > >>>>>>>> successful result must be the following: > >>> > > > > >>>>>>>> > >>> > > > > >>>>>>>> [k1 => v1a] > >>> > > > > >>>>>>>> [k2 => v2ab] > >>> > > > > >>>>>>>> [k3 => v3b] > >>> > > > > >>>>>>>> [k1x => v1a] > >>> > > > > >>>>>>>> [k2x => v2ab] > >>> > > > > >>>>>>>> [k3x => v3b] > >>> > > > > >>>>>>>> > >>> > > > > >>>>>>>> + [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] > in > >>> some > >>> > > > > >>>>> separate > >>> > > > > >>>>>>>> atomic cache > >>> > > > > >>>>>>>> > >>> > > > > >>>>>>>> But Committer Service also has to check that no one > >>> updated > >>> > > the > >>> > > > > >>>>>> original > >>> > > > > >>>>>>>> values before us, because otherwise we can not give > any > >>> > > > > >>>>> serializability > >>> > > > > >>>>>>>> guarantee for these distributed transactions. Here we > >>> may > >>> > need > >>> > > > > >> to > >>> > > > > >>>>> check > >>> > > > > >>>>>>> not > >>> > > > > >>>>>>>> only versions of the updated keys, but also versions > of > >>> any > >>> > > > > >> other > >>> > > > > >>>>> keys > >>> > > > > >>>>>>> end > >>> > > > > >>>>>>>> result depends on. > >>> > > > > >>>>>>>> > >>> > > > > >>>>>>>> After that Committer Service has to do a cleanup (may > be > >>> > > > > >> outside > >>> > > > > >>> of > >>> > > > > >>>>> the > >>> > > > > >>>>>>>> committing tx) to come to the following final state: > >>> > > > > >>>>>>>> > >>> > > > > >>>>>>>> [k1 => v1a] > >>> > > > > >>>>>>>> [k2 => v2ab] > >>> > > > > >>>>>>>> [k3 => v3b] > >>> > > > > >>>>>>>> > >>> > > > > >>>>>>>> Makes sense? > >>> > > > > >>>>>>>> > >>> > > > > >>>>>>>> Sergi > >>> > > > > >>>>>>>> > >>> > > > > >>>>>>>> > >>> > > > > >>>>>>>> 2017-03-15 16:54 GMT+03:00 ALEKSEY KUZNETSOV < > >>> > > > > >>>>> alkuznetsov...@gmail.com > >>> > > > > >>>>>>> : > >>> > > > > >>>>>>>> > >>> > > > > >>>>>>>>> - what do u mean by saying " > >>> > > > > >>>>>>>>> *in a single transaction checks value versions for > all > >>> the > >>> > > > > >> old > >>> > > > > >>>>> values > >>> > > > > >>>>>>>>> and replaces them with calculated new ones *"? > Every > >>> > time > >>> > > > > >>> you > >>> > > > > >>>>>>> change > >>> > > > > >>>>>>>>> value(in some service), you store it to *some > special > >>> > > > > >> atomic > >>> > > > > >>>>>> cache* > >>> > > > > >>>>>>> , > >>> > > > > >>>>>>>> so > >>> > > > > >>>>>>>>> when all services ceased working, Service commiter > >>> got a > >>> > > > > >>>> values > >>> > > > > >>>>>> with > >>> > > > > >>>>>>>> the > >>> > > > > >>>>>>>>> last versions. > >>> > > > > >>>>>>>>> - After "*does cleanup of temporary keys and > values*" > >>> > > > > >>> Service > >>> > > > > >>>>>>> commiter > >>> > > > > >>>>>>>>> persists them into permanent store, isn't it ? > >>> > > > > >>>>>>>>> - I cant grasp your though, you say "*in case of > >>> version > >>> > > > > >>>>> mismatch > >>> > > > > >>>>>> or > >>> > > > > >>>>>>>> TX > >>> > > > > >>>>>>>>> timeout just rollbacks*". But what versions would > it > >>> > > > > >> match? > >>> > > > > >>>>>>>>> > >>> > > > > >>>>>>>>> > >>> > > > > >>>>>>>>> ср, 15 мар. 2017 г. в 15:34, Sergi Vladykin < > >>> > > > > >>>>>> sergi.vlady...@gmail.com > >>> > > > > >>>>>>>> : > >>> > > > > >>>>>>>>> > >>> > > > > >>>>>>>>>> Ok, here is what you actually need to implement at > the > >>> > > > > >>>>> application > >>> > > > > >>>>>>>> level. > >>> > > > > >>>>>>>>>> > >>> > > > > >>>>>>>>>> Lets say we have to call 2 services in the following > >>> > order: > >>> > > > > >>>>>>>>>> - Service A: wants to update keys [k1 => v1, k2 => > >>> v2] > >>> > > > > >> to > >>> > > > > >>>>> [k1 > >>> > > > > >>>>>> => > >>> > > > > >>>>>>>>> v1a, > >>> > > > > >>>>>>>>>> k2 => v2a] > >>> > > > > >>>>>>>>>> - Service B: wants to update keys [k2 => v2a, k3 => > >>> v3] > >>> > > > > >> to > >>> > > > > >>>> [k2 > >>> > > > > >>>>>> => > >>> > > > > >>>>>>>>> v2ab, > >>> > > > > >>>>>>>>>> k3 => v3b] > >>> > > > > >>>>>>>>>> > >>> > > > > >>>>>>>>>> The change > >>> > > > > >>>>>>>>>> from [ k1 => v1, k2 => v2, k3 => v3 ] > >>> > > > > >>>>>>>>>> to [ k1 => v1a, k2 => v2ab, k3 => v3b ] > >>> > > > > >>>>>>>>>> must happen in a single transaction. > >>> > > > > >>>>>>>>>> > >>> > > > > >>>>>>>>>> > >>> > > > > >>>>>>>>>> Optimistic protocol to solve this: > >>> > > > > >>>>>>>>>> > >>> > > > > >>>>>>>>>> Each cache key must have a field `otx`, which is a > >>> unique > >>> > > > > >>>>>>> orchestrator > >>> > > > > >>>>>>>> TX > >>> > > > > >>>>>>>>>> identifier - it must be a parameter passed to all > the > >>> > > > > >>> services. > >>> > > > > >>>>> If > >>> > > > > >>>>>>>> `otx` > >>> > > > > >>>>>>>>> is > >>> > > > > >>>>>>>>>> set to some value it means that it is an > intermediate > >>> key > >>> > > > > >> and > >>> > > > > >>>> is > >>> > > > > >>>>>>>> visible > >>> > > > > >>>>>>>>>> only inside of some transaction, for the finalized > key > >>> > > > > >> `otx` > >>> > > > > >>>> must > >>> > > > > >>>>>> be > >>> > > > > >>>>>>>>> null - > >>> > > > > >>>>>>>>>> it means the key is committed and visible for > >>> everyone. > >>> > > > > >>>>>>>>>> > >>> > > > > >>>>>>>>>> Each cache value must have a field `ver` which is a > >>> > version > >>> > > > > >>> of > >>> > > > > >>>>> that > >>> > > > > >>>>>>>>> value. > >>> > > > > >>>>>>>>>> > >>> > > > > >>>>>>>>>> For both fields (`otx` and `ver`) the safest way is > >>> to use > >>> > > > > >>>> UUID. > >>> > > > > >>>>>>>>>> > >>> > > > > >>>>>>>>>> Workflow is the following: > >>> > > > > >>>>>>>>>> > >>> > > > > >>>>>>>>>> Orchestrator starts the distributed transaction with > >>> `otx` > >>> > > > > >> = > >>> > > > > >>> x > >>> > > > > >>>>> and > >>> > > > > >>>>>>>> passes > >>> > > > > >>>>>>>>>> this parameter to all the services. > >>> > > > > >>>>>>>>>> > >>> > > > > >>>>>>>>>> Service A: > >>> > > > > >>>>>>>>>> - does some computations > >>> > > > > >>>>>>>>>> - stores [k1x => v1a, k2x => v2a] with TTL = Za > >>> > > > > >>>>>>>>>> where > >>> > > > > >>>>>>>>>> Za - left time from max Orchestrator TX > >>> duration > >>> > > > > >>>> after > >>> > > > > >>>>>>>> Service > >>> > > > > >>>>>>>>> A > >>> > > > > >>>>>>>>>> end > >>> > > > > >>>>>>>>>> k1x, k2x - new temporary keys with field > >>> `otx` = > >>> > > > > >> x > >>> > > > > >>>>>>>>>> v2a has updated version `ver` > >>> > > > > >>>>>>>>>> - returns a set of updated keys and all the old > >>> versions > >>> > > > > >> to > >>> > > > > >>>> the > >>> > > > > >>>>>>>>>> orchestrator > >>> > > > > >>>>>>>>>> or just stores it in some special atomic cache > >>> like > >>> > > > > >>>>>>>>>> [x => (k1 -> ver1, k2 -> ver2)] TTL = Za > >>> > > > > >>>>>>>>>> > >>> > > > > >>>>>>>>>> Service B: > >>> > > > > >>>>>>>>>> - retrieves the updated value k2x => v2a because it > >>> knows > >>> > > > > >>>> `otx` > >>> > > > > >>>>> = > >>> > > > > >>>>>> x > >>> > > > > >>>>>>>>>> - does computations > >>> > > > > >>>>>>>>>> - stores [k2x => v2ab, k3x => v3b] TTL = Zb > >>> > > > > >>>>>>>>>> - updates the set of updated keys like [x => (k1 -> > >>> ver1, > >>> > > > > >> k2 > >>> > > > > >>>> -> > >>> > > > > >>>>>>> ver2, > >>> > > > > >>>>>>>> k3 > >>> > > > > >>>>>>>>>> -> ver3)] TTL = Zb > >>> > > > > >>>>>>>>>> > >>> > > > > >>>>>>>>>> Service Committer (may be embedded into > Orchestrator): > >>> > > > > >>>>>>>>>> - takes all the updated keys and versions for `otx` > = > >>> x > >>> > > > > >>>>>>>>>> [x => (k1 -> ver1, k2 -> ver2, k3 -> ver3)] > >>> > > > > >>>>>>>>>> - in a single transaction checks value versions for > >>> all > >>> > > > > >> the > >>> > > > > >>>> old > >>> > > > > >>>>>>> values > >>> > > > > >>>>>>>>>> and replaces them with calculated new ones > >>> > > > > >>>>>>>>>> - does cleanup of temporary keys and values > >>> > > > > >>>>>>>>>> - in case of version mismatch or TX timeout just > >>> rollbacks > >>> > > > > >>> and > >>> > > > > >>>>>>> signals > >>> > > > > >>>>>>>>>> to Orchestrator to restart the job with new > >>> `otx` > >>> > > > > >>>>>>>>>> > >>> > > > > >>>>>>>>>> PROFIT!! > >>> > > > > >>>>>>>>>> > >>> > > > > >>>>>>>>>> This approach even allows you to run independent > >>> parts of > >>> > > > > >> the > >>> > > > > >>>>> graph > >>> > > > > >>>>>>> in > >>> > > > > >>>>>>>>>> parallel (with TX transfer you will always run only > >>> one at > >>> > > > > >> a > >>> > > > > >>>>> time). > >>> > > > > >>>>>>>> Also > >>> > > > > >>>>>>>>> it > >>> > > > > >>>>>>>>>> does not require inventing any special fault > tolerance > >>> > > > > >>> technics > >>> > > > > >>>>>>> because > >>> > > > > >>>>>>>>>> Ignite caches are already fault tolerant and all the > >>> > > > > >>>> intermediate > >>> > > > > >>>>>>>> results > >>> > > > > >>>>>>>>>> are virtually invisible and stored with TTL, thus in > >>> case > >>> > > > > >> of > >>> > > > > >>>> any > >>> > > > > >>>>>>> crash > >>> > > > > >>>>>>>>> you > >>> > > > > >>>>>>>>>> will not have inconsistent state or garbage. > >>> > > > > >>>>>>>>>> > >>> > > > > >>>>>>>>>> Sergi > >>> > > > > >>>>>>>>>> > >>> > > > > >>>>>>>>>> > >>> > > > > >>>>>>>>>> 2017-03-15 11:42 GMT+03:00 ALEKSEY KUZNETSOV < > >>> > > > > >>>>>>> alkuznetsov...@gmail.com > >>> > > > > >>>>>>>>> : > >>> > > > > >>>>>>>>>> > >>> > > > > >>>>>>>>>>> Okay, we are open for proposals on business task. I > >>> mean, > >>> > > > > >>> we > >>> > > > > >>>>> can > >>> > > > > >>>>>>> make > >>> > > > > >>>>>>>>> use > >>> > > > > >>>>>>>>>>> of some other thing, not distributed transaction. > Not > >>> > > > > >>>>> transaction > >>> > > > > >>>>>>>> yet. > >>> > > > > >>>>>>>>>>> > >>> > > > > >>>>>>>>>>> ср, 15 мар. 2017 г. в 11:24, Vladimir Ozerov < > >>> > > > > >>>>>> voze...@gridgain.com > >>> > > > > >>>>>>>> : > >>> > > > > >>>>>>>>>>> > >>> > > > > >>>>>>>>>>>> IMO the use case makes sense. However, as Sergi > >>> already > >>> > > > > >>>>>>> mentioned, > >>> > > > > >>>>>>>>> the > >>> > > > > >>>>>>>>>>>> problem is far more complex, than simply passing > TX > >>> > > > > >> state > >>> > > > > >>>>> over > >>> > > > > >>>>>> a > >>> > > > > >>>>>>>>> wire. > >>> > > > > >>>>>>>>>>> Most > >>> > > > > >>>>>>>>>>>> probably a kind of coordinator will be required > >>> still > >>> > > > > >> to > >>> > > > > >>>>> manage > >>> > > > > >>>>>>> all > >>> > > > > >>>>>>>>>> kinds > >>> > > > > >>>>>>>>>>>> of failures. This task should be started with > clean > >>> > > > > >>> design > >>> > > > > >>>>>>> proposal > >>> > > > > >>>>>>>>>>>> explaining how we handle all these concurrent > >>> events. > >>> > > > > >> And > >>> > > > > >>>>> only > >>> > > > > >>>>>>>> then, > >>> > > > > >>>>>>>>>> when > >>> > > > > >>>>>>>>>>>> we understand all implications, we should move to > >>> > > > > >>>> development > >>> > > > > >>>>>>>> stage. > >>> > > > > >>>>>>>>>>>> > >>> > > > > >>>>>>>>>>>> On Wed, Mar 15, 2017 at 10:38 AM, ALEKSEY > KUZNETSOV > >>> < > >>> > > > > >>>>>>>>>>>> alkuznetsov...@gmail.com> wrote: > >>> > > > > >>>>>>>>>>>> > >>> > > > > >>>>>>>>>>>>> Right > >>> > > > > >>>>>>>>>>>>> > >>> > > > > >>>>>>>>>>>>> ср, 15 мар. 2017 г. в 10:35, Sergi Vladykin < > >>> > > > > >>>>>>>>>> sergi.vlady...@gmail.com > >>> > > > > >>>>>>>>>>>> : > >>> > > > > >>>>>>>>>>>>> > >>> > > > > >>>>>>>>>>>>>> Good! Basically your orchestrator just takes > some > >>> > > > > >>>>>> predefined > >>> > > > > >>>>>>>>> graph > >>> > > > > >>>>>>>>>> of > >>> > > > > >>>>>>>>>>>>>> distributed services to be invoked, calls them > by > >>> > > > > >>> some > >>> > > > > >>>>> kind > >>> > > > > >>>>>>> of > >>> > > > > >>>>>>>>> RPC > >>> > > > > >>>>>>>>>>> and > >>> > > > > >>>>>>>>>>>>>> passes the needed parameters between them, > right? > >>> > > > > >>>>>>>>>>>>>> > >>> > > > > >>>>>>>>>>>>>> Sergi > >>> > > > > >>>>>>>>>>>>>> > >>> > > > > >>>>>>>>>>>>>> 2017-03-14 22:46 GMT+03:00 ALEKSEY KUZNETSOV < > >>> > > > > >>>>>>>>>>> alkuznetsov...@gmail.com > >>> > > > > >>>>>>>>>>>>> : > >>> > > > > >>>>>>>>>>>>>> > >>> > > > > >>>>>>>>>>>>>>> orchestrator is a custom thing. He is > responsible > >>> > > > > >>> for > >>> > > > > >>>>>>>> managing > >>> > > > > >>>>>>>>>>>> business > >>> > > > > >>>>>>>>>>>>>>> scenarios flows. Many nodes are involved in > >>> > > > > >>>> scenarios. > >>> > > > > >>>>>> They > >>> > > > > >>>>>>>>>>> exchange > >>> > > > > >>>>>>>>>>>>> data > >>> > > > > >>>>>>>>>>>>>>> and folow one another. If you acquinted with > BPMN > >>> > > > > >>>>>>> framework, > >>> > > > > >>>>>>>> so > >>> > > > > >>>>>>>>>>>>>>> orchestrator is like bpmn engine. > >>> > > > > >>>>>>>>>>>>>>> > >>> > > > > >>>>>>>>>>>>>>> вт, 14 Мар 2017 г., 18:56 Sergi Vladykin < > >>> > > > > >>>>>>>>>> sergi.vlady...@gmail.com > >>> > > > > >>>>>>>>>>>> : > >>> > > > > >>>>>>>>>>>>>>> > >>> > > > > >>>>>>>>>>>>>>>> What is Orchestrator for you? Is it a thing > >>> > > > > >> from > >>> > > > > >>>>>>> Microsoft > >>> > > > > >>>>>>>> or > >>> > > > > >>>>>>>>>>> your > >>> > > > > >>>>>>>>>>>>>> custom > >>> > > > > >>>>>>>>>>>>>>>> in-house software? > >>> > > > > >>>>>>>>>>>>>>>> > >>> > > > > >>>>>>>>>>>>>>>> Sergi > >>> > > > > >>>>>>>>>>>>>>>> > >>> > > > > >>>>>>>>>>>>>>>> 2017-03-14 18:00 GMT+03:00 ALEKSEY KUZNETSOV < > >>> > > > > >>>>>>>>>>>>> alkuznetsov...@gmail.com > >>> > > > > >>>>>>>>>>>>>>> : > >>> > > > > >>>>>>>>>>>>>>>> > >>> > > > > >>>>>>>>>>>>>>>>> Fine. Let's say we've got multiple servers > >>> > > > > >>> which > >>> > > > > >>>>>>> fulfills > >>> > > > > >>>>>>>>>>> custom > >>> > > > > >>>>>>>>>>>>>> logic. > >>> > > > > >>>>>>>>>>>>>>>>> This servers compound oriented graph (BPMN > >>> > > > > >>>> process) > >>> > > > > >>>>>>> which > >>> > > > > >>>>>>>>>>>>> controlled > >>> > > > > >>>>>>>>>>>>>> by > >>> > > > > >>>>>>>>>>>>>>>>> Orchestrator. > >>> > > > > >>>>>>>>>>>>>>>>> For instance, *server1 *creates *variable A > >>> > > > > >>>> *with > >>> > > > > >>>>>>> value > >>> > > > > >>>>>>>> 1, > >>> > > > > >>>>>>>>>>>>> persists > >>> > > > > >>>>>>>>>>>>>> it > >>> > > > > >>>>>>>>>>>>>>>> to > >>> > > > > >>>>>>>>>>>>>>>>> IGNITE cache and creates *variable B *and > >>> > > > > >> sends > >>> > > > > >>>> it > >>> > > > > >>>>>> to* > >>> > > > > >>>>>>>>>> server2. > >>> > > > > >>>>>>>>>>>>> *The > >>> > > > > >>>>>>>>>>>>>>>>> latests receives *variable B*, do some logic > >>> > > > > >>> with > >>> > > > > >>>>> it > >>> > > > > >>>>>>> and > >>> > > > > >>>>>>>>>> stores > >>> > > > > >>>>>>>>>>>> to > >>> > > > > >>>>>>>>>>>>>>>> IGNITE. > >>> > > > > >>>>>>>>>>>>>>>>> All the work made by both servers must be > >>> > > > > >>>> fulfilled > >>> > > > > >>>>>> in > >>> > > > > >>>>>>>>> *one* > >>> > > > > >>>>>>>>>>>>>>> transaction. > >>> > > > > >>>>>>>>>>>>>>>>> Because we need all information done, or > >>> > > > > >>>>>>>>> nothing(rollbacked). > >>> > > > > >>>>>>>>>>> The > >>> > > > > >>>>>>>>>>>>>>>> scenario > >>> > > > > >>>>>>>>>>>>>>>>> is managed by orchestrator. > >>> > > > > >>>>>>>>>>>>>>>>> > >>> > > > > >>>>>>>>>>>>>>>>> вт, 14 мар. 2017 г. в 17:31, Sergi Vladykin < > >>> > > > > >>>>>>>>>>>>>> sergi.vlady...@gmail.com > >>> > > > > >>>>>>>>>>>>>>>> : > >>> > > > > >>>>>>>>>>>>>>>>> > >>> > > > > >>>>>>>>>>>>>>>>>> Ok, it is not a business case, it is your > >>> > > > > >>> wrong > >>> > > > > >>>>>>>> solution > >>> > > > > >>>>>>>>>> for > >>> > > > > >>>>>>>>>>>> it. > >>> > > > > >>>>>>>>>>>>>>>>>> Lets try again, what is the business case? > >>> > > > > >>>>>>>>>>>>>>>>>> > >>> > > > > >>>>>>>>>>>>>>>>>> Sergi > >>> > > > > >>>>>>>>>>>>>>>>>> > >>> > > > > >>>>>>>>>>>>>>>>>> 2017-03-14 16:42 GMT+03:00 ALEKSEY > >>> > > > > >> KUZNETSOV > >>> > > > > >>> < > >>> > > > > >>>>>>>>>>>>>>> alkuznetsov...@gmail.com > >>> > > > > >>>>>>>>>>>>>>>>> : > >>> > > > > >>>>>>>>>>>>>>>>>> > >>> > > > > >>>>>>>>>>>>>>>>>>> The case is the following, One starts > >>> > > > > >>>>> transaction > >>> > > > > >>>>>>> in > >>> > > > > >>>>>>>>> one > >>> > > > > >>>>>>>>>>>> node, > >>> > > > > >>>>>>>>>>>>>> and > >>> > > > > >>>>>>>>>>>>>>>>> commit > >>> > > > > >>>>>>>>>>>>>>>>>>> this transaction in another jvm node(or > >>> > > > > >>>>> rollback > >>> > > > > >>>>>> it > >>> > > > > >>>>>>>>>>>> remotely). > >>> > > > > >>>>>>>>>>>>>>>>>>> > >>> > > > > >>>>>>>>>>>>>>>>>>> вт, 14 мар. 2017 г. в 16:30, Sergi > >>> > > > > >>> Vladykin < > >>> > > > > >>>>>>>>>>>>>>>> sergi.vlady...@gmail.com > >>> > > > > >>>>>>>>>>>>>>>>>> : > >>> > > > > >>>>>>>>>>>>>>>>>>> > >>> > > > > >>>>>>>>>>>>>>>>>>>> Because even if you make it work for > >>> > > > > >> some > >>> > > > > >>>>>>>> simplistic > >>> > > > > >>>>>>>>>>>>> scenario, > >>> > > > > >>>>>>>>>>>>>>> get > >>> > > > > >>>>>>>>>>>>>>>>>> ready > >>> > > > > >>>>>>>>>>>>>>>>>>> to > >>> > > > > >>>>>>>>>>>>>>>>>>>> write many fault tolerance tests and > >>> > > > > >> make > >>> > > > > >>>>> sure > >>> > > > > >>>>>>> that > >>> > > > > >>>>>>>>> you > >>> > > > > >>>>>>>>>>> TXs > >>> > > > > >>>>>>>>>>>>>> work > >>> > > > > >>>>>>>>>>>>>>>>>>> gracefully > >>> > > > > >>>>>>>>>>>>>>>>>>>> in all modes in case of crashes. Also > >>> > > > > >>> make > >>> > > > > >>>>> sure > >>> > > > > >>>>>>>> that > >>> > > > > >>>>>>>>> we > >>> > > > > >>>>>>>>>>> do > >>> > > > > >>>>>>>>>>>>> not > >>> > > > > >>>>>>>>>>>>>>> have > >>> > > > > >>>>>>>>>>>>>>>>> any > >>> > > > > >>>>>>>>>>>>>>>>>>>> performance drops after all your > >>> > > > > >> changes > >>> > > > > >>> in > >>> > > > > >>>>>>>> existing > >>> > > > > >>>>>>>>>>>>>> benchmarks. > >>> > > > > >>>>>>>>>>>>>>>> All > >>> > > > > >>>>>>>>>>>>>>>>> in > >>> > > > > >>>>>>>>>>>>>>>>>>> all > >>> > > > > >>>>>>>>>>>>>>>>>>>> I don't believe these conditions will > >>> > > > > >> be > >>> > > > > >>>> met > >>> > > > > >>>>>> and > >>> > > > > >>>>>>>> your > >>> > > > > >>>>>>>>>>>>>>> contribution > >>> > > > > >>>>>>>>>>>>>>>>> will > >>> > > > > >>>>>>>>>>>>>>>>>>> be > >>> > > > > >>>>>>>>>>>>>>>>>>>> accepted. > >>> > > > > >>>>>>>>>>>>>>>>>>>> > >>> > > > > >>>>>>>>>>>>>>>>>>>> Better solution to what problem? > >>> > > > > >> Sending > >>> > > > > >>> TX > >>> > > > > >>>>> to > >>> > > > > >>>>>>>>> another > >>> > > > > >>>>>>>>>>>> node? > >>> > > > > >>>>>>>>>>>>>> The > >>> > > > > >>>>>>>>>>>>>>>>>> problem > >>> > > > > >>>>>>>>>>>>>>>>>>>> statement itself is already wrong. What > >>> > > > > >>>>>> business > >>> > > > > >>>>>>>> case > >>> > > > > >>>>>>>>>> you > >>> > > > > >>>>>>>>>>>> are > >>> > > > > >>>>>>>>>>>>>>>> trying > >>> > > > > >>>>>>>>>>>>>>>>> to > >>> > > > > >>>>>>>>>>>>>>>>>>>> solve? I'm sure everything you need can > >>> > > > > >>> be > >>> > > > > >>>>> done > >>> > > > > >>>>>>> in > >>> > > > > >>>>>>>> a > >>> > > > > >>>>>>>>>> much > >>> > > > > >>>>>>>>>>>>> more > >>> > > > > >>>>>>>>>>>>>>>> simple > >>> > > > > >>>>>>>>>>>>>>>>>> and > >>> > > > > >>>>>>>>>>>>>>>>>>>> efficient way at the application level. > >>> > > > > >>>>>>>>>>>>>>>>>>>> > >>> > > > > >>>>>>>>>>>>>>>>>>>> Sergi > >>> > > > > >>>>>>>>>>>>>>>>>>>> > >>> > > > > >>>>>>>>>>>>>>>>>>>> 2017-03-14 16:03 GMT+03:00 ALEKSEY > >>> > > > > >>>> KUZNETSOV > >>> > > > > >>>>> < > >>> > > > > >>>>>>>>>>>>>>>>> alkuznetsov...@gmail.com > >>> > > > > >>>>>>>>>>>>>>>>>>> : > >>> > > > > >>>>>>>>>>>>>>>>>>>> > >>> > > > > >>>>>>>>>>>>>>>>>>>>> Why wrong ? You know the better > >>> > > > > >>> solution? > >>> > > > > >>>>>>>>>>>>>>>>>>>>> > >>> > > > > >>>>>>>>>>>>>>>>>>>>> вт, 14 мар. 2017 г. в 15:46, Sergi > >>> > > > > >>>>> Vladykin < > >>> > > > > >>>>>>>>>>>>>>>>>> sergi.vlady...@gmail.com > >>> > > > > >>>>>>>>>>>>>>>>>>>> : > >>> > > > > >>>>>>>>>>>>>>>>>>>>> > >>> > > > > >>>>>>>>>>>>>>>>>>>>>> Just serializing TX object and > >>> > > > > >>>>>> deserializing > >>> > > > > >>>>>>> it > >>> > > > > >>>>>>>>> on > >>> > > > > >>>>>>>>>>>>> another > >>> > > > > >>>>>>>>>>>>>>> node > >>> > > > > >>>>>>>>>>>>>>>>> is > >>> > > > > >>>>>>>>>>>>>>>>>>>>>> meaningless, because other nodes > >>> > > > > >>>>>>> participating > >>> > > > > >>>>>>>> in > >>> > > > > >>>>>>>>>> the > >>> > > > > >>>>>>>>>>>> TX > >>> > > > > >>>>>>>>>>>>>> have > >>> > > > > >>>>>>>>>>>>>>>> to > >>> > > > > >>>>>>>>>>>>>>>>>> know > >>> > > > > >>>>>>>>>>>>>>>>>>>>> about > >>> > > > > >>>>>>>>>>>>>>>>>>>>>> the new coordinator. This will > >>> > > > > >>> require > >>> > > > > >>>>>>> protocol > >>> > > > > >>>>>>>>>>>> changes, > >>> > > > > >>>>>>>>>>>>> we > >>> > > > > >>>>>>>>>>>>>>>>>>> definitely > >>> > > > > >>>>>>>>>>>>>>>>>>>>> will > >>> > > > > >>>>>>>>>>>>>>>>>>>>>> have fault tolerance and > >>> > > > > >> performance > >>> > > > > >>>>>> issues. > >>> > > > > >>>>>>>> IMO > >>> > > > > >>>>>>>>>> the > >>> > > > > >>>>>>>>>>>>> whole > >>> > > > > >>>>>>>>>>>>>>> idea > >>> > > > > >>>>>>>>>>>>>>>>> is > >>> > > > > >>>>>>>>>>>>>>>>>>>> wrong > >>> > > > > >>>>>>>>>>>>>>>>>>>>>> and it makes no sense to waste time > >>> > > > > >>> on > >>> > > > > >>>>> it. > >>> > > > > >>>>>>>>>>>>>>>>>>>>>> > >>> > > > > >>>>>>>>>>>>>>>>>>>>>> Sergi > >>> > > > > >>>>>>>>>>>>>>>>>>>>>> > >>> > > > > >>>>>>>>>>>>>>>>>>>>>> 2017-03-14 10:57 GMT+03:00 ALEKSEY > >>> > > > > >>>>>> KUZNETSOV > >>> > > > > >>>>>>> < > >>> > > > > >>>>>>>>>>>>>>>>>>> alkuznetsov...@gmail.com > >>> > > > > >>>>>>>>>>>>>>>>>>>>> : > >>> > > > > >>>>>>>>>>>>>>>>>>>>>> > >>> > > > > >>>>>>>>>>>>>>>>>>>>>>> IgniteTransactionState > >>> > > > > >>>> implememntation > >>> > > > > >>>>>>>> contains > >>> > > > > >>>>>>>>>>>>>>>> IgniteTxEntry's > >>> > > > > >>>>>>>>>>>>>>>>>>> which > >>> > > > > >>>>>>>>>>>>>>>>>>>>> is > >>> > > > > >>>>>>>>>>>>>>>>>>>>>>> supposed to be transferable > >>> > > > > >>>>>>>>>>>>>>>>>>>>>>> > >>> > > > > >>>>>>>>>>>>>>>>>>>>>>> пн, 13 мар. 2017 г. в 19:32, > >>> > > > > >>> Dmitriy > >>> > > > > >>>>>>>> Setrakyan > >>> > > > > >>>>>>>>> < > >>> > > > > >>>>>>>>>>>>>>>>>>>> dsetrak...@apache.org > >>> > > > > >>>>>>>>>>>>>>>>>>>>>> : > >>> > > > > >>>>>>>>>>>>>>>>>>>>>>> > >>> > > > > >>>>>>>>>>>>>>>>>>>>>>>> It sounds a little scary to me > >>> > > > > >>> that > >>> > > > > >>>>> we > >>> > > > > >>>>>>> are > >>> > > > > >>>>>>>>>>> passing > >>> > > > > >>>>>>>>>>>>>>>>> transaction > >>> > > > > >>>>>>>>>>>>>>>>>>>>> objects > >>> > >>> -- > > *Best Regards,* > > *Kuznetsov Aleksey* >
