Few thoughts about LOCAL caches on thin client: 1. If partition awareness is disabled: a. Inconsistent behaviour if node to which client is connected goes down. 2. If partition awareness is enabled: a. For Java and .NET -- same as 1a b. For C++ and python -- use random routing for caches that are not PARTITIONED, so inconsistent behaviour from the beginning.
So I suppose we should ban creation of LOCAL caches from thin client in 2.12 (fail attempt to create such caches in ClientRequestHandler вт, 14 сент. 2021 г. в 11:31, Ivan Daschinsky <ivanda...@gmail.com>: > >> Unsupported operation exception. > Binary protocol doesn't have a concept of exception, only error status and > message, but it is just a remark > I suppose that response with error status and message is ok, but may be > others have different opinion? > > >> Removal should happen at 2.13. > A few thin clients are released separately. I suppose that it is better to > remove this feature from pyignite a little bit earlier. > > вт, 14 сент. 2021 г. в 11:21, Anton Vinogradov <a...@apache.org>: > >> > 1. What is expected behaviour if an old thin client requests creation of >> > LOCAL cache on the newest ignite cluster? >> Unsupported operation exception. >> >> > 2. Should we completely remove LOCAL caches support in thin clients >> (i.e. >> pyignite) before 2.13 release? >> Removal should happen at 2.13. >> >> On Tue, Sep 14, 2021 at 10:30 AM Ivan Daschinsky <ivanda...@gmail.com> >> wrote: >> >> > >> 2. 2.13 - complete removal LOCAL caches from codebase. >> > Let's discuss this step with more details. >> > 1. What is expected behaviour if an old thin client requests creation of >> > LOCAL cache on the newest ignite cluster? >> > 2. Should we completely remove LOCAL caches support in thin clients >> (i.e. >> > pyignite) before 2.13 release? >> > >> > вт, 14 сент. 2021 г. в 10:11, Nikolay Izhikov <nizhi...@apache.org>: >> > >> > > I proposed the following plan: >> > > >> > > 1. 2.12 - deprecation of LOCAL caches. >> > > 2. 2.13 - complete removal LOCAL caches from codebase. >> > > >> > > > 13 сент. 2021 г., в 13:30, Ivan Daschinsky <ivanda...@gmail.com> >> > > написал(а): >> > > > >> > > > I personally support deprecation, but we should at least have a >> plan. >> > > > I suppose that putting annotations and removing documentation are >> not >> > > > enough. >> > > > >> > > > >> > > > пн, 13 сент. 2021 г. в 13:22, Maxim Muzafarov <mmu...@apache.org>: >> > > > >> > > >> Ivan, >> > > >> >> > > >> I don't think we can remove LOCAL caches at the nearest time, so >> there >> > > >> is no plan for that. I can only imagine a single release that will >> > > >> contain all the breaking changes we want to apply in 2.x version. >> > > >> >> > > >> My point here is only about deprecation: >> > > >> - there are a lot of motivation points to remove written in this >> > thread; >> > > >> - I always hear from the support team that they do not recommend >> using >> > > >> local caches; >> > > >> - I haven't seen any bugs fixed for a long time for local caches >> > > >> (suppose that we are not maintaining them); >> > > >> >> > > >> I just want to make sure that all these points are reflected in the >> > > >> code base, so propose to mark them as deprecated. >> > > >> >> > > >> On Mon, 13 Sept 2021 at 11:29, Ivan Daschinsky < >> ivanda...@gmail.com> >> > > >> wrote: >> > > >>> >> > > >>> Hi, Maxim. And what is the plan of removing this functionality? >> And I >> > > >> also >> > > >>> have some questions regarding deprecation in binary protocol >> > > >>> >> > > >>> Currently thin client binary protocol >> > > >>> 1. Does support LOCAL caches >> > > >>> 2. Does not support node filters. >> > > >>> >> > > >>> I can hardly imagine the usefulness of this feature on thin >> clients, >> > > >>> especially with partition awareness, but nevertheless. >> > > >>> What is expected behaviour if this feature is removed from newest >> > > version >> > > >>> of Apache Ignite server and and and old client is requesting >> > > >>> creation of LOCAL cache? >> > > >>> >> > > >>> вс, 12 сент. 2021 г. в 15:10, Maxim Muzafarov <mmu...@apache.org >> >: >> > > >>> >> > > >>>> Folks, >> > > >>>> >> > > >>>> Let's get back to the discussion of obsolete LOCAL caches since a >> > lot >> > > >>>> of time has passed since the last discussion. >> > > >>>> I've created an issue [1] for deprecation. Let's deprecate them >> at >> > > >>>> least at the next 2.12 release. >> > > >>>> >> > > >>>> WDYT? >> > > >>>> >> > > >>>> >> > > >>>> [1] https://issues.apache.org/jira/browse/IGNITE-15499 >> > > >>>> >> > > >>>> On Fri, 27 Jul 2018 at 20:59, Valentin Kulichenko >> > > >>>> <valentin.kuliche...@gmail.com> wrote: >> > > >>>>> >> > > >>>>> Guys, >> > > >>>>> >> > > >>>>> Use cases for local caches are rare, but they definitely exist. >> I >> > > >> don't >> > > >>>>> think it's a very good idea to deprecate this functionality at >> this >> > > >>>> point. >> > > >>>>> >> > > >>>>> At the same point, it's obviously not the most critical part of >> the >> > > >>>>> product, so maintaining the whole separate implementation for >> it is >> > > >>>>> probably an overkill. We had exact same story with replicated >> > caches >> > > >> btw >> > > >>>> - >> > > >>>>> they were implemented separately which caused maintainability >> > > >> issues, and >> > > >>>>> we ended up removing this separate implementation. If we have >> the >> > > >> same >> > > >>>>> situation here, let's use the same solution. >> > > >>>>> >> > > >>>>> -Val >> > > >>>>> >> > > >>>>> On Fri, Jul 27, 2018 at 3:05 AM Dmitry Pavlov < >> > dpavlov....@gmail.com >> > > >>> >> > > >>>> wrote: >> > > >>>>> >> > > >>>>>> Hi Dmitriy, >> > > >>>>>> >> > > >>>>>> I would like to stress this: I'm not saying local cache it >> > > >> useless. I'm >> > > >>>>>> supposing it is not used widely. I want to figure out if I'm >> > > >> mistaking. >> > > >>>>>> >> > > >>>>>> All folks involved into user list says it is not used, so why >> not >> > > >> to >> > > >>>>>> deprecate? If we make a mistake, somebody will come to user >> list >> > > >> and >> > > >>>> say, >> > > >>>>>> 'Hey, why did you deprecate this, it is used for... in my >> project' >> > > >>>>>> >> > > >>>>>> Being very experienced Igniter you probably know real life >> usage >> > > >>>> examples. >> > > >>>>>> And I appreciate if you or somebody else in community could >> share >> > > >> it. >> > > >>>>>> >> > > >>>>>> Sincerely, >> > > >>>>>> Dmitriy Pavlov >> > > >>>>>> >> > > >>>>>> пт, 27 июл. 2018 г. в 1:04, Dmitriy Setrakyan < >> > > >> dsetrak...@apache.org>: >> > > >>>>>> >> > > >>>>>>> Guys, >> > > >>>>>>> >> > > >>>>>>> I just want to make sure we are all on the same page. The main >> > > >> use >> > > >>>> case >> > > >>>>>> for >> > > >>>>>>> LOCAL caches is to have a local hash map querable with SQL and >> > > >>>>>>> automatically persisted to a 3rd party DB. >> > > >>>>>>> >> > > >>>>>>> I want to discourage people from saying "nobody needs some >> > > >> feature". >> > > >>>> None >> > > >>>>>>> of the people in this discussion are users of any features - >> we >> > > >> are >> > > >>>> all >> > > >>>>>>> developers of the features. Instead of guessing whether to >> > > >> deprecate >> > > >>>>>>> something or not, I would actually see if it is even worth a >> > > >>>> discussion. >> > > >>>>>>> How much effort is required to fix the bug found in the LOCAL >> > > >> cache? >> > > >>>>>>> >> > > >>>>>>> D. >> > > >>>>>>> >> > > >>>>>>> On Thu, Jul 26, 2018 at 12:19 PM, Dmitry Pavlov < >> > > >>>> dpavlov....@gmail.com> >> > > >>>>>>> wrote: >> > > >>>>>>> >> > > >>>>>>>> Hi Alexey, >> > > >>>>>>>> >> > > >>>>>>>> There is nothing to be sorry about :) Сommunity appreciates >> an >> > > >>>>>>> alternative >> > > >>>>>>>> vision, this allows us to make as informed decisions as it >> > > >>>> possible. >> > > >>>>>>>> >> > > >>>>>>>> Thank you for finding this fact, it is very interesting. >> > > >>>>>>>> >> > > >>>>>>>> I'm not sure all these examples were prepared by experienced >> > > >> Ignite >> > > >>>>>>> users. >> > > >>>>>>>> So idea of deprecation may have one more argument. >> Deprecation >> > > >> will >> > > >>>>>> help >> > > >>>>>>> us >> > > >>>>>>>> to inform users about LOCAL cache: Probably local cache is >> not >> > > >> what >> > > >>>>>> they >> > > >>>>>>>> need. >> > > >>>>>>>> >> > > >>>>>>>> Sincerely, >> > > >>>>>>>> Dmitriy Pavlov >> > > >>>>>>>> >> > > >>>>>>>> чт, 26 июл. 2018 г. в 16:57, Alexey Zinoviev < >> > > >>>> zaleslaw....@gmail.com>: >> > > >>>>>>>> >> > > >>>>>>>>> Sorry, guys, I'll put my 1 cent >> > > >>>>>>>>> >> > > >>>>>>>>> I'd like this idea "Implement LOCAL caches as PARTITIONED >> > > >> caches >> > > >>>>>> over >> > > >>>>>>>> the >> > > >>>>>>>>> local node." >> > > >>>>>>>>> It make sense for examples/testing in pseudo-distributed >> mode >> > > >>>> and so >> > > >>>>>>> far. >> > > >>>>>>>>> >> > > >>>>>>>>> But I think that the deprecation based on user-list mentions >> > > >> is a >> > > >>>>>> wrong >> > > >>>>>>>>> way. Please look here >> > > >>>>>>>>> >> > > >>>>>> >> > > >> >> > https://github.com/search?q=%22CacheMode.LOCAL%22+%26+ignite&type=Code >> > > >>>>>>>>> There a lot of hello world examples with LOCAL mode. >> > > >>>>>>>>> >> > > >>>>>>>>> And of course, we can ask about that on user-list, not here, >> > > >> to >> > > >>>> vote >> > > >>>>>>> for >> > > >>>>>>>>> the deprecation like this. >> > > >>>>>>>>> >> > > >>>>>>>>> 2018-07-26 11:23 GMT+03:00 Vladimir Ozerov < >> > > >> voze...@gridgain.com >> > > >>>>> : >> > > >>>>>>>>> >> > > >>>>>>>>>> I meant LOCAL + non-LOCAL transactions of course. >> > > >>>>>>>>>> >> > > >>>>>>>>>> On Wed, Jul 25, 2018 at 10:42 PM Dmitriy Setrakyan < >> > > >>>>>>>>> dsetrak...@apache.org> >> > > >>>>>>>>>> wrote: >> > > >>>>>>>>>> >> > > >>>>>>>>>>> Vladimir, >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> Are you suggesting that a user cannot span more than one >> > > >>>> local >> > > >>>>>>> cache >> > > >>>>>>>>> in a >> > > >>>>>>>>>>> cross cache LOCAL transactions. This is extremely >> > > >> surprising >> > > >>>> to >> > > >>>>>> me, >> > > >>>>>>>> as >> > > >>>>>>>>> it >> > > >>>>>>>>>>> would require almost no effort to support it. As far as >> > > >>>> mixing >> > > >>>>>> the >> > > >>>>>>>>> local >> > > >>>>>>>>>>> caches with distributed caches, then I agree, cross-cache >> > > >>>>>>>> transactions >> > > >>>>>>>>> do >> > > >>>>>>>>>>> not make sense. >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> I am not sure why deprecating local caches has become a >> > > >>>> pressing >> > > >>>>>>>>> issue. I >> > > >>>>>>>>>>> can see that there are a few bugs, but why not just fix >> > > >> them >> > > >>>> and >> > > >>>>>>> move >> > > >>>>>>>>> on? >> > > >>>>>>>>>>> Can someone explain why supporting LOCAL caches is such a >> > > >>>> burden? >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> Having said that, I am not completely opposed to >> > > >> deprecating >> > > >>>>>> LOCAL >> > > >>>>>>>>>> caches. >> > > >>>>>>>>>>> I just want to know why. >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> D. >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> On Wed, Jul 25, 2018 at 10:55 AM, Vladimir Ozerov < >> > > >>>>>>>>> voze...@gridgain.com> >> > > >>>>>>>>>>> wrote: >> > > >>>>>>>>>>> >> > > >>>>>>>>>>>> Dima, >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> LOCAL cache adds very little value to the product. It >> > > >>>> doesn't >> > > >>>>>>>> support >> > > >>>>>>>>>>>> cross-cache transactions, consumes a lot of memory, >> > > >> much >> > > >>>> slower >> > > >>>>>>>> than >> > > >>>>>>>>>> any >> > > >>>>>>>>>>>> widely-used concurrent hash map. Let's go the same way >> > > >> as >> > > >>>> Java >> > > >>>>>> - >> > > >>>>>>>> mark >> > > >>>>>>>>>>> LOCAL >> > > >>>>>>>>>>>> cache as "deprecated for removal", and then remove it >> > > >> in >> > > >>>> 3.0. >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> On Wed, Jul 25, 2018 at 12:10 PM Dmitrii Ryabov < >> > > >>>>>>>>> somefire...@gmail.com >> > > >>>>>>>>>>> >> > > >>>>>>>>>>>> wrote: >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>>> +1 to make LOCAL as filtered PARTITIONED cache. I >> > > >> think >> > > >>>> it >> > > >>>>>>> would >> > > >>>>>>>> be >> > > >>>>>>>>>>> much >> > > >>>>>>>>>>>>> easier and faster than fixing all bugs. >> > > >>>>>>>>>>>>> >> > > >>>>>>>>>>>>> 2018-07-25 11:51 GMT+03:00 Dmitriy Setrakyan < >> > > >>>>>>>>> dsetrak...@apache.org >> > > >>>>>>>>>>> : >> > > >>>>>>>>>>>>> >> > > >>>>>>>>>>>>>> I would stay away from deprecating such huge >> > > >> pieces as >> > > >>>> a >> > > >>>>>>> whole >> > > >>>>>>>>>> LOCAL >> > > >>>>>>>>>>>>> cache. >> > > >>>>>>>>>>>>>> In retrospect, we should probably not even have >> > > >> LOCAL >> > > >>>>>> caches, >> > > >>>>>>>> but >> > > >>>>>>>>>>> now I >> > > >>>>>>>>>>>>> am >> > > >>>>>>>>>>>>>> certain that it is used by many users. >> > > >>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>> I would do one of the following, whichever one is >> > > >>>> easier: >> > > >>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>> - Fix the issues found with LOCAL caches, >> > > >> including >> > > >>>>>>>>> persistence >> > > >>>>>>>>>>>>> support >> > > >>>>>>>>>>>>>> - Implement LOCAL caches as PARTITIONED caches >> > > >> over >> > > >>>> the >> > > >>>>>>>> local >> > > >>>>>>>>>>> node. >> > > >>>>>>>>>>>> In >> > > >>>>>>>>>>>>>> this case, we would have to hide any >> > > >>>>>> distribution-related >> > > >>>>>>>>> config >> > > >>>>>>>>>>>> from >> > > >>>>>>>>>>>>>> users, like affinity function, for example. >> > > >>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>> D. >> > > >>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>> On Wed, Jul 25, 2018 at 9:05 AM, Valentin >> > > >> Kulichenko < >> > > >>>>>>>>>>>>>> valentin.kuliche...@gmail.com> wrote: >> > > >>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>> It sounds like the main drawback of LOCAL cache >> > > >> is >> > > >>>> that >> > > >>>>>>> it's >> > > >>>>>>>>>>>>> implemented >> > > >>>>>>>>>>>>>>> separately and therefore has to be maintained >> > > >>>> separately. >> > > >>>>>>> If >> > > >>>>>>>>>> that's >> > > >>>>>>>>>>>> the >> > > >>>>>>>>>>>>>>> only issue, why not keep LOCAL cache mode on >> > > >> public >> > > >>>> API, >> > > >>>>>>> but >> > > >>>>>>>>>>>> implement >> > > >>>>>>>>>>>>> it >> > > >>>>>>>>>>>>>>> as a PARTITIONED cache with a node filter >> > > >> forcefully >> > > >>>> set? >> > > >>>>>>>>> That's >> > > >>>>>>>>>>>>> similar >> > > >>>>>>>>>>>>>> to >> > > >>>>>>>>>>>>>>> what we do with REPLICATED caches which are >> > > >> actually >> > > >>>>>>>>> PARTITIONED >> > > >>>>>>>>>>> with >> > > >>>>>>>>>>>>>>> infinite number of backups. >> > > >>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>> This way we fix the issues described by Stan and >> > > >>>> don't >> > > >>>>>> have >> > > >>>>>>>> to >> > > >>>>>>>>>>>>> deprecate >> > > >>>>>>>>>>>>>>> anything. >> > > >>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>> -Val >> > > >>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>> On Wed, Jul 25, 2018 at 12:53 AM Stanislav >> > > >> Lukyanov < >> > > >>>>>>>>>>>>>>> stanlukya...@gmail.com> >> > > >>>>>>>>>>>>>>> wrote: >> > > >>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>> Hi Igniters, >> > > >>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>> I’d like to start a discussion about the >> > > >>>> deprecation of >> > > >>>>>>> the >> > > >>>>>>>>>> LOCAL >> > > >>>>>>>>>>>>>> caches. >> > > >>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>> LOCAL caches are an edge-case functionality >> > > >>>>>>>>>>>>>>>> I haven’t done any formal analysis, but from my >> > > >>>>>>> experience >> > > >>>>>>>>>> LOCAL >> > > >>>>>>>>>>>>> caches >> > > >>>>>>>>>>>>>>>> are needed very rarely, if ever. >> > > >>>>>>>>>>>>>>>> I think most usages of LOCAL caches I’ve seen >> > > >> were >> > > >>>>>>> misuses: >> > > >>>>>>>>> the >> > > >>>>>>>>>>>> users >> > > >>>>>>>>>>>>>>>> actually needed a simple HashMap, or an actual >> > > >>>>>>> PARTITIONED >> > > >>>>>>>>>> cache. >> > > >>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>> LOCAL caches are easy to implement on top of >> > > >>>>>> PARTITIONED >> > > >>>>>>>>>>>>>>>> If one requires a LOCAL cache (which is itself >> > > >>>>>>>> questionable, >> > > >>>>>>>>> as >> > > >>>>>>>>>>>>>> discussed >> > > >>>>>>>>>>>>>>>> above) it is quite easy to implement one on >> > > >> top of >> > > >>>>>>>>> PARTITIONED >> > > >>>>>>>>>>>> cache. >> > > >>>>>>>>>>>>>>>> A node filter of form `node -> node.id >> > > >>>>>>>>> ().equals(localNodeId)` >> > > >>>>>>>>>> is >> > > >>>>>>>>>>>>>> enough >> > > >>>>>>>>>>>>>>>> to make the cache to be stored on the node that >> > > >>>> created >> > > >>>>>>> it. >> > > >>>>>>>>>>>>>>>> Locality of access to the cache (i.e. making it >> > > >>>>>>> unavailable >> > > >>>>>>>>>> from >> > > >>>>>>>>>>>>> other >> > > >>>>>>>>>>>>>>>> nodes) can be achieved on the application >> > > >> level. >> > > >>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>> LOCAL caches are hard to maintain >> > > >>>>>>>>>>>>>>>> A quick look at the open issues mentioning >> > > >> “local >> > > >>>>>> cache” >> > > >>>>>>>>>> suggests >> > > >>>>>>>>>>>>> that >> > > >>>>>>>>>>>>>>>> this is a corner case for implementation of >> > > >> many >> > > >>>> Ignite >> > > >>>>>>>>>> features: >> > > >>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>> >> > > >>>>>> https://issues.apache.org/jira/issues/?jql=text%20~%20% >> > > >>>>>>>>>>>>>>> >> > > >> 22local%20cache%22%20and%20%20project%20%3D%20IGNITE% >> > > >>>>>>>>>>>>>>> 20and%20status%20%3D%20open >> > > >>>>>>>>>>>>>>>> In particular, a recent SO question brought up >> > > >> the >> > > >>>> fact >> > > >>>>>>>> that >> > > >>>>>>>>>>> LOCAL >> > > >>>>>>>>>>>>>> caches >> > > >>>>>>>>>>>>>>>> don’t support native persistence: >> > > >>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>> >> > > >>>> https://stackoverflow.com/questions/51511892/how-to- >> > > >>>>>>>>>>>>>>> >> > > >> configure-persistent-storage-for-apache-ignite-cache >> > > >>>>>>>>>>>>>>>> Having to ask ourselves “how does it play with >> > > >>>> LOCAL >> > > >>>>>>>> caches” >> > > >>>>>>>>>>> every >> > > >>>>>>>>>>>>> time >> > > >>>>>>>>>>>>>>> we >> > > >>>>>>>>>>>>>>>> write any code in Ignite seems way to much for >> > > >> the >> > > >>>>>>> benefits >> > > >>>>>>>>> we >> > > >>>>>>>>>>> gain >> > > >>>>>>>>>>>>>> from >> > > >>>>>>>>>>>>>>> it. >> > > >>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>> Proposal >> > > >>>>>>>>>>>>>>>> Let’s deprecate LOCAL caches in 2.x and remove >> > > >>>> them in >> > > >>>>>>> 3.0. >> > > >>>>>>>>>>>>>>>> As a part of deprecation let’s do the >> > > >> following: >> > > >>>>>>>>>>>>>>>> - Put @Deprecated on the CacheMode.LOCAL >> > > >>>>>>>>>>>>>>>> - Print a warning every time a LOCAL cache is >> > > >>>> created >> > > >>>>>>>>>>>>>>>> - Remove all mentions of LOCAL caches from >> > > >>>> readme.io, >> > > >>>>>> if >> > > >>>>>>>>> any, >> > > >>>>>>>>>>>> except >> > > >>>>>>>>>>>>>> for >> > > >>>>>>>>>>>>>>>> the page about cache modes >> > > >>>>>>>>>>>>>>>> - On the page about cache modes explain that >> > > >> LOCAL >> > > >>>> is >> > > >>>>>>>>>> deprecated >> > > >>>>>>>>>>>> and >> > > >>>>>>>>>>>>>> can >> > > >>>>>>>>>>>>>>>> be replaced with a PARTITIONED cache with a >> > > >> node >> > > >>>> filter >> > > >>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>> Thanks, >> > > >>>>>>>>>>>>>>>> Stan >> > > >>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>> >> > > >>>>>>>>>>>>> >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>> >> > > >>>>>>>>>> >> > > >>>>>>>>> >> > > >>>>>>>> >> > > >>>>>>> >> > > >>>>>> >> > > >>>> >> > > >>> >> > > >>> >> > > >>> -- >> > > >>> Sincerely yours, Ivan Daschinskiy >> > > >> >> > > > >> > > > >> > > > -- >> > > > Sincerely yours, Ivan Daschinskiy >> > > >> > > >> > >> > -- >> > Sincerely yours, Ivan Daschinskiy >> > >> > > > -- > Sincerely yours, Ivan Daschinskiy > -- Sincerely yours, Ivan Daschinskiy
