Hi Igniters! I’m working on the similar feature for my own project. I would like to suggest use in-line compression and write encoded bytes in network channel by bytes array buffer. It allows us avoiding expensive memory allocation. The described design may be implemented in TcpCommunicationSpi level. We can introduce pluggable compressor on TCP level where we will be able to describe our compression strategy, for example, exclude some small messages and many other.
If the community doesn't mind I will file the ticket and will start implementing it. Any thoughts? 2017-11-23 12:06 GMT+03:00 Vladimir Ozerov <voze...@gridgain.com>: > Denis, > > Regarding zipped marshaller - this would be inefficient, because > compression rate will be lower. > > On Thu, Nov 23, 2017 at 1:01 AM, Denis Magda <dma...@apache.org> wrote: > > > Nikita, > > > > Your solution sounds reasonable from the first glance. However, the > > communication layer processes a dozen of small system messages that > should > > be excluded from the compression. Guess, that we will spend more time on > > compressing/decompressing them thus diminishing the positive effect of > the > > compression. > > > > Alexey K., Vladimir O., > > > > What if we create Zip version of the binary marshaller the same way we > > implemented GridClientZipOptimizedMarshaller? > > > > — > > Denis > > > > > On Nov 22, 2017, at 5:36 AM, Alexey Kuznetsov <akuznet...@apache.org> > > wrote: > > > > > > I think it is very useful feature. > > > I also have experience when server nodes connected via fast network. > > > But client nodes via very slow network. > > > > > > I implemeted GridClientZipOptimizedMarshaller and that solved my > issue. > > > But this marshaller works only with old > > > and org.apache.ignite.internal.client.GridClient and has a lot of > > > limitations. > > > But compression was about 6-20x times. > > > > > > We need a solution for Ignite 2.x and client nodes. > > > > > > > > > On Wed, Nov 22, 2017 at 7:48 PM, Nikita Amelchev <nsamelc...@gmail.com > > > > > wrote: > > > > > >> Hello, Igniters! > > >> > > >> I think it is a useful feature. I suggest to implement it to > > communication > > >> SPI like SSL encryption implemented. I have experience with this > feature > > >> and I can try to develop it. > > >> > > >> 2017-11-22 12:01 GMT+03:00 Alexey Kukushkin < > kukushkinale...@gmail.com > > >: > > >> > > >>> Forwarding to DEV list: Ignite developers, could you please share > your > > >>> thoughts on how hard it is to extend Ignite to compress data on the > > >>> network. > > >>> > > >>> On Wed, Nov 22, 2017 at 10:04 AM, Gordon Reid (Nine Mile) < > > >>> gordon.r...@ninemilefinancial.com> wrote: > > >>> > > >>>> Hi Igniters, > > >>>> > > >>>> > > >>>> > > >>>> I see there is a lot of discussion in certain threads about > > >> compression. > > >>>> This seems to have diverged into conversations about object versus > > >> field > > >>>> compression, and even throwing encryption into the mix. For my use > > >> case, > > >>> I > > >>>> am not interested in compressing the cache stored in memory, I have > > >>> plenty > > >>>> of memory for my application. What I don’t have is a good network. I > > >>> have a > > >>>> high latency, low bandwidth network between my C# ignite client and > my > > >>> Java > > >>>> ignite server. I only want to compress data when it is sent over the > > >>>> network to remote nodes. It should be stored in the local memory > > >>>> uncompressed. How can we achive this? Can the TcpCommunicationSpi > > >> support > > >>>> compression? > > >>>> > > >>>> > > >>>> > > >>>> Thanks, > > >>>> > > >>>> Gordon. > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > >>>> This email and any attachments are proprietary & confidential and > are > > >>>> intended solely for the use of the individuals to whom it is > > addressed. > > >>> Any > > >>>> views or opinions expressed are solely for those of the author and > do > > >> not > > >>>> necessarily reflect those of Nine Mile Financial Pty. Limited. If > you > > >>> have > > >>>> received this email in error, please let us know immediately by > reply > > >>> email > > >>>> and delete from your system. Nine Mile Financial Pty. Limited. ABN: > > 346 > > >>>> 1349 0252 > > >>>> > > >>> > > >>> > > >>> > > >>> -- > > >>> Best regards, > > >>> Alexey > > >>> > > >> > > >> > > >> > > >> -- > > >> Best wishes, > > >> Amelchev Nikita > > >> > > > > > > > > > > > > -- > > > Alexey Kuznetsov > > > > > -- Best wishes, Amelchev Nikita
