Hi, Igniters. I've prepared some benchmarking. Results [1].
And I've prepared the evaluation in the form of diagrams [2]. I hope that helps to interest the community and accelerates a reaction to this improvment :) [1] https://github.com/daradurvs/ignite-compression/tree/master/src/main/resources/result [2] https://drive.google.com/file/d/0B2CeUAOgrHkoMklyZ25YTEdKcEk/view 2017-05-24 9:49 GMT+03:00 Vyacheslav Daradur <daradu...@gmail.com>: > Guys, any thoughts? > > 2017-05-16 13:40 GMT+03:00 Vyacheslav Daradur <daradu...@gmail.com>: > >> Hi guys, >> >> I've prepared the PR to show my idea. >> https://github.com/apache/ignite/pull/1951/files >> >> About querying - I've just copied existing tests and have annotated the >> testing data. >> https://github.com/apache/ignite/pull/1951/files#diff-c19a9d >> f4058141d059bb577e75244764 >> >> It means fields which will be marked by @BinaryCompression will be >> compressed at marshalling via BinaryMarshaller. >> >> This solution has no effect on existing data or project architecture. >> >> I'll be glad to see your thougths. >> >> >> 2017-05-15 19:18 GMT+03:00 Vyacheslav Daradur <daradu...@gmail.com>: >> >>> Dmitriy, >>> >>> I have ready prototype. I want to show it. >>> It is always easier to discuss on example. >>> >>> 2017-05-15 19:02 GMT+03:00 Dmitriy Setrakyan <dsetrak...@apache.org>: >>> >>>> Vyacheslav, >>>> >>>> I think it is a bit premature to provide a PR without getting a >>>> community >>>> consensus on the dev list. Please allow some time for the community to >>>> respond. >>>> >>>> D. >>>> >>>> On Mon, May 15, 2017 at 6:36 AM, Vyacheslav Daradur < >>>> daradu...@gmail.com> >>>> wrote: >>>> >>>> > I created the ticket: https://issues.apache.org/jira >>>> /browse/IGNITE-5226 >>>> > >>>> > I'll prepare a PR with described solution in couple of days. >>>> > >>>> > 2017-05-15 15:05 GMT+03:00 Vyacheslav Daradur <daradu...@gmail.com>: >>>> > >>>> > > Hi, Igniters! >>>> > > >>>> > > Apache 2.0 is released. >>>> > > >>>> > > Let's continue the discussion about a compression design. >>>> > > >>>> > > At the moment, I found only one solution which is compatible with >>>> > querying >>>> > > and indexing, this is per-objects-field compression. >>>> > > Per-fields compression means that metadata (a header) of an object >>>> won't >>>> > > be compressed, only serialized values of an object fields (in bytes >>>> array >>>> > > form) will be compressed. >>>> > > >>>> > > This solution have some contentious issues: >>>> > > - small values, like primitives and short arrays - there isn't >>>> sense to >>>> > > compress them; >>>> > > - there is no possible to use compression with java-predefined >>>> types; >>>> > > >>>> > > We can provide an annotation, @IgniteCompression - for example, >>>> which can >>>> > > be used by users for marking fields to compress. >>>> > > >>>> > > Any thoughts? >>>> > > >>>> > > Maybe someone already have ready design? >>>> > > >>>> > > 2017-04-10 11:06 GMT+03:00 Vyacheslav Daradur <daradu...@gmail.com >>>> >: >>>> > > >>>> > >> Alexey, >>>> > >> >>>> > >> Yes, I've read it. >>>> > >> >>>> > >> Ok, let's discuss about public API design. >>>> > >> >>>> > >> I think we need to add some a configure entity to >>>> CacheConfiguration, >>>> > >> which will contain the Compressor interface implementation and some >>>> > usefull >>>> > >> parameters. >>>> > >> Or maybe to provide a BinaryMarshaller decorator, which will be >>>> compress >>>> > >> data after marshalling. >>>> > >> >>>> > >> >>>> > >> 2017-04-10 10:40 GMT+03:00 Alexey Kuznetsov <akuznet...@apache.org >>>> >: >>>> > >> >>>> > >>> Vyacheslav, >>>> > >>> >>>> > >>> Did you read initial discussion [1] about compression? >>>> > >>> As far as I remember we agreed to add only some "top-level" API in >>>> > order >>>> > >>> to >>>> > >>> provide a way for >>>> > >>> Ignite users to inject some sort of custom compression. >>>> > >>> >>>> > >>> >>>> > >>> [1] >>>> > >>> http://apache-ignite-developers.2346864.n4.nabble.com/Data-c >>>> > >>> ompression-in-Ignite-2-0-td10099.html >>>> > >>> >>>> > >>> On Mon, Apr 10, 2017 at 2:19 PM, daradurvs <daradu...@gmail.com> >>>> > wrote: >>>> > >>> >>>> > >>> > Hi Igniters! >>>> > >>> > >>>> > >>> > I am interested in this task. >>>> > >>> > Provide some kind of pluggable compression SPI support >>>> > >>> > <https://issues.apache.org/jira/browse/IGNITE-3592> >>>> > >>> > >>>> > >>> > I developed a solution on BinaryMarshaller-level, but reviewer >>>> has >>>> > >>> rejected >>>> > >>> > it. >>>> > >>> > >>>> > >>> > Let's continue discussion of task goals and solution design. >>>> > >>> > As I understood that, the main goal of this task is to store >>>> data in >>>> > >>> > compressed form. >>>> > >>> > This is what I need from Ignite as its user. Compression >>>> provides >>>> > >>> economy >>>> > >>> > on >>>> > >>> > servers. >>>> > >>> > We can store more data on same servers at the cost of >>>> increasing CPU >>>> > >>> > utilization. >>>> > >>> > >>>> > >>> > I'm researching a possibility of implementation of compression >>>> at the >>>> > >>> > cache-level. >>>> > >>> > >>>> > >>> > Any thoughts? >>>> > >>> > >>>> > >>> > -- >>>> > >>> > Best regards, >>>> > >>> > Vyacheslav >>>> > >>> > >>>> > >>> > >>>> > >>> > >>>> > >>> > >>>> > >>> > -- >>>> > >>> > View this message in context: http://apache-ignite- >>>> > >>> > developers.2346864.n4.nabble.com/Data-compression-in- >>>> > >>> > Ignite-2-0-tp10099p16317.html >>>> > >>> > Sent from the Apache Ignite Developers mailing list archive at >>>> > >>> Nabble.com. >>>> > >>> > >>>> > >>> >>>> > >>> >>>> > >>> >>>> > >>> -- >>>> > >>> Alexey Kuznetsov >>>> > >>> >>>> > >> >>>> > >> >>>> > >> >>>> > >> -- >>>> > >> Best Regards, Vyacheslav >>>> > >> >>>> > > >>>> > > >>>> > > >>>> > > -- >>>> > > Best Regards, Vyacheslav >>>> > > >>>> > >>>> > >>>> > >>>> > -- >>>> > Best Regards, Vyacheslav >>>> > >>>> >>> >>> >>> >>> -- >>> Best Regards, Vyacheslav >>> >> >> >> >> -- >> Best Regards, Vyacheslav >> > > > > -- > Best Regards, Vyacheslav > -- Best Regards, Vyacheslav
