Hi Igniters!
I'd like to continue developing and discussing about compression in Ignite.
Vladimir, could you propose a design of compression feature in Ignite, that
suits you?
2017-06-15 16:13 GMT+03:00 Vyacheslav Daradur <daradu...@gmail.com>:
> Hi Igniters.
>
> Vladimir, I want to propose another design of an implementation of the
> per-field compression.
>
> 1) We will add new step in the method prepareForCache (for example) of
> CacheObject, or in GridCacheMapEntry.
>
> At the step, after marshalling of an object, we will compress fields of
> the object which described in advance.
> User will describe class fields which he wants to compess in an another
> entity like Metadata.
>
> For compression, we will introduce another entity, for example
> CompressionProcessor, which will work with bytes array (marshalled object).
> The entity will read bytes array of described fields, compress it and
> rewrite binary representation of the whole object.
> After processing the object will be put in the cache.
>
> In this case design not to relate to binary infrastructure.
> But there is big overhead to heap-memory for the buffer.
>
> 2) Another solution is to compress bytes array of whole object on copying
> to off-heap.
> But, in this case I don't understand yet, how to provide support of
> querying and indexing.
>
>
> 2017-06-09 11:21 GMT+03:00 Sergey Kozlov <skoz...@gridgain.com>:
>
>> Hi
>>
>> * "Per-field compression" is applicable for huge BLOB fields and will
>> impose the restrictions like unable ot index such fields, slower getting
>> data, potential OOM issues if compression ration is too high.
>> But for some cases it makes sense
>>
>> On Fri, Jun 9, 2017 at 11:11 AM, Антон Чураев <churaev...@gmail.com>
>> wrote:
>>
>> > Seems that Dmitry is referring to transparent data encryption. It is
>> used
>> > throughout the whale database industry.
>> >
>> > 2017-06-09 10:50 GMT+03:00 Vladimir Ozerov <voze...@gridgain.com>:
>> >
>> > > Dima,
>> > >
>> > > Encryption of certain fields is as bad as compression. First, it is a
>> > huge
>> > > change, which makes already complex binary protocol even more complex.
>> > > Second, it have to be ported to CPP, .NET platforms, as well as to
>> JDBC
>> > and
>> > > ODBC.
>> > > Last, but the most important - this is not our headache to encrypt
>> > > sensitive data. This is user responsibility. Nobody in a sane mind
>> will
>> > > store passwords in plain form. Instead, user should encrypt it on his
>> > own,
>> > > choosing proper encryption parameters - algorithms, key lengths,
>> salts,
>> > > etc.. How are you going to expose this in API or configuration?
>> > >
>> > > We should not implement data encryption on binary level, this is out
>> of
>> > > question. Encryption should be implemented on application level (user
>> > > efforts), transport layer (SSL - we already have it), and possibly on
>> > > disk-level (there are tools for this already).
>> > >
>> > >
>> > > On Fri, Jun 9, 2017 at 9:06 AM, Vyacheslav Daradur <
>> daradu...@gmail.com>
>> > > wrote:
>> > >
>> > > > >> which is much less useful.
>> > > > I note, in some cases there is profit more than twice per size of an
>> > > > object.
>> > > >
>> > > > >> Would it be possible to change your implementation to handle the
>> > > > encryption instead?
>> > > > Yes, of cource, there's not much difference between compression and
>> > > > encryption, including in my implementation of per-field-compression.
>> > > >
>> > > > 2017-06-09 8:55 GMT+03:00 Dmitriy Setrakyan <dsetrak...@apache.org
>> >:
>> > > >
>> > > > > Vyacheslav,
>> > > > >
>> > > > > When this feature started out as data compression in Ignite, it
>> > sounded
>> > > > > very useful. Now it is unfolding as a per-field compression,
>> which is
>> > > > much
>> > > > > less useful. In fact, it is questionable whether it is useful at
>> all.
>> > > The
>> > > > > fact that this feature is implemented does not make it mandatory
>> for
>> > > the
>> > > > > community to accept it.
>> > > > >
>> > > > > However, as I mentioned before, per-field encryption is very
>> useful,
>> > as
>> > > > it
>> > > > > would allow users automatically encrypt certain sensitive fields,
>> > like
>> > > > > passwords, credit card numbers, etc. There is not much conceptual
>> > > > > difference between compressing a field vs encrypting a field.
>> Would
>> > it
>> > > be
>> > > > > possible to change your implementation to handle the encryption
>> > > instead?
>> > > > >
>> > > > > D.
>> > > > >
>> > > > > On Thu, Jun 8, 2017 at 10:42 PM, Vyacheslav Daradur <
>> > > daradu...@gmail.com
>> > > > >
>> > > > > wrote:
>> > > > >
>> > > > > > Guys, I want to be clear:
>> > > > > > * "Per-field compression" design is the result of a research of
>> the
>> > > > > binary
>> > > > > > infrastructure of Ignite and some other its places (querying,
>> > > indexing,
>> > > > > > etc.)
>> > > > > > * Full-compression of object will be more effective, but in this
>> > case
>> > > > > there
>> > > > > > is no capability with querying and indexing (or there is large
>> > > overhead
>> > > > > by
>> > > > > > way of decompressing of full object (or caches pages) on demand)
>> > > > > > * "Per-field compression" is a one of ways to implement the
>> > > compression
>> > > > > > feature
>> > > > > >
>> > > > > > I'm new to Ignite also I can be mistaken in some things.
>> > > > > > Last 3-4 month I've tryed to start dicussion about a design, but
>> > > nobody
>> > > > > > answers nothing (except Dmitry and Valentin who was interested
>> how
>> > it
>> > > > > > works).
>> > > > > > But I understand that this is community and nobody is obliged to
>> > > > anybody.
>> > > > > >
>> > > > > > There are strong Ignite experts.
>> > > > > > If they can help me and community with a design of the
>> compression
>> > > > > feature
>> > > > > > it will be great.
>> > > > > > At the moment I have a desire and time to be engaged in
>> development
>> > > of
>> > > > > > compression feature in Ignite.
>> > > > > > Let's use this opportunity :)
>> > > > > >
>> > > > > > 2017-06-09 5:36 GMT+03:00 Dmitriy Setrakyan <
>> dsetrak...@apache.org
>> > >:
>> > > > > >
>> > > > > > > Igniters,
>> > > > > > >
>> > > > > > > I have never seen a single Ignite user asking about
>> compressing a
>> > > > > single
>> > > > > > > field. However, we have had requests to secure certain fields,
>> > e.g.
>> > > > > > > passwords.
>> > > > > > >
>> > > > > > > I personally do not think per-field compression is needed,
>> unless
>> > > we
>> > > > > can
>> > > > > > > point out some concrete real life use cases.
>> > > > > > >
>> > > > > > > D.
>> > > > > > >
>> > > > > > > On Thu, Jun 8, 2017 at 3:42 AM, Vyacheslav Daradur <
>> > > > > daradu...@gmail.com>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > Anton,
>> > > > > > > >
>> > > > > > > > >> I thought that if there will storing compressed data in
>> the
>> > > > > memory,
>> > > > > > > data
>> > > > > > > > >> will transmit over wire in compression too. Is it right?
>> > > > > > > >
>> > > > > > > > In per-field compression case - yes.
>> > > > > > > >
>> > > > > > > > 2017-06-08 13:36 GMT+03:00 Антон Чураев <
>> churaev...@gmail.com
>> > >:
>> > > > > > > >
>> > > > > > > > > Guys, could you please help me.
>> > > > > > > > > I thought that if there will storing compressed data in
>> the
>> > > > memory,
>> > > > > > > data
>> > > > > > > > > will transmit over wire in compression too. Is it right?
>> > > > > > > > >
>> > > > > > > > > 2017-06-08 13:30 GMT+03:00 Vyacheslav Daradur <
>> > > > daradu...@gmail.com
>> > > > > >:
>> > > > > > > > >
>> > > > > > > > > > Vladimir,
>> > > > > > > > > >
>> > > > > > > > > > The main problem which I'am trying to solve is storing
>> data
>> > > in
>> > > > > > memory
>> > > > > > > > in
>> > > > > > > > > a
>> > > > > > > > > > compression form via Ignite.
>> > > > > > > > > > The main goal is using memory more effectivelly.
>> > > > > > > > > >
>> > > > > > > > > > >> here the much simpler step would be to full
>> > > > > > > > > > compression on per-cache basis rather than dealing with
>> > > > > per-fields
>> > > > > > > > case.
>> > > > > > > > > >
>> > > > > > > > > > Please explain your idea. Compess data by memory-page?
>> > > > > > > > > > Is it compatible with quering and indexing?
>> > > > > > > > > >
>> > > > > > > > > > >> In the end, if user would like to compress particular
>> > > field,
>> > > > > he
>> > > > > > > can
>> > > > > > > > > > always to it on his own
>> > > > > > > > > > I think we mustn't think in this way, if user need
>> > something
>> > > he
>> > > > > > > trying
>> > > > > > > > to
>> > > > > > > > > > choose a tool which has this feature OOTB.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > 2017-06-08 12:53 GMT+03:00 Vladimir Ozerov <
>> > > > voze...@gridgain.com
>> > > > > >:
>> > > > > > > > > >
>> > > > > > > > > > > Igniters,
>> > > > > > > > > > >
>> > > > > > > > > > > Honestly I still do not see how to apply it gracefully
>> > this
>> > > > > > feature
>> > > > > > > > ti
>> > > > > > > > > > > Ignite. And overall approach to compress only
>> particular
>> > > > fields
>> > > > > > > looks
>> > > > > > > > > > > overcomplicated to me. Remember, that our main use
>> case
>> > is
>> > > an
>> > > > > > > > > application
>> > > > > > > > > > > without classes on the server. It means that any kind
>> of
>> > > > > > > annotations
>> > > > > > > > > are
>> > > > > > > > > > > inapplicable. To be more precise: proper API should be
>> > > > > > implemented
>> > > > > > > to
>> > > > > > > > > > > handle no-class case (e.g. how would build such an
>> object
>> > > > > through
>> > > > > > > > > > > BinaryBuilder without a class?), and only then add
>> > > > annotations
>> > > > > as
>> > > > > > > > > > > convenient addition to more basic API.
>> > > > > > > > > > >
>> > > > > > > > > > > It seems to me that full implementation, which takes
>> in
>> > > count
>> > > > > > > proper
>> > > > > > > > > > > "classless" API, changes to binary metadata to reflect
>> > > > > compressed
>> > > > > > > > > fields,
>> > > > > > > > > > > changes to SQL, changes to binary protocol, and
>> porting
>> > to
>> > > > .NET
>> > > > > > and
>> > > > > > > > > CPP,
>> > > > > > > > > > > will yield very complex solution with little value to
>> the
>> > > > > > product.
>> > > > > > > > > > >
>> > > > > > > > > > > Instead, as I proposed earlier, it seems that we'd
>> better
>> > > > start
>> > > > > > > with
>> > > > > > > > > the
>> > > > > > > > > > > problem we are trying to solve. Basically, compression
>> > > could
>> > > > > help
>> > > > > > > in
>> > > > > > > > > two
>> > > > > > > > > > > cases:
>> > > > > > > > > > > 1) Transmitting data over wire - it should be
>> implemented
>> > > on
>> > > > > > > > > > communication
>> > > > > > > > > > > layer and should not affect binary serialization
>> > component
>> > > a
>> > > > > lot.
>> > > > > > > > > > > 2) Storing data in memory - here the much simpler step
>> > > would
>> > > > be
>> > > > > > to
>> > > > > > > > full
>> > > > > > > > > > > compression on per-cache basis rather than dealing
>> with
>> > > > > > per-fields
>> > > > > > > > > case.
>> > > > > > > > > > >
>> > > > > > > > > > > In the end, if user would like to compress particular
>> > > field,
>> > > > he
>> > > > > > can
>> > > > > > > > > > always
>> > > > > > > > > > > to it on his own, and set already compressed field to
>> our
>> > > > > > > > BinaryObject.
>> > > > > > > > > > >
>> > > > > > > > > > > Vladimir.
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > On Thu, Jun 8, 2017 at 12:37 PM, Vyacheslav Daradur <
>> > > > > > > > > daradu...@gmail.com
>> > > > > > > > > > >
>> > > > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > Valentin,
>> > > > > > > > > > > >
>> > > > > > > > > > > > Yes, I have the prototype[1][2]
>> > > > > > > > > > > >
>> > > > > > > > > > > > You can see an example of Java class[3] that I used
>> in
>> > my
>> > > > > > > > benchmark.
>> > > > > > > > > > > > For example:
>> > > > > > > > > > > > class Foo {
>> > > > > > > > > > > > @BinaryCompression
>> > > > > > > > > > > > String data;
>> > > > > > > > > > > > }
>> > > > > > > > > > > > If user make decision to store the object in
>> compressed
>> > > > form,
>> > > > > > he
>> > > > > > > > can
>> > > > > > > > > > use
>> > > > > > > > > > > > the annotation @BinaryCompression as shown above.
>> > > > > > > > > > > > It means annotated field 'data' will be compressed
>> at
>> > > > > > > marshalling.
>> > > > > > > > > > > >
>> > > > > > > > > > > > [1] https://github.com/apache/ignite/pull/1951
>> > > > > > > > > > > > [2] https://issues.apache.org/jira
>> /browse/IGNITE-5226
>> > > > > > > > > > > > [3]
>> > > > > > > > > > > > https://github.com/daradurvs/i
>> gnite-compression/blob/
>> > > > > > > > > > > > master/src/main/java/ru/daradu
>> rvs/ignite/compression/
>> > > > > > > > > > model/Audit1F.java
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > 2017-06-08 2:04 GMT+03:00 Valentin Kulichenko <
>> > > > > > > > > > > > valentin.kuliche...@gmail.com
>> > > > > > > > > > > > >:
>> > > > > > > > > > > >
>> > > > > > > > > > > > > Vyacheslav, Anton,
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Are there any ideas and/or prototypes for the API?
>> > Your
>> > > > > > design
>> > > > > > > > > > > > suggestions
>> > > > > > > > > > > > > seem to make sense, but I would like to see how it
>> > all
>> > > > this
>> > > > > > > will
>> > > > > > > > > like
>> > > > > > > > > > > > from
>> > > > > > > > > > > > > user's standpoint.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > -Val
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > On Wed, Jun 7, 2017 at 1:06 AM, Антон Чураев <
>> > > > > > > > churaev...@gmail.com
>> > > > > > > > > >
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > > Vyacheslav, correct me if something wrong
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > We could provide opportunity of choose between
>> CPU
>> > > > usage
>> > > > > > and
>> > > > > > > > > > MEM/NET
>> > > > > > > > > > > > > usage
>> > > > > > > > > > > > > > for users by compression some attributes of
>> stored
>> > > > > objects.
>> > > > > > > > > > > > > > You have learned design, and it is possible to
>> > > localize
>> > > > > > > changes
>> > > > > > > > > in
>> > > > > > > > > > > > > > marshalling without performance affect and
>> current
>> > > > > > > > functionality.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > I think, that it's usefull for our project and
>> > users.
>> > > > > > > > > > > > > > Community, what do you think about this
>> proposal?
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > 2017-06-06 17:29 GMT+03:00 Vyacheslav Daradur <
>> > > > > > > > > daradu...@gmail.com
>> > > > > > > > > > >:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > In short,
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > During marshalling a fields is represented as
>> > > > > > > > > BinaryFieldAccessor
>> > > > > > > > > > > > which
>> > > > > > > > > > > > > > > manages its marshalling. It checks if the
>> field
>> > is
>> > > > > marked
>> > > > > > > by
>> > > > > > > > > > > > annotation
>> > > > > > > > > > > > > > > @BinaryCompression, in that case - binary
>> > > > > representation
>> > > > > > > of
>> > > > > > > > > > field
>> > > > > > > > > > > > > (bytes
>> > > > > > > > > > > > > > > array) will be compressed. It will be marked
>> as
>> > > > > > compressed
>> > > > > > > by
>> > > > > > > > > > types
>> > > > > > > > > > > > > > > constant (GridBinaryMarshaller.COMPRESSED),
>> > after
>> > > > this
>> > > > > > the
>> > > > > > > > > > > > compressed
>> > > > > > > > > > > > > > > bytes
>> > > > > > > > > > > > > > > array wiil be include in binary
>> representation of
>> > > > whole
>> > > > > > > > object.
>> > > > > > > > > > > Note,
>> > > > > > > > > > > > > > > header of marshalled object will not be
>> > compressed.
>> > > > > > > > Compression
>> > > > > > > > > > > > > affected
>> > > > > > > > > > > > > > > only object's field representation.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Objects in IgniteCache is represented as
>> > > BinaryObject
>> > > > > > which
>> > > > > > > > is
>> > > > > > > > > > > > wrapper
>> > > > > > > > > > > > > > over
>> > > > > > > > > > > > > > > bytes array of marshalled object.
>> > > > > > > > > > > > > > > BinaryObject provides some usefull methods,
>> which
>> > > are
>> > > > > > used
>> > > > > > > by
>> > > > > > > > > > > Ignite
>> > > > > > > > > > > > > > > systems.
>> > > > > > > > > > > > > > > For example, the Queries use
>> BinaryObject#field
>> > > > method,
>> > > > > > > which
>> > > > > > > > > > > > > > deserializes
>> > > > > > > > > > > > > > > only field of object, without deserializing of
>> > > whole
>> > > > > > > object.
>> > > > > > > > > > > > > > > BinaryObject#field method during
>> deserialization,
>> > > if
>> > > > > > meets
>> > > > > > > > the
>> > > > > > > > > > > > constant
>> > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > compressed type, decompress this bytes array,
>> > then
>> > > > > > continue
>> > > > > > > > > > > > > unmarshalling
>> > > > > > > > > > > > > > > as usual.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Now, I introduced the Compressor interface in
>> > > > > > > > > > IgniteConfigurations,
>> > > > > > > > > > > > it
>> > > > > > > > > > > > > > > allows user to use own implementation of
>> > > compressor -
>> > > > > it
>> > > > > > is
>> > > > > > > > the
>> > > > > > > > > > > > > > requirement
>> > > > > > > > > > > > > > > in the task[1].
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > As far as I know, Vladimir Ozerov doesn't like
>> > the
>> > > > idea
>> > > > > > of
>> > > > > > > > > > granting
>> > > > > > > > > > > > > this
>> > > > > > > > > > > > > > > opportunity to the user.
>> > > > > > > > > > > > > > > In that case we can choose a compression
>> > algorithm
>> > > > > which
>> > > > > > we
>> > > > > > > > > will
>> > > > > > > > > > > > > provide
>> > > > > > > > > > > > > > by
>> > > > > > > > > > > > > > > default and will move the interface to
>> internals
>> > of
>> > > > > > binary
>> > > > > > > > > > > > > > infractructure.
>> > > > > > > > > > > > > > > For this case I've prepared benchmarked, which
>> > I've
>> > > > > sent
>> > > > > > > > > earlier.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > I vote for ZSTD algorithm[2], it provides good
>> > > > > > compression
>> > > > > > > > > ratio
>> > > > > > > > > > > and
>> > > > > > > > > > > > > good
>> > > > > > > > > > > > > > > throughput. It has implementation in Java,
>> .NET
>> > and
>> > > > > C++,
>> > > > > > > and
>> > > > > > > > > has
>> > > > > > > > > > > > > > > ASF-friendly license, we can use it in the all
>> > > Ignite
>> > > > > > > > > platforms.
>> > > > > > > > > > > > > > > You can look at an assessment of this
>> algorithm
>> > in
>> > > my
>> > > > > > > > > benchmark's
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > [1] https://issues.apache.org/
>> > > > jira/browse/IGNITE-3592
>> > > > > > > > > > > > > > > [2]https://github.com/facebook/zstd
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > 2017-06-06 16:02 GMT+03:00 Антон Чураев <
>> > > > > > > > churaev...@gmail.com
>> > > > > > > > > >:
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Looks good for me.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Could You propose design of implementation
>> in
>> > > > couple
>> > > > > of
>> > > > > > > > > > > sentences?
>> > > > > > > > > > > > > > > > So that we can estimate the completeness and
>> > > > > complexity
>> > > > > > > of
>> > > > > > > > > the
>> > > > > > > > > > > > > > proposal.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > 2017-06-06 15:26 GMT+03:00 Vyacheslav
>> Daradur <
>> > > > > > > > > > > daradu...@gmail.com
>> > > > > > > > > > > > >:
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > Anton,
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > Of course, the solution does not affect on
>> > > > existing
>> > > > > > > > > > > > > implementation. I
>> > > > > > > > > > > > > > > > mean,
>> > > > > > > > > > > > > > > > > there is no changes if user not use the
>> > > > annotation
>> > > > > > > > > > > > > > @BinaryCompression.
>> > > > > > > > > > > > > > > > (no
>> > > > > > > > > > > > > > > > > performance changes)
>> > > > > > > > > > > > > > > > > Only if user make decision to use
>> compression
>> > > on
>> > > > > > > specific
>> > > > > > > > > > field
>> > > > > > > > > > > > or
>> > > > > > > > > > > > > > > fields
>> > > > > > > > > > > > > > > > > of a class - in that case compression
>> will be
>> > > > used
>> > > > > at
>> > > > > > > > > > > marshalling
>> > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > relation to annotated fields.
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > 2017-06-06 15:10 GMT+03:00 Антон Чураев <
>> > > > > > > > > > churaev...@gmail.com
>> > > > > > > > > > > >:
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Vyacheslav,
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Is it possible to propose implementation
>> > that
>> > > > can
>> > > > > > be
>> > > > > > > > > > switched
>> > > > > > > > > > > > on
>> > > > > > > > > > > > > > > > > on-demand?
>> > > > > > > > > > > > > > > > > > In this case it should not affect
>> > performance
>> > > > of
>> > > > > > > > current
>> > > > > > > > > > > > > solution.
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > I mean, that users should make decision
>> > what
>> > > is
>> > > > > > more
>> > > > > > > > > > > important
>> > > > > > > > > > > > > for
>> > > > > > > > > > > > > > > > them:
>> > > > > > > > > > > > > > > > > > throutput or memory/net usage.
>> > > > > > > > > > > > > > > > > > May be they will be choose not all
>> objects,
>> > > or
>> > > > > only
>> > > > > > > > some
>> > > > > > > > > > > > > attributes
>> > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > objects for compress.
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > 2017-06-06 14:48 GMT+03:00 Vyacheslav
>> > > Daradur <
>> > > > > > > > > > > > > daradu...@gmail.com
>> > > > > > > > > > > > > > >:
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > Conclusion:
>> > > > > > > > > > > > > > > > > > > Provided solution allows reduce size
>> of
>> > an
>> > > > > object
>> > > > > > > in
>> > > > > > > > > > > > > IgniteCache
>> > > > > > > > > > > > > > at
>> > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > cost of throughput reduction (small -
>> in
>> > > some
>> > > > > > > cases),
>> > > > > > > > > it
>> > > > > > > > > > > > > depends
>> > > > > > > > > > > > > > on
>> > > > > > > > > > > > > > > > > part
>> > > > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > object which will be compressed and
>> > > > compression
>> > > > > > > > > > algorithm.
>> > > > > > > > > > > > > > > > > > > I mean, we can make more effective
>> use of
>> > > > > memory,
>> > > > > > > and
>> > > > > > > > > in
>> > > > > > > > > > > some
>> > > > > > > > > > > > > > cases
>> > > > > > > > > > > > > > > > it
>> > > > > > > > > > > > > > > > > > can
>> > > > > > > > > > > > > > > > > > > reduce loading of the interconnect.
>> > > > > (replication,
>> > > > > > > > > > > > rebalancing)
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > Especially, it will be particularly
>> > useful
>> > > > for
>> > > > > > > > object's
>> > > > > > > > > > > > fields
>> > > > > > > > > > > > > > > which
>> > > > > > > > > > > > > > > > > are
>> > > > > > > > > > > > > > > > > > > large text (>~ 250 bytes) and can be
>> > > > > effectively
>> > > > > > > > > > > compressed.
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > 2017-06-06 12:00 GMT+03:00 Антон
>> Чураев <
>> > > > > > > > > > > > churaev...@gmail.com
>> > > > > > > > > > > > > >:
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > Vyacheslav, thank you! But could you
>> > > please
>> > > > > > > > provide a
>> > > > > > > > > > > > > > conclusions
>> > > > > > > > > > > > > > > > or
>> > > > > > > > > > > > > > > > > > > > proposals based on this benchmarks?
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > 2017-06-06 11:28 GMT+03:00
>> Vyacheslav
>> > > > > Daradur <
>> > > > > > > > > > > > > > > daradu...@gmail.com
>> > > > > > > > > > > > > > > > >:
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > Dmitry,
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > Excel-pages:
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > 1). "Compression ratio (2)" -
>> shows
>> > > > object
>> > > > > > > size,
>> > > > > > > > > with
>> > > > > > > > > > > > > > > compression
>> > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > > without compression. (Conditions:
>> > > literal
>> > > > > > text)
>> > > > > > > > > > > > > > > > > > > > > 1st graph shows compression
>> ratios of
>> > > > using
>> > > > > > > > > different
>> > > > > > > > > > > > > > > compression
>> > > > > > > > > > > > > > > > > > > > algrithms
>> > > > > > > > > > > > > > > > > > > > > depending on size of compressed
>> > field.
>> > > > > > > > > > > > > > > > > > > > > 2nd graph shows evaluation of
>> size of
>> > > > > objects
>> > > > > > > > > > depending
>> > > > > > > > > > > > on
>> > > > > > > > > > > > > > > sizes
>> > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > > compression algorithms.
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > 2). "Compression ratio (1)" -
>> shows
>> > > > object
>> > > > > > > size,
>> > > > > > > > > with
>> > > > > > > > > > > > > > > compression
>> > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > > without compression. (Conditions:
>> > > badly
>> > > > > > > > compressed
>> > > > > > > > > > > > > character
>> > > > > > > > > > > > > > > > > > sequence)
>> > > > > > > > > > > > > > > > > > > > > 1st graph shows compression
>> ratios of
>> > > > using
>> > > > > > > > > different
>> > > > > > > > > > > > > > > compression
>> > > > > > > > > > > > > > > > > > > > > algrithms depending on size of
>> > > compressed
>> > > > > > > field.
>> > > > > > > > > > > > > > > > > > > > > 2nd graph shows evaluation of
>> size of
>> > > > > objects
>> > > > > > > > > > depending
>> > > > > > > > > > > > on
>> > > > > > > > > > > > > > > sizes
>> > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > > compression algorithms.
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > 3) 'put-avg" - shows average time
>> of
>> > > the
>> > > > > > "put"
>> > > > > > > > > > > operation
>> > > > > > > > > > > > > > > > depending
>> > > > > > > > > > > > > > > > > on
>> > > > > > > > > > > > > > > > > > > > size
>> > > > > > > > > > > > > > > > > > > > > and compression algorithms.
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > 4) 'put-thrpt" - shows throughput
>> of
>> > > the
>> > > > > > "put"
>> > > > > > > > > > > operation
>> > > > > > > > > > > > > > > > depending
>> > > > > > > > > > > > > > > > > on
>> > > > > > > > > > > > > > > > > > > > size
>> > > > > > > > > > > > > > > > > > > > > and compression algorithms.
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > 5) 'get-avg" - shows average time
>> of
>> > > the
>> > > > > > "get"
>> > > > > > > > > > > operation
>> > > > > > > > > > > > > > > > depending
>> > > > > > > > > > > > > > > > > on
>> > > > > > > > > > > > > > > > > > > > size
>> > > > > > > > > > > > > > > > > > > > > and compression algorithms.
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > 6) 'get-thrpt" - shows throughput
>> of
>> > > the
>> > > > > > "get"
>> > > > > > > > > > > operation
>> > > > > > > > > > > > > > > > depending
>> > > > > > > > > > > > > > > > > on
>> > > > > > > > > > > > > > > > > > > > size
>> > > > > > > > > > > > > > > > > > > > > and compression algorithms.
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > 2017-06-06 10:59 GMT+03:00 Dmitriy
>> > > > > Setrakyan
>> > > > > > <
>> > > > > > > > > > > > > > > > > dsetrak...@apache.org
>> > > > > > > > > > > > > > > > > > >:
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > Vladimir, I am not sure how to
>> > > > interpret
>> > > > > > the
>> > > > > > > > > > graphs?
>> > > > > > > > > > > > What
>> > > > > > > > > > > > > > are
>> > > > > > > > > > > > > > > > we
>> > > > > > > > > > > > > > > > > > > > looking
>> > > > > > > > > > > > > > > > > > > > > > at?
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > On Tue, Jun 6, 2017 at 12:33 AM,
>> > > > > Vyacheslav
>> > > > > > > > > > Daradur <
>> > > > > > > > > > > > > > > > > > > > daradu...@gmail.com
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > Hi, Igniters.
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > I've prepared some
>> benchmarking.
>> > > > > Results
>> > > > > > > [1].
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > And I've prepared the
>> evaluation
>> > in
>> > > > the
>> > > > > > > form
>> > > > > > > > of
>> > > > > > > > > > > > > diagrams
>> > > > > > > > > > > > > > > [2].
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > I hope that helps to interest
>> the
>> > > > > > community
>> > > > > > > > and
>> > > > > > > > > > > > > > > accelerates a
>> > > > > > > > > > > > > > > > > > > > reaction
>> > > > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > > > this improvment :)
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > [1]
>> > > > > > > > > > > > > > > > > > > > > > > https://github.com/daradurvs/
>> > > > > > > > > > > > ignite-compression/tree/
>> > > > > > > > > > > > > > > > > > > > > > > master/src/main/resources/resu
>> lt
>> > > > > > > > > > > > > > > > > > > > > > > [2]
>> > https://drive.google.com/file/
>> > > d/
>> > > > > > > > > > > > > > > > > > 0B2CeUAOgrHkoMklyZ25YTEdKcEk/
>> > > > > > > > > > > > > > > > > > > > view
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > 2017-05-24 9:49 GMT+03:00
>> > > Vyacheslav
>> > > > > > > Daradur
>> > > > > > > > <
>> > > > > > > > > > > > > > > > > > daradu...@gmail.com
>> > > > > > > > > > > > > > > > > > > >:
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > Guys, any thoughts?
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > 2017-05-16 13:40 GMT+03:00
>> > > > Vyacheslav
>> > > > > > > > > Daradur <
>> > > > > > > > > > > > > > > > > > > daradu...@gmail.com
>> > > > > > > > > > > > > > > > > > > > >:
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > >> Hi guys,
>> > > > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > > > >> I've prepared the PR to
>> show
>> > my
>> > > > > idea.
>> > > > > > > > > > > > > > > > > > > > > > > >> https://github.com/apache/
>> > > > > > > > > > > ignite/pull/1951/files
>> > > > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > > > >> About querying - I've just
>> > > copied
>> > > > > > > existing
>> > > > > > > > > > tests
>> > > > > > > > > > > > and
>> > > > > > > > > > > > > > > have
>> > > > > > > > > > > > > > > > > > > > annotated
>> > > > > > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > > > >> testing data.
>> > > > > > > > > > > > > > > > > > > > > > > >> https://github.com/apache/
>> > > > > > > > > > > > > > ignite/pull/1951/files#diff-
>> > > > > > > > > > > > > > > > > c19a9d
>> > > > > > > > > > > > > > > > > > > > > > > >> f4058141d059bb577e75244764
>> > > > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > > > >> It means fields which will
>> be
>> > > > marked
>> > > > > > by
>> > > > > > > > > > > > > > > @BinaryCompression
>> > > > > > > > > > > > > > > > > > will
>> > > > > > > > > > > > > > > > > > > be
>> > > > > > > > > > > > > > > > > > > > > > > >> compressed at marshalling
>> via
>> > > > > > > > > > BinaryMarshaller.
>> > > > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > > > >> This solution has no
>> effect on
>> > > > > > existing
>> > > > > > > > data
>> > > > > > > > > > or
>> > > > > > > > > > > > > > project
>> > > > > > > > > > > > > > > > > > > > > architecture.
>> > > > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > > > >> I'll be glad to see your
>> > > thougths.
>> > > > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > > > >> 2017-05-15 19:18 GMT+03:00
>> > > > > Vyacheslav
>> > > > > > > > > Daradur
>> > > > > > > > > > <
>> > > > > > > > > > > > > > > > > > > > daradu...@gmail.com
>> > > > > > > > > > > > > > > > > > > > > >:
>> > > > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > > > >>> Dmitriy,
>> > > > > > > > > > > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > > > > > > > > > > >>> I have ready prototype. I
>> > want
>> > > to
>> > > > > > show
>> > > > > > > > it.
>> > > > > > > > > > > > > > > > > > > > > > > >>> It is always easier to
>> > discuss
>> > > on
>> > > > > > > > example.
>> > > > > > > > > > > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > > > > > > > > > > >>> 2017-05-15 19:02 GMT+03:00
>> > > > Dmitriy
>> > > > > > > > > Setrakyan
>> > > > > > > > > > <
>> > > > > > > > > > > > > > > > > > > > > dsetrak...@apache.org
>> > > > > > > > > > > > > > > > > > > > > > >:
>> > > > > > > > > > > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > > > > > > > > > > >>>> Vyacheslav,
>> > > > > > > > > > > > > > > > > > > > > > > >>>>
>> > > > > > > > > > > > > > > > > > > > > > > >>>> I think it is a bit
>> > premature
>> > > to
>> > > > > > > > provide a
>> > > > > > > > > > PR
>> > > > > > > > > > > > > > without
>> > > > > > > > > > > > > > > > > > getting
>> > > > > > > > > > > > > > > > > > > a
>> > > > > > > > > > > > > > > > > > > > > > > >>>> community
>> > > > > > > > > > > > > > > > > > > > > > > >>>> consensus on the dev
>> list.
>> > > > Please
>> > > > > > > allow
>> > > > > > > > > some
>> > > > > > > > > > > > time
>> > > > > > > > > > > > > > for
>> > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > > > community
>> > > > > > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > > > >>>> respond.
>> > > > > > > > > > > > > > > > > > > > > > > >>>>
>> > > > > > > > > > > > > > > > > > > > > > > >>>> D.
>> > > > > > > > > > > > > > > > > > > > > > > >>>>
>> > > > > > > > > > > > > > > > > > > > > > > >>>> On Mon, May 15, 2017 at
>> 6:36
>> > > AM,
>> > > > > > > > > Vyacheslav
>> > > > > > > > > > > > > Daradur
>> > > > > > > > > > > > > > <
>> > > > > > > > > > > > > > > > > > > > > > > >>>> daradu...@gmail.com>
>> > > > > > > > > > > > > > > > > > > > > > > >>>> wrote:
>> > > > > > > > > > > > > > > > > > > > > > > >>>>
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > I created the ticket:
>> > > > > > > > > > > > > > > https://issues.apache.org/jira
>> > > > > > > > > > > > > > > > > > > > > > > >>>> /browse/IGNITE-5226
>> > > > > > > > > > > > > > > > > > > > > > > >>>> >
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > I'll prepare a PR with
>> > > > described
>> > > > > > > > > solution
>> > > > > > > > > > in
>> > > > > > > > > > > > > > couple
>> > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > days.
>> > > > > > > > > > > > > > > > > > > > > > > >>>> >
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > 2017-05-15 15:05
>> GMT+03:00
>> > > > > > > Vyacheslav
>> > > > > > > > > > > Daradur
>> > > > > > > > > > > > <
>> > > > > > > > > > > > > > > > > > > > > > daradu...@gmail.com
>> > > > > > > > > > > > > > > > > > > > > > > >:
>> > > > > > > > > > > > > > > > > > > > > > > >>>> >
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > > Hi, Igniters!
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > > Apache 2.0 is
>> released.
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > > Let's continue the
>> > > > discussion
>> > > > > > > about
>> > > > > > > > a
>> > > > > > > > > > > > > > compression
>> > > > > > > > > > > > > > > > > > design.
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > > At the moment, I
>> found
>> > > only
>> > > > > one
>> > > > > > > > > solution
>> > > > > > > > > > > > which
>> > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > > > > compatible
>> > > > > > > > > > > > > > > > > > > > > > with
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > querying
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > > and indexing, this is
>> > > > > > > > > per-objects-field
>> > > > > > > > > > > > > > > compression.
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > > Per-fields
>> compression
>> > > means
>> > > > > > that
>> > > > > > > > > > metadata
>> > > > > > > > > > > > (a
>> > > > > > > > > > > > > > > > header)
>> > > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > an
>> > > > > > > > > > > > > > > > > > > > > > object
>> > > > > > > > > > > > > > > > > > > > > > > >>>> won't
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > > be compressed, only
>> > > > serialized
>> > > > > > > > values
>> > > > > > > > > of
>> > > > > > > > > > > an
>> > > > > > > > > > > > > > object
>> > > > > > > > > > > > > > > > > > fields
>> > > > > > > > > > > > > > > > > > > > (in
>> > > > > > > > > > > > > > > > > > > > > > > bytes
>> > > > > > > > > > > > > > > > > > > > > > > >>>> array
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > > form) will be
>> > compressed.
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > > This solution have
>> some
>> > > > > > > contentious
>> > > > > > > > > > > issues:
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > > - small values, like
>> > > > > primitives
>> > > > > > > and
>> > > > > > > > > > short
>> > > > > > > > > > > > > > arrays -
>> > > > > > > > > > > > > > > > > there
>> > > > > > > > > > > > > > > > > > > > isn't
>> > > > > > > > > > > > > > > > > > > > > > > >>>> sense to
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > > compress them;
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > > - there is no
>> possible
>> > to
>> > > > use
>> > > > > > > > > > compression
>> > > > > > > > > > > > with
>> > > > > > > > > > > > > > > > > > > > java-predefined
>> > > > > > > > > > > > > > > > > > > > > > > >>>> types;
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > > We can provide an
>> > > > annotation,
>> > > > > > > > > > > > > > @IgniteCompression -
>> > > > > > > > > > > > > > > > for
>> > > > > > > > > > > > > > > > > > > > > example,
>> > > > > > > > > > > > > > > > > > > > > > > >>>> which can
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > > be used by users for
>> > > marking
>> > > > > > > fields
>> > > > > > > > to
>> > > > > > > > > > > > > compress.
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > > Any thoughts?
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > > Maybe someone already
>> > have
>> > > > > ready
>> > > > > > > > > design?
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > > 2017-04-10 11:06
>> > GMT+03:00
>> > > > > > > > Vyacheslav
>> > > > > > > > > > > > Daradur
>> > > > > > > > > > > > > <
>> > > > > > > > > > > > > > > > > > > > > > > daradu...@gmail.com
>> > > > > > > > > > > > > > > > > > > > > > > >>>> >:
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >> Alexey,
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >> Yes, I've read it.
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >> Ok, let's discuss
>> about
>> > > > > public
>> > > > > > > API
>> > > > > > > > > > > design.
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >> I think we need to
>> add
>> > > > some a
>> > > > > > > > > configure
>> > > > > > > > > > > > > entity
>> > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > > > >>>> CacheConfiguration,
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >> which will contain
>> the
>> > > > > > Compressor
>> > > > > > > > > > > interface
>> > > > > > > > > > > > > > > > > > > implementation
>> > > > > > > > > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > > > > > some
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > usefull
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >> parameters.
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >> Or maybe to provide
>> a
>> > > > > > > > > BinaryMarshaller
>> > > > > > > > > > > > > > decorator,
>> > > > > > > > > > > > > > > > > which
>> > > > > > > > > > > > > > > > > > > > will
>> > > > > > > > > > > > > > > > > > > > > be
>> > > > > > > > > > > > > > > > > > > > > > > >>>> compress
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >> data after
>> marshalling.
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >> 2017-04-10 10:40
>> > > GMT+03:00
>> > > > > > Alexey
>> > > > > > > > > > > > Kuznetsov <
>> > > > > > > > > > > > > > > > > > > > > > > akuznet...@apache.org
>> > > > > > > > > > > > > > > > > > > > > > > >>>> >:
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> Vyacheslav,
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>>
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> Did you read
>> initial
>> > > > > > discussion
>> > > > > > > > [1]
>> > > > > > > > > > > about
>> > > > > > > > > > > > > > > > > compression?
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> As far as I
>> remember
>> > we
>> > > > > agreed
>> > > > > > > to
>> > > > > > > > > add
>> > > > > > > > > > > only
>> > > > > > > > > > > > > > some
>> > > > > > > > > > > > > > > > > > > > "top-level"
>> > > > > > > > > > > > > > > > > > > > > > API
>> > > > > > > > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > order
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> to
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> provide a way for
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> Ignite users to
>> inject
>> > > > some
>> > > > > > sort
>> > > > > > > > of
>> > > > > > > > > > > custom
>> > > > > > > > > > > > > > > > > > compression.
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>>
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>>
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> [1]
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>>
>> > > > > > http://apache-ignite-developer
>> > > > > > > > > > > > > > > s.2346864.n4.nabble
>> > > > > > > > > > > > > > > > .
>> > > > > > > > > > > > > > > > > > > > > com/Data-c
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>>
>> > > ompression-in-Ignite-2-0-
>> > > > > > > > > td10099.html
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>>
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> On Mon, Apr 10,
>> 2017
>> > at
>> > > > 2:19
>> > > > > > PM,
>> > > > > > > > > > > > daradurvs <
>> > > > > > > > > > > > > > > > > > > > > > daradu...@gmail.com
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > wrote:
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>>
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> > Hi Igniters!
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> >
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> > I am interested
>> in
>> > > this
>> > > > > > task.
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> > Provide some
>> kind of
>> > > > > > pluggable
>> > > > > > > > > > > > compression
>> > > > > > > > > > > > > > SPI
>> > > > > > > > > > > > > > > > > > support
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> > <
>> > > > > https://issues.apache.org/
>> > > > > > > > > > > > > > > > > jira/browse/IGNITE-3592>
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> >
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> > I developed a
>> > solution
>> > > > on
>> > > > > > > > > > > > > > > > BinaryMarshaller-level,
>> > > > > > > > > > > > > > > > > > but
>> > > > > > > > > > > > > > > > > > > > > > reviewer
>> > > > > > > > > > > > > > > > > > > > > > > >>>> has
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> rejected
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> > it.
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> >
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> > Let's continue
>> > > > discussion
>> > > > > of
>> > > > > > > > task
>> > > > > > > > > > > goals
>> > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > solution
>> > > > > > > > > > > > > > > > > > > > > design.
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> > As I understood
>> > that,
>> > > > the
>> > > > > > main
>> > > > > > > > > goal
>> > > > > > > > > > of
>> > > > > > > > > > > > > this
>> > > > > > > > > > > > > > > task
>> > > > > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > > > store
>> > > > > > > > > > > > > > > > > > > > > > > >>>> data in
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> > compressed form.
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> > This is what I
>> need
>> > > from
>> > > > > > > Ignite
>> > > > > > > > as
>> > > > > > > > > > its
>> > > > > > > > > > > > > user.
>> > > > > > > > > > > > > > > > > > > Compression
>> > > > > > > > > > > > > > > > > > > > > > > >>>> provides
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> economy
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> > on
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> > servers.
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> > We can store more
>> > data
>> > > > on
>> > > > > > same
>> > > > > > > > > > servers
>> > > > > > > > > > > > at
>> > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > cost
>> > > > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > > > > > >>>> increasing CPU
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> > utilization.
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> >
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> > I'm researching a
>> > > > > > possibility
>> > > > > > > of
>> > > > > > > > > > > > > > > implementation
>> > > > > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > > > > > compression
>> > > > > > > > > > > > > > > > > > > > > > > >>>> at the
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> > cache-level.
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> >
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> > Any thoughts?
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> >
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> > --
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> > Best regards,
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> > Vyacheslav
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> >
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> >
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> >
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> >
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> > --
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> > View this
>> message in
>> > > > > > context:
>> > > > > > > > > > > > > > > > > http://apache-ignite-
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> >
>> > > > > > developers.2346864.n4.nabble.
>> > > > > > > > > > > > > > > > > > com/Data-compression-in-
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> >
>> > > > > > Ignite-2-0-tp10099p16317.html
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> > Sent from the
>> Apache
>> > > > > Ignite
>> > > > > > > > > > Developers
>> > > > > > > > > > > > > > mailing
>> > > > > > > > > > > > > > > > > list
>> > > > > > > > > > > > > > > > > > > > > archive
>> > > > > > > > > > > > > > > > > > > > > > at
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> Nabble.com.
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> >
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>>
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>>
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>>
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> --
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>> Alexey Kuznetsov
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>>
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >> --
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >> Best Regards,
>> > Vyacheslav
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >>
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > > --
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > > Best Regards,
>> Vyacheslav
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > >
>> > > > > > > > > > > > > > > > > > > > > > > >>>> >
>> > > > > > > > > > > > > > > > > > > > > > > >>>> >
>> > > > > > > > > > > > > > > > > > > > > > > >>>> >
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > --
>> > > > > > > > > > > > > > > > > > > > > > > >>>> > Best Regards,
>> Vyacheslav
>> > > > > > > > > > > > > > > > > > > > > > > >>>> >
>> > > > > > > > > > > > > > > > > > > > > > > >>>>
>> > > > > > > > > > > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > > > > > > > > > > >>> --
>> > > > > > > > > > > > > > > > > > > > > > > >>> Best Regards, Vyacheslav
>> > > > > > > > > > > > > > > > > > > > > > > >>>
>> > > > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > > > >> --
>> > > > > > > > > > > > > > > > > > > > > > > >> Best Regards, Vyacheslav
>> > > > > > > > > > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > > --
>> > > > > > > > > > > > > > > > > > > > > > > > Best Regards, Vyacheslav
>> > > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > > > --
>> > > > > > > > > > > > > > > > > > > > > > > Best Regards, Vyacheslav
>> > > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > > --
>> > > > > > > > > > > > > > > > > > > > > Best Regards, Vyacheslav
>> > > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > --
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > > Best Regards, Anton Churaev
>> > > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > --
>> > > > > > > > > > > > > > > > > > > Best Regards, Vyacheslav
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > --
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Best Regards, Anton Churaev
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > --
>> > > > > > > > > > > > > > > > > Best Regards, Vyacheslav
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > --
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Best Regards, Anton Churaev
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > --
>> > > > > > > > > > > > > > > Best Regards, Vyacheslav
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > --
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Best Regards, Anton Churaev
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > --
>> > > > > > > > > > > > Best Regards, Vyacheslav
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > --
>> > > > > > > > > > Best Regards, Vyacheslav
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > --
>> > > > > > > > >
>> > > > > > > > > Best Regards, Anton Churaev
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > --
>> > > > > > > > Best Regards, Vyacheslav
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > Best Regards, Vyacheslav
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Best Regards, Vyacheslav
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> >
>> > Best Regards, Anton Churaev
>> >
>>
>>
>>
>> --
>> Sergey Kozlov
>> GridGain Systems
>> www.gridgain.com
>>
>
>
>
> --
> Best Regards, Vyacheslav
>
--
Best Regards, Vyacheslav D.
