IGNITE-5229 is ready for review

[Roman Shtykh]

Igniters,
IGNITE-5229 is ready for review.Now Redis can have its default cache specified 
and used as a template when switching to other caches, as discussed earlier in 
this ml (please see the JIRA issue).

[1] https://issues.apache.org/jira/browse/IGNITE-5229
-- Roman

[GitHub] ignite pull request #2098: IGNITE-5229: Specify caches when using Redis prot...

[shroman]

GitHub user shroman opened a pull request:

https://github.com/apache/ignite/pull/2098

IGNITE-5229: Specify caches when using Redis protocol. Implemented vi…

…a SELECT command.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shroman/ignite IGNITE-5229

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2098.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2098


commit 90c6d9810d4ead9c47b18a3b87d96c9947c39896
Author: shroman 
Date:   2017-06-07T05:43:09Z

IGNITE-5229: Specify caches when using Redis protocol. Implemented via 
SELECT command.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Key type name and value type name for CREATE TABLE

[Valentin Kulichenko]

Vova,

If you add unique suffix losing human-readable type names, how will the
builder approach work? Maybe it makes sense to add an API call that returns
current type name for a table?

-Val

On Tue, Jun 6, 2017 at 7:43 PM Dmitriy Setrakyan 
wrote:

> Vova,
>
> I am not sure I like the key type name the way it is. Can we add some
> separator between the table name and key name, like "_". To me "PERSON_KEY"
> reads a lot better than "PERSONKey".
>
> D.
>
> On Tue, Jun 6, 2017 at 4:00 AM, Sergi Vladykin 
> wrote:
>
> > Unique suffix is a good idea.
> >
> > Sergi
> >
> > 2017-06-06 13:51 GMT+03:00 Vladimir Ozerov :
> >
> > > Igniters,
> > >
> > > In the very first implementation of CREATE TABLE we applied the
> following
> > > rule to key and value type names:
> > > keyTypeName == tableName + "Key"
> > > valTypeName == tableName
> > >
> > > E.g.:
> > > CREATE TABLE Person ...
> > > keyTypeName == PERSONKey
> > > valTypeName == PERSON
> > >
> > > After that user could potentially create objects with these type names
> > > manually and add them to cache through native Ignite API:
> > >
> > > BinaryObject key =
> IgniteBinary.builder("PERSONKey").addField().build();
> > > BinaryObject val = IgniteBinary.builder("PERSON").addField().build();
> > > IgniteCache.put(key, val);
> > >
> > > This approach has two problems:
> > > 1) Type names are not unique between different tables. it means that if
> > two
> > > tables with the same name are created in different schemas, we got a
> > > conflict.
> > > 2) Type names are bound to binary metadata, so if user decides to do
> the
> > > following, he will receive and error about incompatible metadata:
> > > CREATE TABLE Person (id INT PRIMARY KEY);
> > > DROP TABLE Person;
> > > CREATE TABLE Person(in BIGINT PRIMARY KEY); // Fail because old meta
> > still
> > > has type "Integer".
> > >
> > > In order to avoid that I am going to add unique suffix or so (say,
> UUID)
> > to
> > > type names. This way there will be no human-readable type names any
> more,
> > > but there will be no conflicts either. In future releases we will relax
> > > this somehow.
> > >
> > > Thoughts?
> > >
> > > Vladimir.
> > >
> >
>

Re: Key type name and value type name for CREATE TABLE

[Dmitriy Setrakyan]

Vova,

I am not sure I like the key type name the way it is. Can we add some
separator between the table name and key name, like "_". To me "PERSON_KEY"
reads a lot better than "PERSONKey".

D.

On Tue, Jun 6, 2017 at 4:00 AM, Sergi Vladykin 
wrote:

> Unique suffix is a good idea.
>
> Sergi
>
> 2017-06-06 13:51 GMT+03:00 Vladimir Ozerov :
>
> > Igniters,
> >
> > In the very first implementation of CREATE TABLE we applied the following
> > rule to key and value type names:
> > keyTypeName == tableName + "Key"
> > valTypeName == tableName
> >
> > E.g.:
> > CREATE TABLE Person ...
> > keyTypeName == PERSONKey
> > valTypeName == PERSON
> >
> > After that user could potentially create objects with these type names
> > manually and add them to cache through native Ignite API:
> >
> > BinaryObject key = IgniteBinary.builder("PERSONKey").addField().build();
> > BinaryObject val = IgniteBinary.builder("PERSON").addField().build();
> > IgniteCache.put(key, val);
> >
> > This approach has two problems:
> > 1) Type names are not unique between different tables. it means that if
> two
> > tables with the same name are created in different schemas, we got a
> > conflict.
> > 2) Type names are bound to binary metadata, so if user decides to do the
> > following, he will receive and error about incompatible metadata:
> > CREATE TABLE Person (id INT PRIMARY KEY);
> > DROP TABLE Person;
> > CREATE TABLE Person(in BIGINT PRIMARY KEY); // Fail because old meta
> still
> > has type "Integer".
> >
> > In order to avoid that I am going to add unique suffix or so (say, UUID)
> to
> > type names. This way there will be no human-readable type names any more,
> > but there will be no conflicts either. In future releases we will relax
> > this somehow.
> >
> > Thoughts?
> >
> > Vladimir.
> >
>

Re: Only lateAffinityAssignment for 2.1

[Dmitriy Setrakyan]

On Tue, Jun 6, 2017 at 4:23 AM, Semyon Boikov  wrote:

> Now affinity version is switched only when migration for new primaries
> finishes for all partitions and for all caches.
>

Sounds great. Semyon, is there documentation that needs to be updated as
well?

[jira] [Created] (IGNITE-5428) Failed to build ignite-jta because a dependent library is not downloadable

[Kengo Seki (JIRA)]

Kengo Seki created IGNITE-5428:
--

 Summary: Failed to build ignite-jta because a dependent library is 
not downloadable
 Key: IGNITE-5428
 URL: https://issues.apache.org/jira/browse/IGNITE-5428
 Project: Ignite
  Issue Type: Bug
  Components: build
Affects Versions: 2.0
Reporter: Kengo Seki


I tried to build the master branch and got the following error. Maven 
repository ow2.org providing org.jacorb:jacorb:jar:2.2.3-jonas-patch-20071018 
seems down.

{code}
$ mvn clean package -DskipTests

(snip)

[INFO] 
[INFO] Building ignite-jta 2.1.0-SNAPSHOT
[INFO] 
Downloading: 
http://maven.ow2.org/maven2/org/jacorb/jacorb/2.2.3-jonas-patch-20071018/jacorb-2.2.3-jonas-patch-20071018.pom
Downloading: 
http://maven.ow2.org/maven2/org/jacorb/jacorb-idl/2.2.3-jonas-patch-20071018/jacorb-idl-2.2.3-jonas-patch-20071018.pom
[INFO] 
[INFO] Reactor Summary:
[INFO]
[INFO] ignite-apache-license-gen .. SUCCESS [  4.212 s]
[INFO] ignite-tools ... SUCCESS [ 28.157 s]
[INFO] ignite-core  SUCCESS [04:29 min]
[INFO] ignite-log4j ... SUCCESS [  6.677 s]
[INFO] ignite-indexing  SUCCESS [ 27.618 s]
[INFO] ignite-urideploy ... SUCCESS [  7.765 s]
[INFO] ignite-spring .. SUCCESS [  9.357 s]
[INFO] ignite-hadoop .. SUCCESS [ 31.444 s]
[INFO] ignite-extdata-p2p . SUCCESS [  8.357 s]
[INFO] ignite-extdata-uri-dep . SUCCESS [  3.947 s]
[INFO] ignite-extdata-uri . SUCCESS [  1.689 s]
[INFO] ignite-rest-http ... SUCCESS [  6.224 s]
[INFO] ignite-clients . SUCCESS [  3.442 s]
[INFO] ignite-spring-data . SUCCESS [  6.970 s]
[INFO] ignite-web . SUCCESS [  7.636 s]
[INFO] ignite-aop . SUCCESS [  6.271 s]
[INFO] ignite-ssh . SUCCESS [  8.456 s]
[INFO] ignite-jta . FAILURE [ 12.698 s]
[INFO] ignite-aws . SKIPPED
[INFO] ignite-log4j2 .. SKIPPED
[INFO] ignite-slf4j ... SKIPPED
[INFO] ignite-jcl . SKIPPED
[INFO] ignite-codegen . SKIPPED
[INFO] ignite-gce . SKIPPED
[INFO] ignite-cloud ... SKIPPED
[INFO] ignite-mesos ... SKIPPED
[INFO] ignite-kafka ... SKIPPED
[INFO] ignite-flume ... SKIPPED
[INFO] ignite-yarn  SKIPPED
[INFO] ignite-jms11 ... SKIPPED
[INFO] ignite-twitter . SKIPPED
[INFO] ignite-mqtt  SKIPPED
[INFO] ignite-zookeeper ... SKIPPED
[INFO] ignite-hibernate-core .. SKIPPED
[INFO] ignite-camel ... SKIPPED
[INFO] ignite-storm ... SKIPPED
[INFO] ignite-osgi-paxlogging . SKIPPED
[INFO] ignite-osgi-karaf .. SKIPPED
[INFO] ignite-osgi  SKIPPED
[INFO] ignite-appserver-test .. SKIPPED
[INFO] ignite-websphere-test .. SKIPPED
[INFO] ignite-cassandra ... SKIPPED
[INFO] ignite-cassandra-store . SKIPPED
[INFO] ignite-cassandra-serializers ... SKIPPED
[INFO] ignite-flink ... SKIPPED
[INFO] ignite-kubernetes .. SKIPPED
[INFO] ignite-zeromq .. SKIPPED
[INFO] ignite-rocketmq  SKIPPED
[INFO] ignite-scalar .. SKIPPED
[INFO] ignite-spark ... SKIPPED
[INFO] ignite-visor-console ... SKIPPED
[INFO] ignite-visor-plugins ... SKIPPED
[INFO] apache-ignite 

Re: Persistent Distributed Store Metrics

[Denis Magda]

> On Jun 5, 2017, at 8:02 PM, Konstantin Boudnik  wrote:
> 
> On Mon, Jun 05, 2017 at 07:41PM, Dmitriy Setrakyan wrote:
>> On Mon, Jun 5, 2017 at 6:46 PM, Konstantin Boudnik  wrote:
>> 
>>> Wow, hold on - as far as I remember there was a VOTE to accept the
>>> contribution of the code into the project _on a branch_. We haven't vetted
>>> its
>>> inclusion into the next release, We are still at the phase of getting
>>> familiar
>>> with the code.
>>> 
>> 
>> Cos, the community has been vetting the inclusion of the new code for over
>> 3 weeks already (the process and dates are documented here [2]). To be
>> honest, I am not sure what the appropriate time frame should be, but I
>> would think that a month would be a good check-in point.
> 
> I would think it should be as long as we, as the community, are comfortable
> with the massive change coming into the release. We already been through the
> discussion on the timing, etc. and I don't want to harp on that.

Just in case I’m working on the high-level storage documentation. Here is a 
draft for your reference:
https://apacheignite.readme.io/docs/distributed-persistent-store 


The page is invisible for wide audience at the moment. Only for Ignite 
committers.

—
Denis

> On Jun 5, 2017, at 8:02 PM, Konstantin Boudnik  wrote:
> 
> On Mon, Jun 05, 2017 at 07:41PM, Dmitriy Setrakyan wrote:
>> On Mon, Jun 5, 2017 at 6:46 PM, Konstantin Boudnik  wrote:
>> 
>>> Wow, hold on - as far as I remember there was a VOTE to accept the
>>> contribution of the code into the project _on a branch_. We haven't vetted
>>> its
>>> inclusion into the next release, We are still at the phase of getting
>>> familiar
>>> with the code.
>>> 
>> 
>> Cos, the community has been vetting the inclusion of the new code for over
>> 3 weeks already (the process and dates are documented here [2]). To be
>> honest, I am not sure what the appropriate time frame should be, but I
>> would think that a month would be a good check-in point.
> 
> I would think it should be as long as we, as the community, are comfortable
> with the massive change coming into the release. We already been through the
> discussion on the timing, etc. and I don't want to harp on that.
> 
>> There is also an active stabilization thread for the persistence branch
>> [3]. I encouraged the community to get involved and post any questions or
>> concerns there as well.
>> 
>> There is an upcoming in-memory computing conference that is coming up in
>> June in Amsterdam [4], where there are many Ignite talks scheduled. It
>> would be great to be able to talk about the persistence features of Ignite
>> there as well. I would like to ask the community to mobilize with reviewing
>> the donated code, so we could have something concrete to tell the audience
>> on the conference.
> 
> I am sure that having the code on the branch is good enough to be able to talk
> about this. Having this in the release isn't really a contingency to be able
> to make a presentation, right?
> 
> Cos
> 
>> [2]
>> http://incubator.apache.org/ip-clearance/persistent-distributed-store-ignite.html
>> [3]
>> http://apache-ignite-developers.2346864.n4.nabble.com/Persistent-Store-Stabilization-for-release-td18288.html
>> [4] https://imcsummit.org/
>> 
>> 
>> 
>>> And from what I am seeing in the discussions like this [1], we need to be
>>> extra careful.
>>> 
>> 
>> I would keep the discussion in [1] separate from the persistence store.
>> These are 2 unrelated issues. I will respond on [1] either today or
>> tomorrow, but I agree in general that it should be fixed ASAP.
>> 
>> 
>>> BTW, you have sent this email 9 days before the vote had happened! A bit
>>> too
>>> soon, if you ask me.
>>> 
>> 
>> Cos, this email was sent 1 week after the initial donated code branch was
>> presented to the community (see [2] above). The developers involved were
>> eager to make it available to the users as soon as possible, but no code
>> has been merged to the master branch yet.
>> 
>> I would like to encourage everyone in the community to participate in the
>> persistence branch coding discussions, like the one in this thread, and
>> familiarize themselves with the code.
>> 
>> 
>>> 
>>> [1] https://is.gd/UQCr51
>>> 
>>> Cos
>>> 
>>> On Wed, May 17, 2017 at 11:16AM, Dmitriy Govorukhin wrote:
 Folk,
 
 As you know, ignite 2.1 will contain new module (pds), it will be
 provide ability to store data on disk. Let's discuss what type of
 metrics we need for this?
 I think it must be metrics per memory policy, per cache, checkpoint,
 and global metrics which will be aggregate all metrics.
 
 I did sketch.
 
 PersistentStoreMetrics.java
 
 public interface PersistentStoreMetrics {
 
// Global metrics.
 
public long getMemorySize();
 
public long getDiskSize();

Re: is IGNITE-3142 actual now

[Denis Magda]

Vadim,

Please try to run the reproducer attached. If the error pops up then the bug 
needs to be fixed.

—
Denis

> On Jun 6, 2017, at 4:36 AM, Вадим Опольский  wrote:
> 
> Hello guys!
> 
> I want to resolve issue https://issues.apache.org/jira/browse/IGNITE-3142 
> 
> 
> Is it actual now?
> 
> Vadim Opolski

[GitHub] ignite pull request #2096: ignite-5383

[daradurvs]

GitHub user daradurvs opened a pull request:

https://github.com/apache/ignite/pull/2096

ignite-5383

…inaryMarshaller is used"

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/daradurvs/ignite ignite-5383

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2096.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2096


commit adec739baee06a20d9c6b279040c1266ce93af1a
Author: Vyacheslav Daradur 
Date:   2017-06-06T16:25:18Z

ignite-5383: implemented: "Do not perform cache key validation when 
BinaryMarshaller is used"




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-5427) Add cluster activation/deactivation events

[Alexey Goncharuk (JIRA)]

Alexey Goncharuk created IGNITE-5427:


 Summary: Add cluster activation/deactivation events
 Key: IGNITE-5427
 URL: https://issues.apache.org/jira/browse/IGNITE-5427
 Project: Ignite
  Issue Type: Improvement
  Components: general
Affects Versions: 2.0
Reporter: Alexey Goncharuk
 Fix For: 2.2






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: "Review workflow" changes to prevent "broken review" issues.

[Anton Vinogradov]

Igniters,

Since we found that proposed approach can help,
no one mind that I'll add text listed above to the wiki?

On Tue, Jun 6, 2017 at 1:19 PM, Anton Vinogradov 
wrote:

> Dmitry,
>
> 1) See my initial email, it contains instruction how to find a reviewer.
> And it's pretty easy to do when you have something to review (you did some
> code changes).
>
> I want to add following to our wiki:
>
> "
> Ask commiter to review changes.
> Check affected file's git history to find commiter most likely able to
> review changes.
> In case it's hard to determine who's able to review by git history use
> maintainers list presented above.
> Add "review request" comment to the ticket starting with a commiter
> username.
>
> for example: "[~avinogradov], Please review my changes."
>
> Commiter will gain notification and review your changes and/or find
> another commiter to do this.
>
> Important: Each comment should be started with [~username].
> "
>
> 2) It will be a huge help to the community!
>
> On Tue, Jun 6, 2017 at 1:12 PM, Dmitry Pavlov 
> wrote:
>
>> Anton,
>>
>>
>> Thank you for explanation. Personal ask instead of group broadcast may
>> really help. I understand the idea now.
>>
>>
>> One argument against solution way 1) it may be not easy for contributor,
>> especially newcomer, to find a right person.
>>
>>
>> What do you think about way 2? Personally, I'm ready to help with analysis
>> and assignment of these 66 tasks from next week.
>>
>>
>>
>> вт, 6 июн. 2017 г. в 12:57, Anton Vinogradov :
>>
>> > Dmitry Pavlov,
>> >
>> > There is *HUGE *difference between "Devlist, please review my changes"
>> > and "Dmitry Pavlov, please review my changes".
>> >
>> > In case you're busy right now, you'll, most likely, ignore appeal to
>> > devlist, but, I'm pretty sure, you'll check appeal to yourself.
>> > Am I right?
>> >
>> > So, my idea is: appeal to devlist is a useless spam maker approach, but
>> > appeal to person(s) works.
>> >
>> > On Tue, Jun 6, 2017 at 2:40 AM, Dmitriy Setrakyan <
>> dsetrak...@apache.org>
>> > wrote:
>> >
>> > > Wow, we have 66 tickets waiting for review this is pretty bad.
>> > > Something must definitely change.
>> > >
>> > > From my side, having a contributor shop around for a reviewer is not
>> > really
>> > > fair. However, I would support the idea of Apache Ignite community
>> > electing
>> > > a person responsible to find reviewers for all contributions.
>> > >
>> > > D.
>> > >
>> > > On Mon, Jun 5, 2017 at 11:23 AM, Dmitry Pavlov > >
>> > > wrote:
>> > >
>> > > > 1) There is waiting for review list here
>> > > > https://cwiki.apache.org/confluence/display/IGNITE/
>> > > > Issues+waiting+for+review
>> > > >
>> > > > 2) some of contributions are supplemented by dev-list messages
>> (please
>> > > > review my PR…)
>> > > >
>> > > >
>> > > > And these two tools sometimes do not help. I suppose it is because
>> of
>> > > > reviewers already have some things to do, but not because of lack of
>> > tool
>> > > > support. Do you have other explanations?
>> > > >
>> > > >
>> > > > But still, Igor’s suggestion to notify to dev list sounds
>> reasonable to
>> > > me.
>> > > >
>> > > >
>> > > >
>> > > > Anton, could it solve your requirement to know about issue needed to
>> > > > review?
>> > > >
>> > > > пн, 5 июн. 2017 г. в 21:06, Igor Sapego :
>> > > >
>> > > > > By the way, there are emails being sent from Jira to devlist every
>> > > > > time someone adds comment to ticket, or, for example, edits its
>> > > > > title which helps a lot to keep a track of ticket's state.
>> > > > >
>> > > > > But by some reason there is no such notification when ticket
>> silently
>> > > > > getting moved to "Patch available" state. I believe, that would
>> help
>> > if
>> > > > > there was a notification for that. Is it possible to configure?
>> > > > >
>> > > > > Best Regards,
>> > > > > Igor
>> > > > >
>> > > > > On Mon, Jun 5, 2017 at 9:00 PM, Denis Magda 
>> > wrote:
>> > > > >
>> > > > > > In general, I tend to agree with Anton that something needs to
>> be
>> > > > changed
>> > > > > > in this direction.
>> > > > > >
>> > > > > > How many of you flip through dev list, JIRA or GitHub
>> notifications
>> > > in
>> > > > an
>> > > > > > attempt to find tickets that demand your attention? I bet the
>> > > > percentage
>> > > > > is
>> > > > > > pretty low.
>> > > > > >
>> > > > > > To solve this issue I see two options:
>> > > > > > 1) Proposed by Anton.
>> > > > > > 2) Having a guy in the community who’ll keep an eye on all the
>> > > incoming
>> > > > > > pull-requests shuffling them between committer in the same way
>> > > proposed
>> > > > > in
>> > > > > > 1.
>> > > > > >
>> > > > > > Personally, I’m for 1.
>> > > > > >
>> > > > > > —
>> > > > > > Denis
>> > > > > >
>> > > > > > > On Jun 5, 2017, at 10:28 AM, Dmitry Pavlov <
>> > dpavlov@gmail.com>
>> > 

[jira] [Created] (IGNITE-5425) JDBC thin: support client info

[Taras Ledkov (JIRA)]

Taras Ledkov created IGNITE-5425:


 Summary: JDBC thin: support client info
 Key: IGNITE-5425
 URL: https://issues.apache.org/jira/browse/IGNITE-5425
 Project: Ignite
  Issue Type: Bug
  Components: sql
Affects Versions: 2.0
Reporter: Taras Ledkov
 Fix For: 2.2


The methods of JDBC {{Connection}} must be supported:
- {{setClientInfo(String,String)}};
- {{getClientInfo(String)}};
- {{setClientInfo(Properties)}};
- {{getClientInfo()}};





--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Data compression in Ignite 2.0

[Vyacheslav Daradur]

In short,

During marshalling a fields is represented as BinaryFieldAccessor which
manages its marshalling. It checks if the field is marked by annotation
@BinaryCompression, in that case - binary  representation of field (bytes
array) will be compressed. It will be marked as compressed by types
constant (GridBinaryMarshaller.COMPRESSED), after this the compressed bytes
array wiil be include in binary representation of whole object. Note,
header of marshalled object will not be compressed. Compression affected
only object's field representation.

Objects in IgniteCache is represented as BinaryObject which is wrapper over
bytes array of marshalled object.
BinaryObject provides some usefull methods, which are used by Ignite
systems.
For example, the Queries use BinaryObject#field method, which deserializes
only field of object, without deserializing of whole object.
BinaryObject#field method during deserialization, if meets the constant of
compressed type, decompress this bytes array, then continue unmarshalling
as usual.

Now, I introduced the Compressor interface in IgniteConfigurations, it
allows user to use own implementation of compressor - it is the requirement
in the task[1].

As far as I know, Vladimir Ozerov doesn't like the idea of granting this
opportunity to the user.
In that case we can choose a compression algorithm which we will provide by
default and will move the interface to internals of binary infractructure.
For this case I've prepared benchmarked, which I've sent earlier.

I vote for ZSTD algorithm[2], it provides good compression ratio and good
throughput. It has implementation in Java, .NET and C++, and has
ASF-friendly license, we can use it in the all Ignite platforms.
You can look at an assessment of this algorithm in my benchmark's

[1] https://issues.apache.org/jira/browse/IGNITE-3592
[2]https://github.com/facebook/zstd


2017-06-06 16:02 GMT+03:00 Антон Чураев :

> Looks good for me.
>
> Could You propose design of implementation in couple of sentences?
> So that we can estimate the completeness and complexity of the proposal.
>
> 2017-06-06 15:26 GMT+03:00 Vyacheslav Daradur :
>
> > Anton,
> >
> > Of course, the solution does not affect on existing implementation. I
> mean,
> > there is no changes if user not use the annotation @BinaryCompression.
> (no
> > performance changes)
> > Only if user make decision to use compression on specific field or fields
> > of a class - in that case compression will be used at marshalling in
> > relation to annotated fields.
> >
> > 2017-06-06 15:10 GMT+03:00 Антон Чураев :
> >
> > > Vyacheslav,
> > >
> > > Is it possible to propose implementation that can be switched on
> > on-demand?
> > > In this case it should not affect performance of current solution.
> > >
> > > I mean, that users should make decision what is more important for
> them:
> > > throutput or memory/net usage.
> > > May be they will be choose not all objects, or only some attributes of
> > > objects for compress.
> > >
> > > 2017-06-06 14:48 GMT+03:00 Vyacheslav Daradur :
> > >
> > > > Conclusion:
> > > > Provided solution allows reduce size of an object in IgniteCache at
> the
> > > > cost of throughput reduction (small - in some cases), it depends on
> > part
> > > of
> > > > object which will be compressed and compression algorithm.
> > > > I mean, we can make more effective use of memory, and in some cases
> it
> > > can
> > > > reduce loading of the interconnect. (replication, rebalancing)
> > > >
> > > > Especially, it will be particularly useful for object's fields which
> > are
> > > > large text (>~ 250 bytes) and can be effectively compressed.
> > > >
> > > > 2017-06-06 12:00 GMT+03:00 Антон Чураев :
> > > >
> > > > > Vyacheslav, thank you! But could you please provide a conclusions
> or
> > > > > proposals based on this benchmarks?
> > > > >
> > > > > 2017-06-06 11:28 GMT+03:00 Vyacheslav Daradur  >:
> > > > >
> > > > > > Dmitry,
> > > > > >
> > > > > > Excel-pages:
> > > > > >
> > > > > > 1). "Compression ratio (2)" - shows object size, with compression
> > and
> > > > > > without compression. (Conditions: literal text)
> > > > > > 1st graph shows compression ratios of using different compression
> > > > > algrithms
> > > > > > depending on size of compressed field.
> > > > > > 2nd graph shows evaluation of size of objects depending on sizes
> > and
> > > > > > compression algorithms.
> > > > > >
> > > > > > 2). "Compression ratio (1)" - shows object size, with compression
> > and
> > > > > > without compression. (Conditions:  badly compressed character
> > > sequence)
> > > > > > 1st graph shows compression ratios of using different compression
> > > > > > algrithms depending on size of compressed field.
> > > > > > 2nd graph shows evaluation of size of objects depending on sizes
> > and
> > > > > > compression algorithms.
> > > > > >

[GitHub] ignite pull request #2084: Ignite 5410

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2084


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Data compression in Ignite 2.0

[Антон Чураев]

Looks good for me.

Could You propose design of implementation in couple of sentences?
So that we can estimate the completeness and complexity of the proposal.

2017-06-06 15:26 GMT+03:00 Vyacheslav Daradur :

> Anton,
>
> Of course, the solution does not affect on existing implementation. I mean,
> there is no changes if user not use the annotation @BinaryCompression. (no
> performance changes)
> Only if user make decision to use compression on specific field or fields
> of a class - in that case compression will be used at marshalling in
> relation to annotated fields.
>
> 2017-06-06 15:10 GMT+03:00 Антон Чураев :
>
> > Vyacheslav,
> >
> > Is it possible to propose implementation that can be switched on
> on-demand?
> > In this case it should not affect performance of current solution.
> >
> > I mean, that users should make decision what is more important for them:
> > throutput or memory/net usage.
> > May be they will be choose not all objects, or only some attributes of
> > objects for compress.
> >
> > 2017-06-06 14:48 GMT+03:00 Vyacheslav Daradur :
> >
> > > Conclusion:
> > > Provided solution allows reduce size of an object in IgniteCache at the
> > > cost of throughput reduction (small - in some cases), it depends on
> part
> > of
> > > object which will be compressed and compression algorithm.
> > > I mean, we can make more effective use of memory, and in some cases it
> > can
> > > reduce loading of the interconnect. (replication, rebalancing)
> > >
> > > Especially, it will be particularly useful for object's fields which
> are
> > > large text (>~ 250 bytes) and can be effectively compressed.
> > >
> > > 2017-06-06 12:00 GMT+03:00 Антон Чураев :
> > >
> > > > Vyacheslav, thank you! But could you please provide a conclusions or
> > > > proposals based on this benchmarks?
> > > >
> > > > 2017-06-06 11:28 GMT+03:00 Vyacheslav Daradur :
> > > >
> > > > > Dmitry,
> > > > >
> > > > > Excel-pages:
> > > > >
> > > > > 1). "Compression ratio (2)" - shows object size, with compression
> and
> > > > > without compression. (Conditions: literal text)
> > > > > 1st graph shows compression ratios of using different compression
> > > > algrithms
> > > > > depending on size of compressed field.
> > > > > 2nd graph shows evaluation of size of objects depending on sizes
> and
> > > > > compression algorithms.
> > > > >
> > > > > 2). "Compression ratio (1)" - shows object size, with compression
> and
> > > > > without compression. (Conditions:  badly compressed character
> > sequence)
> > > > > 1st graph shows compression ratios of using different compression
> > > > > algrithms depending on size of compressed field.
> > > > > 2nd graph shows evaluation of size of objects depending on sizes
> and
> > > > > compression algorithms.
> > > > >
> > > > > 3) 'put-avg" - shows average time of the "put" operation depending
> on
> > > > size
> > > > > and compression algorithms.
> > > > >
> > > > > 4) 'put-thrpt" - shows throughput of the "put" operation depending
> on
> > > > size
> > > > > and compression algorithms.
> > > > >
> > > > > 5) 'get-avg" - shows average time of the "get" operation depending
> on
> > > > size
> > > > > and compression algorithms.
> > > > >
> > > > > 6) 'get-thrpt" - shows throughput of the "get" operation depending
> on
> > > > size
> > > > > and compression algorithms.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > 2017-06-06 10:59 GMT+03:00 Dmitriy Setrakyan <
> dsetrak...@apache.org
> > >:
> > > > >
> > > > > > Vladimir, I am not sure how to interpret the graphs? What are we
> > > > looking
> > > > > > at?
> > > > > >
> > > > > > On Tue, Jun 6, 2017 at 12:33 AM, Vyacheslav Daradur <
> > > > daradu...@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi, Igniters.
> > > > > > >
> > > > > > > I've prepared some benchmarking. Results [1].
> > > > > > >
> > > > > > > And I've prepared the evaluation in the form of diagrams [2].
> > > > > > >
> > > > > > > I hope that helps to interest the community and accelerates a
> > > > reaction
> > > > > to
> > > > > > > this improvment :)
> > > > > > >
> > > > > > > [1]
> > > > > > > https://github.com/daradurvs/ignite-compression/tree/
> > > > > > > master/src/main/resources/result
> > > > > > > [2] https://drive.google.com/file/d/
> > 0B2CeUAOgrHkoMklyZ25YTEdKcEk/
> > > > view
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > 2017-05-24 9:49 GMT+03:00 Vyacheslav Daradur <
> > daradu...@gmail.com
> > > >:
> > > > > > >
> > > > > > > > Guys, any thoughts?
> > > > > > > >
> > > > > > > > 2017-05-16 13:40 GMT+03:00 Vyacheslav Daradur <
> > > daradu...@gmail.com
> > > > >:
> > > > > > > >
> > > > > > > >> Hi guys,
> > > > > > > >>
> > > > > > > >> I've prepared the PR to show my idea.
> > > > > > > >> https://github.com/apache/ignite/pull/1951/files
> > > > > > > >>
> > > > > > > >> About querying - I've just copied existing 

[GitHub] ignite pull request #2094: IGNITE-5306 persist sql flag

[kdudkov]

GitHub user kdudkov opened a pull request:

https://github.com/apache/ignite/pull/2094

IGNITE-5306 persist sql flag



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-5306

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2094.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2094


commit cc4e3147d11416d29849a4173dde96e4fd31116b
Author: Konstantin Dudkov 
Date:   2017-06-06T12:02:28Z

IGNITE-5306 persist sql flag




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Data compression in Ignite 2.0

[Антон Чураев]

Vyacheslav,

Is it possible to propose implementation that can be switched on on-demand?
In this case it should not affect performance of current solution.

I mean, that users should make decision what is more important for them:
throutput or memory/net usage.
May be they will be choose not all objects, or only some attributes of
objects for compress.

2017-06-06 14:48 GMT+03:00 Vyacheslav Daradur :

> Conclusion:
> Provided solution allows reduce size of an object in IgniteCache at the
> cost of throughput reduction (small - in some cases), it depends on part of
> object which will be compressed and compression algorithm.
> I mean, we can make more effective use of memory, and in some cases it can
> reduce loading of the interconnect. (replication, rebalancing)
>
> Especially, it will be particularly useful for object's fields which are
> large text (>~ 250 bytes) and can be effectively compressed.
>
> 2017-06-06 12:00 GMT+03:00 Антон Чураев :
>
> > Vyacheslav, thank you! But could you please provide a conclusions or
> > proposals based on this benchmarks?
> >
> > 2017-06-06 11:28 GMT+03:00 Vyacheslav Daradur :
> >
> > > Dmitry,
> > >
> > > Excel-pages:
> > >
> > > 1). "Compression ratio (2)" - shows object size, with compression and
> > > without compression. (Conditions: literal text)
> > > 1st graph shows compression ratios of using different compression
> > algrithms
> > > depending on size of compressed field.
> > > 2nd graph shows evaluation of size of objects depending on sizes and
> > > compression algorithms.
> > >
> > > 2). "Compression ratio (1)" - shows object size, with compression and
> > > without compression. (Conditions:  badly compressed character sequence)
> > > 1st graph shows compression ratios of using different compression
> > > algrithms depending on size of compressed field.
> > > 2nd graph shows evaluation of size of objects depending on sizes and
> > > compression algorithms.
> > >
> > > 3) 'put-avg" - shows average time of the "put" operation depending on
> > size
> > > and compression algorithms.
> > >
> > > 4) 'put-thrpt" - shows throughput of the "put" operation depending on
> > size
> > > and compression algorithms.
> > >
> > > 5) 'get-avg" - shows average time of the "get" operation depending on
> > size
> > > and compression algorithms.
> > >
> > > 6) 'get-thrpt" - shows throughput of the "get" operation depending on
> > size
> > > and compression algorithms.
> > >
> > >
> > >
> > >
> > > 2017-06-06 10:59 GMT+03:00 Dmitriy Setrakyan :
> > >
> > > > Vladimir, I am not sure how to interpret the graphs? What are we
> > looking
> > > > at?
> > > >
> > > > On Tue, Jun 6, 2017 at 12:33 AM, Vyacheslav Daradur <
> > daradu...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi, Igniters.
> > > > >
> > > > > I've prepared some benchmarking. Results [1].
> > > > >
> > > > > And I've prepared the evaluation in the form of diagrams [2].
> > > > >
> > > > > I hope that helps to interest the community and accelerates a
> > reaction
> > > to
> > > > > this improvment :)
> > > > >
> > > > > [1]
> > > > > https://github.com/daradurvs/ignite-compression/tree/
> > > > > master/src/main/resources/result
> > > > > [2] https://drive.google.com/file/d/0B2CeUAOgrHkoMklyZ25YTEdKcEk/
> > view
> > > > >
> > > > >
> > > > >
> > > > > 2017-05-24 9:49 GMT+03:00 Vyacheslav Daradur  >:
> > > > >
> > > > > > Guys, any thoughts?
> > > > > >
> > > > > > 2017-05-16 13:40 GMT+03:00 Vyacheslav Daradur <
> daradu...@gmail.com
> > >:
> > > > > >
> > > > > >> Hi guys,
> > > > > >>
> > > > > >> I've prepared the PR to show my idea.
> > > > > >> https://github.com/apache/ignite/pull/1951/files
> > > > > >>
> > > > > >> About querying - I've just copied existing tests and have
> > annotated
> > > > the
> > > > > >> testing data.
> > > > > >> https://github.com/apache/ignite/pull/1951/files#diff-c19a9d
> > > > > >> f4058141d059bb577e75244764
> > > > > >>
> > > > > >> It means fields which will be marked by @BinaryCompression will
> be
> > > > > >> compressed at marshalling via BinaryMarshaller.
> > > > > >>
> > > > > >> This solution has no effect on existing data or project
> > > architecture.
> > > > > >>
> > > > > >> I'll be glad to see your thougths.
> > > > > >>
> > > > > >>
> > > > > >> 2017-05-15 19:18 GMT+03:00 Vyacheslav Daradur <
> > daradu...@gmail.com
> > > >:
> > > > > >>
> > > > > >>> Dmitriy,
> > > > > >>>
> > > > > >>> I have ready prototype. I want to show it.
> > > > > >>> It is always easier to discuss on example.
> > > > > >>>
> > > > > >>> 2017-05-15 19:02 GMT+03:00 Dmitriy Setrakyan <
> > > dsetrak...@apache.org
> > > > >:
> > > > > >>>
> > > > >  Vyacheslav,
> > > > > 
> > > > >  I think it is a bit premature to provide a PR without getting
> a
> > > > >  community
> > > > >  consensus on the dev list. Please allow some time for the
> > > community
> > > > to
> > 

Re: Data compression in Ignite 2.0

[Vyacheslav Daradur]

I wish to note, that the results of benchmarking shows metrics from
stress-testing.
I mean in real scenarios, for example business operations, which take
milliseconds or seсonds, increase in time of put-get-operation will be
insignificant.

2017-06-06 14:48 GMT+03:00 Vyacheslav Daradur :

> Conclusion:
> Provided solution allows reduce size of an object in IgniteCache at the
> cost of throughput reduction (small - in some cases), it depends on part of
> object which will be compressed and compression algorithm.
> I mean, we can make more effective use of memory, and in some cases it can
> reduce loading of the interconnect. (replication, rebalancing)
>
> Especially, it will be particularly useful for object's fields which are
> large text (>~ 250 bytes) and can be effectively compressed.
>
> 2017-06-06 12:00 GMT+03:00 Антон Чураев :
>
>> Vyacheslav, thank you! But could you please provide a conclusions or
>> proposals based on this benchmarks?
>>
>> 2017-06-06 11:28 GMT+03:00 Vyacheslav Daradur :
>>
>> > Dmitry,
>> >
>> > Excel-pages:
>> >
>> > 1). "Compression ratio (2)" - shows object size, with compression and
>> > without compression. (Conditions: literal text)
>> > 1st graph shows compression ratios of using different compression
>> algrithms
>> > depending on size of compressed field.
>> > 2nd graph shows evaluation of size of objects depending on sizes and
>> > compression algorithms.
>> >
>> > 2). "Compression ratio (1)" - shows object size, with compression and
>> > without compression. (Conditions:  badly compressed character sequence)
>> > 1st graph shows compression ratios of using different compression
>> > algrithms depending on size of compressed field.
>> > 2nd graph shows evaluation of size of objects depending on sizes and
>> > compression algorithms.
>> >
>> > 3) 'put-avg" - shows average time of the "put" operation depending on
>> size
>> > and compression algorithms.
>> >
>> > 4) 'put-thrpt" - shows throughput of the "put" operation depending on
>> size
>> > and compression algorithms.
>> >
>> > 5) 'get-avg" - shows average time of the "get" operation depending on
>> size
>> > and compression algorithms.
>> >
>> > 6) 'get-thrpt" - shows throughput of the "get" operation depending on
>> size
>> > and compression algorithms.
>> >
>> >
>> >
>> >
>> > 2017-06-06 10:59 GMT+03:00 Dmitriy Setrakyan :
>> >
>> > > Vladimir, I am not sure how to interpret the graphs? What are we
>> looking
>> > > at?
>> > >
>> > > On Tue, Jun 6, 2017 at 12:33 AM, Vyacheslav Daradur <
>> daradu...@gmail.com
>> > >
>> > > wrote:
>> > >
>> > > > Hi, Igniters.
>> > > >
>> > > > I've prepared some benchmarking. Results [1].
>> > > >
>> > > > And I've prepared the evaluation in the form of diagrams [2].
>> > > >
>> > > > I hope that helps to interest the community and accelerates a
>> reaction
>> > to
>> > > > this improvment :)
>> > > >
>> > > > [1]
>> > > > https://github.com/daradurvs/ignite-compression/tree/
>> > > > master/src/main/resources/result
>> > > > [2] https://drive.google.com/file/d/0B2CeUAOgrHkoMklyZ25YTEdKcEk
>> /view
>> > > >
>> > > >
>> > > >
>> > > > 2017-05-24 9:49 GMT+03:00 Vyacheslav Daradur :
>> > > >
>> > > > > Guys, any thoughts?
>> > > > >
>> > > > > 2017-05-16 13:40 GMT+03:00 Vyacheslav Daradur <
>> daradu...@gmail.com>:
>> > > > >
>> > > > >> Hi guys,
>> > > > >>
>> > > > >> I've prepared the PR to show my idea.
>> > > > >> https://github.com/apache/ignite/pull/1951/files
>> > > > >>
>> > > > >> About querying - I've just copied existing tests and have
>> annotated
>> > > the
>> > > > >> testing data.
>> > > > >> https://github.com/apache/ignite/pull/1951/files#diff-c19a9d
>> > > > >> f4058141d059bb577e75244764
>> > > > >>
>> > > > >> It means fields which will be marked by @BinaryCompression will
>> be
>> > > > >> compressed at marshalling via BinaryMarshaller.
>> > > > >>
>> > > > >> This solution has no effect on existing data or project
>> > architecture.
>> > > > >>
>> > > > >> I'll be glad to see your thougths.
>> > > > >>
>> > > > >>
>> > > > >> 2017-05-15 19:18 GMT+03:00 Vyacheslav Daradur <
>> daradu...@gmail.com
>> > >:
>> > > > >>
>> > > > >>> Dmitriy,
>> > > > >>>
>> > > > >>> I have ready prototype. I want to show it.
>> > > > >>> It is always easier to discuss on example.
>> > > > >>>
>> > > > >>> 2017-05-15 19:02 GMT+03:00 Dmitriy Setrakyan <
>> > dsetrak...@apache.org
>> > > >:
>> > > > >>>
>> > > >  Vyacheslav,
>> > > > 
>> > > >  I think it is a bit premature to provide a PR without getting a
>> > > >  community
>> > > >  consensus on the dev list. Please allow some time for the
>> > community
>> > > to
>> > > >  respond.
>> > > > 
>> > > >  D.
>> > > > 
>> > > >  On Mon, May 15, 2017 at 6:36 AM, Vyacheslav Daradur <
>> > > >  daradu...@gmail.com>
>> > > >  wrote:
>> > > > 
>> > > >  > I created the ticket: 

[jira] [Created] (IGNITE-5424) GridServiceProxy does not unwraps exception message from InvocationTargetException

[Dmitry Karachentsev (JIRA)]

Dmitry Karachentsev created IGNITE-5424:
---

 Summary: GridServiceProxy does not unwraps exception message from 
InvocationTargetException
 Key: IGNITE-5424
 URL: https://issues.apache.org/jira/browse/IGNITE-5424
 Project: Ignite
  Issue Type: Bug
  Components: managed services
Affects Versions: 2.1
Reporter: Dmitry Karachentsev
Assignee: Dmitry Karachentsev
 Fix For: 2.2


Instead of correct message 'null' passed.

{noformat}
class org.apache.ignite.IgniteException: null
at 
org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:957)
at 
org.apache.ignite.internal.processors.service.GridServiceProxy.invokeMethod(GridServiceProxy.java:208)
at 
org.apache.ignite.internal.processors.service.GridServiceProxy$ProxyInvocationHandler.invoke(GridServiceProxy.java:356)
at org.apache.ignite.internal.processors.service.$Proxy25.go(Unknown 
Source)
at 
org.apache.ignite.internal.processors.service.GridServiceProcessorProxySelfTest.testException(GridServiceProcessorProxySelfTest.java:78)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at junit.framework.TestCase.runTest(TestCase.java:176)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:1979)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:130)
at 
org.apache.ignite.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:1894)
at java.lang.Thread.run(Thread.java:745)
Caused by: class org.apache.ignite.IgniteCheckedException: null
at 
org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7262)
at 
org.apache.ignite.internal.util.future.GridFutureAdapter.resolve(GridFutureAdapter.java:258)
at 
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:189)
at 
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:139)
at 
org.apache.ignite.internal.processors.service.GridServiceProxy.invokeMethod(GridServiceProxy.java:197)
... 12 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.ignite.internal.processors.service.GridServiceProxy$ServiceProxyCallable.call(GridServiceProxy.java:417)
at 
org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2.execute(GridClosureProcessor.java:1847)
at 
org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:566)
at 
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6641)
at 
org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:560)
at 
org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:489)
at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at 
org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1180)
at 
org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1907)
at 
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1550)
at 
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1178)
at 
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:124)
at 
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1095)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more
Caused by: java.lang.Exception: Test exception
at 
org.apache.ignite.internal.processors.service.GridServiceProcessorAbstractSelfTest$ErrorServiceImpl.go(GridServiceProcessorAbstractSelfTest.java:823)
... 20 more
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (IGNITE-5423) Web Console: We need configure agents remotely

[Alexey Kuznetsov (JIRA)]

Alexey Kuznetsov created IGNITE-5423:


 Summary: Web Console: We need configure agents remotely
 Key: IGNITE-5423
 URL: https://issues.apache.org/jira/browse/IGNITE-5423
 Project: Ignite
  Issue Type: Task
  Components: wizards
Affects Versions: 1.7
Reporter: Alexey Kuznetsov
 Fix For: 2.2


For example: change tokens list, node rest-http host and port



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Data compression in Ignite 2.0

[Vyacheslav Daradur]

Conclusion:
Provided solution allows reduce size of an object in IgniteCache at the
cost of throughput reduction (small - in some cases), it depends on part of
object which will be compressed and compression algorithm.
I mean, we can make more effective use of memory, and in some cases it can
reduce loading of the interconnect. (replication, rebalancing)

Especially, it will be particularly useful for object's fields which are
large text (>~ 250 bytes) and can be effectively compressed.

2017-06-06 12:00 GMT+03:00 Антон Чураев :

> Vyacheslav, thank you! But could you please provide a conclusions or
> proposals based on this benchmarks?
>
> 2017-06-06 11:28 GMT+03:00 Vyacheslav Daradur :
>
> > Dmitry,
> >
> > Excel-pages:
> >
> > 1). "Compression ratio (2)" - shows object size, with compression and
> > without compression. (Conditions: literal text)
> > 1st graph shows compression ratios of using different compression
> algrithms
> > depending on size of compressed field.
> > 2nd graph shows evaluation of size of objects depending on sizes and
> > compression algorithms.
> >
> > 2). "Compression ratio (1)" - shows object size, with compression and
> > without compression. (Conditions:  badly compressed character sequence)
> > 1st graph shows compression ratios of using different compression
> > algrithms depending on size of compressed field.
> > 2nd graph shows evaluation of size of objects depending on sizes and
> > compression algorithms.
> >
> > 3) 'put-avg" - shows average time of the "put" operation depending on
> size
> > and compression algorithms.
> >
> > 4) 'put-thrpt" - shows throughput of the "put" operation depending on
> size
> > and compression algorithms.
> >
> > 5) 'get-avg" - shows average time of the "get" operation depending on
> size
> > and compression algorithms.
> >
> > 6) 'get-thrpt" - shows throughput of the "get" operation depending on
> size
> > and compression algorithms.
> >
> >
> >
> >
> > 2017-06-06 10:59 GMT+03:00 Dmitriy Setrakyan :
> >
> > > Vladimir, I am not sure how to interpret the graphs? What are we
> looking
> > > at?
> > >
> > > On Tue, Jun 6, 2017 at 12:33 AM, Vyacheslav Daradur <
> daradu...@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi, Igniters.
> > > >
> > > > I've prepared some benchmarking. Results [1].
> > > >
> > > > And I've prepared the evaluation in the form of diagrams [2].
> > > >
> > > > I hope that helps to interest the community and accelerates a
> reaction
> > to
> > > > this improvment :)
> > > >
> > > > [1]
> > > > https://github.com/daradurvs/ignite-compression/tree/
> > > > master/src/main/resources/result
> > > > [2] https://drive.google.com/file/d/0B2CeUAOgrHkoMklyZ25YTEdKcEk/
> view
> > > >
> > > >
> > > >
> > > > 2017-05-24 9:49 GMT+03:00 Vyacheslav Daradur :
> > > >
> > > > > Guys, any thoughts?
> > > > >
> > > > > 2017-05-16 13:40 GMT+03:00 Vyacheslav Daradur  >:
> > > > >
> > > > >> Hi guys,
> > > > >>
> > > > >> I've prepared the PR to show my idea.
> > > > >> https://github.com/apache/ignite/pull/1951/files
> > > > >>
> > > > >> About querying - I've just copied existing tests and have
> annotated
> > > the
> > > > >> testing data.
> > > > >> https://github.com/apache/ignite/pull/1951/files#diff-c19a9d
> > > > >> f4058141d059bb577e75244764
> > > > >>
> > > > >> It means fields which will be marked by @BinaryCompression will be
> > > > >> compressed at marshalling via BinaryMarshaller.
> > > > >>
> > > > >> This solution has no effect on existing data or project
> > architecture.
> > > > >>
> > > > >> I'll be glad to see your thougths.
> > > > >>
> > > > >>
> > > > >> 2017-05-15 19:18 GMT+03:00 Vyacheslav Daradur <
> daradu...@gmail.com
> > >:
> > > > >>
> > > > >>> Dmitriy,
> > > > >>>
> > > > >>> I have ready prototype. I want to show it.
> > > > >>> It is always easier to discuss on example.
> > > > >>>
> > > > >>> 2017-05-15 19:02 GMT+03:00 Dmitriy Setrakyan <
> > dsetrak...@apache.org
> > > >:
> > > > >>>
> > > >  Vyacheslav,
> > > > 
> > > >  I think it is a bit premature to provide a PR without getting a
> > > >  community
> > > >  consensus on the dev list. Please allow some time for the
> > community
> > > to
> > > >  respond.
> > > > 
> > > >  D.
> > > > 
> > > >  On Mon, May 15, 2017 at 6:36 AM, Vyacheslav Daradur <
> > > >  daradu...@gmail.com>
> > > >  wrote:
> > > > 
> > > >  > I created the ticket: https://issues.apache.org/jira
> > > >  /browse/IGNITE-5226
> > > >  >
> > > >  > I'll prepare a PR with described solution in couple of days.
> > > >  >
> > > >  > 2017-05-15 15:05 GMT+03:00 Vyacheslav Daradur <
> > > daradu...@gmail.com
> > > > >:
> > > >  >
> > > >  > > Hi, Igniters!
> > > >  > >
> > > >  > > Apache 2.0 is released.
> > > >  > >
> > > >  > > Let's continue the discussion about a compression design.
> 

is IGNITE-3142 actual now

[Вадим Опольский]

Hello guys!

I want to resolve issue https://issues.apache.org/jira/browse/IGNITE-3142

Is it actual now?

Vadim Opolski

[GitHub] ignite pull request #2093: IGNITE-5420

[devozerov]

GitHub user devozerov opened a pull request:

https://github.com/apache/ignite/pull/2093

IGNITE-5420



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-5420

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2093.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2093


commit 30676c67c61caebcb5be9d885a4c00b3a585bea9
Author: devozerov 
Date:   2017-06-06T10:59:29Z

Done.

commit 90165393b6415792ed9d6fc45b8a5408afae04c6
Author: devozerov 
Date:   2017-06-06T11:02:47Z

WIP




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Only lateAffinityAssignment for 2.1

[Yakov Zhdanov]

Absolutely agree.

Please make sure that we switch affinity version only when partition N
migrates from current node for all caches. Makes sense?

--Yakov

Re: Key type name and value type name for CREATE TABLE

[Sergi Vladykin]

Unique suffix is a good idea.

Sergi

2017-06-06 13:51 GMT+03:00 Vladimir Ozerov :

> Igniters,
>
> In the very first implementation of CREATE TABLE we applied the following
> rule to key and value type names:
> keyTypeName == tableName + "Key"
> valTypeName == tableName
>
> E.g.:
> CREATE TABLE Person ...
> keyTypeName == PERSONKey
> valTypeName == PERSON
>
> After that user could potentially create objects with these type names
> manually and add them to cache through native Ignite API:
>
> BinaryObject key = IgniteBinary.builder("PERSONKey").addField().build();
> BinaryObject val = IgniteBinary.builder("PERSON").addField().build();
> IgniteCache.put(key, val);
>
> This approach has two problems:
> 1) Type names are not unique between different tables. it means that if two
> tables with the same name are created in different schemas, we got a
> conflict.
> 2) Type names are bound to binary metadata, so if user decides to do the
> following, he will receive and error about incompatible metadata:
> CREATE TABLE Person (id INT PRIMARY KEY);
> DROP TABLE Person;
> CREATE TABLE Person(in BIGINT PRIMARY KEY); // Fail because old meta still
> has type "Integer".
>
> In order to avoid that I am going to add unique suffix or so (say, UUID) to
> type names. This way there will be no human-readable type names any more,
> but there will be no conflicts either. In future releases we will relax
> this somehow.
>
> Thoughts?
>
> Vladimir.
>

[jira] [Created] (IGNITE-5422) CREATE TABLE command should support all-key fields case

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-5422:
---

 Summary: CREATE TABLE command should support all-key fields case
 Key: IGNITE-5422
 URL: https://issues.apache.org/jira/browse/IGNITE-5422
 Project: Ignite
  Issue Type: Task
  Components: sql
Reporter: Vladimir Ozerov
Assignee: Alexander Paschenko
 Fix For: 2.2


Currently it is impossible to create a table containing only {{PRIMARY KEY}} 
columns:
{code}
CREATE TABLE PersonProject {
person_id BIGINT,
project_id BIGINT,
PRIMARY KEY (person_id, project_id);
};
{code}
The reason for this limitation is that we have nothing to be saved as cache 
value and {{null}} values are restricted because they are indistinguishable 
from removal.

We should allow this case somehow. Possible solutions:
1) Save empty object of table type: 
{{IgniteBinary.builder("PersonProject").build()}}
2) Save {{false}} as it is done in Java in Map-to-Set conversion

Note that DML processing should be adjusted accordingly.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Key type name and value type name for CREATE TABLE

[Vladimir Ozerov]

Igniters,

In the very first implementation of CREATE TABLE we applied the following
rule to key and value type names:
keyTypeName == tableName + "Key"
valTypeName == tableName

E.g.:
CREATE TABLE Person ...
keyTypeName == PERSONKey
valTypeName == PERSON

After that user could potentially create objects with these type names
manually and add them to cache through native Ignite API:

BinaryObject key = IgniteBinary.builder("PERSONKey").addField().build();
BinaryObject val = IgniteBinary.builder("PERSON").addField().build();
IgniteCache.put(key, val);

This approach has two problems:
1) Type names are not unique between different tables. it means that if two
tables with the same name are created in different schemas, we got a
conflict.
2) Type names are bound to binary metadata, so if user decides to do the
following, he will receive and error about incompatible metadata:
CREATE TABLE Person (id INT PRIMARY KEY);
DROP TABLE Person;
CREATE TABLE Person(in BIGINT PRIMARY KEY); // Fail because old meta still
has type "Integer".

In order to avoid that I am going to add unique suffix or so (say, UUID) to
type names. This way there will be no human-readable type names any more,
but there will be no conflicts either. In future releases we will relax
this somehow.

Thoughts?

Vladimir.

Only lateAffinityAssignment for 2.1

[Semyon Boikov]

Hi all,

Currently in Igntie we have two modes for cache affinity changing
(IgniteConfiguration.isLateAffinityAssignment=true/false). Late affinity
assignment mode is enabled by default and I do not see when it can be
useful to disable it. Does it make sense for 2.1 to deprecate property
IgniteConfiguration.isLateAffinityAssignment and internally always use late
affinity assignment mode? This will allow to simplify cache code (e.g. get
rid of ForceKeyRequests/Responses).

What do you think?

Thanks!

[jira] [Created] (IGNITE-5421) Web Console: Implement design how to handle lost connection

[Vica Abramova (JIRA)]

Vica Abramova created IGNITE-5421:
-

 Summary: Web Console: Implement design how to handle lost 
connection
 Key: IGNITE-5421
 URL: https://issues.apache.org/jira/browse/IGNITE-5421
 Project: Ignite
  Issue Type: Task
  Components: UI, wizards
Reporter: Vica Abramova






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (IGNITE-5420) Table key type name and value type must be unique

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-5420:
---

 Summary: Table key type name and value type must be unique
 Key: IGNITE-5420
 URL: https://issues.apache.org/jira/browse/IGNITE-5420
 Project: Ignite
  Issue Type: Task
  Components: sql
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
 Fix For: 2.1


We need to make sure that key and value type names assigned to query entity are 
globally unique between all tables in the cluster. Otherwise, we may have 
conflicts between tables in different schemas with equal names (not the case 
for 2.1), or within the same table, when we do CREATE -> DROP -> CREATE and 
certain column type is changes.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: "Review workflow" changes to prevent "broken review" issues.

[Anton Vinogradov]

Dmitry,

1) See my initial email, it contains instruction how to find a reviewer.
And it's pretty easy to do when you have something to review (you did some
code changes).

I want to add following to our wiki:

"
Ask commiter to review changes.
Check affected file's git history to find commiter most likely able to
review changes.
In case it's hard to determine who's able to review by git history use
maintainers list presented above.
Add "review request" comment to the ticket starting with a commiter
username.

for example: "[~avinogradov], Please review my changes."

Commiter will gain notification and review your changes and/or find another
commiter to do this.

Important: Each comment should be started with [~username].
"

2) It will be a huge help to the community!

On Tue, Jun 6, 2017 at 1:12 PM, Dmitry Pavlov  wrote:

> Anton,
>
>
> Thank you for explanation. Personal ask instead of group broadcast may
> really help. I understand the idea now.
>
>
> One argument against solution way 1) it may be not easy for contributor,
> especially newcomer, to find a right person.
>
>
> What do you think about way 2? Personally, I'm ready to help with analysis
> and assignment of these 66 tasks from next week.
>
>
>
> вт, 6 июн. 2017 г. в 12:57, Anton Vinogradov :
>
> > Dmitry Pavlov,
> >
> > There is *HUGE *difference between "Devlist, please review my changes"
> > and "Dmitry Pavlov, please review my changes".
> >
> > In case you're busy right now, you'll, most likely, ignore appeal to
> > devlist, but, I'm pretty sure, you'll check appeal to yourself.
> > Am I right?
> >
> > So, my idea is: appeal to devlist is a useless spam maker approach, but
> > appeal to person(s) works.
> >
> > On Tue, Jun 6, 2017 at 2:40 AM, Dmitriy Setrakyan  >
> > wrote:
> >
> > > Wow, we have 66 tickets waiting for review this is pretty bad.
> > > Something must definitely change.
> > >
> > > From my side, having a contributor shop around for a reviewer is not
> > really
> > > fair. However, I would support the idea of Apache Ignite community
> > electing
> > > a person responsible to find reviewers for all contributions.
> > >
> > > D.
> > >
> > > On Mon, Jun 5, 2017 at 11:23 AM, Dmitry Pavlov 
> > > wrote:
> > >
> > > > 1) There is waiting for review list here
> > > > https://cwiki.apache.org/confluence/display/IGNITE/
> > > > Issues+waiting+for+review
> > > >
> > > > 2) some of contributions are supplemented by dev-list messages
> (please
> > > > review my PR…)
> > > >
> > > >
> > > > And these two tools sometimes do not help. I suppose it is because of
> > > > reviewers already have some things to do, but not because of lack of
> > tool
> > > > support. Do you have other explanations?
> > > >
> > > >
> > > > But still, Igor’s suggestion to notify to dev list sounds reasonable
> to
> > > me.
> > > >
> > > >
> > > >
> > > > Anton, could it solve your requirement to know about issue needed to
> > > > review?
> > > >
> > > > пн, 5 июн. 2017 г. в 21:06, Igor Sapego :
> > > >
> > > > > By the way, there are emails being sent from Jira to devlist every
> > > > > time someone adds comment to ticket, or, for example, edits its
> > > > > title which helps a lot to keep a track of ticket's state.
> > > > >
> > > > > But by some reason there is no such notification when ticket
> silently
> > > > > getting moved to "Patch available" state. I believe, that would
> help
> > if
> > > > > there was a notification for that. Is it possible to configure?
> > > > >
> > > > > Best Regards,
> > > > > Igor
> > > > >
> > > > > On Mon, Jun 5, 2017 at 9:00 PM, Denis Magda 
> > wrote:
> > > > >
> > > > > > In general, I tend to agree with Anton that something needs to be
> > > > changed
> > > > > > in this direction.
> > > > > >
> > > > > > How many of you flip through dev list, JIRA or GitHub
> notifications
> > > in
> > > > an
> > > > > > attempt to find tickets that demand your attention? I bet the
> > > > percentage
> > > > > is
> > > > > > pretty low.
> > > > > >
> > > > > > To solve this issue I see two options:
> > > > > > 1) Proposed by Anton.
> > > > > > 2) Having a guy in the community who’ll keep an eye on all the
> > > incoming
> > > > > > pull-requests shuffling them between committer in the same way
> > > proposed
> > > > > in
> > > > > > 1.
> > > > > >
> > > > > > Personally, I’m for 1.
> > > > > >
> > > > > > —
> > > > > > Denis
> > > > > >
> > > > > > > On Jun 5, 2017, at 10:28 AM, Dmitry Pavlov <
> > dpavlov@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi Anton,
> > > > > > >
> > > > > > >
> > > > > > > It is ok for me if it is advice and hint for faster review, as
> it
> > > is
> > > > > now.
> > > > > > >
> > > > > > >
> > > > > > > We can periodically remind about this opportunity at dev list
> or
> > in
> > > > the
> > > > > > > issue comments. We can remind that tasks in patch 

[jira] [Created] (IGNITE-5419) Try optimize on-heap cache maps created for caches in the same cache group

[Semen Boikov (JIRA)]

Semen Boikov created IGNITE-5419:


 Summary: Try optimize on-heap cache maps created for caches in the 
same cache group
 Key: IGNITE-5419
 URL: https://issues.apache.org/jira/browse/IGNITE-5419
 Project: Ignite
  Issue Type: Task
  Components: cache
Reporter: Semen Boikov
Assignee: Semen Boikov
Priority: Critical
 Fix For: 2.2


Now if cache group contains multiple caches, then DhtLocalPartition will 
contain separate per-cache maps. Need think how this can be optimized (single 
map per-partition where key will contain cacheId? or single on-heap map per 
cache?)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (IGNITE-5418) CPP: Add BINARY SQL type support for SQL queries

[Igor Sapego (JIRA)]

Igor Sapego created IGNITE-5418:
---

 Summary: CPP: Add BINARY SQL type support for SQL queries
 Key: IGNITE-5418
 URL: https://issues.apache.org/jira/browse/IGNITE-5418
 Project: Ignite
  Issue Type: Improvement
  Components: platforms
Affects Versions: 2.0
Reporter: Igor Sapego
 Fix For: 2.2


There are currently no way to add to query argument of type {{BINARY}} (byte 
array) or read field of this type using C++ API. Consider adding support for 
this using {{vector}} or some specific API calls.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: "Review workflow" changes to prevent "broken review" issues.

[Dmitry Pavlov]

Anton,


Thank you for explanation. Personal ask instead of group broadcast may
really help. I understand the idea now.


One argument against solution way 1) it may be not easy for contributor,
especially newcomer, to find a right person.


What do you think about way 2? Personally, I'm ready to help with analysis
and assignment of these 66 tasks from next week.



вт, 6 июн. 2017 г. в 12:57, Anton Vinogradov :

> Dmitry Pavlov,
>
> There is *HUGE *difference between "Devlist, please review my changes"
> and "Dmitry Pavlov, please review my changes".
>
> In case you're busy right now, you'll, most likely, ignore appeal to
> devlist, but, I'm pretty sure, you'll check appeal to yourself.
> Am I right?
>
> So, my idea is: appeal to devlist is a useless spam maker approach, but
> appeal to person(s) works.
>
> On Tue, Jun 6, 2017 at 2:40 AM, Dmitriy Setrakyan 
> wrote:
>
> > Wow, we have 66 tickets waiting for review this is pretty bad.
> > Something must definitely change.
> >
> > From my side, having a contributor shop around for a reviewer is not
> really
> > fair. However, I would support the idea of Apache Ignite community
> electing
> > a person responsible to find reviewers for all contributions.
> >
> > D.
> >
> > On Mon, Jun 5, 2017 at 11:23 AM, Dmitry Pavlov 
> > wrote:
> >
> > > 1) There is waiting for review list here
> > > https://cwiki.apache.org/confluence/display/IGNITE/
> > > Issues+waiting+for+review
> > >
> > > 2) some of contributions are supplemented by dev-list messages (please
> > > review my PR…)
> > >
> > >
> > > And these two tools sometimes do not help. I suppose it is because of
> > > reviewers already have some things to do, but not because of lack of
> tool
> > > support. Do you have other explanations?
> > >
> > >
> > > But still, Igor’s suggestion to notify to dev list sounds reasonable to
> > me.
> > >
> > >
> > >
> > > Anton, could it solve your requirement to know about issue needed to
> > > review?
> > >
> > > пн, 5 июн. 2017 г. в 21:06, Igor Sapego :
> > >
> > > > By the way, there are emails being sent from Jira to devlist every
> > > > time someone adds comment to ticket, or, for example, edits its
> > > > title which helps a lot to keep a track of ticket's state.
> > > >
> > > > But by some reason there is no such notification when ticket silently
> > > > getting moved to "Patch available" state. I believe, that would help
> if
> > > > there was a notification for that. Is it possible to configure?
> > > >
> > > > Best Regards,
> > > > Igor
> > > >
> > > > On Mon, Jun 5, 2017 at 9:00 PM, Denis Magda 
> wrote:
> > > >
> > > > > In general, I tend to agree with Anton that something needs to be
> > > changed
> > > > > in this direction.
> > > > >
> > > > > How many of you flip through dev list, JIRA or GitHub notifications
> > in
> > > an
> > > > > attempt to find tickets that demand your attention? I bet the
> > > percentage
> > > > is
> > > > > pretty low.
> > > > >
> > > > > To solve this issue I see two options:
> > > > > 1) Proposed by Anton.
> > > > > 2) Having a guy in the community who’ll keep an eye on all the
> > incoming
> > > > > pull-requests shuffling them between committer in the same way
> > proposed
> > > > in
> > > > > 1.
> > > > >
> > > > > Personally, I’m for 1.
> > > > >
> > > > > —
> > > > > Denis
> > > > >
> > > > > > On Jun 5, 2017, at 10:28 AM, Dmitry Pavlov <
> dpavlov@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > Hi Anton,
> > > > > >
> > > > > >
> > > > > > It is ok for me if it is advice and hint for faster review, as it
> > is
> > > > now.
> > > > > >
> > > > > >
> > > > > > We can periodically remind about this opportunity at dev list or
> in
> > > the
> > > > > > issue comments. We can remind that tasks in patch available
> status
> > > may
> > > > be
> > > > > > reassigned by contributor.
> > > > > >
> > > > > >
> > > > > > Looking from prospective of overall throughput: it is not clear
> for
> > > me
> > > > > how
> > > > > > this process change will help.
> > > > > >
> > > > > >
> > > > > > Best Regards,
> > > > > >
> > > > > > Dmitriy Pavlov
> > > > > >
> > > > > > пн, 5 июн. 2017 г. в 20:16, Anton Vinogradov :
> > > > > >
> > > > > >> Vova,
> > > > > >>
> > > > > >> Contributors interested to make contributions and I propose them
> > to
> > > > use
> > > > > >> *same* strategy as we (people inside the community) use.
> > > > > >> "-1" will not solve this issue, but my "tips" will.
> > > > > >>
> > > > > >> Dmitry,
> > > > > >>
> > > > > >> The main problem here is that nobody notified that someone is
> > > waiting
> > > > > for
> > > > > >> the review.
> > > > > >> It's not a problem for me to provide tips or to make review, but
> > > it's
> > > > > >> problem to periodically check is there somebody waiting.
> > > > > >>
> > > > > >> Guys,
> > > > > >> Let's try this approach, and I'm pretty 

Re: place for printing info about partition distribution (IGNITE-4756)

[Taras Ledkov]

According to the last Yakov's comments at the ticket:

1. Calculation and printing distribution deviation will be produced only 
for local node.
2. In this case it takes only one additional compare nodeID if we build 
statistic with the assignment calculation.
3. We have return injection of the Ignite instance into the AF to gather 
local node.



On 05.06.2017 20:39, Denis Magda wrote:

Taras, Yakov,

Please join the conversation.

—
Denis

On Jun 4, 2017, at 3:43 AM, Вадим Опольский > wrote:


Guys, lets suggest the format for this log.

Yakov's version is "print only nodes that differ on +/-10% from 
average partitions count we can print problem message looks like:


[cacheName=NAME, nodes=[nodeId=>

Date: 2017-06-04 13:42 GMT+03:00
Subject: Fwd: place for printing info about partition distribution 
(IGNITE-4756)
To: Denis Magda >, 
tled...@gridgain.com , Yakov Zhdanov 
>



Hello guys!

I cant start resolve issue 4756 
(https://issues.apache.org/jira/browse/IGNITE-4756 
) because don't 
understand place for pinting info about partition.


Yakov, Taras do you have any idea about it ?

Vadim Opolski



-- Forwarded message --
From: *Вадим Опольский* >

Date: 2017-05-31 18:24 GMT+03:00
Subject: place for printing info about partition distribution
To: dev@ignite.apache.org , Denis Magda 
>, tled...@gridgain.com 
, Yakov Zhdanov >



Hello guys!

I want to discuss and suggest some format and place for pinting info 
about partition distribution to log from issue 
https://issues.apache.org/jira/browse/IGNITE-4756 



What do you think about it ?

Issue is about RendezvousAffinityFunction and FairAffinityFunction 
classes.


Vadim







--
Taras Ledkov
Mail-To: tled...@gridgain.com

[GitHub] ignite pull request #2059: IGNITE-5075-pds

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2059


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2058: Example of API changes.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2058


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2031: Ignite 5075

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2031


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2036: Ignite 5075 pds snapshots

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2036


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: "Review workflow" changes to prevent "broken review" issues.

[Anton Vinogradov]

Dmitry Pavlov,

There is *HUGE *difference between "Devlist, please review my changes"
and "Dmitry Pavlov, please review my changes".

In case you're busy right now, you'll, most likely, ignore appeal to
devlist, but, I'm pretty sure, you'll check appeal to yourself.
Am I right?

So, my idea is: appeal to devlist is a useless spam maker approach, but
appeal to person(s) works.

On Tue, Jun 6, 2017 at 2:40 AM, Dmitriy Setrakyan 
wrote:

> Wow, we have 66 tickets waiting for review this is pretty bad.
> Something must definitely change.
>
> From my side, having a contributor shop around for a reviewer is not really
> fair. However, I would support the idea of Apache Ignite community electing
> a person responsible to find reviewers for all contributions.
>
> D.
>
> On Mon, Jun 5, 2017 at 11:23 AM, Dmitry Pavlov 
> wrote:
>
> > 1) There is waiting for review list here
> > https://cwiki.apache.org/confluence/display/IGNITE/
> > Issues+waiting+for+review
> >
> > 2) some of contributions are supplemented by dev-list messages (please
> > review my PR…)
> >
> >
> > And these two tools sometimes do not help. I suppose it is because of
> > reviewers already have some things to do, but not because of lack of tool
> > support. Do you have other explanations?
> >
> >
> > But still, Igor’s suggestion to notify to dev list sounds reasonable to
> me.
> >
> >
> >
> > Anton, could it solve your requirement to know about issue needed to
> > review?
> >
> > пн, 5 июн. 2017 г. в 21:06, Igor Sapego :
> >
> > > By the way, there are emails being sent from Jira to devlist every
> > > time someone adds comment to ticket, or, for example, edits its
> > > title which helps a lot to keep a track of ticket's state.
> > >
> > > But by some reason there is no such notification when ticket silently
> > > getting moved to "Patch available" state. I believe, that would help if
> > > there was a notification for that. Is it possible to configure?
> > >
> > > Best Regards,
> > > Igor
> > >
> > > On Mon, Jun 5, 2017 at 9:00 PM, Denis Magda  wrote:
> > >
> > > > In general, I tend to agree with Anton that something needs to be
> > changed
> > > > in this direction.
> > > >
> > > > How many of you flip through dev list, JIRA or GitHub notifications
> in
> > an
> > > > attempt to find tickets that demand your attention? I bet the
> > percentage
> > > is
> > > > pretty low.
> > > >
> > > > To solve this issue I see two options:
> > > > 1) Proposed by Anton.
> > > > 2) Having a guy in the community who’ll keep an eye on all the
> incoming
> > > > pull-requests shuffling them between committer in the same way
> proposed
> > > in
> > > > 1.
> > > >
> > > > Personally, I’m for 1.
> > > >
> > > > —
> > > > Denis
> > > >
> > > > > On Jun 5, 2017, at 10:28 AM, Dmitry Pavlov 
> > > > wrote:
> > > > >
> > > > > Hi Anton,
> > > > >
> > > > >
> > > > > It is ok for me if it is advice and hint for faster review, as it
> is
> > > now.
> > > > >
> > > > >
> > > > > We can periodically remind about this opportunity at dev list or in
> > the
> > > > > issue comments. We can remind that tasks in patch available status
> > may
> > > be
> > > > > reassigned by contributor.
> > > > >
> > > > >
> > > > > Looking from prospective of overall throughput: it is not clear for
> > me
> > > > how
> > > > > this process change will help.
> > > > >
> > > > >
> > > > > Best Regards,
> > > > >
> > > > > Dmitriy Pavlov
> > > > >
> > > > > пн, 5 июн. 2017 г. в 20:16, Anton Vinogradov :
> > > > >
> > > > >> Vova,
> > > > >>
> > > > >> Contributors interested to make contributions and I propose them
> to
> > > use
> > > > >> *same* strategy as we (people inside the community) use.
> > > > >> "-1" will not solve this issue, but my "tips" will.
> > > > >>
> > > > >> Dmitry,
> > > > >>
> > > > >> The main problem here is that nobody notified that someone is
> > waiting
> > > > for
> > > > >> the review.
> > > > >> It's not a problem for me to provide tips or to make review, but
> > it's
> > > > >> problem to periodically check is there somebody waiting.
> > > > >>
> > > > >> Guys,
> > > > >> Let's try this approach, and I'm pretty sure it will help.
> > > > >>
> > > > >> On Mon, Jun 5, 2017 at 7:58 PM, Dmitry Pavlov <
> > dpavlov@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >>> Hi Igniters, Anton,
> > > > >>>
> > > > >>> Let’s imagine that development process as a chain of production
> > > stages
> > > > >>> 1) Developing patch by contributor
> > > > >>> 2) Review changes by committer
> > > > >>>
> > > > >>> Reviews waiting too long time to be done at stage 2 may indicate
> > that
> > > > >> speed
> > > > >>> (potential throughput) of step 2 is less than throughput at step
> 1.
> > > > T2 > > > >>>
> > > > >>> In terms of this model (inspired by Goldratt’s Theory of
> > Constraints
> > > > >>> (TOC)), I have a question:
> > > > >>> Will this 

[GitHub] ignite pull request #2092: IGNITE-5377 ODBC: Expose SqlFieldsQuery hints as ...

[tledkov-gridgain]

Github user tledkov-gridgain closed the pull request at:

https://github.com/apache/ignite/pull/2092


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2092: IGNITE-5377 ODBC: Expose SqlFieldsQuery hints as ...

[tledkov-gridgain]

GitHub user tledkov-gridgain opened a pull request:

https://github.com/apache/ignite/pull/2092

IGNITE-5377 ODBC: Expose SqlFieldsQuery hints as parameters



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-5377

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2092.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2092


commit 8a3787bbd42c99d7eda1d9eb06dc8fe069f2e584
Author: tledkov-gridgain 
Date:   2017-06-06T09:26:27Z

IGNITE-5377 ODBC: Expose SqlFieldsQuery hints as parameters




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Data compression in Ignite 2.0

[Антон Чураев]

Vyacheslav, thank you! But could you please provide a conclusions or
proposals based on this benchmarks?

2017-06-06 11:28 GMT+03:00 Vyacheslav Daradur :

> Dmitry,
>
> Excel-pages:
>
> 1). "Compression ratio (2)" - shows object size, with compression and
> without compression. (Conditions: literal text)
> 1st graph shows compression ratios of using different compression algrithms
> depending on size of compressed field.
> 2nd graph shows evaluation of size of objects depending on sizes and
> compression algorithms.
>
> 2). "Compression ratio (1)" - shows object size, with compression and
> without compression. (Conditions:  badly compressed character sequence)
> 1st graph shows compression ratios of using different compression
> algrithms depending on size of compressed field.
> 2nd graph shows evaluation of size of objects depending on sizes and
> compression algorithms.
>
> 3) 'put-avg" - shows average time of the "put" operation depending on size
> and compression algorithms.
>
> 4) 'put-thrpt" - shows throughput of the "put" operation depending on size
> and compression algorithms.
>
> 5) 'get-avg" - shows average time of the "get" operation depending on size
> and compression algorithms.
>
> 6) 'get-thrpt" - shows throughput of the "get" operation depending on size
> and compression algorithms.
>
>
>
>
> 2017-06-06 10:59 GMT+03:00 Dmitriy Setrakyan :
>
> > Vladimir, I am not sure how to interpret the graphs? What are we looking
> > at?
> >
> > On Tue, Jun 6, 2017 at 12:33 AM, Vyacheslav Daradur  >
> > wrote:
> >
> > > Hi, Igniters.
> > >
> > > I've prepared some benchmarking. Results [1].
> > >
> > > And I've prepared the evaluation in the form of diagrams [2].
> > >
> > > I hope that helps to interest the community and accelerates a reaction
> to
> > > this improvment :)
> > >
> > > [1]
> > > https://github.com/daradurvs/ignite-compression/tree/
> > > master/src/main/resources/result
> > > [2] https://drive.google.com/file/d/0B2CeUAOgrHkoMklyZ25YTEdKcEk/view
> > >
> > >
> > >
> > > 2017-05-24 9:49 GMT+03:00 Vyacheslav Daradur :
> > >
> > > > Guys, any thoughts?
> > > >
> > > > 2017-05-16 13:40 GMT+03:00 Vyacheslav Daradur :
> > > >
> > > >> Hi guys,
> > > >>
> > > >> I've prepared the PR to show my idea.
> > > >> https://github.com/apache/ignite/pull/1951/files
> > > >>
> > > >> About querying - I've just copied existing tests and have annotated
> > the
> > > >> testing data.
> > > >> https://github.com/apache/ignite/pull/1951/files#diff-c19a9d
> > > >> f4058141d059bb577e75244764
> > > >>
> > > >> It means fields which will be marked by @BinaryCompression will be
> > > >> compressed at marshalling via BinaryMarshaller.
> > > >>
> > > >> This solution has no effect on existing data or project
> architecture.
> > > >>
> > > >> I'll be glad to see your thougths.
> > > >>
> > > >>
> > > >> 2017-05-15 19:18 GMT+03:00 Vyacheslav Daradur  >:
> > > >>
> > > >>> Dmitriy,
> > > >>>
> > > >>> I have ready prototype. I want to show it.
> > > >>> It is always easier to discuss on example.
> > > >>>
> > > >>> 2017-05-15 19:02 GMT+03:00 Dmitriy Setrakyan <
> dsetrak...@apache.org
> > >:
> > > >>>
> > >  Vyacheslav,
> > > 
> > >  I think it is a bit premature to provide a PR without getting a
> > >  community
> > >  consensus on the dev list. Please allow some time for the
> community
> > to
> > >  respond.
> > > 
> > >  D.
> > > 
> > >  On Mon, May 15, 2017 at 6:36 AM, Vyacheslav Daradur <
> > >  daradu...@gmail.com>
> > >  wrote:
> > > 
> > >  > I created the ticket: https://issues.apache.org/jira
> > >  /browse/IGNITE-5226
> > >  >
> > >  > I'll prepare a PR with described solution in couple of days.
> > >  >
> > >  > 2017-05-15 15:05 GMT+03:00 Vyacheslav Daradur <
> > daradu...@gmail.com
> > > >:
> > >  >
> > >  > > Hi, Igniters!
> > >  > >
> > >  > > Apache 2.0 is released.
> > >  > >
> > >  > > Let's continue the discussion about a compression design.
> > >  > >
> > >  > > At the moment, I found only one solution which is compatible
> > with
> > >  > querying
> > >  > > and indexing, this is per-objects-field compression.
> > >  > > Per-fields compression means that metadata (a header) of an
> > object
> > >  won't
> > >  > > be compressed, only serialized values of an object fields (in
> > > bytes
> > >  array
> > >  > > form) will be compressed.
> > >  > >
> > >  > > This solution have some contentious issues:
> > >  > > - small values, like primitives and short arrays - there isn't
> > >  sense to
> > >  > > compress them;
> > >  > > - there is no possible to use compression with java-predefined
> > >  types;
> > >  > >
> > >  > > We can provide an annotation, @IgniteCompression - for
> example,
> > >  

Re: Data compression in Ignite 2.0

[Vyacheslav Daradur]

All metrics are taken from app based on custom assembly of AI, containing
the provided PR.

2017-06-06 11:28 GMT+03:00 Vyacheslav Daradur :

> Dmitry,
>
> Excel-pages:
>
> 1). "Compression ratio (2)" - shows object size, with compression and
> without compression. (Conditions: literal text)
> 1st graph shows compression ratios of using different compression
> algrithms depending on size of compressed field.
> 2nd graph shows evaluation of size of objects depending on sizes and
> compression algorithms.
>
> 2). "Compression ratio (1)" - shows object size, with compression and
> without compression. (Conditions:  badly compressed character sequence)
> 1st graph shows compression ratios of using different compression
> algrithms depending on size of compressed field.
> 2nd graph shows evaluation of size of objects depending on sizes and
> compression algorithms.
>
> 3) 'put-avg" - shows average time of the "put" operation depending on size
> and compression algorithms.
>
> 4) 'put-thrpt" - shows throughput of the "put" operation depending on size
> and compression algorithms.
>
> 5) 'get-avg" - shows average time of the "get" operation depending on size
> and compression algorithms.
>
> 6) 'get-thrpt" - shows throughput of the "get" operation depending on size
> and compression algorithms.
>
>
>
>
> 2017-06-06 10:59 GMT+03:00 Dmitriy Setrakyan :
>
>> Vladimir, I am not sure how to interpret the graphs? What are we looking
>> at?
>>
>> On Tue, Jun 6, 2017 at 12:33 AM, Vyacheslav Daradur 
>> wrote:
>>
>> > Hi, Igniters.
>> >
>> > I've prepared some benchmarking. Results [1].
>> >
>> > And I've prepared the evaluation in the form of diagrams [2].
>> >
>> > I hope that helps to interest the community and accelerates a reaction
>> to
>> > this improvment :)
>> >
>> > [1]
>> > https://github.com/daradurvs/ignite-compression/tree/
>> > master/src/main/resources/result
>> > [2] https://drive.google.com/file/d/0B2CeUAOgrHkoMklyZ25YTEdKcEk/view
>> >
>> >
>> >
>> > 2017-05-24 9:49 GMT+03:00 Vyacheslav Daradur :
>> >
>> > > Guys, any thoughts?
>> > >
>> > > 2017-05-16 13:40 GMT+03:00 Vyacheslav Daradur :
>> > >
>> > >> Hi guys,
>> > >>
>> > >> I've prepared the PR to show my idea.
>> > >> https://github.com/apache/ignite/pull/1951/files
>> > >>
>> > >> About querying - I've just copied existing tests and have annotated
>> the
>> > >> testing data.
>> > >> https://github.com/apache/ignite/pull/1951/files#diff-c19a9d
>> > >> f4058141d059bb577e75244764
>> > >>
>> > >> It means fields which will be marked by @BinaryCompression will be
>> > >> compressed at marshalling via BinaryMarshaller.
>> > >>
>> > >> This solution has no effect on existing data or project architecture.
>> > >>
>> > >> I'll be glad to see your thougths.
>> > >>
>> > >>
>> > >> 2017-05-15 19:18 GMT+03:00 Vyacheslav Daradur :
>> > >>
>> > >>> Dmitriy,
>> > >>>
>> > >>> I have ready prototype. I want to show it.
>> > >>> It is always easier to discuss on example.
>> > >>>
>> > >>> 2017-05-15 19:02 GMT+03:00 Dmitriy Setrakyan > >:
>> > >>>
>> >  Vyacheslav,
>> > 
>> >  I think it is a bit premature to provide a PR without getting a
>> >  community
>> >  consensus on the dev list. Please allow some time for the
>> community to
>> >  respond.
>> > 
>> >  D.
>> > 
>> >  On Mon, May 15, 2017 at 6:36 AM, Vyacheslav Daradur <
>> >  daradu...@gmail.com>
>> >  wrote:
>> > 
>> >  > I created the ticket: https://issues.apache.org/jira
>> >  /browse/IGNITE-5226
>> >  >
>> >  > I'll prepare a PR with described solution in couple of days.
>> >  >
>> >  > 2017-05-15 15:05 GMT+03:00 Vyacheslav Daradur <
>> daradu...@gmail.com
>> > >:
>> >  >
>> >  > > Hi, Igniters!
>> >  > >
>> >  > > Apache 2.0 is released.
>> >  > >
>> >  > > Let's continue the discussion about a compression design.
>> >  > >
>> >  > > At the moment, I found only one solution which is compatible
>> with
>> >  > querying
>> >  > > and indexing, this is per-objects-field compression.
>> >  > > Per-fields compression means that metadata (a header) of an
>> object
>> >  won't
>> >  > > be compressed, only serialized values of an object fields (in
>> > bytes
>> >  array
>> >  > > form) will be compressed.
>> >  > >
>> >  > > This solution have some contentious issues:
>> >  > > - small values, like primitives and short arrays - there isn't
>> >  sense to
>> >  > > compress them;
>> >  > > - there is no possible to use compression with java-predefined
>> >  types;
>> >  > >
>> >  > > We can provide an annotation, @IgniteCompression - for example,
>> >  which can
>> >  > > be used by users for marking fields to compress.
>> >  > >
>> >  > > Any thoughts?
>> >  > >
>> > 

Re: Data compression in Ignite 2.0

[Vyacheslav Daradur]

Dmitry,

Excel-pages:

1). "Compression ratio (2)" - shows object size, with compression and
without compression. (Conditions: literal text)
1st graph shows compression ratios of using different compression algrithms
depending on size of compressed field.
2nd graph shows evaluation of size of objects depending on sizes and
compression algorithms.

2). "Compression ratio (1)" - shows object size, with compression and
without compression. (Conditions:  badly compressed character sequence)
1st graph shows compression ratios of using different compression
algrithms depending on size of compressed field.
2nd graph shows evaluation of size of objects depending on sizes and
compression algorithms.

3) 'put-avg" - shows average time of the "put" operation depending on size
and compression algorithms.

4) 'put-thrpt" - shows throughput of the "put" operation depending on size
and compression algorithms.

5) 'get-avg" - shows average time of the "get" operation depending on size
and compression algorithms.

6) 'get-thrpt" - shows throughput of the "get" operation depending on size
and compression algorithms.




2017-06-06 10:59 GMT+03:00 Dmitriy Setrakyan :

> Vladimir, I am not sure how to interpret the graphs? What are we looking
> at?
>
> On Tue, Jun 6, 2017 at 12:33 AM, Vyacheslav Daradur 
> wrote:
>
> > Hi, Igniters.
> >
> > I've prepared some benchmarking. Results [1].
> >
> > And I've prepared the evaluation in the form of diagrams [2].
> >
> > I hope that helps to interest the community and accelerates a reaction to
> > this improvment :)
> >
> > [1]
> > https://github.com/daradurvs/ignite-compression/tree/
> > master/src/main/resources/result
> > [2] https://drive.google.com/file/d/0B2CeUAOgrHkoMklyZ25YTEdKcEk/view
> >
> >
> >
> > 2017-05-24 9:49 GMT+03:00 Vyacheslav Daradur :
> >
> > > Guys, any thoughts?
> > >
> > > 2017-05-16 13:40 GMT+03:00 Vyacheslav Daradur :
> > >
> > >> Hi guys,
> > >>
> > >> I've prepared the PR to show my idea.
> > >> https://github.com/apache/ignite/pull/1951/files
> > >>
> > >> About querying - I've just copied existing tests and have annotated
> the
> > >> testing data.
> > >> https://github.com/apache/ignite/pull/1951/files#diff-c19a9d
> > >> f4058141d059bb577e75244764
> > >>
> > >> It means fields which will be marked by @BinaryCompression will be
> > >> compressed at marshalling via BinaryMarshaller.
> > >>
> > >> This solution has no effect on existing data or project architecture.
> > >>
> > >> I'll be glad to see your thougths.
> > >>
> > >>
> > >> 2017-05-15 19:18 GMT+03:00 Vyacheslav Daradur :
> > >>
> > >>> Dmitriy,
> > >>>
> > >>> I have ready prototype. I want to show it.
> > >>> It is always easier to discuss on example.
> > >>>
> > >>> 2017-05-15 19:02 GMT+03:00 Dmitriy Setrakyan  >:
> > >>>
> >  Vyacheslav,
> > 
> >  I think it is a bit premature to provide a PR without getting a
> >  community
> >  consensus on the dev list. Please allow some time for the community
> to
> >  respond.
> > 
> >  D.
> > 
> >  On Mon, May 15, 2017 at 6:36 AM, Vyacheslav Daradur <
> >  daradu...@gmail.com>
> >  wrote:
> > 
> >  > I created the ticket: https://issues.apache.org/jira
> >  /browse/IGNITE-5226
> >  >
> >  > I'll prepare a PR with described solution in couple of days.
> >  >
> >  > 2017-05-15 15:05 GMT+03:00 Vyacheslav Daradur <
> daradu...@gmail.com
> > >:
> >  >
> >  > > Hi, Igniters!
> >  > >
> >  > > Apache 2.0 is released.
> >  > >
> >  > > Let's continue the discussion about a compression design.
> >  > >
> >  > > At the moment, I found only one solution which is compatible
> with
> >  > querying
> >  > > and indexing, this is per-objects-field compression.
> >  > > Per-fields compression means that metadata (a header) of an
> object
> >  won't
> >  > > be compressed, only serialized values of an object fields (in
> > bytes
> >  array
> >  > > form) will be compressed.
> >  > >
> >  > > This solution have some contentious issues:
> >  > > - small values, like primitives and short arrays - there isn't
> >  sense to
> >  > > compress them;
> >  > > - there is no possible to use compression with java-predefined
> >  types;
> >  > >
> >  > > We can provide an annotation, @IgniteCompression - for example,
> >  which can
> >  > > be used by users for marking fields to compress.
> >  > >
> >  > > Any thoughts?
> >  > >
> >  > > Maybe someone already have ready design?
> >  > >
> >  > > 2017-04-10 11:06 GMT+03:00 Vyacheslav Daradur <
> > daradu...@gmail.com
> >  >:
> >  > >
> >  > >> Alexey,
> >  > >>
> >  > >> Yes, I've read it.
> >  > >>
> >  > >> Ok, let's discuss about public API design.
> >  > >>
> >  > 

[GitHub] ignite pull request #2091: IGNITE-1553 Optimize transaction prepare step whe...

[voipp]

GitHub user voipp opened a pull request:

https://github.com/apache/ignite/pull/2091

IGNITE-1553 Optimize transaction prepare step when store is enabled



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/voipp/ignite IGNITE-1553

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2091.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2091






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Data compression in Ignite 2.0

[Dmitriy Setrakyan]

Vladimir, I am not sure how to interpret the graphs? What are we looking at?

On Tue, Jun 6, 2017 at 12:33 AM, Vyacheslav Daradur 
wrote:

> Hi, Igniters.
>
> I've prepared some benchmarking. Results [1].
>
> And I've prepared the evaluation in the form of diagrams [2].
>
> I hope that helps to interest the community and accelerates a reaction to
> this improvment :)
>
> [1]
> https://github.com/daradurvs/ignite-compression/tree/
> master/src/main/resources/result
> [2] https://drive.google.com/file/d/0B2CeUAOgrHkoMklyZ25YTEdKcEk/view
>
>
>
> 2017-05-24 9:49 GMT+03:00 Vyacheslav Daradur :
>
> > Guys, any thoughts?
> >
> > 2017-05-16 13:40 GMT+03:00 Vyacheslav Daradur :
> >
> >> Hi guys,
> >>
> >> I've prepared the PR to show my idea.
> >> https://github.com/apache/ignite/pull/1951/files
> >>
> >> About querying - I've just copied existing tests and have annotated the
> >> testing data.
> >> https://github.com/apache/ignite/pull/1951/files#diff-c19a9d
> >> f4058141d059bb577e75244764
> >>
> >> It means fields which will be marked by @BinaryCompression will be
> >> compressed at marshalling via BinaryMarshaller.
> >>
> >> This solution has no effect on existing data or project architecture.
> >>
> >> I'll be glad to see your thougths.
> >>
> >>
> >> 2017-05-15 19:18 GMT+03:00 Vyacheslav Daradur :
> >>
> >>> Dmitriy,
> >>>
> >>> I have ready prototype. I want to show it.
> >>> It is always easier to discuss on example.
> >>>
> >>> 2017-05-15 19:02 GMT+03:00 Dmitriy Setrakyan :
> >>>
>  Vyacheslav,
> 
>  I think it is a bit premature to provide a PR without getting a
>  community
>  consensus on the dev list. Please allow some time for the community to
>  respond.
> 
>  D.
> 
>  On Mon, May 15, 2017 at 6:36 AM, Vyacheslav Daradur <
>  daradu...@gmail.com>
>  wrote:
> 
>  > I created the ticket: https://issues.apache.org/jira
>  /browse/IGNITE-5226
>  >
>  > I'll prepare a PR with described solution in couple of days.
>  >
>  > 2017-05-15 15:05 GMT+03:00 Vyacheslav Daradur  >:
>  >
>  > > Hi, Igniters!
>  > >
>  > > Apache 2.0 is released.
>  > >
>  > > Let's continue the discussion about a compression design.
>  > >
>  > > At the moment, I found only one solution which is compatible with
>  > querying
>  > > and indexing, this is per-objects-field compression.
>  > > Per-fields compression means that metadata (a header) of an object
>  won't
>  > > be compressed, only serialized values of an object fields (in
> bytes
>  array
>  > > form) will be compressed.
>  > >
>  > > This solution have some contentious issues:
>  > > - small values, like primitives and short arrays - there isn't
>  sense to
>  > > compress them;
>  > > - there is no possible to use compression with java-predefined
>  types;
>  > >
>  > > We can provide an annotation, @IgniteCompression - for example,
>  which can
>  > > be used by users for marking fields to compress.
>  > >
>  > > Any thoughts?
>  > >
>  > > Maybe someone already have ready design?
>  > >
>  > > 2017-04-10 11:06 GMT+03:00 Vyacheslav Daradur <
> daradu...@gmail.com
>  >:
>  > >
>  > >> Alexey,
>  > >>
>  > >> Yes, I've read it.
>  > >>
>  > >> Ok, let's discuss about public API design.
>  > >>
>  > >> I think we need to add some a configure entity to
>  CacheConfiguration,
>  > >> which will contain the Compressor interface implementation and
> some
>  > usefull
>  > >> parameters.
>  > >> Or maybe to provide a BinaryMarshaller decorator, which will be
>  compress
>  > >> data after marshalling.
>  > >>
>  > >>
>  > >> 2017-04-10 10:40 GMT+03:00 Alexey Kuznetsov <
> akuznet...@apache.org
>  >:
>  > >>
>  > >>> Vyacheslav,
>  > >>>
>  > >>> Did you read initial discussion [1] about compression?
>  > >>> As far as I remember we agreed to add only some "top-level" API
> in
>  > order
>  > >>> to
>  > >>> provide a way for
>  > >>> Ignite users to inject some sort of custom compression.
>  > >>>
>  > >>>
>  > >>> [1]
>  > >>> http://apache-ignite-developers.2346864.n4.nabble.com/Data-c
>  > >>> ompression-in-Ignite-2-0-td10099.html
>  > >>>
>  > >>> On Mon, Apr 10, 2017 at 2:19 PM, daradurvs  >
>  > wrote:
>  > >>>
>  > >>> > Hi Igniters!
>  > >>> >
>  > >>> > I am interested in this task.
>  > >>> > Provide some kind of pluggable compression SPI support
>  > >>> > 
>  > >>> >
>  > >>> > I developed a solution on BinaryMarshaller-level, but reviewer
>  has
>  > >>> rejected
>  > >>> > it.

Re: Writing a helper for KafkaStreamer

[Dmitriy Setrakyan]

Thanks, Mike! Sounds great.

On Tue, Jun 6, 2017 at 12:26 AM, Michael Griggs  wrote:

> We made a change [1] that required users to re-write code that uses
> KafkaStreamer to initialise tuple extractors.  The re-write is not
> immediately obvious, and for simple use cases (streaming single tuples) it
> is easy to write a helper function that will make the transition easier.
>
>
>
> I intend to raise a JIRA and submit a PR to implement this.
>
>
>
> MG
>
>
>
>
>
> [1] https://issues.apache.org/jira/browse/IGNITE-4140
>
>
>
> --
>
> Michael Griggs
>
> Consultant, EMEA
>
> GridGain Systems
>
>
>
>

Re: Data compression in Ignite 2.0

[Vyacheslav Daradur]

Hi, Igniters.

I've prepared some benchmarking. Results [1].

And I've prepared the evaluation in the form of diagrams [2].

I hope that helps to interest the community and accelerates a reaction to
this improvment :)

[1]
https://github.com/daradurvs/ignite-compression/tree/master/src/main/resources/result
[2] https://drive.google.com/file/d/0B2CeUAOgrHkoMklyZ25YTEdKcEk/view



2017-05-24 9:49 GMT+03:00 Vyacheslav Daradur :

> Guys, any thoughts?
>
> 2017-05-16 13:40 GMT+03:00 Vyacheslav Daradur :
>
>> Hi guys,
>>
>> I've prepared the PR to show my idea.
>> https://github.com/apache/ignite/pull/1951/files
>>
>> About querying - I've just copied existing tests and have annotated the
>> testing data.
>> https://github.com/apache/ignite/pull/1951/files#diff-c19a9d
>> f4058141d059bb577e75244764
>>
>> It means fields which will be marked by @BinaryCompression will be
>> compressed at marshalling via BinaryMarshaller.
>>
>> This solution has no effect on existing data or project architecture.
>>
>> I'll be glad to see your thougths.
>>
>>
>> 2017-05-15 19:18 GMT+03:00 Vyacheslav Daradur :
>>
>>> Dmitriy,
>>>
>>> I have ready prototype. I want to show it.
>>> It is always easier to discuss on example.
>>>
>>> 2017-05-15 19:02 GMT+03:00 Dmitriy Setrakyan :
>>>
 Vyacheslav,

 I think it is a bit premature to provide a PR without getting a
 community
 consensus on the dev list. Please allow some time for the community to
 respond.

 D.

 On Mon, May 15, 2017 at 6:36 AM, Vyacheslav Daradur <
 daradu...@gmail.com>
 wrote:

 > I created the ticket: https://issues.apache.org/jira
 /browse/IGNITE-5226
 >
 > I'll prepare a PR with described solution in couple of days.
 >
 > 2017-05-15 15:05 GMT+03:00 Vyacheslav Daradur :
 >
 > > Hi, Igniters!
 > >
 > > Apache 2.0 is released.
 > >
 > > Let's continue the discussion about a compression design.
 > >
 > > At the moment, I found only one solution which is compatible with
 > querying
 > > and indexing, this is per-objects-field compression.
 > > Per-fields compression means that metadata (a header) of an object
 won't
 > > be compressed, only serialized values of an object fields (in bytes
 array
 > > form) will be compressed.
 > >
 > > This solution have some contentious issues:
 > > - small values, like primitives and short arrays - there isn't
 sense to
 > > compress them;
 > > - there is no possible to use compression with java-predefined
 types;
 > >
 > > We can provide an annotation, @IgniteCompression - for example,
 which can
 > > be used by users for marking fields to compress.
 > >
 > > Any thoughts?
 > >
 > > Maybe someone already have ready design?
 > >
 > > 2017-04-10 11:06 GMT+03:00 Vyacheslav Daradur :
 > >
 > >> Alexey,
 > >>
 > >> Yes, I've read it.
 > >>
 > >> Ok, let's discuss about public API design.
 > >>
 > >> I think we need to add some a configure entity to
 CacheConfiguration,
 > >> which will contain the Compressor interface implementation and some
 > usefull
 > >> parameters.
 > >> Or maybe to provide a BinaryMarshaller decorator, which will be
 compress
 > >> data after marshalling.
 > >>
 > >>
 > >> 2017-04-10 10:40 GMT+03:00 Alexey Kuznetsov :
 > >>
 > >>> Vyacheslav,
 > >>>
 > >>> Did you read initial discussion [1] about compression?
 > >>> As far as I remember we agreed to add only some "top-level" API in
 > order
 > >>> to
 > >>> provide a way for
 > >>> Ignite users to inject some sort of custom compression.
 > >>>
 > >>>
 > >>> [1]
 > >>> http://apache-ignite-developers.2346864.n4.nabble.com/Data-c
 > >>> ompression-in-Ignite-2-0-td10099.html
 > >>>
 > >>> On Mon, Apr 10, 2017 at 2:19 PM, daradurvs 
 > wrote:
 > >>>
 > >>> > Hi Igniters!
 > >>> >
 > >>> > I am interested in this task.
 > >>> > Provide some kind of pluggable compression SPI support
 > >>> > 
 > >>> >
 > >>> > I developed a solution on BinaryMarshaller-level, but reviewer
 has
 > >>> rejected
 > >>> > it.
 > >>> >
 > >>> > Let's continue discussion of task goals and solution design.
 > >>> > As I understood that, the main goal of this task is to store
 data in
 > >>> > compressed form.
 > >>> > This is what I need from Ignite as its user. Compression
 provides
 > >>> economy
 > >>> > on
 > >>> > servers.
 > >>> > We can store more data on same servers at the cost of
 increasing CPU
 > >>> > utilization.
 > 

Writing a helper for KafkaStreamer

[Michael Griggs]

We made a change [1] that required users to re-write code that uses
KafkaStreamer to initialise tuple extractors.  The re-write is not
immediately obvious, and for simple use cases (streaming single tuples) it
is easy to write a helper function that will make the transition easier.  

 

I intend to raise a JIRA and submit a PR to implement this.

 

MG

 

 

[1] https://issues.apache.org/jira/browse/IGNITE-4140 

 

--

Michael Griggs

Consultant, EMEA

GridGain Systems

 

[jira] [Created] (IGNITE-5415) Web console: Implement support of 2.1 configuration version.

[Vasiliy Sisko (JIRA)]

Vasiliy Sisko created IGNITE-5415:
-

 Summary: Web console: Implement support of 2.1 configuration 
version.
 Key: IGNITE-5415
 URL: https://issues.apache.org/jira/browse/IGNITE-5415
 Project: Ignite
  Issue Type: Task
  Components: wizards
Affects Versions: 2.0
Reporter: Vasiliy Sisko
Assignee: Vasiliy Sisko
 Fix For: 2.1






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)