SQL usability issue: unclear exception when a key of a wrong type is used

2017-05-17 Thread Denis Magda 
Sergi, Vovan,

One of Ignite users struggled with an SQL issue and asked me to help him 
troubleshooting it. The root of the issue was trivial. The guy used a custom 
object as a key's type but passed the key of “int” type to DML queries.

Honestly, I could spot it only after looking into the code because the 
exception generated by both H2 and Ignite was useless.

Caused by: org.h2.jdbc.JdbcSQLException: Hexadecimal string with odd number of 
characters: "100"; SQL statement:
SELECT
TABLE._KEY,
TABLE.ID,
TABLE.NAME,
TABLE.COUNTRY
FROM TABLE(_KEY OTHER=(?1,), ID BIGINT=(?2,), NAME VARCHAR=(?3,), COUNTRY 
VARCHAR=(?4,)) [90003-195

Let’s improve the usability here. The ticket is ready:
https://issues.apache.org/jira/browse/IGNITE-5250 


Do you think add some hint in the nearest release?

—
Denis

[jira] [Created] (IGNITE-5250) Unhelpful exception when value of wrong type is passed to H2

2017-05-17 Thread Denis Magda (JIRA) 
Denis Magda created IGNITE-5250:
---

 Summary: Unhelpful exception when value of wrong type is passed to 
H2
 Key: IGNITE-5250
 URL: https://issues.apache.org/jira/browse/IGNITE-5250
 Project: Ignite
  Issue Type: Task
Affects Versions: 2.0
Reporter: Denis Magda
Assignee: Sergi Vladykin
 Fix For: 2.1


For instance, if an SQL schema defined this way:
{code}
cfg.setIndexedTypes(AffinityKey.class, Person.class);
{code}

and then, by some reason, the users confuses the type of the key passing 
{{int}} instead of {{AffinityKey}}

{code}
SqlFieldsQuery query = new SqlFieldsQuery("INSERT INTO Person (_key, 
id, name, country ) " +
"VALUES ( ? , ? , ? , ?)");

// Setting the key of a wrong type (AffinityKey instance must be used 
instead).
query.setArgs(100, 1000, "John", "Canada");

// Getting not user friendly exception.
cache.query(query).getAll();
{code}

he will get an exception that doesn't point out to the exact root cause:
{noformat}
Caused by: class org.apache.ignite.IgniteException: Failed to execute SQL query.
at 
org.apache.ignite.internal.processors.query.h2.DmlStatementsProcessor$3.iterator(DmlStatementsProcessor.java:365)
at 
org.apache.ignite.internal.processors.cache.QueryCursorImpl.iterator(QueryCursorImpl.java:94)
at 
org.apache.ignite.internal.processors.query.h2.DmlStatementsProcessor.doInsert(DmlStatementsProcessor.java:836)
at 
org.apache.ignite.internal.processors.query.h2.DmlStatementsProcessor.executeUpdateStatement(DmlStatementsProcessor.java:378)
at 
org.apache.ignite.internal.processors.query.h2.DmlStatementsProcessor.updateSqlFields(DmlStatementsProcessor.java:173)
at 
org.apache.ignite.internal.processors.query.h2.DmlStatementsProcessor.updateSqlFieldsTwoStep(DmlStatementsProcessor.java:207)
at 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.queryDistributedSqlFields(IgniteH2Indexing.java:1657)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor$5.applyx(GridQueryProcessor.java:1701)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor$5.applyx(GridQueryProcessor.java:1699)
at 
org.apache.ignite.internal.util.lang.IgniteOutClosureX.apply(IgniteOutClosureX.java:36)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor.executeQuery(GridQueryProcessor.java:2129)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor.querySqlFields(GridQueryProcessor.java:1706)
at 
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.query(IgniteCacheProxy.java:783)
... 6 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to execute 
SQL query.
at 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery(IgniteH2Indexing.java:1224)
at 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQueryWithTimer(IgniteH2Indexing.java:1276)
at 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.access$1300(IgniteH2Indexing.java:239)
at 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$4.iterator(IgniteH2Indexing.java:1079)
at 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$4.iterator(IgniteH2Indexing.java:1067)
at 
org.apache.ignite.internal.processors.query.h2.DmlStatementsProcessor$3.iterator(DmlStatementsProcessor.java:362)
... 18 more
Caused by: org.h2.jdbc.JdbcSQLException: Hexadecimal string with odd number of 
characters: "100"; SQL statement:
SELECT
TABLE._KEY,
TABLE.ID,
TABLE.NAME,
TABLE.COUNTRY
FROM TABLE(_KEY OTHER=(?1,), ID BIGINT=(?2,), NAME VARCHAR=(?3,), COUNTRY 
VARCHAR=(?4,)) [90003-195]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
at org.h2.message.DbException.get(DbException.java:179)
at org.h2.message.DbException.get(DbException.java:155)
at org.h2.util.StringUtils.convertHexToBytes(StringUtils.java:930)
at org.h2.value.Value.convertTo(Value.java:957)
at org.h2.table.Column.convert(Column.java:167)
at org.h2.expression.TableFunction.getTable(TableFunction.java:118)
at org.h2.expression.TableFunction.getValue(TableFunction.java:41)
at org.h2.table.FunctionTable.getValueResultSet(FunctionTable.java:218)
at org.h2.table.FunctionTable.getResult(FunctionTable.java:189)
at org.h2.index.FunctionIndex.find(FunctionIndex.java:50)
at org.h2.index.BaseIndex.find(BaseIndex.java:128)
at org.h2.index.IndexCursor.find(IndexCursor.java:169)
at org.h2.table.TableFilter.next(TableFilter.java:468)
at 
org.h2.command.dml.Select$LazyResultQueryFlat.fetchNextRow(Select.java:1452)
at

Re: GridGain Donates Persistent Distributed Store To ASF (Apache Ignite)

2017-05-17 Thread Denis Magda 
Cos, Roman,

Your concerns sound reasonable to me. However, that’s my personal point of view 
and not of the overall community. Personally, I accustomed to lean on the 
notion of releases and deadlines. Not sure that it contradicts ASF principles 
or can’t be applied to Apache projects. In any case it’s up to the community to 
decide how to proceed. I’m just trying to form a sort of process ;)

In general, I would avoid all the arguments and look at the storage 
collaboratively and constructively. I didn’t consider to rush you. Just 
forwarded you the message so that you can spot it among many falling in your 
inbox. Take whatever time you need.

In the meanwhile, I’ve prepared the IP Clearance page referring to the template 
below but failed to commit the changes to ASF repo:
http://incubator.apache.org/ip-clearance/ip-clearance-template.html 


Could any of you grant me karma or commit the changes from your accounts?

> We also have this documented contribution process [1]. Is there a good
> reason to circumvent it in this particular case?
> 
> [1] 
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-1.CreateGitHubpull-request
>  
> 
Cos, the IP Clearance/grant process was followed and the following artifacts 
were prepared as a part of it (if you prefer the pull-request then it should be 
possible to do it):

The repository with the donation is ready and available for review:
https://github.com/agoncharuk/ignite/tree/pds-donate 


Big and main part of the sources is aggregated in “modules/pds”. The rest, that 
connects Apache Ignite memory architecture and SQL engine is under “core” and 
“indexing” modules. Alex Goncharuk should be able to point to specific files or 
commits if required.

Here is a description:
* Persistent Store Overview: 
https://cwiki.apache.org/confluence/display/IGNITE/Persistent+Store+Overview 

* Persistent Store Internal Design: 
https://cwiki.apache.org/confluence/display/IGNITE/Persistent+Store+Internal+Design
 


—
Denis

> On May 17, 2017, at 8:01 PM, Roman Shaposhnik  wrote:
> 
> On Wed, May 17, 2017 at 3:04 PM, Konstantin Boudnik  wrote:
>> Well, here's the issue with "simple move from private repo". This is a
>> huge chunk of code. And while employees of Gridgain are quite familiar
>> with it (or so I presume), the rest of the community is not. I, for
>> one, don't consider that the fact it has been tested and integrated
>> with AI 2.0 and, effectively, outside of AI 2.0 is a reasonable "go"
>> criteria.
> 
> Cos is absolutely correct here. Strong +1 to the above.
> 
>> I am sorry that I have to repeat this after 1.5 years after project's
>> graduation from the Incubator. However, I, personally and otherwise,
>> feel like a community process of creating software should be thought
>> through in the spirit of the community, rather than "release dates" or
>> "feature richness". Which means that the community has to be on board
>> with the decisions like this. And "on board" doesn't mean "majority of
>> the votes" as we, fortunately, aren't playing in democracy @apache.
>> Release dates are relevant to an entity, building and selling
>> products. in Apache we're are working on projects, and while releases
>> are important here, they convey a very different meaning.
> 
> Which brings me to a related question: what exactly needs to be released
> on this aggressive schedule and who is a beneficiary of this release?
> 
> What I am trying to say is this: if GirdGain has a product delivery
> deadline -- the
> company can go ahead and release its product with whatever features it needs 
> to.
> 
> But I'm with Cos -- the community has to be given time to get comfortable with
> the code base if for nothing else but for licensing implications.
> 
> Thanks,
> Roman.

Re: GridGain Donates Persistent Distributed Store To ASF (Apache Ignite)

2017-05-17 Thread Roman Shaposhnik 
On Wed, May 17, 2017 at 3:04 PM, Konstantin Boudnik  wrote:
> Well, here's the issue with "simple move from private repo". This is a
> huge chunk of code. And while employees of Gridgain are quite familiar
> with it (or so I presume), the rest of the community is not. I, for
> one, don't consider that the fact it has been tested and integrated
> with AI 2.0 and, effectively, outside of AI 2.0 is a reasonable "go"
> criteria.

Cos is absolutely correct here. Strong +1 to the above.

> I am sorry that I have to repeat this after 1.5 years after project's
> graduation from the Incubator. However, I, personally and otherwise,
> feel like a community process of creating software should be thought
> through in the spirit of the community, rather than "release dates" or
> "feature richness". Which means that the community has to be on board
> with the decisions like this. And "on board" doesn't mean "majority of
> the votes" as we, fortunately, aren't playing in democracy @apache.
> Release dates are relevant to an entity, building and selling
> products. in Apache we're are working on projects, and while releases
> are important here, they convey a very different meaning.

Which brings me to a related question: what exactly needs to be released
on this aggressive schedule and who is a beneficiary of this release?

What I am trying to say is this: if GirdGain has a product delivery
deadline -- the
company can go ahead and release its product with whatever features it needs to.

But I'm with Cos -- the community has to be given time to get comfortable with
the code base if for nothing else but for licensing implications.

Thanks,
Roman.

Re: GridGain Donates Persistent Distributed Store To ASF (Apache Ignite)

2017-05-17 Thread Konstantin Boudnik 
Well, here's the issue with "simple move from private repo". This is a
huge chunk of code. And while employees of Gridgain are quite familiar
with it (or so I presume), the rest of the community is not. I, for
one, don't consider that the fact it has been tested and integrated
with AI 2.0 and, effectively, outside of AI 2.0 is a reasonable "go"
criteria.

I am sorry that I have to repeat this after 1.5 years after project's
graduation from the Incubator. However, I, personally and otherwise,
feel like a community process of creating software should be thought
through in the spirit of the community, rather than "release dates" or
"feature richness". Which means that the community has to be on board
with the decisions like this. And "on board" doesn't mean "majority of
the votes" as we, fortunately, aren't playing in democracy @apache.
Release dates are relevant to an entity, building and selling
products. in Apache we're are working on projects, and while releases
are important here, they convey a very different meaning.

We also have this documented contribution process [1]. Is there a good
reason to circumvent it in this particular case?

[1] 
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-1.CreateGitHubpull-request

Thanks,
  Cos
--
  Take care,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.


On Wed, May 17, 2017 at 12:55 PM, Denis Magda  wrote:
> Hi Cos, thanks,
>
> My view is the following.
>
> Here is an endeavor to release AI 2.1 with improved DDL, .NET and C++ 
> capabilities in June (this is discussed in a separate thread).
>
> It will be much better if the storage can get into that release as well to 
> make it even more solid.
>
> As for the stabilization and testing the feature has already been integrated 
> and perfectly tested with recent AI 2.0 version. This is why, personally, I 
> don’t see any reason why this can affect the vote or potential release date. 
> Simply, we just need to move it from the private repo to the ASF one.
>
> —
> Denis
>
>> On May 17, 2017, at 1:03 PM, Konstantin Boudnik  wrote:
>>
>> Hey guys,
>>
>> I will try to look at it before the week's end. I wonder what's the
>> rush for the vote? The normal development process for any big feature
>> is to bring the code to a brunch, run through a stabilization cycle
>> and then merge into the mainline. Why are we doing something different
>> this time around?
>>
>> Regards,
>>  Cos
>> --
>>  Take care,
>> Konstantin (Cos) Boudnik
>> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>>
>> Disclaimer: Opinions expressed in this email are those of the author,
>> and do not necessarily represent the views of any company the author
>> might be affiliated with at the moment of writing.
>>
>>
>> On Tue, May 16, 2017 at 6:44 PM, Denis Magda  wrote:
>>> Cos, Roman,
>>>
>>> Would you have time to look at the donation in the nearest time? It’s useful
>>> to hear your feedback before the voting is started.
>>>
>>> —
>>> Denis
>>>
>>> Begin forwarded message:
>>>
>>> From: Denis Magda 
>>> Subject: Re: GridGain Donates Persistent Distributed Store To ASF (Apache
>>> Ignite)
>>> Date: May 15, 2017 at 4:37:43 PM PDT
>>> To: dev@ignite.apache.org
>>> Reply-To: dev@ignite.apache.org
>>>
>>> The receipt of the software grant (the persistent store) was acknowledged by
>>> Craig Russel.
>>>
>>> Now, we need to move on with this
>>>
>>> In the meanwhile, I’ve prepared the IP Clearance page referring to the
>>> template below but failed to commit the changes to ASF repo:
>>> http://incubator.apache.org/ip-clearance/ip-clearance-template.html
>>>
>>> *Roman S.*, *Cos*, could you help me with this by granting karma or
>>> committing the form from under your account?
>>>
>>>
>>> Roman, Cos, could you help with this?
>>>
>>> *Alex G.*, please add Apache 2.0 copyrights to all source files that are
>>> going to be donated. Presently there is no copyright at all.
>>>
>>> Everyone interested please spend some time exploring the store's docs and
>>> sources shared in my previous email. If no one has any concerns I will
>>> proceed with the donation formalities.
>>>
>>> —
>>> Denis
>>>
>>> On May 12, 2017, at 2:59 PM, Denis Magda  wrote:
>>>
>>> Folks,
>>>
>>> The repository with the donation is ready and available for review:
>>> https://github.com/agoncharuk/ignite/tree/pds-donate
>>>
>>> Big and main part of the sources is aggregated in “modules/pds”. The rest,
>>> that connects Apache Ignite memory architecture and SQL engine is under
>>> “core” and “indexing” modules. Alex Goncharuk should be able to point to
>>> specific files or commits if required.
>>>
>>> Here is a description:
>>> * Persistent Store

Re: GridGain Donates Persistent Distributed Store To ASF (Apache Ignite)

2017-05-17 Thread Denis Magda 
Hi Cos, thanks,

My view is the following.

Here is an endeavor to release AI 2.1 with improved DDL, .NET and C++ 
capabilities in June (this is discussed in a separate thread). 

It will be much better if the storage can get into that release as well to make 
it even more solid.

As for the stabilization and testing the feature has already been integrated 
and perfectly tested with recent AI 2.0 version. This is why, personally, I 
don’t see any reason why this can affect the vote or potential release date. 
Simply, we just need to move it from the private repo to the ASF one. 

—
Denis

> On May 17, 2017, at 1:03 PM, Konstantin Boudnik  wrote:
> 
> Hey guys,
> 
> I will try to look at it before the week's end. I wonder what's the
> rush for the vote? The normal development process for any big feature
> is to bring the code to a brunch, run through a stabilization cycle
> and then merge into the mainline. Why are we doing something different
> this time around?
> 
> Regards,
>  Cos
> --
>  Take care,
> Konstantin (Cos) Boudnik
> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> 
> Disclaimer: Opinions expressed in this email are those of the author,
> and do not necessarily represent the views of any company the author
> might be affiliated with at the moment of writing.
> 
> 
> On Tue, May 16, 2017 at 6:44 PM, Denis Magda  wrote:
>> Cos, Roman,
>> 
>> Would you have time to look at the donation in the nearest time? It’s useful
>> to hear your feedback before the voting is started.
>> 
>> —
>> Denis
>> 
>> Begin forwarded message:
>> 
>> From: Denis Magda 
>> Subject: Re: GridGain Donates Persistent Distributed Store To ASF (Apache
>> Ignite)
>> Date: May 15, 2017 at 4:37:43 PM PDT
>> To: dev@ignite.apache.org
>> Reply-To: dev@ignite.apache.org
>> 
>> The receipt of the software grant (the persistent store) was acknowledged by
>> Craig Russel.
>> 
>> Now, we need to move on with this
>> 
>> In the meanwhile, I’ve prepared the IP Clearance page referring to the
>> template below but failed to commit the changes to ASF repo:
>> http://incubator.apache.org/ip-clearance/ip-clearance-template.html
>> 
>> *Roman S.*, *Cos*, could you help me with this by granting karma or
>> committing the form from under your account?
>> 
>> 
>> Roman, Cos, could you help with this?
>> 
>> *Alex G.*, please add Apache 2.0 copyrights to all source files that are
>> going to be donated. Presently there is no copyright at all.
>> 
>> Everyone interested please spend some time exploring the store's docs and
>> sources shared in my previous email. If no one has any concerns I will
>> proceed with the donation formalities.
>> 
>> —
>> Denis
>> 
>> On May 12, 2017, at 2:59 PM, Denis Magda  wrote:
>> 
>> Folks,
>> 
>> The repository with the donation is ready and available for review:
>> https://github.com/agoncharuk/ignite/tree/pds-donate
>> 
>> Big and main part of the sources is aggregated in “modules/pds”. The rest,
>> that connects Apache Ignite memory architecture and SQL engine is under
>> “core” and “indexing” modules. Alex Goncharuk should be able to point to
>> specific files or commits if required.
>> 
>> Here is a description:
>> * Persistent Store Overview:
>> https://cwiki.apache.org/confluence/display/IGNITE/Persistent+Store+Overview
>> * Persistent Store Internal Design:
>> https://cwiki.apache.org/confluence/display/IGNITE/Persistent+Store+Internal+Design
>> 
>> The SGA will be signed and sent on Monday.
>> 
>> In the meanwhile, I’ve prepared the IP Clearance page referring to the
>> template below but failed to commit the changes to ASF repo:
>> http://incubator.apache.org/ip-clearance/ip-clearance-template.html
>> 
>> *Roman S.*, *Cos*, could you help me with this by granting karma or
>> committing the form from under your account?
>> 
>> —
>> Denis
>> 
>> On Apr 25, 2017, at 9:56 AM, Konstantin Boudnik  wrote:
>> 
>> While no one is suggesting an IP trap laid out in the non-SGA'ed code
>> in this particular case, we don't want to setup a precedent like this.
>> 
>> From the overall ASF perspective I +1 what Roman has just said.
>> 
>> Thanks,
>> --
>> Take care,
>> Konstantin (Cos) Boudnik
>> 
>> 
>> On Wed, Apr 19, 2017 at 11:41 PM, Roman Shaposhnik 
>> wrote:
>> 
>> On Wed, Apr 19, 2017 at 6:21 PM, Dmitriy Setrakyan
>>  wrote:
>> 
>> On Wed, Apr 19, 2017 at 6:00 PM, Konstantin Boudnik  wrote:
>> 
>> On Tue, Apr 18, 2017 at 11:54PM, Dmitriy Setrakyan wrote:
>> 
>> 
>> Would a standard SGA suffice here?
>> 
>> I believe that ASF guidelines suggest that a discussion should happen
>> first. Once the community gets enough information, we will move to a PMC
>> vote. I was under the impression that once the PMC vote passes, then the
>> SGA should be provided. Or does GridGain need to provide a signed SGA
>> 
>> right
>> 
>> away?
>> 
>> 
>> That reminds me of that

Re: Persistent Distributed Store Metrics

2017-05-17 Thread Denis Magda 
Dmitriy, please give me a couple of days to look at this.

—
Denis

> On May 17, 2017, at 4:16 AM, Dmitriy Govorukhin 
>  wrote:
> 
> Folk,
> 
> As you know, ignite 2.1 will contain new module (pds), it will be
> provide ability to store data on disk. Let's discuss what type of
> metrics we need for this?
> I think it must be metrics per memory policy, per cache, checkpoint,
> and global metrics which will be aggregate all metrics.
> 
> I did sketch.
> 
> PersistentStoreMetrics.java
> 
> public interface PersistentStoreMetrics {
> 
>// Global metrics.
> 
>public long getMemorySize();
> 
>public long getDiskSize();
> 
>public long getPagesInMemory();
> 
>public long getPagesSizeInMemory();
> 
>public long getPagesOnDisk();
> 
>public long getPagesSizeOnDisk();
> 
>public long getFreePages();
> 
>public long getFreePagesSize();
> 
>public long getDirtyPages();
> 
>public long getDirtyPagesSize();
> 
>public long walLog();
> 
>public long walLogSize();
> 
>// Frequency.
> 
>public long getPagesRead();
> 
>public long getPagesWrite();
> 
>public long getFsync();
> 
>public long getWal();
> 
>public long getAverageWalFsyncTime();
> 
>// Per cache.
> 
>public PersistentStoreCacheMetrics cache(String name);
> 
>public PersistentStoreCacheMetrics cache(int cacheId);
> 
>// For last checkpoint.
> 
>public PersistentStoreCheckpointMetrics getLastCheckPoint();
> }
> 
 
> 
> PersistentStoreCacheMetrics.java
> 
> public interface PersistentStoreCacheMetrics {
> 
>public String name();
> 
>public double getFillFactor();
> 
>public double getFillFactor(int part);
> 
>public long getMemorySize();
> 
>public long getDiskSize();
> 
>public long getPagesInMemory();
> 
>public long getPagesSizeInMemory();
> 
>public long getPagesOnDisk();
> 
>public long getPagesSizeOnDisk();
> 
>public long getFreePages();
> 
>public long getFreePagesSize();
> 
>public long getDirtyPages();
> 
>public long getDirtyPagesSize();
> 
>public long getPagesRead();
> 
>public long getPagesWritten();
> }
> 
 
> 
> PersistentStoreCheckpointMetrics.java
> 
> public interface PersistentStoreCheckpointMetrics {
> 
>public long getTotalPages();
> 
>//TODO Page type is internal?
>public long[] pagesType();
> 
>public long getExecutingTime();
> 
>public long getFsyncTime();
> 
>public long getPagesCopyOnWrite();
> }

[jira] [Created] (IGNITE-5249) The release build procedure

2017-05-17 Thread Sergey Kozlov (JIRA) 
Sergey Kozlov created IGNITE-5249:
-

 Summary: The release build procedure
 Key: IGNITE-5249
 URL: https://issues.apache.org/jira/browse/IGNITE-5249
 Project: Ignite
  Issue Type: Task
  Components: build
Reporter: Sergey Kozlov
 Fix For: 2.1


The release build procedure should be placed on the CI/CD server and available 
to run for the release engineer.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Ignite 2.0 TeamCity Run configurations

2017-05-17 Thread Alexey Goncharuk 
Great news! Looking forward to getting rid of the unnecessary TC
configurations.

Once this is done, I think we should also work out if we can use build
once, run tests approach for RunAll configuration. Does anybody have a clue
if this is possible?

2017-05-17 20:46 GMT+03:00 Dmitry Pavlov :

> Hi, Igniters!
>
> Alex (al.psc) has created new Teamcity run configurations optimized for
> running 2.0 version and later. This project and its run configurations
> avoid duplicate tests running for binary and default marshallers because it
> is now the same for 2.0+.
>
> Usage of this run configuration saves us at least 12 machine-hours for each
> test run and in future will allow us to get test results faster.
>
> Do you have concerns or remarks about using new run configurations for all
> 2.0+ pull requests?
>
> We are now doing final checks of compatibility of these run configurations
> with code. Any ideas about what is to be migrated are appreciated.
>
> Best Regards,
> Dmitry Pavlov
>

[jira] [Created] (IGNITE-5248) Best effort check to detect a 32-bit JVM trying to allocate too big amount of memory

2017-05-17 Thread Alexey Goncharuk (JIRA) 
Alexey Goncharuk created IGNITE-5248:


 Summary: Best effort check to detect a 32-bit JVM trying to 
allocate too big amount of memory
 Key: IGNITE-5248
 URL: https://issues.apache.org/jira/browse/IGNITE-5248
 Project: Ignite
  Issue Type: Improvement
  Components: cache
Affects Versions: 2.0
Reporter: Alexey Goncharuk
 Fix For: 2.1


A follow-up on the user list discussion:
http://apache-ignite-users.70518.x6.nabble.com/Ignite2-0-memory-policy-limit-td12840.html

We can catch the IllegalArgumentException and see if the allocated space 
exceeds 2Gb. In this case, we should suggest that user check that he is running 
a 64-bit JVM.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Ignite 2.0 TeamCity Run configurations

2017-05-17 Thread Dmitry Pavlov 
Hi, Igniters!

Alex (al.psc) has created new Teamcity run configurations optimized for
running 2.0 version and later. This project and its run configurations
avoid duplicate tests running for binary and default marshallers because it
is now the same for 2.0+.

Usage of this run configuration saves us at least 12 machine-hours for each
test run and in future will allow us to get test results faster.

Do you have concerns or remarks about using new run configurations for all
2.0+ pull requests?

We are now doing final checks of compatibility of these run configurations
with code. Any ideas about what is to be migrated are appreciated.

Best Regards,
Dmitry Pavlov

[GitHub] ignite pull request #1967: IGNITE-5227: StackOverflowError in GridCacheMapEn...

2017-05-17 Thread mcherkasov 
GitHub user mcherkasov opened a pull request:

https://github.com/apache/ignite/pull/1967

IGNITE-5227: StackOverflowError in GridCacheMapEntry#checkOwnerChanged()

StackOverflowError occurs due recursion in methods:
GridCacheMapEntry.checkOwnerChanged -> 
GridDistributedCacheEntry.checkThreadChain -> GridDistributedCacheEntry.recheck 
-> GridCacheMapEntry.checkOwnerChanged

I broke this cycle by introducing "checkCandidateChain" flag in 
GridCacheMapEntry.checkOwnerChanged method, which allow to miss 
GridDistributedCacheEntry.checkThreadChain. 
The check fo the whole candidate chain is moved to 
GridDistributedCacheEntry.checkThreadChain method, as you can see there  
already was cycle for this, I've just removed break(put it in else branch in 
case we don't have any kes for lock candidates).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mcherkasov/ignite master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1967.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1967


commit fbe059e8003b04568a39a3cfb604a248928d1f6f
Author: mcherkasov 
Date:   2017-05-17T17:12:14Z

IGNITE-5227: StackOverflowError in GridCacheMapEntry#checkOwnerChanged()




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #1966: Ignite 5168

2017-05-17 Thread iveselovskiy 
GitHub user iveselovskiy opened a pull request:

https://github.com/apache/ignite/pull/1966

Ignite 5168



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-5168

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1966.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1966


commit bfb00b6e61f9709718c30971997aeb0ac79e86b4
Author: Alexandr Kuramshin 
Date:   2016-11-18T20:12:28Z

IgniteTcpCommunicationBigClusterTest added

commit 02dd92e605b9b53f5a16c7ec5f8e7b5698b15ba4
Author: Alexandr Kuramshin 
Date:   2016-11-18T21:55:37Z

IgniteTcpCommunicationBigClusterTest update

commit 6acf193a3d356d1bad4c02a53ac76833ed1008d0
Author: Alexandr Kuramshin 
Date:   2016-11-19T09:55:45Z

Have got TcpCommunicationSpi error

commit 4fd39653d24f62f19f70b4dffba8497185cc46fb
Author: Alexandr Kuramshin 
Date:   2016-11-19T16:39:10Z

Some discovery have been done

commit c2c181922c7c24ea457577e32d2af897c8bec87f
Author: Alexandr Kuramshin 
Date:   2016-11-19T20:11:28Z

Prove that problem is not in the onFirstMessage hang

commit f8076edba097f6077229b2090ee3ff1a3369878c
Author: Alexandr Kuramshin 
Date:   2016-11-19T20:26:37Z

Revert: Prove that problem is not in the onFirstMessage hang

commit 6e1f2dfc2acb3dbb8f24aa51ed67b2ee447b4585
Author: Alexandr Kuramshin 
Date:   2016-11-21T08:55:09Z

Revert: pushing unnecessary changes to the master

commit ed794ca815f6bb1471af15779279d287576b39cc
Author: Alexandr Kuramshin 
Date:   2016-11-21T09:08:00Z

Revert: pushing unnecessary changes to the master

commit 3a57d63668bed239e21eca588134272783472e97
Author: iveselovskiy 
Date:   2016-12-09T14:13:37Z

Merge branch 'master' of https://github.com/apache/ignite

commit 166ad21d560c41c42a6de81cf5910537baa3ac92
Author: iveselovskiy 
Date:   2016-12-12T18:46:44Z

Merge branch 'master' of https://github.com/apache/ignite

commit 1eaba3e91bfe54c2bf6f761ac3f238a3b181a8e0
Author: iveselovskiy 
Date:   2017-03-27T18:41:07Z

Merge branch 'master' of https://github.com/apache/ignite

commit cef7775c9dcfc32b11a4cc4678b3a69a1e1b17da
Author: iveselovskiy 
Date:   2017-03-30T13:05:23Z

Merge branch 'master' of https://github.com/apache/ignite

commit 84e26d6fa2f6aec742fafb4d8145fbc9f697da70
Author: iveselovskiy 
Date:   2017-03-31T09:35:07Z

Merge branch 'master' of https://github.com/apache/ignite

commit be360213d74c9f005e4e3a1beb661323262bb7fb
Author: iveselovskiy 
Date:   2017-03-31T12:18:43Z

Merge branch 'master' of https://github.com/apache/ignite

commit ec3de454894f4bba4497b6c0ca0564a730525244
Author: iveselovskiy 
Date:   2017-04-17T15:22:00Z

Merge branch 'master' of https://github.com/apache/ignite

commit 88de0693b76c80fdc6533a799e0475fd53a4b0fc
Author: iveselovskiy 
Date:   2017-05-02T13:20:47Z

Merge branch 'master' of https://github.com/apache/ignite

commit 04d8f4ef69ea92b96642ecc3b797703858574c69
Author: iveselovskiy 
Date:   2017-05-11T13:23:05Z

Merge branch 'master' of https://github.com/apache/ignite

commit a4f3655a11f2b57c190d72588694ca036f10b898
Author: iveselovskiy 
Date:   2017-05-15T14:43:13Z

Merge branch 'master' of https://github.com/apache/ignite

commit a67c15db148735d01ef4d4227d80551224f548ff
Author: iveselovskiy 
Date:   2017-05-15T15:20:40Z

Merge branch 'master' of https://github.com/apache/ignite

commit 3547f8c01148c8a38ef872c2b8f2f6a5631420bb
Author: iveselovskiy 
Date:   2017-05-17T16:20:02Z

Merge branch 'master' of https://github.com/apache/ignite

commit 623a45210fab0b71465ee248ad4d8ce5a6337579
Author: iveselovskiy 
Date:   2017-05-17T17:06:40Z

IGNITE-5168: igfs metrics exposed through mxbeans.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: GridGain Donates Persistent Distributed Store To ASF (Apache Ignite)

2017-05-17 Thread Konstantin Boudnik 
Hey guys,

I will try to look at it before the week's end. I wonder what's the
rush for the vote? The normal development process for any big feature
is to bring the code to a brunch, run through a stabilization cycle
and then merge into the mainline. Why are we doing something different
this time around?

Regards,
  Cos
--
  Take care,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.


On Tue, May 16, 2017 at 6:44 PM, Denis Magda  wrote:
> Cos, Roman,
>
> Would you have time to look at the donation in the nearest time? It’s useful
> to hear your feedback before the voting is started.
>
> —
> Denis
>
> Begin forwarded message:
>
> From: Denis Magda 
> Subject: Re: GridGain Donates Persistent Distributed Store To ASF (Apache
> Ignite)
> Date: May 15, 2017 at 4:37:43 PM PDT
> To: dev@ignite.apache.org
> Reply-To: dev@ignite.apache.org
>
> The receipt of the software grant (the persistent store) was acknowledged by
> Craig Russel.
>
> Now, we need to move on with this
>
> In the meanwhile, I’ve prepared the IP Clearance page referring to the
> template below but failed to commit the changes to ASF repo:
> http://incubator.apache.org/ip-clearance/ip-clearance-template.html
>
> *Roman S.*, *Cos*, could you help me with this by granting karma or
> committing the form from under your account?
>
>
> Roman, Cos, could you help with this?
>
> *Alex G.*, please add Apache 2.0 copyrights to all source files that are
> going to be donated. Presently there is no copyright at all.
>
> Everyone interested please spend some time exploring the store's docs and
> sources shared in my previous email. If no one has any concerns I will
> proceed with the donation formalities.
>
> —
> Denis
>
> On May 12, 2017, at 2:59 PM, Denis Magda  wrote:
>
> Folks,
>
> The repository with the donation is ready and available for review:
> https://github.com/agoncharuk/ignite/tree/pds-donate
>
> Big and main part of the sources is aggregated in “modules/pds”. The rest,
> that connects Apache Ignite memory architecture and SQL engine is under
> “core” and “indexing” modules. Alex Goncharuk should be able to point to
> specific files or commits if required.
>
> Here is a description:
> * Persistent Store Overview:
> https://cwiki.apache.org/confluence/display/IGNITE/Persistent+Store+Overview
> * Persistent Store Internal Design:
> https://cwiki.apache.org/confluence/display/IGNITE/Persistent+Store+Internal+Design
>
> The SGA will be signed and sent on Monday.
>
> In the meanwhile, I’ve prepared the IP Clearance page referring to the
> template below but failed to commit the changes to ASF repo:
> http://incubator.apache.org/ip-clearance/ip-clearance-template.html
>
> *Roman S.*, *Cos*, could you help me with this by granting karma or
> committing the form from under your account?
>
> —
> Denis
>
> On Apr 25, 2017, at 9:56 AM, Konstantin Boudnik  wrote:
>
> While no one is suggesting an IP trap laid out in the non-SGA'ed code
> in this particular case, we don't want to setup a precedent like this.
>
> From the overall ASF perspective I +1 what Roman has just said.
>
> Thanks,
> --
> Take care,
> Konstantin (Cos) Boudnik
>
>
> On Wed, Apr 19, 2017 at 11:41 PM, Roman Shaposhnik 
> wrote:
>
> On Wed, Apr 19, 2017 at 6:21 PM, Dmitriy Setrakyan
>  wrote:
>
> On Wed, Apr 19, 2017 at 6:00 PM, Konstantin Boudnik  wrote:
>
> On Tue, Apr 18, 2017 at 11:54PM, Dmitriy Setrakyan wrote:
>
>
> Would a standard SGA suffice here?
>
> I believe that ASF guidelines suggest that a discussion should happen
> first. Once the community gets enough information, we will move to a PMC
> vote. I was under the impression that once the PMC vote passes, then the
> SGA should be provided. Or does GridGain need to provide a signed SGA
>
> right
>
> away?
>
>
> That reminds me of that Pelosi's self-inflicted conundrum of "In order
> to see the bill, we should pass the bill" ;)
>
>
> Haha :)
>
> SGA != code. In my view, the code should be provided to the community for a
> review. But I am struggling to see why should an SGA be signed prior to the
> community accepting the donation.
>
>
> There's no such thing as SGA without a reference to a code base.
>
> Also, as I explained -- as a community member I would refuse to look
> at the code base that doesn't have a proper licensing attached to it.
> SGA established this kind of proper licensing.
>
> Now, SGA is deinetly not the only way to do so, but it is the easiest
> and since you'd have to do it anyway the most convenient for the
> community.
>
> Thanks,
> Roman.
>
>
>
>

[GitHub] ignite pull request #1965: Ignite gg 8.0.3.ea8

2017-05-17 Thread agoncharuk 
GitHub user agoncharuk opened a pull request:

https://github.com/apache/ignite/pull/1965

Ignite gg 8.0.3.ea8



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-gg-8.0.3.ea8

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1965.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1965






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-5247) TcpCommunicationSpi calls GridNioRecoveryDescriptor with looks unusually large rcvCnt and fail with null.

2017-05-17 Thread Chandra Bose Renganathan (JIRA) 
Chandra Bose Renganathan created IGNITE-5247:


 Summary: TcpCommunicationSpi calls GridNioRecoveryDescriptor with 
looks unusually large rcvCnt and fail with null. 
 Key: IGNITE-5247
 URL: https://issues.apache.org/jira/browse/IGNITE-5247
 Project: Ignite
  Issue Type: Bug
  Components: cache
Affects Versions: 2.0
Reporter: Chandra Bose Renganathan
Priority: Critical


TcpCommunicationSpi calls GridNioRecoveryDescriptor with looks unusually large 
rcvCnt and fail with null. 

This happens when new node try to join existing grid of size 2 or 3, 

Environment linux.

rcvCnt=216172782113783808 ???
The configuration following. PopcornJGroupsTcpDiscoveryIpFinder returns 3 
InetSocketAddress including self.

  IgniteConfiguration cfg = new IgniteConfiguration();
  cfg.setGridLogger(new Slf4jLogger(LOG));
  TcpDiscoverySpi spi = new TcpDiscoverySpi();
  spi.setLocalPort(cachePort);
  
spi.setSocketTimeout(popcornCfg.getLongProperty("popcorn.cache.socketTimeout", 
1L));
  
spi.setLocalPortRange(popcornCfg.getIntProperty("popcorn.cache.portRange", 2));
  spi.setIpFinder(new PopcornJGroupsTcpDiscoveryIpFinder(this));
  Ignition.start(cfg);



08:15:00.838-07:00 [grid-nio-worker-tcp-comm-1-#522%demo-cluster.service%] 
ERROR o.a.i.s.c.t.TcpCommunicationSpi -  - Caught unhandled exception in NIO 
worker thread (restart the node)., "exception":"java.lang.AssertionError: 
Missed message [rcvCnt=216172782113783808, acked=3, 
desc=GridNioRecoveryDescriptor [acked=3, resendCnt=0, rcvCnt=3, sentCnt=3, 
reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode 
[id=acf668c2-188b-4488-bc02-f4a4eca46ab3, addrs=[127.0.0.1, 144.77.76.153], 
sockAddrs=[/127.0.0.1:20103, .com/144.77.76.153:20103], discPort=20103, 
order=2, intOrder=2, lastExchangeTime=1495034080652, loc=false, 
ver=2.0.0#20170430-sha1:d4eef3c6, isClient=false], connected=true, 
connectCnt=1, queueLimit=4096, reserveCnt=1, pairedConnections=false]]", 
"stack":[

"o.a.i.i.u.n.GridNioRecoveryDescriptor.ackReceived(GridNioRecoveryDescriptor.java:207)
 ~[ignite-core-2.0.0.jar:2.0.0]",

"o.a.i.s.c.t.TcpCommunicationSpi$2.onMessage(TcpCommunicationSpi.java:647) 
~[ignite-core-2.0.0.jar:2.0.0]",

"o.a.i.s.c.t.TcpCommunicationSpi$2.onMessage(TcpCommunicationSpi.java:342) 
~[ignite-core-2.0.0.jar:2.0.0]",

"o.a.i.i.u.n.GridNioFilterChain$TailFilter.onMessageReceived(GridNioFilterChain.java:279)
 ~[ignite-core-2.0.0.jar:2.0.0]",

"o.a.i.i.u.n.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
 ~[ignite-core-2.0.0.jar:2.0.0]",

"o.a.i.i.u.n.GridNioCodecFilter.onMessageReceived(GridNioCodecFilter.java:117) 
~[ignite-core-2.0.0.jar:2.0.0]",

"o.a.i.i.u.n.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
 ~[ignite-core-2.0.0.jar:2.0.0]",

"o.a.i.i.u.n.GridConnectionBytesVerifyFilter.onMessageReceived(GridConnectionBytesVerifyFilter.java:88)
 ~[ignite-core-2.0.0.jar:2.0.0]",

"o.a.i.i.u.n.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
 ~[ignite-core-2.0.0.jar:2.0.0]",

"o.a.i.i.u.n.GridNioServer$HeadFilter.onMessageReceived(GridNioServer.java:3062)
 ~[ignite-core-2.0.0.jar:2.0.0]",

"o.a.i.i.u.n.GridNioFilterChain.onMessageReceived(GridNioFilterChain.java:175) 
~[ignite-core-2.0.0.jar:2.0.0]",

"o.a.i.i.u.n.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1121)
 ~[ignite-core-2.0.0.jar:2.0.0]",

"o.a.i.i.u.n.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2031)
 ~[ignite-core-2.0.0.jar:2.0.0]",

"o.a.i.i.u.n.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:1868)
 ~[ignite-core-2.0.0.jar:2.0.0]",

"o.a.i.i.u.n.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1573)
 ~[ignite-core-2.0.0.jar:2.0.0]",
"o.a.i.i.u.w.GridWorker.run(GridWorker.java:110) 
[ignite-core-2.0.0.jar:2.0.0]",
"java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]"]}




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] ignite pull request #1964: IGNITE 5212 Allow custom affinity for system cach...

2017-05-17 Thread vldpyatkov 
GitHub user vldpyatkov opened a pull request:

https://github.com/apache/ignite/pull/1964

IGNITE 5212 Allow custom affinity for system caches



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vldpyatkov/ignite ignite-5212

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1964.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1964


commit 27d152910c2bad48f4c838029bfef0f21780a354
Author: vd-pyatkov 
Date:   2017-05-17T14:50:53Z

IGNITE 5212 Allow custom affinity for system caches




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-5246) Fuzzy c-means (FCM)

2017-05-17 Thread Yury Babak (JIRA) 
Yury Babak created IGNITE-5246:
--

 Summary: Fuzzy c-means (FCM)
 Key: IGNITE-5246
 URL: https://issues.apache.org/jira/browse/IGNITE-5246
 Project: Ignite
  Issue Type: New Feature
  Components: ml
Reporter: Yury Babak


We want to have implementation of fuzzy clusterization.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] ignite pull request #1963: Ignite gg 7170

2017-05-17 Thread vldpyatkov 
GitHub user vldpyatkov opened a pull request:

https://github.com/apache/ignite/pull/1963

Ignite gg 7170



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-gg-7170

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1963.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1963


commit be881725b39926f62df437c2805ac9203fea5737
Author: Ilya Lantukh 
Date:   2016-10-11T16:19:20Z

gg-11595 : WIP.

commit e6d82d8da50b9af9607a914e00da3937d839862a
Author: Ilya Lantukh 
Date:   2016-10-12T16:20:50Z

gg-11595 : WIP.

commit cb9c18ccfc937b4a880eddb5df5d0389de9c7bee
Author: Ilya Lantukh 
Date:   2016-10-13T12:54:39Z

gg-11595 : WIP.

commit ed45f2238361e5bbbe907fdeb4b7a3d7b2b051dd
Author: Ilya Lantukh 
Date:   2016-10-27T13:02:28Z

gg-11595 : Support for restore with concurrent cache operations.

commit e95c68b77cbd3ad30d9f6d0b332137fb26000a41
Author: Ilya Lantukh 
Date:   2016-10-27T13:35:46Z

gg-11595 : Minors.

commit 7a44f1ebccf4fe0f26f54e070bbacbf24fbee3d7
Author: Ilya Lantukh 
Date:   2016-12-16T15:56:24Z

gg-11701 : WIP.

commit 3e2b28075ab5d00dce7fadf4769967f7a0ee2ee8
Author: Ilya Lantukh 
Date:   2016-12-19T12:25:11Z

Merge branches 'ignite-gg-11701' and 'ignite-gg-8.0.2.ea1' of 
https://github.com/gridgain/apache-ignite into ignite-gg-11701

commit cc568f9f6823de04f0beeca45ada4160a58307e4
Author: Ilya Lantukh 
Date:   2016-12-21T13:11:36Z

gg-11701 : WIP

commit e93f8d43a92dd8bde12c4677b5fcc6b0c2de6ce6
Author: Ilya Lantukh 
Date:   2016-12-27T10:45:29Z

Merge branches 'ignite-gg-11701' and 'ignite-gg-8.0.2.ea1' of 
https://github.com/gridgain/apache-ignite into ignite-gg-11701

commit 721f255eb2de1d8207e35328e2dec6514c22500d
Author: Ilya Lantukh 
Date:   2016-12-28T15:58:38Z

gg-11701 : Fixed preloading.

commit bed897dc01960dcbb7219ad948973e0f27bfa564
Author: Ilya Lantukh 
Date:   2016-12-28T16:31:51Z

gg-11701 : Fixed force keys request for single get.

commit 07535d92cd37cc949188434eb7ded2fc8d2e0647
Author: Ilya Lantukh 
Date:   2016-12-28T16:48:15Z

gg-11701 : Added check to multiple get.

commit ff9aa8898445c2a30e80fbc94f668eb5cd29a7d8
Author: Ilya Lantukh 
Date:   2016-12-29T13:04:49Z

Merge branches 'ignite-gg-11701' and 'ignite-gg-8.0.2.ea2' of 
https://github.com/gridgain/apache-ignite into ignite-gg-11701

commit 6cbe8a5a8ea5e4dacb8c5a859cb15aea5b3f38af
Author: Ilya Lantukh 
Date:   2017-01-09T15:22:15Z

gg-11701 : Fixed partition update counters comparison.

commit 3c276c9e092b6f6b4baf0b550ad89cb85ab9c42f
Author: Ilya Lantukh 
Date:   2017-01-10T10:25:09Z

gg-11701 : Implicit lateAffinityAssignment mode when persistence is enabled.

commit b3f7ae3bf903a0a6357163029ed310647877c8fc
Author: Ilya Lantukh 
Date:   2017-01-10T11:25:24Z

gg-11701 : Replaced redundant checks with assertions.

commit 2e55ddb600819dbf4684c0e97cc71a733167a4ce
Author: Ilya Lantukh 
Date:   2017-01-11T15:05:25Z

gg-11701 : Merge with 8.0.2.ea2

commit ecead988090e6a65ffdbb4098252ea26287fe36e
Author: Ilya Lantukh 
Date:   2017-01-11T15:09:25Z

gg-11701 : Merge with 8.0.2.ea2

commit adc0422592a18a7c2635e45185ef17852cd41952
Author: Glukos 
Date:   2017-01-13T17:23:06Z

GG-11595:
Remote exception wrapped on proxy level

commit 4505066481d4fff601a14d344dc8059f5dbe73de
Author: Ivan Rakov 
Date:   2017-01-13T21:32:08Z

GG-11595:
+ serialVersionUid

commit ba847555c86c00a288362e1cfad8c4c30883975c
Author: Ivan Rakov 
Date:   2017-01-16T15:08:35Z

Merge branch 'ignite-gg-8.0.2.ea2#' into ignite-gg-11595

# Conflicts:
#   modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/DynamicCacheChangeRequest.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheProcessor.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/IgniteCacheProxy.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/atomic/GridDhtAtomicCache.java
#   
modules/core/src/test/java/org/apache/ignite/internal/processors/cache/IgniteDynamicCacheStartSelfTest.java

commit 9e6732822061319df1e934361cfaf96e06f91573
Author:

[GitHub] ignite pull request #1962: IGNITE-5054

2017-05-17 Thread devozerov 
GitHub user devozerov opened a pull request:

https://github.com/apache/ignite/pull/1962

IGNITE-5054



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-5054-splitter

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1962.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1962






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-5245) MIN returns NULL if a numeric column of class has NULL values

2017-05-17 Thread Sergey Kozlov (JIRA) 
Sergey Kozlov created IGNITE-5245:
-

 Summary: MIN returns NULL if a numeric column of class has NULL 
values
 Key: IGNITE-5245
 URL: https://issues.apache.org/jira/browse/IGNITE-5245
 Project: Ignite
  Issue Type: Bug
  Components: sql
Affects Versions: 2.0
Reporter: Sergey Kozlov
 Fix For: 2.1


1. Use class with the numeric field (e.g. {{Short col_Short_1}})
2. Put a few entries where where that fields is null
3. Run query {{SELECT MIN(col_Short_1) FROM DefaultTable}}
4. It returns NULL but for same case h2 returns non-null minimal value



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] ignite pull request #1928: Ignite 5193

2017-05-17 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/1928


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-5244) Web Console: Add 'Download Web Agent' button

2017-05-17 Thread Vica Abramova (JIRA) 
Vica Abramova created IGNITE-5244:
-

 Summary: Web Console: Add 'Download Web Agent' button
 Key: IGNITE-5244
 URL: https://issues.apache.org/jira/browse/IGNITE-5244
 Project: Ignite
  Issue Type: Task
  Components: UI, wizards
Reporter: Vica Abramova
Assignee: Vica Abramova


No direct way to download Ignite Web Agent. Let's add the btn.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (IGNITE-5243) Disallow CREATE INDEX and DROP INDEX on statically configured caches

2017-05-17 Thread Vladimir Ozerov (JIRA) 
Vladimir Ozerov created IGNITE-5243:
---

 Summary: Disallow CREATE INDEX and DROP INDEX on statically 
configured caches
 Key: IGNITE-5243
 URL: https://issues.apache.org/jira/browse/IGNITE-5243
 Project: Ignite
  Issue Type: Task
  Components: sql
Affects Versions: 2.0
Reporter: Vladimir Ozerov
Assignee: Alexander Paschenko
 Fix For: 2.1


Same reasoning as in IGNITE-5242.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (IGNITE-5242) Disallow DROP TABLE on statically configured caches

2017-05-17 Thread Vladimir Ozerov (JIRA) 
Vladimir Ozerov created IGNITE-5242:
---

 Summary: Disallow DROP TABLE on statically configured caches
 Key: IGNITE-5242
 URL: https://issues.apache.org/jira/browse/IGNITE-5242
 Project: Ignite
  Issue Type: Task
  Components: sql
Affects Versions: 2.0
Reporter: Vladimir Ozerov
Assignee: Alexander Paschenko
 Fix For: 2.1


Should we allow {{DROP TABLE}} on statically configured cache, there will be a 
room for weird situations when it is impossible to understand on whether to 
start the cache or not.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] ignite pull request #1953: IGNITE-5208: Fixed sigfault on concurrent map acc...

2017-05-17 Thread isapego 
Github user isapego closed the pull request at:

https://github.com/apache/ignite/pull/1953


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #1946: Ignite 5131

2017-05-17 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/1946


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: IGNITE-4447 ready for review

2017-05-17 Thread Вадим Опольский 
Hi Semyon!

I had problem with starting TC test yesterday. Just have queued it -
http://ci.ignite.apache.org/project.html?projectId=IgniteTests_IgniteTests=pull%2F1952%2Fmerge

Vadim

2017-05-16 14:10 GMT+03:00 Semyon Boikov :

> Hi Vadim,
>
> Did you check test pass on TeamCity in this suite, could you please provide
> link for related TeamCity run?
>
> Thanks
>
> On Tue, May 16, 2017 at 1:35 PM, Вадим Опольский 
> wrote:
>
> > Hi guys!
> >
> > IgniteCache150ClientsTest.java was removed from
> > IgniteClientTestSuite.java to IgniteCacheTestSuite.java. Single test
> suite
> > was deleted.
> >
> > Review please this fix - https://github.com/apache/ignite/pull/1952
> >
> > Vadim Opolskii
> >
> >
> >
> > 2017-05-15 11:38 GMT+03:00 Вадим Опольский :
> >
> > > Hi guys!
> > >
> > > Vladimir, is issue https://issues.apache.org/jira/browse/IGNITE-4447
> > > actual now?
> > >
> > > Which tests do u mean? Detail please.
> > >
> > > Vadim
> > >
> >
>

Re: Minimize amount of inner classes, predicates, tuples, etc

2017-05-17 Thread ALEKSEY KUZNETSOV 
yeah, good proposal! In addition i would suggest start adding more javadocs
on current code

ср, 17 мая 2017 г. в 11:08, Vladimir Ozerov :

> Vyacheslav,
>
> When you analyze code full of generic stuff like tuples and closures, it is
> much harder to understand what is going on and navigate the code.
>
> Good coding: GridDhtAtomicCache - specific interface for callback. Easy to
> navigate and understand.
>
> interface UpdateReplyClosure extends
> CI2 {
> ...
> }
>
> Bad coding: CacheContinuousQueryHandler.AcknowledgeBuffer - overcomplicated
> generic, no JavaDocs, nullable semantics. Complexity out of nothing.
>
> @Nullable synchronized IgniteBiTuple,
> Set> onAcknowledged(GridContinuousBatch batch) {
> ...
> }
>
>
> On Tue, May 16, 2017 at 1:22 PM, Vyacheslav Daradur 
> wrote:
>
> > Vladimir, thank you for this suggestion.
> > As newcomer in the Ignite I agree with you, sometimes it is very hard to
> > understand huge classes which contain sets of inner classes.
> >
> > >> 1) Inner classes (both static and non-static)
> > Completelly agree, especially classes which will be serialized.
> >
> > >> 2) Usage of generic tuples
> > Why or in what cases?
> >
> > >> Tuples and closures should be replaced with concrete classes, specific
> > to your case.
> > Can you give an example?
> >
> > 2017-05-16 13:01 GMT+03:00 Vladimir Ozerov :
> >
> > > Igniters,
> > >
> > > Ignite is known to be complex product. Partially this complexity comes
> > from
> > > ... complex problems we are trying to resolve. But partially it comes
> > from
> > > how we write our code. I noticed several anti-patterns which makes our
> > code
> > > hard to manage:
> > >
> > > 1) Inner classes (both static and non-static)
> > > 2) Usage of generic tuples
> > > 3) Usage of many classes from our "lang"package - closures, predicates,
> > > etc..
> > >
> > > When combined these anti-patterns makes our classes huge, leaky in
> terms
> > of
> > > abstractions, hard to follow and refactor. I would like to propose to
> > > restrict usages of these constructs as much as possible in non-test
> code.
> > > Classes should be top-level by default, unless "inner" semantics are
> > > absolutely necessary or saves a lot of code. Tuples and closures should
> > be
> > > replaced with concrete classes, specific to your case.
> > >
> > > Thoughts?
> > >
> > > Vladimir.
> > >
> >
> >
> >
> > --
> > Best Regards, Vyacheslav
> >
>
-- 

*Best Regards,*

*Kuznetsov Aleksey*

Re: Minimize amount of inner classes, predicates, tuples, etc

2017-05-17 Thread Vyacheslav Daradur 
Vladimir,

Thank you for explanation and examples, it is clearly now.

Maybe it will be great to add your suggestions with explanation and
examples to the article[1] "Coding guidelines"

[1] https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines

2017-05-17 11:08 GMT+03:00 Vladimir Ozerov :

> Vyacheslav,
>
> When you analyze code full of generic stuff like tuples and closures, it is
> much harder to understand what is going on and navigate the code.
>
> Good coding: GridDhtAtomicCache - specific interface for callback. Easy to
> navigate and understand.
>
> interface UpdateReplyClosure extends
> CI2 {
> ...
> }
>
> Bad coding: CacheContinuousQueryHandler.AcknowledgeBuffer -
> overcomplicated
> generic, no JavaDocs, nullable semantics. Complexity out of nothing.
>
> @Nullable synchronized IgniteBiTuple,
> Set> onAcknowledged(GridContinuousBatch batch) {
> ...
> }
>
>
> On Tue, May 16, 2017 at 1:22 PM, Vyacheslav Daradur 
> wrote:
>
> > Vladimir, thank you for this suggestion.
> > As newcomer in the Ignite I agree with you, sometimes it is very hard to
> > understand huge classes which contain sets of inner classes.
> >
> > >> 1) Inner classes (both static and non-static)
> > Completelly agree, especially classes which will be serialized.
> >
> > >> 2) Usage of generic tuples
> > Why or in what cases?
> >
> > >> Tuples and closures should be replaced with concrete classes, specific
> > to your case.
> > Can you give an example?
> >
> > 2017-05-16 13:01 GMT+03:00 Vladimir Ozerov :
> >
> > > Igniters,
> > >
> > > Ignite is known to be complex product. Partially this complexity comes
> > from
> > > ... complex problems we are trying to resolve. But partially it comes
> > from
> > > how we write our code. I noticed several anti-patterns which makes our
> > code
> > > hard to manage:
> > >
> > > 1) Inner classes (both static and non-static)
> > > 2) Usage of generic tuples
> > > 3) Usage of many classes from our "lang"package - closures, predicates,
> > > etc..
> > >
> > > When combined these anti-patterns makes our classes huge, leaky in
> terms
> > of
> > > abstractions, hard to follow and refactor. I would like to propose to
> > > restrict usages of these constructs as much as possible in non-test
> code.
> > > Classes should be top-level by default, unless "inner" semantics are
> > > absolutely necessary or saves a lot of code. Tuples and closures should
> > be
> > > replaced with concrete classes, specific to your case.
> > >
> > > Thoughts?
> > >
> > > Vladimir.
> > >
> >
> >
> >
> > --
> > Best Regards, Vyacheslav
> >
>



-- 
Best Regards, Vyacheslav

Persistent Distributed Store Metrics

2017-05-17 Thread Dmitriy Govorukhin 
Folk,

As you know, ignite 2.1 will contain new module (pds), it will be
provide ability to store data on disk. Let's discuss what type of
metrics we need for this?
I think it must be metrics per memory policy, per cache, checkpoint,
and global metrics which will be aggregate all metrics.

I did sketch.

PersistentStoreMetrics.java

public interface PersistentStoreMetrics {

// Global metrics.

public long getMemorySize();

public long getDiskSize();

public long getPagesInMemory();

public long getPagesSizeInMemory();

public long getPagesOnDisk();

public long getPagesSizeOnDisk();

public long getFreePages();

public long getFreePagesSize();

public long getDirtyPages();

public long getDirtyPagesSize();

public long walLog();

public long walLogSize();

// Frequency.

public long getPagesRead();

public long getPagesWrite();

public long getFsync();

public long getWal();

public long getAverageWalFsyncTime();

// Per cache.

public PersistentStoreCacheMetrics cache(String name);

public PersistentStoreCacheMetrics cache(int cacheId);

// For last checkpoint.

public PersistentStoreCheckpointMetrics getLastCheckPoint();
}

>>>

PersistentStoreCacheMetrics.java

public interface PersistentStoreCacheMetrics {

public String name();

public double getFillFactor();

public double getFillFactor(int part);

public long getMemorySize();

public long getDiskSize();

public long getPagesInMemory();

public long getPagesSizeInMemory();

public long getPagesOnDisk();

public long getPagesSizeOnDisk();

public long getFreePages();

public long getFreePagesSize();

public long getDirtyPages();

public long getDirtyPagesSize();

public long getPagesRead();

public long getPagesWritten();
}

>>>

PersistentStoreCheckpointMetrics.java

public interface PersistentStoreCheckpointMetrics {

public long getTotalPages();

//TODO Page type is internal?
public long[] pagesType();

public long getExecutingTime();

public long getFsyncTime();

public long getPagesCopyOnWrite();
}

Re: GridGain Donates Persistent Distributed Store To ASF (Apache Ignite)

2017-05-17 Thread Alexey Goncharuk 
Sure, those classes will be renamed to use Ignite* prefix.

Any other comments regarding Configuration or public API changes?

2017-05-16 17:12 GMT+03:00 Alexey Kuznetsov :

> Alexey Goncharuk,
>
> I take a look at source code and noticed classes with names
> like: GridCacheDatabaseSharedManager.java
> As far as I remember we decided that "Grid" is a kind of "deprecated"
> prefix?
>
> What do you think? Does it make sense to rename?
>
> On Tue, May 16, 2017 at 9:06 PM, Alexey Goncharuk <
> alexey.goncha...@gmail.com> wrote:
>
> > Denis,
> >
> > Headers are updated, RAT check is passing now.
> >
> > 2017-05-16 2:37 GMT+03:00 Denis Magda :
> >
> > > The receipt of the software grant (the persistent store) was
> acknowledged
> > > by Craig Russel.
> > >
> > > Now, we need to move on with this
> > >
> > > > In the meanwhile, I’ve prepared the IP Clearance page referring to
> the
> > > template below but failed to commit the changes to ASF repo:
> > > > http://incubator.apache.org/ip-clearance/ip-clearance-template.html
> > > >
> > > > *Roman S.*, *Cos*, could you help me with this by granting karma or
> > > committing the form from under your account?
> > >
> > > Roman, Cos, could you help with this?
> > >
> > > *Alex G.*, please add Apache 2.0 copyrights to all source files that
> are
> > > going to be donated. Presently there is no copyright at all.
> > >
> > > Everyone interested please spend some time exploring the store's docs
> and
> > > sources shared in my previous email. If no one has any concerns I will
> > > proceed with the donation formalities.
> > >
> > > —
> > > Denis
> > >
> > > > On May 12, 2017, at 2:59 PM, Denis Magda  wrote:
> > > >
> > > > Folks,
> > > >
> > > > The repository with the donation is ready and available for review:
> > > > https://github.com/agoncharuk/ignite/tree/pds-donate
> > > >
> > > > Big and main part of the sources is aggregated in “modules/pds”. The
> > > rest, that connects Apache Ignite memory architecture and SQL engine is
> > > under “core” and “indexing” modules. Alex Goncharuk should be able to
> > point
> > > to specific files or commits if required.
> > > >
> > > > Here is a description:
> > > > * Persistent Store Overview: https://cwiki.apache.org/
> > > confluence/display/IGNITE/Persistent+Store+Overview
> > > > * Persistent Store Internal Design: https://cwiki.apache.org/
> > > confluence/display/IGNITE/Persistent+Store+Internal+Design
> > > >
> > > > The SGA will be signed and sent on Monday.
> > > >
> > > > In the meanwhile, I’ve prepared the IP Clearance page referring to
> the
> > > template below but failed to commit the changes to ASF repo:
> > > > http://incubator.apache.org/ip-clearance/ip-clearance-template.html
> > > >
> > > > *Roman S.*, *Cos*, could you help me with this by granting karma or
> > > committing the form from under your account?
> > > >
> > > > —
> > > > Denis
> > > >
> > > >> On Apr 25, 2017, at 9:56 AM, Konstantin Boudnik 
> > > wrote:
> > > >>
> > > >> While no one is suggesting an IP trap laid out in the non-SGA'ed
> code
> > > >> in this particular case, we don't want to setup a precedent like
> this.
> > > >>
> > > >> From the overall ASF perspective I +1 what Roman has just said.
> > > >>
> > > >> Thanks,
> > > >> --
> > > >> Take care,
> > > >> Konstantin (Cos) Boudnik
> > > >>
> > > >>
> > > >> On Wed, Apr 19, 2017 at 11:41 PM, Roman Shaposhnik <
> > > ro...@shaposhnik.org> wrote:
> > > >>> On Wed, Apr 19, 2017 at 6:21 PM, Dmitriy Setrakyan
> > > >>>  wrote:
> > >  On Wed, Apr 19, 2017 at 6:00 PM, Konstantin Boudnik <
> c...@apache.org
> > >
> > > wrote:
> > > 
> > > > On Tue, Apr 18, 2017 at 11:54PM, Dmitriy Setrakyan wrote:
> > > >>
> > > >> Would a standard SGA suffice here?
> > > >>
> > > >> I believe that ASF guidelines suggest that a discussion should
> > > happen
> > > >> first. Once the community gets enough information, we will move
> to
> > > a PMC
> > > >> vote. I was under the impression that once the PMC vote passes,
> > > then the
> > > >> SGA should be provided. Or does GridGain need to provide a
> signed
> > > SGA
> > > > right
> > > >> away?
> > > >
> > > > That reminds me of that Pelosi's self-inflicted conundrum of "In
> > > order
> > > > to see the bill, we should pass the bill" ;)
> > > >
> > > 
> > >  Haha :)
> > > 
> > >  SGA != code. In my view, the code should be provided to the
> > community
> > > for a
> > >  review. But I am struggling to see why should an SGA be signed
> prior
> > > to the
> > >  community accepting the donation.
> > > >>>
> > > >>> There's no such thing as SGA without a reference to a code base.
> > > >>>
> > > >>> Also, as I explained -- as a community member I would refuse to
> look
> > > >>> at the code base that doesn't have a proper licensing attached to
> it.
> > > >>> SGA

Changing cache name when using Redis protocol

2017-05-17 Thread Roman Shtykh 
Igniters,
Getting back to the discussion on how to switch between caches when using Redis 
(and potentially memcached).# Please note, that this is a problem only with a 
part of Redis data stuctures such as currently implemented STRING. For sets and 
hash tables, set/hash name can be treated as a cache name -- no problem here.
I have several questions to discuss.1. I think this can be done by introducing 
RedisConfiguration where a user can specify a cache name for STRING data.Redis 
is implemented on top of REST processing which is configured with 
ConnectorConfiguration. Is that so wrong then to have 
RedisConfiguration/MemcachedConfiguration as a part of it? (Or having a list of 
configurations for several protocols to use simultaneously in 
IgniteConfiguration instead of one -- currently only one per 
IgniteConfiguration?)
2. Putting aside memcached etc., I can implement "CONFIG SET parameter value" 
command of Redis protocol to set the cache name, so that a user, prior to 
working with Ignite via Redis, issues the command and uses the once-specified 
cache. The question -- is it safe to hold it in GridRestProcessor? Will all 
nodes see this cache name setting request and all its processors safely hold it?

Any ideas/advice on how to do it better?

Jira issue: https://issues.apache.org/jira/browse/IGNITE-5229
Roman