[GitHub] ignite pull request #3244: IGNITE-7182: throttle only changes

2017-12-15 Thread dspavlov 
GitHub user dspavlov opened a pull request:

https://github.com/apache/ignite/pull/3244

IGNITE-7182: throttle only changes

Part of implementation changes throttling policy

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite 
ignite-7182-throttle-only

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3244.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3244


commit dd8a7ac47df1dc401e5a98dc21f9a0988fb50b0f
Author: dpavlov 
Date:   2017-12-01T16:47:45Z

GG-13117: PDS compatibility test flaky failure: debug code added

commit 6c7b4fb24542672944220dca94efffa94a969ee2
Author: dspavlov 
Date:   2017-12-16T07:49:42Z

IGNITE-7182: Throttle only changes, fixes too aggressive throttling at CP 
begin




---

[jira] [Created] (IGNITE-7217) Add abilities to monitor custom thread pools

2017-12-15 Thread Valentin Kulichenko (JIRA) 
Valentin Kulichenko created IGNITE-7217:
---

 Summary: Add abilities to monitor custom thread pools
 Key: IGNITE-7217
 URL: https://issues.apache.org/jira/browse/IGNITE-7217
 Project: Ignite
  Issue Type: Improvement
  Components: general
Affects Versions: 2.3
Reporter: Valentin Kulichenko
 Fix For: 2.4


We have a periodic metrics logger that prints out different stats including the 
ones for public and system thread pools:
{noformat}
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=8, qSize=0]
{noformat}
However, is user configures a custom thread pools via 
{{IgniteConfiguration#setExecutorConfiguration}}, stats for these thread pools 
are not added. We also do not register MBeans for these pools.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Switching Java language level to 8

2017-12-15 Thread Sergey Kozlov 
Hi Denis

Filed the ticket [1]

1. IGNITE-7216 Refactor examples project


On Fri, Dec 15, 2017 at 8:45 PM, Denis Magda  wrote:

> Yuri, Sergey,
>
> These are all good points! Please create sub-tasks fro this umbrella
> tickets (one for ML and the other for examples):
> https://issues.apache.org/jira/browse/IGNITE-7203 <
> https://issues.apache.org/jira/browse/IGNITE-7203>
>
> Don’t want this to be missed in the discussion.
>
> —
> Denis
>
> > On Dec 15, 2017, at 6:03 AM, Sergey Kozlov  wrote:
> >
> > Hi
> >
> > The examples project should be refactored.
> >
> > Now we have regular and Java8 examples  and I suppose to replace regular
> > examples by corresponding Java8 examples if any.
> >
> > On Fri, Dec 15, 2017 at 3:14 PM, vveider  wrote:
> >
> >> Very good. I came to the same thoughts.
> >>
> >> Yet, there is a situation that some classes in *.java.* and .*java8.*
> >> packages that share the name and policy of merging them is not quite
> clear.
> >> Should *.java.* classes be replace by .*java8.*?
> >>
> >>
> >>
> >> --
> >> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> >>
> >
> >
> >
> > --
> > Sergey Kozlov
> > GridGain Systems
> > www.gridgain.com
>
>


-- 
Sergey Kozlov
GridGain Systems
www.gridgain.com

[jira] [Created] (IGNITE-7216) Refactor examples project

2017-12-15 Thread Sergey Kozlov (JIRA) 
Sergey Kozlov created IGNITE-7216:
-

 Summary: Refactor examples project
 Key: IGNITE-7216
 URL: https://issues.apache.org/jira/browse/IGNITE-7216
 Project: Ignite
  Issue Type: Sub-task
Reporter: Sergey Kozlov


Now we have regular and Java8 examples  and I suppose following:
* Replace regular examples by corresponding Java8 examples if any
* Remove Java8 profile
* Set source/target as for main project
* ML profile should be activated by default



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Thin Client Protocol documentation

2017-12-15 Thread Denis Magda 
Hi Andrey,

I’m afraid there are no plans for this features yet (at least I didn’t spot any 
relevant discussion on @dev).

Feel free to start a separate discussion proposing the protocol expansion. It’s 
better not to intermix this with the documentation thread.

—
Denis

> On Dec 15, 2017, at 11:12 AM, Andrey Kornev  wrote:
> 
> Pavel, could you please at least share your time frame wrt adding compute, 
> clustering and services features to the spec?
> 
> Thanks
> Andrey
> 
> From: Pavel Tupitsyn 
> Sent: Tuesday, December 5, 2017 11:15 PM
> To: dev@ignite.apache.org
> Subject: Re: Thin Client Protocol documentation
> 
> Andrey,
> 
> As I understand, you suggest to document every prospective feature right
> now.
> That would be (at least) compute, clustering, transactions, services,
> messages, events, failover, data structures, metrics.
> 
> I don't think it makes sense to do that. Another problem is that it will
> take weeks :)
> But, of course, you are welcome to take the initiative.
> 
> Thanks,
> Pavel
> 
> On Tue, Dec 5, 2017 at 10:20 PM, Andrey Kornev 
> wrote:
> 
>> Pavel,
>> 
>> I have absolutely no doubts that support for all those features will be
>> added eventually. And if so, wouldn't it be the right thing to do to
>> document the operations and their semantics right now without
>> necessarily implementing them? It should really help to ensure that the
>> protocol can accommodate all those use cases before it gets released to the
>> public.
>> 
>> Thanks
>> Andrey
>> --
>> *From:* Pavel Tupitsyn 
>> *Sent:* Tuesday, December 5, 2017 12:07 AM
>> *Cc:* dev@ignite.apache.org
>> *Subject:* Re: Thin Client Protocol documentation
>> 
>> Andrey,
>> 
>> All of this is to come :)
>> 
>> 
>> Prachi,
>> 
>> 1) There are no flags, see the doc
>> 2) String is simply 4 byte length (n) + n bytes
>> 3) Op codes have changed
>> 
>> writeByteLittleEndian(1051, out); // Type code
>> writeIntLittleEndian(20, out); // String length
>> out.writeUTF("myNewCache"); // Cache name
>> 
>> On Tue, Dec 5, 2017 at 4:39 AM, Prachi Garg  wrote:
>> 
>>> Hi Pavel,
>>> 
>>> I am trying to use the OP_CACHE_CREATE_WITH_NAME operation, but can't get
>>> it to work.  Digging deeper into the source code, it seems like I have to
>>> provide a flag, string length, and position, in addition to the type code
>>> and the actual string. Is that correct?
>>> 
>>> Here is the request I am sending to the server-
>>> 
>>> DataOutputStream out = new DataOutputStream(socket.getOutputStream());
>>> 
>>> // Message length
>>> writeIntLittleEndian(24, out);
>>> 
>>> // Op code = OP_CACHE_CREATE_WITH_NAME
>>> writeShortLittleEndian(1051, out);
>>> 
>>> // Request id
>>> long reqId = 1;
>>> writeLongLittleEndian(reqId, out);
>>> 
>>> // String (cache name)
>>> writeByteLittleEndian(9, out); // Type code
>>> writeByteLittleEndian(0, out); // Flag
>>> writeIntLittleEndian(20, out); // String length
>>> writeIntLittleEndian(0, out); // Position
>>> out.writeUTF("myNewCache"); // Cache name
>>> 
>>> // Send request
>>> out.flush();
>>> 
>>> But I get the following error on the server side.
>>> 
>>> [2017-12-04 17:27:39,421][ERROR][client-connector-#53][
>> ClientListenerNioListener]
>>> Failed to parse client request.
>>> java.lang.StringIndexOutOfBoundsException: String index out of range:
>> 2575
>>> at java.lang.String.checkBounds(String.java:385)
>>> at java.lang.String.(String.java:462)
>>> at org.apache.ignite.internal.binary.BinaryUtils.
>>> doReadString(BinaryUtils.java:1314)
>>> at org.apache.ignite.internal.binary.BinaryReaderExImpl.
>>> readString(BinaryReaderExImpl.java:1055)
>>> at org.apache.ignite.internal.processors.platform.client.cache.
>>> ClientCacheCreateWithNameRequest.(ClientCacheCreateWithNameReque
>>> st.java:43)
>>> at org.apache.ignite.internal.processors.platform.client.
>>> ClientMessageParser.decode(ClientMessageParser.java:318)
>>> at org.apache.ignite.internal.processors.platform.client.
>>> ClientMessageParser.decode(ClientMessageParser.java:220)
>>> at org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.
>>> onMessage(ClientListenerNioListener.java:119)
>>> at org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.
>>> onMessage(ClientListenerNioListener.java:40)
>>> at org.apache.ignite.internal.util.nio.GridNioFilterChain$
>>> TailFilter.onMessageReceived(GridNioFilterChain.java:279)
>>> at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.
>>> proceedMessageReceived(GridNioFilterAdapter.java:109)
>>> at org.apache.ignite.internal.util.nio.GridNioAsyncNotifyFilter$3.
>>> body(GridNioAsyncNotifyFilter.java:97)
>>> at org.apache.ignite.internal.util.worker.GridWorker.run(
>>> GridWorker.java:110)
>>> at org.apache.ignite.internal.util.worker.GridWorkerPool$1.
>>> run(GridWorkerPool.java:70)
>>> at

Re: Thin Client Protocol documentation

2017-12-15 Thread Andrey Kornev 
Pavel, could you please at least share your time frame wrt adding compute, 
clustering and services features to the spec?

Thanks
Andrey

From: Pavel Tupitsyn 
Sent: Tuesday, December 5, 2017 11:15 PM
To: dev@ignite.apache.org
Subject: Re: Thin Client Protocol documentation

Andrey,

As I understand, you suggest to document every prospective feature right
now.
That would be (at least) compute, clustering, transactions, services,
messages, events, failover, data structures, metrics.

I don't think it makes sense to do that. Another problem is that it will
take weeks :)
But, of course, you are welcome to take the initiative.

Thanks,
Pavel

On Tue, Dec 5, 2017 at 10:20 PM, Andrey Kornev 
wrote:

> Pavel,
>
> I have absolutely no doubts that support for all those features will be
> added eventually. And if so, wouldn't it be the right thing to do to
> document the operations and their semantics right now without
> necessarily implementing them? It should really help to ensure that the
> protocol can accommodate all those use cases before it gets released to the
> public.
>
> Thanks
> Andrey
> --
> *From:* Pavel Tupitsyn 
> *Sent:* Tuesday, December 5, 2017 12:07 AM
> *Cc:* dev@ignite.apache.org
> *Subject:* Re: Thin Client Protocol documentation
>
> Andrey,
>
> All of this is to come :)
>
>
> Prachi,
>
> 1) There are no flags, see the doc
> 2) String is simply 4 byte length (n) + n bytes
> 3) Op codes have changed
>
> writeByteLittleEndian(1051, out); // Type code
> writeIntLittleEndian(20, out); // String length
> out.writeUTF("myNewCache"); // Cache name
>
> On Tue, Dec 5, 2017 at 4:39 AM, Prachi Garg  wrote:
>
> > Hi Pavel,
> >
> > I am trying to use the OP_CACHE_CREATE_WITH_NAME operation, but can't get
> > it to work.  Digging deeper into the source code, it seems like I have to
> > provide a flag, string length, and position, in addition to the type code
> > and the actual string. Is that correct?
> >
> > Here is the request I am sending to the server-
> >
> > DataOutputStream out = new DataOutputStream(socket.getOutputStream());
> >
> > // Message length
> > writeIntLittleEndian(24, out);
> >
> > // Op code = OP_CACHE_CREATE_WITH_NAME
> > writeShortLittleEndian(1051, out);
> >
> > // Request id
> > long reqId = 1;
> > writeLongLittleEndian(reqId, out);
> >
> > // String (cache name)
> > writeByteLittleEndian(9, out); // Type code
> > writeByteLittleEndian(0, out); // Flag
> > writeIntLittleEndian(20, out); // String length
> > writeIntLittleEndian(0, out); // Position
> > out.writeUTF("myNewCache"); // Cache name
> >
> > // Send request
> > out.flush();
> >
> > But I get the following error on the server side.
> >
> > [2017-12-04 17:27:39,421][ERROR][client-connector-#53][
> ClientListenerNioListener]
> > Failed to parse client request.
> > java.lang.StringIndexOutOfBoundsException: String index out of range:
> 2575
> > at java.lang.String.checkBounds(String.java:385)
> > at java.lang.String.(String.java:462)
> > at org.apache.ignite.internal.binary.BinaryUtils.
> > doReadString(BinaryUtils.java:1314)
> > at org.apache.ignite.internal.binary.BinaryReaderExImpl.
> > readString(BinaryReaderExImpl.java:1055)
> > at org.apache.ignite.internal.processors.platform.client.cache.
> > ClientCacheCreateWithNameRequest.(ClientCacheCreateWithNameReque
> > st.java:43)
> > at org.apache.ignite.internal.processors.platform.client.
> > ClientMessageParser.decode(ClientMessageParser.java:318)
> > at org.apache.ignite.internal.processors.platform.client.
> > ClientMessageParser.decode(ClientMessageParser.java:220)
> > at org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.
> > onMessage(ClientListenerNioListener.java:119)
> > at org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.
> > onMessage(ClientListenerNioListener.java:40)
> > at org.apache.ignite.internal.util.nio.GridNioFilterChain$
> > TailFilter.onMessageReceived(GridNioFilterChain.java:279)
> > at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.
> > proceedMessageReceived(GridNioFilterAdapter.java:109)
> > at org.apache.ignite.internal.util.nio.GridNioAsyncNotifyFilter$3.
> > body(GridNioAsyncNotifyFilter.java:97)
> > at org.apache.ignite.internal.util.worker.GridWorker.run(
> > GridWorker.java:110)
> > at org.apache.ignite.internal.util.worker.GridWorkerPool$1.
> > run(GridWorkerPool.java:70)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1142)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > What am I missing here? Can you provide an example of how to send a
> > 'String' type request?
> >
> > -Prachi
> >
> > On Mon, Dec 4, 2017 at 1:06 PM, Andrey Kornev 
> > wrote:
> >
> >> Pavel,
> >>
> >> Thanks! While

Re: [ARTICLE] In-Memory Technologies: Meeting Healthcare's Fast Data Challenges. Part I

2017-12-15 Thread Denis Magda 
Excellent article! Look forward to the second part where to learn more about a 
real-world use case powered by Ignite.

—
Denis

> On Dec 15, 2017, at 9:13 AM, Akmal Chaudhri  
> wrote:
> 
> Part 1 of a 2-part series is now live:
> 
> https://www.gridgain.com/resources/blog/in-memory-technologies-meeting-healthcares-fast-data-challenges-part-i
> 
> Thank you.

Re: Switching Java language level to 8

2017-12-15 Thread Denis Magda 
Yuri, Sergey,

These are all good points! Please create sub-tasks fro this umbrella tickets 
(one for ML and the other for examples):
https://issues.apache.org/jira/browse/IGNITE-7203 


Don’t want this to be missed in the discussion.

—
Denis

> On Dec 15, 2017, at 6:03 AM, Sergey Kozlov  wrote:
> 
> Hi
> 
> The examples project should be refactored.
> 
> Now we have regular and Java8 examples  and I suppose to replace regular
> examples by corresponding Java8 examples if any.
> 
> On Fri, Dec 15, 2017 at 3:14 PM, vveider  wrote:
> 
>> Very good. I came to the same thoughts.
>> 
>> Yet, there is a situation that some classes in *.java.* and .*java8.*
>> packages that share the name and policy of merging them is not quite clear.
>> Should *.java.* classes be replace by .*java8.*?
>> 
>> 
>> 
>> --
>> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>> 
> 
> 
> 
> -- 
> Sergey Kozlov
> GridGain Systems
> www.gridgain.com

[GitHub] ignite pull request #3242: IGNITE-7213: Empty class descriptions for KNNMode...

2017-12-15 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3242


---

[GitHub] ignite pull request #3243: Ignite 7182: buckets (experimental)

2017-12-15 Thread dspavlov 
GitHub user dspavlov opened a pull request:

https://github.com/apache/ignite/pull/3243

Ignite 7182: buckets (experimental)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7182-buckets

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3243.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3243


commit 45f6340a518d8580c96614e0c12eb379a2c6202d
Author: dpavlov 
Date:   2017-12-13T15:43:39Z

IGNITE-7182: async sorting of CP pages first variant

commit c088390173d04e5be57d3f91bc602733466702da
Author: dpavlov 
Date:   2017-12-13T19:23:45Z

IGNITE-7182: lazy transfer of quick-sorted values to async checkpointer, WIP

commit 7643854c87d81a18239489bf75047add6168929a
Author: dpavlov 
Date:   2017-12-14T12:12:35Z

IGNITE-7182: lazy transfer of quick-sorted values to async checkpointer, WIP

commit dd4894f30f47ba86c505bc42e650de44ab6cee57
Author: dpavlov 
Date:   2017-12-14T12:59:01Z

IGNITE-7182: checkpoint scope class instead of T2

commit 640997dae774be48caebb18e9252b76164668a3d
Author: dpavlov 
Date:   2017-12-14T13:54:39Z

IGNITE-7182: WriteCpPages made callable, reporting to CountDownFuture 
migrated to async checkpointer

commit a2c6bb70332b6b4b458ebde9437c0b9561832ec0
Author: dpavlov 
Date:   2017-12-14T14:43:20Z

IGNITE-7182: quick sort emulating fork join on regular thread

commit a0ca351b01d48bd44173bf8b8e3544dbe8f51274
Author: dpavlov 
Date:   2017-12-14T16:57:07Z

IGNITE-7182: preparedPages was refactored to use arrays

commit 882bbff4314b775fb9406abcb6c4f8785a889eac
Author: dspavlov 
Date:   2017-12-14T20:43:24Z

IGNITE-7182: Checkpoint pages: optimize and parallel sorting

commit aa9eae1b3b067ace1497a180723290621a14df10
Author: dspavlov 
Date:   2017-12-14T20:55:07Z

IGNITE-7182: Checkpoint pages: optimize and parallel sorting

commit e99c3dd6e0b906eae19e4da20d87a3ce7a08600d
Author: dpavlov 
Date:   2017-12-01T16:47:45Z

GG-13117: PDS compatibility test flaky failure: debug code added

commit 00c3e6cf4a105b1dc2f32b3c27c1bed1f3947def
Author: dspavlov 
Date:   2017-12-14T21:01:58Z

Merge branch 'master' into ignite-7182

# Conflicts:
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheDatabaseSharedManager.java

commit d7e21dfc13a544ebcd98b7ffc57e0761e2f12b20
Author: dspavlov 
Date:   2017-12-14T21:09:17Z

IGNITE-7182: Merge fix

commit 42f8bc5f2763db8b0bc63544264b302d2c144fb4
Author: dspavlov 
Date:   2017-12-15T07:01:23Z

IGNITE-7182: new code stubs

commit 745be9b4be857943c8c74a0fc7b45b17f7fdcaf0
Author: dspavlov 
Date:   2017-12-15T07:05:08Z

IGNITE-7182: cosmetic

commit 8da055ebd4a24ce2dcde15715a0647fc82c3b019
Author: dspavlov 
Date:   2017-12-15T07:34:48Z

IGNITE-7182: initial calculation for rates

commit c57c928234af6d5ee14d8cadfc35e366d3d4ccb8
Author: dpavlov 
Date:   2017-12-15T10:05:41Z

IGNITE-7182: fork now/later strategy implemented

commit 366042eb75f9134b299a61a0cfaedd5d3e9e0c63
Author: dpavlov 
Date:   2017-12-15T10:51:31Z

IGNITE-7182: fork now/later strategy applied for splitting writer chunks

commit fc4a840c0931e8a6c8ca2296b45621f810a7fc0b
Author: dpavlov 
Date:   2017-12-15T11:03:20Z

IGNITE-7182: prepare to review.

commit bf6eb5bded2a12684e77efb27bc8ab1ce48b32e4
Author: dpavlov 
Date:   2017-12-15T11:28:45Z

IGNITE-7182: Remove odd sorting under lock

commit 53872f203af6d3bf703e2c10e2fb8ab80562ea25
Author: dpavlov 
Date:   2017-12-15T12:39:19Z

IGNITE-7182: calc CRC restored

commit 464f8d5fb68ec82314025332908ccd700202d7d7
Author: dpavlov 
Date:   2017-12-15T15:09:31Z

IGNITE-7182: Parallel eviction problem fixed

commit 0d7853dbd1be85c015c7b80446ee1d186357c13d
Author: dpavlov 
Date:   2017-12-15T15:21:09Z

IGNITE-7182: Take queue size to account

commit f0e8c3c4abee905e4a41d89330f65bb67ed95312
Author: dpavlov 
Date:   2017-12-15T16:12:38Z

IGNITE-7182: Fork strategy changed

commit ff73e2b692608249e885391b74b30f51717cbb06
Author: dpavlov 
Date:   2017-12-15T17:18:33Z

IGNITE-7182: Buckets created for cache+partition hash code in dirty pages




---

[GitHub] ignite pull request #3242: IGNITE-7213: Empty class descriptions for KNNMode...

2017-12-15 Thread ybabak 
GitHub user ybabak opened a pull request:

https://github.com/apache/ignite/pull/3242

IGNITE-7213: Empty class descriptions for KNNModelFormat

fixed.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7213

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3242.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3242


commit fefebd90dd1d13f4a4c6886eab97f12a19fca30d
Author: YuriBabak 
Date:   2017-12-15T17:07:44Z

IGNITE-7213: Empty class descriptions for KNNModelFormat

fixed.




---

[ARTICLE] In-Memory Technologies: Meeting Healthcare's Fast Data Challenges. Part I

2017-12-15 Thread Akmal Chaudhri 
Part 1 of a 2-part series is now live:

https://www.gridgain.com/resources/blog/in-memory-technologies-meeting-healthcares-fast-data-challenges-part-i

Thank you.

[jira] [Created] (IGNITE-7215) Documentation bug: "Cross-cache Query" page is dead

2017-12-15 Thread Alexey Kukushkin (JIRA) 
Alexey Kukushkin created IGNITE-7215:


 Summary: Documentation bug: "Cross-cache Query" page is dead
 Key: IGNITE-7215
 URL: https://issues.apache.org/jira/browse/IGNITE-7215
 Project: Ignite
  Issue Type: Bug
  Components: documentation
Affects Versions: 2.3
Reporter: Alexey Kukushkin


Could not find "Cross-cache Queries" section in the latest Ignite 2.3 docs. The 
only page that references "cross-cache queries" is [JDBC Client 
Driver|https://apacheignite-sql.readme.io/docs#section-jdbc-client-driver] and 
it points to a [dead 
link|https://apacheignite-sql.readme.io/docs/cache-queries#cross-cache-queries]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-7214) performance measurement for FCM and KNN algorithms

2017-12-15 Thread Oleg Ignatenko (JIRA) 
Oleg Ignatenko created IGNITE-7214:
--

 Summary: performance measurement for FCM and KNN algorithms
 Key: IGNITE-7214
 URL: https://issues.apache.org/jira/browse/IGNITE-7214
 Project: Ignite
  Issue Type: Task
  Components: ml, yardstick
Reporter: Oleg Ignatenko


We want to start tracking our performance of Fuzzy c-means (FCM) and k nearest 
neighbours (KNN) to avoid performance degradation. Also we need some 
performance comparison with other ml libs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-7213) Empty class descriptions for KNNModelFormat

2017-12-15 Thread Yury Babak (JIRA) 
Yury Babak created IGNITE-7213:
--

 Summary: Empty class descriptions for KNNModelFormat
 Key: IGNITE-7213
 URL: https://issues.apache.org/jira/browse/IGNITE-7213
 Project: Ignite
  Issue Type: Bug
  Components: ml
Reporter: Yury Babak
Assignee: Yury Babak
Priority: Critical
 Fix For: 2.4


Javadoc generation failed if we have classes with empty class-javadoc



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: IgniteProjectionStartStopRestartSelfTest is failing for about a week on TC

2017-12-15 Thread Alexey Goncharuk 
This may be related to the recent changes in TC infrastructure. I see the
following in remote nodes output:

nohup: ignoring input
/data/teamcity/work/820be461cd64b574/bin/ignite.sh, ERROR:
The version of JAVA installed in JAVA_HOME= is incorrect.
Please point JAVA_HOME variable to installation of JDK 1.7 or JDK 1.8.
You can also download latest JDK at http://java.com/download


Sergey, Petr, can you take a look?

2017-12-15 15:23 GMT+03:00 Petr Ivanov :

> According to Failure Conditions [1] — it is designed to fail after 30 min.
> So the question is - should we consider 30 min run time to be project’s
> error (and fix the code) or TeamCity error (and fix the build configuration
> by, for example, increasing build failure timeout)?
>
>
> [1] https://ci.ignite.apache.org/admin/editBuildFailureConditions.
> html?id=buildType:Ignite20Tests_IgniteStartNodes <
> https://ci.ignite.apache.org/admin/editBuildFailureConditions.
> html?id=buildType:Ignite20Tests_IgniteStartNodes>
>
>
> > On 15 Dec 2017, at 14:50, Dmitry Pavlov  wrote:
> >
> > Hi Igniters,
> >
> > Who can became Jedi of TeamCity and substitute me in this research?
> >
> > Sincerely,
> > Dmitry Pavlov
> >
> > пт, 15 дек. 2017 г. в 12:38, Anton Vinogradov  >:
> >
> >> Dmitry,
> >>
> >> Could you please investigate?
> >>
> >> On Fri, Dec 15, 2017 at 9:16 AM, Andrey Kuznetsov 
> >> wrote:
> >>
> >>> Hi Anton,
> >>>
> >>> I doubt it was merge conflict. Couple of days ago 'Ignite Start Nodes'
> >>> suite succeeded 1 time after unrelated changes, and now it is failing
> >>> again.
> >>>
> >>> 2017-12-12 16:25 GMT+03:00 Anton Vinogradov  >:
> >>>
>  Hi,
> 
>  Seems we have merge conflict.
> 
>  https://ci.ignite.apache.org/viewType.html?buildTypeId=Ignite20Tests_
>  IgniteStartNodes=buildTypeHistoryList_
>  Ignite20Tests=%3Cdefault%3E
> 
> 
>  --
> >>> Best regards,
> >>>  Andrey Kuznetsov.
> >>>
> >>
>
>

[GitHub] ignite pull request #3241: IGNITE-7097 performance measurement for SparseDis...

2017-12-15 Thread oignatenko 
GitHub user oignatenko opened a pull request:

https://github.com/apache/ignite/pull/3241

IGNITE-7097 performance measurement for SparseDistributedMatrix multi…

…plication

- optimised and unoptimised benchmarks, along with respective documentation
- also a common adjustment to matrix mul benchmarks to make these more 
sensitive to algorithmic changes
-- verified with diffs overview, clean rebuild, trial execution of the 
benchmarks and studying results

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7097

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3241.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3241


commit be918641730af3738dcc4d8cd608b63ba6088732
Author: Oleg Ignatenko 
Date:   2017-12-15T15:58:43Z

IGNITE-7097 performance measurement for SparseDistributedMatrix 
multiplication
- optimised and unoptimised benchmarks, along with respective documentation
- also a common adjustment to matrix mul benchmarks to make these more 
sensitive to algorithmic changes
-- verified with diffs overview, clean rebuild, trial execution of the 
benchmarks and studying results




---

IGNITE-7203 Java 8 by default

2017-12-15 Thread Petr Ivanov 
Hi, all.

Finished IGNITE-7203 [1] and prepared PR [2]
Test TeamCity run can be watched here [3]


[1] https://issues.apache.org/jira/browse/IGNITE-7203 

[2] https://github.com/apache/ignite/pull/3240 

[3] 
https://ci.ignite.apache.org/viewType.html?buildTypeId=Ignite20TestsJava8_RunAll=buildTypeChains_Ignite20TestsJava8=__all_branches__#_expand=block_bt639-1001108=0=0

Re: Rework locking architecture for MVCC and transactional SQL

2017-12-15 Thread Alexey Goncharuk 
That would be great, actually, We have a lot of data structures which size
heavily depends on the topology size and caches number, and all those data
structures are placed on-heap. I am looking forward moving that stuff to a
limited memory region with disk offload.

The DataRegion and disk offload are fairly easy to implement - we have
almost all mechanics in the current page memory implementation. We only
need to get rid of copy-on-write and checkpointing logic and alter page
eviction mechanics a little bit.

Another side note - we can look into range locks mechanics. It will be a
greater effort to implement them, but I think it worth it.

2017-12-15 16:59 GMT+03:00 Vladimir Ozerov :

> Well, there is completely different approach to handle the problem - lock
> escalation. E.g. SQL Server typically require 96 bytes per lock. Much less
> than Ignite, but still. And as soon as some query require more than 5000
> row locks, it is escalated to exclusive table lock. Not only this
> eliminates memory consumption problem, it increases performance of massive
> updates dramatically.
> On the other hand this approach is more prone to deadlocks, since
> transactions updating disjoint data sets now have shared resources to be
> locked.
>
> There is no silver bullet apparently.
>
> On Fri, Dec 15, 2017 at 4:42 PM, Vladimir Ozerov 
> wrote:
>
> > Alex,
> >
> > That might be very good idea. In fact, what you describe closely
> resembles
> > TempDB in MS SQL Server [1]. It is also offloaded to disk, minimally
> logged
> > and purged on restart. Ideally we could try designing this component in
> > generic way, so that it could store a lot different temporal stuff:
> > 1) Locks
> > 2) UNDO data
> > 3) Sort/join data (for SELECT and CREATE INDEX, statistics, whatsoever)
> > 4) If needed - visibility info (e.g. for covering indexes and
> purge/vacuum)
> >
> > WDYT?
> >
> > Vladimir.
> >
> > [1] https://docs.microsoft.com/en-us/sql/relational-
> > databases/databases/tempdb-database
> >
> > On Fri, Dec 15, 2017 at 4:26 PM, Alexey Goncharuk <
> > alexey.goncha...@gmail.com> wrote:
> >
> >> Vladimir,
> >>
> >> What about moving the entire locking mechanism to a separate off-heap
> >> memory region which will be volatile wrt restarts, but will still
> support
> >> off-load to disk. In the current architecture, it means that we will
> need
> >> to allocate a separate DataRegion with no WAL and no crash recovery -
> >> locks
> >> are meaningless after a restart, and we will automatically drop them. I
> >> would be interesting to prototype this because I think we may be on-par
> >> with on-heap lock placement, as we already proved for in-memory caches.
> >>
> >> 2017-12-14 21:53 GMT+03:00 Denis Magda :
> >>
> >> > Vladimir,
> >> >
> >> > No it’s crystal clear, thanks.
> >> >
> >> > If this approach works only for Ignite persistence based deployment,
> how
> >> > will we handle locking for pure in-memory and caching of 3rd party
> >> > databases scenarios? As I understand the tuples still will be stored
> in
> >> the
> >> > page memory while there won’t be any opportunity to fallback to disk
> if
> >> the
> >> > memory usage increases some threshold.
> >> >
> >> > —
> >> > Denis
> >> >
> >> > > On Dec 13, 2017, at 11:21 PM, Vladimir Ozerov  >
> >> > wrote:
> >> > >
> >> > > Denis,
> >> > >
> >> > > Sorry, may be I was not clear enough - "tuple-approach" and
> >> "persistent
> >> > > approach" are the same. By "tuple" I mean a row stored inside a data
> >> > block.
> >> > > Currently we store lock information in Java heap and proposal is to
> >> move
> >> > it
> >> > > to data blocks. The main driver is memory - if there are a rows to
> be
> >> > > locked we will either run out of memory, or produce serious memory
> >> > > pressure. For example, currently update of 1M entries will consume
> >> ~500Mb
> >> > > of heap. With proposed approach it will consume almost nothing. The
> >> > > drawback is increased number of dirty data pages, but it should not
> >> be a
> >> > > problem because in final implementation we will update data rows
> >> before
> >> > > prepare phase anyway, so I do not expect any write amplification in
> >> usual
> >> > > case.
> >> > >
> >> > > This approach is only applicable for Ignite persistence.
> >> > >
> >> > > On Thu, Dec 14, 2017 at 1:53 AM, Denis Magda 
> >> wrote:
> >> > >
> >> > >> Vladimir,
> >> > >>
> >> > >> Thanks for a throughout overview and proposal.
> >> > >>
> >> > >>> Also we could try employing tiered approach
> >> > >>> 1) Try to keep everything in-memory to minimize writes to blocks
> >> > >>> 2) Fallback to persistent lock data if certain threshold is
> reached.
> >> > >>
> >> > >> What are the benefits of the backed-by-persistence approach in
> >> compare
> >> > to
> >> > >> the one based on tuples? Specifically:
> >> > >> - will the persistence approach work for both 3rd party and

[GitHub] ignite pull request #3240: IGNITE-7203 Java 8 by default

2017-12-15 Thread vveider 
GitHub user vveider opened a pull request:

https://github.com/apache/ignite/pull/3240

IGNITE-7203 Java 8 by default



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7203

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3240.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3240


commit 1fd200e5ca712f7ce306807904cccd8b72e171ea
Author: Ivanov Petr 
Date:   2017-12-15T14:35:35Z

IGNITE-7203 Java 8 by default
 * got rid of java-1.8 related profiles
 * remove ml profile and included ml build into main reactor
 * added master properties to maven-compiler-plugin for whole project to be 
compiled only by java-1.8
 * refactored met properties + gathered them in parent module

commit a4db11b84622ec6960de3f44502cf08abe3dc650
Author: Ivanov Petr 
Date:   2017-12-15T15:17:59Z

IGNITE-7203 Java 8 by default
 * refactored examples module: merged java8 and java packages




---

Re: Switching Java language level to 8

2017-12-15 Thread Sergey Kozlov 
Hi

The examples project should be refactored.

Now we have regular and Java8 examples  and I suppose to replace regular
examples by corresponding Java8 examples if any.

On Fri, Dec 15, 2017 at 3:14 PM, vveider  wrote:

> Very good. I came to the same thoughts.
>
> Yet, there is a situation that some classes in *.java.* and .*java8.*
> packages that share the name and policy of merging them is not quite clear.
> Should *.java.* classes be replace by .*java8.*?
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>



-- 
Sergey Kozlov
GridGain Systems
www.gridgain.com

[GitHub] ignite pull request #3239: ignite-2.4.1-p6

2017-12-15 Thread agoncharuk 
GitHub user agoncharuk opened a pull request:

https://github.com/apache/ignite/pull/3239

ignite-2.4.1-p6



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-2.4.1-p6

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3239.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3239


commit e7ca9b65a68de7752195c8f4d2b5180f3c77d19f
Author: Dmitriy Govorukhin 
Date:   2017-11-13T18:52:47Z

ignite-blt-merge -> ignite-2.4.1

commit cc8168fc184bb7f5e3cc3bbb0743397097f78bfb
Author: Dmitriy Govorukhin 
Date:   2017-11-13T19:13:01Z

merge ignite-pitr-rc1 -> ignite-2.4.1

commit 87e6d74cf6a251c7984f9e68c391f790feccc281
Author: Dmitriy Govorukhin 
Date:   2017-11-14T12:49:33Z

ignite-gg-12877 Compact consistent ID in WAL

commit 9f5a22711baea05bd37ab07c8f928a4837dd83a4
Author: Ilya Lantukh 
Date:   2017-11-14T14:12:28Z

Fixed javadoc.

commit d5af2d78dd8eef8eca8ac5391d31d8c779649bb0
Author: Alexey Kuznetsov 
Date:   2017-11-15T08:09:00Z

IGNITE-6913 Baseline: Added new options to controls.sh for baseline 
manipulations.

commit 713924ce865752b6e99b03bd624136541cea5f9f
Author: Sergey Chugunov 
Date:   2017-11-15T09:03:12Z

IGNITE-5850 failover tests for cache operations during BaselineTopology 
changes

commit b65fd134e748d496f732ec2aa0953a0531f544b8
Author: Ilya Lantukh 
Date:   2017-11-15T12:54:35Z

TX read logging if PITR is enabled.

commit 9b2a567c0e04dc33116b51f88bee75f76e9107d1
Author: Ilya Lantukh 
Date:   2017-11-15T13:45:16Z

TX read logging if PITR is enabled.

commit 993058ccf0b2b8d9e80750c3e45a9ffa31d85dfa
Author: Dmitriy Govorukhin 
Date:   2017-11-15T13:51:54Z

ignite-2.4.1 optimization for store full set node more compacted

commit 1eba521f608d39967aec376b397b7fc800234e54
Author: Dmitriy Govorukhin 
Date:   2017-11-15T13:52:22Z

Merge remote-tracking branch 'professional/ignite-2.4.1' into ignite-2.4.1

commit 564b3fd51f8a7d1d81cb6874df66d0270623049c
Author: Sergey Chugunov 
Date:   2017-11-15T14:00:51Z

IGNITE-5850 fixed issue with initialization of data regions on node 
activation, fixed issue with auto-activation when random node joins inactive 
cluster with existing BLT

commit c6d1fa4da7adfadc80abdc7eaf6452b86a4f6aa4
Author: Sergey Chugunov 
Date:   2017-11-15T16:23:08Z

IGNITE-5850 transitionResult is set earlier when request for changing 
BaselineTopology is sent

commit d65674363163e38a4c5fdd73d1c8d8e1c7610797
Author: Sergey Chugunov 
Date:   2017-11-16T11:59:07Z

IGNITE-5850 new failover tests for changing BaselineTopology up (new node 
added to topology)

commit 20552f3851fe8825191b144179be032965e0b5c6
Author: Sergey Chugunov 
Date:   2017-11-16T12:53:43Z

IGNITE-5850 improved error message when online node is removed from baseline

commit 108bbcae4505ac904a6db774643ad600bfb42c21
Author: Sergey Chugunov 
Date:   2017-11-16T13:45:52Z

IGNITE-5850 BaselineTopology should not change on cluster deactivation

commit deb641ad3bdbf260fa60ad6bf607629652e324bd
Author: Dmitriy Govorukhin 
Date:   2017-11-17T09:45:44Z

ignite-2.4.1 truncate wal and checkpoint history on move/delete snapshot

commit 3c8b06f3659af30d1fd148ccc0f40e216a56c998
Author: Alexey Goncharuk 
Date:   2017-11-17T12:48:12Z

IGNITE-6947 Abandon remap after single map if future is done (fixes NPE)

commit ba2047e5ae7d271a677e0c418375d82d78c4023e
Author: devozerov 
Date:   2017-11-14T12:26:31Z

IGNITE-6901: Fixed assertion during 
IgniteH2Indexing.rebuildIndexesFromHash. This closes #3027.

commit abfc0466d6d61d87255d0fe38cbdf11ad46d4f89
Author: Sergey Chugunov 
Date:   2017-11-17T13:40:57Z

IGNITE-5850 tests for queries in presence of BaselineTopology

commit f4eabaf2a905abacc4c60c01d3ca04f6ca9ec188
Author: Sergey Chugunov 
Date:   2017-11-17T17:23:02Z

IGNITE-5850 implementation for setBaselineTopology(long topVer) migrated 
from wc-251

commit 4edeccd3e0b671aa277f58995df9ff9935baa95a
Author: EdShangGG 
Date:   2017-11-17T18:21:17Z

GG-13074 Multiple snapshot test failures after baseline topology is 
introduced
-adding baseline test to suite
-fixing issues with baseline

commit edae228c8f55990c15ef3044be987dcb00d6c81a
Author: EdShangGG

Re: Rework locking architecture for MVCC and transactional SQL

2017-12-15 Thread Vladimir Ozerov 
Alex,

That might be very good idea. In fact, what you describe closely resembles
TempDB in MS SQL Server [1]. It is also offloaded to disk, minimally logged
and purged on restart. Ideally we could try designing this component in
generic way, so that it could store a lot different temporal stuff:
1) Locks
2) UNDO data
3) Sort/join data (for SELECT and CREATE INDEX, statistics, whatsoever)
4) If needed - visibility info (e.g. for covering indexes and purge/vacuum)

WDYT?

Vladimir.

[1]
https://docs.microsoft.com/en-us/sql/relational-databases/databases/tempdb-database

On Fri, Dec 15, 2017 at 4:26 PM, Alexey Goncharuk <
alexey.goncha...@gmail.com> wrote:

> Vladimir,
>
> What about moving the entire locking mechanism to a separate off-heap
> memory region which will be volatile wrt restarts, but will still support
> off-load to disk. In the current architecture, it means that we will need
> to allocate a separate DataRegion with no WAL and no crash recovery - locks
> are meaningless after a restart, and we will automatically drop them. I
> would be interesting to prototype this because I think we may be on-par
> with on-heap lock placement, as we already proved for in-memory caches.
>
> 2017-12-14 21:53 GMT+03:00 Denis Magda :
>
> > Vladimir,
> >
> > No it’s crystal clear, thanks.
> >
> > If this approach works only for Ignite persistence based deployment, how
> > will we handle locking for pure in-memory and caching of 3rd party
> > databases scenarios? As I understand the tuples still will be stored in
> the
> > page memory while there won’t be any opportunity to fallback to disk if
> the
> > memory usage increases some threshold.
> >
> > —
> > Denis
> >
> > > On Dec 13, 2017, at 11:21 PM, Vladimir Ozerov 
> > wrote:
> > >
> > > Denis,
> > >
> > > Sorry, may be I was not clear enough - "tuple-approach" and "persistent
> > > approach" are the same. By "tuple" I mean a row stored inside a data
> > block.
> > > Currently we store lock information in Java heap and proposal is to
> move
> > it
> > > to data blocks. The main driver is memory - if there are a rows to be
> > > locked we will either run out of memory, or produce serious memory
> > > pressure. For example, currently update of 1M entries will consume
> ~500Mb
> > > of heap. With proposed approach it will consume almost nothing. The
> > > drawback is increased number of dirty data pages, but it should not be
> a
> > > problem because in final implementation we will update data rows before
> > > prepare phase anyway, so I do not expect any write amplification in
> usual
> > > case.
> > >
> > > This approach is only applicable for Ignite persistence.
> > >
> > > On Thu, Dec 14, 2017 at 1:53 AM, Denis Magda 
> wrote:
> > >
> > >> Vladimir,
> > >>
> > >> Thanks for a throughout overview and proposal.
> > >>
> > >>> Also we could try employing tiered approach
> > >>> 1) Try to keep everything in-memory to minimize writes to blocks
> > >>> 2) Fallback to persistent lock data if certain threshold is reached.
> > >>
> > >> What are the benefits of the backed-by-persistence approach in compare
> > to
> > >> the one based on tuples? Specifically:
> > >> - will the persistence approach work for both 3rd party and Ignite
> > >> persistence?
> > >> - any performance impacts depending on a chosen method?
> > >> - what’s faster to implement?
> > >>
> > >> —
> > >> Denis
> > >>
> > >>> On Dec 13, 2017, at 2:10 AM, Vladimir Ozerov 
> > >> wrote:
> > >>>
> > >>> Igniters,
> > >>>
> > >>> As you probably we know we work actively on MVCC [1] and
> transactional
> > >> SQL
> > >>> [2] features which could be treated as a single huge improvement. We
> > >> face a
> > >>> number of challenges and one of them is locking.
> > >>>
> > >>> At the moment information about all locks is kept in memory on
> > per-entry
> > >>> basis (see GridCacheMvccManager). For every locked key we maintain
> > >> current
> > >>> lock owner (XID) and the list of would-be-owner transactions. When
> > >>> transaction is about to lock an entry two scenarios are possible:
> > >>> 1) If entry is not locked we obtain the lock immediately
> > >>> 2) if entry is locked we add current transaction to the wait list and
> > >> jumps
> > >>> to the next entry to be locked. Once the first entry is released by
> > >>> conflicting transaction, current transaction becomes an owner of the
> > >> first
> > >>> entry and tries to promote itself for subsequent entries.
> > >>>
> > >>> Once all required locks are obtained, response is sent to the caller.
> > >>>
> > >>> This approach doesn't work well for transactional SQL - if we update
> > >>> millions of rows in a single transaction we will simply run out of
> > >> memory.
> > >>> To mitigate the problem other database vendors keep information about
> > >> locks
> > >>> inside the tuples. I propose to apply the similar design as follows:
> > >>>
> > >>> 1) No per-entry lock

Re: Rework locking architecture for MVCC and transactional SQL

2017-12-15 Thread Alexey Goncharuk 
Vladimir,

What about moving the entire locking mechanism to a separate off-heap
memory region which will be volatile wrt restarts, but will still support
off-load to disk. In the current architecture, it means that we will need
to allocate a separate DataRegion with no WAL and no crash recovery - locks
are meaningless after a restart, and we will automatically drop them. I
would be interesting to prototype this because I think we may be on-par
with on-heap lock placement, as we already proved for in-memory caches.

2017-12-14 21:53 GMT+03:00 Denis Magda :

> Vladimir,
>
> No it’s crystal clear, thanks.
>
> If this approach works only for Ignite persistence based deployment, how
> will we handle locking for pure in-memory and caching of 3rd party
> databases scenarios? As I understand the tuples still will be stored in the
> page memory while there won’t be any opportunity to fallback to disk if the
> memory usage increases some threshold.
>
> —
> Denis
>
> > On Dec 13, 2017, at 11:21 PM, Vladimir Ozerov 
> wrote:
> >
> > Denis,
> >
> > Sorry, may be I was not clear enough - "tuple-approach" and "persistent
> > approach" are the same. By "tuple" I mean a row stored inside a data
> block.
> > Currently we store lock information in Java heap and proposal is to move
> it
> > to data blocks. The main driver is memory - if there are a rows to be
> > locked we will either run out of memory, or produce serious memory
> > pressure. For example, currently update of 1M entries will consume ~500Mb
> > of heap. With proposed approach it will consume almost nothing. The
> > drawback is increased number of dirty data pages, but it should not be a
> > problem because in final implementation we will update data rows before
> > prepare phase anyway, so I do not expect any write amplification in usual
> > case.
> >
> > This approach is only applicable for Ignite persistence.
> >
> > On Thu, Dec 14, 2017 at 1:53 AM, Denis Magda  wrote:
> >
> >> Vladimir,
> >>
> >> Thanks for a throughout overview and proposal.
> >>
> >>> Also we could try employing tiered approach
> >>> 1) Try to keep everything in-memory to minimize writes to blocks
> >>> 2) Fallback to persistent lock data if certain threshold is reached.
> >>
> >> What are the benefits of the backed-by-persistence approach in compare
> to
> >> the one based on tuples? Specifically:
> >> - will the persistence approach work for both 3rd party and Ignite
> >> persistence?
> >> - any performance impacts depending on a chosen method?
> >> - what’s faster to implement?
> >>
> >> —
> >> Denis
> >>
> >>> On Dec 13, 2017, at 2:10 AM, Vladimir Ozerov 
> >> wrote:
> >>>
> >>> Igniters,
> >>>
> >>> As you probably we know we work actively on MVCC [1] and transactional
> >> SQL
> >>> [2] features which could be treated as a single huge improvement. We
> >> face a
> >>> number of challenges and one of them is locking.
> >>>
> >>> At the moment information about all locks is kept in memory on
> per-entry
> >>> basis (see GridCacheMvccManager). For every locked key we maintain
> >> current
> >>> lock owner (XID) and the list of would-be-owner transactions. When
> >>> transaction is about to lock an entry two scenarios are possible:
> >>> 1) If entry is not locked we obtain the lock immediately
> >>> 2) if entry is locked we add current transaction to the wait list and
> >> jumps
> >>> to the next entry to be locked. Once the first entry is released by
> >>> conflicting transaction, current transaction becomes an owner of the
> >> first
> >>> entry and tries to promote itself for subsequent entries.
> >>>
> >>> Once all required locks are obtained, response is sent to the caller.
> >>>
> >>> This approach doesn't work well for transactional SQL - if we update
> >>> millions of rows in a single transaction we will simply run out of
> >> memory.
> >>> To mitigate the problem other database vendors keep information about
> >> locks
> >>> inside the tuples. I propose to apply the similar design as follows:
> >>>
> >>> 1) No per-entry lock information is stored in memory anymore.
> >>> 2) The list of active transactions are maintained in memory still
> >>> 3) When TX locks an entry, it sets special marker to the tuple [3]
> >>> 4) When TX meets already locked entry, it enlists itself to wait queue
> of
> >>> conflicting transaction and suspends
> >>> 5) When first transaction releases conflicting lock, it notifies and
> >> wakes
> >>> up suspended transactions, so they resume locking
> >>> 6) Entry lock data is cleared on transaction commit
> >>> 7) Entry lock data is not cleared on rollback or node restart; Instead,
> >> we
> >>> will could use active transactions list to identify invalid locks and
> >>> overwrite them as needed.
> >>>
> >>> Also we could try employing tiered approach
> >>> 1) Try to keep everything in-memory to minimize writes to blocks
> >>> 2) Fallback to persistent lock data if certain

Re: MVCC version format

2017-12-15 Thread Vladimir Ozerov 
Hi Alex,

Good points.

1) So far we've seen clusters up to ~ 2K nodes in size, and typical size is
much smaller. So this doesn't appear to be a a big problem in common case.
But what if at some point we see, say, 10_000 nodes deployment? in this
case proposed schema may virtually stop working at all :-) On the other
hand, frequent wraparound typically mean that not so much changes happened
in between, so we should be able to cope with freeze fast enough. Also big
deployments would have to take coordinator location in count anyway,
because on coordinator change we cannot start assigning versions
immediately because we need to collect full list of active transactions
from other nodes first, which is painful on large topologies. I.e our
current approach would require careful administration irrespectively of
whether we take proposed approach or not.

2) I think we could simply mark this partition as "definitely frozen"
because when offline partition comes back any active transaction would
definitely see it's data. Would that work?

3) This depends on final implementation heavily. We should do that
constantly in background and track progress on per-partition and possibly
per-block (visibility map?) levels. Provided large time frames, we should
be able to cope with.

On Fri, Dec 15, 2017 at 2:37 PM, Alexey Goncharuk <
alexey.goncha...@gmail.com> wrote:

> Vladimir,
>
> Although at the first glance this seems to be ok for a local solution, I
> have a few concerns wrt distributed version behavior:
>
> 1) I cannot say that epoch switch in this approach will be an extremely
> rare event. Let's say a user regularly deploys a new version of application
> code to a large Ignite cluster and restarts Ignite nodes for such an
> upgrade. If version coordinator is not dedicated, then on average version
> coordinator switch will occur N/2 times (N is the size of the cluster), in
> the worst case the switch will happen N times. At this rate, epoch switch
> may occur maybe once a month depending on a cluster size? The user will
> need to take care of the node order restart.
> 2) What happens if a node went offline, then epoch switched twice, then
> node gets back to cluster again (a classic ABA)? The only solution I see
> here is that we track epoch switches and drop nodes that were offline
> during the epoch switch forever. This may cause problems if a user has
> partition loss policy = READ_WRITE_SAFE. Let's say a partition was offline,
> but updates to other partitions were allowed. If an epoch switch happens at
> this moment, the partition is lost forever.
> 3) How fast will we be able to flip the frozen bit for the whole dataset if
> we are looking at terabytes per node? Given (1), we may be in trouble here.
>
> 2017-12-15 12:57 GMT+03:00 Vladimir Ozerov :
>
> > Igniters,
> >
> > As you probably know, we are working on MVCC feature [1]. When MVCC is
> > enabled every cache operation/tx require one unique version for read and
> > possibly one version for write. Versions are assigned by a single node
> > called *coordinator*.
> >
> > Requirements for version:
> > 1) Must be increasing and comparable, so that we understand which
> operation
> > occurred earlier.
> > 2) Support coordinator failtover
> > 3) As compact as possible
> >
> > Currently we implemented naive approach where version is represented as
> two
> > long values (16 bytes total):
> > 1) Coordinator version (a kind of mix of a timestamp and topology
> version)
> > 2) Local coordinator counter.
> >
> > Other vendors typically assign 4 byte (long) or 8 byte (long) versions,
> > which is definitely better as they are faster to compare and require less
> > space, what could be especially important for small values, such as
> > indexes.
> >
> > I thought a bit, and came to conclusion that we could fit into 64 bit
> > version (8 bytes) as follows:
> > [0..47] - local coordinator counter; this gives capacity 10^14, which
> > should be enough for any sensible workload
> > [48..60] - coordinator version, which is incremented on every coordinator
> > change (relatively rare) or local counter wraparound (very rare); this
> > gives 8192 versions
> > [61] - coordinator epoch
> > [62] - delete marker, set on entry removal, cleared on resurrection (e.g.
> > TX rollback, or index change A->B->A)
> > [63] - freeze marker
> >
> > Mechanics:
> > 1) Every coordinator change increments coordinator version and share it
> via
> > discovery
> > 2) If persistence is enabled, coordinator version is persisted and
> possibly
> > WAL logged
> > 3) Epoch is switched between 0 and 1 on every coordinator version
> > wraparound (extremely rare event)
> > 4) Epoch is cluster-wide properly, every node knows current epoch
> > 5) As epoch switches, nodes start marking entries from previous epoch as
> > "frozen". This is done in a background in a very light-weight mode, as
> > there is no rush
> > 6) Epoch 0 could not be switched to 1 if there are non-frozen entries
> from

Re: IgniteProjectionStartStopRestartSelfTest is failing for about a week on TC

2017-12-15 Thread Petr Ivanov 
According to Failure Conditions [1] — it is designed to fail after 30 min.
So the question is - should we consider 30 min run time to be project’s error 
(and fix the code) or TeamCity error (and fix the build configuration by, for 
example, increasing build failure timeout)?


[1] 
https://ci.ignite.apache.org/admin/editBuildFailureConditions.html?id=buildType:Ignite20Tests_IgniteStartNodes
 



> On 15 Dec 2017, at 14:50, Dmitry Pavlov  wrote:
> 
> Hi Igniters,
> 
> Who can became Jedi of TeamCity and substitute me in this research?
> 
> Sincerely,
> Dmitry Pavlov
> 
> пт, 15 дек. 2017 г. в 12:38, Anton Vinogradov :
> 
>> Dmitry,
>> 
>> Could you please investigate?
>> 
>> On Fri, Dec 15, 2017 at 9:16 AM, Andrey Kuznetsov 
>> wrote:
>> 
>>> Hi Anton,
>>> 
>>> I doubt it was merge conflict. Couple of days ago 'Ignite Start Nodes'
>>> suite succeeded 1 time after unrelated changes, and now it is failing
>>> again.
>>> 
>>> 2017-12-12 16:25 GMT+03:00 Anton Vinogradov :
>>> 
 Hi,
 
 Seems we have merge conflict.
 
 https://ci.ignite.apache.org/viewType.html?buildTypeId=Ignite20Tests_
 IgniteStartNodes=buildTypeHistoryList_
 Ignite20Tests=%3Cdefault%3E
 
 
 --
>>> Best regards,
>>>  Andrey Kuznetsov.
>>> 
>>

[GitHub] ignite pull request #3237: ignite-7195: limit toString of arrays and additio...

2017-12-15 Thread abeliak 
GitHub user abeliak opened a pull request:

https://github.com/apache/ignite/pull/3237

ignite-7195: limit toString of arrays and additional values; add "and…

ignite-7195: limit toString of arrays and additional values; add "and N 
more".

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7195

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3237.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3237


commit 90ab7d4084fde9faf35c9dd385aa2a579c81bf1d
Author: Alexander Belyak 
Date:   2017-12-15T12:16:16Z

ignite-7195: limit toString of arrays and additional values; add "and N 
more".




---

Re: Switching Java language level to 8

2017-12-15 Thread vveider 
Very good. I came to the same thoughts.

Yet, there is a situation that some classes in *.java.* and .*java8.*
packages that share the name and policy of merging them is not quite clear.
Should *.java.* classes be replace by .*java8.*?



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: Switching Java language level to 8

2017-12-15 Thread Yury Babak 
Hi,

currently we use maven profile "ml" for all ML sources. So during this
switching we could remove this profile and add ML module to the normal build
chain.

NB: we use this hack in example and yardstick modules. Also in examples we
have the separate source folder(and profile) for non-ml java 8 examples.
>From my point of view we dont need the separete source folders any more, so
all sources must be moved into "normal" folder(/src/main/java).

Regards,
Yury



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: IgniteProjectionStartStopRestartSelfTest is failing for about a week on TC

2017-12-15 Thread Dmitry Pavlov 
Hi Igniters,

Who can became Jedi of TeamCity and substitute me in this research?

Sincerely,
Dmitry Pavlov

пт, 15 дек. 2017 г. в 12:38, Anton Vinogradov :

> Dmitry,
>
> Could you please investigate?
>
> On Fri, Dec 15, 2017 at 9:16 AM, Andrey Kuznetsov 
> wrote:
>
> > Hi Anton,
> >
> > I doubt it was merge conflict. Couple of days ago 'Ignite Start Nodes'
> > suite succeeded 1 time after unrelated changes, and now it is failing
> > again.
> >
> > 2017-12-12 16:25 GMT+03:00 Anton Vinogradov :
> >
> > > Hi,
> > >
> > > Seems we have merge conflict.
> > >
> > > https://ci.ignite.apache.org/viewType.html?buildTypeId=Ignite20Tests_
> > > IgniteStartNodes=buildTypeHistoryList_
> > > Ignite20Tests=%3Cdefault%3E
> > >
> > >
> > > --
> > Best regards,
> >   Andrey Kuznetsov.
> >
>

[jira] [Created] (IGNITE-7212) Load stoped after server node kill

2017-12-15 Thread Ilya Suntsov (JIRA) 
Ilya Suntsov created IGNITE-7212:


 Summary: Load stoped after server node kill
 Key: IGNITE-7212
 URL: https://issues.apache.org/jira/browse/IGNITE-7212
 Project: Ignite
  Issue Type: Bug
  Components: general
Affects Versions: 2.4
Reporter: Ilya Suntsov
Priority: Critical


Scenario:
* Start 4 servers
* Start load tasks on 5 clients
* Kill 1 server
* Waiting for rebalancing
* Kill 1 server
Result:
After the kill of second servers node load stoped.
In servers logs I see messages like this:
{noformat}
[2017-12-15 11:23:50][DEBUG][grid-nio-worker-tcp-comm-0-#41] Remote client 
closed connection: GridSelectorNioSessionImpl [worker=DirectNioClientWorker 
[super=AbstractNioClientWorker [idx=0, bytesRcvd=130952565, 
bytesSent=131203245, bytesRcvd0=3069200, bytesSent0=3068083, select=true, 
super=GridWorker [name=grid-nio-worker-tcp-comm-0, igniteInstanceName=null, 
finished=false, hashCode=1748650517, interrupted=false, 
runner=grid-nio-worker-tcp-comm-0-#41]]], 
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], 
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], 
inRecovery=GridNioRecoveryDescriptor [acked=1024, resendCnt=0, rcvCnt=1026, 
sentCnt=1029, reserved=true, lastAck=1024, nodeLeft=false, 
node=TcpDiscoveryNode [id=b7cfaa4e-b3b7-4485-a421-c731d9ed869d, 
addrs=[127.0.0.1, 172.31.20.3], 
sockAddrs=[ip-172-31-20-3.us-east-2.compute.internal/172.31.20.3:47500, 
/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, 
lastExchangeTime=1513335739604, loc=false, ver=2.4.0#20171214-sha1:da782958, 
isClient=false], connected=false, connectCnt=1, queueLimit=4096, reserveCnt=1, 
pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=1024, 
resendCnt=0, rcvCnt=1026, sentCnt=1029, reserved=true, lastAck=1024, 
nodeLeft=false, node=TcpDiscoveryNode [id=b7cfaa4e-b3b7-4485-a421-c731d9ed869d, 
addrs=[127.0.0.1, 172.31.20.3], 
sockAddrs=[ip-172-31-20-3.us-east-2.compute.internal/172.31.20.3:47500, 
/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, 
lastExchangeTime=1513335739604, loc=false, ver=2.4.0#20171214-sha1:da782958, 
isClient=false], connected=false, connectCnt=1, queueLimit=4096, reserveCnt=1, 
pairedConnections=false], super=GridNioSessionImpl 
[locAddr=/172.31.23.220:41732, 
rmtAddr=ip-172-31-20-3.us-east-2.compute.internal/172.31.20.3:47100, 
createTime=1513335774008, closeTime=0, bytesSent=131203245, 
bytesRcvd=130952565, bytesSent0=3068083, bytesRcvd0=3069200, 
sndSchedTime=1513335774008, lastSndTime=1513337029027, 
lastRcvTime=1513337029027, readsPaused=false, 
filterChain=FilterChain[filters=[GridNioCodecFilter 
[parser=org.apache.ignite.internal.util.nio.GridDirectParser@11ae7d3b, 
directMode=true], GridConnectionBytesVerifyFilter], accepted=false]]
[2017-12-15 11:23:50][WARN ][tcp-disco-msg-worker-#2] Failed to send message to 
next node [msg=TcpDiscoveryConnectionCheckMessage 
[super=TcpDiscoveryAbstractMessage [sndNodeId=null, 
id=6c7f6d95061-c3cf9fe4-ab13-4d95-ace3-84a54cd73e08, verifierNodeId=null, 
topVer=0, pendingIdx=0, failedNodes=null, isClient=false]], 
next=TcpDiscoveryNode [id=b7cfaa4e-b3b7-4485-a421-c731d9ed869d, 
addrs=[127.0.0.1, 172.31.20.3], 
sockAddrs=[ip-172-31-20-3.us-east-2.compute.internal/172.31.20.3:47500, 
/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, 
lastExchangeTime=1513335739604, loc=false, ver=2.4.0#20171214-sha1:da782958, 
isClient=false], errMsg=Failed to send message to next node 
[msg=TcpDiscoveryConnectionCheckMessage [super=TcpDiscoveryAbstractMessage 
[sndNodeId=null, id=6c7f6d95061-c3cf9fe4-ab13-4d95-ace3-84a54cd73e08, 
verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, 
isClient=false]], next=ClusterNode [id=b7cfaa4e-b3b7-4485-a421-c731d9ed869d, 
order=1, addr=[127.0.0.1, 172.31.20.3], daemon=false]]]
[2017-12-15 11:23:50][DEBUG][grid-nio-worker-tcp-comm-0-#41] Session was closed 
but there are unacknowledged messages, will try to reconnect 
[rmtNode=b7cfaa4e-b3b7-4485-a421-c731d9ed869d]
[2017-12-15 11:23:50][DEBUG][tcp-comm-worker-#1] Recovery reconnect 
[rmtNode=b7cfaa4e-b3b7-4485-a421-c731d9ed869d]
[2017-12-15 11:23:50][DEBUG][tcp-comm-worker-#1] Creating NIO client to node: 
TcpDiscoveryNode [id=b7cfaa4e-b3b7-4485-a421-c731d9ed869d, addrs=[127.0.0.1, 
172.31.20.3], 
sockAddrs=[ip-172-31-20-3.us-east-2.compute.internal/172.31.20.3:47500, 
/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, 
lastExchangeTime=1513335739604, loc=false, ver=2.4.0#20171214-sha1:da782958, 
isClient=false]
[2017-12-15 11:23:50][DEBUG][tcp-comm-worker-#1] Addresses resolved from 
attributes [rmtNode=b7cfaa4e-b3b7-4485-a421-c731d9ed869d, 
addrs=[ip-172-31-20-3.us-east-2.compute.internal/172.31.20.3:47100, 
/127.0.0.1:47100], isRmtAddrsExist=true]
[2017-12-15 11:23:50][DEBUG][tcp-comm-worker-#1] Client creation failed

Re: MVCC version format

2017-12-15 Thread Alexey Goncharuk 
Vladimir,

Although at the first glance this seems to be ok for a local solution, I
have a few concerns wrt distributed version behavior:

1) I cannot say that epoch switch in this approach will be an extremely
rare event. Let's say a user regularly deploys a new version of application
code to a large Ignite cluster and restarts Ignite nodes for such an
upgrade. If version coordinator is not dedicated, then on average version
coordinator switch will occur N/2 times (N is the size of the cluster), in
the worst case the switch will happen N times. At this rate, epoch switch
may occur maybe once a month depending on a cluster size? The user will
need to take care of the node order restart.
2) What happens if a node went offline, then epoch switched twice, then
node gets back to cluster again (a classic ABA)? The only solution I see
here is that we track epoch switches and drop nodes that were offline
during the epoch switch forever. This may cause problems if a user has
partition loss policy = READ_WRITE_SAFE. Let's say a partition was offline,
but updates to other partitions were allowed. If an epoch switch happens at
this moment, the partition is lost forever.
3) How fast will we be able to flip the frozen bit for the whole dataset if
we are looking at terabytes per node? Given (1), we may be in trouble here.

2017-12-15 12:57 GMT+03:00 Vladimir Ozerov :

> Igniters,
>
> As you probably know, we are working on MVCC feature [1]. When MVCC is
> enabled every cache operation/tx require one unique version for read and
> possibly one version for write. Versions are assigned by a single node
> called *coordinator*.
>
> Requirements for version:
> 1) Must be increasing and comparable, so that we understand which operation
> occurred earlier.
> 2) Support coordinator failtover
> 3) As compact as possible
>
> Currently we implemented naive approach where version is represented as two
> long values (16 bytes total):
> 1) Coordinator version (a kind of mix of a timestamp and topology version)
> 2) Local coordinator counter.
>
> Other vendors typically assign 4 byte (long) or 8 byte (long) versions,
> which is definitely better as they are faster to compare and require less
> space, what could be especially important for small values, such as
> indexes.
>
> I thought a bit, and came to conclusion that we could fit into 64 bit
> version (8 bytes) as follows:
> [0..47] - local coordinator counter; this gives capacity 10^14, which
> should be enough for any sensible workload
> [48..60] - coordinator version, which is incremented on every coordinator
> change (relatively rare) or local counter wraparound (very rare); this
> gives 8192 versions
> [61] - coordinator epoch
> [62] - delete marker, set on entry removal, cleared on resurrection (e.g.
> TX rollback, or index change A->B->A)
> [63] - freeze marker
>
> Mechanics:
> 1) Every coordinator change increments coordinator version and share it via
> discovery
> 2) If persistence is enabled, coordinator version is persisted and possibly
> WAL logged
> 3) Epoch is switched between 0 and 1 on every coordinator version
> wraparound (extremely rare event)
> 4) Epoch is cluster-wide properly, every node knows current epoch
> 5) As epoch switches, nodes start marking entries from previous epoch as
> "frozen". This is done in a background in a very light-weight mode, as
> there is no rush
> 6) Epoch 0 could not be switched to 1 if there are non-frozen entries from
> previous epoch 1, and vice versa. This way version comparison is always
> possible.
>
> IMO this should do the trick so that we have 8 byte version with virtually
> zero overhead in runtime. But a little distributed magic would be needed.
>
> Thoughts?
>
> P.S.: Idea is borrowed from PostgreSQL counter wraparound handling.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-3478
>

Re: IgniteProjectionStartStopRestartSelfTest is failing for about a week on TC

2017-12-15 Thread Anton Vinogradov 
Dmitry,

Could you please investigate?

On Fri, Dec 15, 2017 at 9:16 AM, Andrey Kuznetsov  wrote:

> Hi Anton,
>
> I doubt it was merge conflict. Couple of days ago 'Ignite Start Nodes'
> suite succeeded 1 time after unrelated changes, and now it is failing
> again.
>
> 2017-12-12 16:25 GMT+03:00 Anton Vinogradov :
>
> > Hi,
> >
> > Seems we have merge conflict.
> >
> > https://ci.ignite.apache.org/viewType.html?buildTypeId=Ignite20Tests_
> > IgniteStartNodes=buildTypeHistoryList_
> > Ignite20Tests=%3Cdefault%3E
> >
> >
> > --
> Best regards,
>   Andrey Kuznetsov.
>

[jira] [Created] (IGNITE-7211) Web console: Wrong cluster active state on cluster switch.

2017-12-15 Thread Vasiliy Sisko (JIRA) 
Vasiliy Sisko created IGNITE-7211:
-

 Summary: Web console: Wrong cluster active state on cluster switch.
 Key: IGNITE-7211
 URL: https://issues.apache.org/jira/browse/IGNITE-7211
 Project: Ignite
  Issue Type: Bug
Reporter: Vasiliy Sisko
Assignee: Alexey Kuznetsov


# Connect two inactive clusters to Web console.
# Switch cluster in time of some cluster activation.
Active state showed for both clusters as active and do not changed on next 
cluster switch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-7210) Web console: Fix show time of "Connected clusters: N" label

2017-12-15 Thread Vasiliy Sisko (JIRA) 
Vasiliy Sisko created IGNITE-7210:
-

 Summary: Web console: Fix show time of "Connected clusters: N" 
label
 Key: IGNITE-7210
 URL: https://issues.apache.org/jira/browse/IGNITE-7210
 Project: Ignite
  Issue Type: Bug
  Components: website
Reporter: Vasiliy Sisko
Assignee: Alexey Kuznetsov


Now that label showed quickly when login screen still shown or page is fully 
empty.
Should appear together with page content.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)