Re: GridGain Donates Persistent Distributed Store To ASF (Apache Ignite)

[Dmitriy Setrakyan]

Great news and I am glad that GridGain was finally able to open source such
an essential feature to Ignite. Given that I was deeply involved in the
development of this feature for the past year, I would add that one of the
main advantages here is that Ignite becomes fully operational from disk
(SSD or Flash) without any need to preload the data in memory.

With full SQL support available in Ignite, this feature will allow Ignite
serve as a distributed transactional database, both in memory and on disk,
while continuing to support all the existing use cases, including the
in-memory data grid.

Cos, can you point us to any legal paperwork that needs to be signed in
order to complete the donation?

D.

On Mon, Apr 17, 2017 at 7:37 PM, Denis Magda  wrote:

> Igniters,
>
> GridGain, as one of the most active Apache Ignite contributors, has been
> developing a unique distributed persistent store specifically for Apache
> Ignite for more than a year in-house. It’s a fully ACID and ANSI-99 SQL
> compliant fault-tolerant solution.
>
> The store transparently integrates with Apache Ignite as an optional disk
> layer (in addition to the existing RAM layer) via the new page memory
> architecture that to be released in Apache Ignite 2.0. This allows storing
> supersets of data on disk while having a subset in memory not worrying
> about that you forgot to preload (warmup) your caches!
>
> Assuming that the storage goes to ASF as a part of Apache Ignite 2.1
> release the following will be supported by Ignite out-of-the-box:
>
> * SQL queries execution over the data that is both in RAM and on disk: no
> need to preload the whole data set in memory.
>
> * Cluster instantaneous restarts: once your cluster ring is recovered
> after a restart your applications are fully operational even if they highly
> utilize SQL queries.
>
> As for the use cases, it means that Apache Ignite will be possible to use
> as a distributed SQL database with memory-first concept.
>
> And we decided at GridGain that this tremendous feature should be open
> source from the very beginning.
>
> Guys, could you advise how I can start official donation process?
>
> —
> Denis
>
>
>
>
>
>

GridGain Donates Persistent Distributed Store To ASF (Apache Ignite)

[Denis Magda]

Igniters,

GridGain, as one of the most active Apache Ignite contributors, has been 
developing a unique distributed persistent store specifically for Apache Ignite 
for more than a year in-house. It’s a fully ACID and ANSI-99 SQL compliant 
fault-tolerant solution. 

The store transparently integrates with Apache Ignite as an optional disk layer 
(in addition to the existing RAM layer) via the new page memory architecture 
that to be released in Apache Ignite 2.0. This allows storing supersets of data 
on disk while having a subset in memory not worrying about that you forgot to 
preload (warmup) your caches!

Assuming that the storage goes to ASF as a part of Apache Ignite 2.1 release 
the following will be supported by Ignite out-of-the-box:

* SQL queries execution over the data that is both in RAM and on disk: no need 
to preload the whole data set in memory.

* Cluster instantaneous restarts: once your cluster ring is recovered after a 
restart your applications are fully operational even if they highly utilize SQL 
queries.

As for the use cases, it means that Apache Ignite will be possible to use as a 
distributed SQL database with memory-first concept.

And we decided at GridGain that this tremendous feature should be open source 
from the very beginning.

Guys, could you advise how I can start official donation process?

—
Denis


 

 
 

Re: Null as a name for default memory policy?

[Alexey Kuznetsov]

+1 to "default".


On Tue, Apr 18, 2017 at 8:01 AM, Denis Magda  wrote:

> Sounds good to me. Other thoughts?
>
> —
> Denis
>
> > On Apr 17, 2017, at 5:54 PM, Dmitriy Setrakyan 
> wrote:
> >
> > How about we name it "default"?
> >
> > On Mon, Apr 17, 2017 at 5:53 PM, Denis Magda  wrote:
> >
> >> Folks,
> >>
> >> Are we sure we want to use ‘null’ as a name for the default policy if
> it’s
> >> not configured explicitly? Presently, MemoryConfiguration.
> getDefaultMemoryPolicyName
> >> suggests ‘null’ as a default one.
> >>
> >> I’m bringing this to your attention only because in AI 2.0 we’re trying
> to
> >> get rid of ‘null’ as a default for the cache name but, strangely, we
> keep
> >> following this criticized approache in the new API.
> >>
> >> —
> >> Denis
> >>
> >>
>
>


-- 
Alexey Kuznetsov
GridGain Systems
www.gridgain.com

Re: Page Memory behavior with default memory policy

[Dmitriy Setrakyan]

Denis,

If what you are suggesting is true, then we can always allocate about 80%
of available memory by default. By the way, it must also work on Windows,
so we should definitely test it.

Alexey G, can you comment?

D.

On Mon, Apr 17, 2017 at 6:17 PM, Denis Magda  wrote:

> Dmitriy,
>
> > Denis, it sounds like with this approach, in case of the over-allocation,
> > the system will just get slower and slower and users will end up blaming
> > Ignite for it. Am I understanding your suggestion correctly?
>
>
> This will not happen (at least in Unix) unless all the nodes really used
> all the allocated memory by putting data there or touching all the memory
> range somehow else.
>
> > How was this handled in Ignite 1.9?
>
>
> If you are talking about the legacy off-heap impl then we requested small
> chunks of data from an operating system rather than a continuous memory
> region as in the page memory. But I would think of the page memory as of
> Java heap which also can request 8 GB continuous memory region on a 8 GB
> machine following heap settings of an app but an operating system will not
> return the whole range immediately unless Java app occupies the whole heap
> or use special parameters.
>
> At all, I think it’s safe to use approach suggested by me unless I miss
> something.
>
> —
> Denis
>
> > On Apr 17, 2017, at 6:05 PM, Dmitriy Setrakyan 
> wrote:
> >
> > On Mon, Apr 17, 2017 at 6:00 PM, Denis Magda  wrote:
> >
> >> Dmitriy,
> >>
> >> All the nodes will request its own continuous memory region that takes
> >> 70-80% of all RAM from an underlying operation system. However, the
> >> operating system will not outfit the nodes with physical pages mapped to
> >> RAM immediately allowing every node's process to start successfully. The
> >> nodes will communicate to RAM via a virtual memory which in its turn
> will
> >> give an access to physical pages whenever is needed applying low level
> >> eviction and swapping techniques.
> >>
> >
> > Denis, it sounds like with this approach, in case of the over-allocation,
> > the system will just get slower and slower and users will end up blaming
> > Ignite for it. Am I understanding your suggestion correctly?
> >
> > How was this handled in Ignite 1.9?
> >
> > D.
>
>

Re: Page Memory behavior with default memory policy

[Denis Magda]

Dmitriy,

> Denis, it sounds like with this approach, in case of the over-allocation,
> the system will just get slower and slower and users will end up blaming
> Ignite for it. Am I understanding your suggestion correctly?


This will not happen (at least in Unix) unless all the nodes really used all 
the allocated memory by putting data there or touching all the memory range 
somehow else.

> How was this handled in Ignite 1.9?


If you are talking about the legacy off-heap impl then we requested small 
chunks of data from an operating system rather than a continuous memory region 
as in the page memory. But I would think of the page memory as of Java heap 
which also can request 8 GB continuous memory region on a 8 GB machine 
following heap settings of an app but an operating system will not return the 
whole range immediately unless Java app occupies the whole heap or use special 
parameters.

At all, I think it’s safe to use approach suggested by me unless I miss 
something.

—
Denis

> On Apr 17, 2017, at 6:05 PM, Dmitriy Setrakyan  wrote:
> 
> On Mon, Apr 17, 2017 at 6:00 PM, Denis Magda  wrote:
> 
>> Dmitriy,
>> 
>> All the nodes will request its own continuous memory region that takes
>> 70-80% of all RAM from an underlying operation system. However, the
>> operating system will not outfit the nodes with physical pages mapped to
>> RAM immediately allowing every node's process to start successfully. The
>> nodes will communicate to RAM via a virtual memory which in its turn will
>> give an access to physical pages whenever is needed applying low level
>> eviction and swapping techniques.
>> 
> 
> Denis, it sounds like with this approach, in case of the over-allocation,
> the system will just get slower and slower and users will end up blaming
> Ignite for it. Am I understanding your suggestion correctly?
> 
> How was this handled in Ignite 1.9?
> 
> D.

Partition loss policy

[Denis Magda]

Alex G., Dmitriy G.,

Lucky me, preparing the doc for the page memory I’ve spotted a new feature 
added to 2.0 which is partition loss policies:
https://github.com/apache/ignite/blob/9b61a76bd2f5f18e50081b2553e58bf36375b8ab/modules/core/src/main/java/org/apache/ignite/cache/PartitionLossPolicy.java

However, I don’t see any progress in a corresponding ticket:
https://issues.apache.org/jira/browse/IGNITE-1605

What was done specifically and what have to be documented in readme?

—
Denis

Re: Page Memory behavior with default memory policy

[Dmitriy Setrakyan]

On Mon, Apr 17, 2017 at 6:00 PM, Denis Magda  wrote:

> Dmitriy,
>
> All the nodes will request its own continuous memory region that takes
> 70-80% of all RAM from an underlying operation system. However, the
> operating system will not outfit the nodes with physical pages mapped to
> RAM immediately allowing every node's process to start successfully. The
> nodes will communicate to RAM via a virtual memory which in its turn will
> give an access to physical pages whenever is needed applying low level
> eviction and swapping techniques.
>

Denis, it sounds like with this approach, in case of the over-allocation,
the system will just get slower and slower and users will end up blaming
Ignite for it. Am I understanding your suggestion correctly?

How was this handled in Ignite 1.9?

D.

Re: Null as a name for default memory policy?

[Denis Magda]

Sounds good to me. Other thoughts?

—
Denis

> On Apr 17, 2017, at 5:54 PM, Dmitriy Setrakyan  wrote:
> 
> How about we name it "default"?
> 
> On Mon, Apr 17, 2017 at 5:53 PM, Denis Magda  wrote:
> 
>> Folks,
>> 
>> Are we sure we want to use ‘null’ as a name for the default policy if it’s
>> not configured explicitly? Presently, 
>> MemoryConfiguration.getDefaultMemoryPolicyName
>> suggests ‘null’ as a default one.
>> 
>> I’m bringing this to your attention only because in AI 2.0 we’re trying to
>> get rid of ‘null’ as a default for the cache name but, strangely, we keep
>> following this criticized approache in the new API.
>> 
>> —
>> Denis
>> 
>> 

Re: Page Memory behavior with default memory policy

[Denis Magda]

Sorry, misprint 

Alex G., please confirm that we’re *applying* -> *relying* on this low level 
guarantees in our page memory impl.

> On Apr 17, 2017, at 6:00 PM, Denis Magda  wrote:
> 
> Dmitriy,
> 
> All the nodes will request its own continuous memory region that takes 70-80% 
> of all RAM from an underlying operation system. However, the operating system 
> will not outfit the nodes with physical pages mapped to RAM immediately 
> allowing every node's process to start successfully. The nodes will 
> communicate to RAM via a virtual memory which in its turn will give an access 
> to physical pages whenever is needed applying low level eviction and swapping 
> techniques.
> 
> Alex G., please confirm that we’re applying on this low level guarantees in 
> our page memory impl.
> 
> —
> Denis
> 
>> On Apr 17, 2017, at 5:50 PM, Dmitriy Setrakyan  wrote:
>> 
>> On Mon, Apr 17, 2017 at 5:46 PM, Denis Magda  wrote:
>> 
>>> Guys,
>>> 
>>> If a memory region is not expandable can we calculate the size of the
>>> default one automatically rather than setting it to predefined value like 1
>>> GB, 512 MB, etc. ?
>>> 
>>> For instance, when a node is being started it finds out how much RAM is
>>> available and requests 70-80% of it for the default memory region usage.
>>> This should help us avoid this usability issue caused by the fact that we
>>> hard code the size.
>>> 
>> 
>> Denis, what happens if user plans to start multiple nodes on the same box?
> 

Re: Page Memory behavior with default memory policy

[Denis Magda]

Dmitriy,

All the nodes will request its own continuous memory region that takes 70-80% 
of all RAM from an underlying operation system. However, the operating system 
will not outfit the nodes with physical pages mapped to RAM immediately 
allowing every node's process to start successfully. The nodes will communicate 
to RAM via a virtual memory which in its turn will give an access to physical 
pages whenever is needed applying low level eviction and swapping techniques.

Alex G., please confirm that we’re applying on this low level guarantees in our 
page memory impl.

—
Denis

> On Apr 17, 2017, at 5:50 PM, Dmitriy Setrakyan  wrote:
> 
> On Mon, Apr 17, 2017 at 5:46 PM, Denis Magda  wrote:
> 
>> Guys,
>> 
>> If a memory region is not expandable can we calculate the size of the
>> default one automatically rather than setting it to predefined value like 1
>> GB, 512 MB, etc. ?
>> 
>> For instance, when a node is being started it finds out how much RAM is
>> available and requests 70-80% of it for the default memory region usage.
>> This should help us avoid this usability issue caused by the fact that we
>> hard code the size.
>> 
> 
> Denis, what happens if user plans to start multiple nodes on the same box?

Re: Null as a name for default memory policy?

[Dmitriy Setrakyan]

How about we name it "default"?

On Mon, Apr 17, 2017 at 5:53 PM, Denis Magda  wrote:

> Folks,
>
> Are we sure we want to use ‘null’ as a name for the default policy if it’s
> not configured explicitly? Presently, 
> MemoryConfiguration.getDefaultMemoryPolicyName
> suggests ‘null’ as a default one.
>
> I’m bringing this to your attention only because in AI 2.0 we’re trying to
> get rid of ‘null’ as a default for the cache name but, strangely, we keep
> following this criticized approache in the new API.
>
> —
> Denis
>
>

Null as a name for default memory policy?

[Denis Magda]

Folks,

Are we sure we want to use ‘null’ as a name for the default policy if it’s not 
configured explicitly? Presently, 
MemoryConfiguration.getDefaultMemoryPolicyName suggests ‘null’ as a default one.

I’m bringing this to your attention only because in AI 2.0 we’re trying to get 
rid of ‘null’ as a default for the cache name but, strangely, we keep following 
this criticized approache in the new API.

—
Denis

Re: Page Memory behavior with default memory policy

[Dmitriy Setrakyan]

On Mon, Apr 17, 2017 at 5:46 PM, Denis Magda  wrote:

> Guys,
>
> If a memory region is not expandable can we calculate the size of the
> default one automatically rather than setting it to predefined value like 1
> GB, 512 MB, etc. ?
>
> For instance, when a node is being started it finds out how much RAM is
> available and requests 70-80% of it for the default memory region usage.
> This should help us avoid this usability issue caused by the fact that we
> hard code the size.
>

Denis, what happens if user plans to start multiple nodes on the same box?

Re: Memory policies examples fails

[Denis Magda]

Sergey,

I tweaked the example following your suggestion. Now it works, thanks.

In addition, a javadoc of MemoryConfiguraiton, MemoryPolicyConfiguration and 
DataPageEvictionMode has been significantly improved. There are some of the 
parameters like page memory’s concurrency mode and policy’s "empty page pool 
size" that are not obvious. Marked all the blur places in those configurations 
with TODOs and created a ticket [1]. * Alex G *, as an architect of the page 
memory, please look at [1] and try to close it tomorrow so that I can complete 
the readme part.

Finally, as for the out of memory exception I fully support Dmitriy’s opinion - 
let’s be as more concrete as possible. "Failed to allocate new page within 
 MemoryPolicy” sounds to generic and doesn’t help me anyhow. 
Let’s add to the message what was a root cause (policy size, name, etc.) so 
that the user can fix it immediately. Created a ticket [2].

[1] https://issues.apache.org/jira/browse/IGNITE-5007 

[2] https://issues.apache.org/jira/browse/IGNITE-5008

—
Denis

> On Apr 17, 2017, at 2:18 PM, Dmitriy Setrakyan  wrote:
> 
> On Mon, Apr 17, 2017 at 9:06 AM, Sergey Chugunov 
> wrote:
> 
>> Dmitriy,
>> 
>> I think this issue highlights another lack in our documentation that
>> emerged with introduction of PageMemory.
>> 
>> In current documentation we have some recommendations about memory capacity
>> planning.
>> As PageMemory storage data structures introduce their own overhead; this
>> section of documentation must be updated.
>> 
>> As per "null" in OutOfMemoryException message, I've already addressed it so
>> now when PageMemory fails to allocate new page, it throws an
>> IgniteOutOfMemoryException with message like "Failed to allocate new page
>> within  MemoryPolicy".
>> 
> 
> Sergey, the error message should also mention which configuration property
> needs to be changed to fix the issue.

[jira] [Created] (IGNITE-5008) Page Memory Throws Meaningless OOM exception if there is an issue in config

[Denis Magda (JIRA)]

Denis Magda created IGNITE-5008:
---

 Summary: Page Memory Throws Meaningless OOM exception if there is 
an issue in config
 Key: IGNITE-5008
 URL: https://issues.apache.org/jira/browse/IGNITE-5008
 Project: Ignite
  Issue Type: Bug
Reporter: Denis Magda
Assignee: Sergey Chugunov
Priority: Blocker


The ticket was created as a result of this discussion:
http://apache-ignite-developers.2346864.n4.nabble.com/Memory-policies-examples-fails-td16717.html

Presently a page memory can throw an OOM exception due to a misconfiguration. 
The message of the exception has to point out to a problematic place directly 
(size, ect.) so that the user can quickly fix it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Page Memory behavior with default memory policy

[Denis Magda]

Guys,

If a memory region is not expandable can we calculate the size of the default 
one automatically rather than setting it to predefined value like 1 GB, 512 MB, 
etc. ?

For instance, when a node is being started it finds out how much RAM is 
available and requests 70-80% of it for the default memory region usage. This 
should help us avoid this usability issue caused by the fact that we hard code 
the size.

—
Denis

> On Apr 17, 2017, at 2:24 PM, Dmitriy Setrakyan  wrote:
> 
> Sergey,
> 
> What prevents us from making MemoryPolicy expandable? For example, why not
> allocate another 1GB, when user runs out of memory? In my view, this would
> make our memory policies a lot more usable.
> 
> D.
> 
> On Mon, Apr 17, 2017 at 2:42 AM, Sergey Chugunov 
> wrote:
> 
>> Hello Denis,
>> 
>> There is a small piece of documentation in *MemoryConfiguration *class,
>> although I think it should be moved to *MemoryPolicyConfiguration *and
>> detailed a lot.
>> 
>> I'm going to write some documentation soon about validation rules that are
>> applied to memory policies configuration (there are some restrictions about
>> reserved name, default memory policy name and so on). I'll let you know
>> when it is done.
>> 
>> About your question: MemoryPolicy offheap regions were never supposed to be
>> expandable. If user uses default MemoryPolicy (which in fact doesn't have
>> any eviction algorithm configured) he/she will get OutOfMemory error when
>> default size is exhausted.
>> 
>> This OOM behavior was left intentionally to be consistent with the previous
>> behavior as configuring some default eviction policy on default
>> MemoryPolicy may lead to data loss and it is better to communicate wrong
>> memory configuration to user even with OOM than silently drop some cached
>> values.
>> 
>> Thanks,
>> Sergey.
>> 
>> On Mon, Apr 17, 2017 at 12:01 AM, Denis Magda  wrote:
>> 
>>> Guys,
>>> 
>>> I’ve been working on the documentation for page memory [1]. The doc is
>>> still in progress and it will be more profound with code snippets and
>>> pictures int the end, so don’t pay to the fact that it mostly consists of
>>> dry text only.
>>> 
>>> So, the question is different. Presently, javadoc part is useless (and
>>> will be written by me as well) and it’s not clear what’s the default
>>> behavior of the page memory that has only one memory policy (default)
>> that
>>> instantiates a single memory region. Looking at the code I see that the
>>> default police creates 1 GB region that should be expandable because the
>>> policy doesn’t define an eviction algorithm. It means that if my app goes
>>> beyond 1 GB then the page memory will take more memory from an OS. Is my
>>> understanding correct?
>>> 
>>> [1] https://apacheignite.readme.io/docs/page-memory <
>>> https://apacheignite.readme.io/docs/page-memory>
>>> 
>>> —
>>> Denis
>> 

[jira] [Created] (IGNITE-5007) Clarify some parameters of MemoryConfiguration and MemoryPolicyConfiguration

[Denis Magda (JIRA)]

Denis Magda created IGNITE-5007:
---

 Summary: Clarify some parameters of MemoryConfiguration and 
MemoryPolicyConfiguration
 Key: IGNITE-5007
 URL: https://issues.apache.org/jira/browse/IGNITE-5007
 Project: Ignite
  Issue Type: Sub-task
Reporter: Denis Magda
Assignee: Alexey Goncharuk
Priority: Blocker


Alex, please look for TODOs in the javadoc of {{MemoryConfiguration}} and 
{{MemoryPolicyConfiguratino}} files. Once your provide more information for the 
marked parameters assign the ticket back to me.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Page Memory behavior with default memory policy

[Dmitriy Setrakyan]

On Mon, Apr 17, 2017 at 4:40 AM, Alexey Goncharuk <
alexey.goncha...@gmail.com> wrote:

> Agree, I will rename it if there are no objections.
>

+1

Re: Memory policies examples fails

[Dmitriy Setrakyan]

On Mon, Apr 17, 2017 at 9:06 AM, Sergey Chugunov 
wrote:

> Dmitriy,
>
> I think this issue highlights another lack in our documentation that
> emerged with introduction of PageMemory.
>
> In current documentation we have some recommendations about memory capacity
> planning.
> As PageMemory storage data structures introduce their own overhead; this
> section of documentation must be updated.
>
> As per "null" in OutOfMemoryException message, I've already addressed it so
> now when PageMemory fails to allocate new page, it throws an
> IgniteOutOfMemoryException with message like "Failed to allocate new page
> within  MemoryPolicy".
>

Sergey, the error message should also mention which configuration property
needs to be changed to fix the issue.

Re: Binary objects and cache store

[Dmitriy Setrakyan]

On Wed, Feb 8, 2017 at 4:57 PM, Denis Magda  wrote:

> Cross-posting to the dev list.
>
> Igniters, what if we make “storeKeepBinary” = true by default in Ignite
> 2.0? Presently, the user has to tweak the configuration manually.
>
>
Makes sense to me.

Re: cache query operations

[Denis Magda]

Did you have a chance to take a look this documentation that explains how DML 
statements are turned into cache operations?
https://apacheignite.readme.io/docs/dml#dml-operations 


—
Denis

> On Apr 17, 2017, at 7:20 AM, ALEKSEY KUZNETSOV  
> wrote:
> 
> Hi, Igniters! When one fires query like this : ignite().cache(null
> ).query(...)
> The query would be executed against the schema. I wonder, how the schema is
> synchronized with cache, with cache entries?
> 
> What i mean is , cache.query(...update operation...) must eventually update
> the cache entry in cache. But how does it happen?(cache API must be called,
> but i cannot realize how)
> -- 
> 
> *Best Regards,*
> 
> *Kuznetsov Aleksey*

Re: IGNITE-4799 MetricsUpdate after each job

[Denis Magda]

Yasha, Sergey,

As far as I recall discovery SPI exchanges not only compute metrics but cache 
metrics as well over the cluster ring.

Presently, 
org.apache.ignite.configuration.IgniteConfiguration#getMetricsUpdateFrequency 
is compute metrics oriented (at least the javadoc says this). After we use this 
parameter for the all metrics exchanged over discovery we need update the 
javadoc and, obviously, discontinue 0 as a possible value because I don’t see 
how to apply it for cache metrics. Does it make sense to you?

—
Denis

> On Apr 17, 2017, at 6:02 AM, Yakov Zhdanov  wrote:
> 
> Denis M,
> 
> Can you please elaborate on this?
> Currently 
> org.apache.ignite.configuration.IgniteConfiguration#getMetricsUpdateFrequency
> has special values like 0 and -1 that hardly make sense if we use this
> parameter in discovery to issue metrics updates.
> 
> I would agree with Sasha and interpret this parameter as delay between
> metrics update messages thus making 0 and -1 illegal.
> 
> Thanks!
> 
> --Yakov

Re: question: How data are stored in IgniteCache?

[Denis Magda]

It depends on a storage. If it’s a relational database and isStoreKeepBinary is 
false then an object will be deserialized upon the store invocation and the 
storage will insert or update a record using “INSERT * ..” or “UPDATE * …” 
statements. Take a look at the code.

—
Denis

> On Apr 17, 2017, at 5:04 AM, Vyacheslav Daradur  wrote:
> 
>>> When you transfer an object over the wire or put it into a persistent
> store (withKeepBinary property enabled) then only the byte array is used.
> If we put objects into persistent store and withKeepBinary property is
> DISABLED, in wich form it will be stored?
> 
> 2017-04-16 18:29 GMT+03:00 Denis Magda :
> 
>> 1. When you transfer an object over the wire or put it into a persistent
>> store (withKeepBinary property enabled) then only the byte array is used.
>> 2. No
>> 
>> —
>> Denis
>> 
>>> On Apr 15, 2017, at 1:51 PM, Vyacheslav Daradur 
>> wrote:
>>> 
> If we use a cache which is configured to use binary mashaller:
> 1. In which cases a placed in the cache object (serialized object)
>> won't
>>> be wrapped?
> 2. In which cases a placed in the cache object won't be serialized (to
>>> byte array) and will be stored as is?
>>> 
>>> I meant: Are there any cases of described (1,2) situations.
>>> If yes, which cases?
>>> 
>>> 2017-04-15 23:44 GMT+03:00 Vyacheslav Daradur :
>>> 
 If we use a cache which is configured to use binary mashaller:
 1. In which cases a placed in the cache object (serialized object) won't
 be wrapped?
 2. In which cases a placed in the cache object won't be serialized (to
 byte array) and will be stored as is?
 
 2017-04-14 20:03 GMT+03:00 Denis Magda :
 
> If a serialized object is stored in an on-heap cache then it will be
> wrapped by BinaryObjectImp but if it’s an off-heap cache then
> BinaryObjectOffHeapImpl is used instead.
> 
> —
> Denis
> 
>> On Apr 14, 2017, at 12:32 AM, Vyacheslav Daradur >> 
> wrote:
>> 
 when it’s stored in memory in a specific cache partition
>> Does that mean that any serialized object is always stored IN MEMORY
>> as
> is
>> wrapped by BinaryObjectImpl?
>> 
>> 
>> 
>> 2017-04-14 3:42 GMT+03:00 Denis Magda :
>> 
>>> A serialized object is always wrapped by BinaryObjectImpl when it’s
> stored
>>> in memory in a specific cache partition or you access it from your
>>> application in a form of BinaryObject. However, when you transfer the
>>> object over the wire or put it into a persistent store
>> (withKeepBinary
>>> property enabled) then only the byte array is used.
>>> 
>>> —
>>> Denis
>>> 
 On Apr 13, 2017, at 12:21 AM, Vyacheslav Daradur <
>> daradu...@gmail.com
>> 
>>> wrote:
 
 Denis, thank you for answers.
 
 I meant another.
 
 For example:
 Cache queries use a BinaryObjectImpl and a withKeepBinary-mode use
> it, so
 looks like all actions on serialized object are make via a
>>> BinaryObjectImpl.
 
 Does a serialized object always is stored as BinaryObjectImpl or it
> will
>>> be
 wrapped on demand?
 
 2017-04-12 22:34 GMT+03:00 Denis Magda :
 
> A Java wrapper around an actual binary byte array with some
> additional
> fields and methods to work with the serialized data.
> 
> —
> Denis
> 
>> On Apr 12, 2017, at 8:33 AM, Vyacheslav Daradur <
> daradu...@gmail.com>
> wrote:
>> 
>> In what cases BinaryObjecImpl is used?
>> 
>> 2017-04-12 18:08 GMT+03:00 Denis Magda :
>> 
>>> Hi,
>>> 
>>> A cache entry is always stored in a binary format (byte array)
>> in a
> cache.
>>> Even when you transfer an entry from one node to another, as a
> result
>>> of
>>> cache.put(…), operation the entry will be serialized into the
> binary
> format
>>> and transferred over the wire.
>>> 
>>> —
>>> Denis
>>> 
 On Apr 12, 2017, at 1:11 AM, Vyacheslav Daradur <
> daradu...@gmail.com
 
>>> wrote:
 
 Hello Igniters!
 
 I have one conceptual question:
 
 When we put an object in IgniteCache, how it is stored?
 
 As I understand, after marshalling we have an array of bytes,
 1) in a local node it is wrapped in BinaryObjectImpl and stored
>> in
> memory
 2) it is sent to remote node as byte array where it will be
> wrapped
>>> in
 BinaryObjectImpl and be stored 

Re: TouchedExpiryPolicy works incorrect in some cases IGNITE-4401

[Denis Magda]

It doesn’t matter who is right and who is wrong unless someone gets to the 
bottom of the issue debugging it.

I would suggest to create a simple unit test with two caches and trying to 
reproduce the following without computations and other redundant stuff. Would 
you like to work on this?

—
Denis

> On Apr 17, 2017, at 12:44 AM, ALEKSEY KUZNETSOV  
> wrote:
> 
> Why do u think so.
> First of all, the output above is not correct. After 3 iteration key-value
> API strats to return empty value.
> 
> Every 5 seconds(iteration sleep time) repository.getAttributes("1").size()
> is got called. Which makes an entry "touch" and the entry wont be expired
> for as long as 10 seconds.
> 
> Expiry policy says:
> 
> An {@link ExpiryPolicy} that defines the expiry {@link Duration}
> * of a Cache Entry based on when it was last touched. A touch includes
> ** creation, update or **access**.*
> 
> 
> пт, 14 апр. 2017 г. в 18:42, Denis Magda :
> 
>> The iteration happens multiple time which means that the key-value API had
>> to return an empty result set on second or third iteration. But this never
>> happens.
>> 
>> In any case, do you want to find a root of the issue and fix it?
>> Otherwise, we can update the description and wait while someone else fixes
>> it.
>> 
>> —
>> Denis
>> 
>>> On Apr 14, 2017, at 1:33 AM, ALEKSEY KUZNETSOV 
>> wrote:
>>> 
>>> Because expiry time is 10 seconds, while loop iterates every 5 seconds
>>> 
>>> пт, 14 апр. 2017 г. в 11:32, ALEKSEY KUZNETSOV >> :
>>> 
 No, the bug is in SQL query, not key-value storage.
 
 пт, 14 апр. 2017 г. в 11:11, Vladislav Pyatkov :
 
> Denis, Aleksey,
> 
> It is correct, remember I have already said something like[1].
> I have no idea, why this happened in this case with SQL.
> 
> [1]:
> 
> 
>> http://apache-ignite-developers.2346864.n4.nabble.com/TouchedExpiryPolicy-works-incorrect-in-some-cases-IGNITE-4401-td16349.html#a16356
> 
> On Fri, Apr 14, 2017 at 4:29 AM, Denis Magda 
>> wrote:
> 
>> I could reproduce the issue and this should be what Denis K. meant by
>> saying “expiration policy works incorrectly”.
>> 
>> If you remove the expiration policy from the caches' configuration
>> then
>> the issue disappears. In general, SQL engine processes an expiration
> event
>> properly because the SQL queries return an empty result set as
>> expected
> but
>> something doesn’t work well with key-value operations.
>> 
>> *Denis K*, *Vlad P.*, as creators of the ticket please confirm that
>> this
>> is the case.
>> 
>> Please keep debugging this and switch to the latest Ignite version.
>> 
>> —
>> Denis
>> 
>> 
>>> On Apr 13, 2017, at 4:22 AM, ALEKSEY KUZNETSOV <
> alkuznetsov...@gmail.com>
>> wrote:
>>> 
>>> any feedback?
>>> 
>>> чт, 13 апр. 2017 г. в 11:51, ALEKSEY KUZNETSOV <
> alkuznetsov...@gmail.com
>>> :
>>> 
 You should run ExpiryPolicyTest. The output should contain strings
> like
 contains? new AffinityKey("1", "1"): and contains?2 new
>> AffinityKey("1", "
 1"): and empty cursor? =
 If you look at them you will see, that cache contains affinity key
> new
 AffinityKey("1", "1") whereas cursor is empty(on second iteration).
> From
 this output you can conclude SQL query returns icorrect data(empty
>> value)
 
 
 чт, 13 апр. 2017 г. в 3:42, Denis Magda :
 
> Bluntly speaking I have no idea where to look and what to expect.
> This
>> is
> output of the test execution of my machine:
> 
> SQL res: [[1], [d]]
> 2
> Op consume: 303
> Value: org.ignite.test.EDU@22db8f4
> SQL res: []
> 0
> Op consume: 9
> Value: org.ignite.test.EDU@29caf222
> SQL res: []
> 0
> Op consume: 15
> Value: org.ignite.test.EDU@7cd1ac19
> SQL res: []
> 0
> Op consume: 5
> 
> Please be more specific, there are too many files in the code.
> 
> —
> Denis
> 
>> On Apr 12, 2017, at 4:50 AM, ALEKSEY KUZNETSOV <
> alkuznetsov...@gmail.com> wrote:
>> 
>> So what do u think about the issue ?
>> 
>> ср, 12 апр. 2017 г. в 10:42, ALEKSEY KUZNETSOV <
> alkuznetsov...@gmail.com>:
>> 
>>> I have already attached simlified version. Shall i simplify it
> more ?
>>> 
>>> вт, 11 апр. 2017 г. в 19:28, Denis Magda :
>>> 
>>> Can you attach the simplified version? Just want to avoid any
>> side
> effects.
>>> 
>>> —
>>> Denis

Re: Oleg Ignatenko. Request for contribution permissions.

[Denis Magda]

Oleg,

Someone has already added you to the contributors list in JIRA. Please check up 
that you can assign the tickets of interest on yourself.

—
Denis

> On Apr 17, 2017, at 9:13 AM, Oleg Ignatenko  wrote:
> 
> Hello,
> 
> My name is Oleg Ignatenko. I am interested in machine learning features that 
> are suggested to Apache Ignite in IGNITE-4572. I would like to contribute to 
> the project.
> 
> Please grant me the required permissions so I can pick up related tickets (at 
> this time I am specifically interested in IGNITE-4964). My JIRA account is 
> oignatenko
> 
> Looking forward to becoming an active member of the community.
> 
> regards,
> Oleg
> 

Re: Adding ML to Ignite, IGNITE-4572

[Denis Magda]

Thanks Oleg, I’ll review the doc a bit later.

Yuri, I’ve reopened IGNITE-5000. Please consider my latest notes and prepare 
one more pull-request:
https://issues.apache.org/jira/browse/IGNITE-5000 


—
Denis

> On Apr 17, 2017, at 8:48 AM, oignatenko  wrote:
> 
> Denis, I prepared preliminary draft doc for readme.io , it 
> reflects proposed
> rename. You can find it attached to JIRA IGNITE-4964
> 
> With regards to merge, per my reading of recent mail from Yury it is done
> except for module rename which is on its way to master now.
> 
> regards, Oleg
> 
> 
> Denis Magda-2 wrote
>> Oleg, Nikita, Yuri,
>> 
>> Frankly, I don’t like how the module is named presently which is
>> 'ignite-math’.
>> 
>> I propose to rename it to ‘ignite-ml’. Presently, it holds a small
>> fraction of all ML functionality we might have in the future and this is
>> why, thinking of future, it will be a right decision to name it
>> accordingly.
>> 
>> Any concerns?
>> 
>> BTW, have the module been fully merged? I want to try to build it and
>> check the examples.
>> 
>> —
>> Denis
>> 
>>> On Apr 14, 2017, at 7:05 AM, oignatenko 
> 
>> oignatenko@
> 
>>  wrote:
>>> 
>>> Thank you Denis, that sounds like a good plan. I am working on it now. I
>>> didn't reassign IGNITE-4964, just set a watch on it. Will comment on it
>>> when
>>> I have concrete details to share.
>>> 
>>> Side note my build check completed successfully, although it took quite a
>>> bit of time: I cleaned my local maven repo to make sure that things
>>> really
>>> work from the clean state.
>>> 
>>> 
>>> 
>>> --
>>> View this message in context:
>>> http://apache-ignite-developers.2346864.n4.nabble.com/Adding-ML-to-Ignite-IGNITE-4572-tp13936p16677.html
>>> Sent from the Apache Ignite Developers mailing list archive at
>>> Nabble.com.
> 
> 
> 
> 
> 
> --
> View this message in context: 
> http://apache-ignite-developers.2346864.n4.nabble.com/Adding-ML-to-Ignite-IGNITE-4572-tp13936p16758.html
>  
> 
> Sent from the Apache Ignite Developers mailing list archive at Nabble.com 
> .

[GitHub] ignite pull request #1813: mem.OutOfMemoryException was renamed

[sergey-chugunov-1985]

GitHub user sergey-chugunov-1985 opened a pull request:

https://github.com/apache/ignite/pull/1813

mem.OutOfMemoryException was renamed



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite oomexception-rename

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1813.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1813


commit 50c4960fb62680fdfdfb2c8f1c53c3a04d9ce074
Author: Sergey Chugunov 
Date:   2017-04-17T16:29:02Z

mem.OutOfMemoryException was renamed; allocation failure message was 
improved




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #1812: testAllocatedMemory fix

[sergey-chugunov-1985]

GitHub user sergey-chugunov-1985 opened a pull request:

https://github.com/apache/ignite/pull/1812

testAllocatedMemory fix



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite testAllocatedMemory-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1812.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1812


commit edc5762462588e4d4e7cfd4e1284e6fb7042de24
Author: Sergey Chugunov 
Date:   2017-04-17T16:55:03Z

ClusterNodeMetricsSelfTest::testAllocatedMemory was fixed according to 
latest changes in memory management




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Oleg Ignatenko. Request for contribution permissions.

[Oleg Ignatenko]

Hello,

My name is Oleg Ignatenko. I am interested in machine learning features 
that are suggested to Apache Ignite in IGNITE-4572. I would like to 
contribute to the project.


Please grant me the required permissions so I can pick up related 
tickets (at this time I am specifically interested in IGNITE-4964). My 
JIRA account is oignatenko


Looking forward to becoming an active member of the community.

regards,
Oleg

Re: Memory policies examples fails

[Sergey Chugunov]

Dmitriy,

I think this issue highlights another lack in our documentation that
emerged with introduction of PageMemory.

In current documentation we have some recommendations about memory capacity
planning.
As PageMemory storage data structures introduce their own overhead; this
section of documentation must be updated.

As per "null" in OutOfMemoryException message, I've already addressed it so
now when PageMemory fails to allocate new page, it throws an
IgniteOutOfMemoryException with message like "Failed to allocate new page
within  MemoryPolicy".


Thanks,
Sergey.

On Mon, Apr 17, 2017 at 6:52 PM, Dmitriy Setrakyan 
wrote:

> Wow If it happened to Denis, it will happen to many other users as
> well. With the exception error message saying "Null", I doubt any user
> would figure out what to do.
>
> Can we fix the error message?
>
> D.
>
>
> On Mon, Apr 17, 2017 at 6:03 AM, Sergey Chugunov <
> sergey.chugu...@gmail.com>
> wrote:
>
> > Denis,
> >
> > I found an issue with the example: each cache introduces some memory
> > overhead (mostly for cache partitions data structures) and it must be
> > counted in MemoryPolicy size.
> >
> > To fix the example just increase size of *10MB_Region_Eviction *to 15 MB
> > and *5MB_Region_Swapping *to 10 MB.
> >
> > I didn't find any documentation on this, but I think it is part of new
> > PageMemory-based storage so it needs to be documented as part of
> ignite-2.0
> > release.
> >
> > Thanks,
> > Sergey.
> >
> >
> >
> > On Mon, Apr 17, 2017 at 2:50 PM, Sergey Chugunov <
> > sergey.chugu...@gmail.com>
> > wrote:
> >
> > > Denis,
> > >
> > > It looks like partition exchange mechanism allocates a lot of pages
> right
> > > after the cache is started.
> > > I'm trying to figure out what's going on.
> > >
> > > Thanks,
> > > Sergey.
> > >
> > > On Mon, Apr 17, 2017 at 12:05 AM, Denis Magda 
> wrote:
> > >
> > >> Igniters,
> > >>
> > >> I’ve created a simple example to demonstrate how to set up various
> > >> policies:
> > >> https://github.com/apache/ignite/blob/master/examples/src/
> > >> main/java/org/apache/ignite/examples/datagrid/MemoryPolicies
> > Example.java
> > >>
> > >> However, this tiny example fails with
> > >>
> > >> Caused by: class org.apache.ignite.internal.mem.OutOfMemoryException:
> > >> null
> > >> at org.apache.ignite.internal.pagemem.impl.
> PageMemoryNoStoreImp
> > >> l.allocatePage(PageMemoryNoStoreImpl.java:255)
> > >> at org.apache.ignite.internal.processors.cache.
> IgniteCacheOffhe
> > >> apManagerImpl.allocateForTree(IgniteCacheOffheapManagerImpl.java:635)
> > >> at org.apache.ignite.internal.processors.cache.
> IgniteCacheOffhe
> > >> apManagerImpl.createCacheDataStore0(IgniteCacheOffheapManage
> > >> rImpl.java:727)
> > >> at org.apache.ignite.internal.processors.cache.
> IgniteCacheOffhe
> > >> apManagerImpl.createCacheDataStore(IgniteCacheOffheapManager
> > >> Impl.java:711)
> > >> at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> > >> GridDhtLocalPartition.(GridDhtLocalPartition.java:162)
> > >> at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> > >> GridDhtPartitionTopologyImpl.createPartition(GridDhtPartitio
> > >> nTopologyImpl.java:718)
> > >> at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> > >> GridDhtPartitionTopologyImpl.initPartitions0(GridDhtPartitio
> > >> nTopologyImpl.java:405)
> > >> at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> > >> GridDhtPartitionTopologyImpl.beforeExchange(GridDhtPartition
> > >> TopologyImpl.java:569)
> > >> at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> > >> preloader.GridDhtPartitionsExchangeFuture.
> > distributedExchange(GridDhtPar
> > >> titionsExchangeFuture.java:844)
> > >> at org.apache.ignite.internal.processors.cache.distributed.
> dht.
> > >> preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartit
> > >> ionsExchangeFuture.java:573)
> > >> at org.apache.ignite.internal.processors.cache.
> GridCachePartiti
> > >> onExchangeManager$ExchangeWorker.body(GridCacheP
> > >> artitionExchangeManager.java:1800)
> > >> ... 2 more
> > >>
> > >>
> > >>
> > >> What I’m doing wrong? Alex G., could you take a look? Probably I’m
> > >> missing something due to a lack of documentation.
> > >>
> > >> —
> > >> Denis
> > >>
> > >>
> > >
> >
>

Re: Adding ML to Ignite, IGNITE-4572

[oignatenko]

Denis, I prepared preliminary draft doc for readme.io, it reflects proposed
rename. You can find it attached to JIRA IGNITE-4964

With regards to merge, per my reading of recent mail from Yury it is done
except for module rename which is on its way to master now.

regards, Oleg


Denis Magda-2 wrote
> Oleg, Nikita, Yuri,
> 
> Frankly, I don’t like how the module is named presently which is
> 'ignite-math’.
> 
> I propose to rename it to ‘ignite-ml’. Presently, it holds a small
> fraction of all ML functionality we might have in the future and this is
> why, thinking of future, it will be a right decision to name it
> accordingly.
> 
> Any concerns?
> 
> BTW, have the module been fully merged? I want to try to build it and
> check the examples.
> 
> —
> Denis
> 
>> On Apr 14, 2017, at 7:05 AM, oignatenko 

> oignatenko@

>  wrote:
>> 
>> Thank you Denis, that sounds like a good plan. I am working on it now. I
>> didn't reassign IGNITE-4964, just set a watch on it. Will comment on it
>> when
>> I have concrete details to share.
>> 
>> Side note my build check completed successfully, although it took quite a
>> bit of time: I cleaned my local maven repo to make sure that things
>> really
>> work from the clean state.
>> 
>> 
>> 
>> --
>> View this message in context:
>> http://apache-ignite-developers.2346864.n4.nabble.com/Adding-ML-to-Ignite-IGNITE-4572-tp13936p16677.html
>> Sent from the Apache Ignite Developers mailing list archive at
>> Nabble.com.





--
View this message in context: 
http://apache-ignite-developers.2346864.n4.nabble.com/Adding-ML-to-Ignite-IGNITE-4572-tp13936p16758.html
Sent from the Apache Ignite Developers mailing list archive at Nabble.com.

Re: Memory policies examples fails

[Dmitriy Setrakyan]

Wow If it happened to Denis, it will happen to many other users as
well. With the exception error message saying "Null", I doubt any user
would figure out what to do.

Can we fix the error message?

D.


On Mon, Apr 17, 2017 at 6:03 AM, Sergey Chugunov 
wrote:

> Denis,
>
> I found an issue with the example: each cache introduces some memory
> overhead (mostly for cache partitions data structures) and it must be
> counted in MemoryPolicy size.
>
> To fix the example just increase size of *10MB_Region_Eviction *to 15 MB
> and *5MB_Region_Swapping *to 10 MB.
>
> I didn't find any documentation on this, but I think it is part of new
> PageMemory-based storage so it needs to be documented as part of ignite-2.0
> release.
>
> Thanks,
> Sergey.
>
>
>
> On Mon, Apr 17, 2017 at 2:50 PM, Sergey Chugunov <
> sergey.chugu...@gmail.com>
> wrote:
>
> > Denis,
> >
> > It looks like partition exchange mechanism allocates a lot of pages right
> > after the cache is started.
> > I'm trying to figure out what's going on.
> >
> > Thanks,
> > Sergey.
> >
> > On Mon, Apr 17, 2017 at 12:05 AM, Denis Magda  wrote:
> >
> >> Igniters,
> >>
> >> I’ve created a simple example to demonstrate how to set up various
> >> policies:
> >> https://github.com/apache/ignite/blob/master/examples/src/
> >> main/java/org/apache/ignite/examples/datagrid/MemoryPolicies
> Example.java
> >>
> >> However, this tiny example fails with
> >>
> >> Caused by: class org.apache.ignite.internal.mem.OutOfMemoryException:
> >> null
> >> at org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImp
> >> l.allocatePage(PageMemoryNoStoreImpl.java:255)
> >> at org.apache.ignite.internal.processors.cache.IgniteCacheOffhe
> >> apManagerImpl.allocateForTree(IgniteCacheOffheapManagerImpl.java:635)
> >> at org.apache.ignite.internal.processors.cache.IgniteCacheOffhe
> >> apManagerImpl.createCacheDataStore0(IgniteCacheOffheapManage
> >> rImpl.java:727)
> >> at org.apache.ignite.internal.processors.cache.IgniteCacheOffhe
> >> apManagerImpl.createCacheDataStore(IgniteCacheOffheapManager
> >> Impl.java:711)
> >> at org.apache.ignite.internal.processors.cache.distributed.dht.
> >> GridDhtLocalPartition.(GridDhtLocalPartition.java:162)
> >> at org.apache.ignite.internal.processors.cache.distributed.dht.
> >> GridDhtPartitionTopologyImpl.createPartition(GridDhtPartitio
> >> nTopologyImpl.java:718)
> >> at org.apache.ignite.internal.processors.cache.distributed.dht.
> >> GridDhtPartitionTopologyImpl.initPartitions0(GridDhtPartitio
> >> nTopologyImpl.java:405)
> >> at org.apache.ignite.internal.processors.cache.distributed.dht.
> >> GridDhtPartitionTopologyImpl.beforeExchange(GridDhtPartition
> >> TopologyImpl.java:569)
> >> at org.apache.ignite.internal.processors.cache.distributed.dht.
> >> preloader.GridDhtPartitionsExchangeFuture.
> distributedExchange(GridDhtPar
> >> titionsExchangeFuture.java:844)
> >> at org.apache.ignite.internal.processors.cache.distributed.dht.
> >> preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartit
> >> ionsExchangeFuture.java:573)
> >> at org.apache.ignite.internal.processors.cache.GridCachePartiti
> >> onExchangeManager$ExchangeWorker.body(GridCacheP
> >> artitionExchangeManager.java:1800)
> >> ... 2 more
> >>
> >>
> >>
> >> What I’m doing wrong? Alex G., could you take a look? Probably I’m
> >> missing something due to a lack of documentation.
> >>
> >> —
> >> Denis
> >>
> >>
> >
>

[jira] [Created] (IGNITE-5006) .NET: SQL does not work with F# record types

[Pavel Tupitsyn (JIRA)]

Pavel Tupitsyn created IGNITE-5006:
--

 Summary: .NET: SQL does not work with F# record types
 Key: IGNITE-5006
 URL: https://issues.apache.org/jira/browse/IGNITE-5006
 Project: Ignite
  Issue Type: Bug
  Components: platforms
Affects Versions: 1.7
Reporter: Pavel Tupitsyn
Assignee: Pavel Tupitsyn
 Fix For: 2.0


The following code returns null results:

{code}
type Person = 
{ 
[]
id : int; 
[]
name : String 
}

...

cache.QueryFields(new SqlFieldsQuery("from person select id")).GetAll()
{code}

F# record fields have {{@}} symbol in the end. We should clean up or escape 
these.

{{cache.QueryFields(new SqlFieldsQuery("from person select id")).GetAll()}}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] ignite pull request #1811: Ignite 5000

[ybabak]

GitHub user ybabak opened a pull request:

https://github.com/apache/ignite/pull/1811

Ignite 5000

Module ignite-math renamed to ignite-ml and examples moved to the separate 
folder/profile.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-5000

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1811.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1811


commit 40976d619d2d8972301ff42bf61778adc76075c3
Author: Yury Babak 
Date:   2017-04-17T14:01:58Z

IGNITE-5000 (Rename Ignite Math module to Ignite ML module):
   done.

commit 2575b683ac3c427035c1ff84059753fdefe453ab
Author: Yury Babak 
Date:   2017-04-17T15:14:27Z

IGNITE-5000 (Rename Ignite Math module to Ignite ML module):
  Split examples.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #1810: MemoryConfiguration javadoc updated

[sergey-chugunov-1985]

GitHub user sergey-chugunov-1985 opened a pull request:

https://github.com/apache/ignite/pull/1810

MemoryConfiguration javadoc updated



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite memCfg-doc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1810.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1810


commit 8226bdb7cb310cece081b3db00311d791e0fc025
Author: Sergey Chugunov 
Date:   2017-04-17T14:45:02Z

MemoryConfiguration javadoc updated




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #1590: ignite-4680

[kdudkov]

Github user kdudkov closed the pull request at:

https://github.com/apache/ignite/pull/1590


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #1686: IGNITE-4669 .NET: Sort binary object fields

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/1686


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

cache query operations

[ALEKSEY KUZNETSOV]

Hi, Igniters! When one fires query like this : ignite().cache(null
).query(...)
The query would be executed against the schema. I wonder, how the schema is
synchronized with cache, with cache entries?

What i mean is , cache.query(...update operation...) must eventually update
the cache entry in cache. But how does it happen?(cache API must be called,
but i cannot realize how)
-- 

*Best Regards,*

*Kuznetsov Aleksey*

[jira] [Created] (IGNITE-5005) WriteBehindStore - split flusher's to different classes by writeCoalescing

[Alexander Belyak (JIRA)]

Alexander Belyak created IGNITE-5005:


 Summary: WriteBehindStore - split flusher's to different classes 
by writeCoalescing
 Key: IGNITE-5005
 URL: https://issues.apache.org/jira/browse/IGNITE-5005
 Project: Ignite
  Issue Type: Improvement
Reporter: Alexander Belyak


In GridCacheWriteBehindStore.Flusher too many if statements, because its  
behavior depends of writeCoalescing flag too much. Need to split this class 
into two different (with one abstract base Flusher class).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (IGNITE-5004) GridCacheWriteBegindStore - remove StatefulValue

[Alexander Belyak (JIRA)]

Alexander Belyak created IGNITE-5004:


 Summary: GridCacheWriteBegindStore - remove StatefulValue
 Key: IGNITE-5004
 URL: https://issues.apache.org/jira/browse/IGNITE-5004
 Project: Ignite
  Issue Type: Improvement
Affects Versions: 1.9
Reporter: Alexander Belyak


If writeCoalescing=false - GridCacheWriteBehindStore doesn't need to create 
StatefulValue for each KV entry. Need to implement WBStore without this wrapper 
at all (if it is possible to solve ABA problem in cacheMap's) or with thinner 
wrapper without unnecesarry syncronizations/state.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (IGNITE-5003) Parallel write same key in CacheWriteBehindStore

[Alexander Belyak (JIRA)]

Alexander Belyak created IGNITE-5003:


 Summary: Parallel write same key in CacheWriteBehindStore
 Key: IGNITE-5003
 URL: https://issues.apache.org/jira/browse/IGNITE-5003
 Project: Ignite
  Issue Type: Improvement
Affects Versions: 1.9
Reporter: Alexander Belyak


Now GridCacheWriteBehindStore.updateCache wait for writeLock in StatefulValue 
and, moreover, waitForFlush() if value is in pending (flushing) state. We need 
to remove waiting.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (IGNITE-5002) fix tests in master

[Konstantin Dudkov (JIRA)]

Konstantin Dudkov created IGNITE-5002:
-

 Summary: fix tests in master
 Key: IGNITE-5002
 URL: https://issues.apache.org/jira/browse/IGNITE-5002
 Project: Ignite
  Issue Type: Task
Reporter: Konstantin Dudkov
Assignee: Konstantin Dudkov






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (IGNITE-5001) .NET: Tests cleanup

[Pavel Tupitsyn (JIRA)]

Pavel Tupitsyn created IGNITE-5001:
--

 Summary: .NET: Tests cleanup
 Key: IGNITE-5001
 URL: https://issues.apache.org/jira/browse/IGNITE-5001
 Project: Ignite
  Issue Type: Improvement
  Components: platforms
Reporter: Pavel Tupitsyn
Priority: Trivial
 Fix For: 2.0






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] ignite pull request #1808: IGNITE-4587: Found bug - CacheVersionIO is change...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/1808


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #1809: Ignite 4998

[alexpaschenko]

GitHub user alexpaschenko opened a pull request:

https://github.com/apache/ignite/pull/1809

Ignite 4998



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-4998

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1809.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1809


commit fcadf67cf134d603f4b8c9364ab4584cc40efed6
Author: devozerov 
Date:   2017-02-02T08:54:10Z

IGNITE-4570: DDL parsing.

commit 796c933ac70d75f4f642f5b351abc762effd
Author: devozerov 
Date:   2017-03-02T11:29:53Z

Merge branch 'ignite-2.0' into ignite-4565-ddl

commit c05be524a9b73356d26474aa9962434155d111f3
Author: Alexander Paschenko 
Date:   2017-03-03T10:55:41Z

IGNITE-4633 Introduced component type for DDL statements processor.

commit f79ead47a895eb6d205ea0f2a813efb23b67c989
Author: Alexander Paschenko 
Date:   2017-03-03T11:57:10Z

IGNITE-4635: Introduced DDL worker thread.

commit d97716ef9024aea562899d0b8b17c32c49e41f4b
Author: devozerov 
Date:   2017-03-03T12:40:00Z

Added schema version.

commit 19f0568f4653dfcace6148ded0bd357e4be65a48
Author: devozerov 
Date:   2017-03-09T08:06:38Z

Merge branch 'ignite-2.0' into ignite-4565-ddl

commit f6fa2659d88994f7942f1ac5e1fff65e697ac541
Author: devozerov 
Date:   2017-03-09T11:05:27Z

Creating correct path for index create.

commit f4a56fe67739ca6b8666cec6c0b2f7bce288d191
Author: devozerov 
Date:   2017-03-09T11:31:17Z

Moved full-text index into separate property.

commit 157db2b6bfd40a76af3ee695665fe9c6c4a4b54e
Author: devozerov 
Date:   2017-03-09T12:04:09Z

Removed unnecessary GridQueryIndexType.

commit d8d2ad8f93efedd4787ec04a9886b28266f7ac8d
Author: devozerov 
Date:   2017-03-09T12:32:28Z

WIP on general wire-up.

commit fc2cf15fd31ebfb05052bf297aeaec959d61af1e
Author: devozerov 
Date:   2017-03-09T13:07:42Z

Started working on disco messages.

commit 0721eb98f5c856afb8e4bbd3607c4d00492f8ff9
Author: devozerov 
Date:   2017-03-09T13:11:07Z

Minors.

commit 323f77545e30ebfde00b81a741546914dd18b283
Author: devozerov 
Date:   2017-03-09T13:26:29Z

Merge branch 'ignite-2.0' into ignite-4565-ddl

# Conflicts:
#   
modules/core/src/main/java/org/apache/ignite/configuration/CacheConfiguration.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/query/GridCacheQueryManager.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/query/GridQueryProcessor.java
#   
modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/IgniteH2Indexing.java
#   
modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/ddl/DdlTask.java
#   
modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/opt/GridLuceneIndex.java

commit be88fa26f86e31c0ef562f9484412dde684327ef
Author: devozerov 
Date:   2017-03-09T13:49:26Z

Merge branch 'ignite-2.0' into ignite-4565-ddl

# Conflicts:
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/query/GridQueryProcessor.java

commit 539f1d84063c731460bb18bf59a0602c27482d22
Author: devozerov 
Date:   2017-03-09T13:50:37Z

Minors.

commit 752e71490b79f27fb182353ae7c7babb84f0b0fd
Author: devozerov 
Date:   2017-03-09T13:53:54Z

Minors.

commit 20bd9eddaacfa05c5a4db3d951fda544d53d5410
Author: devozerov 
Date:   2017-03-09T14:00:11Z

Merge branch 'ignite-2.0' into ignite-4565-ddl

# Conflicts:
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/query/GridQueryProcessor.java

commit a4d01a632506b82a1cedb9538a843229feb543be
Author: devozerov 
Date:   2017-03-09T14:03:38Z

Finished refactoring.

commit 9622039a540957ef73f558dbc99da0dbe88f38da
Author: devozerov 
Date:   2017-03-09T14:28:14Z

WIP.

commit 318ddedafc810fefdba12ac2334ac467c746dc04
Author: devozerov 
Date:   2017-03-09T14:31:53Z

WIP.

commit 9e32da96efb1cef44c81a360e35a494715b6483b
Author: devozerov 
Date:   2017-03-09T14:54:37Z

WIP.

commit cd477c039a082eb37e10848cb3aee71c01c01e20
Author: devozerov 
Date:   2017-03-09T15:05:21Z

WIP.

commit abf2f3389ee9036039be58a1e3b18c30d52a8f20
Author: devozerov 
Date:   2017-03-09T15:12:12Z

Minors.

Re: Adding ML to Ignite, IGNITE-4572

[Yury Babak]

Denis,

Yes, this module already in apache master branch.

Also I created ticket for module renaming, so we could discuss this renaming
in IGNITE-5000



--
View this message in context: 
http://apache-ignite-developers.2346864.n4.nabble.com/Adding-ML-to-Ignite-IGNITE-4572-tp13936p16743.html
Sent from the Apache Ignite Developers mailing list archive at Nabble.com.

Re: Memory policies examples fails

[Sergey Chugunov]

Denis,

I found an issue with the example: each cache introduces some memory
overhead (mostly for cache partitions data structures) and it must be
counted in MemoryPolicy size.

To fix the example just increase size of *10MB_Region_Eviction *to 15 MB
and *5MB_Region_Swapping *to 10 MB.

I didn't find any documentation on this, but I think it is part of new
PageMemory-based storage so it needs to be documented as part of ignite-2.0
release.

Thanks,
Sergey.



On Mon, Apr 17, 2017 at 2:50 PM, Sergey Chugunov 
wrote:

> Denis,
>
> It looks like partition exchange mechanism allocates a lot of pages right
> after the cache is started.
> I'm trying to figure out what's going on.
>
> Thanks,
> Sergey.
>
> On Mon, Apr 17, 2017 at 12:05 AM, Denis Magda  wrote:
>
>> Igniters,
>>
>> I’ve created a simple example to demonstrate how to set up various
>> policies:
>> https://github.com/apache/ignite/blob/master/examples/src/
>> main/java/org/apache/ignite/examples/datagrid/MemoryPoliciesExample.java
>>
>> However, this tiny example fails with
>>
>> Caused by: class org.apache.ignite.internal.mem.OutOfMemoryException:
>> null
>> at org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImp
>> l.allocatePage(PageMemoryNoStoreImpl.java:255)
>> at org.apache.ignite.internal.processors.cache.IgniteCacheOffhe
>> apManagerImpl.allocateForTree(IgniteCacheOffheapManagerImpl.java:635)
>> at org.apache.ignite.internal.processors.cache.IgniteCacheOffhe
>> apManagerImpl.createCacheDataStore0(IgniteCacheOffheapManage
>> rImpl.java:727)
>> at org.apache.ignite.internal.processors.cache.IgniteCacheOffhe
>> apManagerImpl.createCacheDataStore(IgniteCacheOffheapManager
>> Impl.java:711)
>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>> GridDhtLocalPartition.(GridDhtLocalPartition.java:162)
>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>> GridDhtPartitionTopologyImpl.createPartition(GridDhtPartitio
>> nTopologyImpl.java:718)
>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>> GridDhtPartitionTopologyImpl.initPartitions0(GridDhtPartitio
>> nTopologyImpl.java:405)
>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>> GridDhtPartitionTopologyImpl.beforeExchange(GridDhtPartition
>> TopologyImpl.java:569)
>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>> preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPar
>> titionsExchangeFuture.java:844)
>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>> preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartit
>> ionsExchangeFuture.java:573)
>> at org.apache.ignite.internal.processors.cache.GridCachePartiti
>> onExchangeManager$ExchangeWorker.body(GridCacheP
>> artitionExchangeManager.java:1800)
>> ... 2 more
>>
>>
>>
>> What I’m doing wrong? Alex G., could you take a look? Probably I’m
>> missing something due to a lack of documentation.
>>
>> —
>> Denis
>>
>>
>

Re: IGNITE-4799 MetricsUpdate after each job

[Yakov Zhdanov]

Denis M,

Can you please elaborate on this?
Currently 
org.apache.ignite.configuration.IgniteConfiguration#getMetricsUpdateFrequency
has special values like 0 and -1 that hardly make sense if we use this
parameter in discovery to issue metrics updates.

I would agree with Sasha and interpret this parameter as delay between
metrics update messages thus making 0 and -1 illegal.

Thanks!

--Yakov

[jira] [Created] (IGNITE-5000) Rename Ignite Math module to Ignite ML module

[Yury Babak (JIRA)]

Yury Babak created IGNITE-5000:
--

 Summary: Rename Ignite Math module to Ignite ML module
 Key: IGNITE-5000
 URL: https://issues.apache.org/jira/browse/IGNITE-5000
 Project: Ignite
  Issue Type: Task
Affects Versions: 2.0
Reporter: Yury Babak
Assignee: Yury Babak
 Fix For: 2.0


We want to rename math module to ml(machine learning) because it's a main 
planned functionality for this module.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (IGNITE-4999) Use one thread pool to flush all CacheWriteBehindStore

[Alexander Belyak (JIRA)]

Alexander Belyak created IGNITE-4999:


 Summary: Use one thread pool to flush all CacheWriteBehindStore
 Key: IGNITE-4999
 URL: https://issues.apache.org/jira/browse/IGNITE-4999
 Project: Ignite
  Issue Type: Improvement
Affects Versions: 1.9
Reporter: Alexander Belyak


For now we have flusher threads for each CacheWriteBehindStore so we can't 
create many caches with this mechanism (too many threads).
We should use single thread pool for all CacheWriteBehindStore instances (as 
for TTL).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

IGNITE-4799 MetricsUpdate after each job

[Sasha Belyak]

Now IgniteConfiguration.MetricsUpdate now can be:
-1 - disabled
0 - collect after each job
>0 - collect after specified amount in ms.
If we will use this parameter to send metrics - we should remove this
options and leave only >0 values available. Should I remove any 0 and -1
mentions in 4799?

[GitHub] ignite pull request #1808: IGNITE-4587: Found bug - CacheVersionIO is change...

[glukos]

GitHub user glukos opened a pull request:

https://github.com/apache/ignite/pull/1808

IGNITE-4587: Found bug - CacheVersionIO is changed, but size isn't



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite IGNITE-4587-version-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1808.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1808


commit 3590a6796dc440d49223b8c94e9bac53e4d51e6c
Author: Ivan Rakov 
Date:   2017-04-17T12:34:14Z

IGNITE-4587: Found bug - CacheVersionIO is changed, but size isn't




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #1807: ignite-4982

[kdudkov]

GitHub user kdudkov opened a pull request:

https://github.com/apache/ignite/pull/1807

ignite-4982



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite master-fix-tests

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1807.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1807


commit e7fb17b51ad434e5568f6379ba809479aafb8669
Author: Konstantin Dudkov 
Date:   2017-04-17T12:33:22Z

ignite-4982




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-4998) Add more dynamic indexing tests for various cache modes

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-4998:
---

 Summary: Add more dynamic indexing tests for various cache modes
 Key: IGNITE-4998
 URL: https://issues.apache.org/jira/browse/IGNITE-4998
 Project: Ignite
  Issue Type: Sub-task
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
 Fix For: 2.0






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: IGNITE-4052 ready for review

[Nikolai Tikhonov]

Vadim,

Thank you for your contribution. I'll look at changes. Can you please post
the list improvements to jira ticket?

On Mon, Apr 17, 2017 at 3:23 PM, Вадим Опольский 
wrote:

> Hi, Nikolay!
>
> I've made the following improvements ( https://github.com/apache/
> ignite/pull/1783 ):
>
> 1) Moved framework builder related code to separate method, so it make
> code cleaner and framework testable.
>
> 2) Call framework builder method from test to test role and user.
>
> 3) Added validating mesos role according with mesos role documentation
> http://mesos.apache.org/documentation/latest/roles/
>
> 4) Still using setEnv method because in IgniteFramework we cannot override
> static method and make mock static methods (except powermock, but it would
> require 3 extra dependencies in the module).
>
> P.S. setEnv method designed to worked both on Linux and Windows, and it
> does not left variable in system environment  after testing.
>
> Vadim Opolski
>
>
> 2017-04-14 12:07 GMT+03:00 Nikolay Tikhonov (JIRA) :
>
>>
>> [ https://issues.apache.org/jira/browse/IGNITE-4052?page=com.
>> atlassian.jira.plugin.system.issuetabpanels:comment-tabpane
>> l=15968814#comment-15968814 ]
>>
>> Nikolay Tikhonov commented on IGNITE-4052:
>> --
>>
>> [~javaller]
>> It meens that lets remove #setEnv mathod and will create mock in test
>> which will override {{getUser}} and {{getRole}} methods. Also how do you
>> think might be need to add validation for role? Which valid set of values
>> for this property?
>>
>> > Add ability to set up users for MESOS
>> > -
>> >
>> > Key: IGNITE-4052
>> > URL: https://issues.apache.org/jira/browse/IGNITE-4052
>> > Project: Ignite
>> >  Issue Type: Improvement
>> >  Components: general
>> >Affects Versions: 1.7
>> >Reporter: Nikolay Tikhonov
>> >Assignee: Vadim Opolski
>> >Priority: Trivial
>> >
>> > In current implementation Ignite Mesos Framework connects to MESOS
>> cluster via current user. Need to add ability to configure this parameters
>> via system env properties. Also need to add properties for mesos role.
>> > See org/apache/ignite/mesos/IgniteFramework.java:537
>>
>>
>>
>> --
>> This message was sent by Atlassian JIRA
>> (v6.3.15#6346)
>>
>
>

[jira] [Created] (IGNITE-4997) Make sure dynamic schema state is re-created properly after client re-connect

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-4997:
---

 Summary: Make sure dynamic schema state is re-created properly 
after client re-connect
 Key: IGNITE-4997
 URL: https://issues.apache.org/jira/browse/IGNITE-4997
 Project: Ignite
  Issue Type: Task
  Components: SQL
Affects Versions: 2.0
Reporter: Vladimir Ozerov
Assignee: Alexander Paschenko






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

IGNITE-4052 ready for review

[Вадим Опольский]

Hi, Nikolay!

I've made the following improvements (
https://github.com/apache/ignite/pull/1783 ):

1) Moved framework builder related code to separate method, so it make code
cleaner and framework testable.

2) Call framework builder method from test to test role and user.

3) Added validating mesos role according with mesos role documentation
http://mesos.apache.org/documentation/latest/roles/

4) Still using setEnv method because in IgniteFramework we cannot override
static method and make mock static methods (except powermock, but it would
require 3 extra dependencies in the module).

P.S. setEnv method designed to worked both on Linux and Windows, and it
does not left variable in system environment  after testing.

Vadim Opolski


2017-04-14 12:07 GMT+03:00 Nikolay Tikhonov (JIRA) :

>
> [ https://issues.apache.org/jira/browse/IGNITE-4052?page=
> com.atlassian.jira.plugin.system.issuetabpanels:comment-
> tabpanel=15968814#comment-15968814 ]
>
> Nikolay Tikhonov commented on IGNITE-4052:
> --
>
> [~javaller]
> It meens that lets remove #setEnv mathod and will create mock in test
> which will override {{getUser}} and {{getRole}} methods. Also how do you
> think might be need to add validation for role? Which valid set of values
> for this property?
>
> > Add ability to set up users for MESOS
> > -
> >
> > Key: IGNITE-4052
> > URL: https://issues.apache.org/jira/browse/IGNITE-4052
> > Project: Ignite
> >  Issue Type: Improvement
> >  Components: general
> >Affects Versions: 1.7
> >Reporter: Nikolay Tikhonov
> >Assignee: Vadim Opolski
> >Priority: Trivial
> >
> > In current implementation Ignite Mesos Framework connects to MESOS
> cluster via current user. Need to add ability to configure this parameters
> via system env properties. Also need to add properties for mesos role.
> > See org/apache/ignite/mesos/IgniteFramework.java:537
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.15#6346)
>

Re: question: How data are stored in IgniteCache?

[Vyacheslav Daradur]

>> When you transfer an object over the wire or put it into a persistent
store (withKeepBinary property enabled) then only the byte array is used.
If we put objects into persistent store and withKeepBinary property is
DISABLED, in wich form it will be stored?

2017-04-16 18:29 GMT+03:00 Denis Magda :

> 1. When you transfer an object over the wire or put it into a persistent
> store (withKeepBinary property enabled) then only the byte array is used.
> 2. No
>
> —
> Denis
>
> > On Apr 15, 2017, at 1:51 PM, Vyacheslav Daradur 
> wrote:
> >
> >>> If we use a cache which is configured to use binary mashaller:
> >>> 1. In which cases a placed in the cache object (serialized object)
> won't
> > be wrapped?
> >>> 2. In which cases a placed in the cache object won't be serialized (to
> > byte array) and will be stored as is?
> >
> > I meant: Are there any cases of described (1,2) situations.
> > If yes, which cases?
> >
> > 2017-04-15 23:44 GMT+03:00 Vyacheslav Daradur :
> >
> >> If we use a cache which is configured to use binary mashaller:
> >> 1. In which cases a placed in the cache object (serialized object) won't
> >> be wrapped?
> >> 2. In which cases a placed in the cache object won't be serialized (to
> >> byte array) and will be stored as is?
> >>
> >> 2017-04-14 20:03 GMT+03:00 Denis Magda :
> >>
> >>> If a serialized object is stored in an on-heap cache then it will be
> >>> wrapped by BinaryObjectImp but if it’s an off-heap cache then
> >>> BinaryObjectOffHeapImpl is used instead.
> >>>
> >>> —
> >>> Denis
> >>>
>  On Apr 14, 2017, at 12:32 AM, Vyacheslav Daradur  >
> >>> wrote:
> 
> >> when it’s stored in memory in a specific cache partition
>  Does that mean that any serialized object is always stored IN MEMORY
> as
> >>> is
>  wrapped by BinaryObjectImpl?
> 
> 
> 
>  2017-04-14 3:42 GMT+03:00 Denis Magda :
> 
> > A serialized object is always wrapped by BinaryObjectImpl when it’s
> >>> stored
> > in memory in a specific cache partition or you access it from your
> > application in a form of BinaryObject. However, when you transfer the
> > object over the wire or put it into a persistent store
> (withKeepBinary
> > property enabled) then only the byte array is used.
> >
> > —
> > Denis
> >
> >> On Apr 13, 2017, at 12:21 AM, Vyacheslav Daradur <
> daradu...@gmail.com
> 
> > wrote:
> >>
> >> Denis, thank you for answers.
> >>
> >> I meant another.
> >>
> >> For example:
> >> Cache queries use a BinaryObjectImpl and a withKeepBinary-mode use
> >>> it, so
> >> looks like all actions on serialized object are make via a
> > BinaryObjectImpl.
> >>
> >> Does a serialized object always is stored as BinaryObjectImpl or it
> >>> will
> > be
> >> wrapped on demand?
> >>
> >> 2017-04-12 22:34 GMT+03:00 Denis Magda :
> >>
> >>> A Java wrapper around an actual binary byte array with some
> >>> additional
> >>> fields and methods to work with the serialized data.
> >>>
> >>> —
> >>> Denis
> >>>
>  On Apr 12, 2017, at 8:33 AM, Vyacheslav Daradur <
> >>> daradu...@gmail.com>
> >>> wrote:
> 
>  In what cases BinaryObjecImpl is used?
> 
>  2017-04-12 18:08 GMT+03:00 Denis Magda :
> 
> > Hi,
> >
> > A cache entry is always stored in a binary format (byte array)
> in a
> >>> cache.
> > Even when you transfer an entry from one node to another, as a
> >>> result
> > of
> > cache.put(…), operation the entry will be serialized into the
> >>> binary
> >>> format
> > and transferred over the wire.
> >
> > —
> > Denis
> >
> >> On Apr 12, 2017, at 1:11 AM, Vyacheslav Daradur <
> >>> daradu...@gmail.com
> >>
> > wrote:
> >>
> >> Hello Igniters!
> >>
> >> I have one conceptual question:
> >>
> >> When we put an object in IgniteCache, how it is stored?
> >>
> >> As I understand, after marshalling we have an array of bytes,
> >> 1) in a local node it is wrapped in BinaryObjectImpl and stored
> in
> >>> memory
> >> 2) it is sent to remote node as byte array where it will be
> >>> wrapped
> > in
> >> BinaryObjectImpl and be stored in memory
> >>
> >> --
> >> Best Regards, Vyacheslav
> >
> >
> 
> 
>  --
>  Best Regards, Vyacheslav
> >>>
> >>>
> >>
> >>
> >> --
> >> Best Regards, Vyacheslav
> >
> >
> 
> 
>  --
>  Best Regards, Vyacheslav
> >>>
> >>>
> >>
> >>
> >> --
> >> Best Regards, Vyacheslav
> >>
> >
> >
> >
> > --
> > Best 

Re: Memory policies examples fails

[Sergey Chugunov]

Denis,

It looks like partition exchange mechanism allocates a lot of pages right
after the cache is started.
I'm trying to figure out what's going on.

Thanks,
Sergey.

On Mon, Apr 17, 2017 at 12:05 AM, Denis Magda  wrote:

> Igniters,
>
> I’ve created a simple example to demonstrate how to set up various
> policies:
> https://github.com/apache/ignite/blob/master/examples/
> src/main/java/org/apache/ignite/examples/datagrid/
> MemoryPoliciesExample.java
>
> However, this tiny example fails with
>
> Caused by: class org.apache.ignite.internal.mem.OutOfMemoryException: null
> at org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.
> allocatePage(PageMemoryNoStoreImpl.java:255)
> at org.apache.ignite.internal.processors.cache.
> IgniteCacheOffheapManagerImpl.allocateForTree(
> IgniteCacheOffheapManagerImpl.java:635)
> at org.apache.ignite.internal.processors.cache.
> IgniteCacheOffheapManagerImpl.createCacheDataStore0(
> IgniteCacheOffheapManagerImpl.java:727)
> at org.apache.ignite.internal.processors.cache.
> IgniteCacheOffheapManagerImpl.createCacheDataStore(
> IgniteCacheOffheapManagerImpl.java:711)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.GridDhtLocalPartition.(GridDhtLocalPartition.java:162)
> at org.apache.ignite.internal.processors.cache.distributed.dht.
> GridDhtPartitionTopologyImpl.createPartition(GridDhtPartitionTopologyImpl.
> java:718)
> at org.apache.ignite.internal.processors.cache.distributed.dht.
> GridDhtPartitionTopologyImpl.initPartitions0(GridDhtPartitionTopologyImpl.
> java:405)
> at org.apache.ignite.internal.processors.cache.distributed.dht.
> GridDhtPartitionTopologyImpl.beforeExchange(GridDhtPartitionTopologyImpl.
> java:569)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(
> GridDhtPartitionsExchangeFuture.java:844)
> at org.apache.ignite.internal.processors.cache.distributed.
> dht.preloader.GridDhtPartitionsExchangeFuture.init(
> GridDhtPartitionsExchangeFuture.java:573)
> at org.apache.ignite.internal.processors.cache.
> GridCachePartitionExchangeManager$ExchangeWorker.body(
> GridCachePartitionExchangeManager.java:1800)
> ... 2 more
>
>
>
> What I’m doing wrong? Alex G., could you take a look? Probably I’m missing
> something due to a lack of documentation.
>
> —
> Denis
>
>

Re: NULL strings in REST-HTTP

[Vladimir Ozerov]

+1

On Mon, Apr 17, 2017 at 1:42 PM, Sergey Kozlov  wrote:

> Hi Alexey.
>
> I vote for your fix. Definitely it's wrong to return the empty string
> instead of null.
>
> On Mon, Apr 17, 2017 at 1:30 PM, Alexey Kuznetsov 
> wrote:
>
> > Hi,
> >
> > I would like to change serialization of null strings in our
> > ignite-rest-http module.
> >
> > Current serialization null -> "" (empty string).
> > Proposed serialization null -> null.
> >
> > This will make things more straightforward.
> >
> > Thoughts?
> >
> > --
> > Alexey Kuznetsov
> >
>
>
>
> --
> Sergey Kozlov
> GridGain Systems
> www.gridgain.com
>

Re: Page Memory behavior with default memory policy

[Alexey Goncharuk]

Agree, I will rename it if there are no objections.

2017-04-17 14:31 GMT+03:00 Vladimir Ozerov :

> Guys,
>
> I see that we throw our own "OutOfMemoryException" in this case. I think we
> should rename it to "IgniteOufOfMemoryException" to avoid confusion with
> JDK's "OutOfMemoryError". E.g. I looked at some thread dumps recently and
> was pretty sure that there was real OOME, while in reality it was Ignite's
> exception.
>
> On Mon, Apr 17, 2017 at 12:42 PM, Sergey Chugunov <
> sergey.chugu...@gmail.com
> > wrote:
>
> > Hello Denis,
> >
> > There is a small piece of documentation in *MemoryConfiguration *class,
> > although I think it should be moved to *MemoryPolicyConfiguration *and
> > detailed a lot.
> >
> > I'm going to write some documentation soon about validation rules that
> are
> > applied to memory policies configuration (there are some restrictions
> about
> > reserved name, default memory policy name and so on). I'll let you know
> > when it is done.
> >
> > About your question: MemoryPolicy offheap regions were never supposed to
> be
> > expandable. If user uses default MemoryPolicy (which in fact doesn't have
> > any eviction algorithm configured) he/she will get OutOfMemory error when
> > default size is exhausted.
> >
> > This OOM behavior was left intentionally to be consistent with the
> previous
> > behavior as configuring some default eviction policy on default
> > MemoryPolicy may lead to data loss and it is better to communicate wrong
> > memory configuration to user even with OOM than silently drop some cached
> > values.
> >
> > Thanks,
> > Sergey.
> >
> > On Mon, Apr 17, 2017 at 12:01 AM, Denis Magda  wrote:
> >
> > > Guys,
> > >
> > > I’ve been working on the documentation for page memory [1]. The doc is
> > > still in progress and it will be more profound with code snippets and
> > > pictures int the end, so don’t pay to the fact that it mostly consists
> of
> > > dry text only.
> > >
> > > So, the question is different. Presently, javadoc part is useless (and
> > > will be written by me as well) and it’s not clear what’s the default
> > > behavior of the page memory that has only one memory policy (default)
> > that
> > > instantiates a single memory region. Looking at the code I see that the
> > > default police creates 1 GB region that should be expandable because
> the
> > > policy doesn’t define an eviction algorithm. It means that if my app
> goes
> > > beyond 1 GB then the page memory will take more memory from an OS. Is
> my
> > > understanding correct?
> > >
> > > [1] https://apacheignite.readme.io/docs/page-memory <
> > > https://apacheignite.readme.io/docs/page-memory>
> > >
> > > —
> > > Denis
> >
>

Re: NULL strings in REST-HTTP

[Sergey Kozlov]

Hi Alexey.

I vote for your fix. Definitely it's wrong to return the empty string
instead of null.

On Mon, Apr 17, 2017 at 1:30 PM, Alexey Kuznetsov 
wrote:

> Hi,
>
> I would like to change serialization of null strings in our
> ignite-rest-http module.
>
> Current serialization null -> "" (empty string).
> Proposed serialization null -> null.
>
> This will make things more straightforward.
>
> Thoughts?
>
> --
> Alexey Kuznetsov
>



-- 
Sergey Kozlov
GridGain Systems
www.gridgain.com

NULL strings in REST-HTTP

[Alexey Kuznetsov]

Hi,

I would like to change serialization of null strings in our
ignite-rest-http module.

Current serialization null -> "" (empty string).
Proposed serialization null -> null.

This will make things more straightforward.

Thoughts?

-- 
Alexey Kuznetsov

Re: Page Memory behavior with default memory policy

[Sergey Chugunov]

Hello Denis,

There is a small piece of documentation in *MemoryConfiguration *class,
although I think it should be moved to *MemoryPolicyConfiguration *and
detailed a lot.

I'm going to write some documentation soon about validation rules that are
applied to memory policies configuration (there are some restrictions about
reserved name, default memory policy name and so on). I'll let you know
when it is done.

About your question: MemoryPolicy offheap regions were never supposed to be
expandable. If user uses default MemoryPolicy (which in fact doesn't have
any eviction algorithm configured) he/she will get OutOfMemory error when
default size is exhausted.

This OOM behavior was left intentionally to be consistent with the previous
behavior as configuring some default eviction policy on default
MemoryPolicy may lead to data loss and it is better to communicate wrong
memory configuration to user even with OOM than silently drop some cached
values.

Thanks,
Sergey.

On Mon, Apr 17, 2017 at 12:01 AM, Denis Magda  wrote:

> Guys,
>
> I’ve been working on the documentation for page memory [1]. The doc is
> still in progress and it will be more profound with code snippets and
> pictures int the end, so don’t pay to the fact that it mostly consists of
> dry text only.
>
> So, the question is different. Presently, javadoc part is useless (and
> will be written by me as well) and it’s not clear what’s the default
> behavior of the page memory that has only one memory policy (default) that
> instantiates a single memory region. Looking at the code I see that the
> default police creates 1 GB region that should be expandable because the
> policy doesn’t define an eviction algorithm. It means that if my app goes
> beyond 1 GB then the page memory will take more memory from an OS. Is my
> understanding correct?
>
> [1] https://apacheignite.readme.io/docs/page-memory <
> https://apacheignite.readme.io/docs/page-memory>
>
> —
> Denis

Re: TouchedExpiryPolicy works incorrect in some cases IGNITE-4401

[ALEKSEY KUZNETSOV]

Why do u think so.
First of all, the output above is not correct. After 3 iteration key-value
API strats to return empty value.

Every 5 seconds(iteration sleep time) repository.getAttributes("1").size()
is got called. Which makes an entry "touch" and the entry wont be expired
for as long as 10 seconds.

Expiry policy says:

An {@link ExpiryPolicy} that defines the expiry {@link Duration}
* of a Cache Entry based on when it was last touched. A touch includes
** creation, update or **access**.*


пт, 14 апр. 2017 г. в 18:42, Denis Magda :

> The iteration happens multiple time which means that the key-value API had
> to return an empty result set on second or third iteration. But this never
> happens.
>
> In any case, do you want to find a root of the issue and fix it?
> Otherwise, we can update the description and wait while someone else fixes
> it.
>
> —
> Denis
>
> > On Apr 14, 2017, at 1:33 AM, ALEKSEY KUZNETSOV 
> wrote:
> >
> > Because expiry time is 10 seconds, while loop iterates every 5 seconds
> >
> > пт, 14 апр. 2017 г. в 11:32, ALEKSEY KUZNETSOV  >:
> >
> >> No, the bug is in SQL query, not key-value storage.
> >>
> >> пт, 14 апр. 2017 г. в 11:11, Vladislav Pyatkov :
> >>
> >>> Denis, Aleksey,
> >>>
> >>> It is correct, remember I have already said something like[1].
> >>> I have no idea, why this happened in this case with SQL.
> >>>
> >>> [1]:
> >>>
> >>>
> http://apache-ignite-developers.2346864.n4.nabble.com/TouchedExpiryPolicy-works-incorrect-in-some-cases-IGNITE-4401-td16349.html#a16356
> >>>
> >>> On Fri, Apr 14, 2017 at 4:29 AM, Denis Magda 
> wrote:
> >>>
>  I could reproduce the issue and this should be what Denis K. meant by
>  saying “expiration policy works incorrectly”.
> 
>  If you remove the expiration policy from the caches' configuration
> then
>  the issue disappears. In general, SQL engine processes an expiration
> >>> event
>  properly because the SQL queries return an empty result set as
> expected
> >>> but
>  something doesn’t work well with key-value operations.
> 
>  *Denis K*, *Vlad P.*, as creators of the ticket please confirm that
> this
>  is the case.
> 
>  Please keep debugging this and switch to the latest Ignite version.
> 
>  —
>  Denis
> 
> 
> > On Apr 13, 2017, at 4:22 AM, ALEKSEY KUZNETSOV <
> >>> alkuznetsov...@gmail.com>
>  wrote:
> >
> > any feedback?
> >
> > чт, 13 апр. 2017 г. в 11:51, ALEKSEY KUZNETSOV <
> >>> alkuznetsov...@gmail.com
> > :
> >
> >> You should run ExpiryPolicyTest. The output should contain strings
> >>> like
> >> contains? new AffinityKey("1", "1"): and contains?2 new
>  AffinityKey("1", "
> >> 1"): and empty cursor? =
> >> If you look at them you will see, that cache contains affinity key
> >>> new
> >> AffinityKey("1", "1") whereas cursor is empty(on second iteration).
> >>> From
> >> this output you can conclude SQL query returns icorrect data(empty
>  value)
> >>
> >>
> >> чт, 13 апр. 2017 г. в 3:42, Denis Magda :
> >>
> >>> Bluntly speaking I have no idea where to look and what to expect.
> >>> This
>  is
> >>> output of the test execution of my machine:
> >>>
> >>> SQL res: [[1], [d]]
> >>> 2
> >>> Op consume: 303
> >>> Value: org.ignite.test.EDU@22db8f4
> >>> SQL res: []
> >>> 0
> >>> Op consume: 9
> >>> Value: org.ignite.test.EDU@29caf222
> >>> SQL res: []
> >>> 0
> >>> Op consume: 15
> >>> Value: org.ignite.test.EDU@7cd1ac19
> >>> SQL res: []
> >>> 0
> >>> Op consume: 5
> >>>
> >>> Please be more specific, there are too many files in the code.
> >>>
> >>> —
> >>> Denis
> >>>
>  On Apr 12, 2017, at 4:50 AM, ALEKSEY KUZNETSOV <
> >>> alkuznetsov...@gmail.com> wrote:
> 
>  So what do u think about the issue ?
> 
>  ср, 12 апр. 2017 г. в 10:42, ALEKSEY KUZNETSOV <
> >>> alkuznetsov...@gmail.com>:
> 
> > I have already attached simlified version. Shall i simplify it
> >>> more ?
> >
> > вт, 11 апр. 2017 г. в 19:28, Denis Magda :
> >
> > Can you attach the simplified version? Just want to avoid any
> side
> >>> effects.
> >
> > —
> > Denis
> >
> >> On Apr 11, 2017, at 9:14 AM, ALEKSEY KUZNETSOV <
> >>> alkuznetsov...@gmail.com>
> > wrote:
> >>
> >> I took it from
> https://issues.apache.org/jira/browse/IGNITE-4401
> >>> <
> > https://issues.apache.org/jira/browse/IGNITE-4401> and
> >>> simplified .
> >>> See
> > in attached
> >>
> >>
> >> вт, 11 апр. 2017 г. в 19:03, Denis Magda    > 

[GitHub] ignite pull request #1806: Ignite 4955

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/1806


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-4996) Queries: show Node selection modal in case if user execute SCAN on local cache

[Pavel Konstantinov (JIRA)]

Pavel Konstantinov created IGNITE-4996:
--

 Summary: Queries: show Node selection modal in case if user 
execute SCAN on local cache
 Key: IGNITE-4996
 URL: https://issues.apache.org/jira/browse/IGNITE-4996
 Project: Ignite
  Issue Type: Bug
Reporter: Pavel Konstantinov
Priority: Minor


Also we need to filter nodes list and show only nodes where selected cache is 
configured



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)