Re: Adding sqlline tool to Apache Ignite project

[dsetrakyan]

I like ignitesql.

⁣D.​

On Oct 6, 2017, 4:49 PM, at 4:49 PM, Vladimir Ozerov  
wrote:
>Denis,
>
>Setting default host to 127.0.0.1 is bad idea, because it mean that in
>practice users would have to change the script always. Instead, we
>should
>accept host name as argument. This is perfectly fine from usability
>perspective, most tools work this way (i.e. throw error when started
>without arguments).
>
>Also IMO "ignitedb" is misleading name. Users would like think that it
>is a
>kind of script to start database, rather than to connect to it. We
>should
>think on other names. E.g. "ignitesql".
>
>On Fri, Oct 6, 2017 at 5:23 PM, Sergey Kozlov 
>wrote:
>
>> Denis
>>
>> The link below has included sqlline. Please take a look:
>> https://ci.ignite.apache.org/viewLog.html?buildId=875441;
>> buildTypeId=IgniteRelease_XxxFromMirrorIgniteRelease3Pre
>> pareVote=artifacts#!1rrb2,-wpvx2aopzexz
>>
>> On Thu, Oct 5, 2017 at 7:48 PM, Denis Magda 
>wrote:
>>
>> > Here is the original ticket [1]. Ilya, closed the one created by
>you as a
>> > duplicate.
>> >
>> > In addition to the tool’s jar inclusion in Ignite’s binary releases
>let’s
>> > create a shell script to simplify the connectivity phase:
>> >
>> >- name the script as ignitedb.sh for Unix and ignitedb.bat for
>> Windows.
>> >-
>> >- the script uses the following connection string by default:
>.sqlline
>> >-d org.apache.ignite.IgniteJdbcThinDriver --color=true
>--verbose=true
>> >--showWarnings=true --showNestedErrs=true -u jdbc:ignite:
>> >thin://127.0.0.1/
>> >
>> >
>> >
>> >- make up parameters list to adjust Ignite specific part of the
>> >connection string: Ignite IP and port, streaming mode, etc. The
>full
>> list
>> >of supported parameters is here: https://apacheignite-
>> >sql.readme.io/docs/jdbc-driver#jdbc-thin-driver
>> >
>> >
>> >
>> >
>> >
>> > [1] https://issues.apache.org/jira/browse/IGNITE-5608
>> >
>> > —
>> > Denis
>> >
>> > On Oct 5, 2017, at 9:02 AM, Sergey Kozlov 
>wrote:
>> >
>> > Dmitriy, Denis
>> >
>> > We're in progress to add sqlline in upcoming 2.3
>> >
>> > On Thu, Oct 5, 2017 at 5:30 PM, Dmitriy Setrakyan
>> > > wrote:
>> > Would be nice to get it in 2.3. This is critical functionality for
>our
>> > users and 2.4 seems too far to give anyone comfort.
>> >
>> > On Thu, Oct 5, 2017 at 11:33 AM, Ilya Suntsov
>
>> > wrote:
>> >
>> > > Guys,
>> > >
>> > > I've created the ticket for 2.4 release:
>> > > https://issues.apache.org/jira/browse/IGNITE-6561
>> > >
>> > > 2017-08-30 22:21 GMT+03:00 Julian Hyde :
>> > >
>> > > > Denis,
>> > > >
>> > > > I’m glad you’re thinking of using SQLLine. Under the BSD
>license, you
>> > > > don’t need my permission to distribute, but I grant that
>permission.
>> > > >
>> > > > Drill, Phoenix and Calcite already distribute SQLLine, so
>Ignite is
>> in
>> > > > good company.
>> > > >
>> > > > If you need extensions, please discuss on the dev list, or open
>a
>> > GitHub
>> > > > case or pull request. SQLLine operates in the usual way for a
>GitHub
>> > > > project. It’s unlikely that you’ll need Ignite-specific
>extensions —
>> > > > SQLLine just exposes what comes through the JDBC driver — but
>we can
>> > > > discuss if the need arises. The Hive project forked SQLLine
>into its
>> > own
>> > > > Beeline module and I’d like to avoid a repeat of that.
>> > > >
>> > > > Julian
>> > > >
>> > > > > On Aug 29, 2017, at 6:35 PM, Denis Magda 
>> wrote:
>> > > > >
>> > > > > Igniters,
>> > > > >
>> > > > > Let me introduce Julian Hyde [1], creator of SQLLine tool and
>our
>> > > Apache
>> > > > mate,
>> > > > >
>> > > > > Julian,
>> > > > >
>> > > > > Please grant that Apache Ignite community a permission to
>include
>> > > > SQLLine [2] it in every Ignite deliverable (source, binary).
>It’s
>> > planned
>> > > > to suggest the tool as a default command line SQL utility for
>Ignite
>> > > > clusters. SQLLite and Ignite usage will also be documented on
>> Ignite’s
>> > > > technical documentation.
>> > > > >
>> > > > > [1] https://people.apache.org/~jhyde/ <
>> https://people.apache.org/~jh
>> > > yde/
>> > > > >
>> > > > > [2] https://github.com/julianhyde/sqlline
>> > > > julianhyde/sqlline>
>> > > > >
>> > > > > —
>> > > > > Denis
>> > > > >
>> > > > >> On Aug 25, 2017, at 9:17 AM, Denis Magda > >  > > > > dma...@apache.org>> wrote:
>> > > > >>
>> > > > >> Hi Ilya,
>> > > > >>
>> > > > >> Thanks for the clarification! Referring to the page shared
>by you
>> > [1]
>> > > > if we need to get author’s consent in a written form:
>> > > > >>
>> > > > >> A permissive license similar to the BSD 2-Clause License,
>but
>> with a
>> > > > 3rd clause that prohibits others from using the name of the

Re: [DISCUSS] Ignite Update Checker

[dsetrakyan]

Why would we need googleads for youtube? Any chance to disable it?

⁣D.​

On Sep 30, 2017, 10:25 AM, at 10:25 AM, Prachi Garg  wrote:
>Yes, I see in dev tools that calls to googleads and doubleclick are
>made
>from Youtube.
>
>On Fri, Sep 29, 2017 at 4:33 PM, Denis Magda  wrote:
>
>> I also did a global search on the Ignite website, but didn't find
>anything
>> for googleads or doubleclick.
>>
>>
>> Could you remove and add screencasts block temporary on your local
>> deployment to see if the calls to commercial scripts reported by Cos
>appear
>> in your Chrome dev toolkit?
>>
>> —
>> Denis
>>
>>
>> On Sep 29, 2017, at 3:56 PM, Prachi Garg  wrote:
>>
>> We use the following scripts -
>>
>> https://platform.twitter.com/widgets.js - used on homepage to display
>> tweets
>> https://static.addtoany.com/menu/page.js - used on events page for
>social
>> media sharing
>> https://www.google-analytics.com/analytics.js
>>
>> I also did a global search on the Ignite website, but didn't find
>anything
>> for googleads or doubleclick.
>>
>> -Prachi
>>
>>
>> On Fri, Sep 29, 2017 at 11:03 AM, Denis Magda 
>wrote:
>>
>>> That’s definitely worthwhile checking. Prachi, as the one who
>embedded
>>> the screencast, would you check the theory?
>>>
>>> —
>>> Denis
>>>
>>> On Sep 28, 2017, at 11:50 PM, Alexey Kuznetsov
>
>>> wrote:
>>>
>>> Cos, Denis.
>>>
>>> I think it is because we have embedded videos (on YouTube).
>>> Mauricio or Denis, please check my idea.
>>>
>>> On Fri, Sep 29, 2017 at 8:02 AM, Konstantin Boudnik 
>>> wrote:
>>>
 Sorry guys - I neglected the fact that our lists don't permit
 attachments. I have put the screenshot to an external server [1]

 [1] https://imgur.com/a/p9FJ9

 Thank you!
 --
   With regards,
 Konstantin (Cos) Boudnik
 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

 Disclaimer: Opinions expressed in this email are those of the
>author,
 and do not necessarily represent the views of any company the
>author
 might be affiliated with at the moment of writing.


 On Thu, Sep 28, 2017 at 1:37 PM, Denis Magda 
>wrote:
 > Cos,
 >
 > The screenshot was not attached. Could you share it some other
>way
 (google drive, etc.)? I’ve never seen any commercial on the site.
 >
 > —
 > Denis
 >
 >> On Sep 28, 2017, at 7:23 AM, Konstantin Boudnik 
 wrote:
 >>
 >> I don't see an issue with node version either.
 >>
 >> Just one more, and it might be slightly irrelevant, issue
>though... I
 looked at the Ignite's site and found the following ads and
>trackers (that
 are indeed getting disabled by my browser).
 >> Why are googleads or doubleclick are permitted?
 >>
 >>
 >>
 >> Thanks,
 >>   Cos
 >>
 >>
 >> --
 >>   With regards,
 >> Konstantin (Cos) Boudnik
 >> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
 >>
 >> Disclaimer: Opinions expressed in this email are those of the
>author,
 and do not necessarily represent the views of any company the
>author might
 be affiliated with at the moment of writing.
 >>
 >> On Tue, Sep 26, 2017 at 3:21 PM, Dmitriy Setrakyan <
 dsetrak...@apache.org > wrote:
 >> On Tue, Sep 26, 2017 at 6:20 AM, Vladimir Ozerov <
 voze...@gridgain.com >
 >> wrote:
 >>
 >> > Folks,
 >> >
 >> > Can we add version of current node to web request? This way we
>will
 better
 >> > understand version distribution, what might help us with
>certain API
 >> > update/deprecate decisions
 >> > E.g. http://ignite.apache.org/latest.cgi=2.2.0 <
 http://ignite.apache.org/latest.cgi=2.2.0>
 >>
 >>
 >> Vladimir, I personally do not see a problem with that, as
>sending the
 >> current version to the update checker seems very innocent to me.
>At
 the
 >> same time, it will allow us to examine the usage of each release
>and
 make
 >> decisions about dropping backward compatibility or spotting bugs
>for a
 >> certain release.
 >>
 >> Cos, Raul, any thoughts?
 >>
 >>
 >> >
 >> >
 >> > Vladimir.
 >> >
 >> > On Fri, Sep 8, 2017 at 7:06 AM, Dmitriy Setrakyan <
 dsetrak...@apache.org >
 >> > wrote:
 >> >
 >> > > I think it is safe to assume at this point that everyone is
>in
 general
 >> > > agreement, since there are no active objections.
 >> > >
 >> > > I have filed a ticket for the 2.3 release. Let's try to make
>it
 happen:
 >> > > https://issues.apache.org/jira/browse/IGNITE-6305 <
 https://issues.apache.org/jira/browse/IGNITE-6305>
 >> > >
 >> > > D.
 >> > >
 >> > > On Sat, Aug 26, 

Re: Logical Cache Documented

[dsetrakyan]

Why not? Obviously compression would have to be enabled per group, not per 
cache.

⁣D.​

On Sep 29, 2017, 10:50 PM, at 10:50 PM, Vladimir Ozerov  
wrote:
>And it will continue hitting us in future. For example, when data
>compression is implemented, for logical caches compression rate will be
>poor, as it would be impossbile to build efficient dictionaries in
>mixed
>data pages.
>
>On Sat, Sep 30, 2017 at 8:48 AM, Vladimir Ozerov 
>wrote:
>
>> Folks,
>>
>> Honesly, to me logical caches appears to be a dirty shortcut to
>mitigate
>> some inefficient internal implementation. Why can't we merge
>partition maps
>> in runtime? This should not be a problem for context-independent
>affinity
>> functions (e.g. RendezvousAffinityFunction). From user perspective
>logic
>> caches feature is:
>> 1) Bad API. One cannot define group configuration. All you can do is
>to
>> define group name on cache lavel and hope that nobody started another
>cache
>> in the same group with different configuration before.
>> 2) Performance impact for scans, as you have to iterate over mixed
>data.
>>
>> Couldn't we fix partition map problem without cache groups?
>>
>> On Sat, Sep 30, 2017 at 2:35 AM, Denis Magda 
>wrote:
>>
>>> Guys,
>>>
>>> Another question. Does this capability enabled by default? If yes,
>how do
>>> we decide which group a cache goes to?
>>>
>>> —
>>> Denis
>>>
>>> > On Sep 29, 2017, at 3:58 PM, Denis Magda 
>wrote:
>>> >
>>> > Igniters,
>>> >
>>> > I’ve put on paper the feature from the subj:
>>> > https://apacheignite.readme.io/docs/logical-caches <
>>> https://apacheignite.readme.io/docs/logical-caches>
>>> >
>>> > Sam, will appreciate if you read through it and confirm I
>explained the
>>> topic 100% technically correct.
>>> >
>>> > However, are there any negative impacts of having logical caches?
>This
>>> page has “Possible Impacts” section unfilled:
>>> > https://cwiki.apache.org/confluence/display/IGNITE/Logical+Caches
><
>>> https://cwiki.apache.org/confluence/display/IGNITE/Logical+Caches>
>>> >
>>> > —
>>> > Denis
>>>
>>>
>>

Re: Logical Cache Documented

[dsetrakyan]

Vova, cache groups almost never need to be touched by users. They should be 
configured automatically. To my knowledge,  by default all caches fall into the 
default group (please confirm).

As far as scans, the performance should not be affected, as we now store  
entries in B+trees and the data is sorted by cache. We only iterate over the 
data in a specific cache.

⁣D.​

On Sep 29, 2017, 10:48 PM, at 10:48 PM, Vladimir Ozerov  
wrote:
>Folks,
>
>Honesly, to me logical caches appears to be a dirty shortcut to
>mitigate
>some inefficient internal implementation. Why can't we merge partition
>maps
>in runtime? This should not be a problem for context-independent
>affinity
>functions (e.g. RendezvousAffinityFunction). From user perspective
>logic
>caches feature is:
>1) Bad API. One cannot define group configuration. All you can do is to
>define group name on cache lavel and hope that nobody started another
>cache
>in the same group with different configuration before.
>2) Performance impact for scans, as you have to iterate over mixed
>data.
>
>Couldn't we fix partition map problem without cache groups?
>
>On Sat, Sep 30, 2017 at 2:35 AM, Denis Magda  wrote:
>
>> Guys,
>>
>> Another question. Does this capability enabled by default? If yes,
>how do
>> we decide which group a cache goes to?
>>
>> —
>> Denis
>>
>> > On Sep 29, 2017, at 3:58 PM, Denis Magda  wrote:
>> >
>> > Igniters,
>> >
>> > I’ve put on paper the feature from the subj:
>> > https://apacheignite.readme.io/docs/logical-caches <
>> https://apacheignite.readme.io/docs/logical-caches>
>> >
>> > Sam, will appreciate if you read through it and confirm I explained
>the
>> topic 100% technically correct.
>> >
>> > However, are there any negative impacts of having logical caches?
>This
>> page has “Possible Impacts” section unfilled:
>> > https://cwiki.apache.org/confluence/display/IGNITE/Logical+Caches <
>> https://cwiki.apache.org/confluence/display/IGNITE/Logical+Caches>
>> >
>> > —
>> > Denis
>>
>>

Re: Durable Memory and Ignite Persistence Tuning

[dsetrakyan]

Cos, great point!

I think the Ignite community should start load testing with default settings, 
without any config changes. This should open a lot of holes.

⁣D.​

On Sep 1, 2017, 9:57 PM, at 9:57 PM, Konstantin Boudnik  wrote:
>Just to spice it up: in my experience, having a few hundred parameters
>one can
>tweak (I am making up the number here) is a tough UX call. In JVM, we
>had a
>team that worked on heuristics that would auto-tune a bunch of things
>during
>the VM startup. Hence, providing the best user experience in most
>cases.
>
>Do you think something like this is feasible in case of persistence
>functionality? Documenting it first is a great first step, but perhaps
>wrapping this knowledge into some soft of code, would be even better.
>
>I do have an anecdotal evidence where an experienced Ignite developer
>tried to implement a message processing system with Ignite 2.0. After a
>week
>of getting nowhere performance wise, he dropped it and implemented
>something
>custom with Java and nothing else. Take it or leave it: that's why I
>called
>this "anecdotal" in the first place.
>
>Speaking strictly for myself: when I come across blog posts about
>tuning of
>Apache Kudu or Apache Impala the skin on the back of head starts
>crawling. I
>imagine 95% of the potential users of these applications would turn
>away right
>at that point. If the system doesn't work well enough out of the box -
>screw
>it, there's 355 other alternatives available in the FOSS market.
>
>Thoughts?
>  Cos
>
>On Fri, Sep 01, 2017 at 11:08AM, Denis Magda wrote:
>> Igniters,
>>
>> I see a lot of complains regarding the performance of the subj on the
>user
>> list. At the same time, I do believe that in most scenarios it’s a
>lack of
>> knowledge that we keep in secret.
>>
>> It's time to document Durable Memory and its Native Persistence
>tuning
>> parameters. Let's start doing this for Linux based deployments first.
>Here
>> is what we have for now (which is almost nothing):
>> https://apacheignite.readme.io/docs/durable-memory-tuning
>> 
>>
>> Ideally, at some point we have to come up with doc like this:
>>
>https://access.redhat.com/sites/default/files/attachments/deploying-oracle-12c-on-rhel6_1.2_1.pdf
>>
>> Please share your expertise in a form of settings that have to be put
>on the
>> paper. We put them in JIRA and document afterwords:
>> https://issues.apache.org/jira/browse/IGNITE-6246
>> 
>>
>> —
>> Denis

Re: How a new index is built in runtime?

[dsetrakyan]

Vova, how hard is it to make it multi-threaded?

⁣D.​


On Aug 28, 2017, 10:05 AM, at 10:05 AM, Vladimir Ozerov  
wrote:
>Denis,
>
>We iterate over the whole cache and build the index entry-by-entry.
>Control
>is returned back to the user when index is ready.
>
>On Fri, Aug 18, 2017 at 9:50 AM, Yakov Zhdanov 
>wrote:
>
>> Of course, iteration should have an option to be run from more than 1
>> thread. What will really help, IMO, is ability to insert presorted
>batches
>> in a single tree operation.
>>
>> --
>> Yakov Zhdanov
>>

Re: [IGNITE-5717] improvements of MemoryPolicy default size

[dsetrakyan]

But why? We allocate the memory, so we should know when it runs out. What am i 
missing?

⁣D.​

On Aug 4, 2017, 11:55 AM, at 11:55 AM, Sergey Chugunov 
 wrote:
>I used GC and java only as an example, they are not applicable to
>Ignite
>case where we manage offheap memory.
>
>My point is that there is no easy way to implement this feature in
>Ignite,
>and more time is needed to properly design it and account for all
>risks.
>
>Thanks,
>Sergey.
>
>On Fri, Aug 4, 2017 at 12:44 PM,  wrote:
>
>> Hang on. I thought we were talking about offheap size, GC should not
>be
>> relevant. Am I wrong?
>>
>> ⁣D.​
>>
>> On Aug 4, 2017, 11:38 AM, at 11:38 AM, Sergey Chugunov <
>> sergey.chugu...@gmail.com> wrote:
>> >Do you see an obvious way of implementing it?
>> >
>> >In java there is a heap and GC working on it. And for instance, it
>is
>> >possible to make a decision to throw an OOM based on some gc
>metrics.
>> >
>> >I may be wrong but I don't see a mechanism in Ignite to use it right
>> >away
>> >for such purposes.
>> >And implementing something without thorough planning brings huge
>risk
>> >of
>> >false positives with nodes stopping when they don't have to.
>> >
>> >That's why I think it must be implemented and intensively tested as
>> >part of
>> >a separate ticket.
>> >
>> >Thanks,
>> >Sergey.
>> >
>> >On Fri, Aug 4, 2017 at 12:18 PM,  wrote:
>> >
>> >> Without #3, the #1 and #2 make little sense.
>> >>
>> >> Why is #3 so difficult?
>> >>
>> >> ⁣D.​
>> >>
>> >> On Aug 4, 2017, 10:46 AM, at 10:46 AM, Sergey Chugunov <
>> >> sergey.chugu...@gmail.com> wrote:
>> >> >Dmitriy,
>> >> >
>> >> >Last item makes perfect sense to me, one may think of it as an
>> >> >"OutOfMemoryException" in java.
>> >> >However, it looks like such feature requires considerable efforts
>to
>> >> >properly design and implement it, so I would propose to create a
>> >> >separate
>> >> >ticket and agree upon target version for it.
>> >> >
>> >> >Items #1 and #2 will be implemented under IGNITE-5717. Makes
>sense?
>> >> >
>> >> >Thanks,
>> >> >Sergey.
>> >> >
>> >> >On Thu, Aug 3, 2017 at 4:34 AM, Dmitriy Setrakyan
>> >> >
>> >> >wrote:
>> >> >
>> >> >> Here is what we should do:
>> >> >>
>> >> >>1. Pick an acceptable number. Does not matter if it is 10%
>or
>> >50%.
>> >> >>2. Print the allocated memory in *BOLD* letters into the
>log.
>> >> >>3. Make sure that Ignite server never hangs due to the low
>> >memory
>> >> >issue.
>> >> >>We should sense it and kick the node out automatically,
>again
>> >with
>> >> >a
>> >> >> *BOLD*
>> >> >>message in the log.
>> >> >>
>> >> >>  Is this possible?
>> >> >>
>> >> >> D.
>> >> >>
>> >> >> On Wed, Aug 2, 2017 at 6:09 PM, Vladimir Ozerov
>> >> >
>> >> >> wrote:
>> >> >>
>> >> >> > My proposal is 10% instead of 80%.
>> >> >> >
>> >> >> > ср, 2 авг. 2017 г. в 18:54, Denis Magda :
>> >> >> >
>> >> >> > > Vladimir, Dmitriy P.,
>> >> >> > >
>> >> >> > > Please see inline
>> >> >> > >
>> >> >> > > > On Aug 2, 2017, at 7:20 AM, Vladimir Ozerov
>> >> >
>> >> >> > > wrote:
>> >> >> > > >
>> >> >> > > > Denis,
>> >> >> > > >
>> >> >> > > > The reason is that product should not hang user's
>computer.
>> >How
>> >> >else
>> >> >> > this
>> >> >> > > > could be explained? I am developer. I start Ignite, 1
>node,
>> >2
>> >> >nodes,
>> >> >> X
>> >> >> > > > nodes, observe how they join topology. Add one key, 10
>keys,
>> >1M
>> >> >keys.
>> >> >> > > Then
>> >> >> > > > I do a bug in example and load 100M keys accidentally -
>> >restart
>> >> >the
>> >> >> > > > computer. Correct behavior is to have small "maxMemory"
>by
>> >> >default to
>> >> >> > > avoid
>> >> >> > > > that. User should get exception instead of hang. E.g.
>Java's
>> >> >"-Xmx"
>> >> >> is
>> >> >> > > > typically 25% of RAM - more adequate value, comparing to
>> >> >Ignite.
>> >> >> > > >
>> >> >> > >
>> >> >> > > Right, the developer was educated about the Java heap
>> >parameters
>> >> >and
>> >> >> > > limited the overall space preferring OOM to the laptop
>> >> >suspension. Who
>> >> >> > > knows how he got to the point that 25% RAM should be used.
>> >That
>> >> >might
>> >> >> > have
>> >> >> > > been deep knowledge about JVM or he faced several hangs
>while
>> >> >testing
>> >> >> the
>> >> >> > > application.
>> >> >> > >
>> >> >> > > Anyway, JVM creators didn’t decide to predefine the Java
>heap
>> >to
>> >> >a
>> >> >> static
>> >> >> > > value to avoid the situations like above. So should not we
>as
>> >a
>> >> >> platform.
>> >> >> > > Educate people about the Ignite memory behavior like Sun
>did
>> >for
>> >> >the
>> >> >> Java
>> >> >> > > heap but do not try to solve the lack of knowledge with the
>> >> >default
>> >> >> > static
>> >> >> > > memory size.
>> >> >> > >
>> >> >> > >
>> >> >> > > > It doesn't matter whether you use persistence or not.
>> >> 

Re: [IGNITE-5717] improvements of MemoryPolicy default size

[dsetrakyan]

Hang on. I thought we were talking about offheap size, GC should not be 
relevant. Am I wrong?

⁣D.​

On Aug 4, 2017, 11:38 AM, at 11:38 AM, Sergey Chugunov 
 wrote:
>Do you see an obvious way of implementing it?
>
>In java there is a heap and GC working on it. And for instance, it is
>possible to make a decision to throw an OOM based on some gc metrics.
>
>I may be wrong but I don't see a mechanism in Ignite to use it right
>away
>for such purposes.
>And implementing something without thorough planning brings huge risk
>of
>false positives with nodes stopping when they don't have to.
>
>That's why I think it must be implemented and intensively tested as
>part of
>a separate ticket.
>
>Thanks,
>Sergey.
>
>On Fri, Aug 4, 2017 at 12:18 PM,  wrote:
>
>> Without #3, the #1 and #2 make little sense.
>>
>> Why is #3 so difficult?
>>
>> ⁣D.​
>>
>> On Aug 4, 2017, 10:46 AM, at 10:46 AM, Sergey Chugunov <
>> sergey.chugu...@gmail.com> wrote:
>> >Dmitriy,
>> >
>> >Last item makes perfect sense to me, one may think of it as an
>> >"OutOfMemoryException" in java.
>> >However, it looks like such feature requires considerable efforts to
>> >properly design and implement it, so I would propose to create a
>> >separate
>> >ticket and agree upon target version for it.
>> >
>> >Items #1 and #2 will be implemented under IGNITE-5717. Makes sense?
>> >
>> >Thanks,
>> >Sergey.
>> >
>> >On Thu, Aug 3, 2017 at 4:34 AM, Dmitriy Setrakyan
>> >
>> >wrote:
>> >
>> >> Here is what we should do:
>> >>
>> >>1. Pick an acceptable number. Does not matter if it is 10% or
>50%.
>> >>2. Print the allocated memory in *BOLD* letters into the log.
>> >>3. Make sure that Ignite server never hangs due to the low
>memory
>> >issue.
>> >>We should sense it and kick the node out automatically, again
>with
>> >a
>> >> *BOLD*
>> >>message in the log.
>> >>
>> >>  Is this possible?
>> >>
>> >> D.
>> >>
>> >> On Wed, Aug 2, 2017 at 6:09 PM, Vladimir Ozerov
>> >
>> >> wrote:
>> >>
>> >> > My proposal is 10% instead of 80%.
>> >> >
>> >> > ср, 2 авг. 2017 г. в 18:54, Denis Magda :
>> >> >
>> >> > > Vladimir, Dmitriy P.,
>> >> > >
>> >> > > Please see inline
>> >> > >
>> >> > > > On Aug 2, 2017, at 7:20 AM, Vladimir Ozerov
>> >
>> >> > > wrote:
>> >> > > >
>> >> > > > Denis,
>> >> > > >
>> >> > > > The reason is that product should not hang user's computer.
>How
>> >else
>> >> > this
>> >> > > > could be explained? I am developer. I start Ignite, 1 node,
>2
>> >nodes,
>> >> X
>> >> > > > nodes, observe how they join topology. Add one key, 10 keys,
>1M
>> >keys.
>> >> > > Then
>> >> > > > I do a bug in example and load 100M keys accidentally -
>restart
>> >the
>> >> > > > computer. Correct behavior is to have small "maxMemory" by
>> >default to
>> >> > > avoid
>> >> > > > that. User should get exception instead of hang. E.g. Java's
>> >"-Xmx"
>> >> is
>> >> > > > typically 25% of RAM - more adequate value, comparing to
>> >Ignite.
>> >> > > >
>> >> > >
>> >> > > Right, the developer was educated about the Java heap
>parameters
>> >and
>> >> > > limited the overall space preferring OOM to the laptop
>> >suspension. Who
>> >> > > knows how he got to the point that 25% RAM should be used.
>That
>> >might
>> >> > have
>> >> > > been deep knowledge about JVM or he faced several hangs while
>> >testing
>> >> the
>> >> > > application.
>> >> > >
>> >> > > Anyway, JVM creators didn’t decide to predefine the Java heap
>to
>> >a
>> >> static
>> >> > > value to avoid the situations like above. So should not we as
>a
>> >> platform.
>> >> > > Educate people about the Ignite memory behavior like Sun did
>for
>> >the
>> >> Java
>> >> > > heap but do not try to solve the lack of knowledge with the
>> >default
>> >> > static
>> >> > > memory size.
>> >> > >
>> >> > >
>> >> > > > It doesn't matter whether you use persistence or not.
>> >Persistent case
>> >> > > just
>> >> > > > makes this flaw more obvious - you have virtually unlimited
>> >disk, and
>> >> > yet
>> >> > > > you end up with swapping and hang when using Ignite with
>> >default
>> >> > > > configuration. As already explained, the problem is not
>about
>> >> > allocating
>> >> > > > "maxMemory" right away, but about the value of "maxMemory" -
>it
>> >is
>> >> too
>> >> > > big.
>> >> > > >
>> >> > >
>> >> > > How do you know what should be the default then? Why 1 GB? For
>> >> instance,
>> >> > > if I end up having only 1 GB of free memory left and try to
>start
>> >2
>> >> > server
>> >> > > nodes and an application I will face the laptop suspension
>again.
>> >> > >
>> >> > > —
>> >> > > Denis
>> >> > >
>> >> > > > "We had this behavior before" is never an argument. Previous
>> >offheap
>> >> > > > implementation had a lot of flaws, so let's just forget
>about
>> >it.
>> >> > > >
>> >> > > > On Wed, Aug 2, 2017 at 5:08 PM, Denis Magda

Re: [IGNITE-5717] improvements of MemoryPolicy default size

[dsetrakyan]

Without #3, the #1 and #2 make little sense.

Why is #3 so difficult?

⁣D.​

On Aug 4, 2017, 10:46 AM, at 10:46 AM, Sergey Chugunov 
 wrote:
>Dmitriy,
>
>Last item makes perfect sense to me, one may think of it as an
>"OutOfMemoryException" in java.
>However, it looks like such feature requires considerable efforts to
>properly design and implement it, so I would propose to create a
>separate
>ticket and agree upon target version for it.
>
>Items #1 and #2 will be implemented under IGNITE-5717. Makes sense?
>
>Thanks,
>Sergey.
>
>On Thu, Aug 3, 2017 at 4:34 AM, Dmitriy Setrakyan
>
>wrote:
>
>> Here is what we should do:
>>
>>1. Pick an acceptable number. Does not matter if it is 10% or 50%.
>>2. Print the allocated memory in *BOLD* letters into the log.
>>3. Make sure that Ignite server never hangs due to the low memory
>issue.
>>We should sense it and kick the node out automatically, again with
>a
>> *BOLD*
>>message in the log.
>>
>>  Is this possible?
>>
>> D.
>>
>> On Wed, Aug 2, 2017 at 6:09 PM, Vladimir Ozerov
>
>> wrote:
>>
>> > My proposal is 10% instead of 80%.
>> >
>> > ср, 2 авг. 2017 г. в 18:54, Denis Magda :
>> >
>> > > Vladimir, Dmitriy P.,
>> > >
>> > > Please see inline
>> > >
>> > > > On Aug 2, 2017, at 7:20 AM, Vladimir Ozerov
>
>> > > wrote:
>> > > >
>> > > > Denis,
>> > > >
>> > > > The reason is that product should not hang user's computer. How
>else
>> > this
>> > > > could be explained? I am developer. I start Ignite, 1 node, 2
>nodes,
>> X
>> > > > nodes, observe how they join topology. Add one key, 10 keys, 1M
>keys.
>> > > Then
>> > > > I do a bug in example and load 100M keys accidentally - restart
>the
>> > > > computer. Correct behavior is to have small "maxMemory" by
>default to
>> > > avoid
>> > > > that. User should get exception instead of hang. E.g. Java's
>"-Xmx"
>> is
>> > > > typically 25% of RAM - more adequate value, comparing to
>Ignite.
>> > > >
>> > >
>> > > Right, the developer was educated about the Java heap parameters
>and
>> > > limited the overall space preferring OOM to the laptop
>suspension. Who
>> > > knows how he got to the point that 25% RAM should be used. That
>might
>> > have
>> > > been deep knowledge about JVM or he faced several hangs while
>testing
>> the
>> > > application.
>> > >
>> > > Anyway, JVM creators didn’t decide to predefine the Java heap to
>a
>> static
>> > > value to avoid the situations like above. So should not we as a
>> platform.
>> > > Educate people about the Ignite memory behavior like Sun did for
>the
>> Java
>> > > heap but do not try to solve the lack of knowledge with the
>default
>> > static
>> > > memory size.
>> > >
>> > >
>> > > > It doesn't matter whether you use persistence or not.
>Persistent case
>> > > just
>> > > > makes this flaw more obvious - you have virtually unlimited
>disk, and
>> > yet
>> > > > you end up with swapping and hang when using Ignite with
>default
>> > > > configuration. As already explained, the problem is not about
>> > allocating
>> > > > "maxMemory" right away, but about the value of "maxMemory" - it
>is
>> too
>> > > big.
>> > > >
>> > >
>> > > How do you know what should be the default then? Why 1 GB? For
>> instance,
>> > > if I end up having only 1 GB of free memory left and try to start
>2
>> > server
>> > > nodes and an application I will face the laptop suspension again.
>> > >
>> > > —
>> > > Denis
>> > >
>> > > > "We had this behavior before" is never an argument. Previous
>offheap
>> > > > implementation had a lot of flaws, so let's just forget about
>it.
>> > > >
>> > > > On Wed, Aug 2, 2017 at 5:08 PM, Denis Magda 
>> wrote:
>> > > >
>> > > >> Sergey,
>> > > >>
>> > > >> That’s expectable because as we revealed from this discussion
>the
>> > > >> allocation works different depending on whether the
>persistence is
>> > used
>> > > or
>> > > >> not:
>> > > >>
>> > > >> 1) In-memory mode (the persistence is disabled) - the space
>will be
>> > > >> allocated incrementally until the max threshold is reached.
>Good!
>> > > >>
>> > > >> 2) The persistence mode - the whole space (limited by the max
>> > threshold)
>> > > >> is allocated right away. It’s not surprising that your laptop
>starts
>> > > >> choking.
>> > > >>
>> > > >> So, in my previous response I tried to explain that I can’t
>find any
>> > > >> reason why we should adjust 1). Any reasons except for the
>massive
>> > > >> preloading?
>> > > >>
>> > > >> As for 2), that was a big surprise to reveal this after 2.1
>release.
>> > > >> Definitely we have to fix this somehow.
>> > > >>
>> > > >> —
>> > > >> Denis
>> > > >>
>> > > >>> On Aug 2, 2017, at 6:59 AM, Sergey Chugunov <
>> > sergey.chugu...@gmail.com
>> > > >
>> > > >> wrote:
>> > > >>>
>> > > >>> Denis,
>> > > >>>
>> > > >>> Just a simple example from our own codebase: I tried to
>execute
>> > > >>> 

Re: Cluster auto activation design proposal

[dsetrakyan]

Yakov,

I think it is not just restarts, this set of nodes is minimally required for 
the cluster to function, no?

⁣D.​

On Aug 3, 2017, 11:23 AM, at 11:23 AM, Yakov Zhdanov  
wrote:
>Ю> How about naming it "minimal node set" or "required node set"?
>
>Required for what? I would add restart if there are no confusion.
>
>--Yakov

Re: Cluster auto activation design proposal

[dsetrakyan]

How about naming it "minimal node set" or "required node set"?

⁣D.​

On Aug 3, 2017, 11:15 AM, at 11:15 AM, Yakov Zhdanov  
wrote:
>> * Based on some sort of policies when the actual cluster topology
>differs
>too much from the baseline or when some critical condition happens
>(e.g.,
>when there are no more backups for a partition)
>
>Good point, Alex! I would even go further. If cluster is active and
>under
>load and nodes continue joining and leaving then we can have several
>BT's
>that are possible to restart on - the main condition is to have all the
>up
>to date data partitions. I.e. if you have 4 servers and 3 backups most
>probably you can have all the data with 2, 3 and, of course, 4 nodes.
>Makes
>sense?
>
>I would also think of different name. Topology (for me) also implies
>the
>version, but here only nodes carrying data are important. How about
>"restart nodes set"?
>
>--Yakov

Re: Thin client protocol message format

[dsetrakyan]

Got it, thanks!

On Aug 3, 2017, 11:04 AM, at 11:04 AM, Vladimir Ozerov  
wrote:
>Dima,
>
>Our goal is to have a format, which will work for both synchronous,
>asynchronous, single-threaded and multi-threaded clients. All we need
>to
>achieve this is "request ID" propagated from request to response. This
>way
>3-rd party developers will be free to decide how to implement the
>client.
>
>On Thu, Aug 3, 2017 at 4:52 AM, Dmitriy Setrakyan
>
>wrote:
>
>> Let us not forget that the main purpose of such protocol is to enable
>other
>> users contribute their own client implementations for various
>languages.
>> Also, most JDBC and ODBC use cases work in thread-per-connection
>mode.
>>
>> I think that if we introduce a multi-threaded client here, then it
>will be
>> a lot harder to understand, configure, use, or contribute, so I agree
>with
>> Vladimir.
>>
>> Let's keep it simple for now.
>>
>> D.
>>
>> On Wed, Aug 2, 2017 at 10:37 AM, Yakov Zhdanov 
>> wrote:
>>
>> > Agree with Alex. I think our implementations should share single
>> connection
>> > over threads in the process.
>> >
>> > --Yakov
>> >
>>

Re: [IGNITE-5717] improvements of MemoryPolicy default size

[dsetrakyan]

Vova, 1GB seems a bit too small for me, and frankly i do not want t o guess. 
Why not allocate in increments automatically?

⁣D.​

On Aug 1, 2017, 11:03 PM, at 11:03 PM, Vladimir Ozerov  
wrote:
>Denis,
>No doubts you haven't heard about it - AI 2.1 with persistence, when
>80% of
>RAM is allocated right away, was released several days ago. How do you
>think, how many users tried it already?
>
>Guys,
>Do you really think allocating 80% of available RAM is a normal thing?
>Take
>your laptop and check how many available RAM you have right now. Do you
>fit
>to remaining 20%? If not, then running AI with persistence with all
>defaults will bring your machine down. This is insane. We shold
>allocate no
>more than 1Gb, so that user can play with it without any problems.
>
>On Tue, Aug 1, 2017 at 10:26 PM, Denis Magda  wrote:
>
>> My vote goes for option #1 too. I don’t think that 80% is too
>aggressive
>> to bring it down.
>>
>> IGNITE-5717 was created to fix the issue of the 80% RAM allocation on
>64
>> bit systems when Ignite works on top of 32 bit JVM. I’ve not heard of
>any
>> other complaints in regards the default allocation size.
>>
>> —
>> Denis
>>
>> > On Aug 1, 2017, at 10:58 AM, dsetrak...@apache.org wrote:
>> >
>> > I prefer option #1.
>> >
>> > ⁣D.​
>> >
>> > On Aug 1, 2017, 11:20 AM, at 11:20 AM, Sergey Chugunov <
>> sergey.chugu...@gmail.com> wrote:
>> >> Folks,
>> >>
>> >> I would like to get back to the question about MemoryPolicy
>maxMemory
>> >> defaults.
>> >>
>> >> Although MemoryPolicy may be configured with initial and maxMemory
>> >> settings, when persistence is used MemoryPolicy always allocates
>> >> maxMemory
>> >> size for performance reasons.
>> >>
>> >> As default size of maxMemory is 80% of physical memory it causes
>OOME
>> >> exceptions of 32 bit platforms (either on OS or JVM level) and
>hurts
>> >> performance in setups when multiple Ignite nodes are started on
>the
>> >> same
>> >> physical server.
>> >>
>> >> I suggest to rethink these defaults and switch to other options:
>> >>
>> >> - Check whether platform is 32 or 64 bits and adapt defaults. In
>this
>> >> case we still need to address the issue with multiple nodes on one
>> >> machine
>> >>  even on 64 bit systems.
>> >>
>> >>  - Lower defaults for maxMemory and allocate, for instance,
>max(0.3 *
>> >>  availableMemory, 1Gb).
>> >>  This option allows us to solve all issues with starting on 32 bit
>> >> platforms and reduce instability with multiple nodes on the same
>> >> machine.
>> >>
>> >>
>> >> Thoughts and/or other options?
>> >>
>> >> Thanks,
>> >> Sergey.
>>
>>

Re: Data compression in Ignite 2.0

[dsetrakyan]

I would prefer that we reuse an existing compression protocol, but at the table 
level.

If not possible, then we should go with a shared mapping approach. Any idea how 
hard?

⁣D.​

On Aug 1, 2017, 11:15 AM, at 11:15 AM, Vladimir Ozerov  
wrote:
>Vyacheslav,
>
>This is not about my needs, but about the product :-) BinaryObject is a
>central entity used for both data transfer and data storage. This is
>both
>good and bad at the same time.
>
>Good thing is that as we optimize binary protocol, we improve both
>network
>and storage performance at the same time. We have at least 3 things
>which
>will be included into the product soon: varint encoding [1], optimized
>string encoding [2] and null-field optimization [3]. Bad thing is that
>binary object format is not well suited for data storage optimizations,
>including compression. For example, one good compression technique is
>to
>organize data in column-store format, or to introduce shared
>"dictionary"
>with unique values on cache level. In both cases N equal values are not
>stored N times. Instead, we store one value and N references to it, or
>so.
>This way 2x-10x compression is possible depending on workload type.
>Binary
>object protocol with some compression on top of it cannot give such
>improvement, because it will compress data in individual objects,
>instead
>of compressing the whole cache data in a single context.
>
>That said, I propose to give up adding compression to BinaryObject.
>This is
>a dead end. Instead, we should:
>1) Optimize protocol itself to be more compact, as described in
>aforementioned Ignite tickets
>2) Start new discussion about storage compression
>
>You can read papers of other vendors to get better understanding on
>possible compression options. E.g. Oracle has a lot of compression
>techniques, including heat maps, background compression, per-block
>compression, data dictionaries, etc. [4].
>
>[1] https://issues.apache.org/jira/browse/IGNITE-5097
>[2] https://issues.apache.org/jira/browse/IGNITE-5655
>[3] https://issues.apache.org/jira/browse/IGNITE-3939
>[4]
>http://www.oracle.com/technetwork/database/options/compression/advanced-
>compression-wp-12c-1896128.pdf
>
>Vladimir.
>
>
>On Tue, Jul 11, 2017 at 6:56 PM, Vyacheslav Daradur
>
>wrote:
>
>> Hi Igniters!
>>
>> I'd like to continue developing and discussing about compression in
>Ignite.
>>
>> Vladimir, could you propose a design of compression feature in
>Ignite,
>> that suits you?
>>
>> 2017-06-15 16:13 GMT+03:00 Vyacheslav Daradur :
>>
>>> Hi Igniters.
>>>
>>> Vladimir, I want to propose another design of an implementation of
>the
>>> per-field compression.
>>>
>>> 1) We will add new step in the method prepareForCache (for example)
>of
>>> CacheObject, or in GridCacheMapEntry.
>>>
>>> At the step, after marshalling of an object, we will compress fields
>of
>>> the object which described in advance.
>>> User will describe class fields which he wants to compess in an
>another
>>> entity like Metadata.
>>>
>>> For compression, we will introduce another entity, for example
>>> CompressionProcessor, which will work with bytes array (marshalled
>object).
>>> The entity will read bytes array of described fields, compress it
>and
>>> rewrite binary representation of the whole object.
>>> After processing the object will be put in the cache.
>>>
>>> In this case design not to relate to binary infrastructure.
>>> But there is big overhead to heap-memory for the buffer.
>>>
>>> 2) Another solution is to compress bytes array of whole object on
>copying
>>> to off-heap.
>>> But, in this case I don't understand yet, how to provide support of
>>> querying and indexing.
>>>
>>>
>>> 2017-06-09 11:21 GMT+03:00 Sergey Kozlov :
>>>
 Hi

 * "Per-field compression" is applicable for huge BLOB fields and
>will
 impose the restrictions like unable ot index such fields, slower
>getting
 data, potential OOM issues if compression ration is too high.
 But for some cases it makes sense

 On Fri, Jun 9, 2017 at 11:11 AM, Антон Чураев
>
 wrote:

 > Seems that Dmitry is referring to transparent data encryption. It
>is
 used
 > throughout the whale database industry.
 >
 > 2017-06-09 10:50 GMT+03:00 Vladimir Ozerov
>:
 >
 > > Dima,
 > >
 > > Encryption of certain fields is as bad as compression. First,
>it is a
 > huge
 > > change, which makes already complex binary protocol even more
 complex.
 > > Second, it have to be ported to CPP, .NET platforms, as well as
>to
 JDBC
 > and
 > > ODBC.
 > > Last, but the most important - this is not our headache to
>encrypt
 > > sensitive data. This is user responsibility. Nobody in a sane
>mind
 will
 > > store passwords in plain form. Instead, user should encrypt it
>on his
 > own,
 > > choosing proper 

Re: Create own SQL parser

[dsetrakyan]

Vova, I am not sure what you are proposing... extending H2 parser with new 
syntax or a brand new parser?

⁣D.​

On Aug 1, 2017, 4:26 PM, at 4:26 PM, Vladimir Ozerov  
wrote:
>Andrey,
>
>Note that I am not proposing to remove H2 as a whole. Main point for
>now is
>to support missing pieces of DDL syntax and possibly and some
>extensions.
>Several examples:
>
>1) Currently CREATE TABLE command looks ugly:
>CREATE TABLE Person (id LONG, name VARCHAR) WITH "template=PARTITIONED,
>backups=1, ..."
>
>Commas typically treated in a special way in editors and IDEs, so user
>will
>have to escape them, making usability even worse.
>
>2) What if I need to introduce new template? Currently you have to
>restart
>the node with new config. With our own parser you will do:
>CREATE TEMPLATE my_template MODE=PARTITIONED, BACKUPS=1;
>CREATE TABLE Person (...) TEMPLATE my_template;
>
>No restarts, everything is done dynamically.
>
>
>
>On Tue, Aug 1, 2017 at 4:18 PM, Andrey Mashenkov
>> wrote:
>
>> Vovan,
>>
>> 1. What about ANSI-xx compliant? Will new syntax brake it in some
>cases or
>> just extend?
>>
>> 2. This would be great to have more ways for optimization.
>>
>> Does anyone know or may be have experience with some frameworks or
>open
>> projects which can be helpful? E.g. Apache Calcite?
>>
>> On Tue, Aug 1, 2017 at 3:25 PM, Vladimir Ozerov
>
>> wrote:
>>
>> > Igniters,
>> >
>> > As you know, we rely on H2 for SQL query parsing. This has several
>> > drawbacks:
>> >
>> > 1) Limited and ugly syntax
>> > Ignite has lot's of unique concepts which are in no way supported
>by
>> > traditional RDBMS in general, or by H2 in particular. For example:
>> > - query hints ("distributedJoins", "replicatedOnly", "colocated")
>> > - index hints ("inline size")
>> > - cache configuration (memory policy, affinity key, cache mode,
>etc)
>> > - transaction mode (concurrency, isolation, timeouts) - not needed
>now,
>> but
>> > will be required when transactional SQL is ready
>> >
>> > 2) Performance implications
>> > Typical SQL processing flow looks as follows
>> > - Parse String to H2 object form (prepared statement)
>> > - Convert it to Ignite object form (AST)
>> > - Then convert it back to map and reduce queries in String form
>> > - Convert map and reduce queries from String back to H2
>PreparedStatement
>> > again for final execution
>> >
>> > This is way too much. Moreover, H2 optimizes query during parsing,
>but
>> it's
>> > optimizer is not smart enough. E.g., Ignite "IN" clauses are not
>> optimized
>> > and hence doesn't use indexes, so we force users to use
>intermediate
>> tables
>> > with very ugly syntax, while we should do that on our own instead.
>> Another
>> > example is common expression elimination - H2 cannot do that even
>for
>> > deterministic functions, what cause performance problems
>frequently.
>> >
>> > I propose to start some work in direction of our own parser. We can
>start
>> > with something very simple, e.g. DDL support, which is not that
>complex,
>> > but will improve usability significantly. And then gradually extend
>it to
>> > more complex statements where both rich BNF and optimizer is
>necessary.
>> >
>> > Thoughts?
>> >
>> > Vladimir.
>> >
>>
>>
>>
>> --
>> Best regards,
>> Andrey V. Mashenkov
>>

Re: Thin client protocol message format

[dsetrakyan]

Backward compatible?

⁣D.​

On Aug 1, 2017, 7:04 PM, at 7:04 PM, Pavel Tupitsyn  
wrote:
>Dmitry, we don't need any reserved bytes, because protocol is
>versioned.
>
>On Tue, Aug 1, 2017 at 7:49 PM,  wrote:
>
>> We should also leave 8 bytes of empty space for future changes.
>>
>> ⁣D.​
>>
>> On Aug 1, 2017, 6:41 PM, at 6:41 PM, Pavel Tupitsyn
>
>> wrote:
>> >Alexey, good idea. ODBC and JDBC could also benefit from this.
>> >
>> >On Tue, Aug 1, 2017 at 7:27 PM, Alexey Kuznetsov
>> >
>> >wrote:
>> >
>> >> Pavel,
>> >>
>> >> How about data compression?
>> >> May be it make sense to add a byte with compression algorithm?
>> >> 0 - none
>> >> 1 - ZIP
>> >> 2 - 
>> >> 
>> >>
>> >> On Tue, Aug 1, 2017 at 11:10 PM, Pavel Tupitsyn
>> >
>> >> wrote:
>> >>
>> >> > Igniters,
>> >> >
>> >> > Below is a proposed design for thin client protocol [1] [2]
>socket
>> >data
>> >> > exchange format.
>> >> >
>> >> > * Values are little-endian
>> >> > * Every request and response message starts with 4-byte length
>> >(including
>> >> > handshake)
>> >> > * Ignite binary format is used for value serialization (via
>> >> > GridBinaryMarshaller/BinaryWriter/BinaryReader). Ignite binary
>> >protocol
>> >> > has
>> >> > to be implemented by clients anyway to work with cache values,
>so
>> >it
>> >> makes
>> >> > sense to use for all data exchange.
>> >> >
>> >> >
>> >> > 1) Socket connection is established on a port according
>> >> > to ConnectorConfiguration.port
>> >> >
>> >> > 2) Handshake is performed.
>> >> >  Request:
>> >> >int32 length = 8 // message length
>> >> >byte opCode = 1   // handshake command
>> >> >int16 verMajor
>> >> >int16 verMinor
>> >> >int16 verMaintenance
>> >> >byte clientCode = 2// client type code (odbc, jdbc,
>> >platform)
>> >> >
>> >> >  Response:
>> >> >uint32 length = 1
>> >> >byte success
>> >> >
>> >> > 3) Execute command. Request starts with a command code, then
>goes
>> >> > command-specific data.
>> >> > For example, IgniteCache.put(1, "foo") will
>look
>> >like
>> >> > this:
>> >> >  Request:
>> >> >int16 opCode// OP_CACHE_GET
>> >> >int32 cacheId// GridCacheUtils.cacheId
>> >> >byte flags  // skipStore, noRetry, etc
>> >> >binobject key
>> >> >
>> >> >  Response:
>> >> >byte success
>> >> >binobject value
>> >> >
>> >> > Where binobject corresponds to Ignite BinaryMarshaller format,
>> >which is
>> >> > produced by BinaryWriter.writeObject method. Integer will be
>> >represented
>> >> as
>> >> >   byte typeCode = 3  // GridBinaryMarshaller.INT
>> >> >   int32 value = 1
>> >> >
>> >> > 4) Goto (3)
>> >> >
>> >> >
>> >> > Comments are welcome.
>> >> >
>> >> > Pavel
>> >> >
>> >> > [1]
>> >> > http://apache-ignite-developers.2346864.n4.nabble.
>> >> com/Support-for-Ignite-
>> >> > clients-in-any-language-thin-client-protocol-td20297.html
>> >> > [2] https://issues.apache.org/jira/browse/IGNITE-5896
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Alexey Kuznetsov
>> >>
>>

Re: Thin client protocol message format

[dsetrakyan]

We should also leave 8 bytes of empty space for future changes.

⁣D.​

On Aug 1, 2017, 6:41 PM, at 6:41 PM, Pavel Tupitsyn  
wrote:
>Alexey, good idea. ODBC and JDBC could also benefit from this.
>
>On Tue, Aug 1, 2017 at 7:27 PM, Alexey Kuznetsov
>
>wrote:
>
>> Pavel,
>>
>> How about data compression?
>> May be it make sense to add a byte with compression algorithm?
>> 0 - none
>> 1 - ZIP
>> 2 - 
>> 
>>
>> On Tue, Aug 1, 2017 at 11:10 PM, Pavel Tupitsyn
>
>> wrote:
>>
>> > Igniters,
>> >
>> > Below is a proposed design for thin client protocol [1] [2] socket
>data
>> > exchange format.
>> >
>> > * Values are little-endian
>> > * Every request and response message starts with 4-byte length
>(including
>> > handshake)
>> > * Ignite binary format is used for value serialization (via
>> > GridBinaryMarshaller/BinaryWriter/BinaryReader). Ignite binary
>protocol
>> > has
>> > to be implemented by clients anyway to work with cache values, so
>it
>> makes
>> > sense to use for all data exchange.
>> >
>> >
>> > 1) Socket connection is established on a port according
>> > to ConnectorConfiguration.port
>> >
>> > 2) Handshake is performed.
>> >  Request:
>> >int32 length = 8 // message length
>> >byte opCode = 1   // handshake command
>> >int16 verMajor
>> >int16 verMinor
>> >int16 verMaintenance
>> >byte clientCode = 2// client type code (odbc, jdbc,
>platform)
>> >
>> >  Response:
>> >uint32 length = 1
>> >byte success
>> >
>> > 3) Execute command. Request starts with a command code, then goes
>> > command-specific data.
>> > For example, IgniteCache.put(1, "foo") will look
>like
>> > this:
>> >  Request:
>> >int16 opCode// OP_CACHE_GET
>> >int32 cacheId// GridCacheUtils.cacheId
>> >byte flags  // skipStore, noRetry, etc
>> >binobject key
>> >
>> >  Response:
>> >byte success
>> >binobject value
>> >
>> > Where binobject corresponds to Ignite BinaryMarshaller format,
>which is
>> > produced by BinaryWriter.writeObject method. Integer will be
>represented
>> as
>> >   byte typeCode = 3  // GridBinaryMarshaller.INT
>> >   int32 value = 1
>> >
>> > 4) Goto (3)
>> >
>> >
>> > Comments are welcome.
>> >
>> > Pavel
>> >
>> > [1]
>> > http://apache-ignite-developers.2346864.n4.nabble.
>> com/Support-for-Ignite-
>> > clients-in-any-language-thin-client-protocol-td20297.html
>> > [2] https://issues.apache.org/jira/browse/IGNITE-5896
>> >
>>
>>
>>
>> --
>> Alexey Kuznetsov
>>

Re: Resurrect FairAffinityFunction

[dsetrakyan]

Agree with Val, we should bring it back.

⁣D.​

On Jul 24, 2017, 8:14 PM, at 8:14 PM, Valentin Kulichenko 
 wrote:
>Guys,
>
>Some time ago we removed FairAffinityFunction from the project.
>However, my
>communication with users clearly shows that is was a rush decision.
>Distribution showed by Fair AF is much better than default and for some
>users it's extremely important. Basically, there are cases when
>rendezvous
>function is no-go.
>
>The reason for removal was that it was possible to get inconsistent
>results
>in case multiple caches were created on different topologies. However,
>I
>think this is fixable. As far as I understand, the only thing we need
>to do
>is to maintain a single AffinityFunctionContext for all the caches with
>same affinity function. Currently for each cache we have separate
>context
>which holds the state used by Fair AF. If the state is different, we
>have
>an issue.
>
>The only question is how to check whether two functions are the same or
>not. In case both cache node filter and backup filter are not
>configured,
>this is easy - if number of partitions and excludeNeighbors flag are
>equal
>for two functions, these functions are also equal.
>
>With filters it's a bit more complicated as these are custom
>implementations and in general case we don't know how to compare them.
>Although, to solve this problem, we can enforce user to implement
>equals()
>method for these implementation if Fair AF is used.
>
>I propose to make changes described above and bring Fair AF back.
>
>Thoughts?
>
>-Val

Re: Ignite ML status update

[dsetrakyan]

Sounds good.

On Jul 20, 2017, 1:56 PM, at 1:56 PM, Yury Babak  wrote:
>Dmitriy, I can do it. But as far as I know Oleg Ignatenko already wrote
>blog
>about Ignite ML. But it only about AI 2.0 and only on
>Russian(Habrahabr). So
>we could translate this post in English.
>
>And also I can write a blog about new features ML in AI 2.1 and
>mentioned
>features which we plan for 2.2.
>
>Regards,
>Yury
>
>
>
>--
>View this message in context:
>http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-ML-status-update-tp19829p19875.html
>Sent from the Apache Ignite Developers mailing list archive at
>Nabble.com.

Re: New method for MemoryMetrics: segment fill variance

[dsetrakyan]

Sounds good!

⁣D.​

On Jul 19, 2017, 1:54 PM, at 1:54 PM, Ivan Rakov  wrote:
>Dmitriy,
>
>I agree, hashFunctionScore is more comprehensible.
>I'd correct it to "pageHashFunctionScore" to make it clear that it's
>about function that distributes pages, not about hashCode() of
>user-provided keys.
>
>--
>Best Regards,
>Ivan Rakov
>
>On 18.07.2017 19:34, Dmitriy Setrakyan wrote:
>> Ivan,
>>
>> How about renaming it to hashFunctionScore? This will be more
>digestable to
>> users. "segmentFillVariance" sounds too scientific even to me.
>>
>> D.
>>
>> On Tue, Jul 18, 2017 at 10:39 AM, Ivan Rakov 
>wrote:
>>
>>> Hello Igniters,
>>>
>>> I want to propose one more method for memory metrics -
>>> segmentFillVariance, float value from 0.0 to 1.0.
>>>
>>> Motivation: when persistence is enabled, we split memory policy into
>K
>>> equal segments (K = concurrency level). We calculate hash of pageId
>to
>>> determine segment where page will be stored.
>>> Pages eviction to disk starts when segment is full. If hash function
>is
>>> bad enough, one segment can overflow when there are lots of free
>space, and
>>> evictions can start too early. We want to be able to distinguish
>such
>>> situations.
>>>
>>> I propose to calculate segmentFillVariance as difference between
>maximum
>>> and minimum percentage of segment fill. Greater variance signals
>about
>>> worse hash function.
>>>
>>> Thoughts?
>>>
>>> --
>>> Best Regards,
>>> Ivan Rakov
>>>
>>>

Re: usage analytics

[dsetrakyan]

I would try to ping legal again and see if they respond. If not, I think we 
will need to come up with a simpler approach, that does not require legal 
approval.

⁣D.​

On Jul 18, 2017, 2:23 PM, at 2:23 PM, Nikita Ivanov  wrote:
>Igniters,
>Just a quick update. I haven't gotten response from ASF Legal on this
>thread and I frankly don't know how to proceed here. What's the process
>to
>arrive to a decision point here?
>
>Thanks!
>--
>Nikita Ivanov
>
>
>On Mon, Jul 10, 2017 at 3:11 PM, Konstantin Boudnik 
>wrote:
>
>> On Sat, Jul 08, 2017 at 11:04AM, Nikita Ivanov wrote:
>> > Cos,
>> > Based on my experience having it off by default negates the entire
>> > purpose... We need statistically meaningful data set to make any
>> inferences
>> > from it. Moreover, if we are going to ask folks to turn it on it
>will
>> > significantly skew the resulting data set anyways and show full
>picture.
>> I
>> > think "on" by default is the better option if we are to collect
>usage
>> stats
>> > to begin with.
>>
>> yes, sure. But having this "on" by default is likely to expose us to
>> another
>> shit-storm down the road. An interesting dilemma to have indeed. In
>my
>> experience, whenever I install something like a browser or an
>operating
>> system, it would ask if I want to make the particular piece of
>software
>> better
>> by sending back some anonymized stats. Basically, I am given a way to
>> explicitly opt-out if I wish.
>>
>> By turning the feature "on" by default is like saying: "we'll be
>collecting
>> some stats, but if you don't want to you can go here and there and
>disable
>> the
>> collection. Oh, and by the way - you need to go and figure out the
>exact
>> steps
>> to disable it."
>>
>> > Also, I want to re-iterate it again to avoid misunderstanding:
>there is
>> no
>> > proposal nor will there be a technical way to attribute collected
>data
>> back
>> > to a certain company. That's not what this is all about. We should
>only
>> be
>> > interested in aggregated stats (community size, geo information,
>language
>> > information, components usage).
>>
>> Yes, I think it is clear, but never hurts to re-iterate.
>>
>> Cos
>>
>> > Thoughts?
>> >
>> > --
>> > Nikita Ivanov
>> > Founder & CTO
>> > GridGain Systems
>> >
>> > On Fri, Jul 7, 2017 at 8:17 PM, Konstantin Boudnik 
>> wrote:
>> >
>> > > Actually, that should be OFF by default. It sounds like this
>reduce the
>> > > amount
>> > > of the data collected, but this would address the concerns of
>companies
>> > > like
>> > > Roman's. I know for sure that a few of my clients would sue my
>ass out
>> of
>> > > existence if I gave them the platform collecting their
>data-centers
>> info.
>> > >
>> > > Let's have it, set if off by default and document and easy way to
>turn
>> it
>> > > off.
>> > > Then start making rounds asking our user base to share _some_ of
>the
>> stats
>> > > with the community, so we can track the growth of the install
>base,
>> etc.
>> > >
>> > > Cos
>> > >
>> > > On Thu, Jul 06, 2017 at 08:20AM, Nikita Ivanov wrote:
>> > > > The idea so far is to have a single system property in
>configuration
>> that
>> > > > turns this off completely. I envision that this will be
>prominently
>> > > > featured on Ignite website so that everyone who would like to
>> disable it
>> > > -
>> > > > can do it in seconds.
>> > > >
>> > > > Thoughts?
>> > > >
>> > > > --
>> > > > Nikita Ivanov
>> > > > Founder & CTO
>> > > > GridGain Systems
>> > > >
>> > > > On Wed, Jul 5, 2017 at 9:27 PM, Roman Shtykh
>
>> wrote:
>> > > >
>> > > > > Nikita,
>> > > > >
>> > > > > Sending and storing (somewhere the company cannot securely
>handle)
>> any
>> > > > > information (OS version, IP addresses, etc.) that can be used
>to
>> > > compromise
>> > > > > the services would be unacceptable.
>> > > > > Turning it off might be ok (possibly through the cluster
>settings,
>> not
>> > > via
>> > > > > globally-accessible site), but the thing that there's a risk
>some
>> > > > > information can leak outside (for any reason, starting from a
>human
>> > > > > mistake) is scary.
>> > > > >
>> > > > > -- Roman
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Thursday, July 6, 2017 12:38 PM, Nikita Ivanov <
>> > > niva...@gridgain.com>
>> > > > > wrote:
>> > > > >
>> > > > >
>> > > > > Roman,
>> > > > > Thanks for the feedback. What are those questions
>specifically?
>> Are IP
>> > > > > addresses and OS is what causing it?
>> > > > >
>> > > > > Thanks!
>> > > > >
>> > > > > --
>> > > > > Nikita Ivanov
>> > > > > Founder & CTO
>> > > > > GridGain Systems
>> > > > >
>> > > > > On Wed, Jul 5, 2017 at 6:15 PM, Roman Shtykh
>> > > > >
>> > > > > wrote:
>> > > > >
>> > > > > NIkita,
>> > > > >
>> > > > > While this will help improve Ignite, it will prevent its
>adoption
>> by
>> > > many
>> > > > > projects -- sending and retaining IP adresses, OS versions,
>etc.
>> raises
>> > 

Re: Replace Cron4J with Quartz for ignite-schedule module.

[dsetrakyan]

Probably a good task for a newbie.

⁣D.​

On Jun 21, 2017, 9:41 AM, at 9:41 AM, Alexey Kuznetsov  
wrote:
>Done,
>
>https://issues.apache.org/jira/browse/IGNITE-5565
>
>I think it could take a couple of days in background mode.
>
>On Wed, Jun 21, 2017 at 1:40 PM, Dmitriy Setrakyan
>
>wrote:
>
>> Thanks! Please file a ticket. Do you have an idea on the amount of
>work
>> this would require?
>>
>> On Wed, Jun 21, 2017 at 8:39 AM, Alexey Kuznetsov
>
>> wrote:
>>
>> > Dima,
>> >
>> > IgniteScheduler provides functionality for scheduling jobs locally
>using
>> > UNIX cron-based syntax. Instance of GridScheduler is obtained from
>grid
>> as
>> > follows:
>> >IgniteScheduler s = Ignition.ignite().scheduler();
>> >
>> > Scheduler supports standard UNIX cron format with optional prefix
>of {n1,
>> > n2}, where n1 is delay of scheduling in seconds and n2 is the
>number of
>> > execution.
>> > Both parameters are optional. Here's an example of scheduling a
>closure
>> > that broadcasts a message to all nodes five times, once every
>minute,
>> with
>> > initial delay of two seconds:
>> >Ignition.ignite().scheduler().scheduleLocal(
>> >SchedulerFuture = Ignition.ignite().scheduler().
>> > scheduleLocal(new
>> > Callable() {
>> >@Override public Object call() throws
>IgniteCheckedException {
>> >..
>> >}
>> >}, "{2, 5} * * * * *" // 2 seconds delay with 5 executions
>only.
>> >);
>> >
>> > On Wed, Jun 21, 2017 at 1:31 PM, Dmitriy Setrakyan <
>> dsetrak...@apache.org>
>> > wrote:
>> >
>> > > Alexey,
>> > >
>> > > Can you remind what we use the schedule module in Ignite for?
>> > >
>> > > D.
>> > >
>> > > On Wed, Jun 21, 2017 at 7:26 AM, Alexey Kuznetsov <
>> akuznet...@apache.org
>> > >
>> > > wrote:
>> > >
>> > > > Hi!
>> > > >
>> > > > 1) Cron4J is very old:
>> > > >   Latest Cron4j 2.2.5 released: *28-Dec-2011 *
>> > > >   Latest Quarz 2.3.0 released: *20-Apr-2017*
>> > > >
>> > > > 2) Not very friendly license:
>> > > >   CronJ4 licensed under GNU LESSER GENERAL PUBLIC LICENSE
>> > > >   Quartz is freely usable, licensed under the *Apache 2.0*
>license.
>> > > >
>> > > > So, if we replace Cron4J  with Quartz we can move
>*ignite-schedule*
>> > > module
>> > > >  from lgpl profile to main distribution.
>> > > >
>> > > > Any objections?
>> > > >
>> > > > If no, I will create JIRA issue and implement this change.
>> > > >
>> > > > --
>> > > > Alexey Kuznetsov
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Alexey Kuznetsov
>> >
>>
>
>
>
>--
>Alexey Kuznetsov