Re: [DISCUSS] Ignite Update Checker

[Konstantin Boudnik]

This will surely works to get the update info to the nodes. And it
sounds like a legit approach.
I cannot judge on the feasibility of the GA though - don't know squat
about it ;)

Thanks!
--
  With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.


On Fri, Aug 25, 2017 at 1:56 PM, Dmitriy Setrakyan
 wrote:
> Cos, Raul,
>
> Thanks for the feedback. I completely agree about Maven Central being a 3rd
> party repo (did not think about that initially). All your suggestions make
> sense, but I would like to keep it as simple as possible, and so far
> everything suggested required GIT dependencies and extra work.
>
> How about Yakov's suggestion. We simply add a page to the Ignite website
> which will have only the latest version. Every time a node starts, it
> receives the latest version from the page and suggests that users upgrade
> if needed.
>
> Also, since we have GA enabled for the website, we can track how many times
> this page was accessed, which will be equal to the number of starts. This
> way, the counter information is tracked and monitored by the Ignite PMC.
>
> This approach looks pretty innocent to me and everything is kept and
> managed within Apache.
>
> Thoughts?
>
> D.
>
>
> On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik  wrote:
>
>> I agree with Raul.
>> - providing a ping-back address to a 3rd party might be frown upon by some.
>>   And might have a consequences like stats collection about users'
>>   infrastructure.
>> - checking an ASF git-repo is easy and won't download any binary data:
>>   everything is clear text and could be easily monitored by any of the
>> network
>>   diagnostic tools, shall it be required. But it involves a bit of the
>> release
>>   discipline.
>> - the binary data download in the runtime is my main concern. This is the
>>   vector of MMA. What if someone gains the control over the repository and
>>   replaces the file with some malicious content.
>>
>> As for the particular mechanism: IIRC Ignite used to make a call to an
>> external script to check upon the atest software version available for
>> download. In the past, the endpoint was running on a 3rd party server, I
>> believe the best way would be to put this script on ASF infra and have the
>> "update checker" running in a completely controlled environment. Actually,
>> I
>> recall we had this very discussion during the Incubation; I can probably
>> dig
>> out the corresponding thread.
>>
>> Thoughts?
>>   Cok
>>
>> On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
>> > Hey guys
>> >
>> > In my opinion, maven.org is still owned by a third party (Sonatype). We
>> > don't know what kind of data analysis or intelligence extraction they
>> run.
>> >
>> > If Ignite servers all over the world were hitting maven.org periodically
>> > asking for an Ignite Maven artifact, it gives Sonatype a clear indication
>> > of who is running an Ignite server.
>> >
>> > They could reverse lookup the IP address and find out what corporation it
>> > is.
>> >
>> > How about having Ignite check the ASF Git directly?
>> >
>> > We could use the Git ssh API, but that would require a new dependency,
>> > which I advise against.
>> >
>> > Alternatively, we could have it scrape this HTML for new Git tags:
>> > https://git-wip-us.apache.org/repos/asf?p=ignite.git
>> >
>> > Another option is to place a txt file in the root of the master branch
>> (e.g
>> > LATEST), containing a list of the latest GA versions for each major
>> version
>> > line (1.x, 2.x).
>> >
>> > I would advocate this last option, but it requires somebody remembering
>> to
>> > update the file with every release, unless we automate it with a Maven
>> > plugin.
>> >
>> > Hope that helps!
>> >
>> >
>> > On 24 Aug 2017 19:37, "Denis Magda"  wrote:
>> >
>> > I see nothing wrong with this approach.
>> >
>> > Cos, Roman, Raul, as Apache veterans, what do you think? Is it good to
>> go?
>> >
>> > —
>> > Denis
>> >
>> > > On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan > >
>> > wrote:
>> > >
>> > > Is everyone OK with this approach? Should I file a ticket on it?
>> > >
>> > > On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <
>> dsetrak...@apache.org>
>> > > wrote:
>> > >
>> > >> Igniters,
>> > >>
>> > >> There has been lots of talk of proposals about various usage metrics
>> for
>> > >> Ignite and nothing came of it. I would like to resurrect the topic and
>> > >> propose something very simple and non-intrusive.
>> > >>
>> > >> 1. Update Checker
>> > >> The main purpose of the update checker is not to collect metrics, but
>> to
>> > >> notify users about a new version of Ignite by accessing maven.org and
>> > >> getting the version out of the metadata file:

Re: [DISCUSS] Ignite Update Checker

[Raúl Kripalani]

Hey Dmitriy and all

Also, since we have GA enabled for the website, we can track how many times
> this page was accessed, which will be equal to the number of starts. This
> way, the counter information is tracked and monitored by the Ignite PMC.


Unfortunately this won't work because GA is loaded via Javascript on the
browser, so Google will never receive the page hit.

Given the constraints, the most viable solution is an HTTPS endpoint
running on ASF infra that Ignite invokes via a GET or POST request. The
simplest thing is to write each request in a log file, along with the
timestamp, the version reported by the client, maybe the IP (not sure about
the ASF rules about this concerning privacy, I guess it's OK if you make it
an opt-in) and a unique node identifier, i.e. a UUID the node creates on
first startup or something.

This endpoint would need some basic DDoS protection and blacklisting to
prevent data spoofing.

If we'll be implementing this endpoint anyway, then there's no point
placing another file on Git or elsewhere for reporting the latest versions:
the endpoint itself can return them.

WDYT?

Cheers.

On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan 
wrote:

> Cos, Raul,
>
> Thanks for the feedback. I completely agree about Maven Central being a 3rd
> party repo (did not think about that initially). All your suggestions make
> sense, but I would like to keep it as simple as possible, and so far
> everything suggested required GIT dependencies and extra work.
>
> How about Yakov's suggestion. We simply add a page to the Ignite website
> which will have only the latest version. Every time a node starts, it
> receives the latest version from the page and suggests that users upgrade
> if needed.
>
> Also, since we have GA enabled for the website, we can track how many times
> this page was accessed, which will be equal to the number of starts. This
> way, the counter information is tracked and monitored by the Ignite PMC.
>
> This approach looks pretty innocent to me and everything is kept and
> managed within Apache.
>
> Thoughts?
>
> D.
>
>
> On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik 
> wrote:
>
> > I agree with Raul.
> > - providing a ping-back address to a 3rd party might be frown upon by
> some.
> >   And might have a consequences like stats collection about users'
> >   infrastructure.
> > - checking an ASF git-repo is easy and won't download any binary data:
> >   everything is clear text and could be easily monitored by any of the
> > network
> >   diagnostic tools, shall it be required. But it involves a bit of the
> > release
> >   discipline.
> > - the binary data download in the runtime is my main concern. This is the
> >   vector of MMA. What if someone gains the control over the repository
> and
> >   replaces the file with some malicious content.
> >
> > As for the particular mechanism: IIRC Ignite used to make a call to an
> > external script to check upon the atest software version available for
> > download. In the past, the endpoint was running on a 3rd party server, I
> > believe the best way would be to put this script on ASF infra and have
> the
> > "update checker" running in a completely controlled environment.
> Actually,
> > I
> > recall we had this very discussion during the Incubation; I can probably
> > dig
> > out the corresponding thread.
> >
> > Thoughts?
> >   Cok
> >
> > On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
> > > Hey guys
> > >
> > > In my opinion, maven.org is still owned by a third party (Sonatype).
> We
> > > don't know what kind of data analysis or intelligence extraction they
> > run.
> > >
> > > If Ignite servers all over the world were hitting maven.org
> periodically
> > > asking for an Ignite Maven artifact, it gives Sonatype a clear
> indication
> > > of who is running an Ignite server.
> > >
> > > They could reverse lookup the IP address and find out what corporation
> it
> > > is.
> > >
> > > How about having Ignite check the ASF Git directly?
> > >
> > > We could use the Git ssh API, but that would require a new dependency,
> > > which I advise against.
> > >
> > > Alternatively, we could have it scrape this HTML for new Git tags:
> > > https://git-wip-us.apache.org/repos/asf?p=ignite.git
> > >
> > > Another option is to place a txt file in the root of the master branch
> > (e.g
> > > LATEST), containing a list of the latest GA versions for each major
> > version
> > > line (1.x, 2.x).
> > >
> > > I would advocate this last option, but it requires somebody remembering
> > to
> > > update the file with every release, unless we automate it with a Maven
> > > plugin.
> > >
> > > Hope that helps!
> > >
> > >
> > > On 24 Aug 2017 19:37, "Denis Magda"  wrote:
> > >
> > > I see nothing wrong with this approach.
> > >
> > > Cos, Roman, Raul, as Apache veterans, what do you think? Is it good to
> > go?
> > >
> > > —
> > > Denis
> > >
> > > > On Aug 23, 2017, at 11:17 

Re: Shell script to stop Ignite node

[Dmitriy Setrakyan]

Huge +1

We badly need Linux specific installation packages. Sergey, can you file a
ticket?

D.

On Tue, Aug 22, 2017 at 12:51 AM, Sergey Kozlov 
wrote:

> Hi
>
> Looks like we're trying reinvent a bicycle.
>
> First we have to admit that current installation procedure is very poor
> ("unzip and run"). As result we need scripts to manage Apache Ignite
> process(es)
>
> We should to move to better integration under *nix system (at least),
> namely:
>
> 1. Provide a script to *nix for start/stop/restart/reload Apache Ignite as
> system service (a deamon). At the beginning we can focus on Ubuntu and
> RedHat (CentOS)
>
> 2. Provide the start confugration for Apache Ignite where provide
> non-spring options like number of Apache Ignite instances, JVM options,
> work/log directories etc (at might be ini file)
>
> 3. Provide platform-adopted packages (*.pkg, *deb) in the own repository
> that will install (upgrade) Apache Ignite in proper directories, for
> instance: configuration in /etc/ignite/config, visorcmd in /usr/sbin, start
> scripts in /etc/init.d, work directory in /var/lib/ignite, logs in
> /var/log/ignite/. It will significally reduce the efforts to install Apache
> Ignite for newbies ("just add the reporsitory and run package manager")
>
>
>
> On Tue, Aug 22, 2017 at 2:16 AM, Dmitriy Setrakyan 
> wrote:
>
> > Completely agree. Why do we have Visor CLI join the cluster at all? It
> > should be issuing REST requests to the cluster remotely. Essentially,
> every
> > command should be a REST request/response.
> >
> > Is this possible?
> >
> > D.
> >
> > On Mon, Aug 21, 2017 at 12:25 PM, Denis Magda  wrote:
> >
> > > Igniters,
> > >
> > > I would propose to make this a feature of Visor CLI.
> > >
> > > Recently we’ve added the script to control a cluster activation [1]
> > > explaining that Visor CLI is not flexible for that. If we keep moving
> > this
> > > way at some point there will be the whole zoo of scripts.
> > >
> > > Why don’t we improve Visor CLI letting it connect to the cluster in the
> > > simplest way and embed the proposed feature in it?
> > >
> > > [1] http://apache-ignite-developers.2346864.n4.nabble.
> > > com/Control-sh-script-and-cluster-activation-td20821.html
> > > That’s the second
> > >
> > > —
> > > Denis
> > >
> > > > On Aug 21, 2017, at 11:44 AM, Valentin Kulichenko <
> > > valentin.kuliche...@gmail.com> wrote:
> > > >
> > > > +1
> > > >
> > > > -Val
> > > >
> > > > On Mon, Aug 21, 2017 at 9:06 AM, Yakov Zhdanov 
> > > wrote:
> > > >
> > > >> Guys,
> > > >>
> > > >> Currently to stop node on a particular server (not using visor or
> web
> > > >> console) I need to run "jps" and then "kill" with a pid.
> > > >>
> > > >> What if we create ignite-stop.sh which will display the list of
> > started
> > > >> nodes on local host and allow user to choose one to stop. Also it
> > should
> > > >> support --all param to stop all and probably a signal.
> > > >>
> > > >> Thoughts?
> > > >>
> > > >> --Yakov
> > > >>
> > >
> > >
> >
>
>
>
> --
> Sergey Kozlov
> GridGain Systems
> www.gridgain.com
>

Re: [DISCUSSION] Urgent Ignite bug fix release

[Dmitriy Setrakyan]

I like the format proposed by Denis, very clear.

However, I also do not understand why a user should change the size of some
system cache. How would a user ever know what value to put there? This
value should be configured by Ignite automatically.

D.

On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda  wrote:

> Igor,
>
> Let me suggest this format.
>
> -
> Excessive memory usage by Ignite node process (performance may drop)
> [requested=29251MB, available=15942MB]
>
> Please tune the following settings:
>   [MemoryConfiguration.defaultMemoryPolicySize = suggested value]
>   MemoryConfiguration.systemCacheMaxSize = suggested value
>   [MemoryPolicyConfiguration.maxSize for {policy_name_1} = suggested
> value]
>   [MemoryPolicyConfiguration.maxSize for {policy_name_2} = suggested
> value]
>
> Current settings:
>[DefaultMemoryPolicySize = value]
>[{policy_name_1} size = value]
>[{policy_name_1} size = value]
>SystemCacheInitialSize = value
>SystemCacheMaxSize = value
>Java Heap Init Size = value
>Java Heap Max Size = value
>
> The overall memory usage by all Ignite nodes on the host: value
> ---
>
> Records in […] are optional. If custom memory policy is not set or the
> default memory policy is overridden the output will miss some of the rows.
>
> As for systemCacheMaxSize, it should be show ONLY if the parameter was set
> explicitly by user code. Otherwise, the platform should be wise enough to
> instantiate it properly depending on the host memory usage.
>
> —
> Denis
>
> > On Aug 25, 2017, at 1:49 PM, Seliverstov Igor 
> wrote:
> >
> > The message without logging layout:
> >
> > Not enough memory for current process [required=29251MB,
> available=15942MB].
> > Please change MemoryConfiguration.systemCacheMaxSize and
> > MemoryConfiguration.defaultMemoryPolicySize to decrease memory allocated
> > for each node.
> >
> > Current settings:
> >   HeapInit=250MB
> >   HeapMax=3543MB
> >   DefaultMemoryPolicySize=12753MB
> >   SystemCacheInitialSize=40MB
> >   SystemCacheMaxSize=100MB
> >
> > Other ignite instanses on the server require: 12853MB
> >
> > I think it's make sense to describe what these numbers are consist of.
> >
> > We simple say which parameters have an impact on how much memory the
> > instance needs and their (parameters) actual values.
> >
> > Also we notice that more than one Ignite instance are ran on the server
> or
> > workstation and it also consumes memory.
> >
> > 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
> > dsetrak...@apache.org> написал:
> >
> >> Igor, what is this flood of WARN messaging coming after the text? Are we
> >> really going to print this whole thing out?
> >>
> >> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor  >
> >> wrote:
> >>
> >>> This message appears on topology change in case the available memory is
> >>> exceeded
> >>>
> >>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor :
> >>>
>  An example of current impl:
> 
> 
>  [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
>  GridHomePathSelfTest0%][GridDiscoveryManager]
>  [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
>  GridHomePathSelfTest0%][GridDiscoveryManager] Not enough memory for
>  current process [required=29251MB, available=15942MB].
>  [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
>  GridHomePathSelfTest0%][GridDiscoveryManager] Please change
>  MemoryConfiguration.systemCacheMaxSize and MemoryConfiguration.
> >>> defaultMemoryPolicySize
>  to decrease memory allocated for each node.
>  [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
>  GridHomePathSelfTest0%][GridDiscoveryManager]
>  [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
>  GridHomePathSelfTest0%][GridDiscoveryManager] Current settings:
>  [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
>  GridHomePathSelfTest0%][GridDiscoveryManager]   HeapInit=250MB
>  [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
>  GridHomePathSelfTest0%][GridDiscoveryManager]   HeapMax=3543MB
>  [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
>  GridHomePathSelfTest0%][GridDiscoveryManager]
> >> DefaultMemoryPolicySize=
>  12753MB
>  [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
>  GridHomePathSelfTest0%][GridDiscoveryManager]
>  SystemCacheInitialSize=40MB
>  [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
>  GridHomePathSelfTest0%][GridDiscoveryManager]
> >> SystemCacheMaxSize=100MB
>  [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
>  GridHomePathSelfTest0%][GridDiscoveryManager]
>  [2017-08-25 19:44:37,741][WARN 

Re: [DISCUSSION] Urgent Ignite bug fix release

[Denis Magda]

Igor,

Let me suggest this format. 

-
Excessive memory usage by Ignite node process (performance may drop) 
[requested=29251MB, available=15942MB]

Please tune the following settings:
  [MemoryConfiguration.defaultMemoryPolicySize = suggested value]
  MemoryConfiguration.systemCacheMaxSize = suggested value
  [MemoryPolicyConfiguration.maxSize for {policy_name_1} = suggested value]
  [MemoryPolicyConfiguration.maxSize for {policy_name_2} = suggested value]

Current settings:
   [DefaultMemoryPolicySize = value]
   [{policy_name_1} size = value]
   [{policy_name_1} size = value]
   SystemCacheInitialSize = value
   SystemCacheMaxSize = value
   Java Heap Init Size = value
   Java Heap Max Size = value

The overall memory usage by all Ignite nodes on the host: value
---

Records in […] are optional. If custom memory policy is not set or the default 
memory policy is overridden the output will miss some of the rows.

As for systemCacheMaxSize, it should be show ONLY if the parameter was set 
explicitly by user code. Otherwise, the platform should be wise enough to 
instantiate it properly depending on the host memory usage.

—
Denis
   
> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor  wrote:
> 
> The message without logging layout:
> 
> Not enough memory for current process [required=29251MB, available=15942MB].
> Please change MemoryConfiguration.systemCacheMaxSize and
> MemoryConfiguration.defaultMemoryPolicySize to decrease memory allocated
> for each node.
> 
> Current settings:
>   HeapInit=250MB
>   HeapMax=3543MB
>   DefaultMemoryPolicySize=12753MB
>   SystemCacheInitialSize=40MB
>   SystemCacheMaxSize=100MB
> 
> Other ignite instanses on the server require: 12853MB
> 
> I think it's make sense to describe what these numbers are consist of.
> 
> We simple say which parameters have an impact on how much memory the
> instance needs and their (parameters) actual values.
> 
> Also we notice that more than one Ignite instance are ran on the server or
> workstation and it also consumes memory.
> 
> 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
> dsetrak...@apache.org> написал:
> 
>> Igor, what is this flood of WARN messaging coming after the text? Are we
>> really going to print this whole thing out?
>> 
>> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor 
>> wrote:
>> 
>>> This message appears on topology change in case the available memory is
>>> exceeded
>>> 
>>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor :
>>> 
 An example of current impl:
 
 
 [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
 GridHomePathSelfTest0%][GridDiscoveryManager]
 [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
 GridHomePathSelfTest0%][GridDiscoveryManager] Not enough memory for
 current process [required=29251MB, available=15942MB].
 [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
 GridHomePathSelfTest0%][GridDiscoveryManager] Please change
 MemoryConfiguration.systemCacheMaxSize and MemoryConfiguration.
>>> defaultMemoryPolicySize
 to decrease memory allocated for each node.
 [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
 GridHomePathSelfTest0%][GridDiscoveryManager]
 [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
 GridHomePathSelfTest0%][GridDiscoveryManager] Current settings:
 [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
 GridHomePathSelfTest0%][GridDiscoveryManager]   HeapInit=250MB
 [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
 GridHomePathSelfTest0%][GridDiscoveryManager]   HeapMax=3543MB
 [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
 GridHomePathSelfTest0%][GridDiscoveryManager]
>> DefaultMemoryPolicySize=
 12753MB
 [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
 GridHomePathSelfTest0%][GridDiscoveryManager]
 SystemCacheInitialSize=40MB
 [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
 GridHomePathSelfTest0%][GridDiscoveryManager]
>> SystemCacheMaxSize=100MB
 [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
 GridHomePathSelfTest0%][GridDiscoveryManager]
 [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
 GridHomePathSelfTest0%][GridDiscoveryManager] Other ignite instanses
>> on
 the server require: 12853MB
 [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
 GridHomePathSelfTest0%][GridDiscoveryManager]
 
 
 2017-08-25 17:40 GMT+03:00 Sergey Kozlov :
 
> I suppose we should not forget JVM heap size and suggest to reduce
>> both
> options
> 
> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
>>> dsetrak...@apache.org
>> 
> 

Re: [DISCUSS] Ignite Update Checker

[Dmitriy Setrakyan]

Cos, Raul,

Thanks for the feedback. I completely agree about Maven Central being a 3rd
party repo (did not think about that initially). All your suggestions make
sense, but I would like to keep it as simple as possible, and so far
everything suggested required GIT dependencies and extra work.

How about Yakov's suggestion. We simply add a page to the Ignite website
which will have only the latest version. Every time a node starts, it
receives the latest version from the page and suggests that users upgrade
if needed.

Also, since we have GA enabled for the website, we can track how many times
this page was accessed, which will be equal to the number of starts. This
way, the counter information is tracked and monitored by the Ignite PMC.

This approach looks pretty innocent to me and everything is kept and
managed within Apache.

Thoughts?

D.


On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik  wrote:

> I agree with Raul.
> - providing a ping-back address to a 3rd party might be frown upon by some.
>   And might have a consequences like stats collection about users'
>   infrastructure.
> - checking an ASF git-repo is easy and won't download any binary data:
>   everything is clear text and could be easily monitored by any of the
> network
>   diagnostic tools, shall it be required. But it involves a bit of the
> release
>   discipline.
> - the binary data download in the runtime is my main concern. This is the
>   vector of MMA. What if someone gains the control over the repository and
>   replaces the file with some malicious content.
>
> As for the particular mechanism: IIRC Ignite used to make a call to an
> external script to check upon the atest software version available for
> download. In the past, the endpoint was running on a 3rd party server, I
> believe the best way would be to put this script on ASF infra and have the
> "update checker" running in a completely controlled environment. Actually,
> I
> recall we had this very discussion during the Incubation; I can probably
> dig
> out the corresponding thread.
>
> Thoughts?
>   Cok
>
> On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
> > Hey guys
> >
> > In my opinion, maven.org is still owned by a third party (Sonatype). We
> > don't know what kind of data analysis or intelligence extraction they
> run.
> >
> > If Ignite servers all over the world were hitting maven.org periodically
> > asking for an Ignite Maven artifact, it gives Sonatype a clear indication
> > of who is running an Ignite server.
> >
> > They could reverse lookup the IP address and find out what corporation it
> > is.
> >
> > How about having Ignite check the ASF Git directly?
> >
> > We could use the Git ssh API, but that would require a new dependency,
> > which I advise against.
> >
> > Alternatively, we could have it scrape this HTML for new Git tags:
> > https://git-wip-us.apache.org/repos/asf?p=ignite.git
> >
> > Another option is to place a txt file in the root of the master branch
> (e.g
> > LATEST), containing a list of the latest GA versions for each major
> version
> > line (1.x, 2.x).
> >
> > I would advocate this last option, but it requires somebody remembering
> to
> > update the file with every release, unless we automate it with a Maven
> > plugin.
> >
> > Hope that helps!
> >
> >
> > On 24 Aug 2017 19:37, "Denis Magda"  wrote:
> >
> > I see nothing wrong with this approach.
> >
> > Cos, Roman, Raul, as Apache veterans, what do you think? Is it good to
> go?
> >
> > —
> > Denis
> >
> > > On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan  >
> > wrote:
> > >
> > > Is everyone OK with this approach? Should I file a ticket on it?
> > >
> > > On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan <
> dsetrak...@apache.org>
> > > wrote:
> > >
> > >> Igniters,
> > >>
> > >> There has been lots of talk of proposals about various usage metrics
> for
> > >> Ignite and nothing came of it. I would like to resurrect the topic and
> > >> propose something very simple and non-intrusive.
> > >>
> > >> 1. Update Checker
> > >> The main purpose of the update checker is not to collect metrics, but
> to
> > >> notify users about a new version of Ignite by accessing maven.org and
> > >> getting the version out of the metadata file:
> > >> http://repo2.maven.org/maven2/org/apache/ignite/ignite-core/
> > >> maven-metadata.xml
> > >>
> > >> This way we do not send any information anywhere and, at the same
> time,
> > >> urge our users to download and start using the latest version of
> Ignite.
> > >>
> > >> 2. Startup Counter
> > >> This piece is optional, but we can also get an insight in how many
> times
> > a
> > >> certain Ignite release gets started. This is just a cool metric for
> the
> > >> community to gauge the project popularity. You can think of it as of a
> > page
> > >> visit counter shown on many websites. We can even decide to display
> this
> > >> counter on the Ignite website as well.
> > >>
> > >> To 

Re: [DISCUSSION] Urgent Ignite bug fix release

[Seliverstov Igor]

The message without logging layout:

Not enough memory for current process [required=29251MB, available=15942MB].
Please change MemoryConfiguration.systemCacheMaxSize and
MemoryConfiguration.defaultMemoryPolicySize to decrease memory allocated
for each node.

Current settings:
   HeapInit=250MB
   HeapMax=3543MB
   DefaultMemoryPolicySize=12753MB
   SystemCacheInitialSize=40MB
   SystemCacheMaxSize=100MB

Other ignite instanses on the server require: 12853MB

I think it's make sense to describe what these numbers are consist of.

We simple say which parameters have an impact on how much memory the
instance needs and their (parameters) actual values.

Also we notice that more than one Ignite instance are ran on the server or
workstation and it also consumes memory.

25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" <
dsetrak...@apache.org> написал:

> Igor, what is this flood of WARN messaging coming after the text? Are we
> really going to print this whole thing out?
>
> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor 
> wrote:
>
> > This message appears on topology change in case the available memory is
> > exceeded
> >
> > 2017-08-25 19:47 GMT+03:00 Seliverstov Igor :
> >
> > > An example of current impl:
> > >
> > >
> > > [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager]
> > > [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager] Not enough memory for
> > > current process [required=29251MB, available=15942MB].
> > > [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager] Please change
> > > MemoryConfiguration.systemCacheMaxSize and MemoryConfiguration.
> > defaultMemoryPolicySize
> > > to decrease memory allocated for each node.
> > > [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager]
> > > [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager] Current settings:
> > > [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager]   HeapInit=250MB
> > > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager]   HeapMax=3543MB
> > > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager]
>  DefaultMemoryPolicySize=
> > > 12753MB
> > > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager]
> > > SystemCacheInitialSize=40MB
> > > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager]
>  SystemCacheMaxSize=100MB
> > > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager]
> > > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager] Other ignite instanses
> on
> > > the server require: 12853MB
> > > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > > GridHomePathSelfTest0%][GridDiscoveryManager]
> > >
> > >
> > > 2017-08-25 17:40 GMT+03:00 Sergey Kozlov :
> > >
> > >> I suppose we should not forget JVM heap size and suggest to reduce
> both
> > >> options
> > >>
> > >> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
> > dsetrak...@apache.org
> > >> >
> > >> wrote:
> > >>
> > >> > Igor, I would change the message. How about this:
> > >> >
> > >> > Required RAM size is larger than total physical memory available for
> > OS.
> > >> > > Please change MemoryConfiguration.WhichProperty and
> > >> > > MemoryPolicyConfiguration.WhichProperty to decrease memory
> > allocated
> > >> for
> > >> > > each node.
> > >> >
> > >> >
> > >> > Also, can we calculate what the memory size allocated for each node
> > >> should
> > >> > be? In that case we should suggest it.
> > >> >
> > >> > D.
> > >> >
> > >> >
> > >> > On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <
> > gvvinbl...@gmail.com
> > >> >
> > >> > wrote:
> > >> >
> > >> > > What do you, guys think about next warning?
> > >> > >
> > >> > > [2017-08-25 17:17:04,718][INFO
> > >> > > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> > >> hSelfTest0]
> > >> > > System cache's MemoryPolicy size is configured to 40 MB. Use
> > >> > > MemoryConfiguration.systemCacheMemorySize property to change the
> > >> > setting.
> > >> > > [2017-08-25 17:17:04,718][WARN
> > >> > > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> > >> hSelfTest0]
> > >> > >
> > >> > > >>> Required RAM size is larger than total physical memory
> available
> > >> for
> > >> > > OS.
> > >> > > >>> Check your configuration to avoid swap 

SQL fails silently when Set is used as argument

[Denis Magda]

Igniters,

One of our users reported this bug to me:
https://issues.apache.org/jira/browse/IGNITE-6190 


SQL gurus please take a look.

—
Denis

[jira] [Created] (IGNITE-6190) SQL query fails silently if Set is passed as a parameter

[Denis Magda (JIRA)]

Denis Magda created IGNITE-6190:
---

 Summary: SQL query fails silently if Set is passed as a parameter
 Key: IGNITE-6190
 URL: https://issues.apache.org/jira/browse/IGNITE-6190
 Project: Ignite
  Issue Type: Bug
Reporter: Denis Magda


Seems like the SqlQuery API does not like {{Set}} as the input parameter. 
While this query doesn't work (the Set is used as an input):

{code}
public Map getAccountsForLe(Set leId) {
SqlQuery query =
new SqlQuery(Account.class, "from Account join 
table(id varchar = ?) i on Account.clientLegalEntityId = i.id")
.setArgs(leId);

Map results = new HashMap<>();
_cache.query(query).getAll().stream().forEach(e -> results.put(e.getKey(), 
e.getValue()));
return results;
}
{code}

This one works well (the Set is converted to Array explicitly):

{code}
public Map getAccountsForLe(Set leId) {
SqlQuery query =
new SqlQuery(Account.class, "from Account join 
table(id varchar = ?) i on Account.clientLegalEntityId = i.id")
.setArgs(leId.toArray());

Map results = new HashMap<>();
_cache.query(query).getAll().stream().forEach(e -> results.put(e.getKey(), 
e.getValue()));
return results;
}
{code}

The fact that it fails silently is an issue. IMHO there should be some 
validation to alert the calling code that the type specified is not valid or 
the set has to be transformed to the array on the fly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSSION] Urgent Ignite bug fix release

[Dmitriy Setrakyan]

Igor, what is this flood of WARN messaging coming after the text? Are we
really going to print this whole thing out?

On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor 
wrote:

> This message appears on topology change in case the available memory is
> exceeded
>
> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor :
>
> > An example of current impl:
> >
> >
> > [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > GridHomePathSelfTest0%][GridDiscoveryManager]
> > [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > GridHomePathSelfTest0%][GridDiscoveryManager] Not enough memory for
> > current process [required=29251MB, available=15942MB].
> > [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > GridHomePathSelfTest0%][GridDiscoveryManager] Please change
> > MemoryConfiguration.systemCacheMaxSize and MemoryConfiguration.
> defaultMemoryPolicySize
> > to decrease memory allocated for each node.
> > [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > GridHomePathSelfTest0%][GridDiscoveryManager]
> > [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > GridHomePathSelfTest0%][GridDiscoveryManager] Current settings:
> > [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> > GridHomePathSelfTest0%][GridDiscoveryManager]   HeapInit=250MB
> > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > GridHomePathSelfTest0%][GridDiscoveryManager]   HeapMax=3543MB
> > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > GridHomePathSelfTest0%][GridDiscoveryManager]   DefaultMemoryPolicySize=
> > 12753MB
> > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > GridHomePathSelfTest0%][GridDiscoveryManager]
> > SystemCacheInitialSize=40MB
> > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > GridHomePathSelfTest0%][GridDiscoveryManager]   SystemCacheMaxSize=100MB
> > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > GridHomePathSelfTest0%][GridDiscoveryManager]
> > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > GridHomePathSelfTest0%][GridDiscoveryManager] Other ignite instanses on
> > the server require: 12853MB
> > [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> > GridHomePathSelfTest0%][GridDiscoveryManager]
> >
> >
> > 2017-08-25 17:40 GMT+03:00 Sergey Kozlov :
> >
> >> I suppose we should not forget JVM heap size and suggest to reduce both
> >> options
> >>
> >> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan <
> dsetrak...@apache.org
> >> >
> >> wrote:
> >>
> >> > Igor, I would change the message. How about this:
> >> >
> >> > Required RAM size is larger than total physical memory available for
> OS.
> >> > > Please change MemoryConfiguration.WhichProperty and
> >> > > MemoryPolicyConfiguration.WhichProperty to decrease memory
> allocated
> >> for
> >> > > each node.
> >> >
> >> >
> >> > Also, can we calculate what the memory size allocated for each node
> >> should
> >> > be? In that case we should suggest it.
> >> >
> >> > D.
> >> >
> >> >
> >> > On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor <
> gvvinbl...@gmail.com
> >> >
> >> > wrote:
> >> >
> >> > > What do you, guys think about next warning?
> >> > >
> >> > > [2017-08-25 17:17:04,718][INFO
> >> > > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> >> hSelfTest0]
> >> > > System cache's MemoryPolicy size is configured to 40 MB. Use
> >> > > MemoryConfiguration.systemCacheMemorySize property to change the
> >> > setting.
> >> > > [2017-08-25 17:17:04,718][WARN
> >> > > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> >> hSelfTest0]
> >> > >
> >> > > >>> Required RAM size is larger than total physical memory available
> >> for
> >> > > OS.
> >> > > >>> Check your configuration to avoid swap partition usage.
> >> > > >>> Use MemoryConfiguration and MemoryPolicyConfiguration to change
> >> the
> >> > > settings.
> >> > > >>> Physical memory [required=16397MB, available=15942MB]
> >> > >
> >> > > [2017-08-25 17:17:04,726][WARN
> >> > > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> >> hSelfTest0]
> >> > > Peer class loading is enabled (disable it in production for
> >> performance
> >> > and
> >> > > deployment consistency reasons)
> >> > > [2017-08-25 17:17:04,726][INFO
> >> > > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> >> hSelfTest0]
> >> > > Configured caches [in 'sysMemPlc' memoryPolicy:
> ['ignite-sys-cache']]
> >> > > [2017-08-25 17:17:04,731][INFO
> >> > > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
> >> hSelfTest0]
> >> > > 3-rd party licenses can be found at:
> >> > > /home/gvvinblade/projects/ignite/incubator-ignite/libs/licenses
> >> > >
> >> > >
> >> > > 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov :
> >> > >
> >> > > > Agree, let's release new version including 

Re: [DISCUSS] Ignite Update Checker

[Konstantin Boudnik]

I agree with Raul.
- providing a ping-back address to a 3rd party might be frown upon by some.
  And might have a consequences like stats collection about users'
  infrastructure.
- checking an ASF git-repo is easy and won't download any binary data:
  everything is clear text and could be easily monitored by any of the network
  diagnostic tools, shall it be required. But it involves a bit of the release
  discipline.
- the binary data download in the runtime is my main concern. This is the
  vector of MMA. What if someone gains the control over the repository and
  replaces the file with some malicious content.

As for the particular mechanism: IIRC Ignite used to make a call to an
external script to check upon the atest software version available for
download. In the past, the endpoint was running on a 3rd party server, I
believe the best way would be to put this script on ASF infra and have the
"update checker" running in a completely controlled environment. Actually, I
recall we had this very discussion during the Incubation; I can probably dig
out the corresponding thread.

Thoughts?
  Cok

On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
> Hey guys
> 
> In my opinion, maven.org is still owned by a third party (Sonatype). We
> don't know what kind of data analysis or intelligence extraction they run.
> 
> If Ignite servers all over the world were hitting maven.org periodically
> asking for an Ignite Maven artifact, it gives Sonatype a clear indication
> of who is running an Ignite server.
> 
> They could reverse lookup the IP address and find out what corporation it
> is.
> 
> How about having Ignite check the ASF Git directly?
> 
> We could use the Git ssh API, but that would require a new dependency,
> which I advise against.
> 
> Alternatively, we could have it scrape this HTML for new Git tags:
> https://git-wip-us.apache.org/repos/asf?p=ignite.git
> 
> Another option is to place a txt file in the root of the master branch (e.g
> LATEST), containing a list of the latest GA versions for each major version
> line (1.x, 2.x).
> 
> I would advocate this last option, but it requires somebody remembering to
> update the file with every release, unless we automate it with a Maven
> plugin.
> 
> Hope that helps!
> 
> 
> On 24 Aug 2017 19:37, "Denis Magda"  wrote:
> 
> I see nothing wrong with this approach.
> 
> Cos, Roman, Raul, as Apache veterans, what do you think? Is it good to go?
> 
> —
> Denis
> 
> > On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan 
> wrote:
> >
> > Is everyone OK with this approach? Should I file a ticket on it?
> >
> > On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan 
> > wrote:
> >
> >> Igniters,
> >>
> >> There has been lots of talk of proposals about various usage metrics for
> >> Ignite and nothing came of it. I would like to resurrect the topic and
> >> propose something very simple and non-intrusive.
> >>
> >> 1. Update Checker
> >> The main purpose of the update checker is not to collect metrics, but to
> >> notify users about a new version of Ignite by accessing maven.org and
> >> getting the version out of the metadata file:
> >> http://repo2.maven.org/maven2/org/apache/ignite/ignite-core/
> >> maven-metadata.xml
> >>
> >> This way we do not send any information anywhere and, at the same time,
> >> urge our users to download and start using the latest version of Ignite.
> >>
> >> 2. Startup Counter
> >> This piece is optional, but we can also get an insight in how many times
> a
> >> certain Ignite release gets started. This is just a cool metric for the
> >> community to gauge the project popularity. You can think of it as of a
> page
> >> visit counter shown on many websites. We can even decide to display this
> >> counter on the Ignite website as well.
> >>
> >> To do this, we can simply add a JAR in maven for every release, e.g.
> >> ignite-start-counter.jar, which will contain only 1 byte. Every time an
> >> Ignite node starts, it will download this JAR in the background. Then we
> >> will be able to view the number of the total downloads for this JAR in
> >> Maven Central, which is essentially the number of starts of Ignite nodes.
> >>
> >> *Note that neither of the above suggestions require Ignite to send or
> >> track any user information whatsoever.*
> >>
> >> Please reply suggesting weather you are OK with this approach.
> >>
> >> D.
> >>



signature.asc
Description: Digital signature

Re: [DISCUSSION] Urgent Ignite bug fix release

[Seliverstov Igor]

This message appears on topology change in case the available memory is
exceeded

2017-08-25 19:47 GMT+03:00 Seliverstov Igor :

> An example of current impl:
>
>
> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager]
> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager] Not enough memory for
> current process [required=29251MB, available=15942MB].
> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager] Please change
> MemoryConfiguration.systemCacheMaxSize and 
> MemoryConfiguration.defaultMemoryPolicySize
> to decrease memory allocated for each node.
> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager]
> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager] Current settings:
> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager]   HeapInit=250MB
> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager]   HeapMax=3543MB
> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager]   DefaultMemoryPolicySize=
> 12753MB
> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager]
> SystemCacheInitialSize=40MB
> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager]   SystemCacheMaxSize=100MB
> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager]
> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager] Other ignite instanses on
> the server require: 12853MB
> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29%internal.
> GridHomePathSelfTest0%][GridDiscoveryManager]
>
>
> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov :
>
>> I suppose we should not forget JVM heap size and suggest to reduce both
>> options
>>
>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan > >
>> wrote:
>>
>> > Igor, I would change the message. How about this:
>> >
>> > Required RAM size is larger than total physical memory available for OS.
>> > > Please change MemoryConfiguration.WhichProperty and
>> > > MemoryPolicyConfiguration.WhichProperty to decrease memory allocated
>> for
>> > > each node.
>> >
>> >
>> > Also, can we calculate what the memory size allocated for each node
>> should
>> > be? In that case we should suggest it.
>> >
>> > D.
>> >
>> >
>> > On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor > >
>> > wrote:
>> >
>> > > What do you, guys think about next warning?
>> > >
>> > > [2017-08-25 17:17:04,718][INFO
>> > > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
>> hSelfTest0]
>> > > System cache's MemoryPolicy size is configured to 40 MB. Use
>> > > MemoryConfiguration.systemCacheMemorySize property to change the
>> > setting.
>> > > [2017-08-25 17:17:04,718][WARN
>> > > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
>> hSelfTest0]
>> > >
>> > > >>> Required RAM size is larger than total physical memory available
>> for
>> > > OS.
>> > > >>> Check your configuration to avoid swap partition usage.
>> > > >>> Use MemoryConfiguration and MemoryPolicyConfiguration to change
>> the
>> > > settings.
>> > > >>> Physical memory [required=16397MB, available=15942MB]
>> > >
>> > > [2017-08-25 17:17:04,726][WARN
>> > > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
>> hSelfTest0]
>> > > Peer class loading is enabled (disable it in production for
>> performance
>> > and
>> > > deployment consistency reasons)
>> > > [2017-08-25 17:17:04,726][INFO
>> > > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
>> hSelfTest0]
>> > > Configured caches [in 'sysMemPlc' memoryPolicy: ['ignite-sys-cache']]
>> > > [2017-08-25 17:17:04,731][INFO
>> > > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePat
>> hSelfTest0]
>> > > 3-rd party licenses can be found at:
>> > > /home/gvvinblade/projects/ignite/incubator-ignite/libs/licenses
>> > >
>> > >
>> > > 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov :
>> > >
>> > > > Agree, let's release new version including tickets mentioned by
>> Denis
>> > and
>> > > > Ivan.
>> > > >
>> > > > --Yakov
>> > > >
>> > >
>> >
>>
>>
>>
>> --
>> Sergey Kozlov
>> GridGain Systems
>> www.gridgain.com
>>
>
>

Re: [DISCUSSION] Urgent Ignite bug fix release

[Seliverstov Igor]

An example of current impl:


[2017-08-25 19:44:37,740][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
[2017-08-25 19:44:37,740][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
Not enough memory for current process [required=29251MB, available=15942MB].
[2017-08-25 19:44:37,740][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
Please change MemoryConfiguration.systemCacheMaxSize and
MemoryConfiguration.defaultMemoryPolicySize to decrease memory allocated
for each node.
[2017-08-25 19:44:37,740][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
[2017-08-25 19:44:37,740][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
Current settings:
[2017-08-25 19:44:37,740][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
  HeapInit=250MB
[2017-08-25 19:44:37,741][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
  HeapMax=3543MB
[2017-08-25 19:44:37,741][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
  DefaultMemoryPolicySize=12753MB
[2017-08-25 19:44:37,741][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
  SystemCacheInitialSize=40MB
[2017-08-25 19:44:37,741][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
  SystemCacheMaxSize=100MB
[2017-08-25 19:44:37,741][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
[2017-08-25 19:44:37,741][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]
Other ignite instanses on the server require: 12853MB
[2017-08-25 19:44:37,741][WARN
][disco-event-worker-#29%internal.GridHomePathSelfTest0%][GridDiscoveryManager]


2017-08-25 17:40 GMT+03:00 Sergey Kozlov :

> I suppose we should not forget JVM heap size and suggest to reduce both
> options
>
> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan 
> wrote:
>
> > Igor, I would change the message. How about this:
> >
> > Required RAM size is larger than total physical memory available for OS.
> > > Please change MemoryConfiguration.WhichProperty and
> > > MemoryPolicyConfiguration.WhichProperty to decrease memory allocated
> for
> > > each node.
> >
> >
> > Also, can we calculate what the memory size allocated for each node
> should
> > be? In that case we should suggest it.
> >
> > D.
> >
> >
> > On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor 
> > wrote:
> >
> > > What do you, guys think about next warning?
> > >
> > > [2017-08-25 17:17:04,718][INFO
> > > ][test-runner-#1%internal.GridHomePathSelfTest%][
> GridHomePathSelfTest0]
> > > System cache's MemoryPolicy size is configured to 40 MB. Use
> > > MemoryConfiguration.systemCacheMemorySize property to change the
> > setting.
> > > [2017-08-25 17:17:04,718][WARN
> > > ][test-runner-#1%internal.GridHomePathSelfTest%][
> GridHomePathSelfTest0]
> > >
> > > >>> Required RAM size is larger than total physical memory available
> for
> > > OS.
> > > >>> Check your configuration to avoid swap partition usage.
> > > >>> Use MemoryConfiguration and MemoryPolicyConfiguration to change the
> > > settings.
> > > >>> Physical memory [required=16397MB, available=15942MB]
> > >
> > > [2017-08-25 17:17:04,726][WARN
> > > ][test-runner-#1%internal.GridHomePathSelfTest%][
> GridHomePathSelfTest0]
> > > Peer class loading is enabled (disable it in production for performance
> > and
> > > deployment consistency reasons)
> > > [2017-08-25 17:17:04,726][INFO
> > > ][test-runner-#1%internal.GridHomePathSelfTest%][
> GridHomePathSelfTest0]
> > > Configured caches [in 'sysMemPlc' memoryPolicy: ['ignite-sys-cache']]
> > > [2017-08-25 17:17:04,731][INFO
> > > ][test-runner-#1%internal.GridHomePathSelfTest%][
> GridHomePathSelfTest0]
> > > 3-rd party licenses can be found at:
> > > /home/gvvinblade/projects/ignite/incubator-ignite/libs/licenses
> > >
> > >
> > > 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov :
> > >
> > > > Agree, let's release new version including tickets mentioned by Denis
> > and
> > > > Ivan.
> > > >
> > > > --Yakov
> > > >
> > >
> >
>
>
>
> --
> Sergey Kozlov
> GridGain Systems
> www.gridgain.com
>

[GitHub] ignite pull request #2520: IGNITE-6183 Make "Node crashed in the middle of c...

[glukos]

GitHub user glukos opened a pull request:

https://github.com/apache/ignite/pull/2520

IGNITE-6183 Make "Node crashed in the middle of checkpoint" message s…



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-6183

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2520.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2520


commit d272bbfbd53cb180ae898e94e986180734c10b94
Author: Ivan Rakov 
Date:   2017-08-25T16:24:14Z

IGNITE-6183 Make "Node crashed in the middle of checkpoint" message softer 
and more informative




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Adding sqlline tool to Apache Ignite project

[Denis Magda]

Hi Ilya,

Thanks for the clarification! Referring to the page shared by you [1] if we 
need to get author’s consent in a written form:

A permissive license similar to the BSD 2-Clause License, but with a 3rd clause 
that prohibits others from using the name of the project or its contributors to 
promote derived products without written consent.

[1] https://github.com/julianhyde/sqlline/blob/master/LICENSE 


I’ll reach out the author requesting the permit.

—
Denis

> On Aug 25, 2017, at 9:12 AM, Ilya Kasnacheev  
> wrote:
> 
> Hi Denis,
> 
> There are two kinds of BSD license, 3-clause and 4-clause. The difference
> between them is advertising clause:
> 
> "3. All advertising materials mentioning features or use of this software
> must display the following acknowledgement: This product includes software
> developed by the University of California, Berkeley and its contributors."
> 
> which is causing problems with other licenses and which sqlline's license
> doesn't contain:
> https://github.com/julianhyde/sqlline/blob/master/LICENSE
> So it should be all good for inclusion.
> 
> -- 
> Ilya Kasnacheev.
> 
> 2017-08-25 18:48 GMT+03:00 Denis Magda :
> 
>> Hi Ilya,
>> 
>> It will be a useful addition to Ignite. I would include the tool in one of
>> the nearest releases. It’s license is BSD-3 which is compatible with ours.
>> However, there is a note on ASF side saying that BSD licenses can be
>> included without advertising clause [1]. What does it mean for us? Can we
>> document the tool and mention on our site?
>> 
>> [1] https://www.apache.org/legal/resolved.html#category-a <
>> https://www.apache.org/legal/resolved.html#category-a>
>> 
>> —
>> Denis
>> 
>>> On Aug 25, 2017, at 8:32 AM, Ilya Suntsov  wrote:
>>> 
>>> Hi Igniters,
>>> 
>>> I've found one command line tool that can be used for execute queries
>>> (DDL/DML) with data from Apache Ignite grid via
>>> *org.apache.ignite.IgniteJdbcThinDriver*.
>>> 
>>> Here you can find overview of internal commands of this tool:
>>> https://cwiki.apache.org/confluence/display/IGNITE/Overview+sqlline+tool
>>> 
>>> The most problems with sqlline relate with some issues from our side or
>>> some unsupported features.
>>> 
>>> Would be nice to add tool like this (sqlline or something else) in our
>>> project.
>>> 
>>> Have you any objections or other thoughts?
>>> 
>>> 
>>> --
>>> Ilya Suntsov
>> 
>> 

Re: Adding sqlline tool to Apache Ignite project

[Ilya Kasnacheev]

Hi Denis,

There are two kinds of BSD license, 3-clause and 4-clause. The difference
between them is advertising clause:

"3. All advertising materials mentioning features or use of this software
must display the following acknowledgement: This product includes software
developed by the University of California, Berkeley and its contributors."

which is causing problems with other licenses and which sqlline's license
doesn't contain:
https://github.com/julianhyde/sqlline/blob/master/LICENSE
So it should be all good for inclusion.

-- 
Ilya Kasnacheev.

2017-08-25 18:48 GMT+03:00 Denis Magda :

> Hi Ilya,
>
> It will be a useful addition to Ignite. I would include the tool in one of
> the nearest releases. It’s license is BSD-3 which is compatible with ours.
> However, there is a note on ASF side saying that BSD licenses can be
> included without advertising clause [1]. What does it mean for us? Can we
> document the tool and mention on our site?
>
> [1] https://www.apache.org/legal/resolved.html#category-a <
> https://www.apache.org/legal/resolved.html#category-a>
>
> —
> Denis
>
> > On Aug 25, 2017, at 8:32 AM, Ilya Suntsov  wrote:
> >
> > Hi Igniters,
> >
> > I've found one command line tool that can be used for execute queries
> > (DDL/DML) with data from Apache Ignite grid via
> > *org.apache.ignite.IgniteJdbcThinDriver*.
> >
> > Here you can find overview of internal commands of this tool:
> > https://cwiki.apache.org/confluence/display/IGNITE/Overview+sqlline+tool
> >
> > The most problems with sqlline relate with some issues from our side or
> > some unsupported features.
> >
> > Would be nice to add tool like this (sqlline or something else) in our
> > project.
> >
> > Have you any objections or other thoughts?
> >
> >
> > --
> > Ilya Suntsov
>
>

Re: Testing Ignite Applications Locally

[Denis Magda]

Yasha, 

Sure, I’ll help from the documentation point but will need raw material from 
you, guys - test snippets, essential configuration parameters, etc.

—
Denis

> On Aug 25, 2017, at 8:30 AM, Yakov Zhdanov  wrote:
> 
> Guys,
> 
> I want to discuss the subject again. It is pretty vivid that having wide set 
> of local unit and simple integration tests most likely help to avoid many 
> failures and bugs when going to server environment.
> 
> I participated in many POC and I can say for sure - if developers are not 
> implementing local tests then their application is broken. This is true for 
> the entire industry. Why does anyone think that Ignite and distributed 
> systems in general are exceptions here? Complexity added by distributed 
> nature probably needs local tests even more.
> 
> So, what Ignite already offers here and what can be done further?
> 
> 1. Ignite offers ability to emulate cluster and even many cluster in a single 
> VM. Let's create a page on readme.io  explaining how to 
> start topologies in a single VM and provide couple examples of unit tests for 
> cache operations and, for example, queries. Denis Magda, can you help? (Yes, 
> we don't have the page explaining how to test Ignite locally!)
> 
> 2. Ignite has a large and rich set of tests in its code base. We can provide 
> the link on the page at p1.
> 
> 3. Let's create maven archetype for Ignite. So, executing the command [1] 
> will bring me inited project with valid poms, sample batch scripts, sample 
> Ignite configs, sample logger configuration and test sources folder 
> containing several JUnits (!!).
> 
> [1]  mvn archetype:generate \
>   -DinteractiveMode=false \
>   -DarchetypeGroupId=org.apache.ignite \
>   -DarchetypeArtifactId=ignite-app-archetype \
>   -DgroupId=org.sample \
>   -DartifactId=sampleapp \
>   -Dversion=1.0
> 
> Please share your thoughts and we can file tickets to start moving.
> 
> --Yakov

Re: Adding sqlline tool to Apache Ignite project

[Denis Magda]

Hi Ilya,

It will be a useful addition to Ignite. I would include the tool in one of the 
nearest releases. It’s license is BSD-3 which is compatible with ours. However, 
there is a note on ASF side saying that BSD licenses can be included without 
advertising clause [1]. What does it mean for us? Can we document the tool and 
mention on our site?

[1] https://www.apache.org/legal/resolved.html#category-a 


—
Denis

> On Aug 25, 2017, at 8:32 AM, Ilya Suntsov  wrote:
> 
> Hi Igniters,
> 
> I've found one command line tool that can be used for execute queries
> (DDL/DML) with data from Apache Ignite grid via
> *org.apache.ignite.IgniteJdbcThinDriver*.
> 
> Here you can find overview of internal commands of this tool:
> https://cwiki.apache.org/confluence/display/IGNITE/Overview+sqlline+tool
> 
> The most problems with sqlline relate with some issues from our side or
> some unsupported features.
> 
> Would be nice to add tool like this (sqlline or something else) in our
> project.
> 
> Have you any objections or other thoughts?
> 
> 
> -- 
> Ilya Suntsov

[GitHub] ignite pull request #2519: Ignite 6124 last ver fix

[sergey-chugunov-1985]

GitHub user sergey-chugunov-1985 opened a pull request:

https://github.com/apache/ignite/pull/2519

Ignite 6124 last ver fix



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-6124-lastVer-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2519.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2519


commit 1e08c3fb5c02ec8acafd71b50b6ad3b749259f1a
Author: Andrey V. Mashenkov 
Date:   2017-07-31T11:14:56Z

IGNITE-4800: Lucene query may fails with NPE. This closes #2315.

commit 3fdf453e89a7bd76dff6b6d0646e3821ea3921d5
Author: Andrey V. Mashenkov 
Date:   2017-07-31T14:32:12Z

IGNITE-4800: Lucene query may fails with NPE.
Test fixed.

commit e255a564985a12113984ec02f15a4443495b8ffc
Author: Nikolay Izhikov 
Date:   2017-08-02T08:52:44Z

ignite-5712 Context switching for optimistic transactions

commit 772d462b68c7de8517d1f61e2e05ec8eefb18eac
Author: Alexey Kuznetsov 
Date:   2017-08-03T04:55:15Z

Merge branch ignite-2.1.3 into ignite-2.1.4

commit 0f3b7ca25313083e4dc35e7842931a655abd
Author: tledkov-gridgain 
Date:   2017-08-04T08:46:14Z

IGNITE-5126: Batch support for this JDBC driver. This closes #2162.

commit d1a74a4be8744528e6ed23706174041ddb0f2618
Author: devozerov 
Date:   2017-08-04T09:04:38Z

Merge remote-tracking branch 'upstream/ignite-2.1.4' into ignite-2.1.4

commit 0b3a9a7176f5ae44a96ecf700c8147193dfbf064
Author: Igor Sapego 
Date:   2017-08-04T10:18:00Z

IGNITE-5923: ODBC: SQLGetTypeInfo now works with SQL_ALL_TYPES

(cherry picked from commit 48c914d)

commit 4e0385fbc0f50548f2da3407fdfdfe939b463c67
Author: Igor Sapego 
Date:   2017-08-04T15:34:27Z

IGNITE-5939: ODBC: SQLColAttributes now works with legacy attribute codes.

(cherry picked from commit 70ffa2c)

commit 4f02504475fd1e5cc3b9f4754856e44d20fdc1cb
Author: Alexey Kuznetsov 
Date:   2017-08-07T02:41:22Z

Merge branch ignite-2.1.3 into ignite-2.1.4.

commit b093afb8231135a4904f5fffd62f5b4c332f1d47
Author: tledkov-gridgain 
Date:   2017-08-08T09:05:36Z

IGNITE-5211: Added new constructor: QueryEntity(Class keyCls, Class 
valCls). This closes #2371. This closes #2388. This closes #2407.

commit 879f19106b22e66d5f6ea94424d961d049397410
Author: devozerov 
Date:   2017-08-08T12:16:58Z

IGNITE-5982: GridMapQueryExecutor was split into several pieces.

commit 7c77b869bc3efdf19e53cc2b064f4290fd73e2b2
Author: devozerov 
Date:   2017-08-09T08:47:58Z

IGNITE-5993: Removed unused SQL-related classes and methods (old tree 
index, snapshots, etc). This closes #2414.

commit ab18fdfcc4f6db1e54fb1f3b68ba7fbc31a7f6e7
Author: Andrey V. Mashenkov 
Date:   2017-07-14T17:14:47Z

IGNITE-5452: GridTimeoutProcessor can hang on stop. This closes #2279.

commit 580b6aa8e5a8a887397eab5c4c830ec28f45cd30
Author: Alexey Kuznetsov 
Date:   2017-08-09T10:22:54Z

IGNITE-5734 Web Console: Fixed npm dependencies.
(cherry picked from commit aeafbf1)

commit 841db65e56063605475710bc170de4aea672c31d
Author: Alexey Kuznetsov 
Date:   2017-08-09T11:55:04Z

IGNITE-5987 Added -nq (visor will not quit in batch mode) option for Visor 
Cmd.
(cherry picked from commit 8d6e842)

commit 5c2097856714a7803956d754735c68b21156019c
Author: Alexey Kuznetsov 
Date:   2017-08-11T03:25:36Z

IGNITE-5902 Implemented stop caches at once.
(cherry picked from commit ebb8765)

commit 8b2461942c18f228c0107020aa28c03711bdceda
Author: Alexey Kuznetsov 
Date:   2017-08-11T04:07:26Z

IGNITE-6012 Refactored GridJettyRestHandler.processRequest(): replace 
mapper.writeValueAsString with writeValue(stream, v).
(cherry picked from commit 3a390c8)

commit 3a7d4f4a79e7c0a23244387e3a68535375a66a87
Author: Alexey Kuznetsov 
Date:   2017-08-11T04:18:42Z

IGNITE-6013 Optimized processing response from cluster.
(cherry picked from commit b02c481)

commit 74d6ab9916b3a01c78cdf1ad86211c9fcbb2214d
Author: Alexey Goncharuk 
Date:   2017-08-14T08:08:28Z

Merge branch 'ignite-2.1.3' into ignite-2.1.4

commit fde550bac56fd0cc7c51c62a9c291dd4c3f3030c
Author: Ilya Lantukh 
Date:   2017-08-14T08:32:11Z

IGNITE-5941 - Fixed index name length restrictions. This closes #2408

commit 13f38d79b57b395e43d42a8f3c278cf48336d7c5
Author: Dmitriy Govorukhin 

Adding sqlline tool to Apache Ignite project

[Ilya Suntsov]

Hi Igniters,

I've found one command line tool that can be used for execute queries
(DDL/DML) with data from Apache Ignite grid via
*org.apache.ignite.IgniteJdbcThinDriver*.

Here you can find overview of internal commands of this tool:
https://cwiki.apache.org/confluence/display/IGNITE/Overview+sqlline+tool

The most problems with sqlline relate with some issues from our side or
some unsupported features.

Would be nice to add tool like this (sqlline or something else) in our
project.

Have you any objections or other thoughts?


-- 
Ilya Suntsov

Testing Ignite Applications Locally

[Yakov Zhdanov]

Guys,

I want to discuss the subject again. It is pretty vivid that having wide
set of local unit and simple integration tests most likely help to avoid
many failures and bugs when going to server environment.

I participated in many POC and I can say for sure - if developers are not
implementing local tests then their application is broken. This is true for
the entire industry. Why does anyone think that Ignite and distributed
systems in general are exceptions here? Complexity added by distributed
nature probably needs local tests even more.

So, what Ignite already offers here and what can be done further?

1. Ignite offers ability to emulate cluster and even many cluster in a
single VM. Let's create a page on readme.io explaining how to start
topologies in a single VM and provide couple examples of unit tests for
cache operations and, for example, queries. Denis Magda, can you help?
(Yes, we don't have the page explaining how to test Ignite locally!)

2. Ignite has a large and rich set of tests in its code base. We can
provide the link on the page at p1.

3. Let's create maven archetype for Ignite. So, executing the command [1]
will bring me inited project with valid poms, sample batch scripts, sample
Ignite configs, sample logger configuration and test sources folder
containing several JUnits (!!).

[1]  mvn archetype:generate \
  -DinteractiveMode=false \
  -DarchetypeGroupId=org.apache.ignite \
  -DarchetypeArtifactId=ignite-app-archetype \
  -DgroupId=org.sample \
  -DartifactId=sampleapp \
  -Dversion=1.0

Please share your thoughts and we can file tickets to start moving.

--Yakov

[GitHub] ignite pull request #2518: IGNITE-2779 BinaryMarshaller caches must be clean...

[NSAmelchev]

GitHub user NSAmelchev opened a pull request:

https://github.com/apache/ignite/pull/2518

IGNITE-2779 BinaryMarshaller caches must be cleaned during client reconnect.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/NSAmelchev/ignite IGNITE-2779

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2518.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2518


commit 9170321182797ebb2acada407efea15421875911
Author: NSAmelchev 
Date:   2017-08-25T14:50:10Z

add test and fix




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSSION] Urgent Ignite bug fix release

[Sergey Kozlov]

I suppose we should not forget JVM heap size and suggest to reduce both
options

On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan 
wrote:

> Igor, I would change the message. How about this:
>
> Required RAM size is larger than total physical memory available for OS.
> > Please change MemoryConfiguration.WhichProperty and
> > MemoryPolicyConfiguration.WhichProperty to decrease memory allocated for
> > each node.
>
>
> Also, can we calculate what the memory size allocated for each node should
> be? In that case we should suggest it.
>
> D.
>
>
> On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor 
> wrote:
>
> > What do you, guys think about next warning?
> >
> > [2017-08-25 17:17:04,718][INFO
> > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
> > System cache's MemoryPolicy size is configured to 40 MB. Use
> > MemoryConfiguration.systemCacheMemorySize property to change the
> setting.
> > [2017-08-25 17:17:04,718][WARN
> > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
> >
> > >>> Required RAM size is larger than total physical memory available for
> > OS.
> > >>> Check your configuration to avoid swap partition usage.
> > >>> Use MemoryConfiguration and MemoryPolicyConfiguration to change the
> > settings.
> > >>> Physical memory [required=16397MB, available=15942MB]
> >
> > [2017-08-25 17:17:04,726][WARN
> > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
> > Peer class loading is enabled (disable it in production for performance
> and
> > deployment consistency reasons)
> > [2017-08-25 17:17:04,726][INFO
> > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
> > Configured caches [in 'sysMemPlc' memoryPolicy: ['ignite-sys-cache']]
> > [2017-08-25 17:17:04,731][INFO
> > ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
> > 3-rd party licenses can be found at:
> > /home/gvvinblade/projects/ignite/incubator-ignite/libs/licenses
> >
> >
> > 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov :
> >
> > > Agree, let's release new version including tickets mentioned by Denis
> and
> > > Ivan.
> > >
> > > --Yakov
> > >
> >
>



-- 
Sergey Kozlov
GridGain Systems
www.gridgain.com

[GitHub] ignite pull request #2203: IGNITE-5456 JDBC thin driver: the statement produ...

[tledkov-gridgain]

Github user tledkov-gridgain closed the pull request at:

https://github.com/apache/ignite/pull/2203


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSSION] Urgent Ignite bug fix release

[Dmitriy Setrakyan]

Igor, I would change the message. How about this:

Required RAM size is larger than total physical memory available for OS.
> Please change MemoryConfiguration.WhichProperty and
> MemoryPolicyConfiguration.WhichProperty to decrease memory allocated for
> each node.


Also, can we calculate what the memory size allocated for each node should
be? In that case we should suggest it.

D.


On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor 
wrote:

> What do you, guys think about next warning?
>
> [2017-08-25 17:17:04,718][INFO
> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
> System cache's MemoryPolicy size is configured to 40 MB. Use
> MemoryConfiguration.systemCacheMemorySize property to change the setting.
> [2017-08-25 17:17:04,718][WARN
> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
>
> >>> Required RAM size is larger than total physical memory available for
> OS.
> >>> Check your configuration to avoid swap partition usage.
> >>> Use MemoryConfiguration and MemoryPolicyConfiguration to change the
> settings.
> >>> Physical memory [required=16397MB, available=15942MB]
>
> [2017-08-25 17:17:04,726][WARN
> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
> Peer class loading is enabled (disable it in production for performance and
> deployment consistency reasons)
> [2017-08-25 17:17:04,726][INFO
> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
> Configured caches [in 'sysMemPlc' memoryPolicy: ['ignite-sys-cache']]
> [2017-08-25 17:17:04,731][INFO
> ][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
> 3-rd party licenses can be found at:
> /home/gvvinblade/projects/ignite/incubator-ignite/libs/licenses
>
>
> 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov :
>
> > Agree, let's release new version including tickets mentioned by Denis and
> > Ivan.
> >
> > --Yakov
> >
>

Re: [DISCUSSION] Urgent Ignite bug fix release

[Seliverstov Igor]

What do you, guys think about next warning?

[2017-08-25 17:17:04,718][INFO
][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
System cache's MemoryPolicy size is configured to 40 MB. Use
MemoryConfiguration.systemCacheMemorySize property to change the setting.
[2017-08-25 17:17:04,718][WARN
][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]

>>> Required RAM size is larger than total physical memory available for OS.
>>> Check your configuration to avoid swap partition usage.
>>> Use MemoryConfiguration and MemoryPolicyConfiguration to change the
settings.
>>> Physical memory [required=16397MB, available=15942MB]

[2017-08-25 17:17:04,726][WARN
][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
Peer class loading is enabled (disable it in production for performance and
deployment consistency reasons)
[2017-08-25 17:17:04,726][INFO
][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
Configured caches [in 'sysMemPlc' memoryPolicy: ['ignite-sys-cache']]
[2017-08-25 17:17:04,731][INFO
][test-runner-#1%internal.GridHomePathSelfTest%][GridHomePathSelfTest0]
3-rd party licenses can be found at:
/home/gvvinblade/projects/ignite/incubator-ignite/libs/licenses


2017-08-25 13:26 GMT+03:00 Yakov Zhdanov :

> Agree, let's release new version including tickets mentioned by Denis and
> Ivan.
>
> --Yakov
>

[jira] [Created] (IGNITE-6189) Benchmarks for check LFS used disk space

[Aleksey Chetaev (JIRA)]

Aleksey Chetaev created IGNITE-6189:
---

 Summary: Benchmarks for check LFS used disk space
 Key: IGNITE-6189
 URL: https://issues.apache.org/jira/browse/IGNITE-6189
 Project: Ignite
  Issue Type: Improvement
Reporter: Aleksey Chetaev


Need to create new benchmarks for test how many space we use for store n keys 
in LFS.
Benchmark should content arguments:
1. Range of entries which will be put.
2. How often we show store size.

Case for e.x. params (range 1_000_000, step 100_000):
1. Put 100_000 entries, save time for put
2. Wait for checkpoint finished on all nodes.
3. Calc db size on each server.
4. Write size and time to benchmarks results.
...
6. Repeat first 4 steps while entries count in cache less that 1_000_000.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #2517: Ignite 2.1.3.b2

[mcherkasov]

GitHub user mcherkasov opened a pull request:

https://github.com/apache/ignite/pull/2517

Ignite 2.1.3.b2



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-2.1.3.b2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2517.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2517


commit 84c7427a53a8e1712b1d0b763d7539c9cb844cb6
Author: Sergey Stronchinskiy 
Date:   2017-07-04T11:51:25Z

IGNITE-5532 .NET: Split CacheLinqTest into partial classes

This closes #2226

commit 64c156e9252395504af00f09d934f36b6bc21913
Author: Igor Sapego 
Date:   2017-07-04T16:42:33Z

IGNITE-5663: ODBC: Closing cursor do not reset prepared statement anymore

commit 80c95ff79f344daf1fca3f094733a24bac2a218d
Author: Igor Sapego 
Date:   2017-07-05T15:41:28Z

IGNITE-5576: Added Compute::Run() for C++

commit 836906c89dfb880ac602046c37b3a2dba3ebdc46
Author: samaitra 
Date:   2017-07-06T04:28:15Z

IGNITE-5695 FlinkIgniteSinkSelfTest is failing due to conflicting default 
test timeout and default flush frequency - Fixes #2241.

Signed-off-by: samaitra 

commit 651ffc544bbc32cded55626adcd3ed4cc74f11ce
Author: shroman 
Date:   2017-07-06T05:00:08Z

Removed unnecessary line from the comments.

commit d1d6802378d874b039f775fe787f78c507661bb2
Author: devozerov 
Date:   2017-07-07T09:36:13Z

Merge branch 'ignite-2.1'

commit 45cd87fe73db117f5148ed2006f8de8d2517bbfe
Author: mcherkasov 
Date:   2017-06-30T17:23:55Z

IGNITE-5554 ServiceProcessor may process failed reassignments in timeout 
thread

commit fa974286e8f066a8d6aa57519edf5ec7761be095
Author: Igor Sapego 
Date:   2017-07-07T13:49:15Z

IGNITE-5582: Implemented Compute::Broadcast for C++

commit 01f504ff83cc77f80d37981b5c5a15b653861bbd
Author: NSAmelchev 
Date:   2017-07-10T12:03:01Z

IGNITE-5087 Enum comparison fails after marshal-unmarshal with 
BinaryMarshaller.

commit ecfbc2c97464ad7da3f24afaaf1868a2d2fdb87e
Author: devozerov 
Date:   2017-07-11T09:17:41Z

Merge branch 'ignite-2.1'

# Conflicts:
#   
modules/platforms/dotnet/Apache.Ignite.Core.Tests/Apache.Ignite.Core.Tests.csproj

commit 1be9b40c37efcbf332ebeeefc865c2fe864339e7
Author: sboikov 
Date:   2017-07-11T09:42:54Z

Exchange future cleanup, added special tasks for reassign/force rebalance.

commit 8d4a0c2ca2abc17a1d54fa0d33b161531fa59b12
Author: Pavel Tupitsyn 
Date:   2017-07-11T09:49:16Z

Merge branch 'ignite-2.1'

commit bf25b5c52be044f07076c0800447230c75174db3
Author: Slava Koptilin 
Date:   2017-07-07T12:35:33Z

ignite-5562: assert statements were changed to the 'if' blocks

commit e93b28488693381fcd232de93087ab8ec1d0f5bb
Author: sboikov 
Date:   2017-07-11T11:18:52Z

ignite-5446 Only lateAffinity logic in CacheAffinitySharedManager.

commit 5c363184c80f2fd8b79f1075d1eacbf9af5369a1
Author: Denis Magda 
Date:   2017-07-11T19:20:16Z

Simplified Memory Policies Example

commit b95f76f8a0a3a7e920f78f20b3d814112fc6d522
Author: sboikov 
Date:   2017-07-12T05:47:04Z

ignite-5727 Call TcpCommunicationSpi's discovery listener first

commit 120384fca2b5f6f141207697f776d7859afa857f
Author: devozerov 
Date:   2017-07-12T06:48:51Z

Merge branch 'ignite-2.1'

commit 5394bbdeff4e9fb97d3b413bf30001ede580bdd7
Author: sboikov 
Date:   2017-07-13T10:30:59Z

Unnecessary synchronous rollback in GridDhtTxLocal.prepareAsync

commit 00c6b6c4ba00fa6577f74fc95b378737fb5a789c
Author: Alexander Menshikov 
Date:   2017-07-13T12:24:59Z

IGNITE-5567 Make benchmark Ignite.reentrantLock vs IgniteCache.lock

commit 18bdfe96a1e579371108c661e3374183c58a296d
Author: Alexey Goncharuk 
Date:   2017-07-13T12:42:30Z

Fixed NPE in tests

commit 7338445ac9c1a2343fd41cdd20785de07b727796
Author: dkarachentsev 
Date:   2017-07-13T13:00:08Z

IGNITE-5597 - Fix javadoc in Affinity and AffinityFunction for REPLICATED 
cache. This closes #2268.

commit d9ed07c67e4a4ff3a9de543cbe039ac2a48f03a0
Author: Sergey Chugunov 
Date:   2017-07-13T14:32:06Z

Functionality of muted test is debated now

commit 871d9260f3b32bed5273852dbdb74c758f73d383
Author: Sergey Chugunov 
Date:   2017-07-13T15:34:01Z

Functionality of GridVersionSelfTest is debated 

[jira] [Created] (IGNITE-6188) ODBC: SQLSQLFreeStmt failing if called before all the rows from the result set were fetched.

[Igor Sapego (JIRA)]

Igor Sapego created IGNITE-6188:
---

 Summary: ODBC: SQLSQLFreeStmt failing if called before all the 
rows from the result set were fetched.
 Key: IGNITE-6188
 URL: https://issues.apache.org/jira/browse/IGNITE-6188
 Project: Ignite
  Issue Type: Bug
  Components: odbc
Affects Versions: 2.1
Reporter: Igor Sapego


Steps to reproduce: 
1. Execute select query with more then 1 row in result set.
2. Fetch results and close the cursor before detecting end of result set.
3. Check for statement error.

Error message:
{noformat}
HY000: Failed to find query with ID: 10
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #2505: IGNITE-6168 Need SSL client authentication during...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2505


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-6187) Cache JdbcDatabaseMetadata in JdbcConnection

[Ilya Kasnacheev (JIRA)]

Ilya Kasnacheev created IGNITE-6187:
---

 Summary: Cache JdbcDatabaseMetadata in JdbcConnection
 Key: IGNITE-6187
 URL: https://issues.apache.org/jira/browse/IGNITE-6187
 Project: Ignite
  Issue Type: Improvement
  Components: jdbc
Affects Versions: 2.1
Reporter: Ilya Kasnacheev
Assignee: Ilya Kasnacheev


To avoid re-creating it every time



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Data compression in Ignite 2.0

[Vyacheslav Daradur]

Hi, should I close the initial ticket [1] as "Won't Fix" and add link to
the new discusion about storage compression [2] in comments?

[1] https://issues.apache.org/jira/browse/IGNITE-3592
[2]
http://apache-ignite-developers.2346864.n4.nabble.com/Data-compression-in-Ignite-td20679.html

2017-08-09 23:05 GMT+03:00 Vyacheslav Daradur :

> Vladimir, thank you for detailed explanation.
>
> I think I've understanded the main idea of described storage compression.
>
> I'll join the new discussion after researching of given material and
> comlpetion of varint-optimization [1].
>
> [1] https://issues.apache.org/jira/browse/IGNITE-5097
>
> 2017-08-02 15:43 GMT+03:00 Alexey Kuznetsov :
>
>> Vova,
>>
>> Finally we back to my initial idea - to look how "big databases compress"
>> data :)
>>
>>
>> Just to remind how IBM DB2 do this[1].
>>
>> [1] http://www.ibm.com/developerworks/data/library/techarticle/dm-
>> 1205db210compression/
>> 
>>
>> On Tue, Aug 1, 2017 at 4:15 PM, Vladimir Ozerov 
>> wrote:
>>
>> > Vyacheslav,
>> >
>> > This is not about my needs, but about the product :-) BinaryObject is a
>> > central entity used for both data transfer and data storage. This is
>> both
>> > good and bad at the same time.
>> >
>> > Good thing is that as we optimize binary protocol, we improve both
>> network
>> > and storage performance at the same time. We have at least 3 things
>> which
>> > will be included into the product soon: varint encoding [1], optimized
>> > string encoding [2] and null-field optimization [3]. Bad thing is that
>> > binary object format is not well suited for data storage optimizations,
>> > including compression. For example, one good compression technique is to
>> > organize data in column-store format, or to introduce shared
>> "dictionary"
>> > with unique values on cache level. In both cases N equal values are not
>> > stored N times. Instead, we store one value and N references to it, or
>> so.
>> > This way 2x-10x compression is possible depending on workload type.
>> Binary
>> > object protocol with some compression on top of it cannot give such
>> > improvement, because it will compress data in individual objects,
>> instead
>> > of compressing the whole cache data in a single context.
>> >
>> > That said, I propose to give up adding compression to BinaryObject.
>> This is
>> > a dead end. Instead, we should:
>> > 1) Optimize protocol itself to be more compact, as described in
>> > aforementioned Ignite tickets
>> > 2) Start new discussion about storage compression
>> >
>> > You can read papers of other vendors to get better understanding on
>> > possible compression options. E.g. Oracle has a lot of compression
>> > techniques, including heat maps, background compression, per-block
>> > compression, data dictionaries, etc. [4].
>> >
>> > [1] https://issues.apache.org/jira/browse/IGNITE-5097
>> > [2] https://issues.apache.org/jira/browse/IGNITE-5655
>> > [3] https://issues.apache.org/jira/browse/IGNITE-3939
>> > [4] http://www.oracle.com/technetwork/database/options/
>> > compression/advanced-
>> > compression-wp-12c-1896128.pdf
>> >
>> > Vladimir.
>> >
>> >
>>
>> --
>> Alexey Kuznetsov
>>
>
>
>
> --
> Best Regards, Vyacheslav D.
>



-- 
Best Regards, Vyacheslav D.

[GitHub] ignite pull request #2515: IGNITE-6185 Missed toString IPFinder information.

[vsisko]

GitHub user vsisko opened a pull request:

https://github.com/apache/ignite/pull/2515

IGNITE-6185 Missed toString IPFinder information.

Missed toString IPFinder information.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-6185

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2515.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2515


commit 39c9bd75f6febb5eb036439732a48111670a0f9a
Author: vsisko 
Date:   2017-08-25T10:59:09Z

IGNITE-6185 Missed toString IPFinder information.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSSION] Urgent Ignite bug fix release

[Yakov Zhdanov]

Agree, let's release new version including tickets mentioned by Denis and
Ivan.

--Yakov

Re: [DISCUSS] Ignite Update Checker

[Yakov Zhdanov]

Raul, any ideas on startup counter?

BTW, we have google analytics applied to Ignite site. Can it be used to
count node starts?

--Yakov

[jira] [Created] (IGNITE-6186) Remove redundant parameter of GridFutureAdapter::unregisterWaiter()

[Andrey Kuznetsov (JIRA)]

Andrey Kuznetsov created IGNITE-6186:


 Summary: Remove redundant parameter of 
GridFutureAdapter::unregisterWaiter()
 Key: IGNITE-6186
 URL: https://issues.apache.org/jira/browse/IGNITE-6186
 Project: Ignite
  Issue Type: Improvement
Affects Versions: 2.1
Reporter: Andrey Kuznetsov
Priority: Minor
 Fix For: 2.2


The method is not thread-safe unless actual parameter is currentThread.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Ignite Usability: Deadlocks and Starvation

[Yakov Zhdanov]

Andrey, feel free to file a ticket and remove the parameter =)

--Yakov

[jira] [Created] (IGNITE-6184) No checkClusterState() in IgniteKernal.getOrCreateCaches()

[Ilya Kasnacheev (JIRA)]

Ilya Kasnacheev created IGNITE-6184:
---

 Summary: No checkClusterState() in IgniteKernal.getOrCreateCaches()
 Key: IGNITE-6184
 URL: https://issues.apache.org/jira/browse/IGNITE-6184
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.1
Reporter: Ilya Kasnacheev
Assignee: Ilya Kasnacheev


Check if anything else is off, too.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] Ignite Update Checker

[Raul Kripalani]

Hey guys

In my opinion, maven.org is still owned by a third party (Sonatype). We
don't know what kind of data analysis or intelligence extraction they run.

If Ignite servers all over the world were hitting maven.org periodically
asking for an Ignite Maven artifact, it gives Sonatype a clear indication
of who is running an Ignite server.

They could reverse lookup the IP address and find out what corporation it
is.

How about having Ignite check the ASF Git directly?

We could use the Git ssh API, but that would require a new dependency,
which I advise against.

Alternatively, we could have it scrape this HTML for new Git tags:
https://git-wip-us.apache.org/repos/asf?p=ignite.git

Another option is to place a txt file in the root of the master branch (e.g
LATEST), containing a list of the latest GA versions for each major version
line (1.x, 2.x).

I would advocate this last option, but it requires somebody remembering to
update the file with every release, unless we automate it with a Maven
plugin.

Hope that helps!


On 24 Aug 2017 19:37, "Denis Magda"  wrote:

I see nothing wrong with this approach.

Cos, Roman, Raul, as Apache veterans, what do you think? Is it good to go?

—
Denis

> On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan 
wrote:
>
> Is everyone OK with this approach? Should I file a ticket on it?
>
> On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan 
> wrote:
>
>> Igniters,
>>
>> There has been lots of talk of proposals about various usage metrics for
>> Ignite and nothing came of it. I would like to resurrect the topic and
>> propose something very simple and non-intrusive.
>>
>> 1. Update Checker
>> The main purpose of the update checker is not to collect metrics, but to
>> notify users about a new version of Ignite by accessing maven.org and
>> getting the version out of the metadata file:
>> http://repo2.maven.org/maven2/org/apache/ignite/ignite-core/
>> maven-metadata.xml
>>
>> This way we do not send any information anywhere and, at the same time,
>> urge our users to download and start using the latest version of Ignite.
>>
>> 2. Startup Counter
>> This piece is optional, but we can also get an insight in how many times
a
>> certain Ignite release gets started. This is just a cool metric for the
>> community to gauge the project popularity. You can think of it as of a
page
>> visit counter shown on many websites. We can even decide to display this
>> counter on the Ignite website as well.
>>
>> To do this, we can simply add a JAR in maven for every release, e.g.
>> ignite-start-counter.jar, which will contain only 1 byte. Every time an
>> Ignite node starts, it will download this JAR in the background. Then we
>> will be able to view the number of the total downloads for this JAR in
>> Maven Central, which is essentially the number of starts of Ignite nodes.
>>
>> *Note that neither of the above suggestions require Ignite to send or
>> track any user information whatsoever.*
>>
>> Please reply suggesting weather you are OK with this approach.
>>
>> D.
>>

Re: [DISCUSSION] Urgent Ignite bug fix release

[Ivan Rakov]

Let's include to the release tickets with optimizations of checkpointing 
algorithm:


https://issues.apache.org/jira/browse/IGNITE-6178
https://issues.apache.org/jira/browse/IGNITE-6033
https://issues.apache.org/jira/browse/IGNITE-5961

This will help users who are experiencing problems with slow checkpoints.

Also, let's include https://issues.apache.org/jira/browse/IGNITE-6183 to 
soften "Ignite node crashed in the middle of checkpoint" message as per 
discussion 
.


Best Regards,
Ivan Rakov

On 25.08.2017 3:31, Dmitriy Setrakyan wrote:

+1 on the release, let's do it ASAP.

On Thu, Aug 24, 2017 at 12:32 PM, Denis Magda  wrote:


Igniters,

Presently every Apache Ignite node requests 80% of RAM from an operating
system. We debated a lot about this default [1] and as a practice shows
those who voted for the lower value (20%-25%) were right.

Our users started reporting about bizarre issues ([2], [3], [4]) when a
cluster hangs, freezes for a while or even crashes during standard
preloading phase or minor workloads.

It’s turned out to be that lack of RAM causes extensive swapping and
checkpointing activity. Even more, the issue can be reproduced on your
laptop if you launch a couple of nodes and start data preloading.

All this can be avoided if you allocated less than 80% and tune the
checkpointing if the persistence is used. Unfortunately, only few could get
to this point, the rest would kick Ignite out from their deployments.

Thus, we have to fix the defaults ([5] and [6]) on our own and release an
emergency release with the fixes as soon as possible.

Any objections or other thoughts?

[1] http://apache-ignite-developers.2346864.n4.nabble.
com/IGNITE-5717-improvements-of-MemoryPolicy-default-size-td20264.html
[2] http://apache-ignite-users.70518.x6.nabble.com/
Performance-of-persistent-store-too-low-when-bulb-
loading-tp16247p16337.html
[3] http://apache-ignite-users.70518.x6.nabble.com/Strange-
problems-with-Ignite-native-Persistence-when-Data-exceeds-
Memory-td16187.html#a16194
[4] http://apache-ignite-users.70518.x6.nabble.com/
Activation-slow-and-Ignite-node-crashed-in-the-middle-of-
checkpoint-td16144.html

[5] https://issues.apache.org/jira/browse/IGNITE-6003
[6] https://issues.apache.org/jira/browse/IGNITE-6182

—
Denis

Re: Ignite Usability: Deadlocks and Starvation

[Andrey Kuznetsov]

Yakov, it would be good to get rid of the parameter at all. Strictly
speaking, such tiny things are not bugs, but can lead to bugs, and I'm
curious about the way of improving the situation.


> Andrey, it seems unregisterWaiter() method is supposed to be called only
> with current thread as parameter.
>
> --Yakov
>



-- 
Best regards,
  Andrey Kuznetsov.

[GitHub] ignite pull request #2514: IGNITE-6170: JDBC: make consistent product name a...

[shroman]

GitHub user shroman opened a pull request:

https://github.com/apache/ignite/pull/2514

IGNITE-6170: JDBC: make consistent product name across all drivers



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shroman/ignite IGNITE-6170

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2514.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2514


commit 3dddb61aaaf99d0721e6451c469aeef5b07598b9
Author: shroman 
Date:   2017-08-25T07:34:23Z

IGNITE-6170: JDBC: consistent product name across all drivers




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Ignite Usability: Deadlocks and Starvation

[Yakov Zhdanov]

>It's not public API, just implementation detail.

>GridFutureAdapter::unregisterWaiter is not thread state in general. It
>won't work properly if the argument is not current thread. And also I
>couldn't prove clearly it wouldn't drop some unrelated Node in concurrent
>operation.

Andrey, it seems unregisterWaiter() method is supposed to be called only
with current thread as parameter.

--Yakov