[GitHub] ignite pull request #3538: Ignite 2.3.2 cloud streaming

[alexpaschenko]

GitHub user alexpaschenko opened a pull request:

https://github.com/apache/ignite/pull/3538

Ignite 2.3.2 cloud streaming



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite 
ignite-2.3.2-gg-cloud-streaming

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3538.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3538


commit c5d6decfaf4c8be907975531f7d7bd41dda2da4e
Author: dpavlov 
Date:   2017-10-04T06:46:16Z

IGNITE-6285 Enhance persistent store paths handling - Fixes #2775.

Signed-off-by: Alexey Goncharuk 

commit 2e013fbe85da059580e97d57c15e9cbc92c0aa54
Author: dpavlov 
Date:   2017-10-04T12:05:48Z

IGNITE-6554 Atomic cache remove operations are not logged to WAL - Fixes 
#2800.

Signed-off-by: Alexey Goncharuk 

commit 959bf853b1a77fa674ac8115fcf57c31feb0e6c0
Author: oleg-ostanin 
Date:   2017-10-04T14:35:50Z

Removed excluding ML from examples/src

(cherry picked from commit 78f77b1)

commit d4e5d4fd3e989b69f7bfa49986ee8f8d04cac62f
Author: tledkov-gridgain 
Date:   2017-10-05T08:55:26Z

IGNITE-6529 JDBC: support column metadata 'nullable' property. This closes 
#2793.

commit 81d20410205086ff29ac2e66b7cd17909173d642
Author: tledkov-gridgain 
Date:   2017-10-05T13:32:33Z

IGNITE-6358: JDBC thick: support multiple statements. This closes #2777.

commit 2e046d61078874c12e4fd33bba0826f5453b055d
Author: tledkov-gridgain 
Date:   2017-10-05T13:45:31Z

IGNITE-6556: JDBC thin: fixed setSchema() case sensitivity handling. This 
closes #2805.

commit 62b4b5311109af1f4ac0ae84ac47ad885eaa3533
Author: Alexey Goncharuk 
Date:   2017-10-05T14:37:04Z

IGNITE-5739 Fixed Ignite node crash on deactivation

commit b33be44164d4dc4948ab5c27857d53b75ac4a332
Author: Vasiliy Sisko 
Date:   2017-10-06T07:25:42Z

IGNITE-6287 Web Console: Improved DDL support.
(cherry picked from commit 2410f07)

commit 6c4c41d4ce6f2066a399756f60f798507e0553ad
Author: Alexey Kuznetsov 
Date:   2017-10-06T10:00:39Z

IGNITE-6570 Web Console: Move parsing of JSON to pool of workers.
(cherry picked from commit 74f0400)

commit 6efb0cee753eac1289efd21cc773ca372775aa68
Author: Pavel Tupitsyn 
Date:   2017-10-03T15:33:57Z

.NET: Fix TestRecordLocal

commit fca198bf1ecd4eee8bb72233cd59d0ea2e427c7c
Author: Alexander Paschenko 
Date:   2017-10-06T15:04:44Z

IGNITE-6054: SQL: implemented "WRAP_KEY" and "WRAP_VALUE" modes for CREATE 
TABLE. This closes #2784.

commit 73f092df4792bee918083322ee93026dfa95e3b8
Author: devozerov 
Date:   2017-10-09T07:48:06Z

Fixed JavaDoc.

commit 81b16ada108dc497d38f702b7f678de539345705
Author: devozerov 
Date:   2017-10-09T08:34:51Z

Fixed JavaDoc again.

commit d9f0f4e1bec08ab9f52c6c1ba842524f25a4ba19
Author: devozerov 
Date:   2017-10-09T08:57:06Z

IGNITE-6054: Fixed tests.

commit 9358a88538189e48de68ad2d157c2c1ebf9a219e
Author: tledkov-gridgain 
Date:   2017-10-09T12:14:23Z

IGNITE-6529: JDBC thin: fixed driver protocol compatibility. This closes 
#2819.

commit c4acf54413a0e8cb9cedcb7201084f29ee554ee7
Author: Vasiliy Sisko 
Date:   2017-10-09T12:23:23Z

IGNITE-6287 Web Console: Improved DDL support: added checkbox "Use selected 
cache as default schema name".
(cherry picked from commit a45677c)

commit 831c4d9635c911fde3781987f79a28523d6cf15b
Author: Pavel Tupitsyn 
Date:   2017-10-09T14:33:46Z

IGNITE-6397 .NET thin client: basic cache operations. This closes #2725.

commit 1a6bb2bbf513c8a50f81e9f8def1b3f479c08f4b
Author: Andrey Gura 
Date:   2017-09-27T10:50:26Z

ignite-6305 Ignite update checker enabled

commit 72b409cc233919462b3df1547afdec082d8273e1
Author: Alexander Paschenko 
Date:   2017-10-09T19:17:28Z

IGNITE-6569: Fixed hang when trying to execute "DROP TABLE" on the cache 
this table belongs to. This closes #2823.

commit ab384d4b1692538eca6de81c04ec9b720a73308e
Author: Alexander Paschenko 
Date:   2017-10-10T07:05:12Z

IGNITE-6568: Fixed cache configuration persistence logic. This closes #2815.

commit f06927d13303b780bcb9865a3eea9c95b926a25e
Author: Sergey Chugunov 
Date:   2017-10-09T15:35:11Z

IGNITE-6583 Proper getters for rebalance metrics were added; ignite-style 
getters (without get) were deprecated

Signed-off-by: Andrey Gura 

commit 4a782fb8b8f10917addecd70af77049719f908bd
Author: Alexander Belyak 

Berlin and S.F. Bay Area meetups next week featuring Apache Ignite

[Tom Diederich]

Igniters, 

 

Next week there are a number of events featuring Apache Ignite -- including 
meetups in Berlin and the San Francisco Bay Area. 

 

On Tuesday there will be an excellent webinar titled, “Redis Replaced: Why 
Companies Now Choose Apache® Ignite™ to Improve Application Speed and Scale.” 
This webinar, hosted by Ignite PMC Denis Magda, starts at 11 a.m. Pacific time. 
Be sure to RSVP!

 

Speaking of Denis, in case you missed it, he published a fantastic blog post 
yesterday headlined: “Apache Cassandra vs. Apache Ignite: Strong Consistency 
and Transactions.” This post, part of a series on Apache Cassandra vs. Apache 
Ignite, examines the pros & cons of NoSQL databases in terms of consistency and 
ACID transactions compared with how Apache Ignite handles them. 

 

Meanwhile, starting on Tuesday, Ignite evangelist Akmal Chaudhri touches down 
in Berlin where he’ll begin a three-day meetup roadshow. 

 

First up is the Java Usergroup Berlin-Brandenburg Meetup on Tuesday evening. 
Akmal’s talk is titled, "Skyrocket Java applications with the open-source 
Apache Ignite."


Then on Wednesday he’s back behind the podium – this time at the Berlin 
Kubernetes Meetup. This meetup is sold-out! Great job, Akmal! There are 125 
RSVPs and another 88 on the waiting list. Akmal’s talk is called, "Distributed 
Database DevOps Dilemmas? Kubernetes to the rescue!"

 

On Thursday (Feb. 22), Akmal takes the stage at the Big Data, Berlin Meetup. 
This event’s venue is large enough so there won’t be a need for a waiting list. 
Even so, an impressive 156 people have so far RSVP’d. His talk there is titled, 
“Apache Ignite: the in-memory hammer in your data science toolkit.”

 

Backing up a day and to the Bay is the fabulous Bay Area In-Memory Computing 
Meetup on Feb. 21 in Santa Clara. 

 

Eden Kim, Chair of the SNIA Solid State Storage Technical Work Group will 
present, "Real-world workloads and in-memory performance." His discussion will 
compare real-world datacenter workloads between NVMe SSD block IO and NVDIMM-N 
byte addressable load-store.” Ignite committer Valentin Kulichenko will also 
speak on, "Building consistent and highly available distributed systems with 
Apache Ignite and GridGain." 

 

If you’re at the HQ then RSVP now and make plans to catch all the action.

Read more about all of next week’s event here and have a great weekend!

 

Re: Apache Ignite 2.4 release

[Denis Magda]

Classic relational databases have no choice rather than to use FSYNC by
default. RDBMS is all about consistency.

Distributed databases try to balance consistency and performance. For
instance, why to fsync every update if there is usually 1 backup copy?
This is probably why VoltDB [1] and Cassandra use the modes comparable to
Ignite's LOG_ONLY.

Ignite as a distributed database should care of both consistency and
performance.

My vote goes to FSYNC, LOG_ONLY (default), BACKGROUND, NONE.


[1] https://docs.voltdb.com/UsingVoltDB/CmdLogConfig.php

--
Denis


On Fri, Feb 16, 2018 at 2:14 PM, Dmitriy Setrakyan 
wrote:

> Vova,
>
> I hear your concerns, but at the same time I know that one of the largest
> banks in eastern Europe is using Ignite in LOG_ONLY mode with 3 backups to
> move money. The rational is that the probability of failure of 4 servers at
> hardware level at the same time is very low. However, if the JVM process
> fails on any server, then it can be safely restarted without loosing data.
> In my view, this is why LOG_ONLY mode makes sense as a default.
>
> I still vote to change the default to LOG_ONLY, deprecate the DEFAULT name
> altogether and add FSYNC mode instead.
>
> D.
>
> On Fri, Feb 16, 2018 at 4:05 PM, Vladimir Ozerov 
> wrote:
>
> > Sergey,
> >
> > We do not have backups by default either, so essentially we are loosing
> > data by default. Moreover, backups are less reliable option than fsync
> > because a lot of users cannot afford putting servers into separate power
> > circuits, so a single power failure may easily lead to poweroff of the
> > whole cluster at once, so data is lost still. This is normal practice
> even
> > for enterprise deployments (e.g. asynchronous replication).
> >
> > To make things even worse, we employ PRIMARY_SYNC mode by default! So
> even
> > if you configured backups, you still may loose data due to a single node
> > failure - just shutdown the PRIMARY after commit is confirmed to the
> client
> > and your recent update will disappers.
> >
> > So this is what user should do to make himself safe:
> > 1) Learn about WAL modes
> > 2) Learn about backups
> > 3) Learn about synchronization modes
> > 4) Cross his fingers that he understood everything correctly and that
> there
> > are no other hidden surprises in Ignite which could lead to data loss.
> >
> > Way to much for a product, claiming to be A*C*ID and persistent, don't
> you
> > think so?
> >
> > Leaving deafult WAL mode with fsync resolves all these issues.
> >
> > Vladimir.
> >
> >
> > On Fri, Feb 16, 2018 at 11:43 PM, Sergey Kozlov 
> > wrote:
> >
> > > I suppose some approaches used by classic databases makes no sense for
> > > Ignite. FSYNC requirement for databases has the nature of single host
> > > solution. If you have corrupted db files you have corrupted (lost)
> data.
> > >
> > > For Ignite the enough number of backups and the failure detecting logic
> > can
> > > provide the data consistency in term "cluster data consistency".
> > >
> > >
> > >
> > > On Fri, Feb 16, 2018 at 8:57 PM, Dmitry Pavlov 
> > > wrote:
> > >
> > > > Hi, all WAL modes except NONE protects from data consistency problem
> > > > (B+Tree, pages, etc), which is why I suggest to avoid saying
> > 'corrupted'
> > > > about 'unapplied updates'.
> > > >
> > > > Log Only and Background may cause unapplied updates in case of
> > OS/process
> > > > failures.
> > > >
> > > > None mode IMO is not an option in case data consistency is needed.
> > > >
> > > > пт, 16 февр. 2018 г. в 20:49, Valentin Kulichenko <
> > > > valentin.kuliche...@gmail.com>:
> > > >
> > > > > Guys,
> > > > >
> > > > > While we're on this topic, what is the difference between
> BACKGROUND
> > > and
> > > > > NONE in terms of semantics and provided guarantees? To me it looks
> > like
> > > > > both guarantee to recover the state since last checkpoint and
> > anything
> > > > else
> > > > > can potentially be lost, so from user perspective they are the
> same.
> > > Am I
> > > > > missing something here?
> > > > >
> > > > > Also there is the following in Javadoc for NONE: "If an Ignite node
> > is
> > > > > terminated in NONE mode abruptly, it is likely that the data stored
> > on
> > > > disk
> > > > > is corrupted and work directory will need to be cleared for a node
> > > > > restart.". If this is really the case, I'm not sure NONE makes
> sense
> > at
> > > > > all. Why would I enable persistence if I'm likely to clear the
> > storage
> > > on
> > > > > restart?
> > > > >
> > > > > -Val
> > > > >
> > > > > On Fri, Feb 16, 2018 at 8:39 AM, Vladimir Ozerov <
> > voze...@gridgain.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > What is the reason to have DEFAULT mode at all if you claim
> > LOG_ONLY
> > > to
> > > > > be
> > > > > > completely safe? :)
> > > > > >
> > > > > > And how it could be safe provided that without fsync we loose
> part
> > of
> > > > WAL
> 

[jira] [Created] (IGNITE-7743) JDBC driver allows to connect to non existent schema

[Valentin Kulichenko (JIRA)]

Valentin Kulichenko created IGNITE-7743:
---

 Summary: JDBC driver allows to connect to non existent schema
 Key: IGNITE-7743
 URL: https://issues.apache.org/jira/browse/IGNITE-7743
 Project: Ignite
  Issue Type: Bug
  Components: jdbc
Affects Versions: 2.3
Reporter: Valentin Kulichenko
 Fix For: 2.5


Currently, if one creates a cache without DDL (via {{QueryEntity}} or 
{{indexedTypes}}), separate schema for this cache is created. Schema name is 
case sensitive, so to connect to it with JDBC driver, it's required to provide 
the name in quotes. Here is how it looks like in SqlLine:
{noformat}
./bin/sqlline.sh -u 
jdbc:ignite:thin://127.0.0.1/\"CacheQueryExamplePersons\"{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Apache Ignite 2.4 release

[Dmitriy Setrakyan]

Vova,

I hear your concerns, but at the same time I know that one of the largest
banks in eastern Europe is using Ignite in LOG_ONLY mode with 3 backups to
move money. The rational is that the probability of failure of 4 servers at
hardware level at the same time is very low. However, if the JVM process
fails on any server, then it can be safely restarted without loosing data.
In my view, this is why LOG_ONLY mode makes sense as a default.

I still vote to change the default to LOG_ONLY, deprecate the DEFAULT name
altogether and add FSYNC mode instead.

D.

On Fri, Feb 16, 2018 at 4:05 PM, Vladimir Ozerov 
wrote:

> Sergey,
>
> We do not have backups by default either, so essentially we are loosing
> data by default. Moreover, backups are less reliable option than fsync
> because a lot of users cannot afford putting servers into separate power
> circuits, so a single power failure may easily lead to poweroff of the
> whole cluster at once, so data is lost still. This is normal practice even
> for enterprise deployments (e.g. asynchronous replication).
>
> To make things even worse, we employ PRIMARY_SYNC mode by default! So even
> if you configured backups, you still may loose data due to a single node
> failure - just shutdown the PRIMARY after commit is confirmed to the client
> and your recent update will disappers.
>
> So this is what user should do to make himself safe:
> 1) Learn about WAL modes
> 2) Learn about backups
> 3) Learn about synchronization modes
> 4) Cross his fingers that he understood everything correctly and that there
> are no other hidden surprises in Ignite which could lead to data loss.
>
> Way to much for a product, claiming to be A*C*ID and persistent, don't you
> think so?
>
> Leaving deafult WAL mode with fsync resolves all these issues.
>
> Vladimir.
>
>
> On Fri, Feb 16, 2018 at 11:43 PM, Sergey Kozlov 
> wrote:
>
> > I suppose some approaches used by classic databases makes no sense for
> > Ignite. FSYNC requirement for databases has the nature of single host
> > solution. If you have corrupted db files you have corrupted (lost) data.
> >
> > For Ignite the enough number of backups and the failure detecting logic
> can
> > provide the data consistency in term "cluster data consistency".
> >
> >
> >
> > On Fri, Feb 16, 2018 at 8:57 PM, Dmitry Pavlov 
> > wrote:
> >
> > > Hi, all WAL modes except NONE protects from data consistency problem
> > > (B+Tree, pages, etc), which is why I suggest to avoid saying
> 'corrupted'
> > > about 'unapplied updates'.
> > >
> > > Log Only and Background may cause unapplied updates in case of
> OS/process
> > > failures.
> > >
> > > None mode IMO is not an option in case data consistency is needed.
> > >
> > > пт, 16 февр. 2018 г. в 20:49, Valentin Kulichenko <
> > > valentin.kuliche...@gmail.com>:
> > >
> > > > Guys,
> > > >
> > > > While we're on this topic, what is the difference between BACKGROUND
> > and
> > > > NONE in terms of semantics and provided guarantees? To me it looks
> like
> > > > both guarantee to recover the state since last checkpoint and
> anything
> > > else
> > > > can potentially be lost, so from user perspective they are the same.
> > Am I
> > > > missing something here?
> > > >
> > > > Also there is the following in Javadoc for NONE: "If an Ignite node
> is
> > > > terminated in NONE mode abruptly, it is likely that the data stored
> on
> > > disk
> > > > is corrupted and work directory will need to be cleared for a node
> > > > restart.". If this is really the case, I'm not sure NONE makes sense
> at
> > > > all. Why would I enable persistence if I'm likely to clear the
> storage
> > on
> > > > restart?
> > > >
> > > > -Val
> > > >
> > > > On Fri, Feb 16, 2018 at 8:39 AM, Vladimir Ozerov <
> voze...@gridgain.com
> > >
> > > > wrote:
> > > >
> > > > > What is the reason to have DEFAULT mode at all if you claim
> LOG_ONLY
> > to
> > > > be
> > > > > completely safe? :)
> > > > >
> > > > > And how it could be safe provided that without fsync we loose part
> of
> > > WAL
> > > > > itself in case of crash?
> > > > >
> > > > > пт, 16 февр. 2018 г. в 19:32, Dmitry Pavlov  >:
> > > > >
> > > > > > Thank you. Data can't be corrupted in case crash because of WAL
> > > replay
> > > > > > (since completed checkpoint). Physical records are used to
> restore
> > > > > probably
> > > > > > corrupted pages in persistent store (we overwrite so called 'grey
> > > > zone' -
> > > > > > pages we don't know for sure if they have been written).
> > > > > >
> > > > > > Only one effect is unwritten one or several last transactions. It
> > is
> > > > not
> > > > > > the same with corrupted data.
> > > > > >
> > > > > > пт, 16 февр. 2018 г. в 19:19, Vladimir Ozerov <
> > voze...@gridgain.com
> > > >:
> > > > > >
> > > > > > > Log only mode is not safe - data might be corrupted in case of
> > > system
> > > > > > > crash. Oracle - fsync, Postgres - 

Re: Apache Ignite 2.4 release

[Vladimir Ozerov]

Sergey,

We do not have backups by default either, so essentially we are loosing
data by default. Moreover, backups are less reliable option than fsync
because a lot of users cannot afford putting servers into separate power
circuits, so a single power failure may easily lead to poweroff of the
whole cluster at once, so data is lost still. This is normal practice even
for enterprise deployments (e.g. asynchronous replication).

To make things even worse, we employ PRIMARY_SYNC mode by default! So even
if you configured backups, you still may loose data due to a single node
failure - just shutdown the PRIMARY after commit is confirmed to the client
and your recent update will disappers.

So this is what user should do to make himself safe:
1) Learn about WAL modes
2) Learn about backups
3) Learn about synchronization modes
4) Cross his fingers that he understood everything correctly and that there
are no other hidden surprises in Ignite which could lead to data loss.

Way to much for a product, claiming to be A*C*ID and persistent, don't you
think so?

Leaving deafult WAL mode with fsync resolves all these issues.

Vladimir.


On Fri, Feb 16, 2018 at 11:43 PM, Sergey Kozlov 
wrote:

> I suppose some approaches used by classic databases makes no sense for
> Ignite. FSYNC requirement for databases has the nature of single host
> solution. If you have corrupted db files you have corrupted (lost) data.
>
> For Ignite the enough number of backups and the failure detecting logic can
> provide the data consistency in term "cluster data consistency".
>
>
>
> On Fri, Feb 16, 2018 at 8:57 PM, Dmitry Pavlov 
> wrote:
>
> > Hi, all WAL modes except NONE protects from data consistency problem
> > (B+Tree, pages, etc), which is why I suggest to avoid saying 'corrupted'
> > about 'unapplied updates'.
> >
> > Log Only and Background may cause unapplied updates in case of OS/process
> > failures.
> >
> > None mode IMO is not an option in case data consistency is needed.
> >
> > пт, 16 февр. 2018 г. в 20:49, Valentin Kulichenko <
> > valentin.kuliche...@gmail.com>:
> >
> > > Guys,
> > >
> > > While we're on this topic, what is the difference between BACKGROUND
> and
> > > NONE in terms of semantics and provided guarantees? To me it looks like
> > > both guarantee to recover the state since last checkpoint and anything
> > else
> > > can potentially be lost, so from user perspective they are the same.
> Am I
> > > missing something here?
> > >
> > > Also there is the following in Javadoc for NONE: "If an Ignite node is
> > > terminated in NONE mode abruptly, it is likely that the data stored on
> > disk
> > > is corrupted and work directory will need to be cleared for a node
> > > restart.". If this is really the case, I'm not sure NONE makes sense at
> > > all. Why would I enable persistence if I'm likely to clear the storage
> on
> > > restart?
> > >
> > > -Val
> > >
> > > On Fri, Feb 16, 2018 at 8:39 AM, Vladimir Ozerov  >
> > > wrote:
> > >
> > > > What is the reason to have DEFAULT mode at all if you claim LOG_ONLY
> to
> > > be
> > > > completely safe? :)
> > > >
> > > > And how it could be safe provided that without fsync we loose part of
> > WAL
> > > > itself in case of crash?
> > > >
> > > > пт, 16 февр. 2018 г. в 19:32, Dmitry Pavlov :
> > > >
> > > > > Thank you. Data can't be corrupted in case crash because of WAL
> > replay
> > > > > (since completed checkpoint). Physical records are used to restore
> > > > probably
> > > > > corrupted pages in persistent store (we overwrite so called 'grey
> > > zone' -
> > > > > pages we don't know for sure if they have been written).
> > > > >
> > > > > Only one effect is unwritten one or several last transactions. It
> is
> > > not
> > > > > the same with corrupted data.
> > > > >
> > > > > пт, 16 февр. 2018 г. в 19:19, Vladimir Ozerov <
> voze...@gridgain.com
> > >:
> > > > >
> > > > > > Log only mode is not safe - data might be corrupted in case of
> > system
> > > > > > crash. Oracle - fsync, Postgres - fsync, SQL Server - fsync,
> > > Cassandra
> > > > -
> > > > > > similar to our “background”.
> > > > > >
> > > > > > пт, 16 февр. 2018 г. в 19:11, Dmitry Pavlov <
> dpavlov@gmail.com
> > >:
> > > > > >
> > > > > > > Hi Vladimir,
> > > > > > >
> > > > > > > What you saying is defenetely make sence.
> > > > > > >
> > > > > > > In the same time LOG_ONLY is also safe mode, user will be able
> to
> > > > > restore
> > > > > > > system after crash. If it is not true, we should create
> critical
> > > > ticket
> > > > > > and
> > > > > > > fix it.
> > > > > > >
> > > > > > > Do you know other databases defaults, such as Cassandra,
> Oracle,
> > > > > Postgre?
> > > > > > >
> > > > > > > Sincerely,
> > > > > > > Dmitriy Pavlov
> > > > > > >
> > > > > > > пт, 16 февр. 2018 г. в 18:41, Vladimir Ozerov <
> > > voze...@gridgain.com
> > > > >:
> > > > > > >
> > > > > > > > Igniters,
> > > > 

Re: Apache Ignite 2.4 release

[Sergey Kozlov]

I suppose some approaches used by classic databases makes no sense for
Ignite. FSYNC requirement for databases has the nature of single host
solution. If you have corrupted db files you have corrupted (lost) data.

For Ignite the enough number of backups and the failure detecting logic can
provide the data consistency in term "cluster data consistency".



On Fri, Feb 16, 2018 at 8:57 PM, Dmitry Pavlov 
wrote:

> Hi, all WAL modes except NONE protects from data consistency problem
> (B+Tree, pages, etc), which is why I suggest to avoid saying 'corrupted'
> about 'unapplied updates'.
>
> Log Only and Background may cause unapplied updates in case of OS/process
> failures.
>
> None mode IMO is not an option in case data consistency is needed.
>
> пт, 16 февр. 2018 г. в 20:49, Valentin Kulichenko <
> valentin.kuliche...@gmail.com>:
>
> > Guys,
> >
> > While we're on this topic, what is the difference between BACKGROUND and
> > NONE in terms of semantics and provided guarantees? To me it looks like
> > both guarantee to recover the state since last checkpoint and anything
> else
> > can potentially be lost, so from user perspective they are the same. Am I
> > missing something here?
> >
> > Also there is the following in Javadoc for NONE: "If an Ignite node is
> > terminated in NONE mode abruptly, it is likely that the data stored on
> disk
> > is corrupted and work directory will need to be cleared for a node
> > restart.". If this is really the case, I'm not sure NONE makes sense at
> > all. Why would I enable persistence if I'm likely to clear the storage on
> > restart?
> >
> > -Val
> >
> > On Fri, Feb 16, 2018 at 8:39 AM, Vladimir Ozerov 
> > wrote:
> >
> > > What is the reason to have DEFAULT mode at all if you claim LOG_ONLY to
> > be
> > > completely safe? :)
> > >
> > > And how it could be safe provided that without fsync we loose part of
> WAL
> > > itself in case of crash?
> > >
> > > пт, 16 февр. 2018 г. в 19:32, Dmitry Pavlov :
> > >
> > > > Thank you. Data can't be corrupted in case crash because of WAL
> replay
> > > > (since completed checkpoint). Physical records are used to restore
> > > probably
> > > > corrupted pages in persistent store (we overwrite so called 'grey
> > zone' -
> > > > pages we don't know for sure if they have been written).
> > > >
> > > > Only one effect is unwritten one or several last transactions. It is
> > not
> > > > the same with corrupted data.
> > > >
> > > > пт, 16 февр. 2018 г. в 19:19, Vladimir Ozerov  >:
> > > >
> > > > > Log only mode is not safe - data might be corrupted in case of
> system
> > > > > crash. Oracle - fsync, Postgres - fsync, SQL Server - fsync,
> > Cassandra
> > > -
> > > > > similar to our “background”.
> > > > >
> > > > > пт, 16 февр. 2018 г. в 19:11, Dmitry Pavlov  >:
> > > > >
> > > > > > Hi Vladimir,
> > > > > >
> > > > > > What you saying is defenetely make sence.
> > > > > >
> > > > > > In the same time LOG_ONLY is also safe mode, user will be able to
> > > > restore
> > > > > > system after crash. If it is not true, we should create critical
> > > ticket
> > > > > and
> > > > > > fix it.
> > > > > >
> > > > > > Do you know other databases defaults, such as Cassandra, Oracle,
> > > > Postgre?
> > > > > >
> > > > > > Sincerely,
> > > > > > Dmitriy Pavlov
> > > > > >
> > > > > > пт, 16 февр. 2018 г. в 18:41, Vladimir Ozerov <
> > voze...@gridgain.com
> > > >:
> > > > > >
> > > > > > > Igniters,
> > > > > > >
> > > > > > > Sorry for pouring oil on the flames, but from database
> > perspective
> > > > > moving
> > > > > > > from FSYNC to non-FSYNC mode appears to be a mistake. When you
> > work
> > > > > with
> > > > > > > database, your main expectation is that it will save your data.
> > All
> > > > > > > production database vendor make sure that you are safe, not
> that
> > > you
> > > > > are
> > > > > > > fast. Moreover, some vendors even prevent you from being in
> > unsafe
> > > > mode
> > > > > > > (e.g. you cannot disable fsync in SQL Server at all).
> > > > > > >
> > > > > > > If we continue going in this direction, we will end up with a
> > > > product,
> > > > > > > which is unsafe out of the box and require tons of
> documentation
> > to
> > > > > > > understand how to make it safe. Definitely not the right
> message
> > to
> > > > the
> > > > > > > market. This is like a car without brakes - would you like to
> > drive
> > > > it?
> > > > > > If
> > > > > > > this is Need For Speed game and you have unlimited lives
> > (in-memory
> > > > > cache
> > > > > > > with backing store), then yes. If this is a real life with
> > > > > (persistence)
> > > > > > -
> > > > > > > then no.
> > > > > > >
> > > > > > > On Fri, Feb 16, 2018 at 5:20 PM, Dmitriy Setrakyan <
> > > > > > dsetrak...@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Well, I cannot say that I like the name LOG_ONLY, but I would
> > > 

Re: .NET: ISslContextFactory should be empty

[Pavel Tupitsyn]

Since there are no objections I have removed unused interface members.

Thank you for all the replies.

On Thu, Feb 15, 2018 at 12:51 AM, Pavel Tupitsyn 
wrote:

> Igniters, Alexey, Igor,
>
> I'd like to discuss recently added ISslContextFactory interface.
> https://github.com/apache/ignite/commit/e58aae48b67c74307703d2ae44fe8e
> 3cd4b9649a
>
> "Factory" implies a "CreateInstace" or a similar method, see
> CacheConfigurationCacheStoreFactory
> or ExpiryPolicyFactory as an example.
>
> Another case is a placeholder interface that maps to some Java abstraction
> but has no .NET counterpart (yet), an example is IEvictionPolicy.
>
> In case of ISslContextFactory we have a similar situation, so the
> interface should be empty.
> Key store details do not belong there.
>
> Thoughts?
>
> PS Please add me as a reviewer for .NET tickets
>
> Thanks,
> Pavel
>

Re: Spark data frames

[Valentin Kulichenko]

That's a great idea! Nikolay, let me know if you need any help with the
presentation, I will be happy to help.

-Val

On Fri, Feb 16, 2018 at 12:19 AM, Nikolay Izhikov 
wrote:

> Ok, Igniters.
>
> I will do it in a few weeks.
> I need time to prepare to the talk.
>
> В Пт, 16/02/2018 в 07:16 +, Dmitry Pavlov пишет:
> > +1 for starting new topic from Nikolay when ' Community Meeting to
> > Introduce Ignite Spark Data Frames'  is ready to be announced.
> >
> > пт, 16 февр. 2018 г. в 9:27, Denis Magda :
> >
> > > I'm in :)
> > >
> > > Nikolay, we can use my GoToMeeting account to host the webinar. To draw
> > > more attention I would suggest starting a more specific thread titled
> like
> > > "[RSVP] Community Meeting to Introduce Ignite Spark Data Frames". This
> > > discussion sounds too generic, folks could simply pass by.
> > >
> > > Negotiated?
> > >
> > > --
> > > Denis
> > >
> > > On Wed, Feb 14, 2018 at 6:04 AM, Vyacheslav Daradur <
> daradu...@gmail.com>
> > > wrote:
> > >
> > > > Dmitry, it's a great idea!
> > > >
> > > > Nikolay, I also have the interest to get familiar with Spark Data
> > > > Frames integration.
> > > >
> > > > I'd prefer webinar of the format similar to "Ignite Persistent Store
> > > > Walkthrough" by Denis Magda, which has been presented some times ago.
> > > >
> > > > On Wed, Feb 14, 2018 at 5:03 PM, Dmitriy Setrakyan
> > > >  wrote:
> > > > > I am definitely interested. Great idea!
> > > > >
> > > > > On Wed, Feb 14, 2018 at 4:32 AM, Nikolay Izhikov <
> nizhi...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hello, Dmitry.
> > > > > >
> > > > > > If other community member are also iterested in that kind of
> > > >
> > > > information I
> > > > > > can try to do the talk.
> > > > > >
> > > > > > В Ср, 14/02/2018 в 10:49 +, Dmitry Pavlov пишет:
> > > > > > > Hi Nikolay,
> > > > > > >
> > > > > > > I've notices there are a number of very lively discussions on
> dev
> > >
> > > list
> > > > > > about SparkDataFrames. But I, for example, can't fully
> understand it
> > > > > > because it is not well-known code for me.
> > > > > > >
> > > > > > > I suppose Ignite community has other members, which are not
> aware of
> > > > > >
> > > > > > recent feature SparkDataFrame and its pros.
> > > > > > >
> > > > > > > What do you think about short talk arrangement for community
> to tell
> > > > > >
> > > > > > about this module, e.g. for 30 minutes? Could you please do
> this? I
> > > >
> > > > think
> > > > > > Denis M. can help with infrastructure.
> > > > > > >
> > > > > > > Sincerely,
> > > > > > > Dmitriy Pavlov
> > > >
> > > >
> > > >
> > > > --
> > > > Best Regards, Vyacheslav D.
> > > >
>

Re: Batch size parameter at DataStreamerCacheUpdaters.batched()

[Valentin Kulichenko]

As far as I remember, it used to be public and then was moved to internal.
The main issue with these updaters was that batching is dangerous because
you can get deadlocks if keys are not sorted (which is the case for
BATCHED). There is also BATCHED_SORTED, but it requires keys to be
Comparable and I doubt we do any validation, so usability is questionable.

We can think about how to improve this functionality, but in current state
I would definitely not include it on public API. User can always implement
custom receiver if needed.

-Val

On Fri, Feb 16, 2018 at 5:04 AM, Denis Mekhanikov 
wrote:

> Guys,
>
> I think, it makes sense to move this receiver implementation to public API.
> It has much better performance, than the default one, that performs single
> puts.
>
> I don't see any point in adding a batch size parameter to it though, since
> DataStreamer already has such setting itself.
>
> Denis
>
> пт, 16 февр. 2018 г. в 13:29, Roman Guseinov :
>
> > Val,
> >
> > I got what do you mean. Everything inside the org.apache.ignite.internal
> is
> > considered to be not public API.
> >
> > Best Regards,
> > Roman
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> >
>

Re: Apache Ignite 2.4 release

[Dmitry Pavlov]

Hi, all WAL modes except NONE protects from data consistency problem
(B+Tree, pages, etc), which is why I suggest to avoid saying 'corrupted'
about 'unapplied updates'.

Log Only and Background may cause unapplied updates in case of OS/process
failures.

None mode IMO is not an option in case data consistency is needed.

пт, 16 февр. 2018 г. в 20:49, Valentin Kulichenko <
valentin.kuliche...@gmail.com>:

> Guys,
>
> While we're on this topic, what is the difference between BACKGROUND and
> NONE in terms of semantics and provided guarantees? To me it looks like
> both guarantee to recover the state since last checkpoint and anything else
> can potentially be lost, so from user perspective they are the same. Am I
> missing something here?
>
> Also there is the following in Javadoc for NONE: "If an Ignite node is
> terminated in NONE mode abruptly, it is likely that the data stored on disk
> is corrupted and work directory will need to be cleared for a node
> restart.". If this is really the case, I'm not sure NONE makes sense at
> all. Why would I enable persistence if I'm likely to clear the storage on
> restart?
>
> -Val
>
> On Fri, Feb 16, 2018 at 8:39 AM, Vladimir Ozerov 
> wrote:
>
> > What is the reason to have DEFAULT mode at all if you claim LOG_ONLY to
> be
> > completely safe? :)
> >
> > And how it could be safe provided that without fsync we loose part of WAL
> > itself in case of crash?
> >
> > пт, 16 февр. 2018 г. в 19:32, Dmitry Pavlov :
> >
> > > Thank you. Data can't be corrupted in case crash because of WAL replay
> > > (since completed checkpoint). Physical records are used to restore
> > probably
> > > corrupted pages in persistent store (we overwrite so called 'grey
> zone' -
> > > pages we don't know for sure if they have been written).
> > >
> > > Only one effect is unwritten one or several last transactions. It is
> not
> > > the same with corrupted data.
> > >
> > > пт, 16 февр. 2018 г. в 19:19, Vladimir Ozerov :
> > >
> > > > Log only mode is not safe - data might be corrupted in case of system
> > > > crash. Oracle - fsync, Postgres - fsync, SQL Server - fsync,
> Cassandra
> > -
> > > > similar to our “background”.
> > > >
> > > > пт, 16 февр. 2018 г. в 19:11, Dmitry Pavlov :
> > > >
> > > > > Hi Vladimir,
> > > > >
> > > > > What you saying is defenetely make sence.
> > > > >
> > > > > In the same time LOG_ONLY is also safe mode, user will be able to
> > > restore
> > > > > system after crash. If it is not true, we should create critical
> > ticket
> > > > and
> > > > > fix it.
> > > > >
> > > > > Do you know other databases defaults, such as Cassandra, Oracle,
> > > Postgre?
> > > > >
> > > > > Sincerely,
> > > > > Dmitriy Pavlov
> > > > >
> > > > > пт, 16 февр. 2018 г. в 18:41, Vladimir Ozerov <
> voze...@gridgain.com
> > >:
> > > > >
> > > > > > Igniters,
> > > > > >
> > > > > > Sorry for pouring oil on the flames, but from database
> perspective
> > > > moving
> > > > > > from FSYNC to non-FSYNC mode appears to be a mistake. When you
> work
> > > > with
> > > > > > database, your main expectation is that it will save your data.
> All
> > > > > > production database vendor make sure that you are safe, not that
> > you
> > > > are
> > > > > > fast. Moreover, some vendors even prevent you from being in
> unsafe
> > > mode
> > > > > > (e.g. you cannot disable fsync in SQL Server at all).
> > > > > >
> > > > > > If we continue going in this direction, we will end up with a
> > > product,
> > > > > > which is unsafe out of the box and require tons of documentation
> to
> > > > > > understand how to make it safe. Definitely not the right message
> to
> > > the
> > > > > > market. This is like a car without brakes - would you like to
> drive
> > > it?
> > > > > If
> > > > > > this is Need For Speed game and you have unlimited lives
> (in-memory
> > > > cache
> > > > > > with backing store), then yes. If this is a real life with
> > > > (persistence)
> > > > > -
> > > > > > then no.
> > > > > >
> > > > > > On Fri, Feb 16, 2018 at 5:20 PM, Dmitriy Setrakyan <
> > > > > dsetrak...@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Well, I cannot say that I like the name LOG_ONLY, but I would
> > vote
> > > to
> > > > > > keep
> > > > > > > it for now, given that it is already documented in many places,
> > > > blogs,
> > > > > > and
> > > > > > > examples.
> > > > > > >
> > > > > > > D.
> > > > > > >
> > > > > > > On Fri, Feb 16, 2018 at 8:13 AM, Ivan Rakov <
> > ivan.glu...@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Looks like it's an Ignite term - I've never heard of it
> outside
> > > > > Ignite
> > > > > > > > scope.
> > > > > > > >
> > > > > > > > Though, renaming existing enum value requires keeping old as
> > > > > > deprecated.
> > > > > > > > DEFAULT is confusing enough to pay this price.
> > > > > > > > As for LOG_ONLY, I think we can keep it as long as it has

Re: Apache Ignite 2.4 release

[Valentin Kulichenko]

Guys,

While we're on this topic, what is the difference between BACKGROUND and
NONE in terms of semantics and provided guarantees? To me it looks like
both guarantee to recover the state since last checkpoint and anything else
can potentially be lost, so from user perspective they are the same. Am I
missing something here?

Also there is the following in Javadoc for NONE: "If an Ignite node is
terminated in NONE mode abruptly, it is likely that the data stored on disk
is corrupted and work directory will need to be cleared for a node
restart.". If this is really the case, I'm not sure NONE makes sense at
all. Why would I enable persistence if I'm likely to clear the storage on
restart?

-Val

On Fri, Feb 16, 2018 at 8:39 AM, Vladimir Ozerov 
wrote:

> What is the reason to have DEFAULT mode at all if you claim LOG_ONLY to be
> completely safe? :)
>
> And how it could be safe provided that without fsync we loose part of WAL
> itself in case of crash?
>
> пт, 16 февр. 2018 г. в 19:32, Dmitry Pavlov :
>
> > Thank you. Data can't be corrupted in case crash because of WAL replay
> > (since completed checkpoint). Physical records are used to restore
> probably
> > corrupted pages in persistent store (we overwrite so called 'grey zone' -
> > pages we don't know for sure if they have been written).
> >
> > Only one effect is unwritten one or several last transactions. It is not
> > the same with corrupted data.
> >
> > пт, 16 февр. 2018 г. в 19:19, Vladimir Ozerov :
> >
> > > Log only mode is not safe - data might be corrupted in case of system
> > > crash. Oracle - fsync, Postgres - fsync, SQL Server - fsync, Cassandra
> -
> > > similar to our “background”.
> > >
> > > пт, 16 февр. 2018 г. в 19:11, Dmitry Pavlov :
> > >
> > > > Hi Vladimir,
> > > >
> > > > What you saying is defenetely make sence.
> > > >
> > > > In the same time LOG_ONLY is also safe mode, user will be able to
> > restore
> > > > system after crash. If it is not true, we should create critical
> ticket
> > > and
> > > > fix it.
> > > >
> > > > Do you know other databases defaults, such as Cassandra, Oracle,
> > Postgre?
> > > >
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > > >
> > > > пт, 16 февр. 2018 г. в 18:41, Vladimir Ozerov  >:
> > > >
> > > > > Igniters,
> > > > >
> > > > > Sorry for pouring oil on the flames, but from database perspective
> > > moving
> > > > > from FSYNC to non-FSYNC mode appears to be a mistake. When you work
> > > with
> > > > > database, your main expectation is that it will save your data. All
> > > > > production database vendor make sure that you are safe, not that
> you
> > > are
> > > > > fast. Moreover, some vendors even prevent you from being in unsafe
> > mode
> > > > > (e.g. you cannot disable fsync in SQL Server at all).
> > > > >
> > > > > If we continue going in this direction, we will end up with a
> > product,
> > > > > which is unsafe out of the box and require tons of documentation to
> > > > > understand how to make it safe. Definitely not the right message to
> > the
> > > > > market. This is like a car without brakes - would you like to drive
> > it?
> > > > If
> > > > > this is Need For Speed game and you have unlimited lives (in-memory
> > > cache
> > > > > with backing store), then yes. If this is a real life with
> > > (persistence)
> > > > -
> > > > > then no.
> > > > >
> > > > > On Fri, Feb 16, 2018 at 5:20 PM, Dmitriy Setrakyan <
> > > > dsetrak...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Well, I cannot say that I like the name LOG_ONLY, but I would
> vote
> > to
> > > > > keep
> > > > > > it for now, given that it is already documented in many places,
> > > blogs,
> > > > > and
> > > > > > examples.
> > > > > >
> > > > > > D.
> > > > > >
> > > > > > On Fri, Feb 16, 2018 at 8:13 AM, Ivan Rakov <
> ivan.glu...@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Looks like it's an Ignite term - I've never heard of it outside
> > > > Ignite
> > > > > > > scope.
> > > > > > >
> > > > > > > Though, renaming existing enum value requires keeping old as
> > > > > deprecated.
> > > > > > > DEFAULT is confusing enough to pay this price.
> > > > > > > As for LOG_ONLY, I think we can keep it as long as it has good
> > and
> > > > > > > definitive javadoc.
> > > > > > >
> > > > > > > Best Regards,
> > > > > > > Ivan Rakov
> > > > > > >
> > > > > > >
> > > > > > > On 16.02.2018 17:07, Dmitriy Setrakyan wrote:
> > > > > > >
> > > > > > >> Igniters, just to clarify, does the term LOG_ONLY mean
> anything
> > in
> > > > the
> > > > > > >> industry or is this just an Ignite term?
> > > > > > >>
> > > > > > >> D.
> > > > > > >>
> > > > > > >> On Fri, Feb 16, 2018 at 8:03 AM, Anton Vinogradov <
> > > > > > >> avinogra...@gridgain.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> Log only mode: flushes application buffers.
> > > > > > >>> So, in synced mode without fsync 

[jira] [Created] (IGNITE-7742) AssertionError in IgniteCacheOffheapManagerImpl when Iterate Cache with expire policy and persistence

[Sergey Kosarev (JIRA)]

Sergey Kosarev created IGNITE-7742:
--

 Summary: AssertionError in IgniteCacheOffheapManagerImpl when 
Iterate Cache with expire policy and persistence
 Key: IGNITE-7742
 URL: https://issues.apache.org/jira/browse/IGNITE-7742
 Project: Ignite
  Issue Type: Bug
  Components: persistence
Affects Versions: 2.4
Reporter: Sergey Kosarev


Some assertions were added in IGNITE-6423 

One of them fires here.

We check for 
assert cctx.shared().database().checkpointLockIsHeldByThread();
but we don't have this lock


{code:java}
java.lang.AssertionError
at 
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.remove(IgniteCacheOffheapManagerImpl.java:1372)
at 
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.remove(GridCacheOffheapManager.java:1364)
at 
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.remove(IgniteCacheOffheapManagerImpl.java:370)
at 
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.removeValue(GridCacheMapEntry.java:3602)
at 
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.onExpired(GridCacheMapEntry.java:3355)
at 
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:421)
at 
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.unswap(GridCacheMapEntry.java:369)
at 
org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3043)
at 
org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:2999)
at 
org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
at 
org.apache.ignite.internal.processors.cache.CacheWeakQueryIteratorsHolder$WeakQueryCloseableIterator.onHasNext(CacheWeakQueryIteratorsHolder.java:308)
at 
org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53)
at 
org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45)
at 
org.apache.ignite.internal.processors.database.IgniteDbExpireTest.testIterators(IgniteDbExpireTest.java:132)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at junit.framework.TestCase.runTest(TestCase.java:176)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2001)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:133)
at 
org.apache.ignite.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:1916)
at java.lang.Thread.run(Thread.java:748){code}






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #3522: IGNITE-7452: Make Linear SVM example for multi - ...

[zaleslaw]

Github user zaleslaw closed the pull request at:

https://github.com/apache/ignite/pull/3522


---

Re: Apache Ignite 2.4 release

[Vladimir Ozerov]

What is the reason to have DEFAULT mode at all if you claim LOG_ONLY to be
completely safe? :)

And how it could be safe provided that without fsync we loose part of WAL
itself in case of crash?

пт, 16 февр. 2018 г. в 19:32, Dmitry Pavlov :

> Thank you. Data can't be corrupted in case crash because of WAL replay
> (since completed checkpoint). Physical records are used to restore probably
> corrupted pages in persistent store (we overwrite so called 'grey zone' -
> pages we don't know for sure if they have been written).
>
> Only one effect is unwritten one or several last transactions. It is not
> the same with corrupted data.
>
> пт, 16 февр. 2018 г. в 19:19, Vladimir Ozerov :
>
> > Log only mode is not safe - data might be corrupted in case of system
> > crash. Oracle - fsync, Postgres - fsync, SQL Server - fsync, Cassandra -
> > similar to our “background”.
> >
> > пт, 16 февр. 2018 г. в 19:11, Dmitry Pavlov :
> >
> > > Hi Vladimir,
> > >
> > > What you saying is defenetely make sence.
> > >
> > > In the same time LOG_ONLY is also safe mode, user will be able to
> restore
> > > system after crash. If it is not true, we should create critical ticket
> > and
> > > fix it.
> > >
> > > Do you know other databases defaults, such as Cassandra, Oracle,
> Postgre?
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > > пт, 16 февр. 2018 г. в 18:41, Vladimir Ozerov :
> > >
> > > > Igniters,
> > > >
> > > > Sorry for pouring oil on the flames, but from database perspective
> > moving
> > > > from FSYNC to non-FSYNC mode appears to be a mistake. When you work
> > with
> > > > database, your main expectation is that it will save your data. All
> > > > production database vendor make sure that you are safe, not that you
> > are
> > > > fast. Moreover, some vendors even prevent you from being in unsafe
> mode
> > > > (e.g. you cannot disable fsync in SQL Server at all).
> > > >
> > > > If we continue going in this direction, we will end up with a
> product,
> > > > which is unsafe out of the box and require tons of documentation to
> > > > understand how to make it safe. Definitely not the right message to
> the
> > > > market. This is like a car without brakes - would you like to drive
> it?
> > > If
> > > > this is Need For Speed game and you have unlimited lives (in-memory
> > cache
> > > > with backing store), then yes. If this is a real life with
> > (persistence)
> > > -
> > > > then no.
> > > >
> > > > On Fri, Feb 16, 2018 at 5:20 PM, Dmitriy Setrakyan <
> > > dsetrak...@apache.org>
> > > > wrote:
> > > >
> > > > > Well, I cannot say that I like the name LOG_ONLY, but I would vote
> to
> > > > keep
> > > > > it for now, given that it is already documented in many places,
> > blogs,
> > > > and
> > > > > examples.
> > > > >
> > > > > D.
> > > > >
> > > > > On Fri, Feb 16, 2018 at 8:13 AM, Ivan Rakov  >
> > > > wrote:
> > > > >
> > > > > > Looks like it's an Ignite term - I've never heard of it outside
> > > Ignite
> > > > > > scope.
> > > > > >
> > > > > > Though, renaming existing enum value requires keeping old as
> > > > deprecated.
> > > > > > DEFAULT is confusing enough to pay this price.
> > > > > > As for LOG_ONLY, I think we can keep it as long as it has good
> and
> > > > > > definitive javadoc.
> > > > > >
> > > > > > Best Regards,
> > > > > > Ivan Rakov
> > > > > >
> > > > > >
> > > > > > On 16.02.2018 17:07, Dmitriy Setrakyan wrote:
> > > > > >
> > > > > >> Igniters, just to clarify, does the term LOG_ONLY mean anything
> in
> > > the
> > > > > >> industry or is this just an Ignite term?
> > > > > >>
> > > > > >> D.
> > > > > >>
> > > > > >> On Fri, Feb 16, 2018 at 8:03 AM, Anton Vinogradov <
> > > > > >> avinogra...@gridgain.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> Log only mode: flushes application buffers.
> > > > > >>> So, in synced mode without fsync guarantee. That's why I
> propose
> > to
> > > > > >>> rename
> > > > > >>> it as SYNC.
> > > > > >>>
> > > > > >>> On Fri, Feb 16, 2018 at 4:49 PM, Ilya Lantukh <
> > > ilant...@gridgain.com
> > > > >
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>> I am OK with either FSYNC or STRICT variant.
> > > > > 
> > > > >  LOG_ONLY name means "log without fsync".
> > > > > 
> > > > >  On Fri, Feb 16, 2018 at 4:05 PM, Dmitriy Setrakyan <
> > > > > 
> > > > > >>> dsetrak...@apache.org>
> > > > > >>>
> > > > >  wrote:
> > > > > 
> > > > >  On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov <
> > > ivan.glu...@gmail.com>
> > > > > >
> > > > >  wrote:
> > > > > 
> > > > > > Why create a new term to define something that has already
> been
> > > > > >>
> > > > > > defined?
> > > > > 
> > > > > > That makes sense. I'm ok with FSYNC.
> > > > > >> Anton, I don't understand why we should rename LOG_ONLY to
> > SYNC.
> > > > We
> > > > > >> started this discussion with bad 

[jira] [Created] (IGNITE-7741) Fix javadoc for QR factorization

[Yury Babak (JIRA)]

Yury Babak created IGNITE-7741:
--

 Summary: Fix javadoc for QR factorization
 Key: IGNITE-7741
 URL: https://issues.apache.org/jira/browse/IGNITE-7741
 Project: Ignite
  Issue Type: Bug
  Components: ml
Reporter: Yury Babak


Wrong javadoc for QR factorization.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Apache Ignite 2.4 release

[Dmitry Pavlov]

Thank you. Data can't be corrupted in case crash because of WAL replay
(since completed checkpoint). Physical records are used to restore probably
corrupted pages in persistent store (we overwrite so called 'grey zone' -
pages we don't know for sure if they have been written).

Only one effect is unwritten one or several last transactions. It is not
the same with corrupted data.

пт, 16 февр. 2018 г. в 19:19, Vladimir Ozerov :

> Log only mode is not safe - data might be corrupted in case of system
> crash. Oracle - fsync, Postgres - fsync, SQL Server - fsync, Cassandra -
> similar to our “background”.
>
> пт, 16 февр. 2018 г. в 19:11, Dmitry Pavlov :
>
> > Hi Vladimir,
> >
> > What you saying is defenetely make sence.
> >
> > In the same time LOG_ONLY is also safe mode, user will be able to restore
> > system after crash. If it is not true, we should create critical ticket
> and
> > fix it.
> >
> > Do you know other databases defaults, such as Cassandra, Oracle, Postgre?
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > пт, 16 февр. 2018 г. в 18:41, Vladimir Ozerov :
> >
> > > Igniters,
> > >
> > > Sorry for pouring oil on the flames, but from database perspective
> moving
> > > from FSYNC to non-FSYNC mode appears to be a mistake. When you work
> with
> > > database, your main expectation is that it will save your data. All
> > > production database vendor make sure that you are safe, not that you
> are
> > > fast. Moreover, some vendors even prevent you from being in unsafe mode
> > > (e.g. you cannot disable fsync in SQL Server at all).
> > >
> > > If we continue going in this direction, we will end up with a product,
> > > which is unsafe out of the box and require tons of documentation to
> > > understand how to make it safe. Definitely not the right message to the
> > > market. This is like a car without brakes - would you like to drive it?
> > If
> > > this is Need For Speed game and you have unlimited lives (in-memory
> cache
> > > with backing store), then yes. If this is a real life with
> (persistence)
> > -
> > > then no.
> > >
> > > On Fri, Feb 16, 2018 at 5:20 PM, Dmitriy Setrakyan <
> > dsetrak...@apache.org>
> > > wrote:
> > >
> > > > Well, I cannot say that I like the name LOG_ONLY, but I would vote to
> > > keep
> > > > it for now, given that it is already documented in many places,
> blogs,
> > > and
> > > > examples.
> > > >
> > > > D.
> > > >
> > > > On Fri, Feb 16, 2018 at 8:13 AM, Ivan Rakov 
> > > wrote:
> > > >
> > > > > Looks like it's an Ignite term - I've never heard of it outside
> > Ignite
> > > > > scope.
> > > > >
> > > > > Though, renaming existing enum value requires keeping old as
> > > deprecated.
> > > > > DEFAULT is confusing enough to pay this price.
> > > > > As for LOG_ONLY, I think we can keep it as long as it has good and
> > > > > definitive javadoc.
> > > > >
> > > > > Best Regards,
> > > > > Ivan Rakov
> > > > >
> > > > >
> > > > > On 16.02.2018 17:07, Dmitriy Setrakyan wrote:
> > > > >
> > > > >> Igniters, just to clarify, does the term LOG_ONLY mean anything in
> > the
> > > > >> industry or is this just an Ignite term?
> > > > >>
> > > > >> D.
> > > > >>
> > > > >> On Fri, Feb 16, 2018 at 8:03 AM, Anton Vinogradov <
> > > > >> avinogra...@gridgain.com>
> > > > >> wrote:
> > > > >>
> > > > >> Log only mode: flushes application buffers.
> > > > >>> So, in synced mode without fsync guarantee. That's why I propose
> to
> > > > >>> rename
> > > > >>> it as SYNC.
> > > > >>>
> > > > >>> On Fri, Feb 16, 2018 at 4:49 PM, Ilya Lantukh <
> > ilant...@gridgain.com
> > > >
> > > > >>> wrote:
> > > > >>>
> > > > >>> I am OK with either FSYNC or STRICT variant.
> > > > 
> > > >  LOG_ONLY name means "log without fsync".
> > > > 
> > > >  On Fri, Feb 16, 2018 at 4:05 PM, Dmitriy Setrakyan <
> > > > 
> > > > >>> dsetrak...@apache.org>
> > > > >>>
> > > >  wrote:
> > > > 
> > > >  On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov <
> > ivan.glu...@gmail.com>
> > > > >
> > > >  wrote:
> > > > 
> > > > > Why create a new term to define something that has already been
> > > > >>
> > > > > defined?
> > > > 
> > > > > That makes sense. I'm ok with FSYNC.
> > > > >> Anton, I don't understand why we should rename LOG_ONLY to
> SYNC.
> > > We
> > > > >> started this discussion with bad naming of DEFAULT, but this
> has
> > > > >>
> > > > > nothing
> > > > 
> > > > > to
> > > > >
> > > > >> do with LOG_ONLY (even though it may be scientific - but SYNC
> > > sounds
> > > > >> scientific as well).
> > > > >>
> > > > >> I agree with Ivan, we should not go wild with renaming.
> > However, I
> > > > >
> > > >  would
> > > > >>>
> > > >  like to find out what is the meaning behind the LOG_ONLY name.
> Can
> > > > >
> > > >  someone
> > > > 
> > > > > explain?
> > > > >
> > 

Re: Apache Ignite 2.4 release

[Vladimir Ozerov]

Log only mode is not safe - data might be corrupted in case of system
crash. Oracle - fsync, Postgres - fsync, SQL Server - fsync, Cassandra -
similar to our “background”.

пт, 16 февр. 2018 г. в 19:11, Dmitry Pavlov :

> Hi Vladimir,
>
> What you saying is defenetely make sence.
>
> In the same time LOG_ONLY is also safe mode, user will be able to restore
> system after crash. If it is not true, we should create critical ticket and
> fix it.
>
> Do you know other databases defaults, such as Cassandra, Oracle, Postgre?
>
> Sincerely,
> Dmitriy Pavlov
>
> пт, 16 февр. 2018 г. в 18:41, Vladimir Ozerov :
>
> > Igniters,
> >
> > Sorry for pouring oil on the flames, but from database perspective moving
> > from FSYNC to non-FSYNC mode appears to be a mistake. When you work with
> > database, your main expectation is that it will save your data. All
> > production database vendor make sure that you are safe, not that you are
> > fast. Moreover, some vendors even prevent you from being in unsafe mode
> > (e.g. you cannot disable fsync in SQL Server at all).
> >
> > If we continue going in this direction, we will end up with a product,
> > which is unsafe out of the box and require tons of documentation to
> > understand how to make it safe. Definitely not the right message to the
> > market. This is like a car without brakes - would you like to drive it?
> If
> > this is Need For Speed game and you have unlimited lives (in-memory cache
> > with backing store), then yes. If this is a real life with (persistence)
> -
> > then no.
> >
> > On Fri, Feb 16, 2018 at 5:20 PM, Dmitriy Setrakyan <
> dsetrak...@apache.org>
> > wrote:
> >
> > > Well, I cannot say that I like the name LOG_ONLY, but I would vote to
> > keep
> > > it for now, given that it is already documented in many places, blogs,
> > and
> > > examples.
> > >
> > > D.
> > >
> > > On Fri, Feb 16, 2018 at 8:13 AM, Ivan Rakov 
> > wrote:
> > >
> > > > Looks like it's an Ignite term - I've never heard of it outside
> Ignite
> > > > scope.
> > > >
> > > > Though, renaming existing enum value requires keeping old as
> > deprecated.
> > > > DEFAULT is confusing enough to pay this price.
> > > > As for LOG_ONLY, I think we can keep it as long as it has good and
> > > > definitive javadoc.
> > > >
> > > > Best Regards,
> > > > Ivan Rakov
> > > >
> > > >
> > > > On 16.02.2018 17:07, Dmitriy Setrakyan wrote:
> > > >
> > > >> Igniters, just to clarify, does the term LOG_ONLY mean anything in
> the
> > > >> industry or is this just an Ignite term?
> > > >>
> > > >> D.
> > > >>
> > > >> On Fri, Feb 16, 2018 at 8:03 AM, Anton Vinogradov <
> > > >> avinogra...@gridgain.com>
> > > >> wrote:
> > > >>
> > > >> Log only mode: flushes application buffers.
> > > >>> So, in synced mode without fsync guarantee. That's why I propose to
> > > >>> rename
> > > >>> it as SYNC.
> > > >>>
> > > >>> On Fri, Feb 16, 2018 at 4:49 PM, Ilya Lantukh <
> ilant...@gridgain.com
> > >
> > > >>> wrote:
> > > >>>
> > > >>> I am OK with either FSYNC or STRICT variant.
> > > 
> > >  LOG_ONLY name means "log without fsync".
> > > 
> > >  On Fri, Feb 16, 2018 at 4:05 PM, Dmitriy Setrakyan <
> > > 
> > > >>> dsetrak...@apache.org>
> > > >>>
> > >  wrote:
> > > 
> > >  On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov <
> ivan.glu...@gmail.com>
> > > >
> > >  wrote:
> > > 
> > > > Why create a new term to define something that has already been
> > > >>
> > > > defined?
> > > 
> > > > That makes sense. I'm ok with FSYNC.
> > > >> Anton, I don't understand why we should rename LOG_ONLY to SYNC.
> > We
> > > >> started this discussion with bad naming of DEFAULT, but this has
> > > >>
> > > > nothing
> > > 
> > > > to
> > > >
> > > >> do with LOG_ONLY (even though it may be scientific - but SYNC
> > sounds
> > > >> scientific as well).
> > > >>
> > > >> I agree with Ivan, we should not go wild with renaming.
> However, I
> > > >
> > >  would
> > > >>>
> > >  like to find out what is the meaning behind the LOG_ONLY name. Can
> > > >
> > >  someone
> > > 
> > > > explain?
> > > >
> > > > D.
> > > >
> > > >
> > > 
> > >  --
> > >  Best regards,
> > >  Ilya
> > > 
> > > 
> > > >
> > >
> >
>

Re: Apache Ignite 2.4 release

[Dmitry Pavlov]

Hi Vladimir,

What you saying is defenetely make sence.

In the same time LOG_ONLY is also safe mode, user will be able to restore
system after crash. If it is not true, we should create critical ticket and
fix it.

Do you know other databases defaults, such as Cassandra, Oracle, Postgre?

Sincerely,
Dmitriy Pavlov

пт, 16 февр. 2018 г. в 18:41, Vladimir Ozerov :

> Igniters,
>
> Sorry for pouring oil on the flames, but from database perspective moving
> from FSYNC to non-FSYNC mode appears to be a mistake. When you work with
> database, your main expectation is that it will save your data. All
> production database vendor make sure that you are safe, not that you are
> fast. Moreover, some vendors even prevent you from being in unsafe mode
> (e.g. you cannot disable fsync in SQL Server at all).
>
> If we continue going in this direction, we will end up with a product,
> which is unsafe out of the box and require tons of documentation to
> understand how to make it safe. Definitely not the right message to the
> market. This is like a car without brakes - would you like to drive it? If
> this is Need For Speed game and you have unlimited lives (in-memory cache
> with backing store), then yes. If this is a real life with (persistence) -
> then no.
>
> On Fri, Feb 16, 2018 at 5:20 PM, Dmitriy Setrakyan 
> wrote:
>
> > Well, I cannot say that I like the name LOG_ONLY, but I would vote to
> keep
> > it for now, given that it is already documented in many places, blogs,
> and
> > examples.
> >
> > D.
> >
> > On Fri, Feb 16, 2018 at 8:13 AM, Ivan Rakov 
> wrote:
> >
> > > Looks like it's an Ignite term - I've never heard of it outside Ignite
> > > scope.
> > >
> > > Though, renaming existing enum value requires keeping old as
> deprecated.
> > > DEFAULT is confusing enough to pay this price.
> > > As for LOG_ONLY, I think we can keep it as long as it has good and
> > > definitive javadoc.
> > >
> > > Best Regards,
> > > Ivan Rakov
> > >
> > >
> > > On 16.02.2018 17:07, Dmitriy Setrakyan wrote:
> > >
> > >> Igniters, just to clarify, does the term LOG_ONLY mean anything in the
> > >> industry or is this just an Ignite term?
> > >>
> > >> D.
> > >>
> > >> On Fri, Feb 16, 2018 at 8:03 AM, Anton Vinogradov <
> > >> avinogra...@gridgain.com>
> > >> wrote:
> > >>
> > >> Log only mode: flushes application buffers.
> > >>> So, in synced mode without fsync guarantee. That's why I propose to
> > >>> rename
> > >>> it as SYNC.
> > >>>
> > >>> On Fri, Feb 16, 2018 at 4:49 PM, Ilya Lantukh  >
> > >>> wrote:
> > >>>
> > >>> I am OK with either FSYNC or STRICT variant.
> > 
> >  LOG_ONLY name means "log without fsync".
> > 
> >  On Fri, Feb 16, 2018 at 4:05 PM, Dmitriy Setrakyan <
> > 
> > >>> dsetrak...@apache.org>
> > >>>
> >  wrote:
> > 
> >  On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov 
> > >
> >  wrote:
> > 
> > > Why create a new term to define something that has already been
> > >>
> > > defined?
> > 
> > > That makes sense. I'm ok with FSYNC.
> > >> Anton, I don't understand why we should rename LOG_ONLY to SYNC.
> We
> > >> started this discussion with bad naming of DEFAULT, but this has
> > >>
> > > nothing
> > 
> > > to
> > >
> > >> do with LOG_ONLY (even though it may be scientific - but SYNC
> sounds
> > >> scientific as well).
> > >>
> > >> I agree with Ivan, we should not go wild with renaming. However, I
> > >
> >  would
> > >>>
> >  like to find out what is the meaning behind the LOG_ONLY name. Can
> > >
> >  someone
> > 
> > > explain?
> > >
> > > D.
> > >
> > >
> > 
> >  --
> >  Best regards,
> >  Ilya
> > 
> > 
> > >
> >
>

Re: Apache Ignite 2.4 release

[Vladimir Ozerov]

Igniters,

Sorry for pouring oil on the flames, but from database perspective moving
from FSYNC to non-FSYNC mode appears to be a mistake. When you work with
database, your main expectation is that it will save your data. All
production database vendor make sure that you are safe, not that you are
fast. Moreover, some vendors even prevent you from being in unsafe mode
(e.g. you cannot disable fsync in SQL Server at all).

If we continue going in this direction, we will end up with a product,
which is unsafe out of the box and require tons of documentation to
understand how to make it safe. Definitely not the right message to the
market. This is like a car without brakes - would you like to drive it? If
this is Need For Speed game and you have unlimited lives (in-memory cache
with backing store), then yes. If this is a real life with (persistence) -
then no.

On Fri, Feb 16, 2018 at 5:20 PM, Dmitriy Setrakyan 
wrote:

> Well, I cannot say that I like the name LOG_ONLY, but I would vote to keep
> it for now, given that it is already documented in many places, blogs, and
> examples.
>
> D.
>
> On Fri, Feb 16, 2018 at 8:13 AM, Ivan Rakov  wrote:
>
> > Looks like it's an Ignite term - I've never heard of it outside Ignite
> > scope.
> >
> > Though, renaming existing enum value requires keeping old as deprecated.
> > DEFAULT is confusing enough to pay this price.
> > As for LOG_ONLY, I think we can keep it as long as it has good and
> > definitive javadoc.
> >
> > Best Regards,
> > Ivan Rakov
> >
> >
> > On 16.02.2018 17:07, Dmitriy Setrakyan wrote:
> >
> >> Igniters, just to clarify, does the term LOG_ONLY mean anything in the
> >> industry or is this just an Ignite term?
> >>
> >> D.
> >>
> >> On Fri, Feb 16, 2018 at 8:03 AM, Anton Vinogradov <
> >> avinogra...@gridgain.com>
> >> wrote:
> >>
> >> Log only mode: flushes application buffers.
> >>> So, in synced mode without fsync guarantee. That's why I propose to
> >>> rename
> >>> it as SYNC.
> >>>
> >>> On Fri, Feb 16, 2018 at 4:49 PM, Ilya Lantukh 
> >>> wrote:
> >>>
> >>> I am OK with either FSYNC or STRICT variant.
> 
>  LOG_ONLY name means "log without fsync".
> 
>  On Fri, Feb 16, 2018 at 4:05 PM, Dmitriy Setrakyan <
> 
> >>> dsetrak...@apache.org>
> >>>
>  wrote:
> 
>  On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov 
> >
>  wrote:
> 
> > Why create a new term to define something that has already been
> >>
> > defined?
> 
> > That makes sense. I'm ok with FSYNC.
> >> Anton, I don't understand why we should rename LOG_ONLY to SYNC. We
> >> started this discussion with bad naming of DEFAULT, but this has
> >>
> > nothing
> 
> > to
> >
> >> do with LOG_ONLY (even though it may be scientific - but SYNC sounds
> >> scientific as well).
> >>
> >> I agree with Ivan, we should not go wild with renaming. However, I
> >
>  would
> >>>
>  like to find out what is the meaning behind the LOG_ONLY name. Can
> >
>  someone
> 
> > explain?
> >
> > D.
> >
> >
> 
>  --
>  Best regards,
>  Ilya
> 
> 
> >
>

[jira] [Created] (IGNITE-7740) IGFS caches returned by cache.context().queries().sqlMetadata()

[Alexey Kuznetsov (JIRA)]

Alexey Kuznetsov created IGNITE-7740:


 Summary: IGFS caches returned by 
cache.context().queries().sqlMetadata() 
 Key: IGNITE-7740
 URL: https://issues.apache.org/jira/browse/IGNITE-7740
 Project: Ignite
  Issue Type: Improvement
Reporter: Alexey Kuznetsov
Assignee: Vladimir Ozerov


IGFS are special caches not related to queries and should not returned by 

{code}

cache.context().queries().sqlMetadata();

{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: TeamCity. Ignite RDD tests

[vveider]

Nikolay, I raised your permissions to Ignite Committer status.
Check it out, please.



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[jira] [Created] (IGNITE-7739) Write about Spring Injection and IgniteSpringBean in documentation

[Ilya Kasnacheev (JIRA)]

Ilya Kasnacheev created IGNITE-7739:
---

 Summary: Write about Spring Injection and IgniteSpringBean in 
documentation
 Key: IGNITE-7739
 URL: https://issues.apache.org/jira/browse/IGNITE-7739
 Project: Ignite
  Issue Type: Task
  Components: documentation
Reporter: Ilya Kasnacheev


Currently there is no mention of IgniteSpringBean on readme.io

We have section https://apacheignite.readme.io/docs/resource-injection but it's 
totally not obvious how to make e.g. SpringResource work, what are 
pre-requisites.

Thing is, if Ignite creates Spring context (via Ignition), then Spring 
injection would work out of box.
If Spring context creates Ignite as a bean, that bean should be 
IgniteSpringBean and not Ignite. Otherwise, Spring context and lifecycle would 
not be visible to Ignite and injection won't work.

On unrelated note, maybe we need support for Spring Boot already? See 
https://habrahabr.ru/post/310672/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #3533: IGNITE-7737

[devozerov]

Github user devozerov closed the pull request at:

https://github.com/apache/ignite/pull/3533


---

[GitHub] ignite pull request #3537: IGNITE-7537: [UNFINISHED] SQL COPY: CSV parsing o...

[gg-shq]

GitHub user gg-shq opened a pull request:

https://github.com/apache/ignite/pull/3537

IGNITE-7537: [UNFINISHED] SQL COPY: CSV parsing options

The CSV parser code is not yet finished, however the rest of the code (SQL 
COPY command parameters) is close to completion.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7537

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3537.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3537


commit deb7994f0fd4233e3e0b699794b9066af87195c7
Author: gg-shq 
Date:   2018-01-19T12:13:37Z

IGNITE-6917: Intermediate commit

commit e7747a58c2cdacc6987d625a46d1f79a81863cd3
Author: gg-shq 
Date:   2018-01-19T17:21:50Z

IGNITE-6917: Intermediate commit

commit 6f37e6751285a96bdf757b392e1d4113bb47ee48
Author: gg-shq 
Date:   2018-01-19T17:30:22Z

IGNITE-6917: Intermediate commit

commit 49f0324c77d0bb3b4ec87317b1ecbde1bd6f34b1
Author: gg-shq 
Date:   2018-01-22T10:27:34Z

IGNITE-6917: Intermediate commit

commit a5bec61d41d8dc242cfbf11a7cf03c23bbbcd7c3
Author: gg-shq 
Date:   2018-01-22T12:25:04Z

IGNITE-6917: Intermediate commit

commit e18e18696fc92b93b17decf087721c693625ac36
Author: gg-shq 
Date:   2018-01-22T12:35:56Z

IGNITE-6917: Intermediate commit

commit 990c04919e181535e57290ee2516a9603657c160
Author: gg-shq 
Date:   2018-01-22T16:18:18Z

IGNITE-6917: Intermediate commit

commit 8b163410845a6e6233fa8a2746402651ccea3f69
Author: gg-shq 
Date:   2018-01-22T17:16:20Z

IGNITE-6917: Intermediate commit

commit faf762815ef58865c560d6de722be446e429c61d
Author: gg-shq 
Date:   2018-01-23T12:37:56Z

IGNITE-6917: Intermediate commit

commit f79343f04360b913b32403d5aa0defaf5d04b357
Author: gg-shq 
Date:   2018-01-23T12:40:34Z

IGNITE-6917: Intermediate commit

commit efc5d7ab9bad52aaad0872977495a158b0e47770
Author: gg-shq 
Date:   2018-01-23T13:50:04Z

IGNITE-6917: Added BATCH_SIZE parameter to COPY SQL command for internal 
testing. Adding tests.

commit b61db5de48d9a91a658f6133a2ad2544f358ebbf
Author: gg-shq 
Date:   2018-01-24T11:18:55Z

IGNITE-6917: Adding tests. Clarifying default columns set.

commit aa31488b74c74f881c247339a4b2bd31bf45b849
Author: gg-shq 
Date:   2018-01-24T19:00:17Z

IGNITE-6917: More tests, more logging, cleanups, streaming CSV decoder

commit 01125f4bb68bc4ae958cae1d2f8f7dee493fa55e
Author: gg-shq 
Date:   2018-01-25T11:54:29Z

IGNITE-6917: Javadoc, added BulkLoadCacheWriter.

commit 1a21cd91b3571a23d21bab7cb653478312178bb0
Author: gg-shq 
Date:   2018-01-25T12:29:03Z

IGNITE-6917: Javadoc, javadoc, javadoc.

commit a34060392bd5d86b0118fbd26127460d54f918c3
Author: gg-shq 
Date:   2018-01-25T12:29:57Z

IGNITE-6917: Fixed a syntax error added involuntarily in the previous 
commit.

commit ccaef2e349a728145c77c50d96d24e8a38ac35e1
Author: gg-shq 
Date:   2018-01-25T13:31:40Z

IGNITE-6917: Fixed charset decoder bugs, tests, handling of empty lines

commit b4cf0a4a4fb6cf3c6c35f09fe99ac5954541a679
Author: gg-shq 
Date:   2018-01-26T10:54:01Z

Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/ignite 
into ignite-6917-1

# Conflicts:
#   modules/core/src/main/java/org/apache/ignite/internal/sql/SqlParser.java
#   
modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/h2/IgniteH2Indexing.java

commit 26694bc00b76895d4f22e8416af29000197230ec
Author: gg-shq 
Date:   2018-01-26T12:45:32Z

IGNITE-6917: Moved syntax tests to a separate file, moved truncated rows 
handling from UpdatePlan.processRow() to a different place, minor changes

commit 1d7a9f8818dff4c4bc6a8f9d509a23be394b3e59
Author: gg-shq 
Date:   2018-01-26T12:55:47Z

IGNITE-6917: Find input files by using IgniteUtils.resolveIgnitePath(), 
test fixes.

commit 58c9b2dc7190c149e9c9fa377a2581849bb41420
Author: gg-shq 
Date:   2018-01-26T12:58:58Z

Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/ignite 
into ignite-6917-1

commit 47eace53c82770b076b26bbba87944c872a941ad
Author: gg-shq 
Date:   2018-01-26T13:06:11Z

IGNITE-6917: Javadoc, tidying up.

commit ba0f9c822a0873662a5505892956ca4e68d87e56
Author: gg-shq 
Date:   2018-01-26T14:53:10Z

IGNITE-6917: Added error reporting and tests for batch mode and into jdbc2 
driver.

commit 6e550ef3f0de02ceb0a24e4787906cacb320e54e
Author: gg-shq 
Date:   2018-01-26T14:55:37Z

IGNITE-6917: Minor 

Re: Apache Ignite 2.4 release

[Dmitriy Setrakyan]

Well, I cannot say that I like the name LOG_ONLY, but I would vote to keep
it for now, given that it is already documented in many places, blogs, and
examples.

D.

On Fri, Feb 16, 2018 at 8:13 AM, Ivan Rakov  wrote:

> Looks like it's an Ignite term - I've never heard of it outside Ignite
> scope.
>
> Though, renaming existing enum value requires keeping old as deprecated.
> DEFAULT is confusing enough to pay this price.
> As for LOG_ONLY, I think we can keep it as long as it has good and
> definitive javadoc.
>
> Best Regards,
> Ivan Rakov
>
>
> On 16.02.2018 17:07, Dmitriy Setrakyan wrote:
>
>> Igniters, just to clarify, does the term LOG_ONLY mean anything in the
>> industry or is this just an Ignite term?
>>
>> D.
>>
>> On Fri, Feb 16, 2018 at 8:03 AM, Anton Vinogradov <
>> avinogra...@gridgain.com>
>> wrote:
>>
>> Log only mode: flushes application buffers.
>>> So, in synced mode without fsync guarantee. That's why I propose to
>>> rename
>>> it as SYNC.
>>>
>>> On Fri, Feb 16, 2018 at 4:49 PM, Ilya Lantukh 
>>> wrote:
>>>
>>> I am OK with either FSYNC or STRICT variant.

 LOG_ONLY name means "log without fsync".

 On Fri, Feb 16, 2018 at 4:05 PM, Dmitriy Setrakyan <

>>> dsetrak...@apache.org>
>>>
 wrote:

 On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov 
>
 wrote:

> Why create a new term to define something that has already been
>>
> defined?

> That makes sense. I'm ok with FSYNC.
>> Anton, I don't understand why we should rename LOG_ONLY to SYNC. We
>> started this discussion with bad naming of DEFAULT, but this has
>>
> nothing

> to
>
>> do with LOG_ONLY (even though it may be scientific - but SYNC sounds
>> scientific as well).
>>
>> I agree with Ivan, we should not go wild with renaming. However, I
>
 would
>>>
 like to find out what is the meaning behind the LOG_ONLY name. Can
>
 someone

> explain?
>
> D.
>
>

 --
 Best regards,
 Ilya


>

Re: Apache Ignite 2.4 release

[Ivan Rakov]

Looks like it's an Ignite term - I've never heard of it outside Ignite 
scope.


Though, renaming existing enum value requires keeping old as deprecated.
DEFAULT is confusing enough to pay this price.
As for LOG_ONLY, I think we can keep it as long as it has good and 
definitive javadoc.


Best Regards,
Ivan Rakov

On 16.02.2018 17:07, Dmitriy Setrakyan wrote:

Igniters, just to clarify, does the term LOG_ONLY mean anything in the
industry or is this just an Ignite term?

D.

On Fri, Feb 16, 2018 at 8:03 AM, Anton Vinogradov 
wrote:


Log only mode: flushes application buffers.
So, in synced mode without fsync guarantee. That's why I propose to rename
it as SYNC.

On Fri, Feb 16, 2018 at 4:49 PM, Ilya Lantukh 
wrote:


I am OK with either FSYNC or STRICT variant.

LOG_ONLY name means "log without fsync".

On Fri, Feb 16, 2018 at 4:05 PM, Dmitriy Setrakyan <

dsetrak...@apache.org>

wrote:


On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov 

wrote:

Why create a new term to define something that has already been

defined?

That makes sense. I'm ok with FSYNC.
Anton, I don't understand why we should rename LOG_ONLY to SYNC. We
started this discussion with bad naming of DEFAULT, but this has

nothing

to

do with LOG_ONLY (even though it may be scientific - but SYNC sounds
scientific as well).


I agree with Ivan, we should not go wild with renaming. However, I

would

like to find out what is the meaning behind the LOG_ONLY name. Can

someone

explain?

D.




--
Best regards,
Ilya

Re: Apache Ignite 2.4 release

[Dmitriy Setrakyan]

Igniters, just to clarify, does the term LOG_ONLY mean anything in the
industry or is this just an Ignite term?

D.

On Fri, Feb 16, 2018 at 8:03 AM, Anton Vinogradov 
wrote:

> Log only mode: flushes application buffers.
> So, in synced mode without fsync guarantee. That's why I propose to rename
> it as SYNC.
>
> On Fri, Feb 16, 2018 at 4:49 PM, Ilya Lantukh 
> wrote:
>
> > I am OK with either FSYNC or STRICT variant.
> >
> > LOG_ONLY name means "log without fsync".
> >
> > On Fri, Feb 16, 2018 at 4:05 PM, Dmitriy Setrakyan <
> dsetrak...@apache.org>
> > wrote:
> >
> > > On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov 
> > wrote:
> > >
> > > > Why create a new term to define something that has already been
> > defined?
> > > >>
> > > > That makes sense. I'm ok with FSYNC.
> > > > Anton, I don't understand why we should rename LOG_ONLY to SYNC. We
> > > > started this discussion with bad naming of DEFAULT, but this has
> > nothing
> > > to
> > > > do with LOG_ONLY (even though it may be scientific - but SYNC sounds
> > > > scientific as well).
> > > >
> > >
> > > I agree with Ivan, we should not go wild with renaming. However, I
> would
> > > like to find out what is the meaning behind the LOG_ONLY name. Can
> > someone
> > > explain?
> > >
> > > D.
> > >
> >
> >
> >
> > --
> > Best regards,
> > Ilya
> >
>

Re: Apache Ignite 2.4 release

[Anton Vinogradov]

Log only mode: flushes application buffers.
So, in synced mode without fsync guarantee. That's why I propose to rename
it as SYNC.

On Fri, Feb 16, 2018 at 4:49 PM, Ilya Lantukh  wrote:

> I am OK with either FSYNC or STRICT variant.
>
> LOG_ONLY name means "log without fsync".
>
> On Fri, Feb 16, 2018 at 4:05 PM, Dmitriy Setrakyan 
> wrote:
>
> > On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov 
> wrote:
> >
> > > Why create a new term to define something that has already been
> defined?
> > >>
> > > That makes sense. I'm ok with FSYNC.
> > > Anton, I don't understand why we should rename LOG_ONLY to SYNC. We
> > > started this discussion with bad naming of DEFAULT, but this has
> nothing
> > to
> > > do with LOG_ONLY (even though it may be scientific - but SYNC sounds
> > > scientific as well).
> > >
> >
> > I agree with Ivan, we should not go wild with renaming. However, I would
> > like to find out what is the meaning behind the LOG_ONLY name. Can
> someone
> > explain?
> >
> > D.
> >
>
>
>
> --
> Best regards,
> Ilya
>

[jira] [Created] (IGNITE-7738) Allow 'multiple statements' in thin JDBC streaming mode

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-7738:
---

 Summary: Allow 'multiple statements' in thin JDBC streaming mode
 Key: IGNITE-7738
 URL: https://issues.apache.org/jira/browse/IGNITE-7738
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Reporter: Alexander Paschenko


We need to update thin JDBC protocol to let user run multiple statements via 
Statement.execute(String) when connection is in streamed mode. Currently in 
streaming mode the server always receives all SQL in batches and its batch 
processing logic does not allow multiple statements altogether. If we're able 
to recognize initial nature of the statement, we'll be able to act 
appropriately, and for this to be possible additional information must be 
passed with each query in the batch.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Apache Ignite 2.4 release

[Ilya Lantukh]

I am OK with either FSYNC or STRICT variant.

LOG_ONLY name means "log without fsync".

On Fri, Feb 16, 2018 at 4:05 PM, Dmitriy Setrakyan 
wrote:

> On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov  wrote:
>
> > Why create a new term to define something that has already been defined?
> >>
> > That makes sense. I'm ok with FSYNC.
> > Anton, I don't understand why we should rename LOG_ONLY to SYNC. We
> > started this discussion with bad naming of DEFAULT, but this has nothing
> to
> > do with LOG_ONLY (even though it may be scientific - but SYNC sounds
> > scientific as well).
> >
>
> I agree with Ivan, we should not go wild with renaming. However, I would
> like to find out what is the meaning behind the LOG_ONLY name. Can someone
> explain?
>
> D.
>



-- 
Best regards,
Ilya

[GitHub] ignite pull request #3536: IGNITE-7717 Update discovery caches on topologies...

[Jokser]

GitHub user Jokser opened a pull request:

https://github.com/apache/ignite/pull/3536

IGNITE-7717 Update discovery caches on topologies after exchanges merge



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7717

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3536.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3536


commit 054a71827b0329793afd48720caa084a530a49f9
Author: Pavel Kovalenko 
Date:   2018-02-16T13:36:10Z

IGNITE-7717 Update discovery caches on topologies after exchanges merge.




---

[GitHub] ignite pull request #3510: IGNITE-7676 Add affinity version to snapshot plug...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3510


---

[GitHub] ignite pull request #3523: IGNITE-7699 ObjectBinaryProcessor improvements

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3523


---

Re: TeamCity. Ignite RDD tests

[Nikolay Izhikov]

Hello, Dmitriy.

Seems, that I haven't suffictient access to do test mute.
Can you give me access?
Or please, mute it.

В Пт, 16/02/2018 в 12:03 +, Dmitry Pavlov пишет:
> I think you should be able to mute any test. If not, please write to me.
> 
> пт, 16 февр. 2018 г. в 15:02, Nikolay Izhikov :
> 
> > Thank you, Dmitry.
> > 
> > Can you do it for me?
> > Or I can do it by myself?
> > 
> > 16 февр. 2018 г. 3:00 PM пользователь "Dmitry Pavlov" <
> > dpavlov@gmail.com>
> > написал:
> > 
> > > Hi Nikolay,
> > > 
> > > Ok for me. In this case we have issue to fix.
> > > 
> > > I suggest to mute with comment text with reference to issue in JIRA.
> > > 
> > > Sincerely,
> > > Dmitriy Pavlov
> > > 
> > > пт, 16 февр. 2018 г. в 7:35, Nikolay Izhikov :
> > > 
> > > > Hello, Igniters.
> > > > 
> > > > I'm working on issue [1].
> > > > 
> > > > Team City doesn't collect info about scalatest execution because of
> > 
> > wrong
> > > > pom.xml
> > > > I'm fixed it in PR [3]
> > > > 
> > > > It happens there is a 2 broken tests written in Scala - [4]
> > > > 
> > > > 1. IgniteRDDSpec.IgniteRDD should successfully store data to ignite
> > 
> > using
> > > > saveValues
> > > > 2. IgniteRDDSpec.IgniteRDD should successfully store data to ignite
> > 
> > using
> > > > saveValues with inline transformation
> > > > 
> > > > It seems they are been here for a while.
> > > > I propose to mute or disable it on TeamCity before merging my PR.
> > > > I've created ticket for fixing tests - [5].
> > > > 
> > > > Thoughts?
> > > > 
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-7042
> > > > 
> > > > [2]
> > > > https://ci.ignite.apache.org/viewLog.html?buildId=1096059;;
> > > 
> > > tab=buildResultsDiv=IgniteTests24Java8_IgniteRdd
> > > > 
> > > > [3] https://github.com/apache/ignite/pull/3530
> > > > 
> > > > [4]
> > > > https://ci.ignite.apache.org/viewLog.html?buildId=1095218=
> > > 
> > > IgniteTests24Java8_IgniteRdd=testsInfo
> > > > 
> > > > [5] https://issues.apache.org/jira/browse/IGNITE-7727

signature.asc
Description: This is a digitally signed message part

Re: Apache Ignite 2.4 release

[Dmitriy Setrakyan]

On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov  wrote:

> Why create a new term to define something that has already been defined?
>>
> That makes sense. I'm ok with FSYNC.
> Anton, I don't understand why we should rename LOG_ONLY to SYNC. We
> started this discussion with bad naming of DEFAULT, but this has nothing to
> do with LOG_ONLY (even though it may be scientific - but SYNC sounds
> scientific as well).
>

I agree with Ivan, we should not go wild with renaming. However, I would
like to find out what is the meaning behind the LOG_ONLY name. Can someone
explain?

D.

Re: Batch size parameter at DataStreamerCacheUpdaters.batched()

[Denis Mekhanikov]

Guys,

I think, it makes sense to move this receiver implementation to public API.
It has much better performance, than the default one, that performs single
puts.

I don't see any point in adding a batch size parameter to it though, since
DataStreamer already has such setting itself.

Denis

пт, 16 февр. 2018 г. в 13:29, Roman Guseinov :

> Val,
>
> I got what do you mean. Everything inside the org.apache.ignite.internal is
> considered to be not public API.
>
> Best Regards,
> Roman
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

Re: Apache Ignite 2.4 release

[Ivan Rakov]

Why create a new term to define something that has already been defined?

That makes sense. I'm ok with FSYNC.
Anton, I don't understand why we should rename LOG_ONLY to SYNC. We started 
this discussion with bad naming of DEFAULT, but this has nothing to do with 
LOG_ONLY (even though it may be scientific - but SYNC sounds scientific as 
well).

Best Regards,
Ivan Rakov

On 16.02.2018 15:55, Anton Vinogradov wrote:

  >> I had idea to name old default as FSYNC, but it would be too scientific.
So, then it can be  FSYNC, SYNC, BACKGROUND and NONE!

On Fri, Feb 16, 2018 at 3:49 PM, Dmitriy Setrakyan 
wrote:


On Fri, Feb 16, 2018 at 6:26 AM, Dmitry Pavlov 
wrote:


I had idea to name old default as FSYNC, but it would be too scientific.


I like FSYNC, I do not think it is too scientific. Definitely not more
scientific than LOG_ONLY.

For old DEFAULT, STRICT or STRICT_SYNC - IMO are best options, so I agree

with Ivan.


Not sure I like STRICT. In Linux world, fsync is a well known command which
does exactly what our FSYNC mode will do:
http://man7.org/linux/man-pages/man2/fsync.2.html . STRICT, on the other
hand, is not a commonly understood term. Why create a new term to define
something that has already been defined?

Also, what if tomorrow we need to add an even stricter parameter? Then we
are back to the same problem we are trying to fix right now.

D.

[GitHub] ignite pull request #3535: FIx flacky .Net test ServicesTest.TestCallJavaSer...

[apopovgg]

GitHub user apopovgg opened a pull request:

https://github.com/apache/ignite/pull/3535

FIx flacky .Net test ServicesTest.TestCallJavaService



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-gg-13485

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3535.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3535


commit be91bbf8bcb7e9c71fe6d3bea0f79763f9606558
Author: Krzysztof Chmielewski 
Date:   2017-10-10T14:50:59Z

Fixed "IGNITE-6234 Initialize schemaIds to empty set if schemas field is 
null during the deserialization".

Signed-off-by: nikolay_tikhonov 

commit 08389601728512dc4e7fa5b953f5afe34ae4506f
Author: AMRepo 
Date:   2017-10-10T08:57:20Z

IGNITE-6545: Failure during Ignite Service.cancel() can break normal 
shutdown process. This closes #2807.

(cherry picked from commit 8ffa109)

commit 57547b5afae059a0a6dfa13c08b2e0b6c0e96ebd
Author: devozerov 
Date:   2017-10-13T09:34:35Z

Merge branch 'ignite-2.3.1' into ignite-2.3.2

commit 08798f8e47bdfdd68a557385ed2ce98b4bb1609a
Author: devozerov 
Date:   2017-10-13T11:12:44Z

IGNITE-6605: SQL: common backup filter. This closes #2836.

commit 2b59a241de3935a338842b8fc3221aedc8e11e1d
Author: devozerov 
Date:   2017-10-16T07:33:36Z

IGNITE-6631: Minor improvements to GridH2KeyValueRowOnheap. This closes 
#2855.

commit 98438c954c5f9a08634cf3132361268456397864
Author: devozerov 
Date:   2017-10-16T09:38:54Z

IGNITE-6632: SQL: simplified GridH2Row inheritance tree. This closes #2856.

commit 95b7ab518dd3c3db6fcc5142c2ee85da2516c2b6
Author: devozerov 
Date:   2017-10-16T10:37:11Z

IGNITE-6634: Removed IgniteDistributedJoinTestSuite. It's tests are 
distributed between "Query" and "Query 2" suites. This closes #2857.

commit 9c91deff877ebc0eed84559d4abca71408e3cb0a
Author: devozerov 
Date:   2017-10-16T13:46:13Z

Merge branch 'ignite-2.3.1' into ignite-2.3.2

commit 911ab7ab7a8a6968d219b053adb2338477738506
Author: Alexey Popov 
Date:   2017-10-17T11:45:42Z

IGNITE-6627 .NET: Fix serialization of enums within generic collections

* Fix EnumEqualityComparer serialization
* Fix enum arrays serialization
* Fix empty objects missing metadata

This closes #2864

commit 3ba374c319ac7048a05871692060e2f143d6acdf
Author: Alexey Kuznetsov 
Date:   2017-10-06T17:11:37Z

IGNITE-6463 Web Console: Fixed output of big numbers in SQL query results.
(cherry picked from commit 35589a7)

commit b67feb0f175bfbd6ffbefe82a8d693c8ab7d4213
Author: Vasiliy Sisko 
Date:   2017-10-09T10:55:23Z

IGNITE-5767 Web console: Use byte array type instead of java.lang.Object 
for binary JDBC types.
(cherry picked from commit 3184437)

commit 8e1560322b87d79b3d3250832a3969ac4032d6fc
Author: Alexey Kuznetsov 
Date:   2017-10-06T18:10:08Z

IGNITE-6574 Remove pending requests in case STATUS_AUTH_FAILURE && 
credentials == null.
(cherry picked from commit 85261a3)

commit 7a0300ae35894c389b126e95615f720a99a3d360
Author: devozerov 
Date:   2017-10-18T11:18:08Z

Merge branch 'ignite-2.3.1' into ignite-2.3.2

commit ad01f9b099d0bf92537378859ad6d5a52de57748
Author: Alexey Kuznetsov 
Date:   2017-10-19T02:43:20Z

IGNITE-6647 Web Console: Implemented support of schema migration scripts.
(cherry picked from commit c65399c)

commit 0c66344bc752dac98b256dd140fcab95d1662862
Author: Pavel Tupitsyn 
Date:   2017-10-19T09:36:39Z

IGNITE-6627 .NET: Fix repeated known metadata updates

This closes #2876

commit 1b8abd214ed2afcd3fd1f6a4c71a19d6fe1a4b01
Author: Alexey Kuznetsov 
Date:   2017-10-20T04:23:23Z

IGNITE-6647 Added missing Mongo injector.
(cherry picked from commit 173ecef)

commit a221066b3d029afc392be704a810c0e830fc0c49
Author: Alexey Kuznetsov 
Date:   2017-10-20T14:15:02Z

IGNITE-6647 Web Console: Added folder for modules migrations.
(cherry picked from commit 3700717)

commit da8a9d5a968ba071697a28adb01bc59f80d1893c
Author: Pavel Tupitsyn 
Date:   2017-10-23T08:55:33Z

Merge branch 'ignite-2.3.1' into ignite-2.3.2

# Conflicts:
#   
modules/platforms/dotnet/Apache.Ignite.Core.Tests/Apache.Ignite.Core.Tests.csproj

commit 69fdac3acf768ecb9df80d4412c4de5ffd5bc4f5
Author: Dmitriy Shabalin 
Date:   2017-10-23T09:09:47Z

IGNITE-5909 Added list editable component.
(cherry picked from commit 01daee6)

commit 4a2c38333c112d4956d6394667672c1470503435
Author: apopov 

Re: Apache Ignite 2.4 release

[Anton Vinogradov]

 >> I had idea to name old default as FSYNC, but it would be too scientific.
So, then it can be  FSYNC, SYNC, BACKGROUND and NONE!

On Fri, Feb 16, 2018 at 3:49 PM, Dmitriy Setrakyan 
wrote:

> On Fri, Feb 16, 2018 at 6:26 AM, Dmitry Pavlov 
> wrote:
>
> > I had idea to name old default as FSYNC, but it would be too scientific.
> >
>
> I like FSYNC, I do not think it is too scientific. Definitely not more
> scientific than LOG_ONLY.
>
> For old DEFAULT, STRICT or STRICT_SYNC - IMO are best options, so I agree
> > with Ivan.
> >
>
> Not sure I like STRICT. In Linux world, fsync is a well known command which
> does exactly what our FSYNC mode will do:
> http://man7.org/linux/man-pages/man2/fsync.2.html . STRICT, on the other
> hand, is not a commonly understood term. Why create a new term to define
> something that has already been defined?
>
> Also, what if tomorrow we need to add an even stricter parameter? Then we
> are back to the same problem we are trying to fix right now.
>
> D.
>

Re: Apache Ignite 2.4 release

[Dmitriy Setrakyan]

On Fri, Feb 16, 2018 at 6:26 AM, Dmitry Pavlov 
wrote:

> I had idea to name old default as FSYNC, but it would be too scientific.
>

I like FSYNC, I do not think it is too scientific. Definitely not more
scientific than LOG_ONLY.

For old DEFAULT, STRICT or STRICT_SYNC - IMO are best options, so I agree
> with Ivan.
>

Not sure I like STRICT. In Linux world, fsync is a well known command which
does exactly what our FSYNC mode will do:
http://man7.org/linux/man-pages/man2/fsync.2.html . STRICT, on the other
hand, is not a commonly understood term. Why create a new term to define
something that has already been defined?

Also, what if tomorrow we need to add an even stricter parameter? Then we
are back to the same problem we are trying to fix right now.

D.

[GitHub] ignite pull request #3534: IGNITE-7718 Fixed deserialization of binary objec...

[pvinokurov]

GitHub user pvinokurov opened a pull request:

https://github.com/apache/ignite/pull/3534

IGNITE-7718 Fixed deserialization of binary objects in singleton map …

…and set.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7718

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3534.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3534


commit c19d4e62db0a3db59617c4813d531283970025b6
Author: pvinokurov 
Date:   2018-02-16T12:23:37Z

IGNITE-7718 Fixed deserialization of binary objects in singleton map and 
set.




---

Re: Apache Ignite 2.4 release

[Dmitry Pavlov]

I had idea to name old default as FSYNC, but it would be too scientific.

For old DEFAULT, STRICT or STRICT_SYNC - IMO are best options, so I agree
with Ivan.

пт, 16 февр. 2018 г. в 15:21, Anton Vinogradov :

> typo
> NODE -> NONE
>
> On Fri, Feb 16, 2018 at 3:21 PM, Anton Vinogradov <
> avinogra...@gridgain.com>
> wrote:
>
> > What about
> > FULL_SYNC
> > SYNC -> default
> > BACKGROUND
> > NODE
> > ?
> >
> > On Fri, Feb 16, 2018 at 3:09 PM, Ivan Rakov 
> wrote:
> >
> >> From my point of view, STRICT is the best option. The name signalizes to
> >> user that this mode provides optional strict guarantees.
> >> FULL_SYNC can be messed with CacheWriteSynchronizationMode#FULL_SYNC. I
> >> don't like the idea of naming different things with same names.
> >>
> >> Best Regards,
> >> Ivan Rakov
> >>
> >>
> >> On 16.02.2018 15:01, Dmitriy Setrakyan wrote:
> >>
> >>> BTW, Ilya, why not name the enum value FULL_SYNC instead of STRICT?
> >>>
> >>> On Fri, Feb 16, 2018 at 5:43 AM, Dmitriy Setrakyan <
> >>> dsetrak...@apache.org>
> >>> wrote:
> >>>
> >>> Naming one of the enum constants DEFAULT was a huge mistake. Not sure
> how
>  it passed a code review, but let us all be more careful going forward.
> 
>  I agree with Ilya. The only remedy right now is to deprecate the
> DEFAULT
>  constant.
> 
>  D.
> 
>  On Fri, Feb 16, 2018 at 5:37 AM, Ilya Lantukh 
>  wrote:
> 
>  Hi all,
> >
> > I'd like to suggest to change default WALMode. Currently we have:
> > DEFAULT (write and fsync),
> > LOG_ONLY (write without fsync),
> > BACKGROUND,
> > NONE.
> >
> > It turns out that fsyncs in current DEFAULT mode significantly
> > restricts
> > Ignite performance. Compared to LOG_ONLY, it offers additional
> > guarantees
> > that data won't be lost in case of OS or hardware failure, but such
> > guarantees aren't needed very often, and tradeoff is too big.
> >
> > I suggest to rename current DEFAULT to STRICT and make LOG_ONLY new
> > default
> > mode. We can leave DEFAULT as @Deprecated and treat it as STRICT, so
> > that
> > users with old configs will have the same behaviour.
> >
> > What do you think?
> >
> > On Fri, Feb 16, 2018 at 12:35 AM, Denis Magda 
> > wrote:
> >
> > Vladimir,
> >>
> >> I would suggest not to do this because we still need to spend time
> on
> >> testing, documentation, etc. If someone shows interest in this
> >> features
> >> they can assemble binaries from the master.
> >>
> >> --
> >> Denis
> >>
> >> On Thu, Feb 15, 2018 at 6:43 AM, Nikolay Izhikov <
> nizhi...@apache.org
> >> >
> >> wrote:
> >>
> >> +1
> >>>
> >>> В Чт, 15/02/2018 в 17:27 +0300, Vladimir Ozerov пишет:
> >>>
>  Igniters,
> 
>  AI 2.4 release was shifted a bit and over this time we implemented
> 
> >>> two
> >
> >> important SQL features:
>  1) COPY command for fast file upload to the cluster [1]
>  2) Streaming mode for thin driver [2]
> 
>  Both commands are very important for fast data ingestion into
> Ignite
>  through SQL. I would like to ask community to consider to include
> 
> >>> these
> >
> >> two
> >>>
>  features into AI 2.4 in *experimental* state because both of them
> 
> >>> will
> >
> >> be
> >>
> >>> improved in various ways in the nearest time. If we do so, we will
> 
> >>> be
> >
> >> able
> >>>
>  to collect some feedback from the users before AI 2.5 release.
> What
> 
> >>> do
> >
> >> you
> >>>
>  think?
> 
>  Vladimir.
> 
>  [1] https://issues.apache.org/jira/browse/IGNITE-6917
>  [2] https://issues.apache.org/jira/browse/IGNITE-7253
> 
>  On Tue, Feb 13, 2018 at 1:22 AM, Dmitriy Setrakyan <
> 
> >>> dsetrak...@apache.org>
> >>>
>  wrote:
> 
>  On Mon, Feb 12, 2018 at 9:22 AM, Dmitry Pavlov <
> >
>  dpavlov@gmail.com>
> >>
> >>> wrote:
> >
> > Hi,
> >>
> >> Unfortunately, a quick fix did not give us too much performance
> >>
> > boost.
> >>>
>  I'm going to implement a complete algorithm change for storing
> >>
> > the
> >
> >> page
> >>>
>  identifier. But this change is quite significant and will
> >>
> > require
> >
> >> re-testing. I suggest including
> >> https://issues.apache.org/jira/browse/IGNITE-7638 in the next
> >>
> > version,
> >>>
>  for
> >
> >> example, to 2.5.
> >>
> >> Sincerely,
> 

Re: Apache Ignite 2.4 release

[Anton Vinogradov]

typo
NODE -> NONE

On Fri, Feb 16, 2018 at 3:21 PM, Anton Vinogradov 
wrote:

> What about
> FULL_SYNC
> SYNC -> default
> BACKGROUND
> NODE
> ?
>
> On Fri, Feb 16, 2018 at 3:09 PM, Ivan Rakov  wrote:
>
>> From my point of view, STRICT is the best option. The name signalizes to
>> user that this mode provides optional strict guarantees.
>> FULL_SYNC can be messed with CacheWriteSynchronizationMode#FULL_SYNC. I
>> don't like the idea of naming different things with same names.
>>
>> Best Regards,
>> Ivan Rakov
>>
>>
>> On 16.02.2018 15:01, Dmitriy Setrakyan wrote:
>>
>>> BTW, Ilya, why not name the enum value FULL_SYNC instead of STRICT?
>>>
>>> On Fri, Feb 16, 2018 at 5:43 AM, Dmitriy Setrakyan <
>>> dsetrak...@apache.org>
>>> wrote:
>>>
>>> Naming one of the enum constants DEFAULT was a huge mistake. Not sure how
 it passed a code review, but let us all be more careful going forward.

 I agree with Ilya. The only remedy right now is to deprecate the DEFAULT
 constant.

 D.

 On Fri, Feb 16, 2018 at 5:37 AM, Ilya Lantukh 
 wrote:

 Hi all,
>
> I'd like to suggest to change default WALMode. Currently we have:
> DEFAULT (write and fsync),
> LOG_ONLY (write without fsync),
> BACKGROUND,
> NONE.
>
> It turns out that fsyncs in current DEFAULT mode significantly
> restricts
> Ignite performance. Compared to LOG_ONLY, it offers additional
> guarantees
> that data won't be lost in case of OS or hardware failure, but such
> guarantees aren't needed very often, and tradeoff is too big.
>
> I suggest to rename current DEFAULT to STRICT and make LOG_ONLY new
> default
> mode. We can leave DEFAULT as @Deprecated and treat it as STRICT, so
> that
> users with old configs will have the same behaviour.
>
> What do you think?
>
> On Fri, Feb 16, 2018 at 12:35 AM, Denis Magda 
> wrote:
>
> Vladimir,
>>
>> I would suggest not to do this because we still need to spend time on
>> testing, documentation, etc. If someone shows interest in this
>> features
>> they can assemble binaries from the master.
>>
>> --
>> Denis
>>
>> On Thu, Feb 15, 2018 at 6:43 AM, Nikolay Izhikov > >
>> wrote:
>>
>> +1
>>>
>>> В Чт, 15/02/2018 в 17:27 +0300, Vladimir Ozerov пишет:
>>>
 Igniters,

 AI 2.4 release was shifted a bit and over this time we implemented

>>> two
>
>> important SQL features:
 1) COPY command for fast file upload to the cluster [1]
 2) Streaming mode for thin driver [2]

 Both commands are very important for fast data ingestion into Ignite
 through SQL. I would like to ask community to consider to include

>>> these
>
>> two
>>>
 features into AI 2.4 in *experimental* state because both of them

>>> will
>
>> be
>>
>>> improved in various ways in the nearest time. If we do so, we will

>>> be
>
>> able
>>>
 to collect some feedback from the users before AI 2.5 release. What

>>> do
>
>> you
>>>
 think?

 Vladimir.

 [1] https://issues.apache.org/jira/browse/IGNITE-6917
 [2] https://issues.apache.org/jira/browse/IGNITE-7253

 On Tue, Feb 13, 2018 at 1:22 AM, Dmitriy Setrakyan <

>>> dsetrak...@apache.org>
>>>
 wrote:

 On Mon, Feb 12, 2018 at 9:22 AM, Dmitry Pavlov <
>
 dpavlov@gmail.com>
>>
>>> wrote:
>
> Hi,
>>
>> Unfortunately, a quick fix did not give us too much performance
>>
> boost.
>>>
 I'm going to implement a complete algorithm change for storing
>>
> the
>
>> page
>>>
 identifier. But this change is quite significant and will
>>
> require
>
>> re-testing. I suggest including
>> https://issues.apache.org/jira/browse/IGNITE-7638 in the next
>>
> version,
>>>
 for
>
>> example, to 2.5.
>>
>> Sincerely,
>> Dmitriy Pavlov
>>
>>
>>
>> Dmitriy, thanks for the update! Are there other tickets that are
>
 holding
>>>
 the release at this point? I remember that there was a performance
> degradation issue in FULL_SYNC mode, but I cannot find a ticket.
>
> D.
>
>
>
> --
> Best regards,
> Ilya
>
>

>>
>

Re: Apache Ignite 2.4 release

[Anton Vinogradov]

What about
FULL_SYNC
SYNC -> default
BACKGROUND
NODE
?

On Fri, Feb 16, 2018 at 3:09 PM, Ivan Rakov  wrote:

> From my point of view, STRICT is the best option. The name signalizes to
> user that this mode provides optional strict guarantees.
> FULL_SYNC can be messed with CacheWriteSynchronizationMode#FULL_SYNC. I
> don't like the idea of naming different things with same names.
>
> Best Regards,
> Ivan Rakov
>
>
> On 16.02.2018 15:01, Dmitriy Setrakyan wrote:
>
>> BTW, Ilya, why not name the enum value FULL_SYNC instead of STRICT?
>>
>> On Fri, Feb 16, 2018 at 5:43 AM, Dmitriy Setrakyan > >
>> wrote:
>>
>> Naming one of the enum constants DEFAULT was a huge mistake. Not sure how
>>> it passed a code review, but let us all be more careful going forward.
>>>
>>> I agree with Ilya. The only remedy right now is to deprecate the DEFAULT
>>> constant.
>>>
>>> D.
>>>
>>> On Fri, Feb 16, 2018 at 5:37 AM, Ilya Lantukh 
>>> wrote:
>>>
>>> Hi all,

 I'd like to suggest to change default WALMode. Currently we have:
 DEFAULT (write and fsync),
 LOG_ONLY (write without fsync),
 BACKGROUND,
 NONE.

 It turns out that fsyncs in current DEFAULT mode significantly restricts
 Ignite performance. Compared to LOG_ONLY, it offers additional
 guarantees
 that data won't be lost in case of OS or hardware failure, but such
 guarantees aren't needed very often, and tradeoff is too big.

 I suggest to rename current DEFAULT to STRICT and make LOG_ONLY new
 default
 mode. We can leave DEFAULT as @Deprecated and treat it as STRICT, so
 that
 users with old configs will have the same behaviour.

 What do you think?

 On Fri, Feb 16, 2018 at 12:35 AM, Denis Magda 
 wrote:

 Vladimir,
>
> I would suggest not to do this because we still need to spend time on
> testing, documentation, etc. If someone shows interest in this features
> they can assemble binaries from the master.
>
> --
> Denis
>
> On Thu, Feb 15, 2018 at 6:43 AM, Nikolay Izhikov 
> wrote:
>
> +1
>>
>> В Чт, 15/02/2018 в 17:27 +0300, Vladimir Ozerov пишет:
>>
>>> Igniters,
>>>
>>> AI 2.4 release was shifted a bit and over this time we implemented
>>>
>> two

> important SQL features:
>>> 1) COPY command for fast file upload to the cluster [1]
>>> 2) Streaming mode for thin driver [2]
>>>
>>> Both commands are very important for fast data ingestion into Ignite
>>> through SQL. I would like to ask community to consider to include
>>>
>> these

> two
>>
>>> features into AI 2.4 in *experimental* state because both of them
>>>
>> will

> be
>
>> improved in various ways in the nearest time. If we do so, we will
>>>
>> be

> able
>>
>>> to collect some feedback from the users before AI 2.5 release. What
>>>
>> do

> you
>>
>>> think?
>>>
>>> Vladimir.
>>>
>>> [1] https://issues.apache.org/jira/browse/IGNITE-6917
>>> [2] https://issues.apache.org/jira/browse/IGNITE-7253
>>>
>>> On Tue, Feb 13, 2018 at 1:22 AM, Dmitriy Setrakyan <
>>>
>> dsetrak...@apache.org>
>>
>>> wrote:
>>>
>>> On Mon, Feb 12, 2018 at 9:22 AM, Dmitry Pavlov <

>>> dpavlov@gmail.com>
>
>> wrote:

 Hi,
>
> Unfortunately, a quick fix did not give us too much performance
>
 boost.
>>
>>> I'm going to implement a complete algorithm change for storing
>
 the

> page
>>
>>> identifier. But this change is quite significant and will
>
 require

> re-testing. I suggest including
> https://issues.apache.org/jira/browse/IGNITE-7638 in the next
>
 version,
>>
>>> for

> example, to 2.5.
>
> Sincerely,
> Dmitriy Pavlov
>
>
>
> Dmitriy, thanks for the update! Are there other tickets that are

>>> holding
>>
>>> the release at this point? I remember that there was a performance
 degradation issue in FULL_SYNC mode, but I cannot find a ticket.

 D.



 --
 Best regards,
 Ilya


>>>
>

Re: Apache Ignite 2.4 release

[Ivan Rakov]

From my point of view, STRICT is the best option. The name signalizes 
to user that this mode provides optional strict guarantees.
FULL_SYNC can be messed with CacheWriteSynchronizationMode#FULL_SYNC. I 
don't like the idea of naming different things with same names.


Best Regards,
Ivan Rakov

On 16.02.2018 15:01, Dmitriy Setrakyan wrote:

BTW, Ilya, why not name the enum value FULL_SYNC instead of STRICT?

On Fri, Feb 16, 2018 at 5:43 AM, Dmitriy Setrakyan 
wrote:


Naming one of the enum constants DEFAULT was a huge mistake. Not sure how
it passed a code review, but let us all be more careful going forward.

I agree with Ilya. The only remedy right now is to deprecate the DEFAULT
constant.

D.

On Fri, Feb 16, 2018 at 5:37 AM, Ilya Lantukh 
wrote:


Hi all,

I'd like to suggest to change default WALMode. Currently we have:
DEFAULT (write and fsync),
LOG_ONLY (write without fsync),
BACKGROUND,
NONE.

It turns out that fsyncs in current DEFAULT mode significantly restricts
Ignite performance. Compared to LOG_ONLY, it offers additional guarantees
that data won't be lost in case of OS or hardware failure, but such
guarantees aren't needed very often, and tradeoff is too big.

I suggest to rename current DEFAULT to STRICT and make LOG_ONLY new
default
mode. We can leave DEFAULT as @Deprecated and treat it as STRICT, so that
users with old configs will have the same behaviour.

What do you think?

On Fri, Feb 16, 2018 at 12:35 AM, Denis Magda  wrote:


Vladimir,

I would suggest not to do this because we still need to spend time on
testing, documentation, etc. If someone shows interest in this features
they can assemble binaries from the master.

--
Denis

On Thu, Feb 15, 2018 at 6:43 AM, Nikolay Izhikov 
wrote:


+1

В Чт, 15/02/2018 в 17:27 +0300, Vladimir Ozerov пишет:

Igniters,

AI 2.4 release was shifted a bit and over this time we implemented

two

important SQL features:
1) COPY command for fast file upload to the cluster [1]
2) Streaming mode for thin driver [2]

Both commands are very important for fast data ingestion into Ignite
through SQL. I would like to ask community to consider to include

these

two

features into AI 2.4 in *experimental* state because both of them

will

be

improved in various ways in the nearest time. If we do so, we will

be

able

to collect some feedback from the users before AI 2.5 release. What

do

you

think?

Vladimir.

[1] https://issues.apache.org/jira/browse/IGNITE-6917
[2] https://issues.apache.org/jira/browse/IGNITE-7253

On Tue, Feb 13, 2018 at 1:22 AM, Dmitriy Setrakyan <

dsetrak...@apache.org>

wrote:


On Mon, Feb 12, 2018 at 9:22 AM, Dmitry Pavlov <

dpavlov@gmail.com>

wrote:


Hi,

Unfortunately, a quick fix did not give us too much performance

boost.

I'm going to implement a complete algorithm change for storing

the

page

identifier. But this change is quite significant and will

require

re-testing. I suggest including
https://issues.apache.org/jira/browse/IGNITE-7638 in the next

version,

for

example, to 2.5.

Sincerely,
Dmitriy Pavlov




Dmitriy, thanks for the update! Are there other tickets that are

holding

the release at this point? I remember that there was a performance
degradation issue in FULL_SYNC mode, but I cannot find a ticket.

D.




--
Best regards,
Ilya

[GitHub] ignite pull request #3533: IGNITE-7737

[devozerov]

GitHub user devozerov opened a pull request:

https://github.com/apache/ignite/pull/3533

IGNITE-7737



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7737

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3533.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3533


commit 8a6e5a7c070968a410aec327c1530b06145c3a26
Author: devozerov 
Date:   2018-02-16T11:52:00Z

Done.




---

Re: TeamCity. Ignite RDD tests

[Dmitry Pavlov]

I think you should be able to mute any test. If not, please write to me.

пт, 16 февр. 2018 г. в 15:02, Nikolay Izhikov :

> Thank you, Dmitry.
>
> Can you do it for me?
> Or I can do it by myself?
>
> 16 февр. 2018 г. 3:00 PM пользователь "Dmitry Pavlov" <
> dpavlov@gmail.com>
> написал:
>
> > Hi Nikolay,
> >
> > Ok for me. In this case we have issue to fix.
> >
> > I suggest to mute with comment text with reference to issue in JIRA.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > пт, 16 февр. 2018 г. в 7:35, Nikolay Izhikov :
> >
> > > Hello, Igniters.
> > >
> > > I'm working on issue [1].
> > >
> > > Team City doesn't collect info about scalatest execution because of
> wrong
> > > pom.xml
> > > I'm fixed it in PR [3]
> > >
> > > It happens there is a 2 broken tests written in Scala - [4]
> > >
> > > 1. IgniteRDDSpec.IgniteRDD should successfully store data to ignite
> using
> > > saveValues
> > > 2. IgniteRDDSpec.IgniteRDD should successfully store data to ignite
> using
> > > saveValues with inline transformation
> > >
> > > It seems they are been here for a while.
> > > I propose to mute or disable it on TeamCity before merging my PR.
> > > I've created ticket for fixing tests - [5].
> > >
> > > Thoughts?
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-7042
> > >
> > > [2]
> > > https://ci.ignite.apache.org/viewLog.html?buildId=1096059;
> > tab=buildResultsDiv=IgniteTests24Java8_IgniteRdd
> > >
> > > [3] https://github.com/apache/ignite/pull/3530
> > >
> > > [4]
> > > https://ci.ignite.apache.org/viewLog.html?buildId=1095218=
> > IgniteTests24Java8_IgniteRdd=testsInfo
> > >
> > > [5] https://issues.apache.org/jira/browse/IGNITE-7727
> >
>

Re: TeamCity. Ignite RDD tests

[Nikolay Izhikov]

Thank you, Dmitry.

Can you do it for me?
Or I can do it by myself?

16 февр. 2018 г. 3:00 PM пользователь "Dmitry Pavlov" 
написал:

> Hi Nikolay,
>
> Ok for me. In this case we have issue to fix.
>
> I suggest to mute with comment text with reference to issue in JIRA.
>
> Sincerely,
> Dmitriy Pavlov
>
> пт, 16 февр. 2018 г. в 7:35, Nikolay Izhikov :
>
> > Hello, Igniters.
> >
> > I'm working on issue [1].
> >
> > Team City doesn't collect info about scalatest execution because of wrong
> > pom.xml
> > I'm fixed it in PR [3]
> >
> > It happens there is a 2 broken tests written in Scala - [4]
> >
> > 1. IgniteRDDSpec.IgniteRDD should successfully store data to ignite using
> > saveValues
> > 2. IgniteRDDSpec.IgniteRDD should successfully store data to ignite using
> > saveValues with inline transformation
> >
> > It seems they are been here for a while.
> > I propose to mute or disable it on TeamCity before merging my PR.
> > I've created ticket for fixing tests - [5].
> >
> > Thoughts?
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-7042
> >
> > [2]
> > https://ci.ignite.apache.org/viewLog.html?buildId=1096059;
> tab=buildResultsDiv=IgniteTests24Java8_IgniteRdd
> >
> > [3] https://github.com/apache/ignite/pull/3530
> >
> > [4]
> > https://ci.ignite.apache.org/viewLog.html?buildId=1095218=
> IgniteTests24Java8_IgniteRdd=testsInfo
> >
> > [5] https://issues.apache.org/jira/browse/IGNITE-7727
>

Re: Apache Ignite 2.4 release

[Dmitriy Setrakyan]

BTW, Ilya, why not name the enum value FULL_SYNC instead of STRICT?

On Fri, Feb 16, 2018 at 5:43 AM, Dmitriy Setrakyan 
wrote:

> Naming one of the enum constants DEFAULT was a huge mistake. Not sure how
> it passed a code review, but let us all be more careful going forward.
>
> I agree with Ilya. The only remedy right now is to deprecate the DEFAULT
> constant.
>
> D.
>
> On Fri, Feb 16, 2018 at 5:37 AM, Ilya Lantukh 
> wrote:
>
>> Hi all,
>>
>> I'd like to suggest to change default WALMode. Currently we have:
>> DEFAULT (write and fsync),
>> LOG_ONLY (write without fsync),
>> BACKGROUND,
>> NONE.
>>
>> It turns out that fsyncs in current DEFAULT mode significantly restricts
>> Ignite performance. Compared to LOG_ONLY, it offers additional guarantees
>> that data won't be lost in case of OS or hardware failure, but such
>> guarantees aren't needed very often, and tradeoff is too big.
>>
>> I suggest to rename current DEFAULT to STRICT and make LOG_ONLY new
>> default
>> mode. We can leave DEFAULT as @Deprecated and treat it as STRICT, so that
>> users with old configs will have the same behaviour.
>>
>> What do you think?
>>
>> On Fri, Feb 16, 2018 at 12:35 AM, Denis Magda  wrote:
>>
>> > Vladimir,
>> >
>> > I would suggest not to do this because we still need to spend time on
>> > testing, documentation, etc. If someone shows interest in this features
>> > they can assemble binaries from the master.
>> >
>> > --
>> > Denis
>> >
>> > On Thu, Feb 15, 2018 at 6:43 AM, Nikolay Izhikov 
>> > wrote:
>> >
>> > > +1
>> > >
>> > > В Чт, 15/02/2018 в 17:27 +0300, Vladimir Ozerov пишет:
>> > > > Igniters,
>> > > >
>> > > > AI 2.4 release was shifted a bit and over this time we implemented
>> two
>> > > > important SQL features:
>> > > > 1) COPY command for fast file upload to the cluster [1]
>> > > > 2) Streaming mode for thin driver [2]
>> > > >
>> > > > Both commands are very important for fast data ingestion into Ignite
>> > > > through SQL. I would like to ask community to consider to include
>> these
>> > > two
>> > > > features into AI 2.4 in *experimental* state because both of them
>> will
>> > be
>> > > > improved in various ways in the nearest time. If we do so, we will
>> be
>> > > able
>> > > > to collect some feedback from the users before AI 2.5 release. What
>> do
>> > > you
>> > > > think?
>> > > >
>> > > > Vladimir.
>> > > >
>> > > > [1] https://issues.apache.org/jira/browse/IGNITE-6917
>> > > > [2] https://issues.apache.org/jira/browse/IGNITE-7253
>> > > >
>> > > > On Tue, Feb 13, 2018 at 1:22 AM, Dmitriy Setrakyan <
>> > > dsetrak...@apache.org>
>> > > > wrote:
>> > > >
>> > > > > On Mon, Feb 12, 2018 at 9:22 AM, Dmitry Pavlov <
>> > dpavlov@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Hi,
>> > > > > >
>> > > > > > Unfortunately, a quick fix did not give us too much performance
>> > > boost.
>> > > > > >
>> > > > > > I'm going to implement a complete algorithm change for storing
>> the
>> > > page
>> > > > > > identifier. But this change is quite significant and will
>> require
>> > > > > > re-testing. I suggest including
>> > > > > > https://issues.apache.org/jira/browse/IGNITE-7638 in the next
>> > > version,
>> > > > >
>> > > > > for
>> > > > > > example, to 2.5.
>> > > > > >
>> > > > > > Sincerely,
>> > > > > > Dmitriy Pavlov
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > > > Dmitriy, thanks for the update! Are there other tickets that are
>> > > holding
>> > > > > the release at this point? I remember that there was a performance
>> > > > > degradation issue in FULL_SYNC mode, but I cannot find a ticket.
>> > > > >
>> > > > > D.
>> > > > >
>> > >
>> >
>>
>>
>>
>> --
>> Best regards,
>> Ilya
>>
>
>

Re: TeamCity. Ignite RDD tests

[Dmitry Pavlov]

Hi Nikolay,

Ok for me. In this case we have issue to fix.

I suggest to mute with comment text with reference to issue in JIRA.

Sincerely,
Dmitriy Pavlov

пт, 16 февр. 2018 г. в 7:35, Nikolay Izhikov :

> Hello, Igniters.
>
> I'm working on issue [1].
>
> Team City doesn't collect info about scalatest execution because of wrong
> pom.xml
> I'm fixed it in PR [3]
>
> It happens there is a 2 broken tests written in Scala - [4]
>
> 1. IgniteRDDSpec.IgniteRDD should successfully store data to ignite using
> saveValues
> 2. IgniteRDDSpec.IgniteRDD should successfully store data to ignite using
> saveValues with inline transformation
>
> It seems they are been here for a while.
> I propose to mute or disable it on TeamCity before merging my PR.
> I've created ticket for fixing tests - [5].
>
> Thoughts?
>
> [1] https://issues.apache.org/jira/browse/IGNITE-7042
>
> [2]
> https://ci.ignite.apache.org/viewLog.html?buildId=1096059=buildResultsDiv=IgniteTests24Java8_IgniteRdd
>
> [3] https://github.com/apache/ignite/pull/3530
>
> [4]
> https://ci.ignite.apache.org/viewLog.html?buildId=1095218=IgniteTests24Java8_IgniteRdd=testsInfo
>
> [5] https://issues.apache.org/jira/browse/IGNITE-7727

[jira] [Created] (IGNITE-7737) SQL COPY: rename "batch_size" to "packet_size"

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-7737:
---

 Summary: SQL COPY: rename "batch_size" to "packet_size"
 Key: IGNITE-7737
 URL: https://issues.apache.org/jira/browse/IGNITE-7737
 Project: Ignite
  Issue Type: Task
  Components: sql
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
 Fix For: 2.5






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Apache Ignite 2.4 release

[Dmitriy Setrakyan]

Naming one of the enum constants DEFAULT was a huge mistake. Not sure how
it passed a code review, but let us all be more careful going forward.

I agree with Ilya. The only remedy right now is to deprecate the DEFAULT
constant.

D.

On Fri, Feb 16, 2018 at 5:37 AM, Ilya Lantukh  wrote:

> Hi all,
>
> I'd like to suggest to change default WALMode. Currently we have:
> DEFAULT (write and fsync),
> LOG_ONLY (write without fsync),
> BACKGROUND,
> NONE.
>
> It turns out that fsyncs in current DEFAULT mode significantly restricts
> Ignite performance. Compared to LOG_ONLY, it offers additional guarantees
> that data won't be lost in case of OS or hardware failure, but such
> guarantees aren't needed very often, and tradeoff is too big.
>
> I suggest to rename current DEFAULT to STRICT and make LOG_ONLY new default
> mode. We can leave DEFAULT as @Deprecated and treat it as STRICT, so that
> users with old configs will have the same behaviour.
>
> What do you think?
>
> On Fri, Feb 16, 2018 at 12:35 AM, Denis Magda  wrote:
>
> > Vladimir,
> >
> > I would suggest not to do this because we still need to spend time on
> > testing, documentation, etc. If someone shows interest in this features
> > they can assemble binaries from the master.
> >
> > --
> > Denis
> >
> > On Thu, Feb 15, 2018 at 6:43 AM, Nikolay Izhikov 
> > wrote:
> >
> > > +1
> > >
> > > В Чт, 15/02/2018 в 17:27 +0300, Vladimir Ozerov пишет:
> > > > Igniters,
> > > >
> > > > AI 2.4 release was shifted a bit and over this time we implemented
> two
> > > > important SQL features:
> > > > 1) COPY command for fast file upload to the cluster [1]
> > > > 2) Streaming mode for thin driver [2]
> > > >
> > > > Both commands are very important for fast data ingestion into Ignite
> > > > through SQL. I would like to ask community to consider to include
> these
> > > two
> > > > features into AI 2.4 in *experimental* state because both of them
> will
> > be
> > > > improved in various ways in the nearest time. If we do so, we will be
> > > able
> > > > to collect some feedback from the users before AI 2.5 release. What
> do
> > > you
> > > > think?
> > > >
> > > > Vladimir.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-6917
> > > > [2] https://issues.apache.org/jira/browse/IGNITE-7253
> > > >
> > > > On Tue, Feb 13, 2018 at 1:22 AM, Dmitriy Setrakyan <
> > > dsetrak...@apache.org>
> > > > wrote:
> > > >
> > > > > On Mon, Feb 12, 2018 at 9:22 AM, Dmitry Pavlov <
> > dpavlov@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Unfortunately, a quick fix did not give us too much performance
> > > boost.
> > > > > >
> > > > > > I'm going to implement a complete algorithm change for storing
> the
> > > page
> > > > > > identifier. But this change is quite significant and will require
> > > > > > re-testing. I suggest including
> > > > > > https://issues.apache.org/jira/browse/IGNITE-7638 in the next
> > > version,
> > > > >
> > > > > for
> > > > > > example, to 2.5.
> > > > > >
> > > > > > Sincerely,
> > > > > > Dmitriy Pavlov
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > Dmitriy, thanks for the update! Are there other tickets that are
> > > holding
> > > > > the release at this point? I remember that there was a performance
> > > > > degradation issue in FULL_SYNC mode, but I cannot find a ticket.
> > > > >
> > > > > D.
> > > > >
> > >
> >
>
>
>
> --
> Best regards,
> Ilya
>

Re: Semaphore Stuck when no acquirers to assign permit

[Alexey Goncharuk]

Hi Vladislav,

Can you please finalize the review? I would like to include the fix in one
of the nightly Ignite builds for my own project.

Thanks.
--AG

2018-02-01 4:47 GMT+03:00 Dmitriy Setrakyan :

> Hi Tim, Vlad,
>
> Any update on this ticket? I looked inside, but cannot understand the
> status.
>
> Thanks,
> D.
>
> -- Forwarded message --
> From: Denis Magda 
> Date: Fri, Jan 19, 2018 at 9:14 PM
> Subject: Re: Semaphore Stuck when no acquirers to assign permit
> To: u...@ignite.apache.org, dev@ignite.apache.org
>
>
> Igniters,
>
> Who can check out Tim’s fix for the semaphore?
>
> pull request: https://github.com/apache/ignite/pull/3138 <
> https://github.com/apache/ignite/pull/3138>
> jira: https://issues.apache.org/jira/browse/IGNITE-7090 <
> https://issues.apache.org/jira/browse/IGNITE-7090>
>
> —
> Denis
>
> > On Jan 15, 2018, at 12:02 PM, Timay  wrote:
> >
> > I saw a release date set for 2.4 but have not had any feedback on the
> jira so
> > i wanted to check in on this. Can this make it into the 2.4 release?
> >
> > Thanks
> > Tim
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Transport compression (not store compression)

[Dmitriy Setrakyan]

On Fri, Feb 16, 2018 at 4:22 AM, Vladimir Ozerov 
wrote:

> Dmitry,
>
> IMO we should not compress individual messages. Instead, we should compress
> stream of enqueued messages. This way we will reduce number of array
> copying and improve performance. Makes sense?
>

Yes, it does, as long as we can have a stream of uncompressed messages as
well.

Re: Apache Ignite 2.4 release

[Ilya Lantukh]

Hi all,

I'd like to suggest to change default WALMode. Currently we have:
DEFAULT (write and fsync),
LOG_ONLY (write without fsync),
BACKGROUND,
NONE.

It turns out that fsyncs in current DEFAULT mode significantly restricts
Ignite performance. Compared to LOG_ONLY, it offers additional guarantees
that data won't be lost in case of OS or hardware failure, but such
guarantees aren't needed very often, and tradeoff is too big.

I suggest to rename current DEFAULT to STRICT and make LOG_ONLY new default
mode. We can leave DEFAULT as @Deprecated and treat it as STRICT, so that
users with old configs will have the same behaviour.

What do you think?

On Fri, Feb 16, 2018 at 12:35 AM, Denis Magda  wrote:

> Vladimir,
>
> I would suggest not to do this because we still need to spend time on
> testing, documentation, etc. If someone shows interest in this features
> they can assemble binaries from the master.
>
> --
> Denis
>
> On Thu, Feb 15, 2018 at 6:43 AM, Nikolay Izhikov 
> wrote:
>
> > +1
> >
> > В Чт, 15/02/2018 в 17:27 +0300, Vladimir Ozerov пишет:
> > > Igniters,
> > >
> > > AI 2.4 release was shifted a bit and over this time we implemented two
> > > important SQL features:
> > > 1) COPY command for fast file upload to the cluster [1]
> > > 2) Streaming mode for thin driver [2]
> > >
> > > Both commands are very important for fast data ingestion into Ignite
> > > through SQL. I would like to ask community to consider to include these
> > two
> > > features into AI 2.4 in *experimental* state because both of them will
> be
> > > improved in various ways in the nearest time. If we do so, we will be
> > able
> > > to collect some feedback from the users before AI 2.5 release. What do
> > you
> > > think?
> > >
> > > Vladimir.
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-6917
> > > [2] https://issues.apache.org/jira/browse/IGNITE-7253
> > >
> > > On Tue, Feb 13, 2018 at 1:22 AM, Dmitriy Setrakyan <
> > dsetrak...@apache.org>
> > > wrote:
> > >
> > > > On Mon, Feb 12, 2018 at 9:22 AM, Dmitry Pavlov <
> dpavlov@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Unfortunately, a quick fix did not give us too much performance
> > boost.
> > > > >
> > > > > I'm going to implement a complete algorithm change for storing the
> > page
> > > > > identifier. But this change is quite significant and will require
> > > > > re-testing. I suggest including
> > > > > https://issues.apache.org/jira/browse/IGNITE-7638 in the next
> > version,
> > > >
> > > > for
> > > > > example, to 2.5.
> > > > >
> > > > > Sincerely,
> > > > > Dmitriy Pavlov
> > > > >
> > > > >
> > > > >
> > > >
> > > > Dmitriy, thanks for the update! Are there other tickets that are
> > holding
> > > > the release at this point? I remember that there was a performance
> > > > degradation issue in FULL_SYNC mode, but I cannot find a ticket.
> > > >
> > > > D.
> > > >
> >
>



-- 
Best regards,
Ilya

[jira] [Created] (IGNITE-7736) SQL COPY: streaming model for network packets instead of request-response model

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-7736:
---

 Summary: SQL COPY: streaming model for network packets instead of 
request-response model
 Key: IGNITE-7736
 URL: https://issues.apache.org/jira/browse/IGNITE-7736
 Project: Ignite
  Issue Type: Task
Reporter: Vladimir Ozerov
 Fix For: 2.5


*Problem*
Currently data transfer in COPY command is implemented as a series of 
request-responses. When request is received, it is parsed synchronously and 
passed to the streamer, then response is sent. This is not very efficient 
approach:
# We hardly could utilize long fat network channels efficiently as we spend a 
lot of time waiting for a very small response (ack).
# Parsing takes and adding data to the streamer takes time (especially if we 
reached streamer buffer limitations and are blocked waiting for responses from 
data nodes). During this period network is not utilized and file data is not 
transferred further.

*Solution*
Let's fix the problem iteratively as follows:
# Introduce asynchrony - when request is received, send the response 
immediately before data processing
# Then consider sending one ack for several requests instead of sending ack for 
every request
# When multiple simultaneous requests are enabled (previous point), consider 
moving data processing to separate stream, so that we can read data from the 
socket as fast as possible 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Attention! TeamCity https://ci.ignite.apache.org/

[Vitaliy Osipov]

TeamCity server https://ci.ignite.apache.org/ will be moved to another
hardware server on Saturday 17.02.18  from 22:00 MSK to 6:00 MSK.
It will be unavailable during this time.


-- 
Kind Regards
Vitaliy Osipov
vosi...@gridgain.com
*+7 (921) 397 27 68*

*gridgain.com *Powered by Apache® Ignite™

[jira] [Created] (IGNITE-7735) SQL COPY: expose streamer options to user

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-7735:
---

 Summary: SQL COPY: expose streamer options to user
 Key: IGNITE-7735
 URL: https://issues.apache.org/jira/browse/IGNITE-7735
 Project: Ignite
  Issue Type: Task
  Components: sql
Reporter: Vladimir Ozerov
 Fix For: 2.5


COPY command uses cache streamer underneath. We need to expose usual streamer 
options (per-node-buffer-size, per-node-parallel-ops, allow-overwrite, etc) to 
the user.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7734) SQL COPY: support single quotes for file name

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-7734:
---

 Summary: SQL COPY: support single quotes for file name
 Key: IGNITE-7734
 URL: https://issues.apache.org/jira/browse/IGNITE-7734
 Project: Ignite
  Issue Type: Task
  Components: sql
Reporter: Vladimir Ozerov
 Fix For: 2.5


Currently file name is used in COPY command as follows: \{{"/path/to/file"}}. 
We need to support single-quoted case as well: \{{'/path/to/file'}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7733) SQL COPY: native API support

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-7733:
---

 Summary: SQL COPY: native API support
 Key: IGNITE-7733
 URL: https://issues.apache.org/jira/browse/IGNITE-7733
 Project: Ignite
  Issue Type: Task
  Components: sql
Reporter: Vladimir Ozerov
 Fix For: 2.5


Currently COPY command can only be executed from within thin JDBC driver. Need 
to add native API support as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7732) SQL COPY: ODBC support

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-7732:
---

 Summary: SQL COPY: ODBC support
 Key: IGNITE-7732
 URL: https://issues.apache.org/jira/browse/IGNITE-7732
 Project: Ignite
  Issue Type: Task
  Components: odbc, sql
Reporter: Vladimir Ozerov
Assignee: Igor Sapego
 Fix For: 2.5


Support COPY command for ODBC driver in the same way it is done for JDBC.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Batch size parameter at DataStreamerCacheUpdaters.batched()

[Roman Guseinov]

Val,

I got what do you mean. Everything inside the org.apache.ignite.internal is
considered to be not public API.

Best Regards,
Roman



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: Transport compression (not store compression)

[Vladimir Ozerov]

Dmitry,

IMO we should not compress individual messages. Instead, we should compress
stream of enqueued messages. This way we will reduce number of array
copying and improve performance. Makes sense?

On Fri, Feb 16, 2018 at 12:05 AM, Dmitriy Setrakyan 
wrote:

> Vova, I think your solution is fine, but I think we will always have some
> messages compressed and others not. For example, in many cases, especially
> when messages are relatively small, compressing them will introduce an
> unnecessary overhead, and most likely slow down the cluster.
>
> Why not have compression flag or compression bit on per-message level? We
> check if the bit is turned on, and if it is, then we uncompress the message
> on the receiving side before processing it.
>
> D.
>
> On Thu, Feb 15, 2018 at 12:24 AM, Vladimir Ozerov 
> wrote:
>
> > I think that we should not guess on how the clients are used. They could
> be
> > used in any way - in the same network, in another network, in Docker, in
> > hypervisor, etc.. This holds for both thin and thick clients. It is
> > essential that we design configuration API in a way that compression
> could
> > be enabled only for some participants.
> >
> > What if we do this as follows:
> > 1) Define "IgniteConfiguration.compressionEnabled" flag
> > 2) When two nodes communicate and at least one of them has this flag,
> then
> > all data sent between them is compressed.
> >
> > Makes sense?
> >
> > On Thu, Feb 15, 2018 at 8:50 AM, Nikita Amelchev 
> > wrote:
> >
> > > Hello, Igniters.
> > >
> > > I have not seen such use-cases, where heavy client ignite node placed
> in
> > a
> > > much worse network than the server. I'm not sure we should encourage a
> > bad
> > > cluster architecture.
> > >
> > > Usually, in my use-cases, the servers and clients locate in the same
> > > network. And if the cluster has SSL enabled, it makes sense to enable
> > > compression, even if the network is fast. It also makes sense when we
> > have
> > > a high load on the network, and the CPU is utilized poorly.
> > >
> > > I'll do tests on yardstick for real operations like get, put etc. and
> SQL
> > > requests.
> > >
> > > I propose to add configurable compression for thin client/ODBC/JDBC as
> a
> > > separate issue because it increases the current PR.
> > >
> > > Even if it really makes sense to compress the traffic only between
> > > client-server ignite nodes, it should also be a separate issue, that
> > would
> > > not increase the PR. Especially, since this compression architecture
> may
> > > not be accepted by the community.
> > >
> > > 2018-02-05 13:02 GMT+03:00 Nikita Amelchev :
> > >
> > > > Thanks for your comments,
> > > >
> > > > I will try to separate network compression for clients and servers.
> > > >
> > > > It makes sense to enable compression on servers if we have SSL turned
> > on.
> > > > I tested rebalancing time and compression+ssl is faster. SSL
> throughput
> > > is
> > > > limited by 800 Mbits/sec per connection and if enable compression, it
> > > > boosted up to 1100 Mbits.
> > > >
> > > > 2018-02-02 18:52 GMT+03:00 Alexey Kuznetsov :
> > > >
> > > >> I think Igor is right.
> > > >>
> > > >> Ususally servers connected via fast local network.
> > > >> But clients could be in external and slow network.
> > > >> In this scenario compression will be very useful.
> > > >>
> > > >> Once I had such scenario - client connected to cluster via 300 kb/s
> > > >> network
> > > >> and tries to transfer ~10Mb of uncumpressed data.
> > > >> So it takse ~30 seconds.
> > > >> After I implemented compression it becamed 1M and transfered for ~3
> > > >> seconds.
> > > >>
> > > >> I think we should take care of all mentioned problems with NIO
> threads
> > > in
> > > >> order to not slow down whole cluster.
> > > >>
> > > >>
> > > >> On Fri, Feb 2, 2018 at 10:05 PM, gvvinblade 
> > > wrote:
> > > >>
> > > >> > Nikita,
> > > >> >
> > > >> > Yes, you're right. Maybe I wasn't clear enough.
> > > >> >
> > > >> > Usually server nodes are placed in the same fast network segment
> > (one
> > > >> > datacenter); in any case we need an ability to setup compression
> per
> > > >> > connection using some filter like useCompression(ClusterNode,
> > > >> ClusterNode)
> > > >> > to compress traffic only between servers and client nodes.
> > > >> >
> > > >> > But issue is still there, since the same NIO worker serves both
> > client
> > > >> and
> > > >> > server connections, enabled compression may impact whole cluster
> > > >> > performance
> > > >> > because NIO threads will compress client messages instead of
> > > processing
> > > >> > servers' compute requests. That was my concern.
> > > >> >
> > > >> > Compression for clients is really cool feature and usefull in some
> > > >> cases.
> > > >> > Probably it makes sense to have two NIO servers with and without
> > > >> > compression

Re: Transport compression (not store compression)

[Nikita Amelchev]

Vladimir Ozerov, I also agree that your solution is good.

I will check this flag before adding a client to map of clients. If one of
the nodes have the flag then the session will be marked "compressed".  At
the nearest time, I will provide a solution.

Dmitriy Setrakyan, I will implement and test compressed flag after I write
test with real operations (put, get etc) on yardstick.

2018-02-16 0:05 GMT+03:00 Dmitriy Setrakyan :

> Vova, I think your solution is fine, but I think we will always have some
> messages compressed and others not. For example, in many cases, especially
> when messages are relatively small, compressing them will introduce an
> unnecessary overhead, and most likely slow down the cluster.
>
> Why not have compression flag or compression bit on per-message level? We
> check if the bit is turned on, and if it is, then we uncompress the message
> on the receiving side before processing it.
>
> D.
>
> On Thu, Feb 15, 2018 at 12:24 AM, Vladimir Ozerov 
> wrote:
>
> > I think that we should not guess on how the clients are used. They could
> be
> > used in any way - in the same network, in another network, in Docker, in
> > hypervisor, etc.. This holds for both thin and thick clients. It is
> > essential that we design configuration API in a way that compression
> could
> > be enabled only for some participants.
> >
> > What if we do this as follows:
> > 1) Define "IgniteConfiguration.compressionEnabled" flag
> > 2) When two nodes communicate and at least one of them has this flag,
> then
> > all data sent between them is compressed.
> >
> > Makes sense?
> >
> > On Thu, Feb 15, 2018 at 8:50 AM, Nikita Amelchev 
> > wrote:
> >
> > > Hello, Igniters.
> > >
> > > I have not seen such use-cases, where heavy client ignite node placed
> in
> > a
> > > much worse network than the server. I'm not sure we should encourage a
> > bad
> > > cluster architecture.
> > >
> > > Usually, in my use-cases, the servers and clients locate in the same
> > > network. And if the cluster has SSL enabled, it makes sense to enable
> > > compression, even if the network is fast. It also makes sense when we
> > have
> > > a high load on the network, and the CPU is utilized poorly.
> > >
> > > I'll do tests on yardstick for real operations like get, put etc. and
> SQL
> > > requests.
> > >
> > > I propose to add configurable compression for thin client/ODBC/JDBC as
> a
> > > separate issue because it increases the current PR.
> > >
> > > Even if it really makes sense to compress the traffic only between
> > > client-server ignite nodes, it should also be a separate issue, that
> > would
> > > not increase the PR. Especially, since this compression architecture
> may
> > > not be accepted by the community.
> > >
> > > 2018-02-05 13:02 GMT+03:00 Nikita Amelchev :
> > >
> > > > Thanks for your comments,
> > > >
> > > > I will try to separate network compression for clients and servers.
> > > >
> > > > It makes sense to enable compression on servers if we have SSL turned
> > on.
> > > > I tested rebalancing time and compression+ssl is faster. SSL
> throughput
> > > is
> > > > limited by 800 Mbits/sec per connection and if enable compression, it
> > > > boosted up to 1100 Mbits.
> > > >
> > > > 2018-02-02 18:52 GMT+03:00 Alexey Kuznetsov :
> > > >
> > > >> I think Igor is right.
> > > >>
> > > >> Ususally servers connected via fast local network.
> > > >> But clients could be in external and slow network.
> > > >> In this scenario compression will be very useful.
> > > >>
> > > >> Once I had such scenario - client connected to cluster via 300 kb/s
> > > >> network
> > > >> and tries to transfer ~10Mb of uncumpressed data.
> > > >> So it takse ~30 seconds.
> > > >> After I implemented compression it becamed 1M and transfered for ~3
> > > >> seconds.
> > > >>
> > > >> I think we should take care of all mentioned problems with NIO
> threads
> > > in
> > > >> order to not slow down whole cluster.
> > > >>
> > > >>
> > > >> On Fri, Feb 2, 2018 at 10:05 PM, gvvinblade 
> > > wrote:
> > > >>
> > > >> > Nikita,
> > > >> >
> > > >> > Yes, you're right. Maybe I wasn't clear enough.
> > > >> >
> > > >> > Usually server nodes are placed in the same fast network segment
> > (one
> > > >> > datacenter); in any case we need an ability to setup compression
> per
> > > >> > connection using some filter like useCompression(ClusterNode,
> > > >> ClusterNode)
> > > >> > to compress traffic only between servers and client nodes.
> > > >> >
> > > >> > But issue is still there, since the same NIO worker serves both
> > client
> > > >> and
> > > >> > server connections, enabled compression may impact whole cluster
> > > >> > performance
> > > >> > because NIO threads will compress client messages instead of
> > > processing
> > > >> > servers' compute requests. That was my concern.
> > > >> >
> > > >> > Compression 

[GitHub] ignite pull request #3439: IGNITE-7192 :: JDBC support FQDN to multiple IPs ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3439


---

[jira] [Created] (IGNITE-7731) Restarted node can

[Ksenia Rybakova (JIRA)]

Ksenia Rybakova created IGNITE-7731:
---

 Summary: Restarted node can
 Key: IGNITE-7731
 URL: https://issues.apache.org/jira/browse/IGNITE-7731
 Project: Ignite
  Issue Type: Bug
Reporter: Ksenia Rybakova






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Spark data frames

[Nikolay Izhikov]

Ok, Igniters.

I will do it in a few weeks.
I need time to prepare to the talk.

В Пт, 16/02/2018 в 07:16 +, Dmitry Pavlov пишет:
> +1 for starting new topic from Nikolay when ' Community Meeting to
> Introduce Ignite Spark Data Frames'  is ready to be announced.
> 
> пт, 16 февр. 2018 г. в 9:27, Denis Magda :
> 
> > I'm in :)
> > 
> > Nikolay, we can use my GoToMeeting account to host the webinar. To draw
> > more attention I would suggest starting a more specific thread titled like
> > "[RSVP] Community Meeting to Introduce Ignite Spark Data Frames". This
> > discussion sounds too generic, folks could simply pass by.
> > 
> > Negotiated?
> > 
> > --
> > Denis
> > 
> > On Wed, Feb 14, 2018 at 6:04 AM, Vyacheslav Daradur 
> > wrote:
> > 
> > > Dmitry, it's a great idea!
> > > 
> > > Nikolay, I also have the interest to get familiar with Spark Data
> > > Frames integration.
> > > 
> > > I'd prefer webinar of the format similar to "Ignite Persistent Store
> > > Walkthrough" by Denis Magda, which has been presented some times ago.
> > > 
> > > On Wed, Feb 14, 2018 at 5:03 PM, Dmitriy Setrakyan
> > >  wrote:
> > > > I am definitely interested. Great idea!
> > > > 
> > > > On Wed, Feb 14, 2018 at 4:32 AM, Nikolay Izhikov 
> > > > wrote:
> > > > 
> > > > > Hello, Dmitry.
> > > > > 
> > > > > If other community member are also iterested in that kind of
> > > 
> > > information I
> > > > > can try to do the talk.
> > > > > 
> > > > > В Ср, 14/02/2018 в 10:49 +, Dmitry Pavlov пишет:
> > > > > > Hi Nikolay,
> > > > > > 
> > > > > > I've notices there are a number of very lively discussions on dev
> > 
> > list
> > > > > about SparkDataFrames. But I, for example, can't fully understand it
> > > > > because it is not well-known code for me.
> > > > > > 
> > > > > > I suppose Ignite community has other members, which are not aware of
> > > > > 
> > > > > recent feature SparkDataFrame and its pros.
> > > > > > 
> > > > > > What do you think about short talk arrangement for community to tell
> > > > > 
> > > > > about this module, e.g. for 30 minutes? Could you please do this? I
> > > 
> > > think
> > > > > Denis M. can help with infrastructure.
> > > > > > 
> > > > > > Sincerely,
> > > > > > Dmitriy Pavlov
> > > 
> > > 
> > > 
> > > --
> > > Best Regards, Vyacheslav D.
> > > 

signature.asc
Description: This is a digitally signed message part