Re: Apache Ignite 2.4 release

Hi, I merged performance fix for TCP metrics. Nikolay, I think it is OK to merge your fix today. Let's agree on a final code freeze on Monday, 5 Mar. On Thu, Mar 1, 2018 at 8:24 PM, Denis Magda wrote: > Hi Vladimir, > > The regression, as well as the performance, is

Re: Apache Ignite 2.4 release

Hi Vladimir, The regression, as well as the performance, is definitely bad. Please work with Sergey to see how much time he needs to rerun the whole test set if you merge the changes. I would target the end of the next week as a release date if the vote goes well. Probably, we can run the test

Re: Apache Ignite 2.4 release

Hi, We found one nasty regression in DROP COLUMN feature and fixed it today. Another problem is performance - we found that recent changes to TCP communication metrics caused considerable slowdown in in-memory mode. I almost fixed it and will merge the fix in the nearest day if benchmarks are ok.

Re: Apache Ignite 2.4 release

Good question! We've been waiting for a sign-off from Alexey Goncharuk and Sergey Kozlov who are benchmarking and trying to address performance issues in coopeartion. Guys, please share your results and forecast. -- Denis On Sun, Feb 25, 2018 at 12:10 AM, Pavel Tupitsyn

Re: Apache Ignite 2.4 release

Igniters, Any update on the 2.4 release status? Anything else to merge there? Pavel On Mon, Feb 19, 2018 at 9:30 PM, Dmitriy Setrakyan wrote: > On Mon, Feb 19, 2018 at 1:37 AM, Vladimir Ozerov > wrote: > > > Alex, > > > > You get me right. DEFAULT

Re: Apache Ignite 2.4 release

On Mon, Feb 19, 2018 at 1:37 AM, Vladimir Ozerov wrote: > Alex, > > You get me right. DEFAULT -> LOG_ONLY doesn't introduce any dramatic > changes when comparing 2.3 to 2.4 - Ignite was unsafe out of the box in > 2.3, and it is unsafe in 2.4 as well. > > The very problem is

Re: Apache Ignite 2.4 release

> > Neither Cassandra, nor Mongo or any others claim themselves > to be ACID, so it is not valid to refer to their defaults. This is why I referred to VoltDB as to an example. It supports ACID transactions and persists changes to disk in a mode similar to LOG_ONLY. -- Denis On Sun, Feb 18,

Re: Apache Ignite 2.4 release

Alex, please clarify some of the points data loss because the disk content was shuffled due to an incomplete page > write are you talking about some phase of the checkpointing process? Do you mean that the process just passed over changes to the file but didn't call fsync, thus, offloading this

Re: Apache Ignite 2.4 release

Alex, You get me right. DEFAULT -> LOG_ONLY doesn't introduce any dramatic changes when comparing 2.3 to 2.4 - Ignite was unsafe out of the box in 2.3, and it is unsafe in 2.4 as well. The very problem is that we claim ourselves to be ACID, while in reality we are only "AI" out of the box,

Re: Apache Ignite 2.4 release

Hi Folks, We have performance suggestions logged during Ignite start. Can we create data safety suggestions and list all these options during startup? Arguments about Ignite is distributed and may have backup for protection make sence. Who can provide reference to test of such or similar

Re: Apache Ignite 2.4 release

In terms of 'safety', Ignite default settings are far beyond optimal. For in-memory mode, we have 0 backups by default, which means partition loss in a case of node failure, we have readFromBackup=true and PRIMARY_SYNC by default which effectively cancels linearizability property for cache

Re: Apache Ignite 2.4 release

Classic relational databases have no choice rather than to use FSYNC by default. RDBMS is all about consistency. Distributed databases try to balance consistency and performance. For instance, why to fsync every update if there is usually 1 backup copy? This is probably why VoltDB [1] and

Re: Apache Ignite 2.4 release

Vova, I hear your concerns, but at the same time I know that one of the largest banks in eastern Europe is using Ignite in LOG_ONLY mode with 3 backups to move money. The rational is that the probability of failure of 4 servers at hardware level at the same time is very low. However, if the JVM

Re: Apache Ignite 2.4 release

Sergey, We do not have backups by default either, so essentially we are loosing data by default. Moreover, backups are less reliable option than fsync because a lot of users cannot afford putting servers into separate power circuits, so a single power failure may easily lead to poweroff of the

Re: Apache Ignite 2.4 release

I suppose some approaches used by classic databases makes no sense for Ignite. FSYNC requirement for databases has the nature of single host solution. If you have corrupted db files you have corrupted (lost) data. For Ignite the enough number of backups and the failure detecting logic can provide

Re: Apache Ignite 2.4 release

Hi, all WAL modes except NONE protects from data consistency problem (B+Tree, pages, etc), which is why I suggest to avoid saying 'corrupted' about 'unapplied updates'. Log Only and Background may cause unapplied updates in case of OS/process failures. None mode IMO is not an option in case data

Re: Apache Ignite 2.4 release

Guys, While we're on this topic, what is the difference between BACKGROUND and NONE in terms of semantics and provided guarantees? To me it looks like both guarantee to recover the state since last checkpoint and anything else can potentially be lost, so from user perspective they are the same.

Re: Apache Ignite 2.4 release

What is the reason to have DEFAULT mode at all if you claim LOG_ONLY to be completely safe? :) And how it could be safe provided that without fsync we loose part of WAL itself in case of crash? пт, 16 февр. 2018 г. в 19:32, Dmitry Pavlov : > Thank you. Data can't be

Re: Apache Ignite 2.4 release

Thank you. Data can't be corrupted in case crash because of WAL replay (since completed checkpoint). Physical records are used to restore probably corrupted pages in persistent store (we overwrite so called 'grey zone' - pages we don't know for sure if they have been written). Only one effect is

Re: Apache Ignite 2.4 release

Log only mode is not safe - data might be corrupted in case of system crash. Oracle - fsync, Postgres - fsync, SQL Server - fsync, Cassandra - similar to our “background”. пт, 16 февр. 2018 г. в 19:11, Dmitry Pavlov : > Hi Vladimir, > > What you saying is defenetely make

Re: Apache Ignite 2.4 release

Hi Vladimir, What you saying is defenetely make sence. In the same time LOG_ONLY is also safe mode, user will be able to restore system after crash. If it is not true, we should create critical ticket and fix it. Do you know other databases defaults, such as Cassandra, Oracle, Postgre?

Re: Apache Ignite 2.4 release

Igniters, Sorry for pouring oil on the flames, but from database perspective moving from FSYNC to non-FSYNC mode appears to be a mistake. When you work with database, your main expectation is that it will save your data. All production database vendor make sure that you are safe, not that you are

Re: Apache Ignite 2.4 release

Well, I cannot say that I like the name LOG_ONLY, but I would vote to keep it for now, given that it is already documented in many places, blogs, and examples. D. On Fri, Feb 16, 2018 at 8:13 AM, Ivan Rakov wrote: > Looks like it's an Ignite term - I've never heard of it

Re: Apache Ignite 2.4 release

Looks like it's an Ignite term - I've never heard of it outside Ignite scope. Though, renaming existing enum value requires keeping old as deprecated. DEFAULT is confusing enough to pay this price. As for LOG_ONLY, I think we can keep it as long as it has good and definitive javadoc. Best

Re: Apache Ignite 2.4 release

Igniters, just to clarify, does the term LOG_ONLY mean anything in the industry or is this just an Ignite term? D. On Fri, Feb 16, 2018 at 8:03 AM, Anton Vinogradov wrote: > Log only mode: flushes application buffers. > So, in synced mode without fsync guarantee.

Re: Apache Ignite 2.4 release

Log only mode: flushes application buffers. So, in synced mode without fsync guarantee. That's why I propose to rename it as SYNC. On Fri, Feb 16, 2018 at 4:49 PM, Ilya Lantukh wrote: > I am OK with either FSYNC or STRICT variant. > > LOG_ONLY name means "log without

Re: Apache Ignite 2.4 release

I am OK with either FSYNC or STRICT variant. LOG_ONLY name means "log without fsync". On Fri, Feb 16, 2018 at 4:05 PM, Dmitriy Setrakyan wrote: > On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov wrote: > > > Why create a new term to define something

Re: Apache Ignite 2.4 release

On Fri, Feb 16, 2018 at 7:02 AM, Ivan Rakov wrote: > Why create a new term to define something that has already been defined? >> > That makes sense. I'm ok with FSYNC. > Anton, I don't understand why we should rename LOG_ONLY to SYNC. We > started this discussion with bad

Re: Apache Ignite 2.4 release

Why create a new term to define something that has already been defined? That makes sense. I'm ok with FSYNC. Anton, I don't understand why we should rename LOG_ONLY to SYNC. We started this discussion with bad naming of DEFAULT, but this has nothing to do with LOG_ONLY (even though it may be

Re: Apache Ignite 2.4 release

>> I had idea to name old default as FSYNC, but it would be too scientific. So, then it can be FSYNC, SYNC, BACKGROUND and NONE! On Fri, Feb 16, 2018 at 3:49 PM, Dmitriy Setrakyan wrote: > On Fri, Feb 16, 2018 at 6:26 AM, Dmitry Pavlov > wrote: >

Re: Apache Ignite 2.4 release

On Fri, Feb 16, 2018 at 6:26 AM, Dmitry Pavlov wrote: > I had idea to name old default as FSYNC, but it would be too scientific. > I like FSYNC, I do not think it is too scientific. Definitely not more scientific than LOG_ONLY. For old DEFAULT, STRICT or STRICT_SYNC -

Re: Apache Ignite 2.4 release

I had idea to name old default as FSYNC, but it would be too scientific. For old DEFAULT, STRICT or STRICT_SYNC - IMO are best options, so I agree with Ivan. пт, 16 февр. 2018 г. в 15:21, Anton Vinogradov : > typo > NODE -> NONE > > On Fri, Feb 16, 2018 at 3:21 PM,

Re: Apache Ignite 2.4 release

typo NODE -> NONE On Fri, Feb 16, 2018 at 3:21 PM, Anton Vinogradov wrote: > What about > FULL_SYNC > SYNC -> default > BACKGROUND > NODE > ? > > On Fri, Feb 16, 2018 at 3:09 PM, Ivan Rakov wrote: > >> From my point of view, STRICT is the best

Re: Apache Ignite 2.4 release

What about FULL_SYNC SYNC -> default BACKGROUND NODE ? On Fri, Feb 16, 2018 at 3:09 PM, Ivan Rakov wrote: > From my point of view, STRICT is the best option. The name signalizes to > user that this mode provides optional strict guarantees. > FULL_SYNC can be messed with

Re: Apache Ignite 2.4 release

From my point of view, STRICT is the best option. The name signalizes to user that this mode provides optional strict guarantees. FULL_SYNC can be messed with CacheWriteSynchronizationMode#FULL_SYNC. I don't like the idea of naming different things with same names. Best Regards, Ivan Rakov On

Re: Apache Ignite 2.4 release

BTW, Ilya, why not name the enum value FULL_SYNC instead of STRICT? On Fri, Feb 16, 2018 at 5:43 AM, Dmitriy Setrakyan wrote: > Naming one of the enum constants DEFAULT was a huge mistake. Not sure how > it passed a code review, but let us all be more careful going

Re: Apache Ignite 2.4 release

Naming one of the enum constants DEFAULT was a huge mistake. Not sure how it passed a code review, but let us all be more careful going forward. I agree with Ilya. The only remedy right now is to deprecate the DEFAULT constant. D. On Fri, Feb 16, 2018 at 5:37 AM, Ilya Lantukh

Re: Apache Ignite 2.4 release

Hi all, I'd like to suggest to change default WALMode. Currently we have: DEFAULT (write and fsync), LOG_ONLY (write without fsync), BACKGROUND, NONE. It turns out that fsyncs in current DEFAULT mode significantly restricts Ignite performance. Compared to LOG_ONLY, it offers additional

Re: Apache Ignite 2.4 release

Vladimir, I would suggest not to do this because we still need to spend time on testing, documentation, etc. If someone shows interest in this features they can assemble binaries from the master. -- Denis On Thu, Feb 15, 2018 at 6:43 AM, Nikolay Izhikov wrote: > +1 > > В

Re: Apache Ignite 2.4 release

+1 В Чт, 15/02/2018 в 17:27 +0300, Vladimir Ozerov пишет: > Igniters, > > AI 2.4 release was shifted a bit and over this time we implemented two > important SQL features: > 1) COPY command for fast file upload to the cluster [1] > 2) Streaming mode for thin driver [2] > > Both commands are very

Re: Apache Ignite 2.4 release

Igniters, AI 2.4 release was shifted a bit and over this time we implemented two important SQL features: 1) COPY command for fast file upload to the cluster [1] 2) Streaming mode for thin driver [2] Both commands are very important for fast data ingestion into Ignite through SQL. I would like to

Re: Apache Ignite 2.4 release

On Mon, Feb 12, 2018 at 9:22 AM, Dmitry Pavlov wrote: > Hi, > > Unfortunately, a quick fix did not give us too much performance boost. > > I'm going to implement a complete algorithm change for storing the page > identifier. But this change is quite significant and will

Re: Apache Ignite 2.4 release

Hi, Unfortunately, a quick fix did not give us too much performance boost. I'm going to implement a complete algorithm change for storing the page identifier. But this change is quite significant and will require re-testing. I suggest including https://issues.apache.org/jira/browse/IGNITE-7638

Re: Apache Ignite 2.4 release

Hello, Dmitriy. Data Frame Read feature is documented in https://issues.apache.org/jira/browse/IGNITE-7345 AFAIK page isn't published yet. I will document Data Frame Save feature in a few days. There is a ticket for it - https://issues.apache.org/jira/browse/IGNITE-7655 В Вс, 11/02/2018 в

Re: Apache Ignite 2.4 release

On Sun, Feb 11, 2018 at 2:23 PM, Valentin Kulichenko < valentin.kuliche...@gmail.com> wrote: > IGNITE-7337 (Data frame save functionality) is merged into 2.4. > Thanks Val, this is great news! Also, thanks for Nikolai Izhikov who implemented most of the code. Has this been documented already.

Re: Apache Ignite 2.4 release

IGNITE-7337 (Data frame save functionality) is merged into 2.4. -Val On Fri, Feb 9, 2018 at 11:47 AM, Valentin Kulichenko < valentin.kuliche...@gmail.com> wrote: > Nikolay, > > To merge it to 2.4, you need to merge the change to ignite-2.4 release. > Let's do this if we come to an agreement in

Re: Apache Ignite 2.4 release

Nikolay, To merge it to 2.4, you need to merge the change to ignite-2.4 release. Let's do this if we come to an agreement in the neighbor thread. -Val On Thu, Feb 8, 2018 at 8:21 PM, Nikolay Izhikov wrote: > Hello, Dmitriy. > > IGNITE-7337 are merged to master [1] > > Do

Re: Apache Ignite 2.4 release

Hello, Dmitriy. IGNITE-7337 are merged to master [1] Do I need to do something more to include this feature into 2.4. release? [1] https://github.com/apache/ignite/commit/7c01452990ad0de0fb84ab4c0424a6d71e5bccba В Ср, 07/02/2018 в 11:26 -0800, Dmitriy Setrakyan пишет: > Agree on both, the

Re: Apache Ignite 2.4 release

Agree on both, the performance fix and the spark data frames. Let's get them into the release. However, Raymond is right. We should know how long the performance fix will take. If it adds another month to the development, we should include it into the next release. I am hoping that it can be done

Re: Apache Ignite 2.4 release

Hello, Igniters. Please, consider including IGNITE-7337 - Spark Data Frames: support saving a data frame in Ignite [1] in the 2.4 release. It seems we can merge it into the master in a day or few. [1] https://issues.apache.org/jira/browse/IGNITE-7337 В Ср, 07/02/2018 в 08:35 -0800, Denis

Re: Apache Ignite 2.4 release

I’m voting for the blocker addition into the release. Sergey K. how will it affect your testing cycles? Do you need to re-run everything from scratch and how many days you need? — Denis > On Feb 6, 2018, at 11:29 PM, Alexey Goncharuk > wrote: > > Guys, > >

Re: Apache Ignite 2.4 release

Thanks for the clarification Dmitry. Sent from my iPhone > On 7/02/2018, at 11:13 PM, Dmitry Pavlov wrote: > > Hi Raymond, > > Minor releases were discussed already at dev list. Community does not > support minor releases. In case of emergency fix is required, full

Re: Apache Ignite 2.4 release

Hi Raymond, Minor releases were discussed already at dev list. Community does not support minor releases. In case of emergency fix is required, full release process has to be followed for next release, e.g. for 2.5. Exactly that type of release was 2.2, it was emergency release for fix

Re: Apache Ignite 2.4 release

Guys, Thanks to Dmitriy Pavlov we found the ticket [1] which causes a major slowdown when page replacement starts. Even though it's not a regression, I suggest we consider it a blocker for 2.4 because this is a huge performance issue which can make it virtually impossible to use native

Re: Apache Ignite 2.4 release

Igniters, I will handle 2.4 release if there are no objections. Let's take a bit more time for testing and start vote by the end of this week. Pavel On Sat, Jan 27, 2018 at 3:32 AM, Denis Magda wrote: > Hi Vyacheslav, > > According to the previous review notes the impact of

Re: Apache Ignite 2.4 release

Hi Vyacheslav, According to the previous review notes the impact of the changes might be significant, thus, I would recommend us to move the changes to the next release. BTW, don’t hesitate to ping reviewers more frequently if there is a pending/abandon review. We are all the people who are

Re: Apache Ignite 2.4 release

Hi, Vladimir, it's good news. I'm looking forward to new Ignite release! Could you please share a release schedule for 'varint' optimizations? The task [1] is waiting for review for 5 months. [1] https://issues.apache.org/jira/browse/IGNITE-5097 On Fri, Jan 26, 2018 at 12:51 PM, Vladimir

Re: Apache Ignite 2.4 release

Hi Igniters, As far as I can see all required tasks and fixes were merged. I propose to take several days of silence to test what we've done and start vote at the beginning of the next week. Makes sense? On Mon, Jan 22, 2018 at 8:39 PM, Denis Magda wrote: > Ok, let’s target

Re: Apache Ignite 2.4 release

Ok, let’s target Wednesday as a code freeze date. Community members who are involved in 2.4 release please merge you fixes and optimizations by that time. — Denis > On Jan 22, 2018, at 8:24 AM, Anton Vinogradov wrote: > > Issues >

Re: Apache Ignite 2.4 release

Issues https://issues.apache.org/jira/browse/IGNITE-6711 https://issues.apache.org/jira/browse/IGNITE-6902 are in master and ignite-2.4 On Mon, Jan 22, 2018 at 6:43 PM, Denis Magda wrote: > Igniters, > > What’s left and what prevents us from passing 2.4 through the QA phase?

Re: Apache Ignite 2.4 release

Igniters, What’s left and what prevents us from passing 2.4 through the QA phase? Petr, Anton, Vladimir, Alex G., others please chime in. — Denis > On Jan 17, 2018, at 7:05 PM, Dmitriy Setrakyan wrote: > > Great news, indeed! Looking forward to 2.4 release. > > On

Re: Apache Ignite 2.4 release

Great news, indeed! Looking forward to 2.4 release. On Wed, Jan 17, 2018 at 2:55 AM, Vladimir Ozerov wrote: > Igniters, > > Great news - two very important tickets - baseline topology and Java 8 > support - were merged to master. At this point it makes sense to create a >

Re: Apache Ignite 2.4 release

Igniters, Great news - two very important tickets - baseline topology and Java 8 support - were merged to master. At this point it makes sense to create a branch to stabilize the release and finalize outstanding tickets. I created the branch *ignite-2.4*. Please follow the rules below when

Re: Apache Ignite 2.4 release

Denis, I'll review https://issues.apache.org/jira/browse/IGNITE-6902 On Fri, Jan 12, 2018 at 11:45 PM, Denis Magda wrote: > Here is the page create before: > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.4 < >

Re: Apache Ignite 2.4 release

Here is the page create before: https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.4 This is good news. It will be perfect if we also release as many "required tickets" as we can:

Re: Apache Ignite 2.4 release

+1 Do we have a list of features for 2.4 documented anywhere? D. On Fri, Jan 12, 2018 at 7:43 AM, Pavel Tupitsyn wrote: > +1 > > But some stuff is not yet in master (like baseline topology), so it is too > soon to create a branch, I think. > > Pavel > > On Fri, Jan 12,

Re: Apache Ignite 2.4 release

+1 But some stuff is not yet in master (like baseline topology), so it is too soon to create a branch, I think. Pavel On Fri, Jan 12, 2018 at 5:31 PM, Vladimir Ozerov wrote: > Igniters, > > A number of major improvements have been made (or being finalized at the >