Igniters,
I've completed development of
https://issues.apache.org/jira/browse/IGNITE-7754. TeamCity state is ok.
Please, review my changes.
Please note that it will be possible to track time of WAL fsync on
checkpoint begin by *walCpRecordFsyncDuration *metric in "Checkpoint
started" message.
Also, I've created https://issues.apache.org/jira/browse/IGNITE-8057
with description of possible further improvement of WAL fsync on
checkpoint begin.
Best Regards,
Ivan Rakov
On 26.03.2018 23:45, Valentin Kulichenko wrote:
Ivan,
It's all good then :) Thanks!
-Val
On Mon, Mar 26, 2018 at 1:50 AM, Ivan Rakov <ivan.glu...@gmail.com> wrote:
Val,
There's no any sense to use WalMode.NONE in production environment, it's
kept for testing and debugging purposes (including possible user activities
like capacity planning).
We already print a warning at node start in case WalMode.NONE is set:
U.quietAndWarn(log,"Started write-ahead log manager in NONE mode,
persisted data may be lost in " +
"a case of unexpected node failure. Make sure to deactivate the
cluster before shutdown.");
Best Regards,
Ivan Rakov
On 24.03.2018 1:40, Valentin Kulichenko wrote:
Dmitry,
Thanks for clarification. So it sounds like if we fix all other modes as
we
discuss here, NONE would be the only one allowing corruption. I also don't
see much sense in this and I think we should clearly state this in the
doc,
as well print out a warning if NONE mode is used. Eventually, if it's
confirmed that there are no reasonable use cases for it, we can deprecate
it.
-Val
On Fri, Mar 23, 2018 at 3:26 PM, Dmitry Pavlov <dpavlov....@gmail.com>
wrote:
Hi Val,
NONE means that the WAL log is disabled and not written at all. Use of
the
mode is at your own risk. It is possible that restore state after the
crash
at the middle of checkpoint will not succeed. I do not see much sence in
it, especially in production.
BACKGROUND is full functional WAL mode, but allows some delay before
flush
to disk.
Sincerely,
Dmitriy Pavlov
сб, 24 мар. 2018 г. в 1:07, Valentin Kulichenko <
valentin.kuliche...@gmail.com>:
I agree. In my view, any possibility to get a corrupted storage is a bug
which needs to be fixed.
BTW, can someone explain semantics of NONE mode? What is the difference
from BACKGROUND from user's perspective? Is there any particular use
case
where it can be used?
-Val
On Fri, Mar 23, 2018 at 2:49 AM, Dmitry Pavlov <dpavlov....@gmail.com>
wrote:
Hi Ivan,
IMO we have to add extra FSYNCS for BACKGROUND WAL. Agree?
Sincerely,
Dmitriy Pavlov
пт, 23 мар. 2018 г. в 12:23, Ivan Rakov <ivan.glu...@gmail.com>:
Igniters, there's another important question about this matter.
Do we want to add extra FSYNCS for BACKGROUND WAL mode? I think that
we
have to do it: it will cause similar performance drop, but if we
consider LOG_ONLY broken without these fixes, BACKGROUND is broken as
well.
Best Regards,
Ivan Rakov
On 23.03.2018 10:27, Ivan Rakov wrote:
Fixes are quite simple.
I expect them to be merged in master in a week in worst case.
Best Regards,
Ivan Rakov
On 22.03.2018 17:49, Denis Magda wrote:
Ivan,
How quick are you going to merge the fix into the master? Many
persistence
related optimizations have already stacked up. Probably, we can
release
them sooner if the community agrees.
--
Denis
On Thu, Mar 22, 2018 at 5:22 AM, Ivan Rakov <
ivan.glu...@gmail.com>
wrote:
Thanks all!
We seem to have reached a consensus on this issue. I'll just add
necessary
fsyncs under IGNITE-7754.
Best Regards,
Ivan Rakov
On 22.03.2018 15:13, Ilya Lantukh wrote:
+1 for fixing LOG_ONLY. If current implementation doesn't
protect
from
data
corruption, it doesn't make sence.
On Wed, Mar 21, 2018 at 10:38 PM, Denis Magda <
dma...@apache.org>
wrote:
+1 for the fix of LOG_ONLY
On Wed, Mar 21, 2018 at 11:23 AM, Alexey Goncharuk <
alexey.goncha...@gmail.com> wrote:
+1 for fixing LOG_ONLY to enforce corruption safety given the
provided
performance results.
2018-03-21 18:20 GMT+03:00 Vladimir Ozerov <
voze...@gridgain.com
:
+1 for accepting drop in LOG_ONLY. 7% is not that much and
not a
drop
at
all, provided that we fixing a bug. I.e. should we implement
it
correctly
in the first place we would never notice any "drop".
I do not understand why someone would like to use current
broken
mode.
On Wed, Mar 21, 2018 at 6:11 PM, Dmitry Pavlov
<dpavlov....@gmail.com>
wrote:
Hi, I think option 1 is better. As Val said any mode that
allows
corruption
does not make much sense.
What Ivan mentioned here as drop, in relation to old mode
DEFAULT
(FSYNC
now), is still significant perfromance boost.
Sincerely,
Dmitriy Pavlov
ср, 21 мар. 2018 г. в 17:56, Ivan Rakov <
ivan.glu...@gmail.com
:
I've attached benchmark results to the JIRA ticket.
We observe ~7% drop in "fair" LOG_ONLY_SAFE mode,
independent
of
WAL
compaction enabled flag. It's pretty significant drop: WAL
compaction
itself gives only ~3% drop.
I see two options here:
1) Change LOG_ONLY behavior. That implies that we'll be
ready
to
release
AI 2.5 with 7% drop.
2) Introduce LOG_ONLY_SAFE, make it default, add release
note
to AI
2.5
that we added power loss durability in default mode, but user
may
fallback to previous LOG_ONLY in order to retain
performance.
Thoughts?
Best Regards,
Ivan Rakov
On 20.03.2018 16:00, Ivan Rakov wrote:
Val,
If a storage is in
corrupted state, does it mean that it needs to be
completely
removed
and
cluster needs to be restarted without data?
Yes, there's a chance that in LOG_ONLY all local data will
be
lost,
but only in *power loss**/ OS crash* case.
kill -9, JVM crash, death of critical system thread and
all
other
cases that usually take place are variations of *process
crash*.
All
WAL modes (except NONE, of course) ensure corruption-safety
in
case
of
process crash.
If so, I'm not sure any mode
that allows corruption makes much sense to me.
It depends on performance impact of enforcing power-loss
corruption
safety. Price of full protection from power loss is high -
FSYNC
is
way slower (2-10 times) than other WAL modes. The question is
whether
ensuring weaker guarantees (corruption can't happen, but loss
of
last
updates can) will affect performance as badly as strong
guarantees.
I'll share benchmark results soon.
Best Regards,
Ivan Rakov
On 20.03.2018 5:09, Valentin Kulichenko wrote:
Guys,
What do we understand under "data corruption" here? If a
storage
is
in
corrupted state, does it mean that it needs to be completely
removed
and
cluster needs to be restarted without data? If so, I'm not
sure
any
mode
that allows corruption makes much sense to me. How am I
supposed
to
use a
database, if virtually any failure can end with complete
loss of
data?
In any case, this definitely should not be a default
behavior.
If
user ever
switches to corruption-unsafe mode, there should be a
clear
warning
about
this.
-Val
On Fri, Mar 16, 2018 at 1:06 AM, Ivan Rakov <
ivan.glu...@gmail.com>
wrote:
Ticket to track changes:
https://issues.apache.org/jira/browse/IGNITE-7754
Best Regards,
Ivan Rakov
On 16.03.2018 10:58, Dmitriy Setrakyan wrote:
On Fri, Mar 16, 2018 at 12:55 AM, Ivan Rakov <
ivan.glu...@gmail.com
wrote:
Vladimir,
Unlike BACKGROUND, LOG_ONLY provides strict write
guarantees
unless power
loss has happened.
Seems like we need to measure performance difference
to
decide
whether do
we need separate WAL mode. If it will be invisible,
we'll
just
fix
these
bugs without introducing new mode; if it will be
perceptible,
we'll
continue the discussion about introducing LOG_ONLY_SAFE.
Makes sense?
Yes, this sounds like the right approach.
