date:20180215

Re: WAL Archive Issue

2018-02-15 Thread Dmitry Pavlov

Hi Ivan,

Excellent idea, I've added this to
https://issues.apache.org/jira/browse/IGNITE-7730 ticket.

Sinerely,
Dmitriy Pavlov

ср, 14 февр. 2018 г. в 0:28, Ivan Rakov :

> >   - applying compressor to segments older than 1 completed checkpoint
> ago -
> > saves space.
> By the way: WAL compression is already implemented that way. If there
> are any ".zip" segments in archive dir, they are free to delete.
> This can be a safe workaround for users who experience lack of free
> space - just delete compressed segments. We should mention it in
> documentation for 2.4 release.
>
> Best Regards,
> Ivan Rakov
>
> On 13.02.2018 23:53, Dmitry Pavlov wrote:
> > I see, it seems subgoal 'gain predictable size' can be achieved with
> > following options:
> >   - https://issues.apache.org/jira/browse/IGNITE-6552 implementation (in
> > variant of '...WAL history size in time units and maximum size in
> GBytes',
> > - here we probably should change description or create 2nd issue),
> >   - no-archiver mode ( segments still can be deleted, but in same
> directory
> > it was written) - maximum perfomance on ext* fs.
> >   - applying compressor to segments older than 1 completed checkpoint
> ago -
> > saves space.
> >
> > Is it necessary to store data we can safely remove?
> >
> > Or may be Ignite should handle this by itself and delete unnecessary
> > segments on low space left on device, like Linux decreases page cache in
> > memory if there is no free RAM left.
> >
> > вт, 13 февр. 2018 г. в 23:32, Ivan Rakov :
> >
> >> As far as I understand, the idea is WAL archive with predictable size
> >> ("N checkpoints" is not predictable size), which can be safely removed
> >> (e.g. if free disk space is urgently needed) without losing crash
> recovery.
> >>
> >> No-archiver mode makes sense as well - it should be faster than current
> >> mode (at least, on filesystems different from XFS). It will be useful
> >> for users who has lots of disk space and want to gain maximum
> throughput.
> >>
> >> Best Regards,
> >> Ivan Rakov
> >>
> >> On 13.02.2018 23:14, Dmitry Pavlov wrote:
> >>> Hi, I didn't get the point why it may be required to separate WAL work,
> >> WAL
> >>> uncheckpointed archive (some work outside segment rotation) and
> >>> checkpointed archive (which is better to be compressed using Ignite new
> >>> feature - WAL compressor).
> >>>
> >>> Please consider new no-archiver mode implemented recently.
> >>>
> >>> If archive folder confuses end user, grid admin may set up this mode
> (all
> >>> segments is placed in 1 directory) instead of introducing folders.
> >>>
> >>>
> >>> вт, 13 февр. 2018 г. в 22:11, Ivan Rakov :
> >>>
>  I think, I got the point now.
>  There's no need to copy files from "temp" to "archive" dir - we can
> just
>  move them, which is a constant-time operation.
>  Makes sense.
> 
>  Change is quite complex (we need to synchronize all movings thoroughly
>  to avoid ruining existing WAL read iterators), but feasible.
> 
>  Best Regards,
>  Ivan Rakov
> 
> 
>  On 13.02.2018 22:06, Ivan Rakov wrote:
> > Yakov,
> >
> > This will work. However, I expect performance degradation with this
> > change. Disk storage has a limited number of I/O operations per
> second
> > on hardware level. List of already existing disk I/O activities
> > (writing to WAL work dir, copying from WAL work dir to WAL archive
> > dir, writing partition files during checkpoint) will be updated with
> a
> > new one - copying from WAL work dir to temp dir.
> >
> > Best Regards,
> > Ivan Rakov
> >
> > On 13.02.2018 21:35, Yakov Zhdanov wrote:
> >> Ivan,
> >>
> >> I do not want to create new files. As far as I know, now we copy
> >> segments
> >> to archive dir before they get checkpointed. What I suggest is to
> >> copy them
> >> to a temp dir under wal directory and then move to archive. In my
> >> understanding at the time we copy the files to a temp folder all
> >> changes to
> >> them are already fsynced.
> >>
> >> Correct?
> >>
> >> Yakov Zhdanov,
> >> www.gridgain.com
> >>
> >> 2018-02-13 21:29 GMT+03:00 Ivan Rakov :
> >>
> >>> Yakov,
> >>>
> >>> I see the only one problem with your suggestion - number of
> >>> "uncheckpointed" segments is potentially unlimited.
> >>> Right now we have limited number (10) of file segments with
> immutable
> >>> names in WAL "work" directory. We have to keep this approach due to
> >>> known
> >>> bug in XFS - fsync time is nearly twice bigger for recently created
> >>> files.
> >>>
> >>> Best Regards,
> >>> Ivan Rakov
> >>>
> >>>
> >>> On 13.02.2018 21:22, Yakov Zhdanov wrote:
> >>>
>  I meant we still will be copying segment once and then will be
>

Re: Spark data frames

2018-02-15 Thread Dmitry Pavlov

+1 for starting new topic from Nikolay when ' Community Meeting to
Introduce Ignite Spark Data Frames'  is ready to be announced.

пт, 16 февр. 2018 г. в 9:27, Denis Magda :

> I'm in :)
>
> Nikolay, we can use my GoToMeeting account to host the webinar. To draw
> more attention I would suggest starting a more specific thread titled like
> "[RSVP] Community Meeting to Introduce Ignite Spark Data Frames". This
> discussion sounds too generic, folks could simply pass by.
>
> Negotiated?
>
> --
> Denis
>
> On Wed, Feb 14, 2018 at 6:04 AM, Vyacheslav Daradur 
> wrote:
>
> > Dmitry, it's a great idea!
> >
> > Nikolay, I also have the interest to get familiar with Spark Data
> > Frames integration.
> >
> > I'd prefer webinar of the format similar to "Ignite Persistent Store
> > Walkthrough" by Denis Magda, which has been presented some times ago.
> >
> > On Wed, Feb 14, 2018 at 5:03 PM, Dmitriy Setrakyan
> >  wrote:
> > > I am definitely interested. Great idea!
> > >
> > > On Wed, Feb 14, 2018 at 4:32 AM, Nikolay Izhikov 
> > > wrote:
> > >
> > >> Hello, Dmitry.
> > >>
> > >> If other community member are also iterested in that kind of
> > information I
> > >> can try to do the talk.
> > >>
> > >> В Ср, 14/02/2018 в 10:49 +, Dmitry Pavlov пишет:
> > >> > Hi Nikolay,
> > >> >
> > >> > I've notices there are a number of very lively discussions on dev
> list
> > >> about SparkDataFrames. But I, for example, can't fully understand it
> > >> because it is not well-known code for me.
> > >> >
> > >> > I suppose Ignite community has other members, which are not aware of
> > >> recent feature SparkDataFrame and its pros.
> > >> >
> > >> > What do you think about short talk arrangement for community to tell
> > >> about this module, e.g. for 30 minutes? Could you please do this? I
> > think
> > >> Denis M. can help with infrastructure.
> > >> >
> > >> > Sincerely,
> > >> > Dmitriy Pavlov
> > >>
> >
> >
> >
> > --
> > Best Regards, Vyacheslav D.
> >
>

[jira] [Created] (IGNITE-7730) Improve WAL history size documentation

2018-02-15 Thread Dmitriy Pavlov (JIRA)

Dmitriy Pavlov created IGNITE-7730:
--

Summary: Improve WAL history size documentation
Key: IGNITE-7730
URL: https://issues.apache.org/jira/browse/IGNITE-7730
Project: Ignite
Issue Type: Task
Components: documentation
Affects Versions: 2.1
Reporter: Dmitriy Pavlov
Assignee: Denis Magda
Fix For: 2.5

Until IGNITE-6552 is not implemented, we have only ability to configure WAL
hist. size in checkpoints.

It is needed to improve description for this parameter.

I've added draft notes to wiki
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-Estimatingdiskspace
about ways how wer can estimate WAL sizes without exact bytes/time
specification:
{panel}
WAL Work max used size: walSegmentSize * walSegments = 640Mb (default)

in case Default WAL mode - this size is used always,

in case other modes best case is 1 segment * walSegmentSize

WAL Work+WAL Archive max size may be estimated by

average load or
by maximum size.
1st way is applicable if checkpoints are triggered mostly by timer trigger.
Wal size = 2*Average load(bytes/sec) * trigger interval (sec) * walHistSize
(number of checkpoints)
Where 2 multiplier coming from physical & logical WAL Records.

2nd way: Checkpoint is triggered by segments max dirty pages percent. Use
persisted data regions max sizes:
sum(Max configured DataRegionConfiguration.maxSize) * 75% - est. maximum data
volume to be writen on 1 checkpoint.
Overall WAL size (before archiving) = 2* est. data volume * walHistSize = 1,5 *
sum(DataRegionConfiguration.maxSize) * walHistSize

57 matches

Mail list logo