date:20180215

[jira] [Updated] (IGNITE-7730) Improve WAL history size documentation

2018-02-15 Thread Dmitriy Pavlov (JIRA)


 [ 
https://issues.apache.org/jira/browse/IGNITE-7730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dmitriy Pavlov updated IGNITE-7730:
---
Description: 
Until IGNITE-6552 is not implemented, we have only ability to configure WAL 
hist. size in checkpoints.

It is needed to improve description for this parameter.

I've added draft notes to wiki 
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-Estimatingdiskspace
about ways how wer can estimate WAL sizes without exact bytes/time 
specification:
{panel}
WAL Work max used size: walSegmentSize * walSegments = 640Mb (default)

in case Default WAL mode - this size is used always,

in case other modes best case is 1 segment * walSegmentSize

WAL Work+WAL Archive max size may be estimated by
1. average load or
2. by maximum size.
1st way is applicable if checkpoints are triggered mostly by timer trigger. 
Wal size = 2*Average load(bytes/sec) * trigger interval (sec) * walHistSize 
(number of checkpoints)
Where 2 multiplier coming from physical & logical WAL Records.

2nd way: Checkpoint is triggered by segments max dirty pages percent. Use 
persisted data regions max sizes:
sum(Max configured DataRegionConfiguration.maxSize) * 75% - est. maximum data 
volume to be writen on 1 checkpoint.
Overall WAL size (before archiving) = 2* est. data volume * walHistSize = 1,5 * 
sum(DataRegionConfiguration.maxSize) * walHistSize 

Note applying WAL compressor may significiantly reduce archive size.
{panel}

One more note from [~ivan.glukos] on dev.list we need to include. It is answer 
to question how user can determine if segment from archive folder can be safely 
removed:
{quote}
By the way: WAL compression is already implemented that way. If there
are any ".zip" segments in archive dir, they are free to delete.
This can be a safe workaround for users who experience lack of free
space - just delete compressed segments. We should mention it in
documentation for 2.4 release.
{quote}



  was:
Until IGNITE-6552 is not implemented, we have only ability to configure WAL 
hist. size in checkpoints.

It is needed to improve description for this parameter.

I've added draft notes to wiki 
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-Estimatingdiskspace
about ways how wer can estimate WAL sizes without exact bytes/time 
specification:
{panel}
WAL Work max used size: walSegmentSize * walSegments = 640Mb (default)

in case Default WAL mode - this size is used always,

in case other modes best case is 1 segment * walSegmentSize

WAL Work+WAL Archive max size may be estimated by

average load or
by maximum size.
1st way is applicable if checkpoints are triggered mostly by timer trigger. 
Wal size = 2*Average load(bytes/sec) * trigger interval (sec) * walHistSize 
(number of checkpoints)
Where 2 multiplier coming from physical & logical WAL Records.

2nd way: Checkpoint is triggered by segments max dirty pages percent. Use 
persisted data regions max sizes:
sum(Max configured DataRegionConfiguration.maxSize) * 75% - est. maximum data 
volume to be writen on 1 checkpoint.
Overall WAL size (before archiving) = 2* est. data volume * walHistSize = 1,5 * 
sum(DataRegionConfiguration.maxSize) * walHistSize 

Note applying WAL compressor may significiantly reduce archive size.
{panel}


> Improve WAL history size documentation
> --
>
> Key: IGNITE-7730
> URL: https://issues.apache.org/jira/browse/IGNITE-7730
> Project: Ignite
>  Issue Type: Task
>  Components: documentation
>Affects Versions: 2.1
>Reporter: Dmitriy Pavlov
>Assignee: Denis Magda
>Priority: Major
> Fix For: 2.5
>
>
> Until IGNITE-6552 is not implemented, we have only ability to configure WAL 
> hist. size in checkpoints.
> It is needed to improve description for this parameter.
> I've added draft notes to wiki 
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-Estimatingdiskspace
> about ways how wer can estimate WAL sizes without exact bytes/time 
> specification:
> {panel}
> WAL Work max used size: walSegmentSize * walSegments = 640Mb (default)
> in case Default WAL mode - this size is used always,
> in case other modes best case is 1 segment * walSegmentSize
> WAL Work+WAL Archive max size may be estimated by
> 1. average load or
> 2. by maximum size.
> 1st way is applicable if checkpoints are triggered mostly by timer trigger. 
> Wal size = 2*Average load(bytes/sec) * trigger interval (sec) * walHistSize 
> (number of checkpoints)
> Where 2 multiplier coming from physical & logical WAL Records.
> 2nd way: Checkpoint is triggered by segments max dirty pages percent. Use 
>

[jira] [Created] (IGNITE-7730) Improve WAL history size documentation

2018-02-15 Thread Dmitriy Pavlov (JIRA)

Dmitriy Pavlov created IGNITE-7730:
--

Summary: Improve WAL history size documentation
Key: IGNITE-7730
URL: https://issues.apache.org/jira/browse/IGNITE-7730
Project: Ignite
Issue Type: Task
Components: documentation
Affects Versions: 2.1
Reporter: Dmitriy Pavlov
Assignee: Denis Magda
Fix For: 2.5

Until IGNITE-6552 is not implemented, we have only ability to configure WAL
hist. size in checkpoints.

It is needed to improve description for this parameter.

I've added draft notes to wiki
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-Estimatingdiskspace
about ways how wer can estimate WAL sizes without exact bytes/time
specification:
{panel}
WAL Work max used size: walSegmentSize * walSegments = 640Mb (default)

in case Default WAL mode - this size is used always,

in case other modes best case is 1 segment * walSegmentSize

WAL Work+WAL Archive max size may be estimated by

average load or
by maximum size.
1st way is applicable if checkpoints are triggered mostly by timer trigger.
Wal size = 2*Average load(bytes/sec) * trigger interval (sec) * walHistSize
(number of checkpoints)
Where 2 multiplier coming from physical & logical WAL Records.

2nd way: Checkpoint is triggered by segments max dirty pages percent. Use
persisted data regions max sizes:
sum(Max configured DataRegionConfiguration.maxSize) * 75% - est. maximum data
volume to be writen on 1 checkpoint.
Overall WAL size (before archiving) = 2* est. data volume * walHistSize = 1,5 *
sum(DataRegionConfiguration.maxSize) * walHistSize

1 2 >

1 - 100 of 103 matches

Mail list logo