Repository: ignite Updated Branches: refs/heads/ignite-10639 52671649a -> d463e6ff0
IGNITE-10639 Proof of concept for internal description for packages: Speed based throttling. Project: http://git-wip-us.apache.org/repos/asf/ignite/repo Commit: http://git-wip-us.apache.org/repos/asf/ignite/commit/d463e6ff Tree: http://git-wip-us.apache.org/repos/asf/ignite/tree/d463e6ff Diff: http://git-wip-us.apache.org/repos/asf/ignite/diff/d463e6ff Branch: refs/heads/ignite-10639 Commit: d463e6ff0cdf42957201bc4de78ea9761b1f42cf Parents: 5267164 Author: Dmitriy Pavlov <[email protected]> Authored: Tue Dec 11 15:06:10 2018 +0300 Committer: Dmitriy Pavlov <[email protected]> Committed: Tue Dec 11 15:06:10 2018 +0300 ---------------------------------------------------------------------- .../apache/ignite/internal/package-info.java | 4 +- .../cache/persistence/pagemem/README.md | 18 ++++++ .../cache/persistence/pagemem/package-info.java | 30 ++++++++++ modules/direct-io/README.md | 62 ++++++++++++++++++++ modules/direct-io/src/README.md | 62 -------------------- 5 files changed, 113 insertions(+), 63 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/ignite/blob/d463e6ff/modules/core/src/main/java/org/apache/ignite/internal/package-info.java ---------------------------------------------------------------------- diff --git a/modules/core/src/main/java/org/apache/ignite/internal/package-info.java b/modules/core/src/main/java/org/apache/ignite/internal/package-info.java index d488e32..9d4c786 100644 --- a/modules/core/src/main/java/org/apache/ignite/internal/package-info.java +++ b/modules/core/src/main/java/org/apache/ignite/internal/package-info.java @@ -17,6 +17,8 @@ /** * <!-- Package description. --> - * Contains main implementation. + * Contains main implementation. All classes and interfaces are not public API. + * API Compatibility is not maintained. + * */ package org.apache.ignite.internal; \ No newline at end of file http://git-wip-us.apache.org/repos/asf/ignite/blob/d463e6ff/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/README.md ---------------------------------------------------------------------- diff --git a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/README.md b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/README.md new file mode 100644 index 0000000..16692ff --- /dev/null +++ b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/README.md @@ -0,0 +1,18 @@ + +Speed Based Throttling +---------------------- + +If throttling is enabled in User configuration then Speed based throttlign is applied. + +Speed based throttling is implemented by +[PagesWriteSpeedBasedThrottle.java](PagesWriteSpeedBasedThrottle.java) + +This approach estimates current number of page written and pages to be written. + +<img src="https://docs.google.com/drawings/d/e/2PACX-1vT1u2fuSdIItg67J02ukUGx3cY1tc9B-eebRSa0Hu4zwzkzpJdNSmSCpRD1EmGhYTCxa-kYqSDKOt-v/pub?w=425&h=589";> + + +From source data remained estimated time of checkpoint is calculated. Then we apply this time estimation to our progress of marking pages as dirty. + +<img src="https://docs.google.com/drawings/d/e/2PACX-1vTr9mhBts4rLzoqcRWOy78qPEL2UHMaJLIXGu4_1TlinbdLdtz5aGbhPMzy4uxLWup8dZdDsnZeOUxR/pub?w=441&h=575";> + http://git-wip-us.apache.org/repos/asf/ignite/blob/d463e6ff/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/package-info.java ---------------------------------------------------------------------- diff --git a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/package-info.java b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/package-info.java new file mode 100644 index 0000000..7697969 --- /dev/null +++ b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/package-info.java @@ -0,0 +1,30 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/** + * This package contains page memory implementation for case persitence is enabled. + * Contained clasees with <ul> + * <li>Page Memory implementation {@link org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl}.</li> + * <li>Page write throttling.</li> + * <li>Mapping of full Page IDs to place in memory segment.</li> + * <li>Pages replacement with disk.</li> + * </ul> + * + * See also + * <a href="https://github.com/apache/ignite/tree/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/pagemem";>Package description</a> + */ +package org.apache.ignite.internal.processors.cache.persistence.pagemem; http://git-wip-us.apache.org/repos/asf/ignite/blob/d463e6ff/modules/direct-io/README.md ---------------------------------------------------------------------- diff --git a/modules/direct-io/README.md b/modules/direct-io/README.md new file mode 100644 index 0000000..1bbd22d --- /dev/null +++ b/modules/direct-io/README.md @@ -0,0 +1,62 @@ +Apache Ignite Direct IO Module +------------------------------ + +Apache Ignite Direct IO is plugin, which provides page store with ability to write and read cache partitions +in O_DIRECT mode. + + +OS gets the data and stores it in a file buffer cache (Page Cache). + +<!-- To edit picture please use +https://docs.google.com/drawings/d/19xXbaWC2F2EBcd7F0T9wJ7wZO3DDR0PFR0tBqqIXjUI/edit?usp=sharing +--> +<img src="https://docs.google.com/drawings/d/e/2PACX-1vQBR0OoKFeQ1AOMyDK9QoQEBLDs4kbs7EY6Ed48HnRjlM0J1Ao3g_glD7AR3KZRtUcAVL6hQut6IPVw/pub?w=638&h=499";> + +Similarly, for every write operation, +the OS first writes the data in a cache and then transfers to the disk. To eliminate this process you can enable +Direct I/O in which case the data is read and written directly from/to the disk bypassing the file buffer cache. + +Direct I/O plugin in Ignite is used for the checkpointing process where the dirty pages in RAM are written to the disk. + +Importing Direct I/O Pluging In Maven Project +------------------------------------- + +If you are using Maven to manage dependencies of your project, you can add Direct IO Module +dependency like this (replace '${ignite.version}' with actual Ignite version you are +interested in): + +```xml +<project xmlns="http://maven.apache.org/POM/4.0.0"; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 + http://maven.apache.org/xsd/maven-4.0.0.xsd";> + ... + <dependencies> + ... + <dependency> + <groupId>org.apache.ignite</groupId> + <artifactId>ignite-direct-io</artifactId> + <version>${ignite.version}</version> + </dependency> + ... + </dependencies> + ... +</project> +``` + +Importing Direct I/O Pluging In Gradle Project +------------------------------------- +For gradle you can add compile dependency, where igniteVersion is actual Ignite version: + +```groovy +compile group: 'org.apache.ignite', name: 'ignite-direct-io', version: igniteVersion +``` + +Additional setup is not required. Once plugin is available in classpath, it will be used for Durable Memory IO. + +------------------------------------- +See more information in Apache Ignite documentation: +https://apacheignite.readme.io/docs/durable-memory-tuning#section-enabling-direct-i-o + +and description of internal desing can be found in Wiki: +https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-DirectI/O http://git-wip-us.apache.org/repos/asf/ignite/blob/d463e6ff/modules/direct-io/src/README.md ---------------------------------------------------------------------- diff --git a/modules/direct-io/src/README.md b/modules/direct-io/src/README.md deleted file mode 100644 index 1bbd22d..0000000 --- a/modules/direct-io/src/README.md +++ /dev/null @@ -1,62 +0,0 @@ -Apache Ignite Direct IO Module ------------------------------- - -Apache Ignite Direct IO is plugin, which provides page store with ability to write and read cache partitions -in O_DIRECT mode. - - -OS gets the data and stores it in a file buffer cache (Page Cache). - -<!-- To edit picture please use -https://docs.google.com/drawings/d/19xXbaWC2F2EBcd7F0T9wJ7wZO3DDR0PFR0tBqqIXjUI/edit?usp=sharing ---> -<img src="https://docs.google.com/drawings/d/e/2PACX-1vQBR0OoKFeQ1AOMyDK9QoQEBLDs4kbs7EY6Ed48HnRjlM0J1Ao3g_glD7AR3KZRtUcAVL6hQut6IPVw/pub?w=638&h=499";> - -Similarly, for every write operation, -the OS first writes the data in a cache and then transfers to the disk. To eliminate this process you can enable -Direct I/O in which case the data is read and written directly from/to the disk bypassing the file buffer cache. - -Direct I/O plugin in Ignite is used for the checkpointing process where the dirty pages in RAM are written to the disk. - -Importing Direct I/O Pluging In Maven Project -------------------------------------- - -If you are using Maven to manage dependencies of your project, you can add Direct IO Module -dependency like this (replace '${ignite.version}' with actual Ignite version you are -interested in): - -```xml -<project xmlns="http://maven.apache.org/POM/4.0.0"; - xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; - xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 - http://maven.apache.org/xsd/maven-4.0.0.xsd";> - ... - <dependencies> - ... - <dependency> - <groupId>org.apache.ignite</groupId> - <artifactId>ignite-direct-io</artifactId> - <version>${ignite.version}</version> - </dependency> - ... - </dependencies> - ... -</project> -``` - -Importing Direct I/O Pluging In Gradle Project -------------------------------------- -For gradle you can add compile dependency, where igniteVersion is actual Ignite version: - -```groovy -compile group: 'org.apache.ignite', name: 'ignite-direct-io', version: igniteVersion -``` - -Additional setup is not required. Once plugin is available in classpath, it will be used for Durable Memory IO. - -------------------------------------- -See more information in Apache Ignite documentation: -https://apacheignite.readme.io/docs/durable-memory-tuning#section-enabling-direct-i-o - -and description of internal desing can be found in Wiki: -https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-DirectI/O
