Repository: ignite Updated Branches: refs/heads/master 5a8960ca7 -> d7fd859ca
IGNITE-10639 Proof of concept for internal description for packages: Navigation from internal to throttling - Fixes #5642. Signed-off-by: Dmitriy Pavlov <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/ignite/repo Commit: http://git-wip-us.apache.org/repos/asf/ignite/commit/d7fd859c Tree: http://git-wip-us.apache.org/repos/asf/ignite/tree/d7fd859c Diff: http://git-wip-us.apache.org/repos/asf/ignite/diff/d7fd859c Branch: refs/heads/master Commit: d7fd859ca92f22a4947818a4020a0e4fea11a4cc Parents: 5a8960c Author: Dmitriy Pavlov <[email protected]> Authored: Wed Dec 12 18:21:24 2018 +0300 Committer: Dmitriy Pavlov <[email protected]> Committed: Wed Dec 12 18:21:24 2018 +0300 ---------------------------------------------------------------------- .../java/org/apache/ignite/internal/README.md | 12 ++++ .../apache/ignite/internal/package-info.java | 4 +- .../apache/ignite/internal/processors/README.md | 27 +++++++++ .../ignite/internal/processors/cache/README.md | 8 +++ .../internal/processors/cache/package-info.java | 23 ++++++++ .../processors/cache/persistence/README.md | 10 ++++ .../cache/persistence/package-info.java | 23 ++++++++ .../cache/persistence/pagemem/README.md | 24 ++++++++ .../cache/persistence/pagemem/package-info.java | 30 ++++++++++ .../internal/processors/package-info.java | 8 ++- modules/direct-io/README.md | 62 ++++++++++++++++++++ 11 files changed, 228 insertions(+), 3 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/ignite/blob/d7fd859c/modules/core/src/main/java/org/apache/ignite/internal/README.md ---------------------------------------------------------------------- diff --git a/modules/core/src/main/java/org/apache/ignite/internal/README.md b/modules/core/src/main/java/org/apache/ignite/internal/README.md new file mode 100644 index 0000000..8d0f208 --- /dev/null +++ b/modules/core/src/main/java/org/apache/ignite/internal/README.md @@ -0,0 +1,12 @@ +Apache Ignite Internals +----------------------- + +Contains implementation classes for Apache Ignite. + +*Ignite Components* + +All internal Ignite components implements [GridComponent.java](GridComponent.java) - interface for +- [processors](processors) and for +- [managers](managers) - has associated SPI. + +Service Provider Interface (SPI) abbreviation is here because Apache Ignite was designed as a pluggable product. But Ignite users usually do not define own implementations. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/ignite/blob/d7fd859c/modules/core/src/main/java/org/apache/ignite/internal/package-info.java ---------------------------------------------------------------------- diff --git a/modules/core/src/main/java/org/apache/ignite/internal/package-info.java b/modules/core/src/main/java/org/apache/ignite/internal/package-info.java index d488e32..f897eea 100644 --- a/modules/core/src/main/java/org/apache/ignite/internal/package-info.java +++ b/modules/core/src/main/java/org/apache/ignite/internal/package-info.java @@ -17,6 +17,8 @@ /** * <!-- Package description. --> - * Contains main implementation. + * Contains main implementation. All classes and interfaces are not public API. + * API Compatibility is not maintained. + * See also <a href='https://github.com/apache/ignite/tree/master/modules/core/src/main/java/org/apache/ignite/internal'>GitHub Readme</a> */ package org.apache.ignite.internal; \ No newline at end of file http://git-wip-us.apache.org/repos/asf/ignite/blob/d7fd859c/modules/core/src/main/java/org/apache/ignite/internal/processors/README.md ---------------------------------------------------------------------- diff --git a/modules/core/src/main/java/org/apache/ignite/internal/processors/README.md b/modules/core/src/main/java/org/apache/ignite/internal/processors/README.md new file mode 100644 index 0000000..05a1feb --- /dev/null +++ b/modules/core/src/main/java/org/apache/ignite/internal/processors/README.md @@ -0,0 +1,27 @@ +Apache Ignite Processors +------------------------ + +A processor is Apache Ignite component with the lifecycle. This lifecycle is associated with Ignite Node lifecycle. + +Despite Managers, Ignite processors are not associated with an SPI. + +Cache Processors and Implementation +----------------------------------- +Main grid function from the point of end-user view is a mapping of keys (K) to values (V) +```K->V``` +This mapping is implemented by [cache](cache) + +There is also an affinity key. Usually Key and Affinity Key are equivalent. + +But collocation of data may require transformation from Key to +```K->Affinity Key``` + +Affinity key is mapped to [cache](cache) partition +```K->Affinity Key->Partition``` + +Affinity Key to partition mapping should always be static (any conditions, any JVM). Number of partitions is constant and does not change during grid lifetime, 1024 default + +Affinity Function is also responsible for mapping from partition to target (ideal) node: +```Partition->Node``` + +There are primary nodes and backup (default # of backups = 0 for performance reasons). For replicated cache backups count = nodes count. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/ignite/blob/d7fd859c/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/README.md ---------------------------------------------------------------------- diff --git a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/README.md b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/README.md new file mode 100644 index 0000000..f8c50ce --- /dev/null +++ b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/README.md @@ -0,0 +1,8 @@ +Apache Ignite Cache Processors +------------------------------ + +### Native Persistence +Apache Ignite has its own [Native Persistence](persistence) - Implementation + +### Ignite Cache Entries +Each entry represended by a subclass [GridCacheMapEntry](GridCacheMapEntry.java) \ No newline at end of file http://git-wip-us.apache.org/repos/asf/ignite/blob/d7fd859c/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/package-info.java ---------------------------------------------------------------------- diff --git a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/package-info.java b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/package-info.java new file mode 100644 index 0000000..ab907a0 --- /dev/null +++ b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/package-info.java @@ -0,0 +1,23 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/** + * This package contain cache-related processors & persistence implementation. <br> + * See also + * <a href="https://github.com/apache/ignite/tree/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache";>GitHub Package Readme</a> + */ +package org.apache.ignite.internal.processors.cache; http://git-wip-us.apache.org/repos/asf/ignite/blob/d7fd859c/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/README.md ---------------------------------------------------------------------- diff --git a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/README.md b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/README.md new file mode 100644 index 0000000..372a108 --- /dev/null +++ b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/README.md @@ -0,0 +1,10 @@ +Apache Ignite Persistence +------------------------- +See Native Persistence introduction in [Ignite Persistent Store - under the hood](https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood) + +Contents +-------- +This package contains +- [Page Memory](pagemem) implementation for storage enabled. +- [Write-Ahead Log](wal) implementation for storage enabled. +- [FileNames generation](filename) logic \ No newline at end of file http://git-wip-us.apache.org/repos/asf/ignite/blob/d7fd859c/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/package-info.java ---------------------------------------------------------------------- diff --git a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/package-info.java b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/package-info.java new file mode 100644 index 0000000..42b07b1 --- /dev/null +++ b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/package-info.java @@ -0,0 +1,23 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/** + * This package contain cache with persitence implementation. <br> + * See also + * <a href="https://github.com/apache/ignite/tree/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence";>GitHub Package Readme</a> + */ +package org.apache.ignite.internal.processors.cache.persistence; http://git-wip-us.apache.org/repos/asf/ignite/blob/d7fd859c/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/README.md ---------------------------------------------------------------------- diff --git a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/README.md b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/README.md new file mode 100644 index 0000000..57e623d --- /dev/null +++ b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/README.md @@ -0,0 +1,24 @@ +Apache Ignite Native Peristence Page Memory +------------------------------------------- +This package contains page memory implementation for case persitence is enabled. + +Speed Based Throttling +---------------------- +For an introduction, please see +[wiki PagesWriteThrottling](https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-PagesWriteThrottling) + +If throttling is enabled in User configuration, then Speed based throttling is applied. + +Speed based throttling is implemented by +[PagesWriteSpeedBasedThrottle.java](PagesWriteSpeedBasedThrottle.java) + +Throttling is not active outside of [checkpoint](../checkpoint) process. +But when checkpoint is in progress speed based-throttling approach estimates the current number of pages written and pages to be written. + +<img src="https://docs.google.com/drawings/d/e/2PACX-1vT1u2fuSdIItg67J02ukUGx3cY1tc9B-eebRSa0Hu4zwzkzpJdNSmSCpRD1EmGhYTCxa-kYqSDKOt-v/pub?w=425&h=589";> + + +From source data remained estimated time of checkpoint is calculated. Then we apply this time estimation to our progress of marking pages as dirty. + +<img src="https://docs.google.com/drawings/d/e/2PACX-1vTr9mhBts4rLzoqcRWOy78qPEL2UHMaJLIXGu4_1TlinbdLdtz5aGbhPMzy4uxLWup8dZdDsnZeOUxR/pub?w=441&h=575";> + http://git-wip-us.apache.org/repos/asf/ignite/blob/d7fd859c/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/package-info.java ---------------------------------------------------------------------- diff --git a/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/package-info.java b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/package-info.java new file mode 100644 index 0000000..5bc3a20 --- /dev/null +++ b/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/package-info.java @@ -0,0 +1,30 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/** + * This package contains page memory implementation for case persitence is enabled. + * Contained clasees with <ul> + * <li>Page Memory implementation {@link org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl}.</li> + * <li>Page Write Throttling.</li> + * <li>Mapping of full Page IDs to place in memory segment.</li> + * <li>Pages replacement with disk.</li> + * </ul> + * + * See also + * <a href="https://github.com/apache/ignite/tree/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/pagemem";>Package description</a> + */ +package org.apache.ignite.internal.processors.cache.persistence.pagemem; http://git-wip-us.apache.org/repos/asf/ignite/blob/d7fd859c/modules/core/src/main/java/org/apache/ignite/internal/processors/package-info.java ---------------------------------------------------------------------- diff --git a/modules/core/src/main/java/org/apache/ignite/internal/processors/package-info.java b/modules/core/src/main/java/org/apache/ignite/internal/processors/package-info.java index d76884a..16c8cdd 100644 --- a/modules/core/src/main/java/org/apache/ignite/internal/processors/package-info.java +++ b/modules/core/src/main/java/org/apache/ignite/internal/processors/package-info.java @@ -16,7 +16,11 @@ */ /** - * <!-- Package description. --> - * TODO. + * Package with Apache Ignite internal Processors. + * Processor is Ignite component with lifecycle associated with node lifecycle. + * + * See also + * <a href='https://github.com/apache/ignite/tree/master/modules/core/src/main/java/org/apache/ignite/internal/processors/'>Package Readme</a> + * */ package org.apache.ignite.internal.processors; \ No newline at end of file http://git-wip-us.apache.org/repos/asf/ignite/blob/d7fd859c/modules/direct-io/README.md ---------------------------------------------------------------------- diff --git a/modules/direct-io/README.md b/modules/direct-io/README.md new file mode 100644 index 0000000..ca35214 --- /dev/null +++ b/modules/direct-io/README.md @@ -0,0 +1,62 @@ +Apache Ignite Direct IO Module +------------------------------ + +Apache Ignite Direct IO is plugin, which provides page store with ability to write and read cache partitions +in O_DIRECT mode. + + +OS gets the data and stores it in a file buffer cache (Page Cache). + +<!-- To edit picture please use +https://docs.google.com/drawings/d/19xXbaWC2F2EBcd7F0T9wJ7wZO3DDR0PFR0tBqqIXjUI/edit?usp=sharing +--> +<img src="https://docs.google.com/drawings/d/e/2PACX-1vQBR0OoKFeQ1AOMyDK9QoQEBLDs4kbs7EY6Ed48HnRjlM0J1Ao3g_glD7AR3KZRtUcAVL6hQut6IPVw/pub?w=638&h=499";> + +Similarly, for every write operation, +the OS first writes the data in a cache and then transfers to the disk. To eliminate this process you can enable +Direct I/O in which case the data is read and written directly from/to the disk bypassing the file buffer cache. + +Direct I/O plugin in Ignite is used for the checkpointing process where the dirty pages in RAM are written to the disk. + +Importing Direct I/O Pluging In Maven Project +------------------------------------- + +If you are using Maven to manage dependencies of your project, you can add Direct IO Module +dependency like this (replace '${ignite.version}' with actual Ignite version you are +interested in): + +```xml +<project xmlns="http://maven.apache.org/POM/4.0.0"; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 + http://maven.apache.org/xsd/maven-4.0.0.xsd";> + ... + <dependencies> + ... + <dependency> + <groupId>org.apache.ignite</groupId> + <artifactId>ignite-direct-io</artifactId> + <version>${ignite.version}</version> + </dependency> + ... + </dependencies> + ... +</project> +``` + +Importing Direct I/O Pluging In Gradle Project +------------------------------------- +For gradle you can add compile dependency, where igniteVersion is actual Ignite version: + +```groovy +compile group: 'org.apache.ignite', name: 'ignite-direct-io', version: igniteVersion +``` + +Additional setup is not required. Once plugin is available in classpath, it will be used for Durable Memory IO. + +------------------------------------- +See more information in Apache Ignite documentation: +[How to enable Direct IO](https://apacheignite.readme.io/docs/durable-memory-tuning#section-enabling-direct-i-o) + +and description of internal desing can be found in Wiki: +[Under the hood: Direct IO](https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStore-underthehood-DirectI/O)
