[
https://issues.apache.org/jira/browse/HDFS-14905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh Radhakrishnan updated HDFS-14905:
Fix Version/s: (was: 3.3.0)
3.2.2
Hadoop Flags: Reviewed
Release Note: Non-volatile storage class memory (SCM, also known as
persistent memory) is supported in HDFS cache. To enable SCM cache, user just
needs to configure SCM volume for property “dfs.datanode.cache.pmem.dirs” in
hdfs-site.xml. And all HDFS cache directives keep unchanged. There are two
implementations for HDFS SCM Cache, one is pure java code implementation and
the other is native PMDK based implementation. The latter implementation can
bring user better performance gain in cache write and cache read. If PMDK
native libs could be loaded, it will use PMDK based implementation otherwise it
will fallback to java code implementation. To enable PMDK based implementation,
user should install PMDK library by referring to the official site
http://pmem.io/. Then, build Hadoop with PMDK support by referring to "PMDK
library build options" section in `BUILDING.txt` in the source code. If
multiple SCM volumes are configured, a round-robin policy is used to select an
available volume for caching a block. Consistent with DRAM cache, SCM cache
also has no cache eviction mechanism. When DataNode receives a data read
request from a client, if the corresponding block is cached into SCM, DataNode
will instantiate an InputStream with the block location path on SCM (pure java
implementation) or cache address on SCM (PMDK based implementation). Once the
InputStream is created, DataNode will send the cached data to the client.
Please refer "Centralized Cache Management" guide for more details.
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks [~PhiloHe] for the consolidated patch. +1, I have cherry picked
following 10 commits from {{trunk}} to {{branch-3.2}}
{code}
HDFS-14354 - 15/March/2019
ba50a36a3ead628c3d44d384f7ed4d2b3a55dd07
HDFS-14393 -29/March/2019
f3f51284d57ef2e0c7e968b6eea56eab578f7e93
HDFS-14355 - 31/March/2019
35ff31dd9462cf4fb4ebf5556ee8ae6bcd7c5c3a
HDFS-14401 - 08/May/19
9b0aace1e6c54f201784912c0b623707aa82b761
HDFS-14402 - 29/May/19
37900c5639f8ba8d41b9fedc3d41ee0fbda7d5db
HDFS-14356 - 05/Jun/19
d1aad444907e1fc5314e8e64529e57c51ed7561c
HDFS-14458 - 15/Jul/19
e98adb00b7da8fa913b86ecf2049444b1d8617d4
HDFS-14357 - 15/Jul/19
30a8f840f1572129fe7d02f8a784c47ab57ce89a
HDFS-14700 - 09/Aug/19
f6fa865d6fcb0ef0a25a00615f16f383e5032373
HDFS-14818 - 22/Sep/19
659c88801d008bb352d10a1cb3bd0e401486cc9b
{code}
> Backport HDFS persistent memory read cache support to branch-3.2
>
>
> Key: HDFS-14905
> URL: https://issues.apache.org/jira/browse/HDFS-14905
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: caching, datanode
>Reporter: Feilong He
>Assignee: Feilong He
>Priority: Major
> Fix For: 3.2.2
>
> Attachments: HDFS-14905-branch-3.2-000.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
