[
https://issues.apache.org/jira/browse/HDFS-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095454#comment-16095454
]
Daniel Pol commented on HDFS-7337:
----------------------------------
Maybe you can help me with some EC info. This is with alpha4. Not sure if this
is the correct JIRA to comment under.
I’ve built a custom EC policy and managed to use it properly (RS-5-2-256k just
because I have only 7 datanodes in my setup and want lower overhead while still
supporting 2 failures). Generates data as expected and works fine.
Problem I have is that when I restart HDFS it all gets lost and I have 2
possible cases:
1. If I have it enabled in dfs.namenode.ec.policies.enabled namenode
won’t start up as its not part of the “standard” EC policies. Error logged is:
2017-07-13 18:47:37,339 ERROR namenode.NameNode (NameNode.java:main(1709)) -
Failed to start namenode.
java.lang.IllegalArgumentException: EC policy 'RS-5-2-256k' specified at
dfs.namenode.ec.policies.enabled is not a valid policy. Please choose from list
of available policies: [RS-6-3-64k, RS-3-2-64k, RS-LEGACY-6-3-64k, XOR-2-1-64k,
RS-10-4-64k, ]
at
org.apache.hadoop.hdfs.server.namenode.ErasureCodingPolicyManager.init(ErasureCodingPolicyManager.java:123)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:868)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:693)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:664)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:726)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:949)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:928)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1637)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1704)
2. If I don’t have it enabled, namenode starts up but doesn’t leave safe
mode because it doesn’t know how to read the blocks (without knowing the custom
EC policy)
Right now, I end up deleting all EC data before restarts (or manually leave
safe mode and deleting EC data). Delays my testing quite a bit since I have to
recreate data all the time.
I’m not 100% this is a bug or just something I don’t know how to properly
configure. Maybe there’s some undocumented setting I’m not aware of that can
point to my custom EC policy XML file at startup.
> Configurable and pluggable Erasure Codec and schema
> ---------------------------------------------------
>
> Key: HDFS-7337
> URL: https://issues.apache.org/jira/browse/HDFS-7337
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: erasure-coding
> Reporter: Zhe Zhang
> Priority: Critical
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-7337-prototype-v1.patch,
> HDFS-7337-prototype-v2.zip, HDFS-7337-prototype-v3.zip,
> PluggableErasureCodec.pdf, PluggableErasureCodec-v2.pdf,
> PluggableErasureCodec-v3.pdf, PluggableErasureCodec v4.pdf
>
>
> According to HDFS-7285 and the design, this considers to support multiple
> Erasure Codecs via pluggable approach. It allows to define and configure
> multiple codec schemas with different coding algorithms and parameters. The
> resultant codec schemas can be utilized and specified via command tool for
> different file folders. While design and implement such pluggable framework,
> it’s also to implement a concrete codec by default (Reed Solomon) to prove
> the framework is useful and workable. Separate JIRA could be opened for the
> RS codec implementation.
> Note HDFS-7353 will focus on the very low level codec API and implementation
> to make concrete vendor libraries transparent to the upper layer. This JIRA
> focuses on high level stuffs that interact with configuration, schema and etc.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
