[
https://issues.apache.org/jira/browse/HDFS-12395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16160615#comment-16160615
]
Kai Zheng commented on HDFS-12395:
----------------------------------
Thanks [~Sammi] for working on this. Having looked into the work and it looks
overall close.
1. Overall, similar to HDFS-7859, this exposes lots of not-so-relevant changes
already made in the patch or we need to make, so please feel free to open new
issues to hold such changes separately. The essential changes for this issue is
to *log add/remove/enable/disable erasure coding policy*.
2. You local change in {{pom.xml}} and {{editsStored.xml}}.
3. You changes in {{DFSClient}} looks like some bug fix to existing codes.
4. Refactor: {{getEcPolicy}} => {{getErasureCodingPolicy}};
{{AddECPolicyResponse}} => {{AddErasureCodingPolicyResponse}}
5. Don't quite like the way to sort the map by creating a tree map. And also,
could we improve {{ECSchema}} to ensure {{extraOptions}} is sorted already, so
we don't need to consider doing it in every places? If you'd do this, please in
separate issue.
{code}
+ // Sort extra options based on key
+ extraOptions = new TreeMap<String, String>(extraOptions);
{code}
6. Please use some non-meaningful options for the test purpose to avoid
possible confusion, like "testOption1" or the like.
{code}
+ Map<String, String> extraOptions = new HashMap<String, String>();
+ extraOptions.put("padding", "0");
+ extraOptions.put("recycle", "true");
{code}
7. Please try to use the same order when dump fields of erasure coding policy.
{code}
+ public static void writeErasureCodingPolicy(DataOutputStream out,
+ ErasureCodingPolicy ecPolicy) throws IOException {
+ writeInt(ecPolicy.getCellSize(), out);
+ writeString(ecPolicy.getSchema().getCodecName(), out);
+ writeInt(ecPolicy.getNumDataUnits(), out);
+ writeInt(ecPolicy.getNumParityUnits(), out);
}
{code}
{code}
+ XMLUtils.addSaxString(contentHandler, "CODEC", ecPolicy.getCodecName());
+ XMLUtils.addSaxString(contentHandler, "CELLSIZE",
+ Integer.toString(ecPolicy.getCellSize()));
+ XMLUtils.addSaxString(contentHandler, "DATAUNITS",
+ Integer.toString(ecPolicy.getNumDataUnits()));
+ XMLUtils.addSaxString(contentHandler, "PARITYUNITS",
+ Integer.toString(ecPolicy.getNumParityUnits()));
{code}
8. Not sure why we need to catch and convert the exception here, but not in
other places. Better to pass the {{e}} instead of its {e.getMessage()} when
convert to new IOException.
{code}
+ case OP_ADD_ERASURE_CODING_POLICY:
+ AddErasureCodingPolicyOp addOp = (AddErasureCodingPolicyOp) op;
+ try {
+ fsNamesys.getErasureCodingPolicyManager().addPolicy(
+ addOp.getEcPolicy());
+ } catch (HadoopIllegalArgumentException e) {
+ throw new IOException("Add erasure coding policy failed for:" +
+ e.getMessage());
+ }
{code}
> Support erasure coding policy operations in namenode edit log
> -------------------------------------------------------------
>
> Key: HDFS-12395
> URL: https://issues.apache.org/jira/browse/HDFS-12395
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
> Reporter: SammiChen
> Assignee: SammiChen
> Labels: hdfs-ec-3.0-must-do
> Attachments: editsStored, HDFS-12395.001.patch, HDFS-12395.002.patch
>
>
> Support add, remove, disable, enable erasure coding policy operation in edit
> log.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
