[
https://issues.apache.org/jira/browse/HDFS-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16146400#comment-16146400
]
Lei (Eddy) Xu commented on HDFS-7859:
-------------------------------------
Hi, [~Sammi]
Thanks for the latest patch. I have some questions regarding the latest patch.
{code}
private byte state; // 0x01 - enabled, 0x02 - deleted
{code}
Can we use enum here to be more explicitly?
{code}
message ErasureCodingPolicyManagerSection {
351 // number of enabled EC policies
352 required uint32 numEnabledPolicies = 1;
353 // number of not enabled policies, including system and user
defined policy
354 required uint32 numOtherPolicies = 2;
355 // repeated ErasureCodingPolicyProto enabledPolicies;
356 // repeated ErasureCodingPolicyProto otherPolicies;
357 }
{code}
Are only the number of policies persisted? It looks off to me. It will depends
on the order of system pre-defined policies. So when the pluggable EC policy
being merged, would that impact the correctness of loading / saving fsimage? It
_might_ also make upgrade / downgrade difficult.
{code}
public HashSet<String> getEcPoliciesOnDir() {
239 return ecPoliciesOnDir;
240 }
241
242 public HashSet<Byte> getEcPoliciesOnFile() {
243 return ecPoliciesOnFile;
244 }
245
246 public void setCountEcPolicies(Boolean enabled) {
247 countEcPolicies = enabled;
248 }
249
{code}
What are the consumers of these functions? lso
{code}
public static final class PersistState {
246 public final ErasureCodingPolicyManagerSection section;
247 public final List<ErasureCodingPolicy> enabledPolicies;
248 public final List<ErasureCodingPolicy> otherPolicies;
{code}
Can these fields be {{private final}}?
{code}
// FSImage.java
import org.apache.hadoop.hdfs.protocol.IllegalECPolicyException;
{code}
Seems not necessary? also in {{FSImageFormat.java}}
{code}
private INode loadINode(INodeSection.INode n) throws IOException
{code}
Is this {{IOE}} necessary?
{code}
// TestFSEditLogLoader.java
cluster.shutdown();
754 cluster = null;
755 } finally {
756 if (cluster != null) {
757 cluster.shutdown();
758 }
759 }
{code}
In general, you can use {{try...resource}} for {{MiniDFSCluster}} .
A more general question to [~Sammi], do you think whether it is possible to get
this in beta1?
Thanks!
> Erasure Coding: Persist erasure coding policies in NameNode
> -----------------------------------------------------------
>
> Key: HDFS-7859
> URL: https://issues.apache.org/jira/browse/HDFS-7859
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Kai Zheng
> Assignee: SammiChen
> Attachments: HDFS-7859.001.patch, HDFS-7859.002.patch,
> HDFS-7859.004.patch, HDFS-7859.005.patch, HDFS-7859.006.patch,
> HDFS-7859.007.patch, HDFS-7859.008.patch, HDFS-7859.009.patch,
> HDFS-7859.010.patch, HDFS-7859.011.patch, HDFS-7859-HDFS-7285.002.patch,
> HDFS-7859-HDFS-7285.002.patch, HDFS-7859-HDFS-7285.003.patch
>
>
> In meetup discussion with [~zhz] and [~jingzhao], it's suggested that we
> persist EC schemas in NameNode centrally and reliably, so that EC zones can
> reference them by name efficiently.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
