[
https://issues.apache.org/jira/browse/HDFS-14039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16677408#comment-16677408
]
Xiao Chen commented on HDFS-14039:
----------------------------------
Thanks for the work here, Kitti!
I think there is only 1 problem left, which is the behavior when the default
policy is configured v.s. explicit enable / disable.
The way I understand it, the default policy configuration overrides the enable
/ disable. Specifically:
- enable / disable commands works orthogonal to the default policy configuration
- if default policy is configured, disabling a policy will have an entry in
edits / image, but the policy is still effective and someone can -setPolicy
with it
- if the config is changed across NN restarts, the default policy 'override' is
always just the config.
To do this, it feels to me we should have ECPM cache {{defaultPolicyName}}
during init (or particularly, the parsed policy object of it). Then, instead of
modifying the {{ErasureCodingPolicyInfo}} object, we check if the policy is the
default on other operations: {{enabledPolicies / enabledPolicies}} contains the
default policy, while the state of the {{ErasureCodingPolicyInfo}} in
{{allPolicies}} doesn't change. This way, even if someone does a
{{saveNamespace}}, the default policy state won't be persisted (which happens
currently, IMO a mistake and inconsistency).
> ec -listPolicies doesn't show correct state for the default policy when the
> default is not RS(6,3)
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-14039
> URL: https://issues.apache.org/jira/browse/HDFS-14039
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
> Affects Versions: 3.0.0
> Reporter: Xiao Chen
> Assignee: Kitti Nanasi
> Priority: Major
> Attachments: HDFS-14039.001.patch, HDFS-14039.002.patch,
> HDFS-14039.003.patch
>
>
> {noformat}
> $ hdfs ec -listPolicies
> Erasure Coding Policies:
> ErasureCodingPolicy=[Name=RS-10-4-1024k, Schema=[ECSchema=[Codec=rs,
> numDataUnits=10, numParityUnits=4]], CellSize=1048576, Id=5], State=DISABLED
> ErasureCodingPolicy=[Name=RS-3-2-1024k, Schema=[ECSchema=[Codec=rs,
> numDataUnits=3, numParityUnits=2]], CellSize=1048576, Id=2], State=DISABLED
> ErasureCodingPolicy=[Name=RS-6-3-1024k, Schema=[ECSchema=[Codec=rs,
> numDataUnits=6, numParityUnits=3]], CellSize=1048576, Id=1], State=ENABLED
> ErasureCodingPolicy=[Name=RS-LEGACY-6-3-1024k,
> Schema=[ECSchema=[Codec=rs-legacy, numDataUnits=6, numParityUnits=3]],
> CellSize=1048576, Id=3], State=DISABLED
> ErasureCodingPolicy=[Name=XOR-2-1-1024k, Schema=[ECSchema=[Codec=xor,
> numDataUnits=2, numParityUnits=1]], CellSize=1048576, Id=4], State=DISABLED
> $ hdfs ec -enablePolicy -policy XOR-2-1-1024k
> Erasure coding policy XOR-2-1-1024k is enabled
> $ hdfs ec -listPolicies
> Erasure Coding Policies:
> ErasureCodingPolicy=[Name=RS-10-4-1024k, Schema=[ECSchema=[Codec=rs,
> numDataUnits=10, numParityUnits=4]], CellSize=1048576, Id=5], State=DISABLED
> ErasureCodingPolicy=[Name=RS-3-2-1024k, Schema=[ECSchema=[Codec=rs,
> numDataUnits=3, numParityUnits=2]], CellSize=1048576, Id=2], State=DISABLED
> ErasureCodingPolicy=[Name=RS-6-3-1024k, Schema=[ECSchema=[Codec=rs,
> numDataUnits=6, numParityUnits=3]], CellSize=1048576, Id=1], State=ENABLED
> ErasureCodingPolicy=[Name=RS-LEGACY-6-3-1024k,
> Schema=[ECSchema=[Codec=rs-legacy, numDataUnits=6, numParityUnits=3]],
> CellSize=1048576, Id=3], State=DISABLED
> ErasureCodingPolicy=[Name=XOR-2-1-1024k, Schema=[ECSchema=[Codec=xor,
> numDataUnits=2, numParityUnits=1]], CellSize=1048576, Id=4], State=ENABLED
> ----------------------------------
> $ #set default to be RS-3-2 for dfs.namenode.ec.system.default.policy, and
> restart NN
> (this seems to be what's triggering the failure)
> -----------------------------------
> $ hdfs ec -listPolicies
> Erasure Coding Policies:
> ErasureCodingPolicy=[Name=RS-10-4-1024k, Schema=[ECSchema=[Codec=rs,
> numDataUnits=10, numParityUnits=4]], CellSize=1048576, Id=5], State=DISABLED
> ErasureCodingPolicy=[Name=RS-3-2-1024k, Schema=[ECSchema=[Codec=rs,
> numDataUnits=3, numParityUnits=2]], CellSize=1048576, Id=2], State=DISABLED
> ErasureCodingPolicy=[Name=RS-6-3-1024k, Schema=[ECSchema=[Codec=rs,
> numDataUnits=6, numParityUnits=3]], CellSize=1048576, Id=1], State=ENABLED
> ErasureCodingPolicy=[Name=RS-LEGACY-6-3-1024k,
> Schema=[ECSchema=[Codec=rs-legacy, numDataUnits=6, numParityUnits=3]],
> CellSize=1048576, Id=3], State=DISABLED
> ErasureCodingPolicy=[Name=XOR-2-1-1024k, Schema=[ECSchema=[Codec=xor,
> numDataUnits=2, numParityUnits=1]], CellSize=1048576, Id=4], State=ENABLED
> $ hdfs ec -enablePolicy -policy RS-3-2-1024k
> Erasure coding policy RS-3-2-1024k is enabled
> $ hdfs ec -listPolicies
> Erasure Coding Policies:
> ErasureCodingPolicy=[Name=RS-10-4-1024k, Schema=[ECSchema=[Codec=rs,
> numDataUnits=10, numParityUnits=4]], CellSize=1048576, Id=5], State=DISABLED
> ErasureCodingPolicy=[Name=RS-3-2-1024k, Schema=[ECSchema=[Codec=rs,
> numDataUnits=3, numParityUnits=2]], CellSize=1048576, Id=2], State=DISABLED
> ErasureCodingPolicy=[Name=RS-6-3-1024k, Schema=[ECSchema=[Codec=rs,
> numDataUnits=6, numParityUnits=3]], CellSize=1048576, Id=1], State=ENABLED
> ErasureCodingPolicy=[Name=RS-LEGACY-6-3-1024k,
> Schema=[ECSchema=[Codec=rs-legacy, numDataUnits=6, numParityUnits=3]],
> CellSize=1048576, Id=3], State=DISABLED
> ErasureCodingPolicy=[Name=XOR-2-1-1024k, Schema=[ECSchema=[Codec=xor,
> numDataUnits=2, numParityUnits=1]], CellSize=1048576, Id=4], State=ENABLED
> {noformat}
> The last 2 should show RS-3-2 as ENABLED. RS-6-3 DISABLED if it's not enabled
> before.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
