[jira] [Commented] (HDFS-13512) WebHdfs getFileStatus doesn't return ecPolicy
[ https://issues.apache.org/jira/browse/HDFS-13512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16478202#comment-16478202 ] Ajay Kumar commented on HDFS-13512: --- [~arpitagarwal] thanks for review and commit. [~hanishakoneru], [~shahrs87] thanks for reviews. > WebHdfs getFileStatus doesn't return ecPolicy > - > > Key: HDFS-13512 > URL: https://issues.apache.org/jira/browse/HDFS-13512 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 3.2.0, 3.1.1 > > Attachments: HDFS-13512.00.patch, HDFS-13512.01.patch, > HDFS-13512.02.patch, HDFS-13512.03.patch, HDFS-13512.04.patch, > HDFS-13512.05.patch > > > Currently LISTSTATUS call to WebHdfs returns a json. These jsonArray elements > do have the ecPolicy name. > But when WebHdfsFileSystem converts it back into a FileStatus object, the > ecPolicy is not added. This is because the json contains only the ecPolicy > name and this name is not sufficient to decode it back to ErasureCodingPolicy > object. > While converting json back to HdfsFileStatus we should set ecPolicyName > whenever it is set for give file/dir. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13512) WebHdfs getFileStatus doesn't return ecPolicy
[ https://issues.apache.org/jira/browse/HDFS-13512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477874#comment-16477874 ] Hudson commented on HDFS-13512: --- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #14211 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14211/]) HDFS-13512. WebHdfs getFileStatus doesn't return ecPolicy. Contributed (arp: rev 0fc988e6a3dc6b435cbeea680549c06ef6147e3f) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestJsonUtil.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java > WebHdfs getFileStatus doesn't return ecPolicy > - > > Key: HDFS-13512 > URL: https://issues.apache.org/jira/browse/HDFS-13512 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 3.2.0, 3.1.1 > > Attachments: HDFS-13512.00.patch, HDFS-13512.01.patch, > HDFS-13512.02.patch, HDFS-13512.03.patch, HDFS-13512.04.patch, > HDFS-13512.05.patch > > > Currently LISTSTATUS call to WebHdfs returns a json. These jsonArray elements > do have the ecPolicy name. > But when WebHdfsFileSystem converts it back into a FileStatus object, the > ecPolicy is not added. This is because the json contains only the ecPolicy > name and this name is not sufficient to decode it back to ErasureCodingPolicy > object. > While converting json back to HdfsFileStatus we should set ecPolicyName > whenever it is set for give file/dir. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13512) WebHdfs getFileStatus doesn't return ecPolicy
[ https://issues.apache.org/jira/browse/HDFS-13512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477186#comment-16477186 ] Nitiraj Singh Rathore commented on HDFS-13512: -- [~ajayydv] : will this be available in 3.1.1 release. can you please update the fix version and commit the patch? > WebHdfs getFileStatus doesn't return ecPolicy > - > > Key: HDFS-13512 > URL: https://issues.apache.org/jira/browse/HDFS-13512 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13512.00.patch, HDFS-13512.01.patch, > HDFS-13512.02.patch, HDFS-13512.03.patch, HDFS-13512.04.patch, > HDFS-13512.05.patch > > > Currently LISTSTATUS call to WebHdfs returns a json. These jsonArray elements > do have the ecPolicy name. > But when WebHdfsFileSystem converts it back into a FileStatus object, the > ecPolicy is not added. This is because the json contains only the ecPolicy > name and this name is not sufficient to decode it back to ErasureCodingPolicy > object. > While converting json back to HdfsFileStatus we should set ecPolicyName > whenever it is set for give file/dir. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13512) WebHdfs getFileStatus doesn't return ecPolicy
[
https://issues.apache.org/jira/browse/HDFS-13512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463004#comment-16463004
]
Ajay Kumar commented on HDFS-13512:
---
[~shahrs87]
{quote}I am wondering why we want to preserve ecPolicy in json payload that
server sends.
On the client side, WebhdfsFileSystem is not even reading ecPolicy from
json.{quote}
As pointed out by [~hanishakoneru] removing this may break backward
compatibility. Although we haven't documented it, json returned from webhdfs
has that field. There might be clients using it via curl.
{quote}While going through history of adding erasure coding policy in
DistributedFileSystem, I came across this jira HDFS-11565.
We should adopt the same logic in WebhdfsFileSystem also.{quote}
Its a good optimization but sending only id in case of system policies may not
be very useful for curl. Also this will be less clean solution as we will be
sending id for system defined policies and other field for other type of
policies.
> WebHdfs getFileStatus doesn't return ecPolicy
> -
>
> Key: HDFS-13512
> URL: https://issues.apache.org/jira/browse/HDFS-13512
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HDFS-13512.00.patch, HDFS-13512.01.patch,
> HDFS-13512.02.patch, HDFS-13512.03.patch, HDFS-13512.04.patch,
> HDFS-13512.05.patch
>
>
> Currently LISTSTATUS call to WebHdfs returns a json. These jsonArray elements
> do have the ecPolicy name.
> But when WebHdfsFileSystem converts it back into a FileStatus object, the
> ecPolicy is not added. This is because the json contains only the ecPolicy
> name and this name is not sufficient to decode it back to ErasureCodingPolicy
> object.
> While converting json back to HdfsFileStatus we should set ecPolicyName
> whenever it is set for give file/dir.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13512) WebHdfs getFileStatus doesn't return ecPolicy
[
https://issues.apache.org/jira/browse/HDFS-13512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462975#comment-16462975
]
Rushabh S Shah commented on HDFS-13512:
---
I was reviewing the latest patch #v5.
+JsonUtil.java+
{noformat}
+// to maintain backward comparability
m.put("ecPolicy", status.getErasureCodingPolicy().getName());
+// to re-construct HdfsFileStatus object via WebHdfs
+m.put("ecPolicyObj",
getEcPolicyAsMap(status.getErasureCodingPolicy()));
{noformat}
I am wondering why we want to preserve {{ecPolicy}} in json payload that server
sends.
On the client side, {{WebhdfsFileSystem}} is not even reading {{ecPolicy}} from
json.
Also we don't return ec related bits in {{GETFILESTATUS}} via curl also. Refer
[GETFILESTATUS|https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Status_of_a_File.2FDirectory]
So it is ok to remove {{ecPolicy}} from {{JsonUtil}} and don't worry about
backwards compatibility.
While going through history of adding erasure coding policy in
{{DistributedFileSystem}}, I came across this jira [HDFS-11565].
We should adopt the same logic in {{WebhdfsFileSystem}} also.
Hope it helps.
> WebHdfs getFileStatus doesn't return ecPolicy
> -
>
> Key: HDFS-13512
> URL: https://issues.apache.org/jira/browse/HDFS-13512
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HDFS-13512.00.patch, HDFS-13512.01.patch,
> HDFS-13512.02.patch, HDFS-13512.03.patch, HDFS-13512.04.patch,
> HDFS-13512.05.patch
>
>
> Currently LISTSTATUS call to WebHdfs returns a json. These jsonArray elements
> do have the ecPolicy name.
> But when WebHdfsFileSystem converts it back into a FileStatus object, the
> ecPolicy is not added. This is because the json contains only the ecPolicy
> name and this name is not sufficient to decode it back to ErasureCodingPolicy
> object.
> While converting json back to HdfsFileStatus we should set ecPolicyName
> whenever it is set for give file/dir.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13512) WebHdfs getFileStatus doesn't return ecPolicy
[ https://issues.apache.org/jira/browse/HDFS-13512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460156#comment-16460156 ] Ajay Kumar commented on HDFS-13512: --- [~shahrs87] Our initial plan was to get policy name without wiring more data, specially seeing the concerns around redundancy in [HDFS-11394], but since your earlier feedback was to serialize the whole policy that is the approach we are taking now. > WebHdfs getFileStatus doesn't return ecPolicy > - > > Key: HDFS-13512 > URL: https://issues.apache.org/jira/browse/HDFS-13512 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13512.00.patch, HDFS-13512.01.patch, > HDFS-13512.02.patch, HDFS-13512.03.patch, HDFS-13512.04.patch, > HDFS-13512.05.patch > > > Currently LISTSTATUS call to WebHdfs returns a json. These jsonArray elements > do have the ecPolicy name. > But when WebHdfsFileSystem converts it back into a FileStatus object, the > ecPolicy is not added. This is because the json contains only the ecPolicy > name and this name is not sufficient to decode it back to ErasureCodingPolicy > object. > While converting json back to HdfsFileStatus we should set ecPolicyName > whenever it is set for give file/dir. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13512) WebHdfs getFileStatus doesn't return ecPolicy
[ https://issues.apache.org/jira/browse/HDFS-13512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460127#comment-16460127 ] Rushabh S Shah commented on HDFS-13512: --- bq. Initial patch was work around to get at least the name without increasing redundancy. Do you still want to get just the name or you want more details ? > WebHdfs getFileStatus doesn't return ecPolicy > - > > Key: HDFS-13512 > URL: https://issues.apache.org/jira/browse/HDFS-13512 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13512.00.patch, HDFS-13512.01.patch, > HDFS-13512.02.patch, HDFS-13512.03.patch, HDFS-13512.04.patch, > HDFS-13512.05.patch > > > Currently LISTSTATUS call to WebHdfs returns a json. These jsonArray elements > do have the ecPolicy name. > But when WebHdfsFileSystem converts it back into a FileStatus object, the > ecPolicy is not added. This is because the json contains only the ecPolicy > name and this name is not sufficient to decode it back to ErasureCodingPolicy > object. > While converting json back to HdfsFileStatus we should set ecPolicyName > whenever it is set for give file/dir. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13512) WebHdfs getFileStatus doesn't return ecPolicy
[ https://issues.apache.org/jira/browse/HDFS-13512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460125#comment-16460125 ] Ajay Kumar commented on HDFS-13512: --- [~shahrs87] the goal is that any users of webHdfsFileSystem should be able to retrieve the EC policy from the returned HdfsFileStatus object. Initial patch was work around to get at least the name without increasing redundancy. > WebHdfs getFileStatus doesn't return ecPolicy > - > > Key: HDFS-13512 > URL: https://issues.apache.org/jira/browse/HDFS-13512 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13512.00.patch, HDFS-13512.01.patch, > HDFS-13512.02.patch, HDFS-13512.03.patch, HDFS-13512.04.patch, > HDFS-13512.05.patch > > > Currently LISTSTATUS call to WebHdfs returns a json. These jsonArray elements > do have the ecPolicy name. > But when WebHdfsFileSystem converts it back into a FileStatus object, the > ecPolicy is not added. This is because the json contains only the ecPolicy > name and this name is not sufficient to decode it back to ErasureCodingPolicy > object. > While converting json back to HdfsFileStatus we should set ecPolicyName > whenever it is set for give file/dir. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13512) WebHdfs getFileStatus doesn't return ecPolicy
[
https://issues.apache.org/jira/browse/HDFS-13512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460117#comment-16460117
]
Rushabh S Shah commented on HDFS-13512:
---
Still trying to understand what is the motivation of this jira.
[~ajayydv]: What do you want to see on the client side ? Do you just
ecPolicyName ?
Looking back at {{HDFS-13512.01.patch}}, it looks like you were just interested
in returning the name.
> WebHdfs getFileStatus doesn't return ecPolicy
> -
>
> Key: HDFS-13512
> URL: https://issues.apache.org/jira/browse/HDFS-13512
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HDFS-13512.00.patch, HDFS-13512.01.patch,
> HDFS-13512.02.patch, HDFS-13512.03.patch, HDFS-13512.04.patch,
> HDFS-13512.05.patch
>
>
> Currently LISTSTATUS call to WebHdfs returns a json. These jsonArray elements
> do have the ecPolicy name.
> But when WebHdfsFileSystem converts it back into a FileStatus object, the
> ecPolicy is not added. This is because the json contains only the ecPolicy
> name and this name is not sufficient to decode it back to ErasureCodingPolicy
> object.
> While converting json back to HdfsFileStatus we should set ecPolicyName
> whenever it is set for give file/dir.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
