[jira] [Comment Edited] (HDFS-13512) WebHdfs getFileStatus doesn't return ecPolicy
[ https://issues.apache.org/jira/browse/HDFS-13512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463004#comment-16463004 ] Ajay Kumar edited comment on HDFS-13512 at 5/3/18 9:58 PM: --- [~shahrs87] {quote}I am wondering why we want to preserve ecPolicy in json payload that server sends. On the client side, WebhdfsFileSystem is not even reading ecPolicy from json.{quote} As pointed out by [~hanishakoneru] removing this may break backward compatibility. Although we haven't documented it, json returned from webhdfs has that field. There might be clients using it via curl. {quote}While going through history of adding erasure coding policy in DistributedFileSystem, I came across this jira HDFS-11565. We should adopt the same logic in WebhdfsFileSystem also.{quote} Its a good optimization but sending only id in case of system policies may not be very useful for curl. This will be less clean solution as we will be sending id for system defined policies and other field for other type of policies. Also as you earlier pointed out this is what we return in DFS. was (Author: ajayydv): [~shahrs87] {quote}I am wondering why we want to preserve ecPolicy in json payload that server sends. On the client side, WebhdfsFileSystem is not even reading ecPolicy from json.{quote} As pointed out by [~hanishakoneru] removing this may break backward compatibility. Although we haven't documented it, json returned from webhdfs has that field. There might be clients using it via curl. {quote}While going through history of adding erasure coding policy in DistributedFileSystem, I came across this jira HDFS-11565. We should adopt the same logic in WebhdfsFileSystem also.{quote} Its a good optimization but sending only id in case of system policies may not be very useful for curl. Also this will be less clean solution as we will be sending id for system defined policies and other field for other type of policies. Also as you earlier pointed out this is what we return in DFS. > WebHdfs getFileStatus doesn't return ecPolicy > - > > Key: HDFS-13512 > URL: https://issues.apache.org/jira/browse/HDFS-13512 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13512.00.patch, HDFS-13512.01.patch, > HDFS-13512.02.patch, HDFS-13512.03.patch, HDFS-13512.04.patch, > HDFS-13512.05.patch > > > Currently LISTSTATUS call to WebHdfs returns a json. These jsonArray elements > do have the ecPolicy name. > But when WebHdfsFileSystem converts it back into a FileStatus object, the > ecPolicy is not added. This is because the json contains only the ecPolicy > name and this name is not sufficient to decode it back to ErasureCodingPolicy > object. > While converting json back to HdfsFileStatus we should set ecPolicyName > whenever it is set for give file/dir. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13512) WebHdfs getFileStatus doesn't return ecPolicy
[ https://issues.apache.org/jira/browse/HDFS-13512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463004#comment-16463004 ] Ajay Kumar edited comment on HDFS-13512 at 5/3/18 9:57 PM: --- [~shahrs87] {quote}I am wondering why we want to preserve ecPolicy in json payload that server sends. On the client side, WebhdfsFileSystem is not even reading ecPolicy from json.{quote} As pointed out by [~hanishakoneru] removing this may break backward compatibility. Although we haven't documented it, json returned from webhdfs has that field. There might be clients using it via curl. {quote}While going through history of adding erasure coding policy in DistributedFileSystem, I came across this jira HDFS-11565. We should adopt the same logic in WebhdfsFileSystem also.{quote} Its a good optimization but sending only id in case of system policies may not be very useful for curl. Also this will be less clean solution as we will be sending id for system defined policies and other field for other type of policies. Also as you earlier pointed out this is what we return in DFS. was (Author: ajayydv): [~shahrs87] {quote}I am wondering why we want to preserve ecPolicy in json payload that server sends. On the client side, WebhdfsFileSystem is not even reading ecPolicy from json.{quote} As pointed out by [~hanishakoneru] removing this may break backward compatibility. Although we haven't documented it, json returned from webhdfs has that field. There might be clients using it via curl. {quote}While going through history of adding erasure coding policy in DistributedFileSystem, I came across this jira HDFS-11565. We should adopt the same logic in WebhdfsFileSystem also.{quote} Its a good optimization but sending only id in case of system policies may not be very useful for curl. Also this will be less clean solution as we will be sending id for system defined policies and other field for other type of policies. > WebHdfs getFileStatus doesn't return ecPolicy > - > > Key: HDFS-13512 > URL: https://issues.apache.org/jira/browse/HDFS-13512 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13512.00.patch, HDFS-13512.01.patch, > HDFS-13512.02.patch, HDFS-13512.03.patch, HDFS-13512.04.patch, > HDFS-13512.05.patch > > > Currently LISTSTATUS call to WebHdfs returns a json. These jsonArray elements > do have the ecPolicy name. > But when WebHdfsFileSystem converts it back into a FileStatus object, the > ecPolicy is not added. This is because the json contains only the ecPolicy > name and this name is not sufficient to decode it back to ErasureCodingPolicy > object. > While converting json back to HdfsFileStatus we should set ecPolicyName > whenever it is set for give file/dir. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13512) WebHdfs getFileStatus doesn't return ecPolicy
[ https://issues.apache.org/jira/browse/HDFS-13512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460156#comment-16460156 ] Ajay Kumar edited comment on HDFS-13512 at 5/1/18 9:07 PM: --- [~shahrs87] Our initial plan was to get policy name without wiring more data, specially seeing the concerns around redundancy in [HDFS-11394], but since your earlier feedback was to serialize the whole policy that is the approach we are taking now. Could you plz share your take on this and any concerns you have now? was (Author: ajayydv): [~shahrs87] Our initial plan was to get policy name without wiring more data, specially seeing the concerns around redundancy in [HDFS-11394], but since your earlier feedback was to serialize the whole policy that is the approach we are taking now. > WebHdfs getFileStatus doesn't return ecPolicy > - > > Key: HDFS-13512 > URL: https://issues.apache.org/jira/browse/HDFS-13512 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13512.00.patch, HDFS-13512.01.patch, > HDFS-13512.02.patch, HDFS-13512.03.patch, HDFS-13512.04.patch, > HDFS-13512.05.patch > > > Currently LISTSTATUS call to WebHdfs returns a json. These jsonArray elements > do have the ecPolicy name. > But when WebHdfsFileSystem converts it back into a FileStatus object, the > ecPolicy is not added. This is because the json contains only the ecPolicy > name and this name is not sufficient to decode it back to ErasureCodingPolicy > object. > While converting json back to HdfsFileStatus we should set ecPolicyName > whenever it is set for give file/dir. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13512) WebHdfs getFileStatus doesn't return ecPolicy
[ https://issues.apache.org/jira/browse/HDFS-13512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460117#comment-16460117 ] Rushabh S Shah edited comment on HDFS-13512 at 5/1/18 8:33 PM: --- Still trying to understand what is the motivation of this jira. [~ajayydv]: What do you want to see on the client side ? Do you just want ecPolicyName ? Looking back at {{HDFS-13512.01.patch}}, it looks like you were just interested in returning the name. was (Author: shahrs87): Still trying to understand what is the motivation of this jira. [~ajayydv]: What do you want to see on the client side ? Do you just ecPolicyName ? Looking back at {{HDFS-13512.01.patch}}, it looks like you were just interested in returning the name. > WebHdfs getFileStatus doesn't return ecPolicy > - > > Key: HDFS-13512 > URL: https://issues.apache.org/jira/browse/HDFS-13512 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13512.00.patch, HDFS-13512.01.patch, > HDFS-13512.02.patch, HDFS-13512.03.patch, HDFS-13512.04.patch, > HDFS-13512.05.patch > > > Currently LISTSTATUS call to WebHdfs returns a json. These jsonArray elements > do have the ecPolicy name. > But when WebHdfsFileSystem converts it back into a FileStatus object, the > ecPolicy is not added. This is because the json contains only the ecPolicy > name and this name is not sufficient to decode it back to ErasureCodingPolicy > object. > While converting json back to HdfsFileStatus we should set ecPolicyName > whenever it is set for give file/dir. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org