[jira] [Commented] (HDFS-14851) WebHdfs Returns 200 Status Code for Open of Files with Corrupt Blocks
[ https://issues.apache.org/jira/browse/HDFS-14851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934861#comment-16934861 ] Danny Becker commented on HDFS-14851: - [~jojochuang] HDFS already does this. behavior. The problem here is that WebHdfs will return a 200 code an then fail the read because it has a missing block. HTTP protocol does not allow us to send an error response when it is expecting to read data. So we need to catch this before sending the response code. > WebHdfs Returns 200 Status Code for Open of Files with Corrupt Blocks > - > > Key: HDFS-14851 > URL: https://issues.apache.org/jira/browse/HDFS-14851 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Danny Becker >Assignee: Danny Becker >Priority: Minor > Attachments: HDFS-14851.001.patch > > > WebHdfs returns 200 status code for Open operations on files with missing or > corrupt blocks. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14851) WebHdfs Returns 200 Status Code for Open of Files with Corrupt Blocks
[ https://issues.apache.org/jira/browse/HDFS-14851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934846#comment-16934846 ] Danny Becker commented on HDFS-14851: - [~crh] I understand the concern of iterating over every block to check for a corrupt flag. I disagree with the solution of failing if any block in the file is corrupt. What if the user reads only a portion of the file which doesn't contain corrupt blocks? I don't think parallelism would work for us here either, because it isn't a very computational task, just checking a boolean flag. > WebHdfs Returns 200 Status Code for Open of Files with Corrupt Blocks > - > > Key: HDFS-14851 > URL: https://issues.apache.org/jira/browse/HDFS-14851 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Danny Becker >Assignee: Danny Becker >Priority: Minor > Attachments: HDFS-14851.001.patch > > > WebHdfs returns 200 status code for Open operations on files with missing or > corrupt blocks. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14851) WebHdfs Returns 200 Status Code for Open of Files with Corrupt Blocks
[ https://issues.apache.org/jira/browse/HDFS-14851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933819#comment-16933819 ] Wei-Chiu Chuang commented on HDFS-14851: I don't recall if something similar is implemented in HDFS open. But I would want a function parity if this is implemented in webhdfs. I'm not so concerned about iterating on the list of blocks since for my customers the average block:file ratio is between 1~2. > WebHdfs Returns 200 Status Code for Open of Files with Corrupt Blocks > - > > Key: HDFS-14851 > URL: https://issues.apache.org/jira/browse/HDFS-14851 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Danny Becker >Assignee: Danny Becker >Priority: Minor > Attachments: HDFS-14851.001.patch > > > WebHdfs returns 200 status code for Open operations on files with missing or > corrupt blocks. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14851) WebHdfs Returns 200 Status Code for Open of Files with Corrupt Blocks
[ https://issues.apache.org/jira/browse/HDFS-14851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933784#comment-16933784 ] CR Hota commented on HDFS-14851: [Íñigo Goiri|https://issues.apache.org/jira/secure/ViewProfile.jspa?name=elgoiri] Thanks for tagging me. [Danny Becker|http://jira/secure/ViewProfile.jspa?name=dannytbecker] Thanks for working on this. Yes, we do use webhdfs but haven't come across a scenario like this yet. The change looks quite expensive performance wise for all calls to fix response code. Iterating through all blocks to find what is corrupted or not looks expensive especially when 1048576 is the limit of blocks per file. We may want to rather expose an API through InputStream that exposes List of all corrupted blocks (just like it exposes getAllBlocks), if the size of this list is positive, this web call can throw BlockMissingException. Cc [~xkrogen] [~jojochuang] > WebHdfs Returns 200 Status Code for Open of Files with Corrupt Blocks > - > > Key: HDFS-14851 > URL: https://issues.apache.org/jira/browse/HDFS-14851 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Danny Becker >Assignee: Danny Becker >Priority: Minor > Attachments: HDFS-14851.001.patch > > > WebHdfs returns 200 status code for Open operations on files with missing or > corrupt blocks. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14851) WebHdfs Returns 200 Status Code for Open of Files with Corrupt Blocks
[ https://issues.apache.org/jira/browse/HDFS-14851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16930994#comment-16930994 ] Íñigo Goiri commented on HDFS-14851: [~crh], you guys use WebHDFS right? Have you found anything like this? > WebHdfs Returns 200 Status Code for Open of Files with Corrupt Blocks > - > > Key: HDFS-14851 > URL: https://issues.apache.org/jira/browse/HDFS-14851 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Danny Becker >Assignee: Danny Becker >Priority: Minor > Attachments: HDFS-14851.001.patch > > > WebHdfs returns 200 status code for Open operations on files with missing or > corrupt blocks. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
