[jira] [Updated] (HDFS-12671) [READ] Test NameNode restarts when PROVIDED is configured
[ https://issues.apache.org/jira/browse/HDFS-12671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-12671: -- Resolution: Fixed Status: Resolved (was: Patch Available) > [READ] Test NameNode restarts when PROVIDED is configured > - > > Key: HDFS-12671 > URL: https://issues.apache.org/jira/browse/HDFS-12671 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti > Attachments: HDFS-12671-HDFS-9806.001.patch, > HDFS-12671-HDFS-9806.002.patch, HDFS-12671-HDFS-9806.003.patch, > HDFS-12671-HDFS-9806.004.patch > > > Add test case to ensure namenode restarts can be handled with provided > storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11640) [READ] Datanodes should use a unique identifier when reading from external stores
[ https://issues.apache.org/jira/browse/HDFS-11640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-11640: -- Status: Patch Available (was: Open) > [READ] Datanodes should use a unique identifier when reading from external > stores > - > > Key: HDFS-11640 > URL: https://issues.apache.org/jira/browse/HDFS-11640 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti > Attachments: HDFS-11640-HDFS-9806.001.patch, > HDFS-11640-HDFS-9806.002.patch > > > Use a unique identifier when reading from external stores to ensure that > datanodes read the correct (version of) file. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11640) [READ] Datanodes should use a unique identifier when reading from external stores
[
https://issues.apache.org/jira/browse/HDFS-11640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-11640:
--
Attachment: HDFS-11640-HDFS-9806.002.patch
Posting a patch that uses a {{PathHandle}} to read from remote filesystems. As
{{PathHandle}} is currently not widely supported, the patch defaults to opening
with the remote URI if {{open(PathHandle, HandleOpts[]...)}} is not supported
by the remote {{FileSystem}}.
> [READ] Datanodes should use a unique identifier when reading from external
> stores
> -
>
> Key: HDFS-11640
> URL: https://issues.apache.org/jira/browse/HDFS-11640
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
> Attachments: HDFS-11640-HDFS-9806.001.patch,
> HDFS-11640-HDFS-9806.002.patch
>
>
> Use a unique identifier when reading from external stores to ensure that
> datanodes read the correct (version of) file.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12671) [READ] Test NameNode restarts when PROVIDED is configured
[ https://issues.apache.org/jira/browse/HDFS-12671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242839#comment-16242839 ] Virajith Jalaparti commented on HDFS-12671: --- Thanks! Filed HDFS-12789 to fix the FindBugs error. I will commit v4 of the patch to the feature branch. > [READ] Test NameNode restarts when PROVIDED is configured > - > > Key: HDFS-12671 > URL: https://issues.apache.org/jira/browse/HDFS-12671 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti > Attachments: HDFS-12671-HDFS-9806.001.patch, > HDFS-12671-HDFS-9806.002.patch, HDFS-12671-HDFS-9806.003.patch, > HDFS-12671-HDFS-9806.004.patch > > > Add test case to ensure namenode restarts can be handled with provided > storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12779) [READ] Allow cluster id to be specified to the Image generation tool
[ https://issues.apache.org/jira/browse/HDFS-12779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-12779: -- Status: Patch Available (was: Open) > [READ] Allow cluster id to be specified to the Image generation tool > > > Key: HDFS-12779 > URL: https://issues.apache.org/jira/browse/HDFS-12779 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti >Priority: Trivial > Attachments: HDFS-12779-HDFS-9806.001.patch > > > Setting the cluster id for the FSImage generated for PROVIDED files is > required when the Namenode for PROVIDED files is expected to run in > federation with other Namenodes that manage local storage/data. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12779) [READ] Allow cluster id to be specified to the Image generation tool
[ https://issues.apache.org/jira/browse/HDFS-12779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-12779: -- Status: Open (was: Patch Available) > [READ] Allow cluster id to be specified to the Image generation tool > > > Key: HDFS-12779 > URL: https://issues.apache.org/jira/browse/HDFS-12779 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti >Priority: Trivial > Attachments: HDFS-12779-HDFS-9806.001.patch > > > Setting the cluster id for the FSImage generated for PROVIDED files is > required when the Namenode for PROVIDED files is expected to run in > federation with other Namenodes that manage local storage/data. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12776) [READ] Increasing replication for PROVIDED files should create local replicas
[ https://issues.apache.org/jira/browse/HDFS-12776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-12776: -- Status: Patch Available (was: Open) > [READ] Increasing replication for PROVIDED files should create local replicas > - > > Key: HDFS-12776 > URL: https://issues.apache.org/jira/browse/HDFS-12776 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti > Attachments: HDFS-12776-HDFS-9806.001.patch > > > For PROVIDED files, set replication only works when the target datanode does > not have a PROVIDED volume. In a cluster, where all Datanodes have PROVIDED > volumes, set replication does not work. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12789) [READ] Image generation tool does not close an opened stream
Virajith Jalaparti created HDFS-12789: - Summary: [READ] Image generation tool does not close an opened stream Key: HDFS-12789 URL: https://issues.apache.org/jira/browse/HDFS-12789 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Virajith Jalaparti -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11640) [READ] Datanodes should use a unique identifier when reading from external stores
[ https://issues.apache.org/jira/browse/HDFS-11640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242845#comment-16242845 ] Virajith Jalaparti commented on HDFS-11640: --- [~chris.douglas], can you please take a look? > [READ] Datanodes should use a unique identifier when reading from external > stores > - > > Key: HDFS-11640 > URL: https://issues.apache.org/jira/browse/HDFS-11640 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti > Attachments: HDFS-11640-HDFS-9806.001.patch, > HDFS-11640-HDFS-9806.002.patch > > > Use a unique identifier when reading from external stores to ensure that > datanodes read the correct (version of) file. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12671) [READ] Test NameNode restarts when PROVIDED is configured
[ https://issues.apache.org/jira/browse/HDFS-12671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti reassigned HDFS-12671: - Assignee: Virajith Jalaparti > [READ] Test NameNode restarts when PROVIDED is configured > - > > Key: HDFS-12671 > URL: https://issues.apache.org/jira/browse/HDFS-12671 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti > Attachments: HDFS-12671-HDFS-9806.001.patch, > HDFS-12671-HDFS-9806.002.patch, HDFS-12671-HDFS-9806.003.patch, > HDFS-12671-HDFS-9806.004.patch > > > Add test case to ensure namenode restarts can be handled with provided > storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12779) [READ] Allow cluster id to be specified to the Image generation tool
[ https://issues.apache.org/jira/browse/HDFS-12779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-12779: -- Status: Patch Available (was: Open) > [READ] Allow cluster id to be specified to the Image generation tool > > > Key: HDFS-12779 > URL: https://issues.apache.org/jira/browse/HDFS-12779 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti >Priority: Trivial > Attachments: HDFS-12779-HDFS-9806.001.patch > > > Setting the cluster id for the FSImage generated for PROVIDED files is > required when the Namenode for PROVIDED files is expected to run in > federation with other Namenodes that manage local storage/data. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12779) [READ] Allow cluster id to be specified to the Image generation tool
[ https://issues.apache.org/jira/browse/HDFS-12779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-12779: -- Status: Open (was: Patch Available) > [READ] Allow cluster id to be specified to the Image generation tool > > > Key: HDFS-12779 > URL: https://issues.apache.org/jira/browse/HDFS-12779 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti >Priority: Trivial > Attachments: HDFS-12779-HDFS-9806.001.patch > > > Setting the cluster id for the FSImage generated for PROVIDED files is > required when the Namenode for PROVIDED files is expected to run in > federation with other Namenodes that manage local storage/data. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12671) [READ] Test NameNode restarts when PROVIDED is configured
[ https://issues.apache.org/jira/browse/HDFS-12671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-12671: -- Status: Patch Available (was: Open) > [READ] Test NameNode restarts when PROVIDED is configured > - > > Key: HDFS-12671 > URL: https://issues.apache.org/jira/browse/HDFS-12671 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti > Attachments: HDFS-12671-HDFS-9806.001.patch, > HDFS-12671-HDFS-9806.002.patch, HDFS-12671-HDFS-9806.003.patch, > HDFS-12671-HDFS-9806.004.patch > > > Add test case to ensure namenode restarts can be handled with provided > storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12671) [READ] Test NameNode restarts when PROVIDED is configured
[ https://issues.apache.org/jira/browse/HDFS-12671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-12671: -- Status: Open (was: Patch Available) > [READ] Test NameNode restarts when PROVIDED is configured > - > > Key: HDFS-12671 > URL: https://issues.apache.org/jira/browse/HDFS-12671 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti > Attachments: HDFS-12671-HDFS-9806.001.patch, > HDFS-12671-HDFS-9806.002.patch, HDFS-12671-HDFS-9806.003.patch, > HDFS-12671-HDFS-9806.004.patch > > > Add test case to ensure namenode restarts can be handled with provided > storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12671) [READ] Test NameNode restarts when PROVIDED is configured
[
https://issues.apache.org/jira/browse/HDFS-12671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12671:
--
Attachment: HDFS-12671-HDFS-9806.004.patch
Thanks for taking a look [~elgoiri]. I changed the {{assertTrue}} to
{{assertEquals}} in v4 of the patch. I will file a new JIRA for fixing the
findbugs issue.
> [READ] Test NameNode restarts when PROVIDED is configured
> -
>
> Key: HDFS-12671
> URL: https://issues.apache.org/jira/browse/HDFS-12671
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
> Attachments: HDFS-12671-HDFS-9806.001.patch,
> HDFS-12671-HDFS-9806.002.patch, HDFS-12671-HDFS-9806.003.patch,
> HDFS-12671-HDFS-9806.004.patch
>
>
> Add test case to ensure namenode restarts can be handled with provided
> storage.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9240) Use Builder pattern for BlockLocation constructors
[ https://issues.apache.org/jira/browse/HDFS-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241238#comment-16241238 ] Virajith Jalaparti commented on HDFS-9240: -- The checkstyle issues are from the builder pattern, and the failed tests are unrelated and pass locally. The ASF license warning is spurious. > Use Builder pattern for BlockLocation constructors > -- > > Key: HDFS-9240 > URL: https://issues.apache.org/jira/browse/HDFS-9240 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Xiaoyu Yao >Assignee: Virajith Jalaparti >Priority: Minor > Attachments: HDFS-9240.001.patch, HDFS-9240.002.patch > > > This JIRA is opened to refactor the 8 telescoping constructors of > BlockLocation class with Builder pattern. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12713) [READ] Refactor FileRegion and BlockAliasMap to separate out HDFS metadata and PROVIDED storage metadata
[
https://issues.apache.org/jira/browse/HDFS-12713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16238627#comment-16238627
]
Virajith Jalaparti edited comment on HDFS-12713 at 11/6/17 11:43 PM:
-
[~ehiggs], please see [this
comment|https://issues.apache.org/jira/browse/HDFS-12665?focusedCommentId=16238568=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16238568]
on HDFS-12665. I think this JIRA should be used to add the block pool id to
the {{BlockAliasMap}}, and in turn to its current implementation
{{TextFileRegionAliasMap}}.
was (Author: virajith):
Please see [this
comment|https://issues.apache.org/jira/browse/HDFS-12665?focusedCommentId=16238568=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16238568]
on HDFS-12665. I think this JIRA should be used to add the block pool id to
the {{BlockAliasMap}}, and in turn to its current implementation
{{TextFileRegionAliasMap}}.
> [READ] Refactor FileRegion and BlockAliasMap to separate out HDFS metadata
> and PROVIDED storage metadata
>
>
> Key: HDFS-12713
> URL: https://issues.apache.org/jira/browse/HDFS-12713
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Ewan Higgs
> Attachments: HDFS-12713-HDFS-9806.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12779) [READ] Allow cluster id to be specified to the Image generation tool
[ https://issues.apache.org/jira/browse/HDFS-12779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-12779: -- Status: Patch Available (was: Open) > [READ] Allow cluster id to be specified to the Image generation tool > > > Key: HDFS-12779 > URL: https://issues.apache.org/jira/browse/HDFS-12779 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti >Priority: Trivial > Attachments: HDFS-12779-HDFS-9806.001.patch > > > Setting the cluster id for the FSImage generated for PROVIDED files is > required when the Namenode for PROVIDED files is expected to run in > federation with other Namenodes that manage local storage/data. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12779) [READ] Allow cluster id to be specified to the Image generation tool
[ https://issues.apache.org/jira/browse/HDFS-12779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti reassigned HDFS-12779: - Assignee: Virajith Jalaparti > [READ] Allow cluster id to be specified to the Image generation tool > > > Key: HDFS-12779 > URL: https://issues.apache.org/jira/browse/HDFS-12779 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti >Priority: Trivial > Attachments: HDFS-12779-HDFS-9806.001.patch > > > Setting the cluster id for the FSImage generated for PROVIDED files is > required when the Namenode for PROVIDED files is expected to run in > federation with other Namenodes that manage local storage/data. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12779) [READ] Allow cluster id to be specified to the Image generation tool
[ https://issues.apache.org/jira/browse/HDFS-12779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-12779: -- Attachment: HDFS-12779-HDFS-9806.001.patch The attached patch allows the cluster id to be specified as a command line parameter when running the image generation tool. > [READ] Allow cluster id to be specified to the Image generation tool > > > Key: HDFS-12779 > URL: https://issues.apache.org/jira/browse/HDFS-12779 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti >Priority: Trivial > Attachments: HDFS-12779-HDFS-9806.001.patch > > > Setting the cluster id for the FSImage generated for PROVIDED files is > required when the Namenode for PROVIDED files is expected to run in > federation with other Namenodes that manage local storage/data. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12779) [READ] Allow cluster id to be specified to the Image generation tool
[ https://issues.apache.org/jira/browse/HDFS-12779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-12779: -- Description: Setting the cluster id for the FSImage generated for PROVIDED files is required when the Namenode for PROVIDED files is expected to run in federation with other Namenodes that manage local storage/data. > [READ] Allow cluster id to be specified to the Image generation tool > > > Key: HDFS-12779 > URL: https://issues.apache.org/jira/browse/HDFS-12779 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti >Priority: Trivial > > Setting the cluster id for the FSImage generated for PROVIDED files is > required when the Namenode for PROVIDED files is expected to run in > federation with other Namenodes that manage local storage/data. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12671) [READ] Test NameNode restarts when PROVIDED is configured
[ https://issues.apache.org/jira/browse/HDFS-12671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-12671: -- Attachment: HDFS-12671-HDFS-9806.003.patch > [READ] Test NameNode restarts when PROVIDED is configured > - > > Key: HDFS-12671 > URL: https://issues.apache.org/jira/browse/HDFS-12671 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti > Attachments: HDFS-12671-HDFS-9806.001.patch, > HDFS-12671-HDFS-9806.002.patch, HDFS-12671-HDFS-9806.003.patch > > > Add test case to ensure namenode restarts can be handled with provided > storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12671) [READ] Test NameNode restarts when PROVIDED is configured
[ https://issues.apache.org/jira/browse/HDFS-12671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-12671: -- Status: Patch Available (was: Open) > [READ] Test NameNode restarts when PROVIDED is configured > - > > Key: HDFS-12671 > URL: https://issues.apache.org/jira/browse/HDFS-12671 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti > Attachments: HDFS-12671-HDFS-9806.001.patch, > HDFS-12671-HDFS-9806.002.patch, HDFS-12671-HDFS-9806.003.patch > > > Add test case to ensure namenode restarts can be handled with provided > storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12671) [READ] Test NameNode restarts when PROVIDED is configured
[ https://issues.apache.org/jira/browse/HDFS-12671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-12671: -- Status: Open (was: Patch Available) > [READ] Test NameNode restarts when PROVIDED is configured > - > > Key: HDFS-12671 > URL: https://issues.apache.org/jira/browse/HDFS-12671 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti > Attachments: HDFS-12671-HDFS-9806.001.patch, > HDFS-12671-HDFS-9806.002.patch > > > Add test case to ensure namenode restarts can be handled with provided > storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12776) [READ] Increasing replication for PROVIDED files should create local replicas
[
https://issues.apache.org/jira/browse/HDFS-12776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241015#comment-16241015
]
Virajith Jalaparti edited comment on HDFS-12776 at 11/6/17 10:30 PM:
-
Posting a patch that
# allows local replicas to be created for PROVIDED blocks on Datanodes which
already consist a PROVIDED replica, i.e., allows replacing the PROVIDED block
with a local replica (changes to {{FsDatasetImpl}})
# prefers local replicas to provided replicas when deleting replicas (changes
to {{BlockInfo}}).
# Modifies an existing test case to include the scenario where all Datanodes
have provided and disk volumes.
was (Author: virajith):
Posting a patch that
(i) allows local replicas to be created for PROVIDED blocks on Datanodes which
already consist a PROVIDED replica, i.e., allows replacing the PROVIDED block
with a local replica (changes to {{FsDatasetImpl}})
(ii) prefers local replicas to provided replicas when deleting replicas
(changes to {{BlockInfo}}).
(iii) Modifies an existing test case to include the scenario where all
Datanodes have provided and disk volumes.
> [READ] Increasing replication for PROVIDED files should create local replicas
> -
>
> Key: HDFS-12776
> URL: https://issues.apache.org/jira/browse/HDFS-12776
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
> Attachments: HDFS-12776-HDFS-9806.001.patch
>
>
> For PROVIDED files, set replication only works when the target datanode does
> not have a PROVIDED volume. In a cluster, where all Datanodes have PROVIDED
> volumes, set replication does not work.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12776) [READ] Increasing replication for PROVIDED files should create local replicas
[
https://issues.apache.org/jira/browse/HDFS-12776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12776:
--
Attachment: HDFS-12776-HDFS-9806.001.patch
Posting a patch that
(i) allows local replicas to be created for PROVIDED blocks on Datanodes which
already consist a PROVIDED replica, i.e., allows replacing the PROVIDED block
with a local replica (changes to {{FsDatasetImpl}})
(ii) prefers local replicas to provided replicas when deleting replicas
(changes to {{BlockInfo}}).
(iii) Modifies an existing test case to include the scenario where all
Datanodes have provided and disk volumes.
> [READ] Increasing replication for PROVIDED files should create local replicas
> -
>
> Key: HDFS-12776
> URL: https://issues.apache.org/jira/browse/HDFS-12776
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
> Attachments: HDFS-12776-HDFS-9806.001.patch
>
>
> For PROVIDED files, set replication only works when the target datanode does
> not have a PROVIDED volume. In a cluster, where all Datanodes have PROVIDED
> volumes, set replication does not work.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12776) [READ] Increasing replication for PROVIDED files should create local replicas
[ https://issues.apache.org/jira/browse/HDFS-12776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-12776: -- Status: Patch Available (was: Open) > [READ] Increasing replication for PROVIDED files should create local replicas > - > > Key: HDFS-12776 > URL: https://issues.apache.org/jira/browse/HDFS-12776 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti > Attachments: HDFS-12776-HDFS-9806.001.patch > > > For PROVIDED files, set replication only works when the target datanode does > not have a PROVIDED volume. In a cluster, where all Datanodes have PROVIDED > volumes, set replication does not work. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12671) [READ] Test NameNode restarts when PROVIDED is configured
[ https://issues.apache.org/jira/browse/HDFS-12671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-12671: -- Attachment: HDFS-12671-HDFS-9806.002.patch > [READ] Test NameNode restarts when PROVIDED is configured > - > > Key: HDFS-12671 > URL: https://issues.apache.org/jira/browse/HDFS-12671 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti > Attachments: HDFS-12671-HDFS-9806.001.patch, > HDFS-12671-HDFS-9806.002.patch > > > Add test case to ensure namenode restarts can be handled with provided > storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12671) [READ] Test NameNode restarts when PROVIDED is configured
[ https://issues.apache.org/jira/browse/HDFS-12671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240797#comment-16240797 ] Virajith Jalaparti commented on HDFS-12671: --- Posting a rebased patch. [~goiri], can you take a look at this? Thanks! > [READ] Test NameNode restarts when PROVIDED is configured > - > > Key: HDFS-12671 > URL: https://issues.apache.org/jira/browse/HDFS-12671 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti > Attachments: HDFS-12671-HDFS-9806.001.patch, > HDFS-12671-HDFS-9806.002.patch > > > Add test case to ensure namenode restarts can be handled with provided > storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12671) [READ] Test NameNode restarts when PROVIDED is configured
[ https://issues.apache.org/jira/browse/HDFS-12671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-12671: -- Status: Patch Available (was: Open) > [READ] Test NameNode restarts when PROVIDED is configured > - > > Key: HDFS-12671 > URL: https://issues.apache.org/jira/browse/HDFS-12671 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti > Attachments: HDFS-12671-HDFS-9806.001.patch, > HDFS-12671-HDFS-9806.002.patch > > > Add test case to ensure namenode restarts can be handled with provided > storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12607) [READ] Even one dead datanode with PROVIDED storage results in ProvidedStorageInfo being marked as FAILED
[
https://issues.apache.org/jira/browse/HDFS-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240719#comment-16240719
]
Virajith Jalaparti commented on HDFS-12607:
---
The failing tests are unrelated and pass locally. Thanks for the review
[~goiri]! Committing this to the feature branch.
> [READ] Even one dead datanode with PROVIDED storage results in
> ProvidedStorageInfo being marked as FAILED
> -
>
> Key: HDFS-12607
> URL: https://issues.apache.org/jira/browse/HDFS-12607
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-12607-HDFS-9806.001.patch,
> HDFS-12607-HDFS-9806.002.patch, HDFS-12607-HDFS-9806.003.patch,
> HDFS-12607.repro.patch
>
>
> When a DN configured with PROVIDED storage is marked as dead by the NN, the
> state of {{providedStorageInfo}} in {{ProvidedStorageMap}} is set to FAILED,
> and never becomes NORMAL. The state should change to FAILED only if all
> datanodes with PROVIDED storage are dead, and should be restored back to
> NORMAL when a Datanode with NORMAL DatanodeStorage reports in.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12607) [READ] Even one dead datanode with PROVIDED storage results in ProvidedStorageInfo being marked as FAILED
[
https://issues.apache.org/jira/browse/HDFS-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12607:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
> [READ] Even one dead datanode with PROVIDED storage results in
> ProvidedStorageInfo being marked as FAILED
> -
>
> Key: HDFS-12607
> URL: https://issues.apache.org/jira/browse/HDFS-12607
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-12607-HDFS-9806.001.patch,
> HDFS-12607-HDFS-9806.002.patch, HDFS-12607-HDFS-9806.003.patch,
> HDFS-12607.repro.patch
>
>
> When a DN configured with PROVIDED storage is marked as dead by the NN, the
> state of {{providedStorageInfo}} in {{ProvidedStorageMap}} is set to FAILED,
> and never becomes NORMAL. The state should change to FAILED only if all
> datanodes with PROVIDED storage are dead, and should be restored back to
> NORMAL when a Datanode with NORMAL DatanodeStorage reports in.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12779) [READ] Allow cluster id to be specified to the Image generation tool
Virajith Jalaparti created HDFS-12779: - Summary: [READ] Allow cluster id to be specified to the Image generation tool Key: HDFS-12779 URL: https://issues.apache.org/jira/browse/HDFS-12779 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Virajith Jalaparti -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12713) [READ] Refactor FileRegion and BlockAliasMap to separate out HDFS metadata and PROVIDED storage metadata
[
https://issues.apache.org/jira/browse/HDFS-12713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16238627#comment-16238627
]
Virajith Jalaparti commented on HDFS-12713:
---
Please see [this
comment|https://issues.apache.org/jira/browse/HDFS-12665?focusedCommentId=16238568=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16238568]
on HDFS-12665. I think this JIRA should be used to add the block pool id to
the {{BlockAliasMap}}, and in turn to its current implementation
{{TextFileRegionAliasMap}}.
> [READ] Refactor FileRegion and BlockAliasMap to separate out HDFS metadata
> and PROVIDED storage metadata
>
>
> Key: HDFS-12713
> URL: https://issues.apache.org/jira/browse/HDFS-12713
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Ewan Higgs
>Priority: Major
> Attachments: HDFS-12713-HDFS-9806.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12775) [READ] Fix reporting of Provided volumes
[
https://issues.apache.org/jira/browse/HDFS-12775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12775:
--
Description: Provided Volumes currently report infinite capacity and 0
space used. Further, PROVIDED locations are reported as
{{/default-rack/null:0}} in fsck. This JIRA is for making this more readable,
and replace these with what users would expect. (was: Provided Volumes
currently report infinite capacity and 0 space used. This JIRA aims to replace
this with a capacity report that users would expect.)
> [READ] Fix reporting of Provided volumes
>
>
> Key: HDFS-12775
> URL: https://issues.apache.org/jira/browse/HDFS-12775
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Priority: Major
>
> Provided Volumes currently report infinite capacity and 0 space used.
> Further, PROVIDED locations are reported as {{/default-rack/null:0}} in fsck.
> This JIRA is for making this more readable, and replace these with what users
> would expect.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12775) [READ] Fix reporting for Provided volumes
[ https://issues.apache.org/jira/browse/HDFS-12775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-12775: -- Summary: [READ] Fix reporting for Provided volumes (was: [READ] Fix capacity reporting for Provided volumes) > [READ] Fix reporting for Provided volumes > - > > Key: HDFS-12775 > URL: https://issues.apache.org/jira/browse/HDFS-12775 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti >Priority: Major > > Provided Volumes currently report infinite capacity and 0 space used. This > JIRA aims to replace this with a capacity report that users would expect. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12775) [READ] Fix reporting of Provided volumes
[ https://issues.apache.org/jira/browse/HDFS-12775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-12775: -- Summary: [READ] Fix reporting of Provided volumes (was: [READ] Fix reporting for Provided volumes) > [READ] Fix reporting of Provided volumes > > > Key: HDFS-12775 > URL: https://issues.apache.org/jira/browse/HDFS-12775 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti >Priority: Major > > Provided Volumes currently report infinite capacity and 0 space used. This > JIRA aims to replace this with a capacity report that users would expect. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12777) [READ] Reduce memory and CPU footprint for PROVIDED volumes.
[
https://issues.apache.org/jira/browse/HDFS-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12777:
--
Description:
As opposed to local blocks, each DN keeps track of all blocks in PROVIDED
storage. This can be millions of blocks for 100s of TBs of PROVIDED data.
Storing the data for these blocks can lead to a large memory footprint.
Further, with so many blocks, {{DirectoryScanner}} running on a PROVIDED volume
can increase the memory and CPU utilization.
To reduce these overheads, this JIRA aims to (a) disable the
{{DirectoryScanner}} on PROVIDED volumes (as HDFS-9806 focuses on only
read-only data in PROVIDED volumes), (b) reduce the space occupied by
{{FinalizedProvidedReplicaInfo by using a common URI prefix across all PROVIDED
blocks.
was:
As opposed to local blocks, each DN keeps track of all blocks in PROVIDED
storage. This can be millions of blocks for 100s of TBs of PROVIDED data. This
JIRA aims to reduce the memory footprint of these blocks by using a common URI
prefix across all PROVIDED blocks.
Further, with so many blocks the DirectoryScanner can take up a lot of
> [READ] Reduce memory and CPU footprint for PROVIDED volumes.
>
>
> Key: HDFS-12777
> URL: https://issues.apache.org/jira/browse/HDFS-12777
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Priority: Major
>
> As opposed to local blocks, each DN keeps track of all blocks in PROVIDED
> storage. This can be millions of blocks for 100s of TBs of PROVIDED data.
> Storing the data for these blocks can lead to a large memory footprint.
> Further, with so many blocks, {{DirectoryScanner}} running on a PROVIDED
> volume can increase the memory and CPU utilization.
> To reduce these overheads, this JIRA aims to (a) disable the
> {{DirectoryScanner}} on PROVIDED volumes (as HDFS-9806 focuses on only
> read-only data in PROVIDED volumes), (b) reduce the space occupied by
> {{FinalizedProvidedReplicaInfo by using a common URI prefix across all
> PROVIDED blocks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12777) [READ] Reduce memory and CPU footprint for PROVIDED volumes.
[
https://issues.apache.org/jira/browse/HDFS-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12777:
--
Description:
As opposed to local blocks, each DN keeps track of all blocks in PROVIDED
storage. This can be millions of blocks for 100s of TBs of PROVIDED data.
Storing the data for these blocks can lead to a large memory footprint.
Further, with so many blocks, {{DirectoryScanner}} running on a PROVIDED volume
can increase the memory and CPU utilization.
To reduce these overheads, this JIRA aims to (a) disable the
{{DirectoryScanner}} on PROVIDED volumes (as HDFS-9806 focuses on only
read-only data in PROVIDED volumes), (b) reduce the space occupied by
{{FinalizedProvidedReplicaInfo}} by using a common URI prefix across all
PROVIDED blocks.
was:
As opposed to local blocks, each DN keeps track of all blocks in PROVIDED
storage. This can be millions of blocks for 100s of TBs of PROVIDED data.
Storing the data for these blocks can lead to a large memory footprint.
Further, with so many blocks, {{DirectoryScanner}} running on a PROVIDED volume
can increase the memory and CPU utilization.
To reduce these overheads, this JIRA aims to (a) disable the
{{DirectoryScanner}} on PROVIDED volumes (as HDFS-9806 focuses on only
read-only data in PROVIDED volumes), (b) reduce the space occupied by
{{FinalizedProvidedReplicaInfo by using a common URI prefix across all PROVIDED
blocks.
> [READ] Reduce memory and CPU footprint for PROVIDED volumes.
>
>
> Key: HDFS-12777
> URL: https://issues.apache.org/jira/browse/HDFS-12777
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Priority: Major
>
> As opposed to local blocks, each DN keeps track of all blocks in PROVIDED
> storage. This can be millions of blocks for 100s of TBs of PROVIDED data.
> Storing the data for these blocks can lead to a large memory footprint.
> Further, with so many blocks, {{DirectoryScanner}} running on a PROVIDED
> volume can increase the memory and CPU utilization.
> To reduce these overheads, this JIRA aims to (a) disable the
> {{DirectoryScanner}} on PROVIDED volumes (as HDFS-9806 focuses on only
> read-only data in PROVIDED volumes), (b) reduce the space occupied by
> {{FinalizedProvidedReplicaInfo}} by using a common URI prefix across all
> PROVIDED blocks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12777) [READ] Reduce memory and CPU footprint for PROVIDED volumes.
[ https://issues.apache.org/jira/browse/HDFS-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-12777: -- Summary: [READ] Reduce memory and CPU footprint for PROVIDED volumes. (was: [READ] Share URI prefix across Provided blocks to lower memory footprint.) > [READ] Reduce memory and CPU footprint for PROVIDED volumes. > > > Key: HDFS-12777 > URL: https://issues.apache.org/jira/browse/HDFS-12777 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti >Priority: Major > > As opposed to local blocks, each DN keeps track of all blocks in PROVIDED > storage. This can be millions of blocks for 100s of TBs of PROVIDED data. > This JIRA aims to reduce the memory footprint of these blocks by using a > common URI prefix across all PROVIDED blocks. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12777) [READ] Reduce memory and CPU footprint for PROVIDED volumes.
[ https://issues.apache.org/jira/browse/HDFS-12777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-12777: -- Description: As opposed to local blocks, each DN keeps track of all blocks in PROVIDED storage. This can be millions of blocks for 100s of TBs of PROVIDED data. This JIRA aims to reduce the memory footprint of these blocks by using a common URI prefix across all PROVIDED blocks. Further, with so many blocks the DirectoryScanner can take up a lot of was:As opposed to local blocks, each DN keeps track of all blocks in PROVIDED storage. This can be millions of blocks for 100s of TBs of PROVIDED data. This JIRA aims to reduce the memory footprint of these blocks by using a common URI prefix across all PROVIDED blocks. > [READ] Reduce memory and CPU footprint for PROVIDED volumes. > > > Key: HDFS-12777 > URL: https://issues.apache.org/jira/browse/HDFS-12777 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti >Priority: Major > > As opposed to local blocks, each DN keeps track of all blocks in PROVIDED > storage. This can be millions of blocks for 100s of TBs of PROVIDED data. > This JIRA aims to reduce the memory footprint of these blocks by using a > common URI prefix across all PROVIDED blocks. > Further, with so many blocks the DirectoryScanner can take up a lot of -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12778) [READ] Report multiple locations for PROVIDED blocks
Virajith Jalaparti created HDFS-12778:
-
Summary: [READ] Report multiple locations for PROVIDED blocks
Key: HDFS-12778
URL: https://issues.apache.org/jira/browse/HDFS-12778
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Virajith Jalaparti
Priority: Major
On {{getBlockLocations}}, only one Datanode is returned as the location for all
PROVIDED blocks. This can hurt the performance of applications which typically
3 locations per block. We need to return multiple Datanodes for each PROVIDED
block for better application performance/resilience.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12777) [READ] Share URI prefix across Provided blocks to lower memory footprint.
Virajith Jalaparti created HDFS-12777: - Summary: [READ] Share URI prefix across Provided blocks to lower memory footprint. Key: HDFS-12777 URL: https://issues.apache.org/jira/browse/HDFS-12777 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Virajith Jalaparti Priority: Major As opposed to local blocks, each DN keeps track of all blocks in PROVIDED storage. This can be millions of blocks for 100s of TBs of PROVIDED data. This JIRA aims to reduce the memory footprint of these blocks by using a common URI prefix across all PROVIDED blocks. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12776) [READ] Increasing replication for PROVIDED files should create local replicas
Virajith Jalaparti created HDFS-12776: - Summary: [READ] Increasing replication for PROVIDED files should create local replicas Key: HDFS-12776 URL: https://issues.apache.org/jira/browse/HDFS-12776 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Virajith Jalaparti Priority: Major For PROVIDED files, set replication only works when the target datanode does not have a PROVIDED volume. In a cluster, where all Datanodes have PROVIDED volumes, set replication does not work. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12775) [READ] Fix capacity reporting for Provided volumes
Virajith Jalaparti created HDFS-12775: - Summary: [READ] Fix capacity reporting for Provided volumes Key: HDFS-12775 URL: https://issues.apache.org/jira/browse/HDFS-12775 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Virajith Jalaparti Priority: Major Provided Volumes currently report infinite capacity and 0 space used. This JIRA aims to replace this with a capacity report that users would expect. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12665) [AliasMap] Create a version of the AliasMap that runs in memory in the Namenode (leveldb)
[
https://issues.apache.org/jira/browse/HDFS-12665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16238568#comment-16238568
]
Virajith Jalaparti commented on HDFS-12665:
---
Hi [~ehiggs], thanks for posting the new patch. A couple of comments:
# This patch includes changes that are part of HDFS-11902. Can you post a patch
that does not include these?
# The datanode should check the block pool id associated with a FileRegion
before loading it. The patch eliminates this check (in
{{ProvidedBlockPoolSlice}}). This should be retained as it ensures that the
Datanode doesn't load blocks that shouldn't be associated with a Namenode. For
example, consider the case where a DN reports to two Namenodes, NN1 and NN2, in
federation. Only NN1 is configured with PROVIDED. Both NN1 and NN2 might have a
block with the same id but NN1 refers to a PROVIDED block and NN2 refers to a
local block. The DN needs to distinguish these two blocks with the same id.
One way for the DN to know this is if the {{FileRegion}} or {{AliasMap}} has a
block pool id associated with it. This ensures that the block can be
distinguished in the {{ReplicaMap}} of the {{FsDatasetImpl}} and the two block
aren't fixed up.
My proposal is to have the following as part of the API of {{BlockAliasMap}} so
that we have know the block pool id from the alias map.
{code}
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/blockaliasmap/BlockAliasMap.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/blockaliasmap/BlockAliasMap.java
index d276fb52036..e564097fd2e 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/blockaliasmap/BlockAliasMap.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/blockaliasmap/BlockAliasMap.java
@@ -47,6 +47,7 @@
*/
public abstract U resolve(Block ident) throws IOException;
+public abstract String getBlockPoolID() throws IOException;
}
/**
@@ -74,10 +75,12 @@
/**
* Returns the writer for the alias map.
* @param opts writer options.
+ * @param blockPoolID block pool id to use
* @return {@link Writer} to the alias map.
* @throws IOException
*/
- public abstract Writer getWriter(Writer.Options opts) throws IOException;
+ public abstract Writer getWriter(Writer.Options opts, String blockPoolID)
+ throws IOException;
/**
* Refresh the alias map.
{code}
I think this change along with the change of adding the
{{ProvidedStorageLocation}} should be done as part of HDFS-12713.
> [AliasMap] Create a version of the AliasMap that runs in memory in the
> Namenode (leveldb)
> -
>
> Key: HDFS-12665
> URL: https://issues.apache.org/jira/browse/HDFS-12665
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ewan Higgs
>Assignee: Ewan Higgs
>Priority: Major
> Attachments: HDFS-12665-HDFS-9806.001.patch,
> HDFS-12665-HDFS-9806.002.patch, HDFS-12665-HDFS-9806.003.patch
>
>
> The design of Provided Storage requires the use of an AliasMap to manage the
> mapping between blocks of files on the local HDFS and ranges of files on a
> remote storage system. To reduce load from the Namenode, this can be done
> using a pluggable external service (e.g. AzureTable, Cassandra, Ratis).
> However, to aide adoption and ease of deployment, we propose an in memory
> version.
> This AliasMap will be a wrapper around LevelDB (already a dependency from the
> Timeline Service) and use protobuf for the key (blockpool, blockid, and
> genstamp) and the value (url, offset, length, nonce). The in memory service
> will also have a configurable port on which it will listen for updates from
> Storage Policy Satisfier (SPS) Coordinating Datanodes (C-DN).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12685) [READ] FsVolumeImpl exception when scanning Provided storage volume
[
https://issues.apache.org/jira/browse/HDFS-12685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12685:
--
Attachment: HDFS-12685-HDFS-9806.002.patch
Posting a patch that rebases the earlier patch on the most recent version of
HDFS-9806 branch.
> [READ] FsVolumeImpl exception when scanning Provided storage volume
> ---
>
> Key: HDFS-12685
> URL: https://issues.apache.org/jira/browse/HDFS-12685
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ewan Higgs
>Assignee: Virajith Jalaparti
>Priority: Major
> Attachments: HDFS-12685-HDFS-9806.001.patch,
> HDFS-12685-HDFS-9806.002.patch
>
>
> I left a Datanode running overnight and found this in the logs in the morning:
> {code}
> 2017-10-18 23:51:54,391 ERROR datanode.DirectoryScanner: Error compiling
> report for the volume, StorageId: DS-e75ebc3c-6b12-424e-875a-a4ae1a4dcc29
>
>
> java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException:
> URI scheme is not "file"
>
>
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>
>
>
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.getDiskReport(DirectoryScanner.java:544)
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.scan(DirectoryScanner.java:393)
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:375)
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:320)
>
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
>
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>
>
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
>
> at java.lang.Thread.run(Thread.java:748)
>
>
>
> Caused by: java.lang.IllegalArgumentException: URI scheme is not "file"
>
>
>
> at java.io.File.(File.java:421)
[jira] [Updated] (HDFS-12685) [READ] FsVolumeImpl exception when scanning Provided storage volume
[
https://issues.apache.org/jira/browse/HDFS-12685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12685:
--
Status: Patch Available (was: Open)
> [READ] FsVolumeImpl exception when scanning Provided storage volume
> ---
>
> Key: HDFS-12685
> URL: https://issues.apache.org/jira/browse/HDFS-12685
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ewan Higgs
>Assignee: Virajith Jalaparti
>Priority: Major
> Attachments: HDFS-12685-HDFS-9806.001.patch,
> HDFS-12685-HDFS-9806.002.patch
>
>
> I left a Datanode running overnight and found this in the logs in the morning:
> {code}
> 2017-10-18 23:51:54,391 ERROR datanode.DirectoryScanner: Error compiling
> report for the volume, StorageId: DS-e75ebc3c-6b12-424e-875a-a4ae1a4dcc29
>
>
> java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException:
> URI scheme is not "file"
>
>
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>
>
>
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.getDiskReport(DirectoryScanner.java:544)
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.scan(DirectoryScanner.java:393)
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:375)
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:320)
>
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
>
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>
>
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
>
> at java.lang.Thread.run(Thread.java:748)
>
>
>
> Caused by: java.lang.IllegalArgumentException: URI scheme is not "file"
>
>
>
> at java.io.File.(File.java:421)
>
[jira] [Updated] (HDFS-12685) [READ] FsVolumeImpl exception when scanning Provided storage volume
[
https://issues.apache.org/jira/browse/HDFS-12685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12685:
--
Status: Open (was: Patch Available)
> [READ] FsVolumeImpl exception when scanning Provided storage volume
> ---
>
> Key: HDFS-12685
> URL: https://issues.apache.org/jira/browse/HDFS-12685
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ewan Higgs
>Assignee: Virajith Jalaparti
>Priority: Major
> Attachments: HDFS-12685-HDFS-9806.001.patch
>
>
> I left a Datanode running overnight and found this in the logs in the morning:
> {code}
> 2017-10-18 23:51:54,391 ERROR datanode.DirectoryScanner: Error compiling
> report for the volume, StorageId: DS-e75ebc3c-6b12-424e-875a-a4ae1a4dcc29
>
>
> java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException:
> URI scheme is not "file"
>
>
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>
>
>
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.getDiskReport(DirectoryScanner.java:544)
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.scan(DirectoryScanner.java:393)
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:375)
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:320)
>
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
>
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>
>
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
>
> at java.lang.Thread.run(Thread.java:748)
>
>
>
> Caused by: java.lang.IllegalArgumentException: URI scheme is not "file"
>
>
>
> at java.io.File.(File.java:421)
>
>
[jira] [Commented] (HDFS-12607) [READ] Even one dead datanode with PROVIDED storage results in ProvidedStorageInfo being marked as FAILED
[
https://issues.apache.org/jira/browse/HDFS-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16238423#comment-16238423
]
Virajith Jalaparti commented on HDFS-12607:
---
Posting a rebased patch. It also removes a extra flag ({{hasDNs}}) and uses
{{providedDescriptor.activeProvidedDatanodes()}} instead.
> [READ] Even one dead datanode with PROVIDED storage results in
> ProvidedStorageInfo being marked as FAILED
> -
>
> Key: HDFS-12607
> URL: https://issues.apache.org/jira/browse/HDFS-12607
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
>Priority: Major
> Attachments: HDFS-12607-HDFS-9806.001.patch,
> HDFS-12607-HDFS-9806.002.patch, HDFS-12607-HDFS-9806.003.patch,
> HDFS-12607.repro.patch
>
>
> When a DN configured with PROVIDED storage is marked as dead by the NN, the
> state of {{providedStorageInfo}} in {{ProvidedStorageMap}} is set to FAILED,
> and never becomes NORMAL. The state should change to FAILED only if all
> datanodes with PROVIDED storage are dead, and should be restored back to
> NORMAL when a Datanode with NORMAL DatanodeStorage reports in.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12607) [READ] Even one dead datanode with PROVIDED storage results in ProvidedStorageInfo being marked as FAILED
[
https://issues.apache.org/jira/browse/HDFS-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12607:
--
Status: Patch Available (was: Open)
> [READ] Even one dead datanode with PROVIDED storage results in
> ProvidedStorageInfo being marked as FAILED
> -
>
> Key: HDFS-12607
> URL: https://issues.apache.org/jira/browse/HDFS-12607
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
>Priority: Major
> Attachments: HDFS-12607-HDFS-9806.001.patch,
> HDFS-12607-HDFS-9806.002.patch, HDFS-12607-HDFS-9806.003.patch,
> HDFS-12607.repro.patch
>
>
> When a DN configured with PROVIDED storage is marked as dead by the NN, the
> state of {{providedStorageInfo}} in {{ProvidedStorageMap}} is set to FAILED,
> and never becomes NORMAL. The state should change to FAILED only if all
> datanodes with PROVIDED storage are dead, and should be restored back to
> NORMAL when a Datanode with NORMAL DatanodeStorage reports in.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12607) [READ] Even one dead datanode with PROVIDED storage results in ProvidedStorageInfo being marked as FAILED
[
https://issues.apache.org/jira/browse/HDFS-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12607:
--
Status: Open (was: Patch Available)
> [READ] Even one dead datanode with PROVIDED storage results in
> ProvidedStorageInfo being marked as FAILED
> -
>
> Key: HDFS-12607
> URL: https://issues.apache.org/jira/browse/HDFS-12607
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
>Priority: Major
> Attachments: HDFS-12607-HDFS-9806.001.patch,
> HDFS-12607-HDFS-9806.002.patch, HDFS-12607-HDFS-9806.003.patch,
> HDFS-12607.repro.patch
>
>
> When a DN configured with PROVIDED storage is marked as dead by the NN, the
> state of {{providedStorageInfo}} in {{ProvidedStorageMap}} is set to FAILED,
> and never becomes NORMAL. The state should change to FAILED only if all
> datanodes with PROVIDED storage are dead, and should be restored back to
> NORMAL when a Datanode with NORMAL DatanodeStorage reports in.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12607) [READ] Even one dead datanode with PROVIDED storage results in ProvidedStorageInfo being marked as FAILED
[
https://issues.apache.org/jira/browse/HDFS-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12607:
--
Attachment: HDFS-12607-HDFS-9806.003.patch
> [READ] Even one dead datanode with PROVIDED storage results in
> ProvidedStorageInfo being marked as FAILED
> -
>
> Key: HDFS-12607
> URL: https://issues.apache.org/jira/browse/HDFS-12607
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
>Priority: Major
> Attachments: HDFS-12607-HDFS-9806.001.patch,
> HDFS-12607-HDFS-9806.002.patch, HDFS-12607-HDFS-9806.003.patch,
> HDFS-12607.repro.patch
>
>
> When a DN configured with PROVIDED storage is marked as dead by the NN, the
> state of {{providedStorageInfo}} in {{ProvidedStorageMap}} is set to FAILED,
> and never becomes NORMAL. The state should change to FAILED only if all
> datanodes with PROVIDED storage are dead, and should be restored back to
> NORMAL when a Datanode with NORMAL DatanodeStorage reports in.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12685) [READ] FsVolumeImpl exception when scanning Provided storage volume
[
https://issues.apache.org/jira/browse/HDFS-12685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16238345#comment-16238345
]
Virajith Jalaparti commented on HDFS-12685:
---
[~ehiggs], can you check if this patch avoids the exception?
> [READ] FsVolumeImpl exception when scanning Provided storage volume
> ---
>
> Key: HDFS-12685
> URL: https://issues.apache.org/jira/browse/HDFS-12685
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ewan Higgs
>Assignee: Virajith Jalaparti
>Priority: Major
> Attachments: HDFS-12685-HDFS-9806.001.patch
>
>
> I left a Datanode running overnight and found this in the logs in the morning:
> {code}
> 2017-10-18 23:51:54,391 ERROR datanode.DirectoryScanner: Error compiling
> report for the volume, StorageId: DS-e75ebc3c-6b12-424e-875a-a4ae1a4dcc29
>
>
> java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException:
> URI scheme is not "file"
>
>
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>
>
>
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.getDiskReport(DirectoryScanner.java:544)
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.scan(DirectoryScanner.java:393)
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:375)
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:320)
>
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
>
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>
>
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
>
> at java.lang.Thread.run(Thread.java:748)
>
>
>
> Caused by: java.lang.IllegalArgumentException: URI scheme is not "file"
>
>
>
> at java.io.File.(File.java:421)
>
[jira] [Updated] (HDFS-12685) [READ] FsVolumeImpl exception when scanning Provided storage volume
[
https://issues.apache.org/jira/browse/HDFS-12685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12685:
--
Status: Patch Available (was: Open)
> [READ] FsVolumeImpl exception when scanning Provided storage volume
> ---
>
> Key: HDFS-12685
> URL: https://issues.apache.org/jira/browse/HDFS-12685
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ewan Higgs
>Assignee: Virajith Jalaparti
>Priority: Major
> Attachments: HDFS-12685-HDFS-9806.001.patch
>
>
> I left a Datanode running overnight and found this in the logs in the morning:
> {code}
> 2017-10-18 23:51:54,391 ERROR datanode.DirectoryScanner: Error compiling
> report for the volume, StorageId: DS-e75ebc3c-6b12-424e-875a-a4ae1a4dcc29
>
>
> java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException:
> URI scheme is not "file"
>
>
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>
>
>
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.getDiskReport(DirectoryScanner.java:544)
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.scan(DirectoryScanner.java:393)
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:375)
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:320)
>
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
>
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>
>
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
>
> at java.lang.Thread.run(Thread.java:748)
>
>
>
> Caused by: java.lang.IllegalArgumentException: URI scheme is not "file"
>
>
>
> at java.io.File.(File.java:421)
>
>
[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.
[
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-11902:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
> [READ] Merge BlockFormatProvider and FileRegionProvider.
>
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
>Priority: Major
> Attachments: HDFS-11902-HDFS-9806.001.patch,
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch,
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch,
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch,
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch,
> HDFS-11902-HDFS-9806.010.patch, HDFS-11902-HDFS-9806.011.patch,
> HDFS-11902-HDFS-9806.012.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform
> almost the same function on the Namenode and Datanode respectively. This JIRA
> is to merge them into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12685) [READ] FsVolumeImpl exception when scanning Provided storage volume
[
https://issues.apache.org/jira/browse/HDFS-12685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12685:
--
Status: Open (was: Patch Available)
> [READ] FsVolumeImpl exception when scanning Provided storage volume
> ---
>
> Key: HDFS-12685
> URL: https://issues.apache.org/jira/browse/HDFS-12685
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ewan Higgs
>Priority: Major
> Attachments: HDFS-12685-HDFS-9806.001.patch
>
>
> I left a Datanode running overnight and found this in the logs in the morning:
> {code}
> 2017-10-18 23:51:54,391 ERROR datanode.DirectoryScanner: Error compiling
> report for the volume, StorageId: DS-e75ebc3c-6b12-424e-875a-a4ae1a4dcc29
>
>
> java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException:
> URI scheme is not "file"
>
>
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>
>
>
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.getDiskReport(DirectoryScanner.java:544)
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.scan(DirectoryScanner.java:393)
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:375)
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:320)
>
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
>
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>
>
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
>
> at java.lang.Thread.run(Thread.java:748)
>
>
>
> Caused by: java.lang.IllegalArgumentException: URI scheme is not "file"
>
>
>
> at java.io.File.(File.java:421)
>
>
[jira] [Assigned] (HDFS-12685) [READ] FsVolumeImpl exception when scanning Provided storage volume
[
https://issues.apache.org/jira/browse/HDFS-12685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti reassigned HDFS-12685:
-
Assignee: Virajith Jalaparti
> [READ] FsVolumeImpl exception when scanning Provided storage volume
> ---
>
> Key: HDFS-12685
> URL: https://issues.apache.org/jira/browse/HDFS-12685
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ewan Higgs
>Assignee: Virajith Jalaparti
>Priority: Major
> Attachments: HDFS-12685-HDFS-9806.001.patch
>
>
> I left a Datanode running overnight and found this in the logs in the morning:
> {code}
> 2017-10-18 23:51:54,391 ERROR datanode.DirectoryScanner: Error compiling
> report for the volume, StorageId: DS-e75ebc3c-6b12-424e-875a-a4ae1a4dcc29
>
>
> java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException:
> URI scheme is not "file"
>
>
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>
>
>
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.getDiskReport(DirectoryScanner.java:544)
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.scan(DirectoryScanner.java:393)
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:375)
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:320)
>
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
>
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>
>
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
>
> at java.lang.Thread.run(Thread.java:748)
>
>
>
> Caused by: java.lang.IllegalArgumentException: URI scheme is not "file"
>
>
>
> at java.io.File.(File.java:421)
>
>
[jira] [Commented] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.
[
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16238328#comment-16238328
]
Virajith Jalaparti commented on HDFS-11902:
---
The checkstyle issues are from {{DFSConfigKeys}} and the builder pattern.
{{blockaliasmap/package-info.java}} does have a java doc. Committing v012 of
the patch to HDFS-9806 branch.
> [READ] Merge BlockFormatProvider and FileRegionProvider.
>
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
>Priority: Major
> Attachments: HDFS-11902-HDFS-9806.001.patch,
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch,
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch,
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch,
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch,
> HDFS-11902-HDFS-9806.010.patch, HDFS-11902-HDFS-9806.011.patch,
> HDFS-11902-HDFS-9806.012.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform
> almost the same function on the Namenode and Datanode respectively. This JIRA
> is to merge them into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-9240) Use Builder pattern for BlockLocation constructors
[ https://issues.apache.org/jira/browse/HDFS-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16238168#comment-16238168 ] Virajith Jalaparti edited comment on HDFS-9240 at 11/3/17 6:50 PM: --- Thanks for taking a look [~xyao]. Posting new patch (v002) that fixes the NoWhitespaceAfter and "longer than 80 characters" checkstyle issues. The others are from the builder pattern. was (Author: virajith): Thanks for taking a look [~xyao]. Posting new patch (v002) that fixes the NoWhitespaceAfter and "longer than 80 characters" checkstyle issues. The others from the builder pattern. > Use Builder pattern for BlockLocation constructors > -- > > Key: HDFS-9240 > URL: https://issues.apache.org/jira/browse/HDFS-9240 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Minor > Attachments: HDFS-9240.001.patch, HDFS-9240.002.patch > > > This JIRA is opened to refactor the 8 telescoping constructors of > BlockLocation class with Builder pattern. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9240) Use Builder pattern for BlockLocation constructors
[ https://issues.apache.org/jira/browse/HDFS-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-9240: - Status: Patch Available (was: Open) > Use Builder pattern for BlockLocation constructors > -- > > Key: HDFS-9240 > URL: https://issues.apache.org/jira/browse/HDFS-9240 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Minor > Attachments: HDFS-9240.001.patch, HDFS-9240.002.patch > > > This JIRA is opened to refactor the 8 telescoping constructors of > BlockLocation class with Builder pattern. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9240) Use Builder pattern for BlockLocation constructors
[ https://issues.apache.org/jira/browse/HDFS-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16238168#comment-16238168 ] Virajith Jalaparti commented on HDFS-9240: -- Thanks for taking a look [~xyao]. Posting new patch (v002) that fixes the NoWhitespaceAfter and "longer than 80 characters" checkstyle issues. The others from the builder pattern. > Use Builder pattern for BlockLocation constructors > -- > > Key: HDFS-9240 > URL: https://issues.apache.org/jira/browse/HDFS-9240 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Minor > Attachments: HDFS-9240.001.patch, HDFS-9240.002.patch > > > This JIRA is opened to refactor the 8 telescoping constructors of > BlockLocation class with Builder pattern. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9240) Use Builder pattern for BlockLocation constructors
[ https://issues.apache.org/jira/browse/HDFS-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-9240: - Status: Open (was: Patch Available) > Use Builder pattern for BlockLocation constructors > -- > > Key: HDFS-9240 > URL: https://issues.apache.org/jira/browse/HDFS-9240 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Minor > Attachments: HDFS-9240.001.patch, HDFS-9240.002.patch > > > This JIRA is opened to refactor the 8 telescoping constructors of > BlockLocation class with Builder pattern. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9240) Use Builder pattern for BlockLocation constructors
[ https://issues.apache.org/jira/browse/HDFS-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-9240: - Attachment: HDFS-9240.002.patch > Use Builder pattern for BlockLocation constructors > -- > > Key: HDFS-9240 > URL: https://issues.apache.org/jira/browse/HDFS-9240 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Minor > Attachments: HDFS-9240.001.patch, HDFS-9240.002.patch > > > This JIRA is opened to refactor the 8 telescoping constructors of > BlockLocation class with Builder pattern. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.
[
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-11902:
--
Status: Open (was: Patch Available)
> [READ] Merge BlockFormatProvider and FileRegionProvider.
>
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch,
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch,
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch,
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch,
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch,
> HDFS-11902-HDFS-9806.010.patch, HDFS-11902-HDFS-9806.011.patch,
> HDFS-11902-HDFS-9806.012.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform
> almost the same function on the Namenode and Datanode respectively. This JIRA
> is to merge them into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.
[
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-11902:
--
Attachment: HDFS-11902-HDFS-9806.012.patch
> [READ] Merge BlockFormatProvider and FileRegionProvider.
>
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch,
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch,
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch,
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch,
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch,
> HDFS-11902-HDFS-9806.010.patch, HDFS-11902-HDFS-9806.011.patch,
> HDFS-11902-HDFS-9806.012.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform
> almost the same function on the Namenode and Datanode respectively. This JIRA
> is to merge them into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.
[
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-11902:
--
Status: Patch Available (was: Open)
> [READ] Merge BlockFormatProvider and FileRegionProvider.
>
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch,
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch,
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch,
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch,
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch,
> HDFS-11902-HDFS-9806.010.patch, HDFS-11902-HDFS-9806.011.patch,
> HDFS-11902-HDFS-9806.012.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform
> almost the same function on the Namenode and Datanode respectively. This JIRA
> is to merge them into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.
[
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-11902:
--
Comment: was deleted
(was: The failed test cases are unrelated to the patch, so are the ASF
warnings. Will commit patch v011 to HDFS-9806 feature branch soon.)
> [READ] Merge BlockFormatProvider and FileRegionProvider.
>
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch,
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch,
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch,
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch,
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch,
> HDFS-11902-HDFS-9806.010.patch, HDFS-11902-HDFS-9806.011.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform
> almost the same function on the Namenode and Datanode respectively. This JIRA
> is to merge them into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.
[
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16227772#comment-16227772
]
Virajith Jalaparti commented on HDFS-11902:
---
The failed test cases are unrelated to the patch, so are the ASF warnings. Will
commit patch v011 to HDFS-9806 feature branch soon.
> [READ] Merge BlockFormatProvider and FileRegionProvider.
>
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch,
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch,
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch,
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch,
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch,
> HDFS-11902-HDFS-9806.010.patch, HDFS-11902-HDFS-9806.011.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform
> almost the same function on the Namenode and Datanode respectively. This JIRA
> is to merge them into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12607) [READ] Even one dead datanode with PROVIDED storage results in ProvidedStorageInfo being marked as FAILED
[
https://issues.apache.org/jira/browse/HDFS-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12607:
--
Status: Patch Available (was: Open)
> [READ] Even one dead datanode with PROVIDED storage results in
> ProvidedStorageInfo being marked as FAILED
> -
>
> Key: HDFS-12607
> URL: https://issues.apache.org/jira/browse/HDFS-12607
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-12607-HDFS-9806.001.patch,
> HDFS-12607-HDFS-9806.002.patch, HDFS-12607.repro.patch
>
>
> When a DN configured with PROVIDED storage is marked as dead by the NN, the
> state of {{providedStorageInfo}} in {{ProvidedStorageMap}} is set to FAILED,
> and never becomes NORMAL. The state should change to FAILED only if all
> datanodes with PROVIDED storage are dead, and should be restored back to
> NORMAL when a Datanode with NORMAL DatanodeStorage reports in.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12607) [READ] Even one dead datanode with PROVIDED storage results in ProvidedStorageInfo being marked as FAILED
[
https://issues.apache.org/jira/browse/HDFS-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12607:
--
Status: Open (was: Patch Available)
> [READ] Even one dead datanode with PROVIDED storage results in
> ProvidedStorageInfo being marked as FAILED
> -
>
> Key: HDFS-12607
> URL: https://issues.apache.org/jira/browse/HDFS-12607
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-12607-HDFS-9806.001.patch,
> HDFS-12607-HDFS-9806.002.patch, HDFS-12607.repro.patch
>
>
> When a DN configured with PROVIDED storage is marked as dead by the NN, the
> state of {{providedStorageInfo}} in {{ProvidedStorageMap}} is set to FAILED,
> and never becomes NORMAL. The state should change to FAILED only if all
> datanodes with PROVIDED storage are dead, and should be restored back to
> NORMAL when a Datanode with NORMAL DatanodeStorage reports in.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.
[
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-11902:
--
Attachment: HDFS-11902-HDFS-9806.011.patch
Patch v011 fixes a missing java doc. Will commit this once jenkins comes back.
> [READ] Merge BlockFormatProvider and FileRegionProvider.
>
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch,
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch,
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch,
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch,
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch,
> HDFS-11902-HDFS-9806.010.patch, HDFS-11902-HDFS-9806.011.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform
> almost the same function on the Namenode and Datanode respectively. This JIRA
> is to merge them into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.
[
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-11902:
--
Status: Patch Available (was: Open)
> [READ] Merge BlockFormatProvider and FileRegionProvider.
>
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch,
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch,
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch,
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch,
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch,
> HDFS-11902-HDFS-9806.010.patch, HDFS-11902-HDFS-9806.011.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform
> almost the same function on the Namenode and Datanode respectively. This JIRA
> is to merge them into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.
[
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16227303#comment-16227303
]
Virajith Jalaparti edited comment on HDFS-11902 at 10/31/17 6:42 PM:
-
Patch v011 fixes a missing java doc. Will commit this once jenkins comes back.
Thanks for reviewing this [~ehiggs].
was (Author: virajith):
Patch v011 fixes a missing java doc. Will commit this once jenkins comes back.
> [READ] Merge BlockFormatProvider and FileRegionProvider.
>
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch,
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch,
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch,
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch,
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch,
> HDFS-11902-HDFS-9806.010.patch, HDFS-11902-HDFS-9806.011.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform
> almost the same function on the Namenode and Datanode respectively. This JIRA
> is to merge them into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.
[
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-11902:
--
Status: Open (was: Patch Available)
> [READ] Merge BlockFormatProvider and FileRegionProvider.
>
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch,
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch,
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch,
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch,
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch,
> HDFS-11902-HDFS-9806.010.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform
> almost the same function on the Namenode and Datanode respectively. This JIRA
> is to merge them into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9240) Use Builder pattern for BlockLocation constructors
[ https://issues.apache.org/jira/browse/HDFS-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-9240: - Status: Patch Available (was: Open) > Use Builder pattern for BlockLocation constructors > -- > > Key: HDFS-9240 > URL: https://issues.apache.org/jira/browse/HDFS-9240 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Minor > Attachments: HDFS-9240.001.patch > > > This JIRA is opened to refactor the 8 telescoping constructors of > BlockLocation class with Builder pattern. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9240) Use Builder pattern for BlockLocation constructors
[ https://issues.apache.org/jira/browse/HDFS-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-9240: - Status: Open (was: Patch Available) > Use Builder pattern for BlockLocation constructors > -- > > Key: HDFS-9240 > URL: https://issues.apache.org/jira/browse/HDFS-9240 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Minor > Attachments: HDFS-9240.001.patch > > > This JIRA is opened to refactor the 8 telescoping constructors of > BlockLocation class with Builder pattern. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.
[
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-11902:
--
Status: Patch Available (was: Open)
> [READ] Merge BlockFormatProvider and FileRegionProvider.
>
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch,
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch,
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch,
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch,
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch,
> HDFS-11902-HDFS-9806.010.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform
> almost the same function on the Namenode and Datanode respectively. This JIRA
> is to merge them into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.
[
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-11902:
--
Attachment: (was: HDFS-11902-HDFS-9806.010.patch)
> [READ] Merge BlockFormatProvider and FileRegionProvider.
>
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch,
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch,
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch,
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch,
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch,
> HDFS-11902-HDFS-9806.010.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform
> almost the same function on the Namenode and Datanode respectively. This JIRA
> is to merge them into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.
[
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-11902:
--
Status: Open (was: Patch Available)
> [READ] Merge BlockFormatProvider and FileRegionProvider.
>
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch,
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch,
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch,
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch,
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch,
> HDFS-11902-HDFS-9806.010.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform
> almost the same function on the Namenode and Datanode respectively. This JIRA
> is to merge them into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.
[
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-11902:
--
Attachment: HDFS-11902-HDFS-9806.010.patch
> [READ] Merge BlockFormatProvider and FileRegionProvider.
>
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch,
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch,
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch,
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch,
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch,
> HDFS-11902-HDFS-9806.010.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform
> almost the same function on the Namenode and Datanode respectively. This JIRA
> is to merge them into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.
[
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-11902:
--
Status: Open (was: Patch Available)
> [READ] Merge BlockFormatProvider and FileRegionProvider.
>
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch,
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch,
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch,
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch,
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch,
> HDFS-11902-HDFS-9806.010.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform
> almost the same function on the Namenode and Datanode respectively. This JIRA
> is to merge them into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.
[
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-11902:
--
Status: Patch Available (was: Open)
> [READ] Merge BlockFormatProvider and FileRegionProvider.
>
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch,
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch,
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch,
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch,
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch,
> HDFS-11902-HDFS-9806.010.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform
> almost the same function on the Namenode and Datanode respectively. This JIRA
> is to merge them into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12607) [READ] Even one dead datanode with PROVIDED storage results in ProvidedStorageInfo being marked as FAILED
[
https://issues.apache.org/jira/browse/HDFS-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12607:
--
Status: Patch Available (was: Open)
> [READ] Even one dead datanode with PROVIDED storage results in
> ProvidedStorageInfo being marked as FAILED
> -
>
> Key: HDFS-12607
> URL: https://issues.apache.org/jira/browse/HDFS-12607
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-12607-HDFS-9806.001.patch,
> HDFS-12607-HDFS-9806.002.patch, HDFS-12607.repro.patch
>
>
> When a DN configured with PROVIDED storage is marked as dead by the NN, the
> state of {{providedStorageInfo}} in {{ProvidedStorageMap}} is set to FAILED,
> and never becomes NORMAL. The state should change to FAILED only if all
> datanodes with PROVIDED storage are dead, and should be restored back to
> NORMAL when a Datanode with NORMAL DatanodeStorage reports in.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12607) [READ] Even one dead datanode with PROVIDED storage results in ProvidedStorageInfo being marked as FAILED
[
https://issues.apache.org/jira/browse/HDFS-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12607:
--
Status: Open (was: Patch Available)
> [READ] Even one dead datanode with PROVIDED storage results in
> ProvidedStorageInfo being marked as FAILED
> -
>
> Key: HDFS-12607
> URL: https://issues.apache.org/jira/browse/HDFS-12607
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-12607-HDFS-9806.001.patch,
> HDFS-12607-HDFS-9806.002.patch, HDFS-12607.repro.patch
>
>
> When a DN configured with PROVIDED storage is marked as dead by the NN, the
> state of {{providedStorageInfo}} in {{ProvidedStorageMap}} is set to FAILED,
> and never becomes NORMAL. The state should change to FAILED only if all
> datanodes with PROVIDED storage are dead, and should be restored back to
> NORMAL when a Datanode with NORMAL DatanodeStorage reports in.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.
[
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-11902:
--
Status: Patch Available (was: Open)
> [READ] Merge BlockFormatProvider and FileRegionProvider.
>
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch,
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch,
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch,
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch,
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch,
> HDFS-11902-HDFS-9806.010.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform
> almost the same function on the Namenode and Datanode respectively. This JIRA
> is to merge them into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.
[
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-11902:
--
Attachment: HDFS-11902-HDFS-9806.010.patch
> [READ] Merge BlockFormatProvider and FileRegionProvider.
>
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch,
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch,
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch,
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch,
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch,
> HDFS-11902-HDFS-9806.010.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform
> almost the same function on the Namenode and Datanode respectively. This JIRA
> is to merge them into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.
[
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-11902:
--
Attachment: (was: HDFS-11902-HDFS-9806.010.patch)
> [READ] Merge BlockFormatProvider and FileRegionProvider.
>
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch,
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch,
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch,
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch,
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch,
> HDFS-11902-HDFS-9806.010.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform
> almost the same function on the Namenode and Datanode respectively. This JIRA
> is to merge them into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.
[
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-11902:
--
Status: Open (was: Patch Available)
> [READ] Merge BlockFormatProvider and FileRegionProvider.
>
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch,
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch,
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch,
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch,
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch,
> HDFS-11902-HDFS-9806.010.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform
> almost the same function on the Namenode and Datanode respectively. This JIRA
> is to merge them into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9240) Use Builder pattern for BlockLocation constructors
[
https://issues.apache.org/jira/browse/HDFS-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-9240:
-
Attachment: HDFS-9240.001.patch
>From this description, I don't think this JIRA is related to HDFS-9808 or
>HDFS-11190. HDFS-11190 adds a builder for {{LocatedBlocks}} but this JIRA
>requires a Builder for {{BlockLocation}}.
Attaching a patch for what I think this JIRA was referring to. [~xyao], can you
take a look at it?
> Use Builder pattern for BlockLocation constructors
> --
>
> Key: HDFS-9240
> URL: https://issues.apache.org/jira/browse/HDFS-9240
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
>Priority: Minor
> Attachments: HDFS-9240.001.patch
>
>
> This JIRA is opened to refactor the 8 telescoping constructors of
> BlockLocation class with Builder pattern.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9240) Use Builder pattern for BlockLocation constructors
[ https://issues.apache.org/jira/browse/HDFS-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-9240: - Status: Patch Available (was: Open) > Use Builder pattern for BlockLocation constructors > -- > > Key: HDFS-9240 > URL: https://issues.apache.org/jira/browse/HDFS-9240 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Minor > Attachments: HDFS-9240.001.patch > > > This JIRA is opened to refactor the 8 telescoping constructors of > BlockLocation class with Builder pattern. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12665) [AliasMap] Create a version of the AliasMap that runs in memory in the Namenode (leveldb)
[ https://issues.apache.org/jira/browse/HDFS-12665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16219626#comment-16219626 ] Virajith Jalaparti commented on HDFS-12665: --- [~ehiggs], can we move the part on refactoring FileRegion/BlockAliasMap to a separate JIRA? I filed HDFS-12713 for this. > [AliasMap] Create a version of the AliasMap that runs in memory in the > Namenode (leveldb) > - > > Key: HDFS-12665 > URL: https://issues.apache.org/jira/browse/HDFS-12665 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ewan Higgs >Assignee: Ewan Higgs > Attachments: HDFS-12665-HDFS-9806.001.patch, > HDFS-12665-HDFS-9806.002.patch > > > The design of Provided Storage requires the use of an AliasMap to manage the > mapping between blocks of files on the local HDFS and ranges of files on a > remote storage system. To reduce load from the Namenode, this can be done > using a pluggable external service (e.g. AzureTable, Cassandra, Ratis). > However, to aide adoption and ease of deployment, we propose an in memory > version. > This AliasMap will be a wrapper around LevelDB (already a dependency from the > Timeline Service) and use protobuf for the key (blockpool, blockid, and > genstamp) and the value (url, offset, length, nonce). The in memory service > will also have a configurable port on which it will listen for updates from > Storage Policy Satisfier (SPS) Coordinating Datanodes (C-DN). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12713) [READ] Refactor FileRegion and BlockAliasMap to separate out HDFS metadata and PROVIDED storage metadata
Virajith Jalaparti created HDFS-12713: - Summary: [READ] Refactor FileRegion and BlockAliasMap to separate out HDFS metadata and PROVIDED storage metadata Key: HDFS-12713 URL: https://issues.apache.org/jira/browse/HDFS-12713 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Virajith Jalaparti -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12685) [READ] FsVolumeImpl exception when scanning Provided storage volume
[
https://issues.apache.org/jira/browse/HDFS-12685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12685:
--
Status: Patch Available (was: Open)
> [READ] FsVolumeImpl exception when scanning Provided storage volume
> ---
>
> Key: HDFS-12685
> URL: https://issues.apache.org/jira/browse/HDFS-12685
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ewan Higgs
> Attachments: HDFS-12685-HDFS-9806.001.patch
>
>
> I left a Datanode running overnight and found this in the logs in the morning:
> {code}
> 2017-10-18 23:51:54,391 ERROR datanode.DirectoryScanner: Error compiling
> report for the volume, StorageId: DS-e75ebc3c-6b12-424e-875a-a4ae1a4dcc29
>
>
> java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException:
> URI scheme is not "file"
>
>
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>
>
>
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.getDiskReport(DirectoryScanner.java:544)
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.scan(DirectoryScanner.java:393)
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:375)
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:320)
>
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
>
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>
>
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
>
> at java.lang.Thread.run(Thread.java:748)
>
>
>
> Caused by: java.lang.IllegalArgumentException: URI scheme is not "file"
>
>
>
> at java.io.File.(File.java:421)
>
>
>
>
[jira] [Updated] (HDFS-12685) [READ] FsVolumeImpl exception when scanning Provided storage volume
[
https://issues.apache.org/jira/browse/HDFS-12685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12685:
--
Attachment: HDFS-12685-HDFS-9806.001.patch
Posting a patch that fixes the construction of ScanInfo objects. This should
fix the issue.
> [READ] FsVolumeImpl exception when scanning Provided storage volume
> ---
>
> Key: HDFS-12685
> URL: https://issues.apache.org/jira/browse/HDFS-12685
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ewan Higgs
> Attachments: HDFS-12685-HDFS-9806.001.patch
>
>
> I left a Datanode running overnight and found this in the logs in the morning:
> {code}
> 2017-10-18 23:51:54,391 ERROR datanode.DirectoryScanner: Error compiling
> report for the volume, StorageId: DS-e75ebc3c-6b12-424e-875a-a4ae1a4dcc29
>
>
> java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException:
> URI scheme is not "file"
>
>
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>
>
>
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.getDiskReport(DirectoryScanner.java:544)
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.scan(DirectoryScanner.java:393)
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:375)
>
>
> at
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:320)
>
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
>
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>
>
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
>
> at java.lang.Thread.run(Thread.java:748)
>
>
>
> Caused by: java.lang.IllegalArgumentException: URI scheme is not "file"
>
>
>
> at java.io.File.(File.java:421)
>
[jira] [Updated] (HDFS-12607) [READ] Even one dead datanode with PROVIDED storage results in ProvidedStorageInfo being marked as FAILED
[
https://issues.apache.org/jira/browse/HDFS-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12607:
--
Attachment: HDFS-12607-HDFS-9806.002.patch
Posting a new patch based on [~goiri]'s comments.
Thanks for taking a look [~goiri].
> [READ] Even one dead datanode with PROVIDED storage results in
> ProvidedStorageInfo being marked as FAILED
> -
>
> Key: HDFS-12607
> URL: https://issues.apache.org/jira/browse/HDFS-12607
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-12607-HDFS-9806.001.patch,
> HDFS-12607-HDFS-9806.002.patch, HDFS-12607.repro.patch
>
>
> When a DN configured with PROVIDED storage is marked as dead by the NN, the
> state of {{providedStorageInfo}} in {{ProvidedStorageMap}} is set to FAILED,
> and never becomes NORMAL. The state should change to FAILED only if all
> datanodes with PROVIDED storage are dead, and should be restored back to
> NORMAL when a Datanode with NORMAL DatanodeStorage reports in.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12607) [READ] Even one dead datanode with PROVIDED storage results in ProvidedStorageInfo being marked as FAILED
[
https://issues.apache.org/jira/browse/HDFS-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12607:
--
Status: Patch Available (was: Open)
> [READ] Even one dead datanode with PROVIDED storage results in
> ProvidedStorageInfo being marked as FAILED
> -
>
> Key: HDFS-12607
> URL: https://issues.apache.org/jira/browse/HDFS-12607
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-12607-HDFS-9806.001.patch,
> HDFS-12607-HDFS-9806.002.patch, HDFS-12607.repro.patch
>
>
> When a DN configured with PROVIDED storage is marked as dead by the NN, the
> state of {{providedStorageInfo}} in {{ProvidedStorageMap}} is set to FAILED,
> and never becomes NORMAL. The state should change to FAILED only if all
> datanodes with PROVIDED storage are dead, and should be restored back to
> NORMAL when a Datanode with NORMAL DatanodeStorage reports in.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.
[
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-11902:
--
Status: Open (was: Patch Available)
> [READ] Merge BlockFormatProvider and FileRegionProvider.
>
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch,
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch,
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch,
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch,
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch,
> HDFS-11902-HDFS-9806.010.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform
> almost the same function on the Namenode and Datanode respectively. This JIRA
> is to merge them into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12607) [READ] Even one dead datanode with PROVIDED storage results in ProvidedStorageInfo being marked as FAILED
[
https://issues.apache.org/jira/browse/HDFS-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12607:
--
Status: Open (was: Patch Available)
> [READ] Even one dead datanode with PROVIDED storage results in
> ProvidedStorageInfo being marked as FAILED
> -
>
> Key: HDFS-12607
> URL: https://issues.apache.org/jira/browse/HDFS-12607
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-12607-HDFS-9806.001.patch, HDFS-12607.repro.patch
>
>
> When a DN configured with PROVIDED storage is marked as dead by the NN, the
> state of {{providedStorageInfo}} in {{ProvidedStorageMap}} is set to FAILED,
> and never becomes NORMAL. The state should change to FAILED only if all
> datanodes with PROVIDED storage are dead, and should be restored back to
> NORMAL when a Datanode with NORMAL DatanodeStorage reports in.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11902) [READ] Merge BlockFormatProvider and FileRegionProvider.
[
https://issues.apache.org/jira/browse/HDFS-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-11902:
--
Status: Patch Available (was: Open)
> [READ] Merge BlockFormatProvider and FileRegionProvider.
>
>
> Key: HDFS-11902
> URL: https://issues.apache.org/jira/browse/HDFS-11902
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-11902-HDFS-9806.001.patch,
> HDFS-11902-HDFS-9806.002.patch, HDFS-11902-HDFS-9806.003.patch,
> HDFS-11902-HDFS-9806.004.patch, HDFS-11902-HDFS-9806.005.patch,
> HDFS-11902-HDFS-9806.006.patch, HDFS-11902-HDFS-9806.007.patch,
> HDFS-11902-HDFS-9806.008.patch, HDFS-11902-HDFS-9806.009.patch,
> HDFS-11902-HDFS-9806.010.patch
>
>
> Currently {{BlockFormatProvider}} and {{TextFileRegionProvider}} perform
> almost the same function on the Namenode and Datanode respectively. This JIRA
> is to merge them into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
