[jira] [Commented] (HDFS-10702) Add a Client API and Proxy Provider to enable stale read from Standby
[ https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15429014#comment-15429014 ] Jiayi Zhou commented on HDFS-10702: --- Test failed due to an improper default value for StaleBound. Upload a new patch to fill the problem. > Add a Client API and Proxy Provider to enable stale read from Standby > - > > Key: HDFS-10702 > URL: https://issues.apache.org/jira/browse/HDFS-10702 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10702.001.patch, HDFS-10702.002.patch, > HDFS-10702.003.patch, HDFS-10702.004.patch, HDFS-10702.005.patch, > StaleReadfromStandbyNN.pdf > > > Currently, clients must always talk to the active NameNode when performing > any metadata operation, which means active NameNode could be a bottleneck for > scalability. One way to solve this problem is to send read-only operations to > Standby NameNode. The disadvantage is that it might be a stale read. > Here, I'm thinking of adding a Client API to enable/disable stale read from > Standby which gives Client the power to set the staleness restriction. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10702) Add a Client API and Proxy Provider to enable stale read from Standby
[ https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10702: -- Attachment: HDFS-10702.005.patch > Add a Client API and Proxy Provider to enable stale read from Standby > - > > Key: HDFS-10702 > URL: https://issues.apache.org/jira/browse/HDFS-10702 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10702.001.patch, HDFS-10702.002.patch, > HDFS-10702.003.patch, HDFS-10702.004.patch, HDFS-10702.005.patch, > StaleReadfromStandbyNN.pdf > > > Currently, clients must always talk to the active NameNode when performing > any metadata operation, which means active NameNode could be a bottleneck for > scalability. One way to solve this problem is to send read-only operations to > Standby NameNode. The disadvantage is that it might be a stale read. > Here, I'm thinking of adding a Client API to enable/disable stale read from > Standby which gives Client the power to set the staleness restriction. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-10702) Add a Client API and Proxy Provider to enable stale read from Standby
[ https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428772#comment-15428772 ] Jiayi Zhou edited comment on HDFS-10702 at 8/19/16 8:35 PM: Upload a patch to fix some of the issues commented by Andrew. I haven't worked on the RpcHeader modification yet, want to see if the new patch runs properly first. * Fix nitty issues. * Remove MethodCategory and use annotations instead. Add a new annotation ReadOnly. * Idea of SyncInfo is also to add more fields in the future. NamenodeProtocol#getTransactionID is commented as NameNodeProtocol rather that ClientProtocol, so I add getSyncInfo() for ClientProtocol. Remove state from SyncInfo because we no longer need it. was (Author: clouderajiayi): Upload a path to fix some of the issues commented by Andrew. I haven't worked on the RpcHeader modification yet, want to see if the new patch runs properly first. * Fix nitty issues. * Remove MethodCategory and use annotations instead. Add a new annotation ReadOnly. * Idea of SyncInfo is also to add more fields in the future. NamenodeProtocol#getTransactionID is commented as NameNodeProtocol rather that ClientProtocol, so I add getSyncInfo() for ClientProtocol. Remove state from SyncInfo because we no longer need it. > Add a Client API and Proxy Provider to enable stale read from Standby > - > > Key: HDFS-10702 > URL: https://issues.apache.org/jira/browse/HDFS-10702 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10702.001.patch, HDFS-10702.002.patch, > HDFS-10702.003.patch, HDFS-10702.004.patch, StaleReadfromStandbyNN.pdf > > > Currently, clients must always talk to the active NameNode when performing > any metadata operation, which means active NameNode could be a bottleneck for > scalability. One way to solve this problem is to send read-only operations to > Standby NameNode. The disadvantage is that it might be a stale read. > Here, I'm thinking of adding a Client API to enable/disable stale read from > Standby which gives Client the power to set the staleness restriction. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10702) Add a Client API and Proxy Provider to enable stale read from Standby
[ https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428772#comment-15428772 ] Jiayi Zhou commented on HDFS-10702: --- Upload a path to fix some of the issues commented by Andrew. I haven't worked on the RpcHeader modification yet, want to see if the new patch runs properly first. * Fix nitty issues. * Remove MethodCategory and use annotations instead. Add a new annotation ReadOnly. * Idea of SyncInfo is also to add more fields in the future. NamenodeProtocol#getTransactionID is commented as NameNodeProtocol rather that ClientProtocol, so I add getSyncInfo() for ClientProtocol. Remove state from SyncInfo because we no longer need it. > Add a Client API and Proxy Provider to enable stale read from Standby > - > > Key: HDFS-10702 > URL: https://issues.apache.org/jira/browse/HDFS-10702 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10702.001.patch, HDFS-10702.002.patch, > HDFS-10702.003.patch, HDFS-10702.004.patch, StaleReadfromStandbyNN.pdf > > > Currently, clients must always talk to the active NameNode when performing > any metadata operation, which means active NameNode could be a bottleneck for > scalability. One way to solve this problem is to send read-only operations to > Standby NameNode. The disadvantage is that it might be a stale read. > Here, I'm thinking of adding a Client API to enable/disable stale read from > Standby which gives Client the power to set the staleness restriction. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10702) Add a Client API and Proxy Provider to enable stale read from Standby
[ https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10702: -- Attachment: HDFS-10702.004.patch > Add a Client API and Proxy Provider to enable stale read from Standby > - > > Key: HDFS-10702 > URL: https://issues.apache.org/jira/browse/HDFS-10702 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10702.001.patch, HDFS-10702.002.patch, > HDFS-10702.003.patch, HDFS-10702.004.patch, StaleReadfromStandbyNN.pdf > > > Currently, clients must always talk to the active NameNode when performing > any metadata operation, which means active NameNode could be a bottleneck for > scalability. One way to solve this problem is to send read-only operations to > Standby NameNode. The disadvantage is that it might be a stale read. > Here, I'm thinking of adding a Client API to enable/disable stale read from > Standby which gives Client the power to set the staleness restriction. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10702) Add a Client API and Proxy Provider to enable stale read from Standby
[ https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10702: -- Attachment: HDFS-10702.003.patch Fix checkstyle. Some of the style problems are intentionally preserved. > Add a Client API and Proxy Provider to enable stale read from Standby > - > > Key: HDFS-10702 > URL: https://issues.apache.org/jira/browse/HDFS-10702 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10702.001.patch, HDFS-10702.002.patch, > HDFS-10702.003.patch, StaleReadfromStandbyNN.pdf > > > Currently, clients must always talk to the active NameNode when performing > any metadata operation, which means active NameNode could be a bottleneck for > scalability. One way to solve this problem is to send read-only operations to > Standby NameNode. The disadvantage is that it might be a stale read. > Here, I'm thinking of adding a Client API to enable/disable stale read from > Standby which gives Client the power to set the staleness restriction. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10702) Add a Client API and Proxy Provider to enable stale read from Standby
[ https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10702: -- Attachment: HDFS-10702.002.patch Update a new patch which could apply to trunk. > Add a Client API and Proxy Provider to enable stale read from Standby > - > > Key: HDFS-10702 > URL: https://issues.apache.org/jira/browse/HDFS-10702 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10702.001.patch, HDFS-10702.002.patch, > StaleReadfromStandbyNN.pdf > > > Currently, clients must always talk to the active NameNode when performing > any metadata operation, which means active NameNode could be a bottleneck for > scalability. One way to solve this problem is to send read-only operations to > Standby NameNode. The disadvantage is that it might be a stale read. > Here, I'm thinking of adding a Client API to enable/disable stale read from > Standby which gives Client the power to set the staleness restriction. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10702) Add a Client API and Proxy Provider to enable stale read from Standby
[ https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10702: -- Attachment: (was: HDFS-10702.002.patch) > Add a Client API and Proxy Provider to enable stale read from Standby > - > > Key: HDFS-10702 > URL: https://issues.apache.org/jira/browse/HDFS-10702 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10702.001.patch, StaleReadfromStandbyNN.pdf > > > Currently, clients must always talk to the active NameNode when performing > any metadata operation, which means active NameNode could be a bottleneck for > scalability. One way to solve this problem is to send read-only operations to > Standby NameNode. The disadvantage is that it might be a stale read. > Here, I'm thinking of adding a Client API to enable/disable stale read from > Standby which gives Client the power to set the staleness restriction. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10702) Add a Client API and Proxy Provider to enable stale read from Standby
[ https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10702: -- Attachment: HDFS-10702.002.patch > Add a Client API and Proxy Provider to enable stale read from Standby > - > > Key: HDFS-10702 > URL: https://issues.apache.org/jira/browse/HDFS-10702 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10702.001.patch, HDFS-10702.002.patch, > StaleReadfromStandbyNN.pdf > > > Currently, clients must always talk to the active NameNode when performing > any metadata operation, which means active NameNode could be a bottleneck for > scalability. One way to solve this problem is to send read-only operations to > Standby NameNode. The disadvantage is that it might be a stale read. > Here, I'm thinking of adding a Client API to enable/disable stale read from > Standby which gives Client the power to set the staleness restriction. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10702) Add a Client API and Proxy Provider to enable stale read from Standby
[ https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10702: -- Attachment: HDFS-10702.001.patch > Add a Client API and Proxy Provider to enable stale read from Standby > - > > Key: HDFS-10702 > URL: https://issues.apache.org/jira/browse/HDFS-10702 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10702.001.patch, StaleReadfromStandbyNN.pdf > > > Currently, clients must always talk to the active NameNode when performing > any metadata operation, which means active NameNode could be a bottleneck for > scalability. One way to solve this problem is to send read-only operations to > Standby NameNode. The disadvantage is that it might be a stale read. > Here, I'm thinking of adding a Client API to enable/disable stale read from > Standby which gives Client the power to set the staleness restriction. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10702) Add a Client API and Proxy Provider to enable stale read from Standby
[ https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10702: -- Attachment: (was: HDFS-10702.001.patch) > Add a Client API and Proxy Provider to enable stale read from Standby > - > > Key: HDFS-10702 > URL: https://issues.apache.org/jira/browse/HDFS-10702 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10702.001.patch, StaleReadfromStandbyNN.pdf > > > Currently, clients must always talk to the active NameNode when performing > any metadata operation, which means active NameNode could be a bottleneck for > scalability. One way to solve this problem is to send read-only operations to > Standby NameNode. The disadvantage is that it might be a stale read. > Here, I'm thinking of adding a Client API to enable/disable stale read from > Standby which gives Client the power to set the staleness restriction. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10702) Add a Client API and Proxy Provider to enable stale read from Standby
[ https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10702: -- Attachment: StaleReadfromStandbyNN.pdf > Add a Client API and Proxy Provider to enable stale read from Standby > - > > Key: HDFS-10702 > URL: https://issues.apache.org/jira/browse/HDFS-10702 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10702.001.patch, StaleReadfromStandbyNN.pdf > > > Currently, clients must always talk to the active NameNode when performing > any metadata operation, which means active NameNode could be a bottleneck for > scalability. One way to solve this problem is to send read-only operations to > Standby NameNode. The disadvantage is that it might be a stale read. > Here, I'm thinking of adding a Client API to enable/disable stale read from > Standby which gives Client the power to set the staleness restriction. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10702) Add a Client API and Proxy Provider to enable stale read from Standby
[ https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10702: -- Attachment: HDFS-10702.001.patch > Add a Client API and Proxy Provider to enable stale read from Standby > - > > Key: HDFS-10702 > URL: https://issues.apache.org/jira/browse/HDFS-10702 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10702.001.patch > > > Currently, clients must always talk to the active NameNode when performing > any metadata operation, which means active NameNode could be a bottleneck for > scalability. One way to solve this problem is to send read-only operations to > Standby NameNode. The disadvantage is that it might be a stale read. > Here, I'm thinking of adding a Client API to enable/disable stale read from > Standby which gives Client the power to set the staleness restriction. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10702) Add a Client API and Proxy Provider to enable stale read from Standby
[ https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10702: -- Attachment: (was: StaleReadfromStandbyNN.pdf) > Add a Client API and Proxy Provider to enable stale read from Standby > - > > Key: HDFS-10702 > URL: https://issues.apache.org/jira/browse/HDFS-10702 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > > Currently, clients must always talk to the active NameNode when performing > any metadata operation, which means active NameNode could be a bottleneck for > scalability. One way to solve this problem is to send read-only operations to > Standby NameNode. The disadvantage is that it might be a stale read. > Here, I'm thinking of adding a Client API to enable/disable stale read from > Standby which gives Client the power to set the staleness restriction. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10702) Add a Client API and Proxy Provider to enable stale read from Standby
[ https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10702: -- Attachment: (was: HDFS-10702.001.patch) > Add a Client API and Proxy Provider to enable stale read from Standby > - > > Key: HDFS-10702 > URL: https://issues.apache.org/jira/browse/HDFS-10702 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > > Currently, clients must always talk to the active NameNode when performing > any metadata operation, which means active NameNode could be a bottleneck for > scalability. One way to solve this problem is to send read-only operations to > Standby NameNode. The disadvantage is that it might be a stale read. > Here, I'm thinking of adding a Client API to enable/disable stale read from > Standby which gives Client the power to set the staleness restriction. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10702) Add a Client API and Proxy Provider to enable stale read from Standby
[ https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10702: -- Status: Patch Available (was: Open) > Add a Client API and Proxy Provider to enable stale read from Standby > - > > Key: HDFS-10702 > URL: https://issues.apache.org/jira/browse/HDFS-10702 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10702.001.patch, StaleReadfromStandbyNN.pdf > > > Currently, clients must always talk to the active NameNode when performing > any metadata operation, which means active NameNode could be a bottleneck for > scalability. One way to solve this problem is to send read-only operations to > Standby NameNode. The disadvantage is that it might be a stale read. > Here, I'm thinking of adding a Client API to enable/disable stale read from > Standby which gives Client the power to set the staleness restriction. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDFS-10702) Add a Client API and Proxy Provider to enable stale read from Standby
[ https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10702: -- Comment: was deleted (was: This JIRA gives us a more up-to-date data on Standby NN.) > Add a Client API and Proxy Provider to enable stale read from Standby > - > > Key: HDFS-10702 > URL: https://issues.apache.org/jira/browse/HDFS-10702 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10702.001.patch, StaleReadfromStandbyNN.pdf > > > Currently, clients must always talk to the active NameNode when performing > any metadata operation, which means active NameNode could be a bottleneck for > scalability. One way to solve this problem is to send read-only operations to > Standby NameNode. The disadvantage is that it might be a stale read. > Here, I'm thinking of adding a Client API to enable/disable stale read from > Standby which gives Client the power to set the staleness restriction. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10702) Add a Client API and Proxy Provider to enable stale read from Standby
[ https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10702: -- Attachment: StaleReadfromStandbyNN.pdf HDFS-10702.001.patch Upload the first patch and a design scope. > Add a Client API and Proxy Provider to enable stale read from Standby > - > > Key: HDFS-10702 > URL: https://issues.apache.org/jira/browse/HDFS-10702 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10702.001.patch, StaleReadfromStandbyNN.pdf > > > Currently, clients must always talk to the active NameNode when performing > any metadata operation, which means active NameNode could be a bottleneck for > scalability. One way to solve this problem is to send read-only operations to > Standby NameNode. The disadvantage is that it might be a stale read. > Here, I'm thinking of adding a Client API to enable/disable stale read from > Standby which gives Client the power to set the staleness restriction. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10702) Add a Client API and Proxy Provider to enable stale read from Standby
[ https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398345#comment-15398345 ] Jiayi Zhou commented on HDFS-10702: --- This JIRA gives us a more up-to-date data on Standby NN. > Add a Client API and Proxy Provider to enable stale read from Standby > - > > Key: HDFS-10702 > URL: https://issues.apache.org/jira/browse/HDFS-10702 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > > Currently, clients must always talk to the active NameNode when performing > any metadata operation, which means active NameNode could be a bottleneck for > scalability. One way to solve this problem is to send read-only operations to > Standby NameNode. The disadvantage is that it might be a stale read. > Here, I'm thinking of adding a Client API to enable/disable stale read from > Standby which gives Client the power to set the staleness restriction. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10702) Add a Client API and Proxy Provider to enable stale read from Standby
Jiayi Zhou created HDFS-10702: - Summary: Add a Client API and Proxy Provider to enable stale read from Standby Key: HDFS-10702 URL: https://issues.apache.org/jira/browse/HDFS-10702 Project: Hadoop HDFS Issue Type: Improvement Reporter: Jiayi Zhou Assignee: Jiayi Zhou Priority: Minor Currently, clients must always talk to the active NameNode when performing any metadata operation, which means active NameNode could be a bottleneck for scalability. One way to solve this problem is to send read-only operations to Standby NameNode. The disadvantage is that it might be a stale read. Here, I'm thinking of adding a Client API to enable/disable stale read from Standby which gives Client the power to set the staleness restriction. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10519) Add a configuration option to enable in-progress edit log tailing
[ https://issues.apache.org/jira/browse/HDFS-10519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10519: -- Attachment: HDFS-10519.013.patch Undo some whitespace changes and update test strings instead of adding a new method. > Add a configuration option to enable in-progress edit log tailing > - > > Key: HDFS-10519 > URL: https://issues.apache.org/jira/browse/HDFS-10519 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10519.001.patch, HDFS-10519.002.patch, > HDFS-10519.003.patch, HDFS-10519.004.patch, HDFS-10519.005.patch, > HDFS-10519.006.patch, HDFS-10519.007.patch, HDFS-10519.008.patch, > HDFS-10519.009.patch, HDFS-10519.010.patch, HDFS-10519.011.patch, > HDFS-10519.012.patch, HDFS-10519.013.patch > > > Standby Namenode has the option to do in-progress edit log tailing to improve > the data freshness. In-progress tailing is already implemented, but it's not > enabled as default configuration. And there's no related configuration key to > turn it on. > Adding a related configuration key to let Standby Namenode is reasonable and > would be a basis for further improvement on Standby Namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-10519) Add a configuration option to enable in-progress edit log tailing
[ https://issues.apache.org/jira/browse/HDFS-10519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396027#comment-15396027 ] Jiayi Zhou edited comment on HDFS-10519 at 7/27/16 5:33 PM: Failed test is unrelated. It is filed in another jira. [HDFS-10696|https://issues.apache.org/jira/browse/HDFS-10696?jql=project%20%3D%20HDFS] [~andrew.wang] Can you take a look at the patch and see if we could commit it now? was (Author: clouderajiayi): Failed test is unrelated. It is filed in another jira. [HDFS-10696|https://issues.apache.org/jira/browse/HDFS-10696?jql=project%20%3D%20HDFS] [~andrew.wang]] Can you take a look at the patch and see if we could commit it now? > Add a configuration option to enable in-progress edit log tailing > - > > Key: HDFS-10519 > URL: https://issues.apache.org/jira/browse/HDFS-10519 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10519.001.patch, HDFS-10519.002.patch, > HDFS-10519.003.patch, HDFS-10519.004.patch, HDFS-10519.005.patch, > HDFS-10519.006.patch, HDFS-10519.007.patch, HDFS-10519.008.patch, > HDFS-10519.009.patch, HDFS-10519.010.patch, HDFS-10519.011.patch, > HDFS-10519.012.patch > > > Standby Namenode has the option to do in-progress edit log tailing to improve > the data freshness. In-progress tailing is already implemented, but it's not > enabled as default configuration. And there's no related configuration key to > turn it on. > Adding a related configuration key to let Standby Namenode is reasonable and > would be a basis for further improvement on Standby Namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10519) Add a configuration option to enable in-progress edit log tailing
[ https://issues.apache.org/jira/browse/HDFS-10519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396027#comment-15396027 ] Jiayi Zhou commented on HDFS-10519: --- Failed test is unrelated. It is filed in another jira. [HDFS-10696|https://issues.apache.org/jira/browse/HDFS-10696?jql=project%20%3D%20HDFS] [~andrew.wang]] Can you take a look at the patch and see if we could commit it now? > Add a configuration option to enable in-progress edit log tailing > - > > Key: HDFS-10519 > URL: https://issues.apache.org/jira/browse/HDFS-10519 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10519.001.patch, HDFS-10519.002.patch, > HDFS-10519.003.patch, HDFS-10519.004.patch, HDFS-10519.005.patch, > HDFS-10519.006.patch, HDFS-10519.007.patch, HDFS-10519.008.patch, > HDFS-10519.009.patch, HDFS-10519.010.patch, HDFS-10519.011.patch, > HDFS-10519.012.patch > > > Standby Namenode has the option to do in-progress edit log tailing to improve > the data freshness. In-progress tailing is already implemented, but it's not > enabled as default configuration. And there's no related configuration key to > turn it on. > Adding a related configuration key to let Standby Namenode is reasonable and > would be a basis for further improvement on Standby Namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10519) Add a configuration option to enable in-progress edit log tailing
[ https://issues.apache.org/jira/browse/HDFS-10519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10519: -- Attachment: HDFS-10519.012.patch > Add a configuration option to enable in-progress edit log tailing > - > > Key: HDFS-10519 > URL: https://issues.apache.org/jira/browse/HDFS-10519 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10519.001.patch, HDFS-10519.002.patch, > HDFS-10519.003.patch, HDFS-10519.004.patch, HDFS-10519.005.patch, > HDFS-10519.006.patch, HDFS-10519.007.patch, HDFS-10519.008.patch, > HDFS-10519.009.patch, HDFS-10519.010.patch, HDFS-10519.011.patch, > HDFS-10519.012.patch > > > Standby Namenode has the option to do in-progress edit log tailing to improve > the data freshness. In-progress tailing is already implemented, but it's not > enabled as default configuration. And there's no related configuration key to > turn it on. > Adding a related configuration key to let Standby Namenode is reasonable and > would be a basis for further improvement on Standby Namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10519) Add a configuration option to enable in-progress edit log tailing
[ https://issues.apache.org/jira/browse/HDFS-10519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10519: -- Attachment: (was: HDFS-10519.012.patch) > Add a configuration option to enable in-progress edit log tailing > - > > Key: HDFS-10519 > URL: https://issues.apache.org/jira/browse/HDFS-10519 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10519.001.patch, HDFS-10519.002.patch, > HDFS-10519.003.patch, HDFS-10519.004.patch, HDFS-10519.005.patch, > HDFS-10519.006.patch, HDFS-10519.007.patch, HDFS-10519.008.patch, > HDFS-10519.009.patch, HDFS-10519.010.patch, HDFS-10519.011.patch > > > Standby Namenode has the option to do in-progress edit log tailing to improve > the data freshness. In-progress tailing is already implemented, but it's not > enabled as default configuration. And there's no related configuration key to > turn it on. > Adding a related configuration key to let Standby Namenode is reasonable and > would be a basis for further improvement on Standby Namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10519) Add a configuration option to enable in-progress edit log tailing
[ https://issues.apache.org/jira/browse/HDFS-10519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10519: -- Attachment: HDFS-10519.012.patch > Add a configuration option to enable in-progress edit log tailing > - > > Key: HDFS-10519 > URL: https://issues.apache.org/jira/browse/HDFS-10519 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10519.001.patch, HDFS-10519.002.patch, > HDFS-10519.003.patch, HDFS-10519.004.patch, HDFS-10519.005.patch, > HDFS-10519.006.patch, HDFS-10519.007.patch, HDFS-10519.008.patch, > HDFS-10519.009.patch, HDFS-10519.010.patch, HDFS-10519.011.patch, > HDFS-10519.012.patch > > > Standby Namenode has the option to do in-progress edit log tailing to improve > the data freshness. In-progress tailing is already implemented, but it's not > enabled as default configuration. And there's no related configuration key to > turn it on. > Adding a related configuration key to let Standby Namenode is reasonable and > would be a basis for further improvement on Standby Namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10519) Add a configuration option to enable in-progress edit log tailing
[ https://issues.apache.org/jira/browse/HDFS-10519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10519: -- Attachment: (was: HDFS-10519.012.patch) > Add a configuration option to enable in-progress edit log tailing > - > > Key: HDFS-10519 > URL: https://issues.apache.org/jira/browse/HDFS-10519 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10519.001.patch, HDFS-10519.002.patch, > HDFS-10519.003.patch, HDFS-10519.004.patch, HDFS-10519.005.patch, > HDFS-10519.006.patch, HDFS-10519.007.patch, HDFS-10519.008.patch, > HDFS-10519.009.patch, HDFS-10519.010.patch, HDFS-10519.011.patch > > > Standby Namenode has the option to do in-progress edit log tailing to improve > the data freshness. In-progress tailing is already implemented, but it's not > enabled as default configuration. And there's no related configuration key to > turn it on. > Adding a related configuration key to let Standby Namenode is reasonable and > would be a basis for further improvement on Standby Namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10519) Add a configuration option to enable in-progress edit log tailing
[ https://issues.apache.org/jira/browse/HDFS-10519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10519: -- Attachment: HDFS-10519.012.patch Fix style problems. Failed test is unrelated. > Add a configuration option to enable in-progress edit log tailing > - > > Key: HDFS-10519 > URL: https://issues.apache.org/jira/browse/HDFS-10519 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10519.001.patch, HDFS-10519.002.patch, > HDFS-10519.003.patch, HDFS-10519.004.patch, HDFS-10519.005.patch, > HDFS-10519.006.patch, HDFS-10519.007.patch, HDFS-10519.008.patch, > HDFS-10519.009.patch, HDFS-10519.010.patch, HDFS-10519.011.patch, > HDFS-10519.012.patch > > > Standby Namenode has the option to do in-progress edit log tailing to improve > the data freshness. In-progress tailing is already implemented, but it's not > enabled as default configuration. And there's no related configuration key to > turn it on. > Adding a related configuration key to let Standby Namenode is reasonable and > would be a basis for further improvement on Standby Namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10519) Add a configuration option to enable in-progress edit log tailing
[ https://issues.apache.org/jira/browse/HDFS-10519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10519: -- Attachment: HDFS-10519.011.patch > Add a configuration option to enable in-progress edit log tailing > - > > Key: HDFS-10519 > URL: https://issues.apache.org/jira/browse/HDFS-10519 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10519.001.patch, HDFS-10519.002.patch, > HDFS-10519.003.patch, HDFS-10519.004.patch, HDFS-10519.005.patch, > HDFS-10519.006.patch, HDFS-10519.007.patch, HDFS-10519.008.patch, > HDFS-10519.009.patch, HDFS-10519.010.patch, HDFS-10519.011.patch > > > Standby Namenode has the option to do in-progress edit log tailing to improve > the data freshness. In-progress tailing is already implemented, but it's not > enabled as default configuration. And there's no related configuration key to > turn it on. > Adding a related configuration key to let Standby Namenode is reasonable and > would be a basis for further improvement on Standby Namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10519) Add a configuration option to enable in-progress edit log tailing
[ https://issues.apache.org/jira/browse/HDFS-10519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10519: -- Attachment: HDFS-10519.010.patch Fix some style problems. Remove committedTxId check from RemoteEditLogManifest. Since we don't have dummy send in normal cases, committedTxId will not be up-to-date (especially for a newly-setup cluster). > Add a configuration option to enable in-progress edit log tailing > - > > Key: HDFS-10519 > URL: https://issues.apache.org/jira/browse/HDFS-10519 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10519.001.patch, HDFS-10519.002.patch, > HDFS-10519.003.patch, HDFS-10519.004.patch, HDFS-10519.005.patch, > HDFS-10519.006.patch, HDFS-10519.007.patch, HDFS-10519.008.patch, > HDFS-10519.009.patch, HDFS-10519.010.patch > > > Standby Namenode has the option to do in-progress edit log tailing to improve > the data freshness. In-progress tailing is already implemented, but it's not > enabled as default configuration. And there's no related configuration key to > turn it on. > Adding a related configuration key to let Standby Namenode is reasonable and > would be a basis for further improvement on Standby Namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10519) Add a configuration option to enable in-progress edit log tailing
[ https://issues.apache.org/jira/browse/HDFS-10519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10519: -- Attachment: HDFS-10519.009.patch Fix some problems according to Andrew's recent comment. > Add a configuration option to enable in-progress edit log tailing > - > > Key: HDFS-10519 > URL: https://issues.apache.org/jira/browse/HDFS-10519 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10519.001.patch, HDFS-10519.002.patch, > HDFS-10519.003.patch, HDFS-10519.004.patch, HDFS-10519.005.patch, > HDFS-10519.006.patch, HDFS-10519.007.patch, HDFS-10519.008.patch, > HDFS-10519.009.patch > > > Standby Namenode has the option to do in-progress edit log tailing to improve > the data freshness. In-progress tailing is already implemented, but it's not > enabled as default configuration. And there's no related configuration key to > turn it on. > Adding a related configuration key to let Standby Namenode is reasonable and > would be a basis for further improvement on Standby Namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10519) Add a configuration option to enable in-progress edit log tailing
[ https://issues.apache.org/jira/browse/HDFS-10519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15385131#comment-15385131 ] Jiayi Zhou commented on HDFS-10519: --- Thanks Andrew for the review. I have attached a new patch to solve the problems. * Config key name fixed. * Comment added in hdfs-default.xml. Users want to turn it on if they want more up-to-date data on SBNN. bq. EditLogTailer, regarding the comment: edits themselves are not in-progress, an edit log segment is. Consider renaming the boolean also. * Fixed. Rename to inProgressOk which is just the parameter name in selectInputStream() bq. "isTail" is also not very descriptive, what it's really doing is bounding the tailing to the committed txID length right? * Change it to isBoundedByDurableTxId. It looks a bit long. Any suggestions? Journal.java * Unused boolean has been removed * Rename to getCommittedTxnId * Comment added. numTxn = 0 implies it's a dummy edits send, so we can return early. bq. JournalSet, why is the correct committed txnid to set here 0 rather than the max txId from the set of logs? * Good point. I lazily set it to 0 since it would not be used. Change it to the max txId from the logs. QuorumOutputStream * Fixed. Boolean has been changed to updateCommittedTxnId * Comments added. Committed txnId could be one-batch stale if we don't have this dummy flush. A thread could be added to do this dummy flush, but setting a proper time period needs to be considered. For now, I prefer to leave it sync. bq. QuorumJournalManager, the ternary can be written instead with Math.min yea? I'd prefer to see a little logic rework that avoids the need for the else also. * Fixed. RemoteEditLogManifest * Remove single-arg form of constructor. * Add a validation in checkState. committedTxnId should be larger than the start TxId of the logs. * toString() modified. bq. TestBKSR, TestFJM, you don't need to modify the call since we kept the old overload. * Fixed. In TestFJM, parameter is JournalManager, changed it to FileJournalManager. bq. TestNNSRManager, we should pass through the parameter in the mock, like the other args * Fixed. TestStandbyInProgressTail * Typos fixed * Thread.sleep removed. Use manual doTailEdits() instead. * @Before and @After methods added. Null check is included in tearDown() * Assert methods are static now. * Randomized tests could be tricky. I haven't come up with a good idea yet. A new jira might be filed for it in the future. > Add a configuration option to enable in-progress edit log tailing > - > > Key: HDFS-10519 > URL: https://issues.apache.org/jira/browse/HDFS-10519 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10519.001.patch, HDFS-10519.002.patch, > HDFS-10519.003.patch, HDFS-10519.004.patch, HDFS-10519.005.patch, > HDFS-10519.006.patch, HDFS-10519.007.patch, HDFS-10519.008.patch > > > Standby Namenode has the option to do in-progress edit log tailing to improve > the data freshness. In-progress tailing is already implemented, but it's not > enabled as default configuration. And there's no related configuration key to > turn it on. > Adding a related configuration key to let Standby Namenode is reasonable and > would be a basis for further improvement on Standby Namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10519) Add a configuration option to enable in-progress edit log tailing
[ https://issues.apache.org/jira/browse/HDFS-10519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10519: -- Attachment: HDFS-10519.008.patch > Add a configuration option to enable in-progress edit log tailing > - > > Key: HDFS-10519 > URL: https://issues.apache.org/jira/browse/HDFS-10519 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10519.001.patch, HDFS-10519.002.patch, > HDFS-10519.003.patch, HDFS-10519.004.patch, HDFS-10519.005.patch, > HDFS-10519.006.patch, HDFS-10519.007.patch, HDFS-10519.008.patch > > > Standby Namenode has the option to do in-progress edit log tailing to improve > the data freshness. In-progress tailing is already implemented, but it's not > enabled as default configuration. And there's no related configuration key to > turn it on. > Adding a related configuration key to let Standby Namenode is reasonable and > would be a basis for further improvement on Standby Namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10628) Log HDFS Balancer exit message to its own log
[ https://issues.apache.org/jira/browse/HDFS-10628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10628: -- Attachment: HDFS-10628.002.patch Checkstyle fixed. Failure tests are not related as it's just a simple patch. > Log HDFS Balancer exit message to its own log > - > > Key: HDFS-10628 > URL: https://issues.apache.org/jira/browse/HDFS-10628 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Affects Versions: 2.8.0 >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10628.001.patch, HDFS-10628.002.patch > > > Currently, the exit message is logged to stderr. It would be more convenient > if we also log this to Balancer log when people want to figure out why > Balancer is aborted. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10628) Log HDFS Balancer exit message to its own log
[ https://issues.apache.org/jira/browse/HDFS-10628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15378367#comment-15378367 ] Jiayi Zhou commented on HDFS-10628: --- Versions updated as comment. > Log HDFS Balancer exit message to its own log > - > > Key: HDFS-10628 > URL: https://issues.apache.org/jira/browse/HDFS-10628 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Affects Versions: 2.8.0 >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10628.001.patch > > > Currently, the exit message is logged to stderr. It would be more convenient > if we also log this to Balancer log when people want to figure out why > Balancer is aborted. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10628) Log HDFS Balancer exit message to its own log
[ https://issues.apache.org/jira/browse/HDFS-10628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10628: -- Affects Version/s: 2.8.0 Target Version/s: 2.8.0 > Log HDFS Balancer exit message to its own log > - > > Key: HDFS-10628 > URL: https://issues.apache.org/jira/browse/HDFS-10628 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Affects Versions: 2.8.0 >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10628.001.patch > > > Currently, the exit message is logged to stderr. It would be more convenient > if we also log this to Balancer log when people want to figure out why > Balancer is aborted. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10628) Log HDFS Balancer exit message to its own log
[ https://issues.apache.org/jira/browse/HDFS-10628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10628: -- Attachment: HDFS-10628.001.patch > Log HDFS Balancer exit message to its own log > - > > Key: HDFS-10628 > URL: https://issues.apache.org/jira/browse/HDFS-10628 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou > Attachments: HDFS-10628.001.patch > > > Currently, the exit message is logged to stderr. It would be more convenient > if we also log this to Balancer log when people want to figure out why > Balancer is aborted. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10628) Log HDFS Balancer exit message to its own log
[ https://issues.apache.org/jira/browse/HDFS-10628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10628: -- Status: Patch Available (was: Open) Simply log the exit message in Balancer log > Log HDFS Balancer exit message to its own log > - > > Key: HDFS-10628 > URL: https://issues.apache.org/jira/browse/HDFS-10628 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou > Attachments: HDFS-10628.001.patch > > > Currently, the exit message is logged to stderr. It would be more convenient > if we also log this to Balancer log when people want to figure out why > Balancer is aborted. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10628) Log HDFS Balancer exit message to its own log
[ https://issues.apache.org/jira/browse/HDFS-10628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10628: -- Description: Currently, the exit message is logged to stderr. It would be more convenient if we also log this to Balancer log when people want to figure out why Balancer is aborted. (was: Currently, the exit message is logged to stderr. It would be better if we also log this to Balancer log when people want to figure out why Balancer is aborted.) > Log HDFS Balancer exit message to its own log > - > > Key: HDFS-10628 > URL: https://issues.apache.org/jira/browse/HDFS-10628 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou > > Currently, the exit message is logged to stderr. It would be more convenient > if we also log this to Balancer log when people want to figure out why > Balancer is aborted. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10628) Log HDFS Balancer exit message to its own log
Jiayi Zhou created HDFS-10628: - Summary: Log HDFS Balancer exit message to its own log Key: HDFS-10628 URL: https://issues.apache.org/jira/browse/HDFS-10628 Project: Hadoop HDFS Issue Type: Improvement Reporter: Jiayi Zhou Assignee: Jiayi Zhou Currently, the exit message is logged to stderr. It would be better if we also log this to Balancer log when people want to figure out why Balancer is aborted. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10519) Add a configuration option to enable in-progress edit log tailing
[ https://issues.apache.org/jira/browse/HDFS-10519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10519: -- Attachment: HDFS-10519.007.patch Add a new parameter isTail to selectInputStream() on the NameNode side and a field in RemoteEditLogManifest. When we do in-progress tailing, we'll use committedTxnId rather than highestWrittenTxnId. This won't affect other parts which also need to select in-progress edits. > Add a configuration option to enable in-progress edit log tailing > - > > Key: HDFS-10519 > URL: https://issues.apache.org/jira/browse/HDFS-10519 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10519.001.patch, HDFS-10519.002.patch, > HDFS-10519.003.patch, HDFS-10519.004.patch, HDFS-10519.005.patch, > HDFS-10519.006.patch, HDFS-10519.007.patch > > > Standby Namenode has the option to do in-progress edit log tailing to improve > the data freshness. In-progress tailing is already implemented, but it's not > enabled as default configuration. And there's no related configuration key to > turn it on. > Adding a related configuration key to let Standby Namenode is reasonable and > would be a basis for further improvement on Standby Namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10519) Add a configuration option to enable in-progress edit log tailing
[ https://issues.apache.org/jira/browse/HDFS-10519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10519: -- Attachment: HDFS-10519.006.patch Also add a boolean flag in Journal for the same purpose. > Add a configuration option to enable in-progress edit log tailing > - > > Key: HDFS-10519 > URL: https://issues.apache.org/jira/browse/HDFS-10519 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10519.001.patch, HDFS-10519.002.patch, > HDFS-10519.003.patch, HDFS-10519.004.patch, HDFS-10519.005.patch, > HDFS-10519.006.patch > > > Standby Namenode has the option to do in-progress edit log tailing to improve > the data freshness. In-progress tailing is already implemented, but it's not > enabled as default configuration. And there's no related configuration key to > turn it on. > Adding a related configuration key to let Standby Namenode is reasonable and > would be a basis for further improvement on Standby Namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10519) Add a configuration option to enable in-progress edit log tailing
[ https://issues.apache.org/jira/browse/HDFS-10519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10519: -- Attachment: HDFS-10519.005.patch Fix bugs and checkstyle problems. Add a boolean flag in QuorumOutputStream so that fake send will be triggered only when in-progress tailing is enabled. > Add a configuration option to enable in-progress edit log tailing > - > > Key: HDFS-10519 > URL: https://issues.apache.org/jira/browse/HDFS-10519 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10519.001.patch, HDFS-10519.002.patch, > HDFS-10519.003.patch, HDFS-10519.004.patch, HDFS-10519.005.patch > > > Standby Namenode has the option to do in-progress edit log tailing to improve > the data freshness. In-progress tailing is already implemented, but it's not > enabled as default configuration. And there's no related configuration key to > turn it on. > Adding a related configuration key to let Standby Namenode is reasonable and > would be a basis for further improvement on Standby Namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-10519) Add a configuration option to enable in-progress edit log tailing
[ https://issues.apache.org/jira/browse/HDFS-10519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373773#comment-15373773 ] Jiayi Zhou edited comment on HDFS-10519 at 7/12/16 10:19 PM: - Add a unit test which I forgot to add in the 003 patch. was (Author: clouderajiayi): Add a unit test which I forgot to add in the previous patch. > Add a configuration option to enable in-progress edit log tailing > - > > Key: HDFS-10519 > URL: https://issues.apache.org/jira/browse/HDFS-10519 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10519.001.patch, HDFS-10519.002.patch, > HDFS-10519.003.patch, HDFS-10519.004.patch > > > Standby Namenode has the option to do in-progress edit log tailing to improve > the data freshness. In-progress tailing is already implemented, but it's not > enabled as default configuration. And there's no related configuration key to > turn it on. > Adding a related configuration key to let Standby Namenode is reasonable and > would be a basis for further improvement on Standby Namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10519) Add a configuration option to enable in-progress edit log tailing
[ https://issues.apache.org/jira/browse/HDFS-10519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10519: -- Attachment: HDFS-10519.004.patch Add a unit test which I forgot to add in the previous patch. > Add a configuration option to enable in-progress edit log tailing > - > > Key: HDFS-10519 > URL: https://issues.apache.org/jira/browse/HDFS-10519 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10519.001.patch, HDFS-10519.002.patch, > HDFS-10519.003.patch, HDFS-10519.004.patch > > > Standby Namenode has the option to do in-progress edit log tailing to improve > the data freshness. In-progress tailing is already implemented, but it's not > enabled as default configuration. And there's no related configuration key to > turn it on. > Adding a related configuration key to let Standby Namenode is reasonable and > would be a basis for further improvement on Standby Namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10519) Add a configuration option to enable in-progress edit log tailing
[ https://issues.apache.org/jira/browse/HDFS-10519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10519: -- Attachment: HDFS-10519.003.patch Add a fake sendedits to make committed txn id more up-to-date. This means the standby namenode will only tail the in-progress edits which are committed. Thank you [~tlipcon] for the idea during an offline discussion. > Add a configuration option to enable in-progress edit log tailing > - > > Key: HDFS-10519 > URL: https://issues.apache.org/jira/browse/HDFS-10519 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10519.001.patch, HDFS-10519.002.patch, > HDFS-10519.003.patch > > > Standby Namenode has the option to do in-progress edit log tailing to improve > the data freshness. In-progress tailing is already implemented, but it's not > enabled as default configuration. And there's no related configuration key to > turn it on. > Adding a related configuration key to let Standby Namenode is reasonable and > would be a basis for further improvement on Standby Namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-3077) Quorum-based protocol for reading and writing edit logs
[ https://issues.apache.org/jira/browse/HDFS-3077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-3077: - Assignee: (was: Jiayi Zhou) > Quorum-based protocol for reading and writing edit logs > --- > > Key: HDFS-3077 > URL: https://issues.apache.org/jira/browse/HDFS-3077 > Project: Hadoop HDFS > Issue Type: New Feature > Components: ha, namenode >Reporter: Todd Lipcon > Fix For: QuorumJournalManager (HDFS-3077), 2.0.3-alpha > > Attachments: hdfs-3077-branch-2.txt, hdfs-3077-partial.txt, > hdfs-3077-test-merge.txt, hdfs-3077.txt, hdfs-3077.txt, hdfs-3077.txt, > hdfs-3077.txt, hdfs-3077.txt, hdfs-3077.txt, hdfs-3077.txt, > qjournal-design.pdf, qjournal-design.pdf, qjournal-design.pdf, > qjournal-design.pdf, qjournal-design.pdf, qjournal-design.pdf, > qjournal-design.tex, qjournal-design.tex > > > Currently, one of the weak points of the HA design is that it relies on > shared storage such as an NFS filer for the shared edit log. One alternative > that has been proposed is to depend on BookKeeper, a ZooKeeper subproject > which provides a highly available replicated edit log on commodity hardware. > This JIRA is to implement another alternative, based on a quorum commit > protocol, integrated more tightly in HDFS and with the requirements driven > only by HDFS's needs rather than more generic use cases. More details to > follow. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-3077) Quorum-based protocol for reading and writing edit logs
[ https://issues.apache.org/jira/browse/HDFS-3077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou reassigned HDFS-3077: Assignee: Jiayi Zhou (was: Todd Lipcon) > Quorum-based protocol for reading and writing edit logs > --- > > Key: HDFS-3077 > URL: https://issues.apache.org/jira/browse/HDFS-3077 > Project: Hadoop HDFS > Issue Type: New Feature > Components: ha, namenode >Reporter: Todd Lipcon >Assignee: Jiayi Zhou > Fix For: QuorumJournalManager (HDFS-3077), 2.0.3-alpha > > Attachments: hdfs-3077-branch-2.txt, hdfs-3077-partial.txt, > hdfs-3077-test-merge.txt, hdfs-3077.txt, hdfs-3077.txt, hdfs-3077.txt, > hdfs-3077.txt, hdfs-3077.txt, hdfs-3077.txt, hdfs-3077.txt, > qjournal-design.pdf, qjournal-design.pdf, qjournal-design.pdf, > qjournal-design.pdf, qjournal-design.pdf, qjournal-design.pdf, > qjournal-design.tex, qjournal-design.tex > > > Currently, one of the weak points of the HA design is that it relies on > shared storage such as an NFS filer for the shared edit log. One alternative > that has been proposed is to depend on BookKeeper, a ZooKeeper subproject > which provides a highly available replicated edit log on commodity hardware. > This JIRA is to implement another alternative, based on a quorum commit > protocol, integrated more tightly in HDFS and with the requirements driven > only by HDFS's needs rather than more generic use cases. More details to > follow. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10519) Add a configuration option to enable in-progress edit log tailing
[
https://issues.apache.org/jira/browse/HDFS-10519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15330013#comment-15330013
]
Jiayi Zhou commented on HDFS-10519:
---
Maybe I got something wrong, but isn't the MetaRecoveryContext already null at
the beginning?
In doTailEdits() from EditLogTailer, we have
streams = editLog.selectInputStreams(lastTxnId +
1, 0, null, inProgressTail);
Here, MetaRecoverContext is null. And in image.LoadEdits(), we have
public long
loadEdits(Iterable editStreams,
FSNamesystem target) throws IOException {
return loadEdits(editStreams, target, null,
null);
}
MetaRecoveryContext is also null.
Also, I don't know why toAtLeastTxId is always set to 0 in the implementation,
making the checkForGaps() method no use. If we set it to some other value, like
lastTxnId + 1, the multiple replay should be rejected.
> Add a configuration option to enable in-progress edit log tailing
> -
>
> Key: HDFS-10519
> URL: https://issues.apache.org/jira/browse/HDFS-10519
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ha
>Reporter: Jiayi Zhou
>Assignee: Jiayi Zhou
>Priority: Minor
> Attachments: HDFS-10519.001.patch, HDFS-10519.002.patch
>
>
> Standby Namenode has the option to do in-progress edit log tailing to improve
> the data freshness. In-progress tailing is already implemented, but it's not
> enabled as default configuration. And there's no related configuration key to
> turn it on.
> Adding a related configuration key to let Standby Namenode is reasonable and
> would be a basis for further improvement on Standby Namenode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10519) Add a configuration option to enable in-progress edit log tailing
[ https://issues.apache.org/jira/browse/HDFS-10519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10519: -- Attachment: HDFS-10519.002.patch > Add a configuration option to enable in-progress edit log tailing > - > > Key: HDFS-10519 > URL: https://issues.apache.org/jira/browse/HDFS-10519 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10519.001.patch, HDFS-10519.002.patch > > > Standby Namenode has the option to do in-progress edit log tailing to improve > the data freshness. In-progress tailing is already implemented, but it's not > enabled as default configuration. And there's no related configuration key to > turn it on. > Adding a related configuration key to let Standby Namenode is reasonable and > would be a basis for further improvement on Standby Namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10519) Add a configuration option to enable in-progress edit log tailing
[ https://issues.apache.org/jira/browse/HDFS-10519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10519: -- Release Note: Add a configuration option to enable in-progress edit tailing and a related unit test Status: Patch Available (was: Open) > Add a configuration option to enable in-progress edit log tailing > - > > Key: HDFS-10519 > URL: https://issues.apache.org/jira/browse/HDFS-10519 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10519.001.patch > > > Standby Namenode has the option to do in-progress edit log tailing to improve > the data freshness. In-progress tailing is already implemented, but it's not > enabled as default configuration. And there's no related configuration key to > turn it on. > Adding a related configuration key to let Standby Namenode is reasonable and > would be a basis for further improvement on Standby Namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10519) Add a configuration option to enable in-progress edit log tailing
[ https://issues.apache.org/jira/browse/HDFS-10519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10519: -- Attachment: HDFS-10519.001.patch > Add a configuration option to enable in-progress edit log tailing > - > > Key: HDFS-10519 > URL: https://issues.apache.org/jira/browse/HDFS-10519 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10519.001.patch > > > Standby Namenode has the option to do in-progress edit log tailing to improve > the data freshness. In-progress tailing is already implemented, but it's not > enabled as default configuration. And there's no related configuration key to > turn it on. > Adding a related configuration key to let Standby Namenode is reasonable and > would be a basis for further improvement on Standby Namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10519) Add a configuration option to enable in-progress edit log tailing
Jiayi Zhou created HDFS-10519: - Summary: Add a configuration option to enable in-progress edit log tailing Key: HDFS-10519 URL: https://issues.apache.org/jira/browse/HDFS-10519 Project: Hadoop HDFS Issue Type: Improvement Components: ha Reporter: Jiayi Zhou Assignee: Jiayi Zhou Priority: Minor Standby Namenode has the option to do in-progress edit log tailing to improve the data freshness. In-progress tailing is already implemented, but it's not enabled as default configuration. And there's no related configuration key to turn it on. Adding a related configuration key to let Standby Namenode is reasonable and would be a basis for further improvement on Standby Namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayi Zhou updated HDFS-10375:
--
Attachment: HDFS-10375.004.patch
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10375.002.patch, HDFS-10375.003.patch,
> HDFS-10375.004.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317410#comment-15317410
]
Jiayi Zhou edited comment on HDFS-10375 at 6/6/16 10:31 PM:
I've merged them into 1 commit now. How about this new patch 003?
was (Author: clouderajiayi):
HI've merged them into 1 commit now. How about this new patch 003?
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10375.002.patch, HDFS-10375.003.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317410#comment-15317410
]
Jiayi Zhou commented on HDFS-10375:
---
HI've merged them into 1 commit now. How about this new patch 003?
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10375.002.patch, HDFS-10375.003.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayi Zhou updated HDFS-10375:
--
Attachment: (was: HDFS-10375.003.patch)
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10375.002.patch, HDFS-10375.003.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayi Zhou updated HDFS-10375:
--
Attachment: HDFS-10375.003.patch
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10375.002.patch, HDFS-10375.003.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317341#comment-15317341
]
Jiayi Zhou commented on HDFS-10375:
---
Hi John, how can I run this Hadoop QA again? I think I have fixed the problem
and it should work properly now.
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10375.002.patch, HDFS-10375.003.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayi Zhou updated HDFS-10375:
--
Attachment: HDFS-10375.003.patch
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10375.002.patch, HDFS-10375.003.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316964#comment-15316964
]
Jiayi Zhou commented on HDFS-10375:
---
So I only need to git commit again on my local machine?
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10375.002.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayi Zhou updated HDFS-10375:
--
Attachment: HDFS-10375.002.patch
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Fix For: 2.8.0
>
> Attachments: HDFS-10375.002.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayi Zhou updated HDFS-10375:
--
Attachment: (was: HDFS-10375.001.patch)
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Fix For: 2.8.0
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayi Zhou updated HDFS-10375:
--
Status: Patch Available (was: Open)
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Fix For: 2.8.0
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayi Zhou updated HDFS-10375:
--
Status: Open (was: Patch Available)
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Fix For: 2.8.0
>
> Attachments: HDFS-10375.001.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayi Zhou updated HDFS-10375:
--
Status: Patch Available (was: Open)
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Fix For: 2.8.0
>
> Attachments: HDFS-10375.001.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10375) Remove redundent TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayi Zhou updated HDFS-10375:
--
Status: Open (was: Patch Available)
> Remove redundent TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Fix For: 2.8.0
>
> Attachments: HDFS-10375.001.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10375) Remove redundent TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayi Zhou updated HDFS-10375:
--
Attachment: HDFS-10375.001.patch
> Remove redundent TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Fix For: 2.8.0
>
> Attachments: HDFS-10375.001.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10375) Remove redundent TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayi Zhou updated HDFS-10375:
--
Fix Version/s: 2.8.0
Release Note: Remove redundent TestMiniDFSCluster.testDualClusters to save
time.
Status: Patch Available (was: Open)
> Remove redundent TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Fix For: 2.8.0
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-10375) Remove redundent TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayi Zhou reassigned HDFS-10375:
-
Assignee: Jiayi Zhou
> Remove redundent TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
