[jira] [Updated] (HDFS-12603) Enable async edit logging by default
[
https://issues.apache.org/jira/browse/HDFS-12603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-12603:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0
2.8.3
2.9.0
Status: Resolved (was: Patch Available)
Committed to trunk through branch-2.8.
There was a minor conflict in {{hdfs-default.xml}} for branch-2.8, otherwise
clean cherry-picks.
Thanks for the contribution Andrew!
> Enable async edit logging by default
>
>
> Key: HDFS-12603
> URL: https://issues.apache.org/jira/browse/HDFS-12603
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.8.0, 2.9.0, 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Fix For: 2.9.0, 2.8.3, 3.0.0
>
> Attachments: HDFS-12603.001.patch, HDFS-12603.002.patch,
> HDFS-12603.003.patch, HDFS-12603.004.patch, HDFS-12603.branch-2.01.patch,
> HDFS-12603.branch-2.02.patch, HDFS-12603.injectors-not-working.patch
>
>
> HDFS-7964 added support for async edit logging. Based on further discussion
> in that JIRA, we think it's safe to turn this on by default for better
> out-of-the-box performance.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12603) Enable async edit logging by default
[ https://issues.apache.org/jira/browse/HDFS-12603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-12603: --- Attachment: HDFS-12603.004.patch Here's a patch which undoes the toString additions from 003 and also comments out those two test cases with async edit logging on. > Enable async edit logging by default > > > Key: HDFS-12603 > URL: https://issues.apache.org/jira/browse/HDFS-12603 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.8.0, 2.9.0, 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HDFS-12603.001.patch, HDFS-12603.002.patch, > HDFS-12603.003.patch, HDFS-12603.004.patch, HDFS-12603.branch-2.01.patch, > HDFS-12603.branch-2.02.patch, HDFS-12603.injectors-not-working.patch > > > HDFS-7964 added support for async edit logging. Based on further discussion > in that JIRA, we think it's safe to turn this on by default for better > out-of-the-box performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12603) Enable async edit logging by default
[ https://issues.apache.org/jira/browse/HDFS-12603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-12603: --- Status: Patch Available (was: Reopened) > Enable async edit logging by default > > > Key: HDFS-12603 > URL: https://issues.apache.org/jira/browse/HDFS-12603 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0-alpha1, 2.8.0, 2.9.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HDFS-12603.001.patch, HDFS-12603.002.patch, > HDFS-12603.003.patch, HDFS-12603.004.patch, HDFS-12603.branch-2.01.patch, > HDFS-12603.branch-2.02.patch, HDFS-12603.injectors-not-working.patch > > > HDFS-7964 added support for async edit logging. Based on further discussion > in that JIRA, we think it's safe to turn this on by default for better > out-of-the-box performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12603) Enable async edit logging by default
[ https://issues.apache.org/jira/browse/HDFS-12603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-12603: - Attachment: HDFS-12603.injectors-not-working.patch > Enable async edit logging by default > > > Key: HDFS-12603 > URL: https://issues.apache.org/jira/browse/HDFS-12603 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.8.0, 2.9.0, 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HDFS-12603.001.patch, HDFS-12603.002.patch, > HDFS-12603.003.patch, HDFS-12603.branch-2.01.patch, > HDFS-12603.branch-2.02.patch, HDFS-12603.injectors-not-working.patch > > > HDFS-7964 added support for async edit logging. Based on further discussion > in that JIRA, we think it's safe to turn this on by default for better > out-of-the-box performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12603) Enable async edit logging by default
[ https://issues.apache.org/jira/browse/HDFS-12603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-12603: --- Fix Version/s: (was: 3.0.0) > Enable async edit logging by default > > > Key: HDFS-12603 > URL: https://issues.apache.org/jira/browse/HDFS-12603 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.8.0, 2.9.0, 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HDFS-12603.001.patch, HDFS-12603.002.patch, > HDFS-12603.003.patch, HDFS-12603.branch-2.01.patch, > HDFS-12603.branch-2.02.patch > > > HDFS-7964 added support for async edit logging. Based on further discussion > in that JIRA, we think it's safe to turn this on by default for better > out-of-the-box performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12603) Enable async edit logging by default
[ https://issues.apache.org/jira/browse/HDFS-12603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-12603: - Attachment: HDFS-12603.branch-2.02.patch > Enable async edit logging by default > > > Key: HDFS-12603 > URL: https://issues.apache.org/jira/browse/HDFS-12603 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.8.0, 2.9.0, 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 3.0.0 > > Attachments: HDFS-12603.001.patch, HDFS-12603.002.patch, > HDFS-12603.003.patch, HDFS-12603.branch-2.01.patch, > HDFS-12603.branch-2.02.patch > > > HDFS-7964 added support for async edit logging. Based on further discussion > in that JIRA, we think it's safe to turn this on by default for better > out-of-the-box performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12603) Enable async edit logging by default
[ https://issues.apache.org/jira/browse/HDFS-12603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-12603: --- Fix Version/s: (was: 2.9.0) > Enable async edit logging by default > > > Key: HDFS-12603 > URL: https://issues.apache.org/jira/browse/HDFS-12603 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.8.0, 2.9.0, 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 3.0.0 > > Attachments: HDFS-12603.001.patch, HDFS-12603.002.patch, > HDFS-12603.003.patch, HDFS-12603.branch-2.01.patch > > > HDFS-7964 added support for async edit logging. Based on further discussion > in that JIRA, we think it's safe to turn this on by default for better > out-of-the-box performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12603) Enable async edit logging by default
[ https://issues.apache.org/jira/browse/HDFS-12603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-12603: --- Resolution: Fixed Fix Version/s: 3.0.0 2.9.0 Status: Resolved (was: Patch Available) I've committed this to trunk, branch-3.0, branch-2. branch-2.8 had some conflicts, so I left it out. > Enable async edit logging by default > > > Key: HDFS-12603 > URL: https://issues.apache.org/jira/browse/HDFS-12603 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.8.0, 2.9.0, 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 2.9.0, 3.0.0 > > Attachments: HDFS-12603.001.patch, HDFS-12603.002.patch, > HDFS-12603.003.patch, HDFS-12603.branch-2.01.patch > > > HDFS-7964 added support for async edit logging. Based on further discussion > in that JIRA, we think it's safe to turn this on by default for better > out-of-the-box performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12603) Enable async edit logging by default
[ https://issues.apache.org/jira/browse/HDFS-12603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-12603: - Attachment: HDFS-12603.003.patch > Enable async edit logging by default > > > Key: HDFS-12603 > URL: https://issues.apache.org/jira/browse/HDFS-12603 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.8.0, 2.9.0, 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HDFS-12603.001.patch, HDFS-12603.002.patch, > HDFS-12603.003.patch, HDFS-12603.branch-2.01.patch > > > HDFS-7964 added support for async edit logging. Based on further discussion > in that JIRA, we think it's safe to turn this on by default for better > out-of-the-box performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12603) Enable async edit logging by default
[ https://issues.apache.org/jira/browse/HDFS-12603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-12603: - Attachment: HDFS-12603.branch-2.01.patch Hm, seems adding toString to FSEditLog itself solved it. I think we should do this for the trunk patch too, will attach one shortly. Just curious, how did you pin this down to toString initially? > Enable async edit logging by default > > > Key: HDFS-12603 > URL: https://issues.apache.org/jira/browse/HDFS-12603 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.8.0, 2.9.0, 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HDFS-12603.001.patch, HDFS-12603.002.patch, > HDFS-12603.branch-2.01.patch > > > HDFS-7964 added support for async edit logging. Based on further discussion > in that JIRA, we think it's safe to turn this on by default for better > out-of-the-box performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12603) Enable async edit logging by default
[ https://issues.apache.org/jira/browse/HDFS-12603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-12603: --- Attachment: HDFS-12603.002.patch The strange mocking error in TestFailureToReadEdits was resolved by adding a toString implementation to FSEditLogAsync. ¯\\_(ツ)_/¯ Additionally parameterized that test and fixed a typo I spotted along the way. > Enable async edit logging by default > > > Key: HDFS-12603 > URL: https://issues.apache.org/jira/browse/HDFS-12603 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.8.0, 2.9.0, 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HDFS-12603.001.patch, HDFS-12603.002.patch > > > HDFS-7964 added support for async edit logging. Based on further discussion > in that JIRA, we think it's safe to turn this on by default for better > out-of-the-box performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12603) Enable async edit logging by default
[ https://issues.apache.org/jira/browse/HDFS-12603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-12603: --- Status: Patch Available (was: Open) > Enable async edit logging by default > > > Key: HDFS-12603 > URL: https://issues.apache.org/jira/browse/HDFS-12603 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0-alpha1, 2.8.0, 2.9.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HDFS-12603.001.patch > > > HDFS-7964 added support for async edit logging. Based on further discussion > in that JIRA, we think it's safe to turn this on by default for better > out-of-the-box performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12603) Enable async edit logging by default
[ https://issues.apache.org/jira/browse/HDFS-12603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-12603: --- Target Version/s: 2.9.0, 2.8.3, 3.0.0 (was: 2.9.0, 3.0.0) > Enable async edit logging by default > > > Key: HDFS-12603 > URL: https://issues.apache.org/jira/browse/HDFS-12603 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.8.0, 2.9.0, 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HDFS-12603.001.patch > > > HDFS-7964 added support for async edit logging. Based on further discussion > in that JIRA, we think it's safe to turn this on by default for better > out-of-the-box performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12603) Enable async edit logging by default
[ https://issues.apache.org/jira/browse/HDFS-12603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-12603: --- Attachment: HDFS-12603.001.patch Trivial patch attached. The original patches in HDFS-7964 parameterized all the edit log related tests to run with async edit logging enabled and disabled, so I don't think we need any additional test coverage. > Enable async edit logging by default > > > Key: HDFS-12603 > URL: https://issues.apache.org/jira/browse/HDFS-12603 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.8.0, 2.9.0, 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HDFS-12603.001.patch > > > HDFS-7964 added support for async edit logging. Based on further discussion > in that JIRA, we think it's safe to turn this on by default for better > out-of-the-box performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
