[jira] [Updated] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-10872: - Resolution: Fixed Fix Version/s: 2.7.4 Status: Resolved (was: Patch Available) Thanks Erik! I just verified the reported branch-2.7 test failures (non reproducible locally), as well as local cluster, and committed to branch-2.7. > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha2 > > Attachments: FSLockPerf.java, HDFS-10872-branch-2.012.patch, > HDFS-10872-branch-2.013.patch, HDFS-10872-branch-2.7.013.patch, > HDFS-10872.000.patch, HDFS-10872.001.patch, HDFS-10872.002.patch, > HDFS-10872.003.patch, HDFS-10872.004.patch, HDFS-10872.005.patch, > HDFS-10872.006.patch, HDFS-10872.007.patch, HDFS-10872.008.patch, > HDFS-10872.009.patch, HDFS-10872.010.patch, HDFS-10872.011.patch, > HDFS-10872.012.patch, jmx-output > > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated HDFS-10872: --- Attachment: HDFS-10872-branch-2.7.013.patch Attaching patch for branch-2.7 (v13). > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: FSLockPerf.java, HDFS-10872-branch-2.012.patch, > HDFS-10872-branch-2.013.patch, HDFS-10872-branch-2.7.013.patch, > HDFS-10872.000.patch, HDFS-10872.001.patch, HDFS-10872.002.patch, > HDFS-10872.003.patch, HDFS-10872.004.patch, HDFS-10872.005.patch, > HDFS-10872.006.patch, HDFS-10872.007.patch, HDFS-10872.008.patch, > HDFS-10872.009.patch, HDFS-10872.010.patch, HDFS-10872.011.patch, > HDFS-10872.012.patch, jmx-output > > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-10872: - Fix Version/s: 3.0.0-alpha2 2.8.0 > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: FSLockPerf.java, HDFS-10872-branch-2.012.patch, > HDFS-10872-branch-2.013.patch, HDFS-10872.000.patch, HDFS-10872.001.patch, > HDFS-10872.002.patch, HDFS-10872.003.patch, HDFS-10872.004.patch, > HDFS-10872.005.patch, HDFS-10872.006.patch, HDFS-10872.007.patch, > HDFS-10872.008.patch, HDFS-10872.009.patch, HDFS-10872.010.patch, > HDFS-10872.011.patch, HDFS-10872.012.patch, jmx-output > > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated HDFS-10872: --- Attachment: HDFS-10872-branch-2.013.patch Looks like I introduced two small checkstyle issues when I fixed branch-2 conflicts. Attaching branch-2 v13 patch to fix. > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > Attachments: FSLockPerf.java, HDFS-10872-branch-2.012.patch, > HDFS-10872-branch-2.013.patch, HDFS-10872.000.patch, HDFS-10872.001.patch, > HDFS-10872.002.patch, HDFS-10872.003.patch, HDFS-10872.004.patch, > HDFS-10872.005.patch, HDFS-10872.006.patch, HDFS-10872.007.patch, > HDFS-10872.008.patch, HDFS-10872.009.patch, HDFS-10872.010.patch, > HDFS-10872.011.patch, HDFS-10872.012.patch, jmx-output > > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HDFS-10872:
---
Attachment: HDFS-10872-branch-2.012.patch
Thank you for the commit, [~zhz]!
I've attached a branch-2 patch. You were correct that the conflicts were just
mechanical fixes - touching so many lines in {{FSNamesystem}} in a trivial way
(just adding the operation name) caused many code conflicts but no real logical
conflicts.
> Add MutableRate metrics for FSNamesystemLock operations
> ---
>
> Key: HDFS-10872
> URL: https://issues.apache.org/jira/browse/HDFS-10872
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Erik Krogen
>Assignee: Erik Krogen
> Attachments: FSLockPerf.java, HDFS-10872-branch-2.012.patch,
> HDFS-10872.000.patch, HDFS-10872.001.patch, HDFS-10872.002.patch,
> HDFS-10872.003.patch, HDFS-10872.004.patch, HDFS-10872.005.patch,
> HDFS-10872.006.patch, HDFS-10872.007.patch, HDFS-10872.008.patch,
> HDFS-10872.009.patch, HDFS-10872.010.patch, HDFS-10872.011.patch,
> HDFS-10872.012.patch, jmx-output
>
>
> Add metrics for FSNamesystemLock operations to see, overall, how long each
> operation is holding the lock for. Use MutableRate metrics for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated HDFS-10872: --- Attachment: HDFS-10872.012.patch Looks like when going from v009 to v010 patch I accidentally re-included the YARN-5868 fix. Attaching v012 patch with that removed. Sorry for the mixup. > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > Attachments: FSLockPerf.java, HDFS-10872.000.patch, > HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch, > HDFS-10872.004.patch, HDFS-10872.005.patch, HDFS-10872.006.patch, > HDFS-10872.007.patch, HDFS-10872.008.patch, HDFS-10872.009.patch, > HDFS-10872.010.patch, HDFS-10872.011.patch, HDFS-10872.012.patch, jmx-output > > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-10872: - Attachment: HDFS-10872.011.patch Looks like Yetus was confused because the newest file was the JMX output. I'm attaching v11 patch which is a dup of v10. > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > Attachments: FSLockPerf.java, HDFS-10872.000.patch, > HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch, > HDFS-10872.004.patch, HDFS-10872.005.patch, HDFS-10872.006.patch, > HDFS-10872.007.patch, HDFS-10872.008.patch, HDFS-10872.009.patch, > HDFS-10872.010.patch, HDFS-10872.011.patch, jmx-output > > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HDFS-10872:
---
Attachment: jmx-output
HDFS-10872.010.patch
Thanks for looking, [~liuml07].
Attaching v010 patch with updated names for the metrics:
{{FSN(Read|Write)LockOperationName}}. I've also attached an example of what the
outputted metrics look like on the JMX page ({{jmx-output}}).
> Add MutableRate metrics for FSNamesystemLock operations
> ---
>
> Key: HDFS-10872
> URL: https://issues.apache.org/jira/browse/HDFS-10872
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Erik Krogen
>Assignee: Erik Krogen
> Attachments: FSLockPerf.java, HDFS-10872.000.patch,
> HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch,
> HDFS-10872.004.patch, HDFS-10872.005.patch, HDFS-10872.006.patch,
> HDFS-10872.007.patch, HDFS-10872.008.patch, HDFS-10872.009.patch,
> HDFS-10872.010.patch, jmx-output
>
>
> Add metrics for FSNamesystemLock operations to see, overall, how long each
> operation is holding the lock for. Use MutableRate metrics for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated HDFS-10872: --- Attachment: HDFS-10872.009.patch Looks like YARN-5868 went through faster than expected, reuploading v007 patch as v009 to trigger Jenkins. > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > Attachments: FSLockPerf.java, HDFS-10872.000.patch, > HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch, > HDFS-10872.004.patch, HDFS-10872.005.patch, HDFS-10872.006.patch, > HDFS-10872.007.patch, HDFS-10872.008.patch, HDFS-10872.009.patch > > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated HDFS-10872: --- Attachment: HDFS-10872.008.patch Attaching v008 patch which includes the changes within YARN-5868 just to be able to get a clean Jenkins build. Patch is otherwise identical to v007. > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > Attachments: FSLockPerf.java, HDFS-10872.000.patch, > HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch, > HDFS-10872.004.patch, HDFS-10872.005.patch, HDFS-10872.006.patch, > HDFS-10872.007.patch, HDFS-10872.008.patch > > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-10872: - Target Version/s: 2.7.4 Hadoop Flags: Reviewed Thanks Erik, +1 on v7 patch pending a successful Jenkins run (I just re-triggerred). Will also wait for a day before committing so others can comment. > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > Attachments: FSLockPerf.java, HDFS-10872.000.patch, > HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch, > HDFS-10872.004.patch, HDFS-10872.005.patch, HDFS-10872.006.patch, > HDFS-10872.007.patch > > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated HDFS-10872: --- Attachment: HDFS-10872.007.patch > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > Attachments: FSLockPerf.java, HDFS-10872.000.patch, > HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch, > HDFS-10872.004.patch, HDFS-10872.005.patch, HDFS-10872.006.patch, > HDFS-10872.007.patch > > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated HDFS-10872: --- Attachment: HDFS-10872.006.patch Test failures don't seem related and pass fine locally. Checkstyle is complaining about an import only used within a Javadoc - changed it slightly so that checkstyle won't complain. Attaching v006 patch. > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > Attachments: FSLockPerf.java, HDFS-10872.000.patch, > HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch, > HDFS-10872.004.patch, HDFS-10872.005.patch, HDFS-10872.006.patch > > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HDFS-10872:
---
Attachment: HDFS-10872.005.patch
Now that HADOOP-13782 has provided us with a metrics class which provides fast
concurrent access to {{MutableRate}} metrics ({{MutableRatesWithAggregation}}),
the patch has been refactored to make use of this. This required a minor
modification to {{MetricsRegistry}} to export a
{{newMutableRatesWithAggregation}} method to create a new metrics object in
{{FSNamesystem}} to be able to pass into {{FSNamesystemLock}}. This is
necessary since we want the lock hold metrics to be emitted within the
{{FSNamesystem}} metrics registry but they are generated within
{{FSNamesystemLock}}, so we pass the metric object down into the lock for
modification.
Attaching v005 patch.
> Add MutableRate metrics for FSNamesystemLock operations
> ---
>
> Key: HDFS-10872
> URL: https://issues.apache.org/jira/browse/HDFS-10872
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Erik Krogen
>Assignee: Erik Krogen
> Attachments: FSLockPerf.java, HDFS-10872.000.patch,
> HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch,
> HDFS-10872.004.patch, HDFS-10872.005.patch
>
>
> Add metrics for FSNamesystemLock operations to see, overall, how long each
> operation is holding the lock for. Use MutableRate metrics for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated HDFS-10872: --- Attachment: HDFS-10872.004.patch Attaching v004 to fix checkstyle. Waiting sounds good. Thanks Zhe. > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > Attachments: FSLockPerf.java, HDFS-10872.000.patch, > HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch, > HDFS-10872.004.patch > > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated HDFS-10872: --- Attachment: HDFS-10872.003.patch > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > Attachments: FSLockPerf.java, HDFS-10872.000.patch, > HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch > > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated HDFS-10872: --- Attachment: HDFS-10872.002.patch Attaching v002 patch addressing findbugs and checkstyle. Unit test failure doesn't seem related; passes fine locally. > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > Attachments: FSLockPerf.java, HDFS-10872.000.patch, > HDFS-10872.001.patch, HDFS-10872.002.patch > > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HDFS-10872:
---
Attachment: HDFS-10872.001.patch
Thanks for the review, [~zhz].
1. I agree. Created HDFS-11021 to track.
2. Used a {{final String}} variable for each operation whose name is used more
than once.
3. Per our offline discussion the best place for these metrics to live is
actually {{FSNamesystem}}'s metrics. Thank you for bringing up the issue.
Attaching v001 patch with the above changes and some bug fixes.
> Add MutableRate metrics for FSNamesystemLock operations
> ---
>
> Key: HDFS-10872
> URL: https://issues.apache.org/jira/browse/HDFS-10872
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Erik Krogen
>Assignee: Erik Krogen
> Attachments: FSLockPerf.java, HDFS-10872.000.patch,
> HDFS-10872.001.patch
>
>
> Add metrics for FSNamesystemLock operations to see, overall, how long each
> operation is holding the lock for. Use MutableRate metrics for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated HDFS-10872: --- Status: Patch Available (was: Open) > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > Attachments: FSLockPerf.java, HDFS-10872.000.patch > > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HDFS-10872:
---
Attachment: FSLockPerf.java
Some comments on performance considerations...
I am attaching code ({{FSLockPerf.java}}) that I used to do somewhat
rudimentary microbenchmarking. It's not perfect but hopefully it gives a bit of
an idea of what kind of overhead this may incur. If anyone is interested in
seeing other numbers let me know and I will do my best to generate them. Note
that this feature is disabled by default, so no overhead is incurred for those
not actively opting in to the feature.
I include two different tests, "overall" and "aggTime". In both I focus on the
worst case scenario in which all threads are reader threads, i.e. they are not
hindered by the Namesystem lock and contend solely on metrics. In both cases I
use 200 threads to model what would occur in a highly contested system. Also,
all aggregations involve 50 operations, emulating 50 distinct operation types
occurring at each thread since the last aggregation, which seems a
conservatively high upper bound since most operations are uncommon.
overall tries to be more wholistic but involves a higher degree of variability
since there are actually locks being held and such. This test sets the
aggregation interval at various intervals (including completely disabled and a
high enough interval that aggregation is never triggered) and tests the overall
time it takes each of the 200 threads to complete 500,000 cycles of read
lock/unlock (including all metrics-related operations). Over 1,000 iterations I
got:
{code}
Agg IntervalTotal Time MS (Avg) Total Time MS (StdDev)
0 30518 1777
999 30825 1673
2 30183 1709
1 30272 1681
500030278 1740
100030307 1702
10 30350 1692
{code}
Clearly the metrics processing fits within the noise of locking and such,
especially given that the average of the runs with the logic disabled ended up
being higher than with the logic enabled. Still, these results were not very
satisfying, so I tried to be more specific with aggTime.
aggTime is the more narrow of the two. I assume the local tracking of metrics
is very cheap, simply incrementing a counter within a ThreadLocal, so I focus
on the time to do the more expensive aggregate (involving a {{synchronized}}
method to update the {{MutableRate}} metric). First I run a test with only a
single thread updating metrics, then do the full 200 threads under a few
different conditions: turning on and off aggregation (to get a baseline figure
of performance with many threads running), and including a 1-millisecond sleep
between operations (to emulate slightly less pessimistic conditions of lock
contention). Each thread does 10,000 aggregations and I measure the time per
operation; over 10 trials I got:
{code}
1 aggregation per thread over 100 trials
TestAverage Time (ns) Std Dev (ns)
Single Thread 3107606
No Agg, No Wait 551 551
Agg, No Wait235850 24059
No Agg, Wait1065525 625
Agg, Wait 1158477 8743
{code}
So it seems that even under highly contested conditions an aggregation adds
~100-200 microseconds to the execution path, and without contention only ~3-4
microseconds. Given that a typical aggregation period would be the same as the
metrics collection interval, say 10-60 seconds, this seems reasonable for a
disabled-by-default feature.
> Add MutableRate metrics for FSNamesystemLock operations
> ---
>
> Key: HDFS-10872
> URL: https://issues.apache.org/jira/browse/HDFS-10872
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Erik Krogen
>Assignee: Erik Krogen
> Attachments: FSLockPerf.java, HDFS-10872.000.patch
>
>
> Add metrics for FSNamesystemLock operations to see, overall, how long each
> operation is holding the lock for. Use MutableRate metrics for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated HDFS-10872: --- Attachment: HDFS-10872.000.patch > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > Attachments: HDFS-10872.000.patch > > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated HDFS-10872: --- Status: Open (was: Patch Available) > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated HDFS-10872: --- Status: Patch Available (was: In Progress) > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
