Steve Loughran created HADOOP-18844:
---------------------------------------
Summary: Add a way to get the IOStatistics of active filesystems
in long-lived processes
Key: HADOOP-18844
URL: https://issues.apache.org/jira/browse/HADOOP-18844
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/azure, fs/s3
Affects Versions: 3.3.9
Reporter: Steve Loughran
we can configure abfs and s3a to print out the filesystem IOStats in close(),
but this doesn't let us see the state of long-lived processes, including
latencies, error rates, and other important information.
Proposed: we add a means by which all cached filesystems which provide
IOStatistics can log them to a file/console *while the process continues to run*
* a configurable scheduled thread which enumerates all open filesystem
instances, gets their iostats and if non empty dumps to a log...log
configuration can set that to the console or elsewhere. Good: can run without
intervention; bad: one more thread to manage.
* option to add a signal handler which will do this whenever a SIGUSR2 is
raised. JVM signal handling is a bit of a brittle feature, so it should be off
by default.
{{org.apache.hadoop.service.launcher.IrqHandler}} supports signal wire-up.
good: gives us an equivalent of kill -quit to get a view of iostats. bad: you
need to be on the machine
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
