[jira] [Commented] (HADOOP-10949) Write metrics2 sink plugin for Apache Kafka -- KafkaSink
[
https://issues.apache.org/jira/browse/HADOOP-10949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14709595#comment-14709595
]
Ravi Prakash commented on HADOOP-10949:
---
1. {{dateFormat}} and {{timeFormat}} can be static final members of KafkaSink.
2. Any reason you want to initialize {{hostname}} on every {{putMetrics()}}?
3. Please conform to Hadoop coding style and fix indentation: {code}
ProducerRecordInteger, byte[] data
= new ProducerRecordInteger, byte[](
topic,
jsonLines.toString().getBytes());
{code}
4. Please document the expected behavior of the KafkaProducer in putMetrics(),
of putMetrics and also an example record json.
5. {{flush()}} could be a synchronous send() of an empty record or it should
throw a MetricsException too IMHO.
6. Please document the test. Please follow the coding style guide for the test
as well.
Write metrics2 sink plugin for Apache Kafka -- KafkaSink
Key: HADOOP-10949
URL: https://issues.apache.org/jira/browse/HADOOP-10949
Project: Hadoop Common
Issue Type: New Feature
Components: metrics
Reporter: Babak Behzad
Assignee: Babak Behzad
Attachments: HADOOP-10949-1.patch, HADOOP-10949-2.patch,
HADOOP-10949.patch, HADOOP-10949.patch, HADOOP-10949.patch,
HADOOP-10949.patch, HADOOP-10949.patch, HADOOP-10949.patch,
HADOOP-10949.patch, HADOOP-10949.patch, HADOOP-10949.patch,
HADOOP-10949.patch, HADOOP-10949.patch
Write a metrics2 sink plugin for Hadoop to send metrics directly to Apache
Kafka in addition to the current, Graphite
([Hadoop-9704|https://issues.apache.org/jira/browse/HADOOP-9704]), Ganglia
and File sinks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-12349) Misleading debug message in generic_locate_patch
Kengo Seki created HADOOP-12349:
---
Summary: Misleading debug message in generic_locate_patch
Key: HADOOP-12349
URL: https://issues.apache.org/jira/browse/HADOOP-12349
Project: Hadoop Common
Issue Type: Sub-task
Components: yetus
Affects Versions: HADOOP-12111
Reporter: Kengo Seki
Priority: Minor
The following message in builtin-bugsystem.sh is duplicated with
jira_locate_patch and misleading.
{code}
yetus_debug jira_locate_patch: not a JIRA.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12321) Make JvmPauseMonitor to AbstractService
[
https://issues.apache.org/jira/browse/HADOOP-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil G updated HADOOP-12321:
-
Status: Open (was: Patch Available)
Make JvmPauseMonitor to AbstractService
---
Key: HADOOP-12321
URL: https://issues.apache.org/jira/browse/HADOOP-12321
Project: Hadoop Common
Issue Type: New Feature
Affects Versions: 2.8.0
Reporter: Steve Loughran
Assignee: Sunil G
Attachments: 0001-HADOOP-12321.patch
Original Estimate: 1h
Remaining Estimate: 1h
The new JVM pause monitor has been written with its own start/stop lifecycle
which has already proven brittle to both ordering of operations and, even
after HADOOP-12313, is not thread safe (both start and stop are potentially
re-entrant).
It also requires every class which supports the monitor to add another field
and perform the lifecycle operations in its own lifecycle, which, for all
Yarn services, is the YARN app lifecycle (as implemented in Hadoop common)
Making the monitor a subclass of {{AbstractService}} and moving the
init/start stop operations in {{serviceInit()}}, {{serviceStart()}}
{{serviceStop()}} methods will fix the concurrency and state model issues,
and make it trivial to add as a child to any YARN service which subclasses
{{CompositeService}} (most the NM and RM apps) will be able to hook up the
monitor simply by creating one in the ctor and adding it as a child.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12321) Make JvmPauseMonitor to AbstractService
[
https://issues.apache.org/jira/browse/HADOOP-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14709633#comment-14709633
]
Steve Loughran commented on HADOOP-12321:
-
If you patch NN and DN to move to the new methods, create an HDFS JIRA with
the same entire patch as this one, I'll apply them together
Make JvmPauseMonitor to AbstractService
---
Key: HADOOP-12321
URL: https://issues.apache.org/jira/browse/HADOOP-12321
Project: Hadoop Common
Issue Type: New Feature
Affects Versions: 2.8.0
Reporter: Steve Loughran
Assignee: Sunil G
Attachments: 0001-HADOOP-12321.patch
Original Estimate: 1h
Remaining Estimate: 1h
The new JVM pause monitor has been written with its own start/stop lifecycle
which has already proven brittle to both ordering of operations and, even
after HADOOP-12313, is not thread safe (both start and stop are potentially
re-entrant).
It also requires every class which supports the monitor to add another field
and perform the lifecycle operations in its own lifecycle, which, for all
Yarn services, is the YARN app lifecycle (as implemented in Hadoop common)
Making the monitor a subclass of {{AbstractService}} and moving the
init/start stop operations in {{serviceInit()}}, {{serviceStart()}}
{{serviceStop()}} methods will fix the concurrency and state model issues,
and make it trivial to add as a child to any YARN service which subclasses
{{CompositeService}} (most the NM and RM apps) will be able to hook up the
monitor simply by creating one in the ctor and adding it as a child.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12257) rework build tool support; add gradle
[
https://issues.apache.org/jira/browse/HADOOP-12257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14709693#comment-14709693
]
Allen Wittenauer commented on HADOOP-12257:
---
As I've dug more into this, it's pretty clear that the current life cycle is
confusing when dealing with non-Java build systems. For example, gradle and
autoconf require some work prior to the compile (e.g., gradlew bootstrap)
whereas maven and ant require some work after the compile (e.g., mvn install).
As part of this patch, I want to throw out the current life cycle and rework
it. This is the draft I've got written up and looking for some feedback:
{code}
init
git checkout
check_reexec
...
preapply
find_changed_files
branch
precompile
gradlew bootstrap, autoconf, make deps, ...
compile
source - (.o, .jar, ...)
javac, scalac
postcompile
ant/maven install
postbuild
shellcheck, eclipse, ...
javadoc, scaladoc, ...
clean
apply
find_changed_files
calcdiff
patch (effectively the same as branch)
precompile
compile
postcompile
postbuild
runtests
report
{code}
rework build tool support; add gradle
-
Key: HADOOP-12257
URL: https://issues.apache.org/jira/browse/HADOOP-12257
Project: Hadoop Common
Issue Type: Sub-task
Components: yetus
Affects Versions: HADOOP-12111
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
Attachments: HADOOP-12257.HADOOP-12111.00.patch
We need to rework build tool support to be pluggable as well as add gradle
support so that we cover more of the ecosystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12344) validateSocketPathSecurity0 message could be better
[
https://issues.apache.org/jira/browse/HADOOP-12344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14709665#comment-14709665
]
Hadoop QA commented on HADOOP-12344:
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 5m 27s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 8m 5s | There were no new javac warning
messages. |
| {color:green}+1{color} | release audit | 0m 19s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 26s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 31s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | common tests | 22m 28s | Tests failed in
hadoop-common. |
| | | 38m 21s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.net.unix.TestDomainSocket |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12751651/HADOOP-12344.patch |
| Optional Tests | javac unit |
| git revision | trunk / feaf034 |
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7518/artifact/patchprocess/testrun_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7518/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7518/console |
This message was automatically generated.
validateSocketPathSecurity0 message could be better
---
Key: HADOOP-12344
URL: https://issues.apache.org/jira/browse/HADOOP-12344
Project: Hadoop Common
Issue Type: Improvement
Components: net
Reporter: Casey Brotherton
Assignee: Casey Brotherton
Priority: Trivial
Attachments: HADOOP-12344.patch
When a socket path does not have the correct permissions, an error is thrown.
That error just has the failing component of the path and not the entire path
of the socket.
The entire path of the socket could be printed out to allow for a direct
check of the permissions of the entire path.
{code}
java.io.IOException: the path component: '/' is world-writable. Its
permissions are 0077. Please fix this or select a different socket path.
at
org.apache.hadoop.net.unix.DomainSocket.validateSocketPathSecurity0(Native
Method)
at
org.apache.hadoop.net.unix.DomainSocket.bindAndListen(DomainSocket.java:189)
...
{code}
The error message could also provide the socket path:
{code}
java.io.IOException: the path component: '/' is world-writable. Its
permissions are 0077. Please fix this or select a different socket path than
'/var/run/hdfs-sockets/dn'
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12321) Make JvmPauseMonitor to AbstractService
[
https://issues.apache.org/jira/browse/HADOOP-12321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil G updated HADOOP-12321:
-
Attachment: 0002-HADOOP-12321.patch
Thank you [~steve_l]
I have updated the patch by changing test case.
Also created 3 subjiras in HDFS, YARN and MapReduce separately and linked to
this main Jira. If all are applied together, it will compile successfully.
Make JvmPauseMonitor to AbstractService
---
Key: HADOOP-12321
URL: https://issues.apache.org/jira/browse/HADOOP-12321
Project: Hadoop Common
Issue Type: New Feature
Affects Versions: 2.8.0
Reporter: Steve Loughran
Assignee: Sunil G
Attachments: 0001-HADOOP-12321.patch, 0002-HADOOP-12321.patch
Original Estimate: 1h
Remaining Estimate: 1h
The new JVM pause monitor has been written with its own start/stop lifecycle
which has already proven brittle to both ordering of operations and, even
after HADOOP-12313, is not thread safe (both start and stop are potentially
re-entrant).
It also requires every class which supports the monitor to add another field
and perform the lifecycle operations in its own lifecycle, which, for all
Yarn services, is the YARN app lifecycle (as implemented in Hadoop common)
Making the monitor a subclass of {{AbstractService}} and moving the
init/start stop operations in {{serviceInit()}}, {{serviceStart()}}
{{serviceStop()}} methods will fix the concurrency and state model issues,
and make it trivial to add as a child to any YARN service which subclasses
{{CompositeService}} (most the NM and RM apps) will be able to hook up the
monitor simply by creating one in the ctor and adding it as a child.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12344) validateSocketPathSecurity0 message could be better
[
https://issues.apache.org/jira/browse/HADOOP-12344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14709757#comment-14709757
]
Casey Brotherton commented on HADOOP-12344:
---
Neat. the testcase failure is based on matching the error message.
Sorry I missed that, will revise.
I will also make some wiki page entries to talk about why the path needs to be
secure, and some commands to check the permissions.
Thanks,
Casey
validateSocketPathSecurity0 message could be better
---
Key: HADOOP-12344
URL: https://issues.apache.org/jira/browse/HADOOP-12344
Project: Hadoop Common
Issue Type: Improvement
Components: net
Reporter: Casey Brotherton
Assignee: Casey Brotherton
Priority: Trivial
Attachments: HADOOP-12344.patch
When a socket path does not have the correct permissions, an error is thrown.
That error just has the failing component of the path and not the entire path
of the socket.
The entire path of the socket could be printed out to allow for a direct
check of the permissions of the entire path.
{code}
java.io.IOException: the path component: '/' is world-writable. Its
permissions are 0077. Please fix this or select a different socket path.
at
org.apache.hadoop.net.unix.DomainSocket.validateSocketPathSecurity0(Native
Method)
at
org.apache.hadoop.net.unix.DomainSocket.bindAndListen(DomainSocket.java:189)
...
{code}
The error message could also provide the socket path:
{code}
java.io.IOException: the path component: '/' is world-writable. Its
permissions are 0077. Please fix this or select a different socket path than
'/var/run/hdfs-sockets/dn'
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12325) RPC Metrics : Add the ability track and log slow RPCs
[
https://issues.apache.org/jira/browse/HADOOP-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14709745#comment-14709745
]
Hadoop QA commented on HADOOP-12325:
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 44s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 11m 8s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 13m 21s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 37s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 29s | The applied patch generated 2
new checkstyle issues (total was 511, now 512). |
| {color:green}+1{color} | whitespace | 0m 3s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 2m 16s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 51s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 25s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | common tests | 25m 7s | Tests failed in
hadoop-common. |
| | | 75m 6s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.ipc.TestProtoBufRpc |
| | hadoop.security.token.delegation.web.TestWebDelegationToken |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12751978/HADOOP-12325.005.test.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / feaf034 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7519/artifact/patchprocess/diffcheckstylehadoop-common.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7519/artifact/patchprocess/testrun_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7519/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7519/console |
This message was automatically generated.
RPC Metrics : Add the ability track and log slow RPCs
-
Key: HADOOP-12325
URL: https://issues.apache.org/jira/browse/HADOOP-12325
Project: Hadoop Common
Issue Type: Improvement
Components: ipc, metrics
Affects Versions: 2.7.1
Reporter: Anu Engineer
Assignee: Anu Engineer
Attachments: Callers of WritableRpcEngine.call.png,
HADOOP-12325.001.patch, HADOOP-12325.002.patch, HADOOP-12325.003.patch,
HADOOP-12325.004.patch, HADOOP-12325.005.patch, HADOOP-12325.005.test.patch
This JIRA proposes to add a counter called RpcSlowCalls and also a
configuration setting that allows users to log really slow RPCs. Slow RPCs
are RPCs that fall at 99th percentile. This is useful to troubleshoot why
certain services like name node freezes under heavy load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12325) RPC Metrics : Add the ability track and log slow RPCs
[
https://issues.apache.org/jira/browse/HADOOP-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HADOOP-12325:
--
Attachment: HADOOP-12325.006.patch
[~ajisakaa] Thanks for your review and changes to the test file. Please see my
comments below
bq. 1. Would you add a whitespace before took in the log message?
fixed.
bq. 2. After running the regression test locally, I can't see any logs about
sleep RPC.
On my machine if I open the file
org.apache.hadoop.ipc.TestProtoBufRpc-output.txt in the sure-fire reports
directory, I am able to see the following line.
{code}
2015-08-24 10:52:16,713 WARN ipc.Server (Server.java:logSlowRpcCalls(438)) -
Slow RPC : sleep took 3004 milliseconds to process from client 10.0.1.35:57223
{code}
bq. Attaching a patch to verify that the slow call is logged. Now the test
fails.
With the new call {code} long after = getLongCounter(RpcSlowCalls,
rpcMetrics); {code} somehow the mocking layer is still returning the old
snap-shotted value. I have modified the tests to call server layer directly
and tests are now behaving as expected.
RPC Metrics : Add the ability track and log slow RPCs
-
Key: HADOOP-12325
URL: https://issues.apache.org/jira/browse/HADOOP-12325
Project: Hadoop Common
Issue Type: Improvement
Components: ipc, metrics
Affects Versions: 2.7.1
Reporter: Anu Engineer
Assignee: Anu Engineer
Attachments: Callers of WritableRpcEngine.call.png,
HADOOP-12325.001.patch, HADOOP-12325.002.patch, HADOOP-12325.003.patch,
HADOOP-12325.004.patch, HADOOP-12325.005.patch, HADOOP-12325.005.test.patch,
HADOOP-12325.006.patch
This JIRA proposes to add a counter called RpcSlowCalls and also a
configuration setting that allows users to log really slow RPCs. Slow RPCs
are RPCs that fall at 99th percentile. This is useful to troubleshoot why
certain services like name node freezes under heavy load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-12350) WASB Logging: Improve WASB Logging around deletes, reads and writes
Dushyanth created HADOOP-12350: -- Summary: WASB Logging: Improve WASB Logging around deletes, reads and writes Key: HADOOP-12350 URL: https://issues.apache.org/jira/browse/HADOOP-12350 Project: Hadoop Common Issue Type: Improvement Components: tools Reporter: Dushyanth Fix For: 2.8.0 Logging around the WASB component is very limited and it is disabled by default. This improvement is created to add logging around Reads, Writes and Deletes when Azure Storage Exception to capture the blobs that hit the exception. This information is useful while communicating with the Azure storage team for debugging purposes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12350) WASB Logging: Improve WASB Logging around deletes, reads and writes
[ https://issues.apache.org/jira/browse/HADOOP-12350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12350: --- Assignee: Dushyanth WASB Logging: Improve WASB Logging around deletes, reads and writes --- Key: HADOOP-12350 URL: https://issues.apache.org/jira/browse/HADOOP-12350 Project: Hadoop Common Issue Type: Improvement Components: tools Reporter: Dushyanth Assignee: Dushyanth Fix For: 2.8.0 Logging around the WASB component is very limited and it is disabled by default. This improvement is created to add logging around Reads, Writes and Deletes when Azure Storage Exception to capture the blobs that hit the exception. This information is useful while communicating with the Azure storage team for debugging purposes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12350) WASB Logging: Improve WASB Logging around deletes, reads and writes
[ https://issues.apache.org/jira/browse/HADOOP-12350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12350: --- Attachment: 0001-HADOOP-12350-Added-WASB-Logging-Statement.patch Added first iteration. Patch contains: 1) Logging statement around Read, Write and Deletes. 2) Removed un-necessary imports WASB Logging: Improve WASB Logging around deletes, reads and writes --- Key: HADOOP-12350 URL: https://issues.apache.org/jira/browse/HADOOP-12350 Project: Hadoop Common Issue Type: Improvement Components: tools Reporter: Dushyanth Assignee: Dushyanth Fix For: 2.8.0 Attachments: 0001-HADOOP-12350-Added-WASB-Logging-Statement.patch Logging around the WASB component is very limited and it is disabled by default. This improvement is created to add logging around Reads, Writes and Deletes when Azure Storage Exception to capture the blobs that hit the exception. This information is useful while communicating with the Azure storage team for debugging purposes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12325) RPC Metrics : Add the ability track and log slow RPCs
[
https://issues.apache.org/jira/browse/HADOOP-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14708852#comment-14708852
]
Akira AJISAKA commented on HADOOP-12325:
Thanks [~anu] for creating the patch. Two comments:
1. Would you add a whitespace before took in the log message?
{code}
+LOG.warn(
+Slow RPC : + methodName + took + processingTime +
+ milliseconds to process from client + client);
{code}
2. After running the regression test locally, I can't see any logs about sleep
RPC. I'm thinking the test should output the log as follows:
{noformat}
2015-08-24 15:16:31,771 WARN ipc.Server (Server.java:logSlowRpcCalls(438)) -
Slow RPC : sleep 1001 milliseconds to process from client 127.0.0.1:52257
{noformat}
Would you fix this?
RPC Metrics : Add the ability track and log slow RPCs
-
Key: HADOOP-12325
URL: https://issues.apache.org/jira/browse/HADOOP-12325
Project: Hadoop Common
Issue Type: Improvement
Components: ipc, metrics
Affects Versions: 2.7.1
Reporter: Anu Engineer
Assignee: Anu Engineer
Attachments: Callers of WritableRpcEngine.call.png,
HADOOP-12325.001.patch, HADOOP-12325.002.patch, HADOOP-12325.003.patch,
HADOOP-12325.004.patch, HADOOP-12325.005.patch
This JIRA proposes to add a counter called RpcSlowCalls and also a
configuration setting that allows users to log really slow RPCs. Slow RPCs
are RPCs that fall at 99th percentile. This is useful to troubleshoot why
certain services like name node freezes under heavy load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12344) validateSocketPathSecurity0 message could be better
[
https://issues.apache.org/jira/browse/HADOOP-12344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14708952#comment-14708952
]
Harsh J commented on HADOOP-12344:
--
This would be very useful to add to make it easier to understand what path the
error's talking of. I know its pre-existed, but I wonder what's the use of
asking people to select a different path (bound to have the same problem
somehow).
Patch's changes look good, but I wonder if we should just drop that part of the
message and instruct better on how to fix it?
A message such as:
{code}
java.io.IOException: the path component: '/' in '/var/run/hdfs-sockets/dn' is
world-writable. Its permissions are 0077. Expected permission is XXX. Please
fix this.
{code}
(or similar) … would be more actionable, no?
validateSocketPathSecurity0 message could be better
---
Key: HADOOP-12344
URL: https://issues.apache.org/jira/browse/HADOOP-12344
Project: Hadoop Common
Issue Type: Improvement
Components: net
Reporter: Casey Brotherton
Assignee: Casey Brotherton
Priority: Trivial
Attachments: HADOOP-12344.patch
When a socket path does not have the correct permissions, an error is thrown.
That error just has the failing component of the path and not the entire path
of the socket.
The entire path of the socket could be printed out to allow for a direct
check of the permissions of the entire path.
{code}
java.io.IOException: the path component: '/' is world-writable. Its
permissions are 0077. Please fix this or select a different socket path.
at
org.apache.hadoop.net.unix.DomainSocket.validateSocketPathSecurity0(Native
Method)
at
org.apache.hadoop.net.unix.DomainSocket.bindAndListen(DomainSocket.java:189)
...
{code}
The error message could also provide the socket path:
{code}
java.io.IOException: the path component: '/' is world-writable. Its
permissions are 0077. Please fix this or select a different socket path than
'/var/run/hdfs-sockets/dn'
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12325) RPC Metrics : Add the ability track and log slow RPCs
[ https://issues.apache.org/jira/browse/HADOOP-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-12325: --- Attachment: HADOOP-12325.005.test.patch Attaching a patch to verify that the slow call is logged. Now the test fails. RPC Metrics : Add the ability track and log slow RPCs - Key: HADOOP-12325 URL: https://issues.apache.org/jira/browse/HADOOP-12325 Project: Hadoop Common Issue Type: Improvement Components: ipc, metrics Affects Versions: 2.7.1 Reporter: Anu Engineer Assignee: Anu Engineer Attachments: Callers of WritableRpcEngine.call.png, HADOOP-12325.001.patch, HADOOP-12325.002.patch, HADOOP-12325.003.patch, HADOOP-12325.004.patch, HADOOP-12325.005.patch, HADOOP-12325.005.test.patch This JIRA proposes to add a counter called RpcSlowCalls and also a configuration setting that allows users to log really slow RPCs. Slow RPCs are RPCs that fall at 99th percentile. This is useful to troubleshoot why certain services like name node freezes under heavy load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12325) RPC Metrics : Add the ability track and log slow RPCs
[ https://issues.apache.org/jira/browse/HADOOP-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710087#comment-14710087 ] Xiaoyu Yao commented on HADOOP-12325: - Thanks [~anu] updating the patch! The Jenkins result looks good to me. I will commit it shortly. Ignore the checkstyle issue Variable 'rpcSlowCalls' must be private and have accessor methods. since it is inconsistent with the existing code. RPC Metrics : Add the ability track and log slow RPCs - Key: HADOOP-12325 URL: https://issues.apache.org/jira/browse/HADOOP-12325 Project: Hadoop Common Issue Type: Improvement Components: ipc, metrics Affects Versions: 2.7.1 Reporter: Anu Engineer Assignee: Anu Engineer Attachments: Callers of WritableRpcEngine.call.png, HADOOP-12325.001.patch, HADOOP-12325.002.patch, HADOOP-12325.003.patch, HADOOP-12325.004.patch, HADOOP-12325.005.patch, HADOOP-12325.005.test.patch, HADOOP-12325.006.patch This JIRA proposes to add a counter called RpcSlowCalls and also a configuration setting that allows users to log really slow RPCs. Slow RPCs are RPCs that fall at 99th percentile. This is useful to troubleshoot why certain services like name node freezes under heavy load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12325) RPC Metrics : Add the ability track and log slow RPCs
[ https://issues.apache.org/jira/browse/HADOOP-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HADOOP-12325: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) Thanks [~anu] for the contribution and [~ajisakaa] for the review. I've commit the change to trunk and branch-2. RPC Metrics : Add the ability track and log slow RPCs - Key: HADOOP-12325 URL: https://issues.apache.org/jira/browse/HADOOP-12325 Project: Hadoop Common Issue Type: Improvement Components: ipc, metrics Affects Versions: 2.7.1 Reporter: Anu Engineer Assignee: Anu Engineer Fix For: 2.8.0 Attachments: Callers of WritableRpcEngine.call.png, HADOOP-12325.001.patch, HADOOP-12325.002.patch, HADOOP-12325.003.patch, HADOOP-12325.004.patch, HADOOP-12325.005.patch, HADOOP-12325.005.test.patch, HADOOP-12325.006.patch This JIRA proposes to add a counter called RpcSlowCalls and also a configuration setting that allows users to log really slow RPCs. Slow RPCs are RPCs that fall at 99th percentile. This is useful to troubleshoot why certain services like name node freezes under heavy load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-10949) Write metrics2 sink plugin for Apache Kafka -- KafkaSink
[
https://issues.apache.org/jira/browse/HADOOP-10949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14709938#comment-14709938
]
Hadoop QA commented on HADOOP-10949:
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 43s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 43s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 44s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 9s | The applied patch generated 4
new checkstyle issues (total was 0, now 4). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 39s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 1m 55s | The patch appears to introduce 2
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | common tests | 22m 29s | Tests passed in
hadoop-common. |
| | | 62m 24s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-common |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12751864/HADOOP-10949-2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / b5ce87f |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7520/artifact/patchprocess/diffcheckstylehadoop-common.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7520/artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7520/artifact/patchprocess/testrun_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7520/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7520/console |
This message was automatically generated.
Write metrics2 sink plugin for Apache Kafka -- KafkaSink
Key: HADOOP-10949
URL: https://issues.apache.org/jira/browse/HADOOP-10949
Project: Hadoop Common
Issue Type: New Feature
Components: metrics
Reporter: Babak Behzad
Assignee: Babak Behzad
Attachments: HADOOP-10949-1.patch, HADOOP-10949-2.patch,
HADOOP-10949.patch, HADOOP-10949.patch, HADOOP-10949.patch,
HADOOP-10949.patch, HADOOP-10949.patch, HADOOP-10949.patch,
HADOOP-10949.patch, HADOOP-10949.patch, HADOOP-10949.patch,
HADOOP-10949.patch, HADOOP-10949.patch
Write a metrics2 sink plugin for Hadoop to send metrics directly to Apache
Kafka in addition to the current, Graphite
([Hadoop-9704|https://issues.apache.org/jira/browse/HADOOP-9704]), Ganglia
and File sinks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12351) Can't run test-patch with start-build-env.sh
[ https://issues.apache.org/jira/browse/HADOOP-12351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710128#comment-14710128 ] Allen Wittenauer commented on HADOOP-12351: --- bq. dev-support/test-patch.sh script requires a clean environment with no extra files --dirty-workspace Can't run test-patch with start-build-env.sh Key: HADOOP-12351 URL: https://issues.apache.org/jira/browse/HADOOP-12351 Project: Hadoop Common Issue Type: Bug Components: test, yetus Reporter: Jakob Homan Priority: Minor The Docker instance started by start-build-env.sh drops the user into a root directory wherein other directories (say ~/download) are not accessible, so one cannot pull patches to test into it. The dev-support/test-patch.sh script requires a clean environment with no extra files, such as the patch to test. Between these two restrictions, one can't use the docker env to run test-patch. We should either -v a volume where patches can be stashed, or allow test-patch.sh to ignore patch files' existences. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12257) rework build tool support; add gradle
[ https://issues.apache.org/jira/browse/HADOOP-12257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710101#comment-14710101 ] Allen Wittenauer commented on HADOOP-12257: --- This might help: || Phase || Maven || Ant || Gradle || autoconf || | precompile | mvn validate? | | rebuild gradlew, gradle -b bootstrap | ./configure; make deps? | | compile (see below) | mvn compile | ant jar/tar/whatever | gradle (module to compile) | make (module) | | postcompile | mvn install | | | | | postbuild | mvn javadoc | ant javadoc | gradle (module):scaladoc | ? | * The compile phase is a loop around multijdk. Sets javahome, then does whatever it needs to do a build based upon the build tool. After the build is finished, it calls javac, scalac, etc, to look at its logs, take a look at the object files, etc. These methods no longer do the builds themselves are only checking the state. This removes a lot of extra compiles. This should make it possible to to turn the loop into a matrix in the future. java_home=x,y,z scala_home=a,b,c cc=1,2,3 * build tools now have plugins. Many post-OO revolution build systems appear to do the work of calling the compiler and massaging the output to its own tastes (mvn plugins, gradle plugins, even cmake if you think about it.). So the build systems have a lot more control and should know how things should be output. So some systems like javac allow for the build tool to override its generic warning counters. rework build tool support; add gradle - Key: HADOOP-12257 URL: https://issues.apache.org/jira/browse/HADOOP-12257 Project: Hadoop Common Issue Type: Sub-task Components: yetus Affects Versions: HADOOP-12111 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-12257.HADOOP-12111.00.patch We need to rework build tool support to be pluggable as well as add gradle support so that we cover more of the ecosystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12325) RPC Metrics : Add the ability track and log slow RPCs
[ https://issues.apache.org/jira/browse/HADOOP-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710113#comment-14710113 ] Xiaoyu Yao commented on HADOOP-12325: - Also resolve HDFS-5180 as a dup of HADOOP-12325. RPC Metrics : Add the ability track and log slow RPCs - Key: HADOOP-12325 URL: https://issues.apache.org/jira/browse/HADOOP-12325 Project: Hadoop Common Issue Type: Improvement Components: ipc, metrics Affects Versions: 2.7.1 Reporter: Anu Engineer Assignee: Anu Engineer Fix For: 2.8.0 Attachments: Callers of WritableRpcEngine.call.png, HADOOP-12325.001.patch, HADOOP-12325.002.patch, HADOOP-12325.003.patch, HADOOP-12325.004.patch, HADOOP-12325.005.patch, HADOOP-12325.005.test.patch, HADOOP-12325.006.patch This JIRA proposes to add a counter called RpcSlowCalls and also a configuration setting that allows users to log really slow RPCs. Slow RPCs are RPCs that fall at 99th percentile. This is useful to troubleshoot why certain services like name node freezes under heavy load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12351) Can't run test-patch with start-build-env.sh
[ https://issues.apache.org/jira/browse/HADOOP-12351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710129#comment-14710129 ] Allen Wittenauer commented on HADOOP-12351: --- Probably also worth mentioning that newer versions of test-patch support --docker to fire off its own container to do work in. Can't run test-patch with start-build-env.sh Key: HADOOP-12351 URL: https://issues.apache.org/jira/browse/HADOOP-12351 Project: Hadoop Common Issue Type: Bug Components: test, yetus Reporter: Jakob Homan Priority: Minor The Docker instance started by start-build-env.sh drops the user into a root directory wherein other directories (say ~/download) are not accessible, so one cannot pull patches to test into it. The dev-support/test-patch.sh script requires a clean environment with no extra files, such as the patch to test. Between these two restrictions, one can't use the docker env to run test-patch. We should either -v a volume where patches can be stashed, or allow test-patch.sh to ignore patch files' existences. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12257) rework build tool support; add gradle
[ https://issues.apache.org/jira/browse/HADOOP-12257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710039#comment-14710039 ] Allen Wittenauer commented on HADOOP-12257: --- bq. part of init is detecting the branch and changing, right? Yup. It's all the setup work that happens prior to anything actually changing on the repo. bq. calling branch something like validate will help folks coming from maven. I'm curious if it would hinder ant/gradle/autoconf/whatever. While we're using the term 'branch' today even in the existing code, it doesn't have to stay that way. But anyway, it's called branch to reflect the state of the repo: this is the copy of source from the branch. Log files are written such that they are xyz-testname.log, where xyz is either branch or patch. Then tests can compare branch vs. patch and do whatever. bq. Do you have notes already about how the proposed lifecycle matches up on the non-java build tools? Not yet. Let me write up something regarding the current state of my dev tree and it might make more sense. :) rework build tool support; add gradle - Key: HADOOP-12257 URL: https://issues.apache.org/jira/browse/HADOOP-12257 Project: Hadoop Common Issue Type: Sub-task Components: yetus Affects Versions: HADOOP-12111 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-12257.HADOOP-12111.00.patch We need to rework build tool support to be pluggable as well as add gradle support so that we cover more of the ecosystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12325) RPC Metrics : Add the ability track and log slow RPCs
[
https://issues.apache.org/jira/browse/HADOOP-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710023#comment-14710023
]
Hadoop QA commented on HADOOP-12325:
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 9s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 7m 58s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 52s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 7s | The applied patch generated 2
new checkstyle issues (total was 511, now 512). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 27s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 35s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 56s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | common tests | 22m 49s | Tests passed in
hadoop-common. |
| | | 63m 22s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752053/HADOOP-12325.006.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / b5ce87f |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7521/artifact/patchprocess/diffcheckstylehadoop-common.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7521/artifact/patchprocess/testrun_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7521/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/7521/console |
This message was automatically generated.
RPC Metrics : Add the ability track and log slow RPCs
-
Key: HADOOP-12325
URL: https://issues.apache.org/jira/browse/HADOOP-12325
Project: Hadoop Common
Issue Type: Improvement
Components: ipc, metrics
Affects Versions: 2.7.1
Reporter: Anu Engineer
Assignee: Anu Engineer
Attachments: Callers of WritableRpcEngine.call.png,
HADOOP-12325.001.patch, HADOOP-12325.002.patch, HADOOP-12325.003.patch,
HADOOP-12325.004.patch, HADOOP-12325.005.patch, HADOOP-12325.005.test.patch,
HADOOP-12325.006.patch
This JIRA proposes to add a counter called RpcSlowCalls and also a
configuration setting that allows users to log really slow RPCs. Slow RPCs
are RPCs that fall at 99th percentile. This is useful to troubleshoot why
certain services like name node freezes under heavy load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12257) rework build tool support; add gradle
[ https://issues.apache.org/jira/browse/HADOOP-12257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710003#comment-14710003 ] Sean Busbey commented on HADOOP-12257: -- I think it will help folks grok the lifecycle if we have rough mapping to extant lifecycles (it'll also help make clear why all of the existing build tool lifecycles are insufficient--I presume due to the fact that we have mirrored phases pre/post patch). Initial thoughts: * part of init is detecting the branch and changing, right? * calling branch something like validate will help [folks coming from maven|https://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html#Lifecycle_Reference]. I'm curious if it would hinder ant/gradle/autoconf/whatever. Do you have notes already about how the proposed lifecycle matches up on the non-java build tools? rework build tool support; add gradle - Key: HADOOP-12257 URL: https://issues.apache.org/jira/browse/HADOOP-12257 Project: Hadoop Common Issue Type: Sub-task Components: yetus Affects Versions: HADOOP-12111 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-12257.HADOOP-12111.00.patch We need to rework build tool support to be pluggable as well as add gradle support so that we cover more of the ecosystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12351) Can't run test-patch with start-build-env.sh
Jakob Homan created HADOOP-12351: Summary: Can't run test-patch with start-build-env.sh Key: HADOOP-12351 URL: https://issues.apache.org/jira/browse/HADOOP-12351 Project: Hadoop Common Issue Type: Bug Components: test, yetus Reporter: Jakob Homan Priority: Minor The Docker instance started by start-build-env.sh drops the user into a root directory wherein other directories (say ~/download) are not accessible, so one cannot pull patches to test into it. The dev-support/test-patch.sh script requires a clean environment with no extra files, such as the patch to test. Between these two restrictions, one can't use the docker env to run test-patch. We should either -v a volume where patches can be stashed, or allow test-patch.sh to ignore patch files' existences. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12325) RPC Metrics : Add the ability track and log slow RPCs
[ https://issues.apache.org/jira/browse/HADOOP-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710382#comment-14710382 ] Hudson commented on HADOOP-12325: - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #301 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/301/]) HADOOP-12325. RPC Metrics : Add the ability track and log slow RPCs. Contributed by Anu Engineer (xyao: rev 48774d0a45d95557affbd6bbaf8035cc9575ef36) * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableStat.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/MetricsAsserts.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestProtoBufRpc.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java * hadoop-common-project/hadoop-common/src/test/proto/test_rpc_service.proto * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WritableRpcEngine.java * hadoop-common-project/hadoop-common/src/test/proto/test.proto * hadoop-common-project/hadoop-common/src/main/resources/core-default.xml * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java RPC Metrics : Add the ability track and log slow RPCs - Key: HADOOP-12325 URL: https://issues.apache.org/jira/browse/HADOOP-12325 Project: Hadoop Common Issue Type: Improvement Components: ipc, metrics Affects Versions: 2.7.1 Reporter: Anu Engineer Assignee: Anu Engineer Fix For: 2.8.0 Attachments: Callers of WritableRpcEngine.call.png, HADOOP-12325.001.patch, HADOOP-12325.002.patch, HADOOP-12325.003.patch, HADOOP-12325.004.patch, HADOOP-12325.005.patch, HADOOP-12325.005.test.patch, HADOOP-12325.006.patch This JIRA proposes to add a counter called RpcSlowCalls and also a configuration setting that allows users to log really slow RPCs. Slow RPCs are RPCs that fall at 99th percentile. This is useful to troubleshoot why certain services like name node freezes under heavy load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12325) RPC Metrics : Add the ability track and log slow RPCs
[ https://issues.apache.org/jira/browse/HADOOP-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710308#comment-14710308 ] Hudson commented on HADOOP-12325: - FAILURE: Integrated in Hadoop-trunk-Commit #8345 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8345/]) HADOOP-12325. RPC Metrics : Add the ability track and log slow RPCs. Contributed by Anu Engineer (xyao: rev 48774d0a45d95557affbd6bbaf8035cc9575ef36) * hadoop-common-project/hadoop-common/src/main/resources/core-default.xml * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java * hadoop-common-project/hadoop-common/src/test/proto/test_rpc_service.proto * hadoop-common-project/hadoop-common/src/test/proto/test.proto * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WritableRpcEngine.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestProtoBufRpc.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableStat.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/MetricsAsserts.java RPC Metrics : Add the ability track and log slow RPCs - Key: HADOOP-12325 URL: https://issues.apache.org/jira/browse/HADOOP-12325 Project: Hadoop Common Issue Type: Improvement Components: ipc, metrics Affects Versions: 2.7.1 Reporter: Anu Engineer Assignee: Anu Engineer Fix For: 2.8.0 Attachments: Callers of WritableRpcEngine.call.png, HADOOP-12325.001.patch, HADOOP-12325.002.patch, HADOOP-12325.003.patch, HADOOP-12325.004.patch, HADOOP-12325.005.patch, HADOOP-12325.005.test.patch, HADOOP-12325.006.patch This JIRA proposes to add a counter called RpcSlowCalls and also a configuration setting that allows users to log really slow RPCs. Slow RPCs are RPCs that fall at 99th percentile. This is useful to troubleshoot why certain services like name node freezes under heavy load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12325) RPC Metrics : Add the ability track and log slow RPCs
[ https://issues.apache.org/jira/browse/HADOOP-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710368#comment-14710368 ] Hudson commented on HADOOP-12325: - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #305 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/305/]) HADOOP-12325. RPC Metrics : Add the ability track and log slow RPCs. Contributed by Anu Engineer (xyao: rev 48774d0a45d95557affbd6bbaf8035cc9575ef36) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WritableRpcEngine.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/MetricsAsserts.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/test/proto/test.proto * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java * hadoop-common-project/hadoop-common/src/test/proto/test_rpc_service.proto * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestProtoBufRpc.java * hadoop-common-project/hadoop-common/src/main/resources/core-default.xml * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableStat.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java RPC Metrics : Add the ability track and log slow RPCs - Key: HADOOP-12325 URL: https://issues.apache.org/jira/browse/HADOOP-12325 Project: Hadoop Common Issue Type: Improvement Components: ipc, metrics Affects Versions: 2.7.1 Reporter: Anu Engineer Assignee: Anu Engineer Fix For: 2.8.0 Attachments: Callers of WritableRpcEngine.call.png, HADOOP-12325.001.patch, HADOOP-12325.002.patch, HADOOP-12325.003.patch, HADOOP-12325.004.patch, HADOOP-12325.005.patch, HADOOP-12325.005.test.patch, HADOOP-12325.006.patch This JIRA proposes to add a counter called RpcSlowCalls and also a configuration setting that allows users to log really slow RPCs. Slow RPCs are RPCs that fall at 99th percentile. This is useful to troubleshoot why certain services like name node freezes under heavy load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12325) RPC Metrics : Add the ability track and log slow RPCs
[ https://issues.apache.org/jira/browse/HADOOP-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710421#comment-14710421 ] Hudson commented on HADOOP-12325: - FAILURE: Integrated in Hadoop-Yarn-trunk #1034 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1034/]) HADOOP-12325. RPC Metrics : Add the ability track and log slow RPCs. Contributed by Anu Engineer (xyao: rev 48774d0a45d95557affbd6bbaf8035cc9575ef36) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableStat.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WritableRpcEngine.java * hadoop-common-project/hadoop-common/src/main/resources/core-default.xml * hadoop-common-project/hadoop-common/src/test/proto/test_rpc_service.proto * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/MetricsAsserts.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java * hadoop-common-project/hadoop-common/src/test/proto/test.proto * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestProtoBufRpc.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java * hadoop-common-project/hadoop-common/CHANGES.txt RPC Metrics : Add the ability track and log slow RPCs - Key: HADOOP-12325 URL: https://issues.apache.org/jira/browse/HADOOP-12325 Project: Hadoop Common Issue Type: Improvement Components: ipc, metrics Affects Versions: 2.7.1 Reporter: Anu Engineer Assignee: Anu Engineer Fix For: 2.8.0 Attachments: Callers of WritableRpcEngine.call.png, HADOOP-12325.001.patch, HADOOP-12325.002.patch, HADOOP-12325.003.patch, HADOOP-12325.004.patch, HADOOP-12325.005.patch, HADOOP-12325.005.test.patch, HADOOP-12325.006.patch This JIRA proposes to add a counter called RpcSlowCalls and also a configuration setting that allows users to log really slow RPCs. Slow RPCs are RPCs that fall at 99th percentile. This is useful to troubleshoot why certain services like name node freezes under heavy load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12257) rework build tool support; add gradle
[ https://issues.apache.org/jira/browse/HADOOP-12257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710206#comment-14710206 ] Allen Wittenauer commented on HADOOP-12257: --- Actually, mvn install should be in the precompile. So maybe post compile isn't needed. Hmm... rework build tool support; add gradle - Key: HADOOP-12257 URL: https://issues.apache.org/jira/browse/HADOOP-12257 Project: Hadoop Common Issue Type: Sub-task Components: yetus Affects Versions: HADOOP-12111 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-12257.HADOOP-12111.00.patch We need to rework build tool support to be pluggable as well as add gradle support so that we cover more of the ecosystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12325) RPC Metrics : Add the ability track and log slow RPCs
[ https://issues.apache.org/jira/browse/HADOOP-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710462#comment-14710462 ] Hudson commented on HADOOP-12325: - FAILURE: Integrated in Hadoop-Hdfs-trunk #2231 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2231/]) HADOOP-12325. RPC Metrics : Add the ability track and log slow RPCs. Contributed by Anu Engineer (xyao: rev 48774d0a45d95557affbd6bbaf8035cc9575ef36) * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java * hadoop-common-project/hadoop-common/src/test/proto/test_rpc_service.proto * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/MetricsAsserts.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableStat.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WritableRpcEngine.java * hadoop-common-project/hadoop-common/src/test/proto/test.proto * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestProtoBufRpc.java * hadoop-common-project/hadoop-common/src/main/resources/core-default.xml * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java RPC Metrics : Add the ability track and log slow RPCs - Key: HADOOP-12325 URL: https://issues.apache.org/jira/browse/HADOOP-12325 Project: Hadoop Common Issue Type: Improvement Components: ipc, metrics Affects Versions: 2.7.1 Reporter: Anu Engineer Assignee: Anu Engineer Fix For: 2.8.0 Attachments: Callers of WritableRpcEngine.call.png, HADOOP-12325.001.patch, HADOOP-12325.002.patch, HADOOP-12325.003.patch, HADOOP-12325.004.patch, HADOOP-12325.005.patch, HADOOP-12325.005.test.patch, HADOOP-12325.006.patch This JIRA proposes to add a counter called RpcSlowCalls and also a configuration setting that allows users to log really slow RPCs. Slow RPCs are RPCs that fall at 99th percentile. This is useful to troubleshoot why certain services like name node freezes under heavy load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12325) RPC Metrics : Add the ability track and log slow RPCs
[ https://issues.apache.org/jira/browse/HADOOP-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710593#comment-14710593 ] Akira AJISAKA commented on HADOOP-12325: Thanks [~anu] for the update. Late +1 from me. Thanks [~xyao] for reviewing and committing the patch. RPC Metrics : Add the ability track and log slow RPCs - Key: HADOOP-12325 URL: https://issues.apache.org/jira/browse/HADOOP-12325 Project: Hadoop Common Issue Type: Improvement Components: ipc, metrics Affects Versions: 2.7.1 Reporter: Anu Engineer Assignee: Anu Engineer Fix For: 2.8.0 Attachments: Callers of WritableRpcEngine.call.png, HADOOP-12325.001.patch, HADOOP-12325.002.patch, HADOOP-12325.003.patch, HADOOP-12325.004.patch, HADOOP-12325.005.patch, HADOOP-12325.005.test.patch, HADOOP-12325.006.patch This JIRA proposes to add a counter called RpcSlowCalls and also a configuration setting that allows users to log really slow RPCs. Slow RPCs are RPCs that fall at 99th percentile. This is useful to troubleshoot why certain services like name node freezes under heavy load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12325) RPC Metrics : Add the ability track and log slow RPCs
[ https://issues.apache.org/jira/browse/HADOOP-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710516#comment-14710516 ] Hudson commented on HADOOP-12325: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2250 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2250/]) HADOOP-12325. RPC Metrics : Add the ability track and log slow RPCs. Contributed by Anu Engineer (xyao: rev 48774d0a45d95557affbd6bbaf8035cc9575ef36) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/MetricsAsserts.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java * hadoop-common-project/hadoop-common/src/test/proto/test.proto * hadoop-common-project/hadoop-common/src/main/resources/core-default.xml * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestProtoBufRpc.java * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableStat.java * hadoop-common-project/hadoop-common/src/test/proto/test_rpc_service.proto * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WritableRpcEngine.java RPC Metrics : Add the ability track and log slow RPCs - Key: HADOOP-12325 URL: https://issues.apache.org/jira/browse/HADOOP-12325 Project: Hadoop Common Issue Type: Improvement Components: ipc, metrics Affects Versions: 2.7.1 Reporter: Anu Engineer Assignee: Anu Engineer Fix For: 2.8.0 Attachments: Callers of WritableRpcEngine.call.png, HADOOP-12325.001.patch, HADOOP-12325.002.patch, HADOOP-12325.003.patch, HADOOP-12325.004.patch, HADOOP-12325.005.patch, HADOOP-12325.005.test.patch, HADOOP-12325.006.patch This JIRA proposes to add a counter called RpcSlowCalls and also a configuration setting that allows users to log really slow RPCs. Slow RPCs are RPCs that fall at 99th percentile. This is useful to troubleshoot why certain services like name node freezes under heavy load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12352) Delay in checkpointing Trash can leave trash for 2 intervals before deleting
[ https://issues.apache.org/jira/browse/HADOOP-12352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated HADOOP-12352: - Resolution: Fixed Fix Version/s: 2.8.0 Target Version/s: (was: 2.8.0, 3.0.0) Status: Resolved (was: Patch Available) Committed to branch-2 and trunk. Thank you for the find and fix Casey! Hope to see many more. Delay in checkpointing Trash can leave trash for 2 intervals before deleting Key: HADOOP-12352 URL: https://issues.apache.org/jira/browse/HADOOP-12352 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.7.1 Reporter: Casey Brotherton Assignee: Casey Brotherton Priority: Trivial Fix For: 2.8.0 Attachments: HDFS-8118.001.patch, HDFS-8118.patch When the fs.trash.checkpoint.interval and the fs.trash.interval are set non-zero and the same, it is possible for trash to be left for two intervals. The TrashPolicyDefault will use a floor and ceiling function to ensure that the Trash will be checkpointed every interval of minutes. Each user's trash is checkpointed individually. The time resolution of the checkpoint timestamp is to the second. If the seconds switch while one user is checkpointing, then the next user's timestamp will be later. This will cause the next user's checkpoint to not be deleted at the next interval. I have recreated this in a lab cluster I also have a suggestion for a patch that I can upload later tonight after testing it further. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Moved] (HADOOP-12352) Delay in checkpointing Trash can leave trash for 2 intervals before deleting
[ https://issues.apache.org/jira/browse/HADOOP-12352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J moved HDFS-8118 to HADOOP-12352: Affects Version/s: (was: 2.7.1) 2.7.1 Target Version/s: 2.8.0, 3.0.0 (was: 3.0.0, 2.8.0) Key: HADOOP-12352 (was: HDFS-8118) Project: Hadoop Common (was: Hadoop HDFS) Delay in checkpointing Trash can leave trash for 2 intervals before deleting Key: HADOOP-12352 URL: https://issues.apache.org/jira/browse/HADOOP-12352 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.7.1 Reporter: Casey Brotherton Assignee: Casey Brotherton Priority: Trivial Attachments: HDFS-8118.001.patch, HDFS-8118.patch When the fs.trash.checkpoint.interval and the fs.trash.interval are set non-zero and the same, it is possible for trash to be left for two intervals. The TrashPolicyDefault will use a floor and ceiling function to ensure that the Trash will be checkpointed every interval of minutes. Each user's trash is checkpointed individually. The time resolution of the checkpoint timestamp is to the second. If the seconds switch while one user is checkpointing, then the next user's timestamp will be later. This will cause the next user's checkpoint to not be deleted at the next interval. I have recreated this in a lab cluster I also have a suggestion for a patch that I can upload later tonight after testing it further. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-9382) Add dfs mv overwrite option
[ https://issues.apache.org/jira/browse/HADOOP-9382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710510#comment-14710510 ] Gautam Gopalakrishnan commented on HADOOP-9382: --- [~keegan] Could you respond to Colin's questions please, we can work on getting this feature committed in the next few weeks. Add dfs mv overwrite option --- Key: HADOOP-9382 URL: https://issues.apache.org/jira/browse/HADOOP-9382 Project: Hadoop Common Issue Type: Improvement Reporter: Keegan Witt Assignee: Suresh Srinivas Priority: Minor Labels: BB2015-05-TBR Attachments: HADOOP-9382.1.patch, HADOOP-9382.2.patch, HADOOP-9382.patch Add a -f option to allow overwriting existing destinations in dfs mv command. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12325) RPC Metrics : Add the ability track and log slow RPCs
[ https://issues.apache.org/jira/browse/HADOOP-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710542#comment-14710542 ] Hudson commented on HADOOP-12325: - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #293 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/293/]) HADOOP-12325. RPC Metrics : Add the ability track and log slow RPCs. Contributed by Anu Engineer (xyao: rev 48774d0a45d95557affbd6bbaf8035cc9575ef36) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java * hadoop-common-project/hadoop-common/src/test/proto/test_rpc_service.proto * hadoop-common-project/hadoop-common/src/main/resources/core-default.xml * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableStat.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/MetricsAsserts.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestProtoBufRpc.java * hadoop-common-project/hadoop-common/src/test/proto/test.proto * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WritableRpcEngine.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java RPC Metrics : Add the ability track and log slow RPCs - Key: HADOOP-12325 URL: https://issues.apache.org/jira/browse/HADOOP-12325 Project: Hadoop Common Issue Type: Improvement Components: ipc, metrics Affects Versions: 2.7.1 Reporter: Anu Engineer Assignee: Anu Engineer Fix For: 2.8.0 Attachments: Callers of WritableRpcEngine.call.png, HADOOP-12325.001.patch, HADOOP-12325.002.patch, HADOOP-12325.003.patch, HADOOP-12325.004.patch, HADOOP-12325.005.patch, HADOOP-12325.005.test.patch, HADOOP-12325.006.patch This JIRA proposes to add a counter called RpcSlowCalls and also a configuration setting that allows users to log really slow RPCs. Slow RPCs are RPCs that fall at 99th percentile. This is useful to troubleshoot why certain services like name node freezes under heavy load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
