[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[ https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13712421#comment-13712421 ] Trevor Lorimer commented on HADOOP-9562: Hi Cos, that is correct, in HDFS-5004 I created a new JMX bean to provide the same information. Create REST interface for HDFS health data -- Key: HADOOP-9562 URL: https://issues.apache.org/jira/browse/HADOOP-9562 Project: Hadoop Common Issue Type: Improvement Components: fs Affects Versions: 3.0.0, 2.0.4-alpha Reporter: Trevor Lorimer Priority: Minor Attachments: HADOOP-9562.diff The HDFS health screen (dfshealth.jsp) displays basic Version, Security and Health information concerning the NameNode, currently this information is accessible from classes in the org.apache.hadoop,hdfs.server.namenode package and cannot be accessed outside the NameNode. This becomes prevalent if the data is required to be displayed using a new user interface. The proposal is to create a REST interface to expose the NameNode information displayed on dfshealth.jsp using GET methods. Wrapper classes will be created to serve the data to the REST root resource within the hadoop-hdfs project. This will enable the HDFS health screen information to be accessed remotely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[ https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13711231#comment-13711231 ] Trevor Lorimer commented on HADOOP-9562: I have a created the following ticket to add the functionality using a new NameNode JMX bean. https://issues.apache.org/jira/browse/HDFS-5004 Create REST interface for HDFS health data -- Key: HADOOP-9562 URL: https://issues.apache.org/jira/browse/HADOOP-9562 Project: Hadoop Common Issue Type: Improvement Components: fs Affects Versions: 3.0.0, 2.0.4-alpha Reporter: Trevor Lorimer Priority: Minor Attachments: HADOOP-9562.diff The HDFS health screen (dfshealth.jsp) displays basic Version, Security and Health information concerning the NameNode, currently this information is accessible from classes in the org.apache.hadoop,hdfs.server.namenode package and cannot be accessed outside the NameNode. This becomes prevalent if the data is required to be displayed using a new user interface. The proposal is to create a REST interface to expose the NameNode information displayed on dfshealth.jsp using GET methods. Wrapper classes will be created to serve the data to the REST root resource within the hadoop-hdfs project. This will enable the HDFS health screen information to be accessed remotely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[ https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13711286#comment-13711286 ] Konstantin Boudnik commented on HADOOP-9562: Trevor, my understanding is that HDFS-5004 would be an alternative approach to this new REST interface, right? Create REST interface for HDFS health data -- Key: HADOOP-9562 URL: https://issues.apache.org/jira/browse/HADOOP-9562 Project: Hadoop Common Issue Type: Improvement Components: fs Affects Versions: 3.0.0, 2.0.4-alpha Reporter: Trevor Lorimer Priority: Minor Attachments: HADOOP-9562.diff The HDFS health screen (dfshealth.jsp) displays basic Version, Security and Health information concerning the NameNode, currently this information is accessible from classes in the org.apache.hadoop,hdfs.server.namenode package and cannot be accessed outside the NameNode. This becomes prevalent if the data is required to be displayed using a new user interface. The proposal is to create a REST interface to expose the NameNode information displayed on dfshealth.jsp using GET methods. Wrapper classes will be created to serve the data to the REST root resource within the hadoop-hdfs project. This will enable the HDFS health screen information to be accessed remotely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[ https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13707601#comment-13707601 ] Luke Lu commented on HADOOP-9562: - I think this patch is a little controversial in that # It duplicates functionality already mostly provided by existing mechanisms: JMX and its HTTP bridge. # The jersey stuff doesn't follow Hadoop authentication convention and doesn't honor hadoop.http.filter.initializers to setup SPNEGO and other custom auth filters. # No authorization mechanism at all. # Augmenting existing JMX mechanism is trivial (NameNodeInfoMXBean doesn't have to be implemented by FSNameSystem, it was so for convenience). # Returning data at the whole bean level is a limitation of the current HTTP bridge, which is addressed by HADOOP-9160, which adopts a new popular HTTP/JSON bridge (Jolokia) that supports advanced querying mechanism with both batch and attribute level queries along with first class language bindings for Perl, Python and Javascript. Create REST interface for HDFS health data -- Key: HADOOP-9562 URL: https://issues.apache.org/jira/browse/HADOOP-9562 Project: Hadoop Common Issue Type: Improvement Components: fs Affects Versions: 3.0.0, 2.0.4-alpha Reporter: Trevor Lorimer Priority: Minor Attachments: HADOOP-9562.diff The HDFS health screen (dfshealth.jsp) displays basic Version, Security and Health information concerning the NameNode, currently this information is accessible from classes in the org.apache.hadoop,hdfs.server.namenode package and cannot be accessed outside the NameNode. This becomes prevalent if the data is required to be displayed using a new user interface. The proposal is to create a REST interface to expose the NameNode information displayed on dfshealth.jsp using GET methods. Wrapper classes will be created to serve the data to the REST root resource within the hadoop-hdfs project. This will enable the HDFS health screen information to be accessed remotely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[ https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702102#comment-13702102 ] Suresh Srinivas commented on HADOOP-9562: - Trevor, can you please update the summary and the description of the jira? I will review the patch shortly. Create REST interface for HDFS health data -- Key: HADOOP-9562 URL: https://issues.apache.org/jira/browse/HADOOP-9562 Project: Hadoop Common Issue Type: Improvement Components: fs Affects Versions: 3.0.0, 2.0.4-alpha Reporter: Trevor Lorimer Priority: Minor Attachments: HADOOP-9562.diff The HDFS health screen (dfshealth.jsp) displays basic Version, Security and Health information concerning the NameNode, currently this information is accessible from classes in the org.apache.hadoop,hdfs.server.namenode package and cannot be accessed outside the NameNode. This becomes prevalent if the data is required to be displayed using a new user interface. The proposal is to create a REST interface to expose all the information displayed on dfshealth.jsp using GET methods. Wrapper classes will be created to serve the data to the REST root resource within the hadoop-hdfs project. This will enable the HDFS health screen information to be accessed remotely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[ https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702103#comment-13702103 ] Suresh Srinivas commented on HADOOP-9562: - After looking at the patch briefly, what is the reason for now doing this similar to all the other JMX/Http mechanism? That should require very little change. Create REST interface for HDFS health data -- Key: HADOOP-9562 URL: https://issues.apache.org/jira/browse/HADOOP-9562 Project: Hadoop Common Issue Type: Improvement Components: fs Affects Versions: 3.0.0, 2.0.4-alpha Reporter: Trevor Lorimer Priority: Minor Attachments: HADOOP-9562.diff The HDFS health screen (dfshealth.jsp) displays basic Version, Security and Health information concerning the NameNode, currently this information is accessible from classes in the org.apache.hadoop,hdfs.server.namenode package and cannot be accessed outside the NameNode. This becomes prevalent if the data is required to be displayed using a new user interface. The proposal is to create a REST interface to expose all the information displayed on dfshealth.jsp using GET methods. Wrapper classes will be created to serve the data to the REST root resource within the hadoop-hdfs project. This will enable the HDFS health screen information to be accessed remotely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[ https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13702136#comment-13702136 ] Trevor Lorimer commented on HADOOP-9562: Hi Suresh, The current JMX mechanism exposes data from the FSNameSystem class, but I needed the Status information from the NameNode class which is currently not available in JMX. The other reason was CorruptFiles could contain a List of hundreds of entries which seemed out of place with the amount of current data returned in JMX. So we decided on adding a REST interface to HDFS, that was not much more work than updating the JMX. Thanks, Trevor Create REST interface for HDFS health data -- Key: HADOOP-9562 URL: https://issues.apache.org/jira/browse/HADOOP-9562 Project: Hadoop Common Issue Type: Improvement Components: fs Affects Versions: 3.0.0, 2.0.4-alpha Reporter: Trevor Lorimer Priority: Minor Attachments: HADOOP-9562.diff The HDFS health screen (dfshealth.jsp) displays basic Version, Security and Health information concerning the NameNode, currently this information is accessible from classes in the org.apache.hadoop,hdfs.server.namenode package and cannot be accessed outside the NameNode. This becomes prevalent if the data is required to be displayed using a new user interface. The proposal is to create a REST interface to expose all the information displayed on dfshealth.jsp using GET methods. Wrapper classes will be created to serve the data to the REST root resource within the hadoop-hdfs project. This will enable the HDFS health screen information to be accessed remotely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[ https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13700548#comment-13700548 ] Trevor Lorimer commented on HADOOP-9562: Resubmitted with tests Create REST interface for HDFS health data -- Key: HADOOP-9562 URL: https://issues.apache.org/jira/browse/HADOOP-9562 Project: Hadoop Common Issue Type: Improvement Components: fs Affects Versions: 2.0.4-alpha Reporter: Trevor Lorimer Priority: Minor Attachments: HADOOP-9562.diff The HDFS health screen (dfshealth.jsp) displays basic Version, Security and Health information concerning the NameNode, currently this information is accessible from classes in the org.apache.hadoop,hdfs.server.namenode package and cannot be accessed outside the NameNode. This becomes prevalent if the data is required to be displayed using a new user interface. The proposal is to create a REST interface to expose all the information displayed on dfshealth.jsp using GET methods. Wrapper classes will be created to serve the data to the REST root resource within the hadoop-hdfs project. This will enable the HDFS health screen information to be accessed remotely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[
https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13700583#comment-13700583
]
Hadoop QA commented on HADOOP-9562:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12590971/HADOOP-9562.diff
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 2
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2743//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2743//console
This message is automatically generated.
Create REST interface for HDFS health data
--
Key: HADOOP-9562
URL: https://issues.apache.org/jira/browse/HADOOP-9562
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Affects Versions: 2.0.4-alpha
Reporter: Trevor Lorimer
Priority: Minor
Attachments: HADOOP-9562.diff
The HDFS health screen (dfshealth.jsp) displays basic Version, Security and
Health information concerning the NameNode, currently this information is
accessible from classes in the org.apache.hadoop,hdfs.server.namenode package
and cannot be accessed outside the NameNode. This becomes prevalent if the
data is required to be displayed using a new user interface.
The proposal is to create a REST interface to expose all the information
displayed on dfshealth.jsp using GET methods. Wrapper classes will be created
to serve the data to the REST root resource within the hadoop-hdfs project.
This will enable the HDFS health screen information to be accessed remotely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[
https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13700608#comment-13700608
]
Hadoop QA commented on HADOOP-9562:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12590974/HADOOP-9562.diff
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 2
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2744//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2744//console
This message is automatically generated.
Create REST interface for HDFS health data
--
Key: HADOOP-9562
URL: https://issues.apache.org/jira/browse/HADOOP-9562
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Affects Versions: 2.0.4-alpha
Reporter: Trevor Lorimer
Priority: Minor
Attachments: HADOOP-9562.diff
The HDFS health screen (dfshealth.jsp) displays basic Version, Security and
Health information concerning the NameNode, currently this information is
accessible from classes in the org.apache.hadoop,hdfs.server.namenode package
and cannot be accessed outside the NameNode. This becomes prevalent if the
data is required to be displayed using a new user interface.
The proposal is to create a REST interface to expose all the information
displayed on dfshealth.jsp using GET methods. Wrapper classes will be created
to serve the data to the REST root resource within the hadoop-hdfs project.
This will enable the HDFS health screen information to be accessed remotely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[
https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13701072#comment-13701072
]
Hadoop QA commented on HADOOP-9562:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12591025/HADOOP-9562.diff
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2745//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2745//console
This message is automatically generated.
Create REST interface for HDFS health data
--
Key: HADOOP-9562
URL: https://issues.apache.org/jira/browse/HADOOP-9562
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Affects Versions: 2.0.4-alpha
Reporter: Trevor Lorimer
Priority: Minor
Attachments: HADOOP-9562.diff
The HDFS health screen (dfshealth.jsp) displays basic Version, Security and
Health information concerning the NameNode, currently this information is
accessible from classes in the org.apache.hadoop,hdfs.server.namenode package
and cannot be accessed outside the NameNode. This becomes prevalent if the
data is required to be displayed using a new user interface.
The proposal is to create a REST interface to expose all the information
displayed on dfshealth.jsp using GET methods. Wrapper classes will be created
to serve the data to the REST root resource within the hadoop-hdfs project.
This will enable the HDFS health screen information to be accessed remotely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[ https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13701147#comment-13701147 ] Konstantin Boudnik commented on HADOOP-9562: The patch looks good at the first glance - I will read more into it over the weekend. Create REST interface for HDFS health data -- Key: HADOOP-9562 URL: https://issues.apache.org/jira/browse/HADOOP-9562 Project: Hadoop Common Issue Type: Improvement Components: fs Affects Versions: 3.0.0, 2.0.4-alpha Reporter: Trevor Lorimer Priority: Minor Attachments: HADOOP-9562.diff The HDFS health screen (dfshealth.jsp) displays basic Version, Security and Health information concerning the NameNode, currently this information is accessible from classes in the org.apache.hadoop,hdfs.server.namenode package and cannot be accessed outside the NameNode. This becomes prevalent if the data is required to be displayed using a new user interface. The proposal is to create a REST interface to expose all the information displayed on dfshealth.jsp using GET methods. Wrapper classes will be created to serve the data to the REST root resource within the hadoop-hdfs project. This will enable the HDFS health screen information to be accessed remotely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[
https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13700162#comment-13700162
]
Trevor Lorimer commented on HADOOP-9562:
Added 2 REST calls to HDFS (Status and CorruptFiles).
CorruptFiles returns a List of Strings of corrupt file locations.
Status returns data from the NameNode that was inaccessible from FSNameSystem.
status: {
role: NameNode
state: active
hostAndPort: localhost:8020
securityEnabled: false
}
Create REST interface for HDFS health data
--
Key: HADOOP-9562
URL: https://issues.apache.org/jira/browse/HADOOP-9562
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Affects Versions: 2.0.4-alpha
Reporter: Trevor Lorimer
Priority: Minor
Attachments: HADOOP-9562.diff
The HDFS health screen (dfshealth.jsp) displays basic Version, Security and
Health information concerning the NameNode, currently this information is
accessible from classes in the org.apache.hadoop,hdfs.server.namenode package
and cannot be accessed outside the NameNode. This becomes prevalent if the
data is required to be displayed using a new user interface.
The proposal is to create a REST interface to expose all the information
displayed on dfshealth.jsp using GET methods. Wrapper classes will be created
to serve the data to the REST root resource within the hadoop-hdfs project.
This will enable the HDFS health screen information to be accessed remotely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[
https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13700195#comment-13700195
]
Hadoop QA commented on HADOOP-9562:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12590902/HADOOP-9562.diff
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 2
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2742//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2742//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2742//console
This message is automatically generated.
Create REST interface for HDFS health data
--
Key: HADOOP-9562
URL: https://issues.apache.org/jira/browse/HADOOP-9562
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Affects Versions: 2.0.4-alpha
Reporter: Trevor Lorimer
Priority: Minor
Attachments: HADOOP-9562.diff
The HDFS health screen (dfshealth.jsp) displays basic Version, Security and
Health information concerning the NameNode, currently this information is
accessible from classes in the org.apache.hadoop,hdfs.server.namenode package
and cannot be accessed outside the NameNode. This becomes prevalent if the
data is required to be displayed using a new user interface.
The proposal is to create a REST interface to expose all the information
displayed on dfshealth.jsp using GET methods. Wrapper classes will be created
to serve the data to the REST root resource within the hadoop-hdfs project.
This will enable the HDFS health screen information to be accessed remotely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[ https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665631#comment-13665631 ] Trevor Lorimer commented on HADOOP-9562: Hi, Thanks Philip about 75% of the information can be obtained from JMX. I only need to make these additions to pre-existing JMX interfaces (I will link this to a new JIRA): Started time, Compiled info, Jvm MaxHeap, MaxNonHeap Node Usage stats (i.e. Min, Median, Max, stdev), Number of decommissioned Live and Dead nodes, Journal Status The only information I have had difficulty obtaining is the status header (e.g. NameNode 'localhost:8020' (active)). This information is contained within the NameNode object which does not seem to be exposed through JMX and cannot be referenced from FSNamesystem. Create REST interface for HDFS health data -- Key: HADOOP-9562 URL: https://issues.apache.org/jira/browse/HADOOP-9562 Project: Hadoop Common Issue Type: Improvement Components: fs Affects Versions: 2.0.4-alpha Reporter: Trevor Lorimer Priority: Minor Attachments: 0001-HAD-162-Final-revision-refactor.patch The HDFS health screen (dfshealth.jsp) displays basic Version, Security and Health information concerning the NameNode, currently this information is accessible from classes in the org.apache.hadoop,hdfs.server.namenode package and cannot be accessed outside the NameNode. This becomes prevalent if the data is required to be displayed using a new user interface. The proposal is to create a REST interface to expose all the information displayed on dfshealth.jsp using GET methods. Wrapper classes will be created to serve the data to the REST root resource within the hadoop-hdfs project. This will enable the HDFS health screen information to be accessed remotely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[
https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13663064#comment-13663064
]
Philip Zeyliger commented on HADOOP-9562:
-
Y'all are aware that
{code}
GET http://namenode:50070/jmx?qry=Hadoop:service=NameNode,name=NameNodeInfo
{code}
returns something along the lines of
{code}
{
beans : [ {
name : Hadoop:service=NameNode,name=NameNodeInfo,
modelerType : org.apache.hadoop.hdfs.server.namenode.FSNamesystem,
Threads : 60,
Total : 39793714372608,
ClusterId : cluster19,
BlockPoolId : BP-1187031495-172.29.122.20-1355867299076,
Used : 19269710336134,
PercentUsed : 48.424004,
PercentRemaining : 31.148054,
Version : 2.0.0-cdh4.3.0-SNAPSHOT,
rcb6dc0d3d99891eb095dc6577a440d7dde067789,
Free : 12394968694784,
Safemode : ,
UpgradeFinalized : true,
NonDfsUsedSpace : 8129035341690,
BlockPoolUsedSpace : 19269710336134,
PercentBlockPoolUsed : 48.424004,
TotalBlocks : 64237,
TotalFiles : 27708,
NumberOfMissingBlocks : 0,
LiveNodes :
{\p0431.mtv.cloudera.com\:{\numBlocks\:32249,\usedSpace\:3220494794752,\lastContact\:1,\capacity\:5688127021056,\nonDfsUsedSpace\:560239116288,\adminState\:\In
Service\},\p0429.mtv.cloudera.com\:{\numBlocks\:25709,\usedSpace\:2468956155904,\lastContact\:2,\capacity\:5688127021056,\nonDfsUsedSpace\:1509498798080,\adminState\:\In
Service\},\p0427.mtv.cloudera.com\:{\numBlocks\:16919,\usedSpace\:2056487919683,\lastContact\:0,\capacity\:5676539633664,\nonDfsUsedSpace\:1999484977085,\adminState\:\In
Service\},\p0430.mtv.cloudera.com\:{\numBlocks\:31258,\usedSpace\:3117177036800,\lastContact\:2,\capacity\:5688127021056,\nonDfsUsedSpace\:987432075264,\adminState\:\In
Service\},\p0432.mtv.cloudera.com\:{\numBlocks\:20877,\usedSpace\:1899904856064,\lastContact\:0,\capacity\:5688127021056,\nonDfsUsedSpace\:1620254515200,\adminState\:\In
Service\},\p0433.mtv.cloudera.com\:{\numBlocks\:31995,\usedSpace\:3204067667968,\lastContact\:1,\capacity\:5688127021056,\nonDfsUsedSpace\:172768587776,\adminState\:\In
Service\},\p0428.mtv.cloudera.com\:{\numBlocks\:33680,\usedSpace\:3302621904963,\lastContact\:0,\capacity\:5676539633664,\nonDfsUsedSpace\:1279357271997,\adminState\:\In
Service\}},
DeadNodes : {},
DecomNodes : {},
NameDirStatuses :
{\failed\:{},\active\:{\/data/1/dfs2/nn\:\IMAGE_AND_EDITS\,\/data/2/dfs2/nn\:\IMAGE_AND_EDITS\}}
} ]
}
{code}
Is that sufficient? I'd rather whatever missing information be added to
existing JMX beans, which already accessible via HTTP, than new equivalent APIs
be added.
There's also a java API in DFSClient
{code}
$hdfs dfsadmin -report
Configured Capacity: 39793714372608 (36.19 TB)
Present Capacity: 31664679067648 (28.80 TB)
DFS Remaining: 12394968727552 (11.27 TB)
DFS Used: 19269710340096 (17.53 TB)
DFS Used%: 60.86%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
-
report: Access denied for user philip. Superuser privilege is required
{code}
Create REST interface for HDFS health data
--
Key: HADOOP-9562
URL: https://issues.apache.org/jira/browse/HADOOP-9562
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Affects Versions: 2.0.4-alpha
Reporter: Trevor Lorimer
Priority: Minor
Attachments: 0001-HAD-162-Final-revision-refactor.patch
The HDFS health screen (dfshealth.jsp) displays basic Version, Security and
Health information concerning the NameNode, currently this information is
accessible from classes in the org.apache.hadoop,hdfs.server.namenode package
and cannot be accessed outside the NameNode. This becomes prevalent if the
data is required to be displayed using a new user interface.
The proposal is to create a REST interface to expose all the information
displayed on dfshealth.jsp using GET methods. Wrapper classes will be created
to serve the data to the REST root resource within the hadoop-hdfs project.
This will enable the HDFS health screen information to be accessed remotely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[ https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13663265#comment-13663265 ] Konstantin Shvachko commented on HADOOP-9562: - I think Philip's and Suresh's suggestion makes sense. Let's see what is missing in JMX and add it if anything. Create REST interface for HDFS health data -- Key: HADOOP-9562 URL: https://issues.apache.org/jira/browse/HADOOP-9562 Project: Hadoop Common Issue Type: Improvement Components: fs Affects Versions: 2.0.4-alpha Reporter: Trevor Lorimer Priority: Minor Attachments: 0001-HAD-162-Final-revision-refactor.patch The HDFS health screen (dfshealth.jsp) displays basic Version, Security and Health information concerning the NameNode, currently this information is accessible from classes in the org.apache.hadoop,hdfs.server.namenode package and cannot be accessed outside the NameNode. This becomes prevalent if the data is required to be displayed using a new user interface. The proposal is to create a REST interface to expose all the information displayed on dfshealth.jsp using GET methods. Wrapper classes will be created to serve the data to the REST root resource within the hadoop-hdfs project. This will enable the HDFS health screen information to be accessed remotely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[
https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662586#comment-13662586
]
Hadoop QA commented on HADOOP-9562:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12583939/0001-HAD-162-Final-revision-refactor.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2552//console
This message is automatically generated.
Create REST interface for HDFS health data
--
Key: HADOOP-9562
URL: https://issues.apache.org/jira/browse/HADOOP-9562
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Affects Versions: 2.0.4-alpha
Reporter: Trevor Lorimer
Priority: Minor
Attachments: 0001-HAD-162-Final-revision-refactor.patch
The HDFS health screen (dfshealth.jsp) displays basic Version, Security and
Health information concerning the NameNode, currently this information is
accessible from classes in the org.apache.hadoop,hdfs.server.namenode package
and cannot be accessed outside the NameNode. This becomes prevalent if the
data is required to be displayed using a new user interface.
The proposal is to create a REST interface to expose all the information
displayed on dfshealth.jsp using GET methods. Wrapper classes will be created
to serve the data to the REST root resource within the hadoop-hdfs project.
This will enable the HDFS health screen information to be accessed remotely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[
https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662669#comment-13662669
]
Konstantin Boudnik commented on HADOOP-9562:
Trevor, for a patch to go trough automatic testing system it needs to be
generated against trunk.
Also, a quick look at the patch shows that you have some irrelevant changes in
it:
- .../hadoop-common/src/main/docs/releasenotes.html seems to be included
because it has some CRLF issues in branch-2 as the result of autogeneration or
something.
- the licenses _have_ to be ASL 2.0 and they have to present in all new files
- class {{NodesInfo}} is empty. Is this expected?
Please address these issue in the second version. BTW, you can keep the same
name of the patch file and JIRA will automatically detect new version.
Create REST interface for HDFS health data
--
Key: HADOOP-9562
URL: https://issues.apache.org/jira/browse/HADOOP-9562
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Affects Versions: 2.0.4-alpha
Reporter: Trevor Lorimer
Priority: Minor
Attachments: 0001-HAD-162-Final-revision-refactor.patch
The HDFS health screen (dfshealth.jsp) displays basic Version, Security and
Health information concerning the NameNode, currently this information is
accessible from classes in the org.apache.hadoop,hdfs.server.namenode package
and cannot be accessed outside the NameNode. This becomes prevalent if the
data is required to be displayed using a new user interface.
The proposal is to create a REST interface to expose all the information
displayed on dfshealth.jsp using GET methods. Wrapper classes will be created
to serve the data to the REST root resource within the hadoop-hdfs project.
This will enable the HDFS health screen information to be accessed remotely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[
https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13660825#comment-13660825
]
Steve Loughran commented on HADOOP-9562:
Clearly I was confused. My apologies.
Anyway, I think the proposal makes sense, though the liveness tests we did for
the branch-1 Linux HA on vSphere and Linux HA used an escalating set of probes.
# Pid signal -0
# RPC, HTTP ports open
# HTTP request resolving returning 200. (this is where returning an error
code on dfs safe mode would be invaluable, as it would avoid parsing at all)
# DFS ls operation. This is the one to verify that HDFS is really responding to
requests on the RPC channel.
# for downstream JT liveness: safe mode. state. {{setSafeMode()}} API method
has proven brittle across versions as constants for a nominally internal
operation were moved round. This is also why a safe mode HTTP page appeals to
me.
I suggest then
# some JSON/XML view of DSL health, with XML actually my preference.
# a {{hdfs-live.jspx}} page that returns an HTTP error code if DFS is unhappy.
The
[HappyAxis|http://svn.apache.org/repos/asf/webservices/axis/branches/explicitHeaderWork/java/webapps/axis/happyaxis.jsp]
page I wrote for Apache Axis did a lot more internal state check and
diagnostics of dependencies, env vars etc
Create REST interface for HDFS health data
--
Key: HADOOP-9562
URL: https://issues.apache.org/jira/browse/HADOOP-9562
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Affects Versions: 2.0.4-alpha
Reporter: Trevor Lorimer
Priority: Minor
The HDFS health screen (dfshealth.jsp) displays basic Version, Security and
Health information concerning the NameNode, currently this information is
accessible from classes in the org.apache.hadoop,hdfs.server.namenode package
and cannot be accessed outside the NameNode. This becomes prevalent if the
data is required to be displayed using a new user interface.
The proposal is to create a REST interface to expose all the information
displayed on dfshealth.jsp using GET methods. Wrapper classes will be created
to serve the data to the REST root resource within the hadoop-hdfs project.
This will enable the HDFS health screen information to be accessed remotely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[
https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13660935#comment-13660935
]
Trevor Lorimer commented on HADOOP-9562:
Hi, just to clarify what I was proposing.
All I intend to do is simply serve up all the data currently displayed on the
dfshealth jsp page, using only REST GETs, providing both XML and JSON
responses.
This stems from needing to present the data on a new UI from a different server.
This would be following the same rationale behind the REST APIs of Resource
Manager, Node Manager, Application Manager and History Server
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html
An example REST GET from http://localhost:50070/hdfs/v1/header would return:
header: {
role: NameNode
state: active
hostAndPort: localhost:8020
securityEnabled: false
}
this would correspond to NameNode 'localhost:8020' (active) displayed on the
dfshealth jsp page.
Create REST interface for HDFS health data
--
Key: HADOOP-9562
URL: https://issues.apache.org/jira/browse/HADOOP-9562
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Affects Versions: 2.0.4-alpha
Reporter: Trevor Lorimer
Priority: Minor
The HDFS health screen (dfshealth.jsp) displays basic Version, Security and
Health information concerning the NameNode, currently this information is
accessible from classes in the org.apache.hadoop,hdfs.server.namenode package
and cannot be accessed outside the NameNode. This becomes prevalent if the
data is required to be displayed using a new user interface.
The proposal is to create a REST interface to expose all the information
displayed on dfshealth.jsp using GET methods. Wrapper classes will be created
to serve the data to the REST root resource within the hadoop-hdfs project.
This will enable the HDFS health screen information to be accessed remotely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[
https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13660955#comment-13660955
]
Steve Loughran commented on HADOOP-9562:
OK. I'd like safe mode in there, and the ability to generate a HTTP error if
any value doesn't match expected. I'm not sure {{header}} is the right name,
but that's a detail
{{hdfs/v1/status?format=jsonstate=active}} returns 200 the result listed,
but {{hdfs/v1/status?format=jsonstate=offline}} returns 500+ JSON.
This makes liveness/config probes from curl trivial -you can omit the JSON
parse stage
Create REST interface for HDFS health data
--
Key: HADOOP-9562
URL: https://issues.apache.org/jira/browse/HADOOP-9562
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Affects Versions: 2.0.4-alpha
Reporter: Trevor Lorimer
Priority: Minor
The HDFS health screen (dfshealth.jsp) displays basic Version, Security and
Health information concerning the NameNode, currently this information is
accessible from classes in the org.apache.hadoop,hdfs.server.namenode package
and cannot be accessed outside the NameNode. This becomes prevalent if the
data is required to be displayed using a new user interface.
The proposal is to create a REST interface to expose all the information
displayed on dfshealth.jsp using GET methods. Wrapper classes will be created
to serve the data to the REST root resource within the hadoop-hdfs project.
This will enable the HDFS health screen information to be accessed remotely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[ https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13659727#comment-13659727 ] Konstantin Boudnik commented on HADOOP-9562: Is the http proxy REST friendly/compatible? Create REST interface for HDFS health data -- Key: HADOOP-9562 URL: https://issues.apache.org/jira/browse/HADOOP-9562 Project: Hadoop Common Issue Type: New Feature Components: fs Affects Versions: 2.0.4-alpha Reporter: Trevor Lorimer Priority: Minor The HDFS health screen (dfshealth.jsp) displays basic Version, Security and Health information concerning the NameNode, currently this information is accessible from classes in the org.apache.hadoop,hdfs.server.namenode package and cannot be accessed outside the NameNode. This becomes prevalent if the data is required to be displayed using a new user interface. The proposal is to create a REST interface to expose all the information displayed on dfshealth.jsp using GET methods. Wrapper classes will be created to serve the data to the REST root resource within the hadoop-hdfs project. This will enable the HDFS health screen information to be accessed remotely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[ https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13659736#comment-13659736 ] Steve Loughran commented on HADOOP-9562: I'm thinking more of auth integration. Adding id tags (and tests to verify their consistency) means you aren't scraping, at least not in terms of HTML parsing, instead using XPath queries that would be the same were we to provide any XML status page Create REST interface for HDFS health data -- Key: HADOOP-9562 URL: https://issues.apache.org/jira/browse/HADOOP-9562 Project: Hadoop Common Issue Type: New Feature Components: fs Affects Versions: 2.0.4-alpha Reporter: Trevor Lorimer Priority: Minor The HDFS health screen (dfshealth.jsp) displays basic Version, Security and Health information concerning the NameNode, currently this information is accessible from classes in the org.apache.hadoop,hdfs.server.namenode package and cannot be accessed outside the NameNode. This becomes prevalent if the data is required to be displayed using a new user interface. The proposal is to create a REST interface to expose all the information displayed on dfshealth.jsp using GET methods. Wrapper classes will be created to serve the data to the REST root resource within the hadoop-hdfs project. This will enable the HDFS health screen information to be accessed remotely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[ https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13659737#comment-13659737 ] Steve Loughran commented on HADOOP-9562: Of course, these are features so should only be considered for the 2.1+ release Create REST interface for HDFS health data -- Key: HADOOP-9562 URL: https://issues.apache.org/jira/browse/HADOOP-9562 Project: Hadoop Common Issue Type: New Feature Components: fs Affects Versions: 2.0.4-alpha Reporter: Trevor Lorimer Priority: Minor The HDFS health screen (dfshealth.jsp) displays basic Version, Security and Health information concerning the NameNode, currently this information is accessible from classes in the org.apache.hadoop,hdfs.server.namenode package and cannot be accessed outside the NameNode. This becomes prevalent if the data is required to be displayed using a new user interface. The proposal is to create a REST interface to expose all the information displayed on dfshealth.jsp using GET methods. Wrapper classes will be created to serve the data to the REST root resource within the hadoop-hdfs project. This will enable the HDFS health screen information to be accessed remotely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[ https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13659756#comment-13659756 ] Konstantin Boudnik commented on HADOOP-9562: I musta missed something: the JIRA is for like 3 REST calls to provide convenient RO access to some of the data that is already available either in the form of basic HTML webpage or via JMX. It isn't even a new feature - it is minor improvement and should be treated as such. If there's a need for a new web-backend in HDFS - it's isn't a part of this JIRA: not by a long shot, really. Create REST interface for HDFS health data -- Key: HADOOP-9562 URL: https://issues.apache.org/jira/browse/HADOOP-9562 Project: Hadoop Common Issue Type: Improvement Components: fs Affects Versions: 2.0.4-alpha Reporter: Trevor Lorimer Priority: Minor The HDFS health screen (dfshealth.jsp) displays basic Version, Security and Health information concerning the NameNode, currently this information is accessible from classes in the org.apache.hadoop,hdfs.server.namenode package and cannot be accessed outside the NameNode. This becomes prevalent if the data is required to be displayed using a new user interface. The proposal is to create a REST interface to expose all the information displayed on dfshealth.jsp using GET methods. Wrapper classes will be created to serve the data to the REST root resource within the hadoop-hdfs project. This will enable the HDFS health screen information to be accessed remotely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[ https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13660325#comment-13660325 ] Steve Loughran commented on HADOOP-9562: Well, it's clearly not a bug, which makes it a feature of some form, to go into trunk and then backported with the blessing/acquiescence of the release manager. I'm happy to help review and vote on this, if you are happy to deal with the inconsistencies of adding features to what you declared to be a feature-frozen release. I've proposed a section on Web UI compatibility to the [compatibility definitions|https://issues.apache.org/jira/browse/HADOOP-9517?focusedCommentId=13660321page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13660321], this argues in favour of a a machine-parseable format from the outset. It could still be XML with an XSL for the browser to auto-convert into human-readable content; I know JSON is the format du-jour, but XML tooling is built into Java (and iphone, android, etc), and it is actually more tightly specified than JSON, which has some gaping holes in its spec (what happens if you have duplicate keys, for example). Create REST interface for HDFS health data -- Key: HADOOP-9562 URL: https://issues.apache.org/jira/browse/HADOOP-9562 Project: Hadoop Common Issue Type: Improvement Components: fs Affects Versions: 2.0.4-alpha Reporter: Trevor Lorimer Priority: Minor The HDFS health screen (dfshealth.jsp) displays basic Version, Security and Health information concerning the NameNode, currently this information is accessible from classes in the org.apache.hadoop,hdfs.server.namenode package and cannot be accessed outside the NameNode. This becomes prevalent if the data is required to be displayed using a new user interface. The proposal is to create a REST interface to expose all the information displayed on dfshealth.jsp using GET methods. Wrapper classes will be created to serve the data to the REST root resource within the hadoop-hdfs project. This will enable the HDFS health screen information to be accessed remotely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[ https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13660336#comment-13660336 ] Konstantin Boudnik commented on HADOOP-9562: bq. I'm happy to help review and vote on this, if you are happy to deal with the inconsistencies of adding features to what you declared to be a feature-frozen release. Steve, I'd be really thankful if you can tell me where the notion of adding features to what you declared to be a feature-frozen release coming from? I can bet good money that I haven't said that. Please clarify. Create REST interface for HDFS health data -- Key: HADOOP-9562 URL: https://issues.apache.org/jira/browse/HADOOP-9562 Project: Hadoop Common Issue Type: Improvement Components: fs Affects Versions: 2.0.4-alpha Reporter: Trevor Lorimer Priority: Minor The HDFS health screen (dfshealth.jsp) displays basic Version, Security and Health information concerning the NameNode, currently this information is accessible from classes in the org.apache.hadoop,hdfs.server.namenode package and cannot be accessed outside the NameNode. This becomes prevalent if the data is required to be displayed using a new user interface. The proposal is to create a REST interface to expose all the information displayed on dfshealth.jsp using GET methods. Wrapper classes will be created to serve the data to the REST root resource within the hadoop-hdfs project. This will enable the HDFS health screen information to be accessed remotely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[
https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658168#comment-13658168
]
Suresh Srinivas commented on HADOOP-9562:
-
{quote}
The HDFS health screen (dfshealth.jsp) displays basic Version, Security and
Health information concerning the NameNode, currently this information is
accessible from classes in the org.apache.hadoop,hdfs.server.namenode package
and cannot be accessed outside the NameNode.
{quote}
This is not correct. While doing standardized REST APIs may not be a bad idea,
all of this information is exposed through {{FSNamesystemMBean}} and
{{NameNodeMXBean}} and the corresponding JMX http bridge (see HADOOP-7144 and
related jiras). This is what currently Ambari project uses.
Create REST interface for HDFS health data
--
Key: HADOOP-9562
URL: https://issues.apache.org/jira/browse/HADOOP-9562
Project: Hadoop Common
Issue Type: New Feature
Components: fs
Affects Versions: 2.0.4-alpha
Reporter: Trevor Lorimer
Priority: Minor
The HDFS health screen (dfshealth.jsp) displays basic Version, Security and
Health information concerning the NameNode, currently this information is
accessible from classes in the org.apache.hadoop,hdfs.server.namenode package
and cannot be accessed outside the NameNode. This becomes prevalent if the
data is required to be displayed using a new user interface.
The proposal is to create a REST interface to expose all the information
displayed on dfshealth.jsp using GET methods. Wrapper classes will be created
to serve the data to the REST root resource within the hadoop-hdfs project.
This will enable the HDFS health screen information to be accessed remotely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[
https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658733#comment-13658733
]
Steve Loughran commented on HADOOP-9562:
of course, if it is just GET ops, then making dfshealth a valid XHTML page with
id's for every span/div containing data lets you grab this stuff by
GET-parse-xpath operations, such as {{//html/body/div[id=safemode][0]}}.
You wouldn't need any extra pages, just some tests to verify that the served up
content would parse and the relevant IDs resolve
Create REST interface for HDFS health data
--
Key: HADOOP-9562
URL: https://issues.apache.org/jira/browse/HADOOP-9562
Project: Hadoop Common
Issue Type: New Feature
Components: fs
Affects Versions: 2.0.4-alpha
Reporter: Trevor Lorimer
Priority: Minor
The HDFS health screen (dfshealth.jsp) displays basic Version, Security and
Health information concerning the NameNode, currently this information is
accessible from classes in the org.apache.hadoop,hdfs.server.namenode package
and cannot be accessed outside the NameNode. This becomes prevalent if the
data is required to be displayed using a new user interface.
The proposal is to create a REST interface to expose all the information
displayed on dfshealth.jsp using GET methods. Wrapper classes will be created
to serve the data to the REST root resource within the hadoop-hdfs project.
This will enable the HDFS health screen information to be accessed remotely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[
https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657276#comment-13657276
]
Steve Loughran commented on HADOOP-9562:
This would be good.
# must include DFS safe mode status
# assuming federation is supported, what are the implications?
# what about HFDS HA? Allow all NNs to do an enum of all NNs status?
# are there any security implications? If so, access should be restricted.
# I'd like to have error codes in the pages, so I can do a {{curl -m 60
http://namenode:50020/service/dfslive.jsp}} and expect an error code if dfs is
not up and out out of safemode. The use case would be : let bash scripts (inc
Linux HA scripts) verify FS is live just by looking at the return code of curl
or {{GET}}.
Create REST interface for HDFS health data
--
Key: HADOOP-9562
URL: https://issues.apache.org/jira/browse/HADOOP-9562
Project: Hadoop Common
Issue Type: New Feature
Components: fs
Affects Versions: 2.0.4-alpha
Reporter: Trevor Lorimer
Priority: Minor
The HDFS health screen (dfshealth.jsp) displays basic Version, Security and
Health information concerning the NameNode, currently this information is
accessible from classes in the org.apache.hadoop,hdfs.server.namenode package
and cannot be accessed outside the NameNode. This becomes prevalent if the
data is required to be displayed using a new user interface.
The proposal is to create a REST interface to expose all the information
displayed on dfshealth.jsp using GET methods. Wrapper classes will be created
to serve the data to the REST root resource within the hadoop-hdfs project.
This will enable the HDFS health screen information to be accessed remotely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[
https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657413#comment-13657413
]
Chris Nauroth commented on HADOOP-9562:
---
{quote}
I'd like to have error codes in the pages, so I can do a curl -m 60
http://namenode:50020/service/dfslive.jsp and expect an error code if dfs is
not up and out out of safemode. The use case would be : let bash scripts (inc
Linux HA scripts) verify FS is live just by looking at the return code of curl
or GET.
{quote}
One difficulty is that we don't start the HTTP server until relatively late in
initialization. If you have a ton of edits, then you'll suffer through a long
checkpoint before status is visible, either in the web page or a REST
interface. I have patches queued up in HDFS-4249 and its sub-tasks to start
the HTTP server sooner and display more details about the early phases of
namenode startup.
Create REST interface for HDFS health data
--
Key: HADOOP-9562
URL: https://issues.apache.org/jira/browse/HADOOP-9562
Project: Hadoop Common
Issue Type: New Feature
Components: fs
Affects Versions: 2.0.4-alpha
Reporter: Trevor Lorimer
Priority: Minor
The HDFS health screen (dfshealth.jsp) displays basic Version, Security and
Health information concerning the NameNode, currently this information is
accessible from classes in the org.apache.hadoop,hdfs.server.namenode package
and cannot be accessed outside the NameNode. This becomes prevalent if the
data is required to be displayed using a new user interface.
The proposal is to create a REST interface to expose all the information
displayed on dfshealth.jsp using GET methods. Wrapper classes will be created
to serve the data to the REST root resource within the hadoop-hdfs project.
This will enable the HDFS health screen information to be accessed remotely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
