[jira] [Commented] (YARN-1695) Implement the rest (writable APIs) of RM web-services
[
https://issues.apache.org/jira/browse/YARN-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14241175#comment-14241175
]
Steve Loughran commented on YARN-1695:
--
Now that the functionality is in, is there going to be a replacement for
{{YarnClientImpl}} which we can use to switch to the REST APIs? I can't see an
issue for it.
Implement the rest (writable APIs) of RM web-services
-
Key: YARN-1695
URL: https://issues.apache.org/jira/browse/YARN-1695
Project: Hadoop YARN
Issue Type: New Feature
Reporter: Vinod Kumar Vavilapalli
Assignee: Varun Vasudev
MAPREDUCE-2863 added the REST web-services to RM and NM. But all the APIs
added there were only focused on obtaining information from the cluster. We
need to have the following REST APIs to finish the feature
- Application submission/termination (Priority): This unblocks easy client
interaction with a YARN cluster
- Application Client protocol: For resource scheduling by apps written in an
arbitrary language. Will have to think about throughput concerns
- ContainerManagement Protocol: Again for arbitrary language apps.
One important thing to note here is that we already have client libraries on
all the three protocols that do some some heavy-lifting. One part of the
effort is to figure out if they can be made any thinner and/or how
web-services will implement the same functionality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1695) Implement the rest (writable APIs) of RM web-services
[ https://issues.apache.org/jira/browse/YARN-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13908751#comment-13908751 ] Vinod Kumar Vavilapalli commented on YARN-1695: --- I missed your comment, apologies. We already have a clear security story for web UI and web-services. This JIRA is not adding any more end-points than what are already present. Similarly, we already expose the same information via RPCs, web UI _and_ web-services. So this JIRA isn't adding any more maintenance burden than is already present. Overall, I've seen a lot of use-cases where, at-least on the client side APIs, users want to use web-services directly instead of writing code. That's the goal of this JIRA. As I already pointed out, the remaining protocols from this JIRA are in the grey area of whether to do or not at all and which I am not focusing on at all right-away. Implement the rest (writable APIs) of RM web-services - Key: YARN-1695 URL: https://issues.apache.org/jira/browse/YARN-1695 Project: Hadoop YARN Issue Type: New Feature Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli MAPREDUCE-2863 added the REST web-services to RM and NM. But all the APIs added there were only focused on obtaining information from the cluster. We need to have the following REST APIs to finish the feature - Application submission/termination (Priority): This unblocks easy client interaction with a YARN cluster - Application Client protocol: For resource scheduling by apps written in an arbitrary language. Will have to think about throughput concerns - ContainerManagement Protocol: Again for arbitrary language apps. One important thing to note here is that we already have client libraries on all the three protocols that do some some heavy-lifting. One part of the effort is to figure out if they can be made any thinner and/or how web-services will implement the same functionality. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1695) Implement the rest (writable APIs) of RM web-services
[ https://issues.apache.org/jira/browse/YARN-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13908754#comment-13908754 ] Vinod Kumar Vavilapalli commented on YARN-1695: --- There is a related question here - whether to implement web-services inside RM or pull them out and implement them in a proxy layer similar to WebHDFS. But that's a larger issue with code and feature-set that is already present today. Will create a separate ticket for that discussion. Implement the rest (writable APIs) of RM web-services - Key: YARN-1695 URL: https://issues.apache.org/jira/browse/YARN-1695 Project: Hadoop YARN Issue Type: New Feature Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli MAPREDUCE-2863 added the REST web-services to RM and NM. But all the APIs added there were only focused on obtaining information from the cluster. We need to have the following REST APIs to finish the feature - Application submission/termination (Priority): This unblocks easy client interaction with a YARN cluster - Application Client protocol: For resource scheduling by apps written in an arbitrary language. Will have to think about throughput concerns - ContainerManagement Protocol: Again for arbitrary language apps. One important thing to note here is that we already have client libraries on all the three protocols that do some some heavy-lifting. One part of the effort is to figure out if they can be made any thinner and/or how web-services will implement the same functionality. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1695) Implement the rest (writable APIs) of RM web-services
[ https://issues.apache.org/jira/browse/YARN-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898052#comment-13898052 ] Alejandro Abdelnur commented on YARN-1695: -- err, [~vinodkv], isn't this a duplicate of the JIRA you opened back in DEC, YARN-1538? Implement the rest (writable APIs) of RM web-services - Key: YARN-1695 URL: https://issues.apache.org/jira/browse/YARN-1695 Project: Hadoop YARN Issue Type: New Feature Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli MAPREDUCE-2863 added the REST web-services to RM and NM. But all the APIs added there were only focused on obtaining information from the cluster. We need to have the following REST APIs to finish the feature - Application submission/termination (Priority): This unblocks easy client interaction with a YARN cluster - Application Client protocol: For resource scheduling by apps written in an arbitrary language. Will have to think about throughput concerns - ContainerManagement Protocol: Again for arbitrary language apps. One important thing to note here is that we already have client libraries on all the three protocols that do some some heavy-lifting. One part of the effort is to figure out if they can be made any thinner and/or how web-services will implement the same functionality. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1695) Implement the rest (writable APIs) of RM web-services
[ https://issues.apache.org/jira/browse/YARN-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898089#comment-13898089 ] Vinod Kumar Vavilapalli commented on YARN-1695: --- Ah, thanks. I did search for existing tickets but couldn't quite locate it using JIRA search. Closing one of these as a duplicate. Implement the rest (writable APIs) of RM web-services - Key: YARN-1695 URL: https://issues.apache.org/jira/browse/YARN-1695 Project: Hadoop YARN Issue Type: New Feature Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli MAPREDUCE-2863 added the REST web-services to RM and NM. But all the APIs added there were only focused on obtaining information from the cluster. We need to have the following REST APIs to finish the feature - Application submission/termination (Priority): This unblocks easy client interaction with a YARN cluster - Application Client protocol: For resource scheduling by apps written in an arbitrary language. Will have to think about throughput concerns - ContainerManagement Protocol: Again for arbitrary language apps. One important thing to note here is that we already have client libraries on all the three protocols that do some some heavy-lifting. One part of the effort is to figure out if they can be made any thinner and/or how web-services will implement the same functionality. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1695) Implement the rest (writable APIs) of RM web-services
[ https://issues.apache.org/jira/browse/YARN-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898112#comment-13898112 ] Alejandro Abdelnur commented on YARN-1695: -- Before we start coding work on this, it would be great to see how security will handled (authentication, ACLs, tokens, etc). I'm a bit a concern about introducing a second protocol everywhere. From a maintenance and security risk point of view is doubling our development/support efforts. Granted, HDFS offers data over RPC and HTTP. But, HTTP, when using HttpFS (how I recommend using it HTTP access) is a gateway that ends up doing RPC to HDFS. Thus the only protocol accessing HDFS is RPC. Have we considered having a C implementation of Hadoop RPC, with the multiplatform support of protobuffers that may give us the multi-platform support we are aiming to with a single protocol interface. Thoughts? Implement the rest (writable APIs) of RM web-services - Key: YARN-1695 URL: https://issues.apache.org/jira/browse/YARN-1695 Project: Hadoop YARN Issue Type: New Feature Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli MAPREDUCE-2863 added the REST web-services to RM and NM. But all the APIs added there were only focused on obtaining information from the cluster. We need to have the following REST APIs to finish the feature - Application submission/termination (Priority): This unblocks easy client interaction with a YARN cluster - Application Client protocol: For resource scheduling by apps written in an arbitrary language. Will have to think about throughput concerns - ContainerManagement Protocol: Again for arbitrary language apps. One important thing to note here is that we already have client libraries on all the three protocols that do some some heavy-lifting. One part of the effort is to figure out if they can be made any thinner and/or how web-services will implement the same functionality. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-1695) Implement the rest (writable APIs) of RM web-services
[ https://issues.apache.org/jira/browse/YARN-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13896881#comment-13896881 ] Vinod Kumar Vavilapalli commented on YARN-1695: --- To be clear, the primary focus of this JIRA is definitely the application submission so that tools outside YARN can easily submit and monitor applications without writing against the JAVA or PB interfaces. The later two protocols arguably are not that important to have REST APIs given we already have a cross-platform story in terms of protocol-buffers based interfaces. Implement the rest (writable APIs) of RM web-services - Key: YARN-1695 URL: https://issues.apache.org/jira/browse/YARN-1695 Project: Hadoop YARN Issue Type: New Feature Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli MAPREDUCE-2863 added the REST web-services to RM and NM. But all the APIs added there were only focused on obtaining information from the cluster. We need to have the following REST APIs to finish the feature - Application submission/termination (Priority): This unblocks easy client interaction with a YARN cluster - Application Client protocol: For resource scheduling by apps written in an arbitrary language. Will have to think about throughput concerns - ContainerManagement Protocol: Again for arbitrary language apps. One important thing to note here is that we already have client libraries on all the three protocols that do some some heavy-lifting. One part of the effort is to figure out if they can be made any thinner and/or how web-services will implement the same functionality. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
