[
https://issues.apache.org/jira/browse/KNOX-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16531466#comment-16531466
]
Larry McCay edited comment on KNOX-1334 at 7/3/18 2:17 PM:
-----------------------------------------------------------
Hi [~mert_hoc] - thank you for those details.
A couple observations/questions:
* There doesn't seem to be enough details in the second statement to justify
the conclusion in the third. Can you indicate why there shouldn't be meaningful
differences?
* You have a catch-all rewrite rule for matching things that you do not want
to rewrite - we need to make sure that this doesn't match patterns from other
services as well. It looks to me like it may. Need to test this colocated in a
topology with other services (UIs and APIs) to be sure:
{code:java}
+ <match pattern="\{**}">
+ <!-- This catches the extra matches that we don't actually want to
rewrite. -->
+ </match>
{code}
* Based on your change of the dispatch packagename below it seems that you are
not using 1.0.0 or master (which will soon be 1.1.0) but are on at the latest
0.14.0 release.
{code:java}
- <dispatch classname="org.apache.knox.gateway.dispatch.DefaultDispatch"
ha-classname="org.apache.knox.gateway.rm.dispatch.RMUIHaDispatch"/>
+ <dispatch classname="org.apache.hadoop.gateway.dispatch.DefaultDispatch"
ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
{code}
* so, can you provide what version you are currently using. I wouldn't imagine
that we would land this in anything older than 1.1.0 and TBH, I think we need
to push it out to 1.2.0 which we can make a quick release cycle for a handful
of like things.
* You mentioned that Knox supports Hadoop 2.7 - I assume this is based on the
version of the service definitions that you are changing but you may also be
referring to our pom dependency on hadoop-common (depending on what version you
are using).
** The versions for the service defs are basically "since 2.7.1" and assumes
forward compatibility if there is no newer version.
** 1.0.0 was primarily a repackaging of the class names away from the "hadoop"
name and we bumped our dependency to Hadoop 3
* Depending on the extent of your changes to the UIs that you are proxying
here, a new version may not be the right way to go since newer version numbers
become the default for the service and you have to specify older version in the
topology for them to get picked up instead.
My proposal is that we push this out to 1.2.0 release where we can add some
gateway-site.xml config to specify overrides of the default version behavior
for certain deployments. This will allow us to have EMR deployments specify in
central place which version to use and other distributions specify another.
Overriding only those versions that are not the latest. This will provide a
simple service def profile for different stacks.
Also, since some (most) of the cloud based items in KIP-11 [1] are still in the
works in feature branches and the like, it would be good to turn around some of
these items together for a quick cloud themed release - without holding up
1.1.0 any more.
Thoughts?
1. https://cwiki.apache.org/confluence/display/KNOX/KIP-11+Cloud+Usecases
was (Author: lmccay):
Hi [~mert_hoc] - thank you for those details.
A couple observations/questions:
* There doesn't seem to be enough details in the second statement to justify
the conclusion in the third. Can you indicate why there shouldn't be meaningful
differences?
* You have a catch-all rewrite rule for matching things that you do not want
to rewrite - we need to make sure that this doesn't match patterns from other
services as well. It looks to me like it may. Need to test this colocated in a
topology with other services (UIs and APIs) to be sure:
{code}
+ <match pattern="\{**}">
+ <!-- This catches the extra matches that we don't actually want to
rewrite. -->
+ </match>
{code}
* Based on your change of the dispatch packagename below it seems that you are
not using 1.0.0 or master (which will soon be 1.1.0) but are on at the latest
0.14.0 release.
{code}
- <dispatch classname="org.apache.knox.gateway.dispatch.DefaultDispatch"
ha-classname="org.apache.knox.gateway.rm.dispatch.RMUIHaDispatch"/>
+ <dispatch classname="org.apache.hadoop.gateway.dispatch.DefaultDispatch"
ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
{code}
* so, can you provide what version you are currently using. I wouldn't imagine
that we would land this in anything older than 1.1.0 and TBH, I think we need
to push it out to 1.2.0 which we can make a quick release cycle for a handful
of like things.
* You mentioned that Knox supports Hadoop 2.7 - I assume this is based on the
version of the service definitions that you are changing but you may also be
referring to our pom dependency on hadoop-common (depending on what version you
are using).
** The versions for the service defs are basically "since 2.7.1" and assumes
forward compatibility if there is no newer version.
** 1.0.0 was primarily a repackaging of the class names away from the "hadoop"
name and we bumped our dependency to Hadoop 3
* Depending on the extent of your changes to the UIs that you are proxying
here, a new version may not be the right way to go since newer version numbers
become the default for the service and you have to specify older version in the
topology for them to get picked up instead.
My proposal is that we push this out to 1.2.0 release where we can add some
gateway-site.xml config to specify overrides of the default version behavior
for certain deployments. This will allow us to have EMR deployments specify in
central place which version to use and other distributions specify another.
Overriding only those versions that are not the latest. This will provide a
simple service def profile for different stacks.
Also, since some (most) of the cloud based items in KIP-11 are still in the
works in feature branches and the like, it would be good to turn around some of
these items together for a quick cloud themed release - without holding up
1.1.0 any more.
Thoughts?
> Knox on AWS EMR
> ---------------
>
> Key: KNOX-1334
> URL: https://issues.apache.org/jira/browse/KNOX-1334
> Project: Apache Knox
> Issue Type: Bug
> Reporter: PRAVEEN K RAVIKUMAR
> Priority: Major
> Fix For: 1.1.0
>
> Attachments: KNOX-1343.patch
>
>
> Hi
> I have requirement to send all the request coming to proxy to be forwarded to
> EMR master node.
> Example:
> https://<any dns-name>:<PORT>/ --> Proxy --> EMR master.
> The DNS name could be anything, because there are many slave nodes. I would
> like the proxy to "redirect" dynamically "https://<any dns-name>:<PORT>/" to
> EMR master host. Could you please let me know if a reverse proxy/ Apache Knox
> can do that. I am trying to mimic option2 "SSH Tunnel to the Master Node
> Using Dynamic Port Forwarding"
> https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-ssh-tunnel.html
> Sample URLS:
> YARN ResourceManager https://master-public-dns-name:8088/ YARN NodeManager
> https://slave-public-dns-name:8042/ Spark HistoryServer
> https://master-public-dns-name:18080/
> [https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-ssh-tunnel.html]
>
> If so could you please share me the steps to configure Knox on EMR cluster-
> would be very helpful?
> Praveen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
