I've begun work on a patch for this request here: https://github.com/ekohlwey/knox/tree/KNOX-250 And I've also included some thoughts on the issue under its JIRA ticket: https://issues.apache.org/jira/browse/KNOX-250
Any feedback is much appreciated! On Mon, Feb 10, 2014 at 11:55 AM, larry mccay <[email protected]> wrote: > Hi Ed - > > Absolutely, I would suggest that you discuss your intended approach on the > Jira as you move forward. > This will allow the community to: > > 1. provide valuable insight into the best way forward > 2. aid in the review process when we go to merge if/when contribution is > determined appropriate > > I would caution you up front that modifying the web app artifacts that are > generated at topology deployment time as a way of integrating may be a > dangerous approach. > > Those artifacts are recreated every time the topology is changed and > redeployed and changes made to web.xml or gateway.xml will be lost. > > This is why using the provider mechanism is the preferred way to introduce > (contribute) new artifacts and components into the system. > > Feel free to POC your idea in any manner that you wish and we will try and > support you as needed. > However, the final contribution will need to fit into the architecture of > Knox properly. > > thanks, > > --larry > > > > On Mon, Feb 10, 2014 at 11:29 AM, Ed Kohlwey <[email protected]> wrote: > > > Hi Larry, > > Thanks for the feedback. I think what makes sense is to open a > > Jira(KNOX-250), and proceed with an implementation that fits my teams > > immediate needs, targeting eventual integration in the overall framework. > > Hi Ed - > > > > While this is a usecase that is clearly outside of the primary REST API > > Gateway charter for Knox, it may provide value to a user population that > is > > currently under-served by Knox. We would need to enumerate the value that > > we could add to CLI users with Knox as a bastion. You have already > > identified audit as a possible value-add. > > > > There are complexities involved here that we need to consider and I > haven't > > had the cycles to really spend on them yet. > > > > I am not yet in a position to say whether we would want to include this > in > > Knox but if we were to do so - I think the following would be worth > > considering: > > > > * Apache Mina SSHD Knox Provider (providers are great for introducing > > configuration, filters, listeners, etc.) > > * Each topology (which represents a managed Hadoop cluster) may configure > > an optional sshd-provider > > * The sshd-provider would likely require a ShellFactory that sets up a > > tunneling client into the target cluster (perhaps with the sshd client > > component) > > * The sshd-provider ShellFactory would need to resolve the appropriate > > Hadoop client configuration for it's intended Hadoop cluster > > * The sshd-provider would expose a separate ssh port per cluster for > users > > to connect to > > > > I believe that this is a more natural approach (for Knox) than providing > a > > single port that would have to multiplex clients to clusters. > > > > Feel free to file a Jira for this work - if you like. > > We can continue discussion there and - if/when appropriate - we can > > collaborate on patches as you contribute. > > > > Again, thank you for the interesting in contributing Knox! > > > > --larry > > > > > > > > On Wed, Feb 5, 2014 at 4:42 PM, Ed Kohlwey <[email protected]> wrote: > > > > > Hi Larry, > > > Thanks for the thoughts. > > > > > > Yes, I realize this is somewhat outside of what may be outside of > Knox's > > > main use case as a REST gateway, however I feel it is functionality > that > > > could logically be part of the application. We plan to use Knox to > > satisfy > > > some other security requirements and bundling this functionality in > Knox > > > makes sense to our team. > > > > > > "The users" are system administrators who have sudo access and would > like > > > to use a command terminal to do the usual job of system administrators > > > (changing configuration files, restarting services, chowning things in > > hdfs > > > and local disk, checking on the namenode and resource manager, etc.). I > > > have a requirement that all administrators who maintain the Hadoop > > cluster > > > have every action logged for audit purposes. One way to do this > reliably > > is > > > to force them to tunnel through a bastion host that they do not > control. > > > Presumably by making the bastion administrators different people than > the > > > cluster administrators, a higher level of security is achieved. > > > > > > Doing this within Knox makes sense as it will give us one 'pane of > glass' > > > for our cluster security requirements. > > > > > > For convenience sake, I would like them to be able to do this without > > > running kinit every time they log into the cluster, however this is a > > > secondary goal. This is what I mean by 'interacting normally with > hadoop' > > - > > > using any shell command the way they do before Knox is in the picture > (so > > > hdfs dfs -ls /tmp is one of many examples). > > > > > > I have contemplated two implementations: one is a HTTP based endpoint > > that > > > violates normal REST conventions by keeping the HTTP session open > > > indefinitely. Another is simply discarding HTTP based interaction and > > > providing an SSH-based tunneling facility. The second option is nice > > > because it "looks" normal or almost normal to the system administrator. > > > Another option could be a Knox CLI extension. > > > > > > > > > > > > > > > > > > > > > On Tue, Feb 4, 2014 at 6:27 PM, larry mccay <[email protected]> wrote: > > > > > > > Hi Ed - > > > > > > > > Thanks for the interesting thoughts. > > > > > > > > Unfortunately, I don't think that have the full picture of what you > are > > > > thinking in my mind yet. > > > > > > > > Let's leave out the implementation options and talk purely about the > > > > usecase for a second. > > > > > > > > * You have a set of users (what are their roles - sounds like admins > of > > > > some sort?) > > > > * You'd like them to be able to gain access to the cluster through > Knox > > > > (Knox is obviously a REST API Gateway) > > > > * What is it that these particular users need to do? > > > > * When you say "the user could interact normally with Hadoop" - do > you > > > mean > > > > that they have shell access for running the hadoop CLI's - eg: > "hadoop > > fs > > > > -l /tmp"? > > > > * are you assuming a browser based application for this or is this a > > Knox > > > > CLI extension? > > > > > > > > When we were first talking about the ClientDSL we briefly considered > > > using > > > > Apache Mina sshd but it didn't seem necessary given that we are able > to > > > use > > > > the REST APIs remotely. > > > > > > > > Again, if you can articulate the (a) usecase end to end that would > be a > > > > great starting place. > > > > > > > > Thanks again for your ideas! > > > > > > > > I look forward to contributions from you on this. > > > > > > > > --larry > > > > > > > > > > > > > > > > > > > > On Tue, Feb 4, 2014 at 4:22 PM, Ed Kohlwey <[email protected]> > wrote: > > > > > > > > > I'm interested in implementing a knox service that implements full > > > shell > > > > > access to hosts within a hadoop cluster. The point of doing so is > to > > > > > provide administrative auditing capability for all cluster user > > > accounts, > > > > > including privileged users. A smaller number of users would have > > access > > > > to > > > > > the knox host than the full cluster. > > > > > > > > > > My current thinking on how to do this is to implement a servlet. > > Using > > > a > > > > > GET or POST request, the request body would be passed to standard > in, > > > and > > > > > the response body would contain standard out. Knox itself would use > > ssh > > > > > public key based login to connect as a knox user to the other > cluster > > > > > hosts, and then use a combinaiton of sudo and exec commands to > change > > > the > > > > > shell to the appropriate user. Knox would set up a hadoop > delegation > > > > token > > > > > so the user could interact normally with Hadoop. > > > > > > > > > > Another option could be to start a java based SSH server on another > > > > > (non-22) port and perform normal tunneling, probably by adding an > > > > > interactive shell that asks what host the user would like to > connect > > > to. > > > > > This has the advantage of basically looking like a normal SSH > > > connection. > > > > > > > > > > I noticed this is not how the Knox DSL is implemented, which seems > to > > > use > > > > > some session state. Can anyone discuss the motivations for doing > > this? > > > > Does > > > > > the community have any opinions on how best to go about providing > > > audited > > > > > secure shell access? > > > > > > > > > > > > > > >
