Hi Ed - While this is a usecase that is clearly outside of the primary REST API Gateway charter for Knox, it may provide value to a user population that is currently under-served by Knox. We would need to enumerate the value that we could add to CLI users with Knox as a bastion. You have already identified audit as a possible value-add.
There are complexities involved here that we need to consider and I haven't had the cycles to really spend on them yet. I am not yet in a position to say whether we would want to include this in Knox but if we were to do so - I think the following would be worth considering: * Apache Mina SSHD Knox Provider (providers are great for introducing configuration, filters, listeners, etc.) * Each topology (which represents a managed Hadoop cluster) may configure an optional sshd-provider * The sshd-provider would likely require a ShellFactory that sets up a tunneling client into the target cluster (perhaps with the sshd client component) * The sshd-provider ShellFactory would need to resolve the appropriate Hadoop client configuration for it's intended Hadoop cluster * The sshd-provider would expose a separate ssh port per cluster for users to connect to I believe that this is a more natural approach (for Knox) than providing a single port that would have to multiplex clients to clusters. Feel free to file a Jira for this work - if you like. We can continue discussion there and - if/when appropriate - we can collaborate on patches as you contribute. Again, thank you for the interesting in contributing Knox! --larry On Wed, Feb 5, 2014 at 4:42 PM, Ed Kohlwey <[email protected]> wrote: > Hi Larry, > Thanks for the thoughts. > > Yes, I realize this is somewhat outside of what may be outside of Knox's > main use case as a REST gateway, however I feel it is functionality that > could logically be part of the application. We plan to use Knox to satisfy > some other security requirements and bundling this functionality in Knox > makes sense to our team. > > "The users" are system administrators who have sudo access and would like > to use a command terminal to do the usual job of system administrators > (changing configuration files, restarting services, chowning things in hdfs > and local disk, checking on the namenode and resource manager, etc.). I > have a requirement that all administrators who maintain the Hadoop cluster > have every action logged for audit purposes. One way to do this reliably is > to force them to tunnel through a bastion host that they do not control. > Presumably by making the bastion administrators different people than the > cluster administrators, a higher level of security is achieved. > > Doing this within Knox makes sense as it will give us one 'pane of glass' > for our cluster security requirements. > > For convenience sake, I would like them to be able to do this without > running kinit every time they log into the cluster, however this is a > secondary goal. This is what I mean by 'interacting normally with hadoop' - > using any shell command the way they do before Knox is in the picture (so > hdfs dfs -ls /tmp is one of many examples). > > I have contemplated two implementations: one is a HTTP based endpoint that > violates normal REST conventions by keeping the HTTP session open > indefinitely. Another is simply discarding HTTP based interaction and > providing an SSH-based tunneling facility. The second option is nice > because it "looks" normal or almost normal to the system administrator. > Another option could be a Knox CLI extension. > > > > > > > On Tue, Feb 4, 2014 at 6:27 PM, larry mccay <[email protected]> wrote: > > > Hi Ed - > > > > Thanks for the interesting thoughts. > > > > Unfortunately, I don't think that have the full picture of what you are > > thinking in my mind yet. > > > > Let's leave out the implementation options and talk purely about the > > usecase for a second. > > > > * You have a set of users (what are their roles - sounds like admins of > > some sort?) > > * You'd like them to be able to gain access to the cluster through Knox > > (Knox is obviously a REST API Gateway) > > * What is it that these particular users need to do? > > * When you say "the user could interact normally with Hadoop" - do you > mean > > that they have shell access for running the hadoop CLI's - eg: "hadoop fs > > -l /tmp"? > > * are you assuming a browser based application for this or is this a Knox > > CLI extension? > > > > When we were first talking about the ClientDSL we briefly considered > using > > Apache Mina sshd but it didn't seem necessary given that we are able to > use > > the REST APIs remotely. > > > > Again, if you can articulate the (a) usecase end to end that would be a > > great starting place. > > > > Thanks again for your ideas! > > > > I look forward to contributions from you on this. > > > > --larry > > > > > > > > > > On Tue, Feb 4, 2014 at 4:22 PM, Ed Kohlwey <[email protected]> wrote: > > > > > I'm interested in implementing a knox service that implements full > shell > > > access to hosts within a hadoop cluster. The point of doing so is to > > > provide administrative auditing capability for all cluster user > accounts, > > > including privileged users. A smaller number of users would have access > > to > > > the knox host than the full cluster. > > > > > > My current thinking on how to do this is to implement a servlet. Using > a > > > GET or POST request, the request body would be passed to standard in, > and > > > the response body would contain standard out. Knox itself would use ssh > > > public key based login to connect as a knox user to the other cluster > > > hosts, and then use a combinaiton of sudo and exec commands to change > the > > > shell to the appropriate user. Knox would set up a hadoop delegation > > token > > > so the user could interact normally with Hadoop. > > > > > > Another option could be to start a java based SSH server on another > > > (non-22) port and perform normal tunneling, probably by adding an > > > interactive shell that asks what host the user would like to connect > to. > > > This has the advantage of basically looking like a normal SSH > connection. > > > > > > I noticed this is not how the Knox DSL is implemented, which seems to > use > > > some session state. Can anyone discuss the motivations for doing this? > > Does > > > the community have any opinions on how best to go about providing > audited > > > secure shell access? > > > > > >
