Hi Ed - Absolutely, I would suggest that you discuss your intended approach on the Jira as you move forward. This will allow the community to:
1. provide valuable insight into the best way forward 2. aid in the review process when we go to merge if/when contribution is determined appropriate I would caution you up front that modifying the web app artifacts that are generated at topology deployment time as a way of integrating may be a dangerous approach. Those artifacts are recreated every time the topology is changed and redeployed and changes made to web.xml or gateway.xml will be lost. This is why using the provider mechanism is the preferred way to introduce (contribute) new artifacts and components into the system. Feel free to POC your idea in any manner that you wish and we will try and support you as needed. However, the final contribution will need to fit into the architecture of Knox properly. thanks, --larry On Mon, Feb 10, 2014 at 11:29 AM, Ed Kohlwey <[email protected]> wrote: > Hi Larry, > Thanks for the feedback. I think what makes sense is to open a > Jira(KNOX-250), and proceed with an implementation that fits my teams > immediate needs, targeting eventual integration in the overall framework. > Hi Ed - > > While this is a usecase that is clearly outside of the primary REST API > Gateway charter for Knox, it may provide value to a user population that is > currently under-served by Knox. We would need to enumerate the value that > we could add to CLI users with Knox as a bastion. You have already > identified audit as a possible value-add. > > There are complexities involved here that we need to consider and I haven't > had the cycles to really spend on them yet. > > I am not yet in a position to say whether we would want to include this in > Knox but if we were to do so - I think the following would be worth > considering: > > * Apache Mina SSHD Knox Provider (providers are great for introducing > configuration, filters, listeners, etc.) > * Each topology (which represents a managed Hadoop cluster) may configure > an optional sshd-provider > * The sshd-provider would likely require a ShellFactory that sets up a > tunneling client into the target cluster (perhaps with the sshd client > component) > * The sshd-provider ShellFactory would need to resolve the appropriate > Hadoop client configuration for it's intended Hadoop cluster > * The sshd-provider would expose a separate ssh port per cluster for users > to connect to > > I believe that this is a more natural approach (for Knox) than providing a > single port that would have to multiplex clients to clusters. > > Feel free to file a Jira for this work - if you like. > We can continue discussion there and - if/when appropriate - we can > collaborate on patches as you contribute. > > Again, thank you for the interesting in contributing Knox! > > --larry > > > > On Wed, Feb 5, 2014 at 4:42 PM, Ed Kohlwey <[email protected]> wrote: > > > Hi Larry, > > Thanks for the thoughts. > > > > Yes, I realize this is somewhat outside of what may be outside of Knox's > > main use case as a REST gateway, however I feel it is functionality that > > could logically be part of the application. We plan to use Knox to > satisfy > > some other security requirements and bundling this functionality in Knox > > makes sense to our team. > > > > "The users" are system administrators who have sudo access and would like > > to use a command terminal to do the usual job of system administrators > > (changing configuration files, restarting services, chowning things in > hdfs > > and local disk, checking on the namenode and resource manager, etc.). I > > have a requirement that all administrators who maintain the Hadoop > cluster > > have every action logged for audit purposes. One way to do this reliably > is > > to force them to tunnel through a bastion host that they do not control. > > Presumably by making the bastion administrators different people than the > > cluster administrators, a higher level of security is achieved. > > > > Doing this within Knox makes sense as it will give us one 'pane of glass' > > for our cluster security requirements. > > > > For convenience sake, I would like them to be able to do this without > > running kinit every time they log into the cluster, however this is a > > secondary goal. This is what I mean by 'interacting normally with hadoop' > - > > using any shell command the way they do before Knox is in the picture (so > > hdfs dfs -ls /tmp is one of many examples). > > > > I have contemplated two implementations: one is a HTTP based endpoint > that > > violates normal REST conventions by keeping the HTTP session open > > indefinitely. Another is simply discarding HTTP based interaction and > > providing an SSH-based tunneling facility. The second option is nice > > because it "looks" normal or almost normal to the system administrator. > > Another option could be a Knox CLI extension. > > > > > > > > > > > > > > On Tue, Feb 4, 2014 at 6:27 PM, larry mccay <[email protected]> wrote: > > > > > Hi Ed - > > > > > > Thanks for the interesting thoughts. > > > > > > Unfortunately, I don't think that have the full picture of what you are > > > thinking in my mind yet. > > > > > > Let's leave out the implementation options and talk purely about the > > > usecase for a second. > > > > > > * You have a set of users (what are their roles - sounds like admins of > > > some sort?) > > > * You'd like them to be able to gain access to the cluster through Knox > > > (Knox is obviously a REST API Gateway) > > > * What is it that these particular users need to do? > > > * When you say "the user could interact normally with Hadoop" - do you > > mean > > > that they have shell access for running the hadoop CLI's - eg: "hadoop > fs > > > -l /tmp"? > > > * are you assuming a browser based application for this or is this a > Knox > > > CLI extension? > > > > > > When we were first talking about the ClientDSL we briefly considered > > using > > > Apache Mina sshd but it didn't seem necessary given that we are able to > > use > > > the REST APIs remotely. > > > > > > Again, if you can articulate the (a) usecase end to end that would be a > > > great starting place. > > > > > > Thanks again for your ideas! > > > > > > I look forward to contributions from you on this. > > > > > > --larry > > > > > > > > > > > > > > > On Tue, Feb 4, 2014 at 4:22 PM, Ed Kohlwey <[email protected]> wrote: > > > > > > > I'm interested in implementing a knox service that implements full > > shell > > > > access to hosts within a hadoop cluster. The point of doing so is to > > > > provide administrative auditing capability for all cluster user > > accounts, > > > > including privileged users. A smaller number of users would have > access > > > to > > > > the knox host than the full cluster. > > > > > > > > My current thinking on how to do this is to implement a servlet. > Using > > a > > > > GET or POST request, the request body would be passed to standard in, > > and > > > > the response body would contain standard out. Knox itself would use > ssh > > > > public key based login to connect as a knox user to the other cluster > > > > hosts, and then use a combinaiton of sudo and exec commands to change > > the > > > > shell to the appropriate user. Knox would set up a hadoop delegation > > > token > > > > so the user could interact normally with Hadoop. > > > > > > > > Another option could be to start a java based SSH server on another > > > > (non-22) port and perform normal tunneling, probably by adding an > > > > interactive shell that asks what host the user would like to connect > > to. > > > > This has the advantage of basically looking like a normal SSH > > connection. > > > > > > > > I noticed this is not how the Knox DSL is implemented, which seems to > > use > > > > some session state. Can anyone discuss the motivations for doing > this? > > > Does > > > > the community have any opinions on how best to go about providing > > audited > > > > secure shell access? > > > > > > > > > >
