Isn't using JMX to expose JSON for the web UI misusing JMX? I would think a more appropriate approach would be having /JMX for monitoring integration and a /JSON end point for the UI data.
Thanks. On Mon, Oct 28, 2013 at 4:58 PM, Haohui Mai <h...@hortonworks.com> wrote: > Alejandro, > > If I understand correctly, that is the exact approach that the new web UI > is taking. The new web UI takes the output from JMX and renders them as > HTML at the client side. > > ~Haohui > > > On Mon, Oct 28, 2013 at 4:18 PM, Alejandro Abdelnur <t...@cloudera.com > >wrote: > > > Haohui, > > > > If you have NN and DNs producing JSON instead HTML, then you can build JS > > based web UIs. Take for example Oozie, Oozie produces JSON, it has a > built > > in JS web ui that consumes JSON and Hue has built an external web UI that > > also consumes JSON. In the case of Hue UI, Oozie didn't have to change > > anything to get that UI and improvements on the Hue UI don't require > > changes in Oozie unless it is to produce additional information. > > > > hope this clarifies. > > > > Thx > > > > > > On Mon, Oct 28, 2013 at 4:06 PM, Haohui Mai <h...@hortonworks.com> > wrote: > > > > > Echo my comments on HDFS-5402: > > > > > > bq. If we're going to remove the old web UI, I think the new web UI has > > > to have the same level of unit testing. We shouldn't go backwards in > > > terms of unit testing. > > > > > > I take a look at TestNamenodeJspHelper / TestDatanodeJspHelper / > > > TestClusterJspHelper. It seems to me that we can merge these tests with > > the > > > unit tests on JMX. > > > > > > bq. If we are going to > > > remove this capability, we need to add some other command-line tools > > > to get the same functionality. These tools could use REST if we have > > > that, or JMX, but they need to exist before we can consider removing > > > the old UI. > > > > > > This is a good point. Since all information are available through JMX, > > the > > > easiest way to approach it is to write some scripts using Node.js. The > > > architecture of the new Web UIs is ready for this. > > > > > > > > > On Mon, Oct 28, 2013 at 3:57 PM, Alejandro Abdelnur <t...@cloudera.com > > > >wrote: > > > > > > > Producing JSON would be great. Agree with Colin that we should leave > > for > > > > now the current JSP based web ui. > > > > > > > > thx > > > > > > > > > > > > On Mon, Oct 28, 2013 at 11:16 AM, Colin McCabe < > cmcc...@alumni.cmu.edu > > > > >wrote: > > > > > > > > > This is a really interesting project, Haohui. I think it will make > > > > > our web UI much nicer. > > > > > > > > > > I have a few concerns about removing the old web UI, however: > > > > > > > > > > * If we're going to remove the old web UI, I think the new web UI > has > > > > > to have the same level of unit testing. We shouldn't go backwards > in > > > > > terms of unit testing. > > > > > > > > > > * Most of the deployments of elinks and links out there don't > support > > > > > Javascript. This is just a reality of life when using CentOS 5 or > 6, > > > > > which many users are still using. I have used "links" to diagnose > > > > > problems through the web UI in the past, in systems where access to > > > > > the cluster was available only through telnet. If we are going to > > > > > remove this capability, we need to add some other command-line > tools > > > > > to get the same functionality. These tools could use REST if we > have > > > > > that, or JMX, but they need to exist before we can consider > removing > > > > > the old UI. > > > > > > > > > > best, > > > > > Colin > > > > > > > > > > On Fri, Oct 25, 2013 at 7:31 PM, Haohui Mai <h...@hortonworks.com> > > > > wrote: > > > > > > Thanks for the reply, Luke. Here I just echo my response from the > > > jira: > > > > > > > > > > > > bq. this client-side js only approach, which is less secure than > a > > > > > > progressively enhanced hybrid approach used by YARN. The recent > > gmail > > > > > > XSS fiasco highlights the issue. > > > > > > > > > > > > I'm presenting an informal security analysis to compare the > > security > > > of > > > > > the > > > > > > old and the new web UIs. > > > > > > > > > > > > An attacker launches an XSS attack by injecting malicious code > > which > > > > are > > > > > > usually HTML or JavaScript fragments into the web page, so that > the > > > > > > malicious code can have the same privileges of the web page. > > > > > > > > > > > > First, in the scope of XSS attacks, note that the threat models > of > > > > > > launching XSS attacks on Internet sites Gmail/Linkedin and the > one > > of > > > > the > > > > > > Hadoop UIs are different. They have fundamental different sets of > > > > > external > > > > > > inputs that the attackers have control to. Internet sites have > > little > > > > > > control of these inputs. In the case of Gmail / Linkedin, an > attack > > > can > > > > > > send you a crafted e-mail, or put malicious description in his / > > > > > > her Linkedin profile. The sets of external inputs are > *restricted* > > in > > > > > > Hadoop UIs. The new web UIs take JMX and WebHDFS as inputs. The > > > > > > attacker has to launch a XSS attack by: > > > > > > > > > > > > * Compromise the jars so that the output of JMX / WebHDFS have > the > > > > > > malicious code. > > > > > > * Replace the web UIs completely to include the malicious code. > > > > > > > > > > > > In either case *the attacker has to compromise the hadoop core or > > the > > > > > > namenode*. That means the new web UIs are at least as secure as > the > > > > > hadoop > > > > > > core, and the namenode machine. > > > > > > > > > > > > Second, I argue that using client-side templates are more secure > > than > > > > the > > > > > > current JSP-based server-side templates. To defend against XSS > > > > > > attacks, both techniques have to filter the external inputs at > > > *every* > > > > > > possible execution paths. Several facts much be taken into > > > > > > plays when evaluating the security of both approaches in > real-world > > > > > > environments: > > > > > > > > > > > > * The JavaScript libraries used in the new web UIs have survived > in > > > > > > extremely large-scale production tests. jQuery is used by Google > > and > > > > > > Microsoft, bootstrap is used by Twitter, and dust.js is used by > > > > > Linkedin. > > > > > > All libraries survived from hundreds of thousands of > > > > > > attack attempts on a daily basis. I agree that the libraries > might > > > > still > > > > > > be imperfect, but there's no way that we can test the JSP web > > > > > > UIs to achieve the same level of assurances given the amount of > > > > > resources > > > > > > the community has. > > > > > > * Client-side templates consolidate all filtering logic in one > > > central > > > > > > place. Recall that the goal is to filter all external inputs at > > every > > > > > > execution paths, this is a much more systematic approach > compared > > to > > > > the > > > > > > server-side templates we have today. It is difficult (if not > > > > > > impossible) to do it in a JSP/ASP/PHP application, since such > > > > filtering > > > > > > can be only achieved via ad-hoc approaches ([1] shows some > > > > > > empirical data). Also, HDFS-4901 recently describes a XSS > > > > vulnerability > > > > > in > > > > > > browseDirectory.jsp. > > > > > > > > > > > > bq. You'd require proper SSL (not self signed) setup to avoid JS > > > > > > injection > > > > > > > > > > > > Commodity browsers enforce Same-Origin Policy to defend against > > code > > > > > > injections. It has nothing to do with what kinds of SSL > > certificates > > > > > > you hold. > > > > > > > > > > > > bq. I also have concerns that we commit these changes without > > > matching > > > > > > unit tests > > > > > > > > > > > > The JavaScript code can be automatically tested. The same code > can > > be > > > > run > > > > > > by node.js and the test can compared with pre-defined > > > > > > results. It is also possible to write an adapter to use Rhino to > > > > > accomplish > > > > > > the same task. We can discuss how to integrate them into > > > > > > the maven test routines in a different thread. > > > > > > > > > > > > bq. Client side rendering completely breaks the workflows for ops > > who > > > > > rely > > > > > > on text based terminal/emacs/vim browsers (no js support) to > > > > > > monitor component UI. > > > > > > > > > > > > links / elinks (http://elinks.or.cz/) are text-based web > browsers > > > that > > > > > > support JavaScript. > > > > > > > > > > > > bq. The priority/requirements for UI in core Hadoop should be > > > security > > > > > and > > > > > > correctness, which client side templating cannot address properly > > > > > > so far. > > > > > > > > > > > > I agree that we should focus on security and correctness. The > > > > paragraphs > > > > > > above explain that how the architecture of the new UIs > > > > > > makes the UIs more secure in real-world settings compared to the > UI > > > we > > > > > have > > > > > > today. > > > > > > > > > > > > References: > > > > > > > > > > > > 1. A. Yip et al. Improving Application Security with Data Flow > > > > > Assertions. > > > > > > In SOSP'2009. > > > > > > > > > > > > > > > > > > On Fri, Oct 25, 2013 at 10:02 AM, Luke Lu <l...@apache.org> > wrote: > > > > > > > > > > > >> Echoing my comments on HDFS-3555: > > > > > >> > > > > > >> I have concerns with this client-side js only approach, which is > > > less > > > > > >> secure than a progressively enhanced hybrid approach used by > YARN. > > > The > > > > > >> recent gmail XSS fiasco highlights the issue. I also have > concerns > > > > that > > > > > we > > > > > >> commit these changes without matching unit tests – the fact you > > > cannot > > > > > >> effectively unit test these changes should tell you something > > about > > > > this > > > > > >> approach. > > > > > >> > > > > > >> *Requiring* JS means that an admin cannot turn off js to > > (partially) > > > > use > > > > > >> core Hadoop UI. You'd *require* proper SSL (not self signed) > setup > > > to > > > > > avoid > > > > > >> JS injection, even if security of js libraries used is perfect, > > > which > > > > I > > > > > >> doubt (search gmail/linkedin XSS). Client side rendering > > completely > > > > > breaks > > > > > >> the workflows for ops who rely on text based terminal/emacs/vim > > > > browsers > > > > > >> (no js support) to monitor component UI. > > > > > >> > > > > > >> IMO, JS-only rendering belongs to social networking sites and/or > > > SaaS > > > > > >> front-ends, where full time UI/security specialists babysits UI > > > > > changes. I > > > > > >> think eventually most users will use a self servicing UI in a > SaaS > > > > > >> front-end that uses REST/JMX API to get data from back-end > > > components, > > > > > >> besides their own app master/service UI. The > priority/requirements > > > for > > > > > UI > > > > > >> in core Hadoop should be security and correctness, which client > > side > > > > > >> templating cannot address properly so far. > > > > > >> > > > > > >> > > > > > >> On Tue, Oct 22, 2013 at 3:59 PM, Haohui Mai < > h...@hortonworks.com > > > > > > > > wrote: > > > > > >> > > > > > >> > Hi all, > > > > > >> > > > > > > >> > > > > > > >> > Jing Zhao and I recently have reimplemented the JSP-based web > > UIs > > > in > > > > > >> HTML 5 > > > > > >> > applications (HDFS-5333). Based on our prelimanary testing > > results > > > > we > > > > > >> > believe thst the new web UIs of the namenodes and the datanode > > are > > > > > ready > > > > > >> > for everyday uses. > > > > > >> > > > > > > >> > You're more than welcome to try it out on trunk by visiting > > > http:// > > > > > >> > <namenode>/dfshealth.html > > > > > >> > > > > > > >> > There are a number of benefits from this transition. From a > > > > > developer's > > > > > >> > prospective, the most notable one is *maintainability*: > > > > > >> > > > > > > >> > (1) The abstractions between the UI and the core server are > > > > > well-defined, > > > > > >> > decoupling the UI and the core hadoop servers. > > > > > >> > > > > > > >> > (2) It allows us to deprecate the logic in the JSP pages. The > > old > > > > web > > > > > UIs > > > > > >> > have to duplicate the logic in the JSPs. The logic is often > > > > > out-of-dated > > > > > >> > and not well-tested, which leads to broken pages and security > > > > > >> > vulnerabilities(e.g. HDFS-5251, HDFS-5307, HDFS-5308, > HDFS-5317 > > > and > > > > > >> > HDFS-4901). The architecture of the new UIs prevent these bugs > > at > > > > the > > > > > >> very > > > > > >> > beginning. > > > > > >> > > > > > > >> > > > > > > >> > I propose that deprecate the old, JSP-based web UIs in 2.3. I > > > opened > > > > > >> > HDFS-5402 to track the relevant discussions. > > > > > >> > > > > > > >> > Your feedbacks are highly appreciated. > > > > > >> > > > > > > >> > > > > > > >> > Sincerely, > > > > > >> > > > > > > >> > Haohui > > > > > >> > > > > > > >> > -- > > > > > >> > CONFIDENTIALITY NOTICE > > > > > >> > NOTICE: This message is intended for the use of the individual > > or > > > > > entity > > > > > >> to > > > > > >> > which it is addressed and may contain information that is > > > > > confidential, > > > > > >> > privileged and exempt from disclosure under applicable law. If > > the > > > > > reader > > > > > >> > of this message is not the intended recipient, you are hereby > > > > notified > > > > > >> that > > > > > >> > any printing, copying, dissemination, distribution, disclosure > > or > > > > > >> > forwarding of this communication is strictly prohibited. If > you > > > have > > > > > >> > received this communication in error, please contact the > sender > > > > > >> immediately > > > > > >> > and delete it from your system. Thank You. > > > > > >> > > > > > > >> > > > > > > > > > > > > -- > > > > > > CONFIDENTIALITY NOTICE > > > > > > NOTICE: This message is intended for the use of the individual or > > > > entity > > > > > to > > > > > > which it is addressed and may contain information that is > > > confidential, > > > > > > privileged and exempt from disclosure under applicable law. If > the > > > > reader > > > > > > of this message is not the intended recipient, you are hereby > > > notified > > > > > that > > > > > > any printing, copying, dissemination, distribution, disclosure or > > > > > > forwarding of this communication is strictly prohibited. If you > > have > > > > > > received this communication in error, please contact the sender > > > > > immediately > > > > > > and delete it from your system. Thank You. > > > > > > > > > > > > > > > > > > > > > -- > > > > Alejandro > > > > > > > > > > -- > > > CONFIDENTIALITY NOTICE > > > NOTICE: This message is intended for the use of the individual or > entity > > to > > > which it is addressed and may contain information that is confidential, > > > privileged and exempt from disclosure under applicable law. If the > reader > > > of this message is not the intended recipient, you are hereby notified > > that > > > any printing, copying, dissemination, distribution, disclosure or > > > forwarding of this communication is strictly prohibited. If you have > > > received this communication in error, please contact the sender > > immediately > > > and delete it from your system. Thank You. > > > > > > > > > > > -- > > Alejandro > > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. > -- Alejandro
