are you planning to expose things like hdfs browsing and nn/dn logs over jmx?
thx Alejandro (phone typing) On Oct 28, 2013, at 17:48, Haohui Mai <h...@hortonworks.com> wrote: > It seems more appealing to me that the UI should JMX directly, because: > > * We're support the JMX in the long term for other management software. > * The information provided by the JMX API will be mostly identical of the > JSON API. Today the Web UI covers most of the information provided by JMX. > The Web UI does some trivial work to extract the information and renders it > as HTML. > * We can compatibility and unit tests for free. > > I do agree that the JMX APIs are imperfect and they should be revisited in > the 3.0 timeframe. However, this is orthogonal of the discussions of > transitioning from JSP-based Web UI to client-side JavaScript Web UI. The > architecture of the new Web UI allows easy migration to any JSON-based APIs > whenever they land in the trunk. > > Thanks, > Haohui > > > On Mon, Oct 28, 2013 at 5:13 PM, Alejandro Abdelnur <t...@cloudera.com>wrote: > >> Isn't using JMX to expose JSON for the web UI misusing JMX? >> >> I would think a more appropriate approach would be having /JMX for >> monitoring integration and a /JSON end point for the UI data. >> >> Thanks. >> >> >> On Mon, Oct 28, 2013 at 4:58 PM, Haohui Mai <h...@hortonworks.com> wrote: >> >>> Alejandro, >>> >>> If I understand correctly, that is the exact approach that the new web UI >>> is taking. The new web UI takes the output from JMX and renders them as >>> HTML at the client side. >>> >>> ~Haohui >>> >>> >>> On Mon, Oct 28, 2013 at 4:18 PM, Alejandro Abdelnur <t...@cloudera.com >>>> wrote: >>> >>>> Haohui, >>>> >>>> If you have NN and DNs producing JSON instead HTML, then you can build >> JS >>>> based web UIs. Take for example Oozie, Oozie produces JSON, it has a >>> built >>>> in JS web ui that consumes JSON and Hue has built an external web UI >> that >>>> also consumes JSON. In the case of Hue UI, Oozie didn't have to change >>>> anything to get that UI and improvements on the Hue UI don't require >>>> changes in Oozie unless it is to produce additional information. >>>> >>>> hope this clarifies. >>>> >>>> Thx >>>> >>>> >>>> On Mon, Oct 28, 2013 at 4:06 PM, Haohui Mai <h...@hortonworks.com> >>> wrote: >>>> >>>>> Echo my comments on HDFS-5402: >>>>> >>>>> bq. If we're going to remove the old web UI, I think the new web UI >> has >>>>> to have the same level of unit testing. We shouldn't go backwards in >>>>> terms of unit testing. >>>>> >>>>> I take a look at TestNamenodeJspHelper / TestDatanodeJspHelper / >>>>> TestClusterJspHelper. It seems to me that we can merge these tests >> with >>>> the >>>>> unit tests on JMX. >>>>> >>>>> bq. If we are going to >>>>> remove this capability, we need to add some other command-line tools >>>>> to get the same functionality. These tools could use REST if we have >>>>> that, or JMX, but they need to exist before we can consider removing >>>>> the old UI. >>>>> >>>>> This is a good point. Since all information are available through >> JMX, >>>> the >>>>> easiest way to approach it is to write some scripts using Node.js. >> The >>>>> architecture of the new Web UIs is ready for this. >>>>> >>>>> >>>>> On Mon, Oct 28, 2013 at 3:57 PM, Alejandro Abdelnur < >> t...@cloudera.com >>>>>> wrote: >>>>> >>>>>> Producing JSON would be great. Agree with Colin that we should >> leave >>>> for >>>>>> now the current JSP based web ui. >>>>>> >>>>>> thx >>>>>> >>>>>> >>>>>> On Mon, Oct 28, 2013 at 11:16 AM, Colin McCabe < >>> cmcc...@alumni.cmu.edu >>>>>>> wrote: >>>>>> >>>>>>> This is a really interesting project, Haohui. I think it will >> make >>>>>>> our web UI much nicer. >>>>>>> >>>>>>> I have a few concerns about removing the old web UI, however: >>>>>>> >>>>>>> * If we're going to remove the old web UI, I think the new web UI >>> has >>>>>>> to have the same level of unit testing. We shouldn't go >> backwards >>> in >>>>>>> terms of unit testing. >>>>>>> >>>>>>> * Most of the deployments of elinks and links out there don't >>> support >>>>>>> Javascript. This is just a reality of life when using CentOS 5 >> or >>> 6, >>>>>>> which many users are still using. I have used "links" to >> diagnose >>>>>>> problems through the web UI in the past, in systems where access >> to >>>>>>> the cluster was available only through telnet. If we are going >> to >>>>>>> remove this capability, we need to add some other command-line >>> tools >>>>>>> to get the same functionality. These tools could use REST if we >>> have >>>>>>> that, or JMX, but they need to exist before we can consider >>> removing >>>>>>> the old UI. >>>>>>> >>>>>>> best, >>>>>>> Colin >>>>>>> >>>>>>> On Fri, Oct 25, 2013 at 7:31 PM, Haohui Mai < >> h...@hortonworks.com> >>>>>> wrote: >>>>>>>> Thanks for the reply, Luke. Here I just echo my response from >> the >>>>> jira: >>>>>>>> >>>>>>>> bq. this client-side js only approach, which is less secure >> than >>> a >>>>>>>> progressively enhanced hybrid approach used by YARN. The recent >>>> gmail >>>>>>>> XSS fiasco highlights the issue. >>>>>>>> >>>>>>>> I'm presenting an informal security analysis to compare the >>>> security >>>>> of >>>>>>> the >>>>>>>> old and the new web UIs. >>>>>>>> >>>>>>>> An attacker launches an XSS attack by injecting malicious code >>>> which >>>>>> are >>>>>>>> usually HTML or JavaScript fragments into the web page, so that >>> the >>>>>>>> malicious code can have the same privileges of the web page. >>>>>>>> >>>>>>>> First, in the scope of XSS attacks, note that the threat models >>> of >>>>>>>> launching XSS attacks on Internet sites Gmail/Linkedin and the >>> one >>>> of >>>>>> the >>>>>>>> Hadoop UIs are different. They have fundamental different sets >> of >>>>>>> external >>>>>>>> inputs that the attackers have control to. Internet sites have >>>> little >>>>>>>> control of these inputs. In the case of Gmail / Linkedin, an >>> attack >>>>> can >>>>>>>> send you a crafted e-mail, or put malicious description in his >> / >>>>>>>> her Linkedin profile. The sets of external inputs are >>> *restricted* >>>> in >>>>>>>> Hadoop UIs. The new web UIs take JMX and WebHDFS as inputs. The >>>>>>>> attacker has to launch a XSS attack by: >>>>>>>> >>>>>>>> * Compromise the jars so that the output of JMX / WebHDFS have >>> the >>>>>>>> malicious code. >>>>>>>> * Replace the web UIs completely to include the malicious code. >>>>>>>> >>>>>>>> In either case *the attacker has to compromise the hadoop core >> or >>>> the >>>>>>>> namenode*. That means the new web UIs are at least as secure as >>> the >>>>>>> hadoop >>>>>>>> core, and the namenode machine. >>>>>>>> >>>>>>>> Second, I argue that using client-side templates are more >> secure >>>> than >>>>>> the >>>>>>>> current JSP-based server-side templates. To defend against XSS >>>>>>>> attacks, both techniques have to filter the external inputs at >>>>> *every* >>>>>>>> possible execution paths. Several facts much be taken into >>>>>>>> plays when evaluating the security of both approaches in >>> real-world >>>>>>>> environments: >>>>>>>> >>>>>>>> * The JavaScript libraries used in the new web UIs have >> survived >>> in >>>>>>>> extremely large-scale production tests. jQuery is used by >> Google >>>> and >>>>>>>> Microsoft, bootstrap is used by Twitter, and dust.js is used >> by >>>>>>> Linkedin. >>>>>>>> All libraries survived from hundreds of thousands of >>>>>>>> attack attempts on a daily basis. I agree that the libraries >>> might >>>>>> still >>>>>>>> be imperfect, but there's no way that we can test the JSP web >>>>>>>> UIs to achieve the same level of assurances given the amount >> of >>>>>>> resources >>>>>>>> the community has. >>>>>>>> * Client-side templates consolidate all filtering logic in one >>>>> central >>>>>>>> place. Recall that the goal is to filter all external inputs at >>>> every >>>>>>>> execution paths, this is a much more systematic approach >>> compared >>>> to >>>>>> the >>>>>>>> server-side templates we have today. It is difficult (if not >>>>>>>> impossible) to do it in a JSP/ASP/PHP application, since such >>>>>> filtering >>>>>>>> can be only achieved via ad-hoc approaches ([1] shows some >>>>>>>> empirical data). Also, HDFS-4901 recently describes a XSS >>>>>> vulnerability >>>>>>> in >>>>>>>> browseDirectory.jsp. >>>>>>>> >>>>>>>> bq. You'd require proper SSL (not self signed) setup to avoid >> JS >>>>>>>> injection >>>>>>>> >>>>>>>> Commodity browsers enforce Same-Origin Policy to defend against >>>> code >>>>>>>> injections. It has nothing to do with what kinds of SSL >>>> certificates >>>>>>>> you hold. >>>>>>>> >>>>>>>> bq. I also have concerns that we commit these changes without >>>>> matching >>>>>>>> unit tests >>>>>>>> >>>>>>>> The JavaScript code can be automatically tested. The same code >>> can >>>> be >>>>>> run >>>>>>>> by node.js and the test can compared with pre-defined >>>>>>>> results. It is also possible to write an adapter to use Rhino >> to >>>>>>> accomplish >>>>>>>> the same task. We can discuss how to integrate them into >>>>>>>> the maven test routines in a different thread. >>>>>>>> >>>>>>>> bq. Client side rendering completely breaks the workflows for >> ops >>>> who >>>>>>> rely >>>>>>>> on text based terminal/emacs/vim browsers (no js support) to >>>>>>>> monitor component UI. >>>>>>>> >>>>>>>> links / elinks (http://elinks.or.cz/) are text-based web >>> browsers >>>>> that >>>>>>>> support JavaScript. >>>>>>>> >>>>>>>> bq. The priority/requirements for UI in core Hadoop should be >>>>> security >>>>>>> and >>>>>>>> correctness, which client side templating cannot address >> properly >>>>>>>> so far. >>>>>>>> >>>>>>>> I agree that we should focus on security and correctness. The >>>>>> paragraphs >>>>>>>> above explain that how the architecture of the new UIs >>>>>>>> makes the UIs more secure in real-world settings compared to >> the >>> UI >>>>> we >>>>>>> have >>>>>>>> today. >>>>>>>> >>>>>>>> References: >>>>>>>> >>>>>>>> 1. A. Yip et al. Improving Application Security with Data Flow >>>>>>> Assertions. >>>>>>>> In SOSP'2009. >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Oct 25, 2013 at 10:02 AM, Luke Lu <l...@apache.org> >>> wrote: >>>>>>>> >>>>>>>>> Echoing my comments on HDFS-3555: >>>>>>>>> >>>>>>>>> I have concerns with this client-side js only approach, which >> is >>>>> less >>>>>>>>> secure than a progressively enhanced hybrid approach used by >>> YARN. >>>>> The >>>>>>>>> recent gmail XSS fiasco highlights the issue. I also have >>> concerns >>>>>> that >>>>>>> we >>>>>>>>> commit these changes without matching unit tests – the fact >> you >>>>> cannot >>>>>>>>> effectively unit test these changes should tell you something >>>> about >>>>>> this >>>>>>>>> approach. >>>>>>>>> >>>>>>>>> *Requiring* JS means that an admin cannot turn off js to >>>> (partially) >>>>>> use >>>>>>>>> core Hadoop UI. You'd *require* proper SSL (not self signed) >>> setup >>>>> to >>>>>>> avoid >>>>>>>>> JS injection, even if security of js libraries used is >> perfect, >>>>> which >>>>>> I >>>>>>>>> doubt (search gmail/linkedin XSS). Client side rendering >>>> completely >>>>>>> breaks >>>>>>>>> the workflows for ops who rely on text based >> terminal/emacs/vim >>>>>> browsers >>>>>>>>> (no js support) to monitor component UI. >>>>>>>>> >>>>>>>>> IMO, JS-only rendering belongs to social networking sites >> and/or >>>>> SaaS >>>>>>>>> front-ends, where full time UI/security specialists babysits >> UI >>>>>>> changes. I >>>>>>>>> think eventually most users will use a self servicing UI in a >>> SaaS >>>>>>>>> front-end that uses REST/JMX API to get data from back-end >>>>> components, >>>>>>>>> besides their own app master/service UI. The >>> priority/requirements >>>>> for >>>>>>> UI >>>>>>>>> in core Hadoop should be security and correctness, which >> client >>>> side >>>>>>>>> templating cannot address properly so far. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Oct 22, 2013 at 3:59 PM, Haohui Mai < >>> h...@hortonworks.com >>>>> >>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Jing Zhao and I recently have reimplemented the JSP-based >> web >>>> UIs >>>>> in >>>>>>>>> HTML 5 >>>>>>>>>> applications (HDFS-5333). Based on our prelimanary testing >>>> results >>>>>> we >>>>>>>>>> believe thst the new web UIs of the namenodes and the >> datanode >>>> are >>>>>>> ready >>>>>>>>>> for everyday uses. >>>>>>>>>> >>>>>>>>>> You're more than welcome to try it out on trunk by visiting >>>>> http:// >>>>>>>>>> <namenode>/dfshealth.html >>>>>>>>>> >>>>>>>>>> There are a number of benefits from this transition. From a >>>>>>> developer's >>>>>>>>>> prospective, the most notable one is *maintainability*: >>>>>>>>>> >>>>>>>>>> (1) The abstractions between the UI and the core server are >>>>>>> well-defined, >>>>>>>>>> decoupling the UI and the core hadoop servers. >>>>>>>>>> >>>>>>>>>> (2) It allows us to deprecate the logic in the JSP pages. >> The >>>> old >>>>>> web >>>>>>> UIs >>>>>>>>>> have to duplicate the logic in the JSPs. The logic is often >>>>>>> out-of-dated >>>>>>>>>> and not well-tested, which leads to broken pages and >> security >>>>>>>>>> vulnerabilities(e.g. HDFS-5251, HDFS-5307, HDFS-5308, >>> HDFS-5317 >>>>> and >>>>>>>>>> HDFS-4901). The architecture of the new UIs prevent these >> bugs >>>> at >>>>>> the >>>>>>>>> very >>>>>>>>>> beginning. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I propose that deprecate the old, JSP-based web UIs in 2.3. >> I >>>>> opened >>>>>>>>>> HDFS-5402 to track the relevant discussions. >>>>>>>>>> >>>>>>>>>> Your feedbacks are highly appreciated. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Sincerely, >>>>>>>>>> >>>>>>>>>> Haohui >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> CONFIDENTIALITY NOTICE >>>>>>>>>> NOTICE: This message is intended for the use of the >> individual >>>> or >>>>>>> entity >>>>>>>>> to >>>>>>>>>> which it is addressed and may contain information that is >>>>>>> confidential, >>>>>>>>>> privileged and exempt from disclosure under applicable law. >> If >>>> the >>>>>>> reader >>>>>>>>>> of this message is not the intended recipient, you are >> hereby >>>>>> notified >>>>>>>>> that >>>>>>>>>> any printing, copying, dissemination, distribution, >> disclosure >>>> or >>>>>>>>>> forwarding of this communication is strictly prohibited. If >>> you >>>>> have >>>>>>>>>> received this communication in error, please contact the >>> sender >>>>>>>>> immediately >>>>>>>>>> and delete it from your system. Thank You. >>>>>>>> >>>>>>>> -- >>>>>>>> CONFIDENTIALITY NOTICE >>>>>>>> NOTICE: This message is intended for the use of the individual >> or >>>>>> entity >>>>>>> to >>>>>>>> which it is addressed and may contain information that is >>>>> confidential, >>>>>>>> privileged and exempt from disclosure under applicable law. If >>> the >>>>>> reader >>>>>>>> of this message is not the intended recipient, you are hereby >>>>> notified >>>>>>> that >>>>>>>> any printing, copying, dissemination, distribution, disclosure >> or >>>>>>>> forwarding of this communication is strictly prohibited. If you >>>> have >>>>>>>> received this communication in error, please contact the sender >>>>>>> immediately >>>>>>>> and delete it from your system. Thank You. >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Alejandro >>>>> >>>>> -- >>>>> CONFIDENTIALITY NOTICE >>>>> NOTICE: This message is intended for the use of the individual or >>> entity >>>> to >>>>> which it is addressed and may contain information that is >> confidential, >>>>> privileged and exempt from disclosure under applicable law. If the >>> reader >>>>> of this message is not the intended recipient, you are hereby >> notified >>>> that >>>>> any printing, copying, dissemination, distribution, disclosure or >>>>> forwarding of this communication is strictly prohibited. If you have >>>>> received this communication in error, please contact the sender >>>> immediately >>>>> and delete it from your system. Thank You. >>>> >>>> >>>> >>>> -- >>>> Alejandro >>> >>> -- >>> CONFIDENTIALITY NOTICE >>> NOTICE: This message is intended for the use of the individual or entity >> to >>> which it is addressed and may contain information that is confidential, >>> privileged and exempt from disclosure under applicable law. If the reader >>> of this message is not the intended recipient, you are hereby notified >> that >>> any printing, copying, dissemination, distribution, disclosure or >>> forwarding of this communication is strictly prohibited. If you have >>> received this communication in error, please contact the sender >> immediately >>> and delete it from your system. Thank You. >> >> >> >> -- >> Alejandro > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You.
