Yes you are right. If it is true by default we probably want to update the documentation for the web services to indicate this. Could you file a JIRA for improving that documentation?
Thanks, Bobby On 7/27/12 3:11 AM, "Prajakta Kalmegh" <pkalm...@gmail.com> wrote: >:) Yes, you are right. The yarn.acl.enable property in yarn-default.xml is >set true. If the property is true by default, then this makes it mandatory >for users to either specify a value for hadoop.http.staticuser.user >property explicitly or to change the acl's to false. Am I right to assume >this? > >Regards, >Prajakta > > > >On Thu, Jul 26, 2012 at 11:59 PM, Robert Evans <ev...@yahoo-inc.com> >wrote: > >> OK I think I understand it now. You probably have ACLs enabled, but no >> web filter on the RM to let you sign in as a given user. As such the >> default filter is making you be Dr. Who, or whomever else it is, but the >> ACL check in the web service is rejecting Dr Who, because that is not >>the >> correct user. You will probably run into this issue again if anyone >>else >> but you runs something. You could fix this by either disabling the ACL >> check, which makes a lot of since for a cluster without security, or you >> could implement a servlet Filter for the RM that would let you sign on >>as >> a given user. >> >> --Bobby Evans >> >> >> On 7/26/12 12:48 AM, "Prajakta Kalmegh" <pkalm...@gmail.com> wrote: >> >> >Hi Bobby >> > >> >Thanks for the reply. My REST calls are working fine since I set the >> >'hadoop.http.staticuser.user' property to 'prajakta' instead of Dr.Who >>in >> >core-site.xml . I didn't get time to figure out the reason behind it >>as I >> >just moved on to further coding :) >> > >> >Thanks, >> >Prajakta >> > >> > >> > >> >On Thu, Jul 26, 2012 at 1:40 AM, Robert Evans <ev...@yahoo-inc.com> >> wrote: >> > >> >> Hmm, that is very odd. It only checks the user if security is >>enabled >> >>to >> >> warn the user about potentially accessing something unsafe. I am not >> >>sure >> >> why that would cause an issue. >> >> >> >> --Bobby Evans >> >> >> >> On 7/9/12 6:07 AM, "Prajakta Kalmegh" <pkalm...@gmail.com> wrote: >> >> >> >> >Hi Robert >> >> > >> >> >I figured out the problem just now. To avoid the below error, I had >>to >> >>set >> >> >the 'hadoop.http.staticuser.user' property in core-site.xml >>(defaults >> >>to >> >> >dr.who). I can now get runtime data from AppMaster using *curl* as >> >>well as >> >> >in GUI. >> >> > >> >> >I wonder if we have to set this property even when we are not >> >>specifying >> >> >the yarn web-proxy address (when it runs as part of RM by default) >>as >> >> >well. >> >> >If yes, was it documented somewhere which I failed to see? :( >> >> > >> >> >Anyways, thanks for your response so far. >> >> > >> >> >Regards, >> >> >Prajakta >> >> > >> >> > >> >> > >> >> >On Mon, Jul 9, 2012 at 3:29 PM, Prajakta Kalmegh >><pkalm...@gmail.com> >> >> >wrote: >> >> > >> >> >> Hi Robert >> >> >> >> >> >> I started the proxyserver explicitly by specifying a value for the >> >> >> yarn.web-proxy.address in yarn-site.xml. The proxyserver did start >> >>and I >> >> >> tried getting the JSON response using the following command : >> >> >> >> >> >> curl --compressed -H "Accept: application/json" -X GET " >> >> >> >> >> >> >> >> >> >> >> >>http://localhost:8090/proxy/application_1341823967331_0001/ws/v1/mapreduc >> >> >>e/jobs/job_1341823967331_0001 >> >> >> " >> >> >> >> >> >> However, it refused connection and below is the excerpt from the >> >> >> Proxyserver logs: >> >> >> --------- >> >> >> 2012-07-09 14:26:40,402 INFO org.mortbay.log: Extract >> >> >> >> >> >> >>>>>>jar:file:/home/prajakta/Projects/IRL/hadoop-common/hadoop-dist/target >>>>>>/h >> >>>>ad >> >> >> >>>>>>oop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-yarn-common-3.0.0-SN >>>>>>AP >> >>>>SH >> >> >>OT.jar!/webapps/proxy >> >> >> to /tmp/Jetty_localhost_8090_proxy____.ak3o30/webapp >> >> >> 2012-07-09 14:26:40,992 INFO org.mortbay.log: Started >> >> >> SelectChannelConnector@localhost:8090 >> >> >> 2012-07-09 14:26:40,993 INFO >> >> >> org.apache.hadoop.yarn.service.AbstractService: >> >> >> Service:org.apache.hadoop.yarn.server.webproxy.WebAppProxy is >> >>started. >> >> >> 2012-07-09 14:26:40,993 INFO >> >> >> org.apache.hadoop.yarn.service.AbstractService: >> >> >> Service:org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer >>is >> >> >>started. >> >> >> 2012-07-09 14:33:26,039 INFO >> >> >> org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet: dr.who >>is >> >> >> accessing unchecked >> >> >> http://prajakta:44314/ws/v1/mapreduce/jobs/job_1341823967331_0001 >> >>which >> >> >> is the app master GUI of application_1341823967331_0001 owned by >> >> >>prajakta >> >> >> 2012-07-09 14:33:29,277 INFO >> >> >> org.apache.commons.httpclient.HttpMethodDirector: I/O exception >> >> >> (org.apache.commons.httpclient.NoHttpResponseException) caught >>when >> >> >> processing request: The server prajakta failed to respond >> >> >> 2012-07-09 14:33:29,277 INFO >> >> >> org.apache.commons.httpclient.HttpMethodDirector: Retrying request >> >> >> 2012-07-09 14:33:29,284 WARN org.mortbay.log: >> >> >> >> >> >> >>>>>>/proxy/application_1341823967331_0001/ws/v1/mapreduce/jobs/job_134182 >>>>>>39 >> >>>>67 >> >> >>331_0001: >> >> >> java.net.SocketException: Connection reset >> >> >> 2012-07-09 14:37:33,834 INFO >> >> >> org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet: dr.who >>is >> >> >> accessing unchecked >> >> >> >> >> >> >> >> >> >> >> >>http://prajakta:19888/jobhistory/job/job_1341823967331_0001/jobhistory/jo >> >> >>b/job_1341823967331_0001which is the app master GUI of >> >> >>application_1341823967331_0001 owned by >> >> >> prajakta >> >> >> --------------- >> >> >> >> >> >> I am not sure why http request object is setting my remoteUser to >> >> >>dr.who. >> >> >> :( >> >> >> >> >> >> I gather from >><https://issues.apache.org/jira/browse/MAPREDUCE-2858> >> >> >>that >> >> >> this warning is posted only in case where security is disabled. I >> >>assume >> >> >> that the proxy server is not disabled if security is disabled. >> >> >> >> >> >> Any idea what could be the reason for this I/O exception? Am I >> >>missing >> >> >> setting any property for proper access. Please let me know. >> >> >> >> >> >> Regards, >> >> >> Prajakta >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Fri, Jul 6, 2012 at 10:59 PM, Prajakta Kalmegh >> >> >><pkalm...@gmail.com>wrote: >> >> >> >> >> >>> I am using hadoop trunk (forked from github). It supports RESTful >> >>APIs >> >> >>>as >> >> >>> I am able to retrieve JSON objects for RM (cluster/nodes info)+ >> >> >>> Historyserver. The only issue is with AppMaster REST API. >> >> >>> >> >> >>> Regards, >> >> >>> Prajakta >> >> >>> >> >> >>> >> >> >>> >> >> >>> On Fri, Jul 6, 2012 at 10:55 PM, Robert Evans >> >> >>><ev...@yahoo-inc.com>wrote: >> >> >>> >> >> >>>> What version of hadoop are you using? It could be that the >>version >> >> >>>>you >> >> >>>> have does not have the RESTful APIs in it yet, and the proxy is >> >> >>>>working >> >> >>>> just fine. >> >> >>>> >> >> >>>> --Bobby Evans >> >> >>>> >> >> >>>> On 7/6/12 12:06 PM, "Prajakta Kalmegh" <pkalm...@gmail.com> >>wrote: >> >> >>>> >> >> >>>> >Robert , Thanks for the response. If I do not provide any >>explicit >> >> >>>> >configuration for the proxy server, do I still need to start it >> >>using >> >> >>>> the >> >> >>>> >'yarn start proxy server'? I am currently not doing it. >> >> >>>> > >> >> >>>> >Also, I am able to access the html page for proxy using the >> >> >>>> ><http://localhost:8088/proxy/{appid}/mapreduce/jobs> URL. (Note >> >>this >> >> >>>> url >> >> >>>> >does not have the '/ws/v1/ part in it. I get the html response >> >>when I >> >> >>>> >query >> >> >>>> >for this URL in runtime. >> >> >>>> > >> >> >>>> >So I assume the proxy server must be starting fine since I am >> >>able to >> >> >>>> >access this URL. I will try logging more details tomorrow from >>my >> >> >>>>office >> >> >>>> >machine and will let you know the result. >> >> >>>> > >> >> >>>> >Regards, >> >> >>>> >Prajakta >> >> >>>> > >> >> >>>> > >> >> >>>> > >> >> >>>> >On Fri, Jul 6, 2012 at 10:22 PM, Robert Evans >> >><ev...@yahoo-inc.com> >> >> >>>> wrote: >> >> >>>> > >> >> >>>> >> Sorry I did not respond sooner. The default behavior is to >>have >> >> >>>>the >> >> >>>> >>proxy >> >> >>>> >> server run as part of the RM. I am not really sure why it is >> >>not >> >> >>>> doing >> >> >>>> >> this in your case. If you set the config yourself to be a >>URI >> >> >>>>that is >> >> >>>> >> different from that of the RM then you need to launch a >> >>standalone >> >> >>>> proxy >> >> >>>> >> server. You can do this by running >> >> >>>> >> >> >> >>>> >> yarn start proxy server >> >> >>>> >> >> >> >>>> >> Without sitting down with you it is going to be somewhat >> >>difficult >> >> >>>>to >> >> >>>> >> debug why this is happening. However, in retrospect it >>would be >> >> >>>>nice >> >> >>>> to >> >> >>>> >> add in some extra logging to help indicate why the proxy >>server >> >>is >> >> >>>>not >> >> >>>> >> functioning as desired. If you could file a JIRA to add in >>the >> >> >>>> logging >> >> >>>> >>I >> >> >>>> >> would be happy to provide a patch to you and we can try and >> >>debug >> >> >>>>the >> >> >>>> >> issue further. Please file it under the MAPREDUCE JIRA >>project. >> >> >>>> >> >> >> >>>> >> --Bobby >> >> >>>> >> >> >> >>>> >> On 7/6/12 3:29 AM, "Prajakta Kalmegh" <pkalm...@gmail.com> >> >>wrote: >> >> >>>> >> >> >> >>>> >> >Re-posting as I haven't got a solution yet. Sorry for >> >>spamming. I >> >> >>>> >>won't be >> >> >>>> >> >able to proceed in my code until I get a JSON response using >> >> >>>> AppMaster >> >> >>>> >> >REST >> >> >>>> >> >URL. :( >> >> >>>> >> > >> >> >>>> >> >Thanks, >> >> >>>> >> >Prajakta >> >> >>>> >> > >> >> >>>> >> > >> >> >>>> >> >On Wed, Jul 4, 2012 at 5:55 PM, Prajakta Kalmegh >> >> >>>><pkalm...@gmail.com >> >> >>>> > >> >> >>>> >> >wrote: >> >> >>>> >> > >> >> >>>> >> >> Hi Robert/Harsh >> >> >>>> >> >> >> >> >>>> >> >> Thanks for your reply. >> >> >>>> >> >> >> >> >>>> >> >> My RM is starting just fine. The problem is with the use >>of >> >> >>>> >> >>http://<proxy >>httpddress:port>/proxy/{appid}/ws/v1/mapreduce >> >> >>>> >> >> to get the JSON response. >> >> >>>> >> >> >> >> >>>> >> >> As I said before, I had not configured the >> >> >>>>yarn.web-proxy.address >> >> >>>> >> >>property in yarn-site.xml. I assumed it will use the RM's >> >> >>>> >> >>yarn.resourcemanager.webapp.address property value as >>default. >> >> >>>> >>However, >> >> >>>> >> >>it gives me a '404-Page not found error'. Today I tried >> >> >>>>specifying >> >> >>>> a >> >> >>>> >> >>value explicitly for the yarn.web-proxy.address property. >> >> >>>> >> >> >> >> >>>> >> >> On running the wordcount example, it even gives a url >> >> >>>> >> >><http://localhost:8090>/proxy/{appid}/> to track the App >>Mast >> >> >>>>info. >> >> >>>> >> >>However, I am still not able to get a json response. >> >> >>>> >> >> >> >> >>>> >> >> Also, I tried to get the data from historyserver instead >>of >> >> >>>>runtime >> >> >>>> >> >>using the instructions given on page >> >> >>>> >> >>< >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >>http://hadoop.apache.org/common/docs/r2.0.0-alpha/hadoop-yarn/hadoop-ya >> >> >>>>r >> >> >>>> >> >>n-site/HistoryServerRest.html> >> >> >>>> >> >> >> >> >>>> >> >> HistoryServer REST response does not give me jobids >> >> >>>>corresponding >> >> >>>> to >> >> >>>> >>an >> >> >>>> >> >>application. It just lists all the jobs run until now. By >>the >> >> >>>>way, >> >> >>>> the >> >> >>>> >> >>documentation does say >> >> >>>> >> >> >> >> >>>> >> >> ---------- >> >> >>>> >> >> >> >> >>>> >> >> "Both of the following URI's give you the history server >> >> >>>> information, >> >> >>>> >> >>from an application id identified by the appid value. >> >> >>>> >> >> * http://<history server http >>address:port>/ws/v1/history >> >> >>>> >> >> * http://<history server http >> >> >>>>address:port>/ws/v1/history/info" >> >> >>>> >> >> --------- >> >> >>>> >> >> >> >> >>>> >> >> But there is no provision to specify the application id >>with >> >> >>>>these >> >> >>>> >>REST >> >> >>>> >> >>URLs. >> >> >>>> >> >> >> >> >>>> >> >> Any idea how I can get the Application Master REST working >> >>and >> >> >>>>also >> >> >>>> >> >>linking jobids to application id using the >>HistoryServerREST >> >>API? >> >> >>>> >> >> >> >> >>>> >> >> Any help is appreciated. Thanks in advance. >> >> >>>> >> >> Regards, >> >> >>>> >> >> Prajakta >> >> >>>> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> >>>> >> >> On Fri, Jun 29, 2012 at 8:55 PM, Robert Evans >> >> >>>><ev...@yahoo-inc.com >> >> >>>> > >> >> >>>> >> >>wrote: >> >> >>>> >> >> >> >> >>>> >> >>> Please don't file that JIRA. The proxy server is >>intended >> >>to >> >> >>>> front >> >> >>>> >>the >> >> >>>> >> >>> web server for all calls to the AM. This is so you only >> >>have >> >> >>>>to >> >> >>>> go >> >> >>>> >>to >> >> >>>> >> >>>a >> >> >>>> >> >>> single location to get to any AM's web service. The >>proxy >> >> >>>>server >> >> >>>> >>is a >> >> >>>> >> >>> very simple proxy and just forwards the extra part of the >> >>path >> >> >>>>on >> >> >>>> to >> >> >>>> >> >>>the >> >> >>>> >> >>> AM. >> >> >>>> >> >>> >> >> >>>> >> >>> If you are having issues with this please include the >> >>version >> >> >>>>you >> >> >>>> >>are >> >> >>>> >> >>> having problems with. Also please look at the logs for >>the >> >>RM >> >> >>>>on >> >> >>>> >> >>>startup >> >> >>>> >> >>> to see if there is anything there indicating why it is >>not >> >> >>>> starting >> >> >>>> >>up. >> >> >>>> >> >>> >> >> >>>> >> >>> --Bobby Evans >> >> >>>> >> >>> >> >> >>>> >> >>> On 6/28/12 9:46 AM, "Harsh J" <ha...@cloudera.com> wrote: >> >> >>>> >> >>> >> >> >>>> >> >>> >As far as I can tell, the MR WebApp, as the name itself >> >> >>>>indicates >> >> >>>> >>on >> >> >>>> >> >>> >its doc page, starts only at the MR AM (which may be >> >>running >> >> >>>>at >> >> >>>> any >> >> >>>> >> >>> >NM), and it starts as an ephemeral port logged at in >>the AM >> >> >>>>logs >> >> >>>> >> >>> >usually as: >> >> >>>> >> >>> > >> >> >>>> >> >>> >INFO Web app /mapreduce started at [PORT] >> >> >>>> >> >>> > >> >> >>>> >> >>> >That it starts its own server with an ephemeral access >> >>point >> >> >>>> makes >> >> >>>> >> >>> >sense, since each job uses its own AM and having a >>common >> >> >>>> location >> >> >>>> >>may >> >> >>>> >> >>> >not work with the form of REST API documented at your >>link. >> >> >>>>Can >> >> >>>> you >> >> >>>> >> >>> >please file a JIRA to fix the doc and remove the proxy >> >>server >> >> >>>> refs, >> >> >>>> >> >>> >which are misleading? >> >> >>>> >> >>> > >> >> >>>> >> >>> >Do correct me if I'm wrong. >> >> >>>> >> >>> > >> >> >>>> >> >>> >On Thu, Jun 28, 2012 at 6:13 PM, Prajakta Kalmegh >> >> >>>> >><pkalm...@gmail.com >> >> >>>> >> > >> >> >>>> >> >>> >wrote: >> >> >>>> >> >>> >> Hi >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> I am trying to get the ApplicationMaster info using >>the >> >> >>>> >> >>><http://<proxy >> >> >>>> >> >>> >>http >> >> >>>> >> >>> >> address:port>/proxy/{appid}/ws/v1/mapreduce/info> >>link as >> >> >>>> >>described >> >> >>>> >> >>>on >> >> >>>> >> >>> >>the < >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >>http://hadoop.apache.org/common/docs/r2.0.0-alpha/hadoop-yarn/hadoop-ya >> >> >>>>r >> >> >>>> >> >>>n >> >> >>>> >> >>> >>-site/MapredAppMasterRest.html> >> >> >>>> >> >>> >> page. >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> I am able to access and retrieve JSON response for >>other >> >> >>>> modules >> >> >>>> >> >>> >> (ResourceManager, NodeManager and HistoryServer). >> >>However, >> >> >>>>I am >> >> >>>> >> >>>getting >> >> >>>> >> >>> >> 'Page not found' when I try to use my ResourceManager >> >>Http >> >> >>>> >>address >> >> >>>> >> >>>to >> >> >>>> >> >>> >> access the ApplicationMaster info. I am using < >> >> >>>> >> >>> >> >>http://localhost:8088/proxy/{appid}/ws/v1/mapreduce/info >> > >> >> to >> >> >>>> >> >>>retrieve >> >> >>>> >> >>> >>JSON >> >> >>>> >> >>> >> response. >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> The instructions say "The application master should be >> >> >>>>accessed >> >> >>>> >>via >> >> >>>> >> >>>the >> >> >>>> >> >>> >> proxy. This proxy is configurable to run either on the >> >> >>>>resource >> >> >>>> >> >>>manager >> >> >>>> >> >>> >>or >> >> >>>> >> >>> >> on a separate host." >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> My yarn-default.xml contains: >> >> >>>> >> >>> >> <property> >> >> >>>> >> >>> >> <description>The address for the web proxy as >> >>HOST:PORT, >> >> >>>>if >> >> >>>> >>this >> >> >>>> >> >>>is >> >> >>>> >> >>> >>not >> >> >>>> >> >>> >> given then the proxy will run as part of the >> >> >>>> RM</description> >> >> >>>> >> >>> >> <name>yarn.web-proxy.address</name> >> >> >>>> >> >>> >> <value/> >> >> >>>> >> >>> >> </property> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> and I did not set a value explicitly in yarn-site.xml. >> >>Any >> >> >>>> idea >> >> >>>> >> >>>how I >> >> >>>> >> >>> >>can >> >> >>>> >> >>> >> get this working? Thanks in advance. >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> Regards, >> >> >>>> >> >>> >> Prajakta >> >> >>>> >> >>> > >> >> >>>> >> >>> > >> >> >>>> >> >>> > >> >> >>>> >> >>> >-- >> >> >>>> >> >>> >Harsh J >> >> >>>> >> >>> >> >> >>>> >> >>> >> >> >>>> >> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >>>> >> >> >>> >> >> >> >> >> >> >> >> >>
