:) Yes, you are right. The yarn.acl.enable property in yarn-default.xml is set true. If the property is true by default, then this makes it mandatory for users to either specify a value for hadoop.http.staticuser.user property explicitly or to change the acl's to false. Am I right to assume this?
Regards, Prajakta On Thu, Jul 26, 2012 at 11:59 PM, Robert Evans <ev...@yahoo-inc.com> wrote: > OK I think I understand it now. You probably have ACLs enabled, but no > web filter on the RM to let you sign in as a given user. As such the > default filter is making you be Dr. Who, or whomever else it is, but the > ACL check in the web service is rejecting Dr Who, because that is not the > correct user. You will probably run into this issue again if anyone else > but you runs something. You could fix this by either disabling the ACL > check, which makes a lot of since for a cluster without security, or you > could implement a servlet Filter for the RM that would let you sign on as > a given user. > > --Bobby Evans > > > On 7/26/12 12:48 AM, "Prajakta Kalmegh" <pkalm...@gmail.com> wrote: > > >Hi Bobby > > > >Thanks for the reply. My REST calls are working fine since I set the > >'hadoop.http.staticuser.user' property to 'prajakta' instead of Dr.Who in > >core-site.xml . I didn't get time to figure out the reason behind it as I > >just moved on to further coding :) > > > >Thanks, > >Prajakta > > > > > > > >On Thu, Jul 26, 2012 at 1:40 AM, Robert Evans <ev...@yahoo-inc.com> > wrote: > > > >> Hmm, that is very odd. It only checks the user if security is enabled > >>to > >> warn the user about potentially accessing something unsafe. I am not > >>sure > >> why that would cause an issue. > >> > >> --Bobby Evans > >> > >> On 7/9/12 6:07 AM, "Prajakta Kalmegh" <pkalm...@gmail.com> wrote: > >> > >> >Hi Robert > >> > > >> >I figured out the problem just now. To avoid the below error, I had to > >>set > >> >the 'hadoop.http.staticuser.user' property in core-site.xml (defaults > >>to > >> >dr.who). I can now get runtime data from AppMaster using *curl* as > >>well as > >> >in GUI. > >> > > >> >I wonder if we have to set this property even when we are not > >>specifying > >> >the yarn web-proxy address (when it runs as part of RM by default) as > >> >well. > >> >If yes, was it documented somewhere which I failed to see? :( > >> > > >> >Anyways, thanks for your response so far. > >> > > >> >Regards, > >> >Prajakta > >> > > >> > > >> > > >> >On Mon, Jul 9, 2012 at 3:29 PM, Prajakta Kalmegh <pkalm...@gmail.com> > >> >wrote: > >> > > >> >> Hi Robert > >> >> > >> >> I started the proxyserver explicitly by specifying a value for the > >> >> yarn.web-proxy.address in yarn-site.xml. The proxyserver did start > >>and I > >> >> tried getting the JSON response using the following command : > >> >> > >> >> curl --compressed -H "Accept: application/json" -X GET " > >> >> > >> >> > >> > >> > http://localhost:8090/proxy/application_1341823967331_0001/ws/v1/mapreduc > >> >>e/jobs/job_1341823967331_0001 > >> >> " > >> >> > >> >> However, it refused connection and below is the excerpt from the > >> >> Proxyserver logs: > >> >> --------- > >> >> 2012-07-09 14:26:40,402 INFO org.mortbay.log: Extract > >> >> > >> > >>>>jar:file:/home/prajakta/Projects/IRL/hadoop-common/hadoop-dist/target/h > >>>>ad > >> > >>>>oop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-yarn-common-3.0.0-SNAP > >>>>SH > >> >>OT.jar!/webapps/proxy > >> >> to /tmp/Jetty_localhost_8090_proxy____.ak3o30/webapp > >> >> 2012-07-09 14:26:40,992 INFO org.mortbay.log: Started > >> >> SelectChannelConnector@localhost:8090 > >> >> 2012-07-09 14:26:40,993 INFO > >> >> org.apache.hadoop.yarn.service.AbstractService: > >> >> Service:org.apache.hadoop.yarn.server.webproxy.WebAppProxy is > >>started. > >> >> 2012-07-09 14:26:40,993 INFO > >> >> org.apache.hadoop.yarn.service.AbstractService: > >> >> Service:org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer is > >> >>started. > >> >> 2012-07-09 14:33:26,039 INFO > >> >> org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet: dr.who is > >> >> accessing unchecked > >> >> http://prajakta:44314/ws/v1/mapreduce/jobs/job_1341823967331_0001 > >>which > >> >> is the app master GUI of application_1341823967331_0001 owned by > >> >>prajakta > >> >> 2012-07-09 14:33:29,277 INFO > >> >> org.apache.commons.httpclient.HttpMethodDirector: I/O exception > >> >> (org.apache.commons.httpclient.NoHttpResponseException) caught when > >> >> processing request: The server prajakta failed to respond > >> >> 2012-07-09 14:33:29,277 INFO > >> >> org.apache.commons.httpclient.HttpMethodDirector: Retrying request > >> >> 2012-07-09 14:33:29,284 WARN org.mortbay.log: > >> >> > >> > >>>>/proxy/application_1341823967331_0001/ws/v1/mapreduce/jobs/job_13418239 > >>>>67 > >> >>331_0001: > >> >> java.net.SocketException: Connection reset > >> >> 2012-07-09 14:37:33,834 INFO > >> >> org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet: dr.who is > >> >> accessing unchecked > >> >> > >> >> > >> > >> > http://prajakta:19888/jobhistory/job/job_1341823967331_0001/jobhistory/jo > >> >>b/job_1341823967331_0001which is the app master GUI of > >> >>application_1341823967331_0001 owned by > >> >> prajakta > >> >> --------------- > >> >> > >> >> I am not sure why http request object is setting my remoteUser to > >> >>dr.who. > >> >> :( > >> >> > >> >> I gather from <https://issues.apache.org/jira/browse/MAPREDUCE-2858> > >> >>that > >> >> this warning is posted only in case where security is disabled. I > >>assume > >> >> that the proxy server is not disabled if security is disabled. > >> >> > >> >> Any idea what could be the reason for this I/O exception? Am I > >>missing > >> >> setting any property for proper access. Please let me know. > >> >> > >> >> Regards, > >> >> Prajakta > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> On Fri, Jul 6, 2012 at 10:59 PM, Prajakta Kalmegh > >> >><pkalm...@gmail.com>wrote: > >> >> > >> >>> I am using hadoop trunk (forked from github). It supports RESTful > >>APIs > >> >>>as > >> >>> I am able to retrieve JSON objects for RM (cluster/nodes info)+ > >> >>> Historyserver. The only issue is with AppMaster REST API. > >> >>> > >> >>> Regards, > >> >>> Prajakta > >> >>> > >> >>> > >> >>> > >> >>> On Fri, Jul 6, 2012 at 10:55 PM, Robert Evans > >> >>><ev...@yahoo-inc.com>wrote: > >> >>> > >> >>>> What version of hadoop are you using? It could be that the version > >> >>>>you > >> >>>> have does not have the RESTful APIs in it yet, and the proxy is > >> >>>>working > >> >>>> just fine. > >> >>>> > >> >>>> --Bobby Evans > >> >>>> > >> >>>> On 7/6/12 12:06 PM, "Prajakta Kalmegh" <pkalm...@gmail.com> wrote: > >> >>>> > >> >>>> >Robert , Thanks for the response. If I do not provide any explicit > >> >>>> >configuration for the proxy server, do I still need to start it > >>using > >> >>>> the > >> >>>> >'yarn start proxy server'? I am currently not doing it. > >> >>>> > > >> >>>> >Also, I am able to access the html page for proxy using the > >> >>>> ><http://localhost:8088/proxy/{appid}/mapreduce/jobs> URL. (Note > >>this > >> >>>> url > >> >>>> >does not have the '/ws/v1/ part in it. I get the html response > >>when I > >> >>>> >query > >> >>>> >for this URL in runtime. > >> >>>> > > >> >>>> >So I assume the proxy server must be starting fine since I am > >>able to > >> >>>> >access this URL. I will try logging more details tomorrow from my > >> >>>>office > >> >>>> >machine and will let you know the result. > >> >>>> > > >> >>>> >Regards, > >> >>>> >Prajakta > >> >>>> > > >> >>>> > > >> >>>> > > >> >>>> >On Fri, Jul 6, 2012 at 10:22 PM, Robert Evans > >><ev...@yahoo-inc.com> > >> >>>> wrote: > >> >>>> > > >> >>>> >> Sorry I did not respond sooner. The default behavior is to have > >> >>>>the > >> >>>> >>proxy > >> >>>> >> server run as part of the RM. I am not really sure why it is > >>not > >> >>>> doing > >> >>>> >> this in your case. If you set the config yourself to be a URI > >> >>>>that is > >> >>>> >> different from that of the RM then you need to launch a > >>standalone > >> >>>> proxy > >> >>>> >> server. You can do this by running > >> >>>> >> > >> >>>> >> yarn start proxy server > >> >>>> >> > >> >>>> >> Without sitting down with you it is going to be somewhat > >>difficult > >> >>>>to > >> >>>> >> debug why this is happening. However, in retrospect it would be > >> >>>>nice > >> >>>> to > >> >>>> >> add in some extra logging to help indicate why the proxy server > >>is > >> >>>>not > >> >>>> >> functioning as desired. If you could file a JIRA to add in the > >> >>>> logging > >> >>>> >>I > >> >>>> >> would be happy to provide a patch to you and we can try and > >>debug > >> >>>>the > >> >>>> >> issue further. Please file it under the MAPREDUCE JIRA project. > >> >>>> >> > >> >>>> >> --Bobby > >> >>>> >> > >> >>>> >> On 7/6/12 3:29 AM, "Prajakta Kalmegh" <pkalm...@gmail.com> > >>wrote: > >> >>>> >> > >> >>>> >> >Re-posting as I haven't got a solution yet. Sorry for > >>spamming. I > >> >>>> >>won't be > >> >>>> >> >able to proceed in my code until I get a JSON response using > >> >>>> AppMaster > >> >>>> >> >REST > >> >>>> >> >URL. :( > >> >>>> >> > > >> >>>> >> >Thanks, > >> >>>> >> >Prajakta > >> >>>> >> > > >> >>>> >> > > >> >>>> >> >On Wed, Jul 4, 2012 at 5:55 PM, Prajakta Kalmegh > >> >>>><pkalm...@gmail.com > >> >>>> > > >> >>>> >> >wrote: > >> >>>> >> > > >> >>>> >> >> Hi Robert/Harsh > >> >>>> >> >> > >> >>>> >> >> Thanks for your reply. > >> >>>> >> >> > >> >>>> >> >> My RM is starting just fine. The problem is with the use of > >> >>>> >> >>http://<proxy httpddress:port>/proxy/{appid}/ws/v1/mapreduce > >> >>>> >> >> to get the JSON response. > >> >>>> >> >> > >> >>>> >> >> As I said before, I had not configured the > >> >>>>yarn.web-proxy.address > >> >>>> >> >>property in yarn-site.xml. I assumed it will use the RM's > >> >>>> >> >>yarn.resourcemanager.webapp.address property value as default. > >> >>>> >>However, > >> >>>> >> >>it gives me a '404-Page not found error'. Today I tried > >> >>>>specifying > >> >>>> a > >> >>>> >> >>value explicitly for the yarn.web-proxy.address property. > >> >>>> >> >> > >> >>>> >> >> On running the wordcount example, it even gives a url > >> >>>> >> >><http://localhost:8090>/proxy/{appid}/> to track the App Mast > >> >>>>info. > >> >>>> >> >>However, I am still not able to get a json response. > >> >>>> >> >> > >> >>>> >> >> Also, I tried to get the data from historyserver instead of > >> >>>>runtime > >> >>>> >> >>using the instructions given on page > >> >>>> >> >>< > >> >>>> >> > >> >>>> > >> >>>> > >> http://hadoop.apache.org/common/docs/r2.0.0-alpha/hadoop-yarn/hadoop-ya > >> >>>>r > >> >>>> >> >>n-site/HistoryServerRest.html> > >> >>>> >> >> > >> >>>> >> >> HistoryServer REST response does not give me jobids > >> >>>>corresponding > >> >>>> to > >> >>>> >>an > >> >>>> >> >>application. It just lists all the jobs run until now. By the > >> >>>>way, > >> >>>> the > >> >>>> >> >>documentation does say > >> >>>> >> >> > >> >>>> >> >> ---------- > >> >>>> >> >> > >> >>>> >> >> "Both of the following URI's give you the history server > >> >>>> information, > >> >>>> >> >>from an application id identified by the appid value. > >> >>>> >> >> * http://<history server http address:port>/ws/v1/history > >> >>>> >> >> * http://<history server http > >> >>>>address:port>/ws/v1/history/info" > >> >>>> >> >> --------- > >> >>>> >> >> > >> >>>> >> >> But there is no provision to specify the application id with > >> >>>>these > >> >>>> >>REST > >> >>>> >> >>URLs. > >> >>>> >> >> > >> >>>> >> >> Any idea how I can get the Application Master REST working > >>and > >> >>>>also > >> >>>> >> >>linking jobids to application id using the HistoryServerREST > >>API? > >> >>>> >> >> > >> >>>> >> >> Any help is appreciated. Thanks in advance. > >> >>>> >> >> Regards, > >> >>>> >> >> Prajakta > >> >>>> >> >> > >> >>>> >> >> > >> >>>> >> >> > >> >>>> >> >> > >> >>>> >> >> On Fri, Jun 29, 2012 at 8:55 PM, Robert Evans > >> >>>><ev...@yahoo-inc.com > >> >>>> > > >> >>>> >> >>wrote: > >> >>>> >> >> > >> >>>> >> >>> Please don't file that JIRA. The proxy server is intended > >>to > >> >>>> front > >> >>>> >>the > >> >>>> >> >>> web server for all calls to the AM. This is so you only > >>have > >> >>>>to > >> >>>> go > >> >>>> >>to > >> >>>> >> >>>a > >> >>>> >> >>> single location to get to any AM's web service. The proxy > >> >>>>server > >> >>>> >>is a > >> >>>> >> >>> very simple proxy and just forwards the extra part of the > >>path > >> >>>>on > >> >>>> to > >> >>>> >> >>>the > >> >>>> >> >>> AM. > >> >>>> >> >>> > >> >>>> >> >>> If you are having issues with this please include the > >>version > >> >>>>you > >> >>>> >>are > >> >>>> >> >>> having problems with. Also please look at the logs for the > >>RM > >> >>>>on > >> >>>> >> >>>startup > >> >>>> >> >>> to see if there is anything there indicating why it is not > >> >>>> starting > >> >>>> >>up. > >> >>>> >> >>> > >> >>>> >> >>> --Bobby Evans > >> >>>> >> >>> > >> >>>> >> >>> On 6/28/12 9:46 AM, "Harsh J" <ha...@cloudera.com> wrote: > >> >>>> >> >>> > >> >>>> >> >>> >As far as I can tell, the MR WebApp, as the name itself > >> >>>>indicates > >> >>>> >>on > >> >>>> >> >>> >its doc page, starts only at the MR AM (which may be > >>running > >> >>>>at > >> >>>> any > >> >>>> >> >>> >NM), and it starts as an ephemeral port logged at in the AM > >> >>>>logs > >> >>>> >> >>> >usually as: > >> >>>> >> >>> > > >> >>>> >> >>> >INFO Web app /mapreduce started at [PORT] > >> >>>> >> >>> > > >> >>>> >> >>> >That it starts its own server with an ephemeral access > >>point > >> >>>> makes > >> >>>> >> >>> >sense, since each job uses its own AM and having a common > >> >>>> location > >> >>>> >>may > >> >>>> >> >>> >not work with the form of REST API documented at your link. > >> >>>>Can > >> >>>> you > >> >>>> >> >>> >please file a JIRA to fix the doc and remove the proxy > >>server > >> >>>> refs, > >> >>>> >> >>> >which are misleading? > >> >>>> >> >>> > > >> >>>> >> >>> >Do correct me if I'm wrong. > >> >>>> >> >>> > > >> >>>> >> >>> >On Thu, Jun 28, 2012 at 6:13 PM, Prajakta Kalmegh > >> >>>> >><pkalm...@gmail.com > >> >>>> >> > > >> >>>> >> >>> >wrote: > >> >>>> >> >>> >> Hi > >> >>>> >> >>> >> > >> >>>> >> >>> >> I am trying to get the ApplicationMaster info using the > >> >>>> >> >>><http://<proxy > >> >>>> >> >>> >>http > >> >>>> >> >>> >> address:port>/proxy/{appid}/ws/v1/mapreduce/info> link as > >> >>>> >>described > >> >>>> >> >>>on > >> >>>> >> >>> >>the < > >> >>>> >> >>> >> > >> >>>> >> >>> >> > >> >>>> >> >>> > >> >>>> >> >>> > >> >>>> >> > >> >>>> > >> >>>> > >> http://hadoop.apache.org/common/docs/r2.0.0-alpha/hadoop-yarn/hadoop-ya > >> >>>>r > >> >>>> >> >>>n > >> >>>> >> >>> >>-site/MapredAppMasterRest.html> > >> >>>> >> >>> >> page. > >> >>>> >> >>> >> > >> >>>> >> >>> >> I am able to access and retrieve JSON response for other > >> >>>> modules > >> >>>> >> >>> >> (ResourceManager, NodeManager and HistoryServer). > >>However, > >> >>>>I am > >> >>>> >> >>>getting > >> >>>> >> >>> >> 'Page not found' when I try to use my ResourceManager > >>Http > >> >>>> >>address > >> >>>> >> >>>to > >> >>>> >> >>> >> access the ApplicationMaster info. I am using < > >> >>>> >> >>> >> http://localhost:8088/proxy/{appid}/ws/v1/mapreduce/info > > > >> to > >> >>>> >> >>>retrieve > >> >>>> >> >>> >>JSON > >> >>>> >> >>> >> response. > >> >>>> >> >>> >> > >> >>>> >> >>> >> The instructions say "The application master should be > >> >>>>accessed > >> >>>> >>via > >> >>>> >> >>>the > >> >>>> >> >>> >> proxy. This proxy is configurable to run either on the > >> >>>>resource > >> >>>> >> >>>manager > >> >>>> >> >>> >>or > >> >>>> >> >>> >> on a separate host." > >> >>>> >> >>> >> > >> >>>> >> >>> >> My yarn-default.xml contains: > >> >>>> >> >>> >> <property> > >> >>>> >> >>> >> <description>The address for the web proxy as > >>HOST:PORT, > >> >>>>if > >> >>>> >>this > >> >>>> >> >>>is > >> >>>> >> >>> >>not > >> >>>> >> >>> >> given then the proxy will run as part of the > >> >>>> RM</description> > >> >>>> >> >>> >> <name>yarn.web-proxy.address</name> > >> >>>> >> >>> >> <value/> > >> >>>> >> >>> >> </property> > >> >>>> >> >>> >> > >> >>>> >> >>> >> and I did not set a value explicitly in yarn-site.xml. > >>Any > >> >>>> idea > >> >>>> >> >>>how I > >> >>>> >> >>> >>can > >> >>>> >> >>> >> get this working? Thanks in advance. > >> >>>> >> >>> >> > >> >>>> >> >>> >> Regards, > >> >>>> >> >>> >> Prajakta > >> >>>> >> >>> > > >> >>>> >> >>> > > >> >>>> >> >>> > > >> >>>> >> >>> >-- > >> >>>> >> >>> >Harsh J > >> >>>> >> >>> > >> >>>> >> >>> > >> >>>> >> >> > >> >>>> >> > >> >>>> >> > >> >>>> > >> >>>> > >> >>> > >> >> > >> > >> > >
