Awesome ! glad it worked, no idea why Knox sometimes caches old files, may be we should revisit it later. Happy I could help :) I will try to take a look at the patch later today !
Best, Sandeep On Mon, Nov 21, 2016 at 4:51 AM, McParland, John <[email protected]> wrote: > Hi Sandeep, > > I didn't have a SOLR in the topology file, however when I cleared out > ${GATEWAY_HOME}/data/deployments and re-tested it worked! I would appear > the gateway was using an old cache of the topology. > > I've put a patch on https://issues.apache.org/jira/browse/KNOX-528 now - > I'm much obliged for you help in getting the URL patterns right and for > general debugging help. > > Thanks, > > John McParland MIET CEng | System Architect, ODSC > Health Local and Scotland | CGI > CGI Ltd (UK) > Second Floor, Inovo Building, 121 George St, Glasgow, UK, G1 1RD > M: +44 7920 183 019 > [email protected] | www.cgi-group.co.uk > > CGI IT UK Limited. A CGI Group Inc. Company > Registered Office: 250 Brook Drive, Green Park, Reading RG2 6UA, > United Kingdom. Registered in England & Wales - Number 947968 > > CONFIDENTIALITY NOTICE: Proprietary/Confidential Information belonging to > CGI Group Inc. and its affiliates may be contained in this message. If you > are not a recipient indicated or intended in this message (or responsible > for delivery of this message to such person), or you think for any reason > that this message may have been addressed to you in error, you may not use > or copy or deliver this message to anyone else. In such case, you should > destroy this message and are asked to notify the sender by reply e-mail. > > ________________________________________ > From: Sandeep More [[email protected]] > Sent: 20 November 2016 00:28 > To: [email protected] > Subject: Re: Re-write rules not being applied to new service > > This is weird, > > Do you have SOLR entry in your topology file (alongside the new SOLRAPI) by > any chance ? > > A long shot but can you try stopping the gateway and deleting all the data > in {KNOX}/data/deployments/ > and then starting gateway back up. In the deployments directory you can see > WAR file for your topology it will contain the rewrite.xml file (under > %2F/WEB-XML) you should see the SOLAPI mapping there and should not see > (ideally) the old SOLR mapping. > > I tried on my local machine and the rewriting seems to be working fine. > This is the code snippet from my logs. > > 2016-11-19 19:07:57,944 DEBUG hadoop.gateway > (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL: > https://localhost:8443/gateway/sandbox/solr/gettingstarted/select?indent= > on&q=*&wt=json, > direction: IN via explicit rule: SOLRAPI/solr/inbound/query to URL: > http://localhost:8983/solr/gettingstarted/select?q=*&indent=on&wt=json > > And I have my topology file updated as follows. > <service> > <role>SOLRAPI</role> > <url>http://localhost:8983/solr</url> > </service> > > Best, > Sandeep > > On Sat, Nov 19, 2016 at 3:51 PM, McParland, John <[email protected]> > wrote: > > > Thanks Sandeep, > > > > that makes a good deal of sense. Obviously I need those {collection=**} > > to capture the collection itself etc. > > > > Final hurdle now seems to be on renaming the service to SOLRAPI (I've no > > intention to proxy the UI as this time). > > > > New service/rewrite here: https://github.com/ > mcparlandjcgi/knox/tree/KNOX- > > 528/gateway-service-definitions/src/main/resources/services/solr/5.5.0 > > > > And the updated topology includes the following: > > <service> > > <role>SOLRAPI</role> > > <url>http://sandbox.hortonworks.com:8983/solr</url> > > </service> > > > > But the logs, when I send a request through, show this; > > > > 2016-11-19 20:43:11,605 DEBUG hadoop.gateway (UrlRewriteProcessor.java: > rewrite(166)) > > - Rewrote URL: https://hdp24sandbox:8443/gateway/sandbox/solr/ > > KnoxIntegrationConfig/select?q=*.*&wt=json&indent=true, direction: IN > via > > explicit rule: SOLRAPI/solr/inbound/query to URL: > > SOLR/KnoxIntegrationConfig/select?indent=true&q=*.*&wt=json > > > > I don't understand why the old "SOLR" is being used at the start of the > > re-written URL. It's almost as if the gateway (which I restart using > > bin/gateway.sh stop / bin/gateway.sh start in-between changes) isn't > > picking up the new topology correctly. > > > > Thanks, > > > > John McParland MIET CEng | System Architect, ODSC > > Health Local and Scotland | CGI > > CGI Ltd (UK) > > Second Floor, Inovo Building, 121 George St, Glasgow, UK, G1 1RD > > M: +44 7920 183 019 > > [email protected] | www.cgi-group.co.uk > > > > CGI IT UK Limited. A CGI Group Inc. Company > > Registered Office: 250 Brook Drive, Green Park, Reading RG2 6UA, > > United Kingdom. Registered in England & Wales - Number 947968 > > > > CONFIDENTIALITY NOTICE: Proprietary/Confidential Information belonging to > > CGI Group Inc. and its affiliates may be contained in this message. If > you > > are not a recipient indicated or intended in this message (or responsible > > for delivery of this message to such person), or you think for any reason > > that this message may have been addressed to you in error, you may not > use > > or copy or deliver this message to anyone else. In such case, you should > > destroy this message and are asked to notify the sender by reply e-mail. > > > > ________________________________________ > > From: Sandeep More [[email protected]] > > Sent: 19 November 2016 17:56 > > To: [email protected] > > Subject: Re: Re-write rules not being applied to new service > > > > Hello John, > > > > You have some minor errors that are causing the paths to not match they > way > > you want them to. I did some local testing and the following service > > definitions appear to work locally on my machine ( using your github > > example as baseline ) > > > > *service.xml (complete file)* > > > > <service role="SOLR" name="solr" version="5.5.0"> > > <policies> > > <policy role="webappsec"/> > > <policy role="authentication" name="Anonymous"/> > > <policy role="rewrite"/> > > <policy role="authorization"/> > > </policies> > > <routes> > > > > <route path="/solr/**/**?**"> > > <rewrite apply="SOLR/solr/inbound/query" to="request.url"/> > > </route> > > > > </routes> > > </service> > > > > *rewrite.xml (complete file)* > > > > <rules> > > > > <rule dir="IN" name="SOLR/solr/inbound/query" > > pattern="*://*:*/**/solr/{collection=**}/{query=**}?{**}"> > > <rewrite > > template="{$serviceUrl[SOLR]}/{collection=**}/{query=**}?{**}"/> > > </rule> > > > > </rules> > > > > your topology file seems right. I was able to get the basic > > "gettingstarted" solr query to work. > > You probably need to add some rules for the UI if you like to support it. > > Also, you might want to call the above service something like "SOLRAPI" > > rather then "SOLR" just to differentiate if from UI traffic. > > > > There are minor regex changes I had to do to make it work. Let me know if > > it works ! > > > > To understand UI Proxying with Knox refer to this excellent guide by > Sumit > > Gupta - > > https://cwiki.apache.org/confluence/display/KNOX/ > Proxying+a+UI+using+Knox > > > > Another good resource on adding a service to Knox by Kevin Minder is - - > > https://cwiki.apache.org/confluence/display/KNOX/2015/ > > 12/17/Adding+a+service+to+Apache+Knox > > > > These blogs explain in-detail process to add services and are a step by > > step guides to adding support for knox proxying. > > > > You also asked about debugging, generally for adding UIs I just tail the > > logs and try to figure out from logs and change the logging level if > > needed. If you would like to debug using an IDE you can use the command > > "ant > > start-debug-gateway" from the knox source directory. > > > > Sorry for the lengthy email, hope it addresses your questions ! > > > > Best, > > Sandeep > > > > On Fri, Nov 18, 2016 at 4:29 PM, McParland, John <[email protected] > > > > wrote: > > > > > Thanks again Sandeep, > > > > > > I tried your suggestion, and while I can understand it and believe it > > > should have worked, it still doesn't! I have the same error message as > > > below. > > > > > > From my own understanding, I think the rules in rewrite.xml are fine, > and > > > the JUnit test proves those should work. > > > > > > However I haven't understood how the route in service.xml is > > > picked/matched with the incoming URL. And this appears (again from > what > > I > > > understand) to be where the problem is. My understanding of the > > processing > > > of an incoming URL is; > > > > > > 1) Check URL against routes (not clear on this) > > > 2) Find re-write rule associated with it (presume this is just a > lookup > > > of a map of URL -> rule name) > > > 3) Check the URL matches the pattern in the re-write rule > > > 4) Apply the transformation in the template of the re-write rule. > > > > > > Is there anything else I can do to debug this part (e.g. see what > routes > > > we have at runtime)? > > > > > > Thanks, > > > > > > John McParland MIET CEng | System Architect, ODSC > > > Health Local and Scotland | CGI > > > CGI Ltd (UK) > > > Second Floor, Inovo Building, 121 George St, Glasgow, UK, G1 1RD > > > M: +44 7920 183 019 > > > [email protected] | www.cgi-group.co.uk > > > > > > CGI IT UK Limited. A CGI Group Inc. Company > > > Registered Office: 250 Brook Drive, Green Park, Reading RG2 6UA, > > > United Kingdom. Registered in England & Wales - Number 947968 > > > > > > CONFIDENTIALITY NOTICE: Proprietary/Confidential Information belonging > to > > > CGI Group Inc. and its affiliates may be contained in this message. If > > you > > > are not a recipient indicated or intended in this message (or > responsible > > > for delivery of this message to such person), or you think for any > reason > > > that this message may have been addressed to you in error, you may not > > use > > > or copy or deliver this message to anyone else. In such case, you > should > > > destroy this message and are asked to notify the sender by reply > e-mail. > > > > > > ________________________________________ > > > From: Sandeep More [[email protected]] > > > Sent: 18 November 2016 16:37 > > > To: [email protected] > > > Subject: Re: Re-write rules not being applied to new service > > > > > > Ah ! got it now and the pattern seems perfectly fine. > > > In which case I think just the following should work (without any > > > additional rules): > > > > > > service.xml > > > > > > <routes> > > > <route path="/solr/**/**?{**}" > > > > <rewrite apply="SOLR/solr/inbound/query" > > > to="request.url"/> > > > </route> > > > </routes> > > > > > > And in rewrite.xml > > > > > > <!--Only supporting Solr queries via Knox --> > > > <rule dir="IN" name="SOLR/solr/inbound/query" > > > pattern="*://*:*/**/solr/{collection}/{query}?{**}"> > > > <rewrite template="{$serviceUrl[SOLR]}/ > > > {collection}/{query}?{**}"/> > > > </rule> > > > > > > Let me know if you run into issues, I can take a look at the code. > > > > > > Best, > > > Sandeep > > > > > > > > > On Fri, Nov 18, 2016 at 11:05 AM, McParland, John < > > [email protected]> > > > wrote: > > > > > > > Hi Sandeep, > > > > > > > > thanks for your help. A part of the difficulty I'm having is that > the > > > > "KnoxSolrIntegration" part of the URL is variable - the user sending > a > > > > request to Knox needs to specify it (it happens to the be the > > collection > > > in > > > > Solr which they wish to query). > > > > > > > > E.g. in a normal Solr Query; > > > > > > > > curl -X GET 'http://hdp24sandbox:8983/solr/KnoxIntegrationConfig/ > > > > select?q=*.*&wt=json&indent=true > > > > > > > > All of the pieces after the "/solr/" are variable - the user needs to > > > > specify it regardless of whether going through Knox or directly to > > Solr. > > > > > > > > So my belief is a Knox equivalent of the above curl command is > > > > > > > > curl -k -u guest:guest-password -X GET 'https://hdp24sandbox:8443/ > > > > gateway/sandbox/solr/KnoxIntegrationConfig/select? > > > > q=*.*&wt=json&indent=true' > > > > > > > > Therefore, I thought the pattern to match (in rewrite.xml) would be > > > > > > > > *://*.*//**/solr/**/**?{**} > > > > > > > > As in my head > > > > > > > > - first * = protocol (http/https) > > > > - second * = domain name / IP address of Knox server > > > > - third * = port of Knox on it's server > > > > > > > > - first ** = knox context and topology (gateway/sandbox in my > example) > > > > - second ** = Solr collection to search (KnoxIntegrationConfig in my > > > > example, but something I expect the caller to provide) > > > > - third ** = query type for Solr (e.g. "select") > > > > - {**} = the search parameters for Solr (q=*.*&wt=json&indent=true in > > my > > > > example) > > > > > > > > I suspect either (or both) > > > > a) I've not captured my pattern correctly (but I think so....) > > > > b) I haven't linked the service to the re-write rule to the routes in > > the > > > > service definition correctly [as I'm not sure if/how this works from > > the > > > > tutorial] > > > > > > > > Thanks, > > > > > > > > John McParland MIET CEng | System Architect, ODSC > > > > Health Local and Scotland | CGI > > > > CGI Ltd (UK) > > > > Second Floor, Inovo Building, 121 George St, Glasgow, UK, G1 1RD > > > > M: +44 7920 183 019 > > > > [email protected] | www.cgi-group.co.uk > > > > > > > > CGI IT UK Limited. A CGI Group Inc. Company > > > > Registered Office: 250 Brook Drive, Green Park, Reading RG2 6UA, > > > > United Kingdom. Registered in England & Wales - Number 947968 > > > > > > > > CONFIDENTIALITY NOTICE: Proprietary/Confidential Information > belonging > > to > > > > CGI Group Inc. and its affiliates may be contained in this message. > If > > > you > > > > are not a recipient indicated or intended in this message (or > > responsible > > > > for delivery of this message to such person), or you think for any > > reason > > > > that this message may have been addressed to you in error, you may > not > > > use > > > > or copy or deliver this message to anyone else. In such case, you > > should > > > > destroy this message and are asked to notify the sender by reply > > e-mail. > > > > > > > > ________________________________________ > > > > From: Sandeep More [[email protected]] > > > > Sent: 18 November 2016 14:11 > > > > To: [email protected] > > > > Subject: Re: Re-write rules not being applied to new service > > > > > > > > Hello John, > > > > > > > > I think you are missing the definitions for the path > > > > '/solr/KnoxSolrIntegration' > > > > > > > > Looking at your definition files you could perhaps have something > like > > > > this: > > > > > > > > service.xml > > > > > > > > <routes> > > > > <route path="/solr/**/**?{**}" > > > > > <rewrite apply="SOLR/solr/inbound/query" > > > > to="request.url"/> > > > > </route> > > > > > > > > <route path="/solr/KnoxSolrIntegration**" > > > > > <rewrite apply="SOLR/solr/inbound/ > KnoxSolrIntegration" > > > > to="request.url"/> > > > > </route> > > > > </routes> > > > > > > > > And in rewrite.xml > > > > > > > > <!--Only supporting Solr queries via Knox --> > > > > <rule dir="IN" name="SOLR/solr/inbound/query" > > > > pattern="*://*:*/**/solr/{collection}/{query}?{**}"> > > > > <rewrite template="{$serviceUrl[SOLR]}/ > > > > {collection}/{query}?{**}"/> > > > > </rule> > > > > <rule dir="IN" name="SOLR/solr/inbound/KnoxSolrIntegration" > > > > pattern="*://*:*/**/solr/KnoxSolrIntegration**"> > > > > <rewrite template="{$serviceUrl[SOLR]}/ > > KnoxSolrIntegration{**}"/> > > > > </rule> > > > > > > > > Looks like you are missing the <route> for 'SOLR/solr/inbound/query' > in > > > > your service.xml file (like in the above example). > > > > > > > > Couple of things that helped me were > > > > 1) Try to keep the rules as tight as possible e.g. > > > > /solr/KnoxSolrIntegration** > > > > instead of /solr/** (provided an example with rule #2 in the above > > > example) > > > > 2) In your Service.xml file you have multiple <routes>, you could > > > > consolidate them into single <routes> > > > > > > > > I haven't tested locally it so apologies in advance if it does not > > work. > > > > > > > > > > > > Hope that helps ! > > > > > > > > Best, > > > > Sandeep > > > > > > > > On Fri, Nov 18, 2016 at 7:13 AM, McParland, John < > > [email protected] > > > > > > > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > I'm working to add support for Solr's API to Apache Knox. To do > so, > > > I've > > > > > created a service.xml and rewrite.xml, and deployed to > > > > > ${GATEWAY_HOME}/data/conf/services/solr/5.5.0 > > > > > > > > > > See the files here: https://github.com/mcparlandjc > > > > > gi/knox/tree/KNOX-528/gateway-service-definitions/src/main/r > > > > > esources/services/solr/5.5.0 > > > > > I also wrote a test to check that the re-write rules work as > expected > > > > (see > > > > > https://github.com/mcparlandjcgi/knox/blob/KNOX-528/gateway- > > > > > provider-rewrite/src/test/java/org/apache/hadoop/ > > > > > gateway/filter/rewrite/api/UrlRewriteProcessorTest.java#L281) > > > > > > > > > > I have defined the Solr service in my sandbox topology like so; > > > > > > > > > > <service> > > > > > <role>SOLR</role> > > > > > <url>http://sandbox.hortonworks.com:8983/solr</url> > > > > > </service> > > > > > > > > > > However when I attempt to execute a Solr Query via Knox, I find > that > > > the > > > > > rule isn't applied and indeed, no matching rule is found at all. > > > > > > > > > > 2016-11-18 12:04:23,221 TRACE gateway.access > > > (AccessHandler.java:log(49)) > > > > > - |||158.234.220.6|GET|/gateway/sandbox/solr/KnoxSolrIntegrati > > > > > on/select?q=*.*&wt=json&indent=true|-1|200|0|6 > > > > > 2016-11-18 12:04:23,231 TRACE http.request (TraceRequest.java: > > > > traceRequestDetails(66)) > > > > > - ||82e1c7f7-4eb5-499d-8f29-7602652079f1|Request=GET > > > > > /gateway/sandbox/solr/KnoxSolrIntegration/select?q=* > > > > .*&wt=json&indent=true > > > > > Header[User-Agent]=curl/7.47.0 > > > > > Header[Authorization]=Basic Z3Vlc3Q6Z3Vlc3QtcGFzc3dvcmQ= > > > > > Header[Accept]=*/* > > > > > Header[Host]=hdp24sandbox:8443 > > > > > 2016-11-18 12:04:23,786 DEBUG hadoop.gateway > > > > (GatewayFilter.java:doFilter(116)) > > > > > - Received request: GET /solr/KnoxSolrIntegration/select > > > > > 2016-11-18 12:04:23,888 TRACE hadoop.gateway > > (UrlRewriteProcessor.java: > > > > rewrite(177)) > > > > > - No rule matching URL: hdp24sandbox:8443, direction: IN > > > > > 2016-11-18 12:04:23,889 TRACE hadoop.gateway > > (UrlRewriteProcessor.java: > > > > rewrite(177)) > > > > > - No rule matching URL: 8443, direction: IN > > > > > 2016-11-18 12:04:23,889 TRACE hadoop.gateway > > (UrlRewriteProcessor.java: > > > > rewrite(177)) > > > > > - No rule matching URL: https, direction: IN > > > > > 2016-11-18 12:04:25,052 TRACE hadoop.gateway > > (UrlRewriteProcessor.java: > > > > rewrite(177)) > > > > > - No rule matching URL: https://hdp24sandbox:8443/ > > > gateway/sandbox/solr/ > > > > KnoxSolrIntegrati > > > > > on/select?q=*.*&wt=json&indent=true, direction: IN > > > > > 2016-11-18 12:04:25,062 TRACE hadoop.gateway > > (UrlRewriteProcessor.java: > > > > rewrite(177)) > > > > > - No rule matching URL: 158.234.220.6, direction: IN > > > > > 2016-11-18 12:04:25,062 TRACE hadoop.gateway > > (UrlRewriteProcessor.java: > > > > rewrite(177)) > > > > > - No rule matching URL: https, direction: IN > > > > > 2016-11-18 12:04:25,063 TRACE hadoop.gateway > > (UrlRewriteProcessor.java: > > > > rewrite(177)) > > > > > - No rule matching URL: 8443, direction: IN > > > > > 2016-11-18 12:04:25,063 TRACE hadoop.gateway > > (UrlRewriteProcessor.java: > > > > rewrite(177)) > > > > > - No rule matching URL: hdp24sandbox:8443, direction: IN > > > > > 2016-11-18 12:04:25,063 TRACE hadoop.gateway > > (UrlRewriteProcessor.java: > > > > rewrite(177)) > > > > > - No rule matching URL: hdp24sandbox, direction: IN > > > > > 2016-11-18 12:04:25,063 TRACE hadoop.gateway > > (UrlRewriteProcessor.java: > > > > rewrite(177)) > > > > > - No rule matching URL: /gateway/sandbox, direction: IN > > > > > 2016-11-18 12:04:25,064 TRACE hadoop.gateway > > (UrlRewriteProcessor.java: > > > > rewrite(177)) > > > > > - No rule matching URL: curl/7.47.0, direction: IN > > > > > 2016-11-18 12:04:25,069 TRACE hadoop.gateway > > (UrlRewriteProcessor.java: > > > > rewrite(177)) > > > > > - No rule matching URL: Basic Z3Vlc3Q6Z3Vlc3QtcGFzc3dvcmQ=, > > direction: > > > IN > > > > > 2016-11-18 12:04:25,070 TRACE hadoop.gateway > > (UrlRewriteProcessor.java: > > > > rewrite(177)) > > > > > - No rule matching URL: */*, direction: IN > > > > > 2016-11-18 12:04:25,070 TRACE hadoop.gateway > > (UrlRewriteProcessor.java: > > > > rewrite(177)) > > > > > - No rule matching URL: hdp24sandbox:8443, direction: IN > > > > > 2016-11-18 12:04:25,072 DEBUG hadoop.gateway (DefaultDispatch.java: > > > > executeOutboundRequest(120)) > > > > > - Dispatch request: GET https://hdp24sandbox:8443/gate > > > > > way/sandbox/solr/KnoxSolrIntegration/select?q=* > > .*&wt=json&indent=true > > > > > > > > > > What I expect at the end is the Dispatch request to go to > > > > > > > > > > http://sandbox.hortonworks.com:8983/solr/KnoxSolrIntegration > > > > > /select?q=*.*&wt=json&indent=true > > > > > > > > > > Is there a missing relationship or problem in my service/rules > which > > > > > prevents the rule being matched with the request URL? > > > > > > > > > > Thanks, > > > > > > > > > > John McParland MIET CEng | System Architect, ODSC > > > > > Health Local and Scotland | CGI > > > > > CGI Ltd (UK) > > > > > Second Floor, Inovo Building, 121 George St, Glasgow, UK, G1 1RD > > > > > M: +44 7920 183 019 > > > > > [email protected] | www.cgi-group.co.uk > > > > > > > > > > CGI IT UK Limited. A CGI Group Inc. Company > > > > > Registered Office: 250 Brook Drive, Green Park, Reading RG2 6UA, > > > > > United Kingdom. Registered in England & Wales - Number 947968 > > > > > > > > > > CONFIDENTIALITY NOTICE: Proprietary/Confidential Information > > belonging > > > to > > > > > CGI Group Inc. and its affiliates may be contained in this message. > > If > > > > you > > > > > are not a recipient indicated or intended in this message (or > > > responsible > > > > > for delivery of this message to such person), or you think for any > > > reason > > > > > that this message may have been addressed to you in error, you may > > not > > > > use > > > > > or copy or deliver this message to anyone else. In such case, you > > > should > > > > > destroy this message and are asked to notify the sender by reply > > > e-mail. > > > > > > > > > > > > > > >
