Thanks again Sandeep, I tried your suggestion, and while I can understand it and believe it should have worked, it still doesn't! I have the same error message as below.
>From my own understanding, I think the rules in rewrite.xml are fine, and the >JUnit test proves those should work. However I haven't understood how the route in service.xml is picked/matched with the incoming URL. And this appears (again from what I understand) to be where the problem is. My understanding of the processing of an incoming URL is; 1) Check URL against routes (not clear on this) 2) Find re-write rule associated with it (presume this is just a lookup of a map of URL -> rule name) 3) Check the URL matches the pattern in the re-write rule 4) Apply the transformation in the template of the re-write rule. Is there anything else I can do to debug this part (e.g. see what routes we have at runtime)? Thanks, John McParland MIET CEng | System Architect, ODSC Health Local and Scotland | CGI CGI Ltd (UK) Second Floor, Inovo Building, 121 George St, Glasgow, UK, G1 1RD M: +44 7920 183 019 [email protected] | www.cgi-group.co.uk CGI IT UK Limited. A CGI Group Inc. Company Registered Office: 250 Brook Drive, Green Park, Reading RG2 6UA, United Kingdom. Registered in England & Wales - Number 947968 CONFIDENTIALITY NOTICE: Proprietary/Confidential Information belonging to CGI Group Inc. and its affiliates may be contained in this message. If you are not a recipient indicated or intended in this message (or responsible for delivery of this message to such person), or you think for any reason that this message may have been addressed to you in error, you may not use or copy or deliver this message to anyone else. In such case, you should destroy this message and are asked to notify the sender by reply e-mail. ________________________________________ From: Sandeep More [[email protected]] Sent: 18 November 2016 16:37 To: [email protected] Subject: Re: Re-write rules not being applied to new service Ah ! got it now and the pattern seems perfectly fine. In which case I think just the following should work (without any additional rules): service.xml <routes> <route path="/solr/**/**?{**}" > <rewrite apply="SOLR/solr/inbound/query" to="request.url"/> </route> </routes> And in rewrite.xml <!--Only supporting Solr queries via Knox --> <rule dir="IN" name="SOLR/solr/inbound/query" pattern="*://*:*/**/solr/{collection}/{query}?{**}"> <rewrite template="{$serviceUrl[SOLR]}/{collection}/{query}?{**}"/> </rule> Let me know if you run into issues, I can take a look at the code. Best, Sandeep On Fri, Nov 18, 2016 at 11:05 AM, McParland, John <[email protected]> wrote: > Hi Sandeep, > > thanks for your help. A part of the difficulty I'm having is that the > "KnoxSolrIntegration" part of the URL is variable - the user sending a > request to Knox needs to specify it (it happens to the be the collection in > Solr which they wish to query). > > E.g. in a normal Solr Query; > > curl -X GET 'http://hdp24sandbox:8983/solr/KnoxIntegrationConfig/ > select?q=*.*&wt=json&indent=true > > All of the pieces after the "/solr/" are variable - the user needs to > specify it regardless of whether going through Knox or directly to Solr. > > So my belief is a Knox equivalent of the above curl command is > > curl -k -u guest:guest-password -X GET 'https://hdp24sandbox:8443/ > gateway/sandbox/solr/KnoxIntegrationConfig/select? > q=*.*&wt=json&indent=true' > > Therefore, I thought the pattern to match (in rewrite.xml) would be > > *://*.*//**/solr/**/**?{**} > > As in my head > > - first * = protocol (http/https) > - second * = domain name / IP address of Knox server > - third * = port of Knox on it's server > > - first ** = knox context and topology (gateway/sandbox in my example) > - second ** = Solr collection to search (KnoxIntegrationConfig in my > example, but something I expect the caller to provide) > - third ** = query type for Solr (e.g. "select") > - {**} = the search parameters for Solr (q=*.*&wt=json&indent=true in my > example) > > I suspect either (or both) > a) I've not captured my pattern correctly (but I think so....) > b) I haven't linked the service to the re-write rule to the routes in the > service definition correctly [as I'm not sure if/how this works from the > tutorial] > > Thanks, > > John McParland MIET CEng | System Architect, ODSC > Health Local and Scotland | CGI > CGI Ltd (UK) > Second Floor, Inovo Building, 121 George St, Glasgow, UK, G1 1RD > M: +44 7920 183 019 > [email protected] | www.cgi-group.co.uk > > CGI IT UK Limited. A CGI Group Inc. Company > Registered Office: 250 Brook Drive, Green Park, Reading RG2 6UA, > United Kingdom. Registered in England & Wales - Number 947968 > > CONFIDENTIALITY NOTICE: Proprietary/Confidential Information belonging to > CGI Group Inc. and its affiliates may be contained in this message. If you > are not a recipient indicated or intended in this message (or responsible > for delivery of this message to such person), or you think for any reason > that this message may have been addressed to you in error, you may not use > or copy or deliver this message to anyone else. In such case, you should > destroy this message and are asked to notify the sender by reply e-mail. > > ________________________________________ > From: Sandeep More [[email protected]] > Sent: 18 November 2016 14:11 > To: [email protected] > Subject: Re: Re-write rules not being applied to new service > > Hello John, > > I think you are missing the definitions for the path > '/solr/KnoxSolrIntegration' > > Looking at your definition files you could perhaps have something like > this: > > service.xml > > <routes> > <route path="/solr/**/**?{**}" > > <rewrite apply="SOLR/solr/inbound/query" > to="request.url"/> > </route> > > <route path="/solr/KnoxSolrIntegration**" > > <rewrite apply="SOLR/solr/inbound/KnoxSolrIntegration" > to="request.url"/> > </route> > </routes> > > And in rewrite.xml > > <!--Only supporting Solr queries via Knox --> > <rule dir="IN" name="SOLR/solr/inbound/query" > pattern="*://*:*/**/solr/{collection}/{query}?{**}"> > <rewrite template="{$serviceUrl[SOLR]}/ > {collection}/{query}?{**}"/> > </rule> > <rule dir="IN" name="SOLR/solr/inbound/KnoxSolrIntegration" > pattern="*://*:*/**/solr/KnoxSolrIntegration**"> > <rewrite template="{$serviceUrl[SOLR]}/KnoxSolrIntegration{**}"/> > </rule> > > Looks like you are missing the <route> for 'SOLR/solr/inbound/query' in > your service.xml file (like in the above example). > > Couple of things that helped me were > 1) Try to keep the rules as tight as possible e.g. > /solr/KnoxSolrIntegration** > instead of /solr/** (provided an example with rule #2 in the above example) > 2) In your Service.xml file you have multiple <routes>, you could > consolidate them into single <routes> > > I haven't tested locally it so apologies in advance if it does not work. > > > Hope that helps ! > > Best, > Sandeep > > On Fri, Nov 18, 2016 at 7:13 AM, McParland, John <[email protected]> > wrote: > > > Hi all, > > > > I'm working to add support for Solr's API to Apache Knox. To do so, I've > > created a service.xml and rewrite.xml, and deployed to > > ${GATEWAY_HOME}/data/conf/services/solr/5.5.0 > > > > See the files here: https://github.com/mcparlandjc > > gi/knox/tree/KNOX-528/gateway-service-definitions/src/main/r > > esources/services/solr/5.5.0 > > I also wrote a test to check that the re-write rules work as expected > (see > > https://github.com/mcparlandjcgi/knox/blob/KNOX-528/gateway- > > provider-rewrite/src/test/java/org/apache/hadoop/ > > gateway/filter/rewrite/api/UrlRewriteProcessorTest.java#L281) > > > > I have defined the Solr service in my sandbox topology like so; > > > > <service> > > <role>SOLR</role> > > <url>http://sandbox.hortonworks.com:8983/solr</url> > > </service> > > > > However when I attempt to execute a Solr Query via Knox, I find that the > > rule isn't applied and indeed, no matching rule is found at all. > > > > 2016-11-18 12:04:23,221 TRACE gateway.access (AccessHandler.java:log(49)) > > - |||158.234.220.6|GET|/gateway/sandbox/solr/KnoxSolrIntegrati > > on/select?q=*.*&wt=json&indent=true|-1|200|0|6 > > 2016-11-18 12:04:23,231 TRACE http.request (TraceRequest.java: > traceRequestDetails(66)) > > - ||82e1c7f7-4eb5-499d-8f29-7602652079f1|Request=GET > > /gateway/sandbox/solr/KnoxSolrIntegration/select?q=* > .*&wt=json&indent=true > > Header[User-Agent]=curl/7.47.0 > > Header[Authorization]=Basic Z3Vlc3Q6Z3Vlc3QtcGFzc3dvcmQ= > > Header[Accept]=*/* > > Header[Host]=hdp24sandbox:8443 > > 2016-11-18 12:04:23,786 DEBUG hadoop.gateway > (GatewayFilter.java:doFilter(116)) > > - Received request: GET /solr/KnoxSolrIntegration/select > > 2016-11-18 12:04:23,888 TRACE hadoop.gateway (UrlRewriteProcessor.java: > rewrite(177)) > > - No rule matching URL: hdp24sandbox:8443, direction: IN > > 2016-11-18 12:04:23,889 TRACE hadoop.gateway (UrlRewriteProcessor.java: > rewrite(177)) > > - No rule matching URL: 8443, direction: IN > > 2016-11-18 12:04:23,889 TRACE hadoop.gateway (UrlRewriteProcessor.java: > rewrite(177)) > > - No rule matching URL: https, direction: IN > > 2016-11-18 12:04:25,052 TRACE hadoop.gateway (UrlRewriteProcessor.java: > rewrite(177)) > > - No rule matching URL: https://hdp24sandbox:8443/gateway/sandbox/solr/ > KnoxSolrIntegrati > > on/select?q=*.*&wt=json&indent=true, direction: IN > > 2016-11-18 12:04:25,062 TRACE hadoop.gateway (UrlRewriteProcessor.java: > rewrite(177)) > > - No rule matching URL: 158.234.220.6, direction: IN > > 2016-11-18 12:04:25,062 TRACE hadoop.gateway (UrlRewriteProcessor.java: > rewrite(177)) > > - No rule matching URL: https, direction: IN > > 2016-11-18 12:04:25,063 TRACE hadoop.gateway (UrlRewriteProcessor.java: > rewrite(177)) > > - No rule matching URL: 8443, direction: IN > > 2016-11-18 12:04:25,063 TRACE hadoop.gateway (UrlRewriteProcessor.java: > rewrite(177)) > > - No rule matching URL: hdp24sandbox:8443, direction: IN > > 2016-11-18 12:04:25,063 TRACE hadoop.gateway (UrlRewriteProcessor.java: > rewrite(177)) > > - No rule matching URL: hdp24sandbox, direction: IN > > 2016-11-18 12:04:25,063 TRACE hadoop.gateway (UrlRewriteProcessor.java: > rewrite(177)) > > - No rule matching URL: /gateway/sandbox, direction: IN > > 2016-11-18 12:04:25,064 TRACE hadoop.gateway (UrlRewriteProcessor.java: > rewrite(177)) > > - No rule matching URL: curl/7.47.0, direction: IN > > 2016-11-18 12:04:25,069 TRACE hadoop.gateway (UrlRewriteProcessor.java: > rewrite(177)) > > - No rule matching URL: Basic Z3Vlc3Q6Z3Vlc3QtcGFzc3dvcmQ=, direction: IN > > 2016-11-18 12:04:25,070 TRACE hadoop.gateway (UrlRewriteProcessor.java: > rewrite(177)) > > - No rule matching URL: */*, direction: IN > > 2016-11-18 12:04:25,070 TRACE hadoop.gateway (UrlRewriteProcessor.java: > rewrite(177)) > > - No rule matching URL: hdp24sandbox:8443, direction: IN > > 2016-11-18 12:04:25,072 DEBUG hadoop.gateway (DefaultDispatch.java: > executeOutboundRequest(120)) > > - Dispatch request: GET https://hdp24sandbox:8443/gate > > way/sandbox/solr/KnoxSolrIntegration/select?q=*.*&wt=json&indent=true > > > > What I expect at the end is the Dispatch request to go to > > > > http://sandbox.hortonworks.com:8983/solr/KnoxSolrIntegration > > /select?q=*.*&wt=json&indent=true > > > > Is there a missing relationship or problem in my service/rules which > > prevents the rule being matched with the request URL? > > > > Thanks, > > > > John McParland MIET CEng | System Architect, ODSC > > Health Local and Scotland | CGI > > CGI Ltd (UK) > > Second Floor, Inovo Building, 121 George St, Glasgow, UK, G1 1RD > > M: +44 7920 183 019 > > [email protected] | www.cgi-group.co.uk > > > > CGI IT UK Limited. A CGI Group Inc. Company > > Registered Office: 250 Brook Drive, Green Park, Reading RG2 6UA, > > United Kingdom. Registered in England & Wales - Number 947968 > > > > CONFIDENTIALITY NOTICE: Proprietary/Confidential Information belonging to > > CGI Group Inc. and its affiliates may be contained in this message. If > you > > are not a recipient indicated or intended in this message (or responsible > > for delivery of this message to such person), or you think for any reason > > that this message may have been addressed to you in error, you may not > use > > or copy or deliver this message to anyone else. In such case, you should > > destroy this message and are asked to notify the sender by reply e-mail. > > >
