This is weird,
Do you have SOLR entry in your topology file (alongside the new SOLRAPI) by
any chance ?
A long shot but can you try stopping the gateway and deleting all the data
in {KNOX}/data/deployments/
and then starting gateway back up. In the deployments directory you can see
WAR file for your topology it will contain the rewrite.xml file (under
%2F/WEB-XML) you should see the SOLAPI mapping there and should not see
(ideally) the old SOLR mapping.
I tried on my local machine and the rewriting seems to be working fine.
This is the code snippet from my logs.
2016-11-19 19:07:57,944 DEBUG hadoop.gateway
(UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL:
https://localhost:8443/gateway/sandbox/solr/gettingstarted/select?indent=on&q=*&wt=json,
direction: IN via explicit rule: SOLRAPI/solr/inbound/query to URL:
http://localhost:8983/solr/gettingstarted/select?q=*&indent=on&wt=json
And I have my topology file updated as follows.
<service>
<role>SOLRAPI</role>
<url>http://localhost:8983/solr</url>
</service>
Best,
Sandeep
On Sat, Nov 19, 2016 at 3:51 PM, McParland, John <[email protected]>
wrote:
> Thanks Sandeep,
>
> that makes a good deal of sense. Obviously I need those {collection=**}
> to capture the collection itself etc.
>
> Final hurdle now seems to be on renaming the service to SOLRAPI (I've no
> intention to proxy the UI as this time).
>
> New service/rewrite here: https://github.com/mcparlandjcgi/knox/tree/KNOX-
> 528/gateway-service-definitions/src/main/resources/services/solr/5.5.0
>
> And the updated topology includes the following:
> <service>
> <role>SOLRAPI</role>
> <url>http://sandbox.hortonworks.com:8983/solr</url>
> </service>
>
> But the logs, when I send a request through, show this;
>
> 2016-11-19 20:43:11,605 DEBUG hadoop.gateway
> (UrlRewriteProcessor.java:rewrite(166))
> - Rewrote URL: https://hdp24sandbox:8443/gateway/sandbox/solr/
> KnoxIntegrationConfig/select?q=*.*&wt=json&indent=true, direction: IN via
> explicit rule: SOLRAPI/solr/inbound/query to URL:
> SOLR/KnoxIntegrationConfig/select?indent=true&q=*.*&wt=json
>
> I don't understand why the old "SOLR" is being used at the start of the
> re-written URL. It's almost as if the gateway (which I restart using
> bin/gateway.sh stop / bin/gateway.sh start in-between changes) isn't
> picking up the new topology correctly.
>
> Thanks,
>
> John McParland MIET CEng | System Architect, ODSC
> Health Local and Scotland | CGI
> CGI Ltd (UK)
> Second Floor, Inovo Building, 121 George St, Glasgow, UK, G1 1RD
> M: +44 7920 183 019
> [email protected] | www.cgi-group.co.uk
>
> CGI IT UK Limited. A CGI Group Inc. Company
> Registered Office: 250 Brook Drive, Green Park, Reading RG2 6UA,
> United Kingdom. Registered in England & Wales - Number 947968
>
> CONFIDENTIALITY NOTICE: Proprietary/Confidential Information belonging to
> CGI Group Inc. and its affiliates may be contained in this message. If you
> are not a recipient indicated or intended in this message (or responsible
> for delivery of this message to such person), or you think for any reason
> that this message may have been addressed to you in error, you may not use
> or copy or deliver this message to anyone else. In such case, you should
> destroy this message and are asked to notify the sender by reply e-mail.
>
> ________________________________________
> From: Sandeep More [[email protected]]
> Sent: 19 November 2016 17:56
> To: [email protected]
> Subject: Re: Re-write rules not being applied to new service
>
> Hello John,
>
> You have some minor errors that are causing the paths to not match they way
> you want them to. I did some local testing and the following service
> definitions appear to work locally on my machine ( using your github
> example as baseline )
>
> *service.xml (complete file)*
>
> <service role="SOLR" name="solr" version="5.5.0">
> <policies>
> <policy role="webappsec"/>
> <policy role="authentication" name="Anonymous"/>
> <policy role="rewrite"/>
> <policy role="authorization"/>
> </policies>
> <routes>
>
> <route path="/solr/**/**?**">
> <rewrite apply="SOLR/solr/inbound/query" to="request.url"/>
> </route>
>
> </routes>
> </service>
>
> *rewrite.xml (complete file)*
>
> <rules>
>
> <rule dir="IN" name="SOLR/solr/inbound/query"
> pattern="*://*:*/**/solr/{collection=**}/{query=**}?{**}">
> <rewrite
> template="{$serviceUrl[SOLR]}/{collection=**}/{query=**}?{**}"/>
> </rule>
>
> </rules>
>
> your topology file seems right. I was able to get the basic
> "gettingstarted" solr query to work.
> You probably need to add some rules for the UI if you like to support it.
> Also, you might want to call the above service something like "SOLRAPI"
> rather then "SOLR" just to differentiate if from UI traffic.
>
> There are minor regex changes I had to do to make it work. Let me know if
> it works !
>
> To understand UI Proxying with Knox refer to this excellent guide by Sumit
> Gupta -
> https://cwiki.apache.org/confluence/display/KNOX/Proxying+a+UI+using+Knox
>
> Another good resource on adding a service to Knox by Kevin Minder is - -
> https://cwiki.apache.org/confluence/display/KNOX/2015/
> 12/17/Adding+a+service+to+Apache+Knox
>
> These blogs explain in-detail process to add services and are a step by
> step guides to adding support for knox proxying.
>
> You also asked about debugging, generally for adding UIs I just tail the
> logs and try to figure out from logs and change the logging level if
> needed. If you would like to debug using an IDE you can use the command
> "ant
> start-debug-gateway" from the knox source directory.
>
> Sorry for the lengthy email, hope it addresses your questions !
>
> Best,
> Sandeep
>
> On Fri, Nov 18, 2016 at 4:29 PM, McParland, John <[email protected]>
> wrote:
>
> > Thanks again Sandeep,
> >
> > I tried your suggestion, and while I can understand it and believe it
> > should have worked, it still doesn't! I have the same error message as
> > below.
> >
> > From my own understanding, I think the rules in rewrite.xml are fine, and
> > the JUnit test proves those should work.
> >
> > However I haven't understood how the route in service.xml is
> > picked/matched with the incoming URL. And this appears (again from what
> I
> > understand) to be where the problem is. My understanding of the
> processing
> > of an incoming URL is;
> >
> > 1) Check URL against routes (not clear on this)
> > 2) Find re-write rule associated with it (presume this is just a lookup
> > of a map of URL -> rule name)
> > 3) Check the URL matches the pattern in the re-write rule
> > 4) Apply the transformation in the template of the re-write rule.
> >
> > Is there anything else I can do to debug this part (e.g. see what routes
> > we have at runtime)?
> >
> > Thanks,
> >
> > John McParland MIET CEng | System Architect, ODSC
> > Health Local and Scotland | CGI
> > CGI Ltd (UK)
> > Second Floor, Inovo Building, 121 George St, Glasgow, UK, G1 1RD
> > M: +44 7920 183 019
> > [email protected] | www.cgi-group.co.uk
> >
> > CGI IT UK Limited. A CGI Group Inc. Company
> > Registered Office: 250 Brook Drive, Green Park, Reading RG2 6UA,
> > United Kingdom. Registered in England & Wales - Number 947968
> >
> > CONFIDENTIALITY NOTICE: Proprietary/Confidential Information belonging to
> > CGI Group Inc. and its affiliates may be contained in this message. If
> you
> > are not a recipient indicated or intended in this message (or responsible
> > for delivery of this message to such person), or you think for any reason
> > that this message may have been addressed to you in error, you may not
> use
> > or copy or deliver this message to anyone else. In such case, you should
> > destroy this message and are asked to notify the sender by reply e-mail.
> >
> > ________________________________________
> > From: Sandeep More [[email protected]]
> > Sent: 18 November 2016 16:37
> > To: [email protected]
> > Subject: Re: Re-write rules not being applied to new service
> >
> > Ah ! got it now and the pattern seems perfectly fine.
> > In which case I think just the following should work (without any
> > additional rules):
> >
> > service.xml
> >
> > <routes>
> > <route path="/solr/**/**?{**}" >
> > <rewrite apply="SOLR/solr/inbound/query"
> > to="request.url"/>
> > </route>
> > </routes>
> >
> > And in rewrite.xml
> >
> > <!--Only supporting Solr queries via Knox -->
> > <rule dir="IN" name="SOLR/solr/inbound/query"
> > pattern="*://*:*/**/solr/{collection}/{query}?{**}">
> > <rewrite template="{$serviceUrl[SOLR]}/
> > {collection}/{query}?{**}"/>
> > </rule>
> >
> > Let me know if you run into issues, I can take a look at the code.
> >
> > Best,
> > Sandeep
> >
> >
> > On Fri, Nov 18, 2016 at 11:05 AM, McParland, John <
> [email protected]>
> > wrote:
> >
> > > Hi Sandeep,
> > >
> > > thanks for your help. A part of the difficulty I'm having is that the
> > > "KnoxSolrIntegration" part of the URL is variable - the user sending a
> > > request to Knox needs to specify it (it happens to the be the
> collection
> > in
> > > Solr which they wish to query).
> > >
> > > E.g. in a normal Solr Query;
> > >
> > > curl -X GET 'http://hdp24sandbox:8983/solr/KnoxIntegrationConfig/
> > > select?q=*.*&wt=json&indent=true
> > >
> > > All of the pieces after the "/solr/" are variable - the user needs to
> > > specify it regardless of whether going through Knox or directly to
> Solr.
> > >
> > > So my belief is a Knox equivalent of the above curl command is
> > >
> > > curl -k -u guest:guest-password -X GET 'https://hdp24sandbox:8443/
> > > gateway/sandbox/solr/KnoxIntegrationConfig/select?
> > > q=*.*&wt=json&indent=true'
> > >
> > > Therefore, I thought the pattern to match (in rewrite.xml) would be
> > >
> > > *://*.*//**/solr/**/**?{**}
> > >
> > > As in my head
> > >
> > > - first * = protocol (http/https)
> > > - second * = domain name / IP address of Knox server
> > > - third * = port of Knox on it's server
> > >
> > > - first ** = knox context and topology (gateway/sandbox in my example)
> > > - second ** = Solr collection to search (KnoxIntegrationConfig in my
> > > example, but something I expect the caller to provide)
> > > - third ** = query type for Solr (e.g. "select")
> > > - {**} = the search parameters for Solr (q=*.*&wt=json&indent=true in
> my
> > > example)
> > >
> > > I suspect either (or both)
> > > a) I've not captured my pattern correctly (but I think so....)
> > > b) I haven't linked the service to the re-write rule to the routes in
> the
> > > service definition correctly [as I'm not sure if/how this works from
> the
> > > tutorial]
> > >
> > > Thanks,
> > >
> > > John McParland MIET CEng | System Architect, ODSC
> > > Health Local and Scotland | CGI
> > > CGI Ltd (UK)
> > > Second Floor, Inovo Building, 121 George St, Glasgow, UK, G1 1RD
> > > M: +44 7920 183 019
> > > [email protected] | www.cgi-group.co.uk
> > >
> > > CGI IT UK Limited. A CGI Group Inc. Company
> > > Registered Office: 250 Brook Drive, Green Park, Reading RG2 6UA,
> > > United Kingdom. Registered in England & Wales - Number 947968
> > >
> > > CONFIDENTIALITY NOTICE: Proprietary/Confidential Information belonging
> to
> > > CGI Group Inc. and its affiliates may be contained in this message. If
> > you
> > > are not a recipient indicated or intended in this message (or
> responsible
> > > for delivery of this message to such person), or you think for any
> reason
> > > that this message may have been addressed to you in error, you may not
> > use
> > > or copy or deliver this message to anyone else. In such case, you
> should
> > > destroy this message and are asked to notify the sender by reply
> e-mail.
> > >
> > > ________________________________________
> > > From: Sandeep More [[email protected]]
> > > Sent: 18 November 2016 14:11
> > > To: [email protected]
> > > Subject: Re: Re-write rules not being applied to new service
> > >
> > > Hello John,
> > >
> > > I think you are missing the definitions for the path
> > > '/solr/KnoxSolrIntegration'
> > >
> > > Looking at your definition files you could perhaps have something like
> > > this:
> > >
> > > service.xml
> > >
> > > <routes>
> > > <route path="/solr/**/**?{**}" >
> > > <rewrite apply="SOLR/solr/inbound/query"
> > > to="request.url"/>
> > > </route>
> > >
> > > <route path="/solr/KnoxSolrIntegration**" >
> > > <rewrite apply="SOLR/solr/inbound/KnoxSolrIntegration"
> > > to="request.url"/>
> > > </route>
> > > </routes>
> > >
> > > And in rewrite.xml
> > >
> > > <!--Only supporting Solr queries via Knox -->
> > > <rule dir="IN" name="SOLR/solr/inbound/query"
> > > pattern="*://*:*/**/solr/{collection}/{query}?{**}">
> > > <rewrite template="{$serviceUrl[SOLR]}/
> > > {collection}/{query}?{**}"/>
> > > </rule>
> > > <rule dir="IN" name="SOLR/solr/inbound/KnoxSolrIntegration"
> > > pattern="*://*:*/**/solr/KnoxSolrIntegration**">
> > > <rewrite template="{$serviceUrl[SOLR]}/
> KnoxSolrIntegration{**}"/>
> > > </rule>
> > >
> > > Looks like you are missing the <route> for 'SOLR/solr/inbound/query' in
> > > your service.xml file (like in the above example).
> > >
> > > Couple of things that helped me were
> > > 1) Try to keep the rules as tight as possible e.g.
> > > /solr/KnoxSolrIntegration**
> > > instead of /solr/** (provided an example with rule #2 in the above
> > example)
> > > 2) In your Service.xml file you have multiple <routes>, you could
> > > consolidate them into single <routes>
> > >
> > > I haven't tested locally it so apologies in advance if it does not
> work.
> > >
> > >
> > > Hope that helps !
> > >
> > > Best,
> > > Sandeep
> > >
> > > On Fri, Nov 18, 2016 at 7:13 AM, McParland, John <
> [email protected]
> > >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'm working to add support for Solr's API to Apache Knox. To do so,
> > I've
> > > > created a service.xml and rewrite.xml, and deployed to
> > > > ${GATEWAY_HOME}/data/conf/services/solr/5.5.0
> > > >
> > > > See the files here: https://github.com/mcparlandjc
> > > > gi/knox/tree/KNOX-528/gateway-service-definitions/src/main/r
> > > > esources/services/solr/5.5.0
> > > > I also wrote a test to check that the re-write rules work as expected
> > > (see
> > > > https://github.com/mcparlandjcgi/knox/blob/KNOX-528/gateway-
> > > > provider-rewrite/src/test/java/org/apache/hadoop/
> > > > gateway/filter/rewrite/api/UrlRewriteProcessorTest.java#L281)
> > > >
> > > > I have defined the Solr service in my sandbox topology like so;
> > > >
> > > > <service>
> > > > <role>SOLR</role>
> > > > <url>http://sandbox.hortonworks.com:8983/solr</url>
> > > > </service>
> > > >
> > > > However when I attempt to execute a Solr Query via Knox, I find that
> > the
> > > > rule isn't applied and indeed, no matching rule is found at all.
> > > >
> > > > 2016-11-18 12:04:23,221 TRACE gateway.access
> > (AccessHandler.java:log(49))
> > > > - |||158.234.220.6|GET|/gateway/sandbox/solr/KnoxSolrIntegrati
> > > > on/select?q=*.*&wt=json&indent=true|-1|200|0|6
> > > > 2016-11-18 12:04:23,231 TRACE http.request (TraceRequest.java:
> > > traceRequestDetails(66))
> > > > - ||82e1c7f7-4eb5-499d-8f29-7602652079f1|Request=GET
> > > > /gateway/sandbox/solr/KnoxSolrIntegration/select?q=*
> > > .*&wt=json&indent=true
> > > > Header[User-Agent]=curl/7.47.0
> > > > Header[Authorization]=Basic Z3Vlc3Q6Z3Vlc3QtcGFzc3dvcmQ=
> > > > Header[Accept]=*/*
> > > > Header[Host]=hdp24sandbox:8443
> > > > 2016-11-18 12:04:23,786 DEBUG hadoop.gateway
> > > (GatewayFilter.java:doFilter(116))
> > > > - Received request: GET /solr/KnoxSolrIntegration/select
> > > > 2016-11-18 12:04:23,888 TRACE hadoop.gateway
> (UrlRewriteProcessor.java:
> > > rewrite(177))
> > > > - No rule matching URL: hdp24sandbox:8443, direction: IN
> > > > 2016-11-18 12:04:23,889 TRACE hadoop.gateway
> (UrlRewriteProcessor.java:
> > > rewrite(177))
> > > > - No rule matching URL: 8443, direction: IN
> > > > 2016-11-18 12:04:23,889 TRACE hadoop.gateway
> (UrlRewriteProcessor.java:
> > > rewrite(177))
> > > > - No rule matching URL: https, direction: IN
> > > > 2016-11-18 12:04:25,052 TRACE hadoop.gateway
> (UrlRewriteProcessor.java:
> > > rewrite(177))
> > > > - No rule matching URL: https://hdp24sandbox:8443/
> > gateway/sandbox/solr/
> > > KnoxSolrIntegrati
> > > > on/select?q=*.*&wt=json&indent=true, direction: IN
> > > > 2016-11-18 12:04:25,062 TRACE hadoop.gateway
> (UrlRewriteProcessor.java:
> > > rewrite(177))
> > > > - No rule matching URL: 158.234.220.6, direction: IN
> > > > 2016-11-18 12:04:25,062 TRACE hadoop.gateway
> (UrlRewriteProcessor.java:
> > > rewrite(177))
> > > > - No rule matching URL: https, direction: IN
> > > > 2016-11-18 12:04:25,063 TRACE hadoop.gateway
> (UrlRewriteProcessor.java:
> > > rewrite(177))
> > > > - No rule matching URL: 8443, direction: IN
> > > > 2016-11-18 12:04:25,063 TRACE hadoop.gateway
> (UrlRewriteProcessor.java:
> > > rewrite(177))
> > > > - No rule matching URL: hdp24sandbox:8443, direction: IN
> > > > 2016-11-18 12:04:25,063 TRACE hadoop.gateway
> (UrlRewriteProcessor.java:
> > > rewrite(177))
> > > > - No rule matching URL: hdp24sandbox, direction: IN
> > > > 2016-11-18 12:04:25,063 TRACE hadoop.gateway
> (UrlRewriteProcessor.java:
> > > rewrite(177))
> > > > - No rule matching URL: /gateway/sandbox, direction: IN
> > > > 2016-11-18 12:04:25,064 TRACE hadoop.gateway
> (UrlRewriteProcessor.java:
> > > rewrite(177))
> > > > - No rule matching URL: curl/7.47.0, direction: IN
> > > > 2016-11-18 12:04:25,069 TRACE hadoop.gateway
> (UrlRewriteProcessor.java:
> > > rewrite(177))
> > > > - No rule matching URL: Basic Z3Vlc3Q6Z3Vlc3QtcGFzc3dvcmQ=,
> direction:
> > IN
> > > > 2016-11-18 12:04:25,070 TRACE hadoop.gateway
> (UrlRewriteProcessor.java:
> > > rewrite(177))
> > > > - No rule matching URL: */*, direction: IN
> > > > 2016-11-18 12:04:25,070 TRACE hadoop.gateway
> (UrlRewriteProcessor.java:
> > > rewrite(177))
> > > > - No rule matching URL: hdp24sandbox:8443, direction: IN
> > > > 2016-11-18 12:04:25,072 DEBUG hadoop.gateway (DefaultDispatch.java:
> > > executeOutboundRequest(120))
> > > > - Dispatch request: GET https://hdp24sandbox:8443/gate
> > > > way/sandbox/solr/KnoxSolrIntegration/select?q=*
> .*&wt=json&indent=true
> > > >
> > > > What I expect at the end is the Dispatch request to go to
> > > >
> > > > http://sandbox.hortonworks.com:8983/solr/KnoxSolrIntegration
> > > > /select?q=*.*&wt=json&indent=true
> > > >
> > > > Is there a missing relationship or problem in my service/rules which
> > > > prevents the rule being matched with the request URL?
> > > >
> > > > Thanks,
> > > >
> > > > John McParland MIET CEng | System Architect, ODSC
> > > > Health Local and Scotland | CGI
> > > > CGI Ltd (UK)
> > > > Second Floor, Inovo Building, 121 George St, Glasgow, UK, G1 1RD
> > > > M: +44 7920 183 019
> > > > [email protected] | www.cgi-group.co.uk
> > > >
> > > > CGI IT UK Limited. A CGI Group Inc. Company
> > > > Registered Office: 250 Brook Drive, Green Park, Reading RG2 6UA,
> > > > United Kingdom. Registered in England & Wales - Number 947968
> > > >
> > > > CONFIDENTIALITY NOTICE: Proprietary/Confidential Information
> belonging
> > to
> > > > CGI Group Inc. and its affiliates may be contained in this message.
> If
> > > you
> > > > are not a recipient indicated or intended in this message (or
> > responsible
> > > > for delivery of this message to such person), or you think for any
> > reason
> > > > that this message may have been addressed to you in error, you may
> not
> > > use
> > > > or copy or deliver this message to anyone else. In such case, you
> > should
> > > > destroy this message and are asked to notify the sender by reply
> > e-mail.
> > > >
> > >
> >
>
