Hi Nick - I am happy to hear that the topology touch helped! That gets me so often.
It is going to take me a while to go through your observations and questions in this note. For what it's worth, I also have to relearn a good bit of these things each time I run into something new. The fact that we really started with REST APIs only means that we are constantly trying to chase UI specific patterns now and hoping things line up as we need. It does occur to me that it may be really useful to not just document these things but to have a living breathing example with rewrite rules for various patterns. Like the Java petstore app for service definition authoring. This is how you deal with css and js. This is how you deal with common framework files that are in different apps but they are in the same topology. This is how you deal with rewriting hostnames inside html bodies. This is how you deal with hostnames that are dynamically created in js. This is how you deal with a redirect header. etc... Don't know if we would make this just an article or put it in the examples or what but we should definitely consider this. Anyway, I will try and find some time to go through your questions and shed some light on (and probably learn myself) and get back to you. thanks, --larry On Sat, Jan 20, 2018 at 9:11 AM, Piper, Nick <[email protected]> wrote: > Hi Larry and the Knox mailing list - > > Thanks for the pointer on touching the topology file - indeed that was > something I'd been forgetting. I've got small examples working, but > while building up the full configuration I've found I don't understand > fully how the rewriting syntax works. > > We've got some follow up questions/clarifications to help our > understanding of how rewriting is configured. Here's the starting > point (Knox 0.14.0): > > <topology> > <gateway> > <provider> > <role>identity-assertion</role> > <name>Default</name> > <enabled>true</enabled> > </provider> > </gateway> > <service> > <role>EXAMPLEUI</role> > <url>http://localhost:8000</url> > </service> > </topology> > > <rules> > <rule dir="IN" name="EXAMPLEUI/exampleui/inbound/path" > pattern="*://*:*/**/example/{**}"> > <rewrite template="{$serviceUrl[EXAMPLEUI]}/{**}"/> > </rule> > </rules> > > <service role="EXAMPLEUI" name="exampleui" version="0.0.1"> > <policies> > <policy role="webappsec"/> > <policy role="authentication" name="Anonymous"/> > <policy role="rewrite"/> > <policy role="authorization"/> > </policies> > <routes> > <route path="/example/**"> > </route> > </routes> > <dispatch classname="org.apache.hadoop.gateway.dispatch. > PassAllHeadersDispatch"/> > </service> > > I've discovered the following things, please could you someone let me > know if they are right? > > This rule would be applied 'implicitly', to any HTML which matches > /example/** in the URL (after the gateway/sandbox part): > > <rule dir="OUT" name="EXAMPLEUI/exampleui/outbound/cssrule" > pattern="{filename=*.css}"> > <rewrite template="{$frontend[path]}/example/{filename=*.css}"/> > </rule> > > No <filter> is needed. Is this because for text/html (in fact, > anything at > https://github.com/apache/knox/blob/92e2ec59a5940a9e7c67ec5cd29044 > f811dee40a/gateway-provider-rewrite/src/main/java/org/ > apache/knox/gateway/filter/rewrite/impl/html/HtmlUrlRewriteStreamFilter. > java#L33) > all rules are applied? And for other content-types, such as > application/javascript, filter is needed to explicitly invoke rules. > > It seems that the following are not valid, but were iterations I > attempted (from memory): > > ---------- > > <rule dir="OUT" name="EXAMPLEUI/exampleui/outbound/cssrule" > pattern="{*.css}"> > <rewrite template="{$frontend[path]}/example/{*.css}"/> > </rule> > > Matches everything, which is a surprise. Is the .css part ignored? > > 2018-01-19 18:40:12,410 DEBUG hadoop.gateway > (UrlRewriteProcessor.java:rewrite(164)) > - Rewrote URL: text/javascript, direction: OUT via implicit rule: > EXAMPLEUI/exampleui/outbound/cssrule to URL: > /gateway/sandbox/example/text/javascript > > ---------- > > <rule dir="OUT" name="EXAMPLEUI/exampleui/outbound/cssrule" > pattern="{*}.css"> > <rewrite template="{$frontend[path]}/example/{*}.css"/> > </rule> > > This doesn't match anything, which is also a surprise. > > ---------- > > <rule dir="OUT" name="EXAMPLEUI/exampleui/outbound/cssrule" > pattern="{**}.css"> > <rewrite template="{$frontend[path]}/example/{**}"/> > </rule> > > Still doesn't match anything. I think I tried with and without .css in > the 'template' attribute (although I'd never expect that to influence > if there is a match or or not in the first place.) > > ---------- > > <rule dir="OUT" name="EXAMPLEUI/exampleui/outbound/cssrule" > pattern="{**.css}"> > <rewrite template="{$frontend[path]}/example/{**.css}"/> > </rule> > > Matches too much again, including things that don't end .css: > > 2018-01-19 19:07:05,853 DEBUG hadoop.gateway > (UrlRewriteProcessor.java:rewrite(164)) > - Rewrote URL: vendor.bundle.js, direction: OUT via implicit rule: > EXAMPLEUI/exampleui/outbound/cssrule to URL: /gateway/sandbox/example/ > vendor.xxxxx.bundle.js > 2018-01-19 19:07:05,853 DEBUG hadoop.gateway > (UrlRewriteProcessor.java:rewrite(164)) > - Rewrote URL: text/javascript, direction: OUT via implicit rule: > EXAMPLEUI/exampleui/outbound/cssrule to URL: > /gateway/sandbox/example/text/javascript > > ---------- > > <rule dir="OUT" name="EXAMPLEUI/exampleui/outbound/cssrule" > pattern="{filename=*}.css"> > <rewrite template="{$frontend[path]}/example/{filename=*}.css"/> > </rule> > > Doesn't match anything > > ---------- > > At https://cwiki.apache.org/confluence/display/KNOX/2017/ > 08/14/Understanding+Rewrite+Rules+for+Apache+Knox# > UnderstandingRewriteRulesforApacheKnox-URLTemplates there is this > information: > > {path} => {path=*} > {path=*} // Match single path level. (ie wildcard) > {path=**} // Match multiple path levels. (ie glob) > {path=*.ext} // Match single level with simplified regex pattern. > > Do I paraphrase correctly: > > There are the following match types: > *, **, *.ext > > They are neither regular expressions nor globs, and each match type > has to match the full contents of that part of the URL, after > splitting by / (e.g., {part=*.ext} is ok, but {part=*}.ext is not OK. > > The matched content is placed into a buffer named after variable name > before the =, and can be referred to in the rewrite template attribute by > that name, and also by that name followed by = and then anything. > <rewrite template="{$frontend[path]}/example/{filename=*.css}"/> > <rewrite template="{$frontend[path]}/example/{filename}"/> > <rewrite template="{$frontend[path]}/example/{filename=*.different}"/> > > If no name is used, in the case of {*} and {**}, then those same exact > strings can be used in the template attribute and are expanded to the > matching characters. > > --------------------- > > The above are questions regarding how the rewrite syntax works, but I > also have some questions about filters and routes. > > It appears this implicit firing of rules for text/html stops when a > specific rewrite apply/to is added *even for a different route > path*. e.g., this rewrites the html (in a URL matching > example/index.html): > > <routes> > <route path="/example/**/*.js"> > </route> > <route path="/example/**"> > </route> > </routes> > > but this STOPs writing the html, even though the <rewrite> is inside a > route with path which is unrelated: > > <routes> > <route path="/example/**/*.js"> > <rewrite apply="EXAMPLEUI/exampleui/outbound/jsfilter" > to="response.body"/> > </route> > <route path="/example/**"> > </route> > </routes> > > --------------------- > > (The following diagrams need a fixed-width font) > > ...This matches the URL (after gateway/sandbox) > . - unsure what syntax is used > . > . > <route path="/example/**/*.js"> > <rewrite apply="EXAMPLEUI/exampleui/outbound/jsfilter" > to="response.body"/> > </route> ^ > ^ > . > . > . > . > . What things can go > here? > . > . > This is the name of a filter (not a rule) > > > > ...This has to be globally unique, > . since all filters are combined together > . > . > <filter name="EXAMPLEUI/exampleui/outbound/jsfilter"> > > .....This both decides which set of 'apply' is > . applied, and also which parsing engine > is used > . > . > <content type="application/javascript"> > <apply path=".*\.js" rule="EXAMPLEUI/exampleui/outbound/api"/> > </content> . . > </filter> . . > . This is the name of a rule > . defined elsewhere > . > . > This matches a string in the content that the parser has > tokenized out (e.g., the URL in the content, such as href > attribute) it's a regular expression. Unsure how this > relates to the pattern attribute in the rule which is > referenced > > > --------------------- > > Many thanks for help with our understanding. We will update > https://cwiki.apache.org/confluence/display/KNOX/2017/ > 08/14/Understanding+Rewrite+Rules+for+Apache+Knox > to fold in any clarifications from the mailing list (although, that > looks like a 'blog' entry, so maybe we should copy and update a > non-dated location? - probably https://knox.apache.org/books/ > knox-0-14-0/dev-guide.html#Rewrite+Provider ) > > Regards, > > Nick > > ________________________________________ > From: larry mccay [[email protected]] > Sent: 17 January 2018 6:00 PM > To: [email protected] > Subject: Re: Problems getting started with URL rewriting > > Hi Stuart - > > At first glance, it seems to look okay. > > I am wondering whether you are just running into the difficult dance of > what you need to do to get the service definitions changes updated. > > I believe that you will find that you need to do the following when you > iterate over changes in rewrite rules: > > 1. Change rewrite rule and save > 2. restart server > 3. touch the topology that hosts the service that you changed the rewrite > for > > #3 is the one that always burns me. > Keep in mind that webapps are generated by topologies and regenerated > whenever the topology is changed. > > Even if the server is aware of the changes the previous webapp still is > deployed until the topology is changed. > > HTH, > > --larry > > > On Wed, Jan 17, 2018 at 12:25 PM, Hickinbottom, Stuart < > [email protected]> wrote: > > > Using Knox 0.11 and reproduced with Knox 0.14, we're having trouble > > getting basic URL rewriting to work. The real problem is with an > > ExpressJS application, and the URL is part of some JavaScript, but I > > wanted to get this basic example working first. Please may I ask for > > help to spot what I'm doing wrong? The HTML that goes to the browser > > is entirely unchanged from the HTML served by the back end > > application, while I'd expected the href URLs to be changed on the way > > out. > > > > How much parsing does the rules/rewrite filters attempt? How might a > > Webpack encoding affect this? (Although this isn't relevant to this > > simple HTML example.) > > > > Is there a way to turn up logging (already have > > log4j.logger.org.apache.hadoop.gateway=DEBUG) to see what rules and > > filters are being applied (or not)? > > > > example.html > > > > <html> > > <body> > > <a href="/api/v1/user">hello</a> > > <a href="/api/v1">hello</a> > > </body> > > </html> > > > > service.xml > > > > <service role="EXAMPLEUI" name="exampleui" version="0.0.1"> > > <policies> > > <policy role="webappsec"/> > > <policy role="authentication" name="Anonymous"/> > > <policy role="rewrite"/> > > <policy role="authorization"/> > > </policies> > > <routes> > > <route path="/exampleui"> > > <rewrite apply="EXAMPLEUI/exampleui/inbound/root" > > to="request.url"/> > > <rewrite apply="EXAMPLEUI/exampleui/outbound/html/filter" > > to="response.body"/> > > </route> > > <route path="/exampleui/**"> > > <rewrite apply="EXAMPLEUI/exampleui/inbound/path" > > to="request.url"/> > > <rewrite apply="EXAMPLEUI/exampleui/outbound/html/filter" > > to="response.body"/> > > </route> > > </routes> > > <dispatch classname="org.apache.hadoop.gateway.dispatch. > > PassAllHeadersDispatch"/> > > </service> > > > > rewrite.xml > > > > <rules> > > <rule dir="IN" name="EXAMPLEUI/exampleui/inbound/root" > > pattern="*://*:*/**/exampleui/"> > > <rewrite template="{$serviceUrl[EXAMPLEUI]}/"/> > > </rule> > > <rule dir="IN" name="EXAMPLEUI/exampleui/inbound/path" > > pattern="*://*:*/**/exampleui/{**}"> > > <rewrite template="{$serviceUrl[EXAMPLEUI]}/{**}"/> > > </rule> > > > > <rule dir="OUT" name="EXAMPLEUI/exampleui/outbound/apipath"> > > <rewrite template="{$frontend[path]}/exampleui/api/v1"/> > > </rule> > > <filter name="EXAMPLEUI/exampleui/outbound/html/filter"> > > <content type="text/html"> > > <apply path="/api/v1" rule="EXAMPLEUI/exampleui/ > > outbound/apipath"/> > > </content> > > </filter> > > > > </rules> > > > > > > > > ==> /var/log/hadoop/knox/gateway.log <== > > 2018-01-17 16:54:46,934 DEBUG hadoop.gateway (DefaultDispatch.java: > executeOutboundRequest(133)) > > - Dispatch response status: 200 > > 2018-01-17 16:54:46,935 DEBUG hadoop.gateway (DefaultDispatch.java: > > getInboundResponseContentType(202)) - Using explicit character set UTF-8 > > for entity of type text/html > > 2018-01-17 16:54:46,935 DEBUG hadoop.gateway (DefaultDispatch.java: > > getInboundResponseContentType(210)) - Inbound response entity content > > type: text/html; charset=UTF-8 > > > > ==> /var/log/hadoop/knox/gateway-audit.log <== > > 18/01/17 16:54:46 ||47193300-cc6c-48e6-aad0- > e2b04f2db613|audit|EXAMPLEUI|| > > ||dispatch|uri|http://server.project:4201/example.html|success|Response > > status: 200 > > > > ==> /var/log/hadoop/knox/gateway.log <== > > > > ==> /var/log/hadoop/knox/gateway-audit.log <== > > 18/01/17 16:54:46 ||47193300-cc6c-48e6-aad0- > e2b04f2db613|audit|EXAMPLEUI|| > > ||access|uri|/gateway/project/exampleui/example.html|success|Response > > status: 200 > > > > > > > > Many thanks for any pointers. > > > > We've read through: > > > > https://github.com/apache/knox/blob/master/gateway- > > service-definitions/src/main/resources/services/ambariui/2. > 2.0/rewrite.xml > > https://cwiki.apache.org/confluence/display/KNOX/ > Proxying+a+UI+using+Knox > > https://cwiki.apache.org/confluence/display/KNOX/2017/ > > 08/14/Understanding+Rewrite+Rules+for+Apache+Knox# > > UnderstandingRewriteRulesforApacheKnox-JavaScriptURLRewriteFilter > > > > Regards, > > Stuart > > > > >
