I like the idea of having something analogous to the Java pet store for rewriting rule definitions.
On Sat, Jan 20, 2018 at 6:53 AM, larry mccay <[email protected]> wrote: > Hi Nick - > > I am happy to hear that the topology touch helped! > That gets me so often. > > It is going to take me a while to go through your observations and > questions in this note. > For what it's worth, I also have to relearn a good bit of these things each > time I run into something new. > > The fact that we really started with REST APIs only means that we are > constantly trying to chase UI specific patterns now and hoping things line > up as we need. > > It does occur to me that it may be really useful to not just document these > things but to have a living breathing example with rewrite rules for > various patterns. > Like the Java petstore app for service definition authoring. > > This is how you deal with css and js. > This is how you deal with common framework files that are in different apps > but they are in the same topology. > This is how you deal with rewriting hostnames inside html bodies. > This is how you deal with hostnames that are dynamically created in js. > This is how you deal with a redirect header. > etc... > > Don't know if we would make this just an article or put it in the examples > or what but we should definitely consider this. > > Anyway, I will try and find some time to go through your questions and shed > some light on (and probably learn myself) and get back to you. > > thanks, > > --larry > > > On Sat, Jan 20, 2018 at 9:11 AM, Piper, Nick <[email protected]> wrote: > > > Hi Larry and the Knox mailing list - > > > > Thanks for the pointer on touching the topology file - indeed that was > > something I'd been forgetting. I've got small examples working, but > > while building up the full configuration I've found I don't understand > > fully how the rewriting syntax works. > > > > We've got some follow up questions/clarifications to help our > > understanding of how rewriting is configured. Here's the starting > > point (Knox 0.14.0): > > > > <topology> > > <gateway> > > <provider> > > <role>identity-assertion</role> > > <name>Default</name> > > <enabled>true</enabled> > > </provider> > > </gateway> > > <service> > > <role>EXAMPLEUI</role> > > <url>http://localhost:8000</url> > > </service> > > </topology> > > > > <rules> > > <rule dir="IN" name="EXAMPLEUI/exampleui/inbound/path" > > pattern="*://*:*/**/example/{**}"> > > <rewrite template="{$serviceUrl[EXAMPLEUI]}/{**}"/> > > </rule> > > </rules> > > > > <service role="EXAMPLEUI" name="exampleui" version="0.0.1"> > > <policies> > > <policy role="webappsec"/> > > <policy role="authentication" name="Anonymous"/> > > <policy role="rewrite"/> > > <policy role="authorization"/> > > </policies> > > <routes> > > <route path="/example/**"> > > </route> > > </routes> > > <dispatch classname="org.apache.hadoop.gateway.dispatch. > > PassAllHeadersDispatch"/> > > </service> > > > > I've discovered the following things, please could you someone let me > > know if they are right? > > > > This rule would be applied 'implicitly', to any HTML which matches > > /example/** in the URL (after the gateway/sandbox part): > > > > <rule dir="OUT" name="EXAMPLEUI/exampleui/outbound/cssrule" > > pattern="{filename=*.css}"> > > <rewrite template="{$frontend[path]}/example/{filename=*.css}"/> > > </rule> > > > > No <filter> is needed. Is this because for text/html (in fact, > > anything at > > https://github.com/apache/knox/blob/92e2ec59a5940a9e7c67ec5cd29044 > > f811dee40a/gateway-provider-rewrite/src/main/java/org/ > > apache/knox/gateway/filter/rewrite/impl/html/HtmlUrlRewriteStreamFilter. > > java#L33) > > all rules are applied? And for other content-types, such as > > application/javascript, filter is needed to explicitly invoke rules. > > > > It seems that the following are not valid, but were iterations I > > attempted (from memory): > > > > ---------- > > > > <rule dir="OUT" name="EXAMPLEUI/exampleui/outbound/cssrule" > > pattern="{*.css}"> > > <rewrite template="{$frontend[path]}/example/{*.css}"/> > > </rule> > > > > Matches everything, which is a surprise. Is the .css part ignored? > > > > 2018-01-19 18:40:12,410 DEBUG hadoop.gateway (UrlRewriteProcessor.java: > rewrite(164)) > > - Rewrote URL: text/javascript, direction: OUT via implicit rule: > > EXAMPLEUI/exampleui/outbound/cssrule to URL: > > /gateway/sandbox/example/text/javascript > > > > ---------- > > > > <rule dir="OUT" name="EXAMPLEUI/exampleui/outbound/cssrule" > > pattern="{*}.css"> > > <rewrite template="{$frontend[path]}/example/{*}.css"/> > > </rule> > > > > This doesn't match anything, which is also a surprise. > > > > ---------- > > > > <rule dir="OUT" name="EXAMPLEUI/exampleui/outbound/cssrule" > > pattern="{**}.css"> > > <rewrite template="{$frontend[path]}/example/{**}"/> > > </rule> > > > > Still doesn't match anything. I think I tried with and without .css in > > the 'template' attribute (although I'd never expect that to influence > > if there is a match or or not in the first place.) > > > > ---------- > > > > <rule dir="OUT" name="EXAMPLEUI/exampleui/outbound/cssrule" > > pattern="{**.css}"> > > <rewrite template="{$frontend[path]}/example/{**.css}"/> > > </rule> > > > > Matches too much again, including things that don't end .css: > > > > 2018-01-19 19:07:05,853 DEBUG hadoop.gateway (UrlRewriteProcessor.java: > rewrite(164)) > > - Rewrote URL: vendor.bundle.js, direction: OUT via implicit rule: > > EXAMPLEUI/exampleui/outbound/cssrule to URL: /gateway/sandbox/example/ > > vendor.xxxxx.bundle.js > > 2018-01-19 19:07:05,853 DEBUG hadoop.gateway (UrlRewriteProcessor.java: > rewrite(164)) > > - Rewrote URL: text/javascript, direction: OUT via implicit rule: > > EXAMPLEUI/exampleui/outbound/cssrule to URL: > > /gateway/sandbox/example/text/javascript > > > > ---------- > > > > <rule dir="OUT" name="EXAMPLEUI/exampleui/outbound/cssrule" > > pattern="{filename=*}.css"> > > <rewrite template="{$frontend[path]}/example/{filename=*}.css"/> > > </rule> > > > > Doesn't match anything > > > > ---------- > > > > At https://cwiki.apache.org/confluence/display/KNOX/2017/ > > 08/14/Understanding+Rewrite+Rules+for+Apache+Knox# > > UnderstandingRewriteRulesforApacheKnox-URLTemplates there is this > > information: > > > > {path} => {path=*} > > {path=*} // Match single path level. (ie wildcard) > > {path=**} // Match multiple path levels. (ie glob) > > {path=*.ext} // Match single level with simplified regex pattern. > > > > Do I paraphrase correctly: > > > > There are the following match types: > > *, **, *.ext > > > > They are neither regular expressions nor globs, and each match type > > has to match the full contents of that part of the URL, after > > splitting by / (e.g., {part=*.ext} is ok, but {part=*}.ext is not OK. > > > > The matched content is placed into a buffer named after variable name > > before the =, and can be referred to in the rewrite template attribute by > > that name, and also by that name followed by = and then anything. > > <rewrite template="{$frontend[path]}/example/{filename=*.css}"/> > > <rewrite template="{$frontend[path]}/example/{filename}"/> > > <rewrite template="{$frontend[path]}/example/{filename=*.different} > "/> > > > > If no name is used, in the case of {*} and {**}, then those same exact > > strings can be used in the template attribute and are expanded to the > > matching characters. > > > > --------------------- > > > > The above are questions regarding how the rewrite syntax works, but I > > also have some questions about filters and routes. > > > > It appears this implicit firing of rules for text/html stops when a > > specific rewrite apply/to is added *even for a different route > > path*. e.g., this rewrites the html (in a URL matching > > example/index.html): > > > > <routes> > > <route path="/example/**/*.js"> > > </route> > > <route path="/example/**"> > > </route> > > </routes> > > > > but this STOPs writing the html, even though the <rewrite> is inside a > > route with path which is unrelated: > > > > <routes> > > <route path="/example/**/*.js"> > > <rewrite apply="EXAMPLEUI/exampleui/outbound/jsfilter" > > to="response.body"/> > > </route> > > <route path="/example/**"> > > </route> > > </routes> > > > > --------------------- > > > > (The following diagrams need a fixed-width font) > > > > ...This matches the URL (after gateway/sandbox) > > . - unsure what syntax is used > > . > > . > > <route path="/example/**/*.js"> > > <rewrite apply="EXAMPLEUI/exampleui/outbound/jsfilter" > > to="response.body"/> > > </route> ^ > > ^ > > . > > . > > . > > . > > . What things can > go > > here? > > . > > . > > This is the name of a filter (not a rule) > > > > > > > > ...This has to be globally unique, > > . since all filters are combined together > > . > > . > > <filter name="EXAMPLEUI/exampleui/outbound/jsfilter"> > > > > .....This both decides which set of 'apply' > is > > . applied, and also which parsing engine > > is used > > . > > . > > <content type="application/javascript"> > > <apply path=".*\.js" rule="EXAMPLEUI/exampleui/outbound/api"/> > > </content> . . > > </filter> . . > > . This is the name of a rule > > . defined elsewhere > > . > > . > > This matches a string in the content that the parser has > > tokenized out (e.g., the URL in the content, such as href > > attribute) it's a regular expression. Unsure how this > > relates to the pattern attribute in the rule which is > > referenced > > > > > > --------------------- > > > > Many thanks for help with our understanding. We will update > > https://cwiki.apache.org/confluence/display/KNOX/2017/ > > 08/14/Understanding+Rewrite+Rules+for+Apache+Knox > > to fold in any clarifications from the mailing list (although, that > > looks like a 'blog' entry, so maybe we should copy and update a > > non-dated location? - probably https://knox.apache.org/books/ > > knox-0-14-0/dev-guide.html#Rewrite+Provider ) > > > > Regards, > > > > Nick > > > > ________________________________________ > > From: larry mccay [[email protected]] > > Sent: 17 January 2018 6:00 PM > > To: [email protected] > > Subject: Re: Problems getting started with URL rewriting > > > > Hi Stuart - > > > > At first glance, it seems to look okay. > > > > I am wondering whether you are just running into the difficult dance of > > what you need to do to get the service definitions changes updated. > > > > I believe that you will find that you need to do the following when you > > iterate over changes in rewrite rules: > > > > 1. Change rewrite rule and save > > 2. restart server > > 3. touch the topology that hosts the service that you changed the rewrite > > for > > > > #3 is the one that always burns me. > > Keep in mind that webapps are generated by topologies and regenerated > > whenever the topology is changed. > > > > Even if the server is aware of the changes the previous webapp still is > > deployed until the topology is changed. > > > > HTH, > > > > --larry > > > > > > On Wed, Jan 17, 2018 at 12:25 PM, Hickinbottom, Stuart < > > [email protected]> wrote: > > > > > Using Knox 0.11 and reproduced with Knox 0.14, we're having trouble > > > getting basic URL rewriting to work. The real problem is with an > > > ExpressJS application, and the URL is part of some JavaScript, but I > > > wanted to get this basic example working first. Please may I ask for > > > help to spot what I'm doing wrong? The HTML that goes to the browser > > > is entirely unchanged from the HTML served by the back end > > > application, while I'd expected the href URLs to be changed on the way > > > out. > > > > > > How much parsing does the rules/rewrite filters attempt? How might a > > > Webpack encoding affect this? (Although this isn't relevant to this > > > simple HTML example.) > > > > > > Is there a way to turn up logging (already have > > > log4j.logger.org.apache.hadoop.gateway=DEBUG) to see what rules and > > > filters are being applied (or not)? > > > > > > example.html > > > > > > <html> > > > <body> > > > <a href="/api/v1/user">hello</a> > > > <a href="/api/v1">hello</a> > > > </body> > > > </html> > > > > > > service.xml > > > > > > <service role="EXAMPLEUI" name="exampleui" version="0.0.1"> > > > <policies> > > > <policy role="webappsec"/> > > > <policy role="authentication" name="Anonymous"/> > > > <policy role="rewrite"/> > > > <policy role="authorization"/> > > > </policies> > > > <routes> > > > <route path="/exampleui"> > > > <rewrite apply="EXAMPLEUI/exampleui/inbound/root" > > > to="request.url"/> > > > <rewrite apply="EXAMPLEUI/exampleui/outbound/html/filter" > > > to="response.body"/> > > > </route> > > > <route path="/exampleui/**"> > > > <rewrite apply="EXAMPLEUI/exampleui/inbound/path" > > > to="request.url"/> > > > <rewrite apply="EXAMPLEUI/exampleui/outbound/html/filter" > > > to="response.body"/> > > > </route> > > > </routes> > > > <dispatch classname="org.apache.hadoop.gateway.dispatch. > > > PassAllHeadersDispatch"/> > > > </service> > > > > > > rewrite.xml > > > > > > <rules> > > > <rule dir="IN" name="EXAMPLEUI/exampleui/inbound/root" > > > pattern="*://*:*/**/exampleui/"> > > > <rewrite template="{$serviceUrl[EXAMPLEUI]}/"/> > > > </rule> > > > <rule dir="IN" name="EXAMPLEUI/exampleui/inbound/path" > > > pattern="*://*:*/**/exampleui/{**}"> > > > <rewrite template="{$serviceUrl[EXAMPLEUI]}/{**}"/> > > > </rule> > > > > > > <rule dir="OUT" name="EXAMPLEUI/exampleui/outbound/apipath"> > > > <rewrite template="{$frontend[path]}/exampleui/api/v1"/> > > > </rule> > > > <filter name="EXAMPLEUI/exampleui/outbound/html/filter"> > > > <content type="text/html"> > > > <apply path="/api/v1" rule="EXAMPLEUI/exampleui/ > > > outbound/apipath"/> > > > </content> > > > </filter> > > > > > > </rules> > > > > > > > > > > > > ==> /var/log/hadoop/knox/gateway.log <== > > > 2018-01-17 16:54:46,934 DEBUG hadoop.gateway (DefaultDispatch.java: > > executeOutboundRequest(133)) > > > - Dispatch response status: 200 > > > 2018-01-17 16:54:46,935 DEBUG hadoop.gateway (DefaultDispatch.java: > > > getInboundResponseContentType(202)) - Using explicit character set > UTF-8 > > > for entity of type text/html > > > 2018-01-17 16:54:46,935 DEBUG hadoop.gateway (DefaultDispatch.java: > > > getInboundResponseContentType(210)) - Inbound response entity content > > > type: text/html; charset=UTF-8 > > > > > > ==> /var/log/hadoop/knox/gateway-audit.log <== > > > 18/01/17 16:54:46 ||47193300-cc6c-48e6-aad0- > > e2b04f2db613|audit|EXAMPLEUI|| > > > ||dispatch|uri|http://server.project:4201/example.html| > success|Response > > > status: 200 > > > > > > ==> /var/log/hadoop/knox/gateway.log <== > > > > > > ==> /var/log/hadoop/knox/gateway-audit.log <== > > > 18/01/17 16:54:46 ||47193300-cc6c-48e6-aad0- > > e2b04f2db613|audit|EXAMPLEUI|| > > > ||access|uri|/gateway/project/exampleui/example.html|success|Response > > > status: 200 > > > > > > > > > > > > Many thanks for any pointers. > > > > > > We've read through: > > > > > > https://github.com/apache/knox/blob/master/gateway- > > > service-definitions/src/main/resources/services/ambariui/2. > > 2.0/rewrite.xml > > > https://cwiki.apache.org/confluence/display/KNOX/ > > Proxying+a+UI+using+Knox > > > https://cwiki.apache.org/confluence/display/KNOX/2017/ > > > 08/14/Understanding+Rewrite+Rules+for+Apache+Knox# > > > UnderstandingRewriteRulesforApacheKnox-JavaScriptURLRewriteFilter > > > > > > Regards, > > > Stuart > > > > > > > > >
