Hi Sandeep, Back to fighting with this. Through some nginx debugging on my backend I've come to the conclusion that Knox isn't sending the query parameters on to the backend, regardless of what rewrite rules I specify.
I.e. I send /gateway/pnda/pndaconsole/socket.io/?EIO=3&transport= polling&t=MDGlYhd <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd> Knox sends /socket.io/ <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd> This yields a 400 error, and Knox drops the websocket to the backend. What was confusing me was the 101 in the browser, but I see now that Knox is terminating the connection with the browser on one side and opening another connection with the backend on the other. I see this https://git-wip-us.apache.org/repos/asf?p=knox.git;h=98a08fc but I'm wondering if this code needs further work to support query parameters? What do you think? Thanks in advance! /ailuropod4 On Thu, May 17, 2018 at 8:57 AM, T Smith <[email protected]> wrote: > Hi Sandeep, > > Looking at my nginx server, it never sees the transport=websocket ws:// > protocol request, but it does see a http request for / at the corresponding > time, so I think perhaps the root problem here is in that rewrite, as you > mention. By the way, the reason for ws:// and not wss:// is I switched off > TLS to remove another potential source of issue, so it's all plain http. I > saw the same behaviour over https. > > My services, now, look like this - > > <route path="/pndaconsole/socket.io"> > <rewrite apply="PNDACONSOLE/pndaconsole/inbound/socket" > to="request.url"/> > </route> > > <route path="/pndaconsole/metrics"> > <rewrite apply="PNDACONSOLE/pndaconsole/inbound/metrics" > to="request.url"/> > </route> > > <route path="/pndaconsole/"> > <rewrite apply="PNDACONSOLE/pndaconsole/inbound/root" > to="request.url"/> > </route> > > <route path="/pndaconsole/**"> > <rewrite apply="PNDACONSOLE/pndaconsole/inbound/path" > to="request.url"/> > <rewrite apply="PNDACONSOLE/pndaconsole/outbound/app" > to="response.body"/> > </route> > > My inbound rewrites, now, look like this - > > <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/root" > pattern="*://*:*/**/pndaconsole/"> > <rewrite template="{$serviceUrl[PNDACONSOLE]}/"/> > </rule> > > <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket" > pattern="*://*:*/**/pndaconsole/socket.io/?{**} > <http://socket.io/?%7B**%7D>"> > <rewrite template="{$serviceUrl[PNDACONSOLE]}/socket.io/?{**} > <http://socket.io/?%7B**%7D>"/> > </rule> > > <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/path" > pattern="*://*:*/**/pndaconsole/{**}"> > <rewrite template="{$serviceUrl[PNDACONSOLE]}/{**}"/> > </rule> > > <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/metrics" > pattern="*://*:*/**/pndaconsole/metrics/?{**}"> > <rewrite template="{$serviceUrl[PNDACONSOLE]}/metrics/?{**}"/> > </rule> > > The ws://...transport=websocket requests are rewritten as /, so I tried > adding an entry in my topology for WEBSOCKET and an additional inbound rule > like this - > > <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket" > pattern="ws://*:*/**/pndaconsole/socket.io/?{**} > <http://socket.io/?%7B**%7D>"> > <rewrite template="{$serviceUrl[WEBSSOCKET]}/socket.io/?{**} > <http://socket.io/?%7B**%7D>"/> > </rule> > > Where the topology entry is the same as PNDACONSOLE except it uses the > ws:// protocol specifier. This didn't work at all, and in fact at that > point, nothing got through and everything fell through to the default > rewrite rule. > > I've attached the gateway log with debug turned on everywhere (tarballed > as it's huge). > > I guess what I'm missing is a clear idea of what I'm supposed to do in the > service/rewrite rules to deal with the websockets request which will arrive > with a different protocol specifier and a different query parameter. Should > I explicitly map those through to the back end, or is this something Knox > handles internally? > > Any insight into this appreciated, it seems very close to working, and in > fact socket.io does it's usual fallback so it is functional after a > fashion but it seems like this should fully work given what I see with > Zeppelin. > > > Cheers, > /ailuropod4 > > > On Sun, May 13, 2018 at 3:22 PM, Sandeep Moré <[email protected]> > wrote: > >> Hello ailuropod4 >> >> You should not have to reroute to a different host port, this should >> work. >> >> Looking at the responses, it looks like protocol switching happens at the >> Knox end, between browser and Knox (going by the url ws:// >> 34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/? >> EIO=3&transport=websocket&sid=n_JBdgXCw_sZL-Q5AAOW) but it should have >> been wss:// wonder why this is ws://. >> >> Also, can you include seperate "ws" rules for your service, similar to >> Zeppelin. >> Also, make sure you have, inbound routes in service.xml for e.g. >> >> <routes> >> <route path="/zeppelin/ws"> >> <rewrite apply="ZEPPELINWS/zeppelin/ws/inbound" to="request.url"/> >> </route> >> >> <route path="/zeppelin/ws**"> >> <rewrite apply="ZEPPELINWS/zeppelin/inbound" to="request.url"/> >> </route> >> </routes> >> >> and then reference it in rewrite.xml. >> >> Looking at the error, it looks like the issue is not with rewrite rules >> but with the connection between Knox and the backend. >> We can see that Knox is initiating an upgrade request but then for some >> reason the backend service is closing/refusing it. >> Can you turn up the logging in Knox to Debug and share it if you can >> (this should atleast tell us whether rewrite rules are working). >> >> Also, do you see any errors at the backend service ? it woud be good to >> know what the backend is seeing and reasons for it to refuse the upgrade >> and also the error code >> Unfortunately the error code is gobbled up somewhere in the process. >> >> Best, >> Sandeep >> >> On Sat, May 12, 2018 at 3:16 AM, T Smith <[email protected]> wrote: >> >>> And if it helps, the stack trace as a result of this from Knox follows. >>> >>> Can I match a url containing &transport=websocket in one of the query >>> parameters and map it through to a service defined as ws:// in the >>> topology? At the moment, as above, everything is sent to a http:// >>> service, maybe this is the crux of the problem? >>> >>> 2018-05-12 07:09:47,361 ERROR gateway.websockets >>> (ProxyWebSocketAdapter.java:cleanupOnError(171)) - Error: >>> org.eclipse.jetty.websocket.api.UpgradeException: Didn't switch >>> protocols >>> 2018-05-12 07:09:47,362 ERROR gateway.websockets >>> (ProxyWebSocketAdapter.java:onWebSocketConnect(105)) - Unable to >>> connect to websocket server: java.io.IOException: Connect failure >>> java.io.IOException: Connect failure >>> at org.eclipse.jetty.websocket.jsr356.ClientContainer.connect(C >>> lientContainer.java:157) >>> at org.eclipse.jetty.websocket.jsr356.ClientContainer.connectTo >>> Server(ClientContainer.java:180) >>> at org.apache.knox.gateway.websockets.ProxyWebSocketAdapter.onW >>> ebSocketConnect(ProxyWebSocketAdapter.java:97) >>> at org.eclipse.jetty.websocket.common.events.JettyListenerEvent >>> Driver.onConnect(JettyListenerEventDriver.java:87) >>> at org.eclipse.jetty.websocket.common.events.AbstractEventDrive >>> r.openSession(AbstractEventDriver.java:227) >>> at org.eclipse.jetty.websocket.common.WebSocketSession.open(Web >>> SocketSession.java:421) >>> at org.eclipse.jetty.websocket.server.WebSocketServerConnection >>> .onOpen(WebSocketServerConnection.java:72) >>> at org.eclipse.jetty.io.AbstractEndPoint.upgrade(AbstractEndPoi >>> nt.java:185) >>> at org.eclipse.jetty.server.HttpConnection.completed(HttpConnec >>> tion.java:345) >>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java >>> :436) >>> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConne >>> ction.java:257) >>> at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnec >>> tion.java:544) >>> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queued >>> ThreadPool.java:635) >>> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedT >>> hreadPool.java:555) >>> at java.lang.Thread.run(Thread.java:748) >>> Caused by: org.eclipse.jetty.websocket.api.UpgradeException: Didn't >>> switch protocols >>> at org.eclipse.jetty.websocket.client.io.UpgradeConnection.vali >>> dateResponse(UpgradeConnection.java:314) >>> at org.eclipse.jetty.websocket.client.io.UpgradeConnection.read >>> (UpgradeConnection.java:241) >>> at org.eclipse.jetty.websocket.client.io.UpgradeConnection.onFi >>> llable(UpgradeConnection.java:163) >>> ... 4 more >>> 2018-05-12 07:09:47,373 WARN websockets.ProxyWebSocketAdapter >>> (AbstractEventDriver.java:unhandled(245)) - Unhandled Error (closing >>> connection) >>> org.eclipse.jetty.io.RuntimeIOException: java.io.IOException: Connect >>> failure >>> at org.apache.knox.gateway.websockets.ProxyWebSocketAdapter.onW >>> ebSocketConnect(ProxyWebSocketAdapter.java:106) >>> at org.eclipse.jetty.websocket.common.events.JettyListenerEvent >>> Driver.onConnect(JettyListenerEventDriver.java:87) >>> at org.eclipse.jetty.websocket.common.events.AbstractEventDrive >>> r.openSession(AbstractEventDriver.java:227) >>> at org.eclipse.jetty.websocket.common.WebSocketSession.open(Web >>> SocketSession.java:421) >>> at org.eclipse.jetty.websocket.server.WebSocketServerConnection >>> .onOpen(WebSocketServerConnection.java:72) >>> at org.eclipse.jetty.io.AbstractEndPoint.upgrade(AbstractEndPoi >>> nt.java:185) >>> at org.eclipse.jetty.server.HttpConnection.completed(HttpConnec >>> tion.java:345) >>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java >>> :436) >>> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConne >>> ction.java:257) >>> at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnec >>> tion.java:544) >>> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queued >>> ThreadPool.java:635) >>> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedT >>> hreadPool.java:555) >>> at java.lang.Thread.run(Thread.java:748) >>> Caused by: java.io.IOException: Connect failure >>> at org.eclipse.jetty.websocket.jsr356.ClientContainer.connect(C >>> lientContainer.java:157) >>> at org.eclipse.jetty.websocket.jsr356.ClientContainer.connectTo >>> Server(ClientContainer.java:180) >>> at org.apache.knox.gateway.websockets.ProxyWebSocketAdapter.onW >>> ebSocketConnect(ProxyWebSocketAdapter.java:97) >>> ... 12 more >>> Caused by: org.eclipse.jetty.websocket.api.UpgradeException: Didn't >>> switch protocols >>> at org.eclipse.jetty.websocket.client.io.UpgradeConnection.vali >>> dateResponse(UpgradeConnection.java:314) >>> at org.eclipse.jetty.websocket.client.io.UpgradeConnection.read >>> (UpgradeConnection.java:241) >>> at org.eclipse.jetty.websocket.client.io.UpgradeConnection.onFi >>> llable(UpgradeConnection.java:163) >>> ... 4 more >>> >>> >>> On Fri, May 11, 2018 at 9:04 PM, T Smith <[email protected]> wrote: >>> >>>> Sorry, the other question was version. I'm using 1.0.0. >>>> >>>> On Fri, May 11, 2018 at 9:03 PM, T Smith <[email protected]> wrote: >>>> >>>>> Hi Sandeep, >>>>> >>>>> Here's what's happening - >>>>> >>>>> >>>>> 1. Request URL: >>>>> http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io >>>>> /?EIO=3&transport=polling&t=MDGlYhd >>>>> >>>>> <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=polling&t=MDGlYhd> >>>>> 2. Request Method: >>>>> GET >>>>> 3. Status Code: >>>>> 200 OK >>>>> >>>>> >>>>> 1. Request URL: >>>>> ws://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/? >>>>> EIO=3&transport=websocket&sid=n_JBdgXCw_sZL-Q5AAOW >>>>> >>>>> <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3&transport=websocket&sid=n_JBdgXCw_sZL-Q5AAOW> >>>>> 2. Request Method: >>>>> GET >>>>> 3. Status Code: >>>>> 101 Switching Protocols >>>>> >>>>> So far, so good. But, the next couple of requests fail, with 400 or >>>>> 500 range errors, and the gateway log complains about protocol errors. I >>>>> can supply these if you need them, I'm trying to be concise here to get >>>>> the >>>>> big picture across. >>>>> >>>>> Looking at tcpdump, what's happening is that ws:// request is being >>>>> nerfed to a GET /, which on my backend happens to resolve to a nice front >>>>> page of HTML. Websockets obviously panics as this isn't what it expected, >>>>> and then the whole process repeats. On my backend, /socket.io is >>>>> routed to the socket endpoint, whereas / isn't. So, my theory is that if I >>>>> can somehow convince Knox to send out the ws:// request with the correct >>>>> path including /socket.io, it might work. >>>>> >>>>> At the moment my service/rewrite sections are very simple, I pick up >>>>> requests with socket.io and send them off to the backend. >>>>> >>>>> services.xml - >>>>> >>>>> <policies> >>>>> <policy role="webappsec"/> >>>>> <policy role="authentication" name="Anonymous"/> >>>>> <policy role="rewrite"/> >>>>> <policy role="authorization"/> >>>>> </policies> >>>>> <routes> >>>>> <route path="/pndaconsole"> >>>>> </route> >>>>> <route path="/pndaconsole/**"> >>>>> <rewrite apply="PNDACONSOLE/pndaconsole/outbound/app" >>>>> to="response.body"/> >>>>> </route> >>>>> </routes> >>>>> >>>>> rewrite.xml - >>>>> >>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/root" >>>>> pattern="*://*:*/**/pndaconsole/"> >>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/"/> >>>>> </rule> >>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/socket" >>>>> pattern="*://*:*/**/pndaconsole/socket.io/?{**} >>>>> <http://socket.io/?%7B**%7D>"> >>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/socket.io/?{**} >>>>> <http://socket.io/?%7B**%7D>"/> >>>>> </rule> >>>>> <rule dir="IN" name="PNDACONSOLE/pndaconsole/inbound/path" >>>>> pattern="*://*:*/**/pndaconsole/{**}"> >>>>> <rewrite template="{$serviceUrl[PNDACONSOLE]}/{**}"/> >>>>> </rule> >>>>> (I've snipped the rest of the rules handling outbound rewrites as I >>>>> don't believe they're relevant, but let me know if you think otherwise) >>>>> >>>>> By all accounts it looks like it's *nearly* working, but I'm stumped >>>>> on where my /socket.io path is going. I was considering routing all >>>>> websockets stuff to a completely separate backend/port so that it would go >>>>> to the right place regardless of how it was rewritten, but that seems a >>>>> bit >>>>> extreme and I was hoping to use this approach. >>>>> >>>>> My end to end is - >>>>> >>>>> client, an angularjs app --> knox --> nginx --> reverse proxied >>>>> websockets backend in nodejs >>>>> >>>>> --> flat files, being an angularjs app >>>>> >>>>> We don't have to use socket.io, I guess, but it is currently used >>>>> everywhere so I'd rather avoid swapping that out to get this working if >>>>> possible. It's a fairly common library. >>>>> >>>>> Cheers, >>>>> >>>>> /ailuropod4 <[email protected]> >>>>> >>>>> >>>>> >>>>> On Fri, May 11, 2018 at 8:18 PM, Sandeep Moré <[email protected]> >>>>> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> I am not very familiar with socket.io apps, so you might have to >>>>>> fill me in about what you are trying to do. >>>>>> Looks like you are having issue rewriting Websocket url. >>>>>> >>>>>> What version of Knox are you using ? >>>>>> Knox 0.13.0 and up supports better URL rewriting, see KNOX-776 >>>>>> <https://issues.apache.org/jira/browse/KNOX-776>, it also has some >>>>>> examples in the comments. >>>>>> >>>>>> What is the ws:// url you are trying to rewrite and the rules you are >>>>>> using ? >>>>>> >>>>>> Best, >>>>>> Sandeep >>>>>> >>>>>> On Fri, May 11, 2018 at 2:54 PM, T Smith <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> I'm struggling to get a simple socket.io based application working >>>>>>> through Knox. >>>>>>> >>>>>>> I see websockets is supported, but the upgrade handshake seems to be >>>>>>> nerfed on the way out, specifically the /socket.io/? etc part >>>>>>> simply becomes /. >>>>>>> >>>>>>> I've tried lots of permutations - does anyone have a working example >>>>>>> they can point me towards? I've looked at the Zeppelin approach, but >>>>>>> this >>>>>>> isn't quite the same thing (not socket.io). >>>>>>> >>>>>>> Any help appreciated! >>>>>>> >>>>>>> /ailuropod4 <[email protected]> >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >
