Re: socket.io in knox

[Sandeep Moré]

Hello,

I am not very familiar with socket.io apps, so you might have to fill me in
about what you are trying to do.
Looks like you are having issue rewriting Websocket url.

What version of Knox are you using ?
Knox 0.13.0 and up supports better URL rewriting, see KNOX-776
, it also has some examples
in the comments.

What is the ws:// url you are trying to rewrite and the rules you are using
?

Best,
Sandeep

On Fri, May 11, 2018 at 2:54 PM, T Smith  wrote:

> Hi all,
>
> I'm struggling to get a simple socket.io based application working
> through Knox.
>
> I see websockets is supported, but the upgrade handshake seems to be
> nerfed on the way out, specifically the /socket.io/? etc part simply
> becomes /.
>
> I've tried lots of permutations - does anyone have a working example they
> can point me towards? I've looked at the Zeppelin approach, but this isn't
> quite the same thing (not socket.io).
>
> Any help appreciated!
>
> /ailuropod4 
>

Re: Knox failing to match path

[Sandeep Moré]

Hello Rajat,

Can you clear your deployments folder and restart Knox and try again,
sometimes I have seen they don't get properly updated.
Also, can you turn on DEBUG logging for Knox, and check if you can see that
the service defs are properly added, you will see them in the logs like

2017-10-19 15:44:38,734 DEBUG hadoop.gateway
(ServiceDefinitionsLoader.java:loadServiceDefinitions(66)) - Added Service
definition name: ambariui, role : AMBARIUI, version : 2.2.0
2017-10-19 15:44:38,736 DEBUG hadoop.gateway
(ServiceDefinitionsLoader.java:loadServiceDefinitions(66)) - Added Service
definition name: avatica, role : AVATICA, version : 1.9.0
2017-10-19 15:44:38,739 DEBUG hadoop.gateway
(ServiceDefinitionsLoader.java:loadServiceDefinitions(66)) - Added Service
definition name: druid-broker, role : DRUID-BROKER, version : 0.0.1
2017-10-19 15:44:38,741 DEBUG hadoop.gateway
(ServiceDefinitionsLoader.java:loadServiceDefinitions(66)) - Added Service
definition name: druid-coordinator, role : DRUID-COORDINATOR, version :
0.0.1


On Tue, May 22, 2018 at 10:04 AM, Rajat Goel  wrote:

> Hi,
>
> I am new to Knox. I am trying add a simple application service to my
> default Knox topology. Here’s what my service definition looks like:
>
> 
>
> 
>
> 
>
> 
>
> 
>
> 
>
> 
>
> 
>
> 
>
>
> Rewrite.xml has rules:
>
> 
>
>  "*://*:*/**/app/{**}">
>
> 
>
> 
>
> 
>
>
> I have added my service definition in Knox default topology:
> 
> TESTAPP
> http://19.2.168.154.194:8080
> 
>
> Now when I try to access my service through Knox gateway on web browser,
> using url:
> https://192.168.154.194:8443/gateway/default/app/api1#/visualize?_g=
> 
> ()
>
> Its not working. Knox gateway.log says: 2018-05-22 18:59:42,056 WARN  
> hadoop.gateway
> (GatewayFilter.java:doFilter(162)) - Failed to match path /app/api1
>
> Can someone tell me whats the issue here?
>
> Thanks,
> Rajat
>

Re: accessing zeppelin via knox returns empty page

[Sandeep Moré]

Also, can you try adding forward slash to your url, i.e.
https://test-namenode.subnet1.hadoop.oraclevcn.com:8443/gateway/ui/zeppelin/

Best,
Sandeep

On Mon, Jun 11, 2018 at 9:32 PM larry mccay  wrote:

> Hi Lian -
>
> If the same response is coming back from curl going both direct and
> through Knox then it seems that Knox proxying is working properly - at
> least until that point.
>
> Have you looked at the javascript console in the browser to see whether
> there are any errors?
> In order for Zeppelin to work you will also need to proxy the websocket
> connections - so make sure that you follow the docs [1] completely.
>
> Dev tools and the javascript console should provide you with some more
> details hopefully.
>
> thanks,
>
> --larry
>
> 1. http://knox.apache.org/books/knox-1-0-0/user-guide.html#Zeppelin+UI
>
> On Mon, Jun 11, 2018 at 7:56 PM, Lian Jiang  wrote:
>
>> Hi,
>>
>> I am using HDP2.6, knox, ranger, ldap. I can access zeppelin directly
>> using admin/admin as username/password. However, accessing zepplin via knox
>> gets 200 status code but empty page.
>>
>> Using curl, I found the responses (at the end) are the same for:
>>
>> without knox:
>> curl -ik -H 'Cache-Control: no-cache' -u admin:"password"
>> http://test-namenode.subnet1.hadoop.oraclevcn.com:9995
>>
>> with knox:
>> curl -ik -H 'Cache-Control: no-cache' -u admin:"password"
>> https://test-namenode.subnet1.hadoop.oraclevcn.com:8443/gateway/ui/zeppelin
>>
>> Any idea why the browser does not show the login page? Appreciate any
>> help!
>>
>>
>>
>> Response:
>>
>> HTTP/1.1 200 OK
>> Date: Mon, 11 Jun 2018 23:51:00 GMT
>> Date: Monday, June 11, 2018 11:51:00 PM GMT
>> Access-Control-Allow-Credentials: true
>> Access-Control-Allow-Headers: authorization,Content-Type
>> Access-Control-Allow-Methods: POST, GET, OPTIONS, PUT, HEAD, DELETE
>> X-FRAME-OPTIONS: SAMEORIGIN
>> X-XSS-Protection: 1
>> Accept-Ranges: bytes
>> Content-Type: text/html
>> Last-Modified: Thu, 04 Jan 2018 10:59:02 GMT
>> Server: Jetty(9.2.15.v20160210)
>> Content-Length: 3783
>>
>>> http-equiv="X-UA-Compatible" content="IE=edge"> 
>>   > http-equiv="cache-control" content="max-age=0"> > http-equiv="cache-control" content="no-cache"> > http-equiv="cache-control" content="no-store"> > content="0">   > name="description" content=""> > content="width=device-width">  div.ace_editor.ace_autocomplete
>> .ace_marker-layer .ace_active-line { z-index: -1 !important; }  > rel="stylesheet"
>> href="/gateway/ui/zeppelin/styles/vendor.ead2f1215177b914.css"> > rel="stylesheet"
>> href="/gateway/ui/zeppelin/styles/main.5e840e18559d1996.css"> > rel="stylesheet"
>> ng-href="/gateway/ui/zeppelin/assets/styles/looknfeel/{{looknfeel}}.css">
>> > href="/gateway/ui/zeppelin/assets/styles/printMode.css">  > ng-class="{'bodyAsIframe': asIframe}">   > src="'components/navbar/navbar.html?v=1515063488639'">   > ng-view>   > src="'components/modal-shortcut/modal-shortcut.html?v=1515063488639'">
>>  > id="note-modal-container" ng-include
>> src="'components/noteName-create/note-name-dialog.html?v=1515063488639'">
>>   > id="note-import-container" ng-include
>> src="'components/noteName-import/note-import-dialog.html?v=1515063488639'">
>>   > id="login-container" ng-include
>> src="'components/login/login.html?v=1515063488639'">  > ng-controller="RenameCtrl"> > src="'components/rename/rename.html?v=1515063488639'">   var config = {
>> extensions: ["tex2jax.js"],
>> jax: ["input/TeX", "output/HTML-CSS"],
>> tex2jax: {
>>   inlineMath: [ ["\\(","\\)"] ],
>>   displayMath: [ ['$$','$$'], ["\\[","\\]"] ],
>>   processEscapes: true
>> },
>> "HTML-CSS": { availableFonts: ["TeX"] },
>> messageStyle: "none"
>>   }
>>   // add root only if it's not dev mode
>>   if (Number(location.port) !== 9000) {
>> config.root = '.';
>>   }
>>   MathJax.Hub.Config(config); > src="/gateway/ui/zeppelin/scripts/vendor.b8e51cdeeccd208d.js">
>> > src="/gateway/ui/zeppelin/app.05bbdae750681c30f521.js">
>>
>>
>>
>

Re: socket.io in knox

[Sandeep Moré]

may be you are right about not not properly handeling the query params and
we might need to fix this regex:

final static String REGEX_SPLIT_SERVICE_PATH = "^((?:[^/]*/){3}[^/]*)";

You should be able to do a quick test using a this Unit Test
<https://git-wip-us.apache.org/repos/asf?p=knox.git;a=blobdiff;f=gateway-server/src/test/java/org/apache/hadoop/gateway/websockets/WebsocketEchoTest.java;h=4b0fe085d7a97e9848d9e3ec06417099df587a41;hp=5d1a1175cae606b5ec118ed7b322a9ea3a789a99;hb=98a08fc;hpb=4c2675aab4bfb4ae3a08250d75f349e3387c1cb1>

Best,
Sandeep




On Mon, Jun 18, 2018 at 1:55 PM T Smith  wrote:

> Sorry, bad example.
>
> I send ws://:/gateway/pnda/pndaconsole/
> socket.io/?EIO=3=websocket
>
> Knox sends /socket.io/
>
> That yields a 400 error, Knox throws
> org.eclipse.jetty.websocket.api.UpgradeException: Didn't switch protocols
>
>
> On Mon, Jun 18, 2018 at 6:49 PM, T Smith  wrote:
>
>> Hi Sandeep,
>>
>> Back to fighting with this. Through some nginx debugging on my backend
>> I've come to the conclusion that Knox isn't sending the query parameters on
>> to the backend, regardless of what rewrite rules I specify.
>>
>> I.e. I send
>> /gateway/pnda/pndaconsole/socket.io/?EIO=3=polling=MDGlYhd
>> <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3=polling=MDGlYhd>
>>
>> Knox sends /socket.io/
>> <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3=polling=MDGlYhd>
>>
>> This yields a 400 error, and Knox drops the websocket to the backend.
>> What was confusing me was the 101 in the browser, but I see now that Knox
>> is terminating the connection with the browser on one side and opening
>> another connection with the backend on the other.
>>
>> I see this https://git-wip-us.apache.org/repos/asf?p=knox.git;h=98a08fc
>> but I'm wondering if this code needs further work to support query
>> parameters? What do you think?
>>
>> Thanks in advance!
>>
>> /ailuropod4
>>
>> On Thu, May 17, 2018 at 8:57 AM, T Smith  wrote:
>>
>>> Hi Sandeep,
>>>
>>> Looking at my nginx server, it never sees the transport=websocket ws://
>>> protocol request, but it does see a http request for / at the corresponding
>>> time, so I think perhaps the root problem here is in that rewrite, as you
>>> mention. By the way, the reason for ws:// and not wss:// is I switched off
>>> TLS to remove another potential source of issue, so it's all plain http. I
>>> saw the same behaviour over https.
>>>
>>> My services, now, look like this -
>>>
>>> 
>>>   >> to="request.url"/>
>>> 
>>>
>>> 
>>>   >> to="request.url"/>
>>> 
>>>
>>> 
>>>   >> to="request.url"/>
>>> 
>>>
>>> 
>>>   >> to="request.url"/>
>>>   >> to="response.body"/>
>>> 
>>>
>>> My inbound rewrites, now, look like this -
>>>
>>> >> pattern="*://*:*/**/pndaconsole/">
>>> 
>>> 
>>>
>>> >> pattern="*://*:*/**/pndaconsole/socket.io/?{**}
>>> <http://socket.io/?%7B**%7D>">
>>> http://socket.io/?%7B**%7D>"/>
>>> 
>>>
>>> >> pattern="*://*:*/**/pndaconsole/{**}">
>>> 
>>> 
>>>
>>> >> pattern="*://*:*/**/pndaconsole/metrics/?{**}">
>>> 
>>> 
>>>
>>> The ws://...transport=websocket requests are rewritten as /, so I tried
>>> adding an entry in my topology for WEBSOCKET and an additional inbound rule
>>> like this -
>>>
>>> >> pattern="ws://*:*/**/pndaconsole/socket.io/?{**}
>>> <http://socket.io/?%7B**%7D>">
>>> http://socket.io/?%7B**%7D>"/>
>>> 
>>>
>>> Where the topology entry is the same as PNDACONSOLE except it uses the
>>> ws:// protocol specifier. This didn't work at all, and in fact at that
>>> point, nothing got through and everything fell through to the default
>>> rewrite rule.
>>>
>>> I've attached the gateway log with debug turned on everywhere (tarballed
>>> as it's huge).
>>>
>>> I guess what I'm missing is a clear idea of what I'm supposed to do in
>>> the service/rewrit

Re: Exclude/skip paths in route or filter?

[Sandeep Moré]

Ah, so you would like to skip the entire filter for a specific path ?

On Fri, Jun 15, 2018 at 10:56 AM Sean Roberts 
wrote:

> Sandeep – Thanks.
>
>
>
> In this case the problem is this filter:
>
> 
>
> 
>
> 
>
>  "AMBARI/ambari/context_path/outbound"/>
>
> 
>
>
>
> Thoughts on pre-empting it for a specific path?
>
>
>
> --
>
> Sean Roberts
>
>
>
> *From: *Sandeep Moré 
> *Reply-To: *"user@knox.apache.org" 
> *Date: *Friday, 15 June 2018 at 15:53
> *To: *"user@knox.apache.org" 
> *Cc: *Andras Hegedus , Péter Vámos <
> pva...@hortonworks.com>
> *Subject: *Re: Exclude/skip paths in route or filter?
>
>
>
> Hello Sean
>
>
>
> It is a bit tricky to exclude a path ( I am assuming you want to do this
> in some specific scenario ). There are few options I can think of:
>
>
>
> Option 1. You can try to make the rules more specific so it can apply only
> to the URLs we want, the downside is, since we are narrowing the rule there
> is a high chance of other path failures.
>
>
>
> Option 2. Create an exact rule for the patch you want to exclude and make
> the rule empty or just rewrite the existing path, make sure you add the
> corresponding patch to service.xml file.
>
>
>
> Option 3. Use the flow=AND and flow=OR conditions with option 2. This is
> not ideal for excluding a path but it could be a better option.
>
>
>
> e.g.
>
> 
> 
>
> 
> 
>
>  template="test-scheme-output://test-host-output:777/test-path-output/test-home/~{path=**}?{**}"/>
> 
> 
>  template="test-scheme-output://test-host-output:42/test-path-output/{path}?{**}"/>
> 
> 
>
> Where
>
> OR = At least one condition proceeding a set of one or more actions must
> succeed for the actions to be executed.
> AND = All conditions proceeding a set of one or more actions must succeed
> for the actions to be executed.
>
>
>
>
>
>
>
> Best,
>
> Sandeep
>
>
>
> On Fri, Jun 15, 2018 at 10:08 AM Sean Roberts 
> wrote:
>
> Knox experts – How would a path be excluded from a route or filter within
> a Knox service?
>
>
>
> For example, in the ‘ambari’ service the filter below converts
> “text/plain” content to “application/json”.
>
>
>
> Many paths needs to be excluded from that conversion.
>
>
>
> service.xml
> <https://github.com/apache/knox/blob/master/gateway-service-definitions/src/main/resources/services/ambari/2.2.0/service.xml#L24-L28>
> :
>
> 
>
>  />
>
> 
>
> 
>
>
>
> rewrite.xml
> <https://github.com/apache/knox/blob/master/gateway-service-definitions/src/main/resources/services/ambari/2.2.0/rewrite.xml#L29-L33>
> :
>
> 
>
> 
>
> 
>
>  "AMBARI/ambari/context_path/outbound"/>
>
> 
>
>
>
> --
>
> Sean Roberts
>
>

Re: Exclude/skip paths in route or filter?

[Sandeep Moré]

Oh ok, how about we disable it from the service.xml by commenting out that
rule for path, something like

 




and then selectively enabling only for the path that needs this conversion.

By doing this you will also lose the rewrite rules associated with it, if
you want to keep the rules, create another filter add it to the service and
remove the asType conversion.

Best,
Sandeep

On Fri, Jun 15, 2018 at 11:10 AM Sean Roberts 
wrote:

> Sandeep – Yeah. The interpreting everything that’s text/plain as
> application/json breaks many paths.
>
>
>
> --
>
> Sean Roberts
>
>
>
> *From: *Sandeep Moré 
> *Reply-To: *"user@knox.apache.org" 
> *Date: *Friday, 15 June 2018 at 16:01
> *To: *"user@knox.apache.org" 
> *Subject: *Re: Exclude/skip paths in route or filter?
>
>
>
> Ah, so you would like to skip the entire filter for a specific path ?
>
> On Fri, Jun 15, 2018 at 10:56 AM Sean Roberts 
> wrote:
>
> Sandeep – Thanks.
>
>
>
> In this case the problem is this filter:
>
> 
>
> 
>
> 
>
>  "AMBARI/ambari/context_path/outbound"/>
>
> 
>
>
>
> Thoughts on pre-empting it for a specific path?
>
>
>
> --
>
> Sean Roberts
>
>
>
> *From: *Sandeep Moré 
> *Reply-To: *"user@knox.apache.org" 
> *Date: *Friday, 15 June 2018 at 15:53
> *To: *"user@knox.apache.org" 
> *Cc: *Andras Hegedus , Péter Vámos <
> pva...@hortonworks.com>
> *Subject: *Re: Exclude/skip paths in route or filter?
>
>
>
> Hello Sean
>
>
>
> It is a bit tricky to exclude a path ( I am assuming you want to do this
> in some specific scenario ). There are few options I can think of:
>
>
>
> Option 1. You can try to make the rules more specific so it can apply only
> to the URLs we want, the downside is, since we are narrowing the rule there
> is a high chance of other path failures.
>
>
>
> Option 2. Create an exact rule for the patch you want to exclude and make
> the rule empty or just rewrite the existing path, make sure you add the
> corresponding patch to service.xml file.
>
>
>
> Option 3. Use the flow=AND and flow=OR conditions with option 2. This is
> not ideal for excluding a path but it could be a better option.
>
>
>
> e.g.
>
> 
> 
>
> 
> 
>
>  template="test-scheme-output://test-host-output:777/test-path-output/test-home/~{path=**}?{**}"/>
> 
> 
>  template="test-scheme-output://test-host-output:42/test-path-output/{path}?{**}"/>
> 
> 
>
> Where
>
> OR = At least one condition proceeding a set of one or more actions must
> succeed for the actions to be executed.
> AND = All conditions proceeding a set of one or more actions must succeed
> for the actions to be executed.
>
>
>
>
>
>
>
> Best,
>
> Sandeep
>
>
>
> On Fri, Jun 15, 2018 at 10:08 AM Sean Roberts 
> wrote:
>
> Knox experts – How would a path be excluded from a route or filter within
> a Knox service?
>
>
>
> For example, in the ‘ambari’ service the filter below converts
> “text/plain” content to “application/json”.
>
>
>
> Many paths needs to be excluded from that conversion.
>
>
>
> service.xml
> <https://github.com/apache/knox/blob/master/gateway-service-definitions/src/main/resources/services/ambari/2.2.0/service.xml#L24-L28>
> :
>
> 
>
>  />
>
> 
>
> 
>
>
>
> rewrite.xml
> <https://github.com/apache/knox/blob/master/gateway-service-definitions/src/main/resources/services/ambari/2.2.0/rewrite.xml#L29-L33>
> :
>
> 
>
> 
>
> 
>
>  "AMBARI/ambari/context_path/outbound"/>
>
> 
>
>
>
> --
>
> Sean Roberts
>
>

Re: Exclude/skip paths in route or filter?

[Sandeep Moré]

Hello Sean

It is a bit tricky to exclude a path ( I am assuming you want to do this in
some specific scenario ). There are few options I can think of:

Option 1. You can try to make the rules more specific so it can apply only
to the URLs we want, the downside is, since we are narrowing the rule there
is a high chance of other path failures.

Option 2. Create an exact rule for the patch you want to exclude and make
the rule empty or just rewrite the existing path, make sure you add the
corresponding patch to service.xml file.

Option 3. Use the flow=AND and flow=OR conditions with option 2. This is
not ideal for excluding a path but it could be a better option.

e.g.











Where
OR = At least one condition proceeding a set of one or more actions must
succeed for the actions to be executed.
AND = All conditions proceeding a set of one or more actions must succeed
for the actions to be executed.



Best,
Sandeep

On Fri, Jun 15, 2018 at 10:08 AM Sean Roberts 
wrote:

> Knox experts – How would a path be excluded from a route or filter within
> a Knox service?
>
>
>
> For example, in the ‘ambari’ service the filter below converts
> “text/plain” content to “application/json”.
>
>
>
> Many paths needs to be excluded from that conversion.
>
>
>
> service.xml
> 
> :
>
> 
>
>  />
>
> 
>
> 
>
>
>
> rewrite.xml
> 
> :
>
> 
>
> 
>
> 
>
>  "AMBARI/ambari/context_path/outbound"/>
>
> 
>
>
>
> --
>
> Sean Roberts
>

Re: socket.io in knox

[Sandeep Moré]

Hello ailuropod4,

Excellent analysis !
you are right about (a), (b), (c) and (d).
(a) and (b) can be easily fixed, we should have caught this earlier :( we
do need to account for url query string and url fragment, I am thinking, in
the createWebSocket() method we can extract query and fragment along with
path and concatenate it to path, we then use this path to pass it down to
getMatchedBackendURL(), like you pointed out this should hopefully help
you.

It would be awesome if you could create a JIRA for this issue (if we do not
have already) and create a patch along with a UnitTest, this would be
extremely useful and should take care of (a) and (b)

I think we need to look closely at (c), can you open a JIRA for (c) as well
so we can investigate. I was hoping for the request to hit
UrlRewriteServletFilter.doFilter() before reaching websocket handler,
specifically hitting the following call

UrlRewriteRequest rewriteRequest = new UrlRewriteRequest( config, request );

>From your observation it appears that this might not be the case which is a
bug !
I might not be able to look at it closely for few more days, it would be
awesome if you could also take a look at it.

Again thanks for pointing these issues :)

Best,
Sandeep

On Tue, Jun 19, 2018 at 4:52 AM T Smith  wrote:

> Hi Sandeep,
>
> I've had a look at this and here's what I think - maybe we should take
> this to knox-developers, I'm happy to help work out a solution but I'm
> going to need some insight into the design intent around this module.
>
> a) The existing unit test is hard-coded to test a backend URL that ends in
> /ws. This corresponds to a special case in the code for Zeppelin, so the
> other paths are never tested, and hence the unit test always passes.
> b) createWebSocket passes only a path, so the query component could never
> be considered by getMatchedBackendURL
> c)  getMatchedBackendURL doesn't seem to base its logic on the rewrite
> rules at all, in the case where the backend URL doesn't end in /ws it
> appends the remainder of the path to the backend (you can demonstrate this
> by altering the test case to remove the /ws from the backend URL and
> running the existing test - the rewrite rule is {**}/channels but the
> result is simply /channels).
> d) Due to (a) this is a moot point, but the unit test doesn't check that
> the path was rewritten as expected, so it will pass regardless.
>
> I think if we address (b) to pass a concatenation of the path and query
> then it might work for cases like mine, but this doesn't address (c) which
> will affect anyone wanting to control rewrites. I guess addressing (a) and
> (d) would help in the longer term.
>
> Hope this makes sense and I haven't completely missed the point :-) It
> does explain the behaviour I'm seeing. How would you like to proceed here?
>
> Cheers,
> /ailuropod4
>
>
>
>
> On Mon, Jun 18, 2018 at 7:26 PM, Sandeep Moré 
> wrote:
>
>> may be you are right about not not properly handeling the query params
>> and we might need to fix this regex:
>>
>> final static String REGEX_SPLIT_SERVICE_PATH = "^((?:[^/]*/){3}[^/]*)";
>>
>> You should be able to do a quick test using a this Unit Test
>> <https://git-wip-us.apache.org/repos/asf?p=knox.git;a=blobdiff;f=gateway-server/src/test/java/org/apache/hadoop/gateway/websockets/WebsocketEchoTest.java;h=4b0fe085d7a97e9848d9e3ec06417099df587a41;hp=5d1a1175cae606b5ec118ed7b322a9ea3a789a99;hb=98a08fc;hpb=4c2675aab4bfb4ae3a08250d75f349e3387c1cb1>
>>
>> Best,
>> Sandeep
>>
>>
>>
>>
>> On Mon, Jun 18, 2018 at 1:55 PM T Smith  wrote:
>>
>>> Sorry, bad example.
>>>
>>> I send ws://:/gateway/pnda/pndaconsole/
>>> socket.io/?EIO=3=websocket
>>>
>>> Knox sends /socket.io/
>>>
>>> That yields a 400 error, Knox throws
>>> org.eclipse.jetty.websocket.api.UpgradeException: Didn't switch protocols
>>>
>>>
>>> On Mon, Jun 18, 2018 at 6:49 PM, T Smith  wrote:
>>>
>>>> Hi Sandeep,
>>>>
>>>> Back to fighting with this. Through some nginx debugging on my backend
>>>> I've come to the conclusion that Knox isn't sending the query parameters on
>>>> to the backend, regardless of what rewrite rules I specify.
>>>>
>>>> I.e. I send
>>>> /gateway/pnda/pndaconsole/socket.io/?EIO=3=polling=MDGlYhd
>>>> <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3=polling=MDGlYhd>
>>>>
>>>> Knox sends /socket.io/
>>>> <http://34.244.121.78:8443/gateway/pnda/pndaconsole/socket.io/?EIO=3=polling=MDGlYhd>
>>>>
>>>> This yields a 400 error, and

Re: knox OS auth does not work if /tmp has noexec

[Sandeep Moré]

Hello Lian,
What errors do you see in the PAM logs (/var/log/secure, /var/log/messages)
for this login attempt ?

Best,
Sandeep

On Tue, Jul 3, 2018 at 1:25 AM Lian Jiang  wrote:

> Thanks Larry.
>
> Setting "-Djava.io.tmpdir={other_tmp_folder} 
> -D*jna*.tmpdir={other_tmp_folder}"
> in knoxcli.sh made it throw a different error.
>
> [lianjia@prod1-namenode knox-server]$ sudo bin/knoxcli.sh user-auth-test
> --cluster ui --u guest --p "{PASSWORD}" --d
> org.apache.shiro.authc.AuthenticationException:
> org.jvnet.libpam.PAMException: pam_authenticate failed : Authentication
> failure
> pam_authenticate failed : Authentication failure
> org.apache.shiro.authc.AuthenticationException:
> org.jvnet.libpam.PAMException: pam_authenticate failed : Authentication
> failure
> at
> org.apache.hadoop.gateway.shirorealm.KnoxPamRealm.handleAuthFailure(KnoxPamRealm.java:157)
> at
> org.apache.hadoop.gateway.shirorealm.KnoxPamRealm.doGetAuthenticationInfo(KnoxPamRealm.java:137)
> at
> org.apache.shiro.realm.AuthenticatingRealm.getAuthenticationInfo(AuthenticatingRealm.java:568)
> at
> org.apache.shiro.authc.pam.ModularRealmAuthenticator.doSingleRealmAuthentication(ModularRealmAuthenticator.java:180)
> at
> org.apache.shiro.authc.pam.ModularRealmAuthenticator.doAuthenticate(ModularRealmAuthenticator.java:267)
> at
> org.apache.shiro.authc.AbstractAuthenticator.authenticate(AbstractAuthenticator.java:198)
> at
> org.apache.shiro.mgt.AuthenticatingSecurityManager.authenticate(AuthenticatingSecurityManager.java:106)
> at
> org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:270)
> at
> org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256)
> at
> org.apache.hadoop.gateway.util.KnoxCLI$LDAPCommand.authenticateUser(KnoxCLI.java:1171)
> at
> org.apache.hadoop.gateway.util.KnoxCLI$LDAPCommand.authenticateUser(KnoxCLI.java:1206)
> at
> org.apache.hadoop.gateway.util.KnoxCLI$LDAPAuthCommand.execute(KnoxCLI.java:1502)
> at org.apache.hadoop.gateway.util.KnoxCLI.run(KnoxCLI.java:143)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.gateway.util.KnoxCLI.main(KnoxCLI.java:1777)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.gateway.launcher.Invoker.invokeMainMethod(Invoker.java:70)
> at org.apache.hadoop.gateway.launcher.Invoker.invoke(Invoker.java:39)
> at org.apache.hadoop.gateway.launcher.Command.run(Command.java:99)
> at org.apache.hadoop.gateway.launcher.Launcher.run(Launcher.java:69)
> at org.apache.hadoop.gateway.launcher.Launcher.main(Launcher.java:46)
> Caused by: org.jvnet.libpam.PAMException: pam_authenticate failed :
> Authentication failure
> at org.jvnet.libpam.PAM.check(PAM.java:106)
> at org.jvnet.libpam.PAM.authenticate(PAM.java:124)
> at
> org.apache.hadoop.gateway.shirorealm.KnoxPamRealm.doGetAuthenticationInfo(KnoxPamRealm.java:135)
> ... 22 more
> ERR: Unable to authenticate user: guest
>
>
>
>
> Looks like the /tmp error is gone. However, I found no clue about 
> "Authentication
> failure" even pamtest works:
>
> [lianjia@prod1-namenode knox-server]$ sudo pamtester -v login guest
> authenticate
> pamtester: invoking pam_start(login, guest, ...)
> pamtester: performing operation - authenticate
> Password:
> pamtester: successfully authenticated
>
> Not sure how to go deeper.  Still investigating. Any hint is highly
> appreciated.
>
> On Mon, Jul 2, 2018 at 12:32 PM, larry mccay 
> wrote:
>
>> Hi Lian -
>>
>> I haven't encountered this before. You will likely need to dig into the
>> shiro PAM  support itself if not even lower into the Pam module code.
>>
>> I will try and find some time to dig a bit myself.
>>
>> Thanks,
>>
>> -larry
>>
>> On Mon, Jul 2, 2018, 2:58 PM Lian Jiang  wrote:
>>
>>> Hi,
>>>
>>> When /tmp has noexec, Knox OS auth throws error:
>>>
>>> [lianjia@prod1-namenode knox-server]$ sudo bin/knoxcli.sh
>>> user-auth-test --cluster ui --u guest --p "{PASSWORD}" --d
>>> org.apache.shiro.authc.AuthenticationException: Authentication failed
>>> for token submission [org.apache.shiro.authc.UsernamePasswordToken - guest,
>>> rememberMe=false].  Possible unexpected error? (Typical or expected login
>>> exceptions should extend from AuthenticationException).
>>> /tmp/jna-3506402/jna4211705767471308463.tmp:
>>> /tmp/jna-3506402/jna4211705767471308463.tmp: failed to map segment from
>>> shared object: Operation not permitted
>>> org.apache.shiro.authc.AuthenticationException: Authentication failed
>>> for token submission [org.apache.shiro.authc.UsernamePasswordToken - guest,
>>> rememberMe=false].  Possible unexpected error? (Typical or 

Re: Memory leak in Knox gateway process

[Sandeep Moré]

Hello Vaibhav,

Did you try setting memory options for Knox ?
i.e. setting APP_MEM_OPTS in gateway.sh.

Sometimes, in cases of large memory GC does not run as frequently as it
should.

Best
Sandeep

On Tue, Feb 6, 2018 at 6:49 AM, Vaibhav Kulkarni  wrote:

> Hi,
>
>
> I am using Knox server gateway (version 0.12.0) with Hortonworks
> HDP-2.6.2.0 .  There seems to be a memory leak with the knox process
> because of which it keeps acquiring more and more memory. My test workload
> does very frequent webhdfs rest apis from hundreds of connections. When the
> workload is executing, I notice that the knox gateway process memory keeps
> increasing.
>
>
>
>   PIDVSZSZ   RSS %CPU TIME CMD
>
> 24703 44988224 11247056 2195740 78.8 01:37:02
> /usr/jdk64/jdk1.8.0_112/bin/java 
> -Djava.library.path=/usr/hdp/current/knox-server/ext/native
> -jar /usr/hdp/current/knox-server/bin/ga
>
> teway.jar
>
> 24703 44988224 11247056 2195740 78.8 01:37:03
> /usr/jdk64/jdk1.8.0_112/bin/java 
> -Djava.library.path=/usr/hdp/current/knox-server/ext/native
> -jar /usr/hdp/current/knox-server/bin/ga
>
> teway.jar
>
> 24703 44989420 11247355 2335764 84.3 01:44:13
> /usr/jdk64/jdk1.8.0_112/bin/java 
> -Djava.library.path=/usr/hdp/current/knox-server/ext/native
> -jar /usr/hdp/current/knox-server/bin/ga
>
> teway.jar
>
> 24703 44989576 11247394 2326344 89.5 01:51:04
> /usr/jdk64/jdk1.8.0_112/bin/java 
> -Djava.library.path=/usr/hdp/current/knox-server/ext/native
> -jar /usr/hdp/current/knox-server/bin/ga
>
> teway.jar
>
> 24703 44989744 11247436 2345940 94.8 01:58:06
> /usr/jdk64/jdk1.8.0_112/bin/java 
> -Djava.library.path=/usr/hdp/current/knox-server/ext/native
> -jar /usr/hdp/current/knox-server/bin/ga
>
> teway.jar
>
> 24703 44989908 11247477 2515100 100 02:05:13 /usr/jdk64/jdk1.8.0_112/bin/java
> -Djava.library.path=/usr/hdp/current/knox-server/ext/native -jar
> /usr/hdp/current/knox-server/bin/gat
>
> eway.jar
>
> 24703 44990072 11247518 2653252 105 02:12:18 /usr/jdk64/jdk1.8.0_112/bin/java
> -Djava.library.path=/usr/hdp/current/knox-server/ext/native -jar
> /usr/hdp/current/knox-server/bin/gat
>
> eway.jar
>
> 24703 44990244 11247561 2777268 110 02:19:26 /usr/jdk64/jdk1.8.0_112/bin/java
> -Djava.library.path=/usr/hdp/current/knox-server/ext/native -jar
> /usr/hdp/current/knox-server/bin/gat
>
> eway.jar
>
> 24703 44990324 11247581 2878924 113 02:23:04 /usr/jdk64/jdk1.8.0_112/bin/java
> -Djava.library.path=/usr/hdp/current/knox-server/ext/native -jar
> /usr/hdp/current/knox-server/bin/gat
>
> eway.jar
>
> 24703 44990324 11247581 2878924 112 02:23:04 /usr/jdk64/jdk1.8.0_112/bin/java
> -Djava.library.path=/usr/hdp/current/knox-server/ext/native -jar
> /usr/hdp/current/knox-server/bin/gat
>
> eway.jar
>
> 24703 44990324 11247581 2878872 112 02:23:04 /usr/jdk64/jdk1.8.0_112/bin/java
> -Djava.library.path=/usr/hdp/current/knox-server/ext/native -jar
> /usr/hdp/current/knox-server/bin/gat
>
> eway.jar
>
> 24703 44990324 11247581 2878872 111 02:23:04 /usr/jdk64/jdk1.8.0_112/bin/java
> -Djava.library.path=/usr/hdp/current/knox-server/ext/native -jar
> /usr/hdp/current/knox-server/bin/gat
>
> eway.jar
>
> 24703 44990324 11247581 2878792 111 02:23:05 /usr/jdk64/jdk1.8.0_112/bin/java
> -Djava.library.path=/usr/hdp/current/knox-server/ext/native -jar
> /usr/hdp/current/knox-server/bin/gat
>
> eway.jar
>
> 24703 44990324 11247581 2878792 110 02:23:05 /usr/jdk64/jdk1.8.0_112/bin/java
> -Djava.library.path=/usr/hdp/current/knox-server/ext/native -jar
> /usr/hdp/current/knox-server/bin/gat
>
> eway.jar
>
> 24703 44990324 11247581 2878748 110 02:23:05 /usr/jdk64/jdk1.8.0_112/bin/java
> -Djava.library.path=/usr/hdp/current/knox-server/ext/native -jar
> /usr/hdp/current/knox-server/bin/gat
>
> eway.jar
>
> 24703 44990324 11247581 2878748 110 02:23:05 /usr/jdk64/jdk1.8.0_112/bin/java
> -Djava.library.path=/usr/hdp/current/knox-server/ext/native -jar
> /usr/hdp/current/knox-server/bin/gat
>
> eway.jar
>
> 24703 44990324 11247581 2878660 109 02:23:05 /usr/jdk64/jdk1.8.0_112/bin/java
> -Djava.library.path=/usr/hdp/current/knox-server/ext/native -jar
> /usr/hdp/current/knox-server/bin/gat
>
> eway.jar
>
> 24703 44990324 11247581 2878660 109 02:23:05 /usr/jdk64/jdk1.8.0_112/bin/java
> -Djava.library.path=/usr/hdp/current/knox-server/ext/native -jar
> /usr/hdp/current/knox-server/bin/gat
>
> eway.jar
>
> 24703 44990324 11247581 2878580 108 02:23:05 /usr/jdk64/jdk1.8.0_112/bin/java
> -Djava.library.path=/usr/hdp/current/knox-server/ext/native -jar
> /usr/hdp/current/knox-server/bin/gat
>
> eway.jar
>
> 24703 44990324 11247581 2878580 108 02:23:05 /usr/jdk64/jdk1.8.0_112/bin/java
> -Djava.library.path=/usr/hdp/current/knox-server/ext/native -jar
> /usr/hdp/current/knox-server/bin/gat
>
> eway.jar
>
> 24703 44990324 11247581 2878500 107 02:23:05 /usr/jdk64/jdk1.8.0_112/bin/java
> -Djava.library.path=/usr/hdp/current/knox-server/ext/native -jar
> /usr/hdp/current/knox-server/bin/gat
>
> eway.jar
>
> 24703 44990324 11247581 2878500 

Re: KNOX Pac4j Azure AD Open ID

[Sandeep Moré]

Hello Nisha,
Can you share details of "mycluster" topology ? also, can you turn up the
logs to debug and share them along with the audit log that would help us to
understand the problem better.

Best,
Sandeep

On Fri, Feb 16, 2018 at 3:16 AM, Nisha Menon 
wrote:

> Hello,
>
> I have setup KNOX to connect with Azure AD using pac4j.
>
> However, after the authentication at Azure login page, it gets into an
> infinite loop and does not give back the original REST call response.
>
> *Details:*
>
> 1. I try to access the original URL eg: 
> *https://x.x.2.3:8442/gateway/mycluster/webhdfs/v1/user?op=LISTSTATUS
> *
>
> 2. It redirects to *https://**login.microsoftonline.com
> * and asks for credentials.
>
> 3. After successful login at Azure login page, it redirects to 
> *http://x.x.2.3:8442/gateway/knoxsso/api/v1/websso
> * with code, session
> and state variables passed as below:
>
> *https://x.x.2.3:8442/gateway/knoxsso/api/v1/websso?code=AQABAAIAAABHh4k***LFFm7C9cIShE7nggAA=5dzTZBYhEVDBrA*GZRNfANGb5ls_state=42f2447b-621***790eaa2d18
> *
>
> 2. Following this call, it *again *calls the *login.microsoftonline.com
> * like below:
>
> *https://login.microsoftonline.com/f82969ba-b***c1d0557a/oauth2/authorize?response_type=code_id=385***3-a4bdceaa34_uri=https%3A%2F%2Fx.x.2.3%3A8442%2Fgateway%2Fknoxsso%2Fapi%2Fv1%2Fwebsso%3Fpac4jCallback%3Dtrue%26client_name%3DOidcClient≻ope=openid+profile+email=5dzTZBYhEVDBrAInao9VHDRd33uiRp-GZRNfANGb5ls=BvCUroM7_aKFjmbLYxaxbS0Mq9SJ8If0CUpITEGB-bw
> *
>
> After this, step 1 and 2 alternate several times and finally lands up in
> "ERR_TOO_MANY_REDIRECTS"!!!
>
> This is my knoxsso.xml:
>
>
>1. 
>2.   
>3.   
>4.   webappsec
>5.   WebAppSec
>6.   true
>7.   
> xframe.options.enabledtrue
>8.   
>9.   
>10.   federation
>11.   pac4j
>12.   true
>13.   
>14. pac4j.callbackUrl
>15. 
> https://x.x.2.3:8442/gateway/knoxsso/api/v1/websso
>16.   
>17.   
>18. clientName
>19. OidcClient
>20.   
>21.   
>22. oidc.id
>23. 385c2bc*2695eaa34
>24.   
>25.   
>26. oidc.secret
>27. 
> Y30wOwM88BYvYmPp8KMyDY2W+o=
>28.   
>29.   
>30. oidc.discoveryUri
>31. 
> https://login.microsoftonline.com/f82969***1d0557a/.well-known/openid-configuration
>32.   
>33.   
>34.   
>35.   identity-assertion
>36.   Default
>37.   true
>38.   
>39.   
>40.   
>41. knoxauth
>42.   
>43.   
>44.   KNOXSSO
>45.   
>46.   knoxsso.cookie.secure.only
>47.   false
>48.   
>49.   
>50.   knoxsso.token.ttl
>51.   3
>52.   
>53.   
>54.  knoxsso.redirect.whitelist.regex
>55.  
> ^https?:\/\/(dap-e0|x\.x\.2\.3|127\.0\.0\.1|0:0:0:0:0:0:0:1|::1):[0-9].*$/value>
>56.   
>57.   
>58.   
>
> I tried using response_type "id_token", enabling nonces, knoxsso.secure to
> true, preferredJwsAlgorithm as RS256 etc. Nothing helps.
>
> gateway-audit.log when redirection error starts:
>
>
>1. 18/02/15 12:38:02 
> 

Re: socket.io in knox

[Sandeep Moré]

I was thinking about the issue (c) I think the issue here is with the way
GatewayWebsocketHandler is added in GatewayServer.createhandler() class. I
am thinking the reason why the rewrite rules are skipped is because
GatewayWebsocketHandler
is called before other handlers, we need to test the order in which this
handler is added.

Best,
Sandeep

On Wed, Jun 20, 2018 at 1:13 AM Sandeep Moré  wrote:

> Hello ailuropod4,
>
> Excellent analysis !
> you are right about (a), (b), (c) and (d).
> (a) and (b) can be easily fixed, we should have caught this earlier :( we
> do need to account for url query string and url fragment, I am thinking, in
> the createWebSocket() method we can extract query and fragment along with
> path and concatenate it to path, we then use this path to pass it down to
> getMatchedBackendURL(), like you pointed out this should hopefully help
> you.
>
> It would be awesome if you could create a JIRA for this issue (if we do
> not have already) and create a patch along with a UnitTest, this would be
> extremely useful and should take care of (a) and (b)
>
> I think we need to look closely at (c), can you open a JIRA for (c) as
> well so we can investigate. I was hoping for the request to hit
> UrlRewriteServletFilter.doFilter() before reaching websocket handler,
> specifically hitting the following call
>
> UrlRewriteRequest rewriteRequest = new UrlRewriteRequest( config, request
> );
>
> From your observation it appears that this might not be the case which is
> a bug !
> I might not be able to look at it closely for few more days, it would be
> awesome if you could also take a look at it.
>
> Again thanks for pointing these issues :)
>
> Best,
> Sandeep
>
> On Tue, Jun 19, 2018 at 4:52 AM T Smith  wrote:
>
>> Hi Sandeep,
>>
>> I've had a look at this and here's what I think - maybe we should take
>> this to knox-developers, I'm happy to help work out a solution but I'm
>> going to need some insight into the design intent around this module.
>>
>> a) The existing unit test is hard-coded to test a backend URL that ends
>> in /ws. This corresponds to a special case in the code for Zeppelin, so the
>> other paths are never tested, and hence the unit test always passes.
>> b) createWebSocket passes only a path, so the query component could
>> never be considered by getMatchedBackendURL
>> c)  getMatchedBackendURL doesn't seem to base its logic on the rewrite
>> rules at all, in the case where the backend URL doesn't end in /ws it
>> appends the remainder of the path to the backend (you can demonstrate this
>> by altering the test case to remove the /ws from the backend URL and
>> running the existing test - the rewrite rule is {**}/channels but the
>> result is simply /channels).
>> d) Due to (a) this is a moot point, but the unit test doesn't check that
>> the path was rewritten as expected, so it will pass regardless.
>>
>> I think if we address (b) to pass a concatenation of the path and query
>> then it might work for cases like mine, but this doesn't address (c) which
>> will affect anyone wanting to control rewrites. I guess addressing (a) and
>> (d) would help in the longer term.
>>
>> Hope this makes sense and I haven't completely missed the point :-) It
>> does explain the behaviour I'm seeing. How would you like to proceed here?
>>
>> Cheers,
>> /ailuropod4
>>
>>
>>
>>
>> On Mon, Jun 18, 2018 at 7:26 PM, Sandeep Moré 
>> wrote:
>>
>>> may be you are right about not not properly handeling the query params
>>> and we might need to fix this regex:
>>>
>>> final static String REGEX_SPLIT_SERVICE_PATH = "^((?:[^/]*/){3}[^/]*)";
>>>
>>> You should be able to do a quick test using a this Unit Test
>>> <https://git-wip-us.apache.org/repos/asf?p=knox.git;a=blobdiff;f=gateway-server/src/test/java/org/apache/hadoop/gateway/websockets/WebsocketEchoTest.java;h=4b0fe085d7a97e9848d9e3ec06417099df587a41;hp=5d1a1175cae606b5ec118ed7b322a9ea3a789a99;hb=98a08fc;hpb=4c2675aab4bfb4ae3a08250d75f349e3387c1cb1>
>>>
>>> Best,
>>> Sandeep
>>>
>>>
>>>
>>>
>>> On Mon, Jun 18, 2018 at 1:55 PM T Smith  wrote:
>>>
>>>> Sorry, bad example.
>>>>
>>>> I send ws://:/gateway/pnda/pndaconsole/
>>>> socket.io/?EIO=3=websocket
>>>>
>>>> Knox sends /socket.io/
>>>>
>>>> That yields a 400 error, Knox throws
>>>> org.eclipse.jetty.websocket.api.UpgradeException: Didn't switch protocols
>>>>
>>>>
&g

Re: Websockets to Ambari 2.7 through Knox 1.1

[Sandeep Moré]

Hello ailuropod4,

Tested with Ambari 2.7.1 and websockets seems to be working fine (with Knox
1.1.0). Are you using SSL for websockets by chance ? or is the websockets
host behind proxy ?
Looking at the AMBARIWS service it appears that Knox does not add any
authentication. You might want to sign-in into Ambari and then check if
websockets work, that way authentication header might be transmitted. Also,
do you see websockets connection established from browser to Knox in the
browser developer console ?

I did not find anything interesting in the logs, looks like the websocket
upgrade keeps failing.

Best,
Sandeep

On Tue, Jul 31, 2018 at 5:31 PM T Smith  wrote:

> Hi Sandeep,
>
> Here's the debug - I've cut it down to the first occurrence of stomp and
> the last relevant looking occurrence of websocket. You can see the
> exception mid-way through this - it corresponds to the wire exchange that I
> posted.
>
> It doesn't seem to be causing an obvious functional problem as it falls
> back to some kind of polling. Perhaps others are experiencing this but not
> noticing?
>
> Cheers,
> /ailuropod4
>
>
> On Tue, Jul 31, 2018 at 2:43 PM, Sandeep Moré 
> wrote:
>
>> Your topology file looks good, I don't see we do anything with
>> authentication in the websocket layer.
>> Do you get any errors on Knox side ? or in Ambari logs ?
>>
>> Best,
>> Sandeep
>>
>> On Tue, Jul 31, 2018 at 3:32 PM T Smith  wrote:
>>
>>> Hi all,
>>>
>>> I'm using Ambari 2.7 and Knox 1.1. For the websocket connection (stomp)
>>> I see Knox establish everything correctly with the browser (101) but then
>>> fail to establish a corresponding connection with Ambari. It looks like it
>>> is not adding the necessary authentication header.
>>>
>>> GET /api/stomp/v1/websocket HTTP/1.1
>>> Host: knox-update-18642-hadoop-edge:8080
>>> Upgrade: websocket
>>> Connection: Upgrade
>>> Sec-WebSocket-Key: TRtEre7kaIjOTsa2X141Cw==
>>> Sec-WebSocket-Version: 13
>>> Pragma: no-cache
>>> Cache-Control: no-cache
>>> Cookie: io=BI4GrKnjHdccXkqCAAAI
>>> Accept-Encoding: gzip, deflate, br
>>> Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
>>> Origin: https://knox.service.dc1.pnda.local:8443
>>> User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6)
>>> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36
>>>
>>>
>>> HTTP/1.1 403 Missing authentication token
>>> Date: Tue, 31 Jul 2018 19:20:48 GMT
>>> X-Frame-Options: DENY
>>> X-XSS-Protection: 1; mode=block
>>> X-Content-Type-Options: nosniff
>>> Pragma: no-cache
>>> X-Content-Type-Options: nosniff
>>> Content-Type: text/plain;charset=iso-8859-1
>>> Content-Length: 64
>>>
>>> {
>>>   "status": 403,
>>>   "message": "Missing authentication token"
>>> }
>>>
>>> My topology is pretty simple for Ambari.
>>>
>>> 
>>> 
>>> 
>>> authentication
>>> Anonymous
>>> true
>>> 
>>> 
>>> identity-assertion
>>> Default
>>> false
>>> 
>>> 
>>>
>>>  
>>> AMBARI
>>> http://knox-update-18642-hadoop-edge:8080
>>> 
>>>
>>> 
>>> AMBARIUI
>>> http://knox-update-18642-hadoop-edge:8080
>>> 
>>>
>>> 
>>> AMBARIWS
>>> ws://knox-update-18642-hadoop-edge:8080
>>> 
>>>
>>> 
>>>
>>> Did I miss something?
>>>
>>> Cheers,
>>> /ailuropod4
>>>
>>
>

Re: Knox: how to create a custom dispatch class

[Sandeep Moré]

Great, thanks David !
Will it possible for you to create a .patch file ? also, do you know if
this works for Knox 1.1.0 ?

Best,
Sandeep

On Wed, Aug 1, 2018 at 5:16 AM David Morin 
wrote:

> Hi,
>
> Just for share.
> Here after the Jira with the workaround to implement Logsearch with Knox:
> https://issues.apache.org/jira/browse/KNOX-1400
>
>
> Le mer. 1 août 2018 à 01:15, Sandeep Moré  a
> écrit :
>
>> Great !
>> If you stumbled across bugs in logsearch, file a JIRA and a patch if you
>> got it to work, that would be really helpful for the community !
>> Looking at the Knox services, I don't see it being implemented, I think
>> there were few emails on the mailing list about Grafana.
>>
>> Best,
>> Sandeep
>>
>> On Tue, Jul 31, 2018 at 6:39 PM David Morin 
>> wrote:
>>
>>> Hello,
>>>
>>> Thanks for your help.
>>> Concerning Logsearch it's quite done. I'll share my work soon. On my
>>> github at the beginning.
>>> Now, I'm going to implement grafana with Knox
>>> Does this feature has already been done ?
>>>
>>> Le jeu. 26 juil. 2018 à 21:47, David Morin 
>>> a écrit :
>>>
>>>> Yes, I tried PassAllHeadersNoEncodingDispatch but it didn't work for
>>>> me too.
>>>> In waiting a patch or a workaround, if you have a jar with an example
>>>> of Dispatch class it would be great.
>>>> Thanks
>>>>
>>>> Le jeu. 26 juil. 2018 à 20:41, Sandeep Moré  a
>>>> écrit :
>>>>
>>>>> Great, thanks !
>>>>>
>>>>> On Thu, Jul 26, 2018 at 2:33 PM Dhruv Goyal <777.dh...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Yes, I will do that and share here.
>>>>>>
>>>>>> On Fri, 27 Jul 2018 at 12:01 AM, Sandeep Moré 
>>>>>> wrote:
>>>>>>
>>>>>>> Hello Dhruv,
>>>>>>>
>>>>>>> Can you open a JIRA for this issue, let's track it and try to get it
>>>>>>> fixed !
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jul 26, 2018 at 2:26 PM Dhruv Goyal <777.dh...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> This is the similar issue we were facing when we tried implementing
>>>>>>>> grafana with knox, it is encoded twice in grafana as well, I tried 
>>>>>>>> using
>>>>>>>> “PassAllHeadersNoEncodingDispatch”
>>>>>>>> But it didnt worked for me. We will have to write a custom dispatch
>>>>>>>> class.
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> Dhruv
>>>>>>>>
>>>>>>>> On Thu, 26 Jul 2018 at 11:51 PM, Sandeep Moré <
>>>>>>>> moresand...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> That's weird, if Knox is not picking up the custom dispatch and
>>>>>>>>> picking up the XML than are you getting a ClassNotFoundException ?
>>>>>>>>>
>>>>>>>>> You can try putting the jar file under the lib directory and see
>>>>>>>>> if it works, I should work given all the other jars are found there. 
>>>>>>>>> There
>>>>>>>>> is also a "PassAllHeadersNoEncodingDispatch"  dispatch that you can 
>>>>>>>>> try to
>>>>>>>>> use, I believe 0.12.0 has it, that way you don't have to write custom
>>>>>>>>> dispatch.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Sandeep
>>>>>>>>>
>>>>>>>>> On Thu, Jul 26, 2018 at 2:11 PM David Morin <
>>>>>>>>> morin.david@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks Sandeep
>>>>>>>>>> You're right. I have to delete the directory from deployments and
>>>>>>>>>> restart knox.
>>>>>>>>>> In fact my Xml files are well taken into account.
>>>>>>>>>> But my problem is more related to the fact that I face to the
>>

Re: Knox and Zeppelin UI is not working properly.

[Sandeep Moré]

Hello Dhruv,
What version of zeppelin are you testing with ? You will need to enable
websockets or Zeppelin UI to work, make sure you have websocket turned on
in gateway-site.xml.

Can you share gateway.log failure logs as well so we know where exactly
Knox is failing.

Best,
Sandeep

On Fri, Aug 3, 2018 at 10:58 AM Dhruv Goyal <777.dh...@gmail.com> wrote:

> Hi,
>
> I am trying to enable Zeppelin with Knox, I have picked the latest changes
> done in 1.1.0, but I am still facing the issue, when I login to the Knox
> and then when I provide credentials to login to Zeppelin, it logs out (from
> zeppelin and not from knox) and ask for credentials again. It makes this
> calls which throws 405 no such method :
>
>
> https://192.168.134.214:8443/gateway/default/zeppelin/api/login;JSESSIONID=8c7acc28-22c0-4c77-8e1d-b226bbccd8e5
>
> However it does work out of the box when I try to access it without Knox,
> and else the above rest call is not observed when I try to access it
> without Knox.
>
>
> Regards
> Dhruv

Re: Knox and Zeppelin UI is not working properly.

[Sandeep Moré]

Hello Dhruv,

It appears that the error (405) is coming from Zeppelin, do you see any
errors in Zeppelin log ? can you share zeppelin log and gateway-audit.log
for the above request ?
I have never encountered 405, I tried with Knox 1.1.0 and Zeppelin Version
0.6.0.2.5.0.0-1245 and did not encounter the issue.

is your cluster Kerberized ?

Best,
Sandeep

On Fri, Aug 3, 2018 at 11:41 AM Dhruv Goyal <777.dh...@gmail.com> wrote:

> Hello Sandeep,
>
>
> 1. gateway.websocket.feature.enabled this is set to true in gateway site
> 2. Version of Zeppelin - 0.7.3
>
>
> Gateway Log - This is just a part of it I cannot find any error or issue,
> I can send the complete log if you want:
>
> 2018-08-03 14:41:18,554 DEBUG hadoop.gateway
> (DefaultDispatch.java:executeOutboundRequest(121)) - Dispatch request: GET
> http://platacc002-mst-01.gvs.ggn:9995/api/helium/visualizations/load
> 2018-08-03 14:41:18,554 DEBUG hadoop.gateway
> (UrlRewriteProcessor.java:rewrite(164)) - Rewrote URL:
> https://192.168.134.214:8443/gateway/default/zeppelin/, direction: IN via
> implicit rule: ZEPPELINUI/zeppelin/inbound/root to URL:
> http://platacc002-mst-01.gvs.ggn:9995/
> 2018-08-03 14:41:18,554 DEBUG hadoop.gateway
> (DefaultDispatch.java:executeOutboundRequest(121)) - Dispatch request: GET
> http://platacc002-mst-01.gvs.ggn:9995/app/notebook/notebook.html
> 2018-08-03 14:41:18,558 DEBUG hadoop.gateway
> (DefaultDispatch.java:executeOutboundRequest(134)) - Dispatch response
> status: 302
> 2018-08-03 14:41:18,559 DEBUG hadoop.gateway
> (UrlRewriteProcessor.java:rewrite(164)) - Rewrote URL:
> http://192.168.134.214:8443/api/login;JSESSIONID=02c6810f-00ce-448e-afa8-f87fd50652ed,
> direction: OUT via implicit rule: ZEPPELINUI/zeppelin/outbound/api to URL:
> /gateway/default/zeppelin/api/login;JSESSIONID=02c6810f-00ce-448e-afa8-f87fd50652ed
> 2018-08-03 14:41:18,559 DEBUG hadoop.gateway
> (DefaultDispatch.java:getInboundResponseContentType(209)) - Inbound
> response entity content type not provided.
> 2018-08-03 14:41:18,562 DEBUG hadoop.gateway
> (DefaultDispatch.java:executeOutboundRequest(134)) - Dispatch response
> status: 200
> 2018-08-03 14:41:18,563 DEBUG hadoop.gateway
> (DefaultDispatch.java:getInboundResponseContentType(211)) - Inbound
> response entity content type: text/html
> 2018-08-03 14:41:18,563 DEBUG hadoop.gateway
> (GatewayFilter.java:doFilter(116)) - Received request: GET
> /zeppelin/api/login
> 2018-08-03 14:41:18,564 DEBUG hadoop.gateway
> (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL:
> https://192.168.134.214:8443/gateway/default/zeppelin/api/login;JSESSIONID=02c6810f-00ce-448e-afa8-f87fd50652ed,
> direction: IN via explicit rule: ZEPPELINUI/zeppelin/inbound/api to URL:
> http://platacc002-mst-01.gvs.ggn:9995/api/login;JSESSIONID=02c6810f-00ce-448e-afa8-f87fd50652ed
> 2018-08-03 14:41:18,564 DEBUG hadoop.gateway
> (UrlRewriteProcessor.java:rewrite(164)) - Rewrote URL:
> https://192.168.134.214:8443/gateway/default/zeppelin/, direction: IN via
> implicit rule: ZEPPELINUI/zeppelin/inbound/root to URL:
> http://platacc002-mst-01.gvs.ggn:9995/
> 2018-08-03 14:41:18,565 DEBUG hadoop.gateway
> (DefaultDispatch.java:executeOutboundRequest(121)) - Dispatch request: GET
> http://platacc002-mst-01.gvs.ggn:9995/api/login;JSESSIONID=02c6810f-00ce-448e-afa8-f87fd50652ed
> 2018-08-03 14:41:18,568 DEBUG hadoop.gateway
> (DefaultDispatch.java:executeOutboundRequest(134)) - Dispatch response
> status: 405
> 2018-08-03 14:41:18,569 DEBUG hadoop.gateway
> (DefaultDispatch.java:getInboundResponseContentType(203)) - Using explicit
> character set ISO-8859-1 for entity of type text/html
> 2018-08-03 14:41:18,569 DEBUG hadoop.gateway
> (DefaultDispatch.java:getInboundResponseContentType(211)) - Inbound
> response entity content type: text/html; charset=ISO-8859-1
>
>
>
> On 03-Aug-2018, at 8:33 PM, Sandeep Moré  wrote:
>
> Hello Dhruv,
> What version of zeppelin are you testing with ? You will need to enable
> websockets or Zeppelin UI to work, make sure you have websocket turned on
> in gateway-site.xml.
>
> Can you share gateway.log failure logs as well so we know where exactly
> Knox is failing.
>
> Best,
> Sandeep
>
> On Fri, Aug 3, 2018 at 10:58 AM Dhruv Goyal <777.dh...@gmail.com> wrote:
>
>> Hi,
>>
>> I am trying to enable Zeppelin with Knox, I have picked the latest
>> changes done in 1.1.0, but I am still facing the issue, when I login to the
>> Knox and then when I provide credentials to login to Zeppelin, it logs out
>> (from zeppelin and not from knox) and ask for credentials again. It makes
>> this calls which throws 405 no such method :
>>
>>
>> https://192.168.134.214:8443/gateway/default/zeppelin/api/login;JSESSIONID=8c7acc28-22c0-4c77-8e1d-b226bbccd8e5
>>
>> However it does work out of the box when I try to access it without Knox,
>> and else the above rest call is not observed when I try to access it
>> without Knox.
>>
>>
>> Regards
>> Dhruv
>
>
>

Re: Knox and Zeppelin UI is not working properly.

[Sandeep Moré]

Nice !

On Fri, Aug 3, 2018 at 1:17 PM Dhruv Goyal <777.dh...@gmail.com> wrote:

> Hi Sandeep,
>
> I figured it was the issue with Zeppelin configuration for authentication,
> it is resolved now, thanks for your inputs.
>
> We had to use :
> /** = authcBasic
> And we were using :
> /** = authc
> In zeppelin shiro
>
>
> Thanks & Regards
> Dhruv Goyal
>
> On 03-Aug-2018, at 10:21 PM, Sandeep Moré  wrote:
>
> never encountered 405
>
>
>

Re: logsearch with knox 0.12.0: errors without the trailing /

[Sandeep Moré]

David,

You can rewrite the location header, there are a bunch of services that
rewrite the location header.

e.g.


  

  



  

  




On Sat, Jul 28, 2018 at 10:55 AM David Morin 
wrote:

> yes, this is a request going in.
> How can I make I rewrite rule that is equivalent to a redirect HTTP (302) ?
> Because the script uses location.href.lastIndexOf("/") so this is the url
> of the browser.
>
>
> Le sam. 28 juil. 2018 à 16:06, Sandeep Moré  a
> écrit :
>
>> If this is for a request going in then you can create a rewrite rule that
>> matches *logsearch and rewrite it to logsearch/.
>> If this does not work, you can try modifying the request in a custom
>> dispatch, NiFi dispatch
>> <
>> https://github.com/apache/knox/blob/92e2ec59a5940a9e7c67ec5cd29044f811dee40a/gateway-service-nifi/src/main/java/org/apache/knox/gateway/dispatch/NiFiResponseUtil.java#L62
>> >
>> does this.
>>
>> Best,
>> Sandeep
>>
>> On Sat, Jul 28, 2018 at 9:43 AM David Morin 
>> wrote:
>>
>> > -- Forwarded message -
>> > From: David Morin 
>> > Date: sam. 28 juil. 2018 à 15:34
>> > Subject: logsearch with knox 0.12.0: errors without the trailing /
>> > To: 
>> >
>> >
>> > Hello,
>> >
>> > I'm quite near to resolve my problem with the double urlencoding with
>> > logsearch but I still face to an issue that happens when users try the
>> url
>> > without a trailing /
>> > https:///gateway/default/logsearch instead of
>> > https:///gateway/default/logsearch/
>> >
>> > I've found that the Init Js file of Logsearch contains this:
>> >
>> > require.config({baseUrl: location.href.substring(0,
>> > location.href.lastIndexOf("/")+1)+"scripts",
>> >
>> > Thus, without the trailing / in the request url the logsearch context
>> > disappears.
>> >
>> > How can I handle this issue ?
>> > For example, how can I replace the String above ?
>> > Or how to force a redirect that adds the trailing / ?
>> > https:///gateway/default/*logsearch* =>
>> https:///gateway/default/
>> > *logsearch/* <https:///gateway/default/*logsearch/*>
>> >
>> > Thanks in advance
>> >
>> > David
>> >
>>
>

Re: Link between service.xml and rewrite.xml

[Sandeep Moré]

Hello David,

This is most likely because of the behavior of Knox where if it does not
find any specific rewrite rules it tries to match the best one.
Now, for simple use cases this will work. This will also work if you have a
blanked /** in your service.xml file.

Where service.xml comes in handy is, when you have to have fine grained
rules on paths, say for example different rules for
/mypath/rule1 and /mypath/rule2

It will also help you if you want to specify one what part of
request/response you want to apply specific rules i.e. response.headers,
response.body, request.headers, request.body.

Best,
Sandeep

On Mon, Jul 30, 2018 at 3:56 PM David Morin 
wrote:

> Hello,
>
> Despite a read of the Knox dev guide, I still misunderstand clearly the
> link between service and rewrite.xml.
> Sometimes, I reference  in the route of the
> service.xml file but it seems to be optional. My rules are still taken into
> account with an implicit rule according to logs.
> So, I don't know even more. when we need to reference the rules to apply
> for each route in service.xml.
>
> Thanks
> David
>

Re: [VOTE] Release Apache Knox 1.1.0 RC 3

[Sandeep Moré]

Great, thanks Larry !

On Sun, Jul 29, 2018 at 12:50 PM larry mccay  wrote:

> The VOTE for 1.1.0 rc3 passes with:
>
> 3 binding +1's
> 0 -1's
>
> I will be working on promoting RC3 to an official release shortly.
>
> Thank you for taking the time to test this release and contributing to the
> Apache Knox community!
>
>
> On Thu, Jul 26, 2018 at 12:51 PM, Sandeep Moré 
> wrote:
>
>>
>> +1
>>
>> * Downloaded and built from source
>> * Checked LICENSE and NOTICE files
>> * Verified GPG/MD5/SHA signatures for Knox source, Knox and Knoxshell
>> release packages (zip files)
>> * Installed pseudo-distributed instance (Mac OS X )
>> * Ran through knox tests
>> * Checked websocket functionality
>> * Checked Topology Port Mapping feature
>> * Checked KnoxShell samples
>> * Tested HDFSUI (recent changes)
>>
>> Best,
>> Sandeep
>>
>>
>> On Wed, Jul 25, 2018 at 7:28 PM larry mccay  wrote:
>>
>>> All -
>>>
>>> An issue with the OOTB configuration was found and subsequently fixed
>>> based
>>> on testing of RC 2. This is a minimal incremental change over the
>>> previous
>>> RC.
>>>
>>> Release candidate #3 for the Apache Knox 1.1.0 is available at:
>>>
>>> https://dist.apache.org/repos/dist/dev/knox/knox-1.1.0/
>>>
>>> The release candidate is a zip archive of the sources in:
>>>
>>> https://git-wip-us.apache.org/repos/asf/knox.git
>>> Branch v1.1.0 (git checkout -b v1.1.0)
>>> Tag is v1.1.0-rc3
>>>
>>> The KEYS file for signature validation is available at:
>>> https://dist.apache.org/repos/dist/release/knox/KEYS
>>>
>>> Please vote on releasing this package as Apache Knox 1.1.0.
>>> The vote is open for the next 72 hours and passes if a majority of at
>>> least three +1 Apache Knox PMC votes are cast.
>>>
>>> [ ] +1 Release this package as Apache Knox 1.1.0
>>> [ ] -1 Do not release this package because...
>>>
>>> Thanks,
>>>
>>> --larry
>>>
>>
>

Re: Knox: how to create a custom dispatch class

[Sandeep Moré]

Hello David,

This should have worked, if you turn the debug log on you can see what
dispatch Knox is trying to use.
Also, if the changes are in service.xml then we need to touch the topology
file so that Knox reloads it, I am thinking this could be an issue.
If it still does not work, try clearing the deployments dir and restarting
Knox.

Best,
Sandeep

On Thu, Jul 26, 2018 at 12:25 PM David Morin 
wrote:

> ​Hello,
>
> I've just read in detail the Knox dev guide. First of all, congrats !
> Great job for the doc !
> But I face to an issue with logsearch on my HDP cluster.
> This is a HDP 2.6.5 with Knox 0.12.0
> I've created some XML files. These files are in PJs.
> But I face to some 403 requests. In fact, some urls have been "urlencoded"
> twice:
>
> Rewrote URL:
>
> http://XXX:80/gateway/default/logsearch/api/v1/service/logs/histogram?page=0=9=0=
> **%3A**...
> direction: IN
> via implicit rule: LOGSEARCH/logsearch/inbound
> to URL:
> http://XXX:61888/api/v1/service/logs/histogram?q=**%253A**=0...
>
> Thus, we've got the string "q=%3A" replaced by "q=%253A"
>
> How can I resolve this issue ?
> I've written a custom dispatch class and reference it in the service.xml
> but my class seems to be ignored.
>
>  classname="org.apache.hadoop.gateway.logsearch.LogsearchDispatch"/>
>
> I've put my Jar that contains this class in the ext directory. Thus, it
> should be in the classpath.
>
> Thanks in advance
> Regards,
> David
>
>

Re: Knox: how to create a custom dispatch class

[Sandeep Moré]

That's weird, if Knox is not picking up the custom dispatch and picking up
the XML than are you getting a ClassNotFoundException ?

You can try putting the jar file under the lib directory and see if it
works, I should work given all the other jars are found there. There is
also a "PassAllHeadersNoEncodingDispatch"  dispatch that you can try to
use, I believe 0.12.0 has it, that way you don't have to write custom
dispatch.

Best,
Sandeep

On Thu, Jul 26, 2018 at 2:11 PM David Morin 
wrote:

> Thanks Sandeep
> You're right. I have to delete the directory from deployments and restart
> knox.
> In fact my Xml files are well taken into account.
> But my problem is more related to the fact that I face to the double
> urlencode and my custom dispatch class seems to be ignored.
>
>
> Le jeu. 26 juil. 2018 à 19:59, Sandeep Moré  a
> écrit :
>
>> Hello David,
>>
>> This should have worked, if you turn the debug log on you can see what
>> dispatch Knox is trying to use.
>> Also, if the changes are in service.xml then we need to touch the
>> topology file so that Knox reloads it, I am thinking this could be an
>> issue.
>> If it still does not work, try clearing the deployments dir and
>> restarting Knox.
>>
>> Best,
>> Sandeep
>>
>> On Thu, Jul 26, 2018 at 12:25 PM David Morin 
>> wrote:
>>
>>> ​Hello,
>>>
>>> I've just read in detail the Knox dev guide. First of all, congrats !
>>> Great job for the doc !
>>> But I face to an issue with logsearch on my HDP cluster.
>>> This is a HDP 2.6.5 with Knox 0.12.0
>>> I've created some XML files. These files are in PJs.
>>> But I face to some 403 requests. In fact, some urls have been
>>> "urlencoded" twice:
>>>
>>> Rewrote URL:
>>>
>>> http://XXX:80/gateway/default/logsearch/api/v1/service/logs/histogram?page=0=9=0=
>>> **%3A**...
>>> direction: IN
>>> via implicit rule: LOGSEARCH/logsearch/inbound
>>> to URL:
>>> http://XXX:61888/api/v1/service/logs/histogram?q=**%253A**
>>> =0...
>>>
>>> Thus, we've got the string "q=%3A" replaced by "q=%253A"
>>>
>>> How can I resolve this issue ?
>>> I've written a custom dispatch class and reference it in the service.xml
>>> but my class seems to be ignored.
>>>
>>> >> classname="org.apache.hadoop.gateway.logsearch.LogsearchDispatch"/>
>>>
>>> I've put my Jar that contains this class in the ext directory. Thus, it
>>> should be in the classpath.
>>>
>>> Thanks in advance
>>> Regards,
>>> David
>>>
>>>

Re: [VOTE] Release Apache Knox 1.1.0 RC 3

[Sandeep Moré]

+1

* Downloaded and built from source
* Checked LICENSE and NOTICE files
* Verified GPG/MD5/SHA signatures for Knox source, Knox and Knoxshell
release packages (zip files)
* Installed pseudo-distributed instance (Mac OS X )
* Ran through knox tests
* Checked websocket functionality
* Checked Topology Port Mapping feature
* Checked KnoxShell samples
* Tested HDFSUI (recent changes)

Best,
Sandeep


On Wed, Jul 25, 2018 at 7:28 PM larry mccay  wrote:

> All -
>
> An issue with the OOTB configuration was found and subsequently fixed based
> on testing of RC 2. This is a minimal incremental change over the previous
> RC.
>
> Release candidate #3 for the Apache Knox 1.1.0 is available at:
>
> https://dist.apache.org/repos/dist/dev/knox/knox-1.1.0/
>
> The release candidate is a zip archive of the sources in:
>
> https://git-wip-us.apache.org/repos/asf/knox.git
> Branch v1.1.0 (git checkout -b v1.1.0)
> Tag is v1.1.0-rc3
>
> The KEYS file for signature validation is available at:
> https://dist.apache.org/repos/dist/release/knox/KEYS
>
> Please vote on releasing this package as Apache Knox 1.1.0.
> The vote is open for the next 72 hours and passes if a majority of at
> least three +1 Apache Knox PMC votes are cast.
>
> [ ] +1 Release this package as Apache Knox 1.1.0
> [ ] -1 Do not release this package because...
>
> Thanks,
>
> --larry
>

Re: Complicated Javascript Rewrite

[Sandeep Moré]

Hello Billy,

You can use the variable $infix

instead of $frontend.
for e.g. {$infix[return,url,jobs]}

Best,
Sandeep


On Thu, Jul 26, 2018 at 12:10 PM Watson, Billy 
wrote:

> Hello,
>
>
>
> In trying to rewrite some javascript, there are multiple places where a
> path is used as a raw string and as a value somewhere else in the code. For
> instance,
>
>
>
> key: "searchPath",
>
> get: function() {
>
> return "jobs"
>
> }
>
>
>
> And:
>
>
>
> type: "jobs"
>
>
>
> I want to replace the first “jobs”, but not the second one. The best I
> could come up with was this:
>
>
>
>
>
> 
>
> 
>
> 
>
> 
>
>
>
> 
>
> 
>
>  rule="SERVICE/service/outbound/searchpath/jobs" />
>
>
>
>
>
> This results in:
>
>
>
> return "{$frontend[path]}/jobs"
>
>
>
> The {$frontend[path]} shows up raw instead of being replaced. If I put
> that at the beginning of the template string, it gets replaced, but not if
> I put it where I need it. How can I work around this or should I look at
> contributing some code back to knox to change this behavior?
>
>
>
> By the way, I hardcoded /gateway/{topology}/service into the rewrite
> template, which of course answers my own question. But I’m looking for a
> better workaround in case I need to use this service across topologies.
>
>
>
> Thanks,
>
>
>
> *Billy Watson*
>
> Lead Platform Engineer, Parks Data Platform
>
> Walt Disney Attractions Technology
>
> 7055 S Kirkman Rd, 225B, Orlando, FL 32819
>

Re: Websockets to Ambari 2.7 through Knox 1.1

[Sandeep Moré]

Your topology file looks good, I don't see we do anything with
authentication in the websocket layer.
Do you get any errors on Knox side ? or in Ambari logs ?

Best,
Sandeep

On Tue, Jul 31, 2018 at 3:32 PM T Smith  wrote:

> Hi all,
>
> I'm using Ambari 2.7 and Knox 1.1. For the websocket connection (stomp) I
> see Knox establish everything correctly with the browser (101) but then
> fail to establish a corresponding connection with Ambari. It looks like it
> is not adding the necessary authentication header.
>
> GET /api/stomp/v1/websocket HTTP/1.1
> Host: knox-update-18642-hadoop-edge:8080
> Upgrade: websocket
> Connection: Upgrade
> Sec-WebSocket-Key: TRtEre7kaIjOTsa2X141Cw==
> Sec-WebSocket-Version: 13
> Pragma: no-cache
> Cache-Control: no-cache
> Cookie: io=BI4GrKnjHdccXkqCAAAI
> Accept-Encoding: gzip, deflate, br
> Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
> Origin: https://knox.service.dc1.pnda.local:8443
> User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6)
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36
>
>
> HTTP/1.1 403 Missing authentication token
> Date: Tue, 31 Jul 2018 19:20:48 GMT
> X-Frame-Options: DENY
> X-XSS-Protection: 1; mode=block
> X-Content-Type-Options: nosniff
> Pragma: no-cache
> X-Content-Type-Options: nosniff
> Content-Type: text/plain;charset=iso-8859-1
> Content-Length: 64
>
> {
>   "status": 403,
>   "message": "Missing authentication token"
> }
>
> My topology is pretty simple for Ambari.
>
> 
> 
> 
> authentication
> Anonymous
> true
> 
> 
> identity-assertion
> Default
> false
> 
> 
>
>  
> AMBARI
> http://knox-update-18642-hadoop-edge:8080
> 
>
> 
> AMBARIUI
> http://knox-update-18642-hadoop-edge:8080
> 
>
> 
> AMBARIWS
> ws://knox-update-18642-hadoop-edge:8080
> 
>
> 
>
> Did I miss something?
>
> Cheers,
> /ailuropod4
>

Re: Knox: how to create a custom dispatch class

[Sandeep Moré]

Great !
If you stumbled across bugs in logsearch, file a JIRA and a patch if you
got it to work, that would be really helpful for the community !
Looking at the Knox services, I don't see it being implemented, I think
there were few emails on the mailing list about Grafana.

Best,
Sandeep

On Tue, Jul 31, 2018 at 6:39 PM David Morin 
wrote:

> Hello,
>
> Thanks for your help.
> Concerning Logsearch it's quite done. I'll share my work soon. On my
> github at the beginning.
> Now, I'm going to implement grafana with Knox
> Does this feature has already been done ?
>
> Le jeu. 26 juil. 2018 à 21:47, David Morin  a
> écrit :
>
>> Yes, I tried PassAllHeadersNoEncodingDispatch but it didn't work for me
>> too.
>> In waiting a patch or a workaround, if you have a jar with an example of
>> Dispatch class it would be great.
>> Thanks
>>
>> Le jeu. 26 juil. 2018 à 20:41, Sandeep Moré  a
>> écrit :
>>
>>> Great, thanks !
>>>
>>> On Thu, Jul 26, 2018 at 2:33 PM Dhruv Goyal <777.dh...@gmail.com> wrote:
>>>
>>>> Yes, I will do that and share here.
>>>>
>>>> On Fri, 27 Jul 2018 at 12:01 AM, Sandeep Moré 
>>>> wrote:
>>>>
>>>>> Hello Dhruv,
>>>>>
>>>>> Can you open a JIRA for this issue, let's track it and try to get it
>>>>> fixed !
>>>>>
>>>>>
>>>>> On Thu, Jul 26, 2018 at 2:26 PM Dhruv Goyal <777.dh...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> This is the similar issue we were facing when we tried implementing
>>>>>> grafana with knox, it is encoded twice in grafana as well, I tried using
>>>>>> “PassAllHeadersNoEncodingDispatch”
>>>>>> But it didnt worked for me. We will have to write a custom dispatch
>>>>>> class.
>>>>>>
>>>>>> Regards
>>>>>> Dhruv
>>>>>>
>>>>>> On Thu, 26 Jul 2018 at 11:51 PM, Sandeep Moré 
>>>>>> wrote:
>>>>>>
>>>>>>> That's weird, if Knox is not picking up the custom dispatch and
>>>>>>> picking up the XML than are you getting a ClassNotFoundException ?
>>>>>>>
>>>>>>> You can try putting the jar file under the lib directory and see if
>>>>>>> it works, I should work given all the other jars are found there. There 
>>>>>>> is
>>>>>>> also a "PassAllHeadersNoEncodingDispatch"  dispatch that you can try to
>>>>>>> use, I believe 0.12.0 has it, that way you don't have to write custom
>>>>>>> dispatch.
>>>>>>>
>>>>>>> Best,
>>>>>>> Sandeep
>>>>>>>
>>>>>>> On Thu, Jul 26, 2018 at 2:11 PM David Morin <
>>>>>>> morin.david@gmail.com> wrote:
>>>>>>>
>>>>>>>> Thanks Sandeep
>>>>>>>> You're right. I have to delete the directory from deployments and
>>>>>>>> restart knox.
>>>>>>>> In fact my Xml files are well taken into account.
>>>>>>>> But my problem is more related to the fact that I face to the
>>>>>>>> double urlencode and my custom dispatch class seems to be ignored.
>>>>>>>>
>>>>>>>>
>>>>>>>> Le jeu. 26 juil. 2018 à 19:59, Sandeep Moré 
>>>>>>>> a écrit :
>>>>>>>>
>>>>>>>>> Hello David,
>>>>>>>>>
>>>>>>>>> This should have worked, if you turn the debug log on you can see
>>>>>>>>> what dispatch Knox is trying to use.
>>>>>>>>> Also, if the changes are in service.xml then we need to touch the
>>>>>>>>> topology file so that Knox reloads it, I am thinking this could be an
>>>>>>>>> issue.
>>>>>>>>> If it still does not work, try clearing the deployments dir and
>>>>>>>>> restarting Knox.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Sandeep
>>>>>>>>>
>>>>>>>>> On Thu, Jul 26, 2018 at 12:25 PM David Morin <
>>>>>>>>> morin.david@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> I've just read in detail the Knox dev guide. First of all,
>>>>>>>>>> congrats ! Great job for the doc !
>>>>>>>>>> But I face to an issue with logsearch on my HDP cluster.
>>>>>>>>>> This is a HDP 2.6.5 with Knox 0.12.0
>>>>>>>>>> I've created some XML files. These files are in PJs.
>>>>>>>>>> But I face to some 403 requests. In fact, some urls have been
>>>>>>>>>> "urlencoded" twice:
>>>>>>>>>>
>>>>>>>>>> Rewrote URL:
>>>>>>>>>>
>>>>>>>>>> http://XXX:80/gateway/default/logsearch/api/v1/service/logs/histogram?page=0=9=0=
>>>>>>>>>> **%3A**...
>>>>>>>>>> direction: IN
>>>>>>>>>> via implicit rule: LOGSEARCH/logsearch/inbound
>>>>>>>>>> to URL:
>>>>>>>>>> http://XXX:61888/api/v1/service/logs/histogram?q=**%253A**
>>>>>>>>>> =0...
>>>>>>>>>>
>>>>>>>>>> Thus, we've got the string "q=%3A" replaced by "q=%253A"
>>>>>>>>>>
>>>>>>>>>> How can I resolve this issue ?
>>>>>>>>>> I've written a custom dispatch class and reference it in the
>>>>>>>>>> service.xml but my class seems to be ignored.
>>>>>>>>>>
>>>>>>>>>> >>>>>>>>> classname="org.apache.hadoop.gateway.logsearch.LogsearchDispatch"/>
>>>>>>>>>>
>>>>>>>>>> I've put my Jar that contains this class in the ext directory.
>>>>>>>>>> Thus, it should be in the classpath.
>>>>>>>>>>
>>>>>>>>>> Thanks in advance
>>>>>>>>>> Regards,
>>>>>>>>>> David
>>>>>>>>>>
>>>>>>>>>>

Re: Websockets to Ambari 2.7 through Knox 1.1

[Sandeep Moré]

The only difference between my test config is the authentication provider
(other than Ambari), I am using ShiroProvider and I believe you are using
anonymous.
Your default.xml looks good and Apache Knox 1.1.0 should work. i'll try to
reproduce it with Ambari 2.2 and see what I get.


authentication
ShiroProvider
true

sessionTimeout
30


main.ldapRealm
org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm


main.ldapRealm.userDnTemplate
uid={0},ou=people,dc=hadoop,dc=apache,dc=org


main.ldapRealm.contextFactory.url
ldap://{{knox_host_name}}:33389


main.ldapRealm.contextFactory.authenticationMechanism
simple


urls./**
authcBasic




identity-assertion
Default
true



authorization
AclsAuthz
true




On Wed, Aug 1, 2018 at 4:10 PM T Smith  wrote:

> Hi Sandeep,
>
> We're on 2.7.0 and 1.1.0. Just to be specific on the question about SSL,
> we're using HTTPS to Knox but HTTP between Knox and Ambari. The websockets
> host (Ambari) isn't behind a proxy. The websockets connection is
> established after login so at this point we've already logged in. The
> browser reports the websockets connection with Knox properly established
> (well, it reports 101 for the upgrade, after which you see nothing, which
> is normal in Chrome at least).
>
> It seems unlikely that moving from 2.7.0 to 2.7.1 would help as it's Knox
> that should be sending the header? What do you think?
>
> Not sure what else to try - unless it's something in the gateway-site.xml
> ? We have websockets switched on but everything else is default I believe.
> Other environmental dependencies?
>
> Do you a reference config tarball or test system somewhere with which I
> could compare? There's clearly some subtle difference in our set ups...
>
> Just btw we're installing Knox from the Apache project, not from
> Ambari/HDP.
>
> Cheers,
> /ailuropod4
>
>
>
> On Wed, Aug 1, 2018 at 8:50 AM, Sandeep Moré 
> wrote:
>
>> Hello ailuropod4,
>>
>> Tested with Ambari 2.7.1 and websockets seems to be working fine (with
>> Knox 1.1.0). Are you using SSL for websockets by chance ? or is the
>> websockets host behind proxy ?
>> Looking at the AMBARIWS service it appears that Knox does not add any
>> authentication. You might want to sign-in into Ambari and then check if
>> websockets work, that way authentication header might be transmitted. Also,
>> do you see websockets connection established from browser to Knox in the
>> browser developer console ?
>>
>> I did not find anything interesting in the logs, looks like the websocket
>> upgrade keeps failing.
>>
>> Best,
>> Sandeep
>>
>> On Tue, Jul 31, 2018 at 5:31 PM T Smith  wrote:
>>
>>> Hi Sandeep,
>>>
>>> Here's the debug - I've cut it down to the first occurrence of stomp and
>>> the last relevant looking occurrence of websocket. You can see the
>>> exception mid-way through this - it corresponds to the wire exchange that I
>>> posted.
>>>
>>> It doesn't seem to be causing an obvious functional problem as it falls
>>> back to some kind of polling. Perhaps others are experiencing this but not
>>> noticing?
>>>
>>> Cheers,
>>> /ailuropod4
>>>
>>>
>>> On Tue, Jul 31, 2018 at 2:43 PM, Sandeep Moré 
>>> wrote:
>>>
>>>> Your topology file looks good, I don't see we do anything with
>>>> authentication in the websocket layer.
>>>> Do you get any errors on Knox side ? or in Ambari logs ?
>>>>
>>>> Best,
>>>> Sandeep
>>>>
>>>> On Tue, Jul 31, 2018 at 3:32 PM T Smith  wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I'm using Ambari 2.7 and Knox 1.1. For the websocket connection
>>>>> (stomp) I see Knox establish everything correctly with the browser (101)
>>>>> but then fail to establish a corresponding connection with Ambari. It 
>>>>> looks
>>>>> like it is not adding the necessary authentication header.
>>>>>
>>>>> GET /api/stomp/v1/websocket HTTP/1.1
>>>>> Host: knox-update-18642-hadoop-edge:8080
>>>>> Upgrade: websocket
>>>>> Connection: Upgrade
>>>>> Sec-WebSocket-Key: TRtEre7kaIjOTsa2X141Cw==
>>>>> Sec-WebSocket-Version: 13
>>>>> Pragma: no-cache
>>>>> Cache-Control: no-cache
>>>>> Cookie: io=BI4GrKnjHdccXkqCAAAI
>>>>> Accept-Encoding: gzip, deflate, br
>>>>> Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
>>>>> Origin: https://knox.service.

Re: Rewrite problem

[Sandeep Moré]

Looks like you are missing the ' which is part of ng-include
So in your filter try



If this does not work try escaping the ', hopefully it should work.

Best,
Sandeep

On Wed, Aug 1, 2018 at 4:20 PM Benoit Moisan 
wrote:

>
> hello,
>
> i need some help on a rewrite rule. I have a html file with two types of
> url :
>
>
>
>- 
>- 
>
>
> I succeed to modify first one with this filter and rule but it does not
> work with the second :
>
>  pattern="app/{**}">
> 
> 
>
>  pattern="public/{**}">
> 
> 
>
> 
> 
> 
>  rule="GRAFANA/grafana/outbound/public/html"/>
> 
> 
>
>
>
> thanks in advance
>
>

Re: Possible to rewrite query parameter value?

[Sandeep Moré]

It should be possible using parameterized rewrite variables, this is an
example from Spark History rewrite rules



  
  


  

  


or in the above example



Best,
Sandeep



On Fri, Jul 27, 2018 at 1:03 PM Christopher Jackson <
jackson.christopher@gmail.com> wrote:

> Hi All,
>
> Curious if it is possible to rewrite a query parameter value to append a
> string to the original value? I tried the below configuration but I don’t
> think its valid as requests to the resource were just hanging.
>
> Entry from service.xml for the particular path:
>
>  to="response.headers"/>
>
>
> Entries in rewrite.xml:
>
> 
> 
>  template="{$frontend[url]}/myservice/myapp?originalUrl={originalurl}&{**}"/>
> 
>
> 
> 
>  rule="MYSERVICE/myapp/outbound/redirect/headers/location"/>
> 
> 
>
> An additional concern I have with the above attempted approach is I don’t
> think that I can guarantee the order of the query parameters.
>
>
>
> An example of what I would like to see happen is the header content of a
> 302 response with a Location header value of:
>
>
> https://host.example.com:9443/myapp?originalUrl=/some/path/=EI=true
>
> to be rewritten to:
>
>
> https://host.example.com:8443/gateway/default/myservice/myapp?originalUrl=/gateway/default/myservice/myapp/some/path/=EI=true
>
> In summary the result would have the frontend[url] replace the internal
> destination and the originalUrl query parameter would have the
> frontend[path] appended to its original value.
>
> Anyone have any ideas on how to achieve this?
>
> Regards,
> Christopher Jackson

Re: Possible to rewrite query parameter value?

[Sandeep Moré]

You can avoid using those functions, you can do something like this

 pattern=“{scheme}://{host}:{port}/{gateway}/{topology}/myapp?originalUrl={originalurl}&{**}"/>
>



>template="{$frontend[url]}/myservice/myapp?originalUrl=/{gateway}/{topology}/{originalurl}&{**}"/>


The prefix, postfix are a variation of frontend, they just help to add
content around URL. The limitation with these variables (postfix, prefix,
frontend) is that they have to be at the beginning of the template.

Hope this helps.
Sandeep

On Fri, Jul 27, 2018 at 3:18 PM Christopher Jackson <
jackson.christopher@gmail.com> wrote:

> Hi Sandeep,
>
> Unfortunately that function is only available in 1.10 or 0.13 or later
> which I don’t have access to. Do the $postfix, $prefix, and $infix also
> work with frontend ‘path’ instead of ‘url’?
>
> Regards,
> Christopher Jackson
>
>
> On Jul 27, 2018, at 2:16 PM, Sandeep Moré  wrote:
>
> It should be possible using parameterized rewrite variables, this is an
> example from Spark History rewrite rules
>
> 
>  name="SPARKHISTORYUI/sparkhistory/outbound/headers/location/sso">
>pattern="{scheme}://{host}:{port}/{gateway}/{knoxsso}/{api}/{v}/websso?originalUrl={**}"/>
>template="{scheme}://{host}:{port}/{gateway}/{knoxsso}/{api}/{v}/websso?originalUrl={$postfix[url,/sparkhistory/]}"/>
> 
> 
>   
>  rule="SPARKHISTORYUI/sparkhistory/outbound/headers/location/sso"/>
>   
> 
>
> or in the above example
>
> 
>
> Best,
> Sandeep
>
>
>
> On Fri, Jul 27, 2018 at 1:03 PM Christopher Jackson <
> jackson.christopher@gmail.com> wrote:
>
>> Hi All,
>>
>> Curious if it is possible to rewrite a query parameter value to append a
>> string to the original value? I tried the below configuration but I don’t
>> think its valid as requests to the resource were just hanging.
>>
>> Entry from service.xml for the particular path:
>>
>> > to="response.headers"/>
>>
>>
>> Entries in rewrite.xml:
>>
>> 
>> 
>> > template="{$frontend[url]}/myservice/myapp?originalUrl={originalurl}&{**}"/>
>> 
>>
>> 
>> 
>> > rule="MYSERVICE/myapp/outbound/redirect/headers/location"/>
>> 
>> 
>>
>> An additional concern I have with the above attempted approach is I don’t
>> think that I can guarantee the order of the query parameters.
>>
>>
>>
>> An example of what I would like to see happen is the header content of a
>> 302 response with a Location header value of:
>>
>>
>> https://host.example.com:9443/myapp?originalUrl=/some/path/=EI=true
>>
>> to be rewritten to:
>>
>>
>> https://host.example.com:8443/gateway/default/myservice/myapp?originalUrl=/gateway/default/myservice/myapp/some/path/=EI=true
>>
>> In summary the result would have the frontend[url] replace the internal
>> destination and the originalUrl query parameter would have the
>> frontend[path] appended to its original value.
>>
>> Anyone have any ideas on how to achieve this?
>>
>> Regards,
>> Christopher Jackson
>
>
>

Re: [VOTE] Release Apache Knox 1.1.0 RC 2

[Sandeep Moré]

Thanks Larry !
+1

* Downloaded and built from source (with Java 1.8.0_101)
* Checked LICENSE and NOTICE files
* Verified GPG/MD5/SHA signatures for Knox source, Knox and Knoxshell
release packages (zip files)
* Installed pseudo-distributed instance (Mac OS X )
* Ran through knox tests
* Checked websocket functionality
* Checked KnoxShell samples
* Checked the UI (read only for generated topologies)
* Tested HDFSUI

Best,
Sandeep

On Tue, Jul 24, 2018 at 8:21 PM larry mccay  wrote:

> All -
>
> A number of issues were identified and subsequently fixed based
> on testing of RC 1.
>
> Release candidate #2 for the Apache Knox 1.1.0 is available at:
>
> https://dist.apache.org/repos/dist/dev/knox/knox-1.1.0/
>
> The release candidate is a zip archive of the sources in:
>
> https://git-wip-us.apache.org/repos/asf/knox.git
> Branch v1.1.0 (git checkout -b v1.1.0)
> Tag is v1.1.0-rc2
>
> The KEYS file for signature validation is available at:
> https://dist.apache.org/repos/dist/release/knox/KEYS
>
> Please vote on releasing this package as Apache Knox 1.1.0.
> The vote is open for the next 72 hours and passes if a majority of at
> least three +1 Apache Knox PMC votes are cast.
>
> [ ] +1 Release this package as Apache Knox 1.1.0
> [ ] -1 Do not release this package because...
>
> Thanks,
>
> --larry
>

Re: Supporting Hal+JSON rewrites

[Sandeep Moré]

Sure! You can start by filing a Jira and then attach a patch to it.

Glad it worked !

Best,
Sandeep
On Wed, Jul 25, 2018 at 3:58 PM Watson, Billy 
wrote:

> I built an extension that does this and it seems to work. Should I open
> source my extension or submit a patch?
>
>
>
>
>
>
>
> *Billy Watson*
>
>
>
>
>
> *From: *Sandeep Moré 
> *Reply-To: *"user@knox.apache.org" 
> *Date: *Wednesday, July 25, 2018 at 3:54 PM
> *To: *"user@knox.apache.org" 
> *Subject: *Re: Supporting Hal+JSON rewrites
>
>
>
> Hello Billy,
>
>
>
> I don't think Knox supports application/hal+json content-type (seems to me
> that way). I don't know much about application/hal+json, but structurally
> it looks like JSON, you can try adding it [1] to the supported content
> types for JsonUrlRewriteStreamFilter.
>
>
>
> Best,
>
> Sandeep
>
>
>
> [1]
> https://github.com/apache/knox/blob/92e2ec59a5940a9e7c67ec5cd29044f811dee40a/gateway-provider-rewrite/src/main/java/org/apache/knox/gateway/filter/rewrite/impl/json/JsonUrlRewriteStreamFilter.java#L32
>
>
>
>
>
>
>
> On Wed, Jul 25, 2018 at 1:31 PM Watson, Billy 
> wrote:
>
> Hello,
>
>
>
> TL;DR
>
>1. Am I correct in saying that knox will not rewrite
>application/hal+json content-type responses?
>2. If so, can I write an extension to knox to make it do so?
>3. And if so, where do I begin?
>
>
>
> I have a GUI I’m putting behind knox and part of that application returns
> application/hal+json responses (as opposed to application/json). Knox is
> working for everything except those APIs that return the
> application/hal+json responses: the GUI, comes up, I can navigate across
> pages, etc.) but of course some data is missing.
>
>
>
> How can I get knox to rewrite the special JSON? I have tried:
>
>
>
> 
>
> 
>
> 
>
>
>
> I see that knox supports extensions and I’ve seen the code for the
> JsonUrlRewriteStreamFilter, but other than that I don’t know where to begin.
>
>
>
>
>
>
>
>
>
> Thanks,
>
>
>
> *Billy Watson*
>
>
>
>

Re: Oozie workflow view in Ambari 2.7

[Sandeep Moré]

This should not be the case, this should work, especially with Ambari 2.7,
I am assuming this is Knox 1.1.0 right ?

Best,
Sandeep

On Wed, Aug 8, 2018 at 8:39 PM T Smith  wrote:

> Hi all,
>
> It seems like the Oozie workflow manager view Job URL link points to an
> internal cluster node rather than the external knox location for YARN, e.g.
> http://somecluster-hadoop-mgr-1.node.local:8088/proxy/application_1533747863444_0017/
> instead of
> https://knox:8443/gateway/something/yarn/proxy/application_1533747863444_0017/
>
>
> This means that this view has limited usability through Knox as things
> stand.
>
> Do we know i this a Knox issue, an Ambari issue, or a user issue (i.e. was
> I supposed to configure something!) ?
>
> Cheers,
> /ailuropod4
>
>

Re: Knox and Zeppelin UI is not working properly.

[Sandeep Moré]

Praveen,

You can find the topology details for Zeppelin UI in Knox user guide
https://knox.apache.org/books/knox-1-1-0/user-guide.html#Zeppelin+UI

Best,
Sandeep


On Tue, Aug 7, 2018 at 3:03 PM Ravikumar, Praveen Krishnamoorthy <
rpkr...@amazon.com> wrote:

> Hi Guys,
>
>
>
> I’m currently working to enable Zeppelin UI for my use case. Could anyone
> please share me the Topology file, Shiro Config and the list of other steps
> to enable zeppelin UI in Knox – would be very helpful.
>
>
>
>
>
> Thanks,
>
> Praveen.
>
>
>
> *From: *Nixon Rodrigues 
> *Reply-To: *"user@knox.apache.org" 
> *Date: *Saturday, August 4, 2018 at 7:05 AM
> *To: *"user@knox.apache.org" 
> *Subject: *Re: Knox and Zeppelin UI is not working properly.
>
>
>
> Dhruv,
>
>
>
> Can you provide topology XML for review. I had also faced similar issue
> while proxing.
>
>
>
> Nixon
>
>
>
>
>
> On Fri, Aug 3, 2018, 11:28 PM Sandeep Moré  wrote:
>
> Nice !
>
>
>
> On Fri, Aug 3, 2018 at 1:17 PM Dhruv Goyal <777.dh...@gmail.com> wrote:
>
> Hi Sandeep,
>
>
>
> I figured it was the issue with Zeppelin configuration for authentication,
> it is resolved now, thanks for your inputs.
>
>
>
> We had to use :
>
> /** = authcBasic
>
> And we were using :
>
> /** = authc
>
> In zeppelin shiro
>
>
>
>
>
> Thanks & Regards
>
> Dhruv Goyal
>
>
>
> On 03-Aug-2018, at 10:21 PM, Sandeep Moré  wrote:
>
>
>
> never encountered 405
>
>
>
>

Re: Need help on rewrite rules

[Sandeep Moré]

Hello Priyanka,

Sometimes the content types do not match, can you check if the response
type you are getting back is indeed "application/javascript" ?
Also, try using a broader rule on service.xml for testing something like /
Do you see any failures in gateway.log, i.e. failure to apply rules etc.

On Thu, Aug 16, 2018 at 3:40 AM Priyanka Gupta 
wrote:

> Hi,
>
> I need some help on a rewrite rule.I have a java script which is internally 
> calling other java scripts.So I tried to modify the url of those javascript 
> using filters.
>
> Java scripts url : 
>
>
> But these requests are not changing.
>
>
> 
> 
> 
>
> 
> 
> 
>
> 
> 
>  rule="CDAPUI/cdapui/outbound/img"/>
> 
> 
>
>
> Also, I tied the rule to service.xml like:
>
> 
> 
> 
> 
> 
>
>
> Thanks in advance!
>
>
>
>
>
>
>
>

Re: Dispatcher class for Grafana

[Sandeep Moré]

Hello Divya,

Looking at the URL you posted looks like you are using the wrong dispatch
class, URLDecodingDispatch is used for URLs that are already encoded,
looking at your example URL it looks like it is partially encoded (e.g. the
query params &), in-fact looks like it is not properly encoded.

What error are you getting in gateway.log corresponding to 400 (Bad
Request) ?

Best,
Sandeep



On Thu, Aug 16, 2018 at 6:06 AM Divya Narayan 
wrote:

> Hi ,
>
> I tried to integrate grafana UI with knox gateway. I used
> URLDecodingDispatch class but there were some calls which were failing like:
>
>
> *http://platacc002-mgt-01.gvs.ggn:3000/api/datasources/proxy/1/ws/v1/timeline/metrics?metricNames=dfs.NNTopUserOpCounts.windowMs=150.op=__%.user=%=%25=namenode=1534390641=1534401441*
>
>
> The above call on grafana UI failed with below error:
>
> *Caused by: java.lang.IllegalArgumentException: Malformed escape pair at
> index
> 141: 
> http://platacc002-mgt-01.gvs.ggn:3000/api/datasources/proxy/1/ws/v1/timeline/metrics?metricNames=dfs.NNTopUserOpCounts.windowMs=150.op=__%.user=%=%25=namenode=1534390641=1534401441*
> *at java.net.URI.create(URI.java:852)*
> *at
> knoxdisp.URLDecodingDispatch.getDispatchUrl(URLDecodingDispatch.java:49)*
> *at
> org.apache.hadoop.gateway.dispatch.GatewayDispatchFilter$GetAdapter.doMethod(GatewayDispatchFilter.java:132)*
> *at
> org.apache.hadoop.gateway.dispatch.GatewayDispatchFilter.doFilter(GatewayDispatchFilter.java:115)*
> *at org.apache.ha*
> *doop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:61)*
> *... 76 more*
>
>
> Now instead of making rewrite rules for each of such URL, I decided to
> change the dispatcher class itself to handle requests with special
> characters. In dispatcher class I added URL encoding to encode all the
> special characters.
> Below is the code that I tried:
>
> @Override
>
>   public URI getDispatchUrl(final HttpServletRequest request) {
>
> String decoded;
>
> String encodedURL = null;
>
>
> try {
>
>   decoded = URLDecoder.decode(request.getRequestURL().toString(),
> "UTF-8" );
>
> } catch (final Exception e) {
>
>   /* fall back in case of exception */
>
>   decoded = request.getRequestURL().toString();
>
> }
>
>
> final StringBuffer str = new StringBuffer(decoded);
>
> final String query = request.getQueryString();
>
> if ( query != null ) {
>
>   try {
>
> encodedURL = URLEncoder.encode(query,"UTF-8");
>
> } catch (UnsupportedEncodingException e) {
>
> e.printStackTrace();
>
> }
>
>
>   str.append('?');
>
>   str.append(encodedURL);
>
>
>
> }
>
> final URI url = URI.create(str.toString());
>
> return url;
>
>   }
>
> }
>
>
>
> Now after making these changes, malformed error is resolved and all URLs
> gets encoded like :
>
>
>  *Dispatch request: GET
> http://platacc002-mgt-01.gvs.ggn:3000/api/datasources/proxy/1/ws/v1/timeline/metrics?metricNames%3Ddfs.NNTopUserOpCounts.windowMs%3D150.op%3D__%25.user%3D%25%26hostname%3D%2525%26appId%3Dnamenode%26startTime%3D1534400120%26endTime%3D1534410920
> *
>
>
>
> But the calls still doesn’t work. It throws HTTP Error code: 400 (Bad
> Request).  Can anyone help me to resolve these calls.
>
>
>
> Thanks
>
> Divya
>
>

Re: Knox Integration with CDAP UI

[Sandeep Moré]

Hello Priyanka,

I am not aware of CDAP UI support for Knox, we would be glad if you could
contribute !
About the filters they can be a bit tricky, try to turn on the debug log
and see what rewrite rules are failing that can narrow down a lot of errors.
For JS filters you can use regex to rewrite all *.js files, Knox also has
AND / OR logic in rules (called Flows) if you want any exceptions to your
rules.

You can look at this Wiki page
https://cwiki.apache.org/confluence/display/KNOX/2017/08/14/Understanding+Rewrite+Rules+for+Apache+Knox#UnderstandingRewriteRulesforApacheKnox-JavaScriptURLRewriteFilter

it should help you.

Good luck.
Sandeep

On Thu, Aug 16, 2018 at 7:24 AM Priyanka Gupta 
wrote:

> Hi,
>
> I am trying to integrate CDAP UI with Knox. CDAP UI main page internally 
> calls other urls like:
>
> https://192.168.134.119:8443/cdap_assets/img/company_logo.png
>
> https://192.168.134.119:8443/cdap_assets/fonts/icomoon.woff
>
> https://192.168.134.119:8443/_sock/info?t=1534418001305
>
> https://192.168.134.119:8443/backendstatus
>
>
> I tried working with filters and rewrite rules to change the request but that 
> seems not feasible as there are too many URL calls from within cdap.js file.
>
> How we can resolve this?Also the right approach to integrate CDAP UI with 
> Knox.
>
> Thanks,
>
> Priyanka
>
>
>
>
>
>
>
>
>
>
>
>
>

Re: Need help in Websocket request rewrite rules

[Sandeep Moré]

Hello Priyanka,

So, other than this error in the logs you do not see any other issues and
everything works, right ? just making sure I understand.

You can enable DEBUG log, which is very verbose, that should tell us more
about the error.

Best,
Sandeep



On Tue, Aug 28, 2018 at 2:41 AM Priyanka Gupta 
wrote:

> Hi,
>
> I am trying to integrate CDAP UI with Knox. CDAP UI main page send some web 
> socket request like:
>
> ws://192.168.134.119:11011/cdap/ns/default/_sock/719/ixibayd2/websocket
>
>
> Changed request through Knox:
>
> wss://192.168.134.119:8443/gateway/default/cdapui/cdap/ns/default/_sock/719/ixibayd2/websocket
>
> I have written the following rewrite.xml and service.xml for handling this:
>
> *Service.xml:*
>
> 
>   
> 
>   
> 
>   
> 
>
>
> *Rewrite.xml:*
>
> 
>
>pattern="*://*:*/{**}/websocket">
> 
>   
>
> 
>
> My web socket calls are showing status of 101 switching protocols but I can 
> see errors in Knox gateway logs :
>
> 2018-08-28 06:39:13,978 WARN  websockets.ProxyWebSocketAdapter 
> (AbstractEventDriver.java:unhandled(245)) - Unhandled Error (closing 
> connection)
> org.eclipse.jetty.io.RuntimeIOException: java.io.EOFException: Reading 
> WebSocket Upgrade response
> at 
> org.apache.hadoop.gateway.websockets.ProxyWebSocketAdapter.onWebSocketConnect(ProxyWebSocketAdapter.java:94)
> at 
> org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onConnect(JettyListenerEventDriver.java:87)
> at 
> org.eclipse.jetty.websocket.common.events.AbstractEventDriver.openSession(AbstractEventDriver.java:227)
> at 
> org.eclipse.jetty.websocket.common.WebSocketSession.open(WebSocketSession.java:421)
> at 
> org.eclipse.jetty.websocket.server.WebSocketServerConnection.onOpen(WebSocketServerConnection.java:72)
> at 
> org.eclipse.jetty.io.AbstractEndPoint.upgrade(AbstractEndPoint.java:185)
> at 
> org.eclipse.jetty.server.HttpConnection.completed(HttpConnection.java:345)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:436)
> at 
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at 
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
> at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:748)
>
>
>
> Can anybody please help me in knowing why this error is coming?
>
> Thanks,
>
> Priyanka
>
>
>

Re: Facing Unknown SSL protocol error when calling Knox

[Sandeep Moré]

Perhaps the SSL keys have expired ? I have not seen this one.
You can turn on SSL debugging on Knox and see what details you get, that
should help in debugging.

Best,
Sandeep

On Fri, Jul 20, 2018 at 12:44 AM Guang Yang  wrote:

> Hi guys,
>
> Our Knox service was running well for a long time, but suddenly starting
> from this afternoon, we kept seeing this error *curl: (35) Unknown SSL
> protocol error in connection to localhost:19943* when curl Knox. Do you
> guys have any idea what's the possible reason?
>
>
> Thanks,
>
> Guang
>
>

Re: Issue while accessing URIs containing '%'

[Sandeep Moré]

Hello Rajat,

I am not well aware of Grafana UI so I do not know how the URL should look
like so I am assuming you need % to be in the url right ?

Can you try using the following dispatches

   1. *PassAllHeadersNoEncodingDispatch*
   2. *URLDecodingDispatch*

Hopefully one of them should work.

Best,
Sandeep


On Tue, Jul 17, 2018 at 1:02 PM Rajat Goel  wrote:

> Hi,
>
>
>
> I am trying to integrate Grafana UI with Knox. Grafana UI main page
> internally sends a request URI ‘
> https://192.168.134.225:8443/gateway/default/grafana/api/datasources/proxy/1/ws/v1/timeline/metrics?metricNames=regionserver.Server.regionCount._max=%=ams-hbase==1531806914=1531828514’
> to Knox. Notice here that there is a query param: ‘hostname=%’.
>
>
>
> In the logs I see that Knox rewrites the url using rewrite rule but there
> are exceptions:
>
>
>
> *2018-07-17 13:01:21,332 DEBUG hadoop.gateway
> (UrlRewriteProcessor.java:rewrite(166)) - Rewrote *
>
> *URL:* 
> *https://192.168.134.225:8443/gateway/default/grafana/api/datasources/proxy/1/ws/v1/timeline/metrics?metricNames=regionserver.Server.regionCount._max=%=ams-hbase==1531806914=1531828514
> ,
> direction: IN via explicit rule: GRAFANAUI/grafana/inbound/queryvariable to
> URL:* 
> *http://mst-01.mydomain.com:3000/api/datasources/proxy/1/ws/v1/timeline/metrics?metricNames=regionserver.Server.regionCount._max=%25==ams-hbase=1531806914=1531828514
> *
>
> *2018-07-17 13:01:21,335 ERROR hadoop.gateway
> (AbstractGatewayFilter.java:doFilter(69)) - Failed to execute filter:
> java.lang.IllegalArgumentException: URLDecoder: Incomplete trailing escape
> (%) pattern*
>
> *java.lang.IllegalArgumentException: URLDecoder: Incomplete trailing
> escape (%) pattern*
>
> *at java.net.URLDecoder.decode(URLDecoder.java:187)*
>
> *at org.apache.hadoop.gateway.util.HttpUtils.splitQuery(HttpUtils.java:46)*
>
> *at
> org.apache.hadoop.gateway.identityasserter.common.filter.IdentityAsserterHttpServletRequestWrapper.getParams(IdentityAsserterHttpServletRequestWrapper.java:135)*
>
> *at
> org.apache.hadoop.gateway.identityasserter.common.filter.IdentityAsserterHttpServletRequestWrapper.getParams(IdentityAsserterHttpServletRequestWrapper.java:154)*
>
> *at
> org.apache.hadoop.gateway.identityasserter.common.filter.IdentityAsserterHttpServletRequestWrapper.getQueryString(IdentityAsserterHttpServletRequestWrapper.java:162)*
>
> *at
> org.apache.hadoop.gateway.dispatch.AbstractGatewayDispatch.getDispatchUrl(AbstractGatewayDispatch.java:83)*
>
> *at
> org.apache.hadoop.gateway.dispatch.GatewayDispatchFilter$GetAdapter.doMethod(GatewayDispatchFilter.java:132)*
>
> *at
> org.apache.hadoop.gateway.dispatch.GatewayDispatchFilter.doFilter(GatewayDispatchFilter.java:115)*
>
> *at
> org.apache.hadoop.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:61)*
>
>
>
> My rewrite rule is very simple of the form:
>
> * pattern="*://*:*/**/grafana/{**}?{**}">*
>
> **
>
> **
>
>
>
> I am using PassAllHeadersDispatch. Knox rewrires the URL with query param
> ‘hostname=%25’.
>
>
>
> How can I fix this ?
>
>
>
> Thanks & Regards,
>
> Rajat
>

Re: Need help in enabling SAML auth in Apache Knox

[Sandeep Moré]

Hello Praveen,

I am not familiar with PING Federate Identity provider but looks like your
target url could be wrong.
Can you try using
https://%3Cdnsname%3E:8446/gateway/knoxsso/api/v1/websso?pac4jCallback=trueclient_name=SAML2Client
as Target Url (basically entity id as target url), I am assuming this is
where the SAML Assertion is sent.

You can use the SAML-tracer addon for firefox to check see SAML requests
and responses, it helped me a lot.

Best,
Sandeep

On Mon, Jul 16, 2018 at 1:36 PM Ravikumar, Praveen Krishnamoorthy <
rpkr...@amazon.com> wrote:

> Hi,
>
> I'm Praveen. I'm working on POC to setup Apache Knox on the master node of
> an EMR cluster for our client. With the help of documentations I was able
> to install KNOX successfully and was able to run few tests. Currently I'm
> facing an issue on enabling SAML authentication, which I'm kind of blocked
> and I don’t know, how to proceed or troubleshoot the issue. I have provided
> few details regarding the issue and I would love to provide more if needed.
>
>
>
> Could anyone help me in this, would be very helpful for me to proceed
> further.
>
>
>
> TASK:
>
> -
>
> To enable SAML authentication for Apache Knox.
>
>
>
> NOTE: Apache Knox is installed and running in port 8446
>
>
>
> STEP 1: SSO request initiation.
>
> ***
>
> - Our client uses PING Federate Identity provider.
>
> - raised a request to register the application for SSO access.
>
> Entity ID -
> https://:8446/gateway/knoxsso/api/v1/websso?pac4jCallback=trueclient_name=SAML2Client
>
> Target URL - https://:8446(I'm not sure the target URL
> is valid, I suspect the page is getting redirected to this link after auth)
>
> - I received a IDP metadata.xml and certificate.
>
>
>
> STEP 2: Topology config
>
> ***
>
>
>
> KnoxSSO.xml
>
> 
>
> 
>
>
>
>  
>
>  federation
>
>  pac4j
>
>  true
>
>  
>
>   pac4j.callbackUrl
>
>   
> https://:8446/gateway/knoxsso/api/v1/websso
>
>  
>
>  
>
>clientName
>
>SAML2Client
>
>  
>
>  
>
>saml.identityProviderMetadataPath
>
>/tmp/preprod_metadata_SP.xml
>
>  
>
>  
>
>saml.serviceProviderMetadataPath
>
>/tmp/preprod_metadata_SP.xml
>
>  
>
>  
>
>saml.serviceProviderEntityId
>
>
> https://:8446/gateway/knoxsso/api/v1/websso?pac4jCallback=trueclient_name=SAML2Client
>
>  
>
>  
>
>  
>
>  identity-assertion
>
>  Default
>
>  true
>
>  
>
>
>
>
>
>KNOXSSO
>
>
>
>  knoxsso.cookie.secure.only
>
>  true
>
>   
>
>   
>
> knoxsso.token.ttl
>
> 10
>
>   
>
>   
>
>  knoxsso.redirect.whitelist.regex
>
>
> ^https?:\/\/(emr-knox-webui-dev\.us-west-2\.elb\.amazonaws\.com|localhost|127\.0\.0\.1|0:0:0:0:0:0:0:1|::1):[0-9].*$
>
>   
>
>
>
> 
>
>
>
> gate1.xml
>
> -
>
> 
>
> 
>
>   
>
> 
>
> federation
>
> SSOCookieProvider
>
> true
>
> 
>
> sso.authentication.provider.url
>
> 
> https://:8446/gateway/knoxsso/api/v1/websso
>
> 
>
> 
>
> 
>
> identity-assertion
>
> Default
>
> true
>
> 
>
>   
>
>   
>
>   YARNUI
>
>   http://:8088
>
>   
>
> 
>
>
>
>
>
> PROBLEM:
>
> 
>
> on accessing the YarnUI (firefox browser) after starting the gateway, The
> browser gets redirected to the Identity provider URL -> asks for the login
> credentials -> on submitting the user is getting authenticated but the
> application gets landed to https://:8446 and throws page not
> found error.
>
> I'm seeing the SAML request sent and SAML response getting received but it
> gets landed to an invalid page after authentication. I'm unable to figure
> out the page to land after authentication.
>
>
>
>
>
> Hope I have provided the required details. please do let me know if you
> need any additional details.
>
>
>
> Thanks,
>
> Praveen.
>
>
>

Re: Need help in enabling SAML auth in Apache Knox

[Sandeep Moré]

Hello Praveen,

Following are some of the blogs that talk about Knox SAML integration

https://cwiki.apache.org/confluence/display/KNOX/KnoxSSO+and+Okta
https://cwiki.apache.org/confluence/display/KNOX/Configuring+Apache+Knox+SSO+with+Ipsilon+using+SAML2

Knox user guide also has some information on how to configure SAML
https://knox.apache.org/books/knox-1-0-0/user-guide.html#Pac4j+Provider+-+CAS+/+OAuth+/+SAML+/+OpenID+Connect

For more details you can take a look at Pac4J provider
https://github.com/pac4j/pac4j/wiki/Clients#saml-support

Now coming back to your options, I could not find info on Ping IDP for SAML
so I cannot comment on the options, this is just my educated guess.


   1. Entity ID - I am guessing this is your "saml.serviceProviderEntityId"
   value, for me it is
   
https://www.myhost.com:8443/gateway/knoxsso/api/v1/websso?pac4jCallback=trueclient_name=SAML2Client
   2. Baseurl - no idea what this is, you probably have to refer to ping
   IDP docs
   3. ACSURL (where target assertion is Sent) - this is the same as above
   for
   
https://www.myhost.com:8443/gateway/knoxsso/api/v1/websso?pac4jCallback=true_name=SAML2Client





On Mon, Jul 16, 2018 at 5:47 PM Ravikumar, Praveen Krishnamoorthy <
rpkr...@amazon.com> wrote:

> Hi,
>
>
>
> Also, Could anyone please help me what should I configure for
>
>1. Entity ID
>2. BaseURL (that is used for the entire site)
>3. ACSURL (where target assertion is Sent)
>
>
>
> On the Identity provider side, to enable SSO in Knox.
>
>
>
> Thanks,
>
> Praveen.
>
>
>
> *From: *"Ravikumar, Praveen Krishnamoorthy" 
> *Date: *Monday, July 16, 2018 at 10:36 AM
> *To: *"user@knox.apache.org" 
> *Subject: *Need help in enabling SAML auth in Apache Knox
>
>
>
> Hi,
>
> I'm Praveen. I'm working on POC to setup Apache Knox on the master node of
> an EMR cluster for our client. With the help of documentations I was able
> to install KNOX successfully and was able to run few tests. Currently I'm
> facing an issue on enabling SAML authentication, which I'm kind of blocked
> and I don’t know, how to proceed or troubleshoot the issue. I have provided
> few details regarding the issue and I would love to provide more if needed.
>
>
>
> Could anyone help me in this, would be very helpful for me to proceed
> further.
>
>
>
> TASK:
>
> -
>
> To enable SAML authentication for Apache Knox.
>
>
>
> NOTE: Apache Knox is installed and running in port 8446
>
>
>
> STEP 1: SSO request initiation.
>
> ***
>
> - Our client uses PING Federate Identity provider.
>
> - raised a request to register the application for SSO access.
>
> Entity ID -
> https://:8446/gateway/knoxsso/api/v1/websso?pac4jCallback=trueclient_name=SAML2Client
>
> Target URL - https://:8446(I'm not sure the target URL
> is valid, I suspect the page is getting redirected to this link after auth)
>
> - I received a IDP metadata.xml and certificate.
>
>
>
> STEP 2: Topology config
>
> ***
>
>
>
> KnoxSSO.xml
>
> 
>
> 
>
>
>
>  
>
>  federation
>
>  pac4j
>
>  true
>
>  
>
>   pac4j.callbackUrl
>
>   
> https://:8446/gateway/knoxsso/api/v1/websso
>
>  
>
>  
>
>clientName
>
>SAML2Client
>
>  
>
>  
>
>saml.identityProviderMetadataPath
>
>/tmp/preprod_metadata_SP.xml
>
>  
>
>  
>
>saml.serviceProviderMetadataPath
>
>/tmp/preprod_metadata_SP.xml
>
>  
>
>  
>
>saml.serviceProviderEntityId
>
>
> https://:8446/gateway/knoxsso/api/v1/websso?pac4jCallback=trueclient_name=SAML2Client
>
>  
>
>  
>
>  
>
>  identity-assertion
>
>  Default
>
>  true
>
>  
>
>
>
>
>
>KNOXSSO
>
>
>
>  knoxsso.cookie.secure.only
>
>  true
>
>   
>
>   
>
> knoxsso.token.ttl
>
> 10
>
>   
>
>   
>
>  knoxsso.redirect.whitelist.regex
>
>
> ^https?:\/\/(emr-knox-webui-dev\.us-west-2\.elb\.amazonaws\.com|localhost|127\.0\.0\.1|0:0:0:0:0:0:0:1|::1):[0-9].*$
>
>   
>
>
>
> 
>
>
>
> gate1.xml
>
> -
>
> 
>
> 
>
>   
>
> 
>
> federation
>
> SSOCookieProvider
>
> true
>
> 
>
> sso.authentication.provider.url
>
> 
> https://:8446/gateway/knoxsso/api/v1/websso
>
> 
>
> 
>
> 
>
> identity-assertion
>
> Default
>
> true
>
> 
>
>   
>
>   
>
>   YARNUI
>
>   http://:8088
>
>   
>
> 
>
>
>
>
>
> PROBLEM:
>
> 
>
> on accessing the YarnUI (firefox browser) after starting the gateway, The
> browser gets redirected to the Identity provider URL -> asks for the login
> credentials -> on submitting the user is getting authenticated but the
> application gets landed to https://:8446 and throws page not
> found 

Re: Knox rules applied randomly

[Sandeep Moré]

Divya,

I think this is because of the "path" rule is more open than the "queryv"
rule.
Can you try reordering the rules, putting  "path" rule below  "queryv"  and
see what you get.

Also, I would suggest be more specific about the rules to avoid these
issues.

Best,
Sandeep

On Thu, Aug 30, 2018 at 7:20 AM Divya Narayan 
wrote:

> Hi,
>
> While trying to integrate CDAP with knox, we created rewrite and service
> rules for CDAP and content of those files looks like this:
>
>  *service.xml:*
>
> 
> 
> 
> 
> 
> 
> 
> 
>   
>  
> 
> 
>  classname="org.apache.hadoop.gateway.dispatch.PassAllHeadersDispatch"/>
> 
>
>
> *R**ewrite.xml:*
>
> 
>  pattern="*://*:*/**/cdapui/">
> 
> 
>  pattern="*://*:*/**/cdapui/{**}">
> 
> 
> pattern="*://*:*/**/cdapui/{**}?{**}">
> 
> 
> 
>
>
> Now, the expectation from these rules were that any URL with ? should
> match queryv rule. But in actual for calls containing ? , knox sometimes
> matches queryv but some times matches path rule.
> Whenever it apply queryv, the redirected URL is correct but whenever it
> matches path rule for URL with ? , the redirected URL trim the query part
> of the URL (I.e everthing that comes after ? in URL).
>
>
>
> Here is some calsl from gateway.log where knox is applying different rules
> for same pattern of URLs:
>
> 1- Here for
> https://192.168.134.119:8443/gateway/default/cdapui/_sock/717/lmweicc4/htmlfile?c=_jp.aok0aa1
>  URL, knox applies path rule instead of queryv rule, as a result of which
> redirected URL is wrong
>
> *2018-08-30 09:45:34,417 DEBUG hadoop.gateway
> (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL:
> https://192.168.134.119:8443/gateway/default/cdapui/_sock/717/lmweicc4/htmlfile?c=_jp.aok0aa1
> ,
> direction: IN via explicit rule: CDAPUI/cdap/inbound/path to URL:
> http://platacc003-mst-01.gvs.ggn:11011/_sock/717/lmweicc4/htmlfile
> *
>
>
>
> 2- Another example where for URL
> https://192.168.134.119:8443/gateway/default/cdapui/pipelines/ns/_sock/729/5wugs1dc/htmlfile?c=_jp.abxtrqj,
> knox applies queryv rule, as a result of which redirected URL is correct
>
> *2018-08-30 09:45:34,936 DEBUG hadoop.gateway
> (UrlRewriteProcessor.java:rewrite(164)) - Rewrote URL:
> https://192.168.134.119:8443/gateway/default/cdapui/pipelines/ns/_sock/729/5wugs1dc/htmlfile?c=_jp.abxtrqj
> ,
> direction: IN via implicit rule: CDAPUI/cdap/inbound/queryv to URL:
> http://platacc003-mst-01.gvs.ggn:11011/pipelines/ns/_sock/729/5wugs1dc/htmlfile?c=_jp.abxtrqj
> *
>
>
> Any idea why knox is applying different rules for URLs with similar
> pattern??
>
> Thanks
> Divya
>
>
>
>
>
>

Re: zeppelin via knox in HDP3 does not work for SQL

[Sandeep Moré]

Knox master has patch for this issue
https://issues.apache.org/jira/browse/KNOX-1424

You can try applying the patch it should fix the issue.

Best,
Sandeep

On Fri, Aug 31, 2018 at 12:18 PM Lian Jiang  wrote:

> I am using HDP3.0 (having zeppelin 0.8.0).
>
> Below queries do not show any result, even though downloading the csv show
> the data correctly (e.g. if there is no tables, show the header).
>
> %livy2.sql
> show tables
>
> %spark2.sql
> show tables
>
> Infrequently I saw the table show a short time and then disappear. I guess
> it is html rendering issue.
>
> This issue does not happen when I ssh to the zeppelin host and use
> zeppelin via port forwarding. It happens when I use zeppelin via knox.
>
> Any idea? Any setting is related to this? Sql interpreters worked fine in
> HDP2.6 using zeppelin 0.7.3.
>
>
> javascript error:
> Error: [ngRepeat:iidexp] '_item_' in '_item_ in _collection_' should be an
> identifier or '(_key_, _value_)' expression, but got
> '/gateway/ui/zeppelin/app'.
>
> http://errors.angularjs.org/1.5.7/ngRepeat/iidexp?p0=%2Fgateway%2Fui%2Fzeppelin%2Fapp
> b/<@
> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:36:376
> Kg https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:40:30580
> Z@
> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:37:4755
> S@
> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:36:30480
> S@
> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:36:30611
> S@
> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:36:30611
> N@
> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:36:29413
> X/<@
> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:37:440
> d@
> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:36:30833
> m@
> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:37:909
> mg https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:40:17582
> xc/this.$get https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:38:10966
> a/n.prototype.safeDigest@
> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:76:1460
> b@
> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:76:3748
> a/n.prototype._onMessageHandler@
> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:76:3960
> R/<@
> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:36:5633
> " 
>
>
>
> Appreciate any clue.
>

Re: zeppelin via knox in HDP3 does not work for SQL

[Sandeep Moré]

Great !


On Sat, Sep 1, 2018 at 12:49 PM Lian Jiang  wrote:

> Thanks. patching worked after executing step 3.
>
> On Fri, Aug 31, 2018 at 4:44 PM larry mccay  wrote:
>
>> Replacing the service definition files alone isn't quite enough.
>> You have to do the following to make sure that the server picks up the
>> new service defs and redeploys the topology hosting the affected service:
>>
>> 1. change rewrite rules
>> 2. restart gateway so that the gateway is aware of the new service def
>> 3. touch/save the topology hosting the ZEPPELIN services so that they
>> redeploy
>>
>> On Fri, Aug 31, 2018 at 7:04 PM Lian Jiang  wrote:
>>
>>> Thanks Sandeep. Could you please clarify the patching process?
>>>
>>> There is existing rewrite.xml and service.xml under
>>> /var/lib/knox/data-3.0.0.0-1634/services/zeppelinui/0.6.0 on the knox host.
>>> None of below makes zeppelin sql interpreter work:
>>>
>>> 1. Put the new rewrite.xml and service.xml into new created 0.8.0 folder
>>> and restart knox
>>> 2. Replace these two files under 0.6.0 with the two new files and
>>> restart knox
>>>
>>> Appreciate any clue.
>>>
>>> On Fri, Aug 31, 2018 at 1:15 PM Sandeep Moré 
>>> wrote:
>>>
>>>> Knox master has patch for this issue
>>>> https://issues.apache.org/jira/browse/KNOX-1424
>>>>
>>>> You can try applying the patch it should fix the issue.
>>>>
>>>> Best,
>>>> Sandeep
>>>>
>>>> On Fri, Aug 31, 2018 at 12:18 PM Lian Jiang 
>>>> wrote:
>>>>
>>>>> I am using HDP3.0 (having zeppelin 0.8.0).
>>>>>
>>>>> Below queries do not show any result, even though downloading the csv
>>>>> show the data correctly (e.g. if there is no tables, show the header).
>>>>>
>>>>> %livy2.sql
>>>>> show tables
>>>>>
>>>>> %spark2.sql
>>>>> show tables
>>>>>
>>>>> Infrequently I saw the table show a short time and then disappear. I
>>>>> guess it is html rendering issue.
>>>>>
>>>>> This issue does not happen when I ssh to the zeppelin host and use
>>>>> zeppelin via port forwarding. It happens when I use zeppelin via knox.
>>>>>
>>>>> Any idea? Any setting is related to this? Sql interpreters worked fine
>>>>> in HDP2.6 using zeppelin 0.7.3.
>>>>>
>>>>>
>>>>> javascript error:
>>>>> Error: [ngRepeat:iidexp] '_item_' in '_item_ in _collection_' should
>>>>> be an identifier or '(_key_, _value_)' expression, but got
>>>>> '/gateway/ui/zeppelin/app'.
>>>>>
>>>>> http://errors.angularjs.org/1.5.7/ngRepeat/iidexp?p0=%2Fgateway%2Fui%2Fzeppelin%2Fapp
>>>>> b/<@
>>>>> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:36:376
>>>>> Kg>>>> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:40:30580
>>>>> Z@
>>>>> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:37:4755
>>>>> S@
>>>>> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:36:30480
>>>>> S@
>>>>> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:36:30611
>>>>> S@
>>>>> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:36:30611
>>>>> N@
>>>>> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:36:29413
>>>>> X/<@
>>>>> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:37:440
>>>>> d@
>>>>> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:36:30833
>>>>> m@
>>>>> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:37:909
>>>>> mg>>>> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:40:17582
>>>>> xc/this.$get>>>> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:38:10966
>>>>> a/n.prototype.safeDigest@
>>>>> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:76:1460
>>>>> b@
>>>>> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:76:3748
>>>>> a/n.prototype._onMessageHandler@
>>>>> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:76:3960
>>>>> R/<@
>>>>> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:36:5633
>>>>> " 
>>>>>
>>>>>
>>>>>
>>>>> Appreciate any clue.
>>>>>
>>>>

Re: Knox wrong query param encoding help

[Sandeep Moré]

You can try using "org.apache.knox.gateway.dispatch.URLDecodingDispatch"
dispatch in service.xml ( Knox 1.1.0)

Best,
Sandeep

On Tue, Sep 4, 2018 at 2:09 PM Theyaa Matti  wrote:

> I am having issues with Knox encoding the following URL when parsing html
> content.
>
> /proxy/application_33323323_0001/stages/stage?id=0attempt=0
>
>
> This URL is generated by Yarn to track a running spark job. When applying
> regular rules to the above URL, Knox encodes the URL to:
>
>
> /proxy/application_33323323_0001/stages/stage?amp%3Battempt=0=0
>
>
> Which makes the URL unusable and leads to a 404. To make the URL work I
> have to remove "amp%3B" and then it works.
>
>
> Below are the filter and the rule I am using for this URL and I appreciate
> your help/comments for a remediation.
>
>
> 
> 
>  rule="YARNUI/yarn/outbound/apps/history3"/>
> 
> 
>
>
> 
> 
> 
> 
>
>
> Thank you for your help.
>
>
>

Re: Knox wrong query param encoding help

[Sandeep Moré]

You can extend RMUIHaDispatch class and override getDispatchUrl() method
from URLDecodingDispatch

Best,
Sandeep

On Thu, Sep 6, 2018 at 2:28 PM Theyaa Matti  wrote:

> Yes I am using HA for resource manager and in that case what should I do
> to the fix this issue?
>
> Best,
>
> Theyaa.
>
>
> On Thu, Sep 6, 2018 at 2:24 PM Sandeep Moré  wrote:
>
>> Are you using HA setup ? in which case org.apache.
>> hadoop.gateway.rm.dispatch.RMUIHaDispatch dispatch will be used.
>>
>> On Thu, Sep 6, 2018 at 2:19 PM Theyaa Matti 
>> wrote:
>>
>>> I created the class as follows:
>>>
>>> package org.apache.knox.gateway.dispatch;
>>>
>>> import javax.servlet.http.HttpServletRequest;
>>> import java.net.URI;
>>> import java.net.URLDecoder;
>>> import org.apache.hadoop.gateway.dispatch.DefaultDispatch;
>>>
>>> /**
>>>  * Dispatch which decodes the outgoing URLs (to services).
>>>  * This is useful in cases where the url is picked up
>>>  * from the query parameter and is already encoded.
>>>  *
>>>  * @since 1.1.0
>>>  */
>>> public class URLDecodingDispatch extends DefaultDispatch {
>>>
>>> public URLDecodingDispatch() {
>>> super();
>>> }
>>>
>>> @Override
>>> public URI getDispatchUrl(final HttpServletRequest request) {
>>> String decoded;
>>>
>>> try {
>>> decoded = URLDecoder.decode(request.getRequestURL().toString(), 
>>> "UTF-8" );
>>> } catch (final Exception e) {
>>>   /* fall back in case of exception */
>>> decoded = request.getRequestURL().toString();
>>> }
>>>
>>> final StringBuffer str = new StringBuffer(decoded);
>>> final String query = request.getQueryString();
>>> if ( query != null ) {
>>> str.append('?');
>>> str.append(query);
>>> }
>>> final URI url = URI.create(str.toString());
>>> return url;
>>> }
>>> }
>>>
>>>
>>> packaged it into a jar and uploaded it to the knox lib dir. I can see
>>> Knox loading the jar file at start:
>>> resource.FileResource (FileResource.java:checkFileAlias(152)) - ALIAS
>>> abs=/knox/bin/../lib/knox-1.0-SNAPSHOT.jar
>>> can=/knox/lib/knox-1.0-SNAPSHOT.jar
>>>
>>> I modified the service.xml for yarnui as follows.
>>>
>>> >> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>>>
>>>
>>>
>>> Deleted everything under deployments and restarted knox.
>>>
>>>
>>> I do not see any changes in the behavior and do not see that class being
>>> called. Do you please know what I am missing?
>>>
>>>
>>> Regards,
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Sep 6, 2018 at 9:38 AM Sandeep Moré 
>>> wrote:
>>>
>>>> URLDecodingDispatch extends DefaultDispatch so you should be there.
>>>>
>>>> Best,
>>>> Sandeep
>>>>
>>>> On Thu, Sep 6, 2018 at 9:32 AM Theyaa Matti 
>>>> wrote:
>>>>
>>>>> That should be a good idea, but I still have an issue with adding the
>>>>> custom dispatch to the yarnui service.xml. I already have an existing 
>>>>> class
>>>>> there >>>> classname="org.apache.hadoop.gateway.dispatch.DefaultDispatch"
>>>>> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>>>>>
>>>>> Should I replace the DefaultDispatch or I can have more than one there?
>>>>>
>>>>> Regards,
>>>>>
>>>>>
>>>>> On Tue, Sep 4, 2018 at 10:14 PM Dhruv Goyal <777.dh...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> In that case you might have to build the jar yourself or preferably
>>>>>> use your custom dispatch similar to URLDecodingDispatch.
>>>>>>
>>>>>> Regards
>>>>>> Dhruv
>>>>>>
>>>>>> On Wednesday, September 5, 2018, Theyaa Matti 
>>>>>> wrote:
>>>>>>
>>>>>>> Sorry 0.1

Re: Knox wrong query param encoding help

[Sandeep Moré]

Are you using HA setup ? in which case org.apache.
hadoop.gateway.rm.dispatch.RMUIHaDispatch dispatch will be used.

On Thu, Sep 6, 2018 at 2:19 PM Theyaa Matti  wrote:

> I created the class as follows:
>
> package org.apache.knox.gateway.dispatch;
>
> import javax.servlet.http.HttpServletRequest;
> import java.net.URI;
> import java.net.URLDecoder;
> import org.apache.hadoop.gateway.dispatch.DefaultDispatch;
>
> /**
>  * Dispatch which decodes the outgoing URLs (to services).
>  * This is useful in cases where the url is picked up
>  * from the query parameter and is already encoded.
>  *
>  * @since 1.1.0
>  */
> public class URLDecodingDispatch extends DefaultDispatch {
>
> public URLDecodingDispatch() {
> super();
> }
>
> @Override
> public URI getDispatchUrl(final HttpServletRequest request) {
> String decoded;
>
> try {
> decoded = URLDecoder.decode(request.getRequestURL().toString(), 
> "UTF-8" );
> } catch (final Exception e) {
>   /* fall back in case of exception */
> decoded = request.getRequestURL().toString();
> }
>
> final StringBuffer str = new StringBuffer(decoded);
> final String query = request.getQueryString();
> if ( query != null ) {
> str.append('?');
> str.append(query);
> }
> final URI url = URI.create(str.toString());
> return url;
> }
> }
>
>
> packaged it into a jar and uploaded it to the knox lib dir. I can see Knox
> loading the jar file at start:
> resource.FileResource (FileResource.java:checkFileAlias(152)) - ALIAS
> abs=/knox/bin/../lib/knox-1.0-SNAPSHOT.jar
> can=/knox/lib/knox-1.0-SNAPSHOT.jar
>
> I modified the service.xml for yarnui as follows.
>
>  ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>
>
>
> Deleted everything under deployments and restarted knox.
>
>
> I do not see any changes in the behavior and do not see that class being
> called. Do you please know what I am missing?
>
>
> Regards,
>
>
>
>
>
>
> On Thu, Sep 6, 2018 at 9:38 AM Sandeep Moré  wrote:
>
>> URLDecodingDispatch extends DefaultDispatch so you should be there.
>>
>> Best,
>> Sandeep
>>
>> On Thu, Sep 6, 2018 at 9:32 AM Theyaa Matti 
>> wrote:
>>
>>> That should be a good idea, but I still have an issue with adding the
>>> custom dispatch to the yarnui service.xml. I already have an existing class
>>> there >> classname="org.apache.hadoop.gateway.dispatch.DefaultDispatch"
>>> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>>>
>>> Should I replace the DefaultDispatch or I can have more than one there?
>>>
>>> Regards,
>>>
>>>
>>> On Tue, Sep 4, 2018 at 10:14 PM Dhruv Goyal <777.dh...@gmail.com> wrote:
>>>
>>>> In that case you might have to build the jar yourself or preferably use
>>>> your custom dispatch similar to URLDecodingDispatch.
>>>>
>>>> Regards
>>>> Dhruv
>>>>
>>>> On Wednesday, September 5, 2018, Theyaa Matti 
>>>> wrote:
>>>>
>>>>> Sorry 0.12.0
>>>>>
>>>>> On Tue, Sep 4, 2018 at 2:30 PM Theyaa Matti 
>>>>> wrote:
>>>>>
>>>>>> I am using Knox 1.12.0
>>>>>>
>>>>>> On Tue, Sep 4, 2018 at 2:28 PM Sandeep Moré 
>>>>>> wrote:
>>>>>>
>>>>>>> You can try
>>>>>>> using "org.apache.knox.gateway.dispatch.URLDecodingDispatch" dispatch in
>>>>>>> service.xml ( Knox 1.1.0)
>>>>>>>
>>>>>>> Best,
>>>>>>> Sandeep
>>>>>>>
>>>>>>> On Tue, Sep 4, 2018 at 2:09 PM Theyaa Matti 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I am having issues with Knox encoding the following URL when
>>>>>>>> parsing html content.
>>>>>>>>
>>>>>>>> /proxy/application_33323323_0001/stages/stage?id=0attempt=0
>>>>>>>>
>>>>>>>>
>>>>>>>> This URL is generated by Yarn to track a running spark job. When
>>>>>>>> applying regular rules to the above URL, Knox encodes the URL to:
>>>>>>>>
>>>>>>>>
>>>>>>>> /proxy/application_33323323
>>>>>>>> _0001/stages/stage?amp%3Battempt=0=0
>>>>>>>>
>>>>>>>>
>>>>>>>> Which makes the URL unusable and leads to a 404. To make the URL
>>>>>>>> work I have to remove "amp%3B" and then it works.
>>>>>>>>
>>>>>>>>
>>>>>>>> Below are the filter and the rule I am using for this URL and I
>>>>>>>> appreciate your help/comments for a remediation.
>>>>>>>>
>>>>>>>>
>>>>>>>> 
>>>>>>>> 
>>>>>>>> >>>>>>> rule="YARNUI/yarn/outbound/apps/history3"/>
>>>>>>>> 
>>>>>>>> 
>>>>>>>>
>>>>>>>>
>>>>>>>> 
>>>>>>>> 
>>>>>>>> >>>>>>> template="{$frontend[url]}/yarn/proxy/{*}/stages/stage?{**}"/>
>>>>>>>> 
>>>>>>>>
>>>>>>>>
>>>>>>>> Thank you for your help.
>>>>>>>>
>>>>>>>>
>>>>>>>>

Re: NPE from HttpServletRequest in Dispatch when query string contains special characters.

[Sandeep Moré]

Do you have a stack trace for the NPE ?

On Mon, Sep 10, 2018 at 10:21 AM Theyaa Matti  wrote:

> Anyone has any idea how to resolve this please?
>
> On Fri, Sep 7, 2018 at 1:24 PM Theyaa Matti  wrote:
>
>> I am using custom Dispatch class in my service.xml to handle html
>> entities and I am getting NPE from HttpServletRequest.getRequestURL() when
>> the queryString contains special characters like "amp%3B". In this case the
>> request object is null.
>>
>> Any help is appreciated.
>>
>>
>> Regards
>>
>

Re: how to handle hdfs UI redirect in knox

[Sandeep Moré]

There were a bunch of fixes that went into Knox 1.1.0 (currently in the
process of releasing) that fixes a bunch of issues with HDFSUI.
We also added new service defs [1] that has those fixes.

If you are using older version of knox, try adding these to your hdfsui
service folder (services/hdfsui/). Because of a bug [2] Knox does not
always pick up the latest service defs. so you will have to specify HDFSUI
version in your topology (3.0.0 in this case).

Let me know if you run into issues.

Best,
Sandeep


[1]
https://github.com/apache/knox/tree/master/gateway-service-definitions/src/main/resources/services/hdfsui/3.0.0
[2] https://issues.apache.org/jira/browse/KNOX-1349


On Mon, Jul 9, 2018 at 6:16 PM Lian Jiang  wrote:

> Hi,
>
> I am following
> https://community.hortonworks.com/articles/81713/configure-knox-to-access-hdfs-ui.html
> to expose hdfs ui via knox.
>
> I have tested that all below curl commands worked:
>
> on namenode:
> curl http://localhost:50070/
> curl http://localhost:50070/dfshealth.html
>
> on a random machine:
> curl -vvv -k -u guest:"{PASSWORD}" https://{DOMAIN}/gateway/ui/hdfs
>
> However, in web browser, https://{DOMAIN}/gateway/ui/hdfs is redirected
> to https://{DOMAIN}/gateway/ui/dfshealth.html which is not available. I
> observed that
> /var/lib/knox/data-2.6.4.0-91/services/hdfsui/2.7.0/rewrite.xml already has:
>
>  pattern="*://*:*/**/hdfs/dfshealth.html">
> 
>   
>
> what else do I need to do make hdfs ui work via knox? Appreciate any clue.
>

Re: how to handle hdfs UI redirect in knox

[Sandeep Moré]

Sorry, I wasn't clear.
You can do that (download an replace service and rewrite xmls) and keep
your existing service definition intact i.e.

HDFSUI
http://{DOMAIN}:50070


or


   1. You can create a new new folder
/var/lib/knox/data-2.6.4.0-91/services/hdfsui/3.0.0/
   2. Copy the new service.xml and rewrite.xml files there (into 3.0.0)
   3. Now update your topology with the service definition as follows


HDFSUI
3.0.0
http://{DOMAIN}:50070


Best,
Sandeep

On Tue, Jul 10, 2018 at 1:46 PM Lian Jiang  wrote:

> Thanks Sandeep.
>
> I am trying to understand what you said. Should I do below:
>
> 1. download service.xml and rewrite.xml from link [1].
> 2. replace the ones under
> /var/lib/knox/data-2.6.4.0-91/services/hdfsui/2.7.0/
> 3. in the topology file,
> change:
>  
> HDFSUI
> http://{DOMAIN}:50070
> 
> to:
> 
> HDFSUI
> http://{DOMAIN}:50070/*2.7.0/*
> 
>
> Thanks for clarification.
>
> On Mon, Jul 9, 2018 at 3:35 PM, Sandeep Moré 
> wrote:
>
>> There were a bunch of fixes that went into Knox 1.1.0 (currently in the
>> process of releasing) that fixes a bunch of issues with HDFSUI.
>> We also added new service defs [1] that has those fixes.
>>
>> If you are using older version of knox, try adding these to your hdfsui
>> service folder (services/hdfsui/). Because of a bug [2] Knox does not
>> always pick up the latest service defs. so you will have to specify HDFSUI
>> version in your topology (3.0.0 in this case).
>>
>> Let me know if you run into issues.
>>
>> Best,
>> Sandeep
>>
>>
>> [1]
>> https://github.com/apache/knox/tree/master/gateway-service-definitions/src/main/resources/services/hdfsui/3.0.0
>> [2] https://issues.apache.org/jira/browse/KNOX-1349
>>
>>
>> On Mon, Jul 9, 2018 at 6:16 PM Lian Jiang  wrote:
>>
>>> Hi,
>>>
>>> I am following
>>> https://community.hortonworks.com/articles/81713/configure-knox-to-access-hdfs-ui.html
>>> to expose hdfs ui via knox.
>>>
>>> I have tested that all below curl commands worked:
>>>
>>> on namenode:
>>> curl http://localhost:50070/
>>> curl http://localhost:50070/dfshealth.html
>>>
>>> on a random machine:
>>> curl -vvv -k -u guest:"{PASSWORD}" https://{DOMAIN}/gateway/ui/hdfs
>>>
>>> However, in web browser, https://{DOMAIN}/gateway/ui/hdfs is redirected
>>> to https://{DOMAIN}/gateway/ui/dfshealth.html which is not available. I
>>> observed that
>>> /var/lib/knox/data-2.6.4.0-91/services/hdfsui/2.7.0/rewrite.xml already has:
>>>
>>> >> pattern="*://*:*/**/hdfs/dfshealth.html">
>>> 
>>>   
>>>
>>> what else do I need to do make hdfs ui work via knox? Appreciate any
>>> clue.
>>>
>>
>

Re: how to handle hdfs UI redirect in knox

[Sandeep Moré]

What error are you getting ?
Also what is the Hive version ?

On Tue, Jul 10, 2018 at 3:36 PM Lian Jiang  wrote:

> Thanks for the details.
>
> Way1 did not work (as usual, redirect to dfshealth.html which is not
> found) and way 2 throw 500 server error. I have trouble to find related
> log. Any idea regarding debugging?
>
> On Tue, Jul 10, 2018 at 11:08 AM, Sandeep Moré 
> wrote:
>
>> Sorry, I wasn't clear.
>> You can do that (download an replace service and rewrite xmls) and keep
>> your existing service definition intact i.e.
>> 
>> HDFSUI
>> http://{DOMAIN}:50070
>> 
>>
>> or
>>
>>
>>1. You can create a new new folder
>> /var/lib/knox/data-2.6.4.0-91/services/hdfsui/3.0.0/
>>2. Copy the new service.xml and rewrite.xml files there (into 3.0.0)
>>3. Now update your topology with the service definition as follows
>>
>> 
>> HDFSUI
>> 3.0.0
>> http://{DOMAIN}:50070
>> 
>>
>> Best,
>> Sandeep
>>
>> On Tue, Jul 10, 2018 at 1:46 PM Lian Jiang  wrote:
>>
>>> Thanks Sandeep.
>>>
>>> I am trying to understand what you said. Should I do below:
>>>
>>> 1. download service.xml and rewrite.xml from link [1].
>>> 2. replace the ones under
>>> /var/lib/knox/data-2.6.4.0-91/services/hdfsui/2.7.0/
>>> 3. in the topology file,
>>> change:
>>>  
>>> HDFSUI
>>> http://{DOMAIN}:50070
>>> 
>>> to:
>>> 
>>> HDFSUI
>>> http://{DOMAIN}:50070/*2.7.0/*
>>> 
>>>
>>> Thanks for clarification.
>>>
>>> On Mon, Jul 9, 2018 at 3:35 PM, Sandeep Moré 
>>> wrote:
>>>
>>>> There were a bunch of fixes that went into Knox 1.1.0 (currently in the
>>>> process of releasing) that fixes a bunch of issues with HDFSUI.
>>>> We also added new service defs [1] that has those fixes.
>>>>
>>>> If you are using older version of knox, try adding these to your hdfsui
>>>> service folder (services/hdfsui/). Because of a bug [2] Knox does not
>>>> always pick up the latest service defs. so you will have to specify HDFSUI
>>>> version in your topology (3.0.0 in this case).
>>>>
>>>> Let me know if you run into issues.
>>>>
>>>> Best,
>>>> Sandeep
>>>>
>>>>
>>>> [1]
>>>> https://github.com/apache/knox/tree/master/gateway-service-definitions/src/main/resources/services/hdfsui/3.0.0
>>>> [2] https://issues.apache.org/jira/browse/KNOX-1349
>>>>
>>>>
>>>> On Mon, Jul 9, 2018 at 6:16 PM Lian Jiang 
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am following
>>>>> https://community.hortonworks.com/articles/81713/configure-knox-to-access-hdfs-ui.html
>>>>> to expose hdfs ui via knox.
>>>>>
>>>>> I have tested that all below curl commands worked:
>>>>>
>>>>> on namenode:
>>>>> curl http://localhost:50070/
>>>>> curl http://localhost:50070/dfshealth.html
>>>>>
>>>>> on a random machine:
>>>>> curl -vvv -k -u guest:"{PASSWORD}" https://{DOMAIN}/gateway/ui/hdfs
>>>>>
>>>>> However, in web browser, https://{DOMAIN}/gateway/ui/hdfs is
>>>>> redirected to https://{DOMAIN}/gateway/ui/dfshealth.html which is not
>>>>> available. I observed that
>>>>> /var/lib/knox/data-2.6.4.0-91/services/hdfsui/2.7.0/rewrite.xml already 
>>>>> has:
>>>>>
>>>>> >>>> pattern="*://*:*/**/hdfs/dfshealth.html">
>>>>> 
>>>>>   
>>>>>
>>>>> what else do I need to do make hdfs ui work via knox? Appreciate any
>>>>> clue.
>>>>>
>>>>
>>>
>

Re: KnoxSSO with NiFi error: PKIX path building failed...

[Sandeep Moré]

Hello Ryan,

Looks like you need to provision NiFi public cert into Knox keystore that
should do it.

On Wed, Mar 7, 2018 at 7:12 PM, Ryan H 
wrote:

> Hi All,
>
> I seem to be having a really tough time getting Knox to work with a secure
> NiFi cluster set up. I have tried to get this working two different ways.
> Both ways have basically the same set up for knoxsso, where it uses cloud
> foundry UAA as an external identity provider (currently configured for
> OpenID, with the /.well-known/openid-configuration prepended to the UAA
> instance url). I'm not sure if OpenID connect is the correct way to go, I
> believe there are other options with UAA; this is just the route I went as
> I initially was going to configure NiFi OpenID properties with my UAA
> instance. I have since decided (based on other factors) that Knox would be
> a better way to go. I have been focusing on option 1 below, as I think this
> is the preferred way. However, I tried option 2 below just to see if I
> could get around the error temporarily. I've included the errors I am
> running into below as well as relevant config. Any help is greatly
> appreciated.
>
> versions: NiFi 1.6 and Knox 1.1.0
>
> *1. Users will always access NiFi thru Knox (preferred)*
> *Issue Facing: Getting "PKIX path building failed: unable to find valid
> certification path to requested target"*
>
> *knoxsso.xml*
> 
>   
> 
> webappsec
> WebAppSec
> true
> xframe.options.enabledtrue value>
> 
> 
> federation
> pac4j
> true
> 
> pac4j.session.store
> J2ESessionStore
> 
> 
>   pac4j.callbackUrl
>   https://my-knox-host:8443/gateway/knoxsso/api/v1/websso
> 
> 
> 
>   clientName
>   OidcClient
> 
> 
>   oidc.id
>   some_client_id
> 
> 
>   oidc.secret
>   some_client_secret
> 
> 
>   oidc.discoveryUri
>   https://my-uaa-host:443/.well-known/openid-configuration
> 
> 
> 
>   oidc.preferredJwsAlgorithm
>   RS256
> 
> 
> 
>
> 
>   knoxauth
> 
> 
> KNOXSSO
> 
> knoxsso.cookie.secure.only
> false
> 
> 
> knoxsso.enable.session
> true
> 
> 
> knoxsso.cookie.max.age
> session
> 
> 
> knoxsso.token.ttl
> 360
> 
> 
>knoxsso.redirect.whitelist.regex
>^https?:\/\/(localhost|10\.227\.85\.2|[0-9]
> {1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}||127\.0\.0\.1|0:0:
> 0:0:0:0:0:1|::1):[0-9].*$
> 
> 
> 
>
> *sandbox.xml*
> 
>   federation
>   SSOCookieProvider
>   true
>   
>   sso.authentication.provider.url
>   https://my-knox-host:8443/gateway/knoxsso/api/v1/websso
> 
>   
>   
>
>
> 
> identity-assertion
> Default
> true
> 
>
> 
> hostmap
> static
> true
> 
>
> 
>
> 
> NIFI
> https://my-nifi-host:8443
> 
> 
>
> *Stacktrace from Knox:*
>  knox.gateway (DefaultDispatch.java:executeOutboundRequest(147)) -
> Connection exception dispatching request: https://my-nifi-host:8443/
> nifi?user.name=ba2d3b04-6bbd-4473-80f4-c2f528cb1d72 
> javax.net.ssl.SSLHandshakeException:
> sun.security.validator.ValidatorException: PKIX path building failed:
> sun.security.provider.certpath.SunCertPathBuilderException: unable to
> find valid certification path to requested target
> javax.net.ssl.SSLHandshakeException: 
> sun.security.validator.ValidatorException:
> PKIX path building failed: 
> sun.security.provider.certpath.SunCertPathBuilderException:
> unable to find valid certification path to requested target
> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1959)
> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:328)
> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
> at sun.security.ssl.ClientHandshaker.serverCertificate(
> ClientHandshaker.java:1614)
> at sun.security.ssl.ClientHandshaker.processMessage(
> ClientHandshaker.java:216)
> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
> at sun.security.ssl.Handshaker.process_record(Handshaker.java:987)
> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072)
> at sun.security.ssl.SSLSocketImpl.performInitialHandshake(
> SSLSocketImpl.java:1385)
> at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
> at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
> at org.apache.http.conn.ssl.SSLConnectionSocketFactory.
> createLayeredSocket(SSLConnectionSocketFactory.java:396)
> at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(
> 

Re: KnoxSSO with NiFi error: PKIX path building failed...

[Sandeep Moré]

I am not that aware of NiFi specific SSL settings, but if you have a
truststore from NiFi (instead of a cert) you can use that, if it has a
deifferent password you will have to configure it. You can find the
instructions here
https://knox.apache.org/books/knox-1-0-0/user-guide.html#Mutual+Authentication+with+SSL



On Wed, Mar 7, 2018 at 8:27 PM, Ryan H <ryan.howell.developm...@gmail.com>
wrote:

> Hi Sandeep,
>
> So I have the NiFi TLS Toolkit running in Client/Server mode. I have made
> a request to the CA server from the Knox machine by running the TLS Toolkit
> as a Client and received a keystore, truststore, and nifi-cert.pem. I
> understand that I need to get the public cert into the Knox keystore, but
> unsure which one to import and to where. Should the cert be imported into
> the KNOX_HOME/data/security/keystores/gateway.jks store? And do you know
> which one of the files should have the public cert?
>
> Thanks in Advance,
>
> -Ryan
>
> On Wed, Mar 7, 2018 at 8:03 PM, Sandeep Moré <moresand...@gmail.com>
> wrote:
>
>> Hello Ryan,
>>
>> Looks like you need to provision NiFi public cert into Knox keystore that
>> should do it.
>>
>>
>> On Wed, Mar 7, 2018 at 7:12 PM, Ryan H <ryan.howell.developm...@gmail.com
>> > wrote:
>>
>>> Hi All,
>>>
>>> I seem to be having a really tough time getting Knox to work with a
>>> secure NiFi cluster set up. I have tried to get this working two different
>>> ways. Both ways have basically the same set up for knoxsso, where it uses
>>> cloud foundry UAA as an external identity provider (currently configured
>>> for OpenID, with the /.well-known/openid-configuration prepended to the
>>> UAA instance url). I'm not sure if OpenID connect is the correct way to go,
>>> I believe there are other options with UAA; this is just the route I went
>>> as I initially was going to configure NiFi OpenID properties with my UAA
>>> instance. I have since decided (based on other factors) that Knox would be
>>> a better way to go. I have been focusing on option 1 below, as I think this
>>> is the preferred way. However, I tried option 2 below just to see if I
>>> could get around the error temporarily. I've included the errors I am
>>> running into below as well as relevant config. Any help is greatly
>>> appreciated.
>>>
>>> versions: NiFi 1.6 and Knox 1.1.0
>>>
>>> *1. Users will always access NiFi thru Knox (preferred)*
>>> *Issue Facing: Getting "PKIX path building failed: unable to find valid
>>> certification path to requested target"*
>>>
>>> *knoxsso.xml*
>>> 
>>>   
>>> 
>>> webappsec
>>> WebAppSec
>>> true
>>> xframe.options.enabledtrue>> >
>>> 
>>> 
>>> federation
>>> pac4j
>>> true
>>> 
>>> pac4j.session.store
>>> J2ESessionStore
>>> 
>>> 
>>>   pac4j.callbackUrl
>>>   https://my-knox-host:8443/gateway/knoxsso/api/v1/websso
>>> 
>>> 
>>> 
>>>   clientName
>>>   OidcClient
>>> 
>>> 
>>>   oidc.id
>>>   some_client_id
>>> 
>>> 
>>>   oidc.secret
>>>   some_client_secret
>>> 
>>> 
>>>   oidc.discoveryUri
>>>   https://my-uaa-host:443/.well-known/openid-configurat
>>> ion
>>> 
>>> 
>>>   oidc.preferredJwsAlgorithm
>>>   RS256
>>> 
>>> 
>>> 
>>>
>>> 
>>>   knoxauth
>>> 
>>> 
>>> KNOXSSO
>>> 
>>> knoxsso.cookie.secure.only
>>> false
>>> 
>>> 
>>> knoxsso.enable.session
>>> true
>>> 
>>> 
>>> knoxsso.cookie.max.age
>>> session
>>> 
>>> 
>>> knoxsso.token.ttl
>>> 360
>>> 
>>> 
>>>knoxsso.redirect.whitelist.regex
>>>^https?:\/\/(localhost|10\.227\.85\.2|[0-9]{1,3}\.[0
>>> -9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}||127\.0\.0\.1|0:0:0:0:0:0:
>>> 0:1|::1):[0-9].*$
>>> 
&

Re: webhdfs commands fails randomly with Failed to rewrite URL

[Sandeep Moré]

Hello Ashok,

What version of Knox are you using ? I remember we had a bug with older
Knox version where it would fail to properly rewrite when there was a
special character in the url.

Looking at your failed url
 https://:8443/gateway/default/
*webhdfs/data/v1/webhdfs/v1/user/*


It appears to be malformed. Let me know if that's the case.

Best,
Sandeep

On Thu, Mar 1, 2018 at 6:39 AM, Ashok V Jose  wrote:

> Hi Knox,
>
> We have this program which creates 100 users and they keep running and
> keeps creating different files to hdfs with these commands.
> url='https:// address>:8443/gateway/default/webhdfs/v1/user//test17/'
> + filetag + '_' + str(num) + '.txt?op=CREATE=true'
> req=requests.put(url,verify=False,auth=('','<
> password>'),allow_redirects=False,headers=headers)
>
>
> We see that randomly some of the files are not getting created and we dont
> see any network issues or configuration issues.
>
>
> Here is the snapshot this error message in /var/log/knox/gateway.log when
> the file thread85_49.txt is not getting created
>
> 2018-02-28 06:51:28,752 DEBUG realm.AuthenticatingRealm
> (AuthenticatingRealm.java:cacheAuthenticationInfoIfPossible(507)) -
> AuthenticationInfo caching is disabled for info . Submitted token:
> [org.apache.shiro.authc.UsernamePasswordToken - ,
> rememberMe=false ()].
> 2018-02-28 06:51:28,752 DEBUG authc.AbstractAuthenticator
> (AbstractAuthenticator.java:authenticate(233)) - Authentication
> successful for token [org.apache.shiro.authc.UsernamePasswordToken -
> , rememberMe=false ()]. Returned account
> 2018-02-28 06:51:28,752 DEBUG support.DefaultSubjectContext
> (DefaultSubjectContext.java:resolveSecurityManager(102)) - No
> SecurityManager available in subject context map. Falling back to
> SecurityUtils.getSecurityManager() lookup.
> 2018-02-28 06:51:28,752 DEBUG support.DefaultSubjectContext
> (DefaultSubjectContext.java:resolveSecurityManager(102)) - No
> SecurityManager available in subject context map. Falling back to
> SecurityUtils.getSecurityManager() lookup.
> 2018-02-28 06:51:28,750 ERROR hadoop.gateway 
> (UrlRewriteProcessor.java:rewrite(169))
> - Failed to rewrite URL: https://:8443/gateway/default/
> webhdfs/data/v1/webhdfs/v1/user//test17/thread34_46.txt?_=
> CBDwym5KTMcDaSWnUImQMyca186758MnkmX8hJmrNhyJ-5aFTi-
> WHglgqcTg5g9LDKyJHZPChwSXP3j9WTVlqvWp_jZv6aUTKh6YVAhBxTJ4MEoUiYA_
> 02ni5Hy8m6ZHU3kHanwLDvDIQ8cO02fYOy-oMqOyRP4l9Poc1bA3Lu8nu3juMddNE
> 7Mzy4EbgzM2uGuFRSqwUZ6OfOXA0nw1xw2lhen6r8F-63s_oC02yywOcVyHF-
> kes9HVe9S6BNpAQznEXOU5u7M1Imi7jhlXasG75YtbrfPbhrPnVlBE8M8a3-
> RTVTnZ9_fgAcJsn7U5-Dq07wXWkLRcfF4g6gQwDygHaWJ-j9dt66dt
> ,
> direction: IN via rule: WEBHDFS/webhdfs/inbound/datanode, status: FAILURE
> 2018-02-28 06:51:28,746 DEBUG hadoop.gateway 
> (UrlRewriteProcessor.java:rewrite(166))
> - Rewrote URL: https://:8443/gateway/default/webhdfs/v1/user/ Name>/test17/thread85_49.txt?op=CREATE=true
> ,
> direction: IN via explicit rule: WEBHDFS/webhdfs/inbound/namenode/file to
> URL: http://:50070/webhdfs/v1/user/ Name>/test17/thread85_49.txt?op=CREATE=true
> 
> 2018-02-28 06:51:28,745 DEBUG ssl.SslConnection
> (SslConnection.java:fill(526)) - 
> SslConnection@166efb61{NEED_UNWRAP,eio=0/-1,di=-1}
> -> HttpConnection@264c413e[FILLING,DecryptedEndPoint@2bf37c3a{/ Address>:39888<->8443,Open,in,out,-,-,350/30,HttpConnection}->
> SelectChannelEndPoint@7e35950c{/:39888<->8443,Open,in,
> out,-,-,12/30,SslConnection}{io=0,kio=0,kro=1}][p=HttpParser{s=START,0
> of 
> 0},g=HttpGenerator{s=START},c=HttpChannelOverHttp@7575b295{r=0,c=false,a=IDLE,uri=}]
> unwrap Status = BUFFER_UNDERFLOW HandshakeStatus = NEED_UNWRAP
>
>
> We had authentication set with ldap like this
>
> param>
> main.ldapRealm
> org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm
> 
> 

Re: idleTimeout values are getting reset

[Sandeep Moré]

Hello Ashok,

This is interesting, Knox opens up two connections for Websockets, one on
the client side (browser/client etc.) and other on the backend (service
host).
The timeouts you are seeing are for the two
1. First debug line - WebSocketServerConnection - opens up websocket
connection to the backend
2. Second debug line - WebSocketClientConnection - opens up websocket to
the browser

As far as the timeouts are concerned the shortest one wins (and can be
configured), in this case 98, since the connection will be closed and the
error value relayed (or should be).

I am curious why you are reducing the timeout value if you are getting
timeout errors, shouldn't you be increasing them, or perhaps you are just
debugging.

Best,
Sandeep



On Thu, Apr 19, 2018 at 7:04 AM, Ashok V Jose  wrote:

> Hi Knox Experts,
>
> Needed a help incase you are aware of this issue.
>
> We have this scenario where in we are creating 20 python kernels
> concurrently, we see that of these around 6-7 kernels fail with "Exception:
> Remote host closed websocket" because of timeout.
>
> When we tried the same testcase outside of the knox, we did not see the
> issue. All the kernels got created ok.
>
> So we suspected something in knox which is causing the issue, since the
> failure is because of the timeout imposed, from the ambari console we
> changed these 2 parameters  “gateway.idle.timeout" and
> "gateway.websocket.idle.timeout" to different values and ran the testcase.
>
> However we see in the knox logs that although we have set the
> "gateway.websocket.idle.timeout" to 98 it is being reset to 30.
>
> We see that idleTimeout is 98 , however it resets and becomes 30 later
> on.
>
> So basically it looks like whatever we set is being overwritten to 30,
> is this something you would know ?
>
> Here are the 2 lines which shows the values of idleTimeout to 98 and
> 30.
>
> 2018-04-13 09:17:38,266 DEBUG events.AbstractEventDriver
> (AbstractEventDriver.java:openSession(223)) -
> openSession(WebSocketSession[websocket=JettyListenerEventDriver[org.
> apache.hadoop.gateway.websockets.ProxyWebSocketAdapter],
> behavior=SERVER,connection=WebSocketServerConnection@3016d0df[IDLE,
> DecryptedEndPoint@65fc705{/xx.xxx.xxx.xxx:x<->8443,Open,
> in,out,-,-,116/98,WebSocketServerConnection}->
> SelectChannelEndPoint@2dd64f54{/xx.xxx.xxx.xxx:x<->8443,
> Open,in,out,-,-,1/98,SslConnection}{io=0,kio=0,kro=
> 1}]{f=Flusher[queueSize=0,aggregateSize=0,failure=null],
> g=Generator[SERVER,validating],p=Parser@faa724f[
> ExtensionStack,s=START,c=0,len=0,f=null,p=WebSocketPolicy@430a35ac[
> behavior=SERVER,maxTextMessageSize=2147483647,maxTextMessageBufferSize=
> 32768,maxBinaryMessageSize=2147483647,maxBinaryMessageBufferSize=
> 32768,asyncWriteTimeout=6,idleTimeout=98,inputBufferSize=4096]]},
> remote=WebSocketRemoteEndpoint@63617fae[batching=true],incoming=
> JettyListenerEventDriver[org.apache.hadoop.gateway.websockets.
> ProxyWebSocketAdapter],outgoing=ExtensionStack[queueSize=0,extensions=[],
> incoming=org.eclipse.jetty.websocket.common.WebSocketSession,outgoing=org.
> eclipse.jetty.websocket.server.WebSocketServerConnection]])
>
> 2018-04-13 09:17:40,535 DEBUG component.ContainerLifeCycle
> (ContainerLifeCycle.java:addBean(324)) - WebSocketSession[websocket=
> JsrAnnotatedEventDriver[websocket=org.apache.hadoop.gateway.websockets.
> ProxyInboundSocket@66217bd6],behavior=CLIENT,connection=
> WebSocketClientConnection@1fcb3472[IDLE,SelectChannelEndPoint@627d9a4b{
> chs-ksj-013-mn003.bi.services.us-south.bluemix.net/172.16.12.7:
> <->38356,Open,in,out,-,-,1/30,UpgradeConnection}{io=0,kio=0,
> kro=1}]{f=Flusher[queueSize=0,aggregateSize=0,failure=null],
> g=Generator[CLIENT,validating],p=Parser@1eafcb18[
> ExtensionStack,s=START,c=0,len=0,f=null,p=WebSocketPolicy@4e802544[
> behavior=CLIENT,maxTextMessageSize=2147483647,maxTextMessageBufferSize=
> 32768,maxBinaryMessageSize=2147483647,maxBinaryMessageBufferSize=
> 32768,asyncWriteTimeout=6,idleTimeout=30,inputBufferSize=4096]]},
> remote=null,incoming=JsrAnnotatedEventDriver[websocket=org.apache.hadoop.
> gateway.websockets.ProxyInboundSocket@66217bd6],outgoing=ExtensionStack[
> queueSize=0,extensions=[],incoming=org.eclipse.jetty.
> websocket.jsr356.JsrSession,outgoing=org.eclipse.jetty.websocket.client.io
> .WebSocketClientConnection]] added {ExtensionStack[queueSize=0,
> extensions=[],incoming=org.eclipse.jetty.websocket.
> jsr356.JsrSession,outgoing=org.eclipse.jetty.websocket.client.io.
> WebSocketClientConnection],AUTO}
>
> This are the versions  : Apache Knox: 0.12.0.2.6.2.0-205 (
> c8ad94e5bd29a6925f5a667660a50cfc6de69cc3)
>   HDP-2.6.2.0-205
>
> Please let me know if anything more would help
>
>
> Thanks & Regards,
> Ashok Jose
>
> **
>
> BigInsights Quality
> EGL C Block, 6th Floor
> IBM India Software Labs,
> Mail-Id : 

Re: Job history ui rewrite rule in yarnui

[Sandeep Moré]

Hello Guang,

For the first url, you cannot rewrite it because it does not have the
host:port, so this is different than the host:port your service is using ?
sorry, I don't remember exactly this works.

For the second case, yes I have see that used, but do not remember how we
route it, I think it get's routed to a different service, you can check the
rewrite.xml for details.

Best,
Sandeep


On Thu, Apr 19, 2018 at 7:12 PM, Guang Yang <k...@uber.com> wrote:

> Hi guys,
>
> About the job urls, I found three more non-functional links. They are
> links for jobmanager.(err/log/out) in the page of node container logs, like
> http://hadoopworker001:8042/node/containerlogs/container_xxx/user_xxx.
>
> Their original urls are like this http://hadoopworker1677-sjc1.prod.uber.internal:8042/node/containerlogs/container_e535_1521074600645_578887_01_01/kedar/jobmanager.err/?start=-4096>
> ">jobmanager.err : Total file length is 0 bytes.. As there is no node
> host and port info in this uri, so when doing the rewrite, it can't be
> written to the right url. Any suggestion about how to fix this issue?
>
> Another interesting finding is, actually we have the node host and port
> info in the web page url which contains these links, which is like "
> https://knox/gateway/sandbox/yarn/nodemanager/node/
> containerlogs/container_xxx/user_xxx?scheme=http=
> hadoopworker001-sjc1.prod.uber.internal=8042". So that means if
> there is a way we can get the parameter from the root url when rewrite some
> other urls, we can probably fix that problem.
>
> Do you have some suggestion? Appreciated.
>
> Best,
> Guang
>
> On Mon, Mar 5, 2018 at 5:07 PM, Guang Yang <k...@uber.com> wrote:
>
>> Any suggestion, guys?
>>
>> On Fri, Mar 2, 2018 at 1:28 PM, Guang Yang <k...@uber.com> wrote:
>>
>>> Hi Sandeep,
>>>
>>> Thanks for looking into it.
>>>
>>> Yes, I tried that before, the value can be replaced correctly, but the
>>> webpage is just stuck there, no redirect happens.
>>>
>>> The result in html is >> content="https://host:port/something;>, which however, should be >> http-equiv="refresh" content="*1; url=*https://host:port/something;>.
>>> It seems the html meta refresh format requires *'1; url='*, where the 1
>>> means stop for 1 seconds before refresh.
>>>
>>> Thanks,
>>> Guang
>>>
>>> On Fri, Mar 2, 2018 at 6:49 AM, Sandeep Moré <moresand...@gmail.com>
>>> wrote:
>>>
>>>> Hello Guang,
>>>>
>>>> This does look like a bug, after some digging it appears it was as a
>>>> result of KNOX-973.
>>>>
>>>> Have you tried using
>>>>
>>>> 
>>>>
>>>> I am curious to see what you get.
>>>>
>>>>
>>>> Best,
>>>> Sandeep
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Mar 1, 2018 at 4:30 PM, Guang Yang <k...@uber.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I'm currently working on the Map Reduce Job History UI rewrite rules,
>>>>> and found several potential bugs here.
>>>>>
>>>>> 
>>>>> For this rewrite template
>>>>> <https://github.com/apache/knox/blob/master/gateway-service-definitions/src/main/resources/services/yarnui/2.7.0/rewrite.xml#L104>,
>>>>> let's not say what the `jobstory` is here for now. I think the target
>>>>> url should start with something like {$frontend[url]}, just like other OUT
>>>>> rules, because the previous one doesn't specify the deployment/environment
>>>>> after word `gateway`.
>>>>>
>>>>> But after I change it to , the variable
>>>>> {$frontend[url]} is not replaced with the right value, it's just literal
>>>>> `{$frontend[url]}` in the target url. And I found that only when the
>>>>> variable following the double quotes, it can be replaced, otherwise it 
>>>>> just
>>>>> stays there as literal text.
>>>>>
>>>>> My question is, anyone knows how to fix this bug? Or how to get 
>>>>> {$frontend[url]}
>>>>> replaced with right value even it's not at the beginning of the template?
>>>>>
>>>>> Btw, I think the right template should be .
>>>>>
>>>>> Appreciate for any help.
>>>>>
>>>>> Thanks,
>>>>> Guang
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Re: NodeJS App through Knox

[Sandeep Moré]

Hello Mike,

You can find the information about configuring Websockets in the docs -
https://knox.apache.org/books/knox-1-0-0/user-guide.html#Websocket+Support
It also has an example rewrite.xml and service.xml rule.

Let us know how it goes.

Best,
Sandeep

On Sun, Mar 18, 2018 at 5:47 PM, Mike Mester  wrote:

> As part of my Hadoop infrastructure I have a node frontend.  Part of the
> app is served up by NGINX and Knox rewrite rules work fine for this.  The
> challenge is that the application is an AngularJS application that connects
> to a backend data provider on a tcp port (3123 in this case).  I have read
> through and have the web portion of the interface working per the guide
> found here.  (https://cwiki.apache.org/confluence/display/KNOX/
> Proxying+a+UI+using+Knox).  I dont see any information on how to handle a
> web socket application however.  Any help would be appreciated.
>
>
>

Re: KnoxSSO OpenID Error: Unable to renew the session. The session store may not support this feature

[Sandeep Moré]

Hello Ryan,

Can you check the value of the State parameter ? you should see it in the
request and response.

The state attribute is stored in the session so if is not getting properly
passed we might have an issue there (KNOX-1190 is tracking this), the
current implementation of SessionStore for Pac4J is a bit limited.

Can you try adding the following to your knoxsso.xml topology with the
current Knox build (1.1.0, you will have to build it from source as this
will not work with the current 1.0.0 release)

  
  pac4j.session.store
  J2ESessionStore
  

I am hoping this should save the "state" variable in the in-memory session
store.

Let us know what you find !

Also, what ODIC are you trying to integrate to, just curious to know what
all works with Knox.


On Sun, Mar 4, 2018 at 11:47 AM, Ryan H 
wrote:

> Hi Knox Users,
>
> I am rethreading this error I am getting as I mentioned it in a different
> thread that was about a different error (sorry to those active on the other
> email thread).
>
> I am running into an issue with KnoxSSO with the pac4j OIDC federation
> provider. When accessing the gateway, I am correctly redirected to my
> configured OpenID provider and upon successful authentication, redirected
> back to Knox but resulting in error. I am posting the relevant config files
> as well as the errors below. I have switched over to testBasicAuth just to
> confirm that I can connect to the NiFi app, which I can. I am not really
> sure where to go from here. I have sifted the internet and Knox
> documentation on this and haven't been able to find anything. I did find
> some info on this error with play and pac4j with the way the session was
> being handled and assumed that Knox would handle this (if not, it is not
> documented that I can find). Any help is appreciated!
>
> Cheers,
>
> Ryan
>
>
> *Error 1: *
> 2018-03-04 11:22:53,701 ERROR engine.DefaultCallbackLogic
> (DefaultCallbackLogic.java:renewSession(123)) - Unable to renew the
> session. The session store may not support this feature
>
> *Error 2:*
> 2018-03-04 10:07:05,578 ERROR knox.gateway 
> (AbstractGatewayFilter.java:doFilter(69))
> - Failed to execute filter: org.pac4j.core.exception.TechnicalException:
> State parameter is different from the one sent in authentication request.
> Session expired or possible threat of cross-site request forgery
> 2018-03-04 10:07:05,578 ERROR knox.gateway (GatewayFilter.java:doFilter(177))
> - Gateway processing failed: javax.servlet.ServletException:
> org.pac4j.core.exception.TechnicalException: State parameter is different
> from the one sent in authentication request. Session expired or possible
> threat of cross-site request forgery
> javax.servlet.ServletException: org.pac4j.core.exception.TechnicalException:
> State parameter is different from the one sent in authentication request.
> Session expired or possible threat of cross-site request forgery
> at org.apache.knox.gateway.filter.AbstractGatewayFilter.doFilter(
> AbstractGatewayFilter.java:70)
> at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(
> GatewayFilter.java:377)
> at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(
> GatewayFilter.java:277)
> at org.apache.knox.gateway.webappsec.filter.XFrameOptionsFilter.doFilter(
> XFrameOptionsFilter.java:58)
> at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(
> GatewayFilter.java:377)
> at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(
> GatewayFilter.java:277)
> at org.apache.knox.gateway.GatewayFilter.doFilter(GatewayFilter.java:171)
> at org.apache.knox.gateway.GatewayFilter.doFilter(GatewayFilter.java:94)
> at org.apache.knox.gateway.GatewayServlet.service(GatewayServlet.java:141)
> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
> at org.eclipse.jetty.servlet.ServletHandler.doHandle(
> ServletHandler.java:587)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:143)
> at org.eclipse.jetty.security.SecurityHandler.handle(
> SecurityHandler.java:577)
> at org.eclipse.jetty.server.session.SessionHandler.
> doHandle(SessionHandler.java:223)
> at org.eclipse.jetty.server.handler.ContextHandler.
> doHandle(ContextHandler.java:1127)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(
> ServletHandler.java:515)
> at org.eclipse.jetty.server.session.SessionHandler.
> doScope(SessionHandler.java:185)
> at org.eclipse.jetty.server.handler.ContextHandler.
> doScope(ContextHandler.java:1061)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:141)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(
> ContextHandlerCollection.java:215)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:97)
> at org.apache.knox.gateway.trace.TraceHandler.handle(TraceHandler.java:51)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> 

Re: KnoxSSO OpenID Error: Unable to renew the session. The session store may not support this feature

[Sandeep Moré]

Great ! good to know.
Just as an FYI this patch will be part of KNOX-1190
<https://issues.apache.org/jira/browse/KNOX-1190> so you should be just
able to move to the release that includes this, hopefully 1.1.0, without
having to worry about the patch.

Best,
Sandeep

On Mon, Mar 5, 2018 at 10:24 AM, Ryan H <ryan.howell.developm...@gmail.com>
wrote:

> Patch worked! That did solve the session issue. Thanks for the patch.
>
> -Ryan
>
> On Mon, Mar 5, 2018 at 10:01 AM Ryan H <ryan.howell.developm...@gmail.com>
> wrote:
>
>> No worries! Just applied patch and rebuilding. I will follow up with the
>> verdict.
>>
>> -Ryan
>>
>> On Mon, Mar 5, 2018 at 9:45 AM, Sandeep Moré <moresand...@gmail.com>
>> wrote:
>>
>>> Ok, sorry, I thought the change that pick up the property "
>>> *J2ESessionStore*" were in master but they are not, I confused it with
>>> some other bug.
>>> Anyways, if you want you can try to apply the attached patch (to master)
>>> which enables Knox to pick up "*J2ESessionStore*"  property and see
>>> what you get !
>>>
>>> Again sorry about that.
>>>
>>> Best,
>>> Sandeep
>>>
>>>
>>>
>>> On Sun, Mar 4, 2018 at 8:45 PM, Ryan H <ryan.howell.development@
>>> gmail.com> wrote:
>>>
>>>> Just tried with the 1.1.0-SNAPSHOT build with same result; but with a
>>>> new warning now (might not be relevant, but including anyways). Same error
>>>> as before if going incognito browser.
>>>>
>>>> Here is the error that I now get:
>>>>
>>>> 2018-03-04 20:37:50,653 ERROR engine.DefaultCallbackLogic
>>>> (DefaultCallbackLogic.java:renewSession(123)) - Unable to renew the
>>>> session. The session store may not support this feature
>>>> 2018-03-04 20:39:10,899 WARN  federation.jwt 
>>>> (AbstractJWTFilter.java:validateToken(305))
>>>> - Failed to verify the token signature.
>>>>
>>>> *Updated knoxsso.xml file:*
>>>> 
>>>>   
>>>> 
>>>> webappsec
>>>> WebAppSec
>>>> true
>>>> xframe.options.enabledtrue>>> value>
>>>> 
>>>> 
>>>> federation
>>>> pac4j
>>>> true
>>>> **
>>>> *pac4j.session.store*
>>>> *J2ESessionStore*
>>>> **
>>>> 
>>>>   pac4j.callbackUrl
>>>>   https://localhost:8443/gateway/knoxsso/api/v1/websso<
>>>> /value>
>>>> 
>>>> 
>>>>   clientName
>>>>   OidcClient
>>>> 
>>>> 
>>>>   oidc.id
>>>>   my_client_id
>>>> 
>>>> 
>>>>   oidc.secret
>>>>   my_client_secret
>>>> 
>>>> 
>>>>   oidc.discoveryUri
>>>>   https://my-openid-provider:443/.well-known/
>>>> openid-configuration
>>>> 
>>>> 
>>>>   oidc.preferredJwsAlgorithm
>>>>   RS256
>>>> 
>>>> 
>>>> 
>>>>
>>>> 
>>>>   knoxauth
>>>> 
>>>>
>>>> 
>>>> KNOXSSO
>>>> 
>>>> knoxsso.cookie.secure.only
>>>> false
>>>> 
>>>> 
>>>> knoxsso.cookie.max.age
>>>> session
>>>> 
>>>> 
>>>> knoxsso.token.ttl
>>>> 3
>>>> 
>>>> 
>>>>knoxsso.redirect.whitelist.regex
>>>>^https?:\/\/(localhost|127\.0\.0\.1|0:0:0:
>>>> 0:0:0:0:1|::1):[0-9].*$
>>>> 
>>>> 
>>>>
>>>> 
>>>>
>>>>
>>>>
>>>> On Sun, Mar 4, 2018 at 8:26 PM, Sandeep Moré <moresand...@gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks for the update, I have a hunch that the "state" param is not
>>>>> getting passed to Pac4J via the SessionStore (Error 1).
>>>>> Keep us posted.
>>>>>
>>&g

Re: KnoxSSO OpenID Error: Unable to renew the session. The session store may not support this feature

[Sandeep Moré]

Ok, sorry, I thought the change that pick up the property "*J2ESessionStore*"
were in master but they are not, I confused it with some other bug.
Anyways, if you want you can try to apply the attached patch (to master)
which enables Knox to pick up "*J2ESessionStore*"  property and see what
you get !

Again sorry about that.

Best,
Sandeep



On Sun, Mar 4, 2018 at 8:45 PM, Ryan H <ryan.howell.developm...@gmail.com>
wrote:

> Just tried with the 1.1.0-SNAPSHOT build with same result; but with a new
> warning now (might not be relevant, but including anyways). Same error as
> before if going incognito browser.
>
> Here is the error that I now get:
>
> 2018-03-04 20:37:50,653 ERROR engine.DefaultCallbackLogic
> (DefaultCallbackLogic.java:renewSession(123)) - Unable to renew the
> session. The session store may not support this feature
> 2018-03-04 20:39:10,899 WARN  federation.jwt 
> (AbstractJWTFilter.java:validateToken(305))
> - Failed to verify the token signature.
>
> *Updated knoxsso.xml file:*
> 
>   
> 
> webappsec
> WebAppSec
> true
> xframe.options.enabledtrue value>
> 
> 
> federation
> pac4j
> true
> **
> *pac4j.session.store*
> *J2ESessionStore*
> **
> 
>   pac4j.callbackUrl
>   https://localhost:8443/gateway/knoxsso/api/v1/websso<
> /value>
> 
> 
>   clientName
>   OidcClient
> 
> 
>   oidc.id
>   my_client_id
> 
> 
>   oidc.secret
>   my_client_secret
> 
> 
>   oidc.discoveryUri
>   https://my-openid-provider:443/.well-known/
> openid-configuration
> 
> 
>   oidc.preferredJwsAlgorithm
>   RS256
> 
> 
> 
>
> 
>   knoxauth
> 
>
> 
> KNOXSSO
> 
> knoxsso.cookie.secure.only
> false
> 
> 
> knoxsso.cookie.max.age
> session
> 
> 
> knoxsso.token.ttl
> 3
> 
> 
>knoxsso.redirect.whitelist.regex
>^https?:\/\/(localhost|127\.0\.0\.1|0:0:0:
> 0:0:0:0:1|::1):[0-9].*$
> 
> 
>
> 
>
>
>
> On Sun, Mar 4, 2018 at 8:26 PM, Sandeep Moré <moresand...@gmail.com>
> wrote:
>
>> Thanks for the update, I have a hunch that the "state" param is not
>> getting passed to Pac4J via the SessionStore (Error 1).
>> Keep us posted.
>>
>> Thanks !
>>
>> On Sun, Mar 4, 2018 at 7:40 PM, Ryan H <ryan.howell.developm...@gmail.com
>> > wrote:
>>
>>> Hey Guys,
>>>
>>> I will pull 1.1.0 this evening and give it a go.
>>>
>>> A little more information on the flow for the error(s):
>>>
>>> When I log in with regular browser (not incognito), I get the following
>>> error:
>>> ERROR engine.DefaultCallbackLogic 
>>> (DefaultCallbackLogic.java:renewSession(123))
>>> - Unable to renew the session. The session store may not support this
>>> feature
>>>
>>> When I try to log in using an incognito browser, I get the following
>>> error:
>>> ERROR knox.gateway (AbstractGatewayFilter.java:doFilter(69)) - Failed
>>> to execute filter: org.pac4j.core.exception.TechnicalException: State
>>> parameter is different from the one sent in authentication request. Session
>>> expired or possible threat of cross-site request forgery
>>> 2018-03-04 19:30:42,151 ERROR knox.gateway 
>>> (GatewayFilter.java:doFilter(177))
>>> - Gateway processing failed: javax.servlet.ServletException:
>>> org.pac4j.core.exception.TechnicalException: State parameter is
>>> different from the one sent in authentication request. Session expired or
>>> possible threat of cross-site request forgery
>>> javax.servlet.ServletException: org.pac4j.core.exception.TechnicalException:
>>> State parameter is different from the one sent in authentication request.
>>> Session expired or possible threat of cross-site request forgery
>>> at org.apache.knox.gateway.filter.AbstractGatewayFilter.doFilte
>>> r(AbstractGatewayFilter.java:70)
>>> ...
>>>
>>> As for the state param:
>>> When I log in using a regular browser (not incognito): the state param
>>> is the same.
>>>
>>> When I use the incognito browser: the state param is different from the
>>> OpenID provider and Knox (w

Re: KNOX Pac4j Azure AD Open ID

[Sandeep Moré]

Hello Jérôme
You are right about the client_name and pac4JCallback parameters.
I do not think this is because of misconfiguration, the issue in this case
is that Azure AD does not support query params in the callback URLs.

Best,
Sandeep

On Mon, Feb 26, 2018 at 1:42 AM, Jérôme LELEU <lel...@gmail.com> wrote:

> Hi,
>
> The callback URL sent by the IdP does not have the client_name parameter,
> nor the pac4jCallback parameter (the parameter has changed between
> versions).
>
> So the callback is not properly detected and an authentication is tried
> over and over again.
>
> So I guess there is a misconfiguration on the IdP side at the callback URL
> level.
>
> Thanks.
> Best regards,
> Jérôme
>
>
> On Wed, Feb 21, 2018 at 4:45 AM, Nisha Menon <nisha.meno...@gmail.com>
> wrote:
>
>> Hi Jerome.
>>
>> PFA the logs after enabling the pac4j logs. Hope this helps.
>>
>> Regards
>> Nisha
>>
>> On Tue, Feb 20, 2018 at 7:50 PM, Jérôme LELEU <lel...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Logs look good in pac4j. Redirecting to the identity provider and then
>>> coming back to the gateway. Maybe the user identity is not properly
>>> created...
>>>
>>> Can you turn on DEBUG logs on org.pac4j and org.apache.knox.gateway.pa
>>> c4j?
>>>
>>> Thanks.
>>> Best regards,
>>> Jérôme
>>>
>>>
>>> On Mon, Feb 19, 2018 at 6:49 PM, Colm O hEigeartaigh <
>>> cohei...@apache.org> wrote:
>>>
>>>> I can reproduce the issue with Google. From the logs I see the
>>>> following:
>>>>
>>>> 2018-02-19 17:21:58,484 DEBUG session.KnoxSessionStore
>>>> (KnoxSessionStore.java:get(91)) - Get from session: pac4jRequestedUrl
>>>> = https://localhost:8443/gateway/knoxssopac4j/api/v1/websso?or
>>>> iginalUrl=https://localhost:8443/gateway/sandbox-ssopac4j/we
>>>> bhdfs/v1/data/LICENSE.txt?op=OPEN
>>>> 2018-02-19 17:21:58,485 DEBUG session.KnoxSessionStore
>>>> (KnoxSessionStore.java:set(107)) - Save in session: pac4jRequestedUrl
>>>> = null
>>>> 2018-02-19 17:21:58,485 DEBUG engine.DefaultCallbackLogic
>>>> (DefaultCallbackLogic.java:redirectToOriginallyRequestedUrl(137)) -
>>>> redirectUrl: https://localhost:8443/gateway
>>>> /knoxssopac4j/api/v1/websso?originalUrl=https://localhost:84
>>>> 43/gateway/sandbox-ssopac4j/webhdfs/v1/data/LICENSE.txt?op=OPEN
>>>> 2018-02-19 17:21:58,488 DEBUG knox.gateway
>>>> (GatewayFilter.java:doFilter(119)) - Received request: GET
>>>> /api/v1/websso
>>>>
>>>> It is getting an access token correctly and trying to redirect back to
>>>> the original URL. However from the logs above it seems to never hit the
>>>> original URL but instead hits the "redirectUrl". Is some parsing supposed
>>>> to take place to extract "originalUrl" from the "pac4jRequestedUrl"
>>>> parameter and redirect to this instead?
>>>>
>>>> Colm.
>>>>
>>>> On Mon, Feb 19, 2018 at 4:16 PM, larry mccay <lmc...@apache.org> wrote:
>>>>
>>>>> No, the hadoop-jwt cookie is for KnoxSSO and the SSOCookieProvider.
>>>>> If it isn't being set, it could be that it isn't getting past the
>>>>> pac4j federation provider to the KnoxSSO service itself because there is a
>>>>> redirect from the pac4j provider or the Set-Cookie just isn't being
>>>>> accepted by the browser.
>>>>>
>>>>> The audit log does seem to be getting a redirect from pac4j.
>>>>>
>>>>> I haven't seen any examples of it working AAD - so we are in
>>>>> unchartered waters here.
>>>>>
>>>>> @Jerome - any insights?
>>>>>
>>>>> On Mon, Feb 19, 2018 at 2:04 AM, Nisha Menon <nisha.meno...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hello Larry,
>>>>>>
>>>>>> hadoop-jwt cookie is not set. Isnt this for JWT provider?
>>>>>> In SSO provider with pac4j, I can see cookies like:
>>>>>> pac4j.session.pac4jRequestedUrl, pac4j.session.oidcStateAttribute,
>>>>>> pac4j.session.oidcNonceAttribute etc.
>>>>>>
>>>>>> IP address and hostnames are mapped, else the *basic auth* also
>>>>>> would have failed.
>>>>>> My issue i

[DISCUSS] KIP-10 KnoxSSO Logout Flow

[Sandeep Moré]

Hello All,

I am kickstarting a discussion about KnoxSSO logout flow, I created a KIP
page for it
https://cwiki.apache.org/confluence/display/KNOX/KIP-10+KnoxSSO+Logout+Flow

In the coming week I am planning on adding more sections for CAS, OpenID
connect etc. as I research them more. In the mean time, if you could take a
look at it and provide feedback/comments/suggestions that would be
extremely helpful. Also, please feel free to let me know of any use case
you might think is useful and should be added.

Best,
Sandeep

Re: Job history ui rewrite rule in yarnui

[Sandeep Moré]

Hello Guang,

This does look like a bug, after some digging it appears it was as a result
of KNOX-973.

Have you tried using



I am curious to see what you get.


Best,
Sandeep




On Thu, Mar 1, 2018 at 4:30 PM, Guang Yang  wrote:

> Hi,
>
> I'm currently working on the Map Reduce Job History UI rewrite rules, and
> found several potential bugs here.
>
> 
> For this rewrite template
> ,
> let's not say what the `jobstory` is here for now. I think the target url
> should start with something like {$frontend[url]}, just like other OUT
> rules, because the previous one doesn't specify the deployment/environment
> after word `gateway`.
>
> But after I change it to , the variable
> {$frontend[url]} is not replaced with the right value, it's just literal
> `{$frontend[url]}` in the target url. And I found that only when the
> variable following the double quotes, it can be replaced, otherwise it just
> stays there as literal text.
>
> My question is, anyone knows how to fix this bug? Or how to get 
> {$frontend[url]}
> replaced with right value even it's not at the beginning of the template?
>
> Btw, I think the right template should be .
>
> Appreciate for any help.
>
> Thanks,
> Guang
>
>
>

Re: idleTimeout values are getting reset

[Sandeep Moré]

You can use the class ProxyInboundSocket.java, this was the one included
with the original commit.
In there find the onClientOpen i.e.

 @OnOpen
  public void onClientOpen(final javax.websocket.Session
backendSession) {


and update 'backendSession' as follows

backendSession.setMaxIdleTimeout(6 * 60 * 24);

I would highly recommend to upgrade since your custom settings will be lost
once you upgrade and it will be difficult to keep track as this class does
not exist any more.

Best,
Sandeep


On Thu, Apr 26, 2018 at 1:27 AM, Ashok V Jose <ashoj...@in.ibm.com> wrote:

> Hi Sandeep,
>
> The knox version (0.12.0.2.6.2.0-205) I am using did not have this file
> ProxyInboundClient.java
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_knox_blob_92e2ec59a5940a9e7c67ec5cd29044f811dee40a_gateway-2Dserver_src_main_java_org_apache_knox_gateway_websockets_ProxyInboundClient.java-23L61=DwMFaQ=jf_iaSHvJObTbx-siA1ZOg=IwA-Q1lswMrNoI5t1-MA_ksLuVkVEL61cf3yFoZAQhs=DdBwaq-A5CjsAq6juBCXi4pLgWL1KtMZOv3_IB7x0FQ=ZDrjApvcdgx2U9mY2vzxvz_20rLGqy0268qKih_frAc=>,
> is there a way I can put in somewhere in my verison ?
>
>
> Thanks & Regards,
> Ashok Jose
>
> **
>
> BigInsights Quality
> EGL C Block, 6th Floor
> IBM India Software Labs,
> Mail-Id : ashoj...@in.ibm.com
>
>
>
>
> - Original message -
> From: "Sandeep Moré" <moresand...@gmail.com>
> To: user@knox.apache.org
> Cc:
> Subject: Re: idleTimeout values are getting reset
> Date: Tue, Apr 24, 2018 12:02 AM
>
> Hello Ashok,
>
> Unfortunately, there is only one setting as of now, you can try adding
> session.setMaxIdleTimeout(98);
> at https://github.com/apache/knox/blob/92e2ec59a5940a9e7c67ec5cd29044
> f811dee40a/gateway-server/src/main/java/org/apache/knox/
> gateway/websockets/ProxyInboundClient.java#L61
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_knox_blob_92e2ec59a5940a9e7c67ec5cd29044f811dee40a_gateway-2Dserver_src_main_java_org_apache_knox_gateway_websockets_ProxyInboundClient.java-23L61=DwMFaQ=jf_iaSHvJObTbx-siA1ZOg=IwA-Q1lswMrNoI5t1-MA_ksLuVkVEL61cf3yFoZAQhs=DdBwaq-A5CjsAq6juBCXi4pLgWL1KtMZOv3_IB7x0FQ=ZDrjApvcdgx2U9mY2vzxvz_20rLGqy0268qKih_frAc=>
>
> and see if that helps.
>
> Best,
> Sandeep
>
>
> On Sun, Apr 22, 2018 at 9:17 PM, Ashok V Jose <ashoj...@in.ibm.com>
> wrote:
>
> Hi Sandeep,
>
> Is there a parameter that I can set the timeouts for both 
> WebSocketServerConnection
> and WebSocketClientConnection to a larger value from the knox settings ?
> One of these is gateway.websocket.idle.timeout, is there an other one ?
> Or are these set from the client program ?
>
>
> Thanks & Regards,
> Ashok Jose
>
> **
>
> BigInsights Quality
> EGL C Block, 6th Floor
> IBM India Software Labs,
> Mail-Id : ashoj...@in.ibm.com
>
>
>
>
> - Original message -
> From: "Sandeep Moré" <moresand...@gmail.com>
> To: user@knox.apache.org
> Cc:
> Subject: Re: idleTimeout values are getting reset
> Date: Thu, Apr 19, 2018 7:15 PM
>
> Hello Ashok,
>
> This is interesting, Knox opens up two connections for Websockets, one on
> the client side (browser/client etc.) and other on the backend (service
> host).
> The timeouts you are seeing are for the two
> 1. First debug line - WebSocketServerConnection - opens up websocket
> connection to the backend
> 2. Second debug line - WebSocketClientConnection - opens up websocket to
> the browser
>
> As far as the timeouts are concerned the shortest one wins (and can be
> configured), in this case 98, since the connection will be closed and the
> error value relayed (or should be).
>
> I am curious why you are reducing the timeout value if you are getting
> timeout errors, shouldn't you be increasing them, or perhaps you are just
> debugging.
>
> Best,
> Sandeep
>
>
>
> On Thu, Apr 19, 2018 at 7:04 AM, Ashok V Jose <ashoj...@in.ibm.com>
> wrote:
>
> Hi Knox Experts,
>
> Needed a help incase you are aware of this issue.
>
> We have this scenario where in we are creating 20 python kernels
> concurrently, we see that of these around 6-7 kernels fail with "Exception:
> Remote host closed websocket" because of timeout.
>
> When we tried the same testcase outside of the knox, we did not see the
> issue. All the kernels got created ok.
>
> So we suspected something in knox which is causing the issue, since the
> failure is because of the timeout imposed, from the ambari console we
> changed these 2 parameters  “gateway.idle.timeout" and
> 

Re: x-ndjson via Knox

[Sandeep Moré]

When you say it does not work what error do you get ? do you see the
rewrite rules for header getting triggered in the debug logs ?
Also, what is the exception you get on the elasticsearch side.

Best,
Sandeep

On Fri, Oct 5, 2018 at 11:23 AM rabii lamriq  wrote:

> Hi All,
>
> I tried to change the content-type of the http header in the
> ElasticsearchDispatcher java class, but doesn't work :(
>
> Anyone can give help where we should change the content-type for make
> "application/x-ndjson" in class of ElasticsearchDispatcher  (attached)
>
> Regards
> Rabii
>
> Le mar. 25 sept. 2018 à 16:20, rabii lamriq  a écrit :
>
>> I am not understand how to apply this for content-type.
>>
>> Le mar. 25 sept. 2018 à 15:18, Sandeep Moré  a
>> écrit :
>>
>>> This is an example in Webhdfs service def.
>>> https://github.com/apache/knox/blob/6e7266ad3649e1804ee869afed04f79a71e48b7d/gateway-service-definitions/src/main/resources/services/webhdfs/2.4.0/rewrite.xml#L64
>>>
>>>
>>>
>>> On Tue, Sep 25, 2018 at 5:37 AM rabii lamriq  wrote:
>>>
>>>> Hi Sandeep,
>>>>
>>>> Yes, I know that is possible, but I don't know how to rewrite the rule
>>>> to access to the header and change the Content-type, I can't find any thing
>>>> related to this customization.
>>>>
>>>> Regards
>>>> Rabii
>>>>
>>>> Le lun. 24 sept. 2018 à 17:19, Sandeep Moré  a
>>>> écrit :
>>>>
>>>>> Yes, you can use the rewrite rules to update Headers, I don't remember
>>>>> exactly which service uses it but there should be few services that 
>>>>> rewrite
>>>>> headers.
>>>>>
>>>>> Best,
>>>>> Sandeep
>>>>>
>>>>> On Mon, Sep 24, 2018 at 10:45 AM rabii lamriq 
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> Rewrite rules for knox (link
>>>>>> <https://cwiki.apache.org/confluence/display/KNOX/2017/08/14/Understanding+Rewrite+Rules+for+Apache+Knox#UnderstandingRewriteRulesforApacheKnox-JSONURLRewriteFIlter>)
>>>>>> can help me to change header? is it possible?
>>>>>>
>>>>>> Le lun. 24 sept. 2018 à 16:43, rabii lamriq  a
>>>>>> écrit :
>>>>>>
>>>>>>> Hello
>>>>>>>
>>>>>>> Any help please??
>>>>>>>
>>>>>>> Le lun. 24 sept. 2018 à 09:31, rabii lamriq  a
>>>>>>> écrit :
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> thanks every One, for your feedbacks,
>>>>>>>>
>>>>>>>> Yes, ES accept both application/json and application/x-ndjson,
>>>>>>>> there is no problem in ES ni in Grafana, when i test my query with 
>>>>>>>> curl to
>>>>>>>> send POST to ES, it work well ether json or x-ndjson.
>>>>>>>>
>>>>>>>> However, through Knox Gateway, it work only for x-ndjson, I don't
>>>>>>>> know why. so, since Grafana send their POST with only json and not
>>>>>>>> x-ndjson, the scenario of Grafana -> Knox -> ES with multi search 
>>>>>>>> doesn't
>>>>>>>> work.
>>>>>>>>
>>>>>>>> I think we have two solution:
>>>>>>>>
>>>>>>>> 1- make some thing to force Grafana change header of POST and put
>>>>>>>> x-ndjson instead of json, or
>>>>>>>> 2- rewrite Header by Knox to forwarded with x-ndjson to ES.
>>>>>>>>
>>>>>>>> If some one have any idea of how to make this or other solution.
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> Rabii
>>>>>>>>
>>>>>>>> Le ven. 21 sept. 2018 à 19:21, Kevin Risden  a
>>>>>>>> écrit :
>>>>>>>>
>>>>>>>>> https://github.com/elastic/elasticsearch/issues/25718
>>>>>>>>>
>>>>>>>>> Elasticsearch endpoints should take both application/json and
>>>>>>>>> application/x-ndjson. There might be something else going on here. 
>>>>&

Re: x-ndjson via Knox

[Sandeep Moré]

Ok, I think the problem here is that the request payload is getting
stripped for some reason.
Looking closely at your curl request it looks like you are using XPOST, try
using POST and see if it works.
I don't think Knox has support for XPOST.

Also, if POST works can you file a bug.

Best,
Sandeep

On Mon, Oct 8, 2018 at 7:22 AM rabii lamriq  wrote:

> Hi Sandeep
>
> The error is :
> {"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation
> Failed: 1: no requests
> added;"}],"type":"action_request_validation_exception","reason":"Validation
> Failed: 1: no requests added;"},"status":400}
>
> I don't find how to rewrite rules.
>
> Best
> Rabii
>
> Le ven. 5 oct. 2018 à 18:53, Sandeep Moré  a
> écrit :
>
>> When you say it does not work what error do you get ? do you see the
>> rewrite rules for header getting triggered in the debug logs ?
>> Also, what is the exception you get on the elasticsearch side.
>>
>> Best,
>> Sandeep
>>
>> On Fri, Oct 5, 2018 at 11:23 AM rabii lamriq  wrote:
>>
>>> Hi All,
>>>
>>> I tried to change the content-type of the http header in the
>>> ElasticsearchDispatcher java class, but doesn't work :(
>>>
>>> Anyone can give help where we should change the content-type for make
>>> "application/x-ndjson" in class of ElasticsearchDispatcher  (attached)
>>>
>>> Regards
>>> Rabii
>>>
>>> Le mar. 25 sept. 2018 à 16:20, rabii lamriq  a écrit :
>>>
>>>> I am not understand how to apply this for content-type.
>>>>
>>>> Le mar. 25 sept. 2018 à 15:18, Sandeep Moré  a
>>>> écrit :
>>>>
>>>>> This is an example in Webhdfs service def.
>>>>> https://github.com/apache/knox/blob/6e7266ad3649e1804ee869afed04f79a71e48b7d/gateway-service-definitions/src/main/resources/services/webhdfs/2.4.0/rewrite.xml#L64
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Sep 25, 2018 at 5:37 AM rabii lamriq  wrote:
>>>>>
>>>>>> Hi Sandeep,
>>>>>>
>>>>>> Yes, I know that is possible, but I don't know how to rewrite the
>>>>>> rule to access to the header and change the Content-type, I can't find 
>>>>>> any
>>>>>> thing related to this customization.
>>>>>>
>>>>>> Regards
>>>>>> Rabii
>>>>>>
>>>>>> Le lun. 24 sept. 2018 à 17:19, Sandeep Moré 
>>>>>> a écrit :
>>>>>>
>>>>>>> Yes, you can use the rewrite rules to update Headers, I don't
>>>>>>> remember exactly which service uses it but there should be few services
>>>>>>> that rewrite headers.
>>>>>>>
>>>>>>> Best,
>>>>>>> Sandeep
>>>>>>>
>>>>>>> On Mon, Sep 24, 2018 at 10:45 AM rabii lamriq 
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Rewrite rules for knox (link
>>>>>>>> <https://cwiki.apache.org/confluence/display/KNOX/2017/08/14/Understanding+Rewrite+Rules+for+Apache+Knox#UnderstandingRewriteRulesforApacheKnox-JSONURLRewriteFIlter>)
>>>>>>>> can help me to change header? is it possible?
>>>>>>>>
>>>>>>>> Le lun. 24 sept. 2018 à 16:43, rabii lamriq  a
>>>>>>>> écrit :
>>>>>>>>
>>>>>>>>> Hello
>>>>>>>>>
>>>>>>>>> Any help please??
>>>>>>>>>
>>>>>>>>> Le lun. 24 sept. 2018 à 09:31, rabii lamriq  a
>>>>>>>>> écrit :
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> thanks every One, for your feedbacks,
>>>>>>>>>>
>>>>>>>>>> Yes, ES accept both application/json and application/x-ndjson,
>>>>>>>>>> there is no problem in ES ni in Grafana, when i test my query with 
>>>>>>>>>> curl to
>>>>>>>>>> send POST to ES, it work well ether json or x-ndjson.
>>>>>>>>>>
>>

Re: WebHDFS performance issue in Knox

[Sandeep Moré]

I think this would be a good test, worth a try, not sure how we can force a
certain cipher to be used perhaps a permutation combination of
ssl.include.ciphers, ssl.exclude.ciphers.

Best,
Sandeep


On Tue, Oct 9, 2018 at 5:29 PM David Villarreal 
wrote:

> Hi Kevin,
>
>
>
> In my humble opinion, this has to do with cpu processing encryption in
> general based on which cipher being used.  Couldn’t the same type of
> principals/improvements (hdfs encryption improvements) be done here for
> let’s say for AES cipher suites?  If the main bottleneck here is CPU
> couldn’t you enhance encryption though hardware acceleration and you may
> see better performance numbers?
>
>
>
> https://calomel.org/aesni_ssl_performance.html
>
>
>
> Try forcing a less secure cipher to be used in your environment.  Do you
> then see better numbers?
>
>
>
> dav
>
>
>
>
>
> *From: *Kevin Risden 
> *Reply-To: *"user@knox.apache.org" 
> *Date: *Tuesday, October 9, 2018 at 1:05 PM
> *To: *"user@knox.apache.org" 
> *Subject: *Re: WebHDFS performance issue in Knox
>
>
>
> @David - Not sure what you mean since this is SSL/TLS and not related to
> RPC encryption like the two JIRAs that you linked.
>
> @Guang - NP just took some time to sit down and look at it.
>
>
>
> Some preliminary investigation shows this may be the JDK implementation of
> TLS/SSL that is slowing down the read path. I need to dig into it further
> but found a few references showing that Java slowness for TLS/SSL affects
> Jetty.
>
>-
>https://nbsoftsolutions.com/blog/the-cost-of-tls-in-java-and-solutions
>-
>https://nbsoftsolutions.com/blog/dropwizard-1-3-upcoming-tls-improvements
>- https://webtide.com/conscrypting-native-ssl-for-jetty/
>
> Locally testing off a Jetty 9.4 branch (for KNOX-1516), I was able to
> enable conscrypting (
> https://www.eclipse.org/jetty/documentation/9.4.x/configuring-ssl.html#conscrypt).
> With that I was able to get read performance on par with non ssl and native
> webhdfs. The write side of the equation still has some performance
> differences that need to be looked at further.
>
>
> Kevin Risden
>
>
>
>
>
> On Tue, Oct 9, 2018 at 2:01 PM Guang Yang  wrote:
>
> Thanks Kevin conducting such experiment! This is exactly what I saw
> before. It doesn't look right the download speed is 10x slower when
> enabling SSL.
>
>
>
> On Tue, Oct 9, 2018 at 10:40 AM David Villarreal <
> dvillarr...@hortonworks.com> wrote:
>
> I bring this up because HDFS encryption saw an increase in performance.
>
> https://issues.apache.org/jira/browse/HDFS-6606
>
>
>
> https://issues.apache.org/jira/browse/HADOOP-10768
>
>
>
> Maybe Knox can make some enhancements in this area?
>
>
>
> *From: *David Villarreal 
> *Date: *Tuesday, October 9, 2018 at 10:34 AM
> *To: *"user@knox.apache.org" 
> *Subject: *Re: WebHDFS performance issue in Knox
>
>
>
> Hi Kevin,
>
> Now increase your CPU processing power and show me the numbers.
>
>
>
> Do we support AES-NI optimization with extended CPU instruction set for
> AES hardware acceleration?
>
> libcrypto.so library that supports hardware acceleration, such as OpenSSL
> 1.0.1e. (Many OS versions have an older version of the library that does
> not support AES-NI.)
>
>
>
>
>
>
> *From: *
>
> *Kevin Risden*
>
>
>
>
>
> *> Reply-To: "user@knox.apache.org
> " > Date:
> Tuesday, October 9, 2018 at 10:26 AM To: "user@knox.apache.org
> " >
> Subject: Re: WebHDFS performance issue in Knox*
>
>
>
> Writes look to have performance impact as well:
>
>- directly to webhdfs - ~2.6 seconds
>- knox no ssl - ~29 seconds
>- knox ssl - ~49.6 seconds
>
> Kevin Risden
>
>
>
>
>
> On Tue, Oct 9, 2018 at 12:39 PM Kevin Risden  wrote:
>
> If I run two downloads concurrently:
>
>
>
> 1,073,741,824 46.1MB/s   in 22s
>
> 1,073,741,824 51.3MB/s   in 22s
>
>
>
> So it isn't a limitation of the Knox gateway itself in total bandwidth but
> a per connection limitation somehow.
>
>
> Kevin Risden
>
>
>
>
>
> On Tue, Oct 9, 2018 at 12:24 PM Kevin Risden  wrote:
>
> So I was able to reproduce a slowdown with SSL with a pseudo distributed
> HDFS setup on a single node with Knox running on the same node. This was
> setup in Virtualbox on my laptop.
>
>
>
> Rough timings with wget for a 1GB random file:
>
>- directly to webhdfs - 1,073,741,824  252MB/s   in 3.8s
>- knox no ssl - 1,073,741,824  264MB/s   in 3.6s
>- knox ssl - 1,073,741,824 54.3MB/s   in 20s
>
> There is a significant decrease with Knox SSL for some reason.
>
>
>
> Kevin Risden
>
>
>
>
>
> On Sun, Sep 23, 2018 at 8:53 PM larry mccay  wrote:
>
> SSL handshake will likely happen at least twice.
>
> Once for the request through Knox to the NN then the redirect from the NN
> to the DN goes all the way back to the client.
>
> So they have to follow the redirect and do the handshake to the DN.
>
>
>
>
>
> On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden  wrote:
>
> So I found this in the Knox issues list in JIRA:
>
>
>
> 

Re: WebHDFS performance issue in Knox

[Sandeep Moré]

Awesome, I had seen GCM suck big time in the past.
Great work !

On Wed, Oct 10, 2018 at 3:48 PM Kevin Risden  wrote:

> I tried disabling GCM ciphers based on the following information:
> * https://www.wowza.com/docs/how-to-improve-ssl-performance-with-java-8
> *
> https://stackoverflow.com/questions/25992131/slow-aes-gcm-encryption-and-decryption-with-java-8u20
>
> The results for the read were:
> * knox ssl no GCM - 1,073,741,824  125MB/s   in 8.7s
> * knox ssl - 1,073,741,824 54.3MB/s   in 20s
>
> This is a little more than a 2x speedup. There is also information in the
> links above that there should be more performance improvements with JDK 9+.
>
> For the write side slow down, I found an issue with how Knox is handing
> the streaming data on writes only. I am looking into fixing this to get the
> write performance for HDFS improved.
>
> Kevin Risden
>
>
> On Wed, Oct 10, 2018 at 1:20 PM David Villarreal <
> dvillarr...@hortonworks.com> wrote:
>
>> I believe Curl has an option of what cipher to use..  You may also be
>> able to force it at the server jvm level using
>> /jre/lib/security/java.security
>>
>>
>>
>>
>>
>> *From: *Sandeep Moré 
>> *Reply-To: *"user@knox.apache.org" 
>> *Date: *Tuesday, October 9, 2018 at 6:39 PM
>> *To: *"user@knox.apache.org" 
>> *Subject: *Re: WebHDFS performance issue in Knox
>>
>>
>>
>> I think this would be a good test, worth a try, not sure how we can force
>> a certain cipher to be used perhaps a permutation combination of
>>
>> ssl.include.ciphers, ssl.exclude.ciphers.
>>
>>
>>
>> Best,
>>
>> Sandeep
>>
>>
>>
>>
>>
>> On Tue, Oct 9, 2018 at 5:29 PM David Villarreal <
>> dvillarr...@hortonworks.com> wrote:
>>
>> Hi Kevin,
>>
>>
>>
>> In my humble opinion, this has to do with cpu processing encryption in
>> general based on which cipher being used.  Couldn’t the same type of
>> principals/improvements (hdfs encryption improvements) be done here for
>> let’s say for AES cipher suites?  If the main bottleneck here is CPU
>> couldn’t you enhance encryption though hardware acceleration and you may
>> see better performance numbers?
>>
>>
>>
>> https://calomel.org/aesni_ssl_performance.html
>>
>>
>>
>> Try forcing a less secure cipher to be used in your environment.  Do you
>> then see better numbers?
>>
>>
>>
>> dav
>>
>>
>>
>>
>>
>>
>> *From:*
>> *Kevin Risden*
>>
>>
>>
>>
>> * > Reply-To:
>> "user@knox.apache.org " > > Date: Tuesday, October 9, 2018 at 1:05 PM To:
>> "user@knox.apache.org " > > Subject: Re: WebHDFS performance issue in Knox*
>>
>>
>>
>> @David - Not sure what you mean since this is SSL/TLS and not related to
>> RPC encryption like the two JIRAs that you linked.
>>
>> @Guang - NP just took some time to sit down and look at it.
>>
>>
>>
>> Some preliminary investigation shows this may be the JDK implementation
>> of TLS/SSL that is slowing down the read path. I need to dig into it
>> further but found a few references showing that Java slowness for TLS/SSL
>> affects Jetty.
>>
>>-
>>https://nbsoftsolutions.com/blog/the-cost-of-tls-in-java-and-solutions
>>-
>>https://nbsoftsolutions.com/blog/dropwizard-1-3-upcoming-tls-improvements
>>- https://webtide.com/conscrypting-native-ssl-for-jetty/
>>
>> Locally testing off a Jetty 9.4 branch (for KNOX-1516), I was able to
>> enable conscrypting (
>> https://www.eclipse.org/jetty/documentation/9.4.x/configuring-ssl.html#conscrypt).
>> With that I was able to get read performance on par with non ssl and native
>> webhdfs. The write side of the equation still has some performance
>> differences that need to be looked at further.
>>
>>
>> Kevin Risden
>>
>>
>>
>>
>>
>> On Tue, Oct 9, 2018 at 2:01 PM Guang Yang  wrote:
>>
>> Thanks Kevin conducting such experiment! This is exactly what I saw
>> before. It doesn't look right the download speed is 10x slower when
>> enabling SSL.
>>
>>
>>
>> On Tue, Oct 9, 2018 at 10:40 AM David Villarreal <
>> dvillarr...@hortonworks.com> wrote:
>>
>> I bring this up because HDFS encryption saw an increase in performance.
>>
>> https://issues.apache.org/jira/browse/HDFS-6606
>>
>>
>>
>

Re: Spark History UI Error WARN HttpParser: Header is too large >8192

[Sandeep Moré]

You can try creating a new rewrite rule that drops that problematic header
if you don't need all those groups.

Best,
Sandeep

On Wed, Oct 10, 2018 at 7:01 PM Theyaa Matti  wrote:

> Hi David, Kevin,
>  I have reached out to the spark users community to see if their are
> alternatives or any work going on to fix this issue. One other thing I am
> trying to test out is the limiting the request header sent from Knox to the
> backend services, like spark history ui.
>
>  The main reason this issue appear on our end is the large number of
> groups the user belongs to that is pushing the request header to go beyond
> 8192. Those groups are not needed in the backend services and I am looking
> to see if there is a way for Knox to exclude them when forwarding the
> request to the backend?
>
> Best,
>
> Theyaa.
>
>
> On Wed, Oct 10, 2018 at 2:26 PM Kevin Risden  wrote:
>
>> Moving this back to the user list since it was sent without the list:
>>
>> Question: Is this error coming from the spark history server or the knox
>> server?
>>
>> Answer: It is coming from the spark history UI. Here is what I see in the
>> log for spark history ui.
>>
>> 18/10/10 12:30:29 WARN HttpParser: Header is too large >8192
>>
>> 18/10/10 12:30:29 WARN HttpParser: badMessage: 413 for
>> HttpChannelOverHttp@2756e900{r=1,c=false,a=IDLE,uri=/?doAs=}
>>
>>
>> With the above information there is probably nothing that can be done
>> from Knox. The bad news is that there is no configuration in Spark for this
>> yet. See the following:
>> * https://issues.apache.org/jira/browse/SPARK-15090
>> *
>> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala#L337
>> * Should see something about:
>> * connector.setRequestHeaderSize(requestHeaderSize);
>> * connector.setResponseHeaderSize(responseHeaderSize);
>>
>> Without a configuration to change, the Spark history UI is going to
>> default to the 8192 header size. A related change was done for most
>> projects to make that header size configurable.
>>
>> Kevin Risden
>>
>>
>> On Tue, Oct 9, 2018 at 8:45 PM Theyaa Matti 
>> wrote:
>>
>>> Hi David,
>>>  Thank you for the quick response. I do have that property set
>>> with the recommended value in Knox, but the issue still persists only with
>>> the Spark History UI. The issue appears only when I enable ssl for knox. If
>>> I turn ssl off in Knox the issue disappears, and if I enable it, it shows
>>> up right away.
>>>
>>>  Are there any equivalent properties for Spark History UI? since
>>> I suspect the issue is with the Spark History UI and not with Knox since
>>> all the other UIs work fine with Knox.
>>>
>>> Best,
>>>
>>> Theyaa.
>>>
>>>
>>>
>>> On Tue, Oct 9, 2018 at 7:11 PM David Villarreal <
>>> dvillarr...@hortonworks.com> wrote:
>>>
 Hi Theyaa,

 Change the size of gateway.httpserver.requestHeaderBuffer property.  I
 think the default is 8912  (8k) change to 16384. See if that helps.

 For the second problem Request is a replay (34))] this message is often
 seen when the timing of one of the servers is off.  Make sure you use NTPD
 on all servers and they are all in sync.  If everything is in sync you can
 work around this issue by turning off krb5 replay cache. With the following
 parameter
 -Dsun.security.krb5.rcache=none

 dav


 On 10/9/18, 9:01 AM, "Theyaa Matti"  wrote:

 Hi,
I am getting this error message "WARN HttpParser: Header is too
 large
 >8192" when trying to access the spark history ui through knox. Any
 idea
 please?

 Also when trying to load the executors page, I get : GSS initiate
 failed
 [Caused by GSSException: Failure unspecified at GSS-API level
 (Mechanism level:
 Request is a replay (34))]

 when knox is requesting executorspage-template.html

 appreciate any help here.

Re: x-ndjson via Knox

[Sandeep Moré]

Hello Rab,

Interesting, do you have the debug logs for this ? they will tell us
exactly what is happening.

Best,
Sandeep

On Fri, Sep 21, 2018 at 9:56 AM rabii lamriq  wrote:

> Hi All,
>
> From Grafana, I Post multisearch query to Elasticsearch using the bulk API
> with the newline delimited JSON (NDJSON) format.
> https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html
> like this:
>
> curl -u myuser -i -H “Content-Type: application/json” -XPOST "
> http://host_ES:900/index/_msearch; --data-binary ’
> { first json doc}
>  {second json doc}
> ’
> when bypassing Knox, The query work well either we made json or x-ndjson,
> However, through Knox gateway, if the content-type=application/json, the
> query retun this error of 400:
>
>  HTTP/1.1 400 Bad Request
>  content-type: application/json
>
> But when we replace json by x-ndjson like mentioned in the bulk API, it
> work well.
>
> My problem is that Grafana send their query with the
> content-type=application/json and not x-ndjson, for this, it is possible to
> make any thing on Knox to solve this problem?
>
> Regards
>  Rab
>

Re: x-ndjson via Knox

[Sandeep Moré]

I don't see any issues from Knox side, what error do you see in Grafana ?
I think this could be because Grafana is expecting certain content-type.

On Fri, Sep 21, 2018 at 11:37 AM rabii lamriq  wrote:

> Hi Sandeep
>
> thanks for your feedback, below is the deboug log for this curl
>
> 2018-09-18 17:23:23,154 DEBUG hadoop.gateway
> (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL: 
> https://:8443/gateway/default/es5//_msearch,
> direction: IN via explicit rule: ES5/inbound/query to URL: http://
> :9200//_msearch
> 2018-09-18 17:23:23,156 DEBUG hadoop.gateway
> (ElasticsearchDispatch.java:executeOutboundRequest(79)) - Dispatch request:
> POST http://:9200//_msearch?doAs=
> 2018-09-18 17:23:23,193 DEBUG hadoop.gateway
> (ElasticsearchDispatch.java:executeOutboundRequest(95)) - Dispatch response
> status: 400
> 2018-09-18 17:23:23,194 DEBUG hadoop.gateway
> (DefaultDispatch.java:getInboundResponseContentType(202)) - Using explicit
> character set UTF-8 for entity of type application/json
> 2018-09-18 17:23:23,195 DEBUG hadoop.gateway
> (DefaultDispatch.java:getInboundResponseContentType(210)) - Inbound
> response entity content type: application/json; charset=UTF-8
> 2018-09-18 17:23:54,519 DEBUG hadoop.gateway
> (GatewayFilter.java:doFilter(116)) - Received request: POST
> /es5//_msearch
> 2018-09-18 17:23:54,521 INFO  hadoop.gateway
> (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true
>
> Regards
> Rabii
>
> Le ven. 21 sept. 2018 à 17:17, Sandeep Moré  a
> écrit :
>
>> Hello Rab,
>>
>> Interesting, do you have the debug logs for this ? they will tell us
>> exactly what is happening.
>>
>> Best,
>> Sandeep
>>
>> On Fri, Sep 21, 2018 at 9:56 AM rabii lamriq  wrote:
>>
>>> Hi All,
>>>
>>> From Grafana, I Post multisearch query to Elasticsearch using the bulk
>>> API with the newline delimited JSON (NDJSON) format.
>>> https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html
>>> like this:
>>>
>>> curl -u myuser -i -H “Content-Type: application/json” -XPOST "
>>> http://host_ES:900/index/_msearch; --data-binary ’
>>> { first json doc}
>>>  {second json doc}
>>> ’
>>> when bypassing Knox, The query work well either we made json or x-ndjson,
>>> However, through Knox gateway, if the content-type=application/json, the
>>> query retun this error of 400:
>>>
>>>  HTTP/1.1 400 Bad Request
>>>  content-type: application/json
>>>
>>> But when we replace json by x-ndjson like mentioned in the bulk API, it
>>> work well.
>>>
>>> My problem is that Grafana send their query with the
>>> content-type=application/json and not x-ndjson, for this, it is possible to
>>> make any thing on Knox to solve this problem?
>>>
>>> Regards
>>>  Rab
>>>
>>

Re: x-ndjson via Knox

[Sandeep Moré]

This is an example in Webhdfs service def.
https://github.com/apache/knox/blob/6e7266ad3649e1804ee869afed04f79a71e48b7d/gateway-service-definitions/src/main/resources/services/webhdfs/2.4.0/rewrite.xml#L64



On Tue, Sep 25, 2018 at 5:37 AM rabii lamriq  wrote:

> Hi Sandeep,
>
> Yes, I know that is possible, but I don't know how to rewrite the rule to
> access to the header and change the Content-type, I can't find any thing
> related to this customization.
>
> Regards
> Rabii
>
> Le lun. 24 sept. 2018 à 17:19, Sandeep Moré  a
> écrit :
>
>> Yes, you can use the rewrite rules to update Headers, I don't remember
>> exactly which service uses it but there should be few services that rewrite
>> headers.
>>
>> Best,
>> Sandeep
>>
>> On Mon, Sep 24, 2018 at 10:45 AM rabii lamriq  wrote:
>>
>>>
>>> Rewrite rules for knox (link
>>> <https://cwiki.apache.org/confluence/display/KNOX/2017/08/14/Understanding+Rewrite+Rules+for+Apache+Knox#UnderstandingRewriteRulesforApacheKnox-JSONURLRewriteFIlter>)
>>> can help me to change header? is it possible?
>>>
>>> Le lun. 24 sept. 2018 à 16:43, rabii lamriq  a écrit :
>>>
>>>> Hello
>>>>
>>>> Any help please??
>>>>
>>>> Le lun. 24 sept. 2018 à 09:31, rabii lamriq  a
>>>> écrit :
>>>>
>>>>> Hi,
>>>>>
>>>>> thanks every One, for your feedbacks,
>>>>>
>>>>> Yes, ES accept both application/json and application/x-ndjson, there
>>>>> is no problem in ES ni in Grafana, when i test my query with curl to send
>>>>> POST to ES, it work well ether json or x-ndjson.
>>>>>
>>>>> However, through Knox Gateway, it work only for x-ndjson, I don't know
>>>>> why. so, since Grafana send their POST with only json and not x-ndjson, 
>>>>> the
>>>>> scenario of Grafana -> Knox -> ES with multi search doesn't work.
>>>>>
>>>>> I think we have two solution:
>>>>>
>>>>> 1- make some thing to force Grafana change header of POST and put
>>>>> x-ndjson instead of json, or
>>>>> 2- rewrite Header by Knox to forwarded with x-ndjson to ES.
>>>>>
>>>>> If some one have any idea of how to make this or other solution.
>>>>>
>>>>>
>>>>> Regards
>>>>> Rabii
>>>>>
>>>>> Le ven. 21 sept. 2018 à 19:21, Kevin Risden  a
>>>>> écrit :
>>>>>
>>>>>> https://github.com/elastic/elasticsearch/issues/25718
>>>>>>
>>>>>> Elasticsearch endpoints should take both application/json and
>>>>>> application/x-ndjson. There might be something else going on here. That 
>>>>>> is
>>>>>> why I asked for the Elasticsearch logs.
>>>>>>
>>>>>> It might not be necessary to rewrite the content type.
>>>>>>
>>>>>> Kevin Risden
>>>>>>
>>>>>>
>>>>>> On Fri, Sep 21, 2018 at 1:17 PM rabii lamriq 
>>>>>> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> My problem is how to change the content-type=application/json to
>>>>>>> application/x-ndjson by rewriting the header in the knox.
>>>>>>>
>>>>>>> Because grafana send their query with content-type=application/json
>>>>>>> and knox return status 400,
>>>>>>>
>>>>>>> Hower when i reproduce the problem by curl comande, I noticed that
>>>>>>> if i replace json by x-ndjson through knox it wirk well.
>>>>>>>
>>>>>>> Sence i can't change the content-type in grafana, is this possible
>>>>>>> by rewriting the header in knox to replace json by x-ndjson.
>>>>>>>
>>>>>>> Regards
>>>>>>> Rabii
>>>>>>>
>>>>>>> Le ven. 21 sept. 2018 à 18:05,
>>>>>>> Kevin Risden
>>>>>>>  a écrit :
>>>>>>>
>>>>>>>> Do you have the error from Elasticsearch logs? It might explain if
>>>>>>>> Knox or Grafana is sending a bad request.
>>>>>>>>
>>>>>&g

Re: url rewrite does not work when adding a new service

[Sandeep Moré]

wayFilter.doFilter(GatewayFilter.java:171)
> at
> org.apache.knox.gateway.GatewayFilter.doFilter(GatewayFilter.java:94)
> at
> org.apache.knox.gateway.GatewayServlet.service(GatewayServlet.java:141)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
> at
> org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:201)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at
> org.apache.knox.gateway.trace.TraceHandler.handle(TraceHandler.java:51)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at
> org.apache.knox.gateway.filter.CorrelationHandler.handle(CorrelationHandler.java:41)
> at
> org.eclipse.jetty.servlets.gzip.GzipHandler.handle(GzipHandler.java:479)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at
> org.apache.knox.gateway.filter.PortMappingHelperHandler.handle(PortMappingHelperHandler.java:152)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at
> org.eclipse.jetty.websocket.server.WebSocketHandler.handle(WebSocketHandler.java:112)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258)
> at
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.SocketTimeoutException: connect timed out
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:589)
> at
> org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75)
> at
> org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
> ... 72 more
>
>
> On Wed, Sep 12, 2018 at 6:19 AM Sandeep Moré 
> wrote:
>
>> Hello Lian,
>>
>> What do you see in gateway.log ?
>> Knox replacing knox_load_balancer.com:80 <http://knox_load_balancer.com/>
>>  with http://yahoo.com/ might be the log output of rewritten URL which
>> is expected given
>>
>> 
>> WEATHER
>> http://yahoo.com/
>>  
>>
>>
>>
>>
>> On Wed, Sep 12, 2018 at 2:22 AM Lian Jiang  wrote:
>>
>>> I am following
>>> http://kminder.github.io/knox/2015/11/16/adding-a-service-to-knox.html
>>> to add a weather service to knox.
>>>
>>> data/services/weather/0.0.1/rewrite.xml:
>>> 
>>> >> pattern="*://*:*/**/weather/{path=**}?{**}">
>>> 
>>> 
>

Re: url rewrite does not work when adding a new service

[Sandeep Moré]

Great, you got it work, touching the topology is a bit that can be so easy
to miss.
I don't think we have working configs for Jupyter notebook, we would love
to have one though.
It would be really awesome if you can contribute it to the community when
you get it to work !

Best,
Sandeep

On Wed, Sep 12, 2018 at 6:41 PM Lian Jiang  wrote:

> Thanks Sandeep.
>
> I got clue from
> http://mail-archives.apache.org/mod_mbox/knox-user/201805.mbox/%3ccacrbfygjshmcc+qm0wnwzhcjcb9ge2galrtngory-zklgag...@mail.gmail.com%3E
> .
>
> Restarting knox is not enough and I must touch the topology file to reload
> it every time even if the topology is not changed. Now I see that from the
> auditing log that  knox_load_balancer.com is replaced with yahoo.com.
>
> My goal is to expose Jupyter via KNOX. It is hard to compose rewrite.xml
> and service.xml from the scratch for proxying Jupyter. I tried to refer
> other UIs (e.g. ambari, ranger and zeppelin)'s routing files but it is
> still hard. I need to inspect the response, header in details which is time
> consuming and unreliable. Do you know where I can find working
> service.xml/rewrite.xml for jupyter? I believe this should be a common
> requirement to expose Jupyter via Knox. Thanks a lot for any hint!
>
>
> On Wed, Sep 12, 2018 at 11:31 AM Sandeep Moré 
> wrote:
>
>> No idea why Knox is redirecting back to itself, perhaps turning on Debug
>> logging will help understand what is happening. I remember I got the
>> weather service working at some point, let me know if you would like the
>> service and XML files for it.
>>
>> On Wed, Sep 12, 2018 at 11:52 AM Lian Jiang 
>> wrote:
>>
>>> This is from gateway.log. It is expected that knox cannot connect to
>>> knox_load_balancer.com because rewriting knox_load_balancer.com to
>>> yahoo.com did not happen. Thanks for any hint.
>>>
>>> 2018-09-12 15:48:38,273 WARN  knox.gateway
>>> (DefaultDispatch.java:executeOutboundRequest(147)) - Connection exception
>>> dispatching request: http:///
>>> knox_load_balancer.com:80/gateway/ui/weather
>>> org.apache.http.conn.ConnectTimeoutException: Connect to /
>>> knox_load_balancer.com:80 [/knox_load_balancer.com/130.35.0.245]
>>> failed: connect timed out
>>> org.apache.http.conn.ConnectTimeoutException: Connect to /
>>> knox_load_balancer.com:80 [/knox_load_balancer.com/130.35.0.245]
>>> failed: connect timed out
>>> at
>>> org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
>>> at
>>> org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373)
>>> at
>>> org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
>>> at
>>> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
>>> at
>>> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
>>> at
>>> org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
>>> at
>>> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
>>> at
>>> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
>>> at
>>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
>>> at
>>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
>>> at
>>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
>>> at
>>> org.apache.knox.gateway.dispatch.DefaultDispatch.executeOutboundRequest(DefaultDispatch.java:130)
>>> at
>>> org.apache.knox.gateway.dispatch.DefaultDispatch.executeRequest(DefaultDispatch.java:116)
>>> at
>>> org.apache.knox.gateway.dispatch.DefaultDispatch.doGet(DefaultDispatch.java:278)
>>> at
>>> org.apache.knox.gateway.dispatch.GatewayDispatchFilter$GetAdapter.doMethod(GatewayDispatchFilter.java:170)
>>> at
>>> org.apache.knox.gateway.dispatch.GatewayDispatchFilter.doFilter(GatewayDispatchFilter.java:122)
>>> at
>>> org.apache.knox.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:61)
>>> at
>>> org.apache.knox.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:372)
>>> at
>>> org.apache.knox.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:272)
>>> at
>>> org.apache.ranger.auth

Re: url rewrite does not work when adding a new service

[Sandeep Moré]

Hello Lian,

What do you see in gateway.log ?
Knox replacing knox_load_balancer.com:80 
 with http://yahoo.com/ might be the log output of rewritten URL which is
expected given


WEATHER
http://yahoo.com/
 




On Wed, Sep 12, 2018 at 2:22 AM Lian Jiang  wrote:

> I am following
> http://kminder.github.io/knox/2015/11/16/adding-a-service-to-knox.html to
> add a weather service to knox.
>
> data/services/weather/0.0.1/rewrite.xml:
> 
>  pattern="*://*:*/**/weather/{path=**}?{**}">
> 
> 
> 
>
> data/services/weather/0.0.1/service.xml:
> 
>   
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>
> in topology ui.xml:
> 
> WEATHER
> http://yahoo.com/
>  
>
> Accessing https://*knox_load_balancer.com/gateway/ui/weather
> * got below output in
> gateway-audit.log, I see:
>
> 18/09/12 06:15:21
> ||f7b1ea3a-73ec-464a-87f4-dbb22e22867c|audit|160.34.88.239|WEATHERaccess|uri|/gateway/ui/weather|unavailable|Request
> method: GET
> 18/09/12 06:15:21
> ||f7b1ea3a-73ec-464a-87f4-dbb22e22867c|audit|160.34.88.239|WEATHER|anonymous|||authentication|uri|/gateway/ui/weather|success|
> 18/09/12 06:15:21
> ||f7b1ea3a-73ec-464a-87f4-dbb22e22867c|audit|160.34.88.239|WEATHER|anonymous|||dispatch|uri|http://*knox_load_balancer.com:80
> */gateway/ui/weather|unavailable|Request
> method: GET
> 18/09/12 06:15:41
> ||f7b1ea3a-73ec-464a-87f4-dbb22e22867c|audit|160.34.88.239|WEATHER|anonymous|||dispatch|uri|http://*knox_load_balancer.com:80
> */gateway/ui/weather|failure|
> 18/09/12 06:15:41
> ||f7b1ea3a-73ec-464a-87f4-dbb22e22867c|audit|160.34.88.239|WEATHER|anonymous|||access|uri|/gateway/ui/weather|failure|
>
> Looks like url rewrite (replace knox_load_balancer.com:80 with
> http://yahoo.com/) does not work. Any idea? Appreciate any clue.
>
>

Re: URL Templating

[Sandeep Moré]

We don't support any templating framework in the topology currently but I
think descriptors [1] and externalized provider configurations [2] might
interest
you.

[1]
https://knox.apache.org/books/knox-1-2-0/user-guide.html#Simplified+Topology+Descriptors
[2]
https://knox.apache.org/books/knox-1-2-0/user-guide.html#Externalized+Provider+Configurations

Best,
Sandeep
On Mon, Feb 25, 2019 at 1:47 PM Watson, Billy 
wrote:

> Hello,
>
>
>
> Is there any support for jinja, et. al. templating in the topology.xml
> files, specifically W.R.T. the URL lists? As we deploy between
> environments, such a feature would be super helpful.
>
>
>
> Thanks,
>
>
>
> Billy
>

KNOX Pac4j Azure AD Open ID

[Sandeep Moré]

Hello Nisha,

I remember a while back you were looking at Azure AD and Knox integration,
I checked in the fix as part of KNOX-1191
. It should be available
in upcoming 1.3.0 release.
You just need to use callback url similar to the one below:
https://www.local.com:8443/gateway/knoxsso/api/v1/websso/pac4jCallback/AzureAdClient

Let me know if you have any questions.

Best,
Sandeep

P.S. This is regarding the discussion
http://mail-archives.apache.org/mod_mbox/knox-user/201802.mbox/browser

Re: Add a filter to intercept Knox requests and be able to add a custom logic

[Sandeep Moré]

Hello Matteo,

I don't think this is a right way to add filters to Knox, if you want to
add some custom logic between your Knox and the backend you can  write
custom dispatch [1]
you can add this new jar in the ext folder and it should be picked up by
Knox on start-up.

[1]
https://knox.apache.org/books/knox-1-3-0/dev-guide.html#Custom+Dispatch+Dependency+Injection


On Thu, Mar 14, 2019 at 12:52 PM Matteo Alessandroni 
wrote:

> Hi,
>
> I'm trying to add a filter on a Knox instance so that I'll be able to add
> a custom logic to (every / specific) REST requests to Knox.
> I thought to use the "Class Path" feature [1], so I created a Maven Java
> project, generated a ".jar" file and placed it in the "$GATEWAY_HOME/ext"
> folder
> Then, I thought it was necessary to link my custom class [2] on the
> "gateway.xml" files of the resources I wanted, e.g.
>
> ./data/deployments/sandbox.topo.16977ef2478/%2F/WEB-INF/gateway.xml
>
> added:
>
> 
> rewrite
> url-rewrite
> com.test.knox.MyUrlRewriteServletFilter
> 
>
> then I made a REST request to Knox:
>
> curl -i -k -u admin:admin-password -X GET '
> https://localhost:8443/gateway/sandbox/webhdfs/v1/?op=LISTSTATUS'
>
> but my filter is not called at all (I cannot see the log) and I'm not sure
> whether it's because the class is not loaded or the filter is placed in the
> wrong place or whatever.
>
> On [4] you can see the project I have built.
>
> So my questions are:
>
>- what am I doing wrong or missing?
>- is there a better way to do that? I just need to add a logic when
>executing REST requests to Knox and make another REST call to an external
>service I need.
>
>
> Thank you!
> Regards,
> Matteo
>
> [1] https://knox.apache.org/books/knox-1-2-0/dev-guide.html#Class+Path
> [2]
> https://github.com/mat-ale/apache-knox-filter/blob/master/src/main/java/com/plainid/ext/MyUrlRewriteServletFilter.java
> [3]
> [4] https://github.com/mat-ale/apache-knox-filter
>
>
>

Re: Add a filter to intercept Knox requests and be able to add a custom logic

[Sandeep Moré]

It should have picked it up, going through the gateway.sh file, I don't see
any place where ext folder is added to class path, can you open a BUG for
this ?

as a workaround for this you can copy your jar into the lib folder and Knox
should pick it up on the startup.

Hopefully, this should help !

Best,
Sandepe


On Fri, Mar 15, 2019 at 5:37 AM Matteo Alessandroni 
wrote:

> Hi Sandeep,
>
> thank you for your answer!
> Ok so I tried to change my project and adding a simple class like this:
>
> package com.test.ext;
>
> import java.io.IOException;
> import java.net.URI;
> import java.net.URISyntaxException;
> import javax.servlet.http.HttpServletRequest;
> import javax.servlet.http.HttpServletResponse;
> import org.apache.knox.gateway.config.Configure;
> import org.apache.knox.gateway.config.Default;
> import org.apache.knox.gateway.dispatch.AbstractGatewayDispatch;
> import org.slf4j.Logger;
> import org.slf4j.LoggerFactory;
>
> public class MyDispatch extends AbstractGatewayDispatch {
>
> private static final Logger LOG =
> LoggerFactory.getLogger(MyDispatch.class);
>
> @Override
> public void destroy() {
> LOG.debug("*** destroy()");
> }
>
> @Configure
> protected void customMethod(@Default("Test") String test) {
> LOG.debug("*** @Configure customMethod(): {}", test);
> }
>
> @Override
> public void doGet(URI url, HttpServletRequest request,
> HttpServletResponse response)
> throws IOException, URISyntaxException {
>
> LOG.debug("*** doGet() request: {}, {}", request.getMethod(),
> new URI(request.getRequestURI()));
>
> super.doGet(url, request, response);
> }
>
> }
>
> made the ".jar" and put in the "ext" folder.
> Executed the REST request again:
>
> curl -i -k -u admin:admin-password -X GET '
> https://localhost:8443/gateway/sandbox/webhdfs/v1/?op=LISTSTATUS'
>
> and I expected to see some logs, e.g. from the "doGet()" method (I'm not
> sure about the "@Configure" method, when should a method with that
> annotation be executed?), but it seems it does not see the class.
> So I tried to configure my topology to use the dispatch (as written on
> [1]):
>
> 
> WEBHDFS
> http://hadoop-namenode:50070/webhdfs
>
> 
> com.test.ext.MyDispatch
> false
> 
> 
>
> and after saving logs say:
>
> ERROR knox.gateway (GatewayFilter.java:doFilter(170)) - Gateway processing
> failed: javax.servlet.ServletException:
> org.apache.shiro.subject.ExecutionException:
> java.security.PrivilegedActionException: javax.servlet.ServletException:
> javax.servlet.ServletException: java.lang.ClassNotFoundException:
> com.test.ext.MyDispatch
> javax.servlet.ServletException:
> org.apache.shiro.subject.ExecutionException:
> java.security.PrivilegedActionException: javax.servlet.ServletException:
> javax.servlet.ServletException: *java.lang.ClassNotFoundException:
> com.test.ext.MyDispatch*
>
> so it's not seeing my class.
> What am I missing?
>
> Thank you!
>
> Matteo
>
>
> [1] https://knox.apache.org/books/knox-1-3-0/dev-guide.html#service.xml
>
>
> On 14/03/19 20:47, Sandeep Moré wrote:
>
> Hello Matteo,
>
> I don't think this is a right way to add filters to Knox, if you want to
> add some custom logic between your Knox and the backend you can  write
> custom dispatch [1]
> you can add this new jar in the ext folder and it should be picked up by
> Knox on start-up.
>
> [1]
> https://knox.apache.org/books/knox-1-3-0/dev-guide.html#Custom+Dispatch+Dependency+Injection
>
>
> On Thu, Mar 14, 2019 at 12:52 PM Matteo Alessandroni 
> wrote:
>
>> Hi,
>>
>> I'm trying to add a filter on a Knox instance so that I'll be able to add
>> a custom logic to (every / specific) REST requests to Knox.
>> I thought to use the "Class Path" feature [1], so I created a Maven Java
>> project, generated a ".jar" file and placed it in the "$GATEWAY_HOME/ext"
>> folder
>> Then, I thought it was necessary to link my custom class [2] on the
>> "gateway.xml" files of the resources I wanted, e.g.
>>
>> ./data/deployments/sandbox.topo.16977ef2478/%2F/WEB-INF/gateway.xml
>>
>> added:
>>
>> 
>> rewrite
>> url-rewrite
>> com.test.knox.MyUrlRewriteServletFilter
>> 
>>
>> then I made a REST request to Knox:
>>
>> curl -i -k -u admin:admin-password -X GET '
>> https://localhost:8443/gateway/sandbox/webhdfs/v1/?op=LISTSTATUS'
>>
>>

Re: Adding a new UI service to Knox

[Sandeep Moré]

Hello Odon,

What does the response from the the ferrero UI look like? for the last
rewrite rule, looks like you are having issues rewriting outbound requests,
what response are you dealing with, JSON, HTML, plain text ? depending on
that you probably need to add a filter for that specific content type. I
think you are very close !

Best,
Sandeep


On Mon, Apr 29, 2019 at 1:47 PM Odon Copon  wrote:

> Hi,
> I have been following the steps from
> https://cwiki.apache.org/confluence/display/KNOX/Proxying+a+UI+using+Knox but
> I'm having some issue making something work, so would double check with you
> to understand if I'm making wrong assumptions.
> I have a UI made with React which URL is ferrerorui:8080/site/ that
> contains some graphs generated by some requests to
> ferrerorapi/production/datapoints. When I put Knox in place, I can access
> the UI by accessing knox:12002/gateway/test/ferrerorui/site/ and also I can
> send requests to the API by doing
> knox:12002/gateway/test/ferrerorapi/production/datapoints.
>
>
> This is the service defined (ferrerorapi, the API the UI consumes):
>
> service.xml
>
> 
>   
> 
>   
> 
>
> rewrite.xml
>
> 
>pattern="*://*:*/**/ferrerorapi/{path=**}?{**}">
> 
>   
> 
>
>
> Obviously, the API requests that generate the graphs are all failing with
> 404 when I access the UI, because they are hitting
> knox:12002/production/datapoints instead of
> knox:12002/gateway/test/ferrerorapi/production/datapoints
>
>
> This is the UI service:
>
> service.xml
>
> 
>   
> 
> 
>to="response.body"/>
>   
> 
>
> rewrite.xml
>
> 
>  pattern="*://*:*/**/ferrerorui/">
> 
>   
>pattern="*://*:*/**/ferrerorui/{path=**}?{**}">
> 
>   
>pattern="*://*:*/**/production/{path=**}?{**}">
> 
>   
> 
>
>
> Tried with this last rewrite to convert on the UI, everything that had
> production as part of the path (api call) to be rewritten, but doesn't seem
> to work. The API call from the UI still don't contain the missing
> "gateway/test/ferrerorapi/".
>
> Is there anything that you spot that I'm not doing correctly?
> Thanks.
>

Re: Pattern Matching with query string

[Sandeep Moré]

I don't think that would work, mostly because of querystrings, they
can be in any order so there is no guarantee that a fixed pattern will
grab them.
You might be better off with a mixture of rewrite rules and custom dispatch IMO.

Best,
Sandeep


On Mon, May 6, 2019 at 5:37 AM Odon Copon  wrote:
>
> Hi,
> Having something like:
> http://host:port/path1/path2/path3/path4?querystring1=value=value
>
> How would you apply a pattern matching to the previous URL to be rewritten 
> similar to the following?
> http://host:port/path1?querystring1=value
>
> Would something like this work?
> {scheme}://{host}:{port}/{path1}/{path=**}?{queryParam=*}&{**}
>
> Thanks

Re: Knox jetty threads stuck and Seeing ConnectionPoolTimeoutException while proxying a custom service

[Sandeep Moré]

Hello Rajat,

Do you see Websocket traffic work as expected the first time when the UI
works ? can you check using the developer console if the initial Websocket
connection was established and data is flowing correctly?

Looks like you are connecting to a secure backend, is the Websocket backend
secure as well ?
Can you also post a redacted gateway.log file with DEBUG logging, it would
be helpful to see the entire log file with the stacktrace.

Also, do you see any errors in your service log files ?

Best,
Sandeep


On Mon, Aug 12, 2019 at 1:08 AM Rajat Goel 
wrote:

> Hello,
>
>
>
> I need some help while trying to integrate my custom service with Knox.
> Here’s how the service is deployed:
>
>
>
> _   Knox (Instance
> 1) _   _  My Service1 (Master).
>
>
> |
> |
>  |
>
> UI Client -> Load Balancer  à | |
> --  For Availability à |
>
>  (HA Proxy)
> | |   (HA
> Proxy) |
>
>  | _  Knox (Instance
> 2) _  |   | _ My Service2 (Standby)
>
>
>
>
>
> My custom service UI client uses both web sockets as well as REST calls to
> talk to My Service backend. I have written service definition xml and
> rewrite xml files and deployed them on a test setup as per the above
> architecture. My Knox version is 1.0 (HDP 3.1). I have added the service in
> Knox default topology and  default topology uses Shiro Provider for
> authentication and ‘gateway.websocket.feature.enabled’ is set to true.
>
>
>
> When Knox service is started, I see the following behavior:
>
>- When I launch the UI from browser, I get Shiro authentication dialog
>popup, I enter my credentials and get redirected to my Service UI.
>Initially, for the first couple of minutes, UI works fine. All calls from
>UI are getting routed properly to backend.
>- After first few minutes, I start seeing following exception in Knox
>gateway logs:
>
>2019-08-10 15:37:32,053 ERROR gateway.websockets
> (ProxyWebSocketAdapter.java:onWebSocketConnect(105)) - Unable to connect to
> websocket server: java.io.IOException: Connect failure
>
> java.io.IOException: Connect failure
>
> Caused by: org.eclipse.jetty.websocket.api.UpgradeException: 0 null
>
> at
> org.eclipse.jetty.websocket.client.WebSocketUpgradeRequest.onComplete(WebSocketUpgradeRequest.java:515)
>
> Caused by: java.io.EOFException: HttpConnectionOverHTTP@1e8a6128
> ::SocketChannelEndPoint@5526bc43{
> rafint001-mgt-01.cloud.in.guavus.com/192.168.141.33:9443<->/
> 192.168.141.31:49860
> ,ISHUT,fill=-,flush=-,to=3/0}{io=0/0,kio=0,kro=1}->HttpConnectionOverHTTP@1e8a6128
> (l:/192.168.141.31:49860 <-> r:
> rafint001-mgt-01.cloud.in.guavus.com/192.168.141.33:9443,closed=false)=
> >HttpChannelOverHTTP@500cd5d(exchange=HttpExchange@75bf9768
> req=TERMINATED/null@null res=PENDING/null@null
> )[send=HttpSenderOverHTTP@6678aae8
> (req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator@6dcd6c0
> {s=START}],recv=HttpReceiverOverHTTP@5358963(rsp=IDLE,failure=null)[HttpParser{s=CLOSED,0
> of -1}]]
>
> ... 13 more
>
>
>
>- UI starts slowing down and response times increase.
>- I also see DEBUG exceptions such as:
>
> DEBUG io.FillInterest (FillInterest.java:onFail(134)) - onFail
> FillInterest@1d803962{null}
>
> java.util.concurrent.TimeoutException: Idle timeout expired: 30/30
> ms
>
> at
> org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166)
>
> at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50)
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:748)
>
>
>
>
>
> DEBUG io.WriteFlusher (WriteFlusher.java:onFail(471)) - ignored:
> WriteFlusher@3a2f4fc0{IDLE}->null
>
> java.nio.channels.ClosedChannelException
>
> at org.eclipse.jetty.io.WriteFlusher.onClose(WriteFlusher.java:502)
>
> at
> org.eclipse.jetty.io.AbstractEndPoint.onClose(AbstractEndPoint.java:353)
>
> at
> org.eclipse.jetty.io.ChannelEndPoint.onClose(ChannelEndPoint.java:216)
>
> at
> org.eclipse.jetty.io.AbstractEndPoint.doOnClose(AbstractEndPoint.java:225)
>
> at
> 

Re: Knox jetty threads stuck and Seeing ConnectionPoolTimeoutException while proxying a custom service

[Sandeep Moré]

This could be multiple sockets, this is weird, looks like for some reason
there are multiple sockets opened between Knox and the backend.
Can you try using just Knox and the backend and see what you get, i.e. get
rid of the load balancers and HA to isolate the problem.

On Sat, Aug 17, 2019 at 8:14 AM Rajat Goel 
wrote:

> Hi Sandeep,
>
>
>
> While reproducing the issue, I observed that ‘netstat’ command shows 32
> connections in ESTABLISHED state with UI backend service:
>
> *tcp6   0  0 192.168.133.69:48510 <http://192.168.133.69:48510>
> 192.168.133.69:9443 <http://192.168.133.69:9443> ESTABLISHED 19122/java*
>
> *tcp6   0  0 192.168.133.69:53848 <http://192.168.133.69:53848>
> 192.168.133.69:9443 <http://192.168.133.69:9443> ESTABLISHED 19122/java*
>
> *tcp6   0  0 192.168.133.69:52994 <http://192.168.133.69:52994>
> 192.168.133.69:9443 <http://192.168.133.69:9443> ESTABLISHED 19122/java*
>
> *…*
>
>
>
> However, when I search for port number in gaeway.log file (full debugs
> logs enabled), there is only log mentioning that particular port:
>
> *[root@rafd001-mst-01 knox]# cat gateway.log | grep -w 48510*
>
> *2019-08-17 10:34:40,717 DEBUG conn.DefaultHttpClientConnectionOperator
> (DefaultHttpClientConnectionOperator.java:connect(146)) - Connection
> established 192.168.133.69
> <http://192.168.133.69>:48510<->192.168.133.69:9443
> <http://192.168.133.69:9443>*
>
>
>
> Looks like the connection was opened but not used? Other connections in
> netstat output show same trend. No logs in gateway.log file as to how the
> connection is getting used and these connections stay in ESTABLISHED state
> forever. They don’t even timeout. Any thoughts on why this could be
> happening ?
>
>
>
> Regards,
>
> Rajat
>
>
>
> *From: *Rajat Goel 
> *Reply to: *"user@knox.apache.org" 
> *Date: *Friday, 16 August 2019 at 11:18 PM
> *To: *"user@knox.apache.org" 
> *Subject: *Re: Knox jetty threads stuck and Seeing
> ConnectionPoolTimeoutException while proxying a custom service
>
>
>
> Hi Sandeep,
>
>
>
> You are right, websocket connection are not being established properly. I
> debugged this more today and looked at Knox’s GatewayWebsocketHandler.java
> code. From the code, I found that Knox uses service Url protocol to create
> backend ws or wss websocket connection whereas my service Url had ‘https’
> as I thought Knox would use incoming request URL (protocol as well as path)
> to construct backend websocket URL. After making some changes, I was able
> to fix UpgradeException issues.  Just a thought there: why not use request
> URI in getMatchedBackendURL() API { GatewayWebsocketHandler.java} to
> generate backend URL. This way we won’t have to write separate service
> definition for http(s) and ws requests, in many cases.
>
>
>
> Coming back to original issue, after fixing UpgradeException I still see
> ClosedChannel Exceptions, ConnectionPool timeout, Service connectivity
> errors, UI slowing down and eventually hang.
>
> ‘netstat’ output shows 32 connections in ESTABLISHED  state between Knox
> and UI backend server, all in Idle state. I had configured socket timeout
> as well as wbesocket idle timeouts to 60 seconds, still connections stay in
> ESTABLISHED state. Why so many connections are getting established ? When U
> is accessed directly i.e. there is no Knox proxy between UI client and
> backend, I only see 4-5 TCP connections between UI client and UI backend.
>
>
>
> Please check.
>
>
>
> Regards,
>
> Rajat
>
>
>
> *From: *Sandeep Moré 
> *Reply to: *"user@knox.apache.org" 
> *Date: *Friday, 16 August 2019 at 10:37 PM
> *To: *"user@knox.apache.org" 
> *Subject: *Re: Knox jetty threads stuck and Seeing
> ConnectionPoolTimeoutException while proxying a custom service
>
>
>
> Rajat,
>
> I could not see data flowing through websocket in the logs. I also noticed
> that your backend uses https but your websocket connection is using ws://,
> shouldn't it be using wss:// ?
>
>
>
> On Fri, Aug 16, 2019 at 7:02 AM Rajat Goel 
> wrote:
>
> Hi Sandeep, Knox Team,
>
>
>
> Need urgent help/pointers in debugging this issue as this is very critical
> for us. So request you to please check on the same.
>
>
>
> Thanks & Regards,
>
> Rajat
>
>
>
> *From: *Rajat Goel 
> *Reply to: *"user@knox.apache.org" 
> *Date: *Thursday, 15 August 2019 at 12:02 AM
> *To: *"user@knox.apache.org" 
> *Subject: *Re: Knox jetty threads stuck and Seeing
> ConnectionPoolTimeoutExceptio

Re: Knox jetty threads stuck and Seeing ConnectionPoolTimeoutException while proxying a custom service

[Sandeep Moré]

Hello Rajat,

I see "avax.net.ssl.SSLException: Received fatal alert:
certificate_unknown" errors in the logs, this appears to be the root cause.
Appears to be a certificate issue.

On Mon, Aug 12, 2019 at 1:51 PM Rajat Goel 
wrote:

> Attaching full debug log as well.
>
>
>
> *From: *Rajat Goel 
> *Reply to: *"user@knox.apache.org" 
> *Date: *Monday, 12 August 2019 at 10:50 PM
> *To: *"user@knox.apache.org" 
> *Subject: *Re: Knox jetty threads stuck and Seeing
> ConnectionPoolTimeoutException while proxying a custom service
>
>
>
> Hi Sandeep,
>
>
>
> Thanks for your reply.
>
>
>
> Yes, the websocket traffic works as expected initially. Connection is
> established and data flows correctly. Websocket backend is secured as well.
> To check if the issue is due to SSL or not, I disabled SSL on My service
> and the issue got reproduced with no secure (non SSL) setup as well.
>
>
>
> Attaching instance logs from one cycle of reproduction. This instance uses
> SSOCookieProvider (for SSO) and had DEBUG logs enabled only for
> org.apache.knox.gateway package. Full debug logs file instance is large in
> size so cannot send via mail but let me know if you need that as well. Also
> attaching thread dump of Knox for one particular instance when Knox threads
> were stuck.
>
>
>
> I don’t see any errors in service log files. The issue is reproducible
> every time so if you need any other information, please do let me know.
>
>
>
> Thanks & Regards,
>
> Rajat
>
>
>
> *From: *Sandeep Moré 
> *Reply to: *"user@knox.apache.org" 
> *Date: *Monday, 12 August 2019 at 7:37 PM
> *To: *"user@knox.apache.org" 
> *Subject: *Re: Knox jetty threads stuck and Seeing
> ConnectionPoolTimeoutException while proxying a custom service
>
>
>
> Hello Rajat,
>
>
>
> Do you see Websocket traffic work as expected the first time when the UI
> works ? can you check using the developer console if the initial Websocket
> connection was established and data is flowing correctly?
>
>
>
> Looks like you are connecting to a secure backend, is the Websocket
> backend secure as well ?
>
> Can you also post a redacted gateway.log file with DEBUG logging, it would
> be helpful to see the entire log file with the stacktrace.
>
>
>
> Also, do you see any errors in your service log files ?
>
>
>
> Best,
>
> Sandeep
>
>
>
>
>
> On Mon, Aug 12, 2019 at 1:08 AM Rajat Goel 
> wrote:
>
> Hello,
>
>
>
> I need some help while trying to integrate my custom service with Knox.
> Here’s how the service is deployed:
>
>
>
> _   Knox (Instance
> 1) _   _  My Service1 (Master).
>
>
> |
> |
>  |
>
> UI Client -> Load Balancer  à | |
> --  For Availability à |
>
>  (HA Proxy)
> | |   (HA
> Proxy) |
>
>  | _  Knox (Instance
> 2) _  |   | _ My Service2 (Standby)
>
>
>
>
>
> My custom service UI client uses both web sockets as well as REST calls to
> talk to My Service backend. I have written service definition xml and
> rewrite xml files and deployed them on a test setup as per the above
> architecture. My Knox version is 1.0 (HDP 3.1). I have added the service in
> Knox default topology and  default topology uses Shiro Provider for
> authentication and ‘gateway.websocket.feature.enabled’ is set to true.
>
>
>
> When Knox service is started, I see the following behavior:
>
>- When I launch the UI from browser, I get Shiro authentication dialog
>popup, I enter my credentials and get redirected to my Service UI.
>Initially, for the first couple of minutes, UI works fine. All calls from
>UI are getting routed properly to backend.
>- After first few minutes, I start seeing following exception in Knox
>gateway logs:
>
>2019-08-10 15:37:32,053 ERROR gateway.websockets
> (ProxyWebSocketAdapter.java:onWebSocketConnect(105)) - Unable to connect to
> websocket server: java.io.IOException: Connect failure
>
> java.io.IOException: Connect failure
>
> Caused by: org.eclipse.jetty.websocket.api.UpgradeException: 0 null
>
> at
> org.eclipse.jetty.websocket.client.WebSocketUpgradeRequest.onComplete(WebSocketUpgradeRequest.java:515)
>
> Caused by: java.io.EOFException: HttpConnectionOverHTTP@1e

Re: Knox jetty threads stuck and Seeing ConnectionPoolTimeoutException while proxying a custom service

[Sandeep Moré]

tgoing-15 << "1c"*
>
> *2019-08-17 10:34:42,714 DEBUG http.wire (Wire.java:wire(73)) -
> http-outgoing-15 << "[\r][\n]"*
>
> *2019-08-17 10:34:42,715 DEBUG http.wire (Wire.java:wire(73)) -
> http-outgoing-15 << "[\n]"*
>
> *2019-08-17 10:34:42,715 DEBUG http.wire (Wire.java:wire(73)) -
> http-outgoing-15 << "p("o");[\n]"*
>
> *2019-08-17 10:34:42,715 DEBUG http.wire (Wire.java:wire(73)) -
> http-outgoing-15 << "[\r][\n]"*
>
> *2019-08-17 10:34:42,716 DEBUG http.wire (Wire.java:wire(73)) -
> http-outgoing-15 << "[\r][\n]"*
>
> *2019-08-17 10:35:07,696 DEBUG http.wire (Wire.java:wire(87)) -
> http-outgoing-15 << "1c"*
>
> *2019-08-17 10:35:07,696 DEBUG http.wire (Wire.java:wire(73)) -
> http-outgoing-15 << "[\r][\n]"*
>
> *2019-08-17 10:35:07,697 DEBUG http.wire (Wire.java:wire(73)) -
> http-outgoing-15 << "[\n]"*
>
> *2019-08-17 10:35:07,697 DEBUG http.wire (Wire.java:wire(73)) -
> http-outgoing-15 << "p("h");[\n]"*
>
> *2019-08-17 10:35:07,697 DEBUG http.wire (Wire.java:wire(73)) -
> http-outgoing-15 << "[\r][\n]"*
>
> *2019-08-17 10:35:07,698 DEBUG http.wire (Wire.java:wire(73)) -
> http-outgoing-15 << "[\r][\n]"*
>
> *….*
>
>
>
> The last few messages in bold is recurrent data sent with a periodic
> heartbeat and is being sent by Node js backend server every 25 seconds.
> Notice that no last/end chunk message is sent from backend server and Knox
> server is just waiting for this last chunk message. This results in
> connection hang. Similar messages are seen for rest of the connections
> which are held up.  Looks like the cause here is non-standard
> implementation of chunked transfer encoding in SockJS library. Am I on the
> right path here or you feel issue might be something else ?
>
>
>
> Looking further into the issue with SockJS library and how it can be
> fixed. Thanks for the help so far.
>
>
>
> Regards,
>
> Rajat
>
>
>
> *From: *Sandeep Moré 
> *Reply to: *"user@knox.apache.org" 
> *Date: *Saturday, 17 August 2019 at 8:00 PM
> *To: *"user@knox.apache.org" 
> *Subject: *Re: Knox jetty threads stuck and Seeing
> ConnectionPoolTimeoutException while proxying a custom service
>
>
>
> This could be multiple sockets, this is weird, looks like for some reason
> there are multiple sockets opened between Knox and the backend.
>
> Can you try using just Knox and the backend and see what you get, i.e. get
> rid of the load balancers and HA to isolate the problem.
>
>
>
> On Sat, Aug 17, 2019 at 8:14 AM Rajat Goel 
> wrote:
>
> Hi Sandeep,
>
>
>
> While reproducing the issue, I observed that ‘netstat’ command shows 32
> connections in ESTABLISHED state with UI backend service:
>
> *tcp6   0  0 192.168.133.69:48510 <http://192.168.133.69:48510>
> 192.168.133.69:9443 <http://192.168.133.69:9443> ESTABLISHED 19122/java*
>
> *tcp6   0  0 192.168.133.69:53848 <http://192.168.133.69:53848>
> 192.168.133.69:9443 <http://192.168.133.69:9443> ESTABLISHED 19122/java*
>
> *tcp6   0  0 192.168.133.69:52994 <http://192.168.133.69:52994>
> 192.168.133.69:9443 <http://192.168.133.69:9443> ESTABLISHED 19122/java*
>
> *…*
>
>
>
> However, when I search for port number in gaeway.log file (full debugs
> logs enabled), there is only log mentioning that particular port:
>
> *[root@rafd001-mst-01 knox]# cat gateway.log | grep -w 48510*
>
> *2019-08-17 10:34:40,717 DEBUG conn.DefaultHttpClientConnectionOperator
> (DefaultHttpClientConnectionOperator.java:connect(146)) - Connection
> established 192.168.133.69
> <http://192.168.133.69>:48510<->192.168.133.69:9443
> <http://192.168.133.69:9443>*
>
>
>
> Looks like the connection was opened but not used? Other connections in
> netstat output show same trend. No logs in gateway.log file as to how the
> connection is getting used and these connections stay in ESTABLISHED state
> forever. They don’t even timeout. Any thoughts on why this could be
> happening ?
>
>
>
> Regards,
>
> Rajat
>
>
>
> *From: *Rajat Goel 
> *Reply to: *"user@knox.apache.org" 
> *Date: *Friday, 16 August 2019 at 11:18 PM
> *To: *"user@knox.apache.org" 
> *Subject: *Re: Knox jetty threads stuck and Seeing
> ConnectionPoolTimeoutException while proxying a custom service
>
>
>
> Hi Sandeep,
>
>
>
> You are right, websocket connection are not being established prop

Re: Does KNOX support redirection links

[Sandeep Moré]

Hello Praveen,

If the UI is using Location header to redirect (which is most of the times)
then Knox can do it, there are lot's of examples in the service defs where
we do it (e.g. yarnuiv2)

Best,
Sandeep

On Thu, Jul 18, 2019 at 3:12 PM Praveen krishnamoorthy Ravikumar <
pr...@njit.edu> wrote:

> Greetings,
>
> I’m working currently on enabling Amazon EMR debugging UIs running in
> private subnet via apache Knox. With the steps defined in the documentation
> I was able to install Knox and access YARN/SparkHistory/Ganglia UIs, which
> was amazing. But I’m facing issues on accessing certain links particularly
> the redirection links in NODE MANAGER UI.
>
> My first question, Does the Knox supports redirection internally ?. For
> instance I was trying to access the container log message in the node
> manager UI (NODEMANAGERUI -> Local Logs [under tools] -> containers/ and
> was getting blocked with the browser URL -
> http://ip-10X.us-east-1.opse.c1.com:8042/logs/containers/?host=ip-10X.us-east-1.opse.c1.com=8042
> 
>
>
> What I noticed was the links that returning Http 3xx are getting broken.
>
> Could anyone please help me resolving this issue ?
>
> Thanks,
> Praveen.
>
> Log message in gateway.log :
> —
>
> 2019-07-18 18:41:50,960 DEBUG knox.gateway
> (GatewayFilter.java:doFilter(119)) - Received request: GET
> /node/logs/containers
>
> 2019-07-18 18:41:50,963 DEBUG knox.gateway
> (UrlRewriteProcessor.java:rewrite(161)) - Rewrote URL:
> https://cas-query-sandbox-.emr.us-east-1.opse.c1.com:443/gateway/cto/node/logs/containers?host=ip-10-X.us-east-1.opse.c1.com=8042
> ,
> direction: IN via implicit rule: NODEUI/logs/containers to URL:
> http://ip-10-156X.us-east-1.opse.c1.com:8042/logs/containers?host=ip-10-156X.us-east-1.opse.c1.com=8042
> 
>
> 2019-07-18 18:41:50,964 DEBUG knox.gateway
> (UrlRewriteProcessor.java:rewrite(161)) - Rewrote URL:
> https://cas-query-sandbox.emr.us-east-1.opse.c1.com/gateway/cto/node/logs/?host=ip-X.us-east-1.opse.c1.com=8042,
> direction: IN via implicit rule: NODEUI/logs to URL:
> http://ip-10X.us-east-1.opse.c1.com:8042/logs/
> 
>
> 2019-07-18 18:41:50,964 DEBUG knox.gateway
> (DefaultDispatch.java:executeOutboundRequest(121)) - Dispatch request: GET
> http://ipX.us-east-1.opse.c1.com:8042/logs/containers?host=ip-10X.us-east-1.opse.c1.com=8042
> 
>
> 2019-07-18 18:41:50,967 DEBUG knox.gateway
> (DefaultDispatch.java:executeOutboundRequest(134)) - Dispatch response
> status: *302*
>
> 2019-07-18 18:41:50,968 DEBUG knox.gateway
> (DefaultDispatch.java:getInboundResponseContentType(203)) - Using explicit
> character set UTF-8 for entity of type text/plain
>
> 2019-07-18 18:41:50,968 DEBUG knox.gateway
> (DefaultDispatch.java:getInboundResponseContentType(211)) - Inbound
> response entity content type: text/plain; charset=utf-8
>
>
>

Re: Does KNOX support redirection links

[Sandeep Moré]

Praveen,

Can you check the response http headers that you get back that i causing
this redirection, just want to make sure we are fighting the right problem.
You can see that in in browser developer console.
The yarnuiv2 example I was talking about modifies the Location header, this
is the example
https://github.com/apache/knox/blob/512147f6fd4677baa8f8ffed6034fac256157337/gateway-service-definitions/src/main/resources/services/yarnuiv2/3.0.0/rewrite.xml#L168

Best,
Sandeep

On Thu, Jul 18, 2019 at 4:55 PM Praveen krishnamoorthy Ravikumar <
pr...@njit.edu> wrote:

> Hello Sandeep,
>
> Thank you so much for your response.
>
> In the below case -> gateway.log
>
> 2019-07-18 20:34:31,279 DEBUG knox.gateway
> (GatewayFilter.java:doFilter(119)) - Received request: GET
> /node/logs/containers
>
> 2019-07-18 20:34:31,280 DEBUG knox.gateway
> (UrlRewriteProcessor.java:rewrite(161)) - Rewrote URL:
> https://ec2-3-80-173-38.compute-1.amazonaws.com:8446/gateway/gate1/node/logs/containers?host=ip-172-31-8-223.ec2.internal=8042,
> direction: IN via implicit rule: NODEUI/logs2 to URL:
> http://ip-172-31-8-223.ec2.internal:8042/logs/containers
>
> 2019-07-18 20:34:31,281 DEBUG knox.gateway
> (UrlRewriteProcessor.java:rewrite(161)) - Rewrote URL:
> https://ec2-3-80-173-38.compute-1.amazonaws.com:8446/gateway/gate1/node/logs/?host=ip-172-31-8-223.ec2.internal=8042,
> direction: IN via implicit rule: NODEUI/logs to URL:
> http://ip-172-31-8-223.ec2.internal:8042/logs/
>
> 2019-07-18 20:34:31,281 DEBUG knox.gateway
> (DefaultDispatch.java:executeOutboundRequest(121)) - Dispatch request: GET
> http://ip-172-31-8-223.ec2.internal:8042/logs/containers?user.name=admin
>
> 2019-07-18 20:34:31,285 DEBUG knox.gateway
> (DefaultDispatch.java:executeOutboundRequest(134)) - Dispatch response
> status: 302
> 2019-07-18 20:34:31,285 DEBUG knox.gateway
> (DefaultDispatch.java:getInboundResponseContentType(203)) - Using explicit
> character set UTF-8 for entity of type text/plain
> 2019-07-18 20:34:31,285 DEBUG knox.gateway
> (DefaultDispatch.java:getInboundResponseContentType(211)) - Inbound
> response entity content type: text/plain; charset=utf-8
>
> I see it is redirecting to “http://ip-172X:8042/logs/containers?user.admin
> <http://ip-172x:8042/logs/containers?user.admin>” - which is the direct
> ip link to the slave node.
>
> So Do I need to define seperate redirection rules to make it work?.
>
> Can you also share me yarnuiv2 service def links that I can refer, would
> be very helpful ?
>
>
> Thanks,
> Praveen.
>
> On Jul 18, 2019, at 4:36 PM, Sandeep Moré  wrote:
>
> Hello Praveen,
>
> If the UI is using Location header to redirect (which is most of the
> times) then Knox can do it, there are lot's of examples in the service defs
> where we do it (e.g. yarnuiv2)
>
> Best,
> Sandeep
>
> On Thu, Jul 18, 2019 at 3:12 PM Praveen krishnamoorthy Ravikumar <
> pr...@njit.edu> wrote:
>
>> Greetings,
>>
>> I’m working currently on enabling Amazon EMR debugging UIs running in
>> private subnet via apache Knox. With the steps defined in the documentation
>> I was able to install Knox and access YARN/SparkHistory/Ganglia UIs, which
>> was amazing. But I’m facing issues on accessing certain links particularly
>> the redirection links in NODE MANAGER UI.
>>
>> My first question, Does the Knox supports redirection internally ?. For
>> instance I was trying to access the container log message in the node
>> manager UI (NODEMANAGERUI -> Local Logs [under tools] -> containers/ and
>> was getting blocked with the browser URL -
>> http://ip-10X.us-east-1.opse.c1.com:8042/logs/containers/?host=ip-10X.us-east-1.opse.c1.com=8042
>> <http://ip-10x.us-east-1.opse.c1.com:8042/logs/containers/?host=ip-10X.us-east-1.opse.c1.com=8042>
>>
>>
>> What I noticed was the links that returning Http 3xx are getting broken.
>>
>> Could anyone please help me resolving this issue ?
>>
>> Thanks,
>> Praveen.
>>
>> Log message in gateway.log :
>> —
>>
>> 2019-07-18 18:41:50,960 DEBUG knox.gateway
>> (GatewayFilter.java:doFilter(119)) - Received request: GET
>> /node/logs/containers
>>
>> 2019-07-18 18:41:50,963 DEBUG knox.gateway
>> (UrlRewriteProcessor.java:rewrite(161)) - Rewrote URL:
>> https://cas-query-sandbox-.emr.us-east-1.opse.c1.com:443/gateway/cto/node/logs/containers?host=ip-10-X.us-east-1.opse.c1.com=8042
>> <https://cas-query-sandbox-sc-866316499622-pp-oeobt24ki6aja.emr.us-east-1.opse.c1.vanguard.com/gateway/cto/node/logs/containers?host=ip-10-156-232-245.us-east-1.opse.c1.vanguard.com=8042>,
>> direction: IN via im

Re: [VOTE] Release RC 2 as Apache Knox 1.3.0

[Sandeep Moré]

I am running into issues testing Zeppelin on a non-secure cluster, I can
create an empty notebook but I cannot open it, when I try to open it I get
prompted for login. I could not find error messages in Knox logs, I was
able to confirm websockets are working and I can see the data going back
and forth. Anybody see this issue ? will try to test it again on a secure
cluster.

Best,
Sandeep



On Wed, Jul 17, 2019 at 8:37 AM larry mccay  wrote:

> All -
>
> A candidate for the Apache Knox 1.3.0 release is available at:
>
> https://dist.apache.org/repos/dist/dev/knox/knox-1.3.0/
>
> This RC addresses multiple issues identified in the testing of RC 1:
>   1. Build problem related to DNS and SAN within the self signed cert
>   2. libpam4j issue
>   3. Upgrade to latest LTS of npm
>
> The release candidate is a zip archive of the sources in:
>
> https://https://gitbox.apache.org/repos/asf/knox.git
> Branch v1.3.0 (git checkout -b v1.3.0)
>
> The KEYS file for signature validation is available at:
> https://dist.apache.org/repos/dist/release/knox/KEYS
>
> Please vote on releasing this package as Apache Knox 1.3.0.
> The vote is open for the next 72 hours and passes if a majority of at
> least three +1 Apache Knox PMC votes are cast.
>
> [ ] +1 Release this package as Apache Knox 1.3.0
> [ ] -1 Do not release this package because...
>
> thanks,
>
> --larry
>

Re: [DISCUSS] - Integrating Knox with Swagger

[Sandeep Moré]

Agreed +1 !

On Wed, Nov 13, 2019 at 4:05 PM larry mccay  wrote:

> + dev@...
>
> Thank you for the idea!
>
> Yes, I am familiar with Swagger and that would be huge for our current
> APIs and others that may come along.
> I think the effort to add a swagger filter or the like will be only one
> part of the larger effort of how it integrates into Knox, the site, the
> Admin UI, the docs, etc.
>
> I think that a KIP detailing the full vision would make a lot of sense.
>
> +1 on moving forward with a KIP and/or JIRA to continue discussion and
> tease out design considerations.
>
> On Wed, Nov 13, 2019 at 3:45 PM Sandor Molnar 
> wrote:
>
>> Hi folks,
>>
>> recently I had some contribution that allows end-users managing service
>> definitions without restarting the Knox Gateway. See
>> https://issues.apache.org/jira/browse/KNOX-2053 and
>> https://issues.apache.org/jira/browse/KNOX-2056 for further details.
>>
>> I've been just about creating a new JIRA to document those new API
>> endpoints in Knox user guide but it has come to my mind that we can do it
>> much better by using Swagger.
>>
>> Given the fact, the Admin API does not consist of hundreds of existing
>> endpoints it should not be 'that' huge work. I personally believe the
>> project would gain a lot by using this very useful tool (e.g. Admin API
>> documentation would be generated out-of-the-box, no more documentation
>> JIRAs required. Moreover, the generated documentation would be in sync with
>> the actual implementation).
>> You can check this out here: https://swagger.io/tools/swagger-ui/ (there
>> is a live demo too; it's worth looking at).
>>
>> Any comments, ideas are welcome!
>>
>> Cheers,
>> Sandor
>>
>

Re: [DISCUSS] Planning for Apache Knox 1.4

[Sandeep Moré]

Thanks for starting the planning thread Larry !
Agree with the theme and the release date for 1.4.0.

+1

Best,
Sandeep

On Thu, Oct 31, 2019 at 11:53 AM larry mccay  wrote:

> Folks -
>
> Out last release with end of July, I apologize for the delay in starting
> the planning thread for 1.4.
>
> We currently have a backlog of ~65 JIRAs slated for a Fix Version of 1.4.
>
> There has been some work going on within KnoxShell to provide a general
> purpose representation for tabular data. This will be leveraged for
> rendering SQL query results as well as CSV files and simple processing
> within KnoxShell. I will be writing up a KIP to represent the overall
> vision for this work and initial set of usecases.
>
> We also have Cloudera Manager based discovery emerging and we should
> target an initial set of services to enable for CM/CDH and CDP deployments
> where CM is available.
>
> With the continued increase in cloud based deployments and Knox Gateway
> use in securely accessing the exposed resources, we will concentrate on
> KnoxShell as a first class environment for this access. This will likely
> include an API for discovering metadata about the resources exposed through
> Knox, the required authentication mechanisms, resource types and public
> certs. It will also include Custom GroovyShell Commands for the KnoxShell
> environment to help interact with the remote clusters and resultsets as
> local in-memory tables. I will be start a KIP to try and articulate this
> vision and related 1.4. usecases as well.
>
> I propose that the CM based Service Discovery and KnoxShell access to
> remote clusters be the primary themes of the Apache Knox 1.4 release.
>
> I also propose that we target the end of November as the release date for
> 1.4.
>
> Thoughts?
>
> --larry
>

Re: ldaps and 2 domain controller

[Sandeep Moré]

Hello,
So, you want to configure Knox for both the ldap hosts is it?
Perhaps, configuring Knox with multiple LDAP realms [1] might help.

Best,
Sandeep


[1]
https://cwiki.apache.org/confluence/display/KNOX/2017/03/01/Apache+Knox+using+multiple+LDAP+Realms



On Tue, Mar 17, 2020 at 3:47 AM  wrote:

> Hi!
>
> I want to use ldaps to a windows domain. The domain is built with to
> domain controller for high availability reasons and ech host has a
> certificate. The domain  4711.company.com for example doesn't have a
> certificate.
>
> So knox works with one host in the ldap url only. Is there a way to
> configure both hosts ? I tried a load balancer vip before the 2 domain
> controllers but knox doesn't accept it.
>
>
> Best Regards
>
>
> Markus
>
>
>
> Fiducia & GAD IT AG | *www.fiduciagad.de* 
> AG Frankfurt a. M. HRB 102381 | Sitz der Gesellschaft: Frankfurt a. M. |
> USt-IdNr. DE 143582320
> Vorstand: Martin Beyer (Vorstandssprecher), Birgit Frohnhoff, Jörg Staff
> Vorsitzender des Aufsichtsrats: Jürgen Brinkmann
>
>

Re: Knox URLDecodingDispatch not working

[Sandeep Moré]

Hello Eric,
Sorry for the late reply. The reason why you might be seeing
DefaultDispatch in logs could be because other dispatches do not have
logging statements in them and extend DefaultDispatch (e.g.
URLDecodingDispatch).

This is weird,this should have fixed the issue. What do you see in the
debug logs? debug logs should spit out the url Knox is dispatching request
to.

On Tue, Feb 23, 2021 at 11:13 AM Chufeng Gao  wrote:

> Hello,
>
>
>
> I am a user of Knox 1.15.0. I tried to use it for the security of airflow
> Web UI. However, I got kind of ‘double encoded’issue here.
>
>
>
> I got some encoded url pattern from Knox like this:
>
> ‘
> https://*.***.***.***:8443/gateway/sandbox/airflow/admin/airflow/trigger?dag_id=run_kaggle=%252Fadmin%252Fairflow%252Ftree%253Fdag_id%253Drun_kaggle
> ’.
>
> It seems the ‘/’in the parameters part has been encoded twice.
>
>
>
> Therefore, I added  classname="org.apache.knox.gateway.dispatch.URLDecodingDispatch"/> in
> service.xml to solve this problem. Then I cleaned all the stuff under
> data/deployments and restarted the gateway. However, nothing changes. I
> turned on the debug log and it seemed knox was still using DefaultDispatch.
>
>
>
> Also I had tried several other dispatches or tried to use configurable
> dispatch. None of them worked. The debug log always showed that I
> DefaultDispatch was used.
>
>
>
> I have been blocked by this issue for quite some time. Any help will be
> appreciated!
>
>
>
> Sincerely,
>
>
>
> Eric Gao
>
>
>

Re: [VOTE] Release Apache Knox 1.6.0 - RC 1

[Sandeep Moré]

Thank you Sandor! and apologies for the delay.
Here is my +1

---
* Downloaded and built from source
* Checked LICENSE and NOTICE files
* Verified GPG/SHA signatures for Knox source, Knox and Knoxshell release
packages
* Installed pseudo-distributed instance (Mac OS X )
* Checked KnoxShell samples

Thanks,
Sandeep

On Fri, Oct 15, 2021 at 4:40 PM Sandor Molnar  wrote:

> Hi folks!
>
> Release candidate #1 for the Apache Knox 1.6.0 release is available at:
>
> https://dist.apache.org/repos/dist/dev/knox/knox-1.6.0/
>
> The release candidate is a zip archive of the sources in:
>
> https://gitbox.apache.org/repos/asf/knox.git
> Branch v1.6.0 (git checkout -b v1.6.0)
>
> The KEYS file for signature validation is available at:
> https://dist.apache.org/repos/dist/release/knox/KEYS
>
> KnoxShell User Guide:
> http://knox.apache.org/books/knox-1-6-0/knoxshell_user_guide.html
>
> Gateway User Guide:
> http://knox.apache.org/books/knox-1-6-0/user-guide.html
>
> Dev Guide:
> http://knox.apache.org/books/knox-1-6-0/dev-guide.html
>
> Please vote on releasing this package as Apache Knox 1.6.0.
> The vote is open for the next 96 hours and passes if a majority of at
> least three +1 Apache Knox PMC votes are cast.
>
> [ ] +1 Release this package as Apache Knox 1.6.0
> [ ] -1 Do not release this package because...
>
> Thanks,
> Sandor
>

Re: [VOTE] Release Apache Knox 1.6.0 - RC 1

[Sandeep Moré]

Sounds great, I’ll file a JIRA with the details.

On Fri, Oct 22, 2021 at 8:26 AM Sandor Molnar 
wrote:

> Thanks, Sandeep!
>
> I'd call this issue a release blocker and we need to file a JIRA to fix
> this and kick-off a new release candidate.
>
> Any objection?
>
> On Fri, Oct 22, 2021 at 2:02 PM Sandeep Moré 
> wrote:
>
> > **Update: vote -1
> > I discovered a bug while testing the logout feature so I need to withdraw
> > my vote.
> > This is the error I see when I click the logout button
> >
> > --
> > HTTP ERROR 500 javax.servlet.ServletException:
> > org.apache.jasper.JasperException: Unable to compile class for JSP: An
> > error occurred at line: [155] in the jsp file: [/logout.jsp] response
> > cannot be resolved 152: } catch (Exception e) { 153: return ""; 154: }
> 155:
> > response.setStatus(HttpServletResponse.SC_TEMPORARY_REDIRECT); 156:
> > response.setHeader("Location", globalLogoutPageURL); 157: return; 158: %>
> > An error occurred at line: [156] in the jsp file: [/logout.jsp] response
> > cannot be resolved 153: return ""; 154: } 155:
> > response.setStatus(HttpServletResponse.SC_TEMPORARY_REDIRECT); 156:
> > response.setHeader("Location", globalLogoutPageURL); 157: return; 158: %>
> > 159:  An error occurred at line: [156] in the jsp file:
> > [/logout.jsp] globalLogoutPageURL cannot be resolved to a variable 153:
> > return ""; 154: } 155:
> > response.setStatus(HttpServletResponse.SC_TEMPORARY_REDIRECT); 156:
> > response.setHeader("Location", globalLogoutPageURL); 157: return; 158: %>
> > 159:  An error occurred at line: [157] in the jsp file:
> > [/logout.jsp] This method must return a result of type String 154: } 155:
> > response.setStatus(HttpServletResponse.SC_TEMPORARY_REDIRECT); 156:
> > response.setHeader("Location", globalLogoutPageURL); 157: return; 158: %>
> > 159:  160: An error occurred at line: [157] in the jsp file:
> > [/logout.jsp] Syntax error, insert "}" to complete MethodBody 154: } 155:
> > response.setStatus(HttpServletResponse.SC_TEMPORARY_REDIRECT); 156:
> > response.setHeader("Location", globalLogoutPageURL); 1
> >
> > On Thu, Oct 21, 2021 at 3:32 PM Sandeep Moré 
> > wrote:
> >
> >> Thank you Sandor! and apologies for the delay.
> >> Here is my +1
> >>
> >> ---
> >> * Downloaded and built from source
> >> * Checked LICENSE and NOTICE files
> >> * Verified GPG/SHA signatures for Knox source, Knox and Knoxshell
> release
> >> packages
> >> * Installed pseudo-distributed instance (Mac OS X )
> >> * Checked KnoxShell samples
> >>
> >> Thanks,
> >> Sandeep
> >>
> >> On Fri, Oct 15, 2021 at 4:40 PM Sandor Molnar 
> >> wrote:
> >>
> >>> Hi folks!
> >>>
> >>> Release candidate #1 for the Apache Knox 1.6.0 release is available at:
> >>>
> >>> https://dist.apache.org/repos/dist/dev/knox/knox-1.6.0/
> >>>
> >>> The release candidate is a zip archive of the sources in:
> >>>
> >>> https://gitbox.apache.org/repos/asf/knox.git
> >>> Branch v1.6.0 (git checkout -b v1.6.0)
> >>>
> >>> The KEYS file for signature validation is available at:
> >>> https://dist.apache.org/repos/dist/release/knox/KEYS
> >>>
> >>> KnoxShell User Guide:
> >>> http://knox.apache.org/books/knox-1-6-0/knoxshell_user_guide.html
> >>>
> >>> Gateway User Guide:
> >>> http://knox.apache.org/books/knox-1-6-0/user-guide.html
> >>>
> >>> Dev Guide:
> >>> http://knox.apache.org/books/knox-1-6-0/dev-guide.html
> >>>
> >>> Please vote on releasing this package as Apache Knox 1.6.0.
> >>> The vote is open for the next 96 hours and passes if a majority of at
> >>> least three +1 Apache Knox PMC votes are cast.
> >>>
> >>> [ ] +1 Release this package as Apache Knox 1.6.0
> >>> [ ] -1 Do not release this package because...
> >>>
> >>> Thanks,
> >>> Sandor
> >>>
> >>
>

Re: [VOTE] Release Apache Knox 1.6.0 - RC 1

[Sandeep Moré]

**Update: vote -1
I discovered a bug while testing the logout feature so I need to withdraw
my vote.
This is the error I see when I click the logout button

--
HTTP ERROR 500 javax.servlet.ServletException:
org.apache.jasper.JasperException: Unable to compile class for JSP: An
error occurred at line: [155] in the jsp file: [/logout.jsp] response
cannot be resolved 152: } catch (Exception e) { 153: return ""; 154: } 155:
response.setStatus(HttpServletResponse.SC_TEMPORARY_REDIRECT); 156:
response.setHeader("Location", globalLogoutPageURL); 157: return; 158: %>
An error occurred at line: [156] in the jsp file: [/logout.jsp] response
cannot be resolved 153: return ""; 154: } 155:
response.setStatus(HttpServletResponse.SC_TEMPORARY_REDIRECT); 156:
response.setHeader("Location", globalLogoutPageURL); 157: return; 158: %>
159:  An error occurred at line: [156] in the jsp file:
[/logout.jsp] globalLogoutPageURL cannot be resolved to a variable 153:
return ""; 154: } 155:
response.setStatus(HttpServletResponse.SC_TEMPORARY_REDIRECT); 156:
response.setHeader("Location", globalLogoutPageURL); 157: return; 158: %>
159:  An error occurred at line: [157] in the jsp file:
[/logout.jsp] This method must return a result of type String 154: } 155:
response.setStatus(HttpServletResponse.SC_TEMPORARY_REDIRECT); 156:
response.setHeader("Location", globalLogoutPageURL); 157: return; 158: %>
159:  160: An error occurred at line: [157] in the jsp file:
[/logout.jsp] Syntax error, insert "}" to complete MethodBody 154: } 155:
response.setStatus(HttpServletResponse.SC_TEMPORARY_REDIRECT); 156:
response.setHeader("Location", globalLogoutPageURL); 1

On Thu, Oct 21, 2021 at 3:32 PM Sandeep Moré  wrote:

> Thank you Sandor! and apologies for the delay.
> Here is my +1
>
> ---
> * Downloaded and built from source
> * Checked LICENSE and NOTICE files
> * Verified GPG/SHA signatures for Knox source, Knox and Knoxshell release
> packages
> * Installed pseudo-distributed instance (Mac OS X )
> * Checked KnoxShell samples
>
> Thanks,
> Sandeep
>
> On Fri, Oct 15, 2021 at 4:40 PM Sandor Molnar 
> wrote:
>
>> Hi folks!
>>
>> Release candidate #1 for the Apache Knox 1.6.0 release is available at:
>>
>> https://dist.apache.org/repos/dist/dev/knox/knox-1.6.0/
>>
>> The release candidate is a zip archive of the sources in:
>>
>> https://gitbox.apache.org/repos/asf/knox.git
>> Branch v1.6.0 (git checkout -b v1.6.0)
>>
>> The KEYS file for signature validation is available at:
>> https://dist.apache.org/repos/dist/release/knox/KEYS
>>
>> KnoxShell User Guide:
>> http://knox.apache.org/books/knox-1-6-0/knoxshell_user_guide.html
>>
>> Gateway User Guide:
>> http://knox.apache.org/books/knox-1-6-0/user-guide.html
>>
>> Dev Guide:
>> http://knox.apache.org/books/knox-1-6-0/dev-guide.html
>>
>> Please vote on releasing this package as Apache Knox 1.6.0.
>> The vote is open for the next 96 hours and passes if a majority of at
>> least three +1 Apache Knox PMC votes are cast.
>>
>> [ ] +1 Release this package as Apache Knox 1.6.0
>> [ ] -1 Do not release this package because...
>>
>> Thanks,
>> Sandor
>>
>

Re: [VOTE] Release Apache Knox 1.6.1 - RC 1

[Sandeep Moré]

Thank you Sandor,
While checking the NOTICE file I realized that we have copyright date as
2021, it is minor and I am okay with it since RC was spun up last year but
given this would be released this year wanted to bring it to everyone's
attention.

Else, here is my +1
* Downloaded and built from source
* Checked LICENSE and NOTICE files
* Verified GPG/SHA signatures for Knox source, Knox and Knoxshell release
packages

Thanks,
sandeep

On Tue, Jan 4, 2022 at 12:56 PM Sandor Molnar  wrote:

> Hello everyone!
>
> This is a gentle reminder that I am still waiting for your votes on Apache
> Knox 1.6.1 RC1.
>
> Thanks!
>
> On Wed, Dec 15, 2021 at 9:50 AM Sandor Molnar 
> wrote:
>
>> Hi folks!
>>
>> Release candidate #1 for the Apache Knox 1.6.1 release is available at:
>> https://dist.apache.org/repos/dist/dev/knox/knox-1.6.1/
>>
>> The release candidate is a zip archive of the sources in:
>> https://gitbox.apache.org/repos/asf/knox.git
>> Branch v1.6.1 (git checkout -b v1.6.1)
>>
>> The KEYS file for signature validation is available at:
>> https://dist.apache.org/repos/dist/release/knox/KEYS
>>
>> Please find the most recent changes here:
>> https://github.com/apache/knox/blob/v1.6.1/CHANGES
>>
>> KnoxShell User Guide:
>> http://knox.apache.org/books/knox-1-6-0/knoxshell_user_guide.html
>>
>> Gateway User Guide:
>> http://knox.apache.org/books/knox-1-6-0/user-guide.html
>>
>> Dev Guide:
>> http://knox.apache.org/books/knox-1-6-0/dev-guide.html
>>
>> Please vote on releasing this package as Apache Knox 1.6.1.
>> The vote is open for the next 72 hours and passes if a majority of at
>> least three +1 Apache Knox PMC votes are cast.
>>
>> [ ] +1 Release this package as Apache Knox 1.6.1
>> [ ] -1 Do not release this package because...
>>
>> Thanks,
>> Sandor
>>
>

Re: Topology redeployment changes

[Sandeep Moré]

Thanks Sandor!
I am also CC'ing the dev mailing list.

Thank you for the patch and the heads up :)

On Tue, Nov 23, 2021 at 4:12 AM Sandor Molnar  wrote:

> Hi again!
>
> I just wanted to add some clarification on the above news. If you actually
> change the topology (adding a comment is not considered a change) and save
> it, Knox will continue to redeploy it w/o invoking the previously
> referenced KnoxCLI *redeploy* command. That command is needed only if you
> want to redeploy your topology without any change (to replace the
> well-known touch mechanism).
>
> Sandor
>
>
> On Tue, Nov 23, 2021 at 9:26 AM Sandor Molnar 
> wrote:
>
>> Hi everyone!
>>
>> I've just merged a commit to fix
>> https://issues.apache.org/jira/browse/KNOX-2689. With my changes in
>> place, you will no longer be able to redeploy an XML-based topology by
>> simply touching it. Instead, you will have to run the following command:
>>
>> knoxcli.sh redeploy --cluster topologyName
>>
>> E.g. knoxcli.sh redeploy --cluster sandbox
>>
>> The related commit is merged into the master branch today that
>> corresponds to v2.0.0 (this version is not yet released). If you build the
>> project from the source and use it your own, please remember the
>> above-written changes.
>>
>> Cheers,
>> Sandor
>>
>

Re: [VOTE] Release Apache Knox 1.6.0 - RC 3

[Sandeep Moré]

Here is my +1

* Downloaded and built from source
* Checked LICENSE and NOTICE files
* Verified GPG/SHA signatures for Knox source, Knox and Knoxshell release
packages
* Installed pseudo-distributed instance (Mac OS X )

On Sun, Oct 31, 2021 at 5:30 PM Sandor Molnar  wrote:

> Hi folks!
>
> Release candidate #3 for the Apache Knox 1.6.0 release is available at:
>
> https://dist.apache.org/repos/dist/dev/knox/knox-1.6.0/
>
> The release candidate is a zip archive of the sources in:
>
> https://gitbox.apache.org/repos/asf/knox.git
> Branch v1.6.0 (git checkout -b v1.6.0)
>
> The KEYS file for signature validation is available at:
> https://dist.apache.org/repos/dist/release/knox/KEYS
>
> KnoxShell User Guide:
> http://knox.apache.org/books/knox-1-6-0/knoxshell_user_guide.html
>
> Gateway User Guide:
> http://knox.apache.org/books/knox-1-6-0/user-guide.html
>
> Dev Guide:
> http://knox.apache.org/books/knox-1-6-0/dev-guide.html
>
> Please vote on releasing this package as Apache Knox 1.6.0.
> The vote is open for the next 96 hours and passes if a majority of at
> least three +1 Apache Knox PMC votes are cast.
>
> [ ] +1 Release this package as Apache Knox 1.6.0
> [ ] -1 Do not release this package because...
>
> Thanks,
> Sandor
>

Re: [VOTE] Release Apache Knox 1.6.1 - RC 1

[Sandeep Moré]

On Tue, Jan 4, 2022 at 3:47 PM Sandeep Moré  wrote:

> Thank you Sandor,
> While checking the NOTICE file I realized that we have copyright date as
> 2021, it is minor and I am okay with it since RC was spun up last year but
> given this would be released this year wanted to bring it to everyone's
> attention.
>
> Else, here is my +1
> * Downloaded and built from source
> * Checked LICENSE and NOTICE files
> * Verified GPG/SHA signatures for Knox source, Knox and Knoxshell release
> packages
>
> Thanks,
> sandeep
>
> On Tue, Jan 4, 2022 at 12:56 PM Sandor Molnar 
> wrote:
>
>> Hello everyone!
>>
>> This is a gentle reminder that I am still waiting for your votes on
>> Apache Knox 1.6.1 RC1.
>>
>> Thanks!
>>
>> On Wed, Dec 15, 2021 at 9:50 AM Sandor Molnar 
>> wrote:
>>
>>> Hi folks!
>>>
>>> Release candidate #1 for the Apache Knox 1.6.1 release is available at:
>>> https://dist.apache.org/repos/dist/dev/knox/knox-1.6.1/
>>>
>>> The release candidate is a zip archive of the sources in:
>>> https://gitbox.apache.org/repos/asf/knox.git
>>> Branch v1.6.1 (git checkout -b v1.6.1)
>>>
>>> The KEYS file for signature validation is available at:
>>> https://dist.apache.org/repos/dist/release/knox/KEYS
>>>
>>> Please find the most recent changes here:
>>> https://github.com/apache/knox/blob/v1.6.1/CHANGES
>>>
>>> KnoxShell User Guide:
>>> http://knox.apache.org/books/knox-1-6-0/knoxshell_user_guide.html
>>>
>>> Gateway User Guide:
>>> http://knox.apache.org/books/knox-1-6-0/user-guide.html
>>>
>>> Dev Guide:
>>> http://knox.apache.org/books/knox-1-6-0/dev-guide.html
>>>
>>> Please vote on releasing this package as Apache Knox 1.6.1.
>>> The vote is open for the next 72 hours and passes if a majority of at
>>> least three +1 Apache Knox PMC votes are cast.
>>>
>>> [ ] +1 Release this package as Apache Knox 1.6.1
>>> [ ] -1 Do not release this package because...
>>>
>>> Thanks,
>>> Sandor
>>>
>>