Re: [RESULT] [VOTE] Release Apache Knox 2.0.0 - RC 2

[larry mccay]

Congrats on the 2.0 milestone everyone - looking forward to the release!

Thanks, @Phil Zampino  for managing the release!

On Fri, Feb 24, 2023 at 3:49 PM Phil Zampino  wrote:

> The VOTE for 2.0.0 RC 2 passes with: 4 binding +1's
> 0 -1's I will be working on promoting 2.0.0 RC 2 to an official release
> shortly. Thank you for taking the time to test this release and
> contributing to the Apache Knox community!
>
> - Phil
>

Re: [VOTE] Release Apache Knox 2.0.0 - RC2

[larry mccay]

+1 for RC2

* Verified signatures
* Checked NOTICE, LICENSE, CHANGES
* Built from source and ran unit tests
* Tested KnoxSSO, Homepage
* Tested Token Management Page
* Tested JWTProvider with JWT as HTTP Basic and Bearer Token
* Tested APIs via KnoxShell: webhdfs, hive
* Tested proxying Livy job submits
* Tested KnoxShellTable operations

On Mon, Feb 13, 2023 at 9:42 AM Sandor Molnar  wrote:

> Here is my +1 for RC2:
>
> I executed the same steps as for RC1:
> - verified the signatures (Good signature from "Phil Zampino (CODE SIGNING
> KEY) " [full])
> - checked NOTICE, LICENSE, CHANGES files
> - downloaded the source and built the project from scratch (excluded
> static code analyzer tools)
> - executed some of the KnoxCLI commands (including the newly created
> 'generate-jwk')
> - ran simple commands (such as ls, pwd, whoami, etc...) on the brand new
> Knox's WebShell UI
> - validated new improvements and bugfixes on Token Generation and Token
> Management UIs (also enabled token impersonation in identity-assertion
> layer)
> - verified the newly introduced Concurrent Session limit feature using
> different browsers (based on @mbalazs's samples)
> - tested new service discovery features in a live CM cluster
> - verified there is no regression in Knox's HadoopAuth authentication
> filter: created a topology with this provider and the KNOXTOKEN service.
> After logging into Kerberos I could fetch a token successfully
>
> Thanks, Phil!
>
> On Mon, Feb 13, 2023 at 11:06 AM Attila Magyar 
> wrote:
>
>> I performed the following feature tests:
>>
>>  - Built the package from source
>>  - Tested SQL DB based topology monitor
>>  - Tested "Accessing parameters of a x-www-form-urlencoded request
>> consumes the request body"
>>  - Tested "Login on Knox UI doesn't work when the password contains
>> special characters"
>>  - Virtual group mapping
>>  - Checked if there is no log4j1 dependency on the classpath
>>  - Enabled and tested webshell
>>
>> +1 votre from me too
>>
>> On Thu, Feb 9, 2023 at 7:19 PM Phil Zampino  wrote:
>>
>>> Release candidate #2 for the Apache Knox 2.0.0 release is available at:
>>> https://dist.apache.org/repos/dist/dev/knox/knox-2.0.0/
>>>
>>> The release candidate is a zip archive of the sources in:
>>> https://gitbox.apache.org/repos/asf/knox.git
>>> Branch v2.0.0 (git checkout -b v2.0.0)
>>>
>>> The KEYS file for signature validation is available at:
>>> https://dist.apache.org/repos/dist/release/knox/KEYS
>>>
>>> Please find the most recent changes here:
>>> https://github.com/apache/knox/blob/v2.0.0/CHANGES
>>>
>>> KnoxShell User Guide:
>>> https://knox.apache.org/books/knox-2-0-0/knoxshell_user_guide.html
>>>
>>> Gateway User Guide:
>>> https://knox.apache.org/books/knox-2-0-0/user-guide.html
>>>
>>> Dev Guide:
>>> https://knox.apache.org/books/knox-2-0-0/dev-guide.html
>>>
>>> Please vote on releasing this package as Apache Knox 2.0.0.
>>> The vote is open for the next 72 hours and passes if a majority of at
>>> least three +1 Apache Knox PMC votes are cast.
>>>
>>> [ ] +1 Release this package as Apache Knox 2.0.0
>>> [ ] -1 Do not release this package because...
>>>
>>> Thanks,
>>>   Phil
>>>
>>
>>
>> --
>> *Attila Magyar* | Staff Software Engineer
>>
>> cloudera.com 
>>
>> [image: Cloudera] 
>>
>> [image: Cloudera on Twitter]  [image:
>> Cloudera on Facebook]  [image:
>> Cloudera on LinkedIn] 
>> --
>>
>

Re: Apache Commons Text issue - CVE 2022-42889

[larry mccay]

I do not believe so but there will be a more indepth investigation and
likely an upgrade in the next release.

:~/Projects/knox$ grep -r StringSubstitutor .
Binary file ./install/knox-2.0.0-SNAPSHOT/dep/commons-text-1.9.jar matches

It appears to only exist as the API in the lib itself.

On Mon, Oct 24, 2022 at 12:36 PM O'Connell, Richard via user <
user@knox.apache.org> wrote:

> Hi,
>
> Is Knox affected by this vulnerability ?
>
> https://commons.apache.org/proper/commons-text/security.html
>
>
>
> ………
>
>
>
> On 2022-10-13, the Apache Commons Text team disclosed CVE-2022-42889
>  . Key takeaways:
>
> · If you rely on software that uses a version of commons-text
> prior to 1.10.0, you are likely still not vulnerable: only if this software
> uses the StringSubstitutor API without properly sanitizing any untrusted
> input.
>
> · If your own software uses commons-text, double-check whether it
> uses the StringSubstitutor API without properly sanitizing any untrusted
> input. If so, an update to 1.10.0 could be a quick workaround, but the
> recommended solution is to also properly validate and sanitize any
> untrusted input.
>
>
>
>
>

Re: Missing groups and/or roles claims with KnoxSSO in hadoop-jwt cookie

[larry mccay]

Hi -

Thank you for your question.
You are correct in that neither roles nor groups are included in the JWT
token for KNOXSSO or KNOXTOKEN services.
I believe we are adding (or may have already added an option) to request
groups as well but there is nothing for roles.

The separation of authentication and group lookup is pretty common
throughout the Hadoop ecosystem as you may have noticed.
Knox being a trusted proxyuser in that ecosystem by definition asserts the
identity of the authenticated user to the backend service when we are
proxying access to it.
There is nothing in the proxyuser contract for propagating the group
memberships.
Group lookup is always done closest to the resource being accessed.

Therefore, we only assert the authenticated user name in the JWT as well.
The service consuming the JWT is obligated to determine the groups that are
relevant for that user principal within the operating environment.
Note that these may be completely different groups than those available
from the original IDP.

If you provide more details around what you are trying to do, we may be
able to provide direction to get the groups that are expected from the
operating environment.
Roles are not something that we can do with existing code however.

Hope that is helpful!

--larry

On Sat, Sep 24, 2022 at 4:46 AM Malthe  wrote:

> When using pac4j with OIDC, I am able to see groups and/or roles in the
> log:
>
> 2022-09-24 10:25:06,600 DEBUG filter.Pac4jIdentityAdapter
> (Pac4jIdentityAdapter.java:doFilter(92)) - User authenticated as:
> #OidcProfile# | id: [REDACTED] | attributes: {sub=[REDACTED],
> amr=["pwd"], roles=["My role"],
> iss=https://sts.windows.net/[REDACTED]/, oid=[REDACTED],
> preferred_username=[REDACTED], tid=[REDACTED], ipaddr=[REDACTED],
> exp=Sat Sep 24 11:25:05 CEST 2022, iat=Sat Sep 24 10:20:05 CEST 2022,
> email=[REDACTED], ver=1.0, groups=["[\"group1\",\"group2\"]"],
> uti=[REDACTED], given_name=[REDACTED], token_expiration_advance=-1,
> aud=[[REDACTED]], unique_name=[REDACTED], nbf=Sat Sep 24 10:20:05 CEST
> 2022, idp=live.com, rh=[REDACTED], name=[REDACTED], expiration=Sat Sep
> 24 11:25:04 CEST 2022, family_name=[REDACTED]} | roles: [] |
> permissions: [] | isRemembered: false | clientName: OidcClient |
> linkedId: null |
>
> In the above, notice that the middle part contains values for "roles"
> and "groups".
>
> Now, neither of those are written into the JWT (hadoop-jwt) cookie.
>
> The JWT payload:
>
> {
>   "sub": "[REDACTED]",
>   "kid": "[REDACTED]",
>   "iss": "KNOXSSO",
>   "exp": 1664094307,
>   "managed.token": "false",
>   "knox.id": "[REDACTED]"
> }
>
> This happens even though I have:
>
> pac4j.session.store.exclude.groups=false
> pac4j.session.store.exclude.roles=false
>
> Are groups and/or roles not supposed to be written into the JWT with
> these settings?
>
> Cheers
>

CVE-2021-42357: DOM based XSS Vulnerability in Apache Knox

[larry mccay]

Severity: moderate

Description:

When using Knox SSO in affected releases, a request could be crafted to
redirect a user to a malicious page due to improper URL parsing.
A request that included a specially crafted
request parameter could be used to redirect the user to a page controlled
by an attacker. This URL would need to be presented to the user outside
the normal request flow through a XSS or phishing campaign.

Mitigation:

1.x users should upgrade to 1.6.1.
Unsupported versions of the 0.x line that include this issue are: 0.13.0,
0.14.0.
and these should upgrade to 1.6.1 as well.
1.0.0 and 1.1.0 are also Unsupported but affected and should upgrade to
1.6.1.


Credit:

Apache Knox would like to thank Kajetan Rostojek for this report

Re: [DISCUSS] - Upgrading Log4j to 2.17.1 on Knox 1.6 line

[larry mccay]

s/dependent/vulnerable/


On Thu, Jan 13, 2022 at 10:34 AM larry mccay  wrote:

> We are not vulnerable to those issues as they are in log4j-core and we
> don't use that in the 1.x line.
> Why would we need to upgrade libs that are not dependent?
>
> On Thu, Jan 13, 2022 at 6:47 AM Sandeep Moré 
> wrote:
>
>> Awesome! that sounds great Sandor, thanks!
>>
>> On Thu, Jan 13, 2022 at 5:46 AM Sandor Molnar
>>  wrote:
>>
>>> Hi folks,
>>>
>>> with our recent v1.6.1 release (an announcement is about to be sent out)
>>> we
>>> are on 2.16.0 to mitigate the infamous  CVE-2021-44228
>>> <https://nvd.nist.gov/vuln/detail/CVE-2021-44228> security
>>> vulnerability.
>>> However, there were subsequent security issues found and those
>>> problems were addressed in later versions. For more information please
>>> read
>>> Log4J's security vulnerability page:
>>> https://logging.apache.org/log4j/2.x/security.html
>>>
>>> I'm proposing to kick off a new 1.6.2 release that includes the fix for
>>> https://issues.apache.org/jira/browse/KNOX-2702.
>>>
>>> Any objection?
>>>
>>> Cheers,
>>> Sandor
>>>
>>

Re: [DISCUSS] - Upgrading Log4j to 2.17.1 on Knox 1.6 line

[larry mccay]

We are not vulnerable to those issues as they are in log4j-core and we
don't use that in the 1.x line.
Why would we need to upgrade libs that are not dependent?

On Thu, Jan 13, 2022 at 6:47 AM Sandeep Moré  wrote:

> Awesome! that sounds great Sandor, thanks!
>
> On Thu, Jan 13, 2022 at 5:46 AM Sandor Molnar 
> wrote:
>
>> Hi folks,
>>
>> with our recent v1.6.1 release (an announcement is about to be sent out)
>> we
>> are on 2.16.0 to mitigate the infamous  CVE-2021-44228
>>  security vulnerability.
>> However, there were subsequent security issues found and those
>> problems were addressed in later versions. For more information please
>> read
>> Log4J's security vulnerability page:
>> https://logging.apache.org/log4j/2.x/security.html
>>
>> I'm proposing to kick off a new 1.6.2 release that includes the fix for
>> https://issues.apache.org/jira/browse/KNOX-2702.
>>
>> Any objection?
>>
>> Cheers,
>> Sandor
>>
>

Re: [VOTE] Release Apache Knox 1.6.1 - RC 1

[larry mccay]

Thanks for spinning this release, Sandor!

* verified signatures
* built from source and ran unit tests
* tested KnoxSSO
* Checked homepage, admin ui

Barring any other issues, the copyright can wait for the next release.

+1


On Wed, Jan 5, 2022 at 7:57 AM Attila Magyar  wrote:

> +1 from me
>
> * I verified the signatures for all files.
> * Checked the dep directory to see if log4j-api-2.16.0.jar is in place.
> * Started knox gateway, ldap server and knoxcli.
> * Tested some basic functionality such as token generations and UI
> functions.
>
> Attila
>
> On Tue, Jan 4, 2022 at 9:48 PM Sandeep Moré  wrote:
>
>> Thank you Sandor,
>> While checking the NOTICE file I realized that we have copyright date as
>> 2021, it is minor and I am okay with it since RC was spun up last year but
>> given this would be released this year wanted to bring it to everyone's
>> attention.
>>
>> Else, here is my +1
>> * Downloaded and built from source
>> * Checked LICENSE and NOTICE files
>> * Verified GPG/SHA signatures for Knox source, Knox and Knoxshell release
>> packages
>>
>> Thanks,
>> sandeep
>>
>> On Tue, Jan 4, 2022 at 12:56 PM Sandor Molnar 
>> wrote:
>>
>>> Hello everyone!
>>>
>>> This is a gentle reminder that I am still waiting for your votes on
>>> Apache Knox 1.6.1 RC1.
>>>
>>> Thanks!
>>>
>>> On Wed, Dec 15, 2021 at 9:50 AM Sandor Molnar 
>>> wrote:
>>>
 Hi folks!

 Release candidate #1 for the Apache Knox 1.6.1 release is available at:
 https://dist.apache.org/repos/dist/dev/knox/knox-1.6.1/

 The release candidate is a zip archive of the sources in:
 https://gitbox.apache.org/repos/asf/knox.git
 Branch v1.6.1 (git checkout -b v1.6.1)

 The KEYS file for signature validation is available at:
 https://dist.apache.org/repos/dist/release/knox/KEYS

 Please find the most recent changes here:
 https://github.com/apache/knox/blob/v1.6.1/CHANGES

 KnoxShell User Guide:
 http://knox.apache.org/books/knox-1-6-0/knoxshell_user_guide.html

 Gateway User Guide:
 http://knox.apache.org/books/knox-1-6-0/user-guide.html

 Dev Guide:
 http://knox.apache.org/books/knox-1-6-0/dev-guide.html

 Please vote on releasing this package as Apache Knox 1.6.1.
 The vote is open for the next 72 hours and passes if a majority of at
 least three +1 Apache Knox PMC votes are cast.

 [ ] +1 Release this package as Apache Knox 1.6.1
 [ ] -1 Do not release this package because...

 Thanks,
 Sandor

>>>
>
> --
> *Attila Magyar* | Staff Software Engineer
>
> cloudera.com 
>
> [image: Cloudera] 
>
> [image: Cloudera on Twitter]  [image:
> Cloudera on Facebook]  [image:
> Cloudera on LinkedIn] 
> --
>

Re: Topology redeployment changes

[larry mccay]

This is an incompatible behavior change.
I wonder whether we could make it configurable and an explicit opt-in
behavior.

I guess that the issue is that in certain deployments that a management
application like Ambari may be writing out topologies based on a restart or
config push even though there is no actual change.
This change will reduce the outage for a topology while it is redeploying
which is good.

I guess the typical flow that would rely on this legacy behavior is a dev
environment where you are working with rewrite rules or one of our Knox
applications that we change locally and just need a redeploy.
There may also be test environments that rely on it but not sure what their
reliance on it would be where there is no actual change.

If we are going to make this default behavior then we need to make sure it
is listed as an incompatible change and not backported to the 1.6.x line.

On Tue, Nov 23, 2021 at 7:38 AM Sandeep Moré  wrote:

> Thanks Sandor!
> I am also CC'ing the dev mailing list.
>
> Thank you for the patch and the heads up :)
>
> On Tue, Nov 23, 2021 at 4:12 AM Sandor Molnar 
> wrote:
>
>> Hi again!
>>
>> I just wanted to add some clarification on the above news. If you
>> actually change the topology (adding a comment is not considered a change)
>> and save it, Knox will continue to redeploy it w/o invoking the previously
>> referenced KnoxCLI *redeploy* command. That command is needed only if
>> you want to redeploy your topology without any change (to replace the
>> well-known touch mechanism).
>>
>> Sandor
>>
>>
>> On Tue, Nov 23, 2021 at 9:26 AM Sandor Molnar 
>> wrote:
>>
>>> Hi everyone!
>>>
>>> I've just merged a commit to fix
>>> https://issues.apache.org/jira/browse/KNOX-2689. With my changes in
>>> place, you will no longer be able to redeploy an XML-based topology by
>>> simply touching it. Instead, you will have to run the following command:
>>>
>>> knoxcli.sh redeploy --cluster topologyName
>>>
>>> E.g. knoxcli.sh redeploy --cluster sandbox
>>>
>>> The related commit is merged into the master branch today that
>>> corresponds to v2.0.0 (this version is not yet released). If you build the
>>> project from the source and use it your own, please remember the
>>> above-written changes.
>>>
>>> Cheers,
>>> Sandor
>>>
>>

Re: [VOTE] Release Apache Knox 1.6.0 - RC 4

[larry mccay]

* Verified signatures
* Checked NOTICE, LICENSE, CHANGES files
* Downloaded and built from source
* Ran unit tests
* Verified basic functionality for HDFS, Hive for both the gateway server
and KnoxShell client
* Tested logout from home page
* Tested the enable/disable of token management feature
* Tested the inability to include userinfo in an originalUrl param to
KnoxSSO

+1


On Wed, Nov 3, 2021 at 4:58 AM Sandor Molnar 
wrote:

> Hi folks!
>
> Release candidate #4 for the Apache Knox 1.6.0 release is available at:
>
> https://dist.apache.org/repos/dist/dev/knox/knox-1.6.0/
>
> The release candidate is a zip archive of the sources in:
>
> https://gitbox.apache.org/repos/asf/knox.git
> Branch v1.6.0 (git checkout -b v1.6.0)
>
> The KEYS file for signature validation is available at:
> https://dist.apache.org/repos/dist/release/knox/KEYS
>
> KnoxShell User Guide:
> http://knox.apache.org/books/knox-1-6-0/knoxshell_user_guide.html
>
> Gateway User Guide:
> http://knox.apache.org/books/knox-1-6-0/user-guide.html
>
> Dev Guide:
> http://knox.apache.org/books/knox-1-6-0/dev-guide.html
>
> Please vote on releasing this package as Apache Knox 1.6.0.
> The vote is open for the next 96 hours and passes if a majority of at
> least three +1 Apache Knox PMC votes are cast.
>
> [ ] +1 Release this package as Apache Knox 1.6.0
> [ ] -1 Do not release this package because...
>
> Thanks,
> Sandor
>

Re: [VOTE] Release Apache Knox 1.6.0 - RC 3

[larry mccay]

Sorry for the delay.
I have provided a fix for KNOX-2655 that I believe we need to get into this
release.
This addresses an issue with improper credential hygiene that we need to
address in the 1.x. line before moving on to 2.0.

It is already committed to master and needs to be cherry-picked to v1.6.0,
in my opinion.

Here is my -1 on RC3.


On Mon, Nov 1, 2021 at 3:53 PM larry mccay  wrote:

> Looking into an issue that may result in me withdrawing my +1.
> I'll let you know when I've decided.
>
>
> On Mon, Nov 1, 2021 at 10:08 AM Sandeep Moré 
> wrote:
>
>> Here is my +1
>>
>> * Downloaded and built from source
>> * Checked LICENSE and NOTICE files
>> * Verified GPG/SHA signatures for Knox source, Knox and Knoxshell release
>> packages
>> * Installed pseudo-distributed instance (Mac OS X )
>>
>> On Sun, Oct 31, 2021 at 5:30 PM Sandor Molnar 
>> wrote:
>>
>>> Hi folks!
>>>
>>> Release candidate #3 for the Apache Knox 1.6.0 release is available at:
>>>
>>> https://dist.apache.org/repos/dist/dev/knox/knox-1.6.0/
>>>
>>> The release candidate is a zip archive of the sources in:
>>>
>>> https://gitbox.apache.org/repos/asf/knox.git
>>> Branch v1.6.0 (git checkout -b v1.6.0)
>>>
>>> The KEYS file for signature validation is available at:
>>> https://dist.apache.org/repos/dist/release/knox/KEYS
>>>
>>> KnoxShell User Guide:
>>> http://knox.apache.org/books/knox-1-6-0/knoxshell_user_guide.html
>>>
>>> Gateway User Guide:
>>> http://knox.apache.org/books/knox-1-6-0/user-guide.html
>>>
>>> Dev Guide:
>>> http://knox.apache.org/books/knox-1-6-0/dev-guide.html
>>>
>>> Please vote on releasing this package as Apache Knox 1.6.0.
>>> The vote is open for the next 96 hours and passes if a majority of at
>>> least three +1 Apache Knox PMC votes are cast.
>>>
>>> [ ] +1 Release this package as Apache Knox 1.6.0
>>> [ ] -1 Do not release this package because...
>>>
>>> Thanks,
>>> Sandor
>>>
>>

Re: [VOTE] Release Apache Knox 1.6.0 - RC 3

[larry mccay]

Looking into an issue that may result in me withdrawing my +1.
I'll let you know when I've decided.


On Mon, Nov 1, 2021 at 10:08 AM Sandeep Moré  wrote:

> Here is my +1
>
> * Downloaded and built from source
> * Checked LICENSE and NOTICE files
> * Verified GPG/SHA signatures for Knox source, Knox and Knoxshell release
> packages
> * Installed pseudo-distributed instance (Mac OS X )
>
> On Sun, Oct 31, 2021 at 5:30 PM Sandor Molnar 
> wrote:
>
>> Hi folks!
>>
>> Release candidate #3 for the Apache Knox 1.6.0 release is available at:
>>
>> https://dist.apache.org/repos/dist/dev/knox/knox-1.6.0/
>>
>> The release candidate is a zip archive of the sources in:
>>
>> https://gitbox.apache.org/repos/asf/knox.git
>> Branch v1.6.0 (git checkout -b v1.6.0)
>>
>> The KEYS file for signature validation is available at:
>> https://dist.apache.org/repos/dist/release/knox/KEYS
>>
>> KnoxShell User Guide:
>> http://knox.apache.org/books/knox-1-6-0/knoxshell_user_guide.html
>>
>> Gateway User Guide:
>> http://knox.apache.org/books/knox-1-6-0/user-guide.html
>>
>> Dev Guide:
>> http://knox.apache.org/books/knox-1-6-0/dev-guide.html
>>
>> Please vote on releasing this package as Apache Knox 1.6.0.
>> The vote is open for the next 96 hours and passes if a majority of at
>> least three +1 Apache Knox PMC votes are cast.
>>
>> [ ] +1 Release this package as Apache Knox 1.6.0
>> [ ] -1 Do not release this package because...
>>
>> Thanks,
>> Sandor
>>
>

Re: [VOTE] Release Apache Knox 1.6.0 - RC 3

[larry mccay]

* Verified signatures
* Checked NOTICE, LICENSE, CHANGES files
* Downloaded and built from source
* Ran unit tests
* Verified basic functionality for HDFS, Hive for both the gateway server
and KnoxShell client
* Tested logout from home page
* Tested the enable/disable of token management feature

+1


On Sun, Oct 31, 2021 at 5:29 PM Sandor Molnar  wrote:

> Hi folks!
>
> Release candidate #3 for the Apache Knox 1.6.0 release is available at:
>
> https://dist.apache.org/repos/dist/dev/knox/knox-1.6.0/
>
> The release candidate is a zip archive of the sources in:
>
> https://gitbox.apache.org/repos/asf/knox.git
> Branch v1.6.0 (git checkout -b v1.6.0)
>
> The KEYS file for signature validation is available at:
> https://dist.apache.org/repos/dist/release/knox/KEYS
>
> KnoxShell User Guide:
> http://knox.apache.org/books/knox-1-6-0/knoxshell_user_guide.html
>
> Gateway User Guide:
> http://knox.apache.org/books/knox-1-6-0/user-guide.html
>
> Dev Guide:
> http://knox.apache.org/books/knox-1-6-0/dev-guide.html
>
> Please vote on releasing this package as Apache Knox 1.6.0.
> The vote is open for the next 96 hours and passes if a majority of at
> least three +1 Apache Knox PMC votes are cast.
>
> [ ] +1 Release this package as Apache Knox 1.6.0
> [ ] -1 Do not release this package because...
>
> Thanks,
> Sandor
>

Re: [VOTE] Release Apache Knox 1.6.0 - RC 1

[larry mccay]

Agreed - that's a blocker.


On Fri, Oct 22, 2021 at 8:38 AM Sandeep Moré  wrote:

> Sounds great, I’ll file a JIRA with the details.
>
> On Fri, Oct 22, 2021 at 8:26 AM Sandor Molnar 
> wrote:
>
>> Thanks, Sandeep!
>>
>> I'd call this issue a release blocker and we need to file a JIRA to fix
>> this and kick-off a new release candidate.
>>
>> Any objection?
>>
>> On Fri, Oct 22, 2021 at 2:02 PM Sandeep Moré 
>> wrote:
>>
>> > **Update: vote -1
>> > I discovered a bug while testing the logout feature so I need to
>> withdraw
>> > my vote.
>> > This is the error I see when I click the logout button
>> >
>> > --
>> > HTTP ERROR 500 javax.servlet.ServletException:
>> > org.apache.jasper.JasperException: Unable to compile class for JSP: An
>> > error occurred at line: [155] in the jsp file: [/logout.jsp] response
>> > cannot be resolved 152: } catch (Exception e) { 153: return ""; 154: }
>> 155:
>> > response.setStatus(HttpServletResponse.SC_TEMPORARY_REDIRECT); 156:
>> > response.setHeader("Location", globalLogoutPageURL); 157: return; 158:
>> %>
>> > An error occurred at line: [156] in the jsp file: [/logout.jsp] response
>> > cannot be resolved 153: return ""; 154: } 155:
>> > response.setStatus(HttpServletResponse.SC_TEMPORARY_REDIRECT); 156:
>> > response.setHeader("Location", globalLogoutPageURL); 157: return; 158:
>> %>
>> > 159:  An error occurred at line: [156] in the jsp file:
>> > [/logout.jsp] globalLogoutPageURL cannot be resolved to a variable 153:
>> > return ""; 154: } 155:
>> > response.setStatus(HttpServletResponse.SC_TEMPORARY_REDIRECT); 156:
>> > response.setHeader("Location", globalLogoutPageURL); 157: return; 158:
>> %>
>> > 159:  An error occurred at line: [157] in the jsp file:
>> > [/logout.jsp] This method must return a result of type String 154: }
>> 155:
>> > response.setStatus(HttpServletResponse.SC_TEMPORARY_REDIRECT); 156:
>> > response.setHeader("Location", globalLogoutPageURL); 157: return; 158:
>> %>
>> > 159:  160: An error occurred at line: [157] in the jsp file:
>> > [/logout.jsp] Syntax error, insert "}" to complete MethodBody 154: }
>> 155:
>> > response.setStatus(HttpServletResponse.SC_TEMPORARY_REDIRECT); 156:
>> > response.setHeader("Location", globalLogoutPageURL); 1
>> >
>> > On Thu, Oct 21, 2021 at 3:32 PM Sandeep Moré 
>> > wrote:
>> >
>> >> Thank you Sandor! and apologies for the delay.
>> >> Here is my +1
>> >>
>> >> ---
>> >> * Downloaded and built from source
>> >> * Checked LICENSE and NOTICE files
>> >> * Verified GPG/SHA signatures for Knox source, Knox and Knoxshell
>> release
>> >> packages
>> >> * Installed pseudo-distributed instance (Mac OS X )
>> >> * Checked KnoxShell samples
>> >>
>> >> Thanks,
>> >> Sandeep
>> >>
>> >> On Fri, Oct 15, 2021 at 4:40 PM Sandor Molnar 
>> >> wrote:
>> >>
>> >>> Hi folks!
>> >>>
>> >>> Release candidate #1 for the Apache Knox 1.6.0 release is available
>> at:
>> >>>
>> >>> https://dist.apache.org/repos/dist/dev/knox/knox-1.6.0/
>> >>>
>> >>> The release candidate is a zip archive of the sources in:
>> >>>
>> >>> https://gitbox.apache.org/repos/asf/knox.git
>> >>> Branch v1.6.0 (git checkout -b v1.6.0)
>> >>>
>> >>> The KEYS file for signature validation is available at:
>> >>> https://dist.apache.org/repos/dist/release/knox/KEYS
>> >>>
>> >>> KnoxShell User Guide:
>> >>> http://knox.apache.org/books/knox-1-6-0/knoxshell_user_guide.html
>> >>>
>> >>> Gateway User Guide:
>> >>> http://knox.apache.org/books/knox-1-6-0/user-guide.html
>> >>>
>> >>> Dev Guide:
>> >>> http://knox.apache.org/books/knox-1-6-0/dev-guide.html
>> >>>
>> >>> Please vote on releasing this package as Apache Knox 1.6.0.
>> >>> The vote is open for the next 96 hours and passes if a majority of at
>> >>> least three +1 Apache Knox PMC votes are cast.
>> >>>
>> >>> [ ] +1 Release this package as Apache Knox 1.6.0
>> >>> [ ] -1 Do not release this package because...
>> >>>
>> >>> Thanks,
>> >>> Sandor
>> >>>
>> >>
>>
>

Re: Proxying WebHDFS on kerberized cluster with CAS server as authenticator

[larry mccay]

Hi Tien Dat PHAN -

It is indeed a valid usecase and should work.
If the documentation available in the user guide [1] is not working then we
may have a bug in 1.5.0.
There was a regression in OIDC support due to an upgraded dependency that
was out of step with one of the others.

Please do let us know and provide the configuration that you are using for
comparison.

Thanks!

--larry

1.
http://knox.apache.org/books/knox-1-5-0/user-guide.html#Pac4j+Provider+-+CAS+/+OAuth+/+SAML+/+OpenID+Connect

On Thu, Sep 9, 2021 at 5:17 AM Tien Dat PHAN  wrote:

> Dear experts,
>
> We are having an HDFS cluster which is secured with Kerberos.
> We also have a CAS server which is used for basically most of our
> authentication activities.
>
> We just wonder if with Apache Knox 1.5.0, is it possible to proxy the
> webHDFS web UI of this HDFS cluster, with our CAS server as the
> authenticator.
>
> We have been following the User guide, but so far, we did not succeed.
> For your information, if we use the LDAP server as the authenticator,
> instead of CAS server via Pac4J, it is working well.
>
> So we just wonder if 1) is it possible our use case? and 2) if it is
> POSSIBLE, what could be the missing configuration we should add? (We can
> share our topology configuration here if it can help)
>
> Best regards
> Tien Dat PHAN
>

Re: Apache Spark History server API

[larry mccay]

Hi Jeff -

Your questions are very Cloudera specific.
It is inappropriate for us to discuss these things in the Apache community.

I can tell you that there are multiple reasons that admins and distros
provide multiple topologies in Apache Knox.
One of the most common is to provide access to the same cluster and to the
same services sometimes via different authentication mechanisms.
The cdp-proxy-api topology is the one to use with HTTP Basic authentication.

One of things that makes this a little confusing is that the UIs often
require their API's to be added to the topology as well so that they can
make calls to their backend.
The same backend is sometimes made available in a topology intended for
only APIs and the UI counterpart will be absent.

Any more information for CDP needs to be acquired through company support,
documentation, etc.

thanks,

--larry

On Thu, Sep 9, 2021 at 8:49 PM Jeffrey Rodriguez 
wrote:

> Though we support Spark History server through Knox Spark History UI
> service. I can reach out through a browser and do SSO authentication and
> access /v1/api/applications and it will return a json body.  This REST Api
> as documented in
> https://spark.apache.org/docs/latest/monitoring.html
> The issue is that I need to use curl or a cli client , when I try the hs
> url with the rest endpoint I got an authentication error since the topology
> for UIs uses SSO and I am trying basic authentication instead.
> Any suggestions? Would I have to add to my cdp-proxy-api? CDE have a jwt
> token option where I can get a bear token but i noticed cdp doesn’t provide
> it.
> I am reaching support for public cdp too. I would hate to make updates on
> our cloud conf for Knox so any help would be appreciated.
> On a different question do you know if  Knox or webhdfs support  abfs://
> files?
> Thanks again I have already most of the info I need thanks to Knox user
> group.
> Jeff Rodriguez
>

Re: Jdbc access through Knox Hive endpoint.

[larry mccay]

Hi Jeff -

Can you share the errors that you are encountering?

Thanks,

--larry

On Fri, Sep 3, 2021, 5:57 PM Jeffrey Rodriguez  wrote:

> Hi Apache Knox users,
>  Working on Hive access through Knox. In CDP cloud it
> advertises endpoint
>
> "jdbc:hive2://mywackyserrver-manager0.pood-st.zc6q-kczr.cloudera.site/;ssl=true;transportMode=http;httpPath=mywackyserver/cdp-proxy-api/hive"
> It would really nice if somebody has figure out how to use Knox Hive
> endpoints to access Hive server 2 (http - which is what mine uses).
> According to my theory this should work
> ```
> import jaydebeapi
>
>
> url="jdbc:hive2://mywackyserrver-manager0.pood-st.zc6q-kczr.cloudera.site:8443/default;user=wacky;password=wacky;ssl=true;sslTrustStore=/mytrustore;trustStorePassword=changeme;transportMode=http;httpPath=mywackyserver/cdp-proxy-api/hive"
>
> #Connect to HiveServer2
> # . classpath to to hadoop, hive client jars..
>
> conn=jaydebeapi.connect("org.apache.hive.jdbc.HiveDriver", url)
> cursor = conn.cursor()
>
> # Execute SQL query
> sql="show databases"
> cursor.execute(sql)
> results = cursor.fetchall()
> print( results )
> import jaydebeapi
>
>
> url="jdbc:hive2://mywackyserrver-manager0.pood-st.zc6q-kczr.cloudera.site:8443/default;user=wacky;password=wacky;ssl=true;sslTrustStore=/mytrustore;trustStorePassword=changeme;transportMode=http;httpPath=mywackyserver/cdp-proxy-api/hive"
>
> #Connect to HiveServer2
> # . classpath to to hadoop, hive client jars..
>
> conn=jaydebeapi.connect("org.apache.hive.jdbc.HiveDriver", url)
> cursor = conn.cursor()
>
> # Execute SQL query
> sql="show databases"
> cursor.execute(sql)
> results = cursor.fetchall()
> print( results )
> ```
>
> Thanks,
>Jeff Rodriguez
>

Re: Apache Knox RESOURCEMANAGER API not returning information

[larry mccay]

Hi Jeff -

I am not sure what your deployment is set up to do here but it looks as
though you are trying to send HTTP Basic credentials to a KnoxSSO protected
cdp-proxy topology that is configured for SAML via pac4j.

I assume that there is a cdp-proxy-api topology in your deployment as well
which would be more appropriate for HTTP Basic credentials.

This seems to be a cloudera specific deployment though - so you should
contact support from the vendor for those details.

thanks,

--larry

On Sun, Aug 22, 2021 at 11:40 AM Jeffrey Rodriguez 
wrote:

> Apache Knox RESOURCEMANAGER API not returning information
>
> Running product on a cluster that has a Resource Manager with a
> failover configuration.
>
> Simply accessing /v1/cluster/information didn't return information.
>
>
> curl -v -L  -k -u jrodrigu -H "Accept: application/xml"  -H
> "Content-Type: application/json" -X GET
> '
> https://mysite.cloudera.site:8443/mysite/cdp-proxy/resourcemanager/v1/cluster/info
> '
> Enter host password for user 'jrodrigu':
> * About to connect() to mysiste,cloudera.site port 8443 (#0)
> *   Trying 10.243.52.27...
> * Connected to mysite.cloudera.site (10.243.52.27) port 8443 (#0)
> * Initializing NSS with certpath: sql:/etc/pki/nssdb
> * skipping SSL peer certificate verification
> * SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
> * Server certificate:
> * subject: CN=mysite.cloudera.site,ST=CA,C=US
> * start date: Jun 30 19:19:20 2021 GMT
> * expire date: Jun 30 23:59:59 2022 GMT
> * common name: mysite.cloudera.site
> * issuer: CN=mysite
> * Server auth using Basic with user 'jrodrigu'
> > GET /mysite/cdp-proxy/resourcemanager/v1/cluster/info HTTP/1.1
> > Authorization: Basic anJvZHJpZ3U6UDl1bmtpblA5IVA5dW5raW5QOSE=
> > User-Agent: curl/7.29.0
> > Host: mysite.dpaas-pr.jsoy-bql6.cloudera.site:8443
> > Accept: application/xml
> > Content-Type: application/json
> >
> < HTTP/1.1 302 Found
> < Date: Sun, 22 Aug 2021 15:04:30 GMT
> < Location:
> https://mysite.cloudera.site:8443/mysite/knoxsso/api/v1/websso?originalUrl=https://mysite..cloudera.site:8443/devaron/cdp-proxy/resourcemanager/v1/cluster/info
> < Content-Length: 0
> <
> * Connection #0 to host
> devaron-manager0.dpaas-pr.jsoy-bql6.cloudera.site left intact
> * Issue another request to this URL:
> '
> https://mysite.cloudera.site:8443/devaron/knoxsso/api/v1/websso?originalUrl=https://mysite..cloudera.site:8443/mysite/cdp-proxy/resourcemanager/v1/cluster/info
> '
> * Found bundle for host mysite.cloudera.site: 0x1928190
> * Re-using existing connection! (#0) with host dmysite.cloudera.site
> * Connected to mysite.cloudera.site (10.243.52.27) port 8443 (#0)
> * Server auth using Basic with user foobar
> > GET /mysite/knoxsso/api/v1/websso?originalUrl=
> https://mysite.cloudera.site:8443/mysite.cdp-proxy/resourcemanager/v1/cluster/info
> HTTP/1.1
> > Authorization: Basic sfsdfskdjflskdjfksjsd
> > User-Agent: curl/7.29.0
> > Host: mysite.cloudera.site:8443
> > Accept: application/xml
> > Content-Type: application/json
> >
> < HTTP/1.1 200 OK
> < Date: Sun, 22 Aug 2021 15:04:30 GMT
> < Cache-Control: no-cache, no-store
> < Pragma: no-cache
> < Expires: Thu, 01 Jan 1970 00:00:00 GMT
> < X-Content-Type-Options: nosniff
> < Strict-Transport-Security: max-age=15768000 ; includeSubDomains
> < X-Frame-Options: DENY
> < X-XSS-Protection: 1; mode=block
> < Set-Cookie:
> pac4j.session.pac4jCsrfToken=CBBAJZgve/KisatD1VK9UgpHJiEA1wslBU586xT0TcItouJ3mkLNRsRGu92iv26GIgDhCjxlry9YaKN4ovVmeUVMY4IQmSmQVPGMGY2PlRlsSXv3HZxuHoeYeA==;
> Domain=.dpaas-pr.jsoy-bql6.cloudera.site; Secure; HttpOnly
> < Set-Cookie: pac4jCsrfToken=3ba09563-8fb4-4d72-b27b-246af32c58c6;
> Path=/; Domain=mysite.cloudera.site
> < Set-Cookie:
> 

Re: Troubleshooting Knox issue

[larry mccay]

Hi Ying -

You didn't mention whether you added a service definition for your REST
service or not.
I would really expect a 404 without one but it sort of depends on what your
topology looks like.

If you would like to send the gateway.log, gateway-audit.log and topology
files, I'll take a look.

thanks,

--larry

On Mon, Jun 14, 2021 at 6:41 PM Ying Chen  wrote:

> Hello -
>
> Recently I have run into an issue on an older cluster using Knox 1.4.0.
>
> I have a REST service that is running on the Hadoop cluster.  I've
> configured a Knox topology to route to WebHDFS as well as this REST service
> that I built.
>
> I am able to connect to WebHDFS thru knox, but for some reason it is not
> able to reach the REST services.  Knox returns 500 Server Error,  /
> IOException Service connectivity error.
>
> The REST service is running, and I am able to curl to that REST service on
> the port specified in the topology file...
>
> Any suggestions as to what to check next ?
>
> Thanks in advance for your help.
>
> Ying
>
>
>
>
>
>
>

Re: java.lang.NoClassDefFoundError in KNOX

[larry mccay]

Hi Ebrahim -

This is obviously a classpath related issue that would not be caused simply
by restarting the gateway.
Either, you have changed jars or the command line in gateway.sh script.

Given that this is a Ranger Plugin related class, I assume that you have
changed that jar or introduced an extension that assumes a different
version of that code.
If you have backed up the jars that you have changed - which you obviously
should have - then put them back and restart.

HTH,

--larry

On Mon, Dec 28, 2020 at 11:43 PM Ebrahim Khalil Abbasi <
ebrahim.khalil.abb...@gmail.com> wrote:

> Dear all,
>
> I am using KNOX and its plugin in Ranger for authorization.
> I restarted KNOX and Ranger and then executed the following command:
> -
>  curl -ik  -u admin: https://
> :8443/gateway/dsgdev/livy/v1/sessions
> -
>
>
> I got the http error 500
>
> 
> HTTP ERROR 500
> Problem accessing /gateway/dsgdev/livy/v1/sessions. Reason:
> Server Errorhttp://eclipse.org/jetty;>Powered
> by Jetty:// 9.4.12.v20180830
> ---
>
> The gateway.log file shows two errors
> 2020-12-29 08:03:42,348 ERROR config.RangerConfiguration
> (RangerConfiguration.java:addResourceIfReadable(63)) -
> addResourceIfReadable(ranger-knox-dsgdev_knox-policymgr-ssl.xml): couldn't
> find resource file location
> 2020-12-29 08:03:43,131 ERROR knox.gateway
> (AbstractGatewayFilter.java:doFilter(69)) - Failed to execute filter:
> java.lang.NoClassDefFoundError: com/kstruct/gethostname4j/Hostname
>
>
> Would you please help me what should I do for this problem?
> Ebrahim
>
>
>
>
>
>
>
>

[ANNOUNCE] - Apache Knox 1.5.0 Release

[larry mccay]

The Apache Knox team is proud to announce the release of Apache Knox 1.5.0!

Apache Knox is a REST API Gateway for providing secure access to the
data and processing resources of Apache Hadoop clusters. More details on
Apache Knox can be found at:
https://knox.apache.org/

Downloads, KEYS and verification details can be found on the Releases page
at:
https://cwiki.apache.org/confluence/display/KNOX/Apache+Knox+Releases

The release bits are available at:
https://www.apache.org/dyn/closer.cgi/knox/1.5.0

This release includes 100+ committed changes with a few new features such
as Apache Flink Dashboard proxying, Logging out from the Knox Homepage SSO
and Load Balancing to proxied services such as Apache Hive.

A number of blogs should follow, to showcase new features and capabilities.

Many bug fixes, improvements and library upgrades are also included in this
release.

We would like to thank all of contributors who made the release
possible and to invite others interested in helping out to engage the
community on the dev and users lists!

--Apache Knox PMC and community

Re: Method Level Authorization for Knox

[larry mccay]

Hi Ebrahim -

I tried replying to the Ranger thread but my subscription seems messed up.

I believe that Bosco was referring to the interface within the Ranger Knox
Plugin code that would need to change ALONG with the Ranger side changes
you already made.
Based on what I see in [1], there is no change needed in the Knox code base
as this is all in Ranger.
You would want to push the HTTP verb from the request that is acquired in
the filter into the authorization interface which is in the same package in
Ranger.

Of course, you could also either extend or create a new Authorization
Provider in Knox as well but that will not give you the central access
policy authoring and management that Ranger provides.

thanks,

--larry


1.
https://github.com/apache/ranger/blob/master/knox-agent/src/main/java/org/apache/ranger/authorization/knox/RangerPDPKnoxFilter.java#L146

On Tue, Dec 8, 2020 at 2:59 PM Ebrahim Khalil Abbasi <
ebrahim.khalil.abb...@gmail.com> wrote:

> Hi there,
> I am using knox to access livy to manage spark sessions. To implement
> authorization I want to provide the method level (get/post/delete/...)
> authorization. I implemented a new HTTP Service plugin in Ranger but I need
> to integrate it to the knox or  the Ranger's knox plugin so that each HTTP
> request to the knox is authorized based on the method.
>
> It seems there are two possibilities, one is to update the Knox's
> authorization interface and another is to update the Ranger's Knox plugin.
>
> Would you please suggest a better solution?
> Thanks
> Ebrahim
>

[RESULT] [VOTE] Release Apache Knox 1.5.0 - RC 1

[larry mccay]

The VOTE for 1.5.0 RC 1 PASSES with:

3 binding +1's
0 nonbinding +1
0 -1's

I will be working on promoting 1.5.0 RC1 to an official release shortly.

Thank you for taking the time to test this release and contributing to the
Apache Knox community!

On Sat, Dec 5, 2020 at 11:34 AM larry mccay  wrote:

> +1 for Apache Knox 1.5.0 RC1
>
>- Verified the signatures for knox-source, knox, and knoxshell
> artifacts.
>- Checked the CHANGES, LICENSE and NOTICE files
>- Downloaded the source artifact, built it and executed the
>unit/functional tests
>- Tested the homepage
>- Tested KnoxSSO and Logout from Homepage
>- Tested proxying of HDFS UI and utilities
>- Tested proxying of Hive
>- Tested Ranger Knox plugin from Ranger for authorization
>- Tested Knox AclsAuthz provider
>- Tested Knox PAM authentication via Shiro provider
>- Tested Proxying of Cloudera Manager from CDP public cloud deployment
>- Tested KnoxShell: FileSystems, DataSources/SQL, CSV related commands
> and KnoxShellTable rendering and operations
>
>
>
> On Tue, Dec 1, 2020 at 3:36 PM Phil Zampino  wrote:
>
>> +1 for Apache Knox 1.5.0 RC1
>>
>> I've performed the following:
>>
>>- Verified the signatures for knox-source, knox, and knoxshell
>> artifacts.
>>- Checked the LICENSE and NOTICE files (already commented on the
>>copyright in the NOTICE)
>>- Downloaded the source artifact, built it and executed the
>>unit/functional tests
>>- Exercised the homepage
>>- Exercised the Admin UI Service Definition browser/editor
>>- Created a descriptor using the Admin UI, and verified discovery and
>>topology generation against a CM cluster therewith
>>    - Tested token API (issue, renew, revoke)
>>- Tested server-managed token state
>>
>>
>> LGTM,
>>   Phil
>>
>> On Sun, Nov 22, 2020 at 2:59 PM larry mccay  wrote:
>>
>> > All -
>> >
>> > Release candidate #1 for the Apache Knox 1.5.0 release is available at:
>> >
>> > https://dist.apache.org/repos/dist/dev/knox/knox-1.5.0/
>> >
>> > The release candidate is a zip archive of the sources in:
>> >
>> > https://https://gitbox.apache.org/repos/asf/knox.git
>> > Branch v1.5.0 (git checkout -b v1.5.0)
>> >
>> > The KEYS file for signature validation is available at:
>> > https://dist.apache.org/repos/dist/release/knox/KEYS
>> >
>> > KnoxShell User Guide at
>> > http://knox.apache.org/books/knox-1-5-0/knoxshell_user_guide.html
>> > Gateway User Guide:
>> > http://knox.apache.org/books/knox-1-5-0/user-guide.html
>> > Dev Guide: http://knox.apache.org/books/knox-1-5-0/dev-guide.html
>> >
>> > Please vote on releasing this package as Apache Knox 1.5.0.
>> > The vote is open for the next 72 hours and passes if a majority of at
>> > least three +1 Apache Knox PMC votes are cast.
>> >
>> > [ ] +1 Release this package as Apache Knox 1.5.0
>> > [ ] -1 Do not release this package because...
>> >
>> > thanks!
>> >
>> > --larry
>> >
>>
>

Re: [VOTE] Release Apache Knox 1.5.0 - RC 1

[larry mccay]

+1 for Apache Knox 1.5.0 RC1

   - Verified the signatures for knox-source, knox, and knoxshell artifacts.
   - Checked the CHANGES, LICENSE and NOTICE files
   - Downloaded the source artifact, built it and executed the
   unit/functional tests
   - Tested the homepage
   - Tested KnoxSSO and Logout from Homepage
   - Tested proxying of HDFS UI and utilities
   - Tested proxying of Hive
   - Tested Ranger Knox plugin from Ranger for authorization
   - Tested Knox AclsAuthz provider
   - Tested Knox PAM authentication via Shiro provider
   - Tested Proxying of Cloudera Manager from CDP public cloud deployment
   - Tested KnoxShell: FileSystems, DataSources/SQL, CSV related commands
and KnoxShellTable rendering and operations



On Tue, Dec 1, 2020 at 3:36 PM Phil Zampino  wrote:

> +1 for Apache Knox 1.5.0 RC1
>
> I've performed the following:
>
>- Verified the signatures for knox-source, knox, and knoxshell
> artifacts.
>- Checked the LICENSE and NOTICE files (already commented on the
>copyright in the NOTICE)
>- Downloaded the source artifact, built it and executed the
>unit/functional tests
>- Exercised the homepage
>- Exercised the Admin UI Service Definition browser/editor
>- Created a descriptor using the Admin UI, and verified discovery and
>topology generation against a CM cluster therewith
>- Tested token API (issue, renew, revoke)
>- Tested server-managed token state
>
>
> LGTM,
>   Phil
>
> On Sun, Nov 22, 2020 at 2:59 PM larry mccay  wrote:
>
> > All -
> >
> > Release candidate #1 for the Apache Knox 1.5.0 release is available at:
> >
> > https://dist.apache.org/repos/dist/dev/knox/knox-1.5.0/
> >
> > The release candidate is a zip archive of the sources in:
> >
> > https://https://gitbox.apache.org/repos/asf/knox.git
> > Branch v1.5.0 (git checkout -b v1.5.0)
> >
> > The KEYS file for signature validation is available at:
> > https://dist.apache.org/repos/dist/release/knox/KEYS
> >
> > KnoxShell User Guide at
> > http://knox.apache.org/books/knox-1-5-0/knoxshell_user_guide.html
> > Gateway User Guide:
> > http://knox.apache.org/books/knox-1-5-0/user-guide.html
> > Dev Guide: http://knox.apache.org/books/knox-1-5-0/dev-guide.html
> >
> > Please vote on releasing this package as Apache Knox 1.5.0.
> > The vote is open for the next 72 hours and passes if a majority of at
> > least three +1 Apache Knox PMC votes are cast.
> >
> > [ ] +1 Release this package as Apache Knox 1.5.0
> > [ ] -1 Do not release this package because...
> >
> > thanks!
> >
> > --larry
> >
>

Re: [VOTE] Release Apache Knox 1.5.0 - RC 1

[larry mccay]

Yes, it should. Not sure whether this is a blocker or not. Thoughts?

On Wed, Nov 25, 2020, 9:59 PM Phil Zampino  wrote:

> Checking the NOTICE file in the RC, it has the copyright date as
> 2012-20*19*.
> Should it rather be 2012-20*20*?
>
>
>
> On Sun, Nov 22, 2020 at 2:59 PM larry mccay  wrote:
>
> > All -
> >
> > Release candidate #1 for the Apache Knox 1.5.0 release is available at:
> >
> > https://dist.apache.org/repos/dist/dev/knox/knox-1.5.0/
> >
> > The release candidate is a zip archive of the sources in:
> >
> > https://https://gitbox.apache.org/repos/asf/knox.git
> > Branch v1.5.0 (git checkout -b v1.5.0)
> >
> > The KEYS file for signature validation is available at:
> > https://dist.apache.org/repos/dist/release/knox/KEYS
> >
> > KnoxShell User Guide at
> > http://knox.apache.org/books/knox-1-5-0/knoxshell_user_guide.html
> > Gateway User Guide:
> > http://knox.apache.org/books/knox-1-5-0/user-guide.html
> > Dev Guide: http://knox.apache.org/books/knox-1-5-0/dev-guide.html
> >
> > Please vote on releasing this package as Apache Knox 1.5.0.
> > The vote is open for the next 72 hours and passes if a majority of at
> > least three +1 Apache Knox PMC votes are cast.
> >
> > [ ] +1 Release this package as Apache Knox 1.5.0
> > [ ] -1 Do not release this package because...
> >
> > thanks!
> >
> > --larry
> >
>

[VOTE] Release Apache Knox 1.5.0 - RC 1

[larry mccay]

All -

Release candidate #1 for the Apache Knox 1.5.0 release is available at:

https://dist.apache.org/repos/dist/dev/knox/knox-1.5.0/

The release candidate is a zip archive of the sources in:

https://https://gitbox.apache.org/repos/asf/knox.git
Branch v1.5.0 (git checkout -b v1.5.0)

The KEYS file for signature validation is available at:
https://dist.apache.org/repos/dist/release/knox/KEYS

KnoxShell User Guide at
http://knox.apache.org/books/knox-1-5-0/knoxshell_user_guide.html
Gateway User Guide: http://knox.apache.org/books/knox-1-5-0/user-guide.html
Dev Guide: http://knox.apache.org/books/knox-1-5-0/dev-guide.html

Please vote on releasing this package as Apache Knox 1.5.0.
The vote is open for the next 72 hours and passes if a majority of at
least three +1 Apache Knox PMC votes are cast.

[ ] +1 Release this package as Apache Knox 1.5.0
[ ] -1 Do not release this package because...

thanks!

--larry

Re: Kerberos Delegation

[larry mccay]

Hi Jeff -

No, Knox is a trusted proxy in the Hadoop ecosystem and uses that pattern
with Livy as well as the other ecosystem services to strongly authenticate
as Knox and assert the identity of the authenticated user via doAs query
param.

HTH,

--larry

On Tue, Oct 6, 2020 at 6:27 PM jeff saremi  wrote:

> Does anyone know if Knox uses Kerberos Delegation when forwarding requests
> to say Livy? Is it capable of doing so if not doing it already?
>
> thanks,
> jeff
>

[ANNOUNCE] - Apache Knox 1.4.0 Release

[larry mccay]

The Apache Knox team is proud to announce the release of Apache Knox 1.4.0!

Apache Knox is a REST API Gateway for providing secure access to the
data and processing resources of Hadoop clusters. More details on
Apache Knox can be found at:
https://knox.apache.org/

Downloads, KEYS and verification details can be found on the Releases page
at:
https://cwiki.apache.org/confluence/display/KNOX/Apache+Knox+Releases

The release bits are available at:
https://www.apache.org/dyn/closer.cgi/knox/1.4.0

NOTE: The CHANGES file in the above directory was unfortuantely not updated
but
we have included a link to the updated file on the Releases page above.

This release includes nearly 400 committed changes with some notable new
features:
* [KNOX-2153] - CM discovery - Monitor Cloudera Manager (#239)
* [KNOX-2002] - Add a KnoxShellTable to Represent and Render Output in
Tabular Format
* [KNOX-2224] - KnoxLine and KnoxShell DataSource and Select Command
Alignment (#255)
* [KNOX-2240] - KnoxShell Custom Command for WEBHDFS Use (#296)
* [KNOX-2052] - KnoxShellTable mean, median, and mode methods (#163)
* [KNOX-1740] - Add Trusted Proxy Support to Knox (#106)
* [KNOX-2056] - Adding Service Definitions management into Admin UI
(#169)
* [KNOX-2053] - New REST API to create/read/update/delete service
definitions (#164)
* [KNOX-2226] - Add home page to Knox (#263)
* [KNOX-1410] - Knox Shell support for remote Alias management (#210)

Many bug fixes, improvements and library upgrades are also included in this
release.

The release also includes a new focus on the KnoxShell interactive shell
with custom shell commands for easily leveraging JDBC/SQL (Hive and
others), Filesystems (WebHDFS) and working with CSV files locally. This in
the first release to focus on this area and with continued work this shell
will be a client for end-to-end flows from local environments to datalake
storage and processing.

Much of the above has come from KIP-14 [1] proposal for KnoxShell
Improvements for Tabular Data.

We would like to thank all of contributors who made the release
possible and to invite others interested in helping out to engage the
community on the dev and users lists!

--Apache Knox PMC and community

1.
https://cwiki.apache.org/confluence/display/KNOX/KIP-14+-+KnoxShell+Improvements+for+Tabular+Data

Re: Switching user going from KNOX to WebHDFS

[larry mccay]

Knox will scrub any incoming query params that are used to potentially
spoof other users.
Both user.name and doas/doAs will be scrubbed from the incoming request and
the appropriate query param will be used for the dispatch to the backend
service.

You can look at the default identity-assertion [1] provider principal
mapping capabilities to map an authenticated principal/username to another
user known within the cluster. This will allow you to authenticate as one
user and the effective user within the cluster will be the mapped user.

1.
http://knox.apache.org/books/knox-1-3-0/user-guide.html#Default+Identity+Assertion+Provider

On Mon, Nov 18, 2019 at 4:15 PM jeff saremi  wrote:

> @Larry, yes your description is pretty well.
> We have different "levels" of security. This is the lowest one and most
> unsecure however since this is on-prem product some customers woulnd't mind
> accepting the risk
>
> We also have Kerberos and AD on top of these for those want a more secure
> environment.
>
> Currently, we need Knox to allow us pass the superuser identity if
> possible.
> thanks
> --
> *From:* jeff saremi 
> *Sent:* Monday, November 18, 2019 1:12 PM
> *To:* larry mccay ; user@knox.apache.org <
> user@knox.apache.org>
> *Subject:* Re: Switching user going from KNOX to WebHDFS
>
> @kevin, yes we're not using Kerberos or any AD
>
> So you're saying that whatever user I authenticate against knox is the one
> that will be passed to webhdfs?
>
> If i were to pass ?user.name=hdfs in the query string targeted for hdfs
> or ?doas=hdfs in the that request, would knox honor those and pass them
> along? or will they get overwritten by Knox?
> --
> *From:* larry mccay 
> *Sent:* Monday, November 18, 2019 1:09 PM
> *To:* user@knox.apache.org 
> *Subject:* Re: Switching user going from KNOX to WebHDFS
>
> Hi Jeff -
>
> Thanks for reaching out!
>
> Rather than try and unpack all of that, I'd like to get to step back to a
> description of what you are trying to accomplish with your deployment and
> the addition of Knox within it.
>
> As you have described it, it seems like a very unsecured environment.
> Whether you are running your process as a root user or not, executing your
> queries and operations as the HDFS user is also very insecure.
> HDFS is a superuser in a Hadoop deployment.
>
> Authenticating to Knox as root and asserting the effective user as hdfs is
> certainly we can do but I don't see what the value is of doing that.
>
> So, let's step back and get a clear picture of what you would like to
> accomplish and we can direct you to appropriate authentication/federation
> providers and possibly identity-assertion providers to meet your needs.
>
> thanks,
>
> --larry
>
> On Mon, Nov 18, 2019 at 2:47 PM Kevin Risden  wrote:
>
> If i am to do an hdfs query, all i need to do is to set HADOOP_USER_NAME
> to 'hdfs' then everything works nicely.
>
>
> This means that you aren't using Kerberos just regular simple auth for
> your cluster.
>
> This is true until we get to knox. We still communicate with Knox using a
> root and an admin password. I believe by default, this user's identity is
> used to call webhdfs?
>
>
> The user identity is asserted by Knox against the backend service. So Knox
> is configured for authentication that username is asserted to the backend.
> So however you are doing authentication in Knox needs to be configured.
> This is usually LDAP out of the box but can be configured with different
> authentication providers like PAM.
>
> Kevin Risden
>
>
> On Mon, Nov 18, 2019 at 2:37 PM jeff saremi 
> wrote:
>
> I'm not sure how to phrase this question and also I don't have any
> experience in these two technologies
>
> Here's the deal: We are switching from running hadoop and related
> technologies from under root to a non-root user
>
> So far we have managed to successfully change our namenodes and datanodes
> such that the process is running under a user named 'hdfs'.
>
> If i am to do an hdfs query, all i need to do is to set HADOOP_USER_NAME
> to 'hdfs' then everything works nicely.
>
> This is true until we get to knox. We still communicate with Knox using a
> root and an admin password. I believe by default, this user's identity is
> used to call webhdfs?
>
> We need to change this behavior. Looking for some pointers on what the
> changes would be.
>
> thanks
> Jeff
>
>

Re: Switching user going from KNOX to WebHDFS

[larry mccay]

Hi Jeff -

Thanks for reaching out!

Rather than try and unpack all of that, I'd like to get to step back to a
description of what you are trying to accomplish with your deployment and
the addition of Knox within it.

As you have described it, it seems like a very unsecured environment.
Whether you are running your process as a root user or not, executing your
queries and operations as the HDFS user is also very insecure.
HDFS is a superuser in a Hadoop deployment.

Authenticating to Knox as root and asserting the effective user as hdfs is
certainly we can do but I don't see what the value is of doing that.

So, let's step back and get a clear picture of what you would like to
accomplish and we can direct you to appropriate authentication/federation
providers and possibly identity-assertion providers to meet your needs.

thanks,

--larry

On Mon, Nov 18, 2019 at 2:47 PM Kevin Risden  wrote:

> If i am to do an hdfs query, all i need to do is to set HADOOP_USER_NAME
>> to 'hdfs' then everything works nicely.
>
>
> This means that you aren't using Kerberos just regular simple auth for
> your cluster.
>
> This is true until we get to knox. We still communicate with Knox using a
>> root and an admin password. I believe by default, this user's identity is
>> used to call webhdfs?
>>
>
> The user identity is asserted by Knox against the backend service. So Knox
> is configured for authentication that username is asserted to the backend.
> So however you are doing authentication in Knox needs to be configured.
> This is usually LDAP out of the box but can be configured with different
> authentication providers like PAM.
>
> Kevin Risden
>
>
> On Mon, Nov 18, 2019 at 2:37 PM jeff saremi 
> wrote:
>
>> I'm not sure how to phrase this question and also I don't have any
>> experience in these two technologies
>>
>> Here's the deal: We are switching from running hadoop and related
>> technologies from under root to a non-root user
>>
>> So far we have managed to successfully change our namenodes and datanodes
>> such that the process is running under a user named 'hdfs'.
>>
>> If i am to do an hdfs query, all i need to do is to set HADOOP_USER_NAME
>> to 'hdfs' then everything works nicely.
>>
>> This is true until we get to knox. We still communicate with Knox using a
>> root and an admin password. I believe by default, this user's identity is
>> used to call webhdfs?
>>
>> We need to change this behavior. Looking for some pointers on what the
>> changes would be.
>>
>> thanks
>> Jeff
>>
>

Re: [DISCUSS] Planning for Apache Knox 1.4

[larry mccay]

The entirety of KIP-14 doesn't need to be scoped for the 1.4 release.
JIRAs do need to be filed for the work detailed there and iterative
progress will be made across multiple releases.

I think that the initial goal should be to fill the role that the KnoxLine
example represented and drove interest in a Knox SQL client.
Being able to work with the results of a SQL query is an added benefit of
this approach and we can have some basic operations for doing just that.

I don't see the overall Knox metadata endpoint making it in the 1.4 release
timeframe.
I do see some of the underlying capabilities of things like datasources
being configurable and used to interact with SQL engines rather than having
to provide connection details for each session. I have an early version of
this already working locally.


On Mon, Nov 11, 2019 at 2:38 PM Kevin Risden  wrote:

> Based on that we are ~1/2 way through November and Thanksgiving in the US
> around the corner, I don't see much of the KnoxShell pieces getting
> integrated before end of November to make an end of November release.
> Specifically the following section:
>
> With the continued increase in cloud based deployments and Knox Gateway use
> > in securely accessing the exposed resources, we will concentrate on
> > KnoxShell as a first class environment for this access. This will likely
> > include an API for discovering metadata about the resources exposed
> through
> > Knox, the required authentication mechanisms, resource types and public
> > certs. It will also include Custom GroovyShell Commands for the KnoxShell
> > environment to help interact with the remote clusters and resultsets as
> > local in-memory tables. I will be start a KIP to try and articulate this
> > vision and related 1.4. usecases as well.
> >
>
> The KIP was just started so would be good to flesh that out more instead of
> rushing it into 1.4.0. There are a lot of moving pieces in that paragraph
> and would be good to make sure the Jiras are created and scoped
> appropriately.
>
> In addition to what was mentioned as features, there have been multiple new
> service definitions (Impala, Kudu, NiFi Registry) added as well as fixes to
> existing service definitions (Atlas, Livy, Ranger, Spark, YARN).
>
> So +1 to an end of November release, but need to make sure not trying to
> rush in new things just because a release will happen. There will be more
> releases.
>
>
> Kevin Risden
>
>
> On Fri, Nov 1, 2019 at 11:51 AM Sandeep Moré 
> wrote:
>
> > Thanks for starting the planning thread Larry !
> > Agree with the theme and the release date for 1.4.0.
> >
> > +1
> >
> > Best,
> > Sandeep
> >
> > On Thu, Oct 31, 2019 at 11:53 AM larry mccay  wrote:
> >
> >> Folks -
> >>
> >> Out last release with end of July, I apologize for the delay in starting
> >> the planning thread for 1.4.
> >>
> >> We currently have a backlog of ~65 JIRAs slated for a Fix Version of
> 1.4.
> >>
> >> There has been some work going on within KnoxShell to provide a general
> >> purpose representation for tabular data. This will be leveraged for
> >> rendering SQL query results as well as CSV files and simple processing
> >> within KnoxShell. I will be writing up a KIP to represent the overall
> >> vision for this work and initial set of usecases.
> >>
> >> We also have Cloudera Manager based discovery emerging and we should
> >> target an initial set of services to enable for CM/CDH and CDP
> deployments
> >> where CM is available.
> >>
> >> With the continued increase in cloud based deployments and Knox Gateway
> >> use in securely accessing the exposed resources, we will concentrate on
> >> KnoxShell as a first class environment for this access. This will likely
> >> include an API for discovering metadata about the resources exposed
> through
> >> Knox, the required authentication mechanisms, resource types and public
> >> certs. It will also include Custom GroovyShell Commands for the
> KnoxShell
> >> environment to help interact with the remote clusters and resultsets as
> >> local in-memory tables. I will be start a KIP to try and articulate this
> >> vision and related 1.4. usecases as well.
> >>
> >> I propose that the CM based Service Discovery and KnoxShell access to
> >> remote clusters be the primary themes of the Apache Knox 1.4 release.
> >>
> >> I also propose that we target the end of November as the release date
> for
> >> 1.4.
> >>
> >> Thoughts?
> >>
> >> --larry
> >>
> >
>

Re: [DISCUSS] - Integrating Knox with Swagger

[larry mccay]

+ dev@...

Thank you for the idea!

Yes, I am familiar with Swagger and that would be huge for our current APIs
and others that may come along.
I think the effort to add a swagger filter or the like will be only one
part of the larger effort of how it integrates into Knox, the site, the
Admin UI, the docs, etc.

I think that a KIP detailing the full vision would make a lot of sense.

+1 on moving forward with a KIP and/or JIRA to continue discussion and
tease out design considerations.

On Wed, Nov 13, 2019 at 3:45 PM Sandor Molnar  wrote:

> Hi folks,
>
> recently I had some contribution that allows end-users managing service
> definitions without restarting the Knox Gateway. See
> https://issues.apache.org/jira/browse/KNOX-2053 and
> https://issues.apache.org/jira/browse/KNOX-2056 for further details.
>
> I've been just about creating a new JIRA to document those new API
> endpoints in Knox user guide but it has come to my mind that we can do it
> much better by using Swagger.
>
> Given the fact, the Admin API does not consist of hundreds of existing
> endpoints it should not be 'that' huge work. I personally believe the
> project would gain a lot by using this very useful tool (e.g. Admin API
> documentation would be generated out-of-the-box, no more documentation
> JIRAs required. Moreover, the generated documentation would be in sync with
> the actual implementation).
> You can check this out here: https://swagger.io/tools/swagger-ui/ (there
> is a live demo too; it's worth looking at).
>
> Any comments, ideas are welcome!
>
> Cheers,
> Sandor
>

[DISCUSS] Planning for Apache Knox 1.4

[larry mccay]

Folks -

Out last release with end of July, I apologize for the delay in starting
the planning thread for 1.4.

We currently have a backlog of ~65 JIRAs slated for a Fix Version of 1.4.

There has been some work going on within KnoxShell to provide a general
purpose representation for tabular data. This will be leveraged for
rendering SQL query results as well as CSV files and simple processing
within KnoxShell. I will be writing up a KIP to represent the overall
vision for this work and initial set of usecases.

We also have Cloudera Manager based discovery emerging and we should target
an initial set of services to enable for CM/CDH and CDP deployments where
CM is available.

With the continued increase in cloud based deployments and Knox Gateway use
in securely accessing the exposed resources, we will concentrate on
KnoxShell as a first class environment for this access. This will likely
include an API for discovering metadata about the resources exposed through
Knox, the required authentication mechanisms, resource types and public
certs. It will also include Custom GroovyShell Commands for the KnoxShell
environment to help interact with the remote clusters and resultsets as
local in-memory tables. I will be start a KIP to try and articulate this
vision and related 1.4. usecases as well.

I propose that the CM based Service Discovery and KnoxShell access to
remote clusters be the primary themes of the Apache Knox 1.4 release.

I also propose that we target the end of November as the release date for
1.4.

Thoughts?

--larry

Re: Adding a web.xml to gateway.jar

[larry mccay]

That is coming from WEBHDFS itself.
The error message tells you exactly the problem with it.

A 400 is usually going to come from application code whether that be a
backend service that we dispatch to or a Knox API.
It indicates that the request you used is bad in the context of HTTP method
in this case or some other assumptions about what should be used as
addresses or some other aspect of the request.

If you are looking for the spot in the Knox code that this 400 is being
throw you will not find it since it is coming from the NameNode.

On Tue, Sep 10, 2019 at 7:07 PM jeff saremi  wrote:

> Do we know where/when/how a 400 error is returned like this?
>
> root@clustertest:/tests/knox# curl -iku root:goodpassword -X GET
> https://gateway-svc:8443/gateway/default/webhdfs/v1/?op=STATUS
> HTTP/1.1 400 Bad Request
> Date: Fri, 06 Sep 2019 23:42:42 GMT
> Set-Cookie:
> KNOXSESSIONID=node01q5krk3jp1c9dzv3fc3t5tkgh4.node0;Path=/gateway/default;Secure;HttpOnly
> Expires: Thu, 01 Jan 1970 00:00:00 GMT
> Set-Cookie: rememberMe=deleteMe; Path=/gateway/default; Max-Age=0;
> Expires=Thu, 05-Sep-2019 23:42:43 GMT
> Date: Fri, 06 Sep 2019 23:42:43 GMT
> Cache-Control: no-cache
> Expires: Fri, 06 Sep 2019 23:42:45 GMT
> Date: Fri, 06 Sep 2019 23:42:45 GMT
> Pragma: no-cache
> X-FRAME-OPTIONS: SAMEORIGIN
> Content-Type: application/json;charset=utf-8
> Transfer-Encoding: chunked
> Server: Jetty(9.4.12.v20180830)
>
> {"RemoteException":{"exception":"IllegalArgumentException","javaClassName":"java.lang.IllegalArgumentException","message":"Invalid
> value for webhdfs parameter \"op\": STATUS is not a valid GET operation."}}
>
>
> I want to see if the same logic applies to 500
> --
> *From:* larry mccay 
> *Sent:* Tuesday, September 10, 2019 2:14 PM
> *To:* user@knox.apache.org 
> *Subject:* Re: Adding a web.xml to gateway.jar
>
> Interesting
>
> Almost the entirety of Knox request/response processing is done in
> ServletFilters that represent each Provider along the chain as well as the
> dispatch to backend services or the JAXRS service of a Knox API.
>
> We could do this inside of a Provider filter but that would limit the
> details provided to the application logic of the GatewayServlet rather than
> being more container managed - I would think.
> That may also be the case for custom error pages in the web.xml - but it
> could at least be pattern matched across multiple servlets in the webapp if
> need be.
>
> I will give it some more thought and maybe find some time to try out the
> web.xml change.
>
> On Tue, Sep 10, 2019 at 2:36 PM jeff saremi 
> wrote:
>
> So continuing with the assumption that the problem must be me, I decided
> to create a small app which uses jetty and a web.xml plus two servlets to
> demonstrate the intent.
> It worked perfectly the first time!
> Here's the link to the application. The web.xml is modeled after what I
> modified last in knox.
> I tested the app with the following URL's and each time I got the expected
> result:
> http://localhost:8080/?error=503
>
> {"error": {"code":503,"uri":"/"}}
>
>
> https://1drv.ms/u/s!AmK9GYfTgrv-psxYrFLkiSd-Qpiv4Q?e=oge5qY
>
> So currently I'm sitting at the point that I cannot seem to be able to
> inject any additional servlets and any error redirections in Knox
>
> If someone tells me the first/last entry point in the application where I
> have the chance to intercept all errors/messages before going back to the
> user I would really appreciate it. This way I can modify the source code
> and be done with this with no messy web.xml and additional servlets.
>
> thanks
>
> --
> *From:* jeff saremi 
> *Sent:* Monday, September 9, 2019 9:53 PM
> *To:* user@knox.apache.org 
> *Subject:* Re: Adding a web.xml to gateway.jar
>
> I got further by making sure that the startup scripts do not touch the
> topology file anymore.
> I can see the deployment staying the same as I modified it
> However there must be something wrong with my web.xml or the servlet ,
> since all requests  mow return a 503 which does not seem to be initiated by
> the gateway logic itself. It could be that my web.xml is invalid or my
> servlet creates an infinite loop or something like that.
> I've got some investigating to do.
> Basically this the last version of the web.xml (truncated):
>
>   
> default-knox-gateway-servlet
> org.apache.knox.gateway.GatewayServlet
> 
>   gatewayDescriptorLocation
>   /WEB-INF/gateway.xml
> 
>   
>   
> exception-handler-servlet

Re: Adding a web.xml to gateway.jar

[larry mccay]

":\""+throwable.getMessage()+"\"}");
> }
> out.write("}");
> } catch (Throwable e) {
> log("Error inside ExceptionHandler: " + e.getMessage());
> }
>
> Sample error returned by knox:
> --
>
> root@clustertest:~# curl -iku user:password -X GET
> https://gateway-svc:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS
> HTTP/1.1 503 Service Unavailable
> Date: Tue, 10 Sep 2019 04:40:19 GMT
> Cache-Control: must-revalidate,no-cache,no-store
> Content-Type: text/html;charset=iso-8859-1
> Content-Length: 364
> Server: Jetty(9.4.12.v20180830)
>
> 
> 
> 
> Error 503 Service Unavailable
> 
> HTTP ERROR 503
> Problem accessing /gateway/default/webhdfs/v1/. Reason:
> Service Unavailablehttp://eclipse.org/jetty;>Powered by Jetty:// 9.4.12.v20180830
>
> 
> 
>
> --
> *From:* jeff saremi 
> *Sent:* Monday, September 9, 2019 8:12 PM
> *To:* user@knox.apache.org 
> *Subject:* Re: Adding a web.xml to gateway.jar
>
> Good pointers.
> We're not using Ambari or any gui to do this. All command line.
> This is the supervisord config file:
>
> root@gateway-0:/opt/supervisor/conf.d# ls
> knox.conf
> root@gateway-0:/opt/supervisor/conf.d# cat knox.conf
> [program:knoxgateway]
> command=/opt/knox/bin/gateway-launcher.sh
> stopasgroup=true
> startretries=3
>
> What makes a deployment to take place?
> What command lines should I be looking for?
> The only thing i see relevant is that the conf/topologies/default.xml is
> touched (re-created from a template file)
> Could that be causing a re-deployment?
>
> --
> *From:* larry mccay 
> *Sent:* Monday, September 9, 2019 4:53 PM
> *To:* user@knox.apache.org 
> *Subject:* Re: Adding a web.xml to gateway.jar
>
> Interesting.
> I'm not really familiar with Supervisord but it seems to be not only
> restarting the process but causing the redeployment of the topologies.
> Is it making an Ambari API call to restart Knox or something like that?
>
> The short term goal needs to be a restart that doesn't redeploy and the
> reuse of the web.xml.
> This would be just to verify that what you want to do works as expected
> then we would need to add the error handling to the ShrinkWrap code that
> generates the web.xml in the Knox deployment machinery.
>
> If you are using a default.xml topology that sounds like you are using
> Ambari and HDP?
> Are you able to remove Knox from the Supervisord monitoring temporarily or
> even just turn it off?
>
> If so, we can then stop the Knox process via Ambari then restart Knox
> manually.
> If you restart from Ambari it will push out configuration again and you
> will get a new deployment - so don't do that.
>
> A simple restart without config being changed will just load the previous
> web.xml and webapp without deploying a new one.
>
> On Mon, Sep 9, 2019 at 7:10 PM jeff saremi  wrote:
>
> Larry
> I created a jar file out of a single ExceptionServlet and placed in
> GATEWAY_HOME/ext:
>
> root@gateway-0:/opt/knox# ls -l ext
> total 12
> -rw-r--r-- 1 root root   83 Sep  6 00:23 README
> -rwxr-xr-- 1 root root 1704 Sep  9 22:41 error-servlet.jar
> drwxr--r-- 2 root root 4096 Sep  6 00:23 native
>
> root@gateway-0:/opt/knox# jar tvf ext/error-servlet.jar
>  0 Mon Sep 09 15:24:44 UTC 2019 META-INF/
> 64 Mon Sep 09 15:24:44 UTC 2019 META-INF/MANIFEST.MF
>   2382 Mon Sep 09 15:23:54 UTC 2019 ExceptionHandlerServlet.class
>
> Since I appear to be using 'default' in my URLs I'm guessing that would be
> the right topology. So i went to data/deployments and modified the
> following web.xml:
> data/deployments/default.topo.16d08ec29f0
>
> updated web.xml:
>
>
> root@gateway-0:/opt/knox# tail -25
> data/deployments/default.topo.16d08ec29f0/%2F/WEB-INF/web.xml
> /WEB-INF/rewrite.xml
>   
>
>   
> 400
> /ExceptionHandler
>   
>   
> 401
> /ExceptionHandler
>   
>   
> 404
> /ExceptionHandler
>   
>   
> java.lang.Throwable
> /ExceptionHandler
>   
>   
> javax.servlet.ServletException
> /ExceptionHandler
>   
>
> 
>
> I then killed the running process (since it was started by the Supervisor
> d) and it restarted itself
>
> The response were identical to before meaning that my servlet was never hit
>
> Looking at the deployment folder, I see that there's another 'default*'
> folder is created with updated timestamp. And the new web.xml there does
> not have any of my changes of course
>
> If I were to make these changes to the web

Re: Adding a web.xml to gateway.jar

[larry mccay]

Interesting.
I'm not really familiar with Supervisord but it seems to be not only
restarting the process but causing the redeployment of the topologies.
Is it making an Ambari API call to restart Knox or something like that?

The short term goal needs to be a restart that doesn't redeploy and the
reuse of the web.xml.
This would be just to verify that what you want to do works as expected
then we would need to add the error handling to the ShrinkWrap code that
generates the web.xml in the Knox deployment machinery.

If you are using a default.xml topology that sounds like you are using
Ambari and HDP?
Are you able to remove Knox from the Supervisord monitoring temporarily or
even just turn it off?

If so, we can then stop the Knox process via Ambari then restart Knox
manually.
If you restart from Ambari it will push out configuration again and you
will get a new deployment - so don't do that.

A simple restart without config being changed will just load the previous
web.xml and webapp without deploying a new one.

On Mon, Sep 9, 2019 at 7:10 PM jeff saremi  wrote:

> Larry
> I created a jar file out of a single ExceptionServlet and placed in
> GATEWAY_HOME/ext:
>
> root@gateway-0:/opt/knox# ls -l ext
> total 12
> -rw-r--r-- 1 root root   83 Sep  6 00:23 README
> -rwxr-xr-- 1 root root 1704 Sep  9 22:41 error-servlet.jar
> drwxr--r-- 2 root root 4096 Sep  6 00:23 native
>
> root@gateway-0:/opt/knox# jar tvf ext/error-servlet.jar
>  0 Mon Sep 09 15:24:44 UTC 2019 META-INF/
> 64 Mon Sep 09 15:24:44 UTC 2019 META-INF/MANIFEST.MF
>   2382 Mon Sep 09 15:23:54 UTC 2019 ExceptionHandlerServlet.class
>
> Since I appear to be using 'default' in my URLs I'm guessing that would be
> the right topology. So i went to data/deployments and modified the
> following web.xml:
> data/deployments/default.topo.16d08ec29f0
>
> updated web.xml:
>
>
> root@gateway-0:/opt/knox# tail -25
> data/deployments/default.topo.16d08ec29f0/%2F/WEB-INF/web.xml
> /WEB-INF/rewrite.xml
>   
>
>   
> 400
> /ExceptionHandler
>   
>   
> 401
> /ExceptionHandler
>   
>   
> 404
> /ExceptionHandler
>   
>   
> java.lang.Throwable
> /ExceptionHandler
>   
>   
> javax.servlet.ServletException
> /ExceptionHandler
>   
>
> 
>
> I then killed the running process (since it was started by the Supervisor
> d) and it restarted itself
>
> The response were identical to before meaning that my servlet was never hit
>
> Looking at the deployment folder, I see that there's another 'default*'
> folder is created with updated timestamp. And the new web.xml there does
> not have any of my changes of course
>
> If I were to make these changes to the web.xml permenant, where would be
> the right location, given that the web.xml seems to be created on the fly.
>
>
>
> --
> *From:* jeff saremi 
> *Sent:* Saturday, September 7, 2019 4:28 PM
> *To:* user@knox.apache.org 
> *Subject:* Re: Adding a web.xml to gateway.jar
>
> Great suggestions! Thanks Larry
> I will work on getting the web.xml and the servlet integrated
> Completely agreed on the vulnerability side. We may expose this in a DEBUG
> version and not the release or provide a config value...
> --
> *From:* larry mccay 
> *Sent:* Friday, September 6, 2019 7:25 PM
> *To:* user@knox.apache.org 
> *Subject:* Re: Adding a web.xml to gateway.jar
>
> Hi Jeff -
>
> This is an interesting idea and we should consider discussing this as a
> feature of Knox rather than just something that you are trying to hack into
> an existing release/deployment.
>
> In order to get this to work, I would first change the web.xml in the
> deployments directory for a given topology and add the servlet to the in a
> jar within {GATEWAY_HOME}/ext directory.
> Stop and start the server and it should hopefully pickup the changed
> web.xml file.
>
> In order to cause a 500, I think just dispatching to an invalid URL would
> result in a 500 with a connection exception.
>
> See if that web.xml will work and we can take it from there.
>
> It should be noted that surfacing the details of a webappexception may
> expose sensitive information about the server and you may not want to
> always have this enabled.
>
> HTH.
>
> --larry
>
> On Fri, Sep 6, 2019 at 9:37 PM jeff saremi  wrote:
>
>
> Ultimately I am trying to make sure when an HTTP 500 error happens the
> exception message and stacktrace are returned in the response, in the
> gateway
> So I decided to add a web.xml and overwrite parts of error handling there
> to the gateway project. (added
> to gateway-server-launcher/src/main/resources/META-IN

Re: Adding a web.xml to gateway.jar

[larry mccay]

Hi Jeff -

This is an interesting idea and we should consider discussing this as a
feature of Knox rather than just something that you are trying to hack into
an existing release/deployment.

In order to get this to work, I would first change the web.xml in the
deployments directory for a given topology and add the servlet to the in a
jar within {GATEWAY_HOME}/ext directory.
Stop and start the server and it should hopefully pickup the changed
web.xml file.

In order to cause a 500, I think just dispatching to an invalid URL would
result in a 500 with a connection exception.

See if that web.xml will work and we can take it from there.

It should be noted that surfacing the details of a webappexception may
expose sensitive information about the server and you may not want to
always have this enabled.

HTH.

--larry

On Fri, Sep 6, 2019 at 9:37 PM jeff saremi  wrote:

>
> Ultimately I am trying to make sure when an HTTP 500 error happens the
> exception message and stacktrace are returned in the response, in the
> gateway
> So I decided to add a web.xml and overwrite parts of error handling there
> to the gateway project. (added
> to gateway-server-launcher/src/main/resources/META-INF/web.xml)
>
> root@gateway-0:/opt/knox/bin# cat META-INF/web.xml
> 
> 
> http://www.w3.org/2001/XMLSchema-instance; xmlns="
> http://java.sun.com/xml/ns/javaee; xsi:schemaLocation="
> http://java.sun.com/xml/ns/javaee
> http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd; version="3.0">
>   
> 400
> /ExceptionHandler
>   
>   
> 401
> /ExceptionHandler
>   
>   
> 404
> /ExceptionHandler
>   
>   
> java.lang.Throwable
> /ExceptionHandler
>   
>   
> javax.servlet.ServletException
> /ExceptionHandler
>   
> 
>
> And then I added the following Servlet (added it
> to 
> gateway-util-common/src/main/java/org/apache/knox/gateway/servlet/ExceptionHandlerServlet.java)
>
>
> @WebServlet("/ExceptionHandler")
> public class ExceptionHandlerServlet extends HttpServlet {
>   private static final long serialVersionUID = 1L;
>
>   protected void service(HttpServletRequest request,
> HttpServletResponse response) throws ServletException, IOException {
> // Analyze the servlet exception
> Throwable throwable = (Throwable)
> request.getAttribute("javax.servlet.error.exception");
> Integer statusCode = (Integer)
> request.getAttribute("javax.servlet.error.status_code");
> String servletName = (String)
> request.getAttribute("javax.servlet.error.servlet_name");
> if (servletName == null) {
>   servletName = "Unknown";
> }
> String requestUri = (String)
> request.getAttribute("javax.servlet.error.request_uri");
> if (requestUri == null) {
>   requestUri = "Unknown";
> }
>
> // Set response content type
> response.setContentType("text/html");
>
> PrintWriter out = response.getWriter();
> out.write("Exception/Error
> Details");
> if(statusCode != 500){
>   out.write("Error Details");
>   out.write("Status Code:"+statusCode+"");
>   out.write("Requested URI:"+requestUri);
> } else {
>   out.write("Exception Details");
>   out.write("Servlet Name:"+servletName+"");
>   out.write("Exception
> Name:"+throwable.getClass().getName()+"");
>   out.write("Requested URI:"+requestUri+"");
>   out.write("Exception Message:"+throwable.getMessage()+"");
>   out.write("");
> }
>
> out.write("");
> out.write("");
>   }
> }
>
>
> I see that the application is launched using gateway.jar. And i also see
> my web.xml inside that jar. However I'm not able to get anything returned
> from this servlet!
>
> I honestly don't know how to repro a 500. But I could do a 400, 401, and
> 404. Neither of them got intercepted by the Exception servlet i wrote.
>
> Here are some examples I ran. Note in the first one, a 400 is returned
> along with some exception message. that's what i want to do for 500 or
> verify that it's being done. However I haven't been able to (using text
> search) find out where in the code this response is formed like this
>
> root@clustertest:/tests/knox# curl -iku root:goodpassword -X GET
> https://gateway-svc:8443/gateway/default/webhdfs/v1/?op=STATUS
> HTTP/1.1 400 Bad Request
> Date: Fri, 06 Sep 2019 23:42:42 GMT
> Set-Cookie:
> KNOXSESSIONID=node01q5krk3jp1c9dzv3fc3t5tkgh4.node0;Path=/gateway/default;Secure;HttpOnly
> Expires: Thu, 01 Jan 1970 00:00:00 GMT
> Set-Cookie: rememberMe=deleteMe; Path=/gateway/default; Max-Age=0;
> Expires=Thu, 05-Sep-2019 23:42:43 GMT
> Date: Fri, 06 Sep 2019 23:42:43 GMT
> Cache-Control: no-cache
> Expires: Fri, 06 Sep 2019 23:42:45 GMT
> Date: Fri, 06 Sep 2019 23:42:45 GMT
> Pragma: no-cache
> X-FRAME-OPTIONS: SAMEORIGIN
> Content-Type: application/json;charset=utf-8
> Transfer-Encoding: chunked
> Server: Jetty(9.4.12.v20180830)
>
> {"RemoteException":{"exception":"IllegalArgumentException","javaClassName":"java.lang.IllegalArgumentException","message":"Invalid
> 

Re: [ANNOUNCE] Apache Knox 1.3.0 Release

[larry mccay]

The Apache Knox team is proud to announce the release of Apache Knox 1.3.0!

Apache Knox is a REST API Gateway for providing secure access to the
data and processing resources of Hadoop clusters. More details on
Apache Knox can be found at:
https://knox.apache.org/

Downloads, KEYS and verification details can be found on the Releases page
at:
https://cwiki.apache.org/confluence/display/KNOX/Apache+Knox+Releases

The release bits are available at:
https://www.apache.org/dyn/closer.cgi/knox/1.3.0

NOTE: The CHANGES file in the above directory was unfortuantely not updated
but
we have included a link to the updated file on the Releases page above.

This release includes over 200 committed changes with some notable new
features:
   * [KNOX-1862] - Proxying support for Hue UI
   * [KNOX-1875] - Cloudera Manager service discovery
   * [KNOX-1855] - Proxying support for Cloudera Manager API
   * [KNOX-1774] - Introduce environment variables in
gateway/knoxcli/ldap/knoxshell scripts for customization (#65)
   * [KNOX-1687] - Hashicorp Vault RemoteAliasService provider
   * [KNOX-1550] - TLS/SSL client certificate authentication provider
   * [KNOX-1418] - Add KnoxShell command to build truststore using the
gateway server's public certificate (#60)
   * [KNOX-1285] - Create Apache Knox Dockerfile and image

Many bug fixes, improvements and library upgrades are also included in this
release.

We would like to thank all of contributors who made the release
possible and to invite others interested in helping out to engage the
community on the dev and users lists!

--Apache Knox PMC and community

On Thu, Jul 25, 2019 at 8:51 PM larry mccay  wrote:

> The Apache Knox team is proud to announce the release of Apache Knox 1.2.0!
>
> Apache Knox is a REST API Gateway for providing secure access to the
> data and processing resources of Hadoop clusters. More details on
> Apache Knox can be found at:
> https://knox.apache.org/
>
> Downloads, KEYS and verification details can be found on the Releases page
> at:
> https://cwiki.apache.org/confluence/display/KNOX/Apache+Knox+Releases
>
> The release bits are available at:
> https://www.apache.org/dyn/closer.cgi/knox/1.3.0
>
> NOTE: The CHANGES file in the above directory was unfortunately not
> updated but
> we have included a link to the updated file on the Releases page above.
>
> This release includes over 200 committed changes with some notable new
> features:
>* [KNOX-1862] - Proxying support for Hue UI
>* [KNOX-1875] - Cloudera Manager service discovery
>* [KNOX-1855] - Proxying support for Cloudera Manager API
>* [KNOX-1687] - Hashicorp Vault RemoteAliasService provider
>* [KNOX-1550] - TLS/SSL Client Cert Authentication Provider
>* [KNOX-1418] - Add KnoxShell command to build truststore
>* [KNOX-1285] - Create Apache Knox Dockerfile and image
>
> Many bug fixes, improvements and library upgrades are also included in
> this release.
>
> We would like to thank all of contributors who made the release
> possible and to invite others interested in helping out to engage the
> community on the dev and users lists!
>
>

[ANNOUNCE] Apache Knox 1.3.0 Release

[larry mccay]

The Apache Knox team is proud to announce the release of Apache Knox 1.2.0!

Apache Knox is a REST API Gateway for providing secure access to the
data and processing resources of Hadoop clusters. More details on
Apache Knox can be found at:
https://knox.apache.org/

Downloads, KEYS and verification details can be found on the Releases page
at:
https://cwiki.apache.org/confluence/display/KNOX/Apache+Knox+Releases

The release bits are available at:
https://www.apache.org/dyn/closer.cgi/knox/1.3.0

NOTE: The CHANGES file in the above directory was unfortunately not updated
but
we have included a link to the updated file on the Releases page above.

This release includes over 200 committed changes with some notable new
features:
   * [KNOX-1862] - Proxying support for Hue UI
   * [KNOX-1875] - Cloudera Manager service discovery
   * [KNOX-1855] - Proxying support for Cloudera Manager API
   * [KNOX-1687] - Hashicorp Vault RemoteAliasService provider
   * [KNOX-1550] - TLS/SSL Client Cert Authentication Provider
   * [KNOX-1418] - Add KnoxShell command to build truststore
   * [KNOX-1285] - Create Apache Knox Dockerfile and image

Many bug fixes, improvements and library upgrades are also included in this
release.

We would like to thank all of contributors who made the release
possible and to invite others interested in helping out to engage the
community on the dev and users lists!

[VOTE] Release Apache Knox 1.3.0

[larry mccay]

All -

A candidate for the Apache Knox 1.3.0 release is available at:

https://dist.apache.org/repos/dist/dev/knox/knox-1.3.0/

The release candidate is a zip archive of the sources in:

https://https://gitbox.apache.org/repos/asf/knox.git
Branch v1.3.0 (git checkout -b v1.3.0)

The KEYS file for signature validation is available at:
https://dist.apache.org/repos/dist/release/knox/KEYS

Please vote on releasing this package as Apache Knox 1.3.0.
The vote is open for the next 72 hours and passes if a majority of at
least three +1 Apache Knox PMC votes are cast.

[ ] +1 Release this package as Apache Knox 1.3.0
[ ] -1 Do not release this package because...

thanks,

--larry

Re: KnoxSSO Logout

[larry mccay]

That looks like the right URL and that error is probably due to not using
"https".


On Thu, Feb 21, 2019 at 12:47 PM Rajat Goel  wrote:

> Hi Larry,
>
>
>
> Thanks for the detailed explanation on the current set of limitations
>  Larry. Currently, I am using default form based provider integrated with
> LDAP so as per comment hopefully it will work. I tried to create a new
> topology with following content but it didn’t work.
>
>
>
> Knoxssout.xml
>
>
>
> 
>
> 
>
> 
>
> authentication
>
> Anonymous
>
> true
>
> 
>
> 
>
> identity-assertion
>
> Default
>
> false
>
> 
>
> 
>
> 
>
> KNOXSSOUT
>
> 
>
> 
>
>
>
> What am I doing wrong here? Also what will be my logout url ? I tried
> hitting http://:8443/gateway/knoxssout/api/v1/webssout from
> browser but I see error ERR_EMPTY_RESPONSE
>
>
>
> Regards,
>
> Rajat
>
>
>
> *From: *larry mccay 
> *Reply-To: *"user@knox.apache.org" 
> *Date: *Thursday, 21 February 2019 at 9:17 PM
> *To: *"user@knox.apache.org" 
> *Subject: *Re: KnoxSSO Logout
>
>
>
> Hi Rajat -
>
>
>
> KNOXSSOUT will work in limited usecases and it isn't really documented or
> anything due to those limitations.
>
> Depending on what your actual SSO IdP is it may not work for you.
>
>
>
> Let me describe the issue in the context of a SAML provider...
>
>
>
> * SSOCookieProvider determines that there is no KnoxSSO cookie and
> redirects you to the KnoxSSO endpoint
>
> * KnoxSSO is configured for Okta or some other SAML provider and redirects
> to the SAML provider endpoint
>
> * SAML provider authenticates the user and posts back to the KnoxSSO
> endpoint
>
> * KnoxSSO sets the hadoop-jwt cookie and redirects to the originally
> requested resource
>
>
>
> The above establishes not only a KnoxSSO session but also a session with
> the SAML provider via IDP specific cookies.
>
>
>
> Now, when you invoke the KNOXSSOUT API from some logout link on an app
> page the KNOXSSOUT service will remove
>
> the KnoxSSO cookie and redirect you back. The SSOCookieProvider will not
> find a cookie and send you back to KnoxSSO
>
> which will send you to the IDP again and that session is still active.
> BAM! You are logged right back in.
>
>
>
> These types of nuances are different from provider to provider.
>
>
>
> Now, if you are only using the default form based provider from Knox - it
> may actually work for you as long as you also remove
>
> any application specific cookies as well as call KNOXSSOUT.
>
>
>
> As for a topology example, all you really need to do is add a topology -
> say knoxssout.xml - and protect it with the Anonymous
>
> authentication provider.
>
>
>
> Hope that is helpful.
>
>
>
> --larry
>
>
>
> On Thu, Feb 21, 2019 at 8:39 AM Rajat Goel  wrote:
>
> Hi,
>
>
>
> I was looking for implementing Logout for my service which is integrated
> with Knox SSO (SSOCookieProvider).  I came across this Jira ticket
> https://issues.apache.org/jira/browse/KNOX-744 where a new service
> KNOXSSOUT is created which should be used in a new topology. Can someone
> please provide a sample topology file for using this service for
> invalidating cookie ?
>
>
>
> Also, my Knox version is 0.12 (HDP 2.6.5). Will the above approach work
> with my Knox version ?
>
>
>
> Thanks,
>
> Rajat
>
>

Re: KnoxSSO Logout

[larry mccay]

Hi Rajat -

KNOXSSOUT will work in limited usecases and it isn't really documented or
anything due to those limitations.
Depending on what your actual SSO IdP is it may not work for you.

Let me describe the issue in the context of a SAML provider...

* SSOCookieProvider determines that there is no KnoxSSO cookie and
redirects you to the KnoxSSO endpoint
* KnoxSSO is configured for Okta or some other SAML provider and redirects
to the SAML provider endpoint
* SAML provider authenticates the user and posts back to the KnoxSSO
endpoint
* KnoxSSO sets the hadoop-jwt cookie and redirects to the originally
requested resource

The above establishes not only a KnoxSSO session but also a session with
the SAML provider via IDP specific cookies.

Now, when you invoke the KNOXSSOUT API from some logout link on an app page
the KNOXSSOUT service will remove
the KnoxSSO cookie and redirect you back. The SSOCookieProvider will not
find a cookie and send you back to KnoxSSO
which will send you to the IDP again and that session is still active. BAM!
You are logged right back in.

These types of nuances are different from provider to provider.

Now, if you are only using the default form based provider from Knox - it
may actually work for you as long as you also remove
any application specific cookies as well as call KNOXSSOUT.

As for a topology example, all you really need to do is add a topology -
say knoxssout.xml - and protect it with the Anonymous
authentication provider.

Hope that is helpful.

--larry

On Thu, Feb 21, 2019 at 8:39 AM Rajat Goel  wrote:

> Hi,
>
>
>
> I was looking for implementing Logout for my service which is integrated
> with Knox SSO (SSOCookieProvider).  I came across this Jira ticket
> https://issues.apache.org/jira/browse/KNOX-744 where a new service
> KNOXSSOUT is created which should be used in a new topology. Can someone
> please provide a sample topology file for using this service for
> invalidating cookie ?
>
>
>
> Also, my Knox version is 0.12 (HDP 2.6.5). Will the above approach work
> with my Knox version ?
>
>
>
> Thanks,
>
> Rajat
>

Re: LDAP configuration using knox.

[larry mccay]

In order to change a service definition in an existing deployment you have
to follow this special little dance to make sure the changes get picked up:

1. change service definition
2. restart the gateway so that it loads the new service definition server
side
3. touch the topology that is hosting the service so that it regenerates
the webapp that is created for each topology and deployed to
{GATEWAY_HOME}/data/deployments.

If you don't do 2 or 3, you won't see your changes.

Also, you don't want to specify a name for the authentication provider at
all - that way you get whatever authentication provider is in the topology.
In fact, you can remove the whole Policies section as you don't need to
force it to be anything other than defaults.

Your backend service will have the authenticated username asserted via a
doAs (in a secure cluster) or a user.name (in an unsecured cluster) query
param.

On Thu, Dec 20, 2018 at 9:20 AM Tomislav Novosel 
wrote:

> I changed line:
>  classname="org.apache.knox.gateway.dispatch.PassAllHeadersDispatch"/>
> in service.xml
> to:
>  
>
> and name of authentication provider I changed from anonymous to
> ShiroProvider.
> Nothing happened. Access is open as it was.
>
> Tom
>
> On Thu, 20 Dec 2018 at 14:39, larry mccay  wrote:
>
>> If you followed the proxying article and your service definition is
>> indicating the anonymous authentication provider then that is the issue.
>> That overrides any provider configured in the topology.
>>
>>
>> On Thu, Dec 20, 2018, 8:27 AM Kevin Risden >
>>> If your service.xml has
>>>
>>> >> classname="org.apache.knox.gateway.dispatch.PassAllHeadersDispatch"/>
>>>
>>> Then all the headers including the AUTHORIZATION header are passed
>>> through to the backend. This would mean that your backend is
>>> responsible for authentication.
>>>
>>> If you want Knox to do authentication, either remove the dispatch line
>>> or change it to:
>>>
>>> 
>>>
>>> This will then pass through Kerberized credentials to your backend. If
>>> you are looking to do SSO with Knox then proxying isn't required.
>>>
>>> Kevin Risden
>>> On Thu, Dec 20, 2018 at 8:22 AM Tomislav Novosel 
>>> wrote:
>>> >
>>> > Hi Knox team,
>>> >
>>> > I'm trying to configure LDAP authentication on Knox. I followed this
>>> link to setup simple web application for testing access. It's a simple
>>> HelloWorld.
>>> >
>>> >
>>> https://cwiki.apache.org/confluence/display/KNOX/Proxying+a+UI+using+Knox
>>> >
>>> > After setup, I've succeeded to access app through Knox gateway.
>>> >
>>> > I want to restrict access on that webapp URL with some username and
>>> password to type, so I tried to configure LDAP using this link:
>>> >
>>> >
>>> https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_Knox_Gateway_Admin_Guide/content/setting_up_ldap_authentication.html
>>> >
>>> > After that, access was still the same, without user/pass prompt. IDK
>>> if that is even posible with Knox.
>>> > In logs I see that access is granted, response code 200.
>>> >
>>> > How can I achieve user/pass prompt when accessing webapp URL through
>>> Knox gateway?
>>> >
>>> > Thank you.
>>> > BR,
>>> > Tom
>>>
>>

Re: LDAP configuration using knox.

[larry mccay]

If you followed the proxying article and your service definition is
indicating the anonymous authentication provider then that is the issue.
That overrides any provider configured in the topology.


On Thu, Dec 20, 2018, 8:27 AM Kevin Risden  If your service.xml has
>
>  classname="org.apache.knox.gateway.dispatch.PassAllHeadersDispatch"/>
>
> Then all the headers including the AUTHORIZATION header are passed
> through to the backend. This would mean that your backend is
> responsible for authentication.
>
> If you want Knox to do authentication, either remove the dispatch line
> or change it to:
>
> 
>
> This will then pass through Kerberized credentials to your backend. If
> you are looking to do SSO with Knox then proxying isn't required.
>
> Kevin Risden
> On Thu, Dec 20, 2018 at 8:22 AM Tomislav Novosel 
> wrote:
> >
> > Hi Knox team,
> >
> > I'm trying to configure LDAP authentication on Knox. I followed this
> link to setup simple web application for testing access. It's a simple
> HelloWorld.
> >
> >
> https://cwiki.apache.org/confluence/display/KNOX/Proxying+a+UI+using+Knox
> >
> > After setup, I've succeeded to access app through Knox gateway.
> >
> > I want to restrict access on that webapp URL with some username and
> password to type, so I tried to configure LDAP using this link:
> >
> >
> https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_Knox_Gateway_Admin_Guide/content/setting_up_ldap_authentication.html
> >
> > After that, access was still the same, without user/pass prompt. IDK if
> that is even posible with Knox.
> > In logs I see that access is granted, response code 200.
> >
> > How can I achieve user/pass prompt when accessing webapp URL through
> Knox gateway?
> >
> > Thank you.
> > BR,
> > Tom
>

Re: Question about shared provides & simplified descriptors

[larry mccay]

Hi Lars -

It is indeed true that shared provider configs are only consumed by simple
descriptors.
They do so by having them pulled into the compiled topologies as you
suspect.
It would be good to make that clear.

The answer to whether you should use one or the other is - it depends...

1. Do you have a source for discovery like Ambari - which is the only one
we currently have support for? This would be a No for you apparently.
2. Do you hate XML with the fury of a thousand suns? Some folks do and
would much rather click and edit text boxes. I am not one of them.

Either of the above answers being 'yes' then I would suggest that you
switch.

Discovery is really nice - if you haven't read much about that and only the
provider configs and descriptors without that context then you should dig
in there a bit more.
Move a NN to another host with Ambari and we find out without having to
restart Knox at all.
A port number changes through Ambari and we find out and regenerate the
topology with the new port number - no changes needed in topologies or
restarts of Knox.
If you include a service in a descriptor that hasn't been installed yet and
it subsequently gets installed by Ambari, we find it and regenerate the
topology and expose the previously unavailable service automatically.

The other thing that is related and is also only supported for provider
configs and descriptors, is remote configuration monitoring.
Instead of having to keep all of your Knox instances in sync with the same
topologies, pushing the configs and descriptors to Zookeeper means that all
the instances will be in sync via monitoring of the remote configuration in
ZK. This makes Knox HA deployments much easier. Again, straight topologies
do not work here. So a possible #3 to the above list is would you like
centralized management of topologies that are not limited to the three
known by Ambari.

Anyway, yeah, its good stuff but we will continue to support straight
topologies as the least common denominator but do not intend to have the
above types of improvements available for them.

Look forward to your docs patch.

thanks,

--larry


On Wed, Dec 5, 2018 at 1:51 PM Lars Francke  wrote:

> Hi,
>
> I read the docs about shared providers and simplified descriptors today.
> The documentation hints at it but doesn't actually describe or mention it:
>
> Shared providers can only be used with simplified descriptors because they
> are actually compiled into XML topology files?
>
> If that understanding is correct I'll add something to the docs.
>
> And if it is correct I have two follow-up questions:
> Should we still be writing XML topology files or is the recommendation to
> switch to the simplified ones?
>
> I think I saw in the code that the discovery type defaults to Ambari if I
> don't provide one. Is there any problem if I don't use Ambari?
>
> Cheers,
> Lars
>

Re: [CANCEL][VOTE] Release Apache Knox 1.2.0 RC 2

[larry mccay]

Good find, Kevin!

On Fri, Nov 30, 2018 at 10:13 AM Kevin Risden  wrote:

> -1 found https://issues.apache.org/jira/browse/KNOX-1645
>
> All other testing went well minus Zeppelin UI due to KNOX-1645. I'll
> respin a new rc later today.
>
> Tested the following:
>
>- src zip - mvn verify -Ppackage,release
>- knoxshell - able to connect to Knox and execute a few examples
>- Ran rc2 against https://github.com/risdenk/knox-performance-tests to
>test unsecure webhdfs, hbase, hive
>- Manual testing of UIs with Kerberos
>- Zeppelin UI with websocket enabled
>- Manual testing of Knox against Kerberized cluster
>- knoxsso Pac4j backed by Okta
>- hadoopauth provider
>- token service and Bearer tokens
>- default topology url
>- topology port mapping
>- ambari discovery
>- hadoop group provider
>
> Kevin Risden
>
>
> On Wed, Nov 28, 2018 at 2:41 PM larry mccay  wrote:
>
>> All -
>>
>> Thanks to Kevin for so much work in cleaning up the backlog and taking on
>> release manager work for 1.2.0!
>>
>> The 1.2.0 release happens to contain many dependency upgrades.
>> Not the least of which is Jetty itself from 9.2.x to 9.4.x.
>>
>> We need to put some key areas through their paces pretty rigorously in
>> order to tease out any corner case regressions.
>>
>> For instance, any Jetty specific features - such as:
>> * Websocket support
>> * SSL configuration params
>> * buffer size tweaks
>> * comb the gateway config and see which things are configuring aspects of
>> Jetty where units of measure defaults may change, etc
>>
>> * Pac4J was upgraded - we will need KnoxSSO testing of SAML via Okta,
>> Google Authenticator, etc.
>>
>> * Classloading is already known to have changed and has caused some issues
>> that have already been found.
>> Give some thought into possible classloading issues: singletons, custom
>> providers added to the ext directory, etc.
>>
>> If anyone can think of other corner cases that may not be immediately
>> covered by default and common configurations please call them out and/or
>> list what you have tested for them.
>>
>> thanks!
>>
>> --larry
>>
>> On Wed, Nov 28, 2018 at 2:16 PM Jeffrey Rodriguez 
>> wrote:
>>
>> > +1 , based on my review and testing.
>> >
>> > Thanks,
>> > Jeffrey E Rodriguez
>> >
>> > On Wed, Nov 28, 2018 at 11:05 AM
>> Kevin Risden
>>  wrote:
>> >
>> > > Release candidate #2 for the Apache Knox 1.2.0 release is available
>> at:
>> > >
>> > > https://dist.apache.org/repos/dist/dev/knox/knox-1.2.0/
>> > >
>> > > The release candidate is a zip archive of the sources in:
>> > >
>> > > https://git-wip-us.apache.org/repos/asf/knox.git
>> > > Branch v1.2.0 (git checkout -b v1.2.0)
>> > > Tag is v1.2.0-rc2 (git checkout -b v1.2.0-rc2)
>> > >
>> > > The KEYS file for signature validation is available at:
>> > > https://dist.apache.org/repos/dist/release/knox/KEYS
>> > >
>> > > Please vote on releasing this package as Apache Knox 1.2.0.
>> > > The vote is open for the next 72 hours and passes if a majority of at
>> > > least three +1 Apache Knox PMC votes are cast.
>> > >
>> > > [ ] +1 Release this package as Apache Knox 1.2.0
>> > > [ ] -1 Do not release this package because...
>> > >
>> > > Kevin Risden
>> > >
>> >
>>
>

Re: Knox LDAP group filer is not working

[larry mccay]

Hi Raja -

I need to better understand why you have a need to do the filtering within
the authentication provider.
This is more easily done within the authorization provider and leaves you
with more options for doing group lookup.
At some point, Instead of doing it from LDAP you may want to use local OS
account groups for instance.

My preference these days is to keep the authentication provider config very
simple to establish the authenticated user, use Hadoop Group Lookup
Provider [1] to look up groups exactly the way is done within Hadoop
itself. Then use the AclsAuthzProvider to provide service level
authorization based on group membership, username and/or ip address [2].

By decoupling these things, you have the most flexibility to compose and
even extend providers to meet your needs.

thanks,

--larry

1.
https://knox.apache.org/books/knox-1-1-0/user-guide.html#Hadoop+Group+Lookup+Provider
2. https://knox.apache.org/books/knox-1-1-0/user-guide.html#Authorization

On Tue, Nov 27, 2018 at 8:45 AM Kevin Risden  wrote:

> The username you used to login doesn't match the regex you gave.
>
> 
>
> If you are going to only specify the username you need to modify the regex
> to only have one capture group.
>
> Kevin Risden
>
>
> On Mon, Nov 26, 2018 at 9:08 PM Raja Marimuthu <
> raja.marimu...@northbaysolutions.com> wrote:
>
>> Kevin,
>>
>>
>>
>> Thank you so much.  When I have tried with prinicipalrgeex, I get below
>> error
>>
>>
>>
>>
>>
>> User DN : CN=Len,OU=US02P01,OU=mmc_users,DC=ds,DC=nb,DC=com
>>
>> Group DN : CN=m_powerusers,OU=Applications,OU=Groups,DC=ds,DC=nb,DC=com
>>
>>
>>
>> I need to provide. Access only to m_powerusers. group
>>
>>
>>
>> Setting 1:
>>
>>   > value="dc=ds,dc=nb,dc=com"/>
>>
>> 
>>
>>
>>
>>   > value="((objectclass=person)(memberOf=cn={1},OU=Applications,OU=Groups,DC=ds,DC=nb,DC=com)(sAMAccountName={2}))"/>
>>
>>
>>
>>
>>
>> 
>>
>>
>>
>>
>>
>> Setting 2 :
>>
>>
>>
>>   > value="dc=ds,dc=nb,dc=com"/>
>>
>> 
>>
>>
>>
>>   > value="((objectclass=person)(memberOf=cn=m_powerusers,OU=Applications,OU=Groups,DC=ds,DC=nb,DC=com)(sAMAccountName={2}))"/>
>>
>>
>>
>>
>>
>> 
>>
>>
>>
>>
>>
>>
>>
>> User DN : CN=Len,OU=US02P01,OU=mmc_users,DC=ds,DC=nb,DC=com
>>
>> Group DN : CN=m_powerusers,OU=Applications,OU=Groups,DC=ds,DC=nb,DC=com
>>
>>
>>
>> Error :
>>
>>
>>
>>
>>
>>
>>
>> 2018-11-27 02:02:13,678 WARN  authc.AbstractAuthenticator
>> (AbstractAuthenticator.java:authenticate(216)) - Authentication failed for
>> token submission [org.apache.shiro.authc.UsernamePasswordToken - len,
>> rememberMe=false (73.230.13.102)].  Possible unexpected error? (Typical or
>> expected login exceptions should extend from AuthenticationException).
>>
>> java.lang.IllegalArgumentException: Principal len does not match
>> (.*?)\\(.*?)
>>
>> at
>> org.apache.knox.gateway.shirorealm.KnoxLdapRealm.matchPrincipal(KnoxLdapRealm.java:658)
>>
>> at
>> org.apache.knox.gateway.shirorealm.KnoxLdapRealm.getUserDn(KnoxLdapRealm.java:681)
>>
>> at
>> org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm.getUserDn(KnoxLdapRealm.java:98)
>>
>> at
>> org.apache.shiro.realm.ldap.JndiLdapRealm.getLdapPrincipal(JndiLdapRealm.java:342)
>>
>> at
>> org.apache.shiro.realm.ldap.JndiLdapRealm.queryForAuthenticationInfo(JndiLdapRealm.java:371)
>>
>> at
>> org.apache.shiro.realm.ldap.JndiLdapRealm.doGetAuthenticationInfo(JndiLdapRealm.java:295)
>>
>> at
>> org.apache.knox.gateway.shirorealm.KnoxLdapRealm.doGetAuthenticationInfo(KnoxLdapRealm.java:200)
>>
>> at
>> org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm.doGetAuthenticationInfo(KnoxLdapRealm.java:54)
>>
>> at
>> org.apache.shiro.realm.AuthenticatingRealm.getAuthenticationInfo(AuthenticatingRealm.java:568)
>>
>> at
>> org.apache.shiro.authc.pam.ModularRealmAuthenticator.doSingleRealmAuthentication(ModularRealmAuthenticator.java:180)
>>
>> at
>> org.apache.shiro.authc.pam.ModularRealmAuthenticator.doAuthenticate(ModularRealmAuthenticator.java:267)
>>
>> at
>> org.apache.shiro.authc.AbstractAuthenticator.authenticate(AbstractAuthenticator.java:198)
>>
>> at
>> org.apache.shiro.mgt.AuthenticatingSecurityManager.authenticate(AuthenticatingSecurityManager.java:106)
>>
>> at
>> org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:270)
>>
>> at
>> org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256)
>>
>> at
>> org.apache.shiro.web.filter.authc.AuthenticatingFilter.executeLogin(AuthenticatingFilter.java:53)
>>
>> at
>> org.apache.shiro.web.filter.authc.BasicHttpAuthenticationFilter.onAccessDenied(BasicHttpAuthenticationFilter.java:190)
>>
>> at
>> org.apache.shiro.web.filter.AccessControlFilter.onAccessDenied(AccessControlFilter.java:133)
>>
>> at
>> org.apache.shiro.web.filter.AccessControlFilter.onPreHandle(AccessControlFilter.java:162)

Re: load balancing of Hiveserver 2 throught Knox

[larry mccay]

Hi Rabii -

HIve HA through Knox does not do LBing across Hive instances.
It leverages the Hive ZK based HA for failover to another instance in the
event of connection failures.

thanks,

--larry

On Mon, Oct 8, 2018 at 7:25 AM rabii lamriq  wrote:

> Hi
>
> Can we configure HA and load balancing of HS2 via knox?
>
> Regards
> Rabii
>

Re: WebHDFS performance issue in Knox

[larry mccay]

SSL handshake will likely happen at least twice.
Once for the request through Knox to the NN then the redirect from the NN
to the DN goes all the way back to the client.
So they have to follow the redirect and do the handshake to the DN.


On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden  wrote:

> So I found this in the Knox issues list in JIRA:
>
> https://issues.apache.org/jira/browse/KNOX-1221
>
> It sounds familiar in terms of a slowdown when going through Knox.
>
> Kevin Risden
>
>
> On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden  wrote:
>
>> Hmmm yea curl for a single file should do the handshake once.
>>
>> What are the system performance statistics during the SSL vs non SSL
>> testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here
>> if using that. Otherwise watching top may be helpful. It would be help to
>> determine if the Knox is working harder during the SSL transfer.
>>
>> Kevin Risden
>>
>>
>> On Wed, Sep 12, 2018 at 2:52 PM Guang Yang  wrote:
>>
>>> I'm just using curl to download a single large file. So I suspect SSL
>>> handshake just happens once?
>>>
>>> On Tue, Sep 11, 2018 at 12:02 PM
>>> Kevin Risden
>>>  wrote:
>>>
>>>> What client are you using to connect Knox? Is this for a single file or
>>>> a bunch of files?
>>>>
>>>> The SSL handshake can be slow if the client doesn't keep the connection
>>>> open.
>>>>
>>>> Kevin Risden
>>>>
>>>> On Tue, Sep 11, 2018, 14:51 Guang Yang  wrote:
>>>>
>>>>> Thanks Larry. But the only difference is this part in my
>>>>> gateway-site.xml.
>>>>>
>>>>> **
>>>>> *ssl.enabled*
>>>>> *false*
>>>>> *Indicates whether SSL is enabled.*
>>>>> **
>>>>>
>>>>> On Tue, Sep 11, 2018 at 11:42 AM, larry mccay 
>>>>> wrote:
>>>>>
>>>>>> I really don't think that kind of difference should be expected from
>>>>>> merely SSL overhead.
>>>>>> I don't however have any metrics to contradict it either since I do
>>>>>> not run Knox without SSL.
>>>>>>
>>>>>> Given the above, I am struggling coming up with a meaningful response
>>>>>> to this. :(
>>>>>> I don't think you should see a 10 fold increase in speed by disabling
>>>>>> SSL though.
>>>>>>
>>>>>> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang  wrote:
>>>>>>
>>>>>>> Any idea guys?
>>>>>>>
>>>>>>> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang  wrote:
>>>>>>>
>>>>>>>> Thanks guys! The issue seems exactly what David pointed out, which
>>>>>>>> is because of encrypted over SSL.
>>>>>>>>
>>>>>>>> Without Knox, the download speed can reach to *400M/s* if I call
>>>>>>>> Namenode directly. And with disabling SSL, the speed can reach to
>>>>>>>> *~400M/s* as well through Knox. But with SSL, the speed drops
>>>>>>>> significantly to *~40M/s*. I know it's because of encrypted, but
>>>>>>>> it does surprised me with such a difference. Is it normal from your
>>>>>>>> perspective?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Guang
>>>>>>>>
>>>>>>>> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal <
>>>>>>>> dvillarr...@hortonworks.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Guang,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Keep in mind the data is being encrypted over SSL.  If you disable
>>>>>>>>> SSL you will most likely see a very significant boost in throughput.  
>>>>>>>>> Some
>>>>>>>>> people have used more powerful computers to make encryption quicker.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> David

Re: WebHDFS performance issue in Knox

[larry mccay]

I really don't think that kind of difference should be expected from merely
SSL overhead.
I don't however have any metrics to contradict it either since I do not run
Knox without SSL.

Given the above, I am struggling coming up with a meaningful response to
this. :(
I don't think you should see a 10 fold increase in speed by disabling SSL
though.

On Tue, Sep 11, 2018 at 2:35 PM Guang Yang  wrote:

> Any idea guys?
>
> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang  wrote:
>
>> Thanks guys! The issue seems exactly what David pointed out, which is
>> because of encrypted over SSL.
>>
>> Without Knox, the download speed can reach to *400M/s* if I call
>> Namenode directly. And with disabling SSL, the speed can reach to
>> *~400M/s* as well through Knox. But with SSL, the speed drops
>> significantly to *~40M/s*. I know it's because of encrypted, but it does
>> surprised me with such a difference. Is it normal from your perspective?
>>
>> Thanks,
>> Guang
>>
>> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal <
>> dvillarr...@hortonworks.com> wrote:
>>
>>> Hi Guang,
>>>
>>>
>>>
>>> Keep in mind the data is being encrypted over SSL.  If you disable SSL
>>> you will most likely see a very significant boost in throughput.  Some
>>> people have used more powerful computers to make encryption quicker.
>>>
>>>
>>>
>>> Thanks,
>>>
>>>
>>>
>>> David
>>>
>>>
>>>
>>> *From: *Sean Roberts 
>>> *Reply-To: *"user@knox.apache.org" 
>>> *Date: *Tuesday, September 4, 2018 at 1:53 AM
>>> *To: *"user@knox.apache.org" 
>>> *Subject: *Re: WebHDFS performance issue in Knox
>>>
>>>
>>>
>>> Guang – This is somewhat to be expected.
>>>
>>>
>>>
>>> When you talk to WebHDFS directly, the client can distribute the request
>>> across many data nodes. Also, you are getting data directly from the source.
>>>
>>> With Knox, all traffic goes through the single Knox host. Knox is
>>> responsible for fetching from the datanodes and consolidating to send to
>>> you. This means overhead as it’s acting as a middle man, and lower network
>>> capacity since only 1 host is serving data to you.
>>>
>>>
>>>
>>> Also, if running on a cloud provider, the Knox host may be a smaller
>>> instance size with lower network capacity.
>>>
>>> --
>>>
>>> Sean Roberts
>>>
>>>
>>>
>>> *From: *Guang Yang 
>>> *Reply-To: *"user@knox.apache.org" 
>>> *Date: *Tuesday, 4 September 2018 at 07:46
>>> *To: *"user@knox.apache.org" 
>>> *Subject: *WebHDFS performance issue in Knox
>>>
>>>
>>>
>>> Hi,
>>>
>>>
>>>
>>> We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file
>>> through WebHDFS in Knox, the download speed is just about 11M/s. However,
>>> if we download directly from datanode, the speed is about 40M/s at least.
>>>
>>>
>>>
>>> Are you guys aware of this problem? Any suggestion?
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Guang
>>>
>>
>>
>

Re: turn off debug logging from org.apache.http.wire

[larry mccay]

.logger.org.apache.http.wire=ERROR*
>
> Even I changed it like this, I can still see lots of DEBUG log in
> *gateway.out*. Seems it only affects* gateway.log*, not *gateway.out.*
>
> On Wed, Sep 5, 2018 at 10:48 AM, larry mccay  wrote:
>
>> Hi Guang -
>>
>> This certainly sounds frustrating.
>> I have never had trouble turning it off.
>> Can you share your gatway-log4j.properties file - just make sure there
>> isn't anything sensitive in there?
>>
>> thanks,
>>
>> --larry
>>
>> On Wed, Sep 5, 2018 at 1:40 PM Guang Yang  wrote:
>>
>>> And we're not using Ambari. We just deploy manually.
>>>
>>> On Tue, Sep 4, 2018 at 11:02 PM, Guang Yang  wrote:
>>>
>>>> Hi guys,
>>>>
>>>> I'm working with Wei and we still don't figure it out. Let me clarify
>>>> the question.
>>>>
>>>> Currently, we're seeing lots of DEBUG logs in file *gateway.out*,
>>>> which is from here
>>>> https://github.com/apache/knox/blob/master/gateway-release/home/bin/gateway.sh#L127.
>>>> On the one hand, it prints the file content just like Wei talked about
>>>> before, on the other hand we suspect it might be related to the performance
>>>> issue when download a file through WEBHDFS. So we're trying to disable all
>>>> these DEBUG logs. We tried simply removing this part *>>$APP_OUT_FILE*,
>>>> although there is no such output file, but actually Knox still prints logs
>>>> to console. So what we want to do is to disable all the DEBUG log
>>>> thoroughly, so the service won't print logs to anywhere.
>>>>
>>>> We almost tried everything in *gateway-log4j.properties*, but it seems
>>>> it only affects app.log.file=${launcher.name}.*log* instead of
>>>> *gateway.out*. So, any idea guys?
>>>>
>>>> Thanks,
>>>> Guang
>>>>
>>>> On Sun, Apr 15, 2018 at 11:08 AM, larry mccay 
>>>> wrote:
>>>>
>>>>> +1 to Kevin's point.
>>>>> Ambari rewrites all configs on server restart.
>>>>>
>>>>> On Sun, Apr 15, 2018 at 1:16 PM, Kevin Risden 
>>>>> wrote:
>>>>>
>>>>>> Are you using Ambari or deploying Knox manually?
>>>>>>
>>>>>> If you using Ambari, then Ambari will force overwrite the log4j
>>>>>> configs during a restart. You must update the log4j settings in Ambari.
>>>>>> Another option if using Ambari is to find the debug setting and set it to
>>>>>> false (I don't have a cluster in front of me so can't look up the 
>>>>>> setting).
>>>>>>
>>>>>> Kevin Risden
>>>>>>
>>>>>> On Sun, Apr 15, 2018 at 10:56 AM, Wei Han  wrote:
>>>>>>
>>>>>>> Interesting. Thanks Larry. I'll dig more on my side.
>>>>>>>
>>>>>>> On Sun, Apr 15, 2018 at 4:54 AM, larry mccay 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> No, I cannot reproduce it.
>>>>>>>> If you are modifying the correct gateway-log4j.properties and
>>>>>>>> restarting the server you should not see that.
>>>>>>>>
>>>>>>>> In fact, turning on DEBUG for wire via:
>>>>>>>> log4j.logger.org.apache.http.wire=DEBUG
>>>>>>>>
>>>>>>>> Doesn't result in output in gateway.out for me but instead
>>>>>>>> gateway.log and turning it on and off certainly works for me.
>>>>>>>>
>>>>>>>> You may have enabled TRACE logging if you are seeing body content -
>>>>>>>> those settings are like the following:
>>>>>>>>
>>>>>>>> #log4j.logger.org.apache.knox.gateway.access=TRACE,httpaccess
>>>>>>>> #log4j.additivity.org.apache.knox.gateway.access=false
>>>>>>>>
>>>>>>>> #log4j.logger.org.apache.knox.gateway.http=TRACE,httpserver
>>>>>>>> #log4j.additivity.org.apache.knox.gateway.http=false
>>>>>>>> ##log4j.logger.org.apache.knox.gateway.http.request.headers=OFF
>>>>>>>> ##log4j.logger.org.apache.knox.gateway.http.response.headers=OFF
>>>>>>>> ##log4j.logger.org.apache.knox.gateway.http.request.body=OFF
>

Re: turn off debug logging from org.apache.http.wire

[larry mccay]

Hi Guang -

This certainly sounds frustrating.
I have never had trouble turning it off.
Can you share your gatway-log4j.properties file - just make sure there
isn't anything sensitive in there?

thanks,

--larry

On Wed, Sep 5, 2018 at 1:40 PM Guang Yang  wrote:

> And we're not using Ambari. We just deploy manually.
>
> On Tue, Sep 4, 2018 at 11:02 PM, Guang Yang  wrote:
>
>> Hi guys,
>>
>> I'm working with Wei and we still don't figure it out. Let me clarify the
>> question.
>>
>> Currently, we're seeing lots of DEBUG logs in file *gateway.out*, which
>> is from here
>> https://github.com/apache/knox/blob/master/gateway-release/home/bin/gateway.sh#L127.
>> On the one hand, it prints the file content just like Wei talked about
>> before, on the other hand we suspect it might be related to the performance
>> issue when download a file through WEBHDFS. So we're trying to disable all
>> these DEBUG logs. We tried simply removing this part *>>$APP_OUT_FILE*,
>> although there is no such output file, but actually Knox still prints logs
>> to console. So what we want to do is to disable all the DEBUG log
>> thoroughly, so the service won't print logs to anywhere.
>>
>> We almost tried everything in *gateway-log4j.properties*, but it seems
>> it only affects app.log.file=${launcher.name}.*log* instead of
>> *gateway.out*. So, any idea guys?
>>
>> Thanks,
>> Guang
>>
>> On Sun, Apr 15, 2018 at 11:08 AM, larry mccay  wrote:
>>
>>> +1 to Kevin's point.
>>> Ambari rewrites all configs on server restart.
>>>
>>> On Sun, Apr 15, 2018 at 1:16 PM, Kevin Risden 
>>> wrote:
>>>
>>>> Are you using Ambari or deploying Knox manually?
>>>>
>>>> If you using Ambari, then Ambari will force overwrite the log4j configs
>>>> during a restart. You must update the log4j settings in Ambari. Another
>>>> option if using Ambari is to find the debug setting and set it to false (I
>>>> don't have a cluster in front of me so can't look up the setting).
>>>>
>>>> Kevin Risden
>>>>
>>>> On Sun, Apr 15, 2018 at 10:56 AM, Wei Han  wrote:
>>>>
>>>>> Interesting. Thanks Larry. I'll dig more on my side.
>>>>>
>>>>> On Sun, Apr 15, 2018 at 4:54 AM, larry mccay 
>>>>> wrote:
>>>>>
>>>>>> No, I cannot reproduce it.
>>>>>> If you are modifying the correct gateway-log4j.properties and
>>>>>> restarting the server you should not see that.
>>>>>>
>>>>>> In fact, turning on DEBUG for wire via:
>>>>>> log4j.logger.org.apache.http.wire=DEBUG
>>>>>>
>>>>>> Doesn't result in output in gateway.out for me but instead
>>>>>> gateway.log and turning it on and off certainly works for me.
>>>>>>
>>>>>> You may have enabled TRACE logging if you are seeing body content -
>>>>>> those settings are like the following:
>>>>>>
>>>>>> #log4j.logger.org.apache.knox.gateway.access=TRACE,httpaccess
>>>>>> #log4j.additivity.org.apache.knox.gateway.access=false
>>>>>>
>>>>>> #log4j.logger.org.apache.knox.gateway.http=TRACE,httpserver
>>>>>> #log4j.additivity.org.apache.knox.gateway.http=false
>>>>>> ##log4j.logger.org.apache.knox.gateway.http.request.headers=OFF
>>>>>> ##log4j.logger.org.apache.knox.gateway.http.response.headers=OFF
>>>>>> ##log4j.logger.org.apache.knox.gateway.http.request.body=OFF
>>>>>> ##log4j.logger.org.apache.knox.gateway.http.response.body=OFF
>>>>>>
>>>>>> I suggest you back up to the gateway-log4j.properties from the
>>>>>> original install and remove any other log4j config that you may have
>>>>>> elsewhere.
>>>>>>
>>>>>> On Sun, Apr 15, 2018 at 1:58 AM, Wei Han  wrote:
>>>>>>
>>>>>>> Hi Larry - Thanks a lot for getting back to me.
>>>>>>>
>>>>>>> Yes I made sure all DEBUG level is turned off in my 
>>>>>>> gateway-log4j.properties
>>>>>>> file, but that doesn't seem to be working. I also tried to
>>>>>>> explicitly set log4j.logger.httpclient.wire.header to WARN (as 
>>>>>>> suggested in
>>>>>>> post
>>>>>>> <https://st

Re: Impersonate/ProxyUser through Knox?

[larry mccay]

Hi Sean -

The mechanism for doing such impersonation is through identity assertion
providers.
We have a number of them out of the box.

In order to do this with the same sort of validation and trust
configuration, a new one would likely be needed that took such
configuration.
You would then assert the effective user as the user in the header or query
param that you are checking.

I don't think that using the typical user.name or doas query params will
work since we currently scrub any incoming requests of such impersonation
attempts as it could be an attempt to spoof another identity by the client.

We could also look into providing the trusted proxy config on top of the
HadoopAuthProvider but that would make such impersonation be tightly
coupled to that provider. Maybe that makes sense since it is a Hadoop
specific pattern but at the same time - much of the use of Knox is to avoid
having to use kerberos.

Anyway, you can certainly file a JIRA for a feature and we can discuss the
usecases more in depth there.

thanks,

--larry

On Fri, Aug 31, 2018 at 5:04 PM Sean Roberts 
wrote:

> David – Would you agree that this is a valid feature request?
>
>
>
> Hortonworks docs suggest replacing HttpFs with Knox, but this is a use
> case where Knox cannot replace HttpFs which has its own proxyuser
> functionality.
>
>
>
>
>
> --
>
> Sean Roberts
>
>
>
> *From: *David Villarreal 
> *Date: *Friday, 31 August 2018 at 21:38
> *To: *Sean Roberts , "user@knox.apache.org" <
> user@knox.apache.org>
> *Subject: *Re: Impersonate/ProxyUser through Knox?
>
>
>
> Hi Sean,
>
>
>
> Proxy/Impersonation is configured on the Hadoop side.  And knox
> user/principal impersonates users.  I think the answer to this question is
> no….   Knox does not have its own proxy impersonation provider.
>
>
>
> What I know Knox does have is
>
>
> https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_security/content/knox_configuring_identity_assertion.html
>
> http://kminder.github.io/knox/2015/11/20/identity-assertion.html
>
> http://knox.apache.org/books/knox-1-1-0/user-guide.html#Identity+Assertion
>
>
>
>
>
> *From: *Sean Roberts 
> *Date: *Friday, August 31, 2018 at 12:43 PM
> *To: *"user@knox.apache.org" 
> *Subject: *Impersonate/ProxyUser through Knox?
>
>
>
> Knox experts – Does Knox provide impersonation/proxyuser functionality
> like direct WebHDFS connections *(hadoop.proxyuser.service-user.users)*
> and HttpFS *(httpfs.proxyuser.service-user.users)*?
>
>
>
> For example:
>
> -  “service-user” authenticates to Knox, then requests to run
> commands as “normal-user”.
>
>
>
> --
>
> Sean Roberts
>

Re: zeppelin via knox in HDP3 does not work for SQL

[larry mccay]

Replacing the service definition files alone isn't quite enough.
You have to do the following to make sure that the server picks up the new
service defs and redeploys the topology hosting the affected service:

1. change rewrite rules
2. restart gateway so that the gateway is aware of the new service def
3. touch/save the topology hosting the ZEPPELIN services so that they
redeploy

On Fri, Aug 31, 2018 at 7:04 PM Lian Jiang  wrote:

> Thanks Sandeep. Could you please clarify the patching process?
>
> There is existing rewrite.xml and service.xml under
> /var/lib/knox/data-3.0.0.0-1634/services/zeppelinui/0.6.0 on the knox host.
> None of below makes zeppelin sql interpreter work:
>
> 1. Put the new rewrite.xml and service.xml into new created 0.8.0 folder
> and restart knox
> 2. Replace these two files under 0.6.0 with the two new files and restart
> knox
>
> Appreciate any clue.
>
> On Fri, Aug 31, 2018 at 1:15 PM Sandeep Moré 
> wrote:
>
>> Knox master has patch for this issue
>> https://issues.apache.org/jira/browse/KNOX-1424
>>
>> You can try applying the patch it should fix the issue.
>>
>> Best,
>> Sandeep
>>
>> On Fri, Aug 31, 2018 at 12:18 PM Lian Jiang 
>> wrote:
>>
>>> I am using HDP3.0 (having zeppelin 0.8.0).
>>>
>>> Below queries do not show any result, even though downloading the csv
>>> show the data correctly (e.g. if there is no tables, show the header).
>>>
>>> %livy2.sql
>>> show tables
>>>
>>> %spark2.sql
>>> show tables
>>>
>>> Infrequently I saw the table show a short time and then disappear. I
>>> guess it is html rendering issue.
>>>
>>> This issue does not happen when I ssh to the zeppelin host and use
>>> zeppelin via port forwarding. It happens when I use zeppelin via knox.
>>>
>>> Any idea? Any setting is related to this? Sql interpreters worked fine
>>> in HDP2.6 using zeppelin 0.7.3.
>>>
>>>
>>> javascript error:
>>> Error: [ngRepeat:iidexp] '_item_' in '_item_ in _collection_' should be
>>> an identifier or '(_key_, _value_)' expression, but got
>>> '/gateway/ui/zeppelin/app'.
>>>
>>> http://errors.angularjs.org/1.5.7/ngRepeat/iidexp?p0=%2Fgateway%2Fui%2Fzeppelin%2Fapp
>>> b/<@
>>> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:36:376
>>> Kg>> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:40:30580
>>> Z@
>>> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:37:4755
>>> S@
>>> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:36:30480
>>> S@
>>> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:36:30611
>>> S@
>>> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:36:30611
>>> N@
>>> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:36:29413
>>> X/<@
>>> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:37:440
>>> d@
>>> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:36:30833
>>> m@
>>> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:37:909
>>> mg>> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:40:17582
>>> xc/this.$get>> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:38:10966
>>> a/n.prototype.safeDigest@
>>> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:76:1460
>>> b@
>>> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:76:3748
>>> a/n.prototype._onMessageHandler@
>>> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:76:3960
>>> R/<@
>>> https://mydomain.com/gateway/ui/zeppelin/scripts/vendor.49d751b0c72342f6.js:36:5633
>>> " 
>>>
>>>
>>>
>>> Appreciate any clue.
>>>
>>

Re: Reason why we don't use higher version of Jetty

[larry mccay]

Did you resolve the issue?

On Thu, Aug 30, 2018 at 7:10 PM Guang Yang  wrote:

> Ok, the issue seems goes away after I upgrade jetty to 9.3.10 on Knox
> 1.1.0. But there was some issue on Knox 0.13.0 + Jetty 9.3.10.
>
> On Tue, Aug 28, 2018 at 1:23 PM, Guang Yang  wrote:
>
>> Hey Larry,
>>
>> We're using 0.13.0 and running on Linux version 4.4.92 (Debian 4.9.2-10),
>> the JDK version is 8.
>>
>> On Tue, Aug 28, 2018 at 1:01 PM, larry mccay  wrote:
>>
>>> Hi Guang -
>>>
>>> I do recall this FD issue from looong ago.
>>> Not sure what was done to address it but I haven't seen it in a few
>>> years.
>>>
>>> What version of Knox are you using?
>>> What OS and JDK versions are you using?
>>>
>>> We generally upgrade jetty based on identification of CVEs on current
>>> version but also try not to jump too many versions as it is often painful
>>> to make larger moves due to compatibility issues.
>>>
>>> thanks,
>>>
>>> --larry
>>>
>>> On Tue, Aug 28, 2018 at 3:03 PM, Guang Yang  wrote:
>>>
>>>> Hi guys,
>>>>
>>>> We're using Knox to proxy WEBHDFS request. The problem is when the
>>>> client shut down the connection, Knox will throw EofException. According to
>>>> our dashboard, tons of file descriptors are leaked. When the open files
>>>> reach to the limit, Knox can't receive any request because there is not
>>>> socket available.
>>>>
>>>> I dig a little bit and found there is a bug in lower version of Jetty
>>>> https://github.com/eclipse/jetty.project/issues/634. According that
>>>> link, the issue has been fixed at Jetty 9.3.10, so I upgrade Jetty in Knox.
>>>> But after sometime, Knox throw some other issue
>>>>
>>>> 2018-08-28 18:45:57,214 INFO  hadoop.gateway
>>>> (DefaultHaDispatch.java:executeRequest(93)) - Could not connect to server:
>>>> http://datanode/webhdfs/v1/sompath org.eclipse.jetty.io.EofException
>>>> org.eclipse.jetty.io.EofException
>>>> at org.eclipse.jetty.io
>>>> .ChannelEndPoint.flush(ChannelEndPoint.java:197)
>>>> at
>>>> org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:839)
>>>> at
>>>> org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.shutdownOutput(SslConnection.java:928)
>>>> at
>>>> org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.close(SslConnection.java:967)
>>>> at
>>>> org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:893)
>>>> at
>>>> org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.shutdownOutput(SslConnection.java:928)
>>>> at
>>>> org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.close(SslConnection.java:967)
>>>> at
>>>> org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:893)
>>>>
>>>> And then Knox stops working again. My question is did anybody see the
>>>> FD leak issue before? and have we ever tried to update Jetty to a higher
>>>> version?
>>>>
>>>> Thanks,
>>>> Guang
>>>>
>>>
>>>
>>
>

Re: Reason why we don't use higher version of Jetty

[larry mccay]

Hi Guang -

I do recall this FD issue from looong ago.
Not sure what was done to address it but I haven't seen it in a few years.

What version of Knox are you using?
What OS and JDK versions are you using?

We generally upgrade jetty based on identification of CVEs on current
version but also try not to jump too many versions as it is often painful
to make larger moves due to compatibility issues.

thanks,

--larry

On Tue, Aug 28, 2018 at 3:03 PM, Guang Yang  wrote:

> Hi guys,
>
> We're using Knox to proxy WEBHDFS request. The problem is when the client
> shut down the connection, Knox will throw EofException. According to our
> dashboard, tons of file descriptors are leaked. When the open files reach
> to the limit, Knox can't receive any request because there is not socket
> available.
>
> I dig a little bit and found there is a bug in lower version of Jetty
> https://github.com/eclipse/jetty.project/issues/634. According that link,
> the issue has been fixed at Jetty 9.3.10, so I upgrade Jetty in Knox. But
> after sometime, Knox throw some other issue
>
> 2018-08-28 18:45:57,214 INFO  hadoop.gateway 
> (DefaultHaDispatch.java:executeRequest(93))
> - Could not connect to server: http://datanode/webhdfs/v1/sompath
> org.eclipse.jetty.io.EofException
> org.eclipse.jetty.io.EofException
> at org.eclipse.jetty.io.ChannelEndPoint.flush(
> ChannelEndPoint.java:197)
> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(
> SslConnection.java:839)
> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.
> shutdownOutput(SslConnection.java:928)
> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.close(
> SslConnection.java:967)
> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(
> SslConnection.java:893)
> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.
> shutdownOutput(SslConnection.java:928)
> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.close(
> SslConnection.java:967)
> at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(
> SslConnection.java:893)
>
> And then Knox stops working again. My question is did anybody see the FD
> leak issue before? and have we ever tried to update Jetty to a higher
> version?
>
> Thanks,
> Guang
>

Re: HDP 3.0: zeppelin via knox shows empty page

[larry mccay]

Hi -

Did you happen to upgrade the 2.6 cluster to 3.0?
If so, this may be due to the versioned data directory not getting the
updated service definition for zeppelin.

Locate the previous version and the new version data directories and track
down the zeppelin service definitions within data/services/zeppelin and
compare the rewrite rules.

HTH,

--larry

On Fri, Aug 17, 2018 at 12:53 PM, Lian Jiang  wrote:

> Hi,
>
> I installed HDP3.0 using ambari 2.7 blueprint. All components show green
> in ambari UI. I also can load zeppelin by directly accessing 9995 port.
> However, I cannot load zeppelin via knox (show empty page) and below is the
> log in gateway.log:
>
> 2018-08-17 16:48:58,286 WARN  knox.gateway (GatewayFilter.java:doFilter(185))
> - Failed to match path /zeppelin/app.3117255b2162669ac38b.css
>
> 2018-08-17 16:48:58,656 WARN  knox.gateway (GatewayFilter.java:doFilter(185))
> - Failed to match path /jax/input/TeX/config.js
>
> 2018-08-17 16:48:58,657 WARN  knox.gateway (GatewayFilter.java:doFilter(185))
> - Failed to match path /jax/output/HTML-CSS/config.js
>
> 2018-08-17 16:48:58,657 WARN  knox.gateway (GatewayFilter.java:doFilter(185))
> - Failed to match path /extensions/tex2jax.js
>
> 2018-08-17 16:48:58,911 WARN  knox.gateway (GatewayFilter.java:doFilter(185))
> - Failed to match path /components/navbar/navbar.html
>
> 2018-08-17 16:48:58,911 WARN  knox.gateway (GatewayFilter.java:doFilter(185))
> - Failed to match path /app/notebook/shortcut.html
>
> 2018-08-17 16:48:58,913 WARN  knox.gateway (GatewayFilter.java:doFilter(185))
> - Failed to match path /components/note-create/note-create.html
>
> 2018-08-17 16:48:58,915 WARN  knox.gateway (GatewayFilter.java:doFilter(185))
> - Failed to match path /components/note-import/note-import.html
>
> 2018-08-17 16:48:58,916 WARN  knox.gateway (GatewayFilter.java:doFilter(185))
> - Failed to match path /components/login/login.html
>
> 2018-08-17 16:48:58,917 WARN  knox.gateway (GatewayFilter.java:doFilter(185))
> - Failed to match path /components/note-rename/note-rename.html
>
> 2018-08-17 16:48:59,099 WARN  knox.gateway (GatewayFilter.java:doFilter(185))
> - Failed to match path /app/home/home.html
>
> 2018-08-17 16:49:00,089 WARN  knox.gateway (GatewayFilter.java:doFilter(185))
> - Failed to match path /extensions/MathMenu.js
>
> 2018-08-17 16:49:01,089 WARN  knox.gateway (GatewayFilter.java:doFilter(185))
> - Failed to match path /extensions/MathZoom.js
>
>
> By using the same blueprint zeppelin worked via knox in HDP2.6 and ambari
> 2.6. Any idea? Appreciate any clue.
>

Re: Knox SSO - throwing null pointer exception on first time login

[larry mccay]

Hi Praveen -

Is there no stacktrace anywhere?
You are only getting the NPE line in the browser?

thanks,

--larry

On Thu, Aug 16, 2018 at 11:52 PM, Ravikumar, Praveen Krishnamoorthy <
rpkr...@amazon.com> wrote:

> Attached the SAML Tracer logs for reference. Could anyone please help me
> in this?
>
>
>
> Thanks,
>
> Praveen.
>
>
>
> *From: *"Ravikumar, Praveen Krishnamoorthy" 
> *Date: *Thursday, August 16, 2018 at 6:54 PM
> *To: *"user@knox.apache.org" 
> *Cc: *"Mohanan, Mahesh" 
> *Subject: *Knox SSO - throwing null pointer exception on first time login
>
>
>
> Greetings,
>
>
>
> I’m facing a weird issue on enabling knox-sso functionality for exposing
> UI’s running in AWS EMR. On enabling knox-sso, I tried to access the YARNUI
> site – after authentication its throwing *java.lang.null.pointer*
> exception but on submitting the url again (without clearing the cookies),
> the application is getting landed to the proper page.
>
>
>
> When I clear the cookies and access the URL again, Its throwing null
> pointer exception. We are unable to figure whether it is an issue on the
> Identity provider site or in the knox as Im not seeing any error log
> messages in both the ends.
>
>
>
> I have attached the log messages. Could any please help me in fixing this?
>
>
>
>
>
> Gatway.log -> After the first time login
>
>
>
> 2018-08-16 22:30:14,369 DEBUG knox.gateway (GatewayFilter.java:doFilter(119))
> - Received request: GET /yarn/
>
> 2018-08-16 22:30:14,370 DEBUG federation.jwt 
> (SSOCookieFederationFilter.java:doFilter(114))
> - Sending redirect to: https://knox.idf-emr-prd.a..com:8446/gateway/
> knoxsso/api/v1/websso?originalUrl=https://knox.idf-
> emr-prd.a.x.com:8446/gateway/gate1/yarn/
>
> 2018-08-16 22:30:14,493 DEBUG knox.gateway (GatewayFilter.java:doFilter(119))
> - Received request: GET /api/v1/websso
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> [image: cid:image001.png@01D43592.A1449230]
>
>
>
> Gateway.log -> Log in second time without clearing browser cookies
>
>
>
> Second time:
>
>
>
> 2018-08-16 22:34:15,242 DEBUG knox.gateway (GatewayFilter.java:doFilter(119))
> - Received request: GET /yarn/
>
> 2018-08-16 22:34:15,242 DEBUG federation.jwt 
> (SSOCookieFederationFilter.java:doFilter(114))
> - Sending redirect to: https://knox.idf-emr-prd.a..com:8446/gateway/
> knoxsso/api/v1/websso?originalUrl=https://knox.idf-
> emr-prd.a..com:8446/gateway/gate1/yarn/
>
> 2018-08-16 22:34:15,362 DEBUG knox.gateway (GatewayFilter.java:doFilter(119))
> - Received request: GET /api/v1/websso
>
> 2018-08-16 22:34:17,055 DEBUG knox.gateway (GatewayFilter.java:doFilter(119))
> - Received request: POST /api/v1/websso
>
> 2018-08-16 22:34:17,279 DEBUG knox.gateway (GatewayFilter.java:doFilter(119))
> - Received request: GET /api/v1/websso
>
> 2018-08-16 22:34:17,279 DEBUG filter.Pac4jIdentityAdapter
> (Pac4jIdentityAdapter.java:doFilter(85)) - User authenticated as:
> #SAML2Profile# | id: praveen_raviku...@.com | attributes: {Mail=[
> praveen_raviku...@.com], UserID=[pravikumar],
> notOnOrAfter=2018-08-16T22:39:16.775Z, 
> sessionindex=MdgrQer1EH_J2n1l.OX54WupIWl,
> notBefore=2018-08-16T22:29:16.775Z} | roles: [] | permissions: [] |
> isRemembered: false | clientName: SAML2Client | linkedId: null |
>
> 2018-08-16 22:34:17,282 INFO  service.knoxsso
> (WebSSOResource.java:init(113)) - The cookie max age is being set to:
> session.
>
> 2018-08-16 22:34:17,282 WARN  service.knoxsso
> (WebSSOResource.java:init(117)) - The SSO cookie max age configuration is
> invalid: session - using default.
>
> 2018-08-16 22:34:17,282 INFO  service.knoxsso 
> (WebSSOResource.java:getCookieValue(330))
> - Unable to find cookie with name: original-url
>
> 2018-08-16 22:34:17,284 DEBUG service.knoxsso 
> (WebSSOResource.java:addJWTHadoopCookie(288))
> - Adding the following JWT token as a cookie: eyJhbGciOiJSUzI1NiJ9.
> eyJzdWIiOiJQcmF2ZWVuX1JhdmlrdW1hckBpbnR1aXQuY29tIiwiaXNzIjoi
> S05PWFNTTyIsImV4cCI6MTUzNDQ1ODk1N30.CKWaSVrxyW18O-
> SJ7mc8ZieehS8izd90bAdQmTd38y-FkuTXhTMSTDy2_RqERLklgeiPGsE1zQ7TdxRA9wgQ3NB
> YIaiAfBKE1epbmgFZ_MyAlmHvaSafXKF4LI49HNHJoUE1m2r4wRXsS1uysAwSMJ2r_
> TGs9obB4z6uYRC09YU
>
> 2018-08-16 22:34:17,284 INFO  service.knoxsso 
> (WebSSOResource.java:addJWTHadoopCookie(304))
> - JWT cookie successfully added.
>
> 2018-08-16 22:34:17,284 INFO  service.knoxsso 
> (WebSSOResource.java:getAuthenticationToken(214))
> - About to redirect to original URL: https://knox.idf-emr-prd.
> a.intuit.com:8446/gateway/gate1/yarn/
>
> 2018-08-16 22:34:17,399 DEBUG knox.gateway (GatewayFilter.java:doFilter(119))
> - Received request: GET /yarn/
>
> 2018-08-16 22:34:17,399 DEBUG federation.jwt 
> (SSOCookieFederationFilter.java:getJWTFromCookie(161))
> - hadoop-jwt Cookie has been found and is being processed.
>
> 2018-08-16 22:34:17,400 DEBUG knox.gateway 
> (UrlRewriteProcessor.java:rewrite(161))
> - Rewrote URL: https://knox.idf-emr-prd.a.xx.com:8446/gateway/
> gate1/yarn/, direction: IN via implicit rule: 

Re: HDP 3.0 installation problem

[larry mccay]

No, this is a new one for me.
I will try and look into it but it may be a better question for Ambari than
for Knox.

I would also direct you to your HDP support team.


On Tue, Jul 31, 2018 at 1:10 PM, Lian Jiang  wrote:

> I workaround this issue by pre-creating the required symlink
> /usr/hdp/current/knox-server.
>
> On Mon, Jul 30, 2018 at 3:56 PM, Lian Jiang  wrote:
>
>> Hi,
>>
>> I am using ambari 2.7 blueprint to install HDP3.0 and knox failed to
>> install due to:
>>
>> resource_management.core.exceptions.ExecutionFailed: Execution of 
>> 'ambari-python-wrap /usr/bin/hdp-select set knox-server 3.0.0.0-1634' 
>> returned 1. symlink target /usr/hdp/current/knox-server for knox already 
>> exists and it is not a symlink.
>>
>>
>> I checked that /usr/hdp/current/knox-server is a folder and I have to
>> delete it to make knox installation successful.
>>
>> This breaks the automation. Any idea why /usr/hdp/current/knox-server
>> folder is created in HDP3.0? This did not happen in HDP2.6 using the
>> same blueprint. Thanks.
>>
>
>

Re: Knox SSO logout.

[larry mccay]

Sorry, the SET-COOKIE may be in a list of forbidden headers - I would need
to track that down and if so - why it would be forbidden.
Since apps like Ambari, Ranger, etc set application specific session
cookies and they make it through when proxying, that shouldn't be the case.

It may also be related to whether you are setting it from the same
origin/domain - forget the details here.
The browser will not allow the set-cookie to take effect if you aren't the
one that set it.

On Mon, Jul 30, 2018 at 2:03 PM, Christopher Jackson <
jackson.christopher@gmail.com> wrote:

> Hi Larry,
>
> Thanks for the quick response. I saw the work item for the Knox SSO logout
> (KNOX-744) and reviewed that code. I was trying to avoid it for a few
> reasons:
>
> 1) Doesn’t seem it supports jwt cookie name other then the default. I’ve
> opened an issue for this https://issues.apache.org/jira/browse/KNOX-1403
> 2) It requires an additional topology which I’m not sure how I’d create
> and manage on HDP since there isn’t a corresponding topology configuration
> file for said service. Any thoughts on this?
>
> We are targeting the default form based provider and I am okay with a
> logout action logging out of all services (this is what I would normally
> expect from a SSO logout service).
>
> Unfortunately your response really didn't address my original question on
> why my approach was not working. I would like to understand why the
> Set-Cookie header isn’t taking effect, do you have some insight there that
> you can share?
>
> If that approach will not work and I must use the KNOXSSOUT service can
> you share what a sample configuration might look like?
>
> Thanks in advance.
>
> Regards,
> Christopher Jackson
>
> On Jul 30, 2018, at 12:54 PM, larry mccay  wrote:
>
> Hi Christopher -
>
> SSO Logout is an ongoing issue for KnoxSSO and one that isn't easily dealt
> with.
> I'll enumerate some of the nuances and challenges and then point you to a
> Knox API to do what you need - if you still want to do this.
>
> * depending on the IDP being used one or all of the following may be true:
> - The actual IDP user session is still valid (like Okta/SAML) - so
> even if you remove the cookie once you are redirected back to KnoxSSO and
> thusly to the configured IDP you will be logged right back in
> - For the default form based provider there will be no separate
> session - so that will work as you expect
> * even when the above works as you expect, do you really intend that a
> logout from your application will log the user out of Ambari, Ranger,
> Atlas, Zeppelin and any others that may have KnoxSSO enabled?
>
> If you truly want to accept the above and other nuances then you might
> consider the KNOXSSOUT API which will do exactly what you are trying to do
> from your logout action and you can likely call it from javascript.
> You will need to add a new topology to host it since you don't want it
> protected by any particular authenticaiton provider (though it may work
> within the knoxsso.xml topology - would have to think through that).
>
> This isn't even a documented service due to the above issues with logging
> out.
>
> The url to the call would be something like:
> https://{GATEWAY_HOST}:{GATEWAY_PORT}/gateway/knoxssout/api/v1/webssout
>
> Hope this is helpful.
>
> --larry
>
> On Mon, Jul 30, 2018 at 12:03 PM, Christopher Jackson <
> jackson.christopher@gmail.com> wrote:
>
>> Hi all,
>>
>> I have Knox SSO configured for a custom service for the most part
>> everything is working, however I’m having an issue with logging out of my
>> UIs now.
>>
>> I have a logout service in my API which is aware of the knox jwt token. I
>> have a logout rest endpoint which overwrites the token value and setting it
>> to expire immediately, this essentially clears the token. The service works
>> fine on its own when knox is not involved. I see that the SET-COOKIE header
>> is properly updating the cookie contents when the service is called.
>> However, when I access it through the knox gateway the SET-COOKIE header is
>> not ultimately being received by the caller.
>>
>> I investigated the logs and see that the request is received by the
>> gateway and forwarded to the internal service. The response from the
>> internal service is working fine (wire logging shows the SET-COOKIE header
>> updating the cookies values). However when the final response from the
>> gateway to the caller is returned the SET-COOKIE header is no longer
>> present.
>>
>> It seems something is happening in the gateway filters that I’m not aware
>> of. Nothing in the logs stood out to me.
>>
>> Any suggestions on what may be the problem here?
>>
>> Regards,
>>
>> Christopher Jackson
>
>
>
>

[ANNOUNCE] Apache Knox 1.1.0 Release

[larry mccay]

The Apache Knox team is proud to announce the release of Apache Knox
1.1.0!

Apache Knox is a REST API Gateway for providing secure access to the data
and processing resources of Hadoop clusters. More details on Apache Knox
can be found at: http://knox.apache.org/

The release bits are available at:
http://www.apache.org/dyn/closer.cgi/knox/1.1.0

The change list is in the CHANGES file in the downloaded archive.

This release includes over 175 features, bug fixes and improvements and

is actually our largest number of commits in a release to date!

The Admin UI has been enhanced to leverage the much simpler and efficient

descriptors and provider config and topology generation for a better admin

experience.

The automated discovery of cluster changes and topology redeployment is

a much better experience than the use of topology files alone can provide.

While direct authoring topologies is still supported, this should be
considered

deprecated at this time.

We would like to thank all of contributors who made the release possible and

to invite others interested in helping out to engage the community on the
dev

and users lists!

Re: [VOTE] Release Apache Knox 1.1.0 RC 3

[larry mccay]

The VOTE for 1.1.0 rc3 passes with:

3 binding +1's
0 -1's

I will be working on promoting RC3 to an official release shortly.

Thank you for taking the time to test this release and contributing to the
Apache Knox community!


On Thu, Jul 26, 2018 at 12:51 PM, Sandeep Moré 
wrote:

>
> +1
>
> * Downloaded and built from source
> * Checked LICENSE and NOTICE files
> * Verified GPG/MD5/SHA signatures for Knox source, Knox and Knoxshell
> release packages (zip files)
> * Installed pseudo-distributed instance (Mac OS X )
> * Ran through knox tests
> * Checked websocket functionality
> * Checked Topology Port Mapping feature
> * Checked KnoxShell samples
> * Tested HDFSUI (recent changes)
>
> Best,
> Sandeep
>
>
> On Wed, Jul 25, 2018 at 7:28 PM larry mccay  wrote:
>
>> All -
>>
>> An issue with the OOTB configuration was found and subsequently fixed
>> based
>> on testing of RC 2. This is a minimal incremental change over the previous
>> RC.
>>
>> Release candidate #3 for the Apache Knox 1.1.0 is available at:
>>
>> https://dist.apache.org/repos/dist/dev/knox/knox-1.1.0/
>>
>> The release candidate is a zip archive of the sources in:
>>
>> https://git-wip-us.apache.org/repos/asf/knox.git
>> Branch v1.1.0 (git checkout -b v1.1.0)
>> Tag is v1.1.0-rc3
>>
>> The KEYS file for signature validation is available at:
>> https://dist.apache.org/repos/dist/release/knox/KEYS
>>
>> Please vote on releasing this package as Apache Knox 1.1.0.
>> The vote is open for the next 72 hours and passes if a majority of at
>> least three +1 Apache Knox PMC votes are cast.
>>
>> [ ] +1 Release this package as Apache Knox 1.1.0
>> [ ] -1 Do not release this package because...
>>
>> Thanks,
>>
>> --larry
>>
>

Re: IPv6 support

[larry mccay]

Well, it seems that you can certainly specify IPv6 using curl for a call to
webhdfs as the following output shows:

new-host-6:knox-1.1.0 lmccay$ curl -ivk6u guest:guest-password
https://localhost:8443/gateway/sandbox/webhdfs/v1/tmp?op=LISTSTATUS
*   *Trying ::1*...
* TCP_NODELAY set
** Connected to localhost (::1) port 8443 (#0)*
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection:
ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=Test; L=Test; O=Hadoop; OU=Test; CN=localhost
*  start date: Jul 26 02:14:01 2018 GMT
*  expire date: Jul 26 02:14:01 2019 GMT
*  issuer: C=US; ST=Test; L=Test; O=Hadoop; OU=Test; CN=localhost
*  SSL certificate verify result: self signed certificate (18), continuing
anyway.
* Server auth using Basic with user 'guest'
> GET /gateway/sandbox/webhdfs/v1/tmp?op=LISTSTATUS HTTP/1.1
> Host: localhost:8443
> Authorization: Basic Z3Vlc3Q6Z3Vlc3QtcGFzc3dvcmQ=
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Fri, 27 Jul 2018 15:30:19 GMT
Date: Fri, 27 Jul 2018 15:30:19 GMT
< Set-Cookie:
JSESSIONID=k4qsr435jctv1sx4bf6r7yo0k;Path=/gateway/sandbox;Secure;HttpOnly
Set-Cookie:
JSESSIONID=k4qsr435jctv1sx4bf6r7yo0k;Path=/gateway/sandbox;Secure;HttpOnly
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Set-Cookie: rememberMe=deleteMe; Path=/gateway/sandbox; Max-Age=0;
Expires=Thu, 26-Jul-2018 15:30:19 GMT
Set-Cookie: rememberMe=deleteMe; Path=/gateway/sandbox; Max-Age=0;
Expires=Thu, 26-Jul-2018 15:30:19 GMT
< Server: nginx/1.15.0
Server: nginx/1.15.0
< Date: Fri, 27 Jul 2018 15:30:20 GMT
Date: Fri, 27 Jul 2018 15:30:20 GMT
< Content-Type: application/json; charset=UTF-8
Content-Type: application/json; charset=UTF-8
< Cache-Control: no-cache
Cache-Control: no-cache
< Expires: Fri, 27 Jul 2018 15:30:19 GMT
Expires: Fri, 27 Jul 2018 15:30:19 GMT
< Pragma: no-cache
Pragma: no-cache
< Expires: Fri, 27 Jul 2018 15:30:19 GMT
Expires: Fri, 27 Jul 2018 15:30:19 GMT
< Pragma: no-cache
Pragma: no-cache
< X-FRAME-OPTIONS: SAMEORIGIN
X-FRAME-OPTIONS: SAMEORIGIN
< Content-Length: 1003
Content-Length: 1003

<
* Connection #0 to host localhost left intact
{"FileStatuses":{"FileStatus":[{"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":17212,"group":"hadoop","length":0,"modificationTime":1529337971112,"owner":"druid","pathSuffix":"druid-indexing","permission":"775","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16393,"group":"hdfs","length":0,"modificationTime":1529333529967,"owner":"hdfs","pathSuffix":"entity-file-history","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":0,"blockSize":0,"childrenNum":2,"fileId":16537,"group":"hdfs","length":0,"modificationTime":1529334778446,"owner":"ambari-qa","pathSuffix":"hive","permission":"733","replication":0,"storagePolicy":0,"type":"DIRECTORY"},{"accessTime":1532571397860,"blockSize":134217728,"childrenNum":0,"fileId":144616,"group":"hdfs","length":96969,"modificationTime":1532571397885,"owner":"sam","pathSuffix":"sample.log","permission":"755","replication":1,"storagePolicy":0,"type":"FILE"}]}}


Hope that is helpful.

On Fri, Jul 27, 2018 at 10:24 AM, Rajat Goel  wrote:

> Hello,
>
>
>
> Can Knox work on IPv6 interface? Specifically, I want to have Client/User
> to Knox communication on IPv6 network but my backend services say
> Ambari/WebHDFS runs on IPv4 network so I want Knox to Service communication
> on IPv4 network. Will this work with Knox ?
>
>
>
> Thanks & Regards,
>
> Rajat
>

Re: Knox connection to Active Directory Federation Services?

[larry mccay]

I don't recall seeing anyone using ADFS yet.
This would certainly be of interest and if you get it to work - it would be
great to get a wiki tutorial for doing so!

I have seen deployments with CAC cards where the challenge is done via
proxy like WebGate or something like that and then Header based Preauth is
used in KnoxSSO to establish the session via HTTP Headers.
This would require a combination of network controls that require access
can only be gained through the authenticating proxy and some form of trust
relationship between Knox and the proxy like mutual authentication.

On Thu, Jul 26, 2018 at 5:45 PM, Rutland, Nathan A CTR DISA BD (US) <
nathan.a.rutland2@mail.mil> wrote:

> I have been looking all over for an example of connecting to Active
> Directory Federation Services.  Has anyone successfully implemented this
> with SAML 2.0?
>
> Our ADFS has a Common Access Card PKI tie-in, so I'm trying to leverage it
> to have Knox pointed to ADFS and Ambari set up as SSO to Knox.  Ultimately,
> when someone tries to connect to Ambari, it will request the PIN from the
> CAC certificate, do an ADFS lookup in Active Directory, and send back a
> SAML token of success or failure.  Possibly some other pieces of
> information to help tie that AD username into the Hadoop cluster.
>
> VR, N
>
>
>

Re: [VOTE] Release Apache Knox 1.1.0 RC 3

[larry mccay]

* Verified signatures
* Built from source
* Checked LICENSE, NOTICE, ISSUES and CHANGES files
* Tested Admin UI
* Tested service discovery and topology generation
* Tested dynamic update of Topologies list in Admin UI
* Tested KnoxSSO with Admin UI
* Tested KnoxToken service
* Tested Knox Token Sessions with KnoxShell and JWTProvider
* Tested Hive by following the KnoxLine SQL Client wiki article
* Tested WebHDFS via simple nodejs client
* Tested YARN resource manager API
* Tested Elastic Search
* Tested Okta integration via SAML v2 and pac4j provider
* Tested AmbariUI proxying
* Tested JOBHISTORYUI with failed whitelist match

Here is my +1

On Thu, Jul 26, 2018 at 11:15 AM, Phil Zampino  wrote:

> Thanks for the quick turnaround on these RCs, Larry!
>
> +1
>
> * Verified GPG/MD5/SHA signatures for Knox source, Knox and KnoxShell release
> packages (zip files)
> * Downloaded and built from source (with Java 1.8.0_144)
> * Checked LICENSE and NOTICE files
> * Checked ISSUES file
> * Checked CHANGES file
> * Ran KnoxShell samples
> * Exercised the Admin UI (provider configuration wizard, descriptor
> wizard, etc...)
> * Verified the Admin UI issues discovered in RC1 (URL field validation,
> topology list refreshing) have been addressed
> * Tested service discovery and topology generation
> * Verified the resolution of the gateway-site.xml duplicate property issue
> discovered in RC2
>
>  -- Phil
>
>
> On Wed, Jul 25, 2018 at 7:28 PM larry mccay  wrote:
>
>> All -
>>
>> An issue with the OOTB configuration was found and subsequently fixed
>> based
>> on testing of RC 2. This is a minimal incremental change over the previous
>> RC.
>>
>> Release candidate #3 for the Apache Knox 1.1.0 is available at:
>>
>> https://dist.apache.org/repos/dist/dev/knox/knox-1.1.0/
>>
>> The release candidate is a zip archive of the sources in:
>>
>> https://git-wip-us.apache.org/repos/asf/knox.git
>> Branch v1.1.0 (git checkout -b v1.1.0)
>> Tag is v1.1.0-rc3
>>
>> The KEYS file for signature validation is available at:
>> https://dist.apache.org/repos/dist/release/knox/KEYS
>>
>> Please vote on releasing this package as Apache Knox 1.1.0.
>> The vote is open for the next 72 hours and passes if a majority of at
>> least three +1 Apache Knox PMC votes are cast.
>>
>> [ ] +1 Release this package as Apache Knox 1.1.0
>> [ ] -1 Do not release this package because...
>>
>> Thanks,
>>
>> --larry
>>
>

[VOTE] Release Apache Knox 1.1.0 RC 3

[larry mccay]

All -

An issue with the OOTB configuration was found and subsequently fixed based
on testing of RC 2. This is a minimal incremental change over the previous
RC.

Release candidate #3 for the Apache Knox 1.1.0 is available at:

https://dist.apache.org/repos/dist/dev/knox/knox-1.1.0/

The release candidate is a zip archive of the sources in:

https://git-wip-us.apache.org/repos/asf/knox.git
Branch v1.1.0 (git checkout -b v1.1.0)
Tag is v1.1.0-rc3

The KEYS file for signature validation is available at:
https://dist.apache.org/repos/dist/release/knox/KEYS

Please vote on releasing this package as Apache Knox 1.1.0.
The vote is open for the next 72 hours and passes if a majority of at
least three +1 Apache Knox PMC votes are cast.

[ ] +1 Release this package as Apache Knox 1.1.0
[ ] -1 Do not release this package because...

Thanks,

--larry

Re: Integrate ranger-knox plugin.

[larry mccay]

This generally means that your configured truststore cannot be opened or is
empty and is plain vanilla SSL related issue not related to Knox or Ranger
specifically.


On Wed, Jul 25, 2018 at 8:17 AM, Dhruv Goyal <777.dh...@gmail.com> wrote:

> Hello,
>
> We are trying to enable ranger-knox plugin to provide authorisation from
> Ranger. I had few queries:
>
> Do we need to have SSL enabled in Ranger to be integrated with Knox?
>
> What are the steps which will be required, what all keystore/truststore we
> will need to create, is there any brief documentation for the same?
>
> We are planning to follow this document: https://docs.
> hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_security/
> content/ch04s20s02s04s01.html
> Is this the right doc to proceed?
>
> We have followed these commands:
> cd /usr/hdp/2.6.5.0-292/knox/data/security/keystores
>
> keytool -exportcert -alias gateway-identity -keystore gateway.jks -file
> ../knox.crt
> pass- "Enter Key"
>
> cd ../
>
> cp /usr/java/latest/jre/lib/security/cacerts cacerts.withknox
>
> keytool -import  -trustcacerts -file knox.crt -alias knox -keystore
> cacerts.withknox
> Pass- changeit
>
>  keytool -import  -trustcacerts -file knox.crt -alias knox -keystore
> cacerts.withknox
>
> But we are getting these errors when we are making the repo and clicking
> on test connection:
>
> org.apache.ranger.plugin.client.HadoopException: Exception on REST call
> to KnoxUrl : https://192.168.134.119:8443/gateway/admin/api/v1/topologies
> ..
> Exception on REST call to KnoxUrl : https://192.168.1.1:8443/
> gateway/admin/api/v1/topologies..
> javax.net.ssl.SSLException: java.lang.RuntimeException: Unexpected error:
> java.security.InvalidAlgorithmParameterException: the trustAnchors
> parameter must be non-empty.
> java.lang.RuntimeException: Unexpected error: java.security.
> InvalidAlgorithmParameterException: the trustAnchors parameter must be
> non-empty.
> Unexpected error: java.security.InvalidAlgorithmParameterException: the
> trustAnchors parameter must be non-empty.
> the trustAnchors parameter must be non-empty.
>
>
> Thanks
> Dhruv
>

[VOTE] Release Apache Knox 1.1.0 RC 2

[larry mccay]

All -

A number of issues were identified and subsequently fixed based
on testing of RC 1.

Release candidate #2 for the Apache Knox 1.1.0 is available at:

https://dist.apache.org/repos/dist/dev/knox/knox-1.1.0/

The release candidate is a zip archive of the sources in:

https://git-wip-us.apache.org/repos/asf/knox.git
Branch v1.1.0 (git checkout -b v1.1.0)
Tag is v1.1.0-rc2

The KEYS file for signature validation is available at:
https://dist.apache.org/repos/dist/release/knox/KEYS

Please vote on releasing this package as Apache Knox 1.1.0.
The vote is open for the next 72 hours and passes if a majority of at
least three +1 Apache Knox PMC votes are cast.

[ ] +1 Release this package as Apache Knox 1.1.0
[ ] -1 Do not release this package because...

Thanks,

--larry

[CANCEL] [VOTE] Release Apache Knox 1.1.0 RC 1

[larry mccay]

-1

I've seen the same issues as Phil and believe that the whitelisting of the
cited APIs are a release blocker.
In addition to the input validation in the UI, I have also noted that the
Topologies list doesn't dynamically update on cloning or deleting of a
topology.
Also, Discovery for WebHBase seems to be picking up the UI host:port rather
than the API - this could perhaps be a KNOWN ISSUE with discovery and
should be noted in the ISSUES file with details on how to work around it
with explicit URLs being set in the descriptor.

I'll file a JIRA for the topologies issue now.

I have successfully tested:

* Discovery and topology generation
* KnoxSSO form based IDP for the Admin UI
* HiveServer2 via KnoxShell and JDBC
* WebHDFS via curl and simple nodejs client
* WebHBase - discovery is wrong as noted above
* KnoxToken service with knox init and knoxshell for access to WebHDFS

I am cancelling the VOTE on this RC due to the issues mentioned above and
by Phil.


On Tue, Jul 24, 2018 at 11:32 AM, Phil Zampino  wrote:

> -1
>
> After some testing, there are some issues which merit another RC, IMO:
> - Some services are unnecessarily held to the dispatch whitelist
> validation (e.g., WEBHDFS, WEBHBASE)
> - The Admin UI URL input field validation requires a port, which it should
> not.
>
> While there are work-arounds for both, they adversely affect the user
> experience.
>
>
> On Sat, Jul 21, 2018 at 3:58 PM larry mccay  wrote:
>
>> All -
>>
>> Release candidate #1 for the Apache Knox 1.1.0 is available at:
>>
>> https://dist.apache.org/repos/dist/dev/knox/knox-1.1.0/
>>
>> The release candidate is a zip archive of the sources in:
>>
>> https://git-wip-us.apache.org/repos/asf/knox.git
>> Branch v1.1.0 (git checkout -b v1.1.0)
>> Tag is v1.1.0-rc1
>>
>> The KEYS file for signature validation is available at:
>> https://dist.apache.org/repos/dist/release/knox/KEYS
>>
>> Please vote on releasing this package as Apache Knox 1.1.0.
>> The vote is open for the next 72 hours and passes if a majority of at
>> least three +1 Apache Knox PMC votes are cast.
>>
>> [ ] +1 Release this package as Apache Knox 1.1.0
>> [ ] -1 Do not release this package because...
>>
>> Thanks,
>>
>> --larry
>>
>

[VOTE] Release Apache Knox 1.1.0 RC 1

[larry mccay]

All -

Release candidate #1 for the Apache Knox 1.1.0 is available at:

https://dist.apache.org/repos/dist/dev/knox/knox-1.1.0/

The release candidate is a zip archive of the sources in:

https://git-wip-us.apache.org/repos/asf/knox.git
Branch v1.1.0 (git checkout -b v1.1.0)
Tag is v1.1.0-rc1

The KEYS file for signature validation is available at:
https://dist.apache.org/repos/dist/release/knox/KEYS

Please vote on releasing this package as Apache Knox 1.1.0.
The vote is open for the next 72 hours and passes if a majority of at
least three +1 Apache Knox PMC votes are cast.

[ ] +1 Release this package as Apache Knox 1.1.0
[ ] -1 Do not release this package because...

Thanks,

--larry

Re: Need help in enabling SAML auth in Apache Knox

[larry mccay]

> 2018-07-18 01:41:06,111 DEBUG session.KnoxSessionStore
> (KnoxSessionStore.java:get(101)) - Get from session: pac4jUserProfiles =
> null
>
> 2018-07-18 01:41:06,111 DEBUG session.KnoxSessionStore
> (KnoxSessionStore.java:set(130)) - Save in session: pac4jRequestedUrl =
> https://:8446/gateway/knoxsso/api/v1/websso?originalUrl=https://
> :8446/gateway/gate1/yarn/
>
> 2018-07-18 01:41:06,345 DEBUG session.KnoxSessionStore
> (KnoxSessionStore.java:get(101)) - Get from session: 
> SAML2Client$attemptedAuthentication
> = null
>
> 2018-07-18 01:41:06,346 DEBUG session.KnoxSessionStore
> (KnoxSessionStore.java:get(101)) - *Get from session: samlRelayState =
> null*
>
> 2018-07-18 01:41:06,346 DEBUG session.KnoxSessionStore
> (KnoxSessionStore.java:set(130)) - *Save in session: samlRelayState =*
>
> 2018-07-18 01:41:08,679 DEBUG knox.gateway (GatewayFilter.java:doFilter(119))
> - Received request: POST /api/v1/websso
>
> 2018-07-18 01:41:08,748 DEBUG session.KnoxSessionStore
> (KnoxSessionStore.java:set(130)) - Save in session: 
> SAML2Client$attemptedAuthentication
> =
>
> 2018-07-18 01:41:08,749 DEBUG session.KnoxSessionStore
> (KnoxSessionStore.java:set(130)) - Save in session: pac4jUserProfiles =
> {SAML2Client=#SAML2Profile# | id: praveen_raviku...@int.com | attributes:
> {Mail=[praveen_raviku...@int.com], notOnOrAfter=2018-07-18T01:46:08.427Z,
> sessionindex=X3R7kH9zunD1jn9CP5eN9sm2N0M, notBefore=2018-07-18T01:36:08.427Z}
> | roles: [] | permissions: [] | isRemembered: false | clientName:
> SAML2Client | linkedId: null |}
>
> *2018-07-18 01:41:08,978 ERROR engine.DefaultCallbackLogic
> (DefaultCallbackLogic.java:renewSession(123)) - Unable to renew the
> session. The session store may not support this feature*
>
> 2018-07-18 01:41:08,979 DEBUG session.KnoxSessionStore
> (KnoxSessionStore.java:get(101)) - Get from session: pac4jRequestedUrl =
> null
>
>
>
> 
>
> << *gateway-audit.log* >>
>
> 
>
> 18/07/18 01:41:06 ||be6bf57b-7b96-4292-93ce-00ed574ecd6e|audit|10.89.78.
> 49|YARNUIaccess|uri|/gateway/gate1/yarn/|unavailable|Request method:
> GET
>
> 18/07/18 01:41:06 |||audit|10.89.78.49|YARNUI|||
> |access|uri|/gateway/gate1/yarn/|success|Response status: 302
>
> 18/07/18 01:41:06 ||668bd6c6-664f-499c-97e5-6c2294b98f04|audit|10.89.78.
> 49|KNOXSSOaccess|uri|/gateway/knoxsso/api/v1/websso?
> originalUrl=https://:8446/gateway/gate1/yarn/|unavailable|Request
> method: GET
>
> 18/07/18 01:41:06 |||audit|10.89.78.49|KNOXSSO||
> ||access|uri|/gateway/knoxsso/api/v1/websso?originalUrl=https://
> :8446/gateway/gate1/yarn/|success|Response status: 200
>
> 18/07/18 01:41:08 ||fdbcaedb-75a6-46a5-8a7d-811f194142d1|audit|10.89.78.
> 49|KNOXSSOaccess|uri|/gateway/knoxsso/api/v1/websso?
> pac4jCallback=true_name=SAML2Client|unavailable|Request method:
> POST
>
> 18/07/18 01:41:08 |||audit|10.89.78.49|KNOXSSO||
> ||access|uri|/gateway/knoxsso/api/v1/websso?pac4jCallback=
> true_name=SAML2Client|success|Response status: 302
>
>
>
> Thanks,
>
> Praveen.R
>
>
>
> *From: *larry mccay 
> *Reply-To: *"user@knox.apache.org" 
> *Date: *Tuesday, July 17, 2018 at 5:56 PM
> *To: *"user@knox.apache.org" 
> *Cc: *"prasada_pra...@intuit.com" , "Mohanan,
> Mahesh" 
> *Subject: *Re: Need help in enabling SAML auth in Apache Knox
>
>
>
> Whitelist - this has nothing to do with determining where to redirect - it
> may not allow you to redirect somewhere if it doesn't match the expression
> but it is not used to determine where to redirect to.
>
>
>
> Not sure why the URL would have to be rewritten when proxying.
>
>
>
> * try to access YARNUI through Knox
>
> * SSOCookie finds no cookie so redirects to KnoxSSO and provides
> originalUrl query param which points to the YARNUI through Knox
>
> * KnoxSSO engages SAML IDP and user logs in
>
> * SAML assertion is POSTED to KnoxSSO endpoint with proper callback URL
>
> * KnoxSSO creates hadoop-jwt cookie and redirects to originalUrl query
> param
>
>
>
> I can't tell if you are saying that the POS to KnoxSSO is going to the
> wrong place or the originalUrl redirect is.
>
>
>
> On Tue, Jul 17, 2018 at 3:31 PM, Ravikumar, Praveen Krishnamoorthy <
> rpkr...@amazon.com> wrote:
>
> Thanks a lot for your inputs Sandeep. I genuinely appreciate for your
> thoughts.
>
>
>
> As you mentioned I followed the below docs and enabled SAML. As of now IDP
> related issues are fixed,
>
> · Knox is now able to contact the IDP
>
> · POST t

Re: Need help in enabling SAML auth in Apache Knox

[larry mccay]

Whitelist - this has nothing to do with determining where to redirect - it
may not allow you to redirect somewhere if it doesn't match the expression
but it is not used to determine where to redirect to.

Not sure why the URL would have to be rewritten when proxying.

* try to access YARNUI through Knox
* SSOCookie finds no cookie so redirects to KnoxSSO and provides
originalUrl query param which points to the YARNUI through Knox
* KnoxSSO engages SAML IDP and user logs in
* SAML assertion is POSTED to KnoxSSO endpoint with proper callback URL
* KnoxSSO creates hadoop-jwt cookie and redirects to originalUrl query param

I can't tell if you are saying that the POS to KnoxSSO is going to the
wrong place or the originalUrl redirect is.

On Tue, Jul 17, 2018 at 3:31 PM, Ravikumar, Praveen Krishnamoorthy <
rpkr...@amazon.com> wrote:

> Thanks a lot for your inputs Sandeep. I genuinely appreciate for your
> thoughts.
>
>
>
> As you mentioned I followed the below docs and enabled SAML. As of now IDP
> related issues are fixed,
>
>- Knox is now able to contact the IDP
>- POST the SAML request to IDP.
>- IDP responds with the window allowing user to provide the
>credentials.
>- Users are getting authenticated.
>- I could see the SAML response in the SAML tracer.
>-
>
> But after authentication the page is getting redirected to
> http://:8446  and pops message as 404 not found. I suspect, this
> is an issue on the knox side.
>
>
>
> I have 2 thoughts, either knoxsso.redirect.whitelist.regex param
> highlighted below in knoxsso.xml topology is redirecting to invalid URL or
> do we need to add any rewrite rules in the application to direct to a valid
> URL.
>
>
>
> Could anyone please help me in this.
>
> KnoxSSO.xml
>
> 
>
> 
>
>
>
>  
>
>  federation
>
>  pac4j
>
>  true
>
>  
>
>   pac4j.callbackUrl
>
>   https://:8446/gateway/knoxsso/api/v1/websso<
> /value>
>
>  
>
>  
>
>clientName
>
>SAML2Client
>
>  
>
>  
>
>saml.identityProviderMetadataPath
>
>/tmp/preprod_metadata_SP.xml
>
>  
>
>  
>
>saml.serviceProviderMetadataPath
>
>/tmp/preprod_metadata_SP.xml
>
>  
>
>  
>
>saml.serviceProviderEntityId
>
>https://:8446/gateway/knoxsso/api/v1/websso?
> pac4jCallback=trueclient_name=SAML2Client
>
>  
>
>  
>
>  
>
>  identity-assertion
>
>  Default
>
>  true
>
>  
>
>
>
>
>
>KNOXSSO
>
>
>
>  knoxsso.cookie.secure.only
>
>  true
>
>   
>
>   
>
> knoxsso.token.ttl
>
> 10
>
>   
>
>   **
>
> * knoxsso.redirect.whitelist.regex*
>
> *
> ^https?:\/\/(|localhost|127\.0\.0\.1|0:0:0:0:0:0:0:1|::1):[0-9].*$*
>
> *  *
>
>
>
> 
>
>
>
> Thanks,
>
> Praveen.
>
>
>
> *From: *Sandeep Moré 
> *Reply-To: *"user@knox.apache.org" 
> *Date: *Monday, July 16, 2018 at 6:25 PM
> *To: *"user@knox.apache.org" 
> *Subject: *Re: Need help in enabling SAML auth in Apache Knox
>
>
>
> Hello Praveen,
>
>
>
> Following are some of the blogs that talk about Knox SAML integration
>
>
>
> https://cwiki.apache.org/confluence/display/KNOX/KnoxSSO+and+Okta
>
> https://cwiki.apache.org/confluence/display/KNOX/
> Configuring+Apache+Knox+SSO+with+Ipsilon+using+SAML2
>
>
>
> Knox user guide also has some information on how to configure SAML
>
> https://knox.apache.org/books/knox-1-0-0/user-guide.html#
> Pac4j+Provider+-+CAS+/+OAuth+/+SAML+/+OpenID+Connect
>
>
>
> For more details you can take a look at Pac4J provider
>
> https://github.com/pac4j/pac4j/wiki/Clients#saml-support
>
>
>
> Now coming back to your options, I could not find info on Ping IDP for
> SAML so I cannot comment on the options, this is just my educated guess.
>
>
>
> 1.   Entity ID - I am guessing this is your
> "saml.serviceProviderEntityId" value, for me it is https://www.myhost.com:
> 8443/gateway/knoxsso/api/v1/websso?pac4jCallback=true
> client_name=SAML2Client
>
> 2.   Baseurl - no idea what this is, you probably have to refer to
> ping IDP docs
>
> 3.   ACSURL (where target assertion is Sent) - this is the same as
> above for https://www.myhost.com:8443/gateway/knoxsso/api/v1/websso?
> pac4jCallback=true_name=SAML2Client
>
>
>
>
>
>
>
>
>
> On Mon, Jul 16, 2018 at 5:47 PM Ravikumar, Praveen Krishnamoorthy <
> rpkr...@amazon.com> wrote:
>
> Hi,
>
>
>
> Also, Could anyone please help me what should I configure for
>
>1. Entity ID
>2. BaseURL (that is used for the entire site)
>3. ACSURL (where target assertion is Sent)
>
>
>
> On the Identity provider side, to enable SSO in Knox.
>
>
>
> Thanks,
>
> Praveen.
>
>
>
> *From: *"Ravikumar, Praveen Krishnamoorthy" 
> *Date: *Monday, July 16, 2018 at 10:36 AM
> *To: *"user@knox.apache.org" 
> *Subject: *Need help in enabling SAML 

Re: [DISCUSS] Planning for Apache Knox 1.1.0 Release

[larry mccay]

Agreed, Phil.
I have cut an RC but we need to address this first.  I'll hold off on
announcing it.

On Fri, Jul 13, 2018, 11:36 AM Phil Zampino  wrote:

> During some testing of the proposed 1.1.0 code, I've discovered some NPEs
> in filters (e.g., AclsAuthorizationFilter, HadoopGroupProviderFilter),
> which are concerning.
>
> I've committed a change to address the AclsAuthorizationFilter, but seeing
> similar behavior for the HadoopGroupProviderFilter has increased my concern
> that there may be a more fundamental problem.
> In both cases, it seems that the filters are being invoked prior to (or
> during) their respective init() methods have been invoked. Thus, members
> which should be initialized in the init() method are not yet initialized.
>
> This can be consistently reproduced, though it is a bit of a pain:
>
>- Install Knox (‘ant install-test-home’, or just unzip knox-1.1.0.zip)
>- Start the gateway
>- Access the Admin UI
>
>
> Note that the latest 1.1.0 source has a *fix* for the
> AclsAuthorizationFilter NPE, but master does not yet have this change. This
> is important because that change effectively hides the issue.
>
> I think we should determine what's happening with this before
> producing/testing a release candidate.
>
>
>
>
> On Sat, Feb 24, 2018 at 12:57 PM larry mccay  wrote:
>
> > All -
> >
> > Sorry for the delay on this topic.
> >
> > We are going to start of this planning thread with ~85 Unresolved JIRAs
> in
> > either 1.1.0 or 0.15.0 fixVersion.
> >
> > project = KNOX AND resolution = Unresolved AND fixVersion in (1.1.0,
> > 0.15.0) ORDER BY  priority DESC, updated DESC
> >
> > I will spend some time migrating all 0.15.0 to 1.1.0 to begin with and
> then
> > we will need to go through and see what is already taken care of or can
> > wait for a 1.2.0 or later.
> >
> > I also have a couple KIPs in mind to target larger features/themes for
> this
> > release.
> >
> > Off the top of my head:
> >
> > * I think we need to address some cloud specific usecases and plan to
> > provide a KIP for that. Hybrid cloud/federated knox instances, Azure AD
> > integration, ID mapping from Hadoop user to IAM users/roles, etc. Perhaps
> > some CASB-like features if they make sense.
> >
> > * I also think we need one for articulating a reasonable flow for Logout
> in
> > KnoxSSO. There are a lot of little nuances to logout across multiple apps
> > and between different IDPs. This will require some discussion.
> >
> > * Another thing that has been tugging at my interest has been the fact
> that
> > we may be able provide some common libraries to help ecosystem
> applications
> > uptake the trusted proxy pattern and KnoxSSO.
> >
> > Anyway, these are my initial thoughts, please feel free to raise
> additional
> > ideas/themes for KIPs, etc.
> >
> > I was thinking that we could try and target an end of March or Mid April
> > 1.1.0 release.
> >
> > Thoughts?
> >
> > --larry
> >
>

Re: KnoxSSO JWT authentication by third party.

[larry mccay]

Sure thing!
Seems it is time to upgrade. :)


On Fri, Jul 13, 2018, 6:52 PM Christopher Jackson <
jackson.christopher@gmail.com> wrote:

> I tested on HDP 2.6.4 and it works as expected, log file below shows that
> the file gets updated, and I verified on disk my changed content was there:
>
> 2018-07-13 15:46:36,199 - Generating config: 
> /usr/hdp/current/knox-server/conf/gateway-site.xml
> 2018-07-13 15:46:36,199 - 
> File['/usr/hdp/current/knox-server/conf/gateway-site.xml'] {'owner': 'knox', 
> 'content': InlineTemplate(...), 'group': 'knox', 'mode': None, 'encoding': 
> 'UTF-8'}
> 2018-07-13 15:46:36,208 - 
> File['/usr/hdp/current/knox-server/conf/gateway-log4j.properties'] 
> {'content': InlineTemplate(...), 'owner': 'knox', 'group': 'knox', 'mode': 
> 0644}
> 2018-07-13 15:46:36,217 - 
> File['/usr/hdp/current/knox-server/conf/topologies/default.xml'] {'content': 
> InlineTemplate(...), 'owner': 'knox', 'group': 'knox'}
> 2018-07-13 15:46:36,219 - 
> File['/usr/hdp/current/knox-server/conf/topologies/admin.xml'] {'content': 
> InlineTemplate(...), 'owner': 'knox', 'group': 'knox'}
> 2018-07-13 15:46:36,226 - 
> File['/usr/hdp/current/knox-server/conf/topologies/knoxsso.xml'] {'content': 
> InlineTemplate(...), 'owner': 'knox', 'group': 'knox'}
> 2018-07-13 15:46:36,227 - Writing 
> File['/usr/hdp/current/knox-server/conf/topologies/knoxsso.xml'] because 
> contents don't match
> 2018-07-13 15:46:36,227 - 
> Execute['/usr/hdp/current/knox-server/bin/knoxcli.sh create-master --master 
> [PROTECTED]'] {'environment': {'JAVA_HOME': u'/usr/jdk64/jdk1.8.0_112'}, 
> 'not_if': "ambari-sudo.sh su knox -l -s /bin/bash -c 'test -f 
> /usr/hdp/current/knox-server/data/security/master'", 'user': 'knox'}
>
>
> It appears to only affect my HDP 2.6.2 instance which I’ve tested a few
> and they all exhibit the same behavior. I checked the files on those
> instances and there are no permissions issues. The cluster was not upgraded
> from a previous version. Again I think it is an issue with that condition,
> although like you, I don’t know exactly what that condition check does as
> I’m not familiar with those built in ambari functions.
>
> I think I am just going to document this as a known issue for this
> particular version and move on as it seems to behave correctly on
> subsequent HDP releases. Thanks for you help in this matter.
>
> Regards,
> Christopher Jackson
>
> On Jul 12, 2018, at 11:43 PM, larry mccay  wrote:
>
> Hi -
>
> I just verified that it works as expected with the latest HDP sandbox for
> 2.6.5.
>
> I started the sandbox, changed the TTL param in the KnoxSSO service in
> knoxsso.xml from 3 to -1, saved and restarted.
> I think used the Knox Admin UI to view the topologies that are deployed to
> the instance and it had the -1 value for the TTL param.
>
> While there may be a difference between the versions, I have not
> encountered this issue before at all.
>
> Is it possible that there is a permissions issue on the files?
> Was this cluster upgraded from a previous version?
>
> At any rate, I think that there is an issue with the cluster rather that
> with the knox.py script though I'm not really sure what that condition is
> even checking.
>
> This doesn't really solve anything for you but hope it is helpful in some
> way.
>
> thanks,
>
> --larry
>
>
> On Thu, Jul 12, 2018 at 8:04 PM, larry mccay  wrote:
>
>> Hi Christopher -
>>
>> I apologize, this dropped off my radar.
>> I need to go back and try and reproduce it still.
>>
>> It would still be surprising to me but I am surprised often. :)
>>
>> thanks,
>>
>> --larry
>>
>> On Thu, Jul 12, 2018 at 5:29 PM, Christopher Jackson <
>> jackson.christopher@gmail.com> wrote:
>>
>>> Hey Larry,
>>>
>>> I looked into this a bit deeper. It appears the knoxsso-topology is NOT
>>> updated because of the following code in
>>> /var/lib/ambari-server/resources/common-services/KNOX/0.5.0.
>>> <http://0.5.0.0/>2.2/package/scripts/knox.py:
>>>
>>>   if params.version_formatted and
>>> check_stack_feature(StackFeature.KNOX_SSO_TOPOLOGY,
>>> params.version_formatted):
>>>   File(os.path.join(params.knox_conf_dir, "topologies",
>>> "knoxsso.xml"),
>>>  group=params.knox_group,
>>>  owner=params.knox_user,
>>>  content=InlineTemplate(params.knoxsso_topology_template)
>>>   )
>>>
>>> I believe the if condition is evaluating to false and thus preventing
>>> the knoxsso.xml from being written a

Re: Rewrite rules necessary for knox sso?

[larry mccay]

Again, Christopher - I verified with HDP sandbox which is 2.6.5 and am
seeing expected behavior.
SSOCookieProvider is redirecting to KnoxSSO with an originalUrl query param
that reflects the gateway URL for webhdfs.

I don't know why you are seeing the behavior that you are describing.

I am glad that I'm not seeing the issues that you are describing but it is
a bit concerning to me that you are.
Can you reproduce this in other clusters as well?

On Thu, Jul 12, 2018 at 10:57 PM, larry mccay  wrote:

> Hi Christopher -
>
> Proxying and SSO together may require a rewrite rule.
> I'll look at what SSO Cookie Provider is doing though.
> I would expect originalUrl to be the gateway url from there.
>
> Thanks,
>
> --larry
>
> On Thu, Jul 12, 2018, 4:17 PM Christopher Jackson <
> jackson.christopher@gmail.com> wrote:
>
>> Hi Folks,
>>
>> I have configured a service to use the SSOCookieProvider (federation
>> role) and am seeing some odd behavior that I didn’t expect. Looking for
>> clarification if this is behaving normally or not.
>>
>> I am making a request to: https://host.example.com:8443/
>> gateway/default/myservice/myapp/ and am then getting redirected  to
>> https://host.example.com:8443/gateway/knoxsso/knoxauth/
>> login.html?originalUrl=https://host.example.com:9443/myapp/ where I am
>> prompted for credentials. I am able to login successfully and then am
>> redirected to the value of the originalUrl query parameter.
>>
>> Essentially what I am seeing is that knox sso is listing the originalUrl
>> as the actual path to the web application instead of the gated value of the
>> web application. For my scenario I would expect that query parameter to
>> contain the value https://host.example.com:8443/
>> gateway/default/myservice/myapp/
>>
>> Why does knox sso behave in this manner? Is this expected? If so, do I
>> need to write additional rewrite rules for this to behave like I would like?
>>
>> Note: If I call https://host.example.com:8443/gateway/knoxsso/knoxauth/
>> login.html?originalUrl=https://host.example.com:8443/
>> gateway/default/myservice/myapp/ directly then knox functions as I would
>> expect it to. IE. I can log in and get redirected to my app in the gated
>> path.
>>
>> I am using HDP version 2.6.2.0 with Knox 0.12.0
>>
>> Thanks in advance,
>>
>> Christopher Jackson
>
>

Re: KnoxSSO JWT authentication by third party.

[larry mccay]

Hi -

I just verified that it works as expected with the latest HDP sandbox for
2.6.5.

I started the sandbox, changed the TTL param in the KnoxSSO service in
knoxsso.xml from 3 to -1, saved and restarted.
I think used the Knox Admin UI to view the topologies that are deployed to
the instance and it had the -1 value for the TTL param.

While there may be a difference between the versions, I have not
encountered this issue before at all.

Is it possible that there is a permissions issue on the files?
Was this cluster upgraded from a previous version?

At any rate, I think that there is an issue with the cluster rather that
with the knox.py script though I'm not really sure what that condition is
even checking.

This doesn't really solve anything for you but hope it is helpful in some
way.

thanks,

--larry


On Thu, Jul 12, 2018 at 8:04 PM, larry mccay  wrote:

> Hi Christopher -
>
> I apologize, this dropped off my radar.
> I need to go back and try and reproduce it still.
>
> It would still be surprising to me but I am surprised often. :)
>
> thanks,
>
> --larry
>
> On Thu, Jul 12, 2018 at 5:29 PM, Christopher Jackson <
> jackson.christopher@gmail.com> wrote:
>
>> Hey Larry,
>>
>> I looked into this a bit deeper. It appears the knoxsso-topology is NOT
>> updated because of the following code in /var/lib/ambari-server/resourc
>> es/common-services/KNOX/0.5.0.2.2/package/scripts/knox.py:
>>
>>   if params.version_formatted and 
>> check_stack_feature(StackFeature.KNOX_SSO_TOPOLOGY,
>> params.version_formatted):
>>   File(os.path.join(params.knox_conf_dir, "topologies",
>> "knoxsso.xml"),
>>  group=params.knox_group,
>>  owner=params.knox_user,
>>  content=InlineTemplate(params.knoxsso_topology_template)
>>   )
>>
>> I believe the if condition is evaluating to false and thus preventing the
>> knoxsso.xml from being written as I do not see a corresponding output entry
>> in the log of a restart of the Knox Service (IE. I would expect to see a
>> File['/usr/hdp/current/knox-server/conf/topologies/knoxsso.xml’] line):
>>
>> 2018-07-12 12:42:43,870 - Generating config: 
>> /usr/hdp/current/knox-server/conf/gateway-site.xml
>> 2018-07-12 12:42:43,871 - 
>> File['/usr/hdp/current/knox-server/conf/gateway-site.xml'] {'owner': 'knox', 
>> 'content': InlineTemplate(...), 'group': 'knox', 'mode': None, 'encoding': 
>> 'UTF-8'}
>> 2018-07-12 12:42:43,879 - 
>> File['/usr/hdp/current/knox-server/conf/gateway-log4j.properties'] 
>> {'content': InlineTemplate(...), 'owner': 'knox', 'group': 'knox', 'mode': 
>> 0644}
>> 2018-07-12 12:42:43,887 - 
>> File['/usr/hdp/current/knox-server/conf/topologies/default.xml'] {'content': 
>> InlineTemplate(...), 'owner': 'knox', 'group': 'knox'}
>> 2018-07-12 12:42:43,890 - 
>> File['/usr/hdp/current/knox-server/conf/topologies/admin.xml'] {'content': 
>> InlineTemplate(...), 'owner': 'knox', 'group': 'knox'}
>> 2018-07-12 12:42:43,891 - 
>> Execute['/usr/hdp/current/knox-server/bin/knoxcli.sh create-master --master 
>> [PROTECTED]'] {'environment': {'JAVA_HOME': u'/usr/jdk64/jdk1.8.0_112'}, 
>> 'not_if': "ambari-sudo.sh su knox -l -s /bin/bash -c 'test -f 
>> /usr/hdp/current/knox-server/data/security/master'", 'user': 'knox'}
>>
>>
>> This is on HDP 2.6.2 using Knox 0.12.0. I’ve created issue
>> https://issues.apache.org/jira/browse/AMBARI-24285 to track.
>>
>> Regards,
>> Christopher Jackson
>>
>>
>>
>> On Jun 27, 2018, at 7:13 PM, larry mccay  wrote:
>>
>> Interesting, I will need to try and reproduce this but it is definitely
>> the first that I have heard of it.
>>
>> On Wed, Jun 27, 2018 at 6:36 PM, Christopher Jackson <
>> jackson.christopher@gmail.com> wrote:
>>
>>> Hi Larry,
>>>
>>> 4) On HDP 2.6.2 I have noticed that when I make changes to the "Advanced
>>> knoxsso-topology” section for the Knox Service in Ambari and then restart
>>> the service that the changes are not persisted to disk at
>>> /usr/hdp/current/knox-server/conf/topologies/knoxsso.xml and thus the
>>> changes are not picked up until that file is hand edited to reflect the
>>> changes. Is this a known issue? For example changes to the
>>> “knoxsso.redirect.whitelist.regex” in the ambari config will not take
>>> effect until the file mentioned above is hand edited.
>>>
>>> The trick is that you have to restart the server in order for Ambari to
>>> actually push any config changes out to the Knox instances.
>

Re: Rewrite rules necessary for knox sso?

[larry mccay]

Hi Christopher -

Proxying and SSO together may require a rewrite rule.
I'll look at what SSO Cookie Provider is doing though.
I would expect originalUrl to be the gateway url from there.

Thanks,

--larry

On Thu, Jul 12, 2018, 4:17 PM Christopher Jackson <
jackson.christopher@gmail.com> wrote:

> Hi Folks,
>
> I have configured a service to use the SSOCookieProvider (federation role)
> and am seeing some odd behavior that I didn’t expect. Looking for
> clarification if this is behaving normally or not.
>
> I am making a request to:
> https://host.example.com:8443/gateway/default/myservice/myapp/ and am
> then getting redirected  to
> https://host.example.com:8443/gateway/knoxsso/knoxauth/login.html?originalUrl=https://host.example.com:9443/myapp/
> where I am prompted for credentials. I am able to login successfully and
> then am redirected to the value of the originalUrl query parameter.
>
> Essentially what I am seeing is that knox sso is listing the originalUrl
> as the actual path to the web application instead of the gated value of the
> web application. For my scenario I would expect that query parameter to
> contain the value
> https://host.example.com:8443/gateway/default/myservice/myapp/
>
> Why does knox sso behave in this manner? Is this expected? If so, do I
> need to write additional rewrite rules for this to behave like I would like?
>
> Note: If I call
> https://host.example.com:8443/gateway/knoxsso/knoxauth/login.html?originalUrl=https://host.example.com:8443/gateway/default/myservice/myapp/
> directly then knox functions as I would expect it to. IE. I can log in and
> get redirected to my app in the gated path.
>
> I am using HDP version 2.6.2.0 with Knox 0.12.0
>
> Thanks in advance,
>
> Christopher Jackson

Re: KnoxSSO JWT authentication by third party.

[larry mccay]

Hi Christopher -

I apologize, this dropped off my radar.
I need to go back and try and reproduce it still.

It would still be surprising to me but I am surprised often. :)

thanks,

--larry

On Thu, Jul 12, 2018 at 5:29 PM, Christopher Jackson <
jackson.christopher@gmail.com> wrote:

> Hey Larry,
>
> I looked into this a bit deeper. It appears the knoxsso-topology is NOT
> updated because of the following code in /var/lib/ambari-server/
> resources/common-services/KNOX/0.5.0.2.2/package/scripts/knox.py:
>
>   if params.version_formatted and check_stack_feature(
> StackFeature.KNOX_SSO_TOPOLOGY, params.version_formatted):
>   File(os.path.join(params.knox_conf_dir, "topologies",
> "knoxsso.xml"),
>  group=params.knox_group,
>  owner=params.knox_user,
>  content=InlineTemplate(params.knoxsso_topology_template)
>   )
>
> I believe the if condition is evaluating to false and thus preventing the
> knoxsso.xml from being written as I do not see a corresponding output entry
> in the log of a restart of the Knox Service (IE. I would expect to see a
> File['/usr/hdp/current/knox-server/conf/topologies/knoxsso.xml’] line):
>
> 2018-07-12 12:42:43,870 - Generating config: 
> /usr/hdp/current/knox-server/conf/gateway-site.xml
> 2018-07-12 12:42:43,871 - 
> File['/usr/hdp/current/knox-server/conf/gateway-site.xml'] {'owner': 'knox', 
> 'content': InlineTemplate(...), 'group': 'knox', 'mode': None, 'encoding': 
> 'UTF-8'}
> 2018-07-12 12:42:43,879 - 
> File['/usr/hdp/current/knox-server/conf/gateway-log4j.properties'] 
> {'content': InlineTemplate(...), 'owner': 'knox', 'group': 'knox', 'mode': 
> 0644}
> 2018-07-12 12:42:43,887 - 
> File['/usr/hdp/current/knox-server/conf/topologies/default.xml'] {'content': 
> InlineTemplate(...), 'owner': 'knox', 'group': 'knox'}
> 2018-07-12 12:42:43,890 - 
> File['/usr/hdp/current/knox-server/conf/topologies/admin.xml'] {'content': 
> InlineTemplate(...), 'owner': 'knox', 'group': 'knox'}
> 2018-07-12 12:42:43,891 - 
> Execute['/usr/hdp/current/knox-server/bin/knoxcli.sh create-master --master 
> [PROTECTED]'] {'environment': {'JAVA_HOME': u'/usr/jdk64/jdk1.8.0_112'}, 
> 'not_if': "ambari-sudo.sh su knox -l -s /bin/bash -c 'test -f 
> /usr/hdp/current/knox-server/data/security/master'", 'user': 'knox'}
>
>
> This is on HDP 2.6.2 using Knox 0.12.0. I’ve created issue
> https://issues.apache.org/jira/browse/AMBARI-24285 to track.
>
> Regards,
> Christopher Jackson
>
>
>
> On Jun 27, 2018, at 7:13 PM, larry mccay  wrote:
>
> Interesting, I will need to try and reproduce this but it is definitely
> the first that I have heard of it.
>
> On Wed, Jun 27, 2018 at 6:36 PM, Christopher Jackson <
> jackson.christopher@gmail.com> wrote:
>
>> Hi Larry,
>>
>> 4) On HDP 2.6.2 I have noticed that when I make changes to the "Advanced
>> knoxsso-topology” section for the Knox Service in Ambari and then restart
>> the service that the changes are not persisted to disk at
>> /usr/hdp/current/knox-server/conf/topologies/knoxsso.xml and thus the
>> changes are not picked up until that file is hand edited to reflect the
>> changes. Is this a known issue? For example changes to the
>> “knoxsso.redirect.whitelist.regex” in the ambari config will not take
>> effect until the file mentioned above is hand edited.
>>
>> The trick is that you have to restart the server in order for Ambari to
>> actually push any config changes out to the Knox instances.
>> This is unfortunate - since Knox can hot deploy topology changes but is
>> what it is.
>> Be aware that if you hand edit the files as you are, the next time you
>> restart via Ambari it will overwrite any changes that you have made there.
>>
>>
>>
>> To be clear. I am restarting the knox service via the ambari UI after I
>> make the changes to the knox configuration in the ambari UI. After restart
>> I am seeing that the above file is NOT updated, hence why I am forced to
>> modify it by hand.
>>
>> Changes made to “Advanced topology” are correctly written to disk after
>> an update to the config and a subsequent restart of the knox service. It
>> seems to just be the “Advanced knoxsso-topology” that has the issue.
>>
>> Regards,
>>
>> Christopher Jackson
>>
>>
>>
>> On Jun 27, 2018, at 1:11 PM, larry mccay  wrote:
>>
>> Hi Christopher -
>>
>> 1) Is it possible to include additional claims that contain group
>> information for the user from LDAP?
>>
>> Not currently - there are a couple issues with this appproach but I
>> wouldn't be agai

Re: knox OS auth does not work if /tmp has noexec

[larry mccay]

Hi Lian -

I haven't encountered this before. You will likely need to dig into the
shiro PAM  support itself if not even lower into the Pam module code.

I will try and find some time to dig a bit myself.

Thanks,

-larry

On Mon, Jul 2, 2018, 2:58 PM Lian Jiang  wrote:

> Hi,
>
> When /tmp has noexec, Knox OS auth throws error:
>
> [lianjia@prod1-namenode knox-server]$ sudo bin/knoxcli.sh user-auth-test
> --cluster ui --u guest --p "{PASSWORD}" --d
> org.apache.shiro.authc.AuthenticationException: Authentication failed for
> token submission [org.apache.shiro.authc.UsernamePasswordToken - guest,
> rememberMe=false].  Possible unexpected error? (Typical or expected login
> exceptions should extend from AuthenticationException).
> /tmp/jna-3506402/jna4211705767471308463.tmp:
> /tmp/jna-3506402/jna4211705767471308463.tmp: failed to map segment from
> shared object: Operation not permitted
> org.apache.shiro.authc.AuthenticationException: Authentication failed for
> token submission [org.apache.shiro.authc.UsernamePasswordToken - guest,
> rememberMe=false].  Possible unexpected error? (Typical or expected login
> exceptions should extend from AuthenticationException).
> at
> org.apache.shiro.authc.AbstractAuthenticator.authenticate(AbstractAuthenticator.java:214)
> at
> org.apache.shiro.mgt.AuthenticatingSecurityManager.authenticate(AuthenticatingSecurityManager.java:106)
> at
> org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:270)
> at
> org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256)
> at
> org.apache.hadoop.gateway.util.KnoxCLI$LDAPCommand.authenticateUser(KnoxCLI.java:1171)
> at
> org.apache.hadoop.gateway.util.KnoxCLI$LDAPCommand.authenticateUser(KnoxCLI.java:1206)
> at
> org.apache.hadoop.gateway.util.KnoxCLI$LDAPAuthCommand.execute(KnoxCLI.java:1502)
> at org.apache.hadoop.gateway.util.KnoxCLI.run(KnoxCLI.java:143)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.gateway.util.KnoxCLI.main(KnoxCLI.java:1777)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.gateway.launcher.Invoker.invokeMainMethod(Invoker.java:70)
> at org.apache.hadoop.gateway.launcher.Invoker.invoke(Invoker.java:39)
> at org.apache.hadoop.gateway.launcher.Command.run(Command.java:99)
> at org.apache.hadoop.gateway.launcher.Launcher.run(Launcher.java:69)
> at org.apache.hadoop.gateway.launcher.Launcher.main(Launcher.java:46)
> Caused by: java.lang.UnsatisfiedLinkError:
> /tmp/jna-3506402/jna4211705767471308463.tmp:
> /tmp/jna-3506402/jna4211705767471308463.tmp: failed to map segment from
> shared object: Operation not permitted
> at java.lang.ClassLoader$NativeLibrary.load(Native Method)
> at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
> at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
> at java.lang.Runtime.load0(Runtime.java:809)
> at java.lang.System.load(System.java:1086)
> at
> com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:761)
> at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:736)
> at com.sun.jna.Native.(Native.java:131)
> at com.sun.jna.Pointer.(Pointer.java:41)
> at com.sun.jna.Structure.(Structure.java:1949)
> at org.jvnet.libpam.PAM.(PAM.java:73)
> at
> org.apache.hadoop.gateway.shirorealm.KnoxPamRealm.doGetAuthenticationInfo(KnoxPamRealm.java:135)
> at
> org.apache.shiro.realm.AuthenticatingRealm.getAuthenticationInfo(AuthenticatingRealm.java:568)
> at
> org.apache.shiro.authc.pam.ModularRealmAuthenticator.doSingleRealmAuthentication(ModularRealmAuthenticator.java:180)
> at
> org.apache.shiro.authc.pam.ModularRealmAuthenticator.doAuthenticate(ModularRealmAuthenticator.java:267)
> at
> org.apache.shiro.authc.AbstractAuthenticator.authenticate(AbstractAuthenticator.java:198)
> ... 18 more
> ERR: Unable to authenticate user: guest
>
> Setting "-Djava.io.tmpdir={other_tmp_folder} 
> -D*jna*.tmpdir={other_tmp_folder}"
> in gateway.sh did not help.
>
> I cannot remove noexec for /tmp since it is required for our production.
> Any idea how to solve this issue? Thanks!
>

Re: Knox OS authentication fail due to "password check failed for user"

[larry mccay]

Are you on the Knox host when testing with Pam tester? The accounts will
need to be on the Knox host.



On Sat, Jun 30, 2018, 2:22 AM Lian Jiang  wrote:

> I am using OS auth for knox and have verified the username and password
> work:
>
> sudo pamtester -v knox guest authenticate
> pamtester: invoking pam_start(knox, guest, ...)
> pamtester: performing operation - authenticate
> Password:
> pamtester: successfully authenticated
>
> However, my curl command failed:
>
> curl -ik  -u guest:"{PASSWORD}"
> http://test-namenode.subnet1.hadoop.oraclevcn.com:8443/gateway/ui/webhdfs/v1/user/?op=LISTSTATUS
>
> The error is:
> Jun 30 06:16:03 test-namenode unix_chkpwd[37385]: check pass; user unknown
> Jun 30 06:16:03 test-namenode unix_chkpwd[37385]: password check failed
> for user (guest)
> Jun 30 06:16:03 test-namenode java: pam_unix(knox:auth): authentication
> failure; logname= uid=2018 euid=2018 tty= ruser= rhost=  user=guest
>
>
> Any idea how I can debug? Appreciate any help.
>
>
>

Re: KnoxSSO JWT authentication by third party.

[larry mccay]

Interesting, I will need to try and reproduce this but it is definitely the
first that I have heard of it.

On Wed, Jun 27, 2018 at 6:36 PM, Christopher Jackson <
jackson.christopher@gmail.com> wrote:

> Hi Larry,
>
> 4) On HDP 2.6.2 I have noticed that when I make changes to the "Advanced
> knoxsso-topology” section for the Knox Service in Ambari and then restart
> the service that the changes are not persisted to disk at
> /usr/hdp/current/knox-server/conf/topologies/knoxsso.xml and thus the
> changes are not picked up until that file is hand edited to reflect the
> changes. Is this a known issue? For example changes to the
> “knoxsso.redirect.whitelist.regex” in the ambari config will not take
> effect until the file mentioned above is hand edited.
>
> The trick is that you have to restart the server in order for Ambari to
> actually push any config changes out to the Knox instances.
> This is unfortunate - since Knox can hot deploy topology changes but is
> what it is.
> Be aware that if you hand edit the files as you are, the next time you
> restart via Ambari it will overwrite any changes that you have made there.
>
>
>
> To be clear. I am restarting the knox service via the ambari UI after I
> make the changes to the knox configuration in the ambari UI. After restart
> I am seeing that the above file is NOT updated, hence why I am forced to
> modify it by hand.
>
> Changes made to “Advanced topology” are correctly written to disk after an
> update to the config and a subsequent restart of the knox service. It seems
> to just be the “Advanced knoxsso-topology” that has the issue.
>
> Regards,
>
> Christopher Jackson
>
>
>
> On Jun 27, 2018, at 1:11 PM, larry mccay  wrote:
>
> Hi Christopher -
>
> 1) Is it possible to include additional claims that contain group
> information for the user from LDAP?
>
> Not currently - there are a couple issues with this appproach but I
> wouldn't be against a patch that optionally enables it.
> * There can be 100's of groups sometimes for a given user
> * No one in the current ecosystem is expecting to extract groups from the
> cookie for authorization purposes and group lookup is done closer to the
> resource itself
> * Given that the token represents an authentication event as a snapshot in
> time, the group membership may change by the time you extract them from the
> token
>
> 2) Does the Knox SSO implementation support JSON Web Key (JWK)?
>
> Not currently.
>
> 3) Where is the signing key stored? I have the desire to validate the JWT
> in a third party web container. I am using Knox 0.12.0 on HDP 2.6.2.
>
> By default it uses the gateway-identity alias within the
> {GATEWAY_HOME}/data/security/keystores/gateway.jks keystore.
> It may also be configured to use custom signing keys [1] - via
> gateway.signing.keystore.name and gateway.signing.key.alias
>
> 4) On HDP 2.6.2 I have noticed that when I make changes to the "Advanced
> knoxsso-topology” section for the Knox Service in Ambari and then restart
> the service that the changes are not persisted to disk at
> /usr/hdp/current/knox-server/conf/topologies/knoxsso.xml and thus the
> changes are not picked up until that file is hand edited to reflect the
> changes. Is this a known issue? For example changes to the
> “knoxsso.redirect.whitelist.regex” in the ambari config will not take
> effect until the file mentioned above is hand edited.
>
> The trick is that you have to restart the server in order for Ambari to
> actually push any config changes out to the Knox instances.
> This is unfortunate - since Knox can hot deploy topology changes but is
> what it is.
> Be aware that if you hand edit the files as you are, the next time you
> restart via Ambari it will overwrite any changes that you have made there.
>
> HTH.
>
> --larry
>
> 1. http://knox.apache.org/books/knox-1-0-0/user-guide.html#Gateway+Server+
> Configuration
>
>
> On Wed, Jun 27, 2018 at 1:00 PM, Christopher Jackson <
> jackson.christopher@gmail.com> wrote:
>
>> Hey Folks,
>>
>> I’ve enabled Knox SSO and I am able to navigate to the Knox SSO UI and
>> enter credentials to log in. I am seeing that the JWT cookie is properly
>> created with the claims that I would expect. Some questions:
>>
>> 1) Is it possible to include additional claims that contain group
>> information for the user from LDAP?
>>
>> 2) Does the Knox SSO implementation support JSON Web Key (JWK)?
>>
>> 3) Where is the signing key stored? I have the desire to validate the JWT
>> in a third party web container. I am using Knox 0.12.0 on HDP 2.6.2.
>>
>> 4) On HDP 2.6.2 I have notic

Re: KnoxSSO JWT authentication by third party.

[larry mccay]

Hi Christopher -

1) Is it possible to include additional claims that contain group
information for the user from LDAP?

Not currently - there are a couple issues with this appproach but I
wouldn't be against a patch that optionally enables it.
* There can be 100's of groups sometimes for a given user
* No one in the current ecosystem is expecting to extract groups from the
cookie for authorization purposes and group lookup is done closer to the
resource itself
* Given that the token represents an authentication event as a snapshot in
time, the group membership may change by the time you extract them from the
token

2) Does the Knox SSO implementation support JSON Web Key (JWK)?

Not currently.

3) Where is the signing key stored? I have the desire to validate the JWT
in a third party web container. I am using Knox 0.12.0 on HDP 2.6.2.

By default it uses the gateway-identity alias within the
{GATEWAY_HOME}/data/security/keystores/gateway.jks keystore.
It may also be configured to use custom signing keys [1] - via
gateway.signing.keystore.name and gateway.signing.key.alias

4) On HDP 2.6.2 I have noticed that when I make changes to the "Advanced
knoxsso-topology” section for the Knox Service in Ambari and then restart
the service that the changes are not persisted to disk at
/usr/hdp/current/knox-server/conf/topologies/knoxsso.xml and thus the
changes are not picked up until that file is hand edited to reflect the
changes. Is this a known issue? For example changes to the
“knoxsso.redirect.whitelist.regex” in the ambari config will not take
effect until the file mentioned above is hand edited.

The trick is that you have to restart the server in order for Ambari to
actually push any config changes out to the Knox instances.
This is unfortunate - since Knox can hot deploy topology changes but is
what it is.
Be aware that if you hand edit the files as you are, the next time you
restart via Ambari it will overwrite any changes that you have made there.

HTH.

--larry

1.
http://knox.apache.org/books/knox-1-0-0/user-guide.html#Gateway+Server+Configuration


On Wed, Jun 27, 2018 at 1:00 PM, Christopher Jackson <
jackson.christopher@gmail.com> wrote:

> Hey Folks,
>
> I’ve enabled Knox SSO and I am able to navigate to the Knox SSO UI and
> enter credentials to log in. I am seeing that the JWT cookie is properly
> created with the claims that I would expect. Some questions:
>
> 1) Is it possible to include additional claims that contain group
> information for the user from LDAP?
>
> 2) Does the Knox SSO implementation support JSON Web Key (JWK)?
>
> 3) Where is the signing key stored? I have the desire to validate the JWT
> in a third party web container. I am using Knox 0.12.0 on HDP 2.6.2.
>
> 4) On HDP 2.6.2 I have noticed that when I make changes to the "Advanced
> knoxsso-topology” section for the Knox Service in Ambari and then restart
> the service that the changes are not persisted to disk at
> /usr/hdp/current/knox-server/conf/topologies/knoxsso.xml and thus the
> changes are not picked up until that file is hand edited to reflect the
> changes. Is this a known issue? For example changes to the
> “knoxsso.redirect.whitelist.regex” in the ambari config will not take
> effect until the file mentioned above is hand edited.
>
> Regards,
>
> Christopher Jackson
> jackson.christopher@gmail.com

Re: livy LDAP authentication

[larry mccay]

Hi Lian -

I believe Livy does kerberos authentication and also leverages a doas user
in the request to determine the effective user.

HTH,

--larry

On Tue, Jun 26, 2018 at 11:57 AM, Lian Jiang  wrote:

> I have a HDP 2.6 cluster which uses knox as gateway and LDAP for
> authentication. I enabled livy via knox. Can livy authenticate the callers
> using LDAP?
>
> https://github.com/apache/incubator-livy/pull/20 shows it is still in
> progress.
>
> If livy cannot use LDAP, what authentication mechanism can be used
> instead? Thanks very much for any hints.
>

Re: Knox and MapReduce Job History Server

[larry mccay]

This should not be the case for the new Yarn UI v2.
But it is an ongoing issue that is starting to feel like pushing a boulder
up hill.

I've considered adding a trusted proxy module to Knox that folks could add
as a dependency.
Problem is, every time that I start doing it - it ends up being what is
already available in hadoop.


On Wed, May 9, 2018 at 10:36 AM, Kevin Risden <kris...@apache.org> wrote:

> Thanks Larry. This also seems to affect YARN as well when looking at the
> YARN UI support in Apache Knox. Same sort of issue: "Logged in as: knox".
>
> Kevin Risden
>
> On Thu, May 3, 2018 at 10:10 AM, larry mccay <lmc...@apache.org> wrote:
>
>> This can only be addressed in Hadoop, AFAICT.
>> There are so many UIs and even APIs not supporting trusted proxies and it
>> is really becoming a problem.
>>
>> We need to file JIRAs where this support is missing and potentially
>> provide patches as it seems folks are reluctant to add proper support for
>> it anymore.
>>
>> On Wed, May 2, 2018 at 2:56 PM, Kevin Risden <kris...@apache.org> wrote:
>>
>>> There has been quite a bit of work done around enabling Knox and the
>>> MapReduce Job History server UI. However, I think we are going to run into
>>> a fundamental problem with the MapReduce Job History server and
>>> impersonation/doAs.
>>>
>>> Based on our research, we found that the MR Job History server doesn't
>>> support the "doAs" pattern that is used by Knox and Hue.
>>>
>>> If I navigate to a topology /jobhistory/ in the top right hand corner
>>> is: "Logged in as: knox". Knox sends a doAs parameter (just like Hue)
>>> and it seems like MR Job History server is ignoring this. Ideally the
>>> "Logged in as:" would be the user who authenticated to Knox and not the
>>> user Knox is running as (according to Kerberos).
>>>
>>> Right now this causes problems for us because the Knox user doesn't have
>>> access to view the job logs. If I navigate to an attempt log it looks to
>>> proxy correctly but end up with this error message:
>>>
>>> User [knox] is not authorized to view the logs for
>>> attempt_1525122616004_0164_m_00_0 in log file
>>>
>>> The same type of error happens in Hue when trying to look at job logs.
>>>
>>>
>>> User [hue] is not authorized to view the logs for job_1521053483563_0133
>>> in log file
>>>
>>>
>>> It looks like this might have been previously reported as KNOX-747 but
>>> was closed as can't reproduce.
>>>
>>> I don't think this is a problem with Knox but bringing it up here in
>>> case there are any ideas on workarounds. This might have to be fixed
>>> upstream in the Hadoop project.
>>>
>>> Any ideas on how to move forward with Knox and MapReduce Job History
>>> server?
>>>
>>> Kevin Risden
>>>
>>
>>
>

Re: Knox and MapReduce Job History Server

[larry mccay]

This can only be addressed in Hadoop, AFAICT.
There are so many UIs and even APIs not supporting trusted proxies and it
is really becoming a problem.

We need to file JIRAs where this support is missing and potentially provide
patches as it seems folks are reluctant to add proper support for it
anymore.

On Wed, May 2, 2018 at 2:56 PM, Kevin Risden  wrote:

> There has been quite a bit of work done around enabling Knox and the
> MapReduce Job History server UI. However, I think we are going to run into
> a fundamental problem with the MapReduce Job History server and
> impersonation/doAs.
>
> Based on our research, we found that the MR Job History server doesn't
> support the "doAs" pattern that is used by Knox and Hue.
>
> If I navigate to a topology /jobhistory/ in the top right hand corner is: 
> "Logged
> in as: knox". Knox sends a doAs parameter (just like Hue) and it seems like
> MR Job History server is ignoring this. Ideally the "Logged in as:" would
> be the user who authenticated to Knox and not the user Knox is running as
> (according to Kerberos).
>
> Right now this causes problems for us because the Knox user doesn't have
> access to view the job logs. If I navigate to an attempt log it looks to
> proxy correctly but end up with this error message:
>
> User [knox] is not authorized to view the logs for
> attempt_1525122616004_0164_m_00_0 in log file
>
> The same type of error happens in Hue when trying to look at job logs.
>
>
> User [hue] is not authorized to view the logs for job_1521053483563_0133
> in log file
>
>
> It looks like this might have been previously reported as KNOX-747 but was
> closed as can't reproduce.
>
> I don't think this is a problem with Knox but bringing it up here in case
> there are any ideas on workarounds. This might have to be fixed upstream in
> the Hadoop project.
>
> Any ideas on how to move forward with Knox and MapReduce Job History
> server?
>
> Kevin Risden
>

Re: turn off debug logging from org.apache.http.wire

[larry mccay]

+1 to Kevin's point.
Ambari rewrites all configs on server restart.

On Sun, Apr 15, 2018 at 1:16 PM, Kevin Risden <kris...@apache.org> wrote:

> Are you using Ambari or deploying Knox manually?
>
> If you using Ambari, then Ambari will force overwrite the log4j configs
> during a restart. You must update the log4j settings in Ambari. Another
> option if using Ambari is to find the debug setting and set it to false (I
> don't have a cluster in front of me so can't look up the setting).
>
> Kevin Risden
>
> On Sun, Apr 15, 2018 at 10:56 AM, Wei Han <wei...@uber.com> wrote:
>
>> Interesting. Thanks Larry. I'll dig more on my side.
>>
>> On Sun, Apr 15, 2018 at 4:54 AM, larry mccay <lmc...@apache.org> wrote:
>>
>>> No, I cannot reproduce it.
>>> If you are modifying the correct gateway-log4j.properties and restarting
>>> the server you should not see that.
>>>
>>> In fact, turning on DEBUG for wire via:
>>> log4j.logger.org.apache.http.wire=DEBUG
>>>
>>> Doesn't result in output in gateway.out for me but instead gateway.log
>>> and turning it on and off certainly works for me.
>>>
>>> You may have enabled TRACE logging if you are seeing body content -
>>> those settings are like the following:
>>>
>>> #log4j.logger.org.apache.knox.gateway.access=TRACE,httpaccess
>>> #log4j.additivity.org.apache.knox.gateway.access=false
>>>
>>> #log4j.logger.org.apache.knox.gateway.http=TRACE,httpserver
>>> #log4j.additivity.org.apache.knox.gateway.http=false
>>> ##log4j.logger.org.apache.knox.gateway.http.request.headers=OFF
>>> ##log4j.logger.org.apache.knox.gateway.http.response.headers=OFF
>>> ##log4j.logger.org.apache.knox.gateway.http.request.body=OFF
>>> ##log4j.logger.org.apache.knox.gateway.http.response.body=OFF
>>>
>>> I suggest you back up to the gateway-log4j.properties from the original
>>> install and remove any other log4j config that you may have elsewhere.
>>>
>>> On Sun, Apr 15, 2018 at 1:58 AM, Wei Han <wei...@uber.com> wrote:
>>>
>>>> Hi Larry - Thanks a lot for getting back to me.
>>>>
>>>> Yes I made sure all DEBUG level is turned off in my 
>>>> gateway-log4j.properties
>>>> file, but that doesn't seem to be working. I also tried to explicitly
>>>> set log4j.logger.httpclient.wire.header to WARN (as suggested in post
>>>> <https://stackoverflow.com/questions/4915414/disable-httpclient-logging>),
>>>> but that also didn't help.
>>>>
>>>> Actually it's very easy to reproduce this(at least on my side). If you
>>>> call knox with a webhdfs request (like 
>>>> webhdfs/v1/tmp/weihan/small.txt?op=OPEN),
>>>> you should be able to see a bunch of below logs in gateway.out. In fact it
>>>> outputs the actual content on the wire(security hole?)
>>>>
>>>>   06:52:49.751 [qtp1473205473-61] DEBUG org.apache.http.wire -
>>>> http-outgoing-2 << "[0x0][0x0
>>>>
>>>> Let me know if you're able to repro this.
>>>>
>>>> Thanks.
>>>>
>>>> On Sat, Apr 14, 2018 at 7:11 AM, larry mccay <lmc...@apache.org> wrote:
>>>>
>>>>> Hi Wei -
>>>>>
>>>>> If you look at your gateway-log4j.properties file, you should see
>>>>> something like the following near the top:
>>>>>
>>>>> app.log.dir=${launcher.dir}/../logs
>>>>> app.log.file=${launcher.name}.log
>>>>> app.audit.file=${launcher.name}-audit.log
>>>>>
>>>>> log4j.rootLogger=ERROR, drfa
>>>>>
>>>>> log4j.logger.org.apache.knox.gateway=INFO
>>>>> #log4j.logger.org.apache.knox.gateway=DEBUG
>>>>>
>>>>> #log4j.logger.org.eclipse.jetty=DEBUG
>>>>> #log4j.logger.org.apache.shiro=DEBUG
>>>>> #log4j.logger.org.apache.http=DEBUG
>>>>> #log4j.logger.org.apache.http.client=DEBUG
>>>>> #log4j.logger.org.apache.http.headers=DEBUG
>>>>> #log4j.logger.org.apache.http.wire=DEBUG
>>>>>
>>>>> Note that all of the DEBUG settings are commented out.
>>>>> Also note that the rootLogger is set to ERROR and not DEBUG.
>>>>>
>>>>> Can you compare and share with us what yours are set to?
>>>>>
>>>>> thanks,
>>>>>
>>>>> --larry
>>>>>
>>>>> On Sat, Apr 14, 2018 at 2:56 AM, Wei Han <wei...@uber.com> wrote:
>>>>>
>>>>>> Hi Knox experts -
>>>>>>
>>>>>> Has anyone successfully turn off the debug loggings
>>>>>> from org.apache.http.wire, like below?
>>>>>>
>>>>>>   06:52:49.751 [qtp1473205473-61] DEBUG org.apache.http.wire -
>>>>>> http-outgoing-2 << "[0x0][0x0
>>>>>>
>>>>>> I've tried almost everything in this stackoverflow post
>>>>>> <https://stackoverflow.com/questions/4915414/disable-httpclient-logging>,
>>>>>> but my gateway.out is still filled with all the debug logs from 
>>>>>> httpclient.
>>>>>>
>>>>>> Any tip would be appreciated.
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Re: turn off debug logging from org.apache.http.wire

[larry mccay]

No, I cannot reproduce it.
If you are modifying the correct gateway-log4j.properties and restarting
the server you should not see that.

In fact, turning on DEBUG for wire via:
log4j.logger.org.apache.http.wire=DEBUG

Doesn't result in output in gateway.out for me but instead gateway.log and
turning it on and off certainly works for me.

You may have enabled TRACE logging if you are seeing body content - those
settings are like the following:

#log4j.logger.org.apache.knox.gateway.access=TRACE,httpaccess
#log4j.additivity.org.apache.knox.gateway.access=false

#log4j.logger.org.apache.knox.gateway.http=TRACE,httpserver
#log4j.additivity.org.apache.knox.gateway.http=false
##log4j.logger.org.apache.knox.gateway.http.request.headers=OFF
##log4j.logger.org.apache.knox.gateway.http.response.headers=OFF
##log4j.logger.org.apache.knox.gateway.http.request.body=OFF
##log4j.logger.org.apache.knox.gateway.http.response.body=OFF

I suggest you back up to the gateway-log4j.properties from the original
install and remove any other log4j config that you may have elsewhere.

On Sun, Apr 15, 2018 at 1:58 AM, Wei Han <wei...@uber.com> wrote:

> Hi Larry - Thanks a lot for getting back to me.
>
> Yes I made sure all DEBUG level is turned off in my gateway-log4j.properties
> file, but that doesn't seem to be working. I also tried to explicitly set
> log4j.logger.httpclient.wire.header to WARN (as suggested in post
> <https://stackoverflow.com/questions/4915414/disable-httpclient-logging>),
> but that also didn't help.
>
> Actually it's very easy to reproduce this(at least on my side). If you
> call knox with a webhdfs request (like 
> webhdfs/v1/tmp/weihan/small.txt?op=OPEN),
> you should be able to see a bunch of below logs in gateway.out. In fact it
> outputs the actual content on the wire(security hole?)
>
>   06:52:49.751 [qtp1473205473-61] DEBUG org.apache.http.wire -
> http-outgoing-2 << "[0x0][0x0
>
> Let me know if you're able to repro this.
>
> Thanks.
>
> On Sat, Apr 14, 2018 at 7:11 AM, larry mccay <lmc...@apache.org> wrote:
>
>> Hi Wei -
>>
>> If you look at your gateway-log4j.properties file, you should see
>> something like the following near the top:
>>
>> app.log.dir=${launcher.dir}/../logs
>> app.log.file=${launcher.name}.log
>> app.audit.file=${launcher.name}-audit.log
>>
>> log4j.rootLogger=ERROR, drfa
>>
>> log4j.logger.org.apache.knox.gateway=INFO
>> #log4j.logger.org.apache.knox.gateway=DEBUG
>>
>> #log4j.logger.org.eclipse.jetty=DEBUG
>> #log4j.logger.org.apache.shiro=DEBUG
>> #log4j.logger.org.apache.http=DEBUG
>> #log4j.logger.org.apache.http.client=DEBUG
>> #log4j.logger.org.apache.http.headers=DEBUG
>> #log4j.logger.org.apache.http.wire=DEBUG
>>
>> Note that all of the DEBUG settings are commented out.
>> Also note that the rootLogger is set to ERROR and not DEBUG.
>>
>> Can you compare and share with us what yours are set to?
>>
>> thanks,
>>
>> --larry
>>
>> On Sat, Apr 14, 2018 at 2:56 AM, Wei Han <wei...@uber.com> wrote:
>>
>>> Hi Knox experts -
>>>
>>> Has anyone successfully turn off the debug loggings
>>> from org.apache.http.wire, like below?
>>>
>>>   06:52:49.751 [qtp1473205473-61] DEBUG org.apache.http.wire -
>>> http-outgoing-2 << "[0x0][0x0
>>>
>>> I've tried almost everything in this stackoverflow post
>>> <https://stackoverflow.com/questions/4915414/disable-httpclient-logging>,
>>> but my gateway.out is still filled with all the debug logs from httpclient.
>>>
>>> Any tip would be appreciated.
>>>
>>
>>
>

Re: turn off debug logging from org.apache.http.wire

[larry mccay]

Hi Wei -

If you look at your gateway-log4j.properties file, you should see something
like the following near the top:

app.log.dir=${launcher.dir}/../logs
app.log.file=${launcher.name}.log
app.audit.file=${launcher.name}-audit.log

log4j.rootLogger=ERROR, drfa

log4j.logger.org.apache.knox.gateway=INFO
#log4j.logger.org.apache.knox.gateway=DEBUG

#log4j.logger.org.eclipse.jetty=DEBUG
#log4j.logger.org.apache.shiro=DEBUG
#log4j.logger.org.apache.http=DEBUG
#log4j.logger.org.apache.http.client=DEBUG
#log4j.logger.org.apache.http.headers=DEBUG
#log4j.logger.org.apache.http.wire=DEBUG

Note that all of the DEBUG settings are commented out.
Also note that the rootLogger is set to ERROR and not DEBUG.

Can you compare and share with us what yours are set to?

thanks,

--larry

On Sat, Apr 14, 2018 at 2:56 AM, Wei Han  wrote:

> Hi Knox experts -
>
> Has anyone successfully turn off the debug loggings
> from org.apache.http.wire, like below?
>
>   06:52:49.751 [qtp1473205473-61] DEBUG org.apache.http.wire -
> http-outgoing-2 << "[0x0][0x0
>
> I've tried almost everything in this stackoverflow post
> ,
> but my gateway.out is still filled with all the debug logs from httpclient.
>
> Any tip would be appreciated.
>

Re: CRL (Certificate Revocation List) & mutual authentication with SSL

[larry mccay]

Hi Dominique -

It does not seem that we have implemented CRL as yet.
It would need to be added to the creation of the SSLContextFactory in the
JettySSLService.

Could I bother you to file a JIRA for this for 1.1.0 release?
Feel free to contribute a patch for it as well, if you like.

thanks!

--larry

On Mon, Mar 19, 2018 at 8:54 AM, Dominique De Vito 
wrote:

> Hi,
>
> When searching into the User Guide (1.0), while I have found "mutual
> authentication with SSL", there is no mention of CRL (Certificate
> Revocation List).
>
> Well, it's useful to have a truststore for authenticating clients.
> But it's useful too to revoke this or this user/client account for some
> cases.
>
> "Mutual authentication with SSL" works "hand in hand" with CRL
> (Certificate Revocation List) usually.
>
> Is CRL just missing for the doc ?
> Or, is CRL feature not implemented in Knox ?
>
> Thanks.
>
> Regards,
> Dominique
>
>

Re: KnoxSSO OpenID Error: Unable to renew the session. The session store may not support this feature

[larry mccay]

There was an issue found with the google oidc integration recently.
This may be the same issue, I will need to dig that up in the dev@ or user@
list and verify.


On Sun, Mar 4, 2018 at 11:47 AM, Ryan H 
wrote:

> Hi Knox Users,
>
> I am rethreading this error I am getting as I mentioned it in a different
> thread that was about a different error (sorry to those active on the other
> email thread).
>
> I am running into an issue with KnoxSSO with the pac4j OIDC federation
> provider. When accessing the gateway, I am correctly redirected to my
> configured OpenID provider and upon successful authentication, redirected
> back to Knox but resulting in error. I am posting the relevant config files
> as well as the errors below. I have switched over to testBasicAuth just to
> confirm that I can connect to the NiFi app, which I can. I am not really
> sure where to go from here. I have sifted the internet and Knox
> documentation on this and haven't been able to find anything. I did find
> some info on this error with play and pac4j with the way the session was
> being handled and assumed that Knox would handle this (if not, it is not
> documented that I can find). Any help is appreciated!
>
> Cheers,
>
> Ryan
>
>
> *Error 1: *
> 2018-03-04 11:22:53,701 ERROR engine.DefaultCallbackLogic
> (DefaultCallbackLogic.java:renewSession(123)) - Unable to renew the
> session. The session store may not support this feature
>
> *Error 2:*
> 2018-03-04 10:07:05,578 ERROR knox.gateway 
> (AbstractGatewayFilter.java:doFilter(69))
> - Failed to execute filter: org.pac4j.core.exception.TechnicalException:
> State parameter is different from the one sent in authentication request.
> Session expired or possible threat of cross-site request forgery
> 2018-03-04 10:07:05,578 ERROR knox.gateway (GatewayFilter.java:doFilter(177))
> - Gateway processing failed: javax.servlet.ServletException:
> org.pac4j.core.exception.TechnicalException: State parameter is different
> from the one sent in authentication request. Session expired or possible
> threat of cross-site request forgery
> javax.servlet.ServletException: org.pac4j.core.exception.TechnicalException:
> State parameter is different from the one sent in authentication request.
> Session expired or possible threat of cross-site request forgery
> at org.apache.knox.gateway.filter.AbstractGatewayFilter.doFilter(
> AbstractGatewayFilter.java:70)
> at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(
> GatewayFilter.java:377)
> at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(
> GatewayFilter.java:277)
> at org.apache.knox.gateway.webappsec.filter.XFrameOptionsFilter.doFilter(
> XFrameOptionsFilter.java:58)
> at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(
> GatewayFilter.java:377)
> at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(
> GatewayFilter.java:277)
> at org.apache.knox.gateway.GatewayFilter.doFilter(GatewayFilter.java:171)
> at org.apache.knox.gateway.GatewayFilter.doFilter(GatewayFilter.java:94)
> at org.apache.knox.gateway.GatewayServlet.service(GatewayServlet.java:141)
> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
> at org.eclipse.jetty.servlet.ServletHandler.doHandle(
> ServletHandler.java:587)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:143)
> at org.eclipse.jetty.security.SecurityHandler.handle(
> SecurityHandler.java:577)
> at org.eclipse.jetty.server.session.SessionHandler.
> doHandle(SessionHandler.java:223)
> at org.eclipse.jetty.server.handler.ContextHandler.
> doHandle(ContextHandler.java:1127)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(
> ServletHandler.java:515)
> at org.eclipse.jetty.server.session.SessionHandler.
> doScope(SessionHandler.java:185)
> at org.eclipse.jetty.server.handler.ContextHandler.
> doScope(ContextHandler.java:1061)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:141)
> at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(
> ContextHandlerCollection.java:215)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:97)
> at org.apache.knox.gateway.trace.TraceHandler.handle(TraceHandler.java:51)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:97)
> at org.apache.knox.gateway.filter.CorrelationHandler.
> handle(CorrelationHandler.java:39)
> at org.eclipse.jetty.servlets.gzip.GzipHandler.handle(
> GzipHandler.java:479)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:97)
> at org.apache.knox.gateway.filter.PortMappingHelperHandler.handle(
> PortMappingHelperHandler.java:152)
> at org.eclipse.jetty.server.handler.HandlerCollection.
> handle(HandlerCollection.java:110)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> at 

Re: Knox 1.0.0 Unknown Host Exception for KnoxSSO Config

[larry mccay]

Maybe you have a proxy configured in your browser to allow you to get there.

On Sat, Mar 3, 2018 at 10:35 PM, Ryan H <ryan.howell.developm...@gmail.com>
wrote:

> Hi Larry,
>
> I can reach the host via browser, but not via ping... Ping results in
> "Unknown Host", I'm seeing a correlation here...
>
> -Ryan
>
> On Sat, Mar 3, 2018 at 10:32 PM, larry mccay <lmc...@apache.org> wrote:
>
>> Hi Ryan -
>>
>> Welcome to Knox-ville!
>>
>> Going to start with a very obvious question - can you ping that host from
>> the machine where the gateway is running?
>>
>> thanks,
>>
>> --larry
>>
>> On Sat, Mar 3, 2018 at 10:07 PM, Ryan H <ryan.howell.development@gmail
>> .com> wrote:
>>
>>> Hi All,
>>>
>>> Disclaimer: I am very new to Knox!
>>>
>>> I am working on setting up KnoxSSO with an OpenID provider (Cloud
>>> Foundry UAA) for AuthN to an application (Apache NiFi). I am running into
>>> an issue where it seems that the oidc.discoverUri is resulting in the
>>> following error:
>>>
>>> 2018-03-03 21:59:37,104 ERROR knox.gateway 
>>> (AbstractGatewayFilter.java:doFilter(69))
>>> - Failed to execute filter: org.pac4j.core.exception.TechnicalException:
>>> java.net.UnknownHostException: {guid-id}.sub-uaa.another.zone
>>> .aws-us01.something.io
>>> 2018-03-03 21:59:37,104 ERROR knox.gateway 
>>> (GatewayFilter.java:doFilter(177))
>>> - Gateway processing failed: javax.servlet.ServletException:
>>> org.pac4j.core.exception.TechnicalException:
>>> java.net.UnknownHostException: {guid-id}.sub-uaa.another.zone
>>> .aws-us01.something.io
>>> javax.servlet.ServletException: org.pac4j.core.exception.TechnicalException:
>>> java.net.UnknownHostException: {guid-id}.sub-uaa.another.zone
>>> .aws-us01.something.io
>>> at org.apache.knox.gateway.filter.AbstractGatewayFilter.doFilte
>>> r(AbstractGatewayFilter.java:70)
>>> at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(Gatewa
>>> yFilter.java:377)
>>> at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(Gateway
>>> Filter.java:277)
>>> at org.apache.knox.gateway.webappsec.filter.XFrameOptionsFilter
>>> .doFilter(XFrameOptionsFilter.java:58)
>>> at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(Gatewa
>>> yFilter.java:377)
>>> at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(Gateway
>>> Filter.java:277)
>>> at org.apache.knox.gateway.GatewayFilter.doFilter(GatewayFilter
>>> .java:171)
>>> at org.apache.knox.gateway.GatewayFilter.doFilter(GatewayFilter.java:94)
>>> at org.apache.knox.gateway.GatewayServlet.service(GatewayServle
>>> t.java:141)
>>> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder
>>> .java:812)
>>>
>>> *Here is my topology from knoxsso.xml config:*
>>>
>>> 
>>>   
>>> 
>>> webappsec
>>> WebAppSec
>>> true
>>> xframe.options.en
>>> abledtrue
>>> 
>>> 
>>> federation
>>> pac4j
>>> true
>>> 
>>>   pac4j.callbackUrl
>>>   https://localhost:8443/
>>> gateway/knoxsso/api/v1/websso
>>> 
>>> 
>>>   clientName
>>>   OidcClient
>>> 
>>> 
>>>   oidc.id
>>>   some_client_id
>>> 
>>> 
>>>   oidc.secret
>>>   some_client_secret
>>> 
>>> 
>>>   oidc.discoveryUri
>>>   https://{guid-id}.sub-u
>>> aa.another.zone.aws-us01.something.io/.well-known/openid-con
>>> figuration
>>> 
>>> 
>>>   oidc.preferredJwsAlgorithm
>>>   RS256
>>> 
>>> 
>>> 
>>> 
>>>   knoxauth
>>> 
>>> 

Re: Knox 1.0.0 Unknown Host Exception for KnoxSSO Config

[larry mccay]

Hi Ryan -

Welcome to Knox-ville!

Going to start with a very obvious question - can you ping that host from
the machine where the gateway is running?

thanks,

--larry

On Sat, Mar 3, 2018 at 10:07 PM, Ryan H 
wrote:

> Hi All,
>
> Disclaimer: I am very new to Knox!
>
> I am working on setting up KnoxSSO with an OpenID provider (Cloud Foundry
> UAA) for AuthN to an application (Apache NiFi). I am running into an issue
> where it seems that the oidc.discoverUri is resulting in the following
> error:
>
> 2018-03-03 21:59:37,104 ERROR knox.gateway 
> (AbstractGatewayFilter.java:doFilter(69))
> - Failed to execute filter: org.pac4j.core.exception.TechnicalException:
> java.net.UnknownHostException: {guid-id}.sub-uaa.another.
> zone.aws-us01.something.io
> 2018-03-03 21:59:37,104 ERROR knox.gateway (GatewayFilter.java:doFilter(177))
> - Gateway processing failed: javax.servlet.ServletException:
> org.pac4j.core.exception.TechnicalException:
> java.net.UnknownHostException: {guid-id}.sub-uaa.another.
> zone.aws-us01.something.io
> javax.servlet.ServletException: org.pac4j.core.exception.TechnicalException:
> java.net.UnknownHostException: {guid-id}.sub-uaa.another.
> zone.aws-us01.something.io
> at org.apache.knox.gateway.filter.AbstractGatewayFilter.doFilter(
> AbstractGatewayFilter.java:70)
> at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(
> GatewayFilter.java:377)
> at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(
> GatewayFilter.java:277)
> at org.apache.knox.gateway.webappsec.filter.XFrameOptionsFilter.doFilter(
> XFrameOptionsFilter.java:58)
> at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(
> GatewayFilter.java:377)
> at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(
> GatewayFilter.java:277)
> at org.apache.knox.gateway.GatewayFilter.doFilter(GatewayFilter.java:171)
> at org.apache.knox.gateway.GatewayFilter.doFilter(GatewayFilter.java:94)
> at org.apache.knox.gateway.GatewayServlet.service(GatewayServlet.java:141)
> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
>
> *Here is my topology from knoxsso.xml config:*
>
> 
>   
> 
> webappsec
> WebAppSec
> true
> xframe.options.
> enabledtrue
> 
> 
> federation
> pac4j
> true
> 
>   pac4j.callbackUrl
>   https://localhost:8443/
> gateway/knoxsso/api/v1/websso
> 
> 
>   clientName
>   OidcClient
> 
> 
>   oidc.id
>   some_client_id
> 
> 
>   oidc.secret
>   some_client_secret
> 
> 
>   oidc.discoveryUri
>   https://{guid-id}.sub-
> uaa.another.zone.aws-us01.something.io/.well-known/
> openid-configuration
> 
> 
>   oidc.preferredJwsAlgorithm
>   RS256
> 
> 
> 
> 
>   knoxauth
> 
> 
> KNOXSSO
> 
> knoxsso.cookie.secure.only
> false
> 
> 
> knoxsso.token.ttl
> 360
> 
> 
>knoxsso.redirect.whitelist.regex
>^https?:\/\/(localhost|127\.0\.0\.1|0:0:0:
> 0:0:0:0:1|::1):[0-9].*$
> 
> 
> 
>
>
> *Here is my topology from sandbox.xml:*
>
> 
>
> 
>
> 
> federation
> SSOCookieProvider
> true
> 
> sso.authentication.provider.url
> https://127.0.0.1:8443/gateway/knoxsso/api/v1/websso<
> /value>
> 
> 
>
> 
> identity-assertion
> Default
> true
> 
>
> 
>
> 
> NIFI
> http://localhost:8080
> 
>
> 
>
> I was able to use the gateway to get to the NiFi app with basic auth as a
> connectivity test, and now I want to drop in the OpenID provider for the
> auth I am really after. Any help is greatly appreciated!
>
>
> Cheers,
>
> Ryan H.
>

Re: Knox SSO JWT consumption in WebSphere Liberty.

[larry mccay]

Hi Christopher -

That sounds very strange is the AUTH_HEADER a standard header that I am
unaware of?
I will try and reproduce this.

thanks,

--larry

On Mon, Feb 26, 2018 at 5:48 PM, Christopher Jackson <
jackson.christopher@gmail.com> wrote:

> Hi All,
>
> I have some questions around the Knox SSO provider.
>
> I have configured my Knox instance for Knox SSO and everything seems to be
> working. I’m observing the following:
>
> - I make a request to a service behind the gateway
> - I get redirected to the Knox SSO form based login page.
> - I provide my credentials and click Sign In.
> - I am successfully authenticated and a cookie named hadoop-jwt is created.
> - I get redirected to my originally requested URL.
>
> I have also configured my Knox instance with a custom provider which uses
> a HttpServletRequestWrapper to override the getHeaderName and getHeader
> methods. “AUTH_HEADER” is added to the getHeaderNames enumeration. When
> getHeader is called for “AUTH_HEADER” it returns a value of “Bearer
> ”. I get the JWT serialized string by reading the
> hadoop-jwt cookie.
>
> Why am I doing this? This is necessary for the authentication mechanism
> in the container (WebSphere Liberty Profile (WLP)) serving our web
> application to consume the Knox issued JWT.
>
> The problem I am having is that it seems a later filter (I believe
> DefaultDispatch) in the chain is over writing the value of the header. As
> the final request that is made to the original requested URL has
> an “Authorization” header of the basic64 encoded username and password
> combination that I provided to the Knox SSO form based login page.
>
> Unfortunately WLP does not currently allow me to configure reading the JWT
> from a cookie. So I need to handle it in Knox before the request is made to
> WLP.
>
> Any suggestions on what I may try to overcome this issue?
>
> Regards,
> Christopher Jackson
>

[DISCUSS] Planning for Apache Knox 1.1.0 Release

[larry mccay]

All -

Sorry for the delay on this topic.

We are going to start of this planning thread with ~85 Unresolved JIRAs in
either 1.1.0 or 0.15.0 fixVersion.

project = KNOX AND resolution = Unresolved AND fixVersion in (1.1.0,
0.15.0) ORDER BY  priority DESC, updated DESC

I will spend some time migrating all 0.15.0 to 1.1.0 to begin with and then
we will need to go through and see what is already taken care of or can
wait for a 1.2.0 or later.

I also have a couple KIPs in mind to target larger features/themes for this
release.

Off the top of my head:

* I think we need to address some cloud specific usecases and plan to
provide a KIP for that. Hybrid cloud/federated knox instances, Azure AD
integration, ID mapping from Hadoop user to IAM users/roles, etc. Perhaps
some CASB-like features if they make sense.

* I also think we need one for articulating a reasonable flow for Logout in
KnoxSSO. There are a lot of little nuances to logout across multiple apps
and between different IDPs. This will require some discussion.

* Another thing that has been tugging at my interest has been the fact that
we may be able provide some common libraries to help ecosystem applications
uptake the trusted proxy pattern and KnoxSSO.

Anyway, these are my initial thoughts, please feel free to raise additional
ideas/themes for KIPs, etc.

I was thinking that we could try and target an end of March or Mid April
1.1.0 release.

Thoughts?

--larry

Re: KNOX Pac4j Azure AD Open ID

[larry mccay]

So, it seems that OidcClient and Auth0 is working but AAD and Google are
not.
While I've never tested AAD, I have tested Google so there may be a
regression there.

I notice that there are specific clients for both AzureAd and Google now -
see: http://www.pac4j.org/docs/clients/openid-connect.html#2-clients
Google:
https://github.com/pac4j/pac4j/blob/master/pac4j-oidc/src/main/java/org/pac4j/oidc/client/GoogleOidcClient.java
AzureAd:
https://github.com/pac4j/pac4j/blob/master/pac4j-oidc/src/main/java/org/pac4j/oidc/client/AzureAdClient.java

They also have corresponding profile creators.

Instead of the generic OidcClient, can we try GoogleOidcClient and
AzureAdOidcClient ?

On Tue, Feb 20, 2018 at 6:14 AM, Colm O hEigeartaigh <cohei...@apache.org>
wrote:

> Trying with "www.local.com" makes no difference for me. The problem is
> that it is not retrieving the stored user profile correctly after it
> redirects back to the Pac4jDispatcherFilter:
>
> 2018-02-20 10:55:07,486 DEBUG engine.DefaultSecurityLogic
> (DefaultSecurityLogic.java:perform(98)) - loadProfilesFromSession: true
> 2018-02-20 10:55:07,487 DEBUG session.KnoxSessionStore
> (KnoxSessionStore.java:get(91)) - Get from session: pac4jUserProfiles =
> null
>
> However, I replaced the KnoxSessionStore with the following:
>
> https://github.com/pac4j/pac4j/blob/master/pac4j-core/
> src/main/java/org/pac4j/core/context/session/J2ESessionStore.java
>
> and it worked! Perhaps we should change from storing the profile in a
> cookie to an attribute on the session instead?
>
> Colm.
>
> On Mon, Feb 19, 2018 at 6:43 PM, larry mccay <lmc...@apache.org> wrote:
>
>> KnoxSSO service (WebSSOResource) uses it to redirect to the originally
>> requested resource.
>>
>> The KnoxSSO service assumes that by the time the request gets to it that
>> authentication/federation of the enduser has been done, that the enduser is
>> authorized to use knoxsso and that the identity assertion has mapped it to
>> whatever the effective user should be.
>>
>> Therefore, the KnoxSSO service can then extract the PrimaryPrincipal from
>> the normalized Java Subject, create the JWT representation of the
>> authentication event and set-cookie.
>> It then redirects the user agent to the originally requested resource
>> where the browser should present the cookie.
>>
>> In this case, SSOCookieProvider will be then looking for the cookie and
>> will redirect back to KnoxSSO if it is not present.
>>
>> On Mon, Feb 19, 2018 at 1:15 PM, Colm O hEigeartaigh <cohei...@apache.org
>> > wrote:
>>
>>> OK I'll check it. A question though - where is the redirection via
>>> originalUrl supposed to take place? From the source I can see "originalUrl"
>>> is referenced in:
>>>
>>> gateway-applications/src/main/resources/applications/knoxaut
>>> h/app/js/knoxauth.js
>>> gateway-applications/src/main/resources/applications/knoxaut
>>> h/app/redirecting.jsp
>>>
>>> However, I'm not using the knoxauth application (with pac4j)?
>>>
>>> Colm.
>>>
>>> On Mon, Feb 19, 2018 at 6:04 PM, larry mccay <lmc...@apache.org> wrote:
>>>
>>>> Hi Colm -
>>>>
>>>> Thanks for trying to reproduce.
>>>> Starting to get concerned about a regression
>>>>
>>>> I see that you are using localhost - that will definitely present
>>>> problems for Set-Cookie with some browsers.
>>>> I know that Chrome will not accept it for sure.
>>>>
>>>> Can you switch to a full domain name?
>>>> I usually just map localhost to www.local.com in my /etc/hosts file.
>>>>
>>>> You will also need to make sure to set the whitelist for the domain
>>>> name.
>>>>
>>>> thanks,
>>>>
>>>> --larry
>>>>
>>>>
>>>> On Mon, Feb 19, 2018 at 12:49 PM, Colm O hEigeartaigh <
>>>> cohei...@apache.org> wrote:
>>>>
>>>>> I can reproduce the issue with Google. From the logs I see the
>>>>> following:
>>>>>
>>>>> 2018-02-19 17:21:58,484 DEBUG session.KnoxSessionStore
>>>>> (KnoxSessionStore.java:get(91)) - Get from session: pac4jRequestedUrl
>>>>> = https://localhost:8443/gateway/knoxssopac4j/api/v1/websso?or
>>>>> iginalUrl=https://localhost:8443/gateway/sandbox-ssopac4j/we
>>>>> bhdfs/v1/data/LICENSE.txt?op=OPEN
>>>>> 2018-02-19 17:21:58,485 DEBUG session.KnoxSessionStore
>>>>> (KnoxS

Re: KNOX Pac4j Azure AD Open ID

[larry mccay]

Hi Colm -

Thanks for trying to reproduce.
Starting to get concerned about a regression

I see that you are using localhost - that will definitely present problems
for Set-Cookie with some browsers.
I know that Chrome will not accept it for sure.

Can you switch to a full domain name?
I usually just map localhost to www.local.com in my /etc/hosts file.

You will also need to make sure to set the whitelist for the domain name.

thanks,

--larry


On Mon, Feb 19, 2018 at 12:49 PM, Colm O hEigeartaigh <cohei...@apache.org>
wrote:

> I can reproduce the issue with Google. From the logs I see the following:
>
> 2018-02-19 17:21:58,484 DEBUG session.KnoxSessionStore
> (KnoxSessionStore.java:get(91)) - Get from session: pac4jRequestedUrl =
> https://localhost:8443/gateway/knoxssopac4j/api/v1/
> websso?originalUrl=https://localhost:8443/gateway/
> sandbox-ssopac4j/webhdfs/v1/data/LICENSE.txt?op=OPEN
> 2018-02-19 17:21:58,485 DEBUG session.KnoxSessionStore
> (KnoxSessionStore.java:set(107)) - Save in session: pac4jRequestedUrl =
> null
> 2018-02-19 17:21:58,485 DEBUG engine.DefaultCallbackLogic
> (DefaultCallbackLogic.java:redirectToOriginallyRequestedUrl(137)) -
> redirectUrl: https://localhost:8443/gateway/knoxssopac4j/api/v1/
> websso?originalUrl=https://localhost:8443/gateway/
> sandbox-ssopac4j/webhdfs/v1/data/LICENSE.txt?op=OPEN
> 2018-02-19 17:21:58,488 DEBUG knox.gateway (GatewayFilter.java:doFilter(119))
> - Received request: GET /api/v1/websso
>
> It is getting an access token correctly and trying to redirect back to the
> original URL. However from the logs above it seems to never hit the
> original URL but instead hits the "redirectUrl". Is some parsing supposed
> to take place to extract "originalUrl" from the "pac4jRequestedUrl"
> parameter and redirect to this instead?
>
> Colm.
>
> On Mon, Feb 19, 2018 at 4:16 PM, larry mccay <lmc...@apache.org> wrote:
>
>> No, the hadoop-jwt cookie is for KnoxSSO and the SSOCookieProvider.
>> If it isn't being set, it could be that it isn't getting past the pac4j
>> federation provider to the KnoxSSO service itself because there is a
>> redirect from the pac4j provider or the Set-Cookie just isn't being
>> accepted by the browser.
>>
>> The audit log does seem to be getting a redirect from pac4j.
>>
>> I haven't seen any examples of it working AAD - so we are in unchartered
>> waters here.
>>
>> @Jerome - any insights?
>>
>> On Mon, Feb 19, 2018 at 2:04 AM, Nisha Menon <nisha.meno...@gmail.com>
>> wrote:
>>
>>> Hello Larry,
>>>
>>> hadoop-jwt cookie is not set. Isnt this for JWT provider?
>>> In SSO provider with pac4j, I can see cookies like:
>>> pac4j.session.pac4jRequestedUrl, pac4j.session.oidcStateAttribute,
>>> pac4j.session.oidcNonceAttribute etc.
>>>
>>> IP address and hostnames are mapped, else the *basic auth* also would
>>> have failed.
>>> My issue is only when I use pac4j with Oidc client and Azure AD.
>>>
>>> On Fri, Feb 16, 2018 at 10:11 PM, larry mccay <lmc...@apache.org> wrote:
>>>
>>>> It looks like you may be using ip addresses for your Knox URLs - to
>>>> webhdfs.
>>>> In order to rule out cookie related issue can you do a couple things:
>>>>
>>>> 1. check whether a cookie called hadoop-jwt is actually set in your
>>>> browser
>>>> 2. if not, you may want to set an actual domain in your /etc/hosts or
>>>> something that you can reference - I use www.local.com to map to
>>>> localhost
>>>>
>>>> I think that ip address should work for this case actually but there
>>>> are differences in browsers that might not let it.
>>>> Also, if you had another service on another ip address, the browser
>>>> would not present the cookie - so this is good to be aware of anyway.
>>>>
>>>> On Fri, Feb 16, 2018 at 8:55 AM, Sandeep Moré <moresand...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hello Nisha,
>>>>> Can you share details of "mycluster" topology ? also, can you turn up
>>>>> the logs to debug and share them along with the audit log that would help
>>>>> us to understand the problem better.
>>>>>
>>>>> Best,
>>>>> Sandeep
>>>>>
>>>>> On Fri, Feb 16, 2018 at 3:16 AM, Nisha Menon <nisha.meno...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I have setup KNOX

Re: KNOX Pac4j Azure AD Open ID

[larry mccay]

No, the hadoop-jwt cookie is for KnoxSSO and the SSOCookieProvider.
If it isn't being set, it could be that it isn't getting past the pac4j
federation provider to the KnoxSSO service itself because there is a
redirect from the pac4j provider or the Set-Cookie just isn't being
accepted by the browser.

The audit log does seem to be getting a redirect from pac4j.

I haven't seen any examples of it working AAD - so we are in unchartered
waters here.

@Jerome - any insights?

On Mon, Feb 19, 2018 at 2:04 AM, Nisha Menon <nisha.meno...@gmail.com>
wrote:

> Hello Larry,
>
> hadoop-jwt cookie is not set. Isnt this for JWT provider?
> In SSO provider with pac4j, I can see cookies like: 
> pac4j.session.pac4jRequestedUrl,
> pac4j.session.oidcStateAttribute, pac4j.session.oidcNonceAttribute etc.
>
> IP address and hostnames are mapped, else the *basic auth* also would
> have failed.
> My issue is only when I use pac4j with Oidc client and Azure AD.
>
> On Fri, Feb 16, 2018 at 10:11 PM, larry mccay <lmc...@apache.org> wrote:
>
>> It looks like you may be using ip addresses for your Knox URLs - to
>> webhdfs.
>> In order to rule out cookie related issue can you do a couple things:
>>
>> 1. check whether a cookie called hadoop-jwt is actually set in your
>> browser
>> 2. if not, you may want to set an actual domain in your /etc/hosts or
>> something that you can reference - I use www.local.com to map to
>> localhost
>>
>> I think that ip address should work for this case actually but there are
>> differences in browsers that might not let it.
>> Also, if you had another service on another ip address, the browser would
>> not present the cookie - so this is good to be aware of anyway.
>>
>> On Fri, Feb 16, 2018 at 8:55 AM, Sandeep Moré <moresand...@gmail.com>
>> wrote:
>>
>>> Hello Nisha,
>>> Can you share details of "mycluster" topology ? also, can you turn up
>>> the logs to debug and share them along with the audit log that would help
>>> us to understand the problem better.
>>>
>>> Best,
>>> Sandeep
>>>
>>> On Fri, Feb 16, 2018 at 3:16 AM, Nisha Menon <nisha.meno...@gmail.com>
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> I have setup KNOX to connect with Azure AD using pac4j.
>>>>
>>>> However, after the authentication at Azure login page, it gets into an
>>>> infinite loop and does not give back the original REST call response.
>>>>
>>>> *Details:*
>>>>
>>>> 1. I try to access the original URL eg: 
>>>> *https://x.x.2.3:8442/gateway/mycluster/webhdfs/v1/user?op=LISTSTATUS
>>>> <https://x.x.2.3:8442/gateway/mycluster/webhdfs/v1/user?op=LISTSTATUS>*
>>>>
>>>> 2. It redirects to *https://**login.microsoftonline.com
>>>> <http://login.microsoftonline.com/>* and asks for credentials.
>>>>
>>>> 3. After successful login at Azure login page, it redirects to 
>>>> *http://x.x.2.3:8442/gateway/knoxsso/api/v1/websso
>>>> <http://x.x.2.3:8442/gateway/knoxsso/api/v1/websso>* with code,
>>>> session and state variables passed as below:
>>>>
>>>> *https://x.x.2.3:8442/gateway/knoxsso/api/v1/websso?code=AQABAAIAAABHh4k***LFFm7C9cIShE7nggAA=5dzTZBYhEVDBrA*GZRNfANGb5ls_state=42f2447b-621***790eaa2d18
>>>> <https://x.x.2.3:8442/gateway/knoxsso/api/v1/websso?code=AQABAAIAAABHh4k***LFFm7C9cIShE7nggAA=5dzTZBYhEVDBrA*GZRNfANGb5ls_state=42f2447b-621***790eaa2d18>*
>>>>
>>>> 2. Following this call, it *again *calls the *login.microsoftonline.com
>>>> <http://login.microsoftonline.com/>* like below:
>>>>
>>>> *https://login.microsoftonline.com/f82969ba-b***c1d0557a/oauth2/authorize?response_type=code_id=385***3-a4bdceaa34_uri=https%3A%2F%2Fx.x.2.3%3A8442%2Fgateway%2Fknoxsso%2Fapi%2Fv1%2Fwebsso%3Fpac4jCallback%3Dtrue%26client_name%3DOidcClient≻ope=openid+profile+email=5dzTZBYhEVDBrAInao9VHDRd33uiRp-GZRNfANGb5ls=BvCUroM7_aKFjmbLYxaxbS0Mq9SJ8If0CUpITEGB-bw
>>>> <https://login.microsoftonline.com/f82969ba-b***c1d0557a/oauth2/authorize?response_type=code_id=385***3-a4bdceaa34_uri=https%3A%2F%2Fx.x.2.3%3A8442%2Fgateway%2Fknoxsso%2Fapi%2Fv1%2Fwebsso%3Fpac4jCallback%3Dtrue%26client_name%3DOidcClient%E2%89%BBope=openid+profile+email=5dzTZBYhEVDBrAInao9VHDRd33uiRp-GZRNfANGb5ls=BvCUroM7_aKFjmbLYxaxbS0Mq9SJ8If0CUpITEGB-bw>*
>>>>
>>>> After this, step 1 and 2 alternate sever

Board Report

[larry mccay]

All -

I have submitted the following as our quarterly board report.
If any corrections or additions are needed it is still possible to edit it
- so please feel free to let me know.

thanks,

--larry

## Description:
 - The Apache Knox Gateway is an Application Gateway for interacting with
the REST APIs and UIs of Apache Hadoop deployments.
The Knox Gateway provides a single access point for all REST and HTTP
interactions with Apache Hadoop clusters. Knox delivers three groups of
user facing services: Proxying, Authentication and Client Services

## Issues:
 - there are no issues requiring board attention at this time

## Activity:
 - The Apache Knox team completed the release of Apache Knox 0.14.0 and
followed up with our 1.0.0 release which was largely the same as 0.14.0
but it included a repackaging of all of our class package names to remove
"hadoop" from the name. This allows downstream consumers to take on new
features in 0.14.0 without being forced to move to new package names in
case of incompatibilities. We have a backlog of JIRAs for 1.1.0 but will
also begin our feature planning through the use of KIP-# wiki pages to
identify larger features and themes for the release.

## Health report:
 - We have continued to use the KIP-# pages to drive the primary focus of
each
release and feel it is working well. The 0.14.0 release was primarily
focused on the addition of the service discovery and simplified descriptors
(KIP-8) to improve and simplify management of topologies. It also includes
many improvements and bug fixes. It took a bit longer than anticipated due
to
issues found during the VOTE with community testing. The Dec 14 2017
release date
allowed for the primary focus for 0.14.0 to be more rounded out and a
better quality
release to roll into out 1.0.0 milestone in early Feb 2018.
 - Community contributions continue to show growth and growing interest and
is
encouraging.

## PMC changes:

 - Currently 16 PMC members.
 - Philip Zampino was added to the PMC on Mon Jan 01 2018

## Committer base changes:

 - Currently 20 committers.
 - Philip Zampino was added as a committer on Fri Dec 29 2017

## Releases:

 - Last release was 1.0.0 on Wed Feb 07 2018

## Mailing list activity:

 - d...@knox.apache.org:
- 86 subscribers (up 2 in the last 3 months):
- 923 emails sent to list (1492 in previous quarter)

 - user@knox.apache.org:
- 114 subscribers (up 6 in the last 3 months):
- 62 emails sent to list (119 in previous quarter)

## JIRA activity:

 - 69 JIRA tickets created in the last 3 months
 - 50 JIRA tickets closed/resolved in the last 3 months

Re: KNOX Pac4j Azure AD Open ID

[larry mccay]

It looks like you may be using ip addresses for your Knox URLs - to webhdfs.
In order to rule out cookie related issue can you do a couple things:

1. check whether a cookie called hadoop-jwt is actually set in your browser
2. if not, you may want to set an actual domain in your /etc/hosts or
something that you can reference - I use www.local.com to map to localhost

I think that ip address should work for this case actually but there are
differences in browsers that might not let it.
Also, if you had another service on another ip address, the browser would
not present the cookie - so this is good to be aware of anyway.

On Fri, Feb 16, 2018 at 8:55 AM, Sandeep Moré  wrote:

> Hello Nisha,
> Can you share details of "mycluster" topology ? also, can you turn up the
> logs to debug and share them along with the audit log that would help us to
> understand the problem better.
>
> Best,
> Sandeep
>
> On Fri, Feb 16, 2018 at 3:16 AM, Nisha Menon 
> wrote:
>
>> Hello,
>>
>> I have setup KNOX to connect with Azure AD using pac4j.
>>
>> However, after the authentication at Azure login page, it gets into an
>> infinite loop and does not give back the original REST call response.
>>
>> *Details:*
>>
>> 1. I try to access the original URL eg: 
>> *https://x.x.2.3:8442/gateway/mycluster/webhdfs/v1/user?op=LISTSTATUS
>> *
>>
>> 2. It redirects to *https://**login.microsoftonline.com
>> * and asks for credentials.
>>
>> 3. After successful login at Azure login page, it redirects to 
>> *http://x.x.2.3:8442/gateway/knoxsso/api/v1/websso
>> * with code, session
>> and state variables passed as below:
>>
>> *https://x.x.2.3:8442/gateway/knoxsso/api/v1/websso?code=AQABAAIAAABHh4k***LFFm7C9cIShE7nggAA=5dzTZBYhEVDBrA*GZRNfANGb5ls_state=42f2447b-621***790eaa2d18
>> *
>>
>> 2. Following this call, it *again *calls the *login.microsoftonline.com
>> * like below:
>>
>> *https://login.microsoftonline.com/f82969ba-b***c1d0557a/oauth2/authorize?response_type=code_id=385***3-a4bdceaa34_uri=https%3A%2F%2Fx.x.2.3%3A8442%2Fgateway%2Fknoxsso%2Fapi%2Fv1%2Fwebsso%3Fpac4jCallback%3Dtrue%26client_name%3DOidcClient≻ope=openid+profile+email=5dzTZBYhEVDBrAInao9VHDRd33uiRp-GZRNfANGb5ls=BvCUroM7_aKFjmbLYxaxbS0Mq9SJ8If0CUpITEGB-bw
>> *
>>
>> After this, step 1 and 2 alternate several times and finally lands up in
>> "ERR_TOO_MANY_REDIRECTS"!!!
>>
>> This is my knoxsso.xml:
>>
>>
>>1. 
>>2.   
>>3.   
>>4.   webappsec
>>5.   WebAppSec
>>6.   true
>>7.   
>> xframe.options.enabledtrue
>>8.   
>>9.   
>>10.   federation
>>11.   pac4j
>>12.   true
>>13.   
>>14. pac4j.callbackUrl
>>15. 
>> https://x.x.2.3:8442/gateway/knoxsso/api/v1/websso
>>16.   
>>17.   
>>18. clientName
>>19. OidcClient
>>20.   
>>21.   
>>22. oidc.id
>>23. 385c2bc*2695eaa34
>>24.   
>>25.   
>>26. oidc.secret
>>27. 
>> Y30wOwM88BYvYmPp8KMyDY2W+o=
>>28.   
>>29.   
>>30. oidc.discoveryUri
>>31. 
>> https://login.microsoftonline.com/f82969***1d0557a/.well-known/openid-configuration
>>32.   
>>33.   
>>34.   
>>35.   identity-assertion
>>36.   Default
>>37.   true
>>38.   
>>39.   
>>40.   
>>41. knoxauth
>>42.   
>>43.   
>>44.   KNOXSSO
>>45.   
>>46.   knoxsso.cookie.secure.only
>>47.   false
>>48.   
>>

Re: conflicting outbound rewrite rules

[larry mccay]

Hi Wei -

Sorry for not responding.

Rewrite rules are global for REST APIs and local for anything else.
So, for the UIs they should be resolving as local.

Only REST APIs that require the rewrite rules from other services need
global resolution.
Things like Oozie.

Do I owe you any other answer here?

thanks,

--larry

On Wed, Dec 27, 2017 at 2:23 PM, Wei Han <wei...@uber.com> wrote:

> Hi Larry - I haven't heard from you about this yet. Could you take a look
> when you get a chance?
>
> On Wed, Dec 13, 2017 at 3:41 PM, Wei Han <wei...@uber.com> wrote:
>
>> Hi Larry - Thanks for your response. Actually hdfsui and sparkhistoryui
>> are two different services. My understanding is a rewrite rule change in
>> sparkhistoryui shouldn't affect hdfsui, is this the right assumption? Are
>> these outbound rules designed to be global, or local to the service?
>>
>> Thanks.
>>
>> On Wed, Dec 13, 2017 at 12:12 PM, larry mccay <lmc...@apache.org> wrote:
>>
>>> Hi Wei -
>>>
>>> Thank you for tracking that down!
>>>
>>> Yes, I believe it is a bug but may be slightly more than that
>>> considering the same service def is being used for multiple logical UIs.
>>> Those being HDFSUI and SPARKHISTORYUI - these likely need to be broken
>>> out separately to avoid such things.
>>>
>>> I suspect that sparkhistoryui isn't going to work now that you removed
>>> that second rule.
>>>
>>> Can you please file a JIRA for this and if you provide a patch, we can
>>> review and possibly commit it as is - as long as sparkhistory ui still
>>> works.
>>>
>>> thanks again!
>>>
>>> --larry
>>>
>>> On Wed, Dec 13, 2017 at 2:55 PM, Wei Han <wei...@uber.com> wrote:
>>>
>>>> I found out that knox didn't rewrite /static/jquery-1.10.2.min.js in
>>>> HDFS home page(url: 
>>>> https://knox_host:port/gateway/test/hdfs/dfshealth.html),
>>>> which was causing some UI issues.
>>>> After some digging, I found out the following two rules, which can both
>>>> match the string (/static/jquery-1.10.2.min.js)
>>>> If I remove rule2, the rewrite will start to work.
>>>>
>>>> Is this a bug? Shouldn't knox only apply rules for the same service(in
>>>> my case hdfsui)?
>>>>
>>>>
>>>> *rule1:*
>>>>
>>>> 
>>>>
>>>> 
>>>>
>>>> 
>>>>
>>>>
>>>> *source:*
>>>>
>>>> https://github.com/apache/knox/blob/master/gateway-service-d
>>>> efinitions/src/main/resources/services/hdfsui/2.7.0/rewrite.xml#L67
>>>>
>>>>
>>>> *rule2:*
>>>>
>>>> >>> pattern="/static/{jquery=jquery*.min.js}">
>>>>
>>>> 
>>>>
>>>> 
>>>>
>>>>
>>>> *source:*
>>>>
>>>> https://github.com/apache/knox/blob/master/gateway-service-d
>>>> efinitions/src/main/resources/services/sparkhistoryui/1.4.0/
>>>> rewrite.xml#L56
>>>>
>>>>
>>>>
>>>>
>>>
>>
>