I feel, you are hitting build and packaging issues.
You might want to see the jar file names used by HDP and make sure you are
replacing exactly those ones with the same names or delete them before adding
yours.
You might want to see which jar files has the classes that are missing after
you add your jar file
You might also make sure you are compiling with the Ranger 1.2 code base
HDP does some soft linking. So you might have to be careful what jar file names
are and where you copy them.
If you feel you are only modifying one or two class files, you could also
consider replacing the classes directly in the jar file that is coming from HDP
Bosco
From: Ebrahim Khalil Abbasi <ebrahim.khalil.abb...@gmail.com>
Reply-To: <user@ranger.apache.org>
Date: Sunday, January 3, 2021 at 2:31 AM
To: <user@ranger.apache.org>
Subject: Re: Method Level Authorization for Knox
The error was a classpath related issue and I returned back the changed jar
files and now it works fine.
But, once again I put my produced ranger jar files in the server, getting the
same issue since the server can not find other required jar files. It seems
that the problem comes from the fact that my produced jar files are not
compatible with the server's ones.
In the root of my working directory which is user/hdp there are three sub
folders:
3.1.0.0-78
current
share
the current folder has a symbolic link to 3.1.0.0-78.
To produce the jar files I changed the version of the project in the pom.xml
file of the ranger to 1.2.0.3.1.0.0-78 and then packaged files using maven.
After that the generated jar files are copied to the following folders in the
server:
/usr/hdp/3.1.0.0-78/knox/ext/ranger-knox-plugin-impl
/usr/hdp/3.1.0.0-78/ranger-admin/ews/webapp/WEB-INF/classes/ranger-plugins/knox
/usr/hdp/3.1.0.0-78/ranger-knox-plugin/lib/ranger-knox-plugin-impl
but after restarting knox and ranger in Ambari I am getting again the 500 error
saying that can not find some classes.
On Tue, Dec 29, 2020 at 12:16 AM Don Bosco Durai <bo...@apache.org> wrote:
Seems you are doing progress. You should check with the Knox team regarding the
500 error.
Regarding the access not allowed, you will need to put additional debug logs
and see what is coming to your API and make sure the resources are coming
properly.
Bosco
From: Ebrahim Khalil Abbasi <ebrahim.khalil.abb...@gmail.com>
Reply-To: <user@ranger.apache.org>
Date: Monday, December 28, 2020 at 12:00 AM
To: <user@ranger.apache.org>
Subject: Re: Method Level Authorization for Knox
Thanks Bosco,
I find out that the problem was due to missing the ldapRealm.groupSearchBase
attribute in the authentication configuration.
I am facing another problem when enabling the debug for knox plugin. I
appended the following configs to the gateway-log4j file:
ranger.knoxagent.logger=DEBUG,KNOXAGENT
ranger.knoxagent.log.file=ranger.knoxagent.log
log4j.logger.org.apache.ranger=${ranger.knoxagent.logger}
log4j.additivity.org.apache.ranger=false
log4j.appender.KNOXAGENT =org.apache.log4j.RollingFileAppender
log4j.appender.KNOXAGENT.File=${app.log.dir}/${ranger.knoxagent.log.file}
log4j.appender.KNOXAGENT.layout=org.apache.log4j.PatternLayout
log4j.appender.KNOXAGENT.layout.ConversionPattern=%d{ISO8601} %p %c{2}:
%m%n %L
log4j.appender.KNOXAGENT.DatePattern=.yyyy-MM-dd
With the first execution of the curl command I am getting HTTP ERROR 500. The
log in the gateway.log shows that some classes are not found, for example:
com/google/common/base/MoreObjects
With the second execution of the curl command, the ranger.knoxagent.log is
generated but showing that the access is not allowed.
On Thu, Dec 24, 2020 at 3:21 AM Don Bosco Durai <bo...@apache.org> wrote:
Ranger plugin is part of Knox process. So you should configure the logs in Knox.
Seems you are already seeing the logs (from your previous emails), so you are
getting the logs. But some of the logs you are looking for might not be there.
So you might have temporarily add to debug your issue.
Bosco
From: Ebrahim Khalil Abbasi <ebrahim.khalil.abb...@gmail.com>
Reply-To: <user@ranger.apache.org>
Date: Wednesday, December 23, 2020 at 11:22 AM
To: <user@ranger.apache.org>
Subject: Re: Method Level Authorization for Knox
How can I enable ranger's logs? For example, I want to see the logs generated
for the RangerBasePlugin class?
On Wed, Dec 23, 2020 at 9:58 AM Don Bosco Durai <bo...@apache.org> wrote:
Seems Ranger is denying it…
1362020-12-22 12:32:01,432 DEBUG knox.RangerPDPKnoxFilter: Access allowed:
false
You might have to put your own debug statements to print the incoming request.
Generally the resource values might not be the same as defined in the policy.
You can try “*” and give access to group “public” to see if goes through.
Getting audit working will be useful to you for debugging.
Bosco
From: Ebrahim Khalil Abbasi <ebrahim.khalil.abb...@gmail.com>
Reply-To: <user@ranger.apache.org>
Date: Tuesday, December 22, 2020 at 12:51 AM
To: <user@ranger.apache.org>
Subject: Re: Method Level Authorization for Knox
I am getting the following in the ranger.knoxagent.log
1362020-12-22 12:32:01,432 DEBUG knox.RangerPDPKnoxFilter: Access allowed:
false
1602020-12-22 12:32:01,433 DEBUG knoxauth.request: [PERF]
RangerPDPKnoxFilter.doFilter(url=/gateway/dsgdev/livy/v1/sessions,
topologyName=dsgdev): 2
1012020-12-22 12:32:01,438 DEBUG classloader.RangerPluginClassLoader: ==>
RangerPluginClassLoader.deactivate()
2442020-12-22 12:32:01,438 DEBUG classloader.RangerPluginClassLoader: <==
RangerPluginClassLoader.deactivate()
2562020-12-22 12:32:01,439 DEBUG knox.RangerPDPKnoxFilter: <==
RangerPDPKnoxFilter.doFilter()
but nothing in the ranger UI's audit tab.
On Tue, Dec 22, 2020 at 10:16 AM Don Bosco Durai <bo...@apache.org> wrote:
Can you see who is giving 403? Ranger plugin also returns 403 if authorization
fails.
You can check Ranger Audit logs
Check debug logs of Knox
Bosco
From: Ebrahim Khalil Abbasi <ebrahim.khalil.abb...@gmail.com>
Reply-To: <user@ranger.apache.org>
Date: Monday, December 21, 2020 at 6:18 AM
To: <user@ranger.apache.org>
Subject: Re: Method Level Authorization for Knox
I made changes and committed my jar files on the server. For the following
command:
curl -ik -u <user>:<pass>
https://<SERVER>:8443/gateway/dsgdev/livy/v1/sessions
I am getting the 403 status code:
----------------------------------------------
HTTP/1.1 403 Forbidden
Date: Mon, 21 Dec 2020 14:28:17 GMT
Set-Cookie:
KNOXSESSIONID=node01uq9dlfwvh49d1sczysbwid9wt23.node0;Path=/gateway/dsgdev;Secure;HttpOnly
Set-Cookie: rememberMe=deleteMe; Path=/gateway/dsgdev; Max-Age=0; Expires=Sun,
20-Dec-2020 14:28:17 GMT
Cache-Control: must-revalidate,no-cache,no-store
Content-Type: text/html;charset=iso-8859-1
Content-Length: 348
Server: Jetty(9.4.12.v20180830)
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 403 Forbidden</title>
</head>
<body><h2>HTTP ERROR 403</h2>
<p>Problem accessing /gateway/dsgdev/livy/v1/sessions. Reason:
<pre> Forbidden</pre></p><hr><a href="http://eclipse.org/jetty";>Powered by
Jetty:// 9.4.12.v20180830</a><hr/>
</body>
</html>
-----------------------------------------------
In the knox's gateway.log the logged value is the following:
2020-12-21 17:58:17,653 INFO knox.gateway (KnoxLdapRealm.java:getUserDn(692))
- Computed userDn: uid=admin,ou=people,dc=hadoop,dc=apache,dc=org using
dnTemplate for principal: admin
On Sat, Dec 19, 2020 at 10:02 PM Don Bosco Durai <bo...@apache.org> wrote:
If there are no changes to the method signature, then overwriting the existing
jar with the same jar name should work.
Bosco
From: Ebrahim Khalil Abbasi <ebrahim.khalil.abb...@gmail.com>
Reply-To: <user@ranger.apache.org>
Date: Saturday, December 19, 2020 at 10:30 AM
To: <user@ranger.apache.org>
Subject: Re: Method Level Authorization for Knox
Thanks.
I checked the current setup with the LIVYSERVER service and it works fine.
Now, I updated the ranger-knox-plugin module and want to copy the generated jar
file in the server.
I have two questions:
1. Is that ok to copy the jar file to the following directories?
/usr/hdp/current/knox-server/ext/ranger-knox-plugin-impl
/usr/hdp/current/ranger-admin/ews/webapp/WEB-INF/classes/ranger-plugins/knox
2. Is it required to also copy other jar files such as ranger-plugins-common,
ranger-plugins-audit to the server?
Best
On Sun, Dec 13, 2020 at 9:20 PM Don Bosco Durai <bo...@apache.org> wrote:
You seemed to getting SSL errors. I will suggest that you try to get the
default without your customization working.
After that, you can try to putting debug statements at the entry points to make
sure you are extracting and passing everything in the Request object.
I also assume, you have created the ServiceDef properly.
Bosco
From: Ebrahim Khalil Abbasi <ebrahim.khalil.abb...@gmail.com>
Reply-To: <user@ranger.apache.org>
Date: Saturday, December 12, 2020 at 11:50 PM
To: <user@ranger.apache.org>
Subject: Re: Method Level Authorization for Knox
Bosco,
Thanks for your reply.
I followed these steps but could not manage to get it working:
1. I added the GET, DELETE, and POST methods to the service definition's access
type and updated the service definition in the server.
2. In the authorization/knox/KnoxRangerPlugin class two methods (actionType and
accessType) are added which are respectively used in building the action and
access type of the RangerAccessRequest instance.
3. In the RangerPDPKnoxFilter class I extracted the method type from the
received ServletRequest and set it as the access type of the
RangerAccessRequest. The action type is set to 'allow'.
4. The finally packaged jar file is copied to the following paths in the
server:
KNOX_SERVER/ext/ranger-knox-plugin-impl
RANGER_ADMIN/ews/webapp/WEB-INF/classes/ranger-plugins/knox
5. In the Ambari's KNOX service and in the advanced topology config file I set
authorization to XASecurePDPKnox and also added my service to be authorized:
<service>
<role>MY-SERVICE</role>
<url>https://<SERVER>:8443/gateway/dsgdev/livy/v1/sessions</url>
</service>
6.In the Ranger UI and for the dsgdev_knox service I added the policy
myservice with the topology value of default and knox service value of
MY-SERVICE.
For the admin user the DELETE permission is set.
Here are issues I faced:
ٌ
1. When creating the dsgdev_knox service I set the knox.url to
https://<server>:8443/gateway/default/api/v1/topologies the connection test is
failed:
---------------
org.apache.ranger.plugin.client.HadoopException: Exception on REST call to
KnoxUrl : https://master01dev.sic.local:8443/gateway/default/api/v1/topologies..
Exception on REST call to KnoxUrl :
https://master01dev.sic.local:8443/gateway/default/api/v1/topologies..
java.net.SocketException: java.security.NoSuchAlgorithmException: Error
constructing implementation (algorithm: Default, provider: SunJSSE, class:
sun.security.ssl.SSLContextImpl$DefaultSSLContext).
java.security.NoSuchAlgorithmException: Error constructing implementation
(algorithm: Default, provider: SunJSSE, class:
sun.security.ssl.SSLContextImpl$DefaultSSLContext).
Error constructing implementation (algorithm: Default, provider: SunJSSE,
class: sun.security.ssl.SSLContextImpl$DefaultSSLContext).
problem accessing trust store.
Keystore was tampered with, or password was incorrect.
Password verification failed.
---------------
2. When I execute GET on the MY-SERVICE with the admin user since this user
only has DELETE permission, the authorization should be failed. BUT authorized.
3. I also got the LookupUser error so had to comment the overrided
getDefaultRangerPolicies() method in the RangerServiceKnox class.
Sorry for this long description.
Thanks in advance for any help
Ebrahim
On Wed, Dec 9, 2020 at 2:43 PM Don Bosco Durai <bo...@apache.org> wrote:
I reviewed the RangerPDPKnoxFilter code. Since Knox has only one resource
(topology), you will have to the following:
Update the Knox servicedef to add your “model” resource type
You have to decide whether you want hierarchy. E.g. topology -> model, model->
topology or model and topology at the same level
Update RangerPDPKnoxFilter to create the request with what you want to send to
model
I might be missing some steps…
Thanks
Bosco
From: Ebrahim Khalil Abbasi <ebrahim.khalil.abb...@gmail.com>
Reply-To: <user@ranger.apache.org>
Date: Tuesday, December 8, 2020 at 10:24 PM
To: <user@ranger.apache.org>
Subject: Re: Method Level Authorization for Knox
Sorry for typos
Am I on the right way?
On Wed, Dec 9, 2020 at 9:53 AM Ebrahim Khalil Abbasi
<ebrahim.khalil.abb...@gmail.com> wrote:
Thanks Bosco.
What I understand is that the only config we have in the Knox's authorization
interface is just set it to XASecurePDPKnox, then the authorization is
controlled by the Ranger's Knox plugin.
The solution I am working on is to update the knox-agent and the
ranger-knox-plugin-shim modules to support the model level authorization.
Am I on the write way?
Thanks
Ebrahim
On Tue, Dec 8, 2020 at 1:27 PM Don Bosco Durai <bo...@apache.org> wrote:
I think, either will need you to update the Knox’s authorization interface.
Please note, Ranger Plugin just implements the interface provided by the host
process, in this case Knox.
Have you posted the same question the Knox’s mailing list?
Thanks
Bosco
From: Ebrahim Khalil Abbasi <ebrahim.khalil.abb...@gmail.com>
Reply-To: <user@ranger.apache.org>
Date: Tuesday, December 8, 2020 at 1:40 AM
To: <user@ranger.apache.org>
Subject: Re: Method Level Authorization for Knox
There is no suggestion what I should do?
On Wed, Dec 2, 2020 at 9:18 AM Ebrahim Khalil Abbasi
<ebrahim.khalil.abb...@gmail.com> wrote:
Hi,
My problem is there. Some suggested me to change the current existing knox
plugin and improve it to support the method level authorization, so no
integration required. Another proposed solution is to configure the Apache knox
so that in addition to the Ranger's knox pluging also use my HTTP service
plugin in the chain of authorization process. I am not sure the second solution
is easy to implement.
On Wed, Dec 2, 2020 at 5:45 AM Velmurugan Periasamy <v...@apache.org> wrote:
Hi - can you please elaborate on how you are planning to integrate with the
existing Knox plugin?
On Tue, Dec 1, 2020 at 12:16 AM Ebrahim Khalil Abbasi
<ebrahim.khalil.abb...@gmail.com> wrote:
Hi there,
I am using knox to access livy to manage spark sessions. To implement
authorization I want to provide the method level (get/post/delete/...)
authorization. I implemented a new HTTP Service plugin in Ranger but I need to
integrate it to the Ranger's knox plugin so that each HTTP request to the knox
is authorized based on the method by the Ranger.
Thanks for your any help!
Ebrahim
