The error was a classpath related issue and I returned back the changed jar
files and now it works fine.
But, once again I put my produced ranger jar files in the server, getting
the same issue since the server can not find other required jar files. It
seems that the problem comes from the fact that my produced jar files are
not compatible with the server's ones.
In the root of my working directory which is *user*/*hdp *there are three
sub folders:
3.1.0.0-78
current
share
the *current *folder has a symbolic link to 3.1.0.0-78.
To produce the jar files I changed the version of the project in the
pom.xml file of the ranger to 1.2.0.3.1.0.0-78 and then packaged files
using maven.
After that the generated jar files are copied to the following folders in
the server:
*/usr/hdp/3.1.0.0-78/knox/ext/ranger-knox-plugin-impl*
*/usr/hdp/3.1.0.0-78/ranger-admin/ews/webapp/WEB-INF/classes/ranger-plugins/knox*
*/usr/hdp/3.1.0.0-78/ranger-knox-plugin/lib/ranger-knox-plugin-impl*
but after restarting knox and ranger in Ambari I am getting again the 500
error saying that can not find some classes.
On Tue, Dec 29, 2020 at 12:16 AM Don Bosco Durai <bo...@apache.org> wrote:
> Seems you are doing progress. You should check with the Knox team
> regarding the 500 error.
>
>
>
> Regarding the access not allowed, you will need to put additional debug
> logs and see what is coming to your API and make sure the resources are
> coming properly.
>
>
>
> Bosco
>
>
>
>
>
> *From: *Ebrahim Khalil Abbasi <ebrahim.khalil.abb...@gmail.com>
> *Reply-To: *<user@ranger.apache.org>
> *Date: *Monday, December 28, 2020 at 12:00 AM
> *To: *<user@ranger.apache.org>
> *Subject: *Re: Method Level Authorization for Knox
>
>
>
> Thanks Bosco,
>
>
>
> I find out that the problem was due to missing the *ldapRealm.groupSearchBase
> *attribute in the authentication configuration.
>
>
>
> I am facing another problem when enabling the debug for knox plugin. I
> appended the following configs to the gateway-log4j file:
>
> ranger.knoxagent.logger=DEBUG,KNOXAGENT
> ranger.knoxagent.log.file=ranger.knoxagent.log
> log4j.logger.org.apache.ranger=${ranger.knoxagent.logger}
> log4j.additivity.org.apache.ranger=false
> log4j.appender.KNOXAGENT =org.apache.log4j.RollingFileAppender
>
> log4j.appender.KNOXAGENT.File=${app.log.dir}/${ranger.knoxagent.log.file}
> log4j.appender.KNOXAGENT.layout=org.apache.log4j.PatternLayout
> log4j.appender.KNOXAGENT.layout.ConversionPattern=%d{ISO8601} %p
> %c{2}: %m%n %L
> log4j.appender.KNOXAGENT.DatePattern=.yyyy-MM-dd
>
>
>
> With the first execution of the *curl *command I am getting HTTP ERROR
> 500. The log in the gateway.log shows that some classes are not found, for
> example: com/google/common/base/MoreObjects
>
>
>
> With the second execution of the *curl *command,
> the ranger.knoxagent.log is generated but showing that the access is not
> allowed.
>
>
>
>
>
>
>
>
>
> On Thu, Dec 24, 2020 at 3:21 AM Don Bosco Durai <bo...@apache.org> wrote:
>
> Ranger plugin is part of Knox process. So you should configure the logs in
> Knox.
>
>
>
> Seems you are already seeing the logs (from your previous emails), so you
> are getting the logs. But some of the logs you are looking for might not be
> there. So you might have temporarily add to debug your issue.
>
>
>
> Bosco
>
>
>
>
>
> *From: *Ebrahim Khalil Abbasi <ebrahim.khalil.abb...@gmail.com>
> *Reply-To: *<user@ranger.apache.org>
> *Date: *Wednesday, December 23, 2020 at 11:22 AM
> *To: *<user@ranger.apache.org>
> *Subject: *Re: Method Level Authorization for Knox
>
>
>
> How can I enable ranger's logs? For example, I want to see the
> logs generated for the RangerBasePlugin class?
>
>
>
> On Wed, Dec 23, 2020 at 9:58 AM Don Bosco Durai <bo...@apache.org> wrote:
>
> Seems Ranger is denying it…
>
> 1362020-12-22 12:32:01,432 DEBUG knox.RangerPDPKnoxFilter: Access
> allowed: false
>
>
>
> You might have to put your own debug statements to print the incoming
> request. Generally the resource values might not be the same as defined in
> the policy. You can try “*” and give access to group “public” to see if
> goes through.
>
>
>
> Getting audit working will be useful to you for debugging.
>
>
>
> Bosco
>
>
>
>
>
> *From: *Ebrahim Khalil Abbasi <ebrahim.khalil.abb...@gmail.com>
> *Reply-To: *<user@ranger.apache.org>
> *Date: *Tuesday, December 22, 2020 at 12:51 AM
> *To: *<user@ranger.apache.org>
> *Subject: *Re: Method Level Authorization for Knox
>
>
>
> I am getting the following in the ranger.knoxagent.log
>
>
>
> 1362020-12-22 12:32:01,432 DEBUG knox.RangerPDPKnoxFilter: Access
> allowed: false
> 1602020-12-22 12:32:01,433 DEBUG knoxauth.request: [PERF]
> RangerPDPKnoxFilter.doFilter(url=/gateway/dsgdev/livy/v1/sessions,
> topologyName=dsgdev): 2
> 1012020-12-22 12:32:01,438 DEBUG classloader.RangerPluginClassLoader: ==>
> RangerPluginClassLoader.deactivate()
> 2442020-12-22 12:32:01,438 DEBUG classloader.RangerPluginClassLoader: <==
> RangerPluginClassLoader.deactivate()
> 2562020-12-22 12:32:01,439 DEBUG knox.RangerPDPKnoxFilter: <==
> RangerPDPKnoxFilter.doFilter()
>
>
>
> but nothing in the ranger UI's audit tab.
>
>
>
>
>
>
>
> On Tue, Dec 22, 2020 at 10:16 AM Don Bosco Durai <bo...@apache.org> wrote:
>
> Can you see who is giving 403? Ranger plugin also returns 403 if
> authorization fails.
>
>
>
> 1. You can check Ranger Audit logs
> 2. Check debug logs of Knox
>
>
>
> Bosco
>
>
>
>
>
> *From: *Ebrahim Khalil Abbasi <ebrahim.khalil.abb...@gmail.com>
> *Reply-To: *<user@ranger.apache.org>
> *Date: *Monday, December 21, 2020 at 6:18 AM
> *To: *<user@ranger.apache.org>
> *Subject: *Re: Method Level Authorization for Knox
>
>
>
> I made changes and committed my jar files on the server. For the
> following command:
>
> curl -ik -u <user>:<pass> https://
> <SERVER>:8443/gateway/dsgdev/livy/v1/sessions
>
> I am getting the 403 status code:
>
> ----------------------------------------------
>
> HTTP/1.1 403 Forbidden
> Date: Mon, 21 Dec 2020 14:28:17 GMT
> Set-Cookie:
> KNOXSESSIONID=node01uq9dlfwvh49d1sczysbwid9wt23.node0;Path=/gateway/dsgdev;Secure;HttpOnly
> Set-Cookie: rememberMe=deleteMe; Path=/gateway/dsgdev; Max-Age=0;
> Expires=Sun, 20-Dec-2020 14:28:17 GMT
> Cache-Control: must-revalidate,no-cache,no-store
> Content-Type: text/html;charset=iso-8859-1
> Content-Length: 348
> Server: Jetty(9.4.12.v20180830)
>
> <html>
> <head>
> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
> <title>Error 403 Forbidden</title>
> </head>
> <body><h2>HTTP ERROR 403</h2>
> <p>Problem accessing /gateway/dsgdev/livy/v1/sessions. Reason:
> <pre> Forbidden</pre></p><hr><a href="http://eclipse.org/jetty";>Powered
> by Jetty:// 9.4.12.v20180830</a><hr/>
>
> </body>
> </html>
>
> -----------------------------------------------
>
>
>
> In the knox's gateway.log the logged value is the following:
>
> 2020-12-21 17:58:17,653 INFO knox.gateway
> (KnoxLdapRealm.java:getUserDn(692)) - Computed userDn:
> uid=admin,ou=people,dc=hadoop,dc=apache,dc=org using dnTemplate for
> principal: admin
>
>
>
>
>
>
>
>
>
>
>
> On Sat, Dec 19, 2020 at 10:02 PM Don Bosco Durai <bo...@apache.org> wrote:
>
> If there are no changes to the method signature, then overwriting the
> existing jar with the same jar name should work.
>
>
>
> Bosco
>
>
>
>
>
> *From: *Ebrahim Khalil Abbasi <ebrahim.khalil.abb...@gmail.com>
> *Reply-To: *<user@ranger.apache.org>
> *Date: *Saturday, December 19, 2020 at 10:30 AM
> *To: *<user@ranger.apache.org>
> *Subject: *Re: Method Level Authorization for Knox
>
>
>
> Thanks.
>
>
>
> I checked the current setup with the LIVYSERVER service and it works
> fine. Now, I updated the ranger-knox-plugin module and want to copy the
> generated jar file in the server.
>
> I have two questions:
>
> 1. Is that ok to copy the jar file to the following directories?
>
> /usr/hdp/current/knox-server/ext/ranger-knox-plugin-impl
>
>
> /usr/hdp/current/ranger-admin/ews/webapp/WEB-INF/classes/ranger-plugins/knox
>
>
>
> 2. Is it required to also copy other jar files such as
> ranger-plugins-common, ranger-plugins-audit to the server?
>
>
>
> Best
>
>
>
>
>
>
>
>
>
> On Sun, Dec 13, 2020 at 9:20 PM Don Bosco Durai <bo...@apache.org> wrote:
>
> You seemed to getting SSL errors. I will suggest that you try to get the
> default without your customization working.
>
>
>
> After that, you can try to putting debug statements at the entry points to
> make sure you are extracting and passing everything in the Request object.
>
>
>
> I also assume, you have created the ServiceDef properly.
>
>
>
> Bosco
>
>
>
>
>
> *From: *Ebrahim Khalil Abbasi <ebrahim.khalil.abb...@gmail.com>
> *Reply-To: *<user@ranger.apache.org>
> *Date: *Saturday, December 12, 2020 at 11:50 PM
> *To: *<user@ranger.apache.org>
> *Subject: *Re: Method Level Authorization for Knox
>
>
>
> Bosco,
>
>
>
> Thanks for your reply.
>
>
>
> I followed these steps but *could not manage to get it working*:
>
>
>
> 1. I added the GET, DELETE, and POST methods to the service definition's
> access type and updated the service definition in the server.
>
> 2. In the authorization/knox/KnoxRangerPlugin class two methods
> (actionType and accessType) are added which are respectively used in
> building the action and access type of the RangerAccessRequest instance.
>
>
>
> 3. In the RangerPDPKnoxFilter class I extracted the method type from the
> received ServletRequest and set it as the access type of the
> RangerAccessRequest. The action type is set to '*allow*'.
>
>
>
> 4. The finally packaged jar file is copied to the following paths in the
> server:
>
> KNOX_SERVER/ext/ranger-knox-plugin-impl
>
> RANGER_ADMIN/ews/webapp/WEB-INF/classes/ranger-plugins/knox
>
>
>
> 5. In the Ambari's KNOX service and in the advanced topology config file I
> set authorization to XASecurePDPKnox and also added my service to be
> authorized:
>
> <service>
> <role>MY-SERVICE</role>
> <url>https://<SERVER>:8443/gateway/dsgdev/livy/v1/sessions</url>
> </service>
>
>
>
> 6.In the Ranger UI and for the *dsgdev_knox* service I added the policy
> *myservice
> *with the topology value of *default* and knox service value of
> *MY-SERVICE. *
>
> For the *admin *user the *DELETE *permission is set.
>
>
>
>
>
> Here are issues I faced:
>
> ٌ
>
> 1. When creating the dsgdev_knox service I set the knox.url to
> https://<server>:8443/gateway/default/api/v1/topologies
> the connection test is failed:
>
> ---------------
>
> org.apache.ranger.plugin.client.HadoopException: Exception on REST call to
> KnoxUrl :
> https://master01dev.sic.local:8443/gateway/default/api/v1/topologies..
> Exception on REST call to KnoxUrl :
> https://master01dev.sic.local:8443/gateway/default/api/v1/topologies..
> java.net.SocketException: java.security.NoSuchAlgorithmException: Error
> constructing implementation (algorithm: Default, provider: SunJSSE, class:
> sun.security.ssl.SSLContextImpl$DefaultSSLContext).
> java.security.NoSuchAlgorithmException: Error constructing implementation
> (algorithm: Default, provider: SunJSSE, class:
> sun.security.ssl.SSLContextImpl$DefaultSSLContext).
> Error constructing implementation (algorithm: Default, provider: SunJSSE,
> class: sun.security.ssl.SSLContextImpl$DefaultSSLContext).
> problem accessing trust store.
> Keystore was tampered with, or password was incorrect.
> Password verification failed.
>
> ---------------
>
>
>
> 2. When I execute GET on the MY-SERVICE with the admin user since this
> user only has DELETE permission, the authorization should be failed. BUT
> authorized.
>
>
>
> 3. I also got the LookupUser error so had to comment the overrided
> getDefaultRangerPolicies() method in the RangerServiceKnox class.
>
>
>
>
>
> Sorry for this long description.
>
>
>
> Thanks in advance for any help
>
>
>
> Ebrahim
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Wed, Dec 9, 2020 at 2:43 PM Don Bosco Durai <bo...@apache.org> wrote:
>
> I reviewed the RangerPDPKnoxFilter code. Since Knox has only one resource
> (topology), you will have to the following:
>
>
>
> 1. Update the Knox servicedef to add your “model” resource type
> 2. You have to decide whether you want hierarchy. E.g. topology ->
> model, model-> topology or model and topology at the same level
> 3. Update RangerPDPKnoxFilter to create the request with what you want
> to send to model
>
>
>
> I might be missing some steps…
>
>
>
> Thanks
>
>
>
> Bosco
>
>
>
>
>
> *From: *Ebrahim Khalil Abbasi <ebrahim.khalil.abb...@gmail.com>
> *Reply-To: *<user@ranger.apache.org>
> *Date: *Tuesday, December 8, 2020 at 10:24 PM
> *To: *<user@ranger.apache.org>
> *Subject: *Re: Method Level Authorization for Knox
>
>
>
> Sorry for typos
>
>
>
> Am I on the *right *way?
>
>
>
> On Wed, Dec 9, 2020 at 9:53 AM Ebrahim Khalil Abbasi <
> ebrahim.khalil.abb...@gmail.com> wrote:
>
> Thanks Bosco.
>
>
>
> What I understand is that the only config we have in the Knox's
> authorization interface is just set it to XASecurePDPKnox, then the
> authorization is controlled by the Ranger's Knox plugin.
>
>
>
> The solution I am working on is to update the knox-agent and the
> ranger-knox-plugin-shim modules to support the model level authorization.
>
> Am I on the write way?
>
>
>
> Thanks
>
> Ebrahim
>
>
>
>
>
>
>
> On Tue, Dec 8, 2020 at 1:27 PM Don Bosco Durai <bo...@apache.org> wrote:
>
> I think, either will need you to update the Knox’s authorization
> interface. Please note, Ranger Plugin just implements the interface
> provided by the host process, in this case Knox.
>
>
>
> Have you posted the same question the Knox’s mailing list?
>
>
>
> Thanks
>
>
>
> Bosco
>
>
>
>
>
> *From: *Ebrahim Khalil Abbasi <ebrahim.khalil.abb...@gmail.com>
> *Reply-To: *<user@ranger.apache.org>
> *Date: *Tuesday, December 8, 2020 at 1:40 AM
> *To: *<user@ranger.apache.org>
> *Subject: *Re: Method Level Authorization for Knox
>
>
>
> There is no suggestion what I should do?
>
>
>
> On Wed, Dec 2, 2020 at 9:18 AM Ebrahim Khalil Abbasi <
> ebrahim.khalil.abb...@gmail.com> wrote:
>
> Hi,
>
> My problem is there. Some suggested me to change the current existing knox
> plugin and improve it to support the method level authorization, so no
> integration required. Another proposed solution is to configure the Apache
> knox so that in addition to the Ranger's knox pluging also use my HTTP
> service plugin in the chain of authorization process. I am not sure the
> second solution is easy to implement.
>
>
>
>
>
> On Wed, Dec 2, 2020 at 5:45 AM Velmurugan Periasamy <v...@apache.org>
> wrote:
>
> Hi - can you please elaborate on how you are planning to integrate with
> the existing Knox plugin?
>
>
>
> On Tue, Dec 1, 2020 at 12:16 AM Ebrahim Khalil Abbasi <
> ebrahim.khalil.abb...@gmail.com> wrote:
>
> Hi there,
>
> I am using knox to access livy to manage spark sessions. To implement
> authorization I want to provide the method level (get/post/delete/...)
> authorization. I implemented a new HTTP Service plugin in Ranger but I need
> to integrate it to the Ranger's knox plugin so that each HTTP request to
> the knox is authorized based on the method by the Ranger.
>
>
>
> Thanks for your any help!
>
> Ebrahim
>
>
>
>
>
>
>
>
>
>
