Thanks Bosco.
Changing my code base from 3 to 1.2 solved the issue.
On Sun, Jan 3, 2021 at 9:24 PM Don Bosco Durai <bo...@apache.org> wrote:
> I feel, you are hitting build and packaging issues.
>
>
>
> 1. You might want to see the jar file names used by HDP and make sure
> you are replacing exactly those ones with the same names or delete them
> before adding yours.
> 2. You might want to see which jar files has the classes that are
> missing after you add your jar file
> 3. You might also make sure you are compiling with the Ranger 1.2 code
> base
> 4. HDP does some soft linking. So you might have to be careful what
> jar file names are and where you copy them.
> 5. If you feel you are only modifying one or two class files, you
> could also consider replacing the classes directly in the jar file that is
> coming from HDP
>
>
>
> Bosco
>
>
>
>
>
>
>
> *From: *Ebrahim Khalil Abbasi <ebrahim.khalil.abb...@gmail.com>
> *Reply-To: *<user@ranger.apache.org>
> *Date: *Sunday, January 3, 2021 at 2:31 AM
> *To: *<user@ranger.apache.org>
> *Subject: *Re: Method Level Authorization for Knox
>
>
>
> The error was a classpath related issue and I returned back the changed
> jar files and now it works fine.
>
>
>
> But, once again I put my produced ranger jar files in the server, getting
> the same issue since the server can not find other required jar files. It
> seems that the problem comes from the fact that my produced jar files are
> not compatible with the server's ones.
>
>
>
> In the root of my working directory which is *user*/*hdp *there are three
> sub folders:
>
> 3.1.0.0-78
>
> current
>
> share
>
> the *current *folder has a symbolic link to 3.1.0.0-78.
>
>
>
> To produce the jar files I changed the version of the project in the
> pom.xml file of the ranger to 1.2.0.3.1.0.0-78 and then packaged files
> using maven.
>
>
>
> After that the generated jar files are copied to the following folders in
> the server:
>
> */usr/hdp/3.1.0.0-78/knox/ext/ranger-knox-plugin-impl*
>
>
> */usr/hdp/3.1.0.0-78/ranger-admin/ews/webapp/WEB-INF/classes/ranger-plugins/knox*
>
> */usr/hdp/3.1.0.0-78/ranger-knox-plugin/lib/ranger-knox-plugin-impl*
>
>
>
>
>
> but after restarting knox and ranger in Ambari I am getting again the 500
> error saying that can not find some classes.
>
>
>
>
>
>
>
>
>
>
>
> On Tue, Dec 29, 2020 at 12:16 AM Don Bosco Durai <bo...@apache.org> wrote:
>
> Seems you are doing progress. You should check with the Knox team
> regarding the 500 error.
>
>
>
> Regarding the access not allowed, you will need to put additional debug
> logs and see what is coming to your API and make sure the resources are
> coming properly.
>
>
>
> Bosco
>
>
>
>
>
> *From: *Ebrahim Khalil Abbasi <ebrahim.khalil.abb...@gmail.com>
> *Reply-To: *<user@ranger.apache.org>
> *Date: *Monday, December 28, 2020 at 12:00 AM
> *To: *<user@ranger.apache.org>
> *Subject: *Re: Method Level Authorization for Knox
>
>
>
> Thanks Bosco,
>
>
>
> I find out that the problem was due to missing the *ldapRealm.groupSearchBase
> *attribute in the authentication configuration.
>
>
>
> I am facing another problem when enabling the debug for knox plugin. I
> appended the following configs to the gateway-log4j file:
>
> ranger.knoxagent.logger=DEBUG,KNOXAGENT
> ranger.knoxagent.log.file=ranger.knoxagent.log
> log4j.logger.org.apache.ranger=${ranger.knoxagent.logger}
> log4j.additivity.org.apache.ranger=false
> log4j.appender.KNOXAGENT =org.apache.log4j.RollingFileAppender
>
> log4j.appender.KNOXAGENT.File=${app.log.dir}/${ranger.knoxagent.log.file}
> log4j.appender.KNOXAGENT.layout=org.apache.log4j.PatternLayout
> log4j.appender.KNOXAGENT.layout.ConversionPattern=%d{ISO8601} %p
> %c{2}: %m%n %L
> log4j.appender.KNOXAGENT.DatePattern=.yyyy-MM-dd
>
>
>
> With the first execution of the *curl *command I am getting HTTP ERROR
> 500. The log in the gateway.log shows that some classes are not found, for
> example: com/google/common/base/MoreObjects
>
>
>
> With the second execution of the *curl *command,
> the ranger.knoxagent.log is generated but showing that the access is not
> allowed.
>
>
>
>
>
>
>
>
>
> On Thu, Dec 24, 2020 at 3:21 AM Don Bosco Durai <bo...@apache.org> wrote:
>
> Ranger plugin is part of Knox process. So you should configure the logs in
> Knox.
>
>
>
> Seems you are already seeing the logs (from your previous emails), so you
> are getting the logs. But some of the logs you are looking for might not be
> there. So you might have temporarily add to debug your issue.
>
>
>
> Bosco
>
>
>
>
>
> *From: *Ebrahim Khalil Abbasi <ebrahim.khalil.abb...@gmail.com>
> *Reply-To: *<user@ranger.apache.org>
> *Date: *Wednesday, December 23, 2020 at 11:22 AM
> *To: *<user@ranger.apache.org>
> *Subject: *Re: Method Level Authorization for Knox
>
>
>
> How can I enable ranger's logs? For example, I want to see the
> logs generated for the RangerBasePlugin class?
>
>
>
> On Wed, Dec 23, 2020 at 9:58 AM Don Bosco Durai <bo...@apache.org> wrote:
>
> Seems Ranger is denying it…
>
> 1362020-12-22 12:32:01,432 DEBUG knox.RangerPDPKnoxFilter: Access
> allowed: false
>
>
>
> You might have to put your own debug statements to print the incoming
> request. Generally the resource values might not be the same as defined in
> the policy. You can try “*” and give access to group “public” to see if
> goes through.
>
>
>
> Getting audit working will be useful to you for debugging.
>
>
>
> Bosco
>
>
>
>
>
> *From: *Ebrahim Khalil Abbasi <ebrahim.khalil.abb...@gmail.com>
> *Reply-To: *<user@ranger.apache.org>
> *Date: *Tuesday, December 22, 2020 at 12:51 AM
> *To: *<user@ranger.apache.org>
> *Subject: *Re: Method Level Authorization for Knox
>
>
>
> I am getting the following in the ranger.knoxagent.log
>
>
>
> 1362020-12-22 12:32:01,432 DEBUG knox.RangerPDPKnoxFilter: Access
> allowed: false
> 1602020-12-22 12:32:01,433 DEBUG knoxauth.request: [PERF]
> RangerPDPKnoxFilter.doFilter(url=/gateway/dsgdev/livy/v1/sessions,
> topologyName=dsgdev): 2
> 1012020-12-22 12:32:01,438 DEBUG classloader.RangerPluginClassLoader: ==>
> RangerPluginClassLoader.deactivate()
> 2442020-12-22 12:32:01,438 DEBUG classloader.RangerPluginClassLoader: <==
> RangerPluginClassLoader.deactivate()
> 2562020-12-22 12:32:01,439 DEBUG knox.RangerPDPKnoxFilter: <==
> RangerPDPKnoxFilter.doFilter()
>
>
>
> but nothing in the ranger UI's audit tab.
>
>
>
>
>
>
>
> On Tue, Dec 22, 2020 at 10:16 AM Don Bosco Durai <bo...@apache.org> wrote:
>
> Can you see who is giving 403? Ranger plugin also returns 403 if
> authorization fails.
>
>
>
> 1. You can check Ranger Audit logs
> 2. Check debug logs of Knox
>
>
>
> Bosco
>
>
>
>
>
> *From: *Ebrahim Khalil Abbasi <ebrahim.khalil.abb...@gmail.com>
> *Reply-To: *<user@ranger.apache.org>
> *Date: *Monday, December 21, 2020 at 6:18 AM
> *To: *<user@ranger.apache.org>
> *Subject: *Re: Method Level Authorization for Knox
>
>
>
> I made changes and committed my jar files on the server. For the
> following command:
>
> curl -ik -u <user>:<pass> https://
> <SERVER>:8443/gateway/dsgdev/livy/v1/sessions
>
> I am getting the 403 status code:
>
> ----------------------------------------------
>
> HTTP/1.1 403 Forbidden
> Date: Mon, 21 Dec 2020 14:28:17 GMT
> Set-Cookie:
> KNOXSESSIONID=node01uq9dlfwvh49d1sczysbwid9wt23.node0;Path=/gateway/dsgdev;Secure;HttpOnly
> Set-Cookie: rememberMe=deleteMe; Path=/gateway/dsgdev; Max-Age=0;
> Expires=Sun, 20-Dec-2020 14:28:17 GMT
> Cache-Control: must-revalidate,no-cache,no-store
> Content-Type: text/html;charset=iso-8859-1
> Content-Length: 348
> Server: Jetty(9.4.12.v20180830)
>
> <html>
> <head>
> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
> <title>Error 403 Forbidden</title>
> </head>
> <body><h2>HTTP ERROR 403</h2>
> <p>Problem accessing /gateway/dsgdev/livy/v1/sessions. Reason:
> <pre> Forbidden</pre></p><hr><a href="http://eclipse.org/jetty";>Powered
> by Jetty:// 9.4.12.v20180830</a><hr/>
>
> </body>
> </html>
>
> -----------------------------------------------
>
>
>
> In the knox's gateway.log the logged value is the following:
>
> 2020-12-21 17:58:17,653 INFO knox.gateway
> (KnoxLdapRealm.java:getUserDn(692)) - Computed userDn:
> uid=admin,ou=people,dc=hadoop,dc=apache,dc=org using dnTemplate for
> principal: admin
>
>
>
>
>
>
>
>
>
>
>
> On Sat, Dec 19, 2020 at 10:02 PM Don Bosco Durai <bo...@apache.org> wrote:
>
> If there are no changes to the method signature, then overwriting the
> existing jar with the same jar name should work.
>
>
>
> Bosco
>
>
>
>
>
> *From: *Ebrahim Khalil Abbasi <ebrahim.khalil.abb...@gmail.com>
> *Reply-To: *<user@ranger.apache.org>
> *Date: *Saturday, December 19, 2020 at 10:30 AM
> *To: *<user@ranger.apache.org>
> *Subject: *Re: Method Level Authorization for Knox
>
>
>
> Thanks.
>
>
>
> I checked the current setup with the LIVYSERVER service and it works
> fine. Now, I updated the ranger-knox-plugin module and want to copy the
> generated jar file in the server.
>
> I have two questions:
>
> 1. Is that ok to copy the jar file to the following directories?
>
> /usr/hdp/current/knox-server/ext/ranger-knox-plugin-impl
>
>
> /usr/hdp/current/ranger-admin/ews/webapp/WEB-INF/classes/ranger-plugins/knox
>
>
>
> 2. Is it required to also copy other jar files such as
> ranger-plugins-common, ranger-plugins-audit to the server?
>
>
>
> Best
>
>
>
>
>
>
>
>
>
> On Sun, Dec 13, 2020 at 9:20 PM Don Bosco Durai <bo...@apache.org> wrote:
>
> You seemed to getting SSL errors. I will suggest that you try to get the
> default without your customization working.
>
>
>
> After that, you can try to putting debug statements at the entry points to
> make sure you are extracting and passing everything in the Request object.
>
>
>
> I also assume, you have created the ServiceDef properly.
>
>
>
> Bosco
>
>
>
>
>
> *From: *Ebrahim Khalil Abbasi <ebrahim.khalil.abb...@gmail.com>
> *Reply-To: *<user@ranger.apache.org>
> *Date: *Saturday, December 12, 2020 at 11:50 PM
> *To: *<user@ranger.apache.org>
> *Subject: *Re: Method Level Authorization for Knox
>
>
>
> Bosco,
>
>
>
> Thanks for your reply.
>
>
>
> I followed these steps but *could not manage to get it working*:
>
>
>
> 1. I added the GET, DELETE, and POST methods to the service definition's
> access type and updated the service definition in the server.
>
> 2. In the authorization/knox/KnoxRangerPlugin class two methods
> (actionType and accessType) are added which are respectively used in
> building the action and access type of the RangerAccessRequest instance.
>
>
>
> 3. In the RangerPDPKnoxFilter class I extracted the method type from the
> received ServletRequest and set it as the access type of the
> RangerAccessRequest. The action type is set to '*allow*'.
>
>
>
> 4. The finally packaged jar file is copied to the following paths in the
> server:
>
> KNOX_SERVER/ext/ranger-knox-plugin-impl
>
> RANGER_ADMIN/ews/webapp/WEB-INF/classes/ranger-plugins/knox
>
>
>
> 5. In the Ambari's KNOX service and in the advanced topology config file I
> set authorization to XASecurePDPKnox and also added my service to be
> authorized:
>
> <service>
> <role>MY-SERVICE</role>
> <url>https://<SERVER>:8443/gateway/dsgdev/livy/v1/sessions</url>
> </service>
>
>
>
> 6.In the Ranger UI and for the *dsgdev_knox* service I added the policy
> *myservice
> *with the topology value of *default* and knox service value of
> *MY-SERVICE. *
>
> For the *admin *user the *DELETE *permission is set.
>
>
>
>
>
> Here are issues I faced:
>
> ٌ
>
> 1. When creating the dsgdev_knox service I set the knox.url to
> https://<server>:8443/gateway/default/api/v1/topologies
> the connection test is failed:
>
> ---------------
>
> org.apache.ranger.plugin.client.HadoopException: Exception on REST call to
> KnoxUrl :
> https://master01dev.sic.local:8443/gateway/default/api/v1/topologies..
> Exception on REST call to KnoxUrl :
> https://master01dev.sic.local:8443/gateway/default/api/v1/topologies..
> java.net.SocketException: java.security.NoSuchAlgorithmException: Error
> constructing implementation (algorithm: Default, provider: SunJSSE, class:
> sun.security.ssl.SSLContextImpl$DefaultSSLContext).
> java.security.NoSuchAlgorithmException: Error constructing implementation
> (algorithm: Default, provider: SunJSSE, class:
> sun.security.ssl.SSLContextImpl$DefaultSSLContext).
> Error constructing implementation (algorithm: Default, provider: SunJSSE,
> class: sun.security.ssl.SSLContextImpl$DefaultSSLContext).
> problem accessing trust store.
> Keystore was tampered with, or password was incorrect.
> Password verification failed.
>
> ---------------
>
>
>
> 2. When I execute GET on the MY-SERVICE with the admin user since this
> user only has DELETE permission, the authorization should be failed. BUT
> authorized.
>
>
>
> 3. I also got the LookupUser error so had to comment the overrided
> getDefaultRangerPolicies() method in the RangerServiceKnox class.
>
>
>
>
>
> Sorry for this long description.
>
>
>
> Thanks in advance for any help
>
>
>
> Ebrahim
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Wed, Dec 9, 2020 at 2:43 PM Don Bosco Durai <bo...@apache.org> wrote:
>
> I reviewed the RangerPDPKnoxFilter code. Since Knox has only one resource
> (topology), you will have to the following:
>
>
>
> 1. Update the Knox servicedef to add your “model” resource type
> 2. You have to decide whether you want hierarchy. E.g. topology ->
> model, model-> topology or model and topology at the same level
> 3. Update RangerPDPKnoxFilter to create the request with what you want
> to send to model
>
>
>
> I might be missing some steps…
>
>
>
> Thanks
>
>
>
> Bosco
>
>
>
>
>
> *From: *Ebrahim Khalil Abbasi <ebrahim.khalil.abb...@gmail.com>
> *Reply-To: *<user@ranger.apache.org>
> *Date: *Tuesday, December 8, 2020 at 10:24 PM
> *To: *<user@ranger.apache.org>
> *Subject: *Re: Method Level Authorization for Knox
>
>
>
> Sorry for typos
>
>
>
> Am I on the *right *way?
>
>
>
> On Wed, Dec 9, 2020 at 9:53 AM Ebrahim Khalil Abbasi <
> ebrahim.khalil.abb...@gmail.com> wrote:
>
> Thanks Bosco.
>
>
>
> What I understand is that the only config we have in the Knox's
> authorization interface is just set it to XASecurePDPKnox, then the
> authorization is controlled by the Ranger's Knox plugin.
>
>
>
> The solution I am working on is to update the knox-agent and the
> ranger-knox-plugin-shim modules to support the model level authorization.
>
> Am I on the write way?
>
>
>
> Thanks
>
> Ebrahim
>
>
>
>
>
>
>
> On Tue, Dec 8, 2020 at 1:27 PM Don Bosco Durai <bo...@apache.org> wrote:
>
> I think, either will need you to update the Knox’s authorization
> interface. Please note, Ranger Plugin just implements the interface
> provided by the host process, in this case Knox.
>
>
>
> Have you posted the same question the Knox’s mailing list?
>
>
>
> Thanks
>
>
>
> Bosco
>
>
>
>
>
> *From: *Ebrahim Khalil Abbasi <ebrahim.khalil.abb...@gmail.com>
> *Reply-To: *<user@ranger.apache.org>
> *Date: *Tuesday, December 8, 2020 at 1:40 AM
> *To: *<user@ranger.apache.org>
> *Subject: *Re: Method Level Authorization for Knox
>
>
>
> There is no suggestion what I should do?
>
>
>
> On Wed, Dec 2, 2020 at 9:18 AM Ebrahim Khalil Abbasi <
> ebrahim.khalil.abb...@gmail.com> wrote:
>
> Hi,
>
> My problem is there. Some suggested me to change the current existing knox
> plugin and improve it to support the method level authorization, so no
> integration required. Another proposed solution is to configure the Apache
> knox so that in addition to the Ranger's knox pluging also use my HTTP
> service plugin in the chain of authorization process. I am not sure the
> second solution is easy to implement.
>
>
>
>
>
> On Wed, Dec 2, 2020 at 5:45 AM Velmurugan Periasamy <v...@apache.org>
> wrote:
>
> Hi - can you please elaborate on how you are planning to integrate with
> the existing Knox plugin?
>
>
>
> On Tue, Dec 1, 2020 at 12:16 AM Ebrahim Khalil Abbasi <
> ebrahim.khalil.abb...@gmail.com> wrote:
>
> Hi there,
>
> I am using knox to access livy to manage spark sessions. To implement
> authorization I want to provide the method level (get/post/delete/...)
> authorization. I implemented a new HTTP Service plugin in Ranger but I need
> to integrate it to the Ranger's knox plugin so that each HTTP request to
> the knox is authorized based on the method by the Ranger.
>
>
>
> Thanks for your any help!
>
> Ebrahim
>
>
>
>
>
>
>
>
>
>
