Re: Minimizing build time for admin server only

[Nigel Jones]

On 2019/07/01 10:13:07, Nigel Jones  wrote: 
> I'm building an image of ranger to support simple test cases with a custom 
> plugin. The ranger code is unmodified.
If anyone would like to look/comment, the image I'm creating is defined at 
https://github.com/odpi/egeria/blob/master/open-metadata-resources/open-metadata-deployment/docker/ranger/Dockerfile
 - I won't pretend it's brilliant and smart, just barely 'sufficient' for my 
needs ;-)

Minimizing build time for admin server only

[Nigel Jones]

I'm building an image of ranger to support simple test cases with a custom 
plugin. The ranger code is unmodified.

Currently I build using
mvn -Pall -DskipTests=true clean compile package install assembly:assembly

All I am currently using is 
ranger-*-admin.tar.gz

which I then configure (via script) use alongside solr in docker container and 
mariadb (other container) 

However the build time is long, so can anyone recommend any of
 * A maintained, built base docker container
 * build maven artifacts (set of) which contain all needed runtime components 
which could be used to create an image
 * Optimizations to the maven invocation to not build all the plugins etc

Thanks
Nigel Jones 

Ranger & maven artifacts : Docker image, admin server runtime

[Nigel Jones]

I need to use Apache Ranger to help demo another open source project (ODPi 
Egeria).

So far I have been building my own docker image by extracting the code from git 
within the image, compiling, configuring, & running security admin alongside 
mariadb. This image is then deployed within a helm chart into a kubernetes 
cluster to allow us to demo some integration.

I needed to add another component (usersync to ldap) to an image I mostly setup 
a year and a half ago, so I considered another approach. Rather than incur a 
long build time when I'm generating the docker image, I had an idea... could I 
make use of official, open, existing built components?

I tried to download the right maven artifacts for the server (choosing 
usersync, ldapconfigcheck & security-admin-web), thinking I could use those. 
However I noticed various things I needed - for example 
security-admin/contrib/solr_for_audit_setup wasn't in the security-admin-web 
artifact, but is needed to configure it. I'm not sure it's in any maven 
artifact?

I'm suspecting I need to abandon this avenue. Before I do, has anyone tried 
this? Do you think it would work? I think it's a nice idea, but it is dependent 
on the maven artifacts having been setup to support this approach (ie to 
include utility scripts etc not just the core jars)

I don't need to keep rebuilding the ranger image, so it's not a big deal, just 
testing the water :-) I can revert to just building from source.

Additionally, has anyone else worked through reuseable docker images for ranger 
? As I mentioned I do have something that works, but I admit it's rather hack-y 
& somewhat tuned to my specific needs ...  See 
https://github.com/odpi/egeria/blob/master/open-metadata-resources/open-metadata-deployment/docker/ranger/Dockerfile

Many thanks
nigel.

[jira] [Resolved] (RANGER-1488) Create Ranger plugin for gaiandb

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones resolved RANGER-1488.
-
Resolution: Won't Do

> Create Ranger plugin for gaiandb
> 
>
> Key: RANGER-1488
> URL: https://issues.apache.org/jira/browse/RANGER-1488
> Project: Ranger
>  Issue Type: New Feature
>  Components: plugins
>    Reporter: Nigel Jones
>    Assignee: Nigel Jones
>Priority: Major
>  Labels: VirtualDataConnector
> Attachments: GaianDebuggingResearch.docx
>
>
> GaianDB is a distributed, federated database built on Apache Derby.
> The documentation is at 
> https://github.com/gaiandb/gaiandb/blob/master/README.md - navigate up for 
> the source.
> As part of a Virtual Connector Project based on Atlas we are using gaianDB to 
> provide a virtualization layer. We need to control access to underlying 
> resources and will be building a Ranger plugin for gaiandb to support this. 
> GaianDB already has support for a form of policy plugin which allows the SQL 
> to be intercepted.
> It is unclear if this code will sit
>  - external to ranger/atlas ie in the gaianDB sources
>  - as a sample in the Atlas project (so that we collate what is needed for 
> the virtual connector project)
>  - as a sample, or component in the Ranger project
> However it seems helpful to describe the proposal here and get 
> feedback/dialogue in the ranger community :-)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (RANGER-1488) Create Ranger plugin for gaiandb

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16741249#comment-16741249
 ] 

Nigel Jones commented on RANGER-1488:
-

The gaian plugin is now to be found in the ODPi Egeria project.

 

See 
[https://github.com/odpi/egeria/tree/master/open-metadata-implementation/adapters/governance-engines-plugins/gaian-ranger-plugin]
 for the plugin, as well as 
[https://github.com/odpi/egeria/tree/master/open-metadata-implementation/adapters/authentication-plugins/gaian-impersonation]

for impersonation support

I will close this Jira as no further activity is expected here in the near term.

> Create Ranger plugin for gaiandb
> 
>
> Key: RANGER-1488
> URL: https://issues.apache.org/jira/browse/RANGER-1488
> Project: Ranger
>  Issue Type: New Feature
>  Components: plugins
>    Reporter: Nigel Jones
>    Assignee: Nigel Jones
>Priority: Major
>  Labels: VirtualDataConnector
> Attachments: GaianDebuggingResearch.docx
>
>
> GaianDB is a distributed, federated database built on Apache Derby.
> The documentation is at 
> https://github.com/gaiandb/gaiandb/blob/master/README.md - navigate up for 
> the source.
> As part of a Virtual Connector Project based on Atlas we are using gaianDB to 
> provide a virtualization layer. We need to control access to underlying 
> resources and will be building a Ranger plugin for gaiandb to support this. 
> GaianDB already has support for a form of policy plugin which allows the SQL 
> to be intercepted.
> It is unclear if this code will sit
>  - external to ranger/atlas ie in the gaianDB sources
>  - as a sample in the Atlas project (so that we collate what is needed for 
> the virtual connector project)
>  - as a sample, or component in the Ranger project
> However it seems helpful to describe the proposal here and get 
> feedback/dialogue in the ranger community :-)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (RANGER-2254) Ranger haven't building by "build_ranger_using_docker.sh"

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-2254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686981#comment-16686981
 ] 

Nigel Jones commented on RANGER-2254:
-

If it helps, my current docker scripts which botj build ranger, and then go 
onto configuring it (in a very basic way, with admin server only) are at

[https://github.com/odpi/egeria/tree/master/open-metadata-resources/open-metadata-deployment/docker/ranger]

 

The intent is to automate deployment of a demo scenario using the egeria open 
metadata project we're working on (which also will use atlas). I was hoping to 
look at how we could add some ranger docker images that could be deployed for 
use in future, but I'm still in the stages of getting the basics working. This 
is now focussed around using k8s too.

 

Anyway I struggled a little recently with some of the build issues around 
packaging, but the above script now does seem to work - so the first part may 
be useful for you?

 

Nigel.

> Ranger haven't building by "build_ranger_using_docker.sh"
> -
>
> Key: RANGER-2254
> URL: https://issues.apache.org/jira/browse/RANGER-2254
> Project: Ranger
>  Issue Type: Bug
>  Components: audit, Ranger
>Affects Versions: 1.2.0
> Environment: docker
>Reporter: Leonid
>Priority: Major
>  Labels: build
> Fix For: 1.2.0
>
> Attachments: first.log, second.log
>
>
> I have tried to build Ranger 1.2.0 with 
> {code:java}
> sh ./build_ranger_using_docker.sh mvn clean compile package install 
> assembly:assembly
> {code}
> and had gotten this:
> {code:java}
> [ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.11:check 
> (default) on project ranger: Too many files with unapproved license: 22 See 
> RAT report in: /ranger/target/rat.txt -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
> goal org.apache.rat:apache-rat-plugin:0.11:check (default) on project ranger: 
> Too many files with unapproved license: 22 See RAT report in: 
> /ranger/target/rat.txt
> {code}
> if I have tried use *_-Drat.skip=true_* I had gotten many errors in Audit 
> Component like *_error: cannot find symbol_*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (RANGER-1699) gaiandb: Create initial plugin configuration file that can be deployed to ranger

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones closed RANGER-1699.
---

> gaiandb: Create initial plugin configuration file that can be deployed to 
> ranger
> 
>
> Key: RANGER-1699
> URL: https://issues.apache.org/jira/browse/RANGER-1699
> Project: Ranger
>  Issue Type: Sub-task
>  Components: plugins
>    Reporter: Nigel Jones
>Priority: Major
> Attachments: 0001-adding-gaiandb-serviceDef.patch, 
> ranger-servicedef-gaian.json
>
>
> Create initial configuration file for the gaiandb plugin for ranger which 
> will define policies, resources.
> Additional jiras will be opened to add extra capabilities later on. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (RANGER-1701) gaiandb: simple resource based policy

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones closed RANGER-1701.
---

> gaiandb: simple resource based policy
> -
>
> Key: RANGER-1701
> URL: https://issues.apache.org/jira/browse/RANGER-1701
> Project: Ranger
>  Issue Type: Sub-task
>  Components: plugins
>    Reporter: Nigel Jones
>Priority: Major
>
> Get the gaiandb plugin to a state where a simple resource based policy is 
> operational (permit/deny), and the ranger gui can be used to define.. and the 
> plugin will then implement this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (RANGER-1703) gaiandb - add data masking (similar to hive) for resource based policy

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones closed RANGER-1703.
---

> gaiandb - add data masking (similar to hive) for resource based policy
> --
>
> Key: RANGER-1703
> URL: https://issues.apache.org/jira/browse/RANGER-1703
> Project: Ranger
>  Issue Type: Sub-task
>  Components: plugins
>    Reporter: Nigel Jones
>Priority: Major
>
> Support data masking in gaindb plugin. Initially implement for a resource 
> based policy



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (RANGER-1909) Build failure in Kafka Security plugin Unit Tests

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones closed RANGER-1909.
---

> Build failure in Kafka Security plugin Unit Tests
> -
>
> Key: RANGER-1909
> URL: https://issues.apache.org/jira/browse/RANGER-1909
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>    Reporter: Nigel Jones
>Priority: Major
> Attachments: ranger-20171124a.log
>
>
> Java 1.8 (151, openjdk)
> Maven 3.5.0
> Build fails at
> [INFO] KAFKA Security Plugin .. FAILURE [22:07 
> min]
> [INFO] Running 
> org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerGSSTest
> log4j:WARN No appenders could be found for logger 
> (org.apache.kerby.kerberos.kerb.identity.backend.AbstractIdentityBackend).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 
> 1,120.51 s <<< FAILURE! - in 
> org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerGSSTest
> [ERROR] 
> testAuthorizedRead(org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerGSSTest)
>   Time elapsed: 1,109.032 s  <<< FAILURE!
> java.lang.AssertionError
>   at 
> org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerGSSTest.testAuthorizedRead(KafkaRangerAuthorizerGSSTest.java:243)
> [INFO] Running 
> org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerTest
> log4j:WARN No appenders could be found for logger 
> (org.apache.zookeeper.server.ZooKeeperServerMain).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 
> 114.827 s - in 
> org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerTest
> [INFO] Running 
> org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerSASLSSLTest
> log4j:WARN No appenders could be found for logger 
> (org.apache.zookeeper.server.ZooKeeperServerMain).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 72.214 
> s - in 
> org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerSASLSSLTest
> [INFO] 
> [INFO] Results:
> [INFO] 
> [ERROR] Failures: 
> [ERROR]   KafkaRangerAuthorizerGSSTest.testAuthorizedRead:243
> [INFO] 
> [ERROR] Tests run: 8, Failures: 1, Errors: 0, Skipped: 0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (RANGER-1850) Impersonation/proxy user support for gaiandb ranger plugin

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones closed RANGER-1850.
---

> Impersonation/proxy user support for gaiandb ranger plugin
> --
>
> Key: RANGER-1850
> URL: https://issues.apache.org/jira/browse/RANGER-1850
> Project: Ranger
>  Issue Type: Sub-task
>  Components: plugins
>    Reporter: Nigel Jones
>Priority: Major
> Attachments: GaianDBAuth.docx
>
>
> Applications/users could connect to gaianDB using their own authentication 
> information - for example userid/password in the simple case. Here the ranger 
> plugin will use that id for policy checks.
> However in a multi tiered architecture a service id (aka non personal 
> account) may be used, and somehow the user to be impersonated is passed via 
> an additional property. This has a number of implications to the system 
> configuration, derby/gaiandb configuration & the plugin implementation. 
> Opening this Jira as a placeholder and will add a document soon (++days) on 
> the same to capture some of the discussion around this area in recent days.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (RANGER-1850) Impersonation/proxy user support for gaiandb ranger plugin

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones resolved RANGER-1850.
-
Resolution: Fixed

> Impersonation/proxy user support for gaiandb ranger plugin
> --
>
> Key: RANGER-1850
> URL: https://issues.apache.org/jira/browse/RANGER-1850
> Project: Ranger
>  Issue Type: Sub-task
>  Components: plugins
>    Reporter: Nigel Jones
>Priority: Major
> Attachments: GaianDBAuth.docx
>
>
> Applications/users could connect to gaianDB using their own authentication 
> information - for example userid/password in the simple case. Here the ranger 
> plugin will use that id for policy checks.
> However in a multi tiered architecture a service id (aka non personal 
> account) may be used, and somehow the user to be impersonated is passed via 
> an additional property. This has a number of implications to the system 
> configuration, derby/gaiandb configuration & the plugin implementation. 
> Opening this Jira as a placeholder and will add a document soon (++days) on 
> the same to capture some of the discussion around this area in recent days.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (RANGER-1702) gaiandb: tag based policies

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones reopened RANGER-1702:
-

> gaiandb: tag based policies
> ---
>
> Key: RANGER-1702
> URL: https://issues.apache.org/jira/browse/RANGER-1702
> Project: Ranger
>  Issue Type: Sub-task
>  Components: plugins
>    Reporter: Nigel Jones
>Priority: Major
>
> Update plugin to allow tag based policies to be used for simple permit/deny 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (RANGER-1702) gaiandb: tag based policies

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones resolved RANGER-1702.
-
Resolution: Fixed

> gaiandb: tag based policies
> ---
>
> Key: RANGER-1702
> URL: https://issues.apache.org/jira/browse/RANGER-1702
> Project: Ranger
>  Issue Type: Sub-task
>  Components: plugins
>    Reporter: Nigel Jones
>Priority: Major
>
> Update plugin to allow tag based policies to be used for simple permit/deny 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (RANGER-1703) gaiandb - add data masking (similar to hive) for resource based policy

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones resolved RANGER-1703.
-
Resolution: Fixed

> gaiandb - add data masking (similar to hive) for resource based policy
> --
>
> Key: RANGER-1703
> URL: https://issues.apache.org/jira/browse/RANGER-1703
> Project: Ranger
>  Issue Type: Sub-task
>  Components: plugins
>    Reporter: Nigel Jones
>Priority: Major
>
> Support data masking in gaindb plugin. Initially implement for a resource 
> based policy



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (RANGER-1700) gaiandb: Integrate with ranger libraries - plugin builds, initializes

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones resolved RANGER-1700.
-
Resolution: Fixed

> gaiandb: Integrate with ranger libraries - plugin builds, initializes
> -
>
> Key: RANGER-1700
> URL: https://issues.apache.org/jira/browse/RANGER-1700
> Project: Ranger
>  Issue Type: Sub-task
>  Components: plugins
>    Reporter: Nigel Jones
>Priority: Major
> Attachments: 0001-adding-gaianDB-connection-and-resource-manage.patch
>
>
> Get the plugin to a state where it will initialize with a simple 
> configuration and can be deployed to gaianDB. It may not actually action any 
> policies at this state



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (RANGER-1699) gaiandb: Create initial plugin configuration file that can be deployed to ranger

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones resolved RANGER-1699.
-
Resolution: Fixed

> gaiandb: Create initial plugin configuration file that can be deployed to 
> ranger
> 
>
> Key: RANGER-1699
> URL: https://issues.apache.org/jira/browse/RANGER-1699
> Project: Ranger
>  Issue Type: Sub-task
>  Components: plugins
>    Reporter: Nigel Jones
>Priority: Major
> Attachments: 0001-adding-gaiandb-serviceDef.patch, 
> ranger-servicedef-gaian.json
>
>
> Create initial configuration file for the gaiandb plugin for ranger which 
> will define policies, resources.
> Additional jiras will be opened to add extra capabilities later on. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (RANGER-1701) gaiandb: simple resource based policy

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones resolved RANGER-1701.
-

> gaiandb: simple resource based policy
> -
>
> Key: RANGER-1701
> URL: https://issues.apache.org/jira/browse/RANGER-1701
> Project: Ranger
>  Issue Type: Sub-task
>  Components: plugins
>    Reporter: Nigel Jones
>Priority: Major
>
> Get the gaiandb plugin to a state where a simple resource based policy is 
> operational (permit/deny), and the ranger gui can be used to define.. and the 
> plugin will then implement this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (RANGER-1186) Ranger Source: eclipse

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones closed RANGER-1186.
---

> Ranger Source: eclipse
> --
>
> Key: RANGER-1186
> URL: https://issues.apache.org/jira/browse/RANGER-1186
> Project: Ranger
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 0.7.0
> Environment: eclipse neon, ubuntu 16.10
>Reporter: Nigel Jones
>Assignee: Pradeep Agrawal
>Priority: Major
>  Labels: VirtualDataConnector
> Attachments: RANGER-1186_Ranger_Source_Eclipse.pdf, ranger_me0.PNG, 
> ranger_me1.PNG
>
>
> I can happily build ranger in my Ubuntu 16.10 environment with
>  - open jdk 1.8.0_102
>  - maven 3.3.9
> I also have eclipse neon installed in this environment
> The build/setup instructions at 
> http://ranger.apache.org/quick_start_guide.html have minimal instructions for 
> eclipse.
> I imported the top level ranger directory using the m2e plugin as provided in 
> neon.
> This failed with numerous errors
> Is anyone actively building with eclipse? If so can we update the website 
> with current setup information?
> If not I'll use this JIRA to record the issues & hope to address them & 
> update the docs ;-)
> Screenshots to follow



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (RANGER-1331) Ranger build for windows not working

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones resolved RANGER-1331.
-
Resolution: Cannot Reproduce

> Ranger build for windows not working
> 
>
> Key: RANGER-1331
> URL: https://issues.apache.org/jira/browse/RANGER-1331
> Project: Ranger
>  Issue Type: Improvement
>        Reporter: Nigel Jones
>Priority: Major
> Attachments: RangerWinBuild1.log
>
>
> The quick start guide at http://ranger.apache.org/quick_start_guide.html has 
> instuctions to build ranger which work fine on Ubuntu 16.04/openjdk 8 / maven 
> 3.3.9
> However I was interested in setting up a more productive IDE environment, and 
> given my laptop is running Windows 10 as the primary OS decided to follow the 
> instructions for using eclipse
> HOWEVER unlike Ranger-1186 I tried this on *WINDOWS* rather than linux
> Is this something anyone is doing successfully (even if skipping building 
> some of the linux native auth pieces)?
> I then tried the build but it failed quickly on credential support
> [INFO] ranger . SUCCESS [  9.332 
> s]
> [INFO] Jdbc SQL Connector . SUCCESS [  4.569 
> s]
> [INFO] Credential Support . FAILURE [  9.579 
> s]
> Including full log
> Could we list on that page the OSs people have had success with?
> Are most using linux?
> Anyone tried MacOS?
> NOTE: edited from original to focus on the OS question rather than IDE



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (RANGER-1331) Ranger build for windows not working

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones closed RANGER-1331.
---

> Ranger build for windows not working
> 
>
> Key: RANGER-1331
> URL: https://issues.apache.org/jira/browse/RANGER-1331
> Project: Ranger
>  Issue Type: Improvement
>        Reporter: Nigel Jones
>Priority: Major
> Attachments: RangerWinBuild1.log
>
>
> The quick start guide at http://ranger.apache.org/quick_start_guide.html has 
> instuctions to build ranger which work fine on Ubuntu 16.04/openjdk 8 / maven 
> 3.3.9
> However I was interested in setting up a more productive IDE environment, and 
> given my laptop is running Windows 10 as the primary OS decided to follow the 
> instructions for using eclipse
> HOWEVER unlike Ranger-1186 I tried this on *WINDOWS* rather than linux
> Is this something anyone is doing successfully (even if skipping building 
> some of the linux native auth pieces)?
> I then tried the build but it failed quickly on credential support
> [INFO] ranger . SUCCESS [  9.332 
> s]
> [INFO] Jdbc SQL Connector . SUCCESS [  4.569 
> s]
> [INFO] Credential Support . FAILURE [  9.579 
> s]
> Including full log
> Could we list on that page the OSs people have had success with?
> Are most using linux?
> Anyone tried MacOS?
> NOTE: edited from original to focus on the OS question rather than IDE



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (RANGER-1908) Build (unit tests) failed in HBase Security Plugin

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones closed RANGER-1908.
---

> Build (unit tests) failed in HBase Security Plugin
> --
>
> Key: RANGER-1908
> URL: https://issues.apache.org/jira/browse/RANGER-1908
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>    Reporter: Nigel Jones
>    Assignee: Nigel Jones
>Priority: Major
> Attachments: ranger-20171124a.log
>
>
> MacOS
> Java 1.8 (152, oracle)
> Maven 3.5.0
> Build fails at
> [INFO] HBase Security Plugin .. FAILURE [11:36 
> min]
> The failing test is:
> [INFO] Running 
> org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest
> Formatting using clusterid: testClusterID
> [ERROR] Tests run: 20, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 
> 689.962 s <<< FAILURE! - in 
> org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest
> [ERROR] 
> testTagBasedTablePolicy(org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest)
>   Time elapsed: 611.879 s  <<< ERROR!
> org.apache.hadoop.hbase.exceptions.TimeoutIOException: 
> java.util.concurrent.TimeoutException: The procedure 18 is still running
>   at 
> org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.testTagBasedTablePolicy(HBaseRangerAuthorizationTest.java:780)
> Caused by: java.util.concurrent.TimeoutException: The procedure 18 is still 
> running
>   at 
> org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.testTagBasedTablePolicy(HBaseRangerAuthorizationTest.java:780)
> The next part of the test is ok:
> [INFO] Running 
> org.apache.ranger.authorization.hbase.RangerAuthorizationFilterTest
> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.505 
> s - in org.apache.ranger.authorization.hbase.RangerAuthorizationFilterTest



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (RANGER-1909) Build failure in Kafka Security plugin Unit Tests

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones resolved RANGER-1909.
-
Resolution: Cannot Reproduce

> Build failure in Kafka Security plugin Unit Tests
> -
>
> Key: RANGER-1909
> URL: https://issues.apache.org/jira/browse/RANGER-1909
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>    Reporter: Nigel Jones
>Priority: Major
> Attachments: ranger-20171124a.log
>
>
> Java 1.8 (151, openjdk)
> Maven 3.5.0
> Build fails at
> [INFO] KAFKA Security Plugin .. FAILURE [22:07 
> min]
> [INFO] Running 
> org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerGSSTest
> log4j:WARN No appenders could be found for logger 
> (org.apache.kerby.kerberos.kerb.identity.backend.AbstractIdentityBackend).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 
> 1,120.51 s <<< FAILURE! - in 
> org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerGSSTest
> [ERROR] 
> testAuthorizedRead(org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerGSSTest)
>   Time elapsed: 1,109.032 s  <<< FAILURE!
> java.lang.AssertionError
>   at 
> org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerGSSTest.testAuthorizedRead(KafkaRangerAuthorizerGSSTest.java:243)
> [INFO] Running 
> org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerTest
> log4j:WARN No appenders could be found for logger 
> (org.apache.zookeeper.server.ZooKeeperServerMain).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 
> 114.827 s - in 
> org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerTest
> [INFO] Running 
> org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerSASLSSLTest
> log4j:WARN No appenders could be found for logger 
> (org.apache.zookeeper.server.ZooKeeperServerMain).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 72.214 
> s - in 
> org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerSASLSSLTest
> [INFO] 
> [INFO] Results:
> [INFO] 
> [ERROR] Failures: 
> [ERROR]   KafkaRangerAuthorizerGSSTest.testAuthorizedRead:243
> [INFO] 
> [ERROR] Tests run: 8, Failures: 1, Errors: 0, Skipped: 0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (RANGER-1970) Build fails - rat issues with storm-agent

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones closed RANGER-1970.
---

> Build fails - rat issues with storm-agent
> -
>
> Key: RANGER-1970
> URL: https://issues.apache.org/jira/browse/RANGER-1970
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>    Reporter: Nigel Jones
>Assignee: Colm O hEigeartaigh
>Priority: Major
> Fix For: 1.0.0
>
> Attachments: RANGER-1970.patch
>
>
> Cloned ranger source into a clean directory (java is 8 u162 (oracle), OS is 
> MacOS, maven is 3.5.2), , tried to build using the incantation recommended in 
> the ranger docs:
> -DskipTests=false clean compile package install assembly:assembly
> Version info:
> 5:23 $ mvn --version
> Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 
> 2017-10-18T08:58:13+01:00)
> Maven home: /usr/local/Cellar/maven/3.5.2/libexec
> Java version: 1.8.0_162, vendor: Oracle Corporation
> Java home: 
> /Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/jre
> Default locale: en_GB, platform encoding: UTF-8
> OS name: "mac os x", version: "10.13.3", arch: "x86_64", family: "mac"
> When doing this the build fails in 
> [INFO] Rat check: Summary of files. Unapproved: 2 unknown: 2 generated: 0 
> approved: 24 licence.
> 
> [INFO] Knox Security Plugin ... SUCCESS [04:50 
> min]
> [INFO] Storm Security Plugin .. FAILURE [ 18.102 
> s]
> [INFO] YARN Security Plugin ... SKIPPED
> ...
> [ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.11:check 
> (default) on project ranger-storm-plugin: Too many files with unapproved 
> license: 2 See RAT report in: 
> /Users/jonesn/IdeaProjects/ranger/storm-agent/target/rat.txt -> [Help 1]
>  
> Looking at the cause I see in the rat.txt:
> Unapproved licenses:
> /Users/jonesn/IdeaProjects/ranger/storm-agent/logs/workers-artifacts/word-count-1-1517582937/1024/worker.yaml
>  
> /Users/jonesn/IdeaProjects/ranger/storm-agent/logs/workers-artifacts/stormdev-2-1517582937/1027/worker.yaml
>  
> And the files have indeed just been created:
> 4:53 $ cd /Users/jonesn/IdeaProjects/ranger/storm-agent/logs/workers-artifacts
> ✔ ~/IdeaProjects/ranger/storm-agent/logs/workers-artifacts [master|…3]
> 14:53 $ ls
> stormdev-2-1517582937 word-count-1-1517582937
> ✔ ~/IdeaProjects/ranger/storm-agent/logs/workers-artifacts [master|…3]
> 14:53 $ ls -a
> . stormdev-2-1517582937
> .. word-count-1-1517582937
> ✔ ~/IdeaProjects/ranger/storm-agent/logs/workers-artifacts [master|…3]
>  
> So this looks like a build oversight with storm agent - either this needs 
> clearing up, or ignoring in rat config?
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (RANGER-1488) Create Ranger plugin for gaiandb

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447791#comment-16447791
 ] 

Nigel Jones commented on RANGER-1488:
-

Just an update ... the plugin has been coded and is working. The code is 
currently at [https://github.com/GaianRangerPlugin/ranger-gaian-plugin] and 
some associated issues in the github tracker

 

It's not been pushed to ranger yet as we work through a few questions around 
gaian & where the source is managed, but I wanted to update people on progress. 
If interested do get in touch and am happy to explain/demo etc..

> Create Ranger plugin for gaiandb
> 
>
> Key: RANGER-1488
> URL: https://issues.apache.org/jira/browse/RANGER-1488
> Project: Ranger
>  Issue Type: New Feature
>  Components: plugins
>    Reporter: Nigel Jones
>Assignee: Nigel Jones
>Priority: Major
>  Labels: VirtualDataConnector
> Attachments: GaianDebuggingResearch.docx
>
>
> GaianDB is a distributed, federated database built on Apache Derby.
> The documentation is at 
> https://github.com/gaiandb/gaiandb/blob/master/README.md - navigate up for 
> the source.
> As part of a Virtual Connector Project based on Atlas we are using gaianDB to 
> provide a virtualization layer. We need to control access to underlying 
> resources and will be building a Ranger plugin for gaiandb to support this. 
> GaianDB already has support for a form of policy plugin which allows the SQL 
> to be intercepted.
> It is unclear if this code will sit
>  - external to ranger/atlas ie in the gaianDB sources
>  - as a sample in the Atlas project (so that we collate what is needed for 
> the virtual connector project)
>  - as a sample, or component in the Ranger project
> However it seems helpful to describe the proposal here and get 
> feedback/dialogue in the ranger community :-)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: 0.7.0 -> 0.7.1 - servicedef changes?

[Nigel Jones]

Not seeing masking policies now.

Again hive still works so it surely must be some issue with our servicedef. Any 
ideas? 

On 2018/03/07 11:22:33, Nigel Jones <nigel.l.jo...@gmail.com> wrote: 
> 
> 
> On 2018/03/06 23:59:32, Nigel Jones <nigel.l.jo...@gmail.com> wrote: 
> > I have a servicedef which is working well when deployed to a Ranger 0.7.0 
> > server (HDP 2.6.4).
> > I can create an instance of the new service, with access control & masking 
> > policies & they work :-)
> > 
> > However we've found that when deployed to a cleanly built 0.7.1 / master 
> > server (at a few dates in Feb 18) the SAME servicedef results in only 
> > access control policies being authorable on the Ranger GUI, NOT tag based 
> > policies
> 
> Rebuilt with current master (was running ~1 Feb) and can confirm this problem 
> does not occur, so looks like it was a temporary master regression
> 

Re: 0.7.0 -> 0.7.1 - servicedef changes?

[Nigel Jones]

On 2018/03/06 23:59:32, Nigel Jones <nigel.l.jo...@gmail.com> wrote: 
> I have a servicedef which is working well when deployed to a Ranger 0.7.0 
> server (HDP 2.6.4).
> I can create an instance of the new service, with access control & masking 
> policies & they work :-)
> 
> However we've found that when deployed to a cleanly built 0.7.1 / master 
> server (at a few dates in Feb 18) the SAME servicedef results in only access 
> control policies being authorable on the Ranger GUI, NOT tag based policies

Rebuilt with current master (was running ~1 Feb) and can confirm this problem 
does not occur, so looks like it was a temporary master regression

0.7.0 -> 0.7.1 - servicedef changes?

[Nigel Jones]

I have a servicedef which is working well when deployed to a Ranger 0.7.0 
server (HDP 2.6.4).
I can create an instance of the new service, with access control & masking 
policies & they work :-)

However we've found that when deployed to a cleanly built 0.7.1 / master server 
(at a few dates in Feb 18) the SAME servicedef results in only access control 
policies being authorable on the Ranger GUI, NOT tag based policies

The hive servicedef pre-loaded into the ranger build does continue to show 
masking based policies.

Are there any significant changes in ranger in that 0.7.0->0.7.1 timeframe that 
might explain?

Our current servicedef is at 
https://github.com/GaianRangerPlugin/ranger-gaian-plugin/blob/master/plugin/src/main/resources/service-defs/ranger-servicedef-gaian.json

Thanks!
Nigel.

Re: servicedef validation - implClass

[Nigel Jones]

Quick update... I worked around this for now by implementing a dummy
resource lookup class and deploying to the server.

On Tue, 27 Feb 2018 at 13:08 Nigel Jones <ni...@cherrybyte.me.uk> wrote:

> Having tried this against master (for about a week ago) I am still getting
> the same issue deploying the plugin on a base ranger install (it works ok
> in HDP)
>
> The exception trace in the log included as an attachment
>
> My ews/lib directory does include
> ranger-plugins-common-1.0.0-SNAPSHOT.jar
>
> and is readable by the user ranger is started with
>
> It is also available within ews/webapp/WEB-INF/lib
>
> Currently my implClass is blank
>
> Any ideas?
>
> On Sat, 3 Feb 2018 at 00:37 Ramesh Mani <rm...@hortonworks.com> wrote:
>
>> Nigel,
>>
>> Could you please provide the exception stack that is printed along with
>> error message you are showing. Please check in xa_portal.log ( Ranger
>> log).
>>
>> Also check in Ranger class path ranger-plugins-common*jar is there. Check
>> in {install.dir}/ranger-admin/ews/lib and
>> {install.dir}/ranger-admin/ews/webapp/WEB-INF/lib and it has the necessary
>> permission for the process user who start ranger admin.
>>
>> As you notice it should have used the base implementation when you leave
>> implClass blank / or not in the service def.
>>
>> Regards,
>> Ramesh
>>
>>
>> On 2/2/18, 6:24 AM, "Nigel Jones" <nigel.l.jo...@gmail.com> wrote:
>>
>> >We're currently writing a Ranger plugin for an open source virtual
>> >database driver known as ŒGaianDB¹.
>> >
>> >We have the basics of a plugin working, but today a colleague tried to
>> >setup our plugin, and on their ranger install they could not create an
>> >instance of our ranger service, hitting the error
>> >
>> >"Gaian failed to find service class
>> >org.apache.ranger.services.gaiandb.RangerServiceGaian. Resource lookup
>> >will not be available. Please make sure plugin jar is in the correct
>> >place.²
>> >
>> >I had expected this to be a warning (fair enough), but it actually
>> >prevents the UI from saving the service instance, and so Gaian related
>> >policies cannot be created.
>> >
>> >They got this on I think a) an old 0.6.x build, and 0.7.1.
>> >Meanwhile I had been using a HDP 2.6.3 setup, which comes with 0.7.0 and
>> >with the same service definition did NOT of course hit this error.
>> >
>> >We haven¹t actually deployed any plugin code to the ranger server (yet)
>> >since we have not implemented the resource lookup capability. We had
>> >though specified an implClass in the servicedef. It¹s a later task :-)
>> >
>> >Looking at the ranger code in ServiceMgr.java it seems as if an empty
>> >string should cause ranger to use a default class. We tried this, as well
>> >as specifying the org.apache.ranger.plugin.service.RangerDefaultService
>> >class, but had the same error in both cases. I was though looking at code
>> >in master and see there have been a few changes
>> >
>> >Any suggestions as to how to persuade 0.7.0 to load the servicedef (other
>> >than implement the class required, albeit a no-op oneŠ?), or an
>> >explanation as to why we might see different behaviour?
>> >
>> >I¹m setting up a ranger install off master to debug/check latest codeŠ.
>> >
>> >For info the servicedef begins:
>> >{
>> >  "id":99,
>> >  "name": "gaian",
>> >  "implClass": "org.apache.ranger.services.gaiandb.RangerServiceGaian",
>> >  "label": "Gaian",
>> >  "description": "Gaian",
>> >  "options": {
>> >"enableDenyAndExceptionsInPolicies": "true"
>> >  },
>> >"guid": "86d10748-e4fc-442b-8991-f6a727054ece",
>> >
>> >  "resources": [
>> >The full version can be found in
>> >https://issues.apache.org/jira/browse/RANGER-1699
>> ><https://issues.apache.org/jira/browse/RANGER-1699> - see the latest
>> >attachment. Not as a patch as currently getting things working in a diff.
>> >Build tree
>> >
>> >If it seems like a bug (rather than a newbie error or specific to hdp)
>> >let me know and I¹ll open a jira!
>> >
>> >Many thanks
>> >Nigel.
>> >
>> >
>> >
>>
>>

Re: servicedef validation - implClass

[Nigel Jones]

Having tried this against master (for about a week ago) I am still getting
the same issue deploying the plugin on a base ranger install (it works ok
in HDP)

The exception trace in the log included as an attachment

My ews/lib directory does include
ranger-plugins-common-1.0.0-SNAPSHOT.jar

and is readable by the user ranger is started with

It is also available within ews/webapp/WEB-INF/lib

Currently my implClass is blank

Any ideas?

On Sat, 3 Feb 2018 at 00:37 Ramesh Mani <rm...@hortonworks.com> wrote:

> Nigel,
>
> Could you please provide the exception stack that is printed along with
> error message you are showing. Please check in xa_portal.log ( Ranger log).
>
> Also check in Ranger class path ranger-plugins-common*jar is there. Check
> in {install.dir}/ranger-admin/ews/lib and
> {install.dir}/ranger-admin/ews/webapp/WEB-INF/lib and it has the necessary
> permission for the process user who start ranger admin.
>
> As you notice it should have used the base implementation when you leave
> implClass blank / or not in the service def.
>
> Regards,
> Ramesh
>
>
> On 2/2/18, 6:24 AM, "Nigel Jones" <nigel.l.jo...@gmail.com> wrote:
>
> >We're currently writing a Ranger plugin for an open source virtual
> >database driver known as ŒGaianDB¹.
> >
> >We have the basics of a plugin working, but today a colleague tried to
> >setup our plugin, and on their ranger install they could not create an
> >instance of our ranger service, hitting the error
> >
> >"Gaian failed to find service class
> >org.apache.ranger.services.gaiandb.RangerServiceGaian. Resource lookup
> >will not be available. Please make sure plugin jar is in the correct
> >place.²
> >
> >I had expected this to be a warning (fair enough), but it actually
> >prevents the UI from saving the service instance, and so Gaian related
> >policies cannot be created.
> >
> >They got this on I think a) an old 0.6.x build, and 0.7.1.
> >Meanwhile I had been using a HDP 2.6.3 setup, which comes with 0.7.0 and
> >with the same service definition did NOT of course hit this error.
> >
> >We haven¹t actually deployed any plugin code to the ranger server (yet)
> >since we have not implemented the resource lookup capability. We had
> >though specified an implClass in the servicedef. It¹s a later task :-)
> >
> >Looking at the ranger code in ServiceMgr.java it seems as if an empty
> >string should cause ranger to use a default class. We tried this, as well
> >as specifying the org.apache.ranger.plugin.service.RangerDefaultService
> >class, but had the same error in both cases. I was though looking at code
> >in master and see there have been a few changes
> >
> >Any suggestions as to how to persuade 0.7.0 to load the servicedef (other
> >than implement the class required, albeit a no-op oneŠ?), or an
> >explanation as to why we might see different behaviour?
> >
> >I¹m setting up a ranger install off master to debug/check latest codeŠ.
> >
> >For info the servicedef begins:
> >{
> >  "id":99,
> >  "name": "gaian",
> >  "implClass": "org.apache.ranger.services.gaiandb.RangerServiceGaian",
> >  "label": "Gaian",
> >  "description": "Gaian",
> >  "options": {
> >"enableDenyAndExceptionsInPolicies": "true"
> >  },
> >"guid": "86d10748-e4fc-442b-8991-f6a727054ece",
> >
> >  "resources": [
> >The full version can be found in
> >https://issues.apache.org/jira/browse/RANGER-1699
> ><https://issues.apache.org/jira/browse/RANGER-1699> - see the latest
> >attachment. Not as a patch as currently getting things working in a diff.
> >Build tree
> >
> >If it seems like a bug (rather than a newbie error or specific to hdp)
> >let me know and I¹ll open a jira!
> >
> >Many thanks
> >Nigel.
> >
> >
> >
>
>

[jira] [Created] (RANGER-1970) Build fails - rat issues with storm-agent

[Nigel Jones (JIRA)]

Nigel Jones created RANGER-1970:
---

 Summary: Build fails - rat issues with storm-agent
 Key: RANGER-1970
 URL: https://issues.apache.org/jira/browse/RANGER-1970
 Project: Ranger
  Issue Type: Bug
  Components: Ranger
Reporter: Nigel Jones


Cloned ranger source into a clean directory (java is 8 u162 (oracle), OS is 
MacOS, maven is 3.5.2), , tried to build using the incantation recommended in 
the ranger docs:

-DskipTests=false clean compile package install assembly:assembly

Version info:

5:23 $ mvn --version
Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 
2017-10-18T08:58:13+01:00)
Maven home: /usr/local/Cellar/maven/3.5.2/libexec
Java version: 1.8.0_162, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/jre
Default locale: en_GB, platform encoding: UTF-8
OS name: "mac os x", version: "10.13.3", arch: "x86_64", family: "mac"

When doing this the build fails in 

[INFO] Rat check: Summary of files. Unapproved: 2 unknown: 2 generated: 0 
approved: 24 licence.



[INFO] Knox Security Plugin ... SUCCESS [04:50 min]
[INFO] Storm Security Plugin .. FAILURE [ 18.102 s]
[INFO] YARN Security Plugin ... SKIPPED

...

[ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.11:check 
(default) on project ranger-storm-plugin: Too many files with unapproved 
license: 2 See RAT report in: 
/Users/jonesn/IdeaProjects/ranger/storm-agent/target/rat.txt -> [Help 1]

 

Looking at the cause I see in the rat.txt:


Unapproved licenses:

/Users/jonesn/IdeaProjects/ranger/storm-agent/logs/workers-artifacts/word-count-1-1517582937/1024/worker.yaml
 
/Users/jonesn/IdeaProjects/ranger/storm-agent/logs/workers-artifacts/stormdev-2-1517582937/1027/worker.yaml

 

And the files have indeed just been created:

4:53 $ cd /Users/jonesn/IdeaProjects/ranger/storm-agent/logs/workers-artifacts
✔ ~/IdeaProjects/ranger/storm-agent/logs/workers-artifacts [master|…3]
14:53 $ ls
stormdev-2-1517582937 word-count-1-1517582937
✔ ~/IdeaProjects/ranger/storm-agent/logs/workers-artifacts [master|…3]
14:53 $ ls -a
. stormdev-2-1517582937
.. word-count-1-1517582937
✔ ~/IdeaProjects/ranger/storm-agent/logs/workers-artifacts [master|…3]

 

So this looks like a build oversight with storm agent - either this needs 
clearing up, or ignoring in rat config?

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

servicedef validation - implClass

[Nigel Jones]

We're currently writing a Ranger plugin for an open source virtual database 
driver known as ‘GaianDB’.

We have the basics of a plugin working, but today a colleague tried to setup 
our plugin, and on their ranger install they could not create an instance of 
our ranger service, hitting the error 

"Gaian failed to find service class 
org.apache.ranger.services.gaiandb.RangerServiceGaian. Resource lookup will not 
be available. Please make sure plugin jar is in the correct place.”

I had expected this to be a warning (fair enough), but it actually prevents the 
UI from saving the service instance, and so Gaian related policies cannot be 
created.

They got this on I think a) an old 0.6.x build, and 0.7.1. 
Meanwhile I had been using a HDP 2.6.3 setup, which comes with 0.7.0 and with 
the same service definition did NOT of course hit this error.

We haven’t actually deployed any plugin code to the ranger server (yet) since 
we have not implemented the resource lookup capability. We had though specified 
an implClass in the servicedef. It’s a later task :-)

Looking at the ranger code in ServiceMgr.java it seems as if an empty string 
should cause ranger to use a default class. We tried this, as well as 
specifying the org.apache.ranger.plugin.service.RangerDefaultService class, but 
had the same error in both cases. I was though looking at code in master and 
see there have been a few changes

Any suggestions as to how to persuade 0.7.0 to load the servicedef (other than 
implement the class required, albeit a no-op one…?), or an explanation as to 
why we might see different behaviour?

I’m setting up a ranger install off master to debug/check latest code….

For info the servicedef begins:
{
  "id":99,
  "name": "gaian",
  "implClass": "org.apache.ranger.services.gaiandb.RangerServiceGaian",
  "label": "Gaian",
  "description": "Gaian",
  "options": {
"enableDenyAndExceptionsInPolicies": "true"
  },
"guid": "86d10748-e4fc-442b-8991-f6a727054ece",

  "resources": [
The full version can be found in 
https://issues.apache.org/jira/browse/RANGER-1699 
 - see the latest 
attachment. Not as a patch as currently getting things working in a diff. Build 
tree

If it seems like a bug (rather than a newbie error or specific to hdp) let me 
know and I’ll open a jira!

Many thanks
Nigel.

[jira] [Commented] (RANGER-1699) gaiandb: Create initial plugin configuration file that can be deployed to ranger

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350391#comment-16350391
 ] 

Nigel Jones commented on RANGER-1699:
-

Added current servicedef (not currently a patch as restructuring build)

> gaiandb: Create initial plugin configuration file that can be deployed to 
> ranger
> 
>
> Key: RANGER-1699
> URL: https://issues.apache.org/jira/browse/RANGER-1699
> Project: Ranger
>  Issue Type: Sub-task
>  Components: plugins
>    Reporter: Nigel Jones
>Priority: Major
> Attachments: 0001-adding-gaiandb-serviceDef.patch, 
> ranger-servicedef-gaian.json
>
>
> Create initial configuration file for the gaiandb plugin for ranger which 
> will define policies, resources.
> Additional jiras will be opened to add extra capabilities later on. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (RANGER-1699) gaiandb: Create initial plugin configuration file that can be deployed to ranger

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones updated RANGER-1699:

Attachment: ranger-servicedef-gaian.json

> gaiandb: Create initial plugin configuration file that can be deployed to 
> ranger
> 
>
> Key: RANGER-1699
> URL: https://issues.apache.org/jira/browse/RANGER-1699
> Project: Ranger
>  Issue Type: Sub-task
>  Components: plugins
>    Reporter: Nigel Jones
>Priority: Major
> Attachments: 0001-adding-gaiandb-serviceDef.patch, 
> ranger-servicedef-gaian.json
>
>
> Create initial configuration file for the gaiandb plugin for ranger which 
> will define policies, resources.
> Additional jiras will be opened to add extra capabilities later on. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (RANGER-1850) Impersonation/proxy user support for gaiandb ranger plugin

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313281#comment-16313281
 ] 

Nigel Jones commented on RANGER-1850:
-

I have an example flow on the 'proxy auth' tab (along bottom) of a few diagrams 
on draw.io - 
https://www.draw.io/#Hplanetf1%2Fgaiandb-policy-ranger%2Fmaster%2Fdoc%2FGaian%20Data%20Access%20Diagrams

To answer
 * userid Ernie is coming in from the JDBC application into gaiandb. As part of 
that jdbc connect proxy-uid=gaiandb, proxy-pass=passw0rd is also sent
 * gaiandb/passw0rd is used by the proxy auth module to authenticate with the 
base gaiandb/derby authentication module 
 * Once connected the application does indeed have access to all the gaiandb 
virtual tables in that there is no control over returned metadata (std jdbc 
metadata) as gaiandb doesn't support policy management there (I think it should 
in future). At a derby level gaiandb does have access to everything
 * If the ranger plugin is installed, then access requests to individual 
schemas, tables & columns will be controlled at that point (using user Ernie)
 * This difference between derby's base auth model using grants etc & what 
ranger does is not ideal... but requires significant work to better integrate 
gaiandb & it's policy model into the derby security manager
 * The userids configured to connect to the underlaying data sources that 
gaiandb is virtualizing are defined in the gaiandb configuration files (on the 
gaiandb code co-located with that data source typically). Whilst the user Ernie 
can access these data sources, he does not have direct access to the 
authentication details as such, or at least doesn't use them, that's the point 
of using gaiandb as the virtualization layer
 * A individual data source will have one set of credentials defined in the 
gaiandb configuration ,so regardless of which user connects the same 
credentials are used, so by definition they are NPAs

One point that does surface from this is that access to the gaiandb stored 
procedures/derby tables that contain auth info for the underlying sources 
should be restricted. I am not sure if this is possible but will look into it. 

> Impersonation/proxy user support for gaiandb ranger plugin
> --
>
> Key: RANGER-1850
> URL: https://issues.apache.org/jira/browse/RANGER-1850
> Project: Ranger
>  Issue Type: Sub-task
>  Components: plugins
>Reporter: Nigel Jones
> Attachments: GaianDBAuth.docx
>
>
> Applications/users could connect to gaianDB using their own authentication 
> information - for example userid/password in the simple case. Here the ranger 
> plugin will use that id for policy checks.
> However in a multi tiered architecture a service id (aka non personal 
> account) may be used, and somehow the user to be impersonated is passed via 
> an additional property. This has a number of implications to the system 
> configuration, derby/gaiandb configuration & the plugin implementation. 
> Opening this Jira as a placeholder and will add a document soon (++days) on 
> the same to capture some of the discussion around this area in recent days.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (RANGER-1850) Impersonation/proxy user support for gaiandb ranger plugin

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312967#comment-16312967
 ] 

Nigel Jones edited comment on RANGER-1850 at 1/5/18 11:16 AM:
--

a) Point noted :-)
b) Yes that is correct (I was aware of it, I should add to the docs), it's a 
current limitation caused by derby/us using backlevel.
e) Correct we are not using this approach as this requires changes to 
applications and means derby sees one user (like 'gaiandb') whilst some the 
ranger plugin operates on another. It also means only sql statement execution 
uses this proxied user id, whilst jdbc metadata queries etc do not. Confusing 
and more divergent from base derby security & the way the derby security 
manager works. By using the proper derby plugin approach derby consistently 
sees the same userid as gaian regardless of the API call. We are also closer 
then to the intent of the derby community in terms of extension points, and can 
build on that approach as we update gaiandb in future
f) The plugin is called when a connection is made from an application, via the 
jdbc driver, into gaiandb. It's right at the top.. It is not called per data 
source (that is something gaiandb manages using the auth details configured in 
gaiandb_config.properties for each source)


was (Author: jonesn):
a) Point accepted :-)

> Impersonation/proxy user support for gaiandb ranger plugin
> --
>
> Key: RANGER-1850
> URL: https://issues.apache.org/jira/browse/RANGER-1850
> Project: Ranger
>  Issue Type: Sub-task
>  Components: plugins
>    Reporter: Nigel Jones
> Attachments: GaianDBAuth.docx
>
>
> Applications/users could connect to gaianDB using their own authentication 
> information - for example userid/password in the simple case. Here the ranger 
> plugin will use that id for policy checks.
> However in a multi tiered architecture a service id (aka non personal 
> account) may be used, and somehow the user to be impersonated is passed via 
> an additional property. This has a number of implications to the system 
> configuration, derby/gaiandb configuration & the plugin implementation. 
> Opening this Jira as a placeholder and will add a document soon (++days) on 
> the same to capture some of the discussion around this area in recent days.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RANGER-1850) Impersonation/proxy user support for gaiandb ranger plugin

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312967#comment-16312967
 ] 

Nigel Jones commented on RANGER-1850:
-

a) Point accepted :-)

> Impersonation/proxy user support for gaiandb ranger plugin
> --
>
> Key: RANGER-1850
> URL: https://issues.apache.org/jira/browse/RANGER-1850
> Project: Ranger
>  Issue Type: Sub-task
>  Components: plugins
>    Reporter: Nigel Jones
> Attachments: GaianDBAuth.docx
>
>
> Applications/users could connect to gaianDB using their own authentication 
> information - for example userid/password in the simple case. Here the ranger 
> plugin will use that id for policy checks.
> However in a multi tiered architecture a service id (aka non personal 
> account) may be used, and somehow the user to be impersonated is passed via 
> an additional property. This has a number of implications to the system 
> configuration, derby/gaiandb configuration & the plugin implementation. 
> Opening this Jira as a placeholder and will add a document soon (++days) on 
> the same to capture some of the discussion around this area in recent days.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RANGER-1850) Impersonation/proxy user support for gaiandb ranger plugin

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312899#comment-16312899
 ] 

Nigel Jones commented on RANGER-1850:
-

Thanks for the feedback
a) proxy auth fail -- I think both approaches make some sense. I'll think about 
which makes sense. Note that to some extent this is a shorter term fix since 
there's a lot more to do (IMO) for a full proxy support. for example I'd like 
to be able to grant specific permissions to the service account to permit it 
the proxy capability, and potentially vice versa to allow a user to be 
impersonated. This starts getting into the derby security manager in more 
detail, so should be based on using more up to date derby code (current) and 
working with the derby community. There's also here a tension between the 
implementation of the policy plugin and the security manager. We can't do that 
just yet so a 'sufficient' solution is all I'm going for at this point.
b) create-schema well.. creates a schema for the user. I wouldn't have expected 
it to be needed either - since my queries are against fully qualified views 
like gaiandb.vcustomer -- but this fails with the older derby code gaiandb is 
using. From the mailing list (derby) it appears this is a bug as it only 
affects views and not tables.. but I can't move to current derby code without 
gaiandb rejigging. We've already started work to rebuild gaiandb and start 
updating libraries, but that is a longer term effort. So for now the pragmatic 
workaround is to add this flag. I did this in the plugin to reduce 
impact/complexity/odd errors from users of the code. Similar to a) it's a 
reflection of a minimal solution pending a much bigger change to bring gaiandb 
up to current code levels
c) Which docs are you referring to? If gaiandb - correct. this is an additional 
plugin to be applied on top. In the future I hope we can make this a core 
capability, but this is a pragmatic approach on existing levels. If the docs 
for this plugin, agree that work isn't complete yet. Feel free to clarify what 
info you'll need and I'll add it in the README.md once the code moves into the 
ranger repo
d) For the use case I'm addressing, auth against oracle is via the NPA defined 
for oracle (different to the gaiandb NPA), and those credentials are managed 
via the gaiandb configuration as is the case today. As far as gaiandb/derby is 
concerned the 'user' is the proxied user ie nigel, david etc as that is what 
the 'User' parm on connection is set to (one reason I used it with additional 
proxy-uid, proxy-password for the uid/password I wanted to use for the 
authentication check). So once the query leaves derby we are purely using the 
oracle NPA, and no attempt is made to 'pass down' the actual end user to the 
db. This would be very useful for audit, but the method to do this will vary by 
each db/source gaiandb supports. It would need more support in gaiandb itself 
for a framework, and then specifics for each supported db. Again we're up 
against the fact changes need to be made in this area in any case, so it didn't 
seem prudent to go down this route at this point.  

> Impersonation/proxy user support for gaiandb ranger plugin
> --
>
> Key: RANGER-1850
> URL: https://issues.apache.org/jira/browse/RANGER-1850
> Project: Ranger
>  Issue Type: Sub-task
>  Components: plugins
>    Reporter: Nigel Jones
> Attachments: GaianDBAuth.docx
>
>
> Applications/users could connect to gaianDB using their own authentication 
> information - for example userid/password in the simple case. Here the ranger 
> plugin will use that id for policy checks.
> However in a multi tiered architecture a service id (aka non personal 
> account) may be used, and somehow the user to be impersonated is passed via 
> an additional property. This has a number of implications to the system 
> configuration, derby/gaiandb configuration & the plugin implementation. 
> Opening this Jira as a placeholder and will add a document soon (++days) on 
> the same to capture some of the discussion around this area in recent days.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RANGER-1850) Impersonation/proxy user support for gaiandb ranger plugin

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16283345#comment-16283345
 ] 

Nigel Jones commented on RANGER-1850:
-

I now have an initial version of this.

Have added a proxy auth class to 
https://github.com/planetf1/gaiandb-policy-ranger/tree/master/src/main/java/org/apache/gaiandb/security
 
This allows two additional parms to be passed on the URL - or in the properties 
object.
 proxy-user
 proxy-pwd
These should be set to the NPA account - ie gaiandb/password. Then the EXISTING 
'user' parm gets set to nigel, roger, ernie etc. A little glitch that password 
has to be non null for now - but is ignored. If proxy auth fails, or those 
parms aren't present, auth falls back to the existing gaiandb mechanism (which 
itself falls back to the native derby authenticator). 

By setting user to the 'real' user, the default schema is now 'jonesn' - which 
I wanted since I think the most secure approach is to allow all of the existing 
derby auth support to work as normal - the ONLY difference is really in how we 
connect.. so queries need to be fully qualified ie 'select * from 
gaiandb.vemployee' 

The proxy auth forces additional properties 'create=true' on the connection as 
otherwise derby will fail even with the fully qualified schema, if that schema 
does not exist. (at least in the older version used in gaiandb)

Bottom line
 - can now authenticate using gaiandb user/password (an NPA), and passing in 
desired user
 - can select from gaiandb virtual tables
 - policy plugin gets invoked with correct user context
 - any other db controls respected using the real user (but this area needs 
very broad review post MVP)
 - can also connect as a regular user
Need to figure out where to host this code. For now the update is in github as 
previously

> Impersonation/proxy user support for gaiandb ranger plugin
> --
>
> Key: RANGER-1850
> URL: https://issues.apache.org/jira/browse/RANGER-1850
> Project: Ranger
>  Issue Type: Sub-task
>  Components: plugins
>    Reporter: Nigel Jones
> Attachments: GaianDBAuth.docx
>
>
> Applications/users could connect to gaianDB using their own authentication 
> information - for example userid/password in the simple case. Here the ranger 
> plugin will use that id for policy checks.
> However in a multi tiered architecture a service id (aka non personal 
> account) may be used, and somehow the user to be impersonated is passed via 
> an additional property. This has a number of implications to the system 
> configuration, derby/gaiandb configuration & the plugin implementation. 
> Opening this Jira as a placeholder and will add a document soon (++days) on 
> the same to capture some of the discussion around this area in recent days.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (RANGER-1488) Create Ranger plugin for gaiandb

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272877#comment-16272877
 ] 

Nigel Jones edited comment on RANGER-1488 at 11/30/17 4:16 PM:
---

Added some notes on what I've seen so far as I run through the gaianDB code. I 
will look at subtask 8 .. but this was needed as prep. Will continue next week

The test data source used was 
https://cwiki.apache.org/confluence/display/ATLAS/Example+of+virtualizing+MySQL+employee+data+via+gaianDB
 - there's further gaiandb info via the atlas wiki


was (Author: jonesn):
Added some notes on what I've seen so far as I run through the gaianDB code. I 
will look at subtask 8 .. but this was needed as prep. Will continue next week

> Create Ranger plugin for gaiandb
> 
>
> Key: RANGER-1488
> URL: https://issues.apache.org/jira/browse/RANGER-1488
> Project: Ranger
>  Issue Type: New Feature
>  Components: plugins
>    Reporter: Nigel Jones
>    Assignee: Nigel Jones
>  Labels: VirtualDataConnector
> Attachments: GaianDebuggingResearch.docx
>
>
> GaianDB is a distributed, federated database built on Apache Derby.
> The documentation is at 
> https://github.com/gaiandb/gaiandb/blob/master/README.md - navigate up for 
> the source.
> As part of a Virtual Connector Project based on Atlas we are using gaianDB to 
> provide a virtualization layer. We need to control access to underlying 
> resources and will be building a Ranger plugin for gaiandb to support this. 
> GaianDB already has support for a form of policy plugin which allows the SQL 
> to be intercepted.
> It is unclear if this code will sit
>  - external to ranger/atlas ie in the gaianDB sources
>  - as a sample in the Atlas project (so that we collate what is needed for 
> the virtual connector project)
>  - as a sample, or component in the Ranger project
> However it seems helpful to describe the proposal here and get 
> feedback/dialogue in the ranger community :-)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RANGER-1488) Create Ranger plugin for gaiandb

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272877#comment-16272877
 ] 

Nigel Jones commented on RANGER-1488:
-

Added some notes on what I've seen so far as I run through the gaianDB code. I 
will look at subtask 8 .. but this was needed as prep. Will continue next week

> Create Ranger plugin for gaiandb
> 
>
> Key: RANGER-1488
> URL: https://issues.apache.org/jira/browse/RANGER-1488
> Project: Ranger
>  Issue Type: New Feature
>  Components: plugins
>    Reporter: Nigel Jones
>    Assignee: Nigel Jones
>  Labels: VirtualDataConnector
> Attachments: GaianDebuggingResearch.docx
>
>
> GaianDB is a distributed, federated database built on Apache Derby.
> The documentation is at 
> https://github.com/gaiandb/gaiandb/blob/master/README.md - navigate up for 
> the source.
> As part of a Virtual Connector Project based on Atlas we are using gaianDB to 
> provide a virtualization layer. We need to control access to underlying 
> resources and will be building a Ranger plugin for gaiandb to support this. 
> GaianDB already has support for a form of policy plugin which allows the SQL 
> to be intercepted.
> It is unclear if this code will sit
>  - external to ranger/atlas ie in the gaianDB sources
>  - as a sample in the Atlas project (so that we collate what is needed for 
> the virtual connector project)
>  - as a sample, or component in the Ranger project
> However it seems helpful to describe the proposal here and get 
> feedback/dialogue in the ranger community :-)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (RANGER-1488) Create Ranger plugin for gaiandb

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones updated RANGER-1488:

Attachment: GaianDebuggingResearch.docx

Some rough notes of working through the gaiandb code

> Create Ranger plugin for gaiandb
> 
>
> Key: RANGER-1488
> URL: https://issues.apache.org/jira/browse/RANGER-1488
> Project: Ranger
>  Issue Type: New Feature
>  Components: plugins
>    Reporter: Nigel Jones
>    Assignee: Nigel Jones
>  Labels: VirtualDataConnector
> Attachments: GaianDebuggingResearch.docx
>
>
> GaianDB is a distributed, federated database built on Apache Derby.
> The documentation is at 
> https://github.com/gaiandb/gaiandb/blob/master/README.md - navigate up for 
> the source.
> As part of a Virtual Connector Project based on Atlas we are using gaianDB to 
> provide a virtualization layer. We need to control access to underlying 
> resources and will be building a Ranger plugin for gaiandb to support this. 
> GaianDB already has support for a form of policy plugin which allows the SQL 
> to be intercepted.
> It is unclear if this code will sit
>  - external to ranger/atlas ie in the gaianDB sources
>  - as a sample in the Atlas project (so that we collate what is needed for 
> the virtual connector project)
>  - as a sample, or component in the Ranger project
> However it seems helpful to describe the proposal here and get 
> feedback/dialogue in the ranger community :-)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (RANGER-1908) Build (unit tests) failed in HBase Security Plugin

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones reassigned RANGER-1908:
---

Assignee: Nigel Jones

> Build (unit tests) failed in HBase Security Plugin
> --
>
> Key: RANGER-1908
> URL: https://issues.apache.org/jira/browse/RANGER-1908
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>    Reporter: Nigel Jones
>    Assignee: Nigel Jones
> Attachments: ranger-20171124a.log
>
>
> MacOS
> Java 1.8 (152, oracle)
> Maven 3.5.0
> Build fails at
> [INFO] HBase Security Plugin .. FAILURE [11:36 
> min]
> The failing test is:
> [INFO] Running 
> org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest
> Formatting using clusterid: testClusterID
> [ERROR] Tests run: 20, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 
> 689.962 s <<< FAILURE! - in 
> org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest
> [ERROR] 
> testTagBasedTablePolicy(org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest)
>   Time elapsed: 611.879 s  <<< ERROR!
> org.apache.hadoop.hbase.exceptions.TimeoutIOException: 
> java.util.concurrent.TimeoutException: The procedure 18 is still running
>   at 
> org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.testTagBasedTablePolicy(HBaseRangerAuthorizationTest.java:780)
> Caused by: java.util.concurrent.TimeoutException: The procedure 18 is still 
> running
>   at 
> org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.testTagBasedTablePolicy(HBaseRangerAuthorizationTest.java:780)
> The next part of the test is ok:
> [INFO] Running 
> org.apache.ranger.authorization.hbase.RangerAuthorizationFilterTest
> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.505 
> s - in org.apache.ranger.authorization.hbase.RangerAuthorizationFilterTest



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (RANGER-1908) Build (unit tests) failed in HBase Security Plugin

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones resolved RANGER-1908.
-
Resolution: Cannot Reproduce

> Build (unit tests) failed in HBase Security Plugin
> --
>
> Key: RANGER-1908
> URL: https://issues.apache.org/jira/browse/RANGER-1908
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>    Reporter: Nigel Jones
>    Assignee: Nigel Jones
> Attachments: ranger-20171124a.log
>
>
> MacOS
> Java 1.8 (152, oracle)
> Maven 3.5.0
> Build fails at
> [INFO] HBase Security Plugin .. FAILURE [11:36 
> min]
> The failing test is:
> [INFO] Running 
> org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest
> Formatting using clusterid: testClusterID
> [ERROR] Tests run: 20, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 
> 689.962 s <<< FAILURE! - in 
> org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest
> [ERROR] 
> testTagBasedTablePolicy(org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest)
>   Time elapsed: 611.879 s  <<< ERROR!
> org.apache.hadoop.hbase.exceptions.TimeoutIOException: 
> java.util.concurrent.TimeoutException: The procedure 18 is still running
>   at 
> org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.testTagBasedTablePolicy(HBaseRangerAuthorizationTest.java:780)
> Caused by: java.util.concurrent.TimeoutException: The procedure 18 is still 
> running
>   at 
> org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.testTagBasedTablePolicy(HBaseRangerAuthorizationTest.java:780)
> The next part of the test is ok:
> [INFO] Running 
> org.apache.ranger.authorization.hbase.RangerAuthorizationFilterTest
> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.505 
> s - in org.apache.ranger.authorization.hbase.RangerAuthorizationFilterTest



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (RANGER-1908) Build (unit tests) failed in HBase Security Plugin

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266840#comment-16266840
 ] 

Nigel Jones edited comment on RANGER-1908 at 11/27/17 2:00 PM:
---

More tests
 * MacOS - 4 consecutive builds all clean
 * VM (a) @IBM. Failed 2/3, worked 1/3. looks timing related unit test case 
failure (testing Kafka security plugin -- not the HBase one)
 * VM (b) @Azure (lower spec) failed 3/3 same as above. timing related

At this point I will close this JIRA as non reproducible after around 8 
builds Will keep an eye on any timing related issues to see if perhaps we 
need to relax some timings and raise a new JIRA if so.


was (Author: jonesn):
More tests
 * MacOS - 4 consecutive builds all clean
 * VM (a) @IBM. Failed 2/3, worked 1/3. looks timing related unit test case 
failure
 * VM (b) @Azure (lower spec) failed 3/3 same as above. timing related

At this point I will close this JIRA as non reproducible. Will keep an eye on 
any timing related issues to see if perhaps we need to relax some timings and 
raise a new JIRA if so.

> Build (unit tests) failed in HBase Security Plugin
> --
>
> Key: RANGER-1908
> URL: https://issues.apache.org/jira/browse/RANGER-1908
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>    Reporter: Nigel Jones
> Attachments: ranger-20171124a.log
>
>
> MacOS
> Java 1.8 (152, oracle)
> Maven 3.5.0
> Build fails at
> [INFO] HBase Security Plugin .. FAILURE [11:36 
> min]
> The failing test is:
> [INFO] Running 
> org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest
> Formatting using clusterid: testClusterID
> [ERROR] Tests run: 20, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 
> 689.962 s <<< FAILURE! - in 
> org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest
> [ERROR] 
> testTagBasedTablePolicy(org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest)
>   Time elapsed: 611.879 s  <<< ERROR!
> org.apache.hadoop.hbase.exceptions.TimeoutIOException: 
> java.util.concurrent.TimeoutException: The procedure 18 is still running
>   at 
> org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.testTagBasedTablePolicy(HBaseRangerAuthorizationTest.java:780)
> Caused by: java.util.concurrent.TimeoutException: The procedure 18 is still 
> running
>   at 
> org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.testTagBasedTablePolicy(HBaseRangerAuthorizationTest.java:780)
> The next part of the test is ok:
> [INFO] Running 
> org.apache.ranger.authorization.hbase.RangerAuthorizationFilterTest
> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.505 
> s - in org.apache.ranger.authorization.hbase.RangerAuthorizationFilterTest



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RANGER-1908) Build (unit tests) failed in HBase Security Plugin

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266840#comment-16266840
 ] 

Nigel Jones commented on RANGER-1908:
-

More tests
 * MacOS - 4 consecutive builds all clean
 * VM (a) @IBM. Failed 2/3, worked 1/3. looks timing related unit test case 
failure
 * VM (b) @Azure (lower spec) failed 3/3 same as above. timing related

At this point I will close this JIRA as non reproducible. Will keep an eye on 
any timing related issues to see if perhaps we need to relax some timings and 
raise a new JIRA if so.

> Build (unit tests) failed in HBase Security Plugin
> --
>
> Key: RANGER-1908
> URL: https://issues.apache.org/jira/browse/RANGER-1908
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>    Reporter: Nigel Jones
> Attachments: ranger-20171124a.log
>
>
> MacOS
> Java 1.8 (152, oracle)
> Maven 3.5.0
> Build fails at
> [INFO] HBase Security Plugin .. FAILURE [11:36 
> min]
> The failing test is:
> [INFO] Running 
> org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest
> Formatting using clusterid: testClusterID
> [ERROR] Tests run: 20, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 
> 689.962 s <<< FAILURE! - in 
> org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest
> [ERROR] 
> testTagBasedTablePolicy(org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest)
>   Time elapsed: 611.879 s  <<< ERROR!
> org.apache.hadoop.hbase.exceptions.TimeoutIOException: 
> java.util.concurrent.TimeoutException: The procedure 18 is still running
>   at 
> org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.testTagBasedTablePolicy(HBaseRangerAuthorizationTest.java:780)
> Caused by: java.util.concurrent.TimeoutException: The procedure 18 is still 
> running
>   at 
> org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.testTagBasedTablePolicy(HBaseRangerAuthorizationTest.java:780)
> The next part of the test is ok:
> [INFO] Running 
> org.apache.ranger.authorization.hbase.RangerAuthorizationFilterTest
> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.505 
> s - in org.apache.ranger.authorization.hbase.RangerAuthorizationFilterTest



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (RANGER-1908) Build (unit tests) failed in HBase Security Plugin

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones updated RANGER-1908:

Summary: Build (unit tests) failed in HBase Security Plugin  (was: Build 
failed in HBase Security Plugin)

> Build (unit tests) failed in HBase Security Plugin
> --
>
> Key: RANGER-1908
> URL: https://issues.apache.org/jira/browse/RANGER-1908
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>    Reporter: Nigel Jones
> Attachments: ranger-20171124a.log
>
>
> MacOS
> Java 1.8 (152, oracle)
> Maven 3.5.0
> Build fails at
> [INFO] HBase Security Plugin .. FAILURE [11:36 
> min]
> The failing test is:
> [INFO] Running 
> org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest
> Formatting using clusterid: testClusterID
> [ERROR] Tests run: 20, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 
> 689.962 s <<< FAILURE! - in 
> org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest
> [ERROR] 
> testTagBasedTablePolicy(org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest)
>   Time elapsed: 611.879 s  <<< ERROR!
> org.apache.hadoop.hbase.exceptions.TimeoutIOException: 
> java.util.concurrent.TimeoutException: The procedure 18 is still running
>   at 
> org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.testTagBasedTablePolicy(HBaseRangerAuthorizationTest.java:780)
> Caused by: java.util.concurrent.TimeoutException: The procedure 18 is still 
> running
>   at 
> org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.testTagBasedTablePolicy(HBaseRangerAuthorizationTest.java:780)
> The next part of the test is ok:
> [INFO] Running 
> org.apache.ranger.authorization.hbase.RangerAuthorizationFilterTest
> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.505 
> s - in org.apache.ranger.authorization.hbase.RangerAuthorizationFilterTest



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RANGER-1909) Build failure in Kafka Security plugin Unit Tests

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265425#comment-16265425
 ] 

Nigel Jones commented on RANGER-1909:
-

This is on Ubuntu 17.10 (azure)
Another linux environment with 16.04 did build the same source fine, though 
macos failed (see RANGER-1908)
This could be timing related, and down to the fact I'm using a 'burstable' vm 
within a small budget (class B2S - 2 CPU. but min cap 20%)
This was initially used to compare against the system that failed with 1908 
(which is a fast macbook pro), but subsequently I used a non-restricted 4 CPU 
environment which does build cleanly.
Opening jira to identify/record cause

> Build failure in Kafka Security plugin Unit Tests
> -
>
> Key: RANGER-1909
> URL: https://issues.apache.org/jira/browse/RANGER-1909
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>    Reporter: Nigel Jones
> Attachments: ranger-20171124a.log
>
>
> Java 1.8 (151, openjdk)
> Maven 3.5.0
> Build fails at
> [INFO] KAFKA Security Plugin .. FAILURE [22:07 
> min]
> [INFO] Running 
> org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerGSSTest
> log4j:WARN No appenders could be found for logger 
> (org.apache.kerby.kerberos.kerb.identity.backend.AbstractIdentityBackend).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 
> 1,120.51 s <<< FAILURE! - in 
> org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerGSSTest
> [ERROR] 
> testAuthorizedRead(org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerGSSTest)
>   Time elapsed: 1,109.032 s  <<< FAILURE!
> java.lang.AssertionError
>   at 
> org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerGSSTest.testAuthorizedRead(KafkaRangerAuthorizerGSSTest.java:243)
> [INFO] Running 
> org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerTest
> log4j:WARN No appenders could be found for logger 
> (org.apache.zookeeper.server.ZooKeeperServerMain).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 
> 114.827 s - in 
> org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerTest
> [INFO] Running 
> org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerSASLSSLTest
> log4j:WARN No appenders could be found for logger 
> (org.apache.zookeeper.server.ZooKeeperServerMain).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 72.214 
> s - in 
> org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerSASLSSLTest
> [INFO] 
> [INFO] Results:
> [INFO] 
> [ERROR] Failures: 
> [ERROR]   KafkaRangerAuthorizerGSSTest.testAuthorizedRead:243
> [INFO] 
> [ERROR] Tests run: 8, Failures: 1, Errors: 0, Skipped: 0



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (RANGER-1909) Build failure in Kafka Security plugin Unit Tests

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones updated RANGER-1909:

Attachment: ranger-20171124a.log

> Build failure in Kafka Security plugin Unit Tests
> -
>
> Key: RANGER-1909
> URL: https://issues.apache.org/jira/browse/RANGER-1909
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>    Reporter: Nigel Jones
> Attachments: ranger-20171124a.log
>
>
> Java 1.8 (151, openjdk)
> Maven 3.5.0
> Build fails at
> [INFO] KAFKA Security Plugin .. FAILURE [22:07 
> min]
> [INFO] Running 
> org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerGSSTest
> log4j:WARN No appenders could be found for logger 
> (org.apache.kerby.kerberos.kerb.identity.backend.AbstractIdentityBackend).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 
> 1,120.51 s <<< FAILURE! - in 
> org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerGSSTest
> [ERROR] 
> testAuthorizedRead(org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerGSSTest)
>   Time elapsed: 1,109.032 s  <<< FAILURE!
> java.lang.AssertionError
>   at 
> org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerGSSTest.testAuthorizedRead(KafkaRangerAuthorizerGSSTest.java:243)
> [INFO] Running 
> org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerTest
> log4j:WARN No appenders could be found for logger 
> (org.apache.zookeeper.server.ZooKeeperServerMain).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 
> 114.827 s - in 
> org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerTest
> [INFO] Running 
> org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerSASLSSLTest
> log4j:WARN No appenders could be found for logger 
> (org.apache.zookeeper.server.ZooKeeperServerMain).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 72.214 
> s - in 
> org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerSASLSSLTest
> [INFO] 
> [INFO] Results:
> [INFO] 
> [ERROR] Failures: 
> [ERROR]   KafkaRangerAuthorizerGSSTest.testAuthorizedRead:243
> [INFO] 
> [ERROR] Tests run: 8, Failures: 1, Errors: 0, Skipped: 0



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (RANGER-1909) Build failure in Kafka Security plugin Unit Tests

[Nigel Jones (JIRA)]

Nigel Jones created RANGER-1909:
---

 Summary: Build failure in Kafka Security plugin Unit Tests
 Key: RANGER-1909
 URL: https://issues.apache.org/jira/browse/RANGER-1909
 Project: Ranger
  Issue Type: Bug
  Components: Ranger
Reporter: Nigel Jones


Java 1.8 (151, openjdk)
Maven 3.5.0
Build fails at
[INFO] KAFKA Security Plugin .. FAILURE [22:07 min]

[INFO] Running 
org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerGSSTest
log4j:WARN No appenders could be found for logger 
(org.apache.kerby.kerberos.kerb.identity.backend.AbstractIdentityBackend).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
info.
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 
1,120.51 s <<< FAILURE! - in 
org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerGSSTest
[ERROR] 
testAuthorizedRead(org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerGSSTest)
  Time elapsed: 1,109.032 s  <<< FAILURE!
java.lang.AssertionError
at 
org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerGSSTest.testAuthorizedRead(KafkaRangerAuthorizerGSSTest.java:243)

[INFO] Running 
org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerTest
log4j:WARN No appenders could be found for logger 
(org.apache.zookeeper.server.ZooKeeperServerMain).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
info.
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 114.827 
s - in 
org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerTest
[INFO] Running 
org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerSASLSSLTest
log4j:WARN No appenders could be found for logger 
(org.apache.zookeeper.server.ZooKeeperServerMain).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
info.
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 72.214 s 
- in 
org.apache.ranger.authorization.kafka.authorizer.KafkaRangerAuthorizerSASLSSLTest
[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   KafkaRangerAuthorizerGSSTest.testAuthorizedRead:243
[INFO] 
[ERROR] Tests run: 8, Failures: 1, Errors: 0, Skipped: 0




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (RANGER-1908) Build failed in HBase Security Plugin

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones updated RANGER-1908:

Attachment: ranger-20171124a.log

> Build failed in HBase Security Plugin
> -
>
> Key: RANGER-1908
> URL: https://issues.apache.org/jira/browse/RANGER-1908
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>    Reporter: Nigel Jones
> Attachments: ranger-20171124a.log
>
>
> MacOS
> Java 1.8 (152, oracle)
> Maven 3.5.0
> Build fails at
> [INFO] HBase Security Plugin .. FAILURE [11:36 
> min]
> The failing test is:
> [INFO] Running 
> org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest
> Formatting using clusterid: testClusterID
> [ERROR] Tests run: 20, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 
> 689.962 s <<< FAILURE! - in 
> org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest
> [ERROR] 
> testTagBasedTablePolicy(org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest)
>   Time elapsed: 611.879 s  <<< ERROR!
> org.apache.hadoop.hbase.exceptions.TimeoutIOException: 
> java.util.concurrent.TimeoutException: The procedure 18 is still running
>   at 
> org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.testTagBasedTablePolicy(HBaseRangerAuthorizationTest.java:780)
> Caused by: java.util.concurrent.TimeoutException: The procedure 18 is still 
> running
>   at 
> org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.testTagBasedTablePolicy(HBaseRangerAuthorizationTest.java:780)
> The next part of the test is ok:
> [INFO] Running 
> org.apache.ranger.authorization.hbase.RangerAuthorizationFilterTest
> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.505 
> s - in org.apache.ranger.authorization.hbase.RangerAuthorizationFilterTest



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (RANGER-1908) Build failed in HBase Security Plugin

[Nigel Jones (JIRA)]

Nigel Jones created RANGER-1908:
---

 Summary: Build failed in HBase Security Plugin
 Key: RANGER-1908
 URL: https://issues.apache.org/jira/browse/RANGER-1908
 Project: Ranger
  Issue Type: Bug
  Components: Ranger
Reporter: Nigel Jones


MacOS
Java 1.8 (152, oracle)
Maven 3.5.0

Build fails at
[INFO] HBase Security Plugin .. FAILURE [11:36 min]

The failing test is:
[INFO] Running 
org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest
Formatting using clusterid: testClusterID
[ERROR] Tests run: 20, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 
689.962 s <<< FAILURE! - in 
org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest
[ERROR] 
testTagBasedTablePolicy(org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest)
  Time elapsed: 611.879 s  <<< ERROR!
org.apache.hadoop.hbase.exceptions.TimeoutIOException: 
java.util.concurrent.TimeoutException: The procedure 18 is still running
at 
org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.testTagBasedTablePolicy(HBaseRangerAuthorizationTest.java:780)
Caused by: java.util.concurrent.TimeoutException: The procedure 18 is still 
running
at 
org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.testTagBasedTablePolicy(HBaseRangerAuthorizationTest.java:780)

The next part of the test is ok:
[INFO] Running 
org.apache.ranger.authorization.hbase.RangerAuthorizationFilterTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.505 s 
- in org.apache.ranger.authorization.hbase.RangerAuthorizationFilterTest





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Build issues - inconsistent

[Nigel Jones]

On 2017-11-22 14:46, "Nigel Jones"<nigel.l.jo...@gmail.com> wrote: 

> Thanks, that helps... I'll get all three environments on 3.5.0 and then retry 
> (after a pull)

Though the enunciator issue is resolved with maven 3.5.0, I'm currently to see 
the same other errors - one clean build (linux), 2 failures (linux, macos). 
I'll raise a JIRA on the two specific failures and take it from there...

Re: Build issues - inconsistent

[Nigel Jones]

On 2017-11-22 11:45, Colm O hEigeartaigh  wrote: 
> I can't reproduce the test failures in Kafka, HBase or Hive. However, I can
> reproduce the build errors in security-admin. It's caused by this bug in
> Maven:
> 
> https://issues.apache.org/jira/browse/MNG-6298
> 
> The solution is to use Apache Maven 3.5.0 instead of 3.5.2.

Thanks, that helps... I'll get all three environments on 3.5.0 and then retry 
(after a pull)

I imagine this kind of error is rare, but wonder if it would be worth using the 
maven-enforcer-plugin to validate that at least the mvn version (and anything 
else we want to check) is a 'sane' version (for us). See 
https://issues.apache.org/jira/browse/ATLAS-2245?filter=-2. We could check only 
min 3.5.0, or I guess exclude 3.5.2 with a error OR WARNING if we wanted, could 
be clearer? I'm happy to propose a patch if this is felt to be useful (once I'm 
over my build issues!)

Re: Build issues - inconsistent

[Nigel Jones]

Consistent in failing, but it's unclear how repeatable any specific error is... 
I'll try some more - this was the test from just 4 attempts so I wanted to 
check I was doing things correct. 

My invocation in each case is 
mvn -DskipTests=false clean compile package install assembly:assembly

Maven versions were
a) 3.5.2 (local mac)
b) 3.5.0 (azure ubuntu 17.10)
c) 3.3.9 (cloudfoundry/openstack ubuntu 16.04)

We hit funny maven issues in Atlas - jenkins is now at 3.5.0. I proposed a 
validation check in ATLAS-2245 to persuade everyone to upgrade.

I'm assuming lots of people here use macOS. Whilst my OS level is forward I 
doubt that's the issue. Is everyone still building with java 1.7? And what 
version of maven? Any other dependent components?

Thanks
Nigel.

On 2017-11-21 15:48, Colm O hEigeartaigh <cohei...@apache.org> wrote: 
> What maven version are you using? Are the test failures consistent or
> sporadic?
> 
> Colm.
> 
> On Tue, Nov 21, 2017 at 3:09 PM, Nigel Jones <nigel.l.jo...@gmail.com>
> wrote:
> 
> > I'm experiencing some issues getting the ranger build to work.
> >  a) macOS (10.13.2 b4), oracle 8 *(152)
> >
> > [INFO] HBase Security Plugin .. FAILURE [11:37
> > min]
> >
> > [INFO] Running org.apache.ranger.authorization.hbase.
> > HBaseRangerAuthorizationTest
> > Formatting using clusterid: testClusterID
> > [ERROR] Tests run: 20, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> > 690.138 s <<< FAILURE! - in org.apache.ranger.authorization.hbase.
> > HBaseRangerAuthorizationTest
> > [ERROR] 
> > testTagBasedTablePolicy(org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest)
> > Time elapsed: 612.923 s  <<< ERROR!
> > org.apache.hadoop.hbase.exceptions.TimeoutIOException:
> > java.util.concurrent.TimeoutException: The procedure 18 is still running
> > at org.apache.ranger.authorization.hbase.
> > HBaseRangerAuthorizationTest.testTagBasedTablePolicy(
> > HBaseRangerAuthorizationTest.java:780)
> > Caused by: java.util.concurrent.TimeoutException: The procedure 18 is
> > still running
> > at org.apache.ranger.authorization.hbase.
> > HBaseRangerAuthorizationTest.testTagBasedTablePolicy(
> > HBaseRangerAuthorizationTest.java:780)
> >
> >
> > >> This looks like potentially a timeout, though at 690s maybe in reality
> > a logical issue
> >
> > b) Ubuntu 17.10, MS Azure, openjdk 8 (151)
> >
> > [INFO] Hive Security Plugin ... FAILURE [
> > 41.084 s]
> > ERROR StatusLogger No log4j2 configuration file found. Using default
> > configuration: logging only errors to the console.
> > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> > 30.286 s <<< FAILURE! - in org.apache.ranger.services.
> > hive.HIVERangerAuthorizerTest
> > [ERROR] org.apache.ranger.services.hive.HIVERangerAuthorizerTest  Time
> > elapsed: 30.286 s  <<< ERROR!
> > java.sql.SQLException: Could not open client transport with JDBC Uri:
> > jdbc:hive2://localhost:36781: java.net.ConnectException: Connection refused
> > (Connection refused)
> > at org.apache.ranger.services.hive.HIVERangerAuthorizerTest.setup(
> > HIVERangerAuthorizerTest.java:103)
> > Caused by: org.apache.thrift.transport.TTransportException:
> > java.net.ConnectException: Connection refused (Connection refused)
> > at org.apache.ranger.services.hive.HIVERangerAuthorizerTest.setup(
> > HIVERangerAuthorizerTest.java:103)
> > Caused by: java.net.ConnectException: Connection refused (Connection
> > refused)
> > at org.apache.ranger.services.hive.HIVERangerAuthorizerTest.setup(
> > HIVERangerAuthorizerTest.java:103)
> >
> >
> > >> This one is clearly a jdbc connection issue, perhaps hive didn't start
> > 
> >
> > c) Ubuntu 16.04, our internal IBM cloudfoundry/openstack environment -
> > open jdk 1.8 (151)
> >
> > [INFO] KAFKA Security Plugin .. FAILURE [18:58
> > min]
> >
> > [INFO] Running org.apache.ranger.authorization.kafka.authorizer.
> > KafkaRangerAuthorizerGSSTest
> > log4j:WARN No appenders could be found for logger
> > (org.apache.kerby.kerberos.kerb.identity.backend.AbstractIdentityBackend).
> > log4j:WARN Please initialize the log4j system properly.
> > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
> > more info.
> > [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> > 1,108.807 s <<< FAILURE! - in org.apache.ranger.a

Build issues - inconsistent

[Nigel Jones]

/.m2/repository/com/webcohesion/enunciate/enunciate-docs/2.8.0/enunciate-docs-2.8.0.jar!/com/webcohesion/enunciate/modules/docs/docs.fmt"
 in macro "processResourceGroup" at line 518, column 5]
[ERROR] - Reached through: @file name=(resourceGroup.slug + ".ht...  
[in template 
"jar:file:/Users/jonesn/.m2/repository/com/webcohesion/enunciate/enunciate-docs/2.8.0/enunciate-docs-2.8.0.jar!/com/webcohesion/enunciate/modules/docs/docs.fmt"
 in macro "processResourceGroup" at line 517, column 3]
[ERROR] - Reached through: @processResourceGroup resourceGroup=r...  
[in template 
"jar:file:/Users/jonesn/.m2/repository/com/webcohesion/enunciate/enunciate-docs/2.8.0/enunciate-docs-2.8.0.jar!/com/webcohesion/enunciate/modules/docs/docs.fmt"
 at line 238, column 11]
[ERROR] ~ Reached through: #nested  [in template 
"jar:file:/Users/jonesn/.m2/repository/com/webcohesion/enunciate/enunciate-docs/2.8.0/enunciate-docs-2.8.0.jar!/com/webcohesion/enunciate/modules/docs/docs.fmt"
 in macro "boilerplate" at line 141, column 9]
[ERROR] ~ Reached through: @boilerplate pagenav=pagenav  [in template 
"jar:file:/Users/jonesn/.m2/repository/com/webcohesion/enunciate/enunciate-docs/2.8.0/enunciate-docs-2.8.0.jar!/com/webcohesion/enunciate/modules/docs/docs.fmt"
 at line 192, column 3]
[ERROR] - Reached through: @file name=indexPageName  [in template 
"jar:file:/Users/jonesn/.m2/repository/com/webcohesion/enunciate/enunciate-docs/2.8.0/enunciate-docs-2.8.0.jar!/com/webcohesion/enunciate/modules/docs/docs.fmt"
 at line 181, column 1]
[ERROR] : InvocationTargetException: 
javax/annotation/security/RolesAllowed: javax.annotation.security.RolesAllowed
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :security-admin-web


>> ---

So that's 4 attempts, 4 different results. I could open JIRAs, but I'm 
concerned at the lack of consistency. What do I need to check in my dev 
environments? Are there additional prereqs not checked for in maven?

My Atlas builds are more consistent (occasional timeout, but generally sharing 
results with others)

Many thanks
Nigel Jones
nigel.l.jo...@gmail.com

Building Ranger under MacOS

[Nigel Jones]

Is ranger (master) expected to build properly under MacOS 

I've been building Apache Atlas ok in my environment (I am using JDK 1.8 not 
1.7, maven 3.5.2) but hit a test issues from mac that I didn't see on linux 
(ubuntu) so before persuing/raising a JIRA I wanted to check what the consensus 
was on how well the mac dev environment works currently for ranger?


[ERROR] Tests run: 20, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 
119.937 s <<< FAILURE! - in 
org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest
[ERROR] 
testTagBasedTablePolicy(org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest)
  Time elapsed: 68.325 s  <<< ERROR!
org.apache.hadoop.hbase.client.RetriesExhaustedException: 
Failed after attempts=351, exceptions:
Wed Nov 08 14:08:03 GMT 2017, null, java.net.SocketTimeoutException: 
callTimeout=6, callDuration=68476: row 'temp3,,' on table 'hbase:meta' at 
region=hbase:meta,,1.1588230740, hostname=nigels-mbp,56205,1510150008524, 
seqNum=0

at 
org.apache.ranger.authorization.hbase.HBaseRangerAuthorizationTest.testTagBasedTablePolicy(HBaseRangerAuthorizationTest.java:763)
Caused by: org.apache.hadoop.ipc.RemoteException: 
Failed after attempts=351, exceptions:
Wed Nov 08 14:08:03 GMT 2017, null, java.net.SocketTimeoutException: 
callTimeout=6, callDuration=68476: row 'temp3,,' on table 'hbase:meta' at 
region=hbase:meta,,1.1588230740, hostname=nigels-mbp,56205,1510150008524, 
seqNum=0


many thanks
Nigel.

[jira] [Updated] (RANGER-1850) Impersonation/proxy user support for gaiandb ranger plugin

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones updated RANGER-1850:

Attachment: GaianDBAuth.docx

very initial draft. For reference. Will add more info here once ready for review

> Impersonation/proxy user support for gaiandb ranger plugin
> --
>
> Key: RANGER-1850
> URL: https://issues.apache.org/jira/browse/RANGER-1850
> Project: Ranger
>  Issue Type: Sub-task
>  Components: plugins
>    Reporter: Nigel Jones
> Attachments: GaianDBAuth.docx
>
>
> Applications/users could connect to gaianDB using their own authentication 
> information - for example userid/password in the simple case. Here the ranger 
> plugin will use that id for policy checks.
> However in a multi tiered architecture a service id (aka non personal 
> account) may be used, and somehow the user to be impersonated is passed via 
> an additional property. This has a number of implications to the system 
> configuration, derby/gaiandb configuration & the plugin implementation. 
> Opening this Jira as a placeholder and will add a document soon (++days) on 
> the same to capture some of the discussion around this area in recent days.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RANGER-1488) Create Ranger plugin for gaiandb

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16157125#comment-16157125
 ] 

Nigel Jones commented on RANGER-1488:
-

Originally it was planned that authorization would be based purely on the user 
that connects to gaianDB (which can support local users, or LDAP - the latter 
being more likely in an enterprise) ie this user would be used within the 
ranger/gaiandb plugin to determine access priviliges.

However feedback this week has suggested that *in addition*, it should be 
possible to pass the userid as a connection parameter (derby supports this), 
since it will be a 'non personal' or 'generic' server user id that may be 
connecting to gaiandb in some use cases. 

> Create Ranger plugin for gaiandb
> 
>
> Key: RANGER-1488
> URL: https://issues.apache.org/jira/browse/RANGER-1488
> Project: Ranger
>  Issue Type: New Feature
>  Components: plugins
>    Reporter: Nigel Jones
>    Assignee: Nigel Jones
>  Labels: VirtualDataConnector
>
> GaianDB is a distributed, federated database built on Apache Derby.
> The documentation is at 
> https://github.com/gaiandb/gaiandb/blob/master/README.md - navigate up for 
> the source.
> As part of a Virtual Connector Project based on Atlas we are using gaianDB to 
> provide a virtualization layer. We need to control access to underlying 
> resources and will be building a Ranger plugin for gaiandb to support this. 
> GaianDB already has support for a form of policy plugin which allows the SQL 
> to be intercepted.
> It is unclear if this code will sit
>  - external to ranger/atlas ie in the gaianDB sources
>  - as a sample in the Atlas project (so that we collate what is needed for 
> the virtual connector project)
>  - as a sample, or component in the Ranger project
> However it seems helpful to describe the proposal here and get 
> feedback/dialogue in the ranger community :-)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RANGER-1488) Create Ranger plugin for gaiandb

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16157116#comment-16157116
 ] 

Nigel Jones commented on RANGER-1488:
-

An outline of the broader use cases within which the gaiandb ranger plugin 
would be used can be found at 
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=69407333

However the plugin should be self-contained, and not dependent on the other 
ongoing work around atlas and open metadata. It should fundamentally work 
within the existing ranger framework, and not be dependent on the ongoing work 
in particular in atlas around open metadata, OMAS interfaces and any proposed 
changes to tagsync, metadata exchange etc.

the capabilities will include
 - simple permit/deny on data access
 - masking ( is sufficient as a first step, but could be elaborate as in 
hive)
 - supporting both resource based policies & tag based policies

Not required by current use cases but a useful future enhancement would include
 - row filtering

A desire has also been expressed such that if a user were to do a 'select *' 
from a gaiandb virtual table, any columns the user cannot see should be 
stripped out. However this may cause inconsistencies with any metadata queries 
(which cannot be filtered by a ranger plugin since gaiandb does not yet support 
this). A consideration, but likely to be tricky at this time. 

> Create Ranger plugin for gaiandb
> 
>
> Key: RANGER-1488
> URL: https://issues.apache.org/jira/browse/RANGER-1488
> Project: Ranger
>  Issue Type: New Feature
>  Components: plugins
>    Reporter: Nigel Jones
>Assignee: Nigel Jones
>  Labels: VirtualDataConnector
>
> GaianDB is a distributed, federated database built on Apache Derby.
> The documentation is at 
> https://github.com/gaiandb/gaiandb/blob/master/README.md - navigate up for 
> the source.
> As part of a Virtual Connector Project based on Atlas we are using gaianDB to 
> provide a virtualization layer. We need to control access to underlying 
> resources and will be building a Ranger plugin for gaiandb to support this. 
> GaianDB already has support for a form of policy plugin which allows the SQL 
> to be intercepted.
> It is unclear if this code will sit
>  - external to ranger/atlas ie in the gaianDB sources
>  - as a sample in the Atlas project (so that we collate what is needed for 
> the virtual connector project)
>  - as a sample, or component in the Ranger project
> However it seems helpful to describe the proposal here and get 
> feedback/dialogue in the ranger community :-)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RANGER-1572) Provide simple example of running ranger with docker

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150756#comment-16150756
 ] 

Nigel Jones commented on RANGER-1572:
-

In ATLAS-2012 we added a docker script to Atlas, this was quite different to my 
github example above.
Ranger however is a little more complex in that beyond understanding the actual 
component, it's more difficult than atlas to get value from as a standalone 
component. In Atlas discussions we've acknowledged this and raised ATLAS-2095 
though its had little detailed thought yet.


> Provide simple example of running ranger with docker
> 
>
> Key: RANGER-1572
> URL: https://issues.apache.org/jira/browse/RANGER-1572
> Project: Ranger
>  Issue Type: New Feature
>  Components: Ranger
>    Reporter: Nigel Jones
>    Assignee: Nigel Jones
>
> When experimenting with Ranger it's very useful to be able to quickly run 
> ranger admin/solr/mysql
> Since I needed this for my education/experimentation I thought it might be 
> useful to others. Work in progress and doesn't work yet but raising this JIRA 
> in case anyone else has done the same, or is interested ;-)
> For now the code (just a few scripts) is at 
> https://github.com/planetf1/ranger-docker whilst I try and get this working. 
> Perhaps this could be added as an example down the line, or even used within 
> the build to create an image -- though there's lots of variables in terms of 
> config.
> As above.. not working yet ;-) WIP!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RANGER-1486) New usersync alternative for Atlas (vdc)

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150711#comment-16150711
 ] 

Nigel Jones commented on RANGER-1486:
-

Thanks... the suggestion is purely to scope the groups based on a list from 
Atlas (we'll call them roles there as that usually how they are used). That's 
really the only change, other than effectively an additional predicate, 
usersync would work in the same way. I'm hoping it's a pragmatic way to 
significantly reduce how much user/group info is retrieved from ldap in a large 
enterprise environment with many many different apps & environments, and within 
which currently only a few roles are relevant to the data lake environment (& 
hence atlas/ranger). Maybe in future there's other ideas, including whether 
ranger even needs to store these (I know it does now..) but that's for another 
day

Makes sense?

Thanks, Nigel.

> New usersync alternative for Atlas (vdc)
> 
>
> Key: RANGER-1486
> URL: https://issues.apache.org/jira/browse/RANGER-1486
> Project: Ranger
>  Issue Type: New Feature
>  Components: usersync
>Reporter: Nigel Jones
>Assignee: Nigel Jones
>  Labels: VirtualDataConnector
>
> As part of the Atlas Virtualization Data Connector work we are using this 
> within a large enterprise with a lot of users & groups stored in ldap.
> The connector -- which has a ranger plugin to apply access control policies 
> -- is used by a relatively small subset of these users. However that can't 
> easily be transcribed to an optimal ldap query.
> Since Atlas will have the definitive list of roles that are being used, this 
> new usersync will instead retrieve a list of roles from Atlas, and will then 
> use this list to retrieve only those users found in this list of roles from 
> LDAP.
> This is an alternative usersync so shouldn't conflict and will use the same 
> ranger APIs



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RANGER-1454) Support of Atlas GAF OMAS

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150546#comment-16150546
 ] 

Nigel Jones commented on RANGER-1454:
-

The assumption in this Jira is that the existing Ranger REST API remains as it 
is today. These changes to support new Atlas APIs are initially focussed only 
at getting tagsync working properly with the new data model & approach to 
classifications.

Once in place, since the Atlas Governance Engine OMAS api is consumer centric, 
it is intended purely for enforcement engines such as ranger, and therefore the 
Atlas can can extend as needed to improve integration with ranger. Better fine 
grained notifications from atlas to ranger may also make it easier to update 
policies more quickly, but would require other changes in ranger.

> Support of Atlas GAF OMAS 
> --
>
> Key: RANGER-1454
> URL: https://issues.apache.org/jira/browse/RANGER-1454
> Project: Ranger
>  Issue Type: New Feature
>  Components: tagsync
>    Reporter: Nigel Jones
>Assignee: Nigel Jones
>  Labels: VirtualDataConnector
>
> A new v2 glossary capability is proposed for Atlas in ATLAS-1410.
> As part of this the glossary model becomes more sophisticated. In order to 
> preserve the current simple tag(trait type):parms(trait instance)-entity 
> relationship a new tagsync process will be developed that makes use of a new 
> API proposed in ATLAS-1662. This would be an alternative to the current one 
> so that existing users could continue unaffected, and a change is only 
> required if moving to the v2 glossary implementation
> This will also allow the process of retrieving tags to be more efficient, and 
> this new API can form a consumer-centric interface to support multiple 
> enforcement technologies, including, but not restricted to, Ranger. 
> feel free to assign to me/modify permissions ;-)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (RANGER-1758) Atlas GE OMAS : Support new Kafka notifications

[Nigel Jones (JIRA)]

Nigel Jones created RANGER-1758:
---

 Summary: Atlas GE OMAS : Support new Kafka notifications
 Key: RANGER-1758
 URL: https://issues.apache.org/jira/browse/RANGER-1758
 Project: Ranger
  Issue Type: Sub-task
  Components: tagsync
Reporter: Nigel Jones


This JIRA adds in support for the new Governance Engine OMAS notifications 
coming from Atlas, avoiding the need to pull the entire list of 
classifications/assets from Atlas anytime there's an update, or periodically. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (RANGER-1757) Atlas GE OMAS : Support new REST api

[Nigel Jones (JIRA)]

Nigel Jones created RANGER-1757:
---

 Summary: Atlas GE OMAS : Support new REST api
 Key: RANGER-1757
 URL: https://issues.apache.org/jira/browse/RANGER-1757
 Project: Ranger
  Issue Type: Sub-task
  Components: tagsync
Reporter: Nigel Jones


This subtask will be used to implement the initial REST api queries, which 
could also be repeated at intervals.

I am splitting out this from the messaging/kafka notifications as the 
dependencies in atlas differ, and to aid in what can be implemented 
independently.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RANGER-1670) Change in Atlas Kafka consumer interface for Atlas tag sync.

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150204#comment-16150204
 ] 

Nigel Jones commented on RANGER-1670:
-

Hi,
 I thought it worth mentioning that beyond the just-being-released 0.8.1, 
there's a fair bit of work going in with Atlas currently, to enrichen the data 
model, and to provide new consumer centric interfaces. The glossary is being 
expanded a lot, to enhance the relationships between assets, business terms & 
classifications. The idea is to surface a more consumer centric view of this 
for ranger (and other enforcement engines). In RANGER-1454 I propose to switch 
to using this new API, but it's fair to say the focus so far has been on the 
atlas side, and it's only now we're getting more into the ranger aspects 
(thinking of!), which was a little tricky before the atlas side started 
firming up.

I'd love your feedback on the ideas and happy to chat more. 

> Change in Atlas Kafka consumer interface for Atlas tag sync.
> 
>
> Key: RANGER-1670
> URL: https://issues.apache.org/jira/browse/RANGER-1670
> Project: Ranger
>  Issue Type: Bug
>  Components: tagsync
>Reporter: Nixon Rodrigues
> Attachments: RANGER-1670.patch
>
>
> This change is dependent on upcoming 0.8.1 Atlas release.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RANGER-1486) New usersync alternative for Atlas (vdc)

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147516#comment-16147516
 ] 

Nigel Jones commented on RANGER-1486:
-

We don't yet have a concept of user roles in the proposed atlas model. I added 
thoughts in ATLAS-1768 though the specific dependency for usersync would be the 
API being delivered via ATLAS-1796 which can be dummied up prior to the 
necessary model updates.

One further thought... to limit interdependencies between atlas and ranger I 
think we should fall back to the old behaviour if the atlas being contacted 
does not support the new API. Initially it might also be wise to have a 
property to enable the new behaviour (so as to not hit an existing/old atlas 
server with a new API request), or to figure out (new JIRA?) an appropriate 
more generic version check or capability strategy

> New usersync alternative for Atlas (vdc)
> 
>
> Key: RANGER-1486
> URL: https://issues.apache.org/jira/browse/RANGER-1486
> Project: Ranger
>  Issue Type: New Feature
>  Components: usersync
>    Reporter: Nigel Jones
>    Assignee: Nigel Jones
>  Labels: VirtualDataConnector
>
> As part of the Atlas Virtualization Data Connector work we are using this 
> within a large enterprise with a lot of users & groups stored in ldap.
> The connector -- which has a ranger plugin to apply access control policies 
> -- is used by a relatively small subset of these users. However that can't 
> easily be transcribed to an optimal ldap query.
> Since Atlas will have the definitive list of roles that are being used, this 
> new usersync will instead retrieve a list of roles from Atlas, and will then 
> use this list to retrieve only those users found in this list of roles from 
> LDAP.
> This is an alternative usersync so shouldn't conflict and will use the same 
> ranger APIs



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RANGER-1486) New usersync alternative for Atlas (vdc)

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147417#comment-16147417
 ] 

Nigel Jones commented on RANGER-1486:
-

* Making the use of restricted Atlas roles configurable makes sense - I'd add 
that
* The role names wouldn't come from configuration - they would come from atlas 
via the 'gaf omas' call (note: this OMAS we are now calling the governance 
engine OMAS) (marketing, hr, datascientist etc). They would match the role 
names defined in LDAP.
* I would apply the ldap filter too - indeed in some ways the list of roles 
from atlas forms an additional predicate. However I'm not a specialist on ldap 
queries. We do need to consider efficiency
* We can get a notification (using the governance engine omas, which will 
deliver it over Kafka) when a new role is added or removed from atlas, which 
should cause us to redo the query. However new users we won't see as they are 
not defined in Atlas, only in LDAP & in general ldap doesn't offer an event. In 
fact I think with both OpenLDAP & Active Directory it would be possible to 
create triggers & generate notifications that ranger/usersync could use it's 
not really standard. Perhaps we should open a separate JIRA on that as it 
sounds like it could be genuinely useful in an enterprise environment (part B 
though would be then having a trigger mechanism to push the update not just to 
the ranger server, but to the plugins, and that's complex since they can be 
based on many technologies. At best an optional feature for a plugin?)
* In terms of a role changing, currently in Atlas I don't think we're thinking 
of any more than the role name. so there's not really anything to change. 
Whilst ideally we should track the guid of the Atlas defined role, since this 
doesn't tie up with ldap in any way it's difficult to see we could get much 
value from this.

I think in general the idea of being able to further restrict an ldap search 
makes sense in a large environment otherwise we're pointlessly pushing far too 
much user/group info into ranger, but we do need consensus on whether the 
community is happy that the scoping could come from Atlas. I hope that as long 
as the feature is optional this would be ok.

> New usersync alternative for Atlas (vdc)
> 
>
> Key: RANGER-1486
> URL: https://issues.apache.org/jira/browse/RANGER-1486
> Project: Ranger
>  Issue Type: New Feature
>  Components: usersync
>Reporter: Nigel Jones
>Assignee: Nigel Jones
>  Labels: VirtualDataConnector
>
> As part of the Atlas Virtualization Data Connector work we are using this 
> within a large enterprise with a lot of users & groups stored in ldap.
> The connector -- which has a ranger plugin to apply access control policies 
> -- is used by a relatively small subset of these users. However that can't 
> easily be transcribed to an optimal ldap query.
> Since Atlas will have the definitive list of roles that are being used, this 
> new usersync will instead retrieve a list of roles from Atlas, and will then 
> use this list to retrieve only those users found in this list of roles from 
> LDAP.
> This is an alternative usersync so shouldn't conflict and will use the same 
> ranger APIs



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (RANGER-1572) Provide simple example of running ranger with docker

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones updated RANGER-1572:

Issue Type: New Feature  (was: Bug)

> Provide simple example of running ranger with docker
> 
>
> Key: RANGER-1572
> URL: https://issues.apache.org/jira/browse/RANGER-1572
> Project: Ranger
>  Issue Type: New Feature
>  Components: Ranger
>    Reporter: Nigel Jones
>    Assignee: Nigel Jones
>
> When experimenting with Ranger it's very useful to be able to quickly run 
> ranger admin/solr/mysql
> Since I needed this for my education/experimentation I thought it might be 
> useful to others. Work in progress and doesn't work yet but raising this JIRA 
> in case anyone else has done the same, or is interested ;-)
> For now the code (just a few scripts) is at 
> https://github.com/planetf1/ranger-docker whilst I try and get this working. 
> Perhaps this could be added as an example down the line, or even used within 
> the build to create an image -- though there's lots of variables in terms of 
> config.
> As above.. not working yet ;-) WIP!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RANGER-676) Audit logging to Elasticsearch

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109098#comment-16109098
 ] 

Nigel Jones commented on RANGER-676:


I've heard colleagues express an interest in Elastic Search too.
Has there been any integration work on using Elastic search done to date?

> Audit logging to Elasticsearch
> --
>
> Key: RANGER-676
> URL: https://issues.apache.org/jira/browse/RANGER-676
> Project: Ranger
>  Issue Type: New Feature
>  Components: audit
>Affects Versions: 0.5.0
> Environment: HDP 2.3
>Reporter: Hari Sekhon
>
> Feature Request to have audit logging output to Elasticsearch since there is 
> an output to Solr but Elasticsearch is easier to operate at scale.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RANGER-1701) gaiandb: simple resource based policy

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109072#comment-16109072
 ] 

Nigel Jones commented on RANGER-1701:
-

this will include the resource mapper

> gaiandb: simple resource based policy
> -
>
> Key: RANGER-1701
> URL: https://issues.apache.org/jira/browse/RANGER-1701
> Project: Ranger
>  Issue Type: Sub-task
>  Components: plugins
>    Reporter: Nigel Jones
>
> Get the gaiandb plugin to a state where a simple resource based policy is 
> operational (permit/deny), and the ranger gui can be used to define.. and the 
> plugin will then implement this



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (RANGER-1720) gaiandb: resource lookup

[Nigel Jones (JIRA)]

Nigel Jones created RANGER-1720:
---

 Summary: gaiandb: resource lookup
 Key: RANGER-1720
 URL: https://issues.apache.org/jira/browse/RANGER-1720
 Project: Ranger
  Issue Type: Sub-task
  Components: plugins
Reporter: Nigel Jones


Add resource lookup for typeahead




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (RANGER-1719) Share ranger metadata with Atlas

[Nigel Jones (JIRA)]

Nigel Jones created RANGER-1719:
---

 Summary: Share ranger metadata with Atlas
 Key: RANGER-1719
 URL: https://issues.apache.org/jira/browse/RANGER-1719
 Project: Ranger
  Issue Type: New Feature
  Components: admin
Reporter: Nigel Jones


In ATLAS-1869 I propose that we capture various metadata from Ranger to make it 
available to atlas for stewardship, reporting, compliance.

Opening this as a placeholder to discuss the nature of required plugin or 
bridge in ranger

My thought is to keep this as loosely coupled as possible, so for example it 
could be a case of adding notifications so that ranger can publish changes in 
policies which a component related to atlas could pick up & mediate & inject 
into atlas. Ranger would remain the master for policy definitions, but once 
Atlas knows about them they could be linked with business terms in atlas & 
business-related policies, and we could drive governance reports using this data



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (RANGER-1454) Support of Atlas GAF OMAS

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones updated RANGER-1454:

Summary: Support of Atlas GAF OMAS   (was: Support of Atlas v2 glossary API 
proposal for tag source)

> Support of Atlas GAF OMAS 
> --
>
> Key: RANGER-1454
> URL: https://issues.apache.org/jira/browse/RANGER-1454
> Project: Ranger
>  Issue Type: New Feature
>  Components: tagsync
>    Reporter: Nigel Jones
>    Assignee: Nigel Jones
>  Labels: VirtualDataConnector
>
> A new v2 glossary capability is proposed for Atlas in ATLAS-1410.
> As part of this the glossary model becomes more sophisticated. In order to 
> preserve the current simple tag(trait type):parms(trait instance)-entity 
> relationship a new tagsync process will be developed that makes use of a new 
> API proposed in ATLAS-1662. This would be an alternative to the current one 
> so that existing users could continue unaffected, and a change is only 
> required if moving to the v2 glossary implementation
> This will also allow the process of retrieving tags to be more efficient, and 
> this new API can form a consumer-centric interface to support multiple 
> enforcement technologies, including, but not restricted to, Ranger. 
> feel free to assign to me/modify permissions ;-)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RANGER-1488) Create Ranger plugin for gaiandb

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094711#comment-16094711
 ] 

Nigel Jones commented on RANGER-1488:
-

I've created a high level breakdown of the stages I see in implementing this 
plugin. Currently the code on github hooks into gaian's policy framework and 
can intercept calls. There is also a simple configuration file defined (which I 
will post in the relevant task) but the code is not yet complete and does not 
initialize with ranger. this is the next step.

Is the community ok with plugin code being contributed into the ranger 
codebase, as long as licensing is ok? If so I will aim to add proposed patches 
to the review board tool .  It's probably reasonable to require the first 3 
subtasks are completed before any code is submitted to the project since that 
is the first point of having some coherent that adds value, but I can add 
sooner if that's helpful?

The work-in-progress can be viewed on github, but hopefully this is just a 
staging area

There is some more information about the reason we're building this plugin 
which is currently being driven on the Atlas side. See 
https://cwiki.apache.org/confluence/display/ATLAS/GaianDB+as+a+virtualizer+with+governance+enforcement

> Create Ranger plugin for gaiandb
> 
>
> Key: RANGER-1488
> URL: https://issues.apache.org/jira/browse/RANGER-1488
> Project: Ranger
>  Issue Type: New Feature
>  Components: plugins
>    Reporter: Nigel Jones
>    Assignee: Nigel Jones
>  Labels: VirtualDataConnector
>
> GaianDB is a distributed, federated database built on Apache Derby.
> The documentation is at 
> https://github.com/gaiandb/gaiandb/blob/master/README.md - navigate up for 
> the source.
> As part of a Virtual Connector Project based on Atlas we are using gaianDB to 
> provide a virtualization layer. We need to control access to underlying 
> resources and will be building a Ranger plugin for gaiandb to support this. 
> GaianDB already has support for a form of policy plugin which allows the SQL 
> to be intercepted.
> It is unclear if this code will sit
>  - external to ranger/atlas ie in the gaianDB sources
>  - as a sample in the Atlas project (so that we collate what is needed for 
> the virtual connector project)
>  - as a sample, or component in the Ranger project
> However it seems helpful to describe the proposal here and get 
> feedback/dialogue in the ranger community :-)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (RANGER-1699) gaiandb: Create initial plugin configuration file that can be deployed to ranger

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones updated RANGER-1699:

Summary: gaiandb: Create initial plugin configuration file that can be 
deployed to ranger  (was: Create initial plugin configuration file that can be 
deployed to ranger)

> gaiandb: Create initial plugin configuration file that can be deployed to 
> ranger
> 
>
> Key: RANGER-1699
> URL: https://issues.apache.org/jira/browse/RANGER-1699
> Project: Ranger
>  Issue Type: Sub-task
>  Components: plugins
>    Reporter: Nigel Jones
>
> Create initial configuration file for the gaiandb plugin for ranger which 
> will define policies, resources.
> Additional jiras will be opened to add extra capabilities later on. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (RANGER-1703) gaiandb - add data masking (similar to hive) for resource based policy

[Nigel Jones (JIRA)]

Nigel Jones created RANGER-1703:
---

 Summary: gaiandb - add data masking (similar to hive) for resource 
based policy
 Key: RANGER-1703
 URL: https://issues.apache.org/jira/browse/RANGER-1703
 Project: Ranger
  Issue Type: Sub-task
  Components: plugins
Reporter: Nigel Jones


Support data masking in gaindb plugin. Initially implement for a resource based 
policy



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (RANGER-1704) gaiandb - add data masking support for tag based policy

[Nigel Jones (JIRA)]

Nigel Jones created RANGER-1704:
---

 Summary: gaiandb - add data masking support for tag based policy
 Key: RANGER-1704
 URL: https://issues.apache.org/jira/browse/RANGER-1704
 Project: Ranger
  Issue Type: Sub-task
  Components: plugins
Reporter: Nigel Jones


Support data masking of columns etc using a tag based policy



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (RANGER-1702) gaiandb: tag based policies

[Nigel Jones (JIRA)]

Nigel Jones created RANGER-1702:
---

 Summary: gaiandb: tag based policies
 Key: RANGER-1702
 URL: https://issues.apache.org/jira/browse/RANGER-1702
 Project: Ranger
  Issue Type: Sub-task
  Components: plugins
Reporter: Nigel Jones


Update plugin to allow tag based policies to be used for simple permit/deny 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (RANGER-1701) gaiandb: simple resource based policy

[Nigel Jones (JIRA)]

Nigel Jones created RANGER-1701:
---

 Summary: gaiandb: simple resource based policy
 Key: RANGER-1701
 URL: https://issues.apache.org/jira/browse/RANGER-1701
 Project: Ranger
  Issue Type: Sub-task
  Components: plugins
Reporter: Nigel Jones


Get the gaiandb plugin to a state where a simple resource based policy is 
operational (permit/deny), and the ranger gui can be used to define.. and the 
plugin will then implement this



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (RANGER-1700) gaiandb: Integrate with ranger libraries - plugin builds, initializes

[Nigel Jones (JIRA)]

Nigel Jones created RANGER-1700:
---

 Summary: gaiandb: Integrate with ranger libraries - plugin builds, 
initializes
 Key: RANGER-1700
 URL: https://issues.apache.org/jira/browse/RANGER-1700
 Project: Ranger
  Issue Type: Sub-task
  Components: plugins
Reporter: Nigel Jones


Get the plugin to a state where it will initialize with a simple configuration 
and can be deployed to gaianDB. It may not actually action any policies at this 
state



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (RANGER-1487) generate rules from Governance definitions in Atlas

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084003#comment-16084003
 ] 

Nigel Jones edited comment on RANGER-1487 at 7/12/17 1:47 PM:
--

Having considered this approach some more, ALTAS-1869 will now discuss a 
mechanism for adding the capability to capture some metadata about ranger rules 
within atlas. This can allow business concepts & organization policies to be 
linked with specific rules that implement these policies. I will add more info 
there and create an similar jira for ranger if we feel that info is useful in 
atlas


was (Author: jonesn):
Having considered this approach some more, RANGER-1869 will now discuss a 
mechanism for adding the capability to capture some metadata about ranger rules 
within atlas. This can allow business concepts & organization policies to be 
linked with specific rules that implement these policies

> generate rules from Governance definitions in Atlas
> ---
>
> Key: RANGER-1487
> URL: https://issues.apache.org/jira/browse/RANGER-1487
> Project: Ranger
>  Issue Type: New Feature
>  Components: Ranger
>Reporter: Nigel Jones
>Assignee: Nigel Jones
>  Labels: VirtualDataConnector
>
> As part of the Atlas virtual connector project we intend to build "rules" 
> that relate to classifications within Atlas itself. These are logical 
> definitions that will then be mapped to specific implementations such as 
> rules/policies in ranger (but could also be in other systems like Oracle, or 
> Apache Sentry -- different enforcement approaches for different data access 
> points).
> Opening this Jira to build a "sync/build/link" tool that will generate ranger 
> rules where possible.
> (Very rough description ... will update & clarify)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RANGER-1487) generate rules from Governance definitions in Atlas

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084003#comment-16084003
 ] 

Nigel Jones commented on RANGER-1487:
-

Having considered this approach some more, RANGER-1869 will now discuss a 
mechanism for adding the capability to capture some metadata about ranger rules 
within atlas. This can allow business concepts & organization policies to be 
linked with specific rules that implement these policies

> generate rules from Governance definitions in Atlas
> ---
>
> Key: RANGER-1487
> URL: https://issues.apache.org/jira/browse/RANGER-1487
> Project: Ranger
>  Issue Type: New Feature
>  Components: Ranger
>    Reporter: Nigel Jones
>Assignee: Nigel Jones
>  Labels: VirtualDataConnector
>
> As part of the Atlas virtual connector project we intend to build "rules" 
> that relate to classifications within Atlas itself. These are logical 
> definitions that will then be mapped to specific implementations such as 
> rules/policies in ranger (but could also be in other systems like Oracle, or 
> Apache Sentry -- different enforcement approaches for different data access 
> points).
> Opening this Jira to build a "sync/build/link" tool that will generate ranger 
> rules where possible.
> (Very rough description ... will update & clarify)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (RANGER-1487) generate rules from Governance definitions in Atlas

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones resolved RANGER-1487.
-
Resolution: Fixed

> generate rules from Governance definitions in Atlas
> ---
>
> Key: RANGER-1487
> URL: https://issues.apache.org/jira/browse/RANGER-1487
> Project: Ranger
>  Issue Type: New Feature
>  Components: Ranger
>    Reporter: Nigel Jones
>    Assignee: Nigel Jones
>  Labels: VirtualDataConnector
>
> As part of the Atlas virtual connector project we intend to build "rules" 
> that relate to classifications within Atlas itself. These are logical 
> definitions that will then be mapped to specific implementations such as 
> rules/policies in ranger (but could also be in other systems like Oracle, or 
> Apache Sentry -- different enforcement approaches for different data access 
> points).
> Opening this Jira to build a "sync/build/link" tool that will generate ranger 
> rules where possible.
> (Very rough description ... will update & clarify)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (RANGER-1454) Support of Atlas v2 glossary API proposal for tag source

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16036957#comment-16036957
 ] 

Nigel Jones commented on RANGER-1454:
-

correct, thanks for spotting. the ATLAS-1662 I referred to does refer to the 
governance action OMAS, but the abstract of this Jira is badly named. It is to 
support the new glossary implementation in Atlas, but that is exposed to ranger 
through the governanance action OMAS. I'll update the title.

> Support of Atlas v2 glossary API proposal for tag source
> 
>
> Key: RANGER-1454
> URL: https://issues.apache.org/jira/browse/RANGER-1454
> Project: Ranger
>  Issue Type: New Feature
>  Components: tagsync
>    Reporter: Nigel Jones
>    Assignee: Nigel Jones
>  Labels: VirtualDataConnector
>
> A new v2 glossary capability is proposed for Atlas in ATLAS-1410.
> As part of this the glossary model becomes more sophisticated. In order to 
> preserve the current simple tag(trait type):parms(trait instance)-entity 
> relationship a new tagsync process will be developed that makes use of a new 
> API proposed in ATLAS-1662. This would be an alternative to the current one 
> so that existing users could continue unaffected, and a change is only 
> required if moving to the v2 glossary implementation
> This will also allow the process of retrieving tags to be more efficient, and 
> this new API can form a consumer-centric interface to support multiple 
> enforcement technologies, including, but not restricted to, Ranger. 
> feel free to assign to me/modify permissions ;-)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (RANGER-1572) Provide simple example of running ranger with docker

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16012031#comment-16012031
 ] 

Nigel Jones commented on RANGER-1572:
-

the latest example on github will now correctly setup and start ranger to the 
point of being able to login to the admin UI
Continuing 

> Provide simple example of running ranger with docker
> 
>
> Key: RANGER-1572
> URL: https://issues.apache.org/jira/browse/RANGER-1572
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>    Reporter: Nigel Jones
>    Assignee: Nigel Jones
>
> When experimenting with Ranger it's very useful to be able to quickly run 
> ranger admin/solr/mysql
> Since I needed this for my education/experimentation I thought it might be 
> useful to others. Work in progress and doesn't work yet but raising this JIRA 
> in case anyone else has done the same, or is interested ;-)
> For now the code (just a few scripts) is at 
> https://github.com/planetf1/ranger-docker whilst I try and get this working. 
> Perhaps this could be added as an example down the line, or even used within 
> the build to create an image -- though there's lots of variables in terms of 
> config.
> As above.. not working yet ;-) WIP!



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (RANGER-1186) Ranger Source: eclipse

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005069#comment-16005069
 ] 

Nigel Jones commented on RANGER-1186:
-

As an aside... I may look back at this, but for now am using Intelli-J as 
it's... simpler.. less fighting the IDE ;-)

> Ranger Source: eclipse
> --
>
> Key: RANGER-1186
> URL: https://issues.apache.org/jira/browse/RANGER-1186
> Project: Ranger
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 0.7.0
> Environment: eclipse neon, ubuntu 16.10
>Reporter: Nigel Jones
>Assignee: Pradeep Agrawal
>  Labels: VirtualDataConnector
> Attachments: RANGER-1186_Ranger_Source_Eclipse.pdf, ranger_me0.PNG, 
> ranger_me1.PNG
>
>
> I can happily build ranger in my Ubuntu 16.10 environment with
>  - open jdk 1.8.0_102
>  - maven 3.3.9
> I also have eclipse neon installed in this environment
> The build/setup instructions at 
> http://ranger.apache.org/quick_start_guide.html have minimal instructions for 
> eclipse.
> I imported the top level ranger directory using the m2e plugin as provided in 
> neon.
> This failed with numerous errors
> Is anyone actively building with eclipse? If so can we update the website 
> with current setup information?
> If not I'll use this JIRA to record the issues & hope to address them & 
> update the docs ;-)
> Screenshots to follow



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (RANGER-1454) Support of Atlas v2 glossary API proposal for tag source

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005066#comment-16005066
 ] 

Nigel Jones commented on RANGER-1454:
-

Yes I've raised this as a placeholder for now. Once the glossary itself becomes 
clearer, then I believe we should have a new API on Atlas which exposes the 
tags in a ranger-friendly form (also the subject of a related jira). 
The v2 glossary is more sophisticated so there are multiple levels of 
classifications .. but the new API (pointed to by a new tagsync, or a variant 
thereof)  can allow ranger to work the same as today. The discussion after that 
may be to consider whether any additional metadata should be available in 
ranger as a first-class entity so that it can easily be used in rules... but 
we're a fair few steps away from that ;-)

> Support of Atlas v2 glossary API proposal for tag source
> 
>
> Key: RANGER-1454
> URL: https://issues.apache.org/jira/browse/RANGER-1454
> Project: Ranger
>  Issue Type: New Feature
>  Components: tagsync
>    Reporter: Nigel Jones
>    Assignee: Nigel Jones
>  Labels: VirtualDataConnector
>
> A new v2 glossary capability is proposed for Atlas in ATLAS-1410.
> As part of this the glossary model becomes more sophisticated. In order to 
> preserve the current simple tag(trait type):parms(trait instance)-entity 
> relationship a new tagsync process will be developed that makes use of a new 
> API proposed in ATLAS-1662. This would be an alternative to the current one 
> so that existing users could continue unaffected, and a change is only 
> required if moving to the v2 glossary implementation
> This will also allow the process of retrieving tags to be more efficient, and 
> this new API can form a consumer-centric interface to support multiple 
> enforcement technologies, including, but not restricted to, Ranger. 
> feel free to assign to me/modify permissions ;-)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (RANGER-1487) generate rules from Governance definitions in Atlas

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones reassigned RANGER-1487:
---

Assignee: Nigel Jones

> generate rules from Governance definitions in Atlas
> ---
>
> Key: RANGER-1487
> URL: https://issues.apache.org/jira/browse/RANGER-1487
> Project: Ranger
>  Issue Type: New Feature
>  Components: Ranger
>    Reporter: Nigel Jones
>    Assignee: Nigel Jones
>  Labels: VirtualDataConnector
>
> As part of the Atlas virtual connector project we intend to build "rules" 
> that relate to classifications within Atlas itself. These are logical 
> definitions that will then be mapped to specific implementations such as 
> rules/policies in ranger (but could also be in other systems like Oracle, or 
> Apache Sentry -- different enforcement approaches for different data access 
> points).
> Opening this Jira to build a "sync/build/link" tool that will generate ranger 
> rules where possible.
> (Very rough description ... will update & clarify)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (RANGER-1563) Ranger build needs gcc... docs/validation

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones reassigned RANGER-1563:
---

Assignee: Nigel Jones

> Ranger build needs gcc... docs/validation
> -
>
> Key: RANGER-1563
> URL: https://issues.apache.org/jira/browse/RANGER-1563
> Project: Ranger
>  Issue Type: Bug
>  Components: documentation
>    Reporter: Nigel Jones
>    Assignee: Nigel Jones
>
> When building ranger on a new Linux system I had a build failure, caused by
> [INFO] Executed tasks
> [INFO]
> [INFO] --- native-maven-plugin:1.0-alpha-8:compile (default-compile) @ 
> credValidator ---
> [INFO] /bin/sh -c cd /home/cloudusr/src/ranger/unixauthnative && gcc 
> -I/home/cloudusr/src/ranger/unixauthnative/src/main/c -o 
> /home/cloudusr/src/ranger/unixauthnative/target/objs/credValidator.o -c 
> /home/cloudusr/src/ranger/unixauthnative/src/main/c/credValidator.c
> /bin/sh: 1: gcc: not found
> Clearly gcc is needed. Along with java, maven. Listing these prereqs would be 
> useful
> Suggest we update the docs at http://ranger.apache.org/quick_start_guide.html
> to make this prereq clear.
> Ideally update maven to do a check early on



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (RANGER-1565) Make SOLR setup config docs consistent with code

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones reassigned RANGER-1565:
---

Assignee: Nigel Jones

> Make SOLR setup config docs consistent with code
> 
>
> Key: RANGER-1565
> URL: https://issues.apache.org/jira/browse/RANGER-1565
> Project: Ranger
>  Issue Type: Bug
>  Components: audit
>    Reporter: Nigel Jones
>    Assignee: Nigel Jones
>
> In the Apache ranger setup docs we have a section on Solr configuration at
> https://cwiki.apache.org/confluence/display/RANGER/Install+and+Configure+Solr+for+Ranger+Audits+-+Apache+Ranger+0.5
> In this doc it refers to setting JAVA_HOME in the install.properties file 
> (the one in the contrib/ solr setup directory). However when I set this the 
> ./setup.sh still failed with:
> # ./setup.sh
> Error: JAVA_HOME environment property not defined, aborting installation.
> Looking at the code it appears it does NOT pull out JAVA_HOME from the 
> properties file at all, but relies on it being specified in the environment
> Ideally we would update the script to also source JAVA_HOME from 
> install.properties, in which case the doc can stay as is. If we don't do this 
> we need to update the docs accordingly. I would go with the properties file 
> overriding the local environment, if the variable is set there.
> My proposal is to change the script. Happy to work this if it sounds ok...



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (RANGER-1572) Provide simple example of running ranger with docker

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones reassigned RANGER-1572:
---

Assignee: Nigel Jones

> Provide simple example of running ranger with docker
> 
>
> Key: RANGER-1572
> URL: https://issues.apache.org/jira/browse/RANGER-1572
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>    Reporter: Nigel Jones
>    Assignee: Nigel Jones
>
> When experimenting with Ranger it's very useful to be able to quickly run 
> ranger admin/solr/mysql
> Since I needed this for my education/experimentation I thought it might be 
> useful to others. Work in progress and doesn't work yet but raising this JIRA 
> in case anyone else has done the same, or is interested ;-)
> For now the code (just a few scripts) is at 
> https://github.com/planetf1/ranger-docker whilst I try and get this working. 
> Perhaps this could be added as an example down the line, or even used within 
> the build to create an image -- though there's lots of variables in terms of 
> config.
> As above.. not working yet ;-) WIP!



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (RANGER-1572) Provide simple example of running ranger with docker

[Nigel Jones (JIRA)]

Nigel Jones created RANGER-1572:
---

 Summary: Provide simple example of running ranger with docker
 Key: RANGER-1572
 URL: https://issues.apache.org/jira/browse/RANGER-1572
 Project: Ranger
  Issue Type: Bug
  Components: Ranger
Reporter: Nigel Jones


When experimenting with Ranger it's very useful to be able to quickly run 
ranger admin/solr/mysql

Since I needed this for my education/experimentation I thought it might be 
useful to others. Work in progress and doesn't work yet but raising this JIRA 
in case anyone else has done the same, or is interested ;-)

For now the code (just a few scripts) is at 
https://github.com/planetf1/ranger-docker whilst I try and get this working. 
Perhaps this could be added as an example down the line, or even used within 
the build to create an image -- though there's lots of variables in terms of 
config.

As above.. not working yet ;-) WIP!



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (RANGER-1565) Make SOLR setup config docs consistent with code

[Nigel Jones (JIRA)]

 [ 
https://issues.apache.org/jira/browse/RANGER-1565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Jones updated RANGER-1565:

Request participants:   (was: )
 Description: 
In the Apache ranger setup docs we have a section on Solr configuration at
https://cwiki.apache.org/confluence/display/RANGER/Install+and+Configure+Solr+for+Ranger+Audits+-+Apache+Ranger+0.5

In this doc it refers to setting JAVA_HOME in the install.properties file (the 
one in the contrib/ solr setup directory). However when I set this the 
./setup.sh still failed with:

# ./setup.sh
Error: JAVA_HOME environment property not defined, aborting installation.

Looking at the code it appears it does NOT pull out JAVA_HOME from the 
properties file at all, but relies on it being specified in the environment

Ideally we would update the script to also source JAVA_HOME from 
install.properties, in which case the doc can stay as is. If we don't do this 
we need to update the docs accordingly. I would go with the properties file 
overriding the local environment, if the variable is set there.

My proposal is to change the script. Happy to work this if it sounds ok...



  was:
In the Apache ranger setup docs we have a section on Solr configuration at
https://cwiki.apache.org/confluence/display/RANGER/Install+and+Configure+Solr+for+Ranger+Audits+-+Apache+Ranger+0.5

In this doc it refers to setting JAVA_HOME in the install.properties file (the 
one in the contrib/ solr setup directory). However when I set this the 
./setup.sh still failed with:

# ./setup.sh
Error: JAVA_HOME environment property not defined, aborting installation.

Looking at the code it appears it does NOT pull out JAVA_HOME from the 
properties file at all, but relies on it being specified in the environment

Ideally we would update the script to also source JAVA_HOME from 
install.properties, in which case the doc can stay as is. If we don't do this 
we need to update the docs accordingly.

My proposal is to change the script. Happy to work this if it sounds ok...




> Make SOLR setup config docs consistent with code
> 
>
> Key: RANGER-1565
> URL: https://issues.apache.org/jira/browse/RANGER-1565
> Project: Ranger
>  Issue Type: Bug
>  Components: audit
>    Reporter: Nigel Jones
>
> In the Apache ranger setup docs we have a section on Solr configuration at
> https://cwiki.apache.org/confluence/display/RANGER/Install+and+Configure+Solr+for+Ranger+Audits+-+Apache+Ranger+0.5
> In this doc it refers to setting JAVA_HOME in the install.properties file 
> (the one in the contrib/ solr setup directory). However when I set this the 
> ./setup.sh still failed with:
> # ./setup.sh
> Error: JAVA_HOME environment property not defined, aborting installation.
> Looking at the code it appears it does NOT pull out JAVA_HOME from the 
> properties file at all, but relies on it being specified in the environment
> Ideally we would update the script to also source JAVA_HOME from 
> install.properties, in which case the doc can stay as is. If we don't do this 
> we need to update the docs accordingly. I would go with the properties file 
> overriding the local environment, if the variable is set there.
> My proposal is to change the script. Happy to work this if it sounds ok...



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (RANGER-1565) Make SOLR setup config docs consistent with code

[Nigel Jones (JIRA)]

Nigel Jones created RANGER-1565:
---

 Summary: Make SOLR setup config docs consistent with code
 Key: RANGER-1565
 URL: https://issues.apache.org/jira/browse/RANGER-1565
 Project: Ranger
  Issue Type: Bug
  Components: audit
Reporter: Nigel Jones


In the Apache ranger setup docs we have a section on Solr configuration at
https://cwiki.apache.org/confluence/display/RANGER/Install+and+Configure+Solr+for+Ranger+Audits+-+Apache+Ranger+0.5

In this doc it refers to setting JAVA_HOME in the install.properties file (the 
one in the contrib/ solr setup directory). However when I set this the 
./setup.sh still failed with:

# ./setup.sh
Error: JAVA_HOME environment property not defined, aborting installation.

Looking at the code it appears it does NOT pull out JAVA_HOME from the 
properties file at all, but relies on it being specified in the environment

Ideally we would update the script to also source JAVA_HOME from 
install.properties, in which case the doc can stay as is. If we don't do this 
we need to update the docs accordingly.

My proposal is to change the script. Happy to work this if it sounds ok...





--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (RANGER-1563) Ranger build needs gcc... docs/validation

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16000594#comment-16000594
 ] 

Nigel Jones commented on RANGER-1563:
-

Ok I'll ask again. But happy to propose this change, good practice for next 
time :-)

> Ranger build needs gcc... docs/validation
> -
>
> Key: RANGER-1563
> URL: https://issues.apache.org/jira/browse/RANGER-1563
> Project: Ranger
>  Issue Type: Bug
>  Components: documentation
>    Reporter: Nigel Jones
>
> When building ranger on a new Linux system I had a build failure, caused by
> [INFO] Executed tasks
> [INFO]
> [INFO] --- native-maven-plugin:1.0-alpha-8:compile (default-compile) @ 
> credValidator ---
> [INFO] /bin/sh -c cd /home/cloudusr/src/ranger/unixauthnative && gcc 
> -I/home/cloudusr/src/ranger/unixauthnative/src/main/c -o 
> /home/cloudusr/src/ranger/unixauthnative/target/objs/credValidator.o -c 
> /home/cloudusr/src/ranger/unixauthnative/src/main/c/credValidator.c
> /bin/sh: 1: gcc: not found
> Clearly gcc is needed. Along with java, maven. Listing these prereqs would be 
> useful
> Suggest we update the docs at http://ranger.apache.org/quick_start_guide.html
> to make this prereq clear.
> Ideally update maven to do a check early on



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (RANGER-1563) Ranger build needs gcc... docs/validation

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16000584#comment-16000584
 ] 

Nigel Jones edited comment on RANGER-1563 at 5/8/17 10:46 AM:
--

Of course, I'd be delighted to (though first time may take a while).
related -- I did post on the mailing list asking for JIRA Contributor, so that 
I can assign to myself - are you able to do this? If not I'll post again


was (Author: jonesn):
Of course, I'd be delighted too (though first time may take a while).
related -- I did post on the mailing list asking for JIRA Contributor, so that 
I can assign to myself - are you able to do this? If not I'll post again

> Ranger build needs gcc... docs/validation
> -
>
> Key: RANGER-1563
> URL: https://issues.apache.org/jira/browse/RANGER-1563
> Project: Ranger
>  Issue Type: Bug
>  Components: documentation
>    Reporter: Nigel Jones
>
> When building ranger on a new Linux system I had a build failure, caused by
> [INFO] Executed tasks
> [INFO]
> [INFO] --- native-maven-plugin:1.0-alpha-8:compile (default-compile) @ 
> credValidator ---
> [INFO] /bin/sh -c cd /home/cloudusr/src/ranger/unixauthnative && gcc 
> -I/home/cloudusr/src/ranger/unixauthnative/src/main/c -o 
> /home/cloudusr/src/ranger/unixauthnative/target/objs/credValidator.o -c 
> /home/cloudusr/src/ranger/unixauthnative/src/main/c/credValidator.c
> /bin/sh: 1: gcc: not found
> Clearly gcc is needed. Along with java, maven. Listing these prereqs would be 
> useful
> Suggest we update the docs at http://ranger.apache.org/quick_start_guide.html
> to make this prereq clear.
> Ideally update maven to do a check early on



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (RANGER-1563) Ranger build needs gcc... docs/validation

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16000584#comment-16000584
 ] 

Nigel Jones commented on RANGER-1563:
-

Of course, I'd be delighted too (though first time may take a while).
related -- I did post on the mailing list asking for JIRA Contributor, so that 
I can assign to myself - are you able to do this? If not I'll post again

> Ranger build needs gcc... docs/validation
> -
>
> Key: RANGER-1563
> URL: https://issues.apache.org/jira/browse/RANGER-1563
> Project: Ranger
>  Issue Type: Bug
>  Components: documentation
>    Reporter: Nigel Jones
>
> When building ranger on a new Linux system I had a build failure, caused by
> [INFO] Executed tasks
> [INFO]
> [INFO] --- native-maven-plugin:1.0-alpha-8:compile (default-compile) @ 
> credValidator ---
> [INFO] /bin/sh -c cd /home/cloudusr/src/ranger/unixauthnative && gcc 
> -I/home/cloudusr/src/ranger/unixauthnative/src/main/c -o 
> /home/cloudusr/src/ranger/unixauthnative/target/objs/credValidator.o -c 
> /home/cloudusr/src/ranger/unixauthnative/src/main/c/credValidator.c
> /bin/sh: 1: gcc: not found
> Clearly gcc is needed. Along with java, maven. Listing these prereqs would be 
> useful
> Suggest we update the docs at http://ranger.apache.org/quick_start_guide.html
> to make this prereq clear.
> Ideally update maven to do a check early on



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (RANGER-1563) Ranger build needs gcc... docs/validation

[Nigel Jones (JIRA)]

Nigel Jones created RANGER-1563:
---

 Summary: Ranger build needs gcc... docs/validation
 Key: RANGER-1563
 URL: https://issues.apache.org/jira/browse/RANGER-1563
 Project: Ranger
  Issue Type: Bug
  Components: documentation
Reporter: Nigel Jones


When building ranger on a new Linux system I had a build failure, caused by


[INFO] Executed tasks
[INFO]
[INFO] --- native-maven-plugin:1.0-alpha-8:compile (default-compile) @ 
credValidator ---
[INFO] /bin/sh -c cd /home/cloudusr/src/ranger/unixauthnative && gcc 
-I/home/cloudusr/src/ranger/unixauthnative/src/main/c -o 
/home/cloudusr/src/ranger/unixauthnative/target/objs/credValidator.o -c 
/home/cloudusr/src/ranger/unixauthnative/src/main/c/credValidator.c
/bin/sh: 1: gcc: not found


Clearly gcc is needed. Along with java, maven. Listing these prereqs would be 
useful

Suggest we update the docs at http://ranger.apache.org/quick_start_guide.html
to make this prereq clear.

Ideally update maven to do a check early on




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (RANGER-1488) Create Ranger plugin for gaiandb

[Nigel Jones (JIRA)]

[ 
https://issues.apache.org/jira/browse/RANGER-1488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16000517#comment-16000517
 ] 

Nigel Jones commented on RANGER-1488:
-

Initial code being put together at 
https://github.com/planetf1/gaiandb-policy-ranger
... it will be a few weeks before this is in a functional state, but I wanted 
to give a heads up about the activity

> Create Ranger plugin for gaiandb
> 
>
> Key: RANGER-1488
> URL: https://issues.apache.org/jira/browse/RANGER-1488
> Project: Ranger
>  Issue Type: New Feature
>  Components: plugins
>    Reporter: Nigel Jones
>  Labels: VirtualDataConnector
>
> GaianDB is a distributed, federated database built on Apache Derby.
> The documentation is at 
> https://github.com/gaiandb/gaiandb/blob/master/README.md - navigate up for 
> the source.
> As part of a Virtual Connector Project based on Atlas we are using gaianDB to 
> provide a virtualization layer. We need to control access to underlying 
> resources and will be building a Ranger plugin for gaiandb to support this. 
> GaianDB already has support for a form of policy plugin which allows the SQL 
> to be intercepted.
> It is unclear if this code will sit
>  - external to ranger/atlas ie in the gaianDB sources
>  - as a sample in the Atlas project (so that we collate what is needed for 
> the virtual connector project)
>  - as a sample, or component in the Ranger project
> However it seems helpful to describe the proposal here and get 
> feedback/dialogue in the ranger community :-)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)