[jira] [Commented] (RANGER-3410) Fix setup for non-ssl database connection
[ https://issues.apache.org/jira/browse/RANGER-3410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414578#comment-17414578 ] Andrew Charneski commented on RANGER-3410: -- https://reviews.apache.org/r/73583/ > Fix setup for non-ssl database connection > - > > Key: RANGER-3410 > URL: https://issues.apache.org/jira/browse/RANGER-3410 > Project: Ranger > Issue Type: Improvement > Components: admin >Reporter: Andrew Charneski >Priority: Major > > When running the security-admin setup script using a non-secure mysql > endpoint, a failure occurs because new versions of the mysql client driver > require useSSL=false to enable non-secure connections. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-3409) Update Jackson and remove Codehaus version
[ https://issues.apache.org/jira/browse/RANGER-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414576#comment-17414576 ] Andrew Charneski commented on RANGER-3409: -- https://reviews.apache.org/r/73582/ > Update Jackson and remove Codehaus version > -- > > Key: RANGER-3409 > URL: https://issues.apache.org/jira/browse/RANGER-3409 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Reporter: Andrew Charneski >Priority: Major > > An old version of Jackson (Codehaus Jackson 1.9.13) is still being used. > Jackson has since moved namespaces with a reorganized library structure. > Update all references to the older version to use the newer version (which is > currently used in some modules). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-3411) Improvements for Elasticsearch Audit Log Provider
[ https://issues.apache.org/jira/browse/RANGER-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17414574#comment-17414574 ] Andrew Charneski commented on RANGER-3411: -- [https://reviews.apache.org/r/73581/|https://reviews.apache.org/r/73581/] > Improvements for Elasticsearch Audit Log Provider > - > > Key: RANGER-3411 > URL: https://issues.apache.org/jira/browse/RANGER-3411 > Project: Ranger > Issue Type: Improvement > Components: audit >Reporter: Andrew Charneski >Priority: Major > Attachments: 0001-RANGER-3411-Elasticsearch-Fixes.patch > > > Several improvements are needed to make the Elasticsearch audit log provider > viable: > # Configuration and setup scripts need to support settings for the port, > scheme, and index > # Audit log search logic can remove the "fetch" call after the > "searchResources" call to Elasticsearch > # xasecure.audit.elasticsearch.is.enabled should be supported to enable this > provider > # AuditProviderFactory.java needs logic changes to allow enabling the provider -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (RANGER-3411) Improvements for Elasticsearch Audit Log Provider
[ https://issues.apache.org/jira/browse/RANGER-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Charneski updated RANGER-3411: - Attachment: 0001-RANGER-3411-Elasticsearch-Fixes.patch > Improvements for Elasticsearch Audit Log Provider > - > > Key: RANGER-3411 > URL: https://issues.apache.org/jira/browse/RANGER-3411 > Project: Ranger > Issue Type: Improvement > Components: audit >Reporter: Andrew Charneski >Priority: Major > Attachments: 0001-RANGER-3411-Elasticsearch-Fixes.patch > > > Several improvements are needed to make the Elasticsearch audit log provider > viable: > # Configuration and setup scripts need to support settings for the port, > scheme, and index > # Audit log search logic can remove the "fetch" call after the > "searchResources" call to Elasticsearch > # xasecure.audit.elasticsearch.is.enabled should be supported to enable this > provider > # AuditProviderFactory.java needs logic changes to allow enabling the provider -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (RANGER-3411) Improvements for Elasticsearch Audit Log Provider
Andrew Charneski created RANGER-3411: Summary: Improvements for Elasticsearch Audit Log Provider Key: RANGER-3411 URL: https://issues.apache.org/jira/browse/RANGER-3411 Project: Ranger Issue Type: Improvement Components: audit Reporter: Andrew Charneski Several improvements are needed to make the Elasticsearch audit log provider viable: # Configuration and setup scripts need to support settings for the port, scheme, and index # Audit log search logic can remove the "fetch" call after the "searchResources" call to Elasticsearch # xasecure.audit.elasticsearch.is.enabled should be supported to enable this provider # AuditProviderFactory.java needs logic changes to allow enabling the provider -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (RANGER-3410) Fix setup for non-ssl database connection
Andrew Charneski created RANGER-3410: Summary: Fix setup for non-ssl database connection Key: RANGER-3410 URL: https://issues.apache.org/jira/browse/RANGER-3410 Project: Ranger Issue Type: Improvement Components: admin Reporter: Andrew Charneski When running the security-admin setup script using a non-secure mysql endpoint, a failure occurs because new versions of the mysql client driver require useSSL=false to enable non-secure connections. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (RANGER-3409) Update Jackson and remove Codehaus version
Andrew Charneski created RANGER-3409: Summary: Update Jackson and remove Codehaus version Key: RANGER-3409 URL: https://issues.apache.org/jira/browse/RANGER-3409 Project: Ranger Issue Type: Improvement Components: Ranger Reporter: Andrew Charneski An old version of Jackson (Codehaus Jackson 1.9.13) is still being used. Jackson has since moved namespaces with a reorganized library structure. Update all references to the older version to use the newer version (which is currently used in some modules). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-2634) Add ElasticSearch audit support
[ https://issues.apache.org/jira/browse/RANGER-2634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117190#comment-17117190 ] Andrew Charneski commented on RANGER-2634: -- [~bpatel] I've added some setup scripts in the location you specified. They do not specify the types for the schema - Implicily defined schema works fine in elasticsearch afaik and it's easier to maintain. I have only implemented basic user/password authentication. Kerberos is not-yet-supported. > Add ElasticSearch audit support > --- > > Key: RANGER-2634 > URL: https://issues.apache.org/jira/browse/RANGER-2634 > Project: Ranger > Issue Type: Wish > Components: audit >Reporter: Andrew Charneski >Priority: Major > > Hello, > Our organization is working on integrating and rolling out Apache Ranger for > our internal toolset, and firstly, we'd like to thank you all for your great > work! > Although we have decided to adopt Ranger, for a variety of reasons we would > very much like to use an alternate indexing service (Elasticsearch) in > preference to the currently-supported Solr. > We are happy to develop this extension on our own, and the initial efforts > are underway at [https://github.com/acharneski/ranger/pull/1] however we > would like to engage the community for guidance and suggestions with the aim > of getting this change merged into the main codebase when it is ready. > Our initial approach is in three main phases: > # Isolate all usages of Solr packages/classes to pass through an API > wrapper, delegating to the original Solr code with minimal changes. (Dev > complete) > # Refactor api package as a pluggable interface with greater simplicity and > implementation agnosticism. > # Provide alternate implementation for the new interface for Elasticsearch. > Any guidance and feedback is appreciated. Thank you for reading! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-2634) Add ElasticSearch support
[
https://issues.apache.org/jira/browse/RANGER-2634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094868#comment-17094868
]
Andrew Charneski commented on RANGER-2634:
--
[~pradeep] I rebased last time I updated the commit.
[~bpatel] I have tested it using Elasticsearch secured using a username and
password.
[~bosco] I've verified this can be pulled and merged into [current
master|https://github.com/apache/ranger/commit/2b5fc4796f6ab1dafaf682a8fb6f647812261d5a]:
{code:sh}
git checkout 2b5fc4796f6ab1dafaf682a8fb6f647812261d5a
rbt patch --server https://reviews.apache.org/ 71921
{code}
> Add ElasticSearch support
> -
>
> Key: RANGER-2634
> URL: https://issues.apache.org/jira/browse/RANGER-2634
> Project: Ranger
> Issue Type: Wish
> Components: audit
>Reporter: Andrew Charneski
>Priority: Major
>
> Hello,
> Our organization is working on integrating and rolling out Apache Ranger for
> our internal toolset, and firstly, we'd like to thank you all for your great
> work!
> Although we have decided to adopt Ranger, for a variety of reasons we would
> very much like to use an alternate indexing service (Elasticsearch) in
> preference to the currently-supported Solr.
> We are happy to develop this extension on our own, and the initial efforts
> are underway at [https://github.com/acharneski/ranger/pull/1] however we
> would like to engage the community for guidance and suggestions with the aim
> of getting this change merged into the main codebase when it is ready.
> Our initial approach is in three main phases:
> # Isolate all usages of Solr packages/classes to pass through an API
> wrapper, delegating to the original Solr code with minimal changes. (Dev
> complete)
> # Refactor api package as a pluggable interface with greater simplicity and
> implementation agnosticism.
> # Provide alternate implementation for the new interface for Elasticsearch.
> Any guidance and feedback is appreciated. Thank you for reading!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-2634) Add ElasticSearch support
[
https://issues.apache.org/jira/browse/RANGER-2634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17080015#comment-17080015
]
Andrew Charneski commented on RANGER-2634:
--
I have coded it against and tested against the latest version, which is/was
7.5.1. This is the setup script I am using:
{code:bash}
wget
https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.5.1-x86_64.rpm
sudo rpm --install elasticsearch-7.5.1-x86_64.rpm
echo '
88a89
> xpack.security.enabled: true
' > ~/elasticsearch.yml.diff
sudo patch /etc/elasticsearch/elasticsearch.yml ~/elasticsearch.yml.diff
sudo -i service elasticsearch start
sleep 10
curl -X PUT "localhost:9200/audit?pretty"
sudo /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive
{code}
[~bosco] I have updated this patch to work off current master.
I have responded to or taken action on all comments. Please re-review.
> Add ElasticSearch support
> -
>
> Key: RANGER-2634
> URL: https://issues.apache.org/jira/browse/RANGER-2634
> Project: Ranger
> Issue Type: Wish
> Components: audit
>Reporter: Andrew Charneski
>Priority: Major
>
> Hello,
> Our organization is working on integrating and rolling out Apache Ranger for
> our internal toolset, and firstly, we'd like to thank you all for your great
> work!
> Although we have decided to adopt Ranger, for a variety of reasons we would
> very much like to use an alternate indexing service (Elasticsearch) in
> preference to the currently-supported Solr.
> We are happy to develop this extension on our own, and the initial efforts
> are underway at [https://github.com/acharneski/ranger/pull/1] however we
> would like to engage the community for guidance and suggestions with the aim
> of getting this change merged into the main codebase when it is ready.
> Our initial approach is in three main phases:
> # Isolate all usages of Solr packages/classes to pass through an API
> wrapper, delegating to the original Solr code with minimal changes. (Dev
> complete)
> # Refactor api package as a pluggable interface with greater simplicity and
> implementation agnosticism.
> # Provide alternate implementation for the new interface for Elasticsearch.
> Any guidance and feedback is appreciated. Thank you for reading!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-2634) Add ElasticSearch support
[ https://issues.apache.org/jira/browse/RANGER-2634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17040194#comment-17040194 ] Andrew Charneski commented on RANGER-2634: -- Just another ping to request a review - I'd like to get this merged eventually :-) > Add ElasticSearch support > - > > Key: RANGER-2634 > URL: https://issues.apache.org/jira/browse/RANGER-2634 > Project: Ranger > Issue Type: Wish > Components: audit >Reporter: Andrew Charneski >Priority: Major > > Hello, > Our organization is working on integrating and rolling out Apache Ranger for > our internal toolset, and firstly, we'd like to thank you all for your great > work! > Although we have decided to adopt Ranger, for a variety of reasons we would > very much like to use an alternate indexing service (Elasticsearch) in > preference to the currently-supported Solr. > We are happy to develop this extension on our own, and the initial efforts > are underway at [https://github.com/acharneski/ranger/pull/1] however we > would like to engage the community for guidance and suggestions with the aim > of getting this change merged into the main codebase when it is ready. > Our initial approach is in three main phases: > # Isolate all usages of Solr packages/classes to pass through an API > wrapper, delegating to the original Solr code with minimal changes. (Dev > complete) > # Refactor api package as a pluggable interface with greater simplicity and > implementation agnosticism. > # Provide alternate implementation for the new interface for Elasticsearch. > Any guidance and feedback is appreciated. Thank you for reading! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (RANGER-2634) Add ElasticSearch support
[ https://issues.apache.org/jira/browse/RANGER-2634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029983#comment-17029983 ] Andrew Charneski edited comment on RANGER-2634 at 2/4/20 4:46 PM: -- [~bosco] Sorry just saw this comment. I've made the patch apply to the tip of master. Also, updated the change to correct the comments you left. Please look again, thank you! was (Author: acharneski): Sorry just saw this comment. I've made the patch apply to the tip of master. Also, updated the change to correct the comments you left. Please look again, thank you! > Add ElasticSearch support > - > > Key: RANGER-2634 > URL: https://issues.apache.org/jira/browse/RANGER-2634 > Project: Ranger > Issue Type: Wish > Components: audit >Reporter: Andrew Charneski >Priority: Major > > Hello, > Our organization is working on integrating and rolling out Apache Ranger for > our internal toolset, and firstly, we'd like to thank you all for your great > work! > Although we have decided to adopt Ranger, for a variety of reasons we would > very much like to use an alternate indexing service (Elasticsearch) in > preference to the currently-supported Solr. > We are happy to develop this extension on our own, and the initial efforts > are underway at [https://github.com/acharneski/ranger/pull/1] however we > would like to engage the community for guidance and suggestions with the aim > of getting this change merged into the main codebase when it is ready. > Our initial approach is in three main phases: > # Isolate all usages of Solr packages/classes to pass through an API > wrapper, delegating to the original Solr code with minimal changes. (Dev > complete) > # Refactor api package as a pluggable interface with greater simplicity and > implementation agnosticism. > # Provide alternate implementation for the new interface for Elasticsearch. > Any guidance and feedback is appreciated. Thank you for reading! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (RANGER-2634) Add ElasticSearch support
[ https://issues.apache.org/jira/browse/RANGER-2634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029983#comment-17029983 ] Andrew Charneski edited comment on RANGER-2634 at 2/4/20 4:46 PM: -- [~bosco] Sorry just saw this comment. I've made the patch apply to the tip of master. Also, updated the change to correct the comments you all left. Please look again, thank you! was (Author: acharneski): [~bosco] Sorry just saw this comment. I've made the patch apply to the tip of master. Also, updated the change to correct the comments you left. Please look again, thank you! > Add ElasticSearch support > - > > Key: RANGER-2634 > URL: https://issues.apache.org/jira/browse/RANGER-2634 > Project: Ranger > Issue Type: Wish > Components: audit >Reporter: Andrew Charneski >Priority: Major > > Hello, > Our organization is working on integrating and rolling out Apache Ranger for > our internal toolset, and firstly, we'd like to thank you all for your great > work! > Although we have decided to adopt Ranger, for a variety of reasons we would > very much like to use an alternate indexing service (Elasticsearch) in > preference to the currently-supported Solr. > We are happy to develop this extension on our own, and the initial efforts > are underway at [https://github.com/acharneski/ranger/pull/1] however we > would like to engage the community for guidance and suggestions with the aim > of getting this change merged into the main codebase when it is ready. > Our initial approach is in three main phases: > # Isolate all usages of Solr packages/classes to pass through an API > wrapper, delegating to the original Solr code with minimal changes. (Dev > complete) > # Refactor api package as a pluggable interface with greater simplicity and > implementation agnosticism. > # Provide alternate implementation for the new interface for Elasticsearch. > Any guidance and feedback is appreciated. Thank you for reading! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-2634) Add ElasticSearch support
[ https://issues.apache.org/jira/browse/RANGER-2634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029983#comment-17029983 ] Andrew Charneski commented on RANGER-2634: -- Sorry just saw this comment. I've made the patch apply to the tip of master. Also, updated the change to correct the comments you left. Please look again, thank you! > Add ElasticSearch support > - > > Key: RANGER-2634 > URL: https://issues.apache.org/jira/browse/RANGER-2634 > Project: Ranger > Issue Type: Wish > Components: audit >Reporter: Andrew Charneski >Priority: Major > > Hello, > Our organization is working on integrating and rolling out Apache Ranger for > our internal toolset, and firstly, we'd like to thank you all for your great > work! > Although we have decided to adopt Ranger, for a variety of reasons we would > very much like to use an alternate indexing service (Elasticsearch) in > preference to the currently-supported Solr. > We are happy to develop this extension on our own, and the initial efforts > are underway at [https://github.com/acharneski/ranger/pull/1] however we > would like to engage the community for guidance and suggestions with the aim > of getting this change merged into the main codebase when it is ready. > Our initial approach is in three main phases: > # Isolate all usages of Solr packages/classes to pass through an API > wrapper, delegating to the original Solr code with minimal changes. (Dev > complete) > # Refactor api package as a pluggable interface with greater simplicity and > implementation agnosticism. > # Provide alternate implementation for the new interface for Elasticsearch. > Any guidance and feedback is appreciated. Thank you for reading! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-2634) Add ElasticSearch support
[ https://issues.apache.org/jira/browse/RANGER-2634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022241#comment-17022241 ] Andrew Charneski commented on RANGER-2634: -- Can I get someone to review this? Thanks! > Add ElasticSearch support > - > > Key: RANGER-2634 > URL: https://issues.apache.org/jira/browse/RANGER-2634 > Project: Ranger > Issue Type: Wish > Components: audit >Reporter: Andrew Charneski >Priority: Major > > Hello, > Our organization is working on integrating and rolling out Apache Ranger for > our internal toolset, and firstly, we'd like to thank you all for your great > work! > Although we have decided to adopt Ranger, for a variety of reasons we would > very much like to use an alternate indexing service (Elasticsearch) in > preference to the currently-supported Solr. > We are happy to develop this extension on our own, and the initial efforts > are underway at [https://github.com/acharneski/ranger/pull/1] however we > would like to engage the community for guidance and suggestions with the aim > of getting this change merged into the main codebase when it is ready. > Our initial approach is in three main phases: > # Isolate all usages of Solr packages/classes to pass through an API > wrapper, delegating to the original Solr code with minimal changes. (Dev > complete) > # Refactor api package as a pluggable interface with greater simplicity and > implementation agnosticism. > # Provide alternate implementation for the new interface for Elasticsearch. > Any guidance and feedback is appreciated. Thank you for reading! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-2689) Support multiple versions of Hive
[ https://issues.apache.org/jira/browse/RANGER-2689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020400#comment-17020400 ] Andrew Charneski commented on RANGER-2689: -- My largest concern with that approach is that it will end up forking a significant amount of code, while the modifications for backwards-compatibility are fairly isolated as demonstrated in the PRs linked in the description. This will produce an ongoing technical burden when making changes to the Hive plugin(s). > Support multiple versions of Hive > - > > Key: RANGER-2689 > URL: https://issues.apache.org/jira/browse/RANGER-2689 > Project: Ranger > Issue Type: Improvement > Components: plugins >Reporter: Andrew Charneski >Priority: Major > > Currently Ranger supports the latest version of Hive, 3.1.2. Unfortunately, > there are large segments of the big data community that still rely on older > versions of Hive. Two major examples: > # Spark SQL uses a forked version of Hive 1.2.1 > (https://spark.apache.org/docs/latest/sql-migration-guide-hive-compatibility.html) > # EMR provides Hive only up to 2.3.5 > (https://docs.aws.amazon.com/emr/latest/ReleaseGuide/Hive-release-history.html) > In order to support these internally, my organization has prepared two > modifications of Ranger to link against these versions. These are illustrated > in the PRs https://github.com/acharneski/ranger/pull/4 and > https://github.com/acharneski/ranger/pull/5 > We would like to eliminate the need for entirely separate builds of Ranger to > support this, and integrate these variants into the main Ranger codebase. We > are willing to do the bulk of the implementation but would first like to > discuss the architecture of this change so as to build it in a way the Ranger > committers would be amenable to adopting. > My initial thought is to split the `hive-agent` module into something like > `hive-agent-base`, `hive-agent-1`, `hive-agent-2`, and `hive-agent-3`. This > would allow us to explicitly link to each major version of Hive while > minimizing the duplication of code. Thoughts? > Thank you! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (RANGER-2689) Support multiple versions of Hive
[ https://issues.apache.org/jira/browse/RANGER-2689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Charneski updated RANGER-2689: - Description: Currently Ranger supports the latest version of Hive, 3.1.2. Unfortunately, there are large segments of the big data community that still rely on older versions of Hive. Two major examples: # Spark SQL uses a forked version of Hive 1.2.1 (https://spark.apache.org/docs/latest/sql-migration-guide-hive-compatibility.html) # EMR provides Hive only up to 2.3.5 (https://docs.aws.amazon.com/emr/latest/ReleaseGuide/Hive-release-history.html) In order to support these internally, my organization has prepared two modifications of Ranger to link against these versions. These are illustrated in the PRs https://github.com/acharneski/ranger/pull/4 and https://github.com/acharneski/ranger/pull/5 We would like to eliminate the need for entirely separate builds of Ranger to support this, and integrate these variants into the main Ranger codebase. We are willing to do the bulk of the implementation but would first like to discuss the architecture of this change so as to build it in a way the Ranger committers would be amenable to adopting. My initial thought is to split the `hive-agent` module into something like `hive-agent-base`, `hive-agent-1`, `hive-agent-2`, and `hive-agent-3`. This would allow us to explicitly link to each major version of Hive while minimizing the duplication of code. Thoughts? Thank you! was: Currently Ranger supports the latest version of Hive, 3.1.2. Unfortunately, there are large segments of the big data community that relies on older versions of Hive. Two major examples: # Spark SQL uses a forked version of Hive 1.2.1 (https://spark.apache.org/docs/latest/sql-migration-guide-hive-compatibility.html) # EMR provides Hive only up to 2.3.5 (https://docs.aws.amazon.com/emr/latest/ReleaseGuide/Hive-release-history.html) In order to support these internally, my organization has prepared two modifications of Ranger to link against these versions. These are illustrated in the PRs https://github.com/acharneski/ranger/pull/4 and https://github.com/acharneski/ranger/pull/5 We would like to eliminate the need for entirely separate builds of Ranger to support this, and integrate these variants into the main Ranger codebase. We are willing to do the bulk of the implementation but would first like to discuss the architecture of this change so as to build it in a way the Ranger committers would be amenable to adopting. My initial thought is to split the `hive-agent` module into something like `hive-agent-base`, `hive-agent-1`, `hive-agent-2`, and `hive-agent-3`. This would allow us to explicitly link to each major version of Hive while minimizing the duplication of code. Thoughts? Thank you! > Support multiple versions of Hive > - > > Key: RANGER-2689 > URL: https://issues.apache.org/jira/browse/RANGER-2689 > Project: Ranger > Issue Type: Improvement > Components: plugins >Reporter: Andrew Charneski >Priority: Major > > Currently Ranger supports the latest version of Hive, 3.1.2. Unfortunately, > there are large segments of the big data community that still rely on older > versions of Hive. Two major examples: > # Spark SQL uses a forked version of Hive 1.2.1 > (https://spark.apache.org/docs/latest/sql-migration-guide-hive-compatibility.html) > # EMR provides Hive only up to 2.3.5 > (https://docs.aws.amazon.com/emr/latest/ReleaseGuide/Hive-release-history.html) > In order to support these internally, my organization has prepared two > modifications of Ranger to link against these versions. These are illustrated > in the PRs https://github.com/acharneski/ranger/pull/4 and > https://github.com/acharneski/ranger/pull/5 > We would like to eliminate the need for entirely separate builds of Ranger to > support this, and integrate these variants into the main Ranger codebase. We > are willing to do the bulk of the implementation but would first like to > discuss the architecture of this change so as to build it in a way the Ranger > committers would be amenable to adopting. > My initial thought is to split the `hive-agent` module into something like > `hive-agent-base`, `hive-agent-1`, `hive-agent-2`, and `hive-agent-3`. This > would allow us to explicitly link to each major version of Hive while > minimizing the duplication of code. Thoughts? > Thank you! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (RANGER-2689) Support multiple versions of Hive
[ https://issues.apache.org/jira/browse/RANGER-2689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Charneski updated RANGER-2689: - Description: Currently Ranger supports the latest version of Hive, 3.1.2. Unfortunately, there are large segments of the big data community that relies on older versions of Hive. Two major examples: # Spark SQL uses a forked version of Hive 1.2.1 (https://spark.apache.org/docs/latest/sql-migration-guide-hive-compatibility.html) # EMR provides Hive only up to 2.3.5 (https://docs.aws.amazon.com/emr/latest/ReleaseGuide/Hive-release-history.html) In order to support these internally, my organization has prepared two modifications of Ranger to link against these versions. These are illustrated in the PRs https://github.com/acharneski/ranger/pull/4 and https://github.com/acharneski/ranger/pull/5 We would like to eliminate the need for entirely separate builds of Ranger to support this, and integrate these variants into the main Ranger codebase. We are willing to do the bulk of the implementation but would first like to discuss the architecture of this change so as to build it in a way the Ranger committers would be amenable to adopting. My initial thought is to split the `hive-agent` module into something like `hive-agent-base`, `hive-agent-1`, `hive-agent-2`, and `hive-agent-3`. This would allow us to explicitly link to each major version of Hive while minimizing the duplication of code. Thoughts? Thank you! was: Currently Ranger supports the latest version of Hive, 3.1.2. Unfortunately, there are large segments of the big data community that relies on older versions of Hive. Two major examples: # Spark SQL uses a forked version of Hive 1.2.1 (https://spark.apache.org/docs/latest/sql-migration-guide-hive-compatibility.html) # EMR provides Hive only up to 2.3.5 (https://docs.aws.amazon.com/emr/latest/ReleaseGuide/Hive-release-history.html) In order to support these internally, my organization has prepared two modifications of Ranger to link against these versions. These are illustrated in the PRs https://github.com/acharneski/ranger/pull/4 and https://github.com/apache/ranger/pull/51 We would like to eliminate the need for entirely separate builds of Ranger to support this, and integrate these variants into the main Ranger codebase. We are willing to do the bulk of the implementation but would first like to discuss the architecture of this change so as to build it in a way the Ranger committers would be amenable to adopting. My initial thought is to split the `hive-agent` module into something like `hive-agent-base`, `hive-agent-1`, `hive-agent-2`, and `hive-agent-3`. This would allow us to explicitly link to each major version of Hive while minimizing the duplication of code. Thoughts? Thank you! > Support multiple versions of Hive > - > > Key: RANGER-2689 > URL: https://issues.apache.org/jira/browse/RANGER-2689 > Project: Ranger > Issue Type: Improvement > Components: plugins >Reporter: Andrew Charneski >Priority: Major > > Currently Ranger supports the latest version of Hive, 3.1.2. Unfortunately, > there are large segments of the big data community that relies on older > versions of Hive. Two major examples: > # Spark SQL uses a forked version of Hive 1.2.1 > (https://spark.apache.org/docs/latest/sql-migration-guide-hive-compatibility.html) > # EMR provides Hive only up to 2.3.5 > (https://docs.aws.amazon.com/emr/latest/ReleaseGuide/Hive-release-history.html) > In order to support these internally, my organization has prepared two > modifications of Ranger to link against these versions. These are illustrated > in the PRs https://github.com/acharneski/ranger/pull/4 and > https://github.com/acharneski/ranger/pull/5 > We would like to eliminate the need for entirely separate builds of Ranger to > support this, and integrate these variants into the main Ranger codebase. We > are willing to do the bulk of the implementation but would first like to > discuss the architecture of this change so as to build it in a way the Ranger > committers would be amenable to adopting. > My initial thought is to split the `hive-agent` module into something like > `hive-agent-base`, `hive-agent-1`, `hive-agent-2`, and `hive-agent-3`. This > would allow us to explicitly link to each major version of Hive while > minimizing the duplication of code. Thoughts? > Thank you! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (RANGER-2689) Support multiple versions of Hive
Andrew Charneski created RANGER-2689: Summary: Support multiple versions of Hive Key: RANGER-2689 URL: https://issues.apache.org/jira/browse/RANGER-2689 Project: Ranger Issue Type: Improvement Components: plugins Reporter: Andrew Charneski Currently Ranger supports the latest version of Hive, 3.1.2. Unfortunately, there are large segments of the big data community that relies on older versions of Hive. Two major examples: # Spark SQL uses a forked version of Hive 1.2.1 (https://spark.apache.org/docs/latest/sql-migration-guide-hive-compatibility.html) # EMR provides Hive only up to 2.3.5 (https://docs.aws.amazon.com/emr/latest/ReleaseGuide/Hive-release-history.html) In order to support these internally, my organization has prepared two modifications of Ranger to link against these versions. These are illustrated in the PRs https://github.com/acharneski/ranger/pull/4 and https://github.com/apache/ranger/pull/51 We would like to eliminate the need for entirely separate builds of Ranger to support this, and integrate these variants into the main Ranger codebase. We are willing to do the bulk of the implementation but would first like to discuss the architecture of this change so as to build it in a way the Ranger committers would be amenable to adopting. My initial thought is to split the `hive-agent` module into something like `hive-agent-base`, `hive-agent-1`, `hive-agent-2`, and `hive-agent-3`. This would allow us to explicitly link to each major version of Hive while minimizing the duplication of code. Thoughts? Thank you! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-2634) Add ElasticSearch support
[ https://issues.apache.org/jira/browse/RANGER-2634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998614#comment-16998614 ] Andrew Charneski commented on RANGER-2634: -- [~bosco] updated the PR with fully tested functionality, and posted to https://reviews.apache.org/r/71921/diff/1/?page=1 > Add ElasticSearch support > - > > Key: RANGER-2634 > URL: https://issues.apache.org/jira/browse/RANGER-2634 > Project: Ranger > Issue Type: Wish > Components: audit >Reporter: Andrew Charneski >Priority: Major > > Hello, > Our organization is working on integrating and rolling out Apache Ranger for > our internal toolset, and firstly, we'd like to thank you all for your great > work! > Although we have decided to adopt Ranger, for a variety of reasons we would > very much like to use an alternate indexing service (Elasticsearch) in > preference to the currently-supported Solr. > We are happy to develop this extension on our own, and the initial efforts > are underway at [https://github.com/acharneski/ranger/pull/1] however we > would like to engage the community for guidance and suggestions with the aim > of getting this change merged into the main codebase when it is ready. > Our initial approach is in three main phases: > # Isolate all usages of Solr packages/classes to pass through an API > wrapper, delegating to the original Solr code with minimal changes. (Dev > complete) > # Refactor api package as a pluggable interface with greater simplicity and > implementation agnosticism. > # Provide alternate implementation for the new interface for Elasticsearch. > Any guidance and feedback is appreciated. Thank you for reading! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (RANGER-2634) Add ElasticSearch support
[ https://issues.apache.org/jira/browse/RANGER-2634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16979343#comment-16979343 ] Andrew Charneski edited comment on RANGER-2634 at 11/25/19 10:55 PM: - Hi [~bosco] - This PR (at [https://github.com/apache/ranger/pull/42|https://github.com/apache/ranger/pull/42]) is ready for review and feedback. Some of the items are merely to get the build to work in my environments. I'm happy to create the review on Reviewboard if that's appropriate at this point, but had one major question first I'm hoping you can provide guidance on. My main concern is that I have been unable to test this functionality in an integrated environment - Currently our organization is only using the Hive plugin and my analysis of the code suggests that only "grant" and "revoke" permission operations (in RangerBasePlugin) make use of RangerAccessResultProcessor, which seems to end up in the audit log. Our test environment does show such events, but I am fairly certain this does not actually use the Audit DB (with the new Elasticsearch support). I have prepared some test code that uses a local Elasticsearch to read and write events (ElasticSearchAccessAuditsServiceTest) but have been unable to test actual audit events in our integration test environment. Do you have any suggestions on how to properly test this? Thank you very much for your guidance and feedback! was (Author: acharneski): Hi [~bosco] - This PR (at [https://github.com/apache/ranger/pull/42|https://github.com/apache/ranger/pull/42)]) is ready for review and feedback. Some of the items are merely to get the build to work in my environments. I'm happy to create the review on Reviewboard if that's appropriate at this point, but had one major question first I'm hoping you can provide guidance on. My main concern is that I have been unable to test this functionality in an integrated environment - Currently our organization is only using the Hive plugin and my analysis of the code suggests that only "grant" and "revoke" permission operations (in RangerBasePlugin) make use of RangerAccessResultProcessor, which seems to end up in the audit log. Our test environment does show such events, but I am fairly certain this does not actually use the Audit DB (with the new Elasticsearch support). I have prepared some test code that uses a local Elasticsearch to read and write events (ElasticSearchAccessAuditsServiceTest) but have been unable to test actual audit events in our integration test environment. Do you have any suggestions on how to properly test this? Thank you very much for your guidance and feedback! > Add ElasticSearch support > - > > Key: RANGER-2634 > URL: https://issues.apache.org/jira/browse/RANGER-2634 > Project: Ranger > Issue Type: Wish > Components: audit >Reporter: Andrew Charneski >Priority: Major > > Hello, > Our organization is working on integrating and rolling out Apache Ranger for > our internal toolset, and firstly, we'd like to thank you all for your great > work! > Although we have decided to adopt Ranger, for a variety of reasons we would > very much like to use an alternate indexing service (Elasticsearch) in > preference to the currently-supported Solr. > We are happy to develop this extension on our own, and the initial efforts > are underway at [https://github.com/acharneski/ranger/pull/1] however we > would like to engage the community for guidance and suggestions with the aim > of getting this change merged into the main codebase when it is ready. > Our initial approach is in three main phases: > # Isolate all usages of Solr packages/classes to pass through an API > wrapper, delegating to the original Solr code with minimal changes. (Dev > complete) > # Refactor api package as a pluggable interface with greater simplicity and > implementation agnosticism. > # Provide alternate implementation for the new interface for Elasticsearch. > Any guidance and feedback is appreciated. Thank you for reading! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (RANGER-2634) Add ElasticSearch support
[ https://issues.apache.org/jira/browse/RANGER-2634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16979343#comment-16979343 ] Andrew Charneski edited comment on RANGER-2634 at 11/25/19 10:54 PM: - Hi [~bosco] - This PR (at [https://github.com/apache/ranger/pull/42|https://github.com/apache/ranger/pull/42)]) is ready for review and feedback. Some of the items are merely to get the build to work in my environments. I'm happy to create the review on Reviewboard if that's appropriate at this point, but had one major question first I'm hoping you can provide guidance on. My main concern is that I have been unable to test this functionality in an integrated environment - Currently our organization is only using the Hive plugin and my analysis of the code suggests that only "grant" and "revoke" permission operations (in RangerBasePlugin) make use of RangerAccessResultProcessor, which seems to end up in the audit log. Our test environment does show such events, but I am fairly certain this does not actually use the Audit DB (with the new Elasticsearch support). I have prepared some test code that uses a local Elasticsearch to read and write events (ElasticSearchAccessAuditsServiceTest) but have been unable to test actual audit events in our integration test environment. Do you have any suggestions on how to properly test this? Thank you very much for your guidance and feedback! was (Author: acharneski): Hi [~bosco] - This PR (at [https://github.com/apache/ranger/pull/42)] is ready for review and feedback. Some of the items are merely to get the build to work in my environments. I'm happy to create the review on Reviewboard if that's appropriate at this point, but had one major question first I'm hoping you can provide guidance on. My main concern is that I have been unable to test this functionality in an integrated environment - Currently our organization is only using the Hive plugin and my analysis of the code suggests that only "grant" and "revoke" permission operations (in RangerBasePlugin) make use of RangerAccessResultProcessor, which seems to end up in the audit log. Our test environment does show such events, but I am fairly certain this does not actually use the Audit DB (with the new Elasticsearch support). I have prepared some test code that uses a local Elasticsearch to read and write events (ElasticSearchAccessAuditsServiceTest) but have been unable to test actual audit events in our integration test environment. Do you have any suggestions on how to properly test this? Thank you very much for your guidance and feedback! > Add ElasticSearch support > - > > Key: RANGER-2634 > URL: https://issues.apache.org/jira/browse/RANGER-2634 > Project: Ranger > Issue Type: Wish > Components: audit >Reporter: Andrew Charneski >Priority: Major > > Hello, > Our organization is working on integrating and rolling out Apache Ranger for > our internal toolset, and firstly, we'd like to thank you all for your great > work! > Although we have decided to adopt Ranger, for a variety of reasons we would > very much like to use an alternate indexing service (Elasticsearch) in > preference to the currently-supported Solr. > We are happy to develop this extension on our own, and the initial efforts > are underway at [https://github.com/acharneski/ranger/pull/1] however we > would like to engage the community for guidance and suggestions with the aim > of getting this change merged into the main codebase when it is ready. > Our initial approach is in three main phases: > # Isolate all usages of Solr packages/classes to pass through an API > wrapper, delegating to the original Solr code with minimal changes. (Dev > complete) > # Refactor api package as a pluggable interface with greater simplicity and > implementation agnosticism. > # Provide alternate implementation for the new interface for Elasticsearch. > Any guidance and feedback is appreciated. Thank you for reading! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-2634) Add ElasticSearch support
[ https://issues.apache.org/jira/browse/RANGER-2634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16979343#comment-16979343 ] Andrew Charneski commented on RANGER-2634: -- Hi [~bosco] - This PR (at [https://github.com/apache/ranger/pull/42)] is ready for review and feedback. Some of the items are merely to get the build to work in my environments. I'm happy to create the review on Reviewboard if that's appropriate at this point, but had one major question first I'm hoping you can provide guidance on. My main concern is that I have been unable to test this functionality in an integrated environment - Currently our organization is only using the Hive plugin and my analysis of the code suggests that only "grant" and "revoke" permission operations (in RangerBasePlugin) make use of RangerAccessResultProcessor, which seems to end up in the audit log. Our test environment does show such events, but I am fairly certain this does not actually use the Audit DB (with the new Elasticsearch support). I have prepared some test code that uses a local Elasticsearch to read and write events (ElasticSearchAccessAuditsServiceTest) but have been unable to test actual audit events in our integration test environment. Do you have any suggestions on how to properly test this? Thank you very much for your guidance and feedback! > Add ElasticSearch support > - > > Key: RANGER-2634 > URL: https://issues.apache.org/jira/browse/RANGER-2634 > Project: Ranger > Issue Type: Wish > Components: audit >Reporter: Andrew Charneski >Priority: Major > > Hello, > Our organization is working on integrating and rolling out Apache Ranger for > our internal toolset, and firstly, we'd like to thank you all for your great > work! > Although we have decided to adopt Ranger, for a variety of reasons we would > very much like to use an alternate indexing service (Elasticsearch) in > preference to the currently-supported Solr. > We are happy to develop this extension on our own, and the initial efforts > are underway at [https://github.com/acharneski/ranger/pull/1] however we > would like to engage the community for guidance and suggestions with the aim > of getting this change merged into the main codebase when it is ready. > Our initial approach is in three main phases: > # Isolate all usages of Solr packages/classes to pass through an API > wrapper, delegating to the original Solr code with minimal changes. (Dev > complete) > # Refactor api package as a pluggable interface with greater simplicity and > implementation agnosticism. > # Provide alternate implementation for the new interface for Elasticsearch. > Any guidance and feedback is appreciated. Thank you for reading! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (RANGER-2634) Add ElasticSearch support
[ https://issues.apache.org/jira/browse/RANGER-2634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Charneski updated RANGER-2634: - Summary: Add ElasticSearch support (was: Modularize Solr dependency) > Add ElasticSearch support > - > > Key: RANGER-2634 > URL: https://issues.apache.org/jira/browse/RANGER-2634 > Project: Ranger > Issue Type: Wish > Components: audit >Reporter: Andrew Charneski >Priority: Major > > Hello, > Our organization is working on integrating and rolling out Apache Ranger for > our internal toolset, and firstly, we'd like to thank you all for your great > work! > Although we have decided to adopt Ranger, for a variety of reasons we would > very much like to use an alternate indexing service (Elasticsearch) in > preference to the currently-supported Solr. > We are happy to develop this extension on our own, and the initial efforts > are underway at [https://github.com/acharneski/ranger/pull/1] however we > would like to engage the community for guidance and suggestions with the aim > of getting this change merged into the main codebase when it is ready. > Our initial approach is in three main phases: > # Isolate all usages of Solr packages/classes to pass through an API > wrapper, delegating to the original Solr code with minimal changes. (Dev > complete) > # Refactor api package as a pluggable interface with greater simplicity and > implementation agnosticism. > # Provide alternate implementation for the new interface for Elasticsearch. > Any guidance and feedback is appreciated. Thank you for reading! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-2634) Modularize Solr dependency
[ https://issues.apache.org/jira/browse/RANGER-2634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967029#comment-16967029 ] Andrew Charneski commented on RANGER-2634: -- Thank you for that pointer! We will go that route, that's a lot easier and more reasonable! More soon. I'm going to rename this story. > Modularize Solr dependency > -- > > Key: RANGER-2634 > URL: https://issues.apache.org/jira/browse/RANGER-2634 > Project: Ranger > Issue Type: Wish > Components: audit >Reporter: Andrew Charneski >Priority: Major > > Hello, > Our organization is working on integrating and rolling out Apache Ranger for > our internal toolset, and firstly, we'd like to thank you all for your great > work! > Although we have decided to adopt Ranger, for a variety of reasons we would > very much like to use an alternate indexing service (Elasticsearch) in > preference to the currently-supported Solr. > We are happy to develop this extension on our own, and the initial efforts > are underway at [https://github.com/acharneski/ranger/pull/1] however we > would like to engage the community for guidance and suggestions with the aim > of getting this change merged into the main codebase when it is ready. > Our initial approach is in three main phases: > # Isolate all usages of Solr packages/classes to pass through an API > wrapper, delegating to the original Solr code with minimal changes. (Dev > complete) > # Refactor api package as a pluggable interface with greater simplicity and > implementation agnosticism. > # Provide alternate implementation for the new interface for Elasticsearch. > Any guidance and feedback is appreciated. Thank you for reading! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (RANGER-2634) Modularize Solr dependency
Andrew Charneski created RANGER-2634: Summary: Modularize Solr dependency Key: RANGER-2634 URL: https://issues.apache.org/jira/browse/RANGER-2634 Project: Ranger Issue Type: Wish Components: audit Reporter: Andrew Charneski Hello, Our organization is working on integrating and rolling out Apache Ranger for our internal toolset, and firstly, we'd like to thank you all for your great work! Although we have decided to adopt Ranger, for a variety of reasons we would very much like to use an alternate indexing service (Elasticsearch) in preference to the currently-supported Solr. We are happy to develop this extension on our own, and the initial efforts are underway at [https://github.com/acharneski/ranger/pull/1] however we would like to engage the community for guidance and suggestions with the aim of getting this change merged into the main codebase when it is ready. Our initial approach is in three main phases: # Isolate all usages of Solr packages/classes to pass through an API wrapper, delegating to the original Solr code with minimal changes. (Dev complete) # Refactor api package as a pluggable interface with greater simplicity and implementation agnosticism. # Provide alternate implementation for the new interface for Elasticsearch. Any guidance and feedback is appreciated. Thank you for reading! -- This message was sent by Atlassian Jira (v8.3.4#803005)
