[ 
https://issues.apache.org/jira/browse/KUDU-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17067100#comment-17067100
 ] 

ASF subversion and git services commented on KUDU-3079:
-------------------------------------------------------

Commit b417479266ef0f09570d4536738ca18cf48d7ff2 in kudu's branch 
refs/heads/master from Attila Bukor
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=b417479 ]

KUDU-3079 Add MiniRanger

Apache Ranger has some setup scripts[1][2] that should be used to
install it but they touch a lot of things on the file system that
unfortunately can't be configured or $PREFIX-ed/chroot-ed, so we've
opted for a more manual approach similar to what Impala have chosen[3].

As Ranger 2.0.0 was released back in August 2019 which didn't contain
the Kudu service definition and it wasn't a clean backport I opted to
use the tip of the master (at the time of writing).

Unfortunately Ranger's scripts were written for Python 2 and there were
several compatibility issues with Python 3 so I had to patch it. Most of
the changes were generated by Python's "2to3" script but it required
some manual changes as well to run on Python 3 (subprocesses return
bytes and they were compared with strings which wasn't a problem in
Python 2 but fails in Python 3 so the subprocess result had to be
decoded).

I had to apply another patch to fix an issue running db_setup.py on Mac
as it runs ranger-admin-services.sh which sets the script path by
running `readlink -f $0` which doesn't work on Mac. Instead of rewriting
the script in a way that would work properly on Mac and Linux I opted to
use an environment variable instead, so I decided to use a separate
patch file for this as this one will need to be applied even after
Ranger decides to support Python 3[4].

The commit also adds support for HTTP Basic auth in EasyCurl along with
specifying headers in POST requests which is used in MiniRanger to talk
to the Ranger admin REST API when creating the Kudu service after
startup and adding policies.

Also adds a CreateSymLink() method to Env.

Starting MiniRanger (fresh install) takes ~20s on my machine (2019
MacBook Pro) and ~30-40s on dist-test server.

Starting Ranger subprocess is not handled in this commit, it is going to
be done in a follow-up patch.

Thanks to Andrew Wong for his help with the Ranger setup[5].

[1] https://github.com/apache/ranger/blob/master/security-admin/scripts/setup.sh
[2] 
https://github.com/apache/ranger/blob/master/security-admin/scripts/set_globals.sh
[3] 
https://github.com/apache/impala/commit/0cb7187841780cabe368607ff559e493be59db22
[4] https://issues.apache.org/jira/browse/RANGER-2759
[5] https://gerrit.cloudera.org/c/15385/5

Change-Id: I15ab1eb8abe71c074c26b286073442882e101bc6
Reviewed-on: http://gerrit.cloudera.org:8080/15483
Reviewed-by: Adar Dembo <a...@cloudera.com>
Tested-by: Attila Bukor <abu...@apache.org>
Reviewed-by: Andrew Wong <aw...@cloudera.com>


> Add MiniRanger for integration tests
> ------------------------------------
>
>                 Key: KUDU-3079
>                 URL: https://issues.apache.org/jira/browse/KUDU-3079
>             Project: Kudu
>          Issue Type: Sub-task
>            Reporter: Attila Bukor
>            Assignee: Attila Bukor
>            Priority: Major
>
> To write full integration tests we need a bundled Ranger service



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to