from:"Velmurugan Periasamy"

Re: Exception while creating encryption zone

2016-09-21 Thread Velmurugan Periasamy

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
16/09/21 14:22:51 ERROR hdfs.DFSClient: Failed to close inode 3322459
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
 No lease on /user/lchanel/testdir/test.txt._COPYING_ (inode 3322459): File 
does not exist. Holder DFSClient_NONMAPREDUCE_1559190789_1 does not have any 
open files.
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3521)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3611)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:3578)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:905)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:544)
at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307)

at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1552)
at org.apache.hadoop.ipc.Client.call(Client.java:1496)
at org.apache.hadoop.ipc.Client.call(Client.java:1396)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at com.sun.proxy.$Proxy10.complete(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:501)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
at com.sun.proxy.$Proxy11.complete(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2361)
at 
org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:2338)
at 
org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2303)
at 
org.apache.hadoop.hdfs.DFSClient.closeAllFilesBeingWritten(DFSClient.java:947)
at 
org.apache.hadoop.hdfs.DFSClient.closeOutputStreams(DFSClient.java:979)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:1192)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2852)
at 
org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2869)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Did I miss something ? Because I definitely gave hdfs the same rights than 
keyadmin user within the interface of Ranger KMS.
Thanks for your help,


Loïc

Loïc CHANEL
System Big Data engineer
MS - WASABI - Worldline (Villeurbanne, France)

2016-09-16 16:49 GMT+02:00 Loïc Chanel 
<loic.cha...@telecomnancy.net<mailto:loic.cha...@telecomnancy.net>>:
You were right indeed. Only keyadmin user was granted these rights (as I 
thought hdfs was not submitted to Ranger authorizations), and it was the root 
issue.
Thanks a lot !

Regards,


Loïc

Loïc CHANEL
System Big Data engineer
MS - WASABI - Worldline (Villeurbanne, France)

2016-09-16 16:41 GMT+02:00 Velmurugan Periasamy 
<vperias...@hortonworks.com<mailto:vperias...@hortonworks.com>>:
HDFS user is superuser only for HDFS, for key operations it needs t

Re: Exception while creating encryption zone

2016-09-16 Thread Velmurugan Periasamy

HDFS user is superuser only for HDFS, for key operations it needs to have 
permissions. Login to Ranger using keyadmin/keyadmin and see if there are KMS 
policies giving access to “hdfs” user. If not, grant these permissions.


From: Loïc Chanel 
<loic.cha...@telecomnancy.net<mailto:loic.cha...@telecomnancy.net>>
Reply-To: 
"user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" 
<user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>>
Date: Friday, September 16, 2016 at 10:38 AM
To: "user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" 
<user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>>
Subject: Re: Exception while creating encryption zone

As he's the superdamin user, he should be able to do so, right ?
If not, how can I test this ?

Loïc CHANEL
System Big Data engineer
MS - WASABI - Worldline (Villeurbanne, France)

2016-09-16 16:20 GMT+02:00 Velmurugan Periasamy 
<vperias...@hortonworks.com<mailto:vperias...@hortonworks.com>>:
Loïc:

Can you make sure hdfs user has permissions for key operations (especially 
GENERATE_EEK and GET_METADATA) and try again?

Thank you,
Vel

From: Loïc Chanel 
<loic.cha...@telecomnancy.net<mailto:loic.cha...@telecomnancy.net>>
Reply-To: 
"user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" 
<user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>>
Date: Friday, September 16, 2016 at 8:53 AM
To: "user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" 
<user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>>
Subject: Re: Exception while creating encryption zone

Hi all,

Using TCPDUMP, I investigated a little bit more, and I found that there isn't 
any call from the host I make my "hdfs crypto -createZone -keyName test_lchanel 
-path /user/lchanel" to the port 9292 of the host where Ranger KMS is located.
So it seems it is a configuration or runtime problem.

Does anyone have an idea about where to investigate next ?

Thanks,


Loïc

Loïc CHANEL
System Big Data engineer
MS - WASABI - Worldline (Villeurbanne, France)

2016-09-13 11:20 GMT+02:00 Loïc Chanel 
<loic.cha...@telecomnancy.net<mailto:loic.cha...@telecomnancy.net>>:
Hi all,

As I was trying to test Ranger KMS, I encountered some troubles.
I created a AES-128 key with ranger KMS named test_lchanel, and as I wanted to 
use it to encrypt my home repository using : hdfs crypto -createZone -keyName 
test_lchanel -path /user/lchanel, I got the following exception :

16/09/13 11:11:26 WARN retry.RetryInvocationHandler: Exception while invoking 
ClientNamenodeProtocolTranslatorPB.createEncryptionZone over null. Not retrying 
because try once and fail.
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1552)
at org.apache.hadoop.ipc.Client.call(Client.java:1496)
at org.apache.hadoop.ipc.Client.call(Client.java:1396)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at com.sun.proxy.$Proxy10.createEncryptionZone(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.createEncryptionZone(ClientNamenodeProtocolTranslatorPB.java:1426)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
at com.sun.proxy.$Proxy11.createEncryptionZone(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSClient.createEncryptionZone(DFSClient.java:3337)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.createEncryptionZone(DistributedFileSystem.java:2233)
at 
org.apache.hadoop.hdfs.client.HdfsAdmin.createEncryptionZone(HdfsAdmin.java:307)
at 
org.apache.hadoop.hdfs.tools.CryptoAdmin$CreateZoneCommand.run(CryptoAdmin.java:142)
at org.apache.hadoop.hdfs.tools.CryptoAdmin.run(CryptoAdmin.java:73)
at org.apache.hadoop.hdfs.tools.CryptoAdmin.main(CryptoAdmin.java:82)
RemoteException:

As I know CPU must support AES to use such things, I checked on each server's 
ILO admin interface and it seems my CPU support AES-128. In addition, hadoop 
checknative returns a correct result :

16/

Re: Need help in integrating apache-ranger-incubating-0.6.1

2016-09-16 Thread Velmurugan Periasamy

Venu:

I am able to build apache-ranger-incubating-0.6.1 without error. Can you
verify JAVA_HOME is pointing to JDK 7+ and your maven settings?

I am also posting this in ranger lists to see if anyone came across this
issue.   

Thank you,
Vel


From:  Venu Nadh Veeralanka <vvn...@gmail.com>
Date:  Thursday, September 15, 2016 at 8:49 PM
To:  Velmurugan Periasamy <v...@apache.org>
Subject:  Need help in integrating apache-ranger-incubating-0.6.1

Hi Vel,

We are trying to evaluate Apache Ranger for our Hadoop security policies.

It is failing while building from source code as mentioned below.

Do you have any known issues or any pointers(discussion groups) that will
help us addressing these issues.

root@hostname apache-ranger-incubating-0.6.1]# mvn clean compile package
assembly:assembly install

[INFO] Scanning for projects...

Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/apac
he/17/apache-17.pom
<https://repository.apache.org/content/repositories/snapshots/org/apache/apa
che/17/apache-17.pom>

Downloading: 
https://repository.apache.org/content/repositories/public/org/apache/apache/
17/apache-17.pom 
<https://repository.apache.org/content/repositories/public/org/apache/apache
/17/apache-17.pom> 

Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/17/apache-17.pom
<https://repo.maven.apache.org/maven2/org/apache/apache/17/apache-17.pom>

[ERROR] [ERROR] Some problems were encountered while processing the POMs:

[ERROR] [ERROR] Some problems were encountered while processing the POMs:

[FATAL] Non-resolvable parent POM for org.apache.ranger:ranger:0.6.1: Could
not transfer artifact org.apache:apache:pom:17 from/to
apache.snapshots.https
(https://repository.apache.org/content/repositories/snapshots
<https://repository.apache.org/content/repositories/snapshots> ): Connect to
repository.apache.org:443 <http://repository.apache.org:443/>
[repository.apache.org/207.244.88.143
<http://repository.apache.org/207.244.88.143> ] failed: Connection timed out
and 'parent.relativePath' points at wrong local POM @ line 19, column 13

 @ 

[ERROR] The build could not read 1 project -> [Help 1]

[ERROR]   

[ERROR]   The project org.apache.ranger:ranger:0.6.1
(/root/cdh-downloads/dev/apache-ranger-incubating-0.6.1/pom.xml) has 1 error

[ERROR] Non-resolvable parent POM for org.apache.ranger:ranger:0.6.1:
Could not transfer artifact org.apache:apache:pom:17 from/to
apache.snapshots.https
(https://repository.apache.org/content/repositories/snapshots
<https://repository.apache.org/content/repositories/snapshots> ): Connect to
repository.apache.org:443 <http://repository.apache.org:443/>
[repository.apache.org/207.244.88.143
<http://repository.apache.org/207.244.88.143> ] failed: Connection timed out
and 'parent.relativePath' points at wrong local POM @ line 19, column 13 ->
[Help 2]

[ERROR] 

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR] 

[ERROR] For more information about the errors and possible solutions, please
read the following articles:

[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
<http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException>

[ERROR] [Help 2] 
http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException
<http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException
> 



Regards,
venu.

Re: Exception while creating encryption zone

2016-09-16 Thread Velmurugan Periasamy

Loïc:

Can you make sure hdfs user has permissions for key operations (especially 
GENERATE_EEK and GET_METADATA) and try again?

Thank you,
Vel

From: Loïc Chanel 
>
Reply-To: 
"user@ranger.incubator.apache.org" 
>
Date: Friday, September 16, 2016 at 8:53 AM
To: "user@ranger.incubator.apache.org" 
>
Subject: Re: Exception while creating encryption zone

Hi all,

Using TCPDUMP, I investigated a little bit more, and I found that there isn't 
any call from the host I make my "hdfs crypto -createZone -keyName test_lchanel 
-path /user/lchanel" to the port 9292 of the host where Ranger KMS is located.
So it seems it is a configuration or runtime problem.

Does anyone have an idea about where to investigate next ?

Thanks,


Loïc

Loïc CHANEL
System Big Data engineer
MS - WASABI - Worldline (Villeurbanne, France)

2016-09-13 11:20 GMT+02:00 Loïc Chanel 
>:
Hi all,

As I was trying to test Ranger KMS, I encountered some troubles.
I created a AES-128 key with ranger KMS named test_lchanel, and as I wanted to 
use it to encrypt my home repository using : hdfs crypto -createZone -keyName 
test_lchanel -path /user/lchanel, I got the following exception :

16/09/13 11:11:26 WARN retry.RetryInvocationHandler: Exception while invoking 
ClientNamenodeProtocolTranslatorPB.createEncryptionZone over null. Not retrying 
because try once and fail.
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1552)
at org.apache.hadoop.ipc.Client.call(Client.java:1496)
at org.apache.hadoop.ipc.Client.call(Client.java:1396)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at com.sun.proxy.$Proxy10.createEncryptionZone(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.createEncryptionZone(ClientNamenodeProtocolTranslatorPB.java:1426)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
at com.sun.proxy.$Proxy11.createEncryptionZone(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSClient.createEncryptionZone(DFSClient.java:3337)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.createEncryptionZone(DistributedFileSystem.java:2233)
at 
org.apache.hadoop.hdfs.client.HdfsAdmin.createEncryptionZone(HdfsAdmin.java:307)
at 
org.apache.hadoop.hdfs.tools.CryptoAdmin$CreateZoneCommand.run(CryptoAdmin.java:142)
at org.apache.hadoop.hdfs.tools.CryptoAdmin.run(CryptoAdmin.java:73)
at org.apache.hadoop.hdfs.tools.CryptoAdmin.main(CryptoAdmin.java:82)
RemoteException:

As I know CPU must support AES to use such things, I checked on each server's 
ILO admin interface and it seems my CPU support AES-128. In addition, hadoop 
checknative returns a correct result :

16/09/13 11:16:48 INFO bzip2.Bzip2Factory: Successfully loaded & initialized 
native-bzip2 library system-native
16/09/13 11:16:48 INFO zlib.ZlibFactory: Successfully loaded & initialized 
native-zlib library
Native library checking:
hadoop:  true /usr/hdp/2.5.0.0-1245/hadoop/lib/native/libhadoop.so.1.0.0
zlib:true /lib64/libz.so.1
snappy:  true /usr/hdp/2.5.0.0-1245/hadoop/lib/native/libsnappy.so.1
lz4: true revision:99
bzip2:   true /lib64/libbz2.so.1
openssl: true /usr/lib64/libcrypto.so

Does someone see where my problem might come from ?

Thanks,


Loïc

Loïc CHANEL
System Big Data engineer
MS - WASABI - Worldline (Villeurbanne, France)

CVE update (CVE-2016-5395) - Fixed in Apache Ranger 0.6.1

2016-08-22 Thread Velmurugan Periasamy

Hello:

Here¹s a CVE update for Ranger 0.6.1 release. Please see below details.

Release details can be found at
https://cwiki.apache.org/confluence/display/RANGER/0.6.1+Release+-+Apache+Ra
nger

Thank you,
Velmurugan Periasamy


---
CVE-2016-5395: Apache Ranger Stored Cross Site Scripting vulnerability

---
Severity: Normal 
Vendor: The Apache Software Foundation
Versions Affected: All 0.5.x versions of Apache Ranger and version 0.6.0
Users Affected: All users of ranger policy admin tool
Description: Apache Ranger was found to be vulnerable to a
Stored Cross-Site Scripting in the create user functionality. Admin users
can
store some arbitrary javascript code to be executed when normal users login
and
access policies. 
Fix details: Added logic to sanitize the user input
Mitigation: Users should upgrade to 0.6.1 or later version of Apache Ranger
with the fix.
Credit: Thanks to Victor Hora from Securus Global for reporting this issue.

---

Re: Ranger-0.6 HDFS authentication failed in secure mode

2016-08-11 Thread Velmurugan Periasamy

Error you posted seems to be related to test connection failing, not download 
policy issue. @Sailaja - can you please chime in for the decrypt password issue?

Can you please share 1] your HDFS repository configuration 2] any errors in 
ranger log during the download policy from HDFS plugin

Thanks,
Vel

From: Aneela Saleem >
Reply-To: 
"user@ranger.incubator.apache.org" 
>
Date: Thursday, August 11, 2016 at 11:32 PM
To: "user@ranger.incubator.apache.org" 
>
Subject: Re: Ranger-0.6 HDFS authentication failed in secure mode

Hi Folks!

I have tried different options like kinit using nn/hadoop-master principal. And 
then enable hdfs plugin and start hadoop. But I am still facing the same issue. 
Any help related to above issue will be appreciable.

Thanks

On Mon, Aug 8, 2016 at 8:47 PM, Aneela Saleem 
> wrote:
Madhan!

I can see following exception in ranger-admin.log file

2016-08-08 17:42:43,501 [timed-executor-pool-0] ERROR 
apache.ranger.services.hdfs.client.HdfsResourceMgr (HdfsResourceMgr.java:49) - 
<== HdfsResourceMgr.testConnection Error: Unable to login to Hadoop environment 
[hdfs]
org.apache.ranger.plugin.client.HadoopException: Unable to login to Hadoop 
environment [hdfs]
at org.apache.ranger.plugin.client.BaseClient.login(BaseClient.java:136)
at org.apache.ranger.plugin.client.BaseClient.(BaseClient.java:59)
at 
org.apache.ranger.services.hdfs.client.HdfsClient.(HdfsClient.java:52)
at 
org.apache.ranger.services.hdfs.client.HdfsClient.connectionTest(HdfsClient.java:221)
at 
org.apache.ranger.services.hdfs.client.HdfsResourceMgr.connectionTest(HdfsResourceMgr.java:47)
at 
org.apache.ranger.services.hdfs.RangerServiceHdfs.validateConfig(RangerServiceHdfs.java:58)
at 
org.apache.ranger.biz.ServiceMgr$ValidateCallable.actualCall(ServiceMgr.java:560)
at 
org.apache.ranger.biz.ServiceMgr$ValidateCallable.actualCall(ServiceMgr.java:547)
at 
org.apache.ranger.biz.ServiceMgr$TimedCallable.call(ServiceMgr.java:508)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Unable to decrypt password due to error
at 
org.apache.ranger.plugin.util.PasswordUtils.decryptPassword(PasswordUtils.java:128)
at org.apache.ranger.plugin.client.BaseClient.login(BaseClient.java:113)
... 12 more
Caused by: javax.crypto.IllegalBlockSizeException: Input length must be 
multiple of 8 when decrypting with padded cipher
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:750)
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:676)
at com.sun.crypto.provider.PBECipherCore.doFinal(PBECipherCore.java:422)
at 
com.sun.crypto.provider.PBEWithMD5AndDESCipher.engineDoFinal(PBEWithMD5AndDESCipher.java:316)
at javax.crypto.Cipher.doFinal(Cipher.java:2131)
at 
org.apache.ranger.plugin.util.PasswordUtils.decryptPassword(PasswordUtils.java:112)
... 13 more

On Mon, Aug 8, 2016 at 8:16 PM, Madhan Neethiraj 
> wrote:
Aneela,

Do you see any errors reported in Ranger Admin log file xa_portal.log, for the 
download request from the HDFS plugin?

Thanks,
Madhan

From: Aneela Saleem >
Reply-To: 
"user@ranger.incubator.apache.org" 
>
Date: Monday, August 8, 2016 at 6:05 AM
To: "user@ranger.incubator.apache.org" 
>
Subject: Ranger-0.6 HDFS authentication failed in secure mode

Hi all,

I have installed Ranger-0.6 version, i successfully installed the usersync 
process. Now i'm trying to enable HDFS plugin on Kerberized Hadoop Cluster. 
When is restart Hadoop after enabling the plugin, i get the following error:

2016-08-08 17:56:55,675 ERROR 
org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting policies. 
secureMode=true, user=nn/hadoop-master@platalyticsrealm (auth:KERBEROS), 
response={"httpStatusCode":401,"statusCode":401,"msgDesc":"Authentication 
Failed"}, serviceName=hdfs
2016-08-08 17:56:55,675 ERROR

Re: HDFS Plugin - Unable to get listing of files for directory [/] from Hadoop environment

2016-06-15 Thread Velmurugan Periasamy

Having active name node in repo config should work just fine. Only resource 
lookup is not available during failover cases, until the repo config is updated.

For HA configuration to work, need to add the below properties in repo config 
(I.e. additional entries in the advanced section). They can be copied from 
hdfs-site.xml.

dfs.nameservices = 
dfs.ha.namenodes. = <nn1,nn2>
dfs.namenode.rpc-address. = 
dfs.namenode.rpc-address. = 
dfs.client.failover.proxy.provider. = 
org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider


From: Dale Bradman <da...@profusion.com<mailto:da...@profusion.com>>
Reply-To: 
"user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" 
<user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>>
Date: Wednesday, June 15, 2016 at 10:51 AM
To: "user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" 
<user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>>
Subject: RE: HDFS Plugin - Unable to get listing of files for directory [/] 
from Hadoop environment

That did not work.

It works when I set:
Hadoop.rpc.protection = -

Then in HDFS plugin:
Namenode URL = hdfs://hdpmaster01:8020
RPC Protection Type = Authentication

The above works. It seems it is the HA configuration that is a problem. Will it 
work with NameNode HA? Is there any risk for it not being configured to HA?

Thanks.
From: Velmurugan Periasamy [mailto:vperias...@hortonworks.com]
Sent: 15 June 2016 14:31
To: user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>
Subject: Re: HDFS Plugin - Unable to get listing of files for directory [/] 
from Hadoop environment

Dale:

Could you set hadoop.rpc.protection to authentication and try?

Thank you,
Vel

From: Dale Bradman <da...@profusion.com<mailto:da...@profusion.com>>
Reply-To: 
"user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" 
<user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>>
Date: Wednesday, June 15, 2016 at 9:28 AM
To: "user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" 
<user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>>
Subject: HDFS Plugin - Unable to get listing of files for directory [/] from 
Hadoop environment

Trying to configure the HDFS plugin for Keberised, HA, HDP 2.4.2.
I have followed this guide 
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_Security_Guide/content/hdfs_plugin_kerberos.html
I have created a “rangerrepouser” in AD and is visible in the Ranger UI.

Advanced ranger-hdfs-pluging properties:
Ranger repository config user = 
rangerrepouser@AD.EXAMPLE<mailto:rangerrepouser@AD.EXAMPLE>
Ranger repository config password = password set in AD
Hadoop.rpc.protection =


HDFS Service props:
Username: rangerrepouser@AD.EXAMPLE<mailto:rangerrepouser@MAILTRACK.LOCAL>
Namenode URL: hdfs://tatooine
Authorization enabled: Yes
Authentication type: Kerberos
hadoop.security.auth_to_local :
RULE:[1:$1@$0](ambari-qa-Tatooine@AD.EXAMPLE)s/.*/ambari-qa/RULE:[1:$1@$0](hdfs-Tatooine@AD.EXAMPLE)s/.*/hdfs/RULE:[1:$1@$0](.*@AD.EXAMPLE)s/@.*//RULE:[2:$1@$0](amshbase@AD.EXAMPLE)s/.*/ams/RULE:[2:$1@$0](amszk@AD.EXAMPLE)s/.*/ams/RULE:[2:$1@$0](dn@AD.EXAMPLE)s/.*/hdfs/RULE:[2:$1@$0](hive@AD.EXAMPLE)s/.*/hive/RULE:[2:$1@$0](jhs@AD.EXAMPLE)s/.*/mapred/RULE:[2:$1@$0](jn@AD.EXAMPLE)s/.*/hdfs/RULE:[2:$1@$0](nm@AD.EXAMPLE)s/.*/yarn/RULE:[2:$1@$0](nn@AD.EXAMPLE)s/.*/hdfs/RULE:[2:$1@$0](rm@AD.EXAMPLE)s/.*/yarn/RULE:[2:$1@$0](yarn@AD.EXAMPLE)s/.*/yarn/DEFAULT<mailto:ambari-qa-Tatooine@MAILTRACK.LOCAL)s/.*/ambari-qa/RULE:%5b1:$1@$0%5d(hdfs-Tatooine@MAILTRACK.LOCAL)s/.*/hdfs/RULE:%5b1:$1@$0%5d(.*@MAILTRACK.LOCAL)s/@.*//RULE:%5b2:$1@$0%5d(amshbase@MAILTRACK.LOCAL)s/.*/ams/RULE:%5b2:$1@$0%5d(amszk@MAILTRACK.LOCAL)s/.*/ams/RULE:%5b2:$1@$0%5d(dn@MAILTRACK.LOCAL)s/.*/hdfs/RULE:%5b2:$1@$0%5d(hive@MAILTRACK.LOCAL)s/.*/hive/RULE:%5b2:$1@$0%5d(jhs@MAILTRACK.LOCAL)s/.*/mapred/RULE:%5b2:$1@$0%5d(jn@MAILTRACK.LOCAL)s/.*/hdfs/RULE:%5b2:$1@$0%5d(nm@MAILTRACK.LOCAL)s/.*/yarn/RULE:%5b2:$1@$0%5d(nn@MAILTRACK.LOCAL)s/.*/hdfs/RULE:%5b2:$1@$0%5d(rm@MAILTRACK.LOCAL)s/.*/yarn/RULE:%5b2:$1@$0%5d(yarn@MAILTRACK.LOCAL)s/.*/yarn/DEFAULT>
Dfs.datanode.kerberos.principal=dn/hdpnode01.hadoop.local@AD.EXAMPLE<mailto:Dfs.datanode.kerberos.principal=dn/hdpnode01.hadoop.local@AD.EXAMPLE>
Dfs.namenode.kerberos.principal= nn/hdpmaster01.hadoop.local@ AD.EXAMPLE
Dfs.secondary.namenode.kerberos.principal nn/hdpmaster01.hadoop.local@ 
AD.EXAMPLE
RPC Protection Type =


Here is the xa_portal.log:
2016-06-15 14:21:05,037 [timed-executor-pool-0] INFO  
org.apache.ranger.plugin.client.BaseClient (BaseClient.java:100) - Init Login: 
using username/password
2016-06-15 14:21:05,194 [timed-executor-pool-0] ERROR 
apache.ranger.service

Re: HDFS Plugin - Unable to get listing of files for directory [/] from Hadoop environment

2016-06-15 Thread Velmurugan Periasamy

Dale:

Could you set hadoop.rpc.protection to authentication and try?

Thank you,
Vel

From: Dale Bradman >
Reply-To: 
"user@ranger.incubator.apache.org" 
>
Date: Wednesday, June 15, 2016 at 9:28 AM
To: "user@ranger.incubator.apache.org" 
>
Subject: HDFS Plugin - Unable to get listing of files for directory [/] from 
Hadoop environment

Trying to configure the HDFS plugin for Keberised, HA, HDP 2.4.2.
I have followed this guide 
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_Security_Guide/content/hdfs_plugin_kerberos.html
I have created a “rangerrepouser” in AD and is visible in the Ranger UI.

Advanced ranger-hdfs-pluging properties:
Ranger repository config user = 
rangerrepouser@AD.EXAMPLE
Ranger repository config password = password set in AD
Hadoop.rpc.protection =


HDFS Service props:
Username: rangerrepouser@AD.EXAMPLE
Namenode URL: hdfs://tatooine
Authorization enabled: Yes
Authentication type: Kerberos
hadoop.security.auth_to_local :
RULE:[1:$1@$0](ambari-qa-Tatooine@AD.EXAMPLE)s/.*/ambari-qa/RULE:[1:$1@$0](hdfs-Tatooine@AD.EXAMPLE)s/.*/hdfs/RULE:[1:$1@$0](.*@AD.EXAMPLE)s/@.*//RULE:[2:$1@$0](amshbase@AD.EXAMPLE)s/.*/ams/RULE:[2:$1@$0](amszk@AD.EXAMPLE)s/.*/ams/RULE:[2:$1@$0](dn@AD.EXAMPLE)s/.*/hdfs/RULE:[2:$1@$0](hive@AD.EXAMPLE)s/.*/hive/RULE:[2:$1@$0](jhs@AD.EXAMPLE)s/.*/mapred/RULE:[2:$1@$0](jn@AD.EXAMPLE)s/.*/hdfs/RULE:[2:$1@$0](nm@AD.EXAMPLE)s/.*/yarn/RULE:[2:$1@$0](nn@AD.EXAMPLE)s/.*/hdfs/RULE:[2:$1@$0](rm@AD.EXAMPLE)s/.*/yarn/RULE:[2:$1@$0](yarn@AD.EXAMPLE)s/.*/yarn/DEFAULT
Dfs.datanode.kerberos.principal=dn/hdpnode01.hadoop.local@AD.EXAMPLE
Dfs.namenode.kerberos.principal= nn/hdpmaster01.hadoop.local@ AD.EXAMPLE
Dfs.secondary.namenode.kerberos.principal nn/hdpmaster01.hadoop.local@ 
AD.EXAMPLE
RPC Protection Type =


Here is the xa_portal.log:
2016-06-15 14:21:05,037 [timed-executor-pool-0] INFO  
org.apache.ranger.plugin.client.BaseClient (BaseClient.java:100) - Init Login: 
using username/password
2016-06-15 14:21:05,194 [timed-executor-pool-0] ERROR 
apache.ranger.services.hdfs.client.HdfsResourceMgr (HdfsResourceMgr.java:48) - 
<== HdfsResourceMgr.testConnection Error: 
org.apache.ranger.plugin.client.HadoopException: Unable to get listing of files 
for directory [/] from Hadoop environment [Tatooine_hadoop].
2016-06-15 14:21:05,194 [timed-executor-pool-0] ERROR 
org.apache.ranger.services.hdfs.RangerServiceHdfs (RangerServiceHdfs.java:59) - 
<== RangerServiceHdfs.validateConfig 
Error:org.apache.ranger.plugin.client.HadoopException: Unable to get listing of 
files for directory [/] from Hadoop environment [Tatooine_hadoop].
2016-06-15 14:21:05,195 [timed-executor-pool-0] ERROR 
org.apache.ranger.biz.ServiceMgr$TimedCallable (ServiceMgr.java:434) - 
TimedCallable.call: Error:org.apache.ranger.plugin.client.HadoopException: 
Unable to get listing of files for directory [/] from Hadoop environment 
[Tatooine_hadoop].
2016-06-15 14:21:05,195 [http-bio-6080-exec-3] ERROR 
org.apache.ranger.biz.ServiceMgr (ServiceMgr.java:120) - ==> 
ServiceMgr.validateConfig Error:java.util.concurrent.ExecutionException: 
org.apache.ranger.plugin.client.HadoopException: Unable to get listing of files 
for directory [/] from Hadoop environment [Tatooine_hadoop].



1.   Any ideas as to why this is not working? Everything seems consistent.

2.   Does the rangerrepouser have to be set up on the Ranger Admin server? 
It is visible on Ranger UI but is only synchronised with my edge node and not 
the Admin server

3.   Does it matter that the namenode and secondary namenode are pointing 
to the same Kerberos principal? Doesn’t work if I point them to their 
respective principals either.

Thanks,
Dale

CVE update (CVE-2016-2174) - Fixed in Ranger 0.5.3

2016-06-01 Thread Velmurugan Periasamy

Hello:

Here’s a CVE update for Ranger 0.5.3 release. Please see below details. 

Release details can be found at 
https://cwiki.apache.org/confluence/display/RANGER/0.5.3+Release+-+Apache+Ranger

Thank you,
Velmurugan Periasamy

---
CVE-2016-2174: Apache Ranger sql injection vulnerability
---
Severity: Normal
Vendor: The Apache Software Foundation
Versions Affected: All versions of Apache Ranger from 0.5.0 (up to 0.5.3)
Users Affected: All admin users of ranger policy admin tool
Description: SQL Injection vulnerability in Audit > Access tab. When the user 
clicks an element from policyId row of the list, there is a call made 
underneath 
with eventTime parameter which contains the vulnerability. Admin users can 
send some arbitrary sql code to be executed along with eventTime parameter
 using /service/plugins/policies/eventTime url.
Fix details: Replaced native queries with JPA named queries
Mitigation: Users should upgrade to 0.5.3 version of Apache Ranger with the fix.
Credit: Thanks to Mateusz Olejarka from SecuRing for reporting this issue.
---

Re: Problem setting up the SSL for Ranger usersync

2016-05-11 Thread Velmurugan Periasamy

Since the error is on usersync side, problem could be in accessing either 
usersync key store or trust store.Please verify the below.

1] usersync is using the right key store. Key password and Store password have 
to be the same.
2] usersync is using the right trust store. If not using the default truststore 
add -Djavax.net.ssl.trustStore= option in ranger-usersync-services.sh script
3] Ranger admin's cert is available in trust store used by usersync
4] Permissions are correct for keystore/truststore files


From: Lune Silver >
Reply-To: 
"user@ranger.incubator.apache.org" 
>
Date: Wednesday, May 11, 2016 at 11:59 AM
To: "user@ranger.incubator.apache.org" 
>
Subject: Problem setting up the SSL for Ranger usersync

hello !

I enabled the ssl for ranger admin successfully, but now I have a problem to 
set up the SSL for usersync.

I followed the following doc :
https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_Security_Guide/content/configure_ambari_ranger_ssl_self_signed_cert_usersync.html

But unfortunately, I still have one problem in the usersync log :
###
11 May 2016 14:20:29  INFO UnixAuthenticationService [main] - Starting User 
Sync Service!
11 May 2016 14:20:29  INFO UnixAuthenticationService [main] - Enabling Unix 
Auth Service!
11 May 2016 14:20:30  INFO UserGroupSync [UnixUserSyncThread] - initializing 
sink: org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder
11 May 2016 14:20:30 ERROR UserGroupSync [UnixUserSyncThread] - Failed to 
initialize UserGroup source/sink. Will retry after 6 milliseconds. Error 
details:
java.lang.RuntimeException: Unable to create SSLConext for communication to 
policy manager
at 
org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.getClient(PolicyMgrUserGroupBuilder.java:729)
at 
org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.buildGroupList(PolicyMgrUserGroupBuilder.java:335)
at 
org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.buildUserGroupInfo(PolicyMgrUserGroupBuilder.java:156)
at 
org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.init(PolicyMgrUserGroupBuilder.java:152)
at 
org.apache.ranger.usergroupsync.UserGroupSync.run(UserGroupSync.java:51)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Keystore was tampered with, or password was 
incorrect
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:772)
at 
sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:55)
at java.security.KeyStore.load(KeyStore.java:1214)
at 
org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder.getClient(PolicyMgrUserGroupBuilder.java:706)
... 5 more
Caused by: java.security.UnrecoverableKeyException: Password verification failed
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:770)
... 8 more
###

The error is clear enough, there is a problem with a password, but which one ?

I set up a password PWD1 for the keystore of ranger admin.
I used the same password PWD1 for the alias rangeradmin in the keystore of 
range admin.

I set up a different password PWD2 for the keystore of usersync.
I set up a different password PWD3 for the trustore of usersync.
I set up a specific password PWD4 for ranger local admin.
And I set up a different password for the Ranger Admin username for Ambari

Do you know which password is concerned by this error message please ?

BR.

Lune.

Re: Ranger does not take into account the logging properties from ambari

2016-05-10 Thread Velmurugan Periasamy

This is a bug that is fixed in Ranger 0.6.0 - 
https://issues.apache.org/jira/browse/RANGER-859

As a workaround, you can create symlinks under /var/log/ranger to point to 


From: Lune Silver >
Reply-To: 
"user@ranger.incubator.apache.org" 
>
Date: Tuesday, May 10, 2016 at 11:08 AM
To: "user@ranger.incubator.apache.org" 
>
Subject: Ranger does not take into account the logging properties from ambari

Hello !

I'm using HDP 2.3.2 with ambari 2.2.1 and I have a problem with ranger which 
does not take into account the logging dir I set in the conf.

I set the following properties in the tab advanced -> Advanced ranger-env :
- ranger_admin_log_dir = /ranger/admin
- ranger_usersync_log_dir = /ranger/usersync

And in the advanced -> Advanced ranger-ugsync-site :
- ranger.usersync.logdir = /ranger/usersync

But when I check in the server where I installed ranger, there is no logs in 
these folders.
The only logs are in the default paths : in /var/log/ranger/admin and 
/var/log/ranger/usersync.

Do you know why ranger keeps writing in the default paths please ?

BR.

Gwenael Le Barzic

Re: Informationn about properties of Ranger

2016-04-21 Thread Velmurugan Periasamy

Lune – unix auth service running as part of usersync is applicable only if unix 
authentication method is chosen in ranger admin. For LDAP/AD authentication 
methods, ranger admin will authenticate the user directly against LDAP/AD.

From: Lune Silver >
Reply-To: 
"user@ranger.incubator.apache.org" 
>
Date: Thursday, April 21, 2016 at 5:09 AM
To: "user@ranger.incubator.apache.org" 
>
Subject: Re: Informationn about properties of Ranger

Hello guys/

Selva : The service running within the usersync provides UNIX password based 
authentication for RANGER-ADMIN UI (using a JAAS  via SSL based connection to 
this service from Ranger Admin UI).

Lune :
So if I understand well, this port is used when a user tries to connect to 
Ranger UI Admin. When this occures, the following process happens :
1. Then Ranger Admin connects to usersync using this port.
2. In usersync, there is a service which will calls the password validator 
program.
Question :
Is it only for unix source or is it the same for ldap source ? If I have an 
ldap source, in usersync, will I have also a service in usersync which will 
calls the password validator program based on the records found in the LDAP ?

Best regards.

Lune.

On Thu, Apr 21, 2016 at 12:41 AM, Dilli Dorai 
> wrote:
Thanks Selva, Sailaja for the information.
Hoping the additional information helps the community.
Dilli

On Wed, Apr 20, 2016 at 2:50 PM, Sailaja Polavarapu 
> wrote:
Hi Dilli,
 You are right. I should have been more specific. This port is for 
UnixAuthenticationService which invokes the password validator program.

- Sailaja.

From: Dilli Dorai >
Reply-To: 
"user@ranger.incubator.apache.org" 
>
Date: Wednesday, April 20, 2016 at 2:25 PM
To: "user@ranger.incubator.apache.org" 
>
Subject: Re: Informationn about properties of Ranger

4. ranger.usersync.port

What is this port for exactly ?
[Sailaja]: This is the port where Usersync service listens on.

Sailaja,
May be I am misunderstanding or forgetting something here.

I thought
usersync makes calls to other services like LDAP, AD and Ranger admin.
Other services do not call usersync.

Could you confirm which services make call to this listen port?
Thanks
Dilli

On Wed, Apr 20, 2016 at 1:50 PM, Sailaja Polavarapu 
> wrote:
Hi Lune,
 Answers inline…
We have documentation on some of these properties available at:
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_Security_Guide/content/ranger_advanced_usersync_settings.html

Hope this helps.

Thanks,
Sailaja.

From: Lune Silver >
Reply-To: 
"user@ranger.incubator.apache.org" 
>
Date: Wednesday, April 20, 2016 at 8:39 AM
To: "user@ranger.incubator.apache.org" 
>
Subject: Informationn about properties of Ranger

Hello !

I contact you because I have some questions related to the following properties.
Hope you can help me.

Here are my questions :

1. ranger.usersync.passwordvalidator.path

The comment says that this is the path for a native prorgam to validate 
password. But in which situation ranger does validate password ?
[Sailaja]: In cases where ranger user sync talks to ranger admin, this program 
is called as part of HTTP basic auth filter. These cases include Usersync 
getting users & groups from ranger admin during initial startup, updating 
Ranger admin with the sync’d users and/or group information, etc… Default value 
for this property is "./native/credValidator.uexe” which as you said is a 
native program to validate password.

2. ranger.usersync.policymanager.maxrecordsperapicall

The help says that this is the maximum records returned by api call, but in 
which context ? Is it when a user uses the Ranger API to get the policies 
implemented in Ranger ?
[Sailaja]: Ranger Usersync gets all the users & groups from Ranger admin 
(stored in Ranger DB) during initial start up. Since these records can be many, 
Usersync retrieves these values in paged manner.  The value from this

Re: Need Help to choose Apache Ranger

2016-04-14 Thread Velmurugan Periasamy

Thanks Bosco for the explanation.

Adding Ranger Dev group.

From: Don Bosco Durai >
Reply-To: 
"user@ranger.incubator.apache.org" 
>
Date: Wednesday, April 13, 2016 at 2:30 PM
To: Rehan Ahmed Ch >
Cc: "user@ranger.incubator.apache.org" 
>
Subject: Re: Need Help to choose Apache Ranger

Copying Ranger user group...

Assuming you are referring for storing Audit records. Ranger currently supports 
4 options (Solr, HDFS, DB and Log4Appender). The framework is extensible and 
you can write your custom destination. We also had a Kafka destination in the 
previous release. However, in this release we are asking users to use 
Log4Appender. E.g. You can use Kafka log4j appender and send it to Kafka. 
Similarly I know users who use log4j TCP appender to send to their custom app.

Regardless which destination you use,  the following features are available:

  1.  Ranger plugins have in-built mechanism to send the audits reliably to the 
destination. If the destination is down, it will write to local file and resume 
when it is available.
  2.  If the destination is slower than the rate the audits are generated, then 
it will spool to local file and throttle the writing. But it will eventually it 
will send the audits (local spool size is configurable and dependent on 
availability of disk space)
  3.  If you are using components like Hbase, Kafka or Solr which generate way 
too many audit records, then it will summarize the audits at the source based 
on unique user+request and send the summarized audits.
  4.  It uses different queues and spool file for each destination. So If you 
have destinations which support different speed (e.g. Solr v/s HDFS), you will 
not lose audits and also the faster destinations will get audit records sooner.

Saying that, you need to decide what you want to do with Audits and pick the 
appropriate destinations. From Ranger Admin UI point of view, we will only 
support Solr and DB. And we will drop support for DB in the next release. So if 
you are not going to use Ranger Admin to view the audit records, then you don't 
have to sent to Solr also.

We choose Solr for the following reasons:

  1.  Can scale to billions of documents
  2.  Transparent and native support for sharding and replications
  3.  Easy to add columns and also auto creates missing columns (like no sql). 
In RDBMS, alter table with large amount of data just doesn't work
  4.  Great searching capabilities
  5.  Native dashboard features like faceting, etc.
  6.  Easy to write your own custom application on top of it
  7.  Apache open source

In other words, Solr is a great product on its own :-)

Thanks

Bosco

From: Rehan Ahmed Ch >
Date: Monday, April 11, 2016 at 2:10 AM
To: Don Bosco Durai >
Subject: Re: Need Help to choose Apache Ranger

Hi Don,

Can you please help to have some alternative of "Solr" in case we have opt to 
implement Ranger?

On Sun, Apr 10, 2016 at 12:09 AM, Rehan Ahmed Ch 
> wrote:

Thank you very much dear Don Bosco for your outright response. Much obliged.

I will let you know in case if any your kind guidance will require. Thank you 
so much.

--
Truly,
Rehan Ahmed

--
Truly,
Rehan Ahmed

Re: [Discuss] Phasing support for DB audit

2016-03-29 Thread Velmurugan Periasamy

https://issues.apache.org/jira/browse/RANGER-271 addressed the migration
utility.


https://issues.apache.org/jira/browse/RANGER-900 is created for removing
DB audit support. 


On 3/23/16, 1:52 AM, "Don Bosco Durai"  wrote:

>+1
>
>
>I suggest that we provide utility to migrate audits from DB to Solr. We
>also need to ensure that upgrade instructions are clear.
>
>Thanks
>
>Bosco
>
>
>On 3/8/16, 12:19 PM, "Balaji Ganesan"  wrote:
>
>>All, any concerns with the proposed approach?
>>
>>On Wed, Mar 2, 2016 at 12:25 PM, Balaji Ganesan 
>>wrote:
>>
>>> All,
>>>
>>> As I had spoken with some of you offline, I would want to propose
>>>phasing
>>> out support for storing audit in database as part of next major
>>>release.
>>> Currently Ranger supports the following methods for storing audit data
>>>
>>> 1. Database
>>> 2. HDFS
>>> 3. Solr
>>> 4. Log4j custom appender
>>>
>>> The Ranger UI can currently read from both the database and Solr.
>>>Moving
>>> forward, I would propose to remove database support for audit and keep
>>>only
>>> Solr as the source of audit for Ranger UI. Users can still send their
>>>audit
>>> to HDFS as well as Log4j
>>>
>>> The proposal would involve removing audit to DB related configuration
>>>as
>>> well removal of any code related to storing audit in the database.
>>>
>>> I wanted to start this thread to discuss this proposal and understand
>>> concerns/questions.
>>>
>>> -Balaji
>>>
>
>

CVE update (CVE-2016-0735) - Fixed in Ranger 0.5.2

2016-03-28 Thread Velmurugan Periasamy

Hello:

Here¹s a CVE update for Ranger 0.5.2 release. Please see below details.

Thank you,
Velmurugan Periasamy

--
CVE-2016-0735: Ranger policy excludes flags processing
--
Severity: Important
Vendor: The Apache Software Foundation
Versions Affected: 0.5.0/0.5.1 versions of Apache Ranger
Users affected: All users that use Ranger to authorize HBase, Hive, and
Knox.
Description: In some cases, presence of an exclude policy at a
resource-level can give the user access at its parent resource-level. For
example, if a hive policy excludes access for a user to a particular column,
then such a user would be able to alter the name of that table. Only a user
who has access at the table level should be able to do so. Due to this bug
however, the user is able to do the operation when an exclude policy is
present at the column-level for that table.
Mitigation: Users should upgrade to Ranger 0.5.2 version
--

Re: [Discuss] Phasing support for DB audit

2016-03-22 Thread Velmurugan Periasamy

Hi Balaji:

I agree storing audit data in DB is not scalable for long term data.

What would be the path forward for users who are already using DB to store
audit data and managing the DB themselves?

Adding user@ranger list.

Thank you,
Vel

On 3/8/16, 3:19 PM, "Balaji Ganesan"  wrote:

>All, any concerns with the proposed approach?
>
>On Wed, Mar 2, 2016 at 12:25 PM, Balaji Ganesan 
>wrote:
>
>> All,
>>
>> As I had spoken with some of you offline, I would want to propose
>>phasing
>> out support for storing audit in database as part of next major release.
>> Currently Ranger supports the following methods for storing audit data
>>
>> 1. Database
>> 2. HDFS
>> 3. Solr
>> 4. Log4j custom appender
>>
>> The Ranger UI can currently read from both the database and Solr. Moving
>> forward, I would propose to remove database support for audit and keep
>>only
>> Solr as the source of audit for Ranger UI. Users can still send their
>>audit
>> to HDFS as well as Log4j
>>
>> The proposal would involve removing audit to DB related configuration as
>> well removal of any code related to storing audit in the database.
>>
>> I wanted to start this thread to discuss this proposal and understand
>> concerns/questions.
>>
>> -Balaji
>>

Re: ranger-admin setup failed

2015-11-28 Thread Velmurugan Periasamy

In that case, these properties should have been already there. Not sure if you 
are using an old version of install.properties. Can you add the below lines in 
install.properties and try again?

sqlanywhere_core_file=db/sqlanywhere/xa_core_db_sqlanywhere.sql
sqlanywhere_audit_file=db/sqlanywhere/xa_audit_db_sqlanywhere.sql

From: Hafiz Mujadid <hafizmujadi...@gmail.com<mailto:hafizmujadi...@gmail.com>>
Reply-To: 
"user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" 
<user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>>
Date: Saturday, November 28, 2015 at 8:34 AM
To: "user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" 
<user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>>
Subject: Re: ranger-admin setup failed

i am using apache ranger main branch code

On Sat, Nov 28, 2015 at 6:33 PM, Velmurugan Periasamy 
<vperias...@hortonworks.com<mailto:vperias...@hortonworks.com>> wrote:
What version of ranger you are setting up?

Can you add the below lines in install.properties and try again?

sqlanywhere_core_file=db/sqlanywhere/xa_core_db_sqlanywhere.sql
sqlanywhere_audit_file=db/sqlanywhere/xa_audit_db_sqlanywhere.sql

From: Hafiz Mujadid <hafizmujadi...@gmail.com<mailto:hafizmujadi...@gmail.com>>
Reply-To: 
"user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" 
<user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>>
Date: Saturday, November 28, 2015 at 8:04 AM
To: "user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" 
<user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>>
Subject: ranger-admin setup failed

Hi!

I am running ./setup.sh script for ranger-admin but it is failing with 
following exception

2015-11-28 17:59:53,880  [I] Privileges granted to 'rangeradmin' on 
'ranger_audit'
2015-11-28 17:59:53,880  [I] -- Ranger Policy Manager DB and User 
Creation Process Completed..  --
2015-11-28 17:59:53,920  [I] DB FLAVOR :MYSQL
Traceback (most recent call last):
  File "db_setup.py", line 1725, in 
main(sys.argv)
  File "db_setup.py", line 1587, in main
sqlanywhere_core_file = globalDict['sqlanywhere_core_file']
KeyError: 'sqlanywhere_core_file'
2015-11-28 17:59:53,925  [E] DB schema setup failed! Please contact 
Administrator.


Any suggestion?

thanks



--
Regards: HAFIZ MUJADID

Re: ranger-admin setup failed

2015-11-28 Thread Velmurugan Periasamy

Patch is committed. Could you please try again?

From: Hafiz Mujadid <hafizmujadi...@gmail.com<mailto:hafizmujadi...@gmail.com>>
Reply-To: 
"user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" 
<user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>>
Date: Saturday, November 28, 2015 at 9:17 AM
To: "user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" 
<user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>>
Subject: Re: ranger-admin setup failed

thanks, I am waiting for the solution

On Sat, Nov 28, 2015 at 7:16 PM, Velmurugan Periasamy 
<vperias...@hortonworks.com<mailto:vperias...@hortonworks.com>> wrote:
Thanks for reporting the issue Hafiz. This issue needs to be fixed. I have 
created https://issues.apache.org/jira/browse/RANGER-751

From: Hafiz Mujadid <hafizmujadi...@gmail.com<mailto:hafizmujadi...@gmail.com>>
Reply-To: 
"user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" 
<user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>>
Date: Saturday, November 28, 2015 at 8:47 AM

To: "user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" 
<user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>>
Subject: Re: ranger-admin setup failed

Added these lines and got following error

Traceback (most recent call last):
  File "update_property.py", line 40, in 

write_properties_to_xml(ranger_admin_site_xml_path,parameter_name,parameter_value)
  File "update_property.py", line 21, in write_properties_to_xml
if(os.path.isfile(xml_path)):
  File "/usr/lib/python2.7/genericpath.py", line 29, in isfile
st = os.stat(path)
TypeError: coercing to Unicode: need string or buffer, NoneType found
2015-11-28 18:46:39,743  [E] Update property failed for:

On Sat, Nov 28, 2015 at 6:36 PM, Velmurugan Periasamy 
<vperias...@hortonworks.com<mailto:vperias...@hortonworks.com>> wrote:
In that case, these properties should have been already there. Not sure if you 
are using an old version of install.properties. Can you add the below lines in 
install.properties and try again?

sqlanywhere_core_file=db/sqlanywhere/xa_core_db_sqlanywhere.sql
sqlanywhere_audit_file=db/sqlanywhere/xa_audit_db_sqlanywhere.sql

From: Hafiz Mujadid <hafizmujadi...@gmail.com<mailto:hafizmujadi...@gmail.com>>
Reply-To: 
"user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" 
<user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>>
Date: Saturday, November 28, 2015 at 8:34 AM
To: "user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" 
<user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>>
Subject: Re: ranger-admin setup failed

i am using apache ranger main branch code

On Sat, Nov 28, 2015 at 6:33 PM, Velmurugan Periasamy 
<vperias...@hortonworks.com<mailto:vperias...@hortonworks.com>> wrote:
What version of ranger you are setting up?

Can you add the below lines in install.properties and try again?

sqlanywhere_core_file=db/sqlanywhere/xa_core_db_sqlanywhere.sql
sqlanywhere_audit_file=db/sqlanywhere/xa_audit_db_sqlanywhere.sql

From: Hafiz Mujadid <hafizmujadi...@gmail.com<mailto:hafizmujadi...@gmail.com>>
Reply-To: 
"user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" 
<user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>>
Date: Saturday, November 28, 2015 at 8:04 AM
To: "user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" 
<user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>>
Subject: ranger-admin setup failed

Hi!

I am running ./setup.sh script for ranger-admin but it is failing with 
following exception

2015-11-28 17:59:53,880  [I] Privileges granted to 'rangeradmin' on 
'ranger_audit'
2015-11-28 17:59:53,880  [I] -- Ranger Policy Manager DB and User 
Creation Process Completed..  --
2015-11-28 17:59:53,920  [I] DB FLAVOR :MYSQL
Traceback (most recent call last):
  File "db_setup.py", line 1725, in 
main(sys.argv)
  File "db_setup.py", line 1587, in main
sqlanywhere_core_file = globalDict['sqlanywhere_core_file']
KeyError: 'sqlanywhere_core_file'
2015-11-28 17:59:53,925  [E] DB schema setup failed! Please contact 
Administrator.

Any suggestion?

thanks

--
Regards: HAFIZ MUJADID

--
Regards: HAFIZ MUJADID

--
Regards: HAFIZ MUJADID

Re: Exception while creating encryption zone

Re: Exception while creating encryption zone

Re: Need help in integrating apache-ranger-incubating-0.6.1

Re: Exception while creating encryption zone

CVE update (CVE-2016-5395) - Fixed in Apache Ranger 0.6.1

Re: Ranger-0.6 HDFS authentication failed in secure mode

Re: HDFS Plugin - Unable to get listing of files for directory [/] from Hadoop environment

Re: HDFS Plugin - Unable to get listing of files for directory [/] from Hadoop environment

CVE update (CVE-2016-2174) - Fixed in Ranger 0.5.3

Re: Problem setting up the SSL for Ranger usersync

Re: Ranger does not take into account the logging properties from ambari

Re: Informationn about properties of Ranger

Re: Need Help to choose Apache Ranger

Re: [Discuss] Phasing support for DB audit

CVE update (CVE-2016-0735) - Fixed in Ranger 0.5.2

Re: [Discuss] Phasing support for DB audit

Re: ranger-admin setup failed

Re: ranger-admin setup failed

18 matches

Site Navigation

Mail list logo

Footer information