[jira] [Commented] (RANGER-3271) Ranger Knox Plugin Unable to Write Knox Audits to HDFS
[
https://issues.apache.org/jira/browse/RANGER-3271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340535#comment-17340535
]
Ramesh Mani commented on RANGER-3271:
-
[~venkatas] Could you please put knox in debug and restart know and attache the
gateway.log here?
Also could you please check in the classpath if other version of
hadoop-hdfs.jar is there?
> Ranger Knox Plugin Unable to Write Knox Audits to HDFS
> ---
>
> Key: RANGER-3271
> URL: https://issues.apache.org/jira/browse/RANGER-3271
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 2.1.0
> Environment: HADOOP : 3.3.0
> KNOX : 1.4.0
> RANGER : 2.1.0
>Reporter: Venkat A
>Priority: Blocker
>
> I see following error when Knox audits being written to HDFS after
> Ranger-Knox plugin enabled.
>
> 3322021-05-01 15:33:34,435 INFO destination.HDFSAuditDestination: Returning
> HDFS Filesystem Config: Configuration: core-default.xml, core-site.xml
> 3322021-05-01 15:33:34,435 INFO destination.HDFSAuditDestination: Returning
> HDFS Filesystem Config: Configuration: core-default.xml, core-site.xml
> 3182021-05-01 15:33:34,439 ERROR provider.BaseAuditHandler: Error writing to
> log file.
> 349{color:#FF}*org.apache.hadoop.fs.UnsupportedFileSystemException: No
> FileSystem for scheme "hdfs"*{color} at
> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3332) at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3352) at
> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124) at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403) at
> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371) at
> org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477) at
> org.apache.ranger.audit.destination.HDFSAuditDestination.getLogFileStream(HDFSAuditDestination.java:277)
> at
> org.apache.ranger.audit.destination.HDFSAuditDestination.access$000(HDFSAuditDestination.java:44)
> at
> org.apache.ranger.audit.destination.HDFSAuditDestination$1.run(HDFSAuditDestination.java:157)
> at
> org.apache.ranger.audit.destination.HDFSAuditDestination$1.run(HDFSAuditDestination.java:154)
> at java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.ranger.audit.provider.MiscUtil.executePrivilegedAction(MiscUtil.java:529)
> at
> org.apache.ranger.audit.destination.HDFSAuditDestination.logJSON(HDFSAuditDestination.java:154)
> at
> org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:879)
> at
> org.apache.ranger.audit.queue.AuditFileSpool.runLogAudit(AuditFileSpool.java:827)
> at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:757)
> at java.lang.Thread.run(Thread.java:748)2021-05-01 15:33:34,439 INFO
> destination.HDFSAuditDestination: Flushing HDFS audit. Event Size:1
> 1802021-05-01 15:33:34,439 ERROR queue.AuditFileSpool: Error sending logs to
> consumer. provider=knox.async.multi_dest.batch,
> consumer=knox.async.multi_dest.batch.hdfs 7092021-05-01 15:33:34,440 INFO
> queue.AuditFileSpool: Destination is down. sleeping for 3 milli seconds.
> indexQueue=0, queueName=knox.async.multi_dest.batch,
> consumer=knox.async.multi_dest.batch.hdfs
>
>
> Tried lot of options to avoid above error. Not sure if it is a bug or some
> sort of Compatibility issue.
> Environment :
> HADOOP : 3.3.0
> KNOX : 1.4.0
> RANGER : 2.1.0
> NOTE:
> Knox able to write audits if i give local path to store audits instead of
> HDFS File System.
> Appreciate your help on this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-3271) Ranger Knox Plugin Unable to Write Knox Audits to HDFS
[
https://issues.apache.org/jira/browse/RANGER-3271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340451#comment-17340451
]
Venkat A commented on RANGER-3271:
--
Seems like someone else have raised similar bug
https://issues.apache.org/jira/browse/RANGER-2784
Not Sure if they have got any solution.
> Ranger Knox Plugin Unable to Write Knox Audits to HDFS
> ---
>
> Key: RANGER-3271
> URL: https://issues.apache.org/jira/browse/RANGER-3271
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 2.1.0
> Environment: HADOOP : 3.3.0
> KNOX : 1.4.0
> RANGER : 2.1.0
>Reporter: Venkat A
>Priority: Blocker
>
> I see following error when Knox audits being written to HDFS after
> Ranger-Knox plugin enabled.
>
> 3322021-05-01 15:33:34,435 INFO destination.HDFSAuditDestination: Returning
> HDFS Filesystem Config: Configuration: core-default.xml, core-site.xml
> 3322021-05-01 15:33:34,435 INFO destination.HDFSAuditDestination: Returning
> HDFS Filesystem Config: Configuration: core-default.xml, core-site.xml
> 3182021-05-01 15:33:34,439 ERROR provider.BaseAuditHandler: Error writing to
> log file.
> 349{color:#FF}*org.apache.hadoop.fs.UnsupportedFileSystemException: No
> FileSystem for scheme "hdfs"*{color} at
> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3332) at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3352) at
> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124) at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403) at
> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371) at
> org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477) at
> org.apache.ranger.audit.destination.HDFSAuditDestination.getLogFileStream(HDFSAuditDestination.java:277)
> at
> org.apache.ranger.audit.destination.HDFSAuditDestination.access$000(HDFSAuditDestination.java:44)
> at
> org.apache.ranger.audit.destination.HDFSAuditDestination$1.run(HDFSAuditDestination.java:157)
> at
> org.apache.ranger.audit.destination.HDFSAuditDestination$1.run(HDFSAuditDestination.java:154)
> at java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.ranger.audit.provider.MiscUtil.executePrivilegedAction(MiscUtil.java:529)
> at
> org.apache.ranger.audit.destination.HDFSAuditDestination.logJSON(HDFSAuditDestination.java:154)
> at
> org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:879)
> at
> org.apache.ranger.audit.queue.AuditFileSpool.runLogAudit(AuditFileSpool.java:827)
> at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:757)
> at java.lang.Thread.run(Thread.java:748)2021-05-01 15:33:34,439 INFO
> destination.HDFSAuditDestination: Flushing HDFS audit. Event Size:1
> 1802021-05-01 15:33:34,439 ERROR queue.AuditFileSpool: Error sending logs to
> consumer. provider=knox.async.multi_dest.batch,
> consumer=knox.async.multi_dest.batch.hdfs 7092021-05-01 15:33:34,440 INFO
> queue.AuditFileSpool: Destination is down. sleeping for 3 milli seconds.
> indexQueue=0, queueName=knox.async.multi_dest.batch,
> consumer=knox.async.multi_dest.batch.hdfs
>
>
> Tried lot of options to avoid above error. Not sure if it is a bug or some
> sort of Compatibility issue.
> Environment :
> HADOOP : 3.3.0
> KNOX : 1.4.0
> RANGER : 2.1.0
> NOTE:
> Knox able to write audits if i give local path to store audits instead of
> HDFS File System.
> Appreciate your help on this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-2999) Ranger build is failing if you do not have already downloaded jar under ~/.m2 for calcite-linq4j-1.16.0-kylin-r2
[
https://issues.apache.org/jira/browse/RANGER-2999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340403#comment-17340403
]
Taylor Conkle commented on RANGER-2999:
---
[~sneethiraj], I have found a fix for this. You can no longer authenticate to
repo.spring.io @ /libs-snapshot. You will need to use /snapshot or /milestone.
I implemented the below changes in my settings.xml for Maven and no longer
receive the error.
You can view the notes from Spring in [this link.|[Notice of Permissions
Changes to repo.spring.io, Fall and Winter,
2020|https://spring.io/blog/2020/10/29/notice-of-permissions-changes-to-repo-spring-io-fall-and-winter-2020]]
{code:java}
springRelease
spring-repo
Spring Repository
https://repo.spring.io/milestone
springLibs
spring-repo
Spring Repository
https://repo.spring.io/snapshot
springRelease
springLibs
{code}
> Ranger build is failing if you do not have already downloaded jar under ~/.m2
> for calcite-linq4j-1.16.0-kylin-r2
>
>
> Key: RANGER-2999
> URL: https://issues.apache.org/jira/browse/RANGER-2999
> Project: Ranger
> Issue Type: Bug
> Components: build-infra
>Reporter: Selvamohan Neethiraj
>Priority: Critical
>
> 21:07:17 [ERROR] Failed to execute goal on project ranger-kylin-plugin: Could
> not resolve dependencies for project
> org.apache.ranger:ranger-kylin-plugin:jar:3.0.0-SNAPSHOT: Failed to collect
> dependencies at org.apache.kylin:kylin-server-base:jar:2.6.6 ->
> org.apache.kylin:kylin-query:jar:2.6.6 ->
> org.apache.kylin:kylin-datasource-sdk:jar:2.6.6 ->
> org.apache.calcite:calcite-linq4j:jar:1.16.0-kylin-r2: Failed to read
> artifact descriptor for
> org.apache.calcite:calcite-linq4j:jar:1.16.0-kylin-r2: Could not transfer
> artifact org.apache.calcite:calcite-linq4j:pom:1.16.0-kylin-r2 from/to
> spring-snapshots (http://repo.spring.io/libs-snapshot): Authorization failed
> for
> http://repo.spring.io/libs-snapshot/org/apache/calcite/calcite-linq4j/1.16.0-kylin-r2/calcite-linq4j-1.16.0-kylin-r2.pom
> 403 Forbidden -> [Help 1]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (RANGER-2999) Ranger build is failing if you do not have already downloaded jar under ~/.m2 for calcite-linq4j-1.16.0-kylin-r2
[
https://issues.apache.org/jira/browse/RANGER-2999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340403#comment-17340403
]
Taylor Conkle edited comment on RANGER-2999 at 5/6/21, 7:55 PM:
[~sneethiraj], I have found a fix for this. You can no longer authenticate to
repo.spring.io @ /libs-snapshot. You will need to use /snapshot or /milestone.
I implemented the below changes in my settings.xml for Maven and no longer
receive the error.
You can view the notes from Spring: [Notice of Permissions Changes to
repo.spring.io, Fall and Winter,
2020|https://spring.io/blog/2020/10/29/notice-of-permissions-changes-to-repo-spring-io-fall-and-winter-2020]
{code:java}
springRelease
spring-repo
Spring Repository
https://repo.spring.io/milestone
springLibs
spring-repo
Spring Repository
https://repo.spring.io/snapshot
springRelease
springLibs
{code}
was (Author: taylor-codes-01):
[~sneethiraj], I have found a fix for this. You can no longer authenticate to
repo.spring.io @ /libs-snapshot. You will need to use /snapshot or /milestone.
I implemented the below changes in my settings.xml for Maven and no longer
receive the error.
You can view the notes from Spring in [this link.|[Notice of Permissions
Changes to repo.spring.io, Fall and Winter,
2020|https://spring.io/blog/2020/10/29/notice-of-permissions-changes-to-repo-spring-io-fall-and-winter-2020]]
{code:java}
springRelease
spring-repo
Spring Repository
https://repo.spring.io/milestone
springLibs
spring-repo
Spring Repository
https://repo.spring.io/snapshot
springRelease
springLibs
{code}
> Ranger build is failing if you do not have already downloaded jar under ~/.m2
> for calcite-linq4j-1.16.0-kylin-r2
>
>
> Key: RANGER-2999
> URL: https://issues.apache.org/jira/browse/RANGER-2999
> Project: Ranger
> Issue Type: Bug
> Components: build-infra
>Reporter: Selvamohan Neethiraj
>Priority: Critical
>
> 21:07:17 [ERROR] Failed to execute goal on project ranger-kylin-plugin: Could
> not resolve dependencies for project
> org.apache.ranger:ranger-kylin-plugin:jar:3.0.0-SNAPSHOT: Failed to collect
> dependencies at org.apache.kylin:kylin-server-base:jar:2.6.6 ->
> org.apache.kylin:kylin-query:jar:2.6.6 ->
> org.apache.kylin:kylin-datasource-sdk:jar:2.6.6 ->
> org.apache.calcite:calcite-linq4j:jar:1.16.0-kylin-r2: Failed to read
> artifact descriptor for
> org.apache.calcite:calcite-linq4j:jar:1.16.0-kylin-r2: Could not transfer
> artifact org.apache.calcite:calcite-linq4j:pom:1.16.0-kylin-r2 from/to
> spring-snapshots (http://repo.spring.io/libs-snapshot): Authorization failed
> for
> http://repo.spring.io/libs-snapshot/org/apache/calcite/calcite-linq4j/1.16.0-kylin-r2/calcite-linq4j-1.16.0-kylin-r2.pom
> 403 Forbidden -> [Help 1]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Review Request 73338: RANGER-3275 : Need to update solr-config.xml in the ranger-audits collection config-set
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73338/#review222952 --- Ship it! Ship It! - Ramesh Mani On May 6, 2021, 4:43 p.m., Vishal Suvagia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73338/ > --- > > (Updated May 6, 2021, 4:43 p.m.) > > > Review request for ranger, Ankita Sinha, Dhaval Shah, Dineshkumar Yadav, > Gautam Borad, Jayendra Parab, Kishor Gollapalliwar, Abhay Kulkarni, Madhan > Neethiraj, Mehul Parikh, Pradeep Agrawal, Ramesh Mani, Sailaja Polavarapu, > and Velmurugan Periasamy. > > > Bugs: RANGER-3275 > https://issues.apache.org/jira/browse/RANGER-3275 > > > Repository: ranger > > > Description > --- > > The solrconfig.xml in the config-set for Ranger needs to be updated in > correct order to use the ttl configuration properly, without which the > documents do not contain the ttl and will not expire or purge out even after > the ttl is set. > > > Diffs > - > > security-admin/contrib/solr_for_audit_setup/conf/solrconfig.xml > 2216f665fd197e066585fe527645b218ab25a221 > > > Diff: https://reviews.apache.org/r/73338/diff/1/ > > > Testing > --- > > Tested the changes locally, ttl is getting applied to the created documents. > > > Thanks, > > Vishal Suvagia > >
[jira] [Updated] (RANGER-3275) Need to update solr-config.xml in the ranger-audits collection config-set
[
https://issues.apache.org/jira/browse/RANGER-3275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vishal Suvagia updated RANGER-3275:
---
Attachment: RANGER-3275.patch
> Need to update solr-config.xml in the ranger-audits collection config-set
> -
>
> Key: RANGER-3275
> URL: https://issues.apache.org/jira/browse/RANGER-3275
> Project: Ranger
> Issue Type: Bug
> Components: admin, Ranger
>Affects Versions: 3.0.0
>Reporter: Vishal Suvagia
>Assignee: Vishal Suvagia
>Priority: Major
> Attachments: RANGER-3275.patch
>
>
> {color:#172b4d}The solrconfig.xml in the config-set for Ranger needs to be
> updated in correct order to use the ttl configuration properly, without which
> the documents will not expire or purge out even after the ttl is set.
> {color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Review Request 73338: RANGER-3275 : Need to update solr-config.xml in the ranger-audits collection config-set
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73338/ --- (Updated May 6, 2021, 4:43 p.m.) Review request for ranger, Ankita Sinha, Dhaval Shah, Dineshkumar Yadav, Gautam Borad, Jayendra Parab, Kishor Gollapalliwar, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Pradeep Agrawal, Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy. Bugs: RANGER-3275 https://issues.apache.org/jira/browse/RANGER-3275 Repository: ranger Description --- The solrconfig.xml in the config-set for Ranger needs to be updated in correct order to use the ttl configuration properly, without which the documents do not contain the ttl and will not expire or purge out even after the ttl is set. Diffs - security-admin/contrib/solr_for_audit_setup/conf/solrconfig.xml 2216f665fd197e066585fe527645b218ab25a221 Diff: https://reviews.apache.org/r/73338/diff/1/ Testing --- Tested the changes locally, ttl is getting applied to the created documents. Thanks, Vishal Suvagia
[jira] [Commented] (RANGER-3271) Ranger Knox Plugin Unable to Write Knox Audits to HDFS
[
https://issues.apache.org/jira/browse/RANGER-3271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340244#comment-17340244
]
Venkat A commented on RANGER-3271:
--
All,
Are there any specific Versions of Hadoop support with Ranger and Knox Plugins
? I'm using Hadoop 3.3.0.
> Ranger Knox Plugin Unable to Write Knox Audits to HDFS
> ---
>
> Key: RANGER-3271
> URL: https://issues.apache.org/jira/browse/RANGER-3271
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 2.1.0
> Environment: HADOOP : 3.3.0
> KNOX : 1.4.0
> RANGER : 2.1.0
>Reporter: Venkat A
>Priority: Blocker
>
> I see following error when Knox audits being written to HDFS after
> Ranger-Knox plugin enabled.
>
> 3322021-05-01 15:33:34,435 INFO destination.HDFSAuditDestination: Returning
> HDFS Filesystem Config: Configuration: core-default.xml, core-site.xml
> 3322021-05-01 15:33:34,435 INFO destination.HDFSAuditDestination: Returning
> HDFS Filesystem Config: Configuration: core-default.xml, core-site.xml
> 3182021-05-01 15:33:34,439 ERROR provider.BaseAuditHandler: Error writing to
> log file.
> 349{color:#FF}*org.apache.hadoop.fs.UnsupportedFileSystemException: No
> FileSystem for scheme "hdfs"*{color} at
> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3332) at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3352) at
> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124) at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403) at
> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371) at
> org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477) at
> org.apache.ranger.audit.destination.HDFSAuditDestination.getLogFileStream(HDFSAuditDestination.java:277)
> at
> org.apache.ranger.audit.destination.HDFSAuditDestination.access$000(HDFSAuditDestination.java:44)
> at
> org.apache.ranger.audit.destination.HDFSAuditDestination$1.run(HDFSAuditDestination.java:157)
> at
> org.apache.ranger.audit.destination.HDFSAuditDestination$1.run(HDFSAuditDestination.java:154)
> at java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at
> org.apache.ranger.audit.provider.MiscUtil.executePrivilegedAction(MiscUtil.java:529)
> at
> org.apache.ranger.audit.destination.HDFSAuditDestination.logJSON(HDFSAuditDestination.java:154)
> at
> org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:879)
> at
> org.apache.ranger.audit.queue.AuditFileSpool.runLogAudit(AuditFileSpool.java:827)
> at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:757)
> at java.lang.Thread.run(Thread.java:748)2021-05-01 15:33:34,439 INFO
> destination.HDFSAuditDestination: Flushing HDFS audit. Event Size:1
> 1802021-05-01 15:33:34,439 ERROR queue.AuditFileSpool: Error sending logs to
> consumer. provider=knox.async.multi_dest.batch,
> consumer=knox.async.multi_dest.batch.hdfs 7092021-05-01 15:33:34,440 INFO
> queue.AuditFileSpool: Destination is down. sleeping for 3 milli seconds.
> indexQueue=0, queueName=knox.async.multi_dest.batch,
> consumer=knox.async.multi_dest.batch.hdfs
>
>
> Tried lot of options to avoid above error. Not sure if it is a bug or some
> sort of Compatibility issue.
> Environment :
> HADOOP : 3.3.0
> KNOX : 1.4.0
> RANGER : 2.1.0
> NOTE:
> Knox able to write audits if i give local path to store audits instead of
> HDFS File System.
> Appreciate your help on this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
