[jira] [Updated] (RANGER-3089) Ranger Setup issue fix
[
https://issues.apache.org/jira/browse/RANGER-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kishor Gollapalliwar updated RANGER-3089:
-
Description:
Ranger setup is failing with below error:
{code:java}
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/commons/compress/archivers/tar/TarArchiveInputStreamException in
thread "main" java.lang.NoClassDefFoundError:
org/apache/commons/compress/archivers/tar/TarArchiveInputStream at
org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfoByNonNativeIO(RawLocalFileSystem.java:753)
at
org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:744)
at
org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:705)
at
org.apache.hadoop.security.alias.JavaKeyStoreProvider.stashOriginalFilePermissions(JavaKeyStoreProvider.java:83)
at
org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.locateKeystore(AbstractJavaKeyStoreProvider.java:320)
at
org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.(AbstractJavaKeyStoreProvider.java:86)
at
org.apache.hadoop.security.alias.JavaKeyStoreProvider.(JavaKeyStoreProvider.java:49)
at
org.apache.hadoop.security.alias.JavaKeyStoreProvider.(JavaKeyStoreProvider.java:41)
at
org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory.createProvider(JavaKeyStoreProvider.java:100)
at
org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:73)
at
org.apache.ranger.credentialapi.CredentialReader.getDecryptedString(CredentialReader.java:59)
at
org.apache.ranger.common.PropertiesUtil.processProperties(PropertiesUtil.java:136)
at
org.springframework.beans.factory.config.PropertyResourceConfigurer.postProcessBeanFactory(PropertyResourceConfigurer.java:86)
at
org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:283)
at
org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:163)
at
org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:687)
at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:524)
at
org.springframework.context.support.ClassPathXmlApplicationContext.(ClassPathXmlApplicationContext.java:139)
at
org.springframework.context.support.ClassPathXmlApplicationContext.(ClassPathXmlApplicationContext.java:93)
at org.apache.ranger.util.CLIUtil.init(CLIUtil.java:54) at
org.apache.ranger.util.CLIUtil.getBean(CLIUtil.java:62) at
org.apache.ranger.patch.cliutil.ChangePasswordUtil.main(ChangePasswordUtil.java:55)Caused
by: java.lang.ClassNotFoundException:
org.apache.commons.compress.archivers.tar.TarArchiveInputStream at
java.net.URLClassLoader.findClass(URLClassLoader.java:382) at
java.lang.ClassLoader.loadClass(ClassLoader.java:418) at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at
java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 22 more{code}
*Steps*:
# Get latest code
# Build Ranger using maven command
# copy *ranger-3.0.0-SNAPSHOT-admin.tar.gz* from target director to location
where you want to setup ranger
# untar *ranger-3.0.0-SNAPSHOT-admin.tar.gz* and move to
*ranger-3.0.0-SNAPSHOT-admin* directory
# configure *install.properties* as per your requirements
# run *setup.sh*
was:
Ranger setup is failing with
*Steps*:
# Build Ranger using maven command
# copy *ranger-3.0.0-SNAPSHOT-admin.tar.gz* from target director to location
where you want to setup ranger
# untar *ranger-3.0.0-SNAPSHOT-admin.tar.gz* and move to
*ranger-3.0.0-SNAPSHOT-admin* directory
# configure *install.properties* as per your requirements
# run *setup.sh*
> Ranger Setup issue fix
> --
>
> Key: RANGER-3089
> URL: https://issues.apache.org/jira/browse/RANGER-3089
> Project: Ranger
> Issue Type: Improvement
> Components: Ranger
>Affects Versions: 3.0.0
>Reporter: Kishor Gollapalliwar
>Assignee: Kishor Gollapalliwar
>Priority: Major
>
> Ranger setup is failing with below error:
> {code:java}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/commons/compress/archivers/tar/TarArchiveInputStreamException in
> thread "main" java.lang.NoClassDefFoundError:
> org/apache/commons/compress/archivers/tar/TarArchiveInputStream at
> org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfoByNonNativeIO(RawLocalFileSystem.java:753)
> at
>
[jira] [Updated] (RANGER-3089) Ranger Setup issue fix
[ https://issues.apache.org/jira/browse/RANGER-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kishor Gollapalliwar updated RANGER-3089: - Description: Ranger setup is failing with *Steps*: # Build Ranger using maven command # copy *ranger-3.0.0-SNAPSHOT-admin.tar.gz* from target director to location where you want to setup ranger # untar *ranger-3.0.0-SNAPSHOT-admin.tar.gz* and move to *ranger-3.0.0-SNAPSHOT-admin* directory # configure *install.properties* as per your requirements # run *setup.sh* was:Fix ranger setup issues > Ranger Setup issue fix > -- > > Key: RANGER-3089 > URL: https://issues.apache.org/jira/browse/RANGER-3089 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 3.0.0 >Reporter: Kishor Gollapalliwar >Assignee: Kishor Gollapalliwar >Priority: Major > > Ranger setup is failing with > *Steps*: > # Build Ranger using maven command > # copy *ranger-3.0.0-SNAPSHOT-admin.tar.gz* from target director to > location where you want to setup ranger > # untar *ranger-3.0.0-SNAPSHOT-admin.tar.gz* and move to > *ranger-3.0.0-SNAPSHOT-admin* directory > # configure *install.properties* as per your requirements > # run *setup.sh* > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (RANGER-3089) Ranger Setup issue fix
[ https://issues.apache.org/jira/browse/RANGER-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kishor Gollapalliwar reassigned RANGER-3089: Assignee: Kishor Gollapalliwar > Ranger Setup issue fix > -- > > Key: RANGER-3089 > URL: https://issues.apache.org/jira/browse/RANGER-3089 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 3.0.0 >Reporter: Kishor Gollapalliwar >Assignee: Kishor Gollapalliwar >Priority: Major > > Fix ranger setup issues -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (RANGER-3089) Ranger Setup issue fix
Kishor Gollapalliwar created RANGER-3089: Summary: Ranger Setup issue fix Key: RANGER-3089 URL: https://issues.apache.org/jira/browse/RANGER-3089 Project: Ranger Issue Type: Improvement Components: Ranger Affects Versions: 3.0.0 Reporter: Kishor Gollapalliwar Fix ranger setup issues -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 73040: RANGER-3088: Build tagged-resource-cache using memory optimization flags identical to policy-cache
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73040/#review32 --- Ship it! Ship It! - Madhan Neethiraj On Nov. 25, 2020, 12:17 a.m., Abhay Kulkarni wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73040/ > --- > > (Updated Nov. 25, 2020, 12:17 a.m.) > > > Review request for ranger, Madhan Neethiraj, Ramesh Mani, and Velmurugan > Periasamy. > > > Bugs: RANGER-3088 > https://issues.apache.org/jira/browse/RANGER-3088 > > > Repository: ranger > > > Description > --- > > Policy engine consists of in-memory cache of policies and tagged entities. > Each cache also has an in-memory index for quick look-up. The structure of > the index and its run-time behavior is controlled with configuration options > provided when the index is constructed. > > These indexes should be constructed with identical options for consistent > memory and performance behavior. > > > Diffs > - > > > agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerAbstractContextEnricher.java > fa84760c6 > > agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerTagEnricher.java > 94ac749aa > > agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyRepository.java > 3886eea8b > > > Diff: https://reviews.apache.org/r/73040/diff/1/ > > > Testing > --- > > Ran all unit tests successfully. > > > Thanks, > > Abhay Kulkarni > >
Review Request 73040: RANGER-3088: Build tagged-resource-cache using memory optimization flags identical to policy-cache
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73040/ --- Review request for ranger, Madhan Neethiraj, Ramesh Mani, and Velmurugan Periasamy. Bugs: RANGER-3088 https://issues.apache.org/jira/browse/RANGER-3088 Repository: ranger Description --- Policy engine consists of in-memory cache of policies and tagged entities. Each cache also has an in-memory index for quick look-up. The structure of the index and its run-time behavior is controlled with configuration options provided when the index is constructed. These indexes should be constructed with identical options for consistent memory and performance behavior. Diffs - agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerAbstractContextEnricher.java fa84760c6 agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerTagEnricher.java 94ac749aa agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyRepository.java 3886eea8b Diff: https://reviews.apache.org/r/73040/diff/1/ Testing --- Ran all unit tests successfully. Thanks, Abhay Kulkarni
[jira] [Created] (RANGER-3088) Build tagged-resource-cache using memory optimization flags identical to policy-cache
Abhay Kulkarni created RANGER-3088: -- Summary: Build tagged-resource-cache using memory optimization flags identical to policy-cache Key: RANGER-3088 URL: https://issues.apache.org/jira/browse/RANGER-3088 Project: Ranger Issue Type: Bug Components: Ranger Reporter: Abhay Kulkarni Assignee: Abhay Kulkarni Policy engine consists of in-memory cache of policies and tagged entities. Each cache also has an in-memory index for quick look-up. The structure of the index and its run-time behavior is controlled with configuration options provided when the index is constructed. These indexes should be constructed with identical options for consistent memory and performance behavior. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (RANGER-3087) Making db_setup.py fool-proof and robust
[
https://issues.apache.org/jira/browse/RANGER-3087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vishal Suvagia updated RANGER-3087:
---
Attachment: RANGER-3087.patch
> Making db_setup.py fool-proof and robust
>
>
> Key: RANGER-3087
> URL: https://issues.apache.org/jira/browse/RANGER-3087
> Project: Ranger
> Issue Type: Improvement
> Components: Ranger
>Affects Versions: 3.0.0
>Reporter: Vishal Suvagia
>Assignee: Vishal Suvagia
>Priority: Major
> Attachments: RANGER-3087.patch
>
>
> When a user configures a small heap size in install.properties, vm creation
> in db_setup.py fails to apply the java patches with below error.
> {code:none}
> Error occurred during initialization of VM
> Initial heap size set to a larger value than the maximum heap size
> {code}
> Need to make db_setup.py more fool-proof and robust to wrongly configured
> heap-size.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (RANGER-3087) Making db_setup.py fool-proof and robust
Vishal Suvagia created RANGER-3087:
--
Summary: Making db_setup.py fool-proof and robust
Key: RANGER-3087
URL: https://issues.apache.org/jira/browse/RANGER-3087
Project: Ranger
Issue Type: Improvement
Components: Ranger
Affects Versions: 3.0.0
Reporter: Vishal Suvagia
Assignee: Vishal Suvagia
When a user configures a small heap size in install.properties, vm creation in
db_setup.py fails to apply the java patches with below error.
{code:none}
Error occurred during initialization of VM
Initial heap size set to a larger value than the maximum heap size
{code}
Need to make db_setup.py more fool-proof and robust to wrongly configured
heap-size.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (RANGER-3086) log4j audit's behavior is different from that configured in hive-log4j2.properties
[
https://issues.apache.org/jira/browse/RANGER-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayi Liu resolved RANGER-3086.
---
Resolution: Won't Fix
> log4j audit's behavior is different from that configured in
> hive-log4j2.properties
> --
>
> Key: RANGER-3086
> URL: https://issues.apache.org/jira/browse/RANGER-3086
> Project: Ranger
> Issue Type: Bug
> Components: audit
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Jiayi Liu
>Priority: Major
>
> I need to output the audit of the ranger to a local file.
> ranger-1.2.0+hive-2.3.5 works very well, but the same configuration in
> ranger-2.x+hive-3.1.2+hadoop-3.1.3 cannot output the log files I need
> correctly, as if the ranger audit did not read hive-log4j2.properties, the
> json log has been output to hivesever2.err.
> Can someone please take a look? Below is my configuration.
> hive-log4j2.properties
> {code:java}
> status = INFO
> name = HiveLog4j2
> packages = org.apache.hadoop.hive.ql.log
> # list of properties
> property.hive.log.level = INFO
> property.hive.root.logger = DRFA
> property.hive.log.dir = ${sys:java.io.tmpdir}/${sys:user.name}
> property.hive.log.file = hive.log
> property.hive.perflogger.log.level = INFO
> # list of all appenders
> appenders = console, DRFA, RANGERAUDIT
> # console appender
> appender.console.type = Console
> appender.console.name = console
> appender.console.target = SYSTEM_ERR
> appender.console.layout.type = PatternLayout
> appender.console.layout.pattern = %d{ISO8601} %5p [%t] %c{2}: %m%n
> # daily rolling file appender
> appender.DRFA.type = RollingRandomAccessFile
> appender.DRFA.name = DRFA
> appender.DRFA.fileName = ${sys:hive.log.dir}/${sys:hive.log.file}
> # Use %pid in the filePattern to append @ to the
> filename if you want separate log files for different CLI session
> appender.DRFA.filePattern =
> ${sys:hive.log.dir}/${sys:hive.log.file}.%d{-MM-dd}
> appender.DRFA.layout.type = PatternLayout
> appender.DRFA.layout.pattern = %d{ISO8601} %5p [%t] %c{2}: %m%n
> appender.DRFA.policies.type = Policies
> appender.DRFA.policies.time.type = TimeBasedTriggeringPolicy
> appender.DRFA.policies.time.interval = 1
> appender.DRFA.policies.time.modulate = true
> appender.DRFA.strategy.type = DefaultRolloverStrategy
> appender.DRFA.strategy.max = 30
> appender.DRFA.strategy.action.type = Delete
> appender.DRFA.strategy.action.basepath = /var/log/hive
> appender.DRFA.strategy.action.followLinks = true
> appender.DRFA.strategy.action.condition.type = IfAccumulatedFileSize
> appender.DRFA.strategy.action.condition.exceeds = 500MB
> appender.DRFA.strategy.action.condition.nested_condition.type = IfFileName
> appender.DRFA.strategy.action.condition.nested_condition.glob =
> ${sys:hive.log.file}.*
> # RANGERAUDIT appender
> appender.RANGERAUDIT.type=file
> appender.RANGERAUDIT.name=RANGERAUDIT
> appender.RANGERAUDIT.fileName=${sys:hive.log.dir}/ranger-audit.log
> appender.RANGERAUDIT.filePermissions=rwxrwxrwx
> appender.RANGERAUDIT.layout.type=PatternLayout
> appender.RANGERAUDIT.layout.pattern=%d{ISO8601} %q %5p [%t] %c{2} (%F:%M(%L))
> - %m%n
> # list of all loggers
> loggers = NIOServerCnxn, ClientCnxnSocketNIO, DataNucleus, Datastore, JPOX,
> PerfLogger, Ranger
> logger.NIOServerCnxn.name = org.apache.zookeeper.server.NIOServerCnxn
> logger.NIOServerCnxn.level = WARN
> logger.ClientCnxnSocketNIO.name = org.apache.zookeeper.ClientCnxnSocketNIO
> logger.ClientCnxnSocketNIO.level = WARN
> logger.DataNucleus.name = DataNucleus
> logger.DataNucleus.level = ERROR
> logger.Datastore.name = Datastore
> logger.Datastore.level = ERROR
> logger.JPOX.name = JPOX
> logger.JPOX.level = ERROR
> logger.PerfLogger.name = org.apache.hadoop.hive.ql.log.PerfLogger
> logger.PerfLogger.level = ${sys:hive.perflogger.log.level}
> #logger.Log4JAuditDestination.name =
> org.apache.ranger.audit.destination.Log4JAuditDestination
> #logger.Log4JAuditDestination.level = INFO
> #logger.Log4JAuditDestination.appenderRefs = RANGERAUDIT
> #logger.Log4JAuditDestination.appenderRef.RANGERAUDIT.ref = RANGERAUDIT
> logger.Ranger.name = xaaudit
> logger.Ranger.level = INFO
> logger.Ranger.appenderRefs = RANGERAUDIT
> logger.Ranger.appenderRef.RANGERAUDIT.ref = RANGERAUDIT
> # root logger
> rootLogger.level = ${sys:hive.log.level}
> rootLogger.appenderRefs = root
> rootLogger.appenderRef.root.ref = ${sys:hive.root.logger}
> {code}
> ranger-hive-audit.xml
> {code:java}
>
> xasecure.audit.log4j.is.enabled
> true
>
>
> xasecure.audit.log4j.is.async
> false
>
>
> xasecure.audit.log4j.async.max.queue.size
> 10240
>
>
> xasecure.audit.log4j.async.max.flush.interval.ms
> 3
>
>
> xasecure.audit.destination.log4j
> true
>
>
>
[jira] [Commented] (RANGER-3086) log4j audit's behavior is different from that configured in hive-log4j2.properties
[
https://issues.apache.org/jira/browse/RANGER-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238211#comment-17238211
]
Jiayi Liu commented on RANGER-3086:
---
I found the reason, it is the log4j conflict between hive3 and hadoop3.
> log4j audit's behavior is different from that configured in
> hive-log4j2.properties
> --
>
> Key: RANGER-3086
> URL: https://issues.apache.org/jira/browse/RANGER-3086
> Project: Ranger
> Issue Type: Bug
> Components: audit
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Jiayi Liu
>Priority: Major
>
> I need to output the audit of the ranger to a local file.
> ranger-1.2.0+hive-2.3.5 works very well, but the same configuration in
> ranger-2.x+hive-3.1.2+hadoop-3.1.3 cannot output the log files I need
> correctly, as if the ranger audit did not read hive-log4j2.properties, the
> json log has been output to hivesever2.err.
> Can someone please take a look? Below is my configuration.
> hive-log4j2.properties
> {code:java}
> status = INFO
> name = HiveLog4j2
> packages = org.apache.hadoop.hive.ql.log
> # list of properties
> property.hive.log.level = INFO
> property.hive.root.logger = DRFA
> property.hive.log.dir = ${sys:java.io.tmpdir}/${sys:user.name}
> property.hive.log.file = hive.log
> property.hive.perflogger.log.level = INFO
> # list of all appenders
> appenders = console, DRFA, RANGERAUDIT
> # console appender
> appender.console.type = Console
> appender.console.name = console
> appender.console.target = SYSTEM_ERR
> appender.console.layout.type = PatternLayout
> appender.console.layout.pattern = %d{ISO8601} %5p [%t] %c{2}: %m%n
> # daily rolling file appender
> appender.DRFA.type = RollingRandomAccessFile
> appender.DRFA.name = DRFA
> appender.DRFA.fileName = ${sys:hive.log.dir}/${sys:hive.log.file}
> # Use %pid in the filePattern to append @ to the
> filename if you want separate log files for different CLI session
> appender.DRFA.filePattern =
> ${sys:hive.log.dir}/${sys:hive.log.file}.%d{-MM-dd}
> appender.DRFA.layout.type = PatternLayout
> appender.DRFA.layout.pattern = %d{ISO8601} %5p [%t] %c{2}: %m%n
> appender.DRFA.policies.type = Policies
> appender.DRFA.policies.time.type = TimeBasedTriggeringPolicy
> appender.DRFA.policies.time.interval = 1
> appender.DRFA.policies.time.modulate = true
> appender.DRFA.strategy.type = DefaultRolloverStrategy
> appender.DRFA.strategy.max = 30
> appender.DRFA.strategy.action.type = Delete
> appender.DRFA.strategy.action.basepath = /var/log/hive
> appender.DRFA.strategy.action.followLinks = true
> appender.DRFA.strategy.action.condition.type = IfAccumulatedFileSize
> appender.DRFA.strategy.action.condition.exceeds = 500MB
> appender.DRFA.strategy.action.condition.nested_condition.type = IfFileName
> appender.DRFA.strategy.action.condition.nested_condition.glob =
> ${sys:hive.log.file}.*
> # RANGERAUDIT appender
> appender.RANGERAUDIT.type=file
> appender.RANGERAUDIT.name=RANGERAUDIT
> appender.RANGERAUDIT.fileName=${sys:hive.log.dir}/ranger-audit.log
> appender.RANGERAUDIT.filePermissions=rwxrwxrwx
> appender.RANGERAUDIT.layout.type=PatternLayout
> appender.RANGERAUDIT.layout.pattern=%d{ISO8601} %q %5p [%t] %c{2} (%F:%M(%L))
> - %m%n
> # list of all loggers
> loggers = NIOServerCnxn, ClientCnxnSocketNIO, DataNucleus, Datastore, JPOX,
> PerfLogger, Ranger
> logger.NIOServerCnxn.name = org.apache.zookeeper.server.NIOServerCnxn
> logger.NIOServerCnxn.level = WARN
> logger.ClientCnxnSocketNIO.name = org.apache.zookeeper.ClientCnxnSocketNIO
> logger.ClientCnxnSocketNIO.level = WARN
> logger.DataNucleus.name = DataNucleus
> logger.DataNucleus.level = ERROR
> logger.Datastore.name = Datastore
> logger.Datastore.level = ERROR
> logger.JPOX.name = JPOX
> logger.JPOX.level = ERROR
> logger.PerfLogger.name = org.apache.hadoop.hive.ql.log.PerfLogger
> logger.PerfLogger.level = ${sys:hive.perflogger.log.level}
> #logger.Log4JAuditDestination.name =
> org.apache.ranger.audit.destination.Log4JAuditDestination
> #logger.Log4JAuditDestination.level = INFO
> #logger.Log4JAuditDestination.appenderRefs = RANGERAUDIT
> #logger.Log4JAuditDestination.appenderRef.RANGERAUDIT.ref = RANGERAUDIT
> logger.Ranger.name = xaaudit
> logger.Ranger.level = INFO
> logger.Ranger.appenderRefs = RANGERAUDIT
> logger.Ranger.appenderRef.RANGERAUDIT.ref = RANGERAUDIT
> # root logger
> rootLogger.level = ${sys:hive.log.level}
> rootLogger.appenderRefs = root
> rootLogger.appenderRef.root.ref = ${sys:hive.root.logger}
> {code}
> ranger-hive-audit.xml
> {code:java}
>
> xasecure.audit.log4j.is.enabled
> true
>
>
> xasecure.audit.log4j.is.async
> false
>
>
> xasecure.audit.log4j.async.max.queue.size
> 10240
>
>
> xasecure.audit.log4j.async.max.flush.interval.ms
>
[jira] [Created] (RANGER-3086) log4j audit's behavior is different from that configured in hive-log4j2.properties
Jiayi Liu created RANGER-3086:
-
Summary: log4j audit's behavior is different from that configured
in hive-log4j2.properties
Key: RANGER-3086
URL: https://issues.apache.org/jira/browse/RANGER-3086
Project: Ranger
Issue Type: Bug
Components: audit
Affects Versions: 2.1.0, 2.0.0
Reporter: Jiayi Liu
I need to output the audit of the ranger to a local file.
ranger-1.2.0+hive-2.3.5 works very well, but the same configuration in
ranger-2.x+hive-3.1.2+hadoop-3.1.3 cannot output the log files I need
correctly, as if the ranger audit did not read hive-log4j2.properties, the json
log has been output to hivesever2.err.
Can someone please take a look? Below is my configuration.
hive-log4j2.properties
{code:java}
status = INFO
name = HiveLog4j2
packages = org.apache.hadoop.hive.ql.log
# list of properties
property.hive.log.level = INFO
property.hive.root.logger = DRFA
property.hive.log.dir = ${sys:java.io.tmpdir}/${sys:user.name}
property.hive.log.file = hive.log
property.hive.perflogger.log.level = INFO
# list of all appenders
appenders = console, DRFA, RANGERAUDIT
# console appender
appender.console.type = Console
appender.console.name = console
appender.console.target = SYSTEM_ERR
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{ISO8601} %5p [%t] %c{2}: %m%n
# daily rolling file appender
appender.DRFA.type = RollingRandomAccessFile
appender.DRFA.name = DRFA
appender.DRFA.fileName = ${sys:hive.log.dir}/${sys:hive.log.file}
# Use %pid in the filePattern to append @ to the
filename if you want separate log files for different CLI session
appender.DRFA.filePattern =
${sys:hive.log.dir}/${sys:hive.log.file}.%d{-MM-dd}
appender.DRFA.layout.type = PatternLayout
appender.DRFA.layout.pattern = %d{ISO8601} %5p [%t] %c{2}: %m%n
appender.DRFA.policies.type = Policies
appender.DRFA.policies.time.type = TimeBasedTriggeringPolicy
appender.DRFA.policies.time.interval = 1
appender.DRFA.policies.time.modulate = true
appender.DRFA.strategy.type = DefaultRolloverStrategy
appender.DRFA.strategy.max = 30
appender.DRFA.strategy.action.type = Delete
appender.DRFA.strategy.action.basepath = /var/log/hive
appender.DRFA.strategy.action.followLinks = true
appender.DRFA.strategy.action.condition.type = IfAccumulatedFileSize
appender.DRFA.strategy.action.condition.exceeds = 500MB
appender.DRFA.strategy.action.condition.nested_condition.type = IfFileName
appender.DRFA.strategy.action.condition.nested_condition.glob =
${sys:hive.log.file}.*
# RANGERAUDIT appender
appender.RANGERAUDIT.type=file
appender.RANGERAUDIT.name=RANGERAUDIT
appender.RANGERAUDIT.fileName=${sys:hive.log.dir}/ranger-audit.log
appender.RANGERAUDIT.filePermissions=rwxrwxrwx
appender.RANGERAUDIT.layout.type=PatternLayout
appender.RANGERAUDIT.layout.pattern=%d{ISO8601} %q %5p [%t] %c{2} (%F:%M(%L)) -
%m%n
# list of all loggers
loggers = NIOServerCnxn, ClientCnxnSocketNIO, DataNucleus, Datastore, JPOX,
PerfLogger, Ranger
logger.NIOServerCnxn.name = org.apache.zookeeper.server.NIOServerCnxn
logger.NIOServerCnxn.level = WARN
logger.ClientCnxnSocketNIO.name = org.apache.zookeeper.ClientCnxnSocketNIO
logger.ClientCnxnSocketNIO.level = WARN
logger.DataNucleus.name = DataNucleus
logger.DataNucleus.level = ERROR
logger.Datastore.name = Datastore
logger.Datastore.level = ERROR
logger.JPOX.name = JPOX
logger.JPOX.level = ERROR
logger.PerfLogger.name = org.apache.hadoop.hive.ql.log.PerfLogger
logger.PerfLogger.level = ${sys:hive.perflogger.log.level}
#logger.Log4JAuditDestination.name =
org.apache.ranger.audit.destination.Log4JAuditDestination
#logger.Log4JAuditDestination.level = INFO
#logger.Log4JAuditDestination.appenderRefs = RANGERAUDIT
#logger.Log4JAuditDestination.appenderRef.RANGERAUDIT.ref = RANGERAUDIT
logger.Ranger.name = xaaudit
logger.Ranger.level = INFO
logger.Ranger.appenderRefs = RANGERAUDIT
logger.Ranger.appenderRef.RANGERAUDIT.ref = RANGERAUDIT
# root logger
rootLogger.level = ${sys:hive.log.level}
rootLogger.appenderRefs = root
rootLogger.appenderRef.root.ref = ${sys:hive.root.logger}
{code}
ranger-hive-audit.xml
{code:java}
xasecure.audit.log4j.is.enabled
true
xasecure.audit.log4j.is.async
false
xasecure.audit.log4j.async.max.queue.size
10240
xasecure.audit.log4j.async.max.flush.interval.ms
3
xasecure.audit.destination.log4j
true
xasecure.audit.destination.log4j.logger
xaaudit
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
