Dale, Log aggregation is the recommended solution for high volume transaction. The log summary will log only the unique request combination (user, resource, action, IP,…) with the count for the interval. The interval is by default is 5 seconds, so in Hbase, if each RegionServer is handling 20K request per second, 100K requests will be compacted to very few audit logs. You can increment/decrement the interval to control the number of audit logs.
Thanks Bosco From: Alok Lal <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Monday, June 8, 2015 at 11:02 AM To: "[email protected]" <[email protected]> Subject: Re: Ranger Audit Page not displaying any results > 1. All audit messages go into a single table. Based on the type of the > database you could create partition the table by date so that it is easy to > drop stale partitions. It should be then possible to setup a job that can > periodically drop stale partitions. Different databases have different ways > to do so. >> 1. If you want to preserve the logs for longer than the database might allow >> then also consider sending audit logging to HDFS in parallel. Note that >> queries are run against data in database not HDFS. Tee’ing logging to HFDS >> would be for archival purposes at timelines longer than the db can support. > 2. It is not possible to piecemeal limit what is logged. Policy permissions > are not at the level of user action, e.g. PUT v/s SCAN, etc. Instead several > of user actions are clustered into a few access types like Read, Write, > Create, Admin. > Hopefully automating the periodic truncation of audit logs per the above means > would help to mitigate the need for this. The ideal solution is provided > however by https://issues.apache.org/jira/browse/RANGER-276. This is available > from 0.5 onwards. This can aggregate logs at source and put less burden on the > database or Solr in 0.5 onwards. > > Thanks > > > On 6/8/15, 5:45 AM, "Bradman, Dale" <[email protected]> wrote: > >> Reason being was because the ranger MySql database had taken up all disk >> space on the machine it was installed on. >> >> As I am running HBase REST server with lots of messages coming through every >> minute, the audit logs are getting absolutely hammered by “PUT” commands. >> >> This is great and all, but is there a way to: >> >> a) prune the logs without having to delete straight from the MySQL database? >> b) Limit what is sent to the MySQL DB? For example, if it is a “PUT” command >> coming in from HBase then do not write to the audit log for this. I know I >> can turn the HBase auditing off completely but I would rather filter through >> what I would and would not like filtered. >> >> Thank you in advance. >> Dale >> >>> On 21 May 2015, at 18:07, Alok Lal <[email protected]> wrote: >>> >>> Perhaps the policy manager logs could point you the problem? They should >>> be under <install-dir>/ews/logs. If not then the command line of the >>> policy manager process contains the location of log files: run "ps -eaf | >>> grep Embed² and then look for logdir argument to the process. >>> >>> Thanks >>> >>> >>> >>> On 5/21/15, 5:22 AM, "Bradman, Dale" <[email protected]> wrote: >>> >>>> Hi, >>>> >>>> When I log in to the Ranger UI and click on the audit tab as an Admin, it >>>> says ³No Access Audit found!² Which is strange because it has been >>>> working fine up until now. I have checked the MySql ranger_audit database >>>> and it is still heavily populated with information. One thing that has >>>> changed is the that there is no more disk space available on the node >>>> that MySql is hosted on. This will be expanded in the next day or so but >>>> in the mean time, surely I should still be able to retrieve the audit >>>> logs? >>>> >>>> Thanks, >>>> Dale >>>> >>>> ________________________________ >>>> >>>> Capgemini is a trading name used by the Capgemini Group of companies >>>> which includes Capgemini UK plc, a company registered in England and >>>> Wales (number 943935) whose registered office is at No. 1, Forge End, >>>> Woking, Surrey, GU21 6DB. >>> >> >> >> ________________________________ >> >> Capgemini is a trading name used by the Capgemini Group of companies which >> includes Capgemini UK plc, a company registered in England and Wales (number >> 943935) whose registered office is at No. 1, Forge End, Woking, Surrey, GU21 >> 6DB. >> This message contains information that may be privileged or confidential and >> is the property of the Capgemini Group. It is intended only for the person to >> whom it is addressed. If you are not the intended recipient, you are not >> authorized to read, print, retain, copy, disseminate, distribute, or use this >> message or any part thereof. If you receive this message in error, please >> notify the sender immediately and delete all copies of this message. >> >>
