> On Dec. 10, 2021, 4:41 a.m., Ramesh Mani wrote: > > agents-audit/src/main/java/org/apache/ranger/audit/queue/AuditFileSpool.java > > Line 189 (original), 197 (patched) > > <https://reviews.apache.org/r/73752/diff/4/?file=2256392#file2256392line199> > > > > creation of .gz spool files should be controlled nu a configuration. > > Current behavior should be there by default and plugin which needs this > > format can be enabling it. > > > > By the way audit spool files are created because either the > > destinations are down or through put of audit generated are high, in which > > case queue buffer size can be increased so spooling wont happen > > > > With the introduction of Audit filter in Ranger-3000 you can configure > > audit filters to reduce uncessary audits.
> creation of .gz spool files should be controlled nu a configuration. Current > behavior should be there by default and plugin which needs this format can be > enabling it. Why? What's the downside of compression? Less disk space (especially for JSON), less disk IOPS for what may be co-located with a heavy IOPS application. It should be the default. As implemented, it is backwards compatible because it will respect any previous '.log' file extensions. If a plugin would like to avoid compression, they can configure the audit file name to not include the '.gz' file extension. > By the way audit spool files are created because either the destinations are > down or through put of audit generated are high, Yes. I am familiar with this usage. It's what drove me to contribute this patch in the first place. I am finding SOLR very tricky to properly scale with Ranger; SOLR often crashes. A couple of times, when it crashed, a busy Kafka cluster went down because the hard drive filled up these audit messages. Having the data compressed would have given me enough time to fix SOLR and clear the audit backlog. > With the introduction of Audit filter in Ranger-3000 you can configure audit > filters to reduce uncessary audits. Yes. Helpful, but then as an admin, I need to know every audit log message format for every plugin and decide which audit messages they produce are helpful or not. This is something I can do after running without filters for some amount of time and as time allows. Better yet, I'd rather just collect all of the audit messages and only filter in Ranger Admin just in case I do want the messages after all. However, none of that diminishes the need to simply compress the data and lower the risk of a service crash. - David ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73752/#review223835 ----------------------------------------------------------- On Dec. 9, 2021, 7:11 p.m., David Mollitor wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73752/ > ----------------------------------------------------------- > > (Updated Dec. 9, 2021, 7:11 p.m.) > > > Review request for ranger. > > > Repository: ranger > > > Description > ------- > > Allow audit plugins to write GZIP compressed audit log files to the spool > directory. > > > Diffs > ----- > > > agents-audit/src/main/java/org/apache/ranger/audit/queue/AuditFileSpool.java > f8c2aa5f3 > > > Diff: https://reviews.apache.org/r/73752/diff/4/ > > > Testing > ------- > > > Thanks, > > David Mollitor > >
