Thanks all for the pointers. Really insightful. Subroto I think that’s part of the enterprise version but yeah even I have seen it. Again not sure of the performance implications.
Sagar. On Sat, 2 Mar 2019 at 5:15 AM, Subroto Barua <sbarua...@yahoo.com.invalid> wrote: > Datastax version has an option to store audit info to dse_audit.audit_log > table; I do not know the performance impact since I use the file option > > Subroto > > > On Mar 1, 2019, at 9:40 AM, Jeremiah D Jordan <jeremiah.jor...@gmail.com> > wrote: > > > > AFAIK the Full Query Logging binary format was already made more general > in order to support using that format for the audit logging. > > > > -Jeremiah > > > >> On Mar 1, 2019, at 11:38 AM, Joshua McKenzie <jmcken...@apache.org> > wrote: > >> > >> Is there a world in which a general purpose, side-channel file storage > >> format for transient things like this (hints, batches, audit logs, etc) > >> could be useful as a first class citizen in the codebase? i.e. a world > in > >> which we refactored some of the hints-specific reader/writer code to be > >> used for things like this if/when they come up? > >> > >>> On Thu, Feb 28, 2019 at 12:04 PM Jonathan Haddad <j...@jonhaddad.com > <mailto:j...@jonhaddad.com>> wrote: > >>> > >>> Agreed with Dinesh and Josh. I would *never* put the audit log back in > >>> Cassandra. > >>> > >>> This is extendable, Sagar, so you're free to do as you want, but I'm > very > >>> opposed to putting a ticking time bomb in Cassandra proper. > >>> > >>> Jon > >>> > >>> > >>> On Thu, Feb 28, 2019 at 8:38 AM Dinesh Joshi > <djos...@icloud.com.invalid> > >>> wrote: > >>> > >>>> I strongly echo Josh’s sentiment. Imagine losing audit entries > because C* > >>>> is overloaded? It’s fine if you don’t care about losing audit entries. > >>>> > >>>> Dinesh > >>>> > >>>>> On Feb 28, 2019, at 6:41 AM, Joshua McKenzie <jmcken...@apache.org> > >>>> wrote: > >>>>> > >>>>> One of the things we've run into historically, on a *lot* of axes, is > >>>> that > >>>>> "just put it in C*" for various functionality looks great from a user > >>> and > >>>>> usability perspective, and proves to be something of a nightmare from > >>> an > >>>>> admin / cluster behavior perspective. > >>>>> > >>>>> i.e. - cluster suffering so you're writing hints? Write them to C* > >>> tables > >>>>> and watch the cluster suffer more! :) > >>>>> Same thing probably holds true for audit logging - at a time frame > when > >>>>> things are getting hairy w/a cluster, if you're writing that audit > >>>> logging > >>>>> into C* proper (and dealing with ser/deser, compaction pressure, > >>> flushing > >>>>> pressure, etc) from that, there's a compounding effect of pressure > and > >>>> pain > >>>>> on the cluster. > >>>>> > >>>>> So the TL;DR we as a project kind of philosophically have been moving > >>>>> towards (I think that's valid to say?) is: use C* for the things it's > >>>>> absolutely great at, and try to side-channel other recovery > operations > >>> as > >>>>> much as you can (see: file-based hints) to stay out of its way. > >>>>> > >>>>> Same thing held true w/design of CDC - I debated "materialize in > memory > >>>> for > >>>>> consumer to take over socket", and "keep the data in another C* > table", > >>>> but > >>>>> the ramifications to perf and core I/O operations in C* the moment > >>> things > >>>>> start to go badly were significant enough that the route we went was > >>> "do > >>>> no > >>>>> harm". For better or for worse, as there's obvious tradeoffs there. > >>>>> > >>>>>> On Thu, Feb 28, 2019 at 7:46 AM Sagar <sagarmeansoc...@gmail.com> > >>>> wrote: > >>>>>> > >>>>>> Thanks all for the pointers. > >>>>>> > >>>>>> @Joseph, > >>>>>> > >>>>>> I have gone through the links shared by you. Also, I have been > looking > >>>> at > >>>>>> the code base. > >>>>>> > >>>>>> I understand the fact that pushing the logs to ES or Solr is a lot > >>>> easier > >>>>>> to do. Having said that, the only reason I thought having something > >>> like > >>>>>> this might help is, if I don't want to add more pieces and still > >>>> provide a > >>>>>> central piece of audit logging within Cassandra itself and still be > >>>>>> queryable. > >>>>>> > >>>>>> In terms of usages, one of them could definitely be CDC related use > >>>> cases. > >>>>>> With data being stored in tables and being queryable, it can become > a > >>>> lot > >>>>>> more easier to expose this data to external systems like Kafka > >>> Connect, > >>>>>> Debezium which have the ability to push data to Kafka for example. > >>> Note > >>>>>> that pushing data to Kafka is just an example, but what I mean is, > if > >>> we > >>>>>> can have data in tables, then instead of everyone writing custom > >>> custom > >>>>>> loggers, they can hook into this table info and take action. > >>>>>> > >>>>>> Regarding the infinite loop question, I have done some analysis, and > >>> in > >>>> my > >>>>>> opinion, instead of tweaking the behaviour of Binlog and the way it > >>>>>> functions currently, we can actually spin up another tailer thread > to > >>>> the > >>>>>> same Chronicle Queue which can do the needful. This way the config > >>>> options > >>>>>> etc all remain the same(apart from the logger ofcourse). > >>>>>> > >>>>>> Let me know if any of it makes sense :D > >>>>>> > >>>>>> Thanks! > >>>>>> Sagar. > >>>>>> > >>>>>> > >>>>>> On Thu, Feb 28, 2019 at 1:09 AM Dinesh Joshi > >>> <djos...@icloud.com.invalid > >>>>> > >>>>>> wrote: > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>>> On Feb 27, 2019, at 10:41 AM, Joseph Lynch <joe.e.ly...@gmail.com > > > >>>>>>> wrote: > >>>>>>>> > >>>>>>>> Vinay can confirm, but as far as I am aware we have no current > plans > >>>> to > >>>>>>>> implement audit logging to a table directly, but the > implementation > >>> is > >>>>>>>> fully pluggable (like compaction, compression, etc ...). Check out > >>> the > >>>>>>> blog > >>>>>>>> post [1] and documentation [2] Vinay wrote for more details, but > the > >>>>>>> short > >>>>>>> > >>>>>>> +1. I am still curious as to why you'd want to store audit log > >>> entries > >>>>>>> back in Cassandra? Depending on the scale it can generate a lot of > >>> load > >>>>>> and > >>>>>>> I think you'd end up in an infinite loop because as you're > inserting > >>>> the > >>>>>>> audit log entry you'll generate a new one and so on unless you > black > >>>> list > >>>>>>> audits to that table / keyspace. > >>>>>>> > >>>>>>> Ideally you'd insert this data into ElasticSearch / Solr or some > >>> other > >>>>>>> place that can be then used for analytics or search. > >>>>>>> > >>>>>>> Dinesh > >>>>>>> > --------------------------------------------------------------------- > >>>>>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > >>>>>>> For additional commands, e-mail: dev-h...@cassandra.apache.org > >>>>>>> > >>>>>>> > >>>>>> > >>>> > >>>> > >>>> --------------------------------------------------------------------- > >>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > >>>> For additional commands, e-mail: dev-h...@cassandra.apache.org > >>>> > >>>> > >>> > >>> -- > >>> Jon Haddad > >>> > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rustyrazorblade.com&d=DwIFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g&m=vyXA1unA3gpHGCpKOfRurmET3jOHaV2bjs1mHVVsb2U&s=EDg90XhABktX19m4FaDHKIjFaU2YAHbXjeEGk7Jx6dk&e= > < > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rustyrazorblade.com&d=DwIFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g&m=vyXA1unA3gpHGCpKOfRurmET3jOHaV2bjs1mHVVsb2U&s=EDg90XhABktX19m4FaDHKIjFaU2YAHbXjeEGk7Jx6dk&e= > > > >>> twitter: rustyrazorblade > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > >
