AFAIK the Full Query Logging binary format was already made more general in order to support using that format for the audit logging.
-Jeremiah > On Mar 1, 2019, at 11:38 AM, Joshua McKenzie <jmcken...@apache.org> wrote: > > Is there a world in which a general purpose, side-channel file storage > format for transient things like this (hints, batches, audit logs, etc) > could be useful as a first class citizen in the codebase? i.e. a world in > which we refactored some of the hints-specific reader/writer code to be > used for things like this if/when they come up? > > On Thu, Feb 28, 2019 at 12:04 PM Jonathan Haddad <j...@jonhaddad.com > <mailto:j...@jonhaddad.com>> wrote: > >> Agreed with Dinesh and Josh. I would *never* put the audit log back in >> Cassandra. >> >> This is extendable, Sagar, so you're free to do as you want, but I'm very >> opposed to putting a ticking time bomb in Cassandra proper. >> >> Jon >> >> >> On Thu, Feb 28, 2019 at 8:38 AM Dinesh Joshi <djos...@icloud.com.invalid> >> wrote: >> >>> I strongly echo Josh’s sentiment. Imagine losing audit entries because C* >>> is overloaded? It’s fine if you don’t care about losing audit entries. >>> >>> Dinesh >>> >>>> On Feb 28, 2019, at 6:41 AM, Joshua McKenzie <jmcken...@apache.org> >>> wrote: >>>> >>>> One of the things we've run into historically, on a *lot* of axes, is >>> that >>>> "just put it in C*" for various functionality looks great from a user >> and >>>> usability perspective, and proves to be something of a nightmare from >> an >>>> admin / cluster behavior perspective. >>>> >>>> i.e. - cluster suffering so you're writing hints? Write them to C* >> tables >>>> and watch the cluster suffer more! :) >>>> Same thing probably holds true for audit logging - at a time frame when >>>> things are getting hairy w/a cluster, if you're writing that audit >>> logging >>>> into C* proper (and dealing with ser/deser, compaction pressure, >> flushing >>>> pressure, etc) from that, there's a compounding effect of pressure and >>> pain >>>> on the cluster. >>>> >>>> So the TL;DR we as a project kind of philosophically have been moving >>>> towards (I think that's valid to say?) is: use C* for the things it's >>>> absolutely great at, and try to side-channel other recovery operations >> as >>>> much as you can (see: file-based hints) to stay out of its way. >>>> >>>> Same thing held true w/design of CDC - I debated "materialize in memory >>> for >>>> consumer to take over socket", and "keep the data in another C* table", >>> but >>>> the ramifications to perf and core I/O operations in C* the moment >> things >>>> start to go badly were significant enough that the route we went was >> "do >>> no >>>> harm". For better or for worse, as there's obvious tradeoffs there. >>>> >>>>> On Thu, Feb 28, 2019 at 7:46 AM Sagar <sagarmeansoc...@gmail.com> >>> wrote: >>>>> >>>>> Thanks all for the pointers. >>>>> >>>>> @Joseph, >>>>> >>>>> I have gone through the links shared by you. Also, I have been looking >>> at >>>>> the code base. >>>>> >>>>> I understand the fact that pushing the logs to ES or Solr is a lot >>> easier >>>>> to do. Having said that, the only reason I thought having something >> like >>>>> this might help is, if I don't want to add more pieces and still >>> provide a >>>>> central piece of audit logging within Cassandra itself and still be >>>>> queryable. >>>>> >>>>> In terms of usages, one of them could definitely be CDC related use >>> cases. >>>>> With data being stored in tables and being queryable, it can become a >>> lot >>>>> more easier to expose this data to external systems like Kafka >> Connect, >>>>> Debezium which have the ability to push data to Kafka for example. >> Note >>>>> that pushing data to Kafka is just an example, but what I mean is, if >> we >>>>> can have data in tables, then instead of everyone writing custom >> custom >>>>> loggers, they can hook into this table info and take action. >>>>> >>>>> Regarding the infinite loop question, I have done some analysis, and >> in >>> my >>>>> opinion, instead of tweaking the behaviour of Binlog and the way it >>>>> functions currently, we can actually spin up another tailer thread to >>> the >>>>> same Chronicle Queue which can do the needful. This way the config >>> options >>>>> etc all remain the same(apart from the logger ofcourse). >>>>> >>>>> Let me know if any of it makes sense :D >>>>> >>>>> Thanks! >>>>> Sagar. >>>>> >>>>> >>>>> On Thu, Feb 28, 2019 at 1:09 AM Dinesh Joshi >> <djos...@icloud.com.invalid >>>> >>>>> wrote: >>>>> >>>>>> >>>>>> >>>>>>> On Feb 27, 2019, at 10:41 AM, Joseph Lynch <joe.e.ly...@gmail.com> >>>>>> wrote: >>>>>>> >>>>>>> Vinay can confirm, but as far as I am aware we have no current plans >>> to >>>>>>> implement audit logging to a table directly, but the implementation >> is >>>>>>> fully pluggable (like compaction, compression, etc ...). Check out >> the >>>>>> blog >>>>>>> post [1] and documentation [2] Vinay wrote for more details, but the >>>>>> short >>>>>> >>>>>> +1. I am still curious as to why you'd want to store audit log >> entries >>>>>> back in Cassandra? Depending on the scale it can generate a lot of >> load >>>>> and >>>>>> I think you'd end up in an infinite loop because as you're inserting >>> the >>>>>> audit log entry you'll generate a new one and so on unless you black >>> list >>>>>> audits to that table / keyspace. >>>>>> >>>>>> Ideally you'd insert this data into ElasticSearch / Solr or some >> other >>>>>> place that can be then used for analytics or search. >>>>>> >>>>>> Dinesh >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org >>>>>> For additional commands, e-mail: dev-h...@cassandra.apache.org >>>>>> >>>>>> >>>>> >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org >>> For additional commands, e-mail: dev-h...@cassandra.apache.org >>> >>> >> >> -- >> Jon Haddad >> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rustyrazorblade.com&d=DwIFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g&m=vyXA1unA3gpHGCpKOfRurmET3jOHaV2bjs1mHVVsb2U&s=EDg90XhABktX19m4FaDHKIjFaU2YAHbXjeEGk7Jx6dk&e= >> >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rustyrazorblade.com&d=DwIFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g&m=vyXA1unA3gpHGCpKOfRurmET3jOHaV2bjs1mHVVsb2U&s=EDg90XhABktX19m4FaDHKIjFaU2YAHbXjeEGk7Jx6dk&e=> >> twitter: rustyrazorblade
