Sure Chris. I would be watching for that ticket's closure. You had pointed out about this ticket already and what I had decided was, while this gets merged etc, I can work with the folks here and get a design out.
Thanks! Sagar. On Tue, Apr 9, 2019 at 9:58 PM Chris Lohfink <clohfin...@gmail.com> wrote: > You should really build it on/wait for > https://issues.apache.org/jira/browse/CASSANDRA-14629 (Can use a reviewer > please btw!) then you can provide each entry of the audit log binlog as a > partition in iterator format. Can use settings to find where binlog output > is and go from there. > > Chris > > On Tue, Apr 9, 2019 at 10:27 AM Sagar <sagarmeansoc...@gmail.com> wrote: > > > Thanks Jon. > > > > So, it will basically have to read the audit log file which can certainly > > be implemented. One of the things to consider is, if for example, we have > > some external system which is polling the audit log table on a given node > > every minute to get incremental audit logs. In that case, it will keep > > track of the last offset read from the file and query from that point > > onward from the virtual table. I think such querying APIs are already > > present in the current Virtual table implementation. So, in this case, > the > > query would be to read the file from the specified offset. We can use > > RandomAccessFile for example for this. > > > > Another thing to consider is, what if the audit log files get rolled > > over(does this happen?) and the fictitious external system from above has > > not read the rolled over file fully. In that case, it will need to read > the > > ex- audit log file fully and the remaining bytes from the new audit log. > > > > Thanks! > > Sagar. > > > > On Thu, Apr 4, 2019 at 8:59 AM Jon Haddad <j...@jonhaddad.com> wrote: > > > > > The virtual table could read the data out of the audit log, just like > > > it could read a hosts file or list the output of the ps command. > > > > > > > > > On Wed, Apr 3, 2019 at 8:02 PM Sagar <sagarmeansoc...@gmail.com> > wrote: > > > > > > > > Thanks Alex. > > > > > > > > I was going through the implementation of Virtual tables thus far and > > the > > > > data that we get when we query against them seems to be more point in > > > time > > > > like CachesTable or fairly static like Settings. Having said that, > > audit > > > > log's nature of data doesn't fall in either of the 2 categories. For > > > Audit > > > > log, it should be more of a stream of events that happen on that node > > and > > > > almost all events need to be captured. The class AbstractDataSet > being > > > used > > > > by the Virtual tables suggests that it can be built on demand and > > thrown > > > > after use(which is what is happening currently) or can be persisted. > > IMO > > > if > > > > we need audit logs on Virtual tables, we will have to go the route of > > > being > > > > able to persist the events generated. > > > > > > > > Sagar. > > > > > > > > On Sun, Mar 31, 2019 at 11:35 PM Alex Ott <alex...@gmail.com> wrote: > > > > > > > > > Hi Sagar > > > > > > > > > > 3.x/4.x are versions for open source variant of drivers, while DSE > > > versions > > > > > are 1.x/2.x > > > > > > > > > > Description of this function is a thttps:// > > > > > docs.datastax.com/en/drivers/java/3.6/ > > > > > > > > > > Sagar at "Tue, 26 Mar 2019 22:12:56 +0530" wrote: > > > > > S> Thanks Andy, > > > > > > > > > > S> This enhancement is in the datastax version and not in the > apache > > > > > cassandra > > > > > S> driver? > > > > > > > > > > S> Thanks! > > > > > S> Sagar. > > > > > > > > > > S> On Tue, Mar 26, 2019 at 3:23 AM Andy Tolbert < > > > > > andrew.tolb...@datastax.com> > > > > > S> wrote: > > > > > > > > > > >> Hello > > > > > >> > > > > > >> 1) yes its local only. The driver by default does connect to > each > > > host > > > > > >> > though so its pretty trivial to have a load balancing policy > > that > > > > > you can > > > > > >> > direct to specific hosts (this should probably be in the > driver > > > so > > > > > people > > > > > >> > dont have to keep reimplementing it). > > > > > >> > > > > > > >> > > > > > >> The capability to target a specific host was added to the java > > > driver > > > > > (and > > > > > >> others) recently in anticipation of Virtual Tables in version > > > 3.6.0+ > > > > > via > > > > > >> Statement.setHost [1]. This will bypass the load balancing > > policy > > > > > >> completely and send the request directly to that that Host > > > (assuming > > > > > it's > > > > > >> connected). > > > > > >> > > > > > >> The drivers also parse virtual table metadata as well. > > > > > >> > > > > > >> [1]: > > > > > >> > > > > > >> > > > > > > > > > > > https://docs.datastax.com/en/drivers/java/3.6/com/datastax/driver/core/Statement.html#setHost-com.datastax.driver.core.Host- > > > > > >> > > > > > >> Thanks! > > > > > >> Andy > > > > > >> > > > > > >> On Mon, Mar 25, 2019 at 11:29 AM Sagar < > > sagarmeansoc...@gmail.com> > > > > > wrote: > > > > > >> > > > > > >> > Thanks Chris. I got caught up with a few things and couldn't > > > reply > > > > > back. > > > > > >> > So, I re-looked this again and I think virtual tables can be > > > used for > > > > > >> audit > > > > > >> > logging. Considering that they don't have any replication - > so > > we > > > > > won't > > > > > >> be > > > > > >> > clogging the network with replication IO. > > > > > >> > > > > > > >> > In terms of storage, from what I understood, virtual tables > > don't > > > > > have > > > > > >> any > > > > > >> > associated SSTables. So, is data stored only in Memtables? > Can > > > you > > > > > please > > > > > >> > shed some light on storage and the retention because of this? > > > > > >> > > > > > > >> > Lastly, the driver changes, I agree, we should make the > driver > > be > > > > > able to > > > > > >> > contact to specific hosts with the correct LBP. If we do go > > this > > > > > route, I > > > > > >> > can start taking a look at it. > > > > > >> > > > > > > >> > Thanks! > > > > > >> > Sagar. > > > > > >> > > > > > > >> > On Wed, Mar 6, 2019 at 10:42 PM Chris Lohfink < > > > clohfin...@gmail.com> > > > > > >> > wrote: > > > > > >> > > > > > > >> > > 1) yes its local only. The driver by default does connect > to > > > each > > > > > host > > > > > >> > > though so its pretty trivial to have a load balancing > policy > > > that > > > > > you > > > > > >> can > > > > > >> > > direct to specific hosts (this should probably be in the > > > driver so > > > > > >> people > > > > > >> > > dont have to keep reimplementing it). > > > > > >> > > > > > > > >> > > 2) yes, easiest way is to setup a whitelist load balancing > > > policy > > > > > like > > > > > >> in > > > > > >> > > cqlsh but like above. Best option is a custom LBP + > > > > > StatementWrapper > > > > > >> that > > > > > >> > > holds the host target which can direct individual queries > to > > > > > specific > > > > > >> > hosts > > > > > >> > > > > > > > >> > > 3) yes, cqlsh makes a connection to local C* instance with > > > > > whitelist > > > > > >> > policy > > > > > >> > > so it only queries that one node. > > > > > >> > > > > > > > >> > > Chris > > > > > >> > > > > > > > >> > > On Wed, Mar 6, 2019 at 9:43 AM Sagar < > > > sagarmeansoc...@gmail.com> > > > > > >> wrote: > > > > > >> > > > > > > > >> > > > So, I went through the ticket for the creation of Virtual > > > > > Tables(must > > > > > >> > say > > > > > >> > > > it was quite a long ticket spanning across 4 years). > > > > > >> > > > > > > > > >> > > > I see that there are a few tables created in the > db.virtual > > > > > package. > > > > > >> > > These > > > > > >> > > > appear to be metrics related tables. > > > > > >> > > > > > > > > >> > > > Couple of questions here: > > > > > >> > > > > > > > > >> > > > 1) Do all the tables pertain only data locally? What I > mean > > > is > > > > > that > > > > > >> in > > > > > >> > a > > > > > >> > > > cluster, each node will have its own ThreadPoolsTable > > > pertaining > > > > > to > > > > > >> > > thread > > > > > >> > > > pools on that node? Is that assumption correct? > > > > > >> > > > 2) In terms of querying, again can we query only > locally? I > > > saw > > > > > a lot > > > > > >> > of > > > > > >> > > > discussion on the ticket for where node = 1.2.3.4. I > guess > > > that > > > > > isn't > > > > > >> > > > supported? So. for any user to query for metrics of a > given > > > > > node, he > > > > > >> > will > > > > > >> > > > have to login and query on that node. > > > > > >> > > > 3) Looks like these metrics are queryable via cqlsh? Is > > that > > > > > >> statement > > > > > >> > > > correct? > > > > > >> > > > > > > > > >> > > > Thanks! > > > > > >> > > > Sagar. > > > > > >> > > > > > > > > >> > > > On Tue, Mar 5, 2019 at 7:30 AM Sagar < > > > sagarmeansoc...@gmail.com> > > > > > >> > wrote: > > > > > >> > > > > > > > > >> > > > > Right, Thanks Jonathan and Chris. > > > > > >> > > > > > > > > > >> > > > > Mean while, I would go through the 2 jira items to try > > and > > > > > >> understand > > > > > >> > > > > about virtual tables. > > > > > >> > > > > > > > > > >> > > > > Thanks! > > > > > >> > > > > Sagar. > > > > > >> > > > > > > > > > >> > > > > On Tue, Mar 5, 2019 at 1:14 AM Jonathan Haddad < > > > > > j...@jonhaddad.com> > > > > > >> > > > wrote: > > > > > >> > > > > > > > > > >> > > > >> Sagar, > > > > > >> > > > >> > > > > > >> > > > >> There isn't going to be much in the way of docs, since > > > it's > > > > > brand > > > > > >> > new > > > > > >> > > > and > > > > > >> > > > >> not really a public facing thing yet. As Chris > pointed > > > out, > > > > > >> there's > > > > > >> > > > other > > > > > >> > > > >> work that would need to be done to work on virtual > > tables > > > for > > > > > >> large > > > > > >> > > > >> datasets. > > > > > >> > > > >> > > > > > >> > > > >> Jon > > > > > >> > > > >> > > > > > >> > > > >> On Mon, Mar 4, 2019 at 6:42 AM Chris Lohfink < > > > > > >> clohfin...@gmail.com> > > > > > >> > > > >> wrote: > > > > > >> > > > >> > > > > > >> > > > >> > While you probably could put a virtual table wrapper > > > over > > > > > the > > > > > >> > > binlogs, > > > > > >> > > > >> you > > > > > >> > > > >> > would want to wait for something like > > > > > >> > > > >> > > > > > > >> > > > > > > >> > > > > > S> > > > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_CASSANDRA-2D14629&d=DwIFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=VHsWqsWT2MoX5jRZ0xZfdAWZxBsrn5KzowynGYCJaXE&m=Hm3Dmun0qF2plgG5tk2ihs3UcucypmoQY1YEE2vpsuE&s=MMYt6iqyvTZ5IilHTp0BhXswf-zCSN-xjXqIbC0IV_I&e= > > > > > >> > to get in so > > > > > >> > > > you > > > > > >> > > > >> > would not OOM. The current virtual table > > implementation > > > > > requires > > > > > >> > you > > > > > >> > > > >> have > > > > > >> > > > >> > the entire result set to be returned at once. > > > > > >> > > > >> > > > > > > >> > > > >> > Chris > > > > > >> > > > >> > > > > > > >> > > > >> > On Mon, Mar 4, 2019 at 5:29 AM Sagar < > > > > > sagarmeansoc...@gmail.com > > > > > >> > > > > > > >> > > > wrote: > > > > > >> > > > >> > > > > > > >> > > > >> > > Hi Jonathan, > > > > > >> > > > >> > > > > > > > >> > > > >> > > I couldn't find much literature on Virtual tables > > > apart > > > > > from > > > > > >> > this > > > > > >> > > > >> ticket: > > > > > >> > > > >> > > > > > > > >> > > > >> > > > > > > > >> > > > > > > >> > > > > > S> > > > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_CASSANDRA-2D7622&d=DwIFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=VHsWqsWT2MoX5jRZ0xZfdAWZxBsrn5KzowynGYCJaXE&m=Hm3Dmun0qF2plgG5tk2ihs3UcucypmoQY1YEE2vpsuE&s=gMx2_o1_2qTFpyS4Lc_mO0wPNXKRH-gn8vO2bpJr2-o&e= > > > > > >> > > > >> > > > > > > > >> > > > >> > > Any insights would be helpful. > > > > > >> > > > >> > > > > > > > >> > > > >> > > Thanks! > > > > > >> > > > >> > > Sagar. > > > > > >> > > > >> > > > > > > > >> > > > >> > > On Sat, Mar 2, 2019 at 7:23 AM Jonathan Haddad < > > > > > >> > j...@jonhaddad.com > > > > > >> > > > > > > > > >> > > > >> > wrote: > > > > > >> > > > >> > > > > > > > >> > > > >> > > > Instead of logging to tables, putting a virtual > > > table > > > > > around > > > > > >> > the > > > > > >> > > > >> audit > > > > > >> > > > >> > / > > > > > >> > > > >> > > > query logs might be an option. Same with the > > commit > > > log > > > > > for > > > > > >> > cdc > > > > > >> > > > >> > > > > > > > > >> > > > >> > > > On Fri, Mar 1, 2019 at 5:25 PM Sagar < > > > > > >> > sagarmeansoc...@gmail.com > > > > > >> > > > > > > > > >> > > > >> > wrote: > > > > > >> > > > >> > > > > > > > > >> > > > >> > > > > Thanks all for the pointers. Really > insightful. > > > > > >> > > > >> > > > > > > > > > >> > > > >> > > > > Subroto I think that’s part of the enterprise > > > version > > > > > but > > > > > >> > yeah > > > > > >> > > > >> even I > > > > > >> > > > >> > > > have > > > > > >> > > > >> > > > > seen it. Again not sure of the performance > > > > > implications. > > > > > >> > > > >> > > > > > > > > > >> > > > >> > > > > Sagar. > > > > > >> > > > >> > > > > > > > > > >> > > > >> > > > > On Sat, 2 Mar 2019 at 5:15 AM, Subroto Barua > > > > > >> > > > >> > > <sbarua...@yahoo.com.invalid > > > > > >> > > > >> > > > > > > > > > >> > > > >> > > > > wrote: > > > > > >> > > > >> > > > > > > > > > >> > > > >> > > > > > Datastax version has an option to store > audit > > > info > > > > > to > > > > > >> > > > >> > > > dse_audit.audit_log > > > > > >> > > > >> > > > > > table; I do not know the performance impact > > > since I > > > > > use > > > > > >> > the > > > > > >> > > > file > > > > > >> > > > >> > > option > > > > > >> > > > >> > > > > > > > > > > >> > > > >> > > > > > Subroto > > > > > >> > > > >> > > > > > > > > > > >> > > > >> > > > > > > On Mar 1, 2019, at 9:40 AM, Jeremiah D > > Jordan > > > < > > > > > >> > > > >> > > > > jeremiah.jor...@gmail.com> > > > > > >> > > > >> > > > > > wrote: > > > > > >> > > > >> > > > > > > > > > > > >> > > > >> > > > > > > AFAIK the Full Query Logging binary format > > was > > > > > already > > > > > >> > > made > > > > > >> > > > >> more > > > > > >> > > > >> > > > > general > > > > > >> > > > >> > > > > > in order to support using that format for > the > > > audit > > > > > >> > logging. > > > > > >> > > > >> > > > > > > > > > > > >> > > > >> > > > > > > -Jeremiah > > > > > >> > > > >> > > > > > > > > > > > >> > > > >> > > > > > >> On Mar 1, 2019, at 11:38 AM, Joshua > > McKenzie > > > < > > > > > >> > > > >> > > jmcken...@apache.org> > > > > > >> > > > >> > > > > > wrote: > > > > > >> > > > >> > > > > > >> > > > > > >> > > > >> > > > > > >> Is there a world in which a general > > purpose, > > > > > >> > side-channel > > > > > >> > > > >> file > > > > > >> > > > >> > > > storage > > > > > >> > > > >> > > > > > >> format for transient things like this > > (hints, > > > > > >> batches, > > > > > >> > > > audit > > > > > >> > > > >> > logs, > > > > > >> > > > >> > > > > etc) > > > > > >> > > > >> > > > > > >> could be useful as a first class citizen > in > > > the > > > > > >> > codebase? > > > > > >> > > > >> i.e. a > > > > > >> > > > >> > > > world > > > > > >> > > > >> > > > > > in > > > > > >> > > > >> > > > > > >> which we refactored some of the > > > hints-specific > > > > > >> > > > reader/writer > > > > > >> > > > >> > code > > > > > >> > > > >> > > to > > > > > >> > > > >> > > > > be > > > > > >> > > > >> > > > > > >> used for things like this if/when they > come > > > up? > > > > > >> > > > >> > > > > > >> > > > > > >> > > > >> > > > > > >>> On Thu, Feb 28, 2019 at 12:04 PM > Jonathan > > > > > Haddad < > > > > > >> > > > >> > > > j...@jonhaddad.com > > > > > >> > > > >> > > > > > <mailto:j...@jonhaddad.com>> wrote: > > > > > >> > > > >> > > > > > >>> > > > > > >> > > > >> > > > > > >>> Agreed with Dinesh and Josh. I would > > > *never* > > > > > put > > > > > >> the > > > > > >> > > > audit > > > > > >> > > > >> log > > > > > >> > > > >> > > > back > > > > > >> > > > >> > > > > in > > > > > >> > > > >> > > > > > >>> Cassandra. > > > > > >> > > > >> > > > > > >>> > > > > > >> > > > >> > > > > > >>> This is extendable, Sagar, so you're > free > > > to do > > > > > as > > > > > >> you > > > > > >> > > > want, > > > > > >> > > > >> > but > > > > > >> > > > >> > > > I'm > > > > > >> > > > >> > > > > > very > > > > > >> > > > >> > > > > > >>> opposed to putting a ticking time bomb > in > > > > > Cassandra > > > > > >> > > > proper. > > > > > >> > > > >> > > > > > >>> > > > > > >> > > > >> > > > > > >>> Jon > > > > > >> > > > >> > > > > > >>> > > > > > >> > > > >> > > > > > >>> > > > > > >> > > > >> > > > > > >>> On Thu, Feb 28, 2019 at 8:38 AM Dinesh > > Joshi > > > > > >> > > > >> > > > > > <djos...@icloud.com.invalid> > > > > > >> > > > >> > > > > > >>> wrote: > > > > > >> > > > >> > > > > > >>> > > > > > >> > > > >> > > > > > >>>> I strongly echo Josh’s sentiment. > Imagine > > > > > losing > > > > > >> > audit > > > > > >> > > > >> entries > > > > > >> > > > >> > > > > > because C* > > > > > >> > > > >> > > > > > >>>> is overloaded? It’s fine if you don’t > > care > > > > > about > > > > > >> > losing > > > > > >> > > > >> audit > > > > > >> > > > >> > > > > entries. > > > > > >> > > > >> > > > > > >>>> > > > > > >> > > > >> > > > > > >>>> Dinesh > > > > > >> > > > >> > > > > > >>>> > > > > > >> > > > >> > > > > > >>>>> On Feb 28, 2019, at 6:41 AM, Joshua > > > McKenzie < > > > > > >> > > > >> > > > jmcken...@apache.org > > > > > >> > > > >> > > > > > > > > > > >> > > > >> > > > > > >>>> wrote: > > > > > >> > > > >> > > > > > >>>>> > > > > > >> > > > >> > > > > > >>>>> One of the things we've run into > > > > > historically, on > > > > > >> a > > > > > >> > > > *lot* > > > > > >> > > > >> of > > > > > >> > > > >> > > > axes, > > > > > >> > > > >> > > > > is > > > > > >> > > > >> > > > > > >>>> that > > > > > >> > > > >> > > > > > >>>>> "just put it in C*" for various > > > functionality > > > > > >> looks > > > > > >> > > > great > > > > > >> > > > >> > from > > > > > >> > > > >> > > a > > > > > >> > > > >> > > > > user > > > > > >> > > > >> > > > > > >>> and > > > > > >> > > > >> > > > > > >>>>> usability perspective, and proves to > be > > > > > something > > > > > >> > of a > > > > > >> > > > >> > > nightmare > > > > > >> > > > >> > > > > from > > > > > >> > > > >> > > > > > >>> an > > > > > >> > > > >> > > > > > >>>>> admin / cluster behavior perspective. > > > > > >> > > > >> > > > > > >>>>> > > > > > >> > > > >> > > > > > >>>>> i.e. - cluster suffering so you're > > writing > > > > > hints? > > > > > >> > > Write > > > > > >> > > > >> them > > > > > >> > > > >> > to > > > > > >> > > > >> > > > C* > > > > > >> > > > >> > > > > > >>> tables > > > > > >> > > > >> > > > > > >>>>> and watch the cluster suffer more! :) > > > > > >> > > > >> > > > > > >>>>> Same thing probably holds true for > audit > > > > > logging - > > > > > >> > at > > > > > >> > > a > > > > > >> > > > >> time > > > > > >> > > > >> > > > frame > > > > > >> > > > >> > > > > > when > > > > > >> > > > >> > > > > > >>>>> things are getting hairy w/a cluster, > if > > > > > you're > > > > > >> > > writing > > > > > >> > > > >> that > > > > > >> > > > >> > > > audit > > > > > >> > > > >> > > > > > >>>> logging > > > > > >> > > > >> > > > > > >>>>> into C* proper (and dealing with > > > ser/deser, > > > > > >> > compaction > > > > > >> > > > >> > > pressure, > > > > > >> > > > >> > > > > > >>> flushing > > > > > >> > > > >> > > > > > >>>>> pressure, etc) from that, there's a > > > > > compounding > > > > > >> > effect > > > > > >> > > > of > > > > > >> > > > >> > > > pressure > > > > > >> > > > >> > > > > > and > > > > > >> > > > >> > > > > > >>>> pain > > > > > >> > > > >> > > > > > >>>>> on the cluster. > > > > > >> > > > >> > > > > > >>>>> > > > > > >> > > > >> > > > > > >>>>> So the TL;DR we as a project kind of > > > > > >> philosophically > > > > > >> > > > have > > > > > >> > > > >> > been > > > > > >> > > > >> > > > > moving > > > > > >> > > > >> > > > > > >>>>> towards (I think that's valid to say?) > > is: > > > > > use C* > > > > > >> > for > > > > > >> > > > the > > > > > >> > > > >> > > things > > > > > >> > > > >> > > > > it's > > > > > >> > > > >> > > > > > >>>>> absolutely great at, and try to > > > side-channel > > > > > other > > > > > >> > > > >> recovery > > > > > >> > > > >> > > > > > operations > > > > > >> > > > >> > > > > > >>> as > > > > > >> > > > >> > > > > > >>>>> much as you can (see: file-based > hints) > > to > > > > > stay > > > > > >> out > > > > > >> > of > > > > > >> > > > its > > > > > >> > > > >> > way. > > > > > >> > > > >> > > > > > >>>>> > > > > > >> > > > >> > > > > > >>>>> Same thing held true w/design of CDC > - I > > > > > debated > > > > > >> > > > >> "materialize > > > > > >> > > > >> > > in > > > > > >> > > > >> > > > > > memory > > > > > >> > > > >> > > > > > >>>> for > > > > > >> > > > >> > > > > > >>>>> consumer to take over socket", and > "keep > > > the > > > > > data > > > > > >> in > > > > > >> > > > >> another > > > > > >> > > > >> > C* > > > > > >> > > > >> > > > > > table", > > > > > >> > > > >> > > > > > >>>> but > > > > > >> > > > >> > > > > > >>>>> the ramifications to perf and core I/O > > > > > operations > > > > > >> in > > > > > >> > > C* > > > > > >> > > > >> the > > > > > >> > > > >> > > > moment > > > > > >> > > > >> > > > > > >>> things > > > > > >> > > > >> > > > > > >>>>> start to go badly were significant > > enough > > > > > that the > > > > > >> > > route > > > > > >> > > > >> we > > > > > >> > > > >> > > went > > > > > >> > > > >> > > > > was > > > > > >> > > > >> > > > > > >>> "do > > > > > >> > > > >> > > > > > >>>> no > > > > > >> > > > >> > > > > > >>>>> harm". For better or for worse, as > > there's > > > > > obvious > > > > > >> > > > >> tradeoffs > > > > > >> > > > >> > > > there. > > > > > >> > > > >> > > > > > >>>>> > > > > > >> > > > >> > > > > > >>>>>> On Thu, Feb 28, 2019 at 7:46 AM > Sagar < > > > > > >> > > > >> > > > sagarmeansoc...@gmail.com> > > > > > >> > > > >> > > > > > >>>> wrote: > > > > > >> > > > >> > > > > > >>>>>> > > > > > >> > > > >> > > > > > >>>>>> Thanks all for the pointers. > > > > > >> > > > >> > > > > > >>>>>> > > > > > >> > > > >> > > > > > >>>>>> @Joseph, > > > > > >> > > > >> > > > > > >>>>>> > > > > > >> > > > >> > > > > > >>>>>> I have gone through the links shared > by > > > you. > > > > > >> Also, > > > > > >> > I > > > > > >> > > > have > > > > > >> > > > >> > been > > > > > >> > > > >> > > > > > looking > > > > > >> > > > >> > > > > > >>>> at > > > > > >> > > > >> > > > > > >>>>>> the code base. > > > > > >> > > > >> > > > > > >>>>>> > > > > > >> > > > >> > > > > > >>>>>> I understand the fact that pushing > the > > > logs > > > > > to ES > > > > > >> > or > > > > > >> > > > Solr > > > > > >> > > > >> > is a > > > > > >> > > > >> > > > lot > > > > > >> > > > >> > > > > > >>>> easier > > > > > >> > > > >> > > > > > >>>>>> to do. Having said that, the only > > reason > > > I > > > > > >> thought > > > > > >> > > > having > > > > > >> > > > >> > > > > something > > > > > >> > > > >> > > > > > >>> like > > > > > >> > > > >> > > > > > >>>>>> this might help is, if I don't want > to > > > add > > > > > more > > > > > >> > > pieces > > > > > >> > > > >> and > > > > > >> > > > >> > > still > > > > > >> > > > >> > > > > > >>>> provide a > > > > > >> > > > >> > > > > > >>>>>> central piece of audit logging within > > > > > Cassandra > > > > > >> > > itself > > > > > >> > > > >> and > > > > > >> > > > >> > > still > > > > > >> > > > >> > > > > be > > > > > >> > > > >> > > > > > >>>>>> queryable. > > > > > >> > > > >> > > > > > >>>>>> > > > > > >> > > > >> > > > > > >>>>>> In terms of usages, one of them could > > > > > definitely > > > > > >> be > > > > > >> > > CDC > > > > > >> > > > >> > > related > > > > > >> > > > >> > > > > use > > > > > >> > > > >> > > > > > >>>> cases. > > > > > >> > > > >> > > > > > >>>>>> With data being stored in tables and > > > being > > > > > >> > queryable, > > > > > >> > > > it > > > > > >> > > > >> can > > > > > >> > > > >> > > > > become > > > > > >> > > > >> > > > > > a > > > > > >> > > > >> > > > > > >>>> lot > > > > > >> > > > >> > > > > > >>>>>> more easier to expose this data to > > > external > > > > > >> systems > > > > > >> > > > like > > > > > >> > > > >> > Kafka > > > > > >> > > > >> > > > > > >>> Connect, > > > > > >> > > > >> > > > > > >>>>>> Debezium which have the ability to > push > > > data > > > > > to > > > > > >> > Kafka > > > > > >> > > > for > > > > > >> > > > >> > > > example. > > > > > >> > > > >> > > > > > >>> Note > > > > > >> > > > >> > > > > > >>>>>> that pushing data to Kafka is just an > > > > > example, > > > > > >> but > > > > > >> > > > what I > > > > > >> > > > >> > mean > > > > > >> > > > >> > > > is, > > > > > >> > > > >> > > > > > if > > > > > >> > > > >> > > > > > >>> we > > > > > >> > > > >> > > > > > >>>>>> can have data in tables, then instead > > of > > > > > everyone > > > > > >> > > > writing > > > > > >> > > > >> > > custom > > > > > >> > > > >> > > > > > >>> custom > > > > > >> > > > >> > > > > > >>>>>> loggers, they can hook into this > table > > > info > > > > > and > > > > > >> > take > > > > > >> > > > >> action. > > > > > >> > > > >> > > > > > >>>>>> > > > > > >> > > > >> > > > > > >>>>>> Regarding the infinite loop > question, I > > > have > > > > > done > > > > > >> > > some > > > > > >> > > > >> > > analysis, > > > > > >> > > > >> > > > > and > > > > > >> > > > >> > > > > > >>> in > > > > > >> > > > >> > > > > > >>>> my > > > > > >> > > > >> > > > > > >>>>>> opinion, instead of tweaking the > > > behaviour of > > > > > >> > Binlog > > > > > >> > > > and > > > > > >> > > > >> the > > > > > >> > > > >> > > way > > > > > >> > > > >> > > > > it > > > > > >> > > > >> > > > > > >>>>>> functions currently, we can actually > > > spin up > > > > > >> > another > > > > > >> > > > >> tailer > > > > > >> > > > >> > > > thread > > > > > >> > > > >> > > > > > to > > > > > >> > > > >> > > > > > >>>> the > > > > > >> > > > >> > > > > > >>>>>> same Chronicle Queue which can do the > > > > > needful. > > > > > >> This > > > > > >> > > way > > > > > >> > > > >> the > > > > > >> > > > >> > > > config > > > > > >> > > > >> > > > > > >>>> options > > > > > >> > > > >> > > > > > >>>>>> etc all remain the same(apart from > the > > > logger > > > > > >> > > > ofcourse). > > > > > >> > > > >> > > > > > >>>>>> > > > > > >> > > > >> > > > > > >>>>>> Let me know if any of it makes sense > :D > > > > > >> > > > >> > > > > > >>>>>> > > > > > >> > > > >> > > > > > >>>>>> Thanks! > > > > > >> > > > >> > > > > > >>>>>> Sagar. > > > > > >> > > > >> > > > > > >>>>>> > > > > > >> > > > >> > > > > > >>>>>> > > > > > >> > > > >> > > > > > >>>>>> On Thu, Feb 28, 2019 at 1:09 AM > Dinesh > > > Joshi > > > > > >> > > > >> > > > > > >>> <djos...@icloud.com.invalid > > > > > >> > > > >> > > > > > >>>>> > > > > > >> > > > >> > > > > > >>>>>> wrote: > > > > > >> > > > >> > > > > > >>>>>> > > > > > >> > > > >> > > > > > >>>>>>> > > > > > >> > > > >> > > > > > >>>>>>> > > > > > >> > > > >> > > > > > >>>>>>>> On Feb 27, 2019, at 10:41 AM, > Joseph > > > Lynch > > > > > < > > > > > >> > > > >> > > > > joe.e.ly...@gmail.com > > > > > >> > > > >> > > > > > > > > > > > >> > > > >> > > > > > >>>>>>> wrote: > > > > > >> > > > >> > > > > > >>>>>>>> > > > > > >> > > > >> > > > > > >>>>>>>> Vinay can confirm, but as far as I > am > > > > > aware we > > > > > >> > have > > > > > >> > > > no > > > > > >> > > > >> > > current > > > > > >> > > > >> > > > > > plans > > > > > >> > > > >> > > > > > >>>> to > > > > > >> > > > >> > > > > > >>>>>>>> implement audit logging to a table > > > > > directly, > > > > > >> but > > > > > >> > > the > > > > > >> > > > >> > > > > > implementation > > > > > >> > > > >> > > > > > >>> is > > > > > >> > > > >> > > > > > >>>>>>>> fully pluggable (like compaction, > > > > > compression, > > > > > >> > etc > > > > > >> > > > >> ...). > > > > > >> > > > >> > > Check > > > > > >> > > > >> > > > > out > > > > > >> > > > >> > > > > > >>> the > > > > > >> > > > >> > > > > > >>>>>>> blog > > > > > >> > > > >> > > > > > >>>>>>>> post [1] and documentation [2] > Vinay > > > wrote > > > > > for > > > > > >> > more > > > > > >> > > > >> > details, > > > > > >> > > > >> > > > but > > > > > >> > > > >> > > > > > the > > > > > >> > > > >> > > > > > >>>>>>> short > > > > > >> > > > >> > > > > > >>>>>>> > > > > > >> > > > >> > > > > > >>>>>>> +1. I am still curious as to why > you'd > > > want > > > > > to > > > > > >> > store > > > > > >> > > > >> audit > > > > > >> > > > >> > > log > > > > > >> > > > >> > > > > > >>> entries > > > > > >> > > > >> > > > > > >>>>>>> back in Cassandra? Depending on the > > > scale > > > > > it can > > > > > >> > > > >> generate a > > > > > >> > > > >> > > lot > > > > > >> > > > >> > > > > of > > > > > >> > > > >> > > > > > >>> load > > > > > >> > > > >> > > > > > >>>>>> and > > > > > >> > > > >> > > > > > >>>>>>> I think you'd end up in an infinite > > loop > > > > > because > > > > > >> > as > > > > > >> > > > >> you're > > > > > >> > > > >> > > > > > inserting > > > > > >> > > > >> > > > > > >>>> the > > > > > >> > > > >> > > > > > >>>>>>> audit log entry you'll generate a > new > > > one > > > > > and so > > > > > >> > on > > > > > >> > > > >> unless > > > > > >> > > > >> > > you > > > > > >> > > > >> > > > > > black > > > > > >> > > > >> > > > > > >>>> list > > > > > >> > > > >> > > > > > >>>>>>> audits to that table / keyspace. > > > > > >> > > > >> > > > > > >>>>>>> > > > > > >> > > > >> > > > > > >>>>>>> Ideally you'd insert this data into > > > > > >> ElasticSearch > > > > > >> > / > > > > > >> > > > >> Solr or > > > > > >> > > > >> > > > some > > > > > >> > > > >> > > > > > >>> other > > > > > >> > > > >> > > > > > >>>>>>> place that can be then used for > > > analytics or > > > > > >> > search. > > > > > >> > > > >> > > > > > >>>>>>> > > > > > >> > > > >> > > > > > >>>>>>> Dinesh > > > > > >> > > > >> > > > > > >>>>>>> > > > > > >> > > > >> > > > > > > > > > > >> > > > >> > > > > > > >> > > > > > > > > --------------------------------------------------------------------- > > > > > >> > > > >> > > > > > >>>>>>> To unsubscribe, e-mail: > > > > > >> > > > >> > dev-unsubscr...@cassandra.apache.org > > > > > >> > > > >> > > > > > >>>>>>> For additional commands, e-mail: > > > > > >> > > > >> > > dev-h...@cassandra.apache.org > > > > > >> > > > >> > > > > > >>>>>>> > > > > > >> > > > >> > > > > > >>>>>>> > > > > > >> > > > >> > > > > > >>>>>> > > > > > >> > > > >> > > > > > >>>> > > > > > >> > > > >> > > > > > >>>> > > > > > >> > > > >> > > > > > >>>> > > > > > >> > > > >> > > > > > > > > > >> > > > >> > > > > > >> > > > > --------------------------------------------------------------------- > > > > > >> > > > >> > > > > > >>>> To unsubscribe, e-mail: > > > > > >> > > > >> dev-unsubscr...@cassandra.apache.org > > > > > >> > > > >> > > > > > >>>> For additional commands, e-mail: > > > > > >> > > > >> > dev-h...@cassandra.apache.org > > > > > >> > > > >> > > > > > >>>> > > > > > >> > > > >> > > > > > >>>> > > > > > >> > > > >> > > > > > >>> > > > > > >> > > > >> > > > > > >>> -- > > > > > >> > > > >> > > > > > >>> Jon Haddad > > > > > >> > > > >> > > > > > >>> > > > > > >> > > > >> > > > > > > > > > > >> > > > >> > > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> > > > > > > > >> > > > >> > > > > > > >> > > > >> > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > S> > > > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rustyrazorblade.com&d=DwIFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g&m=vyXA1unA3gpHGCpKOfRurmET3jOHaV2bjs1mHVVsb2U&s=EDg90XhABktX19m4FaDHKIjFaU2YAHbXjeEGk7Jx6dk&e= > > > > > >> > > > >> > > > > > < > > > > > >> > > > >> > > > > > > > > > > >> > > > >> > > > > > > > > > >> > > > >> > > > > > > > > >> > > > >> > > > > > > > >> > > > >> > > > > > > >> > > > >> > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > S> > > > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rustyrazorblade.com&d=DwIFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g&m=vyXA1unA3gpHGCpKOfRurmET3jOHaV2bjs1mHVVsb2U&s=EDg90XhABktX19m4FaDHKIjFaU2YAHbXjeEGk7Jx6dk&e= > > > > > >> > > > >> > > > > > > > > > > > >> > > > >> > > > > > >>> twitter: rustyrazorblade > > > > > >> > > > >> > > > > > > > > > > > >> > > > >> > > > > > > > > > > >> > > > >> > > > > > > > > > > >> > > > >> > > > > > > > > > > >> > > > >> > > > > > > >> > > > > > > > > --------------------------------------------------------------------- > > > > > >> > > > >> > > > > > To unsubscribe, e-mail: > > > > > >> > > dev-unsubscr...@cassandra.apache.org > > > > > >> > > > >> > > > > > For additional commands, e-mail: > > > > > >> > > > dev-h...@cassandra.apache.org > > > > > >> > > > >> > > > > > > > > > > >> > > > >> > > > > > > > > > > >> > > > >> > > > > > > > > > >> > > > >> > > > -- > > > > > >> > > > >> > > > Jon Haddad > > > > > >> > > > >> > > > > > > > > >> > > > > > > >> > > > > > S> > > > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rustyrazorblade.com&d=DwIFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=VHsWqsWT2MoX5jRZ0xZfdAWZxBsrn5KzowynGYCJaXE&m=Hm3Dmun0qF2plgG5tk2ihs3UcucypmoQY1YEE2vpsuE&s=2McQRIC_i0mUwuhRKH3M0fWYXD78djaxqePdOqpgah8&e= > > > > > >> > > > >> > > > twitter: rustyrazorblade > > > > > >> > > > >> > > > > > > > > >> > > > >> > > > > > > > >> > > > >> > > > > > > >> > > > >> > > > > > >> > > > >> > > > > > >> > > > >> -- > > > > > >> > > > >> Jon Haddad > > > > > >> > > > >> > > > > > >> > > > > > > >> > > > > > S> > > > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rustyrazorblade.com&d=DwIFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=VHsWqsWT2MoX5jRZ0xZfdAWZxBsrn5KzowynGYCJaXE&m=Hm3Dmun0qF2plgG5tk2ihs3UcucypmoQY1YEE2vpsuE&s=2McQRIC_i0mUwuhRKH3M0fWYXD78djaxqePdOqpgah8&e= > > > > > >> > > > >> twitter: rustyrazorblade > > > > > >> > > > >> > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > -- > > > > > With best wishes, Alex Ott > > > > > Solutions Architect EMEA, DataStax > > > > > http://datastax.com/ > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > > > > > >
