Re: Audit logging to tables.

2019-04-09 Thread Sagar
Sure Chris. I would be watching for that ticket's closure. You had pointed out about this ticket already and what I had decided was, while this gets merged etc, I can work with the folks here and get a design out. Thanks! Sagar. On Tue, Apr 9, 2019 at 9:58 PM Chris Lohfink wrote: > You should

Re: Audit logging to tables.

2019-04-09 Thread Chris Lohfink
You should really build it on/wait for https://issues.apache.org/jira/browse/CASSANDRA-14629 (Can use a reviewer please btw!) then you can provide each entry of the audit log binlog as a partition in iterator format. Can use settings to find where binlog output is and go from there. Chris On

Re: Audit logging to tables.

2019-04-09 Thread Sagar
Thanks Jon. So, it will basically have to read the audit log file which can certainly be implemented. One of the things to consider is, if for example, we have some external system which is polling the audit log table on a given node every minute to get incremental audit logs. In that case, it

Re: Audit logging to tables.

2019-04-03 Thread Jon Haddad
The virtual table could read the data out of the audit log, just like it could read a hosts file or list the output of the ps command. On Wed, Apr 3, 2019 at 8:02 PM Sagar wrote: > > Thanks Alex. > > I was going through the implementation of Virtual tables thus far and the > data that we get

Fwd: Audit logging to tables.

2019-04-03 Thread Sagar
Thanks Alex. I was going through the implementation of Virtual tables thus far and the data that we get when we query against them seems to be more point in time like CachesTable or fairly static like Settings. Having said that, audit log's nature of data doesn't fall in either of the 2

Re: Audit logging to tables.

2019-03-31 Thread Alex Ott
Hi Sagar 3.x/4.x are versions for open source variant of drivers, while DSE versions are 1.x/2.x Description of this function is a thttps://docs.datastax.com/en/drivers/java/3.6/ Sagar at "Tue, 26 Mar 2019 22:12:56 +0530" wrote: S> Thanks Andy, S> This enhancement is in the datastax

Re: Audit logging to tables.

2019-03-26 Thread Sagar
Thanks Andy, This enhancement is in the datastax version and not in the apache cassandra driver? Thanks! Sagar. On Tue, Mar 26, 2019 at 3:23 AM Andy Tolbert wrote: > Hello > > 1) yes its local only. The driver by default does connect to each host > > though so its pretty trivial to have a

Re: Audit logging to tables.

2019-03-25 Thread Andy Tolbert
Hello 1) yes its local only. The driver by default does connect to each host > though so its pretty trivial to have a load balancing policy that you can > direct to specific hosts (this should probably be in the driver so people > dont have to keep reimplementing it). > The capability to target

Re: Audit logging to tables.

2019-03-25 Thread Sagar
Thanks Chris. I got caught up with a few things and couldn't reply back. So, I re-looked this again and I think virtual tables can be used for audit logging. Considering that they don't have any replication - so we won't be clogging the network with replication IO. In terms of storage, from what

Re: Audit logging to tables.

2019-03-06 Thread Chris Lohfink
1) yes its local only. The driver by default does connect to each host though so its pretty trivial to have a load balancing policy that you can direct to specific hosts (this should probably be in the driver so people dont have to keep reimplementing it). 2) yes, easiest way is to setup a

Re: Audit logging to tables.

2019-03-06 Thread Sagar
So, I went through the ticket for the creation of Virtual Tables(must say it was quite a long ticket spanning across 4 years). I see that there are a few tables created in the db.virtual package. These appear to be metrics related tables. Couple of questions here: 1) Do all the tables pertain

Re: Audit logging to tables.

2019-03-04 Thread Sagar
Right, Thanks Jonathan and Chris. Mean while, I would go through the 2 jira items to try and understand about virtual tables. Thanks! Sagar. On Tue, Mar 5, 2019 at 1:14 AM Jonathan Haddad wrote: > Sagar, > > There isn't going to be much in the way of docs, since it's brand new and > not

Re: Audit logging to tables.

2019-03-04 Thread Jonathan Haddad
Sagar, There isn't going to be much in the way of docs, since it's brand new and not really a public facing thing yet. As Chris pointed out, there's other work that would need to be done to work on virtual tables for large datasets. Jon On Mon, Mar 4, 2019 at 6:42 AM Chris Lohfink wrote: >

Re: Audit logging to tables.

2019-03-04 Thread Chris Lohfink
While you probably could put a virtual table wrapper over the binlogs, you would want to wait for something like https://issues.apache.org/jira/browse/CASSANDRA-14629 to get in so you would not OOM. The current virtual table implementation requires you have the entire result set to be returned at

Re: Audit logging to tables.

2019-03-04 Thread Sagar
Hi Jonathan, I couldn't find much literature on Virtual tables apart from this ticket: https://issues.apache.org/jira/browse/CASSANDRA-7622 Any insights would be helpful. Thanks! Sagar. On Sat, Mar 2, 2019 at 7:23 AM Jonathan Haddad wrote: > Instead of logging to tables, putting a virtual

Re: Audit logging to tables.

2019-03-01 Thread Jonathan Haddad
Instead of logging to tables, putting a virtual table around the audit / query logs might be an option. Same with the commit log for cdc On Fri, Mar 1, 2019 at 5:25 PM Sagar wrote: > Thanks all for the pointers. Really insightful. > > Subroto I think that’s part of the enterprise version but

Re: Audit logging to tables.

2019-03-01 Thread Sagar
Thanks all for the pointers. Really insightful. Subroto I think that’s part of the enterprise version but yeah even I have seen it. Again not sure of the performance implications. Sagar. On Sat, 2 Mar 2019 at 5:15 AM, Subroto Barua wrote: > Datastax version has an option to store audit info

Re: Audit logging to tables.

2019-03-01 Thread Subroto Barua
Datastax version has an option to store audit info to dse_audit.audit_log table; I do not know the performance impact since I use the file option Subroto > On Mar 1, 2019, at 9:40 AM, Jeremiah D Jordan > wrote: > > AFAIK the Full Query Logging binary format was already made more general in

Re: Audit logging to tables.

2019-03-01 Thread Jeremiah D Jordan
AFAIK the Full Query Logging binary format was already made more general in order to support using that format for the audit logging. -Jeremiah > On Mar 1, 2019, at 11:38 AM, Joshua McKenzie wrote: > > Is there a world in which a general purpose, side-channel file storage > format for

Re: Audit logging to tables.

2019-03-01 Thread Joshua McKenzie
Is there a world in which a general purpose, side-channel file storage format for transient things like this (hints, batches, audit logs, etc) could be useful as a first class citizen in the codebase? i.e. a world in which we refactored some of the hints-specific reader/writer code to be used for

Re: Audit logging to tables.

2019-02-28 Thread Jonathan Haddad
Agreed with Dinesh and Josh. I would *never* put the audit log back in Cassandra. This is extendable, Sagar, so you're free to do as you want, but I'm very opposed to putting a ticking time bomb in Cassandra proper. Jon On Thu, Feb 28, 2019 at 8:38 AM Dinesh Joshi wrote: > I strongly echo

Re: Audit logging to tables.

2019-02-28 Thread Dinesh Joshi
I strongly echo Josh’s sentiment. Imagine losing audit entries because C* is overloaded? It’s fine if you don’t care about losing audit entries. Dinesh > On Feb 28, 2019, at 6:41 AM, Joshua McKenzie wrote: > > One of the things we've run into historically, on a *lot* of axes, is that > "just

Re: Audit logging to tables.

2019-02-28 Thread Joshua McKenzie
One of the things we've run into historically, on a *lot* of axes, is that "just put it in C*" for various functionality looks great from a user and usability perspective, and proves to be something of a nightmare from an admin / cluster behavior perspective. i.e. - cluster suffering so you're

Re: Audit logging to tables.

2019-02-28 Thread Sagar
Thanks all for the pointers. @Joseph, I have gone through the links shared by you. Also, I have been looking at the code base. I understand the fact that pushing the logs to ES or Solr is a lot easier to do. Having said that, the only reason I thought having something like this might help is,

Re: Audit logging to tables.

2019-02-27 Thread Dinesh Joshi
> On Feb 27, 2019, at 10:41 AM, Joseph Lynch wrote: > > Vinay can confirm, but as far as I am aware we have no current plans to > implement audit logging to a table directly, but the implementation is > fully pluggable (like compaction, compression, etc ...). Check out the blog > post [1] and

Re: Audit logging to tables.

2019-02-27 Thread Joseph Lynch
Hi Sagar, Vinay can confirm, but as far as I am aware we have no current plans to implement audit logging to a table directly, but the implementation is fully pluggable (like compaction, compression, etc ...). Check out the blog post [1] and documentation [2] Vinay wrote for more details, but the

Re: Audit logging to tables.

2019-02-27 Thread Rahul Singh
I understand why you’d want it but it would add more data management to the database. Generally for logging you could consider putting into ELK and then it can be more queried on arbitrarily. On Feb 27, 2019, 12:42 PM -0500, Dinesh Joshi , wrote: > I don’t believe there is a plan to do it. If

Re: Audit logging to tables.

2019-02-27 Thread Dinesh Joshi
I don’t believe there is a plan to do it. If it were available in a table how would that help you? Dinesh > On Feb 27, 2019, at 9:32 AM, Sagar wrote: > > Hey All, > > While following some of the recent developments on Cassandra, I found the > new feature on Audit logging quite useful. > > I

Audit logging to tables.

2019-02-27 Thread Sagar
Hey All, While following some of the recent developments on Cassandra, I found the new feature on Audit logging quite useful. I wanted to understand is there any plan of pushing the audit logs to a table? Thanks! Sagar.