[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17344947#comment-17344947
]
Vinay Chella commented on CASSANDRA-12151:
--
{quote}going though this, this still does not have password obfuscation as
[~vinaykumarcse] mentioned (together with [~eanujwa]).
{quote}
That is correct [~stefan.miklosovic], CASSANDRA-16669 to track the same.
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Normal
> Labels: fqltool
> Fix For: 4.0, 4.0-alpha1
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17338977#comment-17338977 ] Stefan Miklosovic commented on CASSANDRA-12151: --- We observe passwords to be present in logs. > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Normal > Labels: fqltool > Fix For: 4.0, 4.0-alpha1 > > Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17338797#comment-17338797 ] Stefan Miklosovic commented on CASSANDRA-12151: --- Hi, going though this, this still does not have password obfuscation as [~vinaykumarcse] mentioned (together with [~eanujwa]). Is this something we are ok to go with in 4.0? Double checking this to be sure. I think we should obfuscate it. > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Normal > Labels: fqltool > Fix For: 4.0, 4.0-alpha1 > > Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030404#comment-17030404 ] Laxmikant Upadhyay commented on CASSANDRA-12151: Alpha-3 is about to release (voting has been done) . Regarding post alpha release you can track by referring to details in mail from Joshua regarding current 4.0 release status. [https://mail.google.com/mail/u/0/#search/label%3Acassandra+alpha/FMfcgxwGDDhhxLTfQtPlFsBhfXKNQSnB] > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Normal > Labels: fqltool > Fix For: 4.0 > > Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030119#comment-17030119 ] Dhawal Mody commented on CASSANDRA-12151: - [~laxmikant99] and team - when do you think cassandra 4.0 post-alpha will be released? > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Normal > Labels: fqltool > Fix For: 4.0 > > Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16800383#comment-16800383 ] Laxmikant Upadhyay commented on CASSANDRA-12151: [~Dhawal Mody] Audit logging is a new feature is Cassandra hence it will not be back-ported in 3.x. The release date of 4.0 has not been announced yet. > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Normal > Labels: fqltool > Fix For: 4.0 > > Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799223#comment-16799223 ] Dhawal Mody commented on CASSANDRA-12151: - [~jasobrown] [~eperott] - we are using 3.11.3 in production and we could use this option too - any chance it will be backported in Cassandra 3.11 ? Also when will Cassandra 4.0 release to the public? > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Normal > Labels: fqltool > Fix For: 4.0 > > Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622014#comment-16622014 ] Per Otterström commented on CASSANDRA-12151: [~mehdic], if you're interested we have created an audit-log plug-in for Cassandra 3.0 and 3.11 which we open sourced a while back. [https://github.com/Ericsson/ecaudit] There is a feature gap, but we're working to close that as much as possible. > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Major > Labels: fqltool > Fix For: 4.0 > > Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596703#comment-16596703 ] Jason Brown commented on CASSANDRA-12151: - [~mehdic] No, as it is a new feature, it goes into trunk/4.0. Further, 3.7 is an unsupported version, so even if we did consider a backport, it would have to go into the 3.11 series. > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Major > Labels: fqltool > Fix For: 4.0 > > Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589219#comment-16589219 ] Mehdi commented on CASSANDRA-12151: --- We are using Cassandra 3.7 and we could use this audit feature. Is there a way we can backport this patch to 3.7? > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Major > Fix For: 4.0 > > Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472417#comment-16472417 ] Stefan Podkowinski commented on CASSANDRA-12151: Pushed a commit at a6bf9c53531 to let users know that they can't expect any prepared statement values in the logs. Thanks for your contribution, [~vinaykumarcse]. > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Major > Fix For: 4.0 > > Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471856#comment-16471856 ] Jason Brown commented on CASSANDRA-12151: - Ugh, I forgot we did this, having -D jvm args for all the (directory) things. I'm not a fan, but apparently this is a thing we do. sgtm. > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Major > Fix For: 4.x > > Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471316#comment-16471316
]
Vinay Chella commented on CASSANDRA-12151:
--
{quote}
Why do you want this to be inconsistent? What is the argument for allowing
this one field, of all the audit log config, to have an additional mechanism
for configuration?
{quote}
Operators set the logging directory different from data directory, consistent
across the nodes and consolidated location for all logs (debug logs, system
logs, gc logs and audit logs), hence followed the same convention that is
followed for configuring other log locations and without extra step to
configure audit log location. However, considering everyone might not follow
the same or operator thinks it is additional/ different mechanism for auditlog,
this location can also be configured using yaml along with other auditlog
configs if they would like to, they don't necessarily need to follow the
convention of setting log location via an environment variable, they can choose
to set it via yaml too. We are consolidating all logs here by managing/
controlling the location via an environment variable, so we end up using the
environment variable. It just avoids yet another location(directory)
configuration to manage. Hopefully, this gives the background.
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468713#comment-16468713 ] Jason Brown commented on CASSANDRA-12151: - bq. Idea is to be inconsistent with rest of the project's log configuration process by setting them via -D jvm arg, *Why* do you want this to be inconsistent? What is the argument for allowing this one field, of all the audit log config, to have an additional mechanism for configuration? > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Major > Fix For: 4.x > > Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468526#comment-16468526
]
Vinay Chella commented on CASSANDRA-12151:
--
Hi Jason,
Yes, we can set audit_logs_dir either from -D jvm arg or from
{{cassandra.yaml}} - {{audit_logging_options::audit_logs_dir}}. Idea is to be
inconsistent with rest of the project's log configuration process by setting
them via {{-D jvm arg}}, hence naming is kept inconsistent with logs -
{{cassandra.logdir.audit}}. I have also updated the documentation for the same.
{quote}Operators cannot, in an obvious manner, adjust the following
AuditLogOptions options:
audit_logs_dir/block/max_queue_weight/max_log_size/roll_cycle. They are
basically hidden yaml options: you can set them, but you have to know they
exist. I understand these are more advanced options, but we should document
them either in the yaml or the doc page.
{quote}
Yes, they intentionally kept there for advanced configurations. Agree with you
on documenting them. Updated the documentation for this.
||*Branch*||*PR*||*uTest*||
|[trunk_CASSANDRA-12151|https://github.com/vinaykumarchella/cassandra/tree/trunk_CASSANDRA-12151]|[PR
for
trunk|https://github.com/vinaykumarchella/cassandra/pull/2/commits]|[!https://circleci.com/gh/vinaykumarchella/cassandra/tree/trunk_CASSANDRA-12151.svg?style=svg!|https://circleci.com/gh/vinaykumarchella/cassandra/tree/trunk_CASSANDRA-12151]|
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468457#comment-16468457
]
Jason Brown commented on CASSANDRA-12151:
-
On the whole, we're almost there. I've done a last-pass review, fixed a few
last dangling items (see below), and pushed a commit up to my branch. Just a
few last questions:
- Do we need the -D jvm arg to set {{AuditLogOptions#audit_logs_dir}}? This
kinda comes out of nowhere and matches nothing else in the audit log config.
- Operators cannot, in an obvious manner, adjust the following
{{AuditLogOptions}} options:
audit_logs_dir/block/max_queue_weight/max_log_size/roll_cycle. They are
basically hidden yaml options: you can set them, but you have to know they
exist. I understand these are more advanced options, but we should document
them either in the yaml or the doc page.
what i've done in latest commit:
- cleaned up yaml comments, and point users to the audit_log docs; else, it's
gonna real messy real quick
- trivial clean ups: NEWS.txt, formatting, white space, comments, removed
unused methods, fixed test method names in {{AuditLoggerTest}}
- reworked {{AuditLogFilter#create()}} to eliminate garbage collection of all
the {{HashSet}} s, which then we pass to {{ImmutableSet#of}}
- Added path checking in {{AuditLogManager}} to ensure when enabling either
FQL or audit logging, the path doesn't conflict with the other. This was easy
enough to do for the {{BinLog}}-related {{IAuditLogger}}, but I have no idea
how to get the directory or file from logback (when using {{FileAuditLogger}}).
Since that's a 'spare' implementation, I've punted for now.
- clarified {{AuditLogEntry#timestamp}} to allow {{Builder}} users to
explicitly set the value. This allows the FQL paths to be more in tune with
their original code (where it would get {{System#currentTimeMillis()}} instead
of using {{queryStartNanoTime}}).
Tests running.
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468212#comment-16468212
]
Vinay Chella commented on CASSANDRA-12151:
--
Thanks, [~jasobrown] for cleanup and fixing [~eperott] comments.
{quote}When using logback as backend, would it make sense to mark audit records
with a specific appender name such as "AUDIT" rather than
"FileAuditLoggerAppender". That way we can easily tell regular log messages
from audit log messages.
{quote}
Yes, certainly. However, AuditLog feature does not ship with appender
configurations. I see that "FileAuditLoggerAppender" is being referenced in the
documentation, have updated and pushed.
{quote}On a similar topic, rather than creating the AuditLogEntryCategory type,
the mapping in AuditLogEntryType and the kespace/scope of (I)AuditLogContext,
would it make sense to use the existing Permission type (SELECT, MODIFY,
CREATE...) and IResource (Data, Role, Function...). We could create a new
resource type to represent Connections (like connection/native,
connection/thrift, connection/jmx) which could be used for managing white-lists
for authentication.
{quote}
I don't think it is a good idea to piggyback on Permission type and IResource
to get the AuditLogType, that makes those 2 features tightly bound and it seems
like a hack rather than cleaner implementation. Also, binding them tightly
makes future extensions on those features tough to manage and we end up
separating eventually. So not sure, if that is a good idea to piggyback on 2
other different features to get the AuditLog needs.
\\
{quote}Sure, I understand we seek to close this ticket. I'm just a bit
concerned with the timing. If this ticket is merged as is and we take a cut for
4.0, then I assume we will have to stick to this way of configure audit logs
for some time.
{quote}
CQL grammar for managing audit log configurations is an interesting idea,
considering the changes needed at this point, hierarchical and composite
requirements that come with it, I agree with @Jason on exploring as a followup.
Please feel free to create followup JIRA on this.
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468087#comment-16468087 ] Jason Brown commented on CASSANDRA-12151: - pushed a few trivial clean up patches (includes some of the things [~eperott] reported), and running dtests and utests: ||12151|| |[branch|https://github.com/jasobrown/cassandra/tree/trunk_CASSANDRA-12151]| |[utests dtests|https://circleci.com/gh/jasobrown/workflows/cassandra/tree/trunk_CASSANDRA-12151]| || bq. should we create some dtests for this as well? I thought about that,as well. However, as this feature is not distributed, we can test all the functionality via unit tests. In fact, [~vinaykumarcse] already has a bunch of unit tests already in this patch. I'm going to give this patch one last major review, and will see if we need more tests. > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Major > Fix For: 4.x > > Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467079#comment-16467079 ] Per Otterström commented on CASSANDRA-12151: bq. This is an intersting idea. However, I'm reticent to introduce CQL grammar changes this late into this ticket. As a followup, I think this could be worth exploring. Sure, I understand we seek to close this ticket. I'm just a bit concerned with the timing. If this ticket is merged as is and we take a cut for 4.0, then I assume we will have to stick to this way of configure audit logs for some time. bq. While there's some conceptual overlap, I don't think it's a good idea to try to force the concepts of 'audit logging' and 'auth' onto one another here. I believe it depends on the use case for doing audits. In some cases it would make perfect sense to have these two concepts in sync. E.g. I want to grant a user to have SELECT permission on a specific table. Then I would grant the same user to have NOAUDIT for SELECT on the very same table. Shoudl the user try to read somethign else, or write to that table, and audit entry would be created. In a case like this it is very helpful for the admin to use the same concepts for these two "permissions". To verify accuracy, should we create some dtests for this as well? > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Major > Fix For: 4.x > > Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466880#comment-16466880 ] Jason Brown commented on CASSANDRA-12151: - bq. Would it not make sense to maintain audit whitelist permissions next to all other permissions in Cassandra? This is an intersting idea. However, I'm reticent to introduce CQL grammar changes this late into this ticket. As a followup, I think this could be worth exploring. bq. would it make sense to use the existing Permission type (SELECT, MODIFY, CREATE...) and IResource (Data, Role, Function...)? While there's some conceptual overlap, I don't think it's a good idea to try to force the concepts of 'audit logging' and 'auth' onto one another here. [~eperott]'s other comments I agree with. > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Major > Fix For: 4.x > > Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466440#comment-16466440 ] Per Otterström commented on CASSANDRA-12151: I've been following this ticket with some interest. Spent some time to review the patch, didn't try things out yet. When using logback as backend, would it make sense to mark audit records with a specific appender name such as "AUDIT" rather than "FileAuditLoggerAppender". That way we can easily tell regular log messages from audit log messages. Having audit include/exclude filters in the yaml file seem a bit impractical when you want to update it. I believe there are quite a few similarities with the permissions-on-resources model in Cassandra. Would it not make sense to maintain audit whitelist permissions next to all other permissions in Cassandra? Example "GRANT NOAUDIT ON KEYSPACE myks TO myuser". I've been experimenting with a similar approach (storing white-lists in the role option field) in our internal audit log plugin and it looks promising. On a similar topic, rather than creating the AuditLogEntryCategory type, the mapping in AuditLogEntryType and the kespace/scope of (I)AuditLogContext, would it make sense to use the existing Permission type (SELECT, MODIFY, CREATE...) and IResource (Data, Role, Function...). We could create a new resource type to represent Connections (like connection/native, connection/thrift, connection/jmx) which could be used for managing white-lists for authentication. Why always mute audit logs for the system keyspaces? IMO, we should make less assumptions on the use cases and let this be configurable like all other keyspaces. The AuditLogManager contain a few redundant null checks and isAuditintEnabled()-checks. The BinLogAuditLogger declare "LoggerFactory.getLogger(FullQueryLogger.class)" > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Major > Fix For: 4.x > > Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465523#comment-16465523
]
Vinay Chella commented on CASSANDRA-12151:
--
Hi [~jasobrown],
Thanks for the changes. Yes, extracting `BinLogAuditLogger` is a cleaner way to
do this.
Updated changeset with minor fixes to pass tests, moved the {{category}} in
AuditLogEntryType to enum.
Filtered unintended messages to FQL.
{quote}Why did you remove the reloadFilters functionality?
{quote}
Updated documentation regarding "ReloadFilters" functionality.
Yes, circleci utests are green, trying to get paid account to run dtests.
\\
||*Branch*||*PR*||*uTest*||
|[trunk_CASSANDRA-12151|https://github.com/vinaykumarchella/cassandra/tree/trunk_CASSANDRA-12151]|[PR
for
trunk|https://github.com/vinaykumarchella/cassandra/pull/2/commits]|[!https://circleci.com/gh/vinaykumarchella/cassandra/tree/trunk_CASSANDRA-12151.svg?style=svg!|https://circleci.com/gh/vinaykumarchella/cassandra/tree/trunk_CASSANDRA-12151]|
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458567#comment-16458567
]
Jason Brown commented on CASSANDRA-12151:
-
After rereading this patch, I've merged the FQL and AuditLog code paths as they
do the same thing from the outside: record user-initiated events. To that end,
I've moved the management of {{FullQueryLogger}} under {{AuditLogManager}},
thus making {{AuditLogManager}} the hub for logging these actions. Further,
instead of {{BinAuditLogger}} inherit from {{FullQueryLogger}} (which never
felt quite right), I've pulled the shared componentry into a base class
({{BinLogAuditLogger}}), and {{BinAuditLogger}} / {{FullQueryLogger}} inherit
from that.
I refactored {{AuditLogEntry}} to use the builder pattern for construction as
all the {{static #getLogEntry()}} with the {{set*()}} methods was rather
confusing. Further, we're using the builder pattern elsewhere in the code.
{{#toString()}} is typically used for debugging, so I'd prefer a different
method name - I used {{#getLogString()}}. If you want the debug dump to have
the same output, you can have {{toString()}} call that renamed function.
We need an extra filter in {{AuditLogManager#log(AuditLogEntry)}} to make sure
we don't send unintended messages to FQL. Can you figure out what that is? I'm
not sure if we can simply use the entry's {{AuditLogEntryType}} or the type's
category.
Why did you remove the reloadFilters functionality?
I feel like the {{category}} in AuditLogEntryType should be an enum, as well.
wdyt?
I haven't looked at the circleci results yet for utests/dtests, but let's make
sure this is right path forward first.
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456004#comment-16456004
]
Vinay Chella commented on CASSANDRA-12151:
--
Hi [~jasobrown],
Implemented {{nodetool enableauditlog}} and {{disableauditlog}} commands,
updated documentation for the same. Fixed whitespaces in rest of the changeset,
also fixed the tests.
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16446557#comment-16446557
]
Vinay Chella commented on CASSANDRA-12151:
--
Thanks, [~jasobrown], for reviewing it and making changes, sure I will take
care of tests.
{quote}
I've asked Vinay to add some nodetool commands for enabling/disabling auditlog
as well (or, at least, to seriously think about it).
{quote}
I agree, will add nodetool commands for enabling/disabling auditlog to help
improve the usability at node level.
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16446271#comment-16446271
]
Jason Brown commented on CASSANDRA-12151:
-
Thanks to [~djoshi3] and [~spo...@gmail.com] for doing the first passes of
review here, I'll take it the rest of the way.
[~vinaykumarcse] and I spoke offline about some things, and on the whole we are
in the right direction. As I started reviewing the patch, the whitespace
inconsistencies nagged at me, so I went ahead and fixed those, as well as some
random code comments. Further, I made {{AuditLogFilters}} atomic and final, and
no longer a singleton; a volatile reference is maintained in
{{AuditLogManager}}. I also moved getLogEntry methods from {{AuditLogManager}}
to {{AuditLogEntry}} as they seemed more apropos there. Code available
[here|https://github.com/jasobrown/cassandra/tree/trunk_CASSANDRA-12151].
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443377#comment-16443377
]
Vinay Chella commented on CASSANDRA-12151:
--
Hi [~spo...@gmail.com],
Sorry, was away from work for last few days. If we are filtering users, to log
any of these failures/ failed attempts, we can include {{system}} user in
include filters.
{quote}question regarding how we should go on with prepared statements. Cause I
can't see how we can get away with not logging these, if we want to address any
compliance or security use cases.
{quote}
As you also suggested on this thread previously, either filtering based on log
entry types or ParameterizedClass for logger should address this concern. I
will open followup JIRAs as soon as we close this PR to keep the changeset
moderate at this moment.
Sounds like a plan?
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435587#comment-16435587 ] Stefan Podkowinski commented on CASSANDRA-12151: I must have missed that log statement when filtering by user. Its a bit strange that failed login attempts don't get logged, if you filter by a particular user. But that's probably more a minor issue compared to the the question regarding how we should go on with prepared statements. Cause I can't see how we can get away with not logging these, if we want to address any compliance or security use cases. Any ideas? > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Major > Fix For: 4.x > > Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435109#comment-16435109
]
Vinay Chella commented on CASSANDRA-12151:
--
Hi [~spo...@gmail.com],
Updated the PR with comments provided. {{SYSTEM_USER}} is used in our internal
backport for thrift code path, I removed it on trunk. I intended to use
{{IAuditLogger.error()}} initially in error scenarios, but I ended up using
{{IAuditLogger.log()}}, hence removed it, if any use case comes up, we can add
it back. Fixed and added more test cases for {{AuditLogFilter.isFiltered()}} to
address the scenarios you requested.
{quote}Username will not be provided for failed authentications
{quote}
Below is the sample audit statement on failed authentications, username is
provided as part of the audit log. Are you referring to something else?
{code:java}
INFO [Native-Transport-Requests-1] 2018-04-12 00:37:20,284
FileAuditLogger.java:35 -
user:system|host:127.0.0.1:7000|source:/127.0.0.1|port:64481|timestamp:1523518640284|type:LOGIN_ERROR|category:AUTH|operation:LOGIN
FAILURE; Provided username vchella and/or password are incorrect
{code}
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431899#comment-16431899
]
Stefan Podkowinski commented on CASSANDRA-12151:
The native transport integration for diagnostic events is basically just
pushing events one by one over the control connection to the subscribed client.
It's not really designed as a generic, fully scalable server-side data push
solution. There are also no delivery guarantees, as it's really just supposed
for debugging and analysis and not for implementing any control instances on
top if it. The use case I have in mind is to have 1-2 clients subscribing to
some kind of event, either ad-hoc or constantly running in the background. But
I don't really see any use case for having a large fanout of e.g. compaction
events. For that, the solution proposed in CASSANDRA-13459 should be
sufficient. But we should probably discuss further details there, as it's
slightly off-topic for this ticket.
{quote}I think specifying a shell script is probably OK although if someone
specifies the script we should run it immediately once Chronicle rolls the
file. Also if the script is specified we probably shouldn't delete artifacts.
{quote}
I've created a new ticket (CASSANDRA-14373) for this, as it's not strictly an
auditing feature.
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431296#comment-16431296 ] Ariel Weisberg commented on CASSANDRA-12151: Adding a way for clients to subscribe to events is great, but there are couple of questions that brings up. How does backpressure work? We have to have a convincing story for what happens so that slow or unavailable clients can't consume unbounded memory. The other nice to have is what happens if a client disconnects and reconnects? Can it get the events that it missed? This ties into being able to consume on disk artifacts like the BinLog. Do the diagnostic events have enough information in them that you can tell where in the stream of events for a particular node you are? Just enough to facilitate at least once delivery with duplicates dropped. Are diagnostic events always really going to be used for diagnostic purposes? I'm just questioning the name. Maybe it should just be SUBSCRIBE and maybe only some of the more problematic things should be locked behind config options. And then we have to think about what happens with large numbers of subscribers although for V1 it could just be a sharp edge. In terms of using the BinLog as more of a store of record. We need to figure out how crashes and restarts are going to be handled. I don't recall exactly what happens, but I suspect that chronicle is just going to leave behind the old files and start a new one for appending so we need to come back and process the old files on startup. I think specifying a shell script is probably OK although if someone specifies the script we should run it immediately once Chronicle rolls the file. Also if the script is specified we probably shouldn't delete artifacts. As these things start to get more complex how are we going to change the YAML and fqltool to be the ri Also this entire change set is starting to get really large. * Generating whole new classes of events, multiple targets for logging * Modifications to the existing BinLog to add new capabilities like using it as a store of record, running a users provided script * Wire protocol changes and a new way to subscribe to have the server push stuff Would be nice to discuss and review these in more isolation. GL Dinesh! > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Major > Fix For: 4.x > > Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431160#comment-16431160
]
Nate McCall commented on CASSANDRA-12151:
-
{quote}I've now managed to update CASSANDRA-13668 by implementing IAuditLogger
and expose audit events as diagnostic events via native transport to subscribed
clients. Went pretty much as expected and seems to work fine.
{quote}
[~spo...@gmail.com] That's a fantastic idea! Thank you.
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430448#comment-16430448
]
Stefan Podkowinski commented on CASSANDRA-12151:
I've now managed to update CASSANDRA-13668 by implementing IAuditLogger and
expose audit events as diagnostic events via native transport to subscribed
clients. Went pretty much as expected and seems to work fine.
Smaller issues I've came across:
* AuditLogUtil doesn't look very useful. DEFAULT_SOURCE is only used once,
SYSTEM_USER not at all.
* Do we really want to keep IAuditLogger.error()? Please provide a description
of the log/error semantics in context with audit logging for possible
subclasses, or get rid of error().
* AuditLogFilter.isFiltered(): proposed logic will ignore includeSet if
excludeSet is provided (may not make sense to do so, but not strictly forbidden
by cassandra.yaml either), e.g. exclude(A), include(A,B,C) should only have B,C
pass
There are also a couple of limitations:
* Username will not be provided for failed authentications
* Bound values will not get logged for prepared statements
I haven't found a quick way to work around these, but being able to avoid the
audit log by using prepared statements, is something we have to address. It's
probably not going to be that much of an issue for my use case logging ad-hoc
commands for regular users, once we have CASSANDRA-8303 and can disable
prepared statement for them. But for logging all activity for application
users, I don't know.
[~laxmikant99]
{quote}Can we have a configurable property like exitOnAuditFailure ? This
fulfills requirement of strict auditing .. I mean in case auditing fails for a
db operation, then the operation should not get executed.
{quote}
Any logging to the BinLogger should block by default. But it doesn't exit the
JVM.
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428102#comment-16428102 ] Laxmikant Upadhyay commented on CASSANDRA-12151: Can we have a configurable property like exitOnAuditFailure ? This fulfills requirement of strict auditing .. I mean in case auditing fails db operation should not get executed. > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Major > Fix For: 4.x > > Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427318#comment-16427318 ] Vinay Chella commented on CASSANDRA-12151: -- Thank you [~spo...@gmail.com] > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Major > Fix For: 4.x > > Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426915#comment-16426915
]
Stefan Podkowinski commented on CASSANDRA-12151:
I was just giving the BinLogger a try and it looks like the way the bin log
files are created, should make it possible to archive them before they get
deleted. The most simple way would probably to run a rsync job for pulling the
latest chronicle bin logs from the nodes. The timestamp based naming of the
files should make this trivial.
Another option would be to allow the user to configure an archival script that
will be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin
log, just as we do in {{CommitLogArchiver}}. WDYT [~aweisberg]?
I'm going to write a custom implementation tomorrow and will get back with some
more feedback.
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16422918#comment-16422918
]
Vinay Chella commented on CASSANDRA-12151:
--
[~spo...@gmail.com],
{quote}We should not provide solutions that we know will have significant
performance issues on busy production systems. I don’t mind keeping the
FileAuditLogger class, as it’s really trivial. But please remove references to
it from cassandra.yaml and the corresponding entries in logback.xml (don’t like
to have an empty audit.log file on all nodes either). Let’s nudge users to use
to BinAuditLogger right away as the recommended solution.
{quote}
Yes, I took care of this. Removed FileAuditLogger references from
cassandra.yaml and also code defaults
{quote}If you don’t think adding an option like include_auditlog_types should
be necessary, that’s fine. But then let me use my own implementation for
filtering logs, if my requirements are different. This means that I’d have to
be able to specify additional parameters (e.g. custom filtering options) along
with the class name for my implementation. So I'd suggest to use
ParameterizedClass for the logger in cassandra.yaml to make that easily
possible.
Looking at IAuditLogger and thinking about how to filter log events makes me a
bit worried about the design in general there. We keep generating AuditLogEntry
instances and create unnecessary garbage, even if we’re only interested in some
specific entry types. Maybe we should move filtering either into the
IAuditLogger implementation or make it possible to use a custom AuditLogFilter
as well (IAuditLogger.getLogFilter() ?).
{quote}
Agreed, just to be clear from my previous reply - I am not against adding a new
filter (include_auditlog_types) or extending the interface, I would like to get
the initial simple version out and iterate on it rather than increasing the
changeset to be too heavy.
{quote}Just looking at AuditLogFilter makes me also think that the isFiltered
logic should be reconsidered, as null values will cause entries to always pass,
even if the include set is not empty.
{quote}
Sure, considered null entries also. Now, {{isFiltered}} logic takes care of
null values when input set is not empty.
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420963#comment-16420963
]
Dinesh Joshi commented on CASSANDRA-12151:
--
[~spo...@gmail.com] The {{BinLog}} does what I suggested. It deletes old files
once you exceed the allocated storage quota. From my reading, it seems that it
segments logs on the basis of time (defaults to hourly).
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420389#comment-16420389
]
Stefan Podkowinski commented on CASSANDRA-12151:
We should not provide solutions that we know will have significant performance
issues on busy production systems. I don’t mind keeping the FileAuditLogger
class, as it’s really trivial. But please remove references to it from
cassandra.yaml and the corresponding entries in logback.xml (don’t like to have
an empty audit.log file on all nodes either). Let’s nudge users to use to
BinAuditLogger right away as the recommended solution.
If you don’t think adding an option like include_auditlog_types should be
necessary, that’s fine. But then let me use my own implementation for filtering
logs, if my requirements are different. This means that I’d have to be able to
specify additional parameters (e.g. custom filtering options) along with the
class name for my implementation. So I'd suggest to use ParameterizedClass for
the logger in cassandra.yaml to make that easily possible.
Looking at IAuditLogger and thinking about how to filter log events makes me a
bit worried about the design in general there. We keep generating AuditLogEntry
instances and create unnecessary garbage, even if we’re only interested in some
specific entry types. Maybe we should move filtering either into the
IAuditLogger implementation or make it possible to use a custom AuditLogFilter
as well (IAuditLogger.getLogFilter() ?).
Just looking at AuditLogFilter makes me also think that the isFiltered logic
should be reconsidered, as null values will cause entries to always pass, even
if the include set is not empty.
{quote}How do you suggest we approach this problem? We cannot keep the audit
log files around indefinitely. Perhaps we can specify a disk quota for audit
log files in the configuration? We can expose this setting in cassandra.yaml?
{quote}
How does this work with BinLogger? I haven't looked at that part in detail yet.
Does it overwrite old entries automatically? How would you suggest users should
archive logs from there?
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420323#comment-16420323
]
Dinesh Joshi commented on CASSANDRA-12151:
--
Hey [~vinaykumarcse], thank you for making the code changes. Minor nit on the
whitespacing but other than that LGTM.
Hey [~spo...@gmail.com],
{quote}Rotating out log files by simply deleting them is probably also not what
you'd expect from a auditing solution.
{quote}
How do you suggest we approach this problem? We cannot keep the audit log files
around indefinitely. Perhaps we can specify a disk quota for audit log files in
the configuration? We can expose this setting in cassandra.yaml?
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419652#comment-16419652
]
Vinay Chella commented on CASSANDRA-12151:
--
[~spo...@gmail.com] ,
{quote}I'd not use the logback logger for query logging in production. As shown
recently in CASSANDRA-14318, the performance impact is significant and your
benchmark results show that as well. Rotating out log files by simply deleting
them is probably also not what you'd expect from an auditing solution.
{quote}
Agreed, yes, BinAuditLogger is the default for obvious reasons based on my perf
numbers, but if any users are willing to use log back based AuditLogger thereby
willing to compromise on performance while benefiting on easy readability, they
can by changing the yaml configuration.
{quote}NIT: check logger.isEnabled before toString in FileAuditLogger
{quote}
FileAuditLogger::log method is called only when AuditLogger is enabled, do you
think the redundant check is needed inside the IAuditLogger implementations as
well?
{quote}But maybe we can keep the logback logger for some simple auth related
logging and make it also useful for users who would not enable auditing in
first place. How about enabling the FileAuditLogger by default and let it
create an auth.log, which would log all failed login attempts? Maybe add a
comment on how to enable successful attempts as well.
{quote}
Logging just auth failure attempts can be achieved by either the design idea
that you brought up or new feature to filter {{include_auditlog_types}}. But
not sure, if everyone would like or need auth failures attempts by default.
I also see good design ideas in terms of {{multiple implementations in
parallel}} and feature requests {{include_auditlog_types}}, however, do you
agree with me that it is a {{nice to have}} but not {{must to have}} feature
considering the scope of the first iteration, to keep the initial
implementation simple, iterate faster and changeset smaller? If you do, I can
create followup JIRAs for the ideas that you proposed.
Hi [~chovatia.jayd...@gmail.com] .
{quote}Currently if audit is enabled then it will log all types of statements
DDL|DML|DCL, people will have different requirements for logging (say only log
DCL OR DCL + DDL etc.) given DML would generate huge volume of data, so I think
we need to give better control to users as to what they want to log
{quote}
Correct, the current design enables users to have control of what to log since
the use cases could be different (eg., DCL or DCL+DDL etc.,) using yaml
configurations
{quote}probably at what frequency as well.
{quote}
I don't know as a database when it promises for audit logging for compliance if
it does based on some probability. However, if you have such one of use case
(e.g., audit logging for debugging needs instead of compliance needs), you can
implement {{IAuditLogger}} or extend one of existing loggers to have
{{FrequencyBasedAuditLogger}} which logs based on some probability.
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419564#comment-16419564
]
Jaydeepkumar Chovatia commented on CASSANDRA-12151:
---
Hi [~vinaykumarcse]
Currently if audit is enabled then it will log all types of statements
{{DDL|DML|DCL}}, people will have different requirements for logging (say only
log {{DCL }} OR {{DCL + DDL}} etc.) given {{DML}} would generate huge volume
of data, so I think we need to give better control to users as to what they
want to log and probably at what frequency as well.
Jaydeep
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418918#comment-16418918
]
Stefan Podkowinski commented on CASSANDRA-12151:
I'd not use the logback logger for query logging in production. As shown
recently in CASSANDRA-14318, the performance impact is significant and your
benchmark results show that as well. Rotating out log files by simply deleting
them is probably also not what you'd expect from a auditing solution. But maybe
we can keep the logback logger for some simple auth related logging and make it
also useful for users who would not enable auditing in first place. How about
enabling the {{FileAuditLogger}} by default and let it create an {{auth.log}},
which would log all failed login attempts? Maybe add a comment on how to enable
successful attempts as well.
Looks like this won't be possible with the current {{included_categories}}
filtering, which is based on the DDL, DML, .. categories. I'd suggest to make
the filter work for both the category and actual AuditLogEntryType (SELECT,
UPDATE, DELETE,..).
Full query logging should be done using the BinLogger or a custom
implementation. It would be nice to be able to use mutliple implementations in
parallel, in case we want to enable FileAuditLogger by default.
NIT: check logger.isEnabled before toString in FileAuditLogger
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416952#comment-16416952
]
Vinay Chella commented on CASSANDRA-12151:
--
Hi [~djoshi3] [~jolynch],
Thanks for the review, implemented review suggestions provided and pushed those
changes in PR.
\\
||[branch|https://github.com/vinaykumarchella/cassandra/tree/trunk_CASSANDRA-12151]||
|[PR for trunk|https://github.com/vinaykumarchella/cassandra/pull/2/commits]|
|[circleci|https://circleci.com/gh/vinaykumarchella/cassandra/tree/trunk_CASSANDRA-12151]|
\\
{quote}Initialization can be simplified like this - excludedUsers =
ImmutableSet.of(DatabaseDescriptor.getAuditLoggingOptions().excluded_users.split(","));
{quote}
{{loadFilters()}} are designed so that filter values can be loaded during the
startup as well as JMX call to reload the filters
{quote}AuditLogFilter:: isFiltered - You can return directly at line 137 and
get rid of isExcluded variable You can get rid of isIncluded as well and return
at line 148. Let the control fall through to line 153 and it will return false.
Furthermore, If you check the sets during initialization and ensure that
they're mutually exclusive, your logic simplifies to if
(includedSet.contains(object)) return false; else if
(excludedSet.contains(object)}) return true;
{quote}
I went with the suggestions provided in former part to simplify the logic in
{{isFiltered}} rather than mutual exclusion at initialization part.
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412272#comment-16412272
]
Dinesh Joshi commented on CASSANDRA-12151:
--
Hey [~vinaykumarcse],
# {{AuditLogEntry.host}} - this should be final. You should use
{{FBUtilities::getBroadcastAddressAndPort}} instead for completeness sake.
# {{AuditLogEntry}} - variables {{source, srcPort, user, type}} could be made
final
# {{AuditLogEntry}} - line 47 Do you need to create a new Date object for each
entry? How about using {{System.currentTimeMillis()}} instead?
# {{AuditLogFilter.logger}} - this needs to be private
# {{AuditLogFilter::loadFilters}} - consider using {{private volatile
ImmutableSet}} instead of {{AtomicReferences}} to mutable sets. The class
invariant should be that once constructed none of the sets should be null.
# Initialization can be simplified like this - {{excludedUsers =
ImmutableSet.of(DatabaseDescriptor.getAuditLoggingOptions().excluded_users.split(","));}}
# Make {{AuditLogFilter}} constructor private
# {{AuditLogFilter}}'s constructor - Do not call a public overridable method
{{loadFilters}} in a constructor. See:
[https://wiki.sei.cmu.edu/confluence/display/java/MET05-J.+Ensure+that+constructors+do+not+call+overridable+methods]
# {{AuditLogFilter:: isFiltered}} - You can return directly at line 137 and
get rid of {{isExcluded}} variable You can get rid of {{isIncluded}} as well
and return at line 148. Let the control fall through to line 153 and it will
return false. Furthermore, If you check the sets during initialization and
ensure that they're mutually exclusive, your logic simplifies to {{if
(includedSet.contains(object)) return false; else if
(excludedSet.contains(object)}) return true;}}
# {{AuditLogManager.logger}} - this needs to be private
# {{AuditLogManager::logError}} - get rid of this method as it is not used
anywhere
# {{AuditLogUtil.DEFAULT_SOURCE}} - this should be {{0.0.0.0/0}} for IPv4 and
{{::/0}} for IPv6. This are used to represent "unknown or unspecified"
addresses.
# {{IAuditLogContext.AuditLogContext}} - please make all the class variables
final.
# {{CreateTypeStatement, AlterTypeStatement, DropTypeStatement}} all have
{{getStringTypeName}}. Since it's not being used anywhere except inside the
class, you can get rid of it.
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410693#comment-16410693 ] Joseph Lynch commented on CASSANDRA-12151: -- I took a look at the patch and it looks good to me. Comments left on the PR, but I like that the interface is pluggable and future implementations can do what they like (percentage based, remote/network logs, different output formats, etc). Should we do some dtests and ccm multi node testing locally? > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Major > Fix For: 4.x > > Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403146#comment-16403146
]
Vinay Chella commented on CASSANDRA-12151:
--
[~djoshi3]
Implemented all the code reviews comments provided in this JIRA thread as well
as Github PR. Except one below
{quote}
Consider refactoring your code to add a netty handler that invokes an auditing
interface. The advantage of this approach would be that, when audit logging is
disabled, you can take this handler out of the netty pipeline. This way there
is zero performance impact when the audit is disabled. You can define a
IAuditLogger interface that has sufficient contextual information to log all
queries. This will help make the audit logging implementation pluggable.
{quote}
I am creating a follow-up JIRA to discuss the more details on this.
On a high level, this changeset includes following changes
# Extended and reused FullQueryLogger in logging audit events
# Combined and Simplified FQL and AuditLog entry points in the request path
# AuditLogEntryType::allStatementsMap - Instead of creating an explicit map of
statements, type of statement is being added to the actual class itself. This
makes new statements easy to manage
# AuditLogFilter::loadFilters - Simplified filter loading logic, easy to add
new filters if needed
# CQL query auditing can now be filtered on user level.
# Added documentation in the doc folder
# Removed ConsistencyLevel in logging details
# Added more test cases
# Implemented code review comments provided in this JIRA as well as Github PR
\\
||[branch|https://github.com/vinaykumarchella/cassandra/tree/trunk_CASSANDRA-12151]||
|[PR for trunk|https://github.com/vinaykumarchella/cassandra/pull/2/commits]|
|[circleci|https://circleci.com/gh/vinaykumarchella/cassandra/tree/trunk_CASSANDRA-12151]|
\\
We ran cassandra stress test with this patch and attached stress test results.
Here is the high level summary
Note: Below tests are run on AWS i2.2xl instance.
\\
{{cass-stress cmd: write n=100 -rate threads=10 -graph
file=CASSANDRA_12151-benchmark.html}}
||WRITE - Test Suite||Throughput||Latency Mean||Latency 95th||Latency 99th||
|trunk|13,925 op/s|0.7 ms|1.1 ms|1.7 ms|
|CASSANDRA-12151:Disabled AuditLog|14,422 op/s|0.7 ms|1.1 ms|1.6 ms|
|CASSANDRA-12151:FQL based AuditLog with Sync|13,372 op/s|0.7 ms|1.2 ms|1.7 ms|
|CASSANDRA-12151:FQL based AuditLog with Async|12,908 op/s|0.8 ms|1.2 ms|1.9 ms|
|CASSANDRA-12151:SLF4j based AuditLog|10,520 op/s|0.9 ms|1.6 ms|2.4 ms|
\\
{{cass-stress cmd: mixed n=100 -rate threads=10 -graph
file=CASSANDRA_12151-benchmark.html}}
||MIXED - Test Suite||Throughput||Latency Mean||Latency 95th||Latency 99th||
|trunk|12,939 op/s [READ: 6,494 op/s, WRITE: 6,444 op/s]|0.7 ms [READ: 0.8 ms,
WRITE: 0.7 ms]|1.2 ms [READ: 1.3 ms, WRITE: 1.2 ms]|1.7 ms [READ: 1.8 ms,
WRITE: 1.7 ms]|
|CASSANDRA-12151: Disabled AuditLog|12,840 op/s [READ: 6,421 op/s, WRITE: 6,419
op/s]|0.8 ms [READ: 0.8 ms, WRITE: 0.7 ms]|1.2 ms [READ: 1.3 ms, WRITE: 1.2
ms]|1.8 ms [READ: 1.8 ms, WRITE: 1.7 ms]|
|CASSANDRA-12151: FQL based AuditLog with Sync|10,932 op/s [READ: 5,452 op/s,
WRITE: 5,481 op/s]|0.9 ms [READ: 1.0 ms, WRITE: 0.8 ms]|1.5 ms [READ: 1.6 ms,
WRITE: 1.4 ms]|2.3 ms [READ: 2.4 ms, WRITE: 2.1 ms]|
|CASSANDRA-12151: FQL based AuditLog with Async|11,146 op/s [READ: 5,565 op/s,
WRITE: 5,581 op/s]|0.9 ms [READ: 0.9 ms, WRITE: 0.8 ms]|1.5 ms [READ: 1.5 ms,
WRITE: 1.4 ms]|2.2 ms [READ: 2.2 ms, WRITE: 2.1 ms]|
|CASSANDRA-12151: SLF4j based AuditLog|9,764 op/s [READ: 4,883 op/s, WRITE:
4,882 op/s]|1.0 ms [READ: 1.0 ms, WRITE: 1.0 ms]|1.7 ms [READ: 1.7 ms, WRITE:
1.6 ms]|2.5 ms [READ: 2.6 ms, WRITE: 2.4 ms]|
\\
Looking at the results, with AuditLog feature disabled, there appears to be no
measurable difference in performance. FQL appears to have little or no overhead
in WRITE only workloads, and a minor overhead in MIXED workload. SLF4J appears
to have minor regressions in both workloads (with mixed slightly worse).
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389656#comment-16389656
]
Stefan Podkowinski commented on CASSANDRA-12151:
To further summarize, let me quote [~jolynch]'s list of collected use cases
once more and add another use case highlighted with bold letters.
{quote} # Logging for security
# Logging for business accounting compliance (e.g. SOX).
# Logging for monetary transaction compliance (e.g. PCI).
# *Logging for data protection compliance (GDPR)*.
# Logging for replay later (e.g. for correctness testing)
# Logging for debugging{quote}
Why is user auditing relevant to comply with GDPR regulations? [Art.
30|https://gdpr-info.eu/art-30-gdpr/] mandates that organizations must maintain
records of processing activities. As already mentioned, this doesn't have to
take place in Cassandra directly and can often be done more effectively at
application level. But if data is manipulated manually, or say you have some
small adhoc tools that change data but won't produce valid logs, you need to be
able to enable auditing logging for these cases, to be able to document how
data was changed.
It's also required by [art. 32|https://gdpr-info.eu/art-32-gdpr/] that any
natural person must only process personal data as instructed, ie. you need to
have an activity log to be able to prove that this is actually the case.
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388931#comment-16388931 ] Vinay Chella commented on CASSANDRA-12151: -- Thanks everyone for your inputs. I am trying to summarize the discussion here so that I can take action items and either incorporate in this patch or create separate JIRAs to track. 1. Simple and incremental approach 2. Reuse the BinLog/ Chronicle work that is put in CASSANDRA-13983 so that we don't duplicate efforts and achieve asynchronous, efficient logging 3. Provide context needed in AuditLogger interfaces so that we could have various implementations based on logging needs (e.g., more information to log, etc.,) 4. Filter/ whitelist users for AuditLog events. 5. Pluggable component that we can fit into the client-facing netty pipeline 6. Code review comments from Dinesh and Jaydeepkumar 7. Auditlog to log percentage of queries instead of logging every query I am currently working on #2, #3, #4 and #6. I will gather more data on #5, #7 and create separate JIRAs as needed. To satisfy #1, we are trying best to get the simple version(simple/ basic configs) out, take feedback and improve on in later versions/ patches. I am also planning to stress test this patch and publish numbers with and without AuditLog so that user knows the cost of this feature. Short story: Everyone wants C* audit logging, but we also know it has a non-trivial cost in terms of performance (not a free lunch). The debate is around what level of details should go in the log, what should be configurable, which can be addressed with #1 approach (incremental approach). However, design(Interface level details) feedback and redundant efforts(BinLog/Chronicle) are being addressed in this patch. Hope this summary helps. Please let me know if I am missing something important. > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Major > Fix For: 4.x > > Attachments: 12151.txt, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387303#comment-16387303 ] Jaydeepkumar Chovatia commented on CASSANDRA-12151: --- As per this [PR|https://github.com/vinaykumarchella/cassandra/pull/2] every single query will be audited once audit is enabled, can we make this selective to not audit all the queries but percentage of queries as people will have different requirements for audit query frequency? > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Major > Fix For: 4.x > > Attachments: 12151.txt, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386288#comment-16386288 ] Ariel Weisberg commented on CASSANDRA-12151: I think logging more things to more formats, and making what is logged more configurable are great additions. I'm -1 on having two approaches to logging database access to a file that are redundant. I don't really care what implementation we go with just that there aren't two. You could completely remove the existing FQL since we haven't shipped it and replace it with something with a similar high performance asynchronous binary logging option as well as the text based one you think is best for audit logs. Or you can iterate on the existing FQL functionality to get what you want from it. Chronicle has a human and machine readable text format. Switching between the two is a flag. You could use the existing full query log plumbing to write one of those if you wanted. You can also convert binary logs to human readable logs using fqltool. You can also tail the full query log in human readable format using fqltool. I think binary vs text is a bit of a red herring. Some things a new implementation needs to preserve is support for asynchronous logging, bounded memory utilization for queueing, and a reasonably efficient approach to appending to a file from multiple threads. It should be possible for a beefy node to log all the operations a node is executing and process them using a single core if configured to use binary format. > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Major > Fix For: 4.x > > Attachments: 12151.txt, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386137#comment-16386137
]
Stefan Podkowinski commented on CASSANDRA-12151:
[~eanujwa] wrote:
{quote}If auditing for a database application user is only at application
level, what would happen in case direct queries are executed on the database
using the the application user credentials (using tools such as cqlsh)?
{quote}
If the operator decided not to enable audit logging for the application users,
nothing would get logged. If someone steals the credentials, none of the
activities would show up in the logs. If you think that this is unacceptable,
then you can enable auditing for the app user as well, which I would not, but
it’s really up to you.
[~jolynch] wrote:
{quote}Doesn't the category filter adequately achieve this (you could exclude
DML or QUERY)? Do we need per user query logging when there is already per user
permissions limiting their access to the database in the first place?
{quote}
Although I understand the motivation behind the query categories from a
developer perspective, it’s probably a feature that is not as easy to
understand and handled by operators. Let alone managers and compliance
officers, who lack the technical understanding for discussing which categories
needs to be enabled for which tables. Once you’ll notice that you have to limit
the audit logging due to the performance impact, it’s going to be a rather
painful discussion how categories and tables should be configured in this
regard. I’d rather give ops people more simpler options that are easy to
understand and reason about, when discussed between ops and non-technical
stakeholders. Offering “all or nothing” audit logging on user basis would fit
this requirements for me (at least for CQL, login events could be a different
story).
I'd also really like to avoid luring people into the idea that Cassandra would
be a one-stop solution for audit logging and you don't have to implement your
own logging on application level anymore, since Cassandra does handle this for
you. It probably won't, as logging all statements for applications will be too
much of a performance impact. Now the question for me is to offer operators
options to optimize their way out of this situation, or simply tell people that
it's not possible to handle this use-case (audit logging for application users
on high-load systems) in Cassandra. I'd clearly go with the later, as many
users think that Cassandra already puts too much of a burden on operators.
{quote}While I agree use case #1 (security) does not require this, use cases #2
and #3 very much do. For #2 or similar you typically have to prove that only
authorized applications manipulated the database and a typical way to do that
is to produce query logs showing that only trusted application IP addresses and
specific credentials made DML statements (but QUERY is less important). For #3
the requirements are even greater, e.g. you may have to be able to prove that
user data was not exfiltrated at all, requiring auditing of QUERY statements.
Yes it's higher overhead but if you can turn it off with the category filters I
think it's fine don't you?
{quote}
Can you really “prove” unauthorised data manipulation did *not* happen through
the audit logs? Say I have an application that is updating a credit score and
now I use that user to do an update manually, how would you ever notice that
from the audit logs? My manual statement looks just like the million others
executed by the app. IP based access restrictions should be in place anyways,
that’s also not something that would necessarily give you additional clues.
As was already suggested, let’s take an incremental approach that will allow us
to introduce a simple audit logging that can be understood by both technical
and non-technical people and configured in little time. If any external
compliance authority raises some concerns, e.g. that DCL statements need to be
handled differently from queries, we can take care of that. But any 90%
solutions would still be a big win over what we currently have.
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16384516#comment-16384516
]
Joseph Lynch commented on CASSANDRA-12151:
--
There are a lot of competing desires in this ticket, and I want to firmly +1 an
incremental approach where we get the basic interface in and start adding new
implementations or configuration options in follow up items. If I understand
the comments above we're trying to solve the following all in one ticket:
# Logging for security
# Logging for business accounting compliance (e.g. SOX).
# Logging for monetary transaction compliance (e.g. PCI).
# Logging for replay later (e.g. for correctness testing)
These require different implementations because they require different
tradeoffs, and they may not all get done in this ticket. For example,
CASSANDRA-13983 is focused on use case #4 and appears to use a highly optimized
binary format which is lower overhead but does not meet requirements for #1/2/3
and requires custom parsers (you can't just hand the file to the auditors),
whereas the patch [~vinaykumarcse] has provided I believe aims more for #1/2/3.
Generally #2/3 require guarantees of logging if a query was attempted,
regardless of it succeeded, and #2/3 generally require a lot more context than
#1/#4 do. I think it would be great if in this ticket we can commit the basic
interface and starting points of the configuration options (e.g. is it
controlled through the cassandra.yaml or table options), and we can work on
improving performance, configuration flexibility, etc in follow up tickets.
[~jasobrown] wrote:
{quote}I agree with [~djoshi3] and [~jjordan]: this functionality should really
leverage the existing behavior of FQL (CASSANDRA-13893). There is no need to
create a parallel or duplicate set of behaviors, unless it's completely
warranted - and I have heard no arguments here that it is.
{quote}
Do you think the patch creates a parallel or duplicate set of behaviors? It
provides a different query logging implementation that makes different
tradeoffs and targets a different use case, but I think everyone is agreeing
that we can unify the two behind one interface (we just have to make sure that
interface has enough query context for all use cases which might be tough as
FQL really doesn't need all the session context like user info but #1,2,3 do).
[~eanujwa] wrote:
{quote}If you are logging the exact query with all the values in case of
regular queries (not prepared), then how would logging bind values of a
prepared statement becomes a security concern?
{quote}
Generally speaking secure applications exclusively use prepared statements as
simple statements are vulnerable to injection. Also, if you're using audit
logging for PCI (or even SOX) the data in DML could easily be sensitive (e.g.
credit cards or user's names), which you probably want to avoid by default. It
could certainly be an option though.
[~spo...@gmail.com] wrote:
{quote}Usually you'll see two kind of users on production systems: privileged
users and application users. Auditing privileged users (admins or developers)
will almost always make sense, in order to be able to detect unauthorized
access and data manipulation. There's only a limited amount of statements to
log, as these will be executed manually. It also shouldn't matter which
keyspaces or tables are access by the users; he is either monitored or not.
{quote}
Doesn't the category filter adequately achieve this (you could exclude DML or
QUERY)? Do we need per user query logging when there is already per user
permissions limiting their access to the database in the first place?
{quote}However, auditing queries of application users has a very limited
security and data privacy benefit, but adds a great deal of load to the
database. Those queries will be automatically generated by the application and
there will be no way to tell if the query or statement was authorized, as you
don't know on behalf of whom it was executed. Any auditing functionality for
these operations must therefor take place at application level.
{quote}
While I agree use case #1 (security) does not require this, use cases #2 and #3
very much do. For #2 or similar you typically have to prove that only
authorized applications manipulated the database and a typical way to do that
is to produce query logs showing that only trusted application IP addresses and
specific credentials made DML statements (but QUERY is less important). For #3
the requirements are even greater, e.g. you may have to be able to prove that
user data was not exfiltrated at all, requiring auditing of QUERY statements.
Yes it's higher overhead but if you can turn it off with the category filters I
think it's fine don't you?
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
>
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383854#comment-16383854 ] Dinesh Joshi commented on CASSANDRA-12151: -- Hi [~eanujwa] - I did review the design. As you can see there are different perspectives on the Audit functionality - granularity, type etc. Personally, defining an interface for Auditing would help keep things generic. For example, based on your needs you could, in theory, have an Audit logger implementation that logs at the keyspace or table level or even make more sophisticated decisions. It frees up everyone to define the Audit implementation to their specific needs. Bear in mind that you do not want to degrade performance for users that do not use Auditing capabilities. > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Major > Fix For: 4.x > > Attachments: 12151.txt, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383690#comment-16383690 ] Anuj Wadehra commented on CASSANDRA-12151: -- [~djoshi3] Can you review the design first and close open points? Once the design is closed, code review can be done for the revised patch. > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Major > Fix For: 4.x > > Attachments: 12151.txt, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383682#comment-16383682 ] Anuj Wadehra commented on CASSANDRA-12151: -- [~djoshi3] Can you review the design in detail and close open points? > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Major > Fix For: 4.x > > Attachments: 12151.txt, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383680#comment-16383680
]
Anuj Wadehra commented on CASSANDRA-12151:
--
[~spo...@gmail.com] My comments:
{quote}Any auditing functionality for these operations must therefor take place
at application level.
{quote}
If auditing for a database application user is only at application level, what
would happen in case direct queries are executed on the database using the the
application user credentials (using tools such as cqlsh)?
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383672#comment-16383672
]
Anuj Wadehra commented on CASSANDRA-12151:
--
[~vinaykumarcse] My comments are as follows:
{quote}I like the idea of keeping the auditing configurations to be simple at
keyspace level
{quote}
I think enabling/disabling auditing should be at table level rather than
keyspace level. Reasoning-Keyspaces are designed based on data replication
needs rather than auditing needs.
Thus, a keyspace k1 may have two tables with same replication needs but
different auditing requirements e.g. t1 with sensitive data which must be
audited and t2 with some temporary data which need not be audited. With
keyspace level auditing, users will end up doing unnecessary auditing and
cluttering of audit logs.
{quote}if we are managing any configurations at table level, config file might
not be a good place, table property?
{quote}
Agree. We can have default auditing configuration for tables in cassandra.yaml
(say enable_audit=true/false, categories=DDL,DCL etc.,extended=true/false etc.)
and table level properties to override the default auditing options in yaml.
This way we will not clutter cassandra.yaml with tables to be audited, there
would not be any additional configuration file and Cassandra configuration
files would not have any application schema awareness.
{quote}All the usecases that I see here are Auditing at the cluster or no
auditing, but not specific to user.
{quote}
If a Cassandra role defined for an application user has limited permissions,
you may not want to audit everything from the application and clutter the
auditing logs.
But If you have a single powerful application user for database, the feature
may not be that useful as the application user may be misused and you may
easily bypass the application and directly login to cqlsh. I dont think
Cassandra has an authentication mechanism to identify and restrict direct login
for specific application users/roles. In case of direct login using an
application user, you will log the operation but will not be able to fix the
responsibility of the operation. More thoughts?
{quote}This would be a security concern to keep the actual data in logs
{quote}
If you are logging the exact query with all the values in case of regular
queries (not prepared), then how would logging bind values of a prepared
statement becomes a security concern?
{quote}
If we think logging individual statement in the batch might not come for free,
I would rather approach it differently in case of batch, we just limit it to
log only statement types and its count or just the prepared statement ids?
{quote}
Can we only log unique query strings in a batch to restrict IO? In case of a
batch with prepared statements and extended configuration set to true, audit
log format could be:
Q1: INSERT INTO t1(variable1,variable2) values(?,?)
Q2: INSERT INTO t2(variable1,variable2,variable3) values(?,?,?)
Q1 [bind_value1,bind_value2]
Q2 [bind_value1,bind_value2,bind_value3]
Q1 [bind_value1,bind_value2]
Q1 [bind_value1,bind_value2]
If separate log statements are used, UUID may be associated with the batch
(like it is done in your patch)
{quote}Why do you think logging CL is required? Is CL adding any value for the
auditor?
{quote}
What are your thoughts on this?
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383606#comment-16383606 ] Jason Brown commented on CASSANDRA-12151: - I agree with [~djoshi3] and [~jjordan]: this functionality should really leverage the existing behavior of FQL (CASSANDRA-13893). There is no need to create a parallel or duplicate set of behaviors, unless it's completely warranted - and I have heard no arguments here that it is. Further, I like [~djoshi3]'s idea about adding a pluggable component that we can fit into the client-facing netty pipeline. It allows somebody to go bananas adding audit checks/details, and yet leaves untouched the standard behavior that 99.9% of users expect. > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Major > Fix For: 4.x > > Attachments: 12151.txt, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383488#comment-16383488 ] Stefan Podkowinski commented on CASSANDRA-12151: [~vinaykumarcse] wrote: bq. Seems like a complicated configuration, would like to understand the use cases more here and see if anyone else needs this functionality. All the usecases that I see here are Auditing at the cluster or no auditing, but not specific to user. I would love to hear if there are any other users with this usecase. Usually you'll see two kind of users on production systems: privileged users and application users. Auditing privileged users (admins or developers) will almost always make sense, in order to be able to detect unauthorized access and data manipulation. There's only a limited amount of statements to log, as these will be executed manually. It also shouldn't matter which keyspaces or tables are access by the users; he is either monitored or not. However, auditing queries of application users has a very limited security and data privacy benefit, but adds a great deal of load to the database. Those queries will be automatically generated by the application and there will be no way to tell if the query or statement was authorized, as you don't know on behalf of whom it was executed. Any auditing functionality for these operations must therefor take place at application level. Eg. a help desk tool, which is used by a support employee to access personal data of a customer in Cassandra, must keep an activity log for that directly. It doesn't make sense to log queries for the generic help desk tool Cassandra user on the database side. Therefor we need a way to enable CQL query auditing on user level. > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Major > Fix For: 4.x > > Attachments: 12151.txt, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383281#comment-16383281
]
Dinesh Joshi commented on CASSANDRA-12151:
--
Hi [~vinaykumarcse], I have gone over your patch. Here is my feedback -
# {{AuditLogEntryType::allStatementsMap}} - Instead of creating an explicit
map of statements consider adding the type of statement in the actual class
itself. That way when new statements are introduced, you don't need to keep
updating this map.
# {{AuditLogFilter::loadFilters}} - Consider adding checks here to make sure
{{includedKeyspacesList}} and {{excludedKeyspacesList}} do not have the same
keyspace specified. This will help simplifying your logic in
{{AuditLogFilter::isFiltered}}. Also consider lowercasing / uppercasing while
storing the keyspace and category names as you are doing a case insensitive
match in {{isFiltered}}.
# {{AuditLogFilter}} consider using {{Set}} instead of
{{List}} for {{includedKeyspacesList}}, {{excludedKeyspacesList}},
{{excludedCategoriesList}}, {{includedCategoriesList}}. Keyspace names and
Categories should be unique.
# {{AuditLogFilter::isFiltered}} can be simplified and made more performant if
you use {{Set}} instead of {{List}}.
# It would be great if you can update the docs in the source tree.
On the implementation design, it would be great to see if you can extend
-CASSANDRA-13983-. It does pretty much what you're doing except logging the
authorization failures.
Consider refactoring your code to add a netty handler that invokes an auditing
interface. The advantage of this approach would be that, when audit logging is
disabled, you can take this handler out of the netty pipeline. This way there
is zero performance impact when audit is disabled. You can define a
IAuditLogger interface that has sufficient contextual information to log all
queries. This will help make the audit logging implementation pluggable.
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382483#comment-16382483
]
Vinay Chella commented on CASSANDRA-12151:
--
I like the idea of keeping the auditing configurations to be simple at keyspace
level, as you mentioned specifying the configurations at table level gets too
messy to manage. Also adding new configurations as on when adding new tables
would be tough to manage and operate. Regarding maintaining separate files for
audit log configurations makes it difficult for the C* operators, sidecars,
config management systems to manage several configuration files. It would
become maintenance overhead to manage configuration files per feature. So I
would keep it simple and manage configurations via a single file in
cassandra.yaml. Even maintaining separate config file does not reduce the
overhead of adding/ removing table level audit log configurations. IMO, if we
are managing any configurations at table level, config file might not be a good
place, table property?
{quote}Configuration of whitelisted users and separate auditing configuration
for each user
{quote}
seems like a complicated configuration, would like to understand the use cases
more here and would like to see if anyone else needs this functionality. All
the usecases that I see here, either Audit the cluster or not, but not specific
to user, I would love to hear if there are any other users with this usecase.
{quote}Auditing bind values of prepared statements and its configuration
{quote}
This would be a security concern to keep the actual data in logs, these logs
could be shipped over the wire to an archival system for log rotation, in which
case data goes over the wire as a plain text and all the effort of securing the
C* cluster are not helpful at that point. Audit log should not become a
security hole in C* design.
{quote}Logging every statement in a Batch separately may have significant
performance hit.
{quote}
Merging the entire batch of statements to a single log line would be a
performance hit or might hit the limitations of message sizes in logger or any
such implementations. Considering the performance of storing the entire batch
of statements and logging them might not be a great idea for GC reasons as
well, there is always a tradeoff between small working set size vs one large
working set in terms of low garbage collection overhead, allocator overhead. If
we think logging individual statement in the batch might not come for free, I
would rather approach it differently in case of batch, we just limit it to log
only statement types and its count or just the prepared statement ids? We also
need to keep the performance overhead that we are adding to the request path
with AuditLog.
{quote} Password Obfuscation for DCL Queries
{quote}
Yes, Agreed - Password Obfuscation for DCL Queries shall be fixed.
{quote}Are you planning to evaluate and implement a Chronicle Queue variant
similar to CASSANDRA-13983?
{quote}
Yes, there is certainly a possible option to provide one more minimal
performance impact implementation for IAuditLogger, and we can consider binlog.
I will take a look at it next week and create a follow-up JIRA as needed.
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381505#comment-16381505 ] Anuj Wadehra commented on CASSANDRA-12151: -- [~vinaykumarcse] I have gone through your patch. Some high level review comments on the patch: # Why do you think logging CL is required? Is CL adding any value for the auditor? # Don’t you think we should have separate configuration file for auditing rather than cassandra.yaml? —there are applications with considerable number of tables. As per the proposed design, user shall be able to individually specify tables which must be audited and this would clutter the cassandra.yaml file. # Logging every statement in a Batch separately may have significant performance hit. Can we just log once per Batch and make sure that all operations in the batch are included in that one log statement? # Are you planning to evaluate and implement a Chronicle Queue variant similar to CASSANDRA-13983? The patch lacks following features from the design document: # Configuration of whitelisted/application users and separate auditing configuration for that. # Configuration of tables to be audited with/without regular expressions e.g. ks1.*,*.table1 etc. # Auditing bind values of prepared statements and its configuration # Password Obfuscation for DCL Queries [~jasobrown] [~vinaykumarcse] We can develop auditing feature (Cassandra code) incrementally. In the first iteration, [~vinaykumarcse] can contribute his patch and then our team shall contribute the remaining 4 features. We are already developing an auditing plugin for Cassandra which aligns with the proposed design and we can port these remaining features from the plugin to the main Cassandra code. > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Major > Fix For: 4.x > > Attachments: 12151.txt, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379130#comment-16379130 ] Jeremiah Jordan commented on CASSANDRA-12151: - One of the goals should be recording queries with minimal impact on workloads, which was also a goal of CASSANDRA-13983, so I would think some re-use might be a good idea rather than coming up with a new way of doing that. > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Anuj Wadehra >Priority: Major > Fix For: 4.x > > Attachments: 12151.txt, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379076#comment-16379076 ] Anuj Wadehra commented on CASSANDRA-12151: -- Thanks for the review comments ! [~vinaykumarcse] I will go through your patch and share my comments. [~jjordan] I had a look at -CASSANDRA-13983- The use cases are quite different. Yes we can have a chronicle-queue variant of Audit logger like [~vinaykumarcse] said but I think we should start with a simple logger implementation unless we have real good reasons to go with chronicle-queue for Audit logging. > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Anuj Wadehra >Priority: Major > Fix For: 4.x > > Attachments: 12151.txt, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377534#comment-16377534
]
Vinay Chella commented on CASSANDRA-12151:
--
Hi [~eanujwa] [~jasobrown],
I’m excited to see the design document and it looks good to us!
Netflix had a similar requirement recently for our internal 2.1 clusters and we
implemented a simple version (no query categories, etc…) for sox auditing. As
your design is very close to what we implemented, just a few differently named
classes for the most part, can we work together on the trunk
[patchset|https://github.com/vinaykumarchella/cassandra/pull/2] to add the
missing components from your design? Alternatively, we could take an
incremental approach, review what we have on the trunk branch of the simple
version and get it committed and then add in some of the more advanced features
next. I believe this patch follows the design goals that you put together.
Please review and let me know if you have any questions or concerns about the
first iteration. If folks are interested in the 3.x/2.x branches I can put
those up on my github as well.
[~jhb]
{quote}I just have one question, do you think enabling/updating/disabling audit
require a node restart?
{quote}
The posted patch allows online auditlog enable/disable or filter updates via
JMX.
[~jjordan]
{quote}You should take a look at the infrastructure added in CASSANDRA-13983
for query logging
{quote}
Yes, we looked and that certainly looks interesting, perhaps this design allows
us to use it as another implementation of {{IAuditLogger}}?
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Anuj Wadehra
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377259#comment-16377259 ] Jeremiah Jordan commented on CASSANDRA-12151: - You should take a look at the infrastructure added in CASSANDRA-13983 for query logging > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Anuj Wadehra >Priority: Major > Fix For: 4.x > > Attachments: 12151.txt, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377175#comment-16377175
]
Michael Shuler commented on CASSANDRA-12151:
Title: {{s/3.x/4.x/}}
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Anuj Wadehra
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376850#comment-16376850 ] Jacques-Henri Berthemet commented on CASSANDRA-12151: - Proposed design looks good to me. I think using a file is better than a table as a rogue user may just truncate/drop/overwrite the table. I just have one question, do you think enabling/updating/disabling audit require a node restart? > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Anuj Wadehra >Priority: Major > Fix For: 4.x > > Attachments: 12151.txt, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375459#comment-16375459 ] Anuj Wadehra commented on CASSANDRA-12151: -- Please find attached the Design proposal for "Database Auditing" feature in Cassandra. The solution suggests file-based audit logs rather than storing audit logs in database. I request everyone to review the proposal and share their feedback so that we can incorporate comments and go ahead with the implementation. [^DesignProposal_AuditingFeature_ApacheCassandra_v1.docx] > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Anuj Wadehra >Priority: Major > Fix For: 4.x > > Attachments: 12151.txt, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326295#comment-16326295 ] Anuj Wadehra commented on CASSANDRA-12151: -- [~jasobrown] Yes. Our team is keen to work on this ticket. I will be sharing the "Proposal" soon. For now, you can assign the JIRA to me. > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Priority: Major > Fix For: 4.x > > Attachments: 12151.txt > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322253#comment-16322253 ] Stefan Podkowinski commented on CASSANDRA-12151: I took a different approach with CASSANDRA-13668 by proposing to send auditing events to external subscribers that would be able to store or analyze them further. It's a different approach from storing auditing events in a local or distributed keyspace, as implemented in this ticket. My assumption was basically that raising alerts on certain events would be easier and more efficient to implement, if consumed by an external client, compared to polling a table. But I'm curious how any discussion as part of this ticket will turn out and what the different requirements are. > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi > Fix For: 4.x > > Attachments: 12151.txt > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1636#comment-1636 ] Jason Brown commented on CASSANDRA-12151: - [~eanujwa] No one is currently assigned to the ticket, and [~stefan setyadi]'s last comments are from Aug 2016. Thus I suspect nobody is working on it at the moment. Are you interested in working on it, or just requesting the feature? > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi > Fix For: 4.x > > Attachments: 12151.txt > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322076#comment-16322076 ] Anuj Wadehra commented on CASSANDRA-12151: -- There is no update since May,2017. Is anyone working on implementing auditing in Cassandra ? > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi > Fix For: 4.x > > Attachments: 12151.txt > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938440#comment-15938440 ] nhanpt14 commented on CASSANDRA-12151: -- Could we have Data Auditing in Apache Cassandra like DataStax Enterprise? I hope it will be support in next release. > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi > Fix For: 3.11.x > > Attachments: 12151.txt > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15401929#comment-15401929 ] stefan setyadi commented on CASSANDRA-12151: After some discussion with my team, we were thinking of the idea of being able to configure the log to each keyspace or table. And so I propose: # Cassandra will still log meta-changes of the server # We can configure it to log every query (update/read) which targets a specific keyspace or table. For our use case, we need to log it even knowing that there will be some performance penalty. all in all, the ability to configure which query will be logged will be helpful in cases where * the table is important, such as employee data, financial information, etc * or where we know the table won't be accessed as much > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi > Fix For: 3.x > > Attachments: 12151.txt > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15401669#comment-15401669 ] stefan setyadi commented on CASSANDRA-12151: okay so first of all, my initial use case was for the audit log to be reviewed and used to detect intrusion. At first I was thinking of logging the queries so it could be used to detect malicious insert/read. I admit now that in hindsight, I didn't have a clear idea of how big the scale of the operations were. You're probably right and we shouldn't log every insert/read query. I agree it is still useful to know any meta-changes but currently I have no clear picture of how the audit log will be used. +1 on the user login idea and the byteman. > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi > Fix For: 3.x > > Attachments: 12151.txt > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399584#comment-15399584 ] Jonathan Ellis commented on CASSANDRA-12151: Remember that people almost always use Cassandra to drive applications at scale, not to do interactive analytics. I can't see that logging 100,000 ops per second of the same ten queries is going to add much value. I don't want to load that gun for people to blow their feet off with... Generally auditing is most useful to see "who *changed* what" not "who *asked for* what." (Again, the "who" for most of the latter is going to be "the application server.") And again, it's not super useful to know that the app server inserted 10,000 new user accounts today, but it IS useful to know when Jonathan added a new column to the users table. (I would also include user logins as an interesting event. This will be dominated by app servers still but much much less noise than logging every query or update.) Besides changes over CQL, this could also include JMX changes, although there are so many entry points to JMX mbeans that this would be ugly to do by hand. Perhaps we could inject this with byteman? > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi > Fix For: 3.x > > Attachments: 12151.txt > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398691#comment-15398691 ] stefan setyadi commented on CASSANDRA-12151: yeah sure, sorry for going ahead and doing my own thing. # I was thinking of global one as it would be easier to be queried. it would also help in case any of the server fails as we probably don't want to lose the logs. # can I ask for clarification on what you meant by every query? at the very least we want to know the usual "who did what and when". Maybe have different levels/choice for which query are logged? In the end the trade off between performance and log is something which I'd rather leave on the hands of the company using it. > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi > Fix For: 3.x > > Attachments: 12151.txt > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397542#comment-15397542 ] Jonathan Ellis commented on CASSANDRA-12151: Can we back up a little and talk about design? Some questions in my mind: # Do we want a global audit log, or server-local? If the former (easier for users to query), it should go in system_distributed keyspace; otherwise just in system (higher performance). # Is there a use case where you'd want to log every query? That seems like it would entail a prohibitive performance penalty. I would think most users would be better served by logging meta-changes (adding roles, altering tables, etc) > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi > Fix For: 3.x > > Attachments: 12151.txt > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393643#comment-15393643
]
stefan setyadi commented on CASSANDRA-12151:
Hi this is my first time submitting a patch to jira so tell me if there's
anything I can improve on.
I attached a patch for feedback, do tell me if I should change my approach.
to start the logging put {noformat}audit_log: true{noformat} in cassandra.yaml
for now, a summary of what I've done:
* I created keyspace and column families to be initialize at server start.
* Right now it's also logging queries by cassandra itself.
* I didn't put the connections attempt as we can get that information from the
clientstate when processing the statement.
* the activity type is just the class name of the statement.
* the column family value when creating keyspace is 'null' as in string value
I haven't tested this on prepared and batch statement.
any feedback is welcome, regards
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
> Fix For: 3.x
>
> Attachments: 12151.txt
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
