[jira] [Comment Edited] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468526#comment-16468526
]
Vinay Chella edited comment on CASSANDRA-12151 at 5/9/18 8:03 AM:
--
Hi [~jasobrown] ,
Yes, we can set audit_logs_dir either from -D jvm arg or from
{{cassandra.yaml}} - {{audit_logging_options::audit_logs_dir}}. Idea is to be
inconsistent with rest of the project's log configuration process by setting
them via {{-D jvm arg}}, hence naming is kept inconsistent with logs -
{{cassandra.logdir.audit}}. I have also updated the documentation for the same.
{quote}Operators cannot, in an obvious manner, adjust the following
AuditLogOptions options:
audit_logs_dir/block/max_queue_weight/max_log_size/roll_cycle. They are
basically hidden yaml options: you can set them, but you have to know they
exist. I understand these are more advanced options, but we should document
them either in the yaml or the doc page.
{quote}
Yes, they intentionally kept there for advanced configurations. Agree with you
on documenting them. Updated the documentation for this.
||*Branch*||*PR*||*uTest*||
|[trunk_CASSANDRA-12151|https://github.com/vinaykumarchella/cassandra/tree/trunk_CASSANDRA-12151]|[PR
for
trunk|https://github.com/vinaykumarchella/cassandra/pull/2/commits]|[!https://circleci.com/gh/vinaykumarchella/cassandra/tree/trunk_CASSANDRA-12151.svg?style=svg!|https://circleci.com/gh/vinaykumarchella/cassandra/tree/trunk_CASSANDRA-12151]|
was (Author: vinaykumarcse):
Hi Jason,
Yes, we can set audit_logs_dir either from -D jvm arg or from
{{cassandra.yaml}} - {{audit_logging_options::audit_logs_dir}}. Idea is to be
inconsistent with rest of the project's log configuration process by setting
them via {{-D jvm arg}}, hence naming is kept inconsistent with logs -
{{cassandra.logdir.audit}}. I have also updated the documentation for the same.
{quote}Operators cannot, in an obvious manner, adjust the following
AuditLogOptions options:
audit_logs_dir/block/max_queue_weight/max_log_size/roll_cycle. They are
basically hidden yaml options: you can set them, but you have to know they
exist. I understand these are more advanced options, but we should document
them either in the yaml or the doc page.
{quote}
Yes, they intentionally kept there for advanced configurations. Agree with you
on documenting them. Updated the documentation for this.
||*Branch*||*PR*||*uTest*||
|[trunk_CASSANDRA-12151|https://github.com/vinaykumarchella/cassandra/tree/trunk_CASSANDRA-12151]|[PR
for
trunk|https://github.com/vinaykumarchella/cassandra/pull/2/commits]|[!https://circleci.com/gh/vinaykumarchella/cassandra/tree/trunk_CASSANDRA-12151.svg?style=svg!|https://circleci.com/gh/vinaykumarchella/cassandra/tree/trunk_CASSANDRA-12151]|
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467079#comment-16467079 ] Per Otterström edited comment on CASSANDRA-12151 at 5/8/18 11:25 AM: - bq. This is an intersting idea. However, I'm reticent to introduce CQL grammar changes this late into this ticket. As a followup, I think this could be worth exploring. Sure, I understand we seek to close this ticket. I'm just a bit concerned with the timing. If this ticket is merged as is and we take a cut for 4.0, then I assume we will have to stick to this way of configure audit logs for some time. bq. While there's some conceptual overlap, I don't think it's a good idea to try to force the concepts of 'audit logging' and 'auth' onto one another here. I believe it depends on the use case for doing audits. In some cases it would make perfect sense to have these two concepts in sync. E.g. I want to grant a user to have SELECT permission on a specific table. Then I would grant the same user to have NOAUDIT for SELECT on the very same table. Should the user try to read somethign else, or write to that table, an audit entry would be created. In a case like this it is very helpful for the admin to use the same concepts for these two "permissions". To verify accuracy, should we create some dtests for this as well? was (Author: eperott): bq. This is an intersting idea. However, I'm reticent to introduce CQL grammar changes this late into this ticket. As a followup, I think this could be worth exploring. Sure, I understand we seek to close this ticket. I'm just a bit concerned with the timing. If this ticket is merged as is and we take a cut for 4.0, then I assume we will have to stick to this way of configure audit logs for some time. bq. While there's some conceptual overlap, I don't think it's a good idea to try to force the concepts of 'audit logging' and 'auth' onto one another here. I believe it depends on the use case for doing audits. In some cases it would make perfect sense to have these two concepts in sync. E.g. I want to grant a user to have SELECT permission on a specific table. Then I would grant the same user to have NOAUDIT for SELECT on the very same table. Shoudl the user try to read somethign else, or write to that table, and audit entry would be created. In a case like this it is very helpful for the admin to use the same concepts for these two "permissions". To verify accuracy, should we create some dtests for this as well? > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Major > Fix For: 4.x > > Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458567#comment-16458567
]
Jason Brown edited comment on CASSANDRA-12151 at 4/30/18 1:29 PM:
--
After rereading this patch, I've merged the FQL and AuditLog code paths as they
do the same thing from the outside: record user-initiated events. To that end,
I've moved the management of {{FullQueryLogger}} under {{AuditLogManager}},
thus making {{AuditLogManager}} the hub for logging these actions. Further,
instead of {{BinAuditLogger}} inherit from {{FullQueryLogger}} (which never
felt quite right), I've pulled the shared componentry into a base class
({{BinLogAuditLogger}}), and {{BinAuditLogger}} / {{FullQueryLogger}} inherit
from that.
I refactored {{AuditLogEntry}} to use the builder pattern for construction as
all the {{static #getLogEntry()}} with the {{set*()}} methods was rather
confusing. Further, we're using the builder pattern elsewhere in the code.
{{#toString()}} is typically used for debugging, so I'd prefer a different
method name - I used {{#getLogString()}}. If you want the debug dump to have
the same output, you can have {{toString()}} call that renamed function.
We need an extra filter in {{AuditLogManager#log(AuditLogEntry)}} to make sure
we don't send unintended messages to FQL. Can you figure out what that is? I'm
not sure if we can simply use the entry's {{AuditLogEntryType}} or the type's
category.
Why did you remove the reloadFilters functionality?
I feel like the {{category}} in AuditLogEntryType should be an enum, as well.
wdyt?
I haven't looked at the circleci results yet for utests/dtests, but let's make
sure this is right path forward first.
UPDATE: branch (squashed as single patch) is
[here|https://github.com/jasobrown/cassandra/tree/trunk_CASSANDRA-12151]
was (Author: jasobrown):
After rereading this patch, I've merged the FQL and AuditLog code paths as they
do the same thing from the outside: record user-initiated events. To that end,
I've moved the management of {{FullQueryLogger}} under {{AuditLogManager}},
thus making {{AuditLogManager}} the hub for logging these actions. Further,
instead of {{BinAuditLogger}} inherit from {{FullQueryLogger}} (which never
felt quite right), I've pulled the shared componentry into a base class
({{BinLogAuditLogger}}), and {{BinAuditLogger}} / {{FullQueryLogger}} inherit
from that.
I refactored {{AuditLogEntry}} to use the builder pattern for construction as
all the {{static #getLogEntry()}} with the {{set*()}} methods was rather
confusing. Further, we're using the builder pattern elsewhere in the code.
{{#toString()}} is typically used for debugging, so I'd prefer a different
method name - I used {{#getLogString()}}. If you want the debug dump to have
the same output, you can have {{toString()}} call that renamed function.
We need an extra filter in {{AuditLogManager#log(AuditLogEntry)}} to make sure
we don't send unintended messages to FQL. Can you figure out what that is? I'm
not sure if we can simply use the entry's {{AuditLogEntryType}} or the type's
category.
Why did you remove the reloadFilters functionality?
I feel like the {{category}} in AuditLogEntryType should be an enum, as well.
wdyt?
I haven't looked at the circleci results yet for utests/dtests, but let's make
sure this is right path forward first.
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16446271#comment-16446271
]
Jason Brown edited comment on CASSANDRA-12151 at 4/20/18 7:46 PM:
--
Thanks to [~djoshi3] and [~spo...@gmail.com] for doing the first passes of
review here, I'll take it the rest of the way.
[~vinaykumarcse] and I spoke offline about some things, and on the whole we are
in the right direction. As I started reviewing the patch, the whitespace
inconsistencies nagged at me, so I went ahead and fixed those, as well as some
random code comments. Further, I made {{AuditLogFilters}} atomic and final, and
no longer a singleton; a volatile reference is maintained in
{{AuditLogManager}}. I also moved getLogEntry methods from {{AuditLogManager}}
to {{AuditLogEntry}} as they seemed more apropos there. Code available
[here|https://github.com/jasobrown/cassandra/tree/trunk_CASSANDRA-12151].
(Note: i didn't run the tests, so you may need to fix those up if necessary)
There are some things wrt the interaction of FQL and AuditLogging that I still
need to work out in my head, and I've asked Vinay to add some nodetool commands
for enabling/disabling auditlog as well (or, at least, to seriously think about
it). However, I wanted to post that first batch of changes to get that out of
the way.
was (Author: jasobrown):
Thanks to [~djoshi3] and [~spo...@gmail.com] for doing the first passes of
review here, I'll take it the rest of the way.
[~vinaykumarcse] and I spoke offline about some things, and on the whole we are
in the right direction. As I started reviewing the patch, the whitespace
inconsistencies nagged at me, so I went ahead and fixed those, as well as some
random code comments. Further, I made {{AuditLogFilters}} atomic and final, and
no longer a singleton; a volatile reference is maintained in
{{AuditLogManager}}. I also moved getLogEntry methods from {{AuditLogManager}}
to {{AuditLogEntry}} as they seemed more apropos there. Code available
[here|https://github.com/jasobrown/cassandra/tree/trunk_CASSANDRA-12151].
There are some things wrt the interaction of FQL and AuditLogging that I still
need to work out in my head, and I've asked Vinay to add some nodetool commands
for enabling/disabling auditlog as well (or, at least, to seriously think about
it). However, I wanted to post that first batch of changes to get that out of
the way.
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16446271#comment-16446271
]
Jason Brown edited comment on CASSANDRA-12151 at 4/20/18 7:45 PM:
--
Thanks to [~djoshi3] and [~spo...@gmail.com] for doing the first passes of
review here, I'll take it the rest of the way.
[~vinaykumarcse] and I spoke offline about some things, and on the whole we are
in the right direction. As I started reviewing the patch, the whitespace
inconsistencies nagged at me, so I went ahead and fixed those, as well as some
random code comments. Further, I made {{AuditLogFilters}} atomic and final, and
no longer a singleton; a volatile reference is maintained in
{{AuditLogManager}}. I also moved getLogEntry methods from {{AuditLogManager}}
to {{AuditLogEntry}} as they seemed more apropos there. Code available
[here|https://github.com/jasobrown/cassandra/tree/trunk_CASSANDRA-12151].
There are some things wrt the interaction of FQL and AuditLogging that I still
need to work out in my head, and I've asked Vinay to add some nodetool commands
for enabling/disabling auditlog as well (or, at least, to seriously think about
it). However, I wanted to post that first batch of changes to get that out of
the way.
was (Author: jasobrown):
Thanks to [~djoshi3] and [~spo...@gmail.com] for doing the first passes of
review here, I'll take it the rest of the way.
[~vinaykumarcse] and I spoke offline about some things, and on the whole we are
in the right direction. As I started reviewing the patch, the whitespace
inconsistencies nagged at me, so I went ahead and fixed those, as well as some
random code comments. Further, I made {{AuditLogFilters}} atomic and final, and
no longer a singleton; a volatile reference is maintained in
{{AuditLogManager}}. I also moved getLogEntry methods from {{AuditLogManager}}
to {{AuditLogEntry}} as they seemed more apropos there. Code available
[here|https://github.com/jasobrown/cassandra/tree/trunk_CASSANDRA-12151].
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430448#comment-16430448
]
Stefan Podkowinski edited comment on CASSANDRA-12151 at 4/9/18 1:08 PM:
I've now managed to update CASSANDRA-13668 by implementing IAuditLogger and
expose audit events as diagnostic events via native transport to subscribed
clients. Went pretty much as expected and seems to work fine.
Smaller issues I've came across:
* AuditLogUtil doesn't look very useful. DEFAULT_SOURCE is only used once,
SYSTEM_USER not at all.
* Do we really want to keep IAuditLogger.error()? Please provide a description
of the log/error semantics in context with audit logging for possible
subclasses, or get rid of error().
* AuditLogFilter.isFiltered(): proposed logic will ignore includeSet if
excludeSet is provided (may not make sense to do so, but not strictly forbidden
by cassandra.yaml either), e.g. exclude(A), include(A,B,C) should only have B,C
pass
There are also a couple of limitations:
* Username will not be provided for failed authentications
* Bound values will not get logged for prepared statements (as already pointed
out in this ticket's discussion)
I haven't found a quick way to work around these, but being able to avoid the
audit log by using prepared statements, is something we have to address. It's
probably not going to be that much of an issue for my use case logging ad-hoc
commands for regular users, once we have CASSANDRA-8303 and can disable
prepared statement for them. But for logging all activity for application
users, I don't know.
[~laxmikant99]
{quote}Can we have a configurable property like exitOnAuditFailure ? This
fulfills requirement of strict auditing .. I mean in case auditing fails for a
db operation, then the operation should not get executed.
{quote}
Any logging to the BinLogger should block by default. But it doesn't exit the
JVM.
was (Author: spo...@gmail.com):
I've now managed to update CASSANDRA-13668 by implementing IAuditLogger and
expose audit events as diagnostic events via native transport to subscribed
clients. Went pretty much as expected and seems to work fine.
Smaller issues I've came across:
* AuditLogUtil doesn't look very useful. DEFAULT_SOURCE is only used once,
SYSTEM_USER not at all.
* Do we really want to keep IAuditLogger.error()? Please provide a description
of the log/error semantics in context with audit logging for possible
subclasses, or get rid of error().
* AuditLogFilter.isFiltered(): proposed logic will ignore includeSet if
excludeSet is provided (may not make sense to do so, but not strictly forbidden
by cassandra.yaml either), e.g. exclude(A), include(A,B,C) should only have B,C
pass
There are also a couple of limitations:
* Username will not be provided for failed authentications
* Bound values will not get logged for prepared statements
I haven't found a quick way to work around these, but being able to avoid the
audit log by using prepared statements, is something we have to address. It's
probably not going to be that much of an issue for my use case logging ad-hoc
commands for regular users, once we have CASSANDRA-8303 and can disable
prepared statement for them. But for logging all activity for application
users, I don't know.
[~laxmikant99]
{quote}Can we have a configurable property like exitOnAuditFailure ? This
fulfills requirement of strict auditing .. I mean in case auditing fails for a
db operation, then the operation should not get executed.
{quote}
Any logging to the BinLogger should block by default. But it doesn't exit the
JVM.
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional
[jira] [Comment Edited] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428102#comment-16428102 ] Laxmikant Upadhyay edited comment on CASSANDRA-12151 at 4/6/18 8:31 AM: Can we have a configurable property like exitOnAuditFailure ? This fulfills requirement of strict auditing .. I mean in case auditing fails for a db operation, then the operation should not get executed. was (Author: laxmikant99): Can we have a configurable property like exitOnAuditFailure ? This fulfills requirement of strict auditing .. I mean in case auditing fails db operation should not get executed. > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Major > Fix For: 4.x > > Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426915#comment-16426915
]
Stefan Podkowinski edited comment on CASSANDRA-12151 at 4/5/18 1:25 PM:
I was just giving the BinLogger a try and it looks like the way the bin log
files are created, should make it possible to archive them before they get
deleted. The most simple way would probably to run a rsync job for pulling the
latest chronicle bin logs from the nodes. The timestamp based naming of the
files should make this trivial.
Another option would be to allow the user to configure an archival script that
will be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin
log, just as we do in {{CommitLogArchiver}}. WDYT [~aweisberg]?
I'm going to write a custom {{IAuditLogger}} implementation tomorrow and will
get back with some more feedback.
was (Author: spo...@gmail.com):
I was just giving the BinLogger a try and it looks like the way the bin log
files are created, should make it possible to archive them before they get
deleted. The most simple way would probably to run a rsync job for pulling the
latest chronicle bin logs from the nodes. The timestamp based naming of the
files should make this trivial.
Another option would be to allow the user to configure an archival script that
will be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin
log, just as we do in {{CommitLogArchiver}}. WDYT [~aweisberg]?
I'm going to write a custom implementation tomorrow and will get back with some
more feedback.
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416952#comment-16416952
]
Vinay Chella edited comment on CASSANDRA-12151 at 3/29/18 5:44 AM:
---
Hi [~djoshi3] [~jolynch],
Thanks for the review, implemented review suggestions provided and pushed those
changes in PR.
\\
||*Branch*||*PR*||*uTest*||
|[trunk_CASSANDRA-12151|https://github.com/vinaykumarchella/cassandra/tree/trunk_CASSANDRA-12151]|[PR
for
trunk|https://github.com/vinaykumarchella/cassandra/pull/2/commits]|[!https://circleci.com/gh/vinaykumarchella/cassandra/tree/trunk_CASSANDRA-12151.svg?style=svg!|https://circleci.com/gh/vinaykumarchella/cassandra/tree/trunk_CASSANDRA-12151]|
\\
{quote}Initialization can be simplified like this - excludedUsers =
ImmutableSet.of(DatabaseDescriptor.getAuditLoggingOptions().excluded_users.split(","));
{quote}
{{loadFilters()}} are designed so that filter values can be loaded during the
startup as well as JMX call to reload the filters
{quote}AuditLogFilter:: isFiltered - You can return directly at line 137 and
get rid of isExcluded variable You can get rid of isIncluded as well and return
at line 148. Let the control fall through to line 153 and it will return false.
Furthermore, If you check the sets during initialization and ensure that
they're mutually exclusive, your logic simplifies to if
(includedSet.contains(object)) return false; else if
(excludedSet.contains(object)}) return true;
{quote}
I went with the suggestions provided in former part to simplify the logic in
{{isFiltered}} rather than mutual exclusion at initialization part.
was (Author: vinaykumarcse):
Hi [~djoshi3] [~jolynch],
Thanks for the review, implemented review suggestions provided and pushed those
changes in PR.
\\
||[branch|https://github.com/vinaykumarchella/cassandra/tree/trunk_CASSANDRA-12151]||
|[PR for trunk|https://github.com/vinaykumarchella/cassandra/pull/2/commits]|
|[circleci|https://circleci.com/gh/vinaykumarchella/cassandra/tree/trunk_CASSANDRA-12151]|
\\
{quote}Initialization can be simplified like this - excludedUsers =
ImmutableSet.of(DatabaseDescriptor.getAuditLoggingOptions().excluded_users.split(","));
{quote}
{{loadFilters()}} are designed so that filter values can be loaded during the
startup as well as JMX call to reload the filters
{quote}AuditLogFilter:: isFiltered - You can return directly at line 137 and
get rid of isExcluded variable You can get rid of isIncluded as well and return
at line 148. Let the control fall through to line 153 and it will return false.
Furthermore, If you check the sets during initialization and ensure that
they're mutually exclusive, your logic simplifies to if
(includedSet.contains(object)) return false; else if
(excludedSet.contains(object)}) return true;
{quote}
I went with the suggestions provided in former part to simplify the logic in
{{isFiltered}} rather than mutual exclusion at initialization part.
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403146#comment-16403146
]
Vinay Chella edited comment on CASSANDRA-12151 at 3/17/18 12:12 AM:
[~djoshi3]
Implemented all the code reviews comments provided in this JIRA thread as well
as Github PR. Except one below
{quote}Consider refactoring your code to add a netty handler that invokes an
auditing interface. The advantage of this approach would be that, when audit
logging is disabled, you can take this handler out of the netty pipeline. This
way there is zero performance impact when the audit is disabled. You can define
a IAuditLogger interface that has sufficient contextual information to log all
queries. This will help make the audit logging implementation pluggable.
{quote}
I am creating a follow-up JIRA to discuss the more details on this.
On a high level, this changeset includes following changes
# Extended and reused FullQueryLogger in logging audit events
# Combined and Simplified FQL and AuditLog entry points in the request path
# AuditLogEntryType::allStatementsMap - Instead of creating an explicit map of
statements, type of statement is being added to the actual class itself. This
makes new statements easy to manage
# AuditLogFilter::loadFilters - Simplified filter loading logic, easy to add
new filters if needed
# CQL query auditing can now be filtered on user level.
# Added documentation in the doc folder
# Removed ConsistencyLevel in logging details
# Added more test cases
# Implemented code review comments provided in this JIRA as well as Github PR
||[branch|https://github.com/vinaykumarchella/cassandra/tree/trunk_CASSANDRA-12151]||
|[PR for trunk|https://github.com/vinaykumarchella/cassandra/pull/2/commits]|
|[circleci|https://circleci.com/gh/vinaykumarchella/cassandra/tree/trunk_CASSANDRA-12151]|
We ran cassandra stress test with this patch and attached stress test results
([^CASSANDRA_12151-benchmark.html]). Here is the high level summary
Note: Below tests are run on AWS i2.2xl instance.
{{cass-stress cmd: write n=100 -rate threads=10 -graph
file=CASSANDRA_12151-benchmark.html}}
||WRITE - Test Suite||Throughput||Latency Mean||Latency 95th||Latency 99th||
|trunk|13,925 op/s|0.7 ms|1.1 ms|1.7 ms|
|CASSANDRA-12151:Disabled AuditLog|14,422 op/s|0.7 ms|1.1 ms|1.6 ms|
|CASSANDRA-12151:FQL based AuditLog with Sync|13,372 op/s|0.7 ms|1.2 ms|1.7 ms|
|CASSANDRA-12151:FQL based AuditLog with Async|12,908 op/s|0.8 ms|1.2 ms|1.9 ms|
|CASSANDRA-12151:SLF4j based AuditLog|10,520 op/s|0.9 ms|1.6 ms|2.4 ms|
{{cass-stress cmd: mixed n=100 -rate threads=10 -graph
file=CASSANDRA_12151-benchmark.html}}
||MIXED - Test Suite||Throughput||Latency Mean||Latency 95th||Latency 99th||
|trunk|12,939 op/s [READ: 6,494 op/s, WRITE: 6,444 op/s]|0.7 ms [READ: 0.8 ms,
WRITE: 0.7 ms]|1.2 ms [READ: 1.3 ms, WRITE: 1.2 ms]|1.7 ms [READ: 1.8 ms,
WRITE: 1.7 ms]|
|CASSANDRA-12151: Disabled AuditLog|12,840 op/s [READ: 6,421 op/s, WRITE: 6,419
op/s]|0.8 ms [READ: 0.8 ms, WRITE: 0.7 ms]|1.2 ms [READ: 1.3 ms, WRITE: 1.2
ms]|1.8 ms [READ: 1.8 ms, WRITE: 1.7 ms]|
|CASSANDRA-12151: FQL based AuditLog with Sync|10,932 op/s [READ: 5,452 op/s,
WRITE: 5,481 op/s]|0.9 ms [READ: 1.0 ms, WRITE: 0.8 ms]|1.5 ms [READ: 1.6 ms,
WRITE: 1.4 ms]|2.3 ms [READ: 2.4 ms, WRITE: 2.1 ms]|
|CASSANDRA-12151: FQL based AuditLog with Async|11,146 op/s [READ: 5,565 op/s,
WRITE: 5,581 op/s]|0.9 ms [READ: 0.9 ms, WRITE: 0.8 ms]|1.5 ms [READ: 1.5 ms,
WRITE: 1.4 ms]|2.2 ms [READ: 2.2 ms, WRITE: 2.1 ms]|
|CASSANDRA-12151: SLF4j based AuditLog|9,764 op/s [READ: 4,883 op/s, WRITE:
4,882 op/s]|1.0 ms [READ: 1.0 ms, WRITE: 1.0 ms]|1.7 ms [READ: 1.7 ms, WRITE:
1.6 ms]|2.5 ms [READ: 2.6 ms, WRITE: 2.4 ms]|
Looking at the results, with AuditLog feature disabled, there appears to be no
measurable difference in performance. FQL appears to have little or no overhead
in WRITE only workloads, and a minor overhead in MIXED workload. SLF4J appears
to have minor regressions in both workloads (with mixed slightly worse).
was (Author: vinaykumarcse):
[~djoshi3]
Implemented all the code reviews comments provided in this JIRA thread as well
as Github PR. Except one below
{quote}
Consider refactoring your code to add a netty handler that invokes an auditing
interface. The advantage of this approach would be that, when audit logging is
disabled, you can take this handler out of the netty pipeline. This way there
is zero performance impact when the audit is disabled. You can define a
IAuditLogger interface that has sufficient contextual information to log all
queries. This will help make the audit logging implementation pluggable.
{quote}
I am creating a follow-up JIRA to discuss the more details on this.
On a high level, this changeset includes following changes
# Extended and reused FullQueryLogger in
[jira] [Comment Edited] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387303#comment-16387303 ] Jaydeepkumar Chovatia edited comment on CASSANDRA-12151 at 3/6/18 5:16 AM: --- As per this [PR|https://github.com/vinaykumarchella/cassandra/pull/2] every single query will be audited once audit is enabled, can we make this selective to not audit all the queries but percentage of queries as people will have different requirements on audit query frequency? was (Author: chovatia.jayd...@gmail.com): As per this [PR|https://github.com/vinaykumarchella/cassandra/pull/2] every single query will be audited once audit is enabled, can we make this selective to not audit all the queries but percentage of queries as people will have different requirements for audit query frequency? > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Major > Fix For: 4.x > > Attachments: 12151.txt, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16384516#comment-16384516
]
Joseph Lynch edited comment on CASSANDRA-12151 at 3/3/18 6:50 AM:
--
There are a lot of competing desires in this ticket, and I want to firmly +1 an
incremental approach where we get the basic interface in and start adding new
implementations or configuration options in follow up items. If I understand
the comments above we're trying to solve the following all in one ticket:
# Logging for security
# Logging for business accounting compliance (e.g. SOX).
# Logging for monetary transaction compliance (e.g. PCI).
# Logging for replay later (e.g. for correctness testing)
# Logging for debugging
These require different implementations because they require different
tradeoffs, and they may not all get done in this ticket. For example,
CASSANDRA-13983 is focused on use case #4 and appears to use a highly optimized
binary format which is lower overhead but does not meet requirements for #1/2/3
and requires custom parsers (you can't just hand the file to the auditors),
whereas the patch [~vinaykumarcse] has provided I believe aims more for #1/2/3.
Generally #2/3 require guarantees of logging if a query was attempted,
regardless of it succeeded, and #2/3 generally require a lot more context than
#1/#4 do. I think it would be great if in this ticket we can commit the basic
interface and starting points of the configuration options (e.g. is it
controlled through the cassandra.yaml or table options), and we can work on
improving performance, configuration flexibility, etc in follow up tickets.
[~jasobrown] wrote:
{quote}I agree with [~djoshi3] and [~jjordan]: this functionality should really
leverage the existing behavior of FQL (CASSANDRA-13893). There is no need to
create a parallel or duplicate set of behaviors, unless it's completely
warranted - and I have heard no arguments here that it is.
{quote}
Do you think the patch creates a parallel or duplicate set of behaviors? It
provides a different query logging implementation that makes different
tradeoffs and targets a different use case, but I think everyone is agreeing
that we can unify the two behind one interface (we just have to make sure that
interface has enough query context for all use cases which might be tough as
FQL really doesn't need all the session context like user info but #1,2,3 do).
[~eanujwa] wrote:
{quote}If you are logging the exact query with all the values in case of
regular queries (not prepared), then how would logging bind values of a
prepared statement becomes a security concern?
{quote}
Generally speaking secure applications exclusively use prepared statements as
simple statements are vulnerable to injection. Also, if you're using audit
logging for PCI (or even SOX) the data in DML could easily be sensitive (e.g.
credit cards or user's names), which you probably want to avoid by default. It
could certainly be an option though.
[~spo...@gmail.com] wrote:
{quote}Usually you'll see two kind of users on production systems: privileged
users and application users. Auditing privileged users (admins or developers)
will almost always make sense, in order to be able to detect unauthorized
access and data manipulation. There's only a limited amount of statements to
log, as these will be executed manually. It also shouldn't matter which
keyspaces or tables are access by the users; he is either monitored or not.
{quote}
Doesn't the category filter adequately achieve this (you could exclude DML or
QUERY)? Do we need per user query logging when there is already per user
permissions limiting their access to the database in the first place?
{quote}However, auditing queries of application users has a very limited
security and data privacy benefit, but adds a great deal of load to the
database. Those queries will be automatically generated by the application and
there will be no way to tell if the query or statement was authorized, as you
don't know on behalf of whom it was executed. Any auditing functionality for
these operations must therefor take place at application level.
{quote}
While I agree use case #1 (security) does not require this, use cases #2 and #3
very much do. For #2 or similar you typically have to prove that only
authorized applications manipulated the database and a typical way to do that
is to produce query logs showing that only trusted application IP addresses and
specific credentials made DML statements (but QUERY is less important). For #3
the requirements are even greater, e.g. you may have to be able to prove that
user data was not exfiltrated at all, requiring auditing of QUERY statements.
Yes it's higher overhead but if you can turn it off with the category filters I
think it's fine don't you?
was (Author: jolynch):
There are a lot of competing
[jira] [Comment Edited] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382483#comment-16382483
]
Vinay Chella edited comment on CASSANDRA-12151 at 3/1/18 8:32 PM:
--
I like the idea of keeping the auditing configurations to be simple at keyspace
level, as you mentioned specifying the configurations at table level gets too
messy to manage. Also adding new configurations as on when adding new tables
would be tough to manage and operate. Regarding maintaining separate files for
audit log configurations, makes it difficult for the C* operators, sidecars,
config management systems to manage several configuration files. It would
become maintenance overhead to manage configuration files per feature. So I
would keep it simple and manage configurations via a single file in
cassandra.yaml. Even maintaining separate config file does not reduce the
overhead of adding/ removing table level audit log configurations. IMO, if we
are managing any configurations at table level, config file might not be a good
place, table property?
{quote}Configuration of whitelisted users and separate auditing configuration
for each user
{quote}
Seems like a complicated configuration, would like to understand the use cases
more here and see if anyone else needs this functionality. All the usecases
that I see here are Auditing at the cluster or no auditing, but not specific to
user. I would love to hear if there are any other users with this usecase.
{quote}Auditing bind values of prepared statements and its configuration
{quote}
This would be a security concern to keep the actual data in logs, these logs
could be shipped over the wire to an archival system for log rotation, in which
case data goes over the wire as a plain text and all the effort of securing the
C* cluster are not helpful at that point. Audit log should not become a
security hole in C* design.
{quote}Logging every statement in a Batch separately may have significant
performance hit.
{quote}
Merging the entire batch of statements to a single log line would be a
performance hit or might hit the limitations of message sizes in logger or any
such implementations. Considering the performance of storing the entire batch
of statements and logging them might not be a great idea for GC reasons as
well, there is always a tradeoff between small working set size vs one large
working set in terms of low garbage collection overhead, allocator overhead. If
we think logging individual statement in the batch might not come for free, I
would rather approach it differently in case of batch, we just limit it to log
only statement types and its count or just the prepared statement ids? We also
need to keep the performance overhead that we are adding to the request path
with AuditLog.
{quote} Password Obfuscation for DCL Queries
{quote}
Yes, Agreed - Password Obfuscation for DCL Queries shall be fixed.
{quote}Are you planning to evaluate and implement a Chronicle Queue variant
similar to CASSANDRA-13983?
{quote}
Yes, there is certainly a possible option to provide one more minimal
performance impact implementation for IAuditLogger, and we can consider binlog.
I will take a look at it next week and create a follow-up JIRA as needed.
was (Author: vinaykumarcse):
I like the idea of keeping the auditing configurations to be simple at keyspace
level, as you mentioned specifying the configurations at table level gets too
messy to manage. Also adding new configurations as on when adding new tables
would be tough to manage and operate. Regarding maintaining separate files for
audit log configurations makes it difficult for the C* operators, sidecars,
config management systems to manage several configuration files. It would
become maintenance overhead to manage configuration files per feature. So I
would keep it simple and manage configurations via a single file in
cassandra.yaml. Even maintaining separate config file does not reduce the
overhead of adding/ removing table level audit log configurations. IMO, if we
are managing any configurations at table level, config file might not be a good
place, table property?
{quote}Configuration of whitelisted users and separate auditing configuration
for each user
{quote}
seems like a complicated configuration, would like to understand the use cases
more here and would like to see if anyone else needs this functionality. All
the usecases that I see here, either Audit the cluster or not, but not specific
to user, I would love to hear if there are any other users with this usecase.
{quote}Auditing bind values of prepared statements and its configuration
{quote}
This would be a security concern to keep the actual data in logs, these logs
could be shipped over the wire to an archival system for log rotation, in which
case data goes over the wire as a plain text and all the
[jira] [Comment Edited] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377534#comment-16377534
]
Vinay Chella edited comment on CASSANDRA-12151 at 2/27/18 4:48 PM:
---
Hi [~eanujwa] [~jasobrown],
I’m excited to see the design document and it looks good to us!
Netflix had a similar requirement recently for our internal 2.1 clusters and we
implemented a simple version (no query categories, etc…) for sox auditing. As
your design is very close to what we implemented, just a few differently named
classes for the most part, can we work together on the trunk
[patchset|https://github.com/vinaykumarchella/cassandra/pull/2] to add the
missing components from your design? Alternatively, we could take an
incremental approach, review what we have on the trunk branch of the simple
version and get it committed and then add in some of the more advanced features
next. I believe this patch follows the design goals that you put together.
Please review and let me know if you have any questions or concerns about the
first iteration. If folks are interested in the 3.x/2.x branches I can put
those up on my github as well.
[~jhb]
{quote}I just have one question, do you think enabling/updating/disabling audit
require a node restart?
{quote}
The posted patch allows online auditlog enable/disable via JMX.
[~jjordan]
{quote}You should take a look at the infrastructure added in CASSANDRA-13983
for query logging
{quote}
Yes, we looked and that certainly looks interesting, perhaps this design allows
us to use it as another implementation of {{IAuditLogger}}?
Here is the patch location:
||[trunk|https://github.com/vinaykumarchella/cassandra]||
|[PR for Trunk|https://github.com/vinaykumarchella/cassandra/pull/2]|
was (Author: vinaykumarcse):
Hi [~eanujwa] [~jasobrown],
I’m excited to see the design document and it looks good to us!
Netflix had a similar requirement recently for our internal 2.1 clusters and we
implemented a simple version (no query categories, etc…) for sox auditing. As
your design is very close to what we implemented, just a few differently named
classes for the most part, can we work together on the trunk
[patchset|https://github.com/vinaykumarchella/cassandra/pull/2] to add the
missing components from your design? Alternatively, we could take an
incremental approach, review what we have on the trunk branch of the simple
version and get it committed and then add in some of the more advanced features
next. I believe this patch follows the design goals that you put together.
Please review and let me know if you have any questions or concerns about the
first iteration. If folks are interested in the 3.x/2.x branches I can put
those up on my github as well.
[~jhb]
{quote}I just have one question, do you think enabling/updating/disabling audit
require a node restart?
{quote}
The posted patch allows online auditlog enable/disable via JMX.
[~jjordan]
{quote}You should take a look at the infrastructure added in CASSANDRA-13983
for query logging
{quote}
Yes, we looked and that certainly looks interesting, perhaps this design allows
us to use it as another implementation of {{IAuditLogger}}?
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Anuj Wadehra
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377534#comment-16377534
]
Vinay Chella edited comment on CASSANDRA-12151 at 2/27/18 2:22 AM:
---
Hi [~eanujwa] [~jasobrown],
I’m excited to see the design document and it looks good to us!
Netflix had a similar requirement recently for our internal 2.1 clusters and we
implemented a simple version (no query categories, etc…) for sox auditing. As
your design is very close to what we implemented, just a few differently named
classes for the most part, can we work together on the trunk
[patchset|https://github.com/vinaykumarchella/cassandra/pull/2] to add the
missing components from your design? Alternatively, we could take an
incremental approach, review what we have on the trunk branch of the simple
version and get it committed and then add in some of the more advanced features
next. I believe this patch follows the design goals that you put together.
Please review and let me know if you have any questions or concerns about the
first iteration. If folks are interested in the 3.x/2.x branches I can put
those up on my github as well.
[~jhb]
{quote}I just have one question, do you think enabling/updating/disabling audit
require a node restart?
{quote}
The posted patch allows online auditlog enable/disable via JMX.
[~jjordan]
{quote}You should take a look at the infrastructure added in CASSANDRA-13983
for query logging
{quote}
Yes, we looked and that certainly looks interesting, perhaps this design allows
us to use it as another implementation of {{IAuditLogger}}?
was (Author: vinaykumarcse):
Hi [~eanujwa] [~jasobrown],
I’m excited to see the design document and it looks good to us!
Netflix had a similar requirement recently for our internal 2.1 clusters and we
implemented a simple version (no query categories, etc…) for sox auditing. As
your design is very close to what we implemented, just a few differently named
classes for the most part, can we work together on the trunk
[patchset|https://github.com/vinaykumarchella/cassandra/pull/2] to add the
missing components from your design? Alternatively, we could take an
incremental approach, review what we have on the trunk branch of the simple
version and get it committed and then add in some of the more advanced features
next. I believe this patch follows the design goals that you put together.
Please review and let me know if you have any questions or concerns about the
first iteration. If folks are interested in the 3.x/2.x branches I can put
those up on my github as well.
[~jhb]
{quote}I just have one question, do you think enabling/updating/disabling audit
require a node restart?
{quote}
The posted patch allows online auditlog enable/disable or filter updates via
JMX.
[~jjordan]
{quote}You should take a look at the infrastructure added in CASSANDRA-13983
for query logging
{quote}
Yes, we looked and that certainly looks interesting, perhaps this design allows
us to use it as another implementation of {{IAuditLogger}}?
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Anuj Wadehra
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15401669#comment-15401669 ] stefan setyadi edited comment on CASSANDRA-12151 at 8/1/16 10:34 AM: - okay so first of all, my initial use case was for the audit log to be reviewed and used to detect intrusion. At first I was thinking of logging the queries so it could be used to detect meta-changes and malicious insert/read. I admit now that in hindsight, I didn't have a clear idea of how big the scale of the operations were. You're probably right and we shouldn't log every insert/read query. I agree it is still useful to know any meta-changes but currently I have no clear picture of how the audit log will be used without the insert/read. +1 on the user login idea and the byteman. was (Author: stefan setyadi): okay so first of all, my initial use case was for the audit log to be reviewed and used to detect intrusion. At first I was thinking of logging the queries so it could be used to detect malicious insert/read. I admit now that in hindsight, I didn't have a clear idea of how big the scale of the operations were. You're probably right and we shouldn't log every insert/read query. I agree it is still useful to know any meta-changes but currently I have no clear picture of how the audit log will be used. +1 on the user login idea and the byteman. > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi > Fix For: 3.x > > Attachments: 12151.txt > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393643#comment-15393643
]
stefan setyadi edited comment on CASSANDRA-12151 at 7/26/16 11:18 AM:
--
Hi this is my first time submitting a patch to jira so tell me if there's
anything I can improve on.
I attached a patch for feedback, do tell me if I should change my approach.
the patch is made on clone of trunk branch
to start the logging put {noformat}audit_log: true{noformat} in cassandra.yaml
for now, a summary of what I've done:
* I created keyspace and column families to be initialize at server start.
* Right now it's also logging queries by cassandra itself.
* I didn't put the connections attempt as we can get that information from the
clientstate when processing the statement.
* the activity type is just the class name of the statement.
* the column family value when creating keyspace is 'null' as in string value
I haven't tested this on prepared and batch statement.
any feedback is welcome, regards
was (Author: stefan setyadi):
Hi this is my first time submitting a patch to jira so tell me if there's
anything I can improve on.
I attached a patch for feedback, do tell me if I should change my approach.
to start the logging put {noformat}audit_log: true{noformat} in cassandra.yaml
for now, a summary of what I've done:
* I created keyspace and column families to be initialize at server start.
* Right now it's also logging queries by cassandra itself.
* I didn't put the connections attempt as we can get that information from the
clientstate when processing the statement.
* the activity type is just the class name of the statement.
* the column family value when creating keyspace is 'null' as in string value
I haven't tested this on prepared and batch statement.
any feedback is welcome, regards
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
> Fix For: 3.x
>
> Attachments: 12151.txt
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
