[jira] [Commented] (CASSANDRA-10789) Allow DBAs to kill individual client sessions from certain IP(s) and temporarily block subsequent connections without bouncing JVM
[
https://issues.apache.org/jira/browse/CASSANDRA-10789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17580699#comment-17580699
]
Stefan Miklosovic commented on CASSANDRA-10789:
---
I updated this ticket to current trunk as the patch was not applying anymore. I
think that it would be nice to expose banned ips / addresses via a virtual
table. We would not have new nodetool commands which would enable / disable
clients. Instead, it would be possible to ban / unban a client via CQL virtual
table which would in turn talk to ConnectionBlacklistHandler. We might
contemplate about getting rid of JMX way of managing this stuff completely and
we would go only via CQL / virtual tables. I do not think that doing one thing
by two ways is a good idea and there is a general trend towards CQL / virtual
tables instead of JMX anyway.
Banning a client would look like:
{code}
INSERT INTO system_views.banned_clients (hostname) values ("192.168.1.10");
{code}
This would in turn called handler and it would added that banned host into the
list.
Unbanning would look like
{code}
DELETE FROM system_views.banned_clients where hostname = "192.168.1.10";
{code}
This CQL functionality would be built on top of
https://issues.apache.org/jira/browse/CASSANDRA-16806 which enables DELETE when
implementation allows it. This would also support TRUNCATE to unban all clients.
> Allow DBAs to kill individual client sessions from certain IP(s) and
> temporarily block subsequent connections without bouncing JVM
> --
>
> Key: CASSANDRA-10789
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10789
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Coordination
>Reporter: Wei Deng
>Assignee: Damien Stevenson
>Priority: Normal
> Labels: 4.0-feature-freeze-review-requested
> Fix For: 4.x
>
> Attachments: 10789-trunk-dtest.txt, 10789-trunk.txt
>
>
> In production, there could be hundreds of clients connected to a Cassandra
> cluster (maybe even from different applications), and if they use DataStax
> Java Driver, each client will establish at least one TCP connection to a
> Cassandra server (see
> https://datastax.github.io/java-driver/2.1.9/features/pooling/). This is all
> normal and at any given time, you can indeed see hundreds of ESTABLISHED
> connections to port 9042 on a C* server (from netstat -na). The problem is
> that sometimes when a C* cluster is under heavy load, when the DBA identifies
> some client session that sends abusive amount of traffic to the C* server and
> would like to stop it, they would like a lightweight approach rather than
> shutting down the JVM or rolling restart the whole cluster to kill all
> hundreds of connections in order to kill a single client session. If the DBA
> had root privilege, they would have been able to do something at the OS
> network level to achieve the same goal but oftentimes enterprise DBA role is
> separate from OS sysadmin role, so the DBAs usually don't have that privilege.
> This is especially helpful when you have a multi-tenant C* cluster and you
> want to have the impact for handling such client to be minimal to the other
> applications. This feature (killing individual session) seems to be a common
> feature in other databases (regardless of whether the client has some
> reconnect logic or not). It could be implemented as a JMX MBean method and
> exposed through nodetool to the DBAs.
> Note due to CQL driver's automated reconnection, simply killing the currently
> connected client session will not work well, so the JMX parameter should be
> an IP address or a list of IP addresses, so that the Cassandra server can
> terminate existing connection with that IP, and block future connection
> attempts from that IP for the remaining time until the JVM is restarted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-10789) Allow DBAs to kill individual client sessions from certain IP(s) and temporarily block subsequent connections without bouncing JVM
[ https://issues.apache.org/jira/browse/CASSANDRA-10789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579800#comment-17579800 ] Stefan Miklosovic commented on CASSANDRA-10789: --- I would like to continue to work on this patch, I briefly went over it and it seems in quite a good shape. Has something changed in the meanwhile which would block this feature to be finished? [~aweisberg] [~jjirsa] > Allow DBAs to kill individual client sessions from certain IP(s) and > temporarily block subsequent connections without bouncing JVM > -- > > Key: CASSANDRA-10789 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10789 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Coordination >Reporter: Wei Deng >Assignee: Damien Stevenson >Priority: Normal > Labels: 4.0-feature-freeze-review-requested > Fix For: 4.x > > Attachments: 10789-trunk-dtest.txt, 10789-trunk.txt > > > In production, there could be hundreds of clients connected to a Cassandra > cluster (maybe even from different applications), and if they use DataStax > Java Driver, each client will establish at least one TCP connection to a > Cassandra server (see > https://datastax.github.io/java-driver/2.1.9/features/pooling/). This is all > normal and at any given time, you can indeed see hundreds of ESTABLISHED > connections to port 9042 on a C* server (from netstat -na). The problem is > that sometimes when a C* cluster is under heavy load, when the DBA identifies > some client session that sends abusive amount of traffic to the C* server and > would like to stop it, they would like a lightweight approach rather than > shutting down the JVM or rolling restart the whole cluster to kill all > hundreds of connections in order to kill a single client session. If the DBA > had root privilege, they would have been able to do something at the OS > network level to achieve the same goal but oftentimes enterprise DBA role is > separate from OS sysadmin role, so the DBAs usually don't have that privilege. > This is especially helpful when you have a multi-tenant C* cluster and you > want to have the impact for handling such client to be minimal to the other > applications. This feature (killing individual session) seems to be a common > feature in other databases (regardless of whether the client has some > reconnect logic or not). It could be implemented as a JMX MBean method and > exposed through nodetool to the DBAs. > Note due to CQL driver's automated reconnection, simply killing the currently > connected client session will not work well, so the JMX parameter should be > an IP address or a list of IP addresses, so that the Cassandra server can > terminate existing connection with that IP, and block future connection > attempts from that IP for the remaining time until the JVM is restarted. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-10789) Allow DBAs to kill individual client sessions from certain IP(s) and temporarily block subsequent connections without bouncing JVM
[ https://issues.apache.org/jira/browse/CASSANDRA-10789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633217#comment-16633217 ] Jeff Jirsa commented on CASSANDRA-10789: Please don't mark issues ready to commit until they've been reviewed and accepted. > Allow DBAs to kill individual client sessions from certain IP(s) and > temporarily block subsequent connections without bouncing JVM > -- > > Key: CASSANDRA-10789 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10789 > Project: Cassandra > Issue Type: Improvement > Components: Coordination >Reporter: Wei Deng >Assignee: Damien Stevenson >Priority: Major > Labels: 4.0-feature-freeze-review-requested > Fix For: 4.x > > Attachments: 10789-trunk-dtest.txt, 10789-trunk.txt > > > In production, there could be hundreds of clients connected to a Cassandra > cluster (maybe even from different applications), and if they use DataStax > Java Driver, each client will establish at least one TCP connection to a > Cassandra server (see > https://datastax.github.io/java-driver/2.1.9/features/pooling/). This is all > normal and at any given time, you can indeed see hundreds of ESTABLISHED > connections to port 9042 on a C* server (from netstat -na). The problem is > that sometimes when a C* cluster is under heavy load, when the DBA identifies > some client session that sends abusive amount of traffic to the C* server and > would like to stop it, they would like a lightweight approach rather than > shutting down the JVM or rolling restart the whole cluster to kill all > hundreds of connections in order to kill a single client session. If the DBA > had root privilege, they would have been able to do something at the OS > network level to achieve the same goal but oftentimes enterprise DBA role is > separate from OS sysadmin role, so the DBAs usually don't have that privilege. > This is especially helpful when you have a multi-tenant C* cluster and you > want to have the impact for handling such client to be minimal to the other > applications. This feature (killing individual session) seems to be a common > feature in other databases (regardless of whether the client has some > reconnect logic or not). It could be implemented as a JMX MBean method and > exposed through nodetool to the DBAs. > Note due to CQL driver's automated reconnection, simply killing the currently > connected client session will not work well, so the JMX parameter should be > an IP address or a list of IP addresses, so that the Cassandra server can > terminate existing connection with that IP, and block future connection > attempts from that IP for the remaining time until the JVM is restarted. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-10789) Allow DBAs to kill individual client sessions from certain IP(s) and temporarily block subsequent connections without bouncing JVM
[ https://issues.apache.org/jira/browse/CASSANDRA-10789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470537#comment-16470537 ] Ariel Weisberg commented on CASSANDRA-10789: bq. The use case here is for operator's diagnosing client issues and cluster recovery. Thanks Kurt. It makes more sense with that context. > Allow DBAs to kill individual client sessions from certain IP(s) and > temporarily block subsequent connections without bouncing JVM > -- > > Key: CASSANDRA-10789 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10789 > Project: Cassandra > Issue Type: Improvement > Components: Coordination >Reporter: Wei Deng >Assignee: Damien Stevenson >Priority: Major > Fix For: 4.x > > Attachments: 10789-trunk-dtest.txt, 10789-trunk.txt > > > In production, there could be hundreds of clients connected to a Cassandra > cluster (maybe even from different applications), and if they use DataStax > Java Driver, each client will establish at least one TCP connection to a > Cassandra server (see > https://datastax.github.io/java-driver/2.1.9/features/pooling/). This is all > normal and at any given time, you can indeed see hundreds of ESTABLISHED > connections to port 9042 on a C* server (from netstat -na). The problem is > that sometimes when a C* cluster is under heavy load, when the DBA identifies > some client session that sends abusive amount of traffic to the C* server and > would like to stop it, they would like a lightweight approach rather than > shutting down the JVM or rolling restart the whole cluster to kill all > hundreds of connections in order to kill a single client session. If the DBA > had root privilege, they would have been able to do something at the OS > network level to achieve the same goal but oftentimes enterprise DBA role is > separate from OS sysadmin role, so the DBAs usually don't have that privilege. > This is especially helpful when you have a multi-tenant C* cluster and you > want to have the impact for handling such client to be minimal to the other > applications. This feature (killing individual session) seems to be a common > feature in other databases (regardless of whether the client has some > reconnect logic or not). It could be implemented as a JMX MBean method and > exposed through nodetool to the DBAs. > Note due to CQL driver's automated reconnection, simply killing the currently > connected client session will not work well, so the JMX parameter should be > an IP address or a list of IP addresses, so that the Cassandra server can > terminate existing connection with that IP, and block future connection > attempts from that IP for the remaining time until the JVM is restarted. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-10789) Allow DBAs to kill individual client sessions from certain IP(s) and temporarily block subsequent connections without bouncing JVM
[ https://issues.apache.org/jira/browse/CASSANDRA-10789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468988#comment-16468988 ] Ariel Weisberg commented on CASSANDRA-10789: I forgot it was a ChannelGroup. I would just leave it for now. > Allow DBAs to kill individual client sessions from certain IP(s) and > temporarily block subsequent connections without bouncing JVM > -- > > Key: CASSANDRA-10789 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10789 > Project: Cassandra > Issue Type: Improvement > Components: Coordination >Reporter: Wei Deng >Assignee: Damien Stevenson >Priority: Major > Fix For: 4.x > > Attachments: 10789-trunk-dtest.txt, 10789-trunk.txt > > > In production, there could be hundreds of clients connected to a Cassandra > cluster (maybe even from different applications), and if they use DataStax > Java Driver, each client will establish at least one TCP connection to a > Cassandra server (see > https://datastax.github.io/java-driver/2.1.9/features/pooling/). This is all > normal and at any given time, you can indeed see hundreds of ESTABLISHED > connections to port 9042 on a C* server (from netstat -na). The problem is > that sometimes when a C* cluster is under heavy load, when the DBA identifies > some client session that sends abusive amount of traffic to the C* server and > would like to stop it, they would like a lightweight approach rather than > shutting down the JVM or rolling restart the whole cluster to kill all > hundreds of connections in order to kill a single client session. If the DBA > had root privilege, they would have been able to do something at the OS > network level to achieve the same goal but oftentimes enterprise DBA role is > separate from OS sysadmin role, so the DBAs usually don't have that privilege. > This is especially helpful when you have a multi-tenant C* cluster and you > want to have the impact for handling such client to be minimal to the other > applications. This feature (killing individual session) seems to be a common > feature in other databases (regardless of whether the client has some > reconnect logic or not). It could be implemented as a JMX MBean method and > exposed through nodetool to the DBAs. > Note due to CQL driver's automated reconnection, simply killing the currently > connected client session will not work well, so the JMX parameter should be > an IP address or a list of IP addresses, so that the Cassandra server can > terminate existing connection with that IP, and block future connection > attempts from that IP for the remaining time until the JVM is restarted. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
