Re: Re : Recommended procedure for enabling SSL on a live production cluster
hi Nate; thanks for the help. Upgrading to 2.1.12 seems to be the solution for client to node encryption on NATIVE port. The other issue we are facing is with the STORAGE port. The reason behind this is that we need to switch back and forth between different internode_encryption modes, and we need C* servers to keep running in transient state or during mode switching. Currently this is not possible. For example, we have a internode_encryption=none cluster in a multi-region AWS environment and want to set internode_encryption=dc by rolling restart C* nodes. However, the node with internode_encryption=dc, does not open to listen to non-ssl port. As a result, we have a splitted brain cluster here. Below is a ticket opened for the exact same issue. Has anybody overcome any such issue on a production cluster? Thanks in advance. https://issues.apache.org/jira/browse/CASSANDRA-8751 thanks Sai On Wed, Jul 20, 2016 at 5:25 PM, Nate McCall <n...@thelastpickle.com> wrote: > If you migrate to the latest 2.1 first, you can make this a non-issue as > 2.1.12 and above support simultaneous SSL and plain on the same port for > exactly this use case: > https://issues.apache.org/jira/browse/CASSANDRA-10559 > > On Thu, Jul 21, 2016 at 3:02 AM, sai krishnam raju potturi < > pskraj...@gmail.com> wrote: > >> hi ; >> if possible could someone shed some light on this. I followed a >> post from the lastpickle which was very informative, but we had some >> concerns when it came to enabling SSL on a live production cluster. >> >> >> http://thelastpickle.com/blog/2015/09/30/hardening-cassandra-step-by-step-part-1-server-to-server.html >> >> 1 : We generally remove application traffic from a DC which has ongoing >> changes, just not to affect end customers if things go south during the >> update. >> >> 2 : So once DC-A has been restarted after enabling SSL, this would be >> missing writes during that period, as the DC-A would be shown as down by >> the other DC's. We will not be able to put back application traffic on DC-A >> until we run inter-dc repairs, which will happen only when SSL has been >> enabled on all DC's. >> >> 3 : Repeating the procedure for every DC will lead to some missed writes >> across all DC's. >> >> 4 : We could do the rolling restart of a DC-A with application traffic >> on, but we are concerned if for any infrastructure related reason we have >> an issue, we will have to serve traffic from another DC-B, which might be >> missing on writes to the DC-A during that period. >> >> We have 4 DC's which 50 nodes each. >> >> >> thanks >> Sai >> >> -- Forwarded message -- >> From: sai krishnam raju potturi <pskraj...@gmail.com> >> Date: Mon, Jul 18, 2016 at 11:06 AM >> Subject: Re : Recommended procedure for enabling SSL on a live production >> cluster >> To: user@cassandra.apache.org >> >> >> Hi; >> We have a Cassandra cluster ( version 2.0.14 ) spanning across 4 >> datacenters with 50 nodes each. We are planning to enable SSL between the >> datacenters. We are following the standard procedure for enabling SSL ( >> http://thelastpickle.com/blog/2015/09/30/hardening-cassandra-step-by-step-part-1-server-to-server.html) >> . We were planning to enable SSL for each datacenter at a time. >> >> During the rolling restart, it's expected that the nodes in the >> datacenter that had the service restarted, will show as down by the nodes >> in other datacenters that have not restarted the service. This would lead >> to missed writes among various nodes during this procedure. >> >> What would be the recommended procedure for enabling SSL on a live >> production cluster without the chaos. >> >> thanks >> Sai >> >> > > > -- > - > Nate McCall > Wellington, NZ > @zznate > > CTO > Apache Cassandra Consulting > http://www.thelastpickle.com >
Re: Re : Recommended procedure for enabling SSL on a live production cluster
Thank you Nate. We are currently on 2.0.14. So the only option we are left with is the upgrade to 2.1.12. So for the earlier versions there is no way to enable SSL once the cluster is in production. Thanks again Nate. Thanks Sai On Wed, Jul 20, 2016, 17:26 Nate McCall <n...@thelastpickle.com> wrote: > If you migrate to the latest 2.1 first, you can make this a non-issue as > 2.1.12 and above support simultaneous SSL and plain on the same port for > exactly this use case: > https://issues.apache.org/jira/browse/CASSANDRA-10559 > > On Thu, Jul 21, 2016 at 3:02 AM, sai krishnam raju potturi < > pskraj...@gmail.com> wrote: > >> hi ; >> if possible could someone shed some light on this. I followed a >> post from the lastpickle which was very informative, but we had some >> concerns when it came to enabling SSL on a live production cluster. >> >> >> http://thelastpickle.com/blog/2015/09/30/hardening-cassandra-step-by-step-part-1-server-to-server.html >> >> 1 : We generally remove application traffic from a DC which has ongoing >> changes, just not to affect end customers if things go south during the >> update. >> >> 2 : So once DC-A has been restarted after enabling SSL, this would be >> missing writes during that period, as the DC-A would be shown as down by >> the other DC's. We will not be able to put back application traffic on DC-A >> until we run inter-dc repairs, which will happen only when SSL has been >> enabled on all DC's. >> >> 3 : Repeating the procedure for every DC will lead to some missed writes >> across all DC's. >> >> 4 : We could do the rolling restart of a DC-A with application traffic >> on, but we are concerned if for any infrastructure related reason we have >> an issue, we will have to serve traffic from another DC-B, which might be >> missing on writes to the DC-A during that period. >> >> We have 4 DC's which 50 nodes each. >> >> >> thanks >> Sai >> >> -- Forwarded message -- >> From: sai krishnam raju potturi <pskraj...@gmail.com> >> Date: Mon, Jul 18, 2016 at 11:06 AM >> Subject: Re : Recommended procedure for enabling SSL on a live production >> cluster >> To: user@cassandra.apache.org >> >> >> Hi; >> We have a Cassandra cluster ( version 2.0.14 ) spanning across 4 >> datacenters with 50 nodes each. We are planning to enable SSL between the >> datacenters. We are following the standard procedure for enabling SSL ( >> http://thelastpickle.com/blog/2015/09/30/hardening-cassandra-step-by-step-part-1-server-to-server.html) >> . We were planning to enable SSL for each datacenter at a time. >> >> During the rolling restart, it's expected that the nodes in the >> datacenter that had the service restarted, will show as down by the nodes >> in other datacenters that have not restarted the service. This would lead >> to missed writes among various nodes during this procedure. >> >> What would be the recommended procedure for enabling SSL on a live >> production cluster without the chaos. >> >> thanks >> Sai >> >> > > > -- > - > Nate McCall > Wellington, NZ > @zznate > > CTO > Apache Cassandra Consulting > http://www.thelastpickle.com >
Re: Re : Recommended procedure for enabling SSL on a live production cluster
If you migrate to the latest 2.1 first, you can make this a non-issue as 2.1.12 and above support simultaneous SSL and plain on the same port for exactly this use case: https://issues.apache.org/jira/browse/CASSANDRA-10559 On Thu, Jul 21, 2016 at 3:02 AM, sai krishnam raju potturi < pskraj...@gmail.com> wrote: > hi ; > if possible could someone shed some light on this. I followed a > post from the lastpickle which was very informative, but we had some > concerns when it came to enabling SSL on a live production cluster. > > > http://thelastpickle.com/blog/2015/09/30/hardening-cassandra-step-by-step-part-1-server-to-server.html > > 1 : We generally remove application traffic from a DC which has ongoing > changes, just not to affect end customers if things go south during the > update. > > 2 : So once DC-A has been restarted after enabling SSL, this would be > missing writes during that period, as the DC-A would be shown as down by > the other DC's. We will not be able to put back application traffic on DC-A > until we run inter-dc repairs, which will happen only when SSL has been > enabled on all DC's. > > 3 : Repeating the procedure for every DC will lead to some missed writes > across all DC's. > > 4 : We could do the rolling restart of a DC-A with application traffic on, > but we are concerned if for any infrastructure related reason we have an > issue, we will have to serve traffic from another DC-B, which might be > missing on writes to the DC-A during that period. > > We have 4 DC's which 50 nodes each. > > > thanks > Sai > > -- Forwarded message -- > From: sai krishnam raju potturi <pskraj...@gmail.com> > Date: Mon, Jul 18, 2016 at 11:06 AM > Subject: Re : Recommended procedure for enabling SSL on a live production > cluster > To: user@cassandra.apache.org > > > Hi; > We have a Cassandra cluster ( version 2.0.14 ) spanning across 4 > datacenters with 50 nodes each. We are planning to enable SSL between the > datacenters. We are following the standard procedure for enabling SSL ( > http://thelastpickle.com/blog/2015/09/30/hardening-cassandra-step-by-step-part-1-server-to-server.html) > . We were planning to enable SSL for each datacenter at a time. > > During the rolling restart, it's expected that the nodes in the > datacenter that had the service restarted, will show as down by the nodes > in other datacenters that have not restarted the service. This would lead > to missed writes among various nodes during this procedure. > > What would be the recommended procedure for enabling SSL on a live > production cluster without the chaos. > > thanks > Sai > > -- - Nate McCall Wellington, NZ @zznate CTO Apache Cassandra Consulting http://www.thelastpickle.com
Re: Re : Recommended procedure for enabling SSL on a live production cluster
hi ; if possible could someone shed some light on this. I followed a post from the lastpickle which was very informative, but we had some concerns when it came to enabling SSL on a live production cluster. http://thelastpickle.com/blog/2015/09/30/hardening-cassandra-step-by-step-part-1-server-to-server.html 1 : We generally remove application traffic from a DC which has ongoing changes, just not to affect end customers if things go south during the update. 2 : So once DC-A has been restarted after enabling SSL, this would be missing writes during that period, as the DC-A would be shown as down by the other DC's. We will not be able to put back application traffic on DC-A until we run inter-dc repairs, which will happen only when SSL has been enabled on all DC's. 3 : Repeating the procedure for every DC will lead to some missed writes across all DC's. 4 : We could do the rolling restart of a DC-A with application traffic on, but we are concerned if for any infrastructure related reason we have an issue, we will have to serve traffic from another DC-B, which might be missing on writes to the DC-A during that period. We have 4 DC's which 50 nodes each. thanks Sai -- Forwarded message -- From: sai krishnam raju potturi <pskraj...@gmail.com> Date: Mon, Jul 18, 2016 at 11:06 AM Subject: Re : Recommended procedure for enabling SSL on a live production cluster To: user@cassandra.apache.org Hi; We have a Cassandra cluster ( version 2.0.14 ) spanning across 4 datacenters with 50 nodes each. We are planning to enable SSL between the datacenters. We are following the standard procedure for enabling SSL ( http://thelastpickle.com/blog/2015/09/30/hardening-cassandra-step-by-step-part-1-server-to-server.html) . We were planning to enable SSL for each datacenter at a time. During the rolling restart, it's expected that the nodes in the datacenter that had the service restarted, will show as down by the nodes in other datacenters that have not restarted the service. This would lead to missed writes among various nodes during this procedure. What would be the recommended procedure for enabling SSL on a live production cluster without the chaos. thanks Sai
Re : Recommended procedure for enabling SSL on a live production cluster
Hi; We have a Cassandra cluster ( version 2.0.14 ) spanning across 4 datacenters with 50 nodes each. We are planning to enable SSL between the datacenters. We are following the standard procedure for enabling SSL ( http://thelastpickle.com/blog/2015/09/30/hardening-cassandra-step-by-step-part-1-server-to-server.html) . We were planning to enable SSL for each datacenter at a time. During the rolling restart, it's expected that the nodes in the datacenter that had the service restarted, will show as down by the nodes in other datacenters that have not restarted the service. This would lead to missed writes among various nodes during this procedure. What would be the recommended procedure for enabling SSL on a live production cluster without the chaos. thanks Sai
