Re: [openstack-dev] [TripleO] Move redis out of Pacemaker

[Pradeep Kilambi]

On Mon, Dec 12, 2016 at 3:48 PM, Michele Baldessari 
wrote:

> Hi Pradeep,
>
> On Mon, Dec 12, 2016 at 02:51:59PM +0100, Giulio Fidente wrote:
> > On 12/09/2016 04:49 PM, Pradeep Kilambi wrote:
> > >I would like to get some thoughts on $Subject. This came up when i was
> > >discussing the standalone roles for telemetry. Currently when we deploy
> > >redis in tripleo, its a pacemaker managed service. So if we were to
> > >deploy telemetry services on a dedicated node we could. But redis will
> > >have to be on a another node? (assuming we dont want to pull in
> > >pacemaker on to telemetry nodes).
>
> Ok so with the composable HA work [1] you should be able to split out
> the redis service on to dedicated nodes and these nodes can be either
> full pacemaker cluster members or only have the pacemaker-remote
> service.
>
> > currently redis instances are not configured as a redis cluster but use
> the
> > master/slave replication model instead and pacemaker is taking care of
> > electing/relocating the redis master as needed
> >
> > there shouldn't be any dependency on the redis profile for the telemetry
> > roles, they should instead just point at the redis_vip
> >
> > the redis_vip is always guaranteed (by haproxy) to point to the redis
> master
> >
> > >With most services moved out of pacemaker in Newton, I think its time to
> > >move redis as well? Are there any constraints in moving redis to be
> > >managed by systemd? Looking at how we do it, It should be easily movable
> > >to systemd? Can we consider doing this for Ocata?
> >
> > I think we could look at using the redis cluster which allows multiple
> > masters, but I am not sure this can happen in Ocata ... yet again, there
> > shouldn't be in the telemetry roles any dependency on redis itself
> >
> > if we were to use the cluster mode the only difference would probably be
> > that the redis_vip will start balancing requests across the nodes
>
> In general I am in favour to split out redis from pacemaker. There is
> the question that in theory we'd have two potentially separate quorums,
> but I think that with redis this should not be a big problem.
>
> Maybe let's start with a prototype and see how things look and iterate
> from there? I think it is a bit late for ocata, but we could at least
> start the work without changing the defaults (i.e. let the operator
> override the tripleo::service with a redis base profile instead of the
> pacemaker one)
>

Makes sense. I understand it might be too late for ocata. We don't really
have
any urgency so long as we can split out redis like you say we can with
composable HA.
I was more curious what the long term plan was and what you said makes
sense.

Thanks.
~ Prad


>
> Does that make sense,
> Michele
>
> [1] https://review.openstack.org/#/q/topic:bp/composable-ha
> --
> Michele Baldessari
> C2A5 9DA3 9961 4FFB E01B  D0BC DDD4 DCCB 7515 5C6D
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Move redis out of Pacemaker

[Michele Baldessari]

Hi Pradeep,

On Mon, Dec 12, 2016 at 02:51:59PM +0100, Giulio Fidente wrote:
> On 12/09/2016 04:49 PM, Pradeep Kilambi wrote:
> >I would like to get some thoughts on $Subject. This came up when i was
> >discussing the standalone roles for telemetry. Currently when we deploy
> >redis in tripleo, its a pacemaker managed service. So if we were to
> >deploy telemetry services on a dedicated node we could. But redis will
> >have to be on a another node? (assuming we dont want to pull in
> >pacemaker on to telemetry nodes).

Ok so with the composable HA work [1] you should be able to split out
the redis service on to dedicated nodes and these nodes can be either
full pacemaker cluster members or only have the pacemaker-remote
service.

> currently redis instances are not configured as a redis cluster but use the
> master/slave replication model instead and pacemaker is taking care of
> electing/relocating the redis master as needed
> 
> there shouldn't be any dependency on the redis profile for the telemetry
> roles, they should instead just point at the redis_vip
> 
> the redis_vip is always guaranteed (by haproxy) to point to the redis master
> 
> >With most services moved out of pacemaker in Newton, I think its time to
> >move redis as well? Are there any constraints in moving redis to be
> >managed by systemd? Looking at how we do it, It should be easily movable
> >to systemd? Can we consider doing this for Ocata?
> 
> I think we could look at using the redis cluster which allows multiple
> masters, but I am not sure this can happen in Ocata ... yet again, there
> shouldn't be in the telemetry roles any dependency on redis itself
> 
> if we were to use the cluster mode the only difference would probably be
> that the redis_vip will start balancing requests across the nodes

In general I am in favour to split out redis from pacemaker. There is
the question that in theory we'd have two potentially separate quorums,
but I think that with redis this should not be a big problem.

Maybe let's start with a prototype and see how things look and iterate
from there? I think it is a bit late for ocata, but we could at least
start the work without changing the defaults (i.e. let the operator
override the tripleo::service with a redis base profile instead of the
pacemaker one)

Does that make sense,
Michele

[1] https://review.openstack.org/#/q/topic:bp/composable-ha
-- 
Michele Baldessari
C2A5 9DA3 9961 4FFB E01B  D0BC DDD4 DCCB 7515 5C6D

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Move redis out of Pacemaker

[Pradeep Kilambi]

On Mon, Dec 12, 2016 at 8:58 AM, Giulio Fidente  wrote:

> On 12/12/2016 02:51 PM, Giulio Fidente wrote:
>
>> On 12/09/2016 04:49 PM, Pradeep Kilambi wrote:
>>
>>> Hey Folks:
>>>
>>> I would like to get some thoughts on $Subject. This came up when i was
>>> discussing the standalone roles for telemetry. Currently when we deploy
>>> redis in tripleo, its a pacemaker managed service. So if we were to
>>> deploy telemetry services on a dedicated node we could. But redis will
>>> have to be on a another node? (assuming we dont want to pull in
>>> pacemaker on to telemetry nodes).
>>>
>>
>> currently redis instances are not configured as a redis cluster but use
>> the master/slave replication model instead and pacemaker is taking care
>> of electing/relocating the redis master as needed
>>
>> there shouldn't be any dependency on the redis profile for the telemetry
>> roles, they should instead just point at the redis_vip
>>
>> the redis_vip is always guaranteed (by haproxy) to point to the redis
>> master
>>
>> With most services moved out of pacemaker in Newton, I think its time to
>>> move redis as well? Are there any constraints in moving redis to be
>>> managed by systemd? Looking at how we do it, It should be easily movable
>>> to systemd? Can we consider doing this for Ocata?
>>>
>>
>> I think we could look at using the redis cluster which allows multiple
>> masters, but I am not sure this can happen in Ocata ... yet again, there
>> shouldn't be in the telemetry roles any dependency on redis itself
>>
>> if we were to use the cluster mode the only difference would probably be
>> that the redis_vip will start balancing requests across the nodes
>>
>
> also note that the cluster model is not replicating data across the nodes,
> but rather sharding the data, with the promise of surviving failures of
> some of the nodes and I'm not sure we want that anyway
>
> https://redis.io/topics/cluster-spec


 I just thought we could use redis sentinel to achieve the HA and avoid
pacemaker headache. Which i think is already integrated a while ago in
tripleo. Its just not used, As pacemaker is taking over the control.

See https://redis.io/topics/sentinel



>
> --
> Giulio Fidente
> GPG KEY: 08D733BA
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Move redis out of Pacemaker

[Giulio Fidente]

On 12/12/2016 02:51 PM, Giulio Fidente wrote:

On 12/09/2016 04:49 PM, Pradeep Kilambi wrote:

Hey Folks:

I would like to get some thoughts on $Subject. This came up when i was
discussing the standalone roles for telemetry. Currently when we deploy
redis in tripleo, its a pacemaker managed service. So if we were to
deploy telemetry services on a dedicated node we could. But redis will
have to be on a another node? (assuming we dont want to pull in
pacemaker on to telemetry nodes).


currently redis instances are not configured as a redis cluster but use
the master/slave replication model instead and pacemaker is taking care
of electing/relocating the redis master as needed

there shouldn't be any dependency on the redis profile for the telemetry
roles, they should instead just point at the redis_vip

the redis_vip is always guaranteed (by haproxy) to point to the redis
master


With most services moved out of pacemaker in Newton, I think its time to
move redis as well? Are there any constraints in moving redis to be
managed by systemd? Looking at how we do it, It should be easily movable
to systemd? Can we consider doing this for Ocata?


I think we could look at using the redis cluster which allows multiple
masters, but I am not sure this can happen in Ocata ... yet again, there
shouldn't be in the telemetry roles any dependency on redis itself

if we were to use the cluster mode the only difference would probably be
that the redis_vip will start balancing requests across the nodes


also note that the cluster model is not replicating data across the 
nodes, but rather sharding the data, with the promise of surviving 
failures of some of the nodes and I'm not sure we want that anyway


https://redis.io/topics/cluster-spec
--
Giulio Fidente
GPG KEY: 08D733BA

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Move redis out of Pacemaker

[Giulio Fidente]

On 12/09/2016 04:49 PM, Pradeep Kilambi wrote:

Hey Folks:

I would like to get some thoughts on $Subject. This came up when i was
discussing the standalone roles for telemetry. Currently when we deploy
redis in tripleo, its a pacemaker managed service. So if we were to
deploy telemetry services on a dedicated node we could. But redis will
have to be on a another node? (assuming we dont want to pull in
pacemaker on to telemetry nodes).


currently redis instances are not configured as a redis cluster but use 
the master/slave replication model instead and pacemaker is taking care 
of electing/relocating the redis master as needed


there shouldn't be any dependency on the redis profile for the telemetry 
roles, they should instead just point at the redis_vip


the redis_vip is always guaranteed (by haproxy) to point to the redis master


With most services moved out of pacemaker in Newton, I think its time to
move redis as well? Are there any constraints in moving redis to be
managed by systemd? Looking at how we do it, It should be easily movable
to systemd? Can we consider doing this for Ocata?


I think we could look at using the redis cluster which allows multiple 
masters, but I am not sure this can happen in Ocata ... yet again, there 
shouldn't be in the telemetry roles any dependency on redis itself


if we were to use the cluster mode the only difference would probably be 
that the redis_vip will start balancing requests across the nodes

--
Giulio Fidente
GPG KEY: 08D733BA

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Move redis out of Pacemaker

[Emilien Macchi]

On Fri, Dec 9, 2016 at 10:49 AM, Pradeep Kilambi  wrote:
> Hey Folks:
>
> I would like to get some thoughts on $Subject. This came up when i was
> discussing the standalone roles for telemetry. Currently when we deploy
> redis in tripleo, its a pacemaker managed service. So if we were to deploy
> telemetry services on a dedicated node we could. But redis will have to be
> on a another node? (assuming we dont want to pull in pacemaker on to
> telemetry nodes).
>
> With most services moved out of pacemaker in Newton, I think its time to
> move redis as well? Are there any constraints in moving redis to be managed
> by systemd? Looking at how we do it, It should be easily movable to systemd?
> Can we consider doing this for Ocata?
>
> Thoughts?

If I understand correctly the work that is being done in Composable
HA, Redis will be able to be deployed with Pacemaker on a dedicated
node:
https://review.openstack.org/#/c/362389

I'm not sure if we can deploy it with systemd only, in a HA manner but
Michele could eventually confirm.

Regarding the timeline to do it in Ocata, I'm really not sure that's
doable, it depends on the Composable HA progress I guess.
-- 
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev