Re: [Openstack] [OpenStack] [keystone] How to make keystone highly available?

[Clint Byrum]

Excerpts from Alexandr Porunov's message of 2016-09-30 18:51:02 +0300:
> > How do you handle database high availability and replication?
> 
> Especially in my case, I don't care about tokens which will be lost after
> first keystone server dies. My services can authenticate again and get new
> tokens. It isn't critical. But if in your case it isn't acceptable then I
> would have used fernet tokens on your place (but they are a little bit
> bigger than uuid tokens). If you need small tokens, fast checks and you
> can't lose tokens then I would have used MariaDB Galera Cluster for the
> token replication.
> 

This assumes you have very little churn in your identity database, which
may not be true if you're using Heat, which creates users for software
configurations unless you use s3 tempurls.

There is also important realtime data with Fernet tokens in the
revocation_event table, but it is highly temporal, so if you lose it,
the effect is that for $token_ttl length of time, you might allow tokens
that were previously revoked to validate.

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [OpenStack] [keystone] How to make keystone highly available?

[Alexandr Porunov]

> How do you handle database high availability and replication?

Especially in my case, I don't care about tokens which will be lost after
first keystone server dies. My services can authenticate again and get new
tokens. It isn't critical. But if in your case it isn't acceptable then I
would have used fernet tokens on your place (but they are a little bit
bigger than uuid tokens). If you need small tokens, fast checks and you
can't lose tokens then I would have used MariaDB Galera Cluster for the
token replication.


On Fri, Sep 30, 2016 at 3:41 PM, Clint Byrum  wrote:

> Excerpts from Alexandr Porunov's message of 2016-09-20 17:09:06 +0300:
> > Hello everyone,
> >
> > Thank you all for your advice!
> > In my case one keystone server can easily hold a load. I don't need to
> > balance a load between two or more keystone servers. However I need two
> > keystone servers for high availability. So, I decided just to use two
> > keystone servers with the same virtual IP address.
> > Pros:
> > - I can easily configure HA
> > - I don't need to use a master/master replication for a database.
>
> How do you handle database high availability and replication?
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [OpenStack] [keystone] How to make keystone highly available?

[Clint Byrum]

Excerpts from Alexandr Porunov's message of 2016-09-20 17:09:06 +0300:
> Hello everyone,
> 
> Thank you all for your advice!
> In my case one keystone server can easily hold a load. I don't need to
> balance a load between two or more keystone servers. However I need two
> keystone servers for high availability. So, I decided just to use two
> keystone servers with the same virtual IP address.
> Pros:
> - I can easily configure HA
> - I don't need to use a master/master replication for a database.

How do you handle database high availability and replication?

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [OpenStack] [keystone] How to make keystone highly available?

[Turbo Fredriksson]

On Sep 21, 2016, at 6:48 AM, Van Leeuwen, Robert wrote:

> If I had these constraints I would add a loadbalancer-config on the same 
> machine that runs the OpenStack apis.

Now that I have multiple Neutron instances, how do I make my routers
HA?

I managed to make the router 'distributed', but there doesn't seem
to be a '--ha' for "neutron router-update" (like there is for
"neutron router-create").
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [OpenStack] [keystone] How to make keystone highly available?

[Alexandr Porunov]

Also, if you haven't changed default parameter for token cache time then it
has 5 min of cache (you can increase this time if you want). So, your 15-16
machines (or less if some of them doesn't use keystone like swift object,
container, account nodes) will use requests to keystone only once in 5
minutes. It is about one request in 18-19 seconds.

If you will use that scheme like I said it kill be about one request in
36-38 seconds.

I don't recomend you to use load balancer on both your keystone servers
because it isn't really effective. Half of your requests will ask one
keystone server to send request to another keystone server to get a token.
It isn't really effective because you send your request to both keystone
servers rather than one keystone server.

But in your situation I think it doesn't matter.

Sincerely,
Alexandr

On Wed, Sep 21, 2016 at 9:29 AM, Alexandr Porunov <
alexandr.poru...@gmail.com> wrote:

> > That is more like a hot-standby (only one server is used at any one time)
> > and I guess that's an option as well. But because I have limited
> resources,
> > I'd prefer, if possible, to use all of them all the time.
>
> Wrong. It depends on how you configured you cluster. You can point half of
> your machines to use keystone1 and other half machines to use keystone2.
> For it you have to configure your keepalived to manage two virtual ip
> address with different priority. Example:
> keystone1: 192.168.0.61, proirity 101
> keystone1: 192.168.0.62, priority 100
> keystone2: 192.168.0.61 proirity 100
> keystone2: 192.168.0.62, priority 101
>
> your 8 machines (nove, swift, glance, and so on) will use 192.168.0.61
> your other 8 machines will use 192.168.0.62
>
> Assuming your keystone1 is dead then keepalived will assign both IPs for
> keystone2 (i.e. 192.168.0.61, 192.168.0.62). Your all machines then
> automatically will use keystone2. After your keystone1 is returned back to
> life your keepalived will reassign IP address 192.168.0.61 back to
> keystone1 bacause 192.168.0.61 has higher priority for keystone1 than
> keystone2.
>
> Sincerely,
> Alexandr
>
> On Wed, Sep 21, 2016 at 12:44 AM, Turbo Fredriksson 
> wrote:
>
>> On Sep 20, 2016, at 10:06 PM, Alexandr Porunov wrote:
>>
>> > If you care about high availability (as I do) then you need to have
>> > additional keystone instance which will prevent your cluster from SPOF.
>>
>> That was the idea. One node is already dedicated for that, but I haven't
>> installed it yet, because I'm not sure what the right way to do this is.
>>
>> I've been reading the high availability docs for the last couple of weeks
>> now, but they don't talk about the most basic things :(. For once, they
>> get right down to it (which I've complained about other docs don't in the
>> past :D).
>>
>>
>> IF (!) I understand things correctly, services report themselves into
>> the catalog (which is basically the *SQL server). This catalog is
>> Keystone.
>>
>> So my first Keystone server have registered itself as:
>>
>> bladeA01:~# openstack endpoint list | grep keystone
>> | 26855d6e55284651a0fcaa5cf25b3d90 | europe-london | keystone |
>> identity| True| internal  | http://10.0.4.1:5000/v2.0
>>   |
>> | 72eb76cdb6cf4db3813eac4a683e4e34 | europe-london | keystone |
>> identity| True| public| http://10.0.4.1:5000/v2.0
>>   |
>> | c95ce66a4efa46b4855185d088279824 | europe-london | keystone |
>> identity| True| admin | http://10.0.4.1:35357/v2.0
>>|
>>
>> Now, I'm assuming that the second one will do the same (on ITS IP of
>> course). So "anyone" needing to contact Keystone, will, I assume,
>> consult this catalog and "pick one".
>>
>>
>> So if I use a DNS round-robin for the two keystones:
>>
>> openstack.domain.tld.1 IN A 10.0.4.1
>> openstack.domain.tld.1 IN A 10.0.4.9
>>
>> the entry for 'openstack.domain.tld.' will be invalidated every second,
>> practically making sure that a new request will be given every time.
>> The downside is that if the .1 keystone is down, the service will have
>> to wait for the time out before it can ask the next one.
>>
>> Round-robin is "poor mans load balancer", and it have many flaws, but
>> it will at least give some form of using all available resources at any
>> one time.
>>
>> > For it I use the same virtual IP address for both keystone instances
>> > managed by keepalived.
>>
>> That is more like a hot-standby (only one server is used at any one time)
>> and I guess that's an option as well. But because I have limited
>> resources,
>> I'd prefer, if possible, to use all of them all the time.
>>
>> Balancing the load is less of an issue than high availability is, but
>> if possible, I'd like to solve both of them :).
>>
>> > Also you can use peacemaker and other stuff to reach high availability
>>
>> Yes, but I'm guessing those need _another_ machine in front of the ones
>> I want to load 

Re: [Openstack] [OpenStack] [keystone] How to make keystone highly available?

[Alexandr Porunov]

> That is more like a hot-standby (only one server is used at any one time)
> and I guess that's an option as well. But because I have limited
resources,
> I'd prefer, if possible, to use all of them all the time.

Wrong. It depends on how you configured you cluster. You can point half of
your machines to use keystone1 and other half machines to use keystone2.
For it you have to configure your keepalived to manage two virtual ip
address with different priority. Example:
keystone1: 192.168.0.61, proirity 101
keystone1: 192.168.0.62, priority 100
keystone2: 192.168.0.61 proirity 100
keystone2: 192.168.0.62, priority 101

your 8 machines (nove, swift, glance, and so on) will use 192.168.0.61
your other 8 machines will use 192.168.0.62

Assuming your keystone1 is dead then keepalived will assign both IPs for
keystone2 (i.e. 192.168.0.61, 192.168.0.62). Your all machines then
automatically will use keystone2. After your keystone1 is returned back to
life your keepalived will reassign IP address 192.168.0.61 back to
keystone1 bacause 192.168.0.61 has higher priority for keystone1 than
keystone2.

Sincerely,
Alexandr

On Wed, Sep 21, 2016 at 12:44 AM, Turbo Fredriksson 
wrote:

> On Sep 20, 2016, at 10:06 PM, Alexandr Porunov wrote:
>
> > If you care about high availability (as I do) then you need to have
> > additional keystone instance which will prevent your cluster from SPOF.
>
> That was the idea. One node is already dedicated for that, but I haven't
> installed it yet, because I'm not sure what the right way to do this is.
>
> I've been reading the high availability docs for the last couple of weeks
> now, but they don't talk about the most basic things :(. For once, they
> get right down to it (which I've complained about other docs don't in the
> past :D).
>
>
> IF (!) I understand things correctly, services report themselves into
> the catalog (which is basically the *SQL server). This catalog is Keystone.
>
> So my first Keystone server have registered itself as:
>
> bladeA01:~# openstack endpoint list | grep keystone
> | 26855d6e55284651a0fcaa5cf25b3d90 | europe-london | keystone |
> identity| True| internal  | http://10.0.4.1:5000/v2.0
>   |
> | 72eb76cdb6cf4db3813eac4a683e4e34 | europe-london | keystone |
> identity| True| public| http://10.0.4.1:5000/v2.0
>   |
> | c95ce66a4efa46b4855185d088279824 | europe-london | keystone |
> identity| True| admin | http://10.0.4.1:35357/v2.0
>  |
>
> Now, I'm assuming that the second one will do the same (on ITS IP of
> course). So "anyone" needing to contact Keystone, will, I assume,
> consult this catalog and "pick one".
>
>
> So if I use a DNS round-robin for the two keystones:
>
> openstack.domain.tld.1 IN A 10.0.4.1
> openstack.domain.tld.1 IN A 10.0.4.9
>
> the entry for 'openstack.domain.tld.' will be invalidated every second,
> practically making sure that a new request will be given every time.
> The downside is that if the .1 keystone is down, the service will have
> to wait for the time out before it can ask the next one.
>
> Round-robin is "poor mans load balancer", and it have many flaws, but
> it will at least give some form of using all available resources at any
> one time.
>
> > For it I use the same virtual IP address for both keystone instances
> > managed by keepalived.
>
> That is more like a hot-standby (only one server is used at any one time)
> and I guess that's an option as well. But because I have limited resources,
> I'd prefer, if possible, to use all of them all the time.
>
> Balancing the load is less of an issue than high availability is, but
> if possible, I'd like to solve both of them :).
>
> > Also you can use peacemaker and other stuff to reach high availability
>
> Yes, but I'm guessing those need _another_ machine in front of the ones
> I want to load balance. And if that goes down, EVERYTHING stops working.
> Unless they are clustered, which require _even more_ machines!
>
> Which I don't have. I don't want to dedicate a dual CPU, eight core
> Intel Xeon E5530 @ 2.40GHz just to swap traffic around! It's a huge
> waste of precious resources. And if I add smaller machines outside of
> the blade center, then ALL traffic needs to go out in the rack and then
> back in, which will affect performance (which is already kind'a bad
> because it's an older setup with only Gbps links).
>
> I had to dedicate one whole switch (there's two Cisco 3020 in the
> blade center) just for the truncated link down to the storage. Which
> is another think I need to solve eventually - it is currently the biggest,
> badest SPOF :( :(.
> --
> Michael Jackson is not going to buried or cremated
> but recycled into shopping bags so he can remain white,
> plastic and dangerous for kids to play with.
>
>
> ___
> Mailing list: 

Re: [Openstack] [OpenStack] [keystone] How to make keystone highly available?

[Van Leeuwen, Robert]

>> Also you can use peacemaker and other stuff to reach high availability
  >Yes, but I'm guessing those need _another_ machine in front of the ones
>   I want to load balance. And if that goes down, EVERYTHING stops working.
>  Unless they are clustered, which require _even more_ machines!

If I had these constraints I would add a loadbalancer-config on the same 
machine that runs the OpenStack apis.
So add keepalived with haproxy on the 2 “OpenStack controllers”.
Keepalived for a VRRP based HA-IP and haproxy for load-balancing (with the nice 
addition you can do SSL offloading on haproxy).
This is somewhat similar to what what most vendors (Mirantis/Ubuntu/RedHat) are 
doing AFAIK.

If you use neutron with the L3 agent you will also have to think about HA of 
that.
That’s a different story though and I am not really up to speed on the latest 
and greatest there.

Cheers,
Robert

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [OpenStack] [keystone] How to make keystone highly available?

[Turbo Fredriksson]

On Sep 20, 2016, at 10:06 PM, Alexandr Porunov wrote:

> If you care about high availability (as I do) then you need to have
> additional keystone instance which will prevent your cluster from SPOF.

That was the idea. One node is already dedicated for that, but I haven't
installed it yet, because I'm not sure what the right way to do this is.

I've been reading the high availability docs for the last couple of weeks
now, but they don't talk about the most basic things :(. For once, they
get right down to it (which I've complained about other docs don't in the
past :D).


IF (!) I understand things correctly, services report themselves into
the catalog (which is basically the *SQL server). This catalog is Keystone.

So my first Keystone server have registered itself as:

bladeA01:~# openstack endpoint list | grep keystone
| 26855d6e55284651a0fcaa5cf25b3d90 | europe-london | keystone | identity
| True| internal  | http://10.0.4.1:5000/v2.0  |
| 72eb76cdb6cf4db3813eac4a683e4e34 | europe-london | keystone | identity
| True| public| http://10.0.4.1:5000/v2.0  |
| c95ce66a4efa46b4855185d088279824 | europe-london | keystone | identity
| True| admin | http://10.0.4.1:35357/v2.0 |

Now, I'm assuming that the second one will do the same (on ITS IP of
course). So "anyone" needing to contact Keystone, will, I assume,
consult this catalog and "pick one".


So if I use a DNS round-robin for the two keystones:

openstack.domain.tld.1 IN A 10.0.4.1
openstack.domain.tld.1 IN A 10.0.4.9

the entry for 'openstack.domain.tld.' will be invalidated every second,
practically making sure that a new request will be given every time.
The downside is that if the .1 keystone is down, the service will have
to wait for the time out before it can ask the next one.

Round-robin is "poor mans load balancer", and it have many flaws, but
it will at least give some form of using all available resources at any
one time.

> For it I use the same virtual IP address for both keystone instances
> managed by keepalived.

That is more like a hot-standby (only one server is used at any one time)
and I guess that's an option as well. But because I have limited resources,
I'd prefer, if possible, to use all of them all the time.

Balancing the load is less of an issue than high availability is, but
if possible, I'd like to solve both of them :).

> Also you can use peacemaker and other stuff to reach high availability

Yes, but I'm guessing those need _another_ machine in front of the ones
I want to load balance. And if that goes down, EVERYTHING stops working.
Unless they are clustered, which require _even more_ machines!

Which I don't have. I don't want to dedicate a dual CPU, eight core
Intel Xeon E5530 @ 2.40GHz just to swap traffic around! It's a huge
waste of precious resources. And if I add smaller machines outside of
the blade center, then ALL traffic needs to go out in the rack and then
back in, which will affect performance (which is already kind'a bad
because it's an older setup with only Gbps links).

I had to dedicate one whole switch (there's two Cisco 3020 in the
blade center) just for the truncated link down to the storage. Which
is another think I need to solve eventually - it is currently the biggest,
badest SPOF :( :(.
--
Michael Jackson is not going to buried or cremated
but recycled into shopping bags so he can remain white,
plastic and dangerous for kids to play with.


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [OpenStack] [keystone] How to make keystone highly available?

[Alexandr Porunov]

I think that I haven't understood your situation correctly but I will try
to suggest something:

If you have a big load on your keystone server you can spread the load
between two or more servers by using load balancers like HAProxy. But it
wouldn't work if keystone instances not shared tokens with each other. For
it you have to share databse to store all tokens which are generated by
 your keystone servers. Personally I prefer MariaDB Galera Cluster because
it isn't hard to install and works very nice. To reach high availability
for load balancing you need to have one more load balancer managed by
keepalived.
(I am assuming that you don't have such a load)

If you care about high availability (as I do) then you need to have
additional keystone instance which will prevent your cluster from SPOF. For
it I use the same virtual IP address for both keystone instances managed by
keepalived.

Also you can use peacemaker and other stuff to reach high availability but
I can't give you advice for those tools because I haven't used them.

In your situation I would installed additional keystone instance in one of
your node and keepalived on both nodes to prevent SPOF.

Sincerely,
Alexandr

On Tue, Sep 20, 2016 at 10:56 PM, Turbo Fredriksson 
wrote:

> On Sep 20, 2016, at 3:09 PM, Alexandr Porunov wrote:
>
> > So, I decided just to use two keystone servers with the same virtual IP
> address.
>
> Now that you've made your decision, I'd like to ask some
> followup questions for my own decision if I may.
>
>
> Most everyone have talked about some kind of load balancer or
> HA solution.
>
> My setup is such that that wouldn't be .. "workable".
>
> My whole Openstack setup is installed on a HP Bladecenter
> with 16 half hight nodes so all communication etc is pretty
> much internal in the blade center. I.e, the only external
> link is the one to the rest of the flat and then to the
> gatway/firewall/NATbox.
>
> Also, having a LB in front of MySQL (and possibly one more
> in front of Keystone), that will introduce a "single point
> of failure" that I'd prefer not to have (even if it was
> practically possible - I don't want to dedicate a whole
> BC node just for that, they're to precious).
>
>
> What would be the impact and/or problem by using DNS round
> robin for MySQL and Keystone for example? And then have
> MySQL in a master-master setup. I've never done that either,
> so I'm not sure how good idea that would be.. Any pointers?
>
> The only DB I've ever had the .. "displeasure" to try to get
> to work in a master-master setup is OpenLDAP and I can remember
> (vividly!) the discussions that's been on the OpenLDAP lists
> about this over the years! In essence, "don't, for the love of
> whoever, do it!!".
> --
> Choose a job you love, and you will never have
> to work a day in your life.
>
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [OpenStack] [keystone] How to make keystone highly available?

[Turbo Fredriksson]

On Sep 20, 2016, at 3:09 PM, Alexandr Porunov wrote:

> So, I decided just to use two keystone servers with the same virtual IP 
> address.

Now that you've made your decision, I'd like to ask some
followup questions for my own decision if I may.


Most everyone have talked about some kind of load balancer or
HA solution.

My setup is such that that wouldn't be .. "workable".

My whole Openstack setup is installed on a HP Bladecenter
with 16 half hight nodes so all communication etc is pretty
much internal in the blade center. I.e, the only external
link is the one to the rest of the flat and then to the
gatway/firewall/NATbox.

Also, having a LB in front of MySQL (and possibly one more
in front of Keystone), that will introduce a "single point
of failure" that I'd prefer not to have (even if it was
practically possible - I don't want to dedicate a whole
BC node just for that, they're to precious).


What would be the impact and/or problem by using DNS round
robin for MySQL and Keystone for example? And then have
MySQL in a master-master setup. I've never done that either,
so I'm not sure how good idea that would be.. Any pointers?

The only DB I've ever had the .. "displeasure" to try to get
to work in a master-master setup is OpenLDAP and I can remember
(vividly!) the discussions that's been on the OpenLDAP lists
about this over the years! In essence, "don't, for the love of
whoever, do it!!".
--
Choose a job you love, and you will never have
to work a day in your life.


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [OpenStack] [keystone] How to make keystone highly available?

[Alexandr Porunov]

Hello everyone,

Thank you all for your advice!
In my case one keystone server can easily hold a load. I don't need to
balance a load between two or more keystone servers. However I need two
keystone servers for high availability. So, I decided just to use two
keystone servers with the same virtual IP address.
Pros:
- I can easily configure HA
- I don't need to use a master/master replication for a database.
- I don't need to use load balancers.
- I don't need to learn peacemaker
Cons:
- It will not increase the overall throughput
- One of the keystone servers will always be at a cold state

However. We can rid of cons by using two virtual IP adresses. So, we can
use a half proxy servers with keystone1 and another half of proxy servers
with keystone2. If one of the keystone servers will be dead then keepalived
will assign both virtual ip addresses to the alive node. After node will be
brought back to live keepalived will reassign IP addresses back (It isn't
the load balancing but our throughput will increase). If our clients will
use proxy servers with both keystone1 and keystone 2 then we have to use
master/master replication for the database.

Best regards,
Alexandr

On Tue, Sep 20, 2016 at 8:49 AM, Van Leeuwen, Robert 
wrote:

> >
>
> > Hello,
>
> >
>
> > I am thinking about using the keystone as an authentication system but I
> am afraid about failures which can affect all the cluster. In fact if the
> keystone server dies then our full cluster will stop. It would be better if
> we could use HA with the keystone. Then if > our primary keystone server
> dies we have to elect a new primary keystoe server. Are there some tools
> which can be used in HA deployment?
>
> >
>
> > Any piece of advice will be valuable
>
>
>
> IMHO: get a proper ha load-balancer solution, make sure you install at
> least 2 of all APIS and load-balance in active-active mode
>
> The OpenStack APIs are stateless so you can run multiple servers at the
> same time as long as they can connect to the same database backend.
>
>
>
> The load-balancer is usable for all APIs, you can use it as the HA-IP for
> mysql and can also be used to offload SSL so you just have one place to
> configure your certificates.
>
> (for galera, make use of a “sorry-server” and not active-active
> load-balancing for writes)
>
>
>
> The great thing about a load-balancer is that you remove all HA complexity
> out of the OpenStack setup.
>
> In my personal opinion the pacemaker setups can function OK but pacemaker
> is a complex piece of software and it is not unlikely to cause downtime
> either to misconfiguration or inexperienced people operating it.
>
> Especially if pacemaker is also starting/stopping mysql/rabbit/openstack
> services and not only moving around a few IPs.
>
> If you are going the pacemaker way make sure you play around with it quite
> a bit and do failure tests so you are comfortable with the commands and
> know what to look for when things go wrong.
>
> Do not forget: any other people operating the pacemaker cluster will need
> that knowledge and a 2 node cluster is not a real cluster (split-brains).
>
>
>
> Cheers,
>
> Robert van Leeuwen
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [OpenStack] [keystone] How to make keystone highly available?

[Clint Byrum]

Excerpts from Alexandr Porunov's message of 2016-09-19 21:46:54 +0300:
> Hello,
> 
> I am thinking about using the keystone as an authentication system but I am
> afraid about failures which can affect all the cluster. In fact if the
> keystone server dies then our full cluster will stop. It would be better if
> we could use HA with the keystone. Then if our primary keystone server dies
> we have to elect a new primary keystoe server. Are there some tools which
> can be used in HA deployment?
> 
> Any piece of advice will be valuable

Hi Alexandr.

Keystone is a stateless application. It stores all of the state in
a SQL database, or in files that are immutable (for Fernet token key
files). So, what you really want is an HA SQL solution, and (if you're
using Fernet tokens) a key sync mechanism.

There are many such standard solutions. If you're fine with one server
worth of capacity, then DRBD+Pacemaker+Corosync are a pretty simple
option to keep MySQL HA. Once you do that, you can just use a load
balancer or something like UCARP/VRRP to make sure HTTP requests arrive
at a working keystone node.

Good luck.

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [OpenStack] [keystone] How to make keystone highly available?

[John van Ommen]

If you'd like to avoid the pain of configuring this manually, Helion
OpenStack has many HA features enabled by default:

http://docs.hpcloud.com/#3.x/helion/planning/high_availability.html

On Mon, Sep 19, 2016 at 2:10 PM, Alexandr Porunov
 wrote:
> Thank you for pointing on it. I haven't yet use pacemaker but I will try to
> configure it.
>
> Sincerely,
> Alexandr
>
> On Mon, Sep 19, 2016 at 10:23 PM, Remo Mattei  wrote:
>>
>> There is already ha options with pacemaker and corosync. Have you looked
>> at that?
>>
>> Inviato da iPhone
>>
>> Il giorno 19 set 2016, alle ore 11:46, Alexandr Porunov
>>  ha scritto:
>>
>> Hello,
>>
>> I am thinking about using the keystone as an authentication system but I
>> am afraid about failures which can affect all the cluster. In fact if the
>> keystone server dies then our full cluster will stop. It would be better if
>> we could use HA with the keystone. Then if our primary keystone server dies
>> we have to elect a new primary keystoe server. Are there some tools which
>> can be used in HA deployment?
>>
>> Any piece of advice will be valuable
>>
>> Sincerely,
>> Alexandr
>> !DSPAM:1,57e036cd282281177211101!
>>
>> ___
>> Mailing list:
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to : openstack@lists.openstack.org
>> Unsubscribe :
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>
>>
>> !DSPAM:1,57e036cd282281177211101!
>
>
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [OpenStack] [keystone] How to make keystone highly available?

[Alexandr Porunov]

Thank you for pointing on it. I haven't yet use pacemaker but I will try to
configure it.

Sincerely,
Alexandr

On Mon, Sep 19, 2016 at 10:23 PM, Remo Mattei  wrote:

> There is already ha options with pacemaker and corosync. Have you looked
> at that?
>
> Inviato da iPhone
>
> Il giorno 19 set 2016, alle ore 11:46, Alexandr Porunov <
> alexandr.poru...@gmail.com> ha scritto:
>
> Hello,
>
> I am thinking about using the keystone as an authentication system but I
> am afraid about failures which can affect all the cluster. In fact if the
> keystone server dies then our full cluster will stop. It would be better if
> we could use HA with the keystone. Then if our primary keystone server dies
> we have to elect a new primary keystoe server. Are there some tools which
> can be used in HA deployment?
>
> Any piece of advice will be valuable
>
> Sincerely,
> Alexandr
> !DSPAM:1,57e036cd282281177211101!
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
>
>
> !DSPAM:1,57e036cd282281177211101!
>
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [OpenStack] [keystone] How to make keystone highly available?

[Remo Mattei]

There is already ha options with pacemaker and corosync. Have you looked at 
that? 

Inviato da iPhone

> Il giorno 19 set 2016, alle ore 11:46, Alexandr Porunov 
>  ha scritto:
> 
> Hello,
> 
> I am thinking about using the keystone as an authentication system but I am 
> afraid about failures which can affect all the cluster. In fact if the 
> keystone server dies then our full cluster will stop. It would be better if 
> we could use HA with the keystone. Then if our primary keystone server dies 
> we have to elect a new primary keystoe server. Are there some tools which can 
> be used in HA deployment?
> 
> Any piece of advice will be valuable
> 
> Sincerely,
> Alexandr
> !DSPAM:1,57e036cd282281177211101!
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> 
> 
> !DSPAM:1,57e036cd282281177211101!
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

[Openstack] [OpenStack] [keystone] How to make keystone highly available?

[Alexandr Porunov]

Hello,

I am thinking about using the keystone as an authentication system but I am
afraid about failures which can affect all the cluster. In fact if the
keystone server dies then our full cluster will stop. It would be better if
we could use HA with the keystone. Then if our primary keystone server dies
we have to elect a new primary keystoe server. Are there some tools which
can be used in HA deployment?

Any piece of advice will be valuable

Sincerely,
Alexandr
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack