That's weird. I initially tested it in US East 2.
I'd check that the packages/versions for python-azure-sdk,
python-msrest, python-msrestazure, python-keyring, etc are the same
on your nodes in Europe vs US locations.
If they are the same you could create a support ticket on Azure to
ask if there's been any changes to the API in the US, or if they can
see what's causing it to fail in the US location and not in Europe.
On 12/02/19 17:37 +0100, Thomas Berreis wrote:
Network fencing is working correctly therefore credentials should be fine.
Maybe there are differences between Azure Cloud in Europe and US because I
get these errors only with our clusters in US Central and US East 2 but not
in Europe West and North. Any hint?
Thomas
-----Original Message-----
From: Users [mailto:[email protected]] On Behalf Of Oyvind
Albrigtsen
Sent: Dienstag, 12. Februar 2019 09:51
To: Cluster Labs - All topics related to open-source clustering welcomed
<[email protected]>
Subject: Re: [ClusterLabs] fence_azure_arm flooding syslog
At least that limits what the issue could be.
I'd check if the credentials allows getting and setting
network-settings:
https://github.com/ClusterLabs/fence-agents/blob/master/agents/azure_arm/fen
ce_azure_arm.py#L204
On 11/02/19 16:46 +0100, Thomas Berreis wrote:
Hi Oyvind,
Debug mode doesn't help me. The error occurs after successfully getting
Bearer token. No other issues are shown:
DEBUG:requests_oauthlib.oauth2_session:Invoking 0 token response hooks.
2019-02-11 15:27:12,687 DEBUG: Invoking 0 token response hooks.
DEBUG:requests_oauthlib.oauth2_session:Obtained token {u'resource':
u'https://management.core.windows.net/', u'access_token': ***',
u'ext_expires_in': u'3599', u'expires_in': u'3599', u'expires_at':
1549902431.687557, u'token_type': u'Bearer', u'not_before':
u'1549898532',
u'expires_on': u'1549902432'}.
2019-02-11 15:27:12,687 DEBUG: Obtained token {u'resource':
u'https://management.core.windows.net/', u'access_token': u'***',
u'ext_expires_in': u'3599', u'expires_in': u'3599', u'expires_at':
1549902431.687557, u'token_type': u'Bearer', u'not_before':
u'1549898532',
u'expires_on': u'1549902432'}.
WARNING:msrestazure.azure_active_directory:Keyring cache token has failed:
No recommended backend was available. Install the keyrings.alt package
if you want to use the non-recommended backends. See README.rst for
details.
2019-02-11 15:27:12,687 WARNING: Keyring cache token has failed: No
recommended backend was available. Install the keyrings.alt package if
you want to use the non-recommended backends. See README.rst for details.
DEBUG:requests_oauthlib.oauth2_session:Encoding `client_id` "***" with
`client_secret` as Basic auth credentials.
2019-02-11 15:27:12,727 DEBUG: Encoding `client_id` "***" with
`client_secret` as Basic auth credentials.
DEBUG:requests_oauthlib.oauth2_session:Requesting url
https://login.microsoftonline.com/***/oauth2/token using method POST.
2019-02-11 15:27:12,727 DEBUG: Requesting url
https://login.microsoftonline.com/***/oauth2/token using method POST.
2019-02-11 15:27:13,072 DEBUG: Obtained token {u'resource':
u'https://management.core.windows.net/', u'access_token': u'e***w',
u'ext_expires_in': u'3600', u'expires_in': u'3600', u'expires_at':
1549902433.072768, u'token_type': u'Bearer', u'not_before':
u'1549898533',
u'expires_on': u'1549902433'}.
WARNING:msrestazure.azure_active_directory:Keyring cache token has failed:
No recommended backend was available. Install the keyrings.alt package
if you want to use the non-recommended backends. See README.rst for
details.
2019-02-11 15:27:13,073 WARNING: Keyring cache token has failed: No
recommended backend was available. Install the keyrings.alt package if
you want to use the non-recommended backends. See README.rst for details.
INFO:root:getting power status for VM node01
Thomas
-----Original Message-----
From: Users [mailto:[email protected]] On Behalf Of Oyvind
Albrigtsen
Sent: Montag, 11. Februar 2019 14:58
To: Cluster Labs - All topics related to open-source clustering
welcomed <[email protected]>
Subject: Re: [ClusterLabs] fence_azure_arm flooding syslog
You can try adding verbose=1 and see if that gives you some more
details on where it's failing.
On 11/02/19 14:47 +0100, Thomas Berreis wrote:
We need this feature because shutdown / reboot takes too much time ( >
5
min) and network fencing fences the virtual machine much faster ( < 5
sec).
We finished all the required steps and network fencing works as
expected but I'm still confused about these errors in the log and the
failure counts showed by pcs ...
fence_azure_arm: Keyring cache token has failed: No recommended
backend was available. Install the keyrings.alt package if you want to
use the non-recommended backends. See README.rst for details.
stonith-ng[7789]: warning: fence_azure_arm[7896] stderr: [ 2019-02-11
13:41:19,178 WARNING: Keyring cache token has failed: No recommended
backend was available. Install the keyrings.alt package if you want to
use the non-recommended backends. See README.rst for details. ]
stonith-net_monitor_60000 on node02 'unknown error' (1): call=369,
status=Timed Out, exitreason='', last-rc-change='Mon Feb 11 10:38:02
2019', queued=0ms, exec=20007ms
Thomas
-----Original Message-----
From: Users [mailto:[email protected]] On Behalf Of Oyvind
Albrigtsen
Sent: Montag, 11. Februar 2019 13:58
To: Cluster Labs - All topics related to open-source clustering
welcomed <[email protected]>
Subject: Re: [ClusterLabs] fence_azure_arm flooding syslog
Oh. Thanks for clarifying.
Network-fencing requires extra setup as explained if you run "pcs
stonith resource describe fence_azure_arm".
You probably dont need network-fencing, so you can just skip that part.
On 11/02/19 13:50 +0100, Thomas Berreis wrote:
It seems like the issue might be the space in "pcmk_host_list= node01".
Sorry, this typo was only in my mail because I anonymized in- and output.
The issue only occurs when I add the parameter "network-fencing=on".
WARNING:msrestazure.azure_active_directory:Keyring cache token has
failed:
No recommended backend was available. Install the keyrings.alt
package if you want to use the non-recommended backends. See
README.rst for
details.
2019-02-11 12:29:37,079 WARNING: Keyring cache token has failed: No
recommended backend was available. Install the keyrings.alt package
if you want to use the non-recommended backends. See README.rst for
details.
Thomas
-----Original Message-----
From: Users [mailto:[email protected]] On Behalf Of
Oyvind Albrigtsen
Sent: Montag, 11. Februar 2019 10:38
To: Cluster Labs - All topics related to open-source clustering
welcomed <[email protected]>
Subject: Re: [ClusterLabs] fence_azure_arm flooding syslog
pcmk_host_list is used by Pacemaker to limit which hosts can be
fenced by the STONITH device, so you'll have to use the plug
parameter for
testing.
Pacemaker uses it internally when it fences a host, so you wouldnt
use plug= as part of the pcs stonith create command.
It seems like the issue might be the space in "pcmk_host_list= node01".
Oyvind
On 10/02/19 14:43 +0100, Thomas Berreis wrote:
Hi Oyvind,
Seems, option "pcmk_host_list" is not supported by fence_azure_arm.
Instead fence_azure_arm requires "plug" and however this is not
supported
by pcs.
What's the correct syntax to add such a stonith device with
fence_azure_arm?
[root ~]# fence_azure_arm
login=***
network-fencing=on
passwd=***
pcmk_host_list= node01
resourceGroup=***
retry_on=0
subscriptionId=***
tenantId=***
WARNING:root:Parse error: Ignoring unknown option 'pcmk_host_list=
node01'
ERROR:root:Failed: You have to enter plug number or machine
identification
2019-02-10 12:54:51,862 ERROR: Failed: You have to enter plug number
or machine identification
------------------------------
[root ~]# fence_azure_arm
login=***
network-fencing=on
passwd=***
pcmk_host_list= node01
resourceGroup=***
retry_on=0
subscriptionId=***
tenantId=***
plug=node01
WARNING:root:Parse error: Ignoring unknown option 'pcmk_host_list=
node01'
WARNING:msrestazure.azure_active_directory:Keyring cache token has
failed:
No recommended backend was available.
2019-02-10 12:56:30,377 WARNING: Keyring cache token has failed: No
recommended backend was available. Install th
WARNING:msrestazure.azure_active_directory:Keyring cache token has
failed:
No recommended backend was available.
2019-02-10 12:56:30,729 WARNING: Keyring cache token has failed: No
recommended backend was available. Install th
Success: Powered OFF
------------------------------
[root ~]# fence_azure_arm
login=***
network-fencing=on
passwd=***
plug=node01
resourceGroup=***
retry_on=0
subscriptionId=***
tenantId=***
Success: Powered OFF
------------------------------
[root ~]# pcs stonith create stonith-net fence_azure_arm [...]
plug=node01 network-fencing=on
Error: invalid stonith option 'plug'
[root ~]# fence_azure_arm -h
Usage:
fence_azure_arm [options]
Options:
[...]
-n, --plug=[id] Physical plug number on device, UUID
or
identification of machine
Thanks!
Thomas
-----Original Message-----
From: Users [mailto:[email protected]] On Behalf Of
Oyvind Albrigtsen
Sent: Freitag, 8. Februar 2019 14:16
To: Cluster Labs - All topics related to open-source clustering
welcomed <[email protected]>
Subject: Re: [ClusterLabs] fence_azure_arm flooding syslog
Hi Thomas,
It looks like it's trying to use credentials from keyring, which
generates these error messages.
Maybe there's a typo in one of the authentitcation parameters in
your config. Can you post the output of "pcs stonith show --full"?
You can try running fence_azure_arm without parameters on the
command line and copy the parameters (one per line) to it, and add e.g.
action=status and end with Ctrl+D to see .
Oyvind
On 08/02/19 14:02 +0100, Thomas Berreis wrote:
Hi,
I'm having a two-node cluster in Azure with fencing enabled via
module fence_azure_arm.
The cluster management is flooding my syslog on both nodes with
these
messages:
** fence_azure_arm: Keyring cache token has failed: No recommended
backend was available. Install the keyrings.alt package if you want
to use the non-recommended backends. See README.rst for details.
** stonith-ng[5220]: warning: fence_azure_arm[14847] stderr: [
2019-02-08
12:52:02,703 WARNING: Keyring cache token has failed: No
recommended backend was available. Install the keyrings.alt package
if you want to use the non-recommended backends. See README.rst for
details. ]
Can anyone give me a hint how to fix these issue?
pcs version: 0.9.165
pacemaker version: 1.1.19-8.el7_6.1
fence_azure_arm version: 4.2.1
corosync 2.4.3
Thanks!
Thomas
_______________________________________________
Users mailing list: [email protected]
https://lists.clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Users mailing list: [email protected]
https://lists.clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Users mailing list: [email protected]
https://lists.clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Users mailing list: [email protected]
https://lists.clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Users mailing list: [email protected]
https://lists.clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Users mailing list: [email protected]
https://lists.clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Users mailing list: [email protected]
https://lists.clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Users mailing list: [email protected]
https://lists.clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Users mailing list: [email protected]
https://lists.clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Users mailing list: [email protected]
https://lists.clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Users mailing list: [email protected]
https://lists.clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Users mailing list: [email protected]
https://lists.clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
