Hello all, I recently found myself with a requirement for Sahara and, for the most part, got the initial setup done without incident. I've created node and cluster templates, installed images, and gotten to the point of creating clusters.
When clusters are created, the structure is set up, but when it actually goes to provision things, it fails creating a trust. 2016-09-01 11:20:13.775 17080 DEBUG keystoneclient.auth.identity.v3.base [req-41d4dbfd-9fec-468c-8bfa-60b225c16fea 88e701edf4a0407593cc03cb74d4df57 0d81fa86513d4e3d9ed88cdc8c0b7d73 - - -] [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] Making authentication request to https://internal-osapi.rwdc1.cirrusseven.net:5000/v3/auth/tokens get_auth_ref /usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v3/base.py:188 2016-09-01 11:20:13.814 17080 DEBUG keystoneclient.session [req-41d4dbfd-9fec-468c-8bfa-60b225c16fea 88e701edf4a0407593cc03cb74d4df57 0d81fa86513d4e3d9ed88cdc8c0b7d73 - - -] [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] Request returned failure status: 401 request /usr/lib/python2.7/site-packages/keystoneclient/session.py:419 2016-09-01 11:20:13.815 17080 ERROR sahara.service.trusts [req-41d4dbfd-9fec-468c-8bfa-60b225c16fea 88e701edf4a0407593cc03cb74d4df57 0d81fa86513d4e3d9ed88cdc8c0b7d73 - - -] [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] Unable to create trust (reason: The request you have made requires authentication. (HTTP 401) (Request-ID: req-6099c99a-eb75-4dd3-b335-61f47b16d265)) 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [req-41d4dbfd-9fec-468c-8bfa-60b225c16fea 88e701edf4a0407593cc03cb74d4df57 0d81fa86513d4e3d9ed88cdc8c0b7d73 - - -] [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] Error during operating on cluster (reason: Failed to create trust 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] Traceback (most recent call last): 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] File "/usr/lib/python2.7/site-packages/sahara/service/ops.py", line 192, in wrapper 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] f(cluster_id, *args, **kwds) 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] File "/usr/lib/python2.7/site-packages/sahara/service/ops.py", line 273, in _provision_cluster 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] ctx, cluster, plugin = _prepare_provisioning(cluster_id) 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] File "/usr/lib/python2.7/site-packages/sahara/service/ops.py", line 254, in _prepare_provisioning 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] _setup_trust_for_cluster(cluster) 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] File "/usr/lib/python2.7/site-packages/sahara/service/ops.py", line 180, in _setup_trust_for_cluster 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] trusts.create_trust_for_cluster(cluster) 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] File "/usr/lib/python2.7/site-packages/sahara/service/trusts.py", line 97, in create_trust_for_cluster 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] allow_redelegation=True) 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] File "/usr/lib/python2.7/site-packages/sahara/service/trusts.py", line 75, in create_trust 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] raise ex.CreationFailed(_('Failed to create trust')) 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] CreationFailed: Failed to create trust 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] Error ID: ec8e5abf-cee8-429b-8258-3a7b98d1e966 2016-09-01 11:20:13.910 17080 ERROR sahara.service.ops [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] 2016-09-01 11:20:14.097 17080 DEBUG keystoneclient.auth.identity.v3.base [req-41d4dbfd-9fec-468c-8bfa-60b225c16fea 88e701edf4a0407593cc03cb74d4df57 0d81fa86513d4e3d9ed88cdc8c0b7d73 - - -] [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] Making authentication request to https://internal-osapi.rwdc1.cirrusseven.net:5000/v3/auth/tokens get_auth_ref /usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v3/base.py:188 2016-09-01 11:20:14.129 17080 DEBUG keystoneclient.session [req-41d4dbfd-9fec-468c-8bfa-60b225c16fea 88e701edf4a0407593cc03cb74d4df57 0d81fa86513d4e3d9ed88cdc8c0b7d73 - - -] [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] Request returned failure status: 401 request /usr/lib/python2.7/site-packages/keystoneclient/session.py:419 2016-09-01 11:20:14.130 17080 ERROR sahara.service.trusts [req-41d4dbfd-9fec-468c-8bfa-60b225c16fea 88e701edf4a0407593cc03cb74d4df57 0d81fa86513d4e3d9ed88cdc8c0b7d73 - - -] [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] Unable to create trust (reason: The request you have made requires authentication. (HTTP 401) (Request-ID: req-ca6ffb9c-ab1b-4c03-af9a-e2570d2fc791)) 2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [req-41d4dbfd-9fec-468c-8bfa-60b225c16fea 88e701edf4a0407593cc03cb74d4df57 0d81fa86513d4e3d9ed88cdc8c0b7d73 - - -] [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] Error during rollback of cluster (reason: Failed to create trust 2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] Traceback (most recent call last): 2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] File "/usr/lib/python2.7/site-packages/sahara/service/ops.py", line 210, in wrapper 2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] if _rollback_cluster(cluster, ex): 2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] File "/usr/lib/python2.7/site-packages/sahara/service/ops.py", line 238, in _rollback_cluster 2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] _setup_trust_for_cluster(cluster) 2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] File "/usr/lib/python2.7/site-packages/sahara/service/ops.py", line 180, in _setup_trust_for_cluster 2016-09-01 11:20:14" .228 17080 ERROR sahara.service.ops [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] trusts.create_trust_for_cluster(cluster) 2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] File "/usr/lib/python2.7/site-packages/sahara/service/trusts.py", line 97, in create_trust_for_cluster 2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] allow_redelegation=True) 2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] File "/usr/lib/python2.7/site-packages/sahara/service/trusts.py", line 75, in create_trust 2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] raise ex.CreationFailed(_('Failed to create trust')) 2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] CreationFailed: Failed to create trust 2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] Error ID: e811d08e-e1a6-450f-aaa7-bb114fc1d2de 2016-09-01 11:20:14.228 17080 ERROR sahara.service.ops [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] 2016-09-01 11:20:14.774 17081 DEBUG sahara.utils.api [-] Rest.route.decorator.handler, kwargs={'tenant_id': u'0d81fa86513d4e3d9ed88cdc8c0b7d73', 'cluster_id': u'e10e2581-e9e4-45c7-a267-70359f19b9db'} handler /usr/lib/python2.7/site-packages/sahara/utils/api.py:71 2016-09-01 11:20:14.829 17080 INFO sahara.utils.cluster [req-41d4dbfd-9fec-468c-8bfa-60b225c16fea 88e701edf4a0407593cc03cb74d4df57 0d81fa86513d4e3d9ed88cdc8c0b7d73 - - -] [instance: none, cluster: e10e2581-e9e4-45c7-a267-70359f19b9db] Cluster status has been changed. New status=Error Any attempt to delete the cluster results in a similar error and leaves the cluster in a "Deleting" state forever. I assume what it's trying to do is create a trust between the requesting user and the Sahara service user (trustor and trustee respectively). I have tried doing this manually with the Openstack client and it was created just fine. Keystone logs don't shed much light on this either. It looks like a normal auth request followed by the old "requires authentication" error. I am using Keystone V3 for everything, and have set "use_identity_api_v3 = True" and "auth_version = 3" as well as all the usual domain and project settings under keystone_authtoken. I'm a bit at a loss where to look next. Thanks in advance for any help you all can provide. Cheers, Erik _______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : [email protected] Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
