Found it! The user_creds.password column in the database is 255 characters. My
value is long enough (800+) that it fills it up and gets truncated. I think
that garbling of the data leads to the padding exception.
If I modify the database so that the column is longer, then the delete goes
through just fine.
Ken
From: Ken Thomas <[email protected]>
To: Zane Bitter <[email protected]>; "[email protected]"
<[email protected]>
Sent: Friday, October 24, 2014 2:43 PM
Subject: Re: [openstack-dev] [Heat] Stack stuck in DELETE_IN_PROGRESS even
though all resources are DELETE_COMPLETE
I've got some additional info for y'all on this. The problem is in
heat.engine.parser.Stack.delete with this statement:
user_creds = db_api.user_creds_get(self.user_creds_id)
In my case, that call is throwing an exception that's not getting caught or
logged. The exact exception is:
ERROR Incorrect paddingTraceback (most recent call last): File
"/usr/lib/python2.6/site-packages/heat/engine/parser.py", line 670, in delete
user_creds = db_api.user_creds_get(self.user_creds_id) File
"/usr/lib/python2.6/site-packages/heat/db/api.py", line 170, in user_creds_get
return IMPL.user_creds_get(context_id) File
"/usr/lib/python2.6/site-packages/heat/db/sqlalchemy/api.py", line 468, in
user_creds_get result['password'] = _decrypt(result['password'],
db_result.decrypt_method) File
"/usr/lib/python2.6/site-packages/heat/db/sqlalchemy/api.py", line 174, in
_decrypt value = decryptor(enc_value) File
"/usr/lib/python2.6/site-packages/heat/common/crypt.py", line 48, in
oslo_decrypt_v1 auth_info, b64decode=True) File
"/usr/lib/python2.6/site-packages/heat/openstack/common/crypto/utils.py", line
158, in decrypt msg = base64.b64decode(msg) File
"/usr/lib64/python2.6/base64.py", line 76, in b64decode raise
TypeError(msg)TypeError: Incorrect padding
Does this ring any bells? I checked the user_creds table and it looks okay as
far as I can tell. Do I have a mismatch between how the value is encoded vs how
it's being decoded?
Ken
From: Zane Bitter <[email protected]>
To: Ken Thomas <[email protected]>; "[email protected]"
<[email protected]>
Sent: Wednesday, October 22, 2014 6:10 AM
Subject: Re: [openstack-dev] [Heat] Stack stuck in DELETE_IN_PROGRESS even
though all resources are DELETE_COMPLETE
Moving this to the openstack@ list...
On 21/10/14 18:56, Ken Thomas wrote:
> Greetings all,
> I'm using Heat from Icehouse and I'm hitting a problem that I'm hoping y'all
> can shed some light on.
> I have no problems doing stack-create. I can watch the MySQL commands go by
> and see it happily update the stack table so that it eventually shows up as
> CREATE_COMPLETE. When I delete the stack, everything seems to be working
> fine. I see the MySQL update that sets the stack to DELETE IN PROGRESS as
> well as the updates that sets my single resource to DELETE_COMPLETE... but I
> never see the final update to the stack table to set it to DELETE COMPLETE.
I don't think you're the first person to report this, but afaik we've
never been able to track it down. There are a bunch of things that Heat
does after deleting the resources, like cleaning up trusts in Keystone,
and perhaps one of those is failing. However, *in theory* there's no way
for the stack to be left in the "DELETE_IN_PROGRESS" state without an
exception being raised that would leave a prominent entry in the logs.
> One very odd thing that I found in the MySQL logs (snippet below), is a query
> that includes that stack name but with an extra '*' character append to it.
> My stack is named 'xyzzy8' but notice the 'xzyzzy8*' in the constraint.
> That's not going to return anything and I'm wondering if that is what's
> preventing the final stack DELETE_COMPLTE update from happening?
The stack name that ends with '*' is the "backup stack", which is used
to hold out-of-date resources during a stack update. When the stack is
deleted, Heat will check for the existence of the backup stack and make
sure any resources left behind in there (from a failed update) are
deleted too. If it doesn't exist, that shouldn't be a problem.
> There are no errors in the any of the heat logs.
>
> Any guidance would be greatly appreciated!
All I can suggest is to (a) make sure you have the very latest version
of Icehouse and (b) make sure your logging is turned up to 11, then try
again.
Since you seem very proficient at debugging, if that doesn't work you
could also try adding extra logging into the method
heat.engine.parser.Stack.delete to try to figure out how far it is getting.
Feel free to raise a bug in Launchpad and attach any logs that might
help us figure out what is happening.
cheers,
Zane.
>
> Ken
> SELECT stack.status_reason AS stack_status_reason, stack.created_at AS
> stack_created_at, stack.deleted_at AS stack_deleted_at, stack.action AS
> stack_action, stack.status AS stack_status, stack.id AS stack_id, stack.name
> AS stack_name, stack.raw_template_id AS stack_raw_template_id, stack.username
> AS stack_username, stack.tenant AS stack_tenant, stack.parameters AS
> stack_parameters, stack.user_creds_id AS stack_user_creds_id, stack.owner_id
> AS stack_owner_id, stack.timeout AS stack_timeout, stack.disable_rollback AS
> stack_disable_rollback, stack.stack_user_project_id AS
> stack_stack_user_project_id, stack.updated_at AS stack_updated_atFROM
> stackWHERE stack.deleted_at IS NULL AND (stack.tenant =
> 'c6c488223aae4e97bf56dda8cef36b3b' OR stack.stack_user_project_id =
> 'c6c488223aae4e97bf56dda8cef36b3b') AND stack.name = 'xyzzy8*' AND
> stack.owner_id = '9a3e56d7-0c10-4c1c-8c54-0e5580cee121'
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : [email protected]
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
