Re: [Openstack] Cinder problems with usage and caching ?
Hey Ollie, thx for your reply. I would but i don't have more informations as in my post before. My DB looks clean: mysql select SUM(size) from volumes where deleted=0\G *** 1. row *** SUM(size): 88 1 row in set (0.00 sec) mysql select SUM(volume_size) from snapshots where deleted=0\G *** 1. row *** SUM(volume_size): 53 1 row in set (0.00 sec) That's the entries of my mysql db but it seems that cinder will sumarize all new created volumes to the usage (that's correct) even they will delete after. I thought first it's a problem with the caching so i've restarted all memcached services but the problem still the same. I don't see anything in the cinder logs instead of the api entries at my post before. It's weired :( but a big problem Thx and Greetings Heiko On 13.06.2013 13:30, Ollie Leahy wrote: Thanks for taking the trouble to do that Heiko, as you can see that's been open a while and we're having trouble reproducing the problem, any information you can supply about your situation would be welcome. eg, errors in log files or the contents of your database as Duncan suggested in that bug. Ollie On Thu 13 Jun 2013 09:55:53 IST, Heiko Krämer wrote: Hey Ollie, yeah thx, I've found yesterday an existing bug report. https://bugs.launchpad.net/cinder/+bug/1174193 Thx and greetings Heiko On 12.06.2013 17:05, Ollie Leahy wrote: This looks like a bug, so you could raise a bug on cinder at https://bugs.launchpad.net/cinder/+filebug When you do you could include information about the version of cinder you are using, is it grizzy, folsom or are you testing on head? Also, if you can include any context information for example had that project id had used more quota in the past and deleted it? It would also be useful to search through any cinder logs for other error warnings, in case there was a failure in the past, when quota was either consumed or recovered by this project and where the operation was not completed successfully. Ollie On 12/06/13 10:02, Heiko Krämer wrote: Hi guys, I'm running in a problem raised by cinder api. I'll show you the log output it's more then my explaination :) 2013-06-12 10:50:13AUDIT [cinder.api.v1.volumes] Create volume of 30 GB 2013-06-12 10:50:13 WARNING [cinder.volume.api] Quota exceeded for d4e1c14691d841f6b53a24b6c4c42a0e, tried to create 30G volume (172G of 200G already consumed) 2013-06-12 10:50:13ERROR [cinder.api.middleware.fault] Caught error: Requested volume or snapshot exceeds allowed Gigabytes quota root@api2:~# cinder list +--++-+--+-+--+--+ | ID | Status | Display Name| Size | Volume Type | Bootable | Attached to | +--++-+--+-+--+--+ | 6ce6f626-2d2b-4a17-8933-13e196fa650c | in-use | bosh| 10 | default | false | 567a4c86-08ab-43cd-b9bc-3b220f2bf262 | | 8822b84b-595e-4b6f-9636-472dae7c33a4 | in-use | volume-64e51c64-5da4-4981-9b05-f7abfc6695b1 | 16 | None| false | 65f33296-c2b0-4824-b887-359ee0462b56 | | d56e5a86-f6d1-43ed-b125-2ff977aefa24 | in-use | volume-363573c1-05d6-4484-9aad-0919e47546e0 | 5 | None| false | fbb809d5-71f3-4a78-9cb7-4913c1e0af83 | | f7506174-4ae4-4a3c-928f-47b785bb35f5 | in-use | volume-385997c8-709c-4fa2-9d5b-ca2bba9d4e87 | 7 | None| false | 0f1ab672-043a-4361-afd5-9f2ddd818ed8 | +--++-+--+-+--+--+ root@api2:~# cinder quota-show d4e1c14691d841f6b53a24b6c4c42a0e +---+---+ | Property | Value | +---+---+ | gigabytes | 200 | | snapshots | 20 | | volumes | 30 | +---+---+ you see I consume only 38GB of 200GB and not 172GB (log). It's anything wrong with caching by cinder ? Have anyone the same problem or any hints ? Greetings Heiko ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Cinder problems with usage and caching ?
Hey Ollie, yeah thx, I've found yesterday an existing bug report. https://bugs.launchpad.net/cinder/+bug/1174193 Thx and greetings Heiko On 12.06.2013 17:05, Ollie Leahy wrote: This looks like a bug, so you could raise a bug on cinder at https://bugs.launchpad.net/cinder/+filebug When you do you could include information about the version of cinder you are using, is it grizzy, folsom or are you testing on head? Also, if you can include any context information for example had that project id had used more quota in the past and deleted it? It would also be useful to search through any cinder logs for other error warnings, in case there was a failure in the past, when quota was either consumed or recovered by this project and where the operation was not completed successfully. Ollie On 12/06/13 10:02, Heiko Krämer wrote: Hi guys, I'm running in a problem raised by cinder api. I'll show you the log output it's more then my explaination :) 2013-06-12 10:50:13AUDIT [cinder.api.v1.volumes] Create volume of 30 GB 2013-06-12 10:50:13 WARNING [cinder.volume.api] Quota exceeded for d4e1c14691d841f6b53a24b6c4c42a0e, tried to create 30G volume (172G of 200G already consumed) 2013-06-12 10:50:13ERROR [cinder.api.middleware.fault] Caught error: Requested volume or snapshot exceeds allowed Gigabytes quota root@api2:~# cinder list +--++-+--+-+--+--+ | ID | Status | Display Name| Size | Volume Type | Bootable | Attached to | +--++-+--+-+--+--+ | 6ce6f626-2d2b-4a17-8933-13e196fa650c | in-use | bosh| 10 | default | false | 567a4c86-08ab-43cd-b9bc-3b220f2bf262 | | 8822b84b-595e-4b6f-9636-472dae7c33a4 | in-use | volume-64e51c64-5da4-4981-9b05-f7abfc6695b1 | 16 | None| false | 65f33296-c2b0-4824-b887-359ee0462b56 | | d56e5a86-f6d1-43ed-b125-2ff977aefa24 | in-use | volume-363573c1-05d6-4484-9aad-0919e47546e0 | 5 | None| false | fbb809d5-71f3-4a78-9cb7-4913c1e0af83 | | f7506174-4ae4-4a3c-928f-47b785bb35f5 | in-use | volume-385997c8-709c-4fa2-9d5b-ca2bba9d4e87 | 7 | None| false | 0f1ab672-043a-4361-afd5-9f2ddd818ed8 | +--++-+--+-+--+--+ root@api2:~# cinder quota-show d4e1c14691d841f6b53a24b6c4c42a0e +---+---+ | Property | Value | +---+---+ | gigabytes | 200 | | snapshots | 20 | | volumes | 30 | +---+---+ you see I consume only 38GB of 200GB and not 172GB (log). It's anything wrong with caching by cinder ? Have anyone the same problem or any hints ? Greetings Heiko ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Cinder problems with usage and caching ?
Thanks for taking the trouble to do that Heiko, as you can see that's been open a while and we're having trouble reproducing the problem, any information you can supply about your situation would be welcome. eg, errors in log files or the contents of your database as Duncan suggested in that bug. Ollie On Thu 13 Jun 2013 09:55:53 IST, Heiko Krämer wrote: Hey Ollie, yeah thx, I've found yesterday an existing bug report. https://bugs.launchpad.net/cinder/+bug/1174193 Thx and greetings Heiko On 12.06.2013 17:05, Ollie Leahy wrote: This looks like a bug, so you could raise a bug on cinder at https://bugs.launchpad.net/cinder/+filebug When you do you could include information about the version of cinder you are using, is it grizzy, folsom or are you testing on head? Also, if you can include any context information for example had that project id had used more quota in the past and deleted it? It would also be useful to search through any cinder logs for other error warnings, in case there was a failure in the past, when quota was either consumed or recovered by this project and where the operation was not completed successfully. Ollie On 12/06/13 10:02, Heiko Krämer wrote: Hi guys, I'm running in a problem raised by cinder api. I'll show you the log output it's more then my explaination :) 2013-06-12 10:50:13AUDIT [cinder.api.v1.volumes] Create volume of 30 GB 2013-06-12 10:50:13 WARNING [cinder.volume.api] Quota exceeded for d4e1c14691d841f6b53a24b6c4c42a0e, tried to create 30G volume (172G of 200G already consumed) 2013-06-12 10:50:13ERROR [cinder.api.middleware.fault] Caught error: Requested volume or snapshot exceeds allowed Gigabytes quota root@api2:~# cinder list +--++-+--+-+--+--+ | ID | Status | Display Name| Size | Volume Type | Bootable | Attached to | +--++-+--+-+--+--+ | 6ce6f626-2d2b-4a17-8933-13e196fa650c | in-use | bosh| 10 | default | false | 567a4c86-08ab-43cd-b9bc-3b220f2bf262 | | 8822b84b-595e-4b6f-9636-472dae7c33a4 | in-use | volume-64e51c64-5da4-4981-9b05-f7abfc6695b1 | 16 | None| false | 65f33296-c2b0-4824-b887-359ee0462b56 | | d56e5a86-f6d1-43ed-b125-2ff977aefa24 | in-use | volume-363573c1-05d6-4484-9aad-0919e47546e0 | 5 | None| false | fbb809d5-71f3-4a78-9cb7-4913c1e0af83 | | f7506174-4ae4-4a3c-928f-47b785bb35f5 | in-use | volume-385997c8-709c-4fa2-9d5b-ca2bba9d4e87 | 7 | None| false | 0f1ab672-043a-4361-afd5-9f2ddd818ed8 | +--++-+--+-+--+--+ root@api2:~# cinder quota-show d4e1c14691d841f6b53a24b6c4c42a0e +---+---+ | Property | Value | +---+---+ | gigabytes | 200 | | snapshots | 20 | | volumes | 30 | +---+---+ you see I consume only 38GB of 200GB and not 172GB (log). It's anything wrong with caching by cinder ? Have anyone the same problem or any hints ? Greetings Heiko ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Cinder problems with usage and caching ?
This looks like a bug, so you could raise a bug on cinder at https://bugs.launchpad.net/cinder/+filebug When you do you could include information about the version of cinder you are using, is it grizzy, folsom or are you testing on head? Also, if you can include any context information for example had that project id had used more quota in the past and deleted it? It would also be useful to search through any cinder logs for other error warnings, in case there was a failure in the past, when quota was either consumed or recovered by this project and where the operation was not completed successfully. Ollie On 12/06/13 10:02, Heiko Krämer wrote: Hi guys, I'm running in a problem raised by cinder api. I'll show you the log output it's more then my explaination :) 2013-06-12 10:50:13AUDIT [cinder.api.v1.volumes] Create volume of 30 GB 2013-06-12 10:50:13 WARNING [cinder.volume.api] Quota exceeded for d4e1c14691d841f6b53a24b6c4c42a0e, tried to create 30G volume (172G of 200G already consumed) 2013-06-12 10:50:13ERROR [cinder.api.middleware.fault] Caught error: Requested volume or snapshot exceeds allowed Gigabytes quota root@api2:~# cinder list +--++-+--+-+--+--+ | ID | Status | Display Name| Size | Volume Type | Bootable | Attached to | +--++-+--+-+--+--+ | 6ce6f626-2d2b-4a17-8933-13e196fa650c | in-use | bosh| 10 | default | false | 567a4c86-08ab-43cd-b9bc-3b220f2bf262 | | 8822b84b-595e-4b6f-9636-472dae7c33a4 | in-use | volume-64e51c64-5da4-4981-9b05-f7abfc6695b1 | 16 | None| false | 65f33296-c2b0-4824-b887-359ee0462b56 | | d56e5a86-f6d1-43ed-b125-2ff977aefa24 | in-use | volume-363573c1-05d6-4484-9aad-0919e47546e0 | 5 | None| false | fbb809d5-71f3-4a78-9cb7-4913c1e0af83 | | f7506174-4ae4-4a3c-928f-47b785bb35f5 | in-use | volume-385997c8-709c-4fa2-9d5b-ca2bba9d4e87 | 7 | None| false | 0f1ab672-043a-4361-afd5-9f2ddd818ed8 | +--++-+--+-+--+--+ root@api2:~# cinder quota-show d4e1c14691d841f6b53a24b6c4c42a0e +---+---+ | Property | Value | +---+---+ | gigabytes | 200 | | snapshots | 20 | | volumes | 30 | +---+---+ you see I consume only 38GB of 200GB and not 172GB (log). It's anything wrong with caching by cinder ? Have anyone the same problem or any hints ? Greetings Heiko ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
