Re: [openstack-dev] [Openstack] Glance on swift problem
Hello,
I make some more debugging on that problem and I found what is happend:
Glance is sending first chunk (in my config it was 200MB) and after
finishing sending it to swift it send some bad http request. Swift has
got in logs something like:
proxy-server ERROR WSGI: code 400, message Bad request syntax
('\x01\x00\x1c[n\x01\x00\x00\x96s\x01\x00\x00*t') (txn:
txa21d64d49ac347bb87023-00542d4e59) (client_ip: X.X.X.X)
and uploading is finished with error in glance
Maybe someone of You know such bug?
Best regards
Slawek Kaplonski
W dniu 2014-09-30 16:20, Sławek Kapłoński napisał(a):
Hello,
I'm using openstack havana release and glance with swift backend.
Today I found that I have problem when I create image with url in
"--copy-from" when image is bigger than my
"swift_store_large_object_size" because then glance is trying to split
image to chunks with size given in
"swift_store_large_object_chunk_size" and when try to upload first
chunk to swift I have error:
2014-09-30 15:05:29.361 18023 ERROR glance.store.swift [-] Error
during chunked upload to backend, deleting stale chunks
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift Traceback (most
recent call last):
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File
"/usr/lib/python2.7/dist-packages/glance/store/swift.py", line 384, in
add
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
content_length=content_length)
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File
"/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1234,
in put_object
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
response_dict=response_dict)
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File
"/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1143,
in _retry
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
reset_func(func, *args, **kwargs)
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File
"/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1215,
in _default_reset
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift %
(container, obj))
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
ClientException: put_object('glance',
'9f56ccec-deeb-4020-95ba-ca7bf1170056-1', ...) failure and no
ability to reset contents for reupload.
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
2014-09-30 15:05:29.362 18023 ERROR glance.store.swift [-] Failed to
add object to Swift.
Got error from Swift: put_object('glance',
'9f56ccec-deeb-4020-95ba-ca7bf1170056-1', ...) failure and no
ability to reset contents for reupload.
2014-09-30 15:05:29.362 18023 ERROR glance.api.v1.upload_utils [-]
Failed to upload image 9f56ccec-deeb-4020-95ba-ca7bf1170056
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
Traceback (most recent call last):
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.7/dist-packages/glance/api/v1/upload_utils.py", line
101, in upload_data_to_store
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
store)
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.7/dist-packages/glance/store/__init__.py", line 333,
in store_add_to_backend
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
(location, size, checksum, metadata) = store.add(image_id, data, size)
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.7/dist-packages/glance/store/swift.py", line 447, in
add
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
raise glance.store.BackendException(msg)
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
BackendException: Failed to add object to Swift.
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils Got
error from Swift: put_object('glance',
'9f56ccec-deeb-4020-95ba-ca7bf1170056-1', ...) failure and no
ability to reset contents for reupload.
Does someone of You got same error and know what is solution of it? I
was searching about that in google but I not found anything what could
solve my problem.
--
Pozdrawiam
Sławek Kapłoński
sla...@kaplonski.pl
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack] Glance on swift problem
Slawek,
did you change Swift proxy-server.conf file too?
On Wed, Oct 1, 2014 at 11:26 PM, Sławek Kapłoński
wrote:
> Hello,
>
> Thanks for Your help but it not helps. I checked that for sure on each
> swift
> node there is a lot of free space. What can confirm that is fact that when
> I
> try to create image with size about 1.7GB and I have
> swift_store_large_object_size set to 1GB than there is error (always after
> send first chunk to swift (200MB). When I only change
> swift_store_large_object_size to 2GB and restart glance-api than the same
> image is created correctly (it is then in one big object).
>
> ---
> Best regards
> Sławek Kapłoński
> sla...@kaplonski.pl
>
> Dnia wtorek, 30 września 2014 22:28:11 Timur Nurlygayanov pisze:
> > Hi Slawek,
> >
> > we faced the same error and this is issue with Swift.
> > We can see 100% disk usage on the Swift node during the file upload and
> > looks like Swift can't send info about status of the file loading in
> time.
> >
> > On our environments we found the workaround for this issue:
> > 1. Set swift_store_large_object_size = 200 in glance.conf.
> > 2. Add to Swift proxy-server.conf:
> >
> > [DEFAULT]
> > ...
> > node_timeout = 90
> >
> > Probably we can set this value as default value for this parameter
> instead
> > of '30'?
> >
> >
> > Regards,
> > Timur
> >
> >
> > On Tue, Sep 30, 2014 at 7:41 PM, Sławek Kapłoński
> >
> > wrote:
> > > Hello,
> > >
> > > I can't find that upload from was previous logs but I now try to upload
> > > same image once again. In glance there was exactly same error. In swift
> > > logs I have:
> > >
> > > Sep 30 17:35:10 127.0.0.1 proxy-server X.X.X.X Y.Y.Y.Y
> > > 30/Sep/2014/15/35/10 HEAD
> /v1/AUTH_7ef5a7661ccd4c069e3ad387a6dceebd/glance
> > > HTTP/1.0 204
> > > Sep 30 17:35:16 127.0.0.1 proxy-server X.X.X.X Y.Y.Y.Y
> > > 30/Sep/2014/15/35/16 PUT /v1/AUTH_7ef5a7661ccd4c069e3ad387a6dcee
> > > bd/glance/fa5dfe09-74f5-4287-9852-d2f1991eebc0-1 HTTP/1.0 201 - -
> > >
> > > Best regards
> > > Slawek Kaplonski
> > >
> > > W dniu 2014-09-30 17:03, Kuo Hugo napisał(a):
> > >> Hi ,
> > >>
> > >> Could you please post the log of related requests in Swift's log ???
> > >>
> > >> Thanks // Hugo
> > >>
> > >> 2014-09-30 22:20 GMT+08:00 Sławek Kapłoński :
> > >> Hello,
> > >>
> > >>> I'm using openstack havana release and glance with swift backend.
> > >>> Today I found that I have problem when I create image with url in
> > >>> "--copy-from" when image is bigger than my
> > >>> "swift_store_large_object_size" because then glance is trying to
> > >>> split image to chunks with size given in
> > >>> "swift_store_large_object_chunk_size" and when try to upload first
> > >>> chunk to swift I have error:
> > >>>
> > >>> 2014-09-30 15:05:29.361 18023 ERROR glance.store.swift [-] Error
> > >>> during chunked upload to backend, deleting stale chunks
> > >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift Traceback
> > >>> (most recent call last):
> > >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File
> > >>> "/usr/lib/python2.7/dist-packages/glance/store/swift.py", line 384,
> > >>> in add
> > >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
> > >>>
> > >>> content_length=content_length)
> > >>>
> > >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File
> > >>> "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1234,
> > >>> in put_object
> > >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
> > >>>
> > >>> response_dict=response_dict)
> > >>>
> > >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File
> > >>> "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1143,
> > >>> in _retry
> > >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
> > >>>
> > >>> reset_func(func, *args, **kwargs)
> > >>>
> > >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File
> > >>> "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1215,
> > >>> in _default_reset
> > >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift %
> > >>> (container, obj))
> > >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
> > >>> ClientException: put_object('glance',
> > >>> '9f56ccec-deeb-4020-95ba-ca7bf1170056-1', ...) failure and no
> > >>> ability to reset contents for reupload.
> > >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
> > >>> 2014-09-30 15:05:29.362 18023 ERROR glance.store.swift [-] Failed
> > >>> to add object to Swift.
> > >>> Got error from Swift: put_object('glance',
> > >>> '9f56ccec-deeb-4020-95ba-ca7bf1170056-1', ...) failure and no
> > >>> ability to reset contents for reupload.
> > >>> 2014-09-30 15:05:29.362 18023 ERROR glance.api.v1.upload_utils [-]
> > >>> Failed to upload image 9f56ccec-deeb-4020-95ba-ca7bf1170056
> > >>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
> > >>> Traceback (most recent call last):
> > >>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
>
Re: [openstack-dev] [Openstack] Glance on swift problem
Hello,
Thanks for Your help but it not helps. I checked that for sure on each swift
node there is a lot of free space. What can confirm that is fact that when I
try to create image with size about 1.7GB and I have
swift_store_large_object_size set to 1GB than there is error (always after
send first chunk to swift (200MB). When I only change
swift_store_large_object_size to 2GB and restart glance-api than the same
image is created correctly (it is then in one big object).
---
Best regards
Sławek Kapłoński
sla...@kaplonski.pl
Dnia wtorek, 30 września 2014 22:28:11 Timur Nurlygayanov pisze:
> Hi Slawek,
>
> we faced the same error and this is issue with Swift.
> We can see 100% disk usage on the Swift node during the file upload and
> looks like Swift can't send info about status of the file loading in time.
>
> On our environments we found the workaround for this issue:
> 1. Set swift_store_large_object_size = 200 in glance.conf.
> 2. Add to Swift proxy-server.conf:
>
> [DEFAULT]
> ...
> node_timeout = 90
>
> Probably we can set this value as default value for this parameter instead
> of '30'?
>
>
> Regards,
> Timur
>
>
> On Tue, Sep 30, 2014 at 7:41 PM, Sławek Kapłoński
>
> wrote:
> > Hello,
> >
> > I can't find that upload from was previous logs but I now try to upload
> > same image once again. In glance there was exactly same error. In swift
> > logs I have:
> >
> > Sep 30 17:35:10 127.0.0.1 proxy-server X.X.X.X Y.Y.Y.Y
> > 30/Sep/2014/15/35/10 HEAD /v1/AUTH_7ef5a7661ccd4c069e3ad387a6dceebd/glance
> > HTTP/1.0 204
> > Sep 30 17:35:16 127.0.0.1 proxy-server X.X.X.X Y.Y.Y.Y
> > 30/Sep/2014/15/35/16 PUT /v1/AUTH_7ef5a7661ccd4c069e3ad387a6dcee
> > bd/glance/fa5dfe09-74f5-4287-9852-d2f1991eebc0-1 HTTP/1.0 201 - -
> >
> > Best regards
> > Slawek Kaplonski
> >
> > W dniu 2014-09-30 17:03, Kuo Hugo napisał(a):
> >> Hi ,
> >>
> >> Could you please post the log of related requests in Swift's log ???
> >>
> >> Thanks // Hugo
> >>
> >> 2014-09-30 22:20 GMT+08:00 Sławek Kapłoński :
> >> Hello,
> >>
> >>> I'm using openstack havana release and glance with swift backend.
> >>> Today I found that I have problem when I create image with url in
> >>> "--copy-from" when image is bigger than my
> >>> "swift_store_large_object_size" because then glance is trying to
> >>> split image to chunks with size given in
> >>> "swift_store_large_object_chunk_size" and when try to upload first
> >>> chunk to swift I have error:
> >>>
> >>> 2014-09-30 15:05:29.361 18023 ERROR glance.store.swift [-] Error
> >>> during chunked upload to backend, deleting stale chunks
> >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift Traceback
> >>> (most recent call last):
> >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File
> >>> "/usr/lib/python2.7/dist-packages/glance/store/swift.py", line 384,
> >>> in add
> >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
> >>>
> >>> content_length=content_length)
> >>>
> >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File
> >>> "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1234,
> >>> in put_object
> >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
> >>>
> >>> response_dict=response_dict)
> >>>
> >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File
> >>> "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1143,
> >>> in _retry
> >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
> >>>
> >>> reset_func(func, *args, **kwargs)
> >>>
> >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File
> >>> "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1215,
> >>> in _default_reset
> >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift %
> >>> (container, obj))
> >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
> >>> ClientException: put_object('glance',
> >>> '9f56ccec-deeb-4020-95ba-ca7bf1170056-1', ...) failure and no
> >>> ability to reset contents for reupload.
> >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
> >>> 2014-09-30 15:05:29.362 18023 ERROR glance.store.swift [-] Failed
> >>> to add object to Swift.
> >>> Got error from Swift: put_object('glance',
> >>> '9f56ccec-deeb-4020-95ba-ca7bf1170056-1', ...) failure and no
> >>> ability to reset contents for reupload.
> >>> 2014-09-30 15:05:29.362 18023 ERROR glance.api.v1.upload_utils [-]
> >>> Failed to upload image 9f56ccec-deeb-4020-95ba-ca7bf1170056
> >>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
> >>> Traceback (most recent call last):
> >>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
> >>>
> >>> File
> >>>
> >>> "/usr/lib/python2.7/dist-packages/glance/api/v1/upload_utils.py",
> >>> line 101, in upload_data_to_store
> >>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
> >>>
> >>> store)
> >>>
> >>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
> >>>
> >>> File "/usr/lib/py
Re: [openstack-dev] [Openstack] Glance on swift problem
Hello,
W dniu 2014-09-30 20:28, Timur Nurlygayanov napisał(a):
Hi Slawek,
we faced the same error and this is issue with Swift.
We can see 100% disk usage on the Swift node during the file upload
and looks like Swift can't send info about status of the file loading
in time.
On our environments we found the workaround for this issue:
1. Set swift_store_large_object_size = 200 in glance.conf.
I have it set to 1024 - should it be lower? Value which I have is too
big now?
2. Add to Swift proxy-server.conf:
[DEFAULT]
...
node_timeout = 90
Probably we can set this value as default value for this parameter
instead of '30'?
Regards,
Timur
On Tue, Sep 30, 2014 at 7:41 PM, Sławek Kapłoński
wrote:
Hello,
I can't find that upload from was previous logs but I now try to
upload same image once again. In glance there was exactly same
error. In swift logs I have:
Sep 30 17:35:10 127.0.0.1 proxy-server X.X.X.X Y.Y.Y.Y
30/Sep/2014/15/35/10 HEAD
/v1/AUTH_7ef5a7661ccd4c069e3ad387a6dceebd/glance HTTP/1.0 204
Sep 30 17:35:16 127.0.0.1 proxy-server X.X.X.X Y.Y.Y.Y
30/Sep/2014/15/35/16 PUT
/v1/AUTH_7ef5a7661ccd4c069e3ad387a6dceebd/glance/fa5dfe09-74f5-4287-9852-d2f1991eebc0-1
HTTP/1.0 201 - -
Best regards
Slawek Kaplonski
W dniu 2014-09-30 17:03, Kuo Hugo napisał(a):
Hi ,
Could you please post the log of related requests in Swift's log
???
Thanks // Hugo
2014-09-30 22:20 GMT+08:00 Sławek Kapłoński
:
Hello,
I'm using openstack havana release and glance with swift backend.
Today I found that I have problem when I create image with url in
"--copy-from" when image is bigger than my
"swift_store_large_object_size" because then glance is trying to
split image to chunks with size given in
"swift_store_large_object_chunk_size" and when try to upload first
chunk to swift I have error:
2014-09-30 15:05:29.361 18023 ERROR glance.store.swift [-] Error
during chunked upload to backend, deleting stale chunks
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift Traceback
(most recent call last):
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File
"/usr/lib/python2.7/dist-packages/glance/store/swift.py", line 384,
in add
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
content_length=content_length)
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File
"/usr/lib/python2.7/dist-packages/swiftclient/client.py", line
1234,
in put_object
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
response_dict=response_dict)
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File
"/usr/lib/python2.7/dist-packages/swiftclient/client.py", line
1143,
in _retry
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
reset_func(func, *args, **kwargs)
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File
"/usr/lib/python2.7/dist-packages/swiftclient/client.py", line
1215,
in _default_reset
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift %
(container, obj))
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
ClientException: put_object('glance',
'9f56ccec-deeb-4020-95ba-ca7bf1170056-1', ...) failure and no
ability to reset contents for reupload.
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
2014-09-30 15:05:29.362 18023 ERROR glance.store.swift [-] Failed
to add object to Swift.
Got error from Swift: put_object('glance',
'9f56ccec-deeb-4020-95ba-ca7bf1170056-1', ...) failure and no
ability to reset contents for reupload.
2014-09-30 15:05:29.362 18023 ERROR glance.api.v1.upload_utils [-]
Failed to upload image 9f56ccec-deeb-4020-95ba-ca7bf1170056
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
Traceback (most recent call last):
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
File
"/usr/lib/python2.7/dist-packages/glance/api/v1/upload_utils.py",
line 101, in upload_data_to_store
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
store)
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
File "/usr/lib/python2.7/dist-packages/glance/store/__init__.py",
line 333, in store_add_to_backend
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
(location, size, checksum, metadata) = store.add(image_id, data,
size)
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
File "/usr/lib/python2.7/dist-packages/glance/store/swift.py",
line 447, in add
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
raise glance.store.BackendException(msg)
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
BackendException: Failed to add object to Swift.
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils Got
error from Swift: put_object('glance',
'9f56ccec-deeb-4020-95ba-ca7bf1170056-1', ...) failure and no
ability to reset contents for reupload.
Does someone of You got same error and know what is solution of it?
I was searching about that in google but I not found anything what
could solve my problem.
--
Best regards
Sławek Kapłoński
sla..
Re: [openstack-dev] [Openstack] Glance on swift problem
Hi Slawek,
we faced the same error and this is issue with Swift.
We can see 100% disk usage on the Swift node during the file upload and
looks like Swift can't send info about status of the file loading in time.
On our environments we found the workaround for this issue:
1. Set swift_store_large_object_size = 200 in glance.conf.
2. Add to Swift proxy-server.conf:
[DEFAULT]
...
node_timeout = 90
Probably we can set this value as default value for this parameter instead
of '30'?
Regards,
Timur
On Tue, Sep 30, 2014 at 7:41 PM, Sławek Kapłoński
wrote:
> Hello,
>
> I can't find that upload from was previous logs but I now try to upload
> same image once again. In glance there was exactly same error. In swift
> logs I have:
>
> Sep 30 17:35:10 127.0.0.1 proxy-server X.X.X.X Y.Y.Y.Y
> 30/Sep/2014/15/35/10 HEAD /v1/AUTH_7ef5a7661ccd4c069e3ad387a6dceebd/glance
> HTTP/1.0 204
> Sep 30 17:35:16 127.0.0.1 proxy-server X.X.X.X Y.Y.Y.Y
> 30/Sep/2014/15/35/16 PUT /v1/AUTH_7ef5a7661ccd4c069e3ad387a6dcee
> bd/glance/fa5dfe09-74f5-4287-9852-d2f1991eebc0-1 HTTP/1.0 201 - -
>
> Best regards
> Slawek Kaplonski
>
> W dniu 2014-09-30 17:03, Kuo Hugo napisał(a):
>
>> Hi ,
>>
>> Could you please post the log of related requests in Swift's log ???
>>
>> Thanks // Hugo
>>
>> 2014-09-30 22:20 GMT+08:00 Sławek Kapłoński :
>>
>> Hello,
>>>
>>> I'm using openstack havana release and glance with swift backend.
>>> Today I found that I have problem when I create image with url in
>>> "--copy-from" when image is bigger than my
>>> "swift_store_large_object_size" because then glance is trying to
>>> split image to chunks with size given in
>>> "swift_store_large_object_chunk_size" and when try to upload first
>>> chunk to swift I have error:
>>>
>>> 2014-09-30 15:05:29.361 18023 ERROR glance.store.swift [-] Error
>>> during chunked upload to backend, deleting stale chunks
>>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift Traceback
>>> (most recent call last):
>>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File
>>> "/usr/lib/python2.7/dist-packages/glance/store/swift.py", line 384,
>>> in add
>>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
>>> content_length=content_length)
>>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File
>>> "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1234,
>>> in put_object
>>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
>>> response_dict=response_dict)
>>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File
>>> "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1143,
>>> in _retry
>>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
>>> reset_func(func, *args, **kwargs)
>>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File
>>> "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1215,
>>> in _default_reset
>>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift %
>>> (container, obj))
>>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
>>> ClientException: put_object('glance',
>>> '9f56ccec-deeb-4020-95ba-ca7bf1170056-1', ...) failure and no
>>> ability to reset contents for reupload.
>>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
>>> 2014-09-30 15:05:29.362 18023 ERROR glance.store.swift [-] Failed
>>> to add object to Swift.
>>> Got error from Swift: put_object('glance',
>>> '9f56ccec-deeb-4020-95ba-ca7bf1170056-1', ...) failure and no
>>> ability to reset contents for reupload.
>>> 2014-09-30 15:05:29.362 18023 ERROR glance.api.v1.upload_utils [-]
>>> Failed to upload image 9f56ccec-deeb-4020-95ba-ca7bf1170056
>>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
>>> Traceback (most recent call last):
>>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
>>> File
>>> "/usr/lib/python2.7/dist-packages/glance/api/v1/upload_utils.py",
>>> line 101, in upload_data_to_store
>>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
>>> store)
>>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
>>> File "/usr/lib/python2.7/dist-packages/glance/store/__init__.py",
>>> line 333, in store_add_to_backend
>>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
>>> (location, size, checksum, metadata) = store.add(image_id, data,
>>> size)
>>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
>>> File "/usr/lib/python2.7/dist-packages/glance/store/swift.py",
>>> line 447, in add
>>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
>>> raise glance.store.BackendException(msg)
>>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
>>> BackendException: Failed to add object to Swift.
>>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils Got
>>> error from Swift: put_object('glance',
>>> '9f56ccec-deeb-4020-95ba-ca7bf1170056-1', ...) failure and no
>>> ability to reset contents for reupload.
>>>
>>> Does someone of You got same error a
Re: [openstack-dev] [Openstack] Glance on swift problem
Hi ,
Could you please post the log of related requests in Swift's log ???
Thanks // Hugo
2014-09-30 22:20 GMT+08:00 Sławek Kapłoński :
> Hello,
>
> I'm using openstack havana release and glance with swift backend. Today I
> found that I have problem when I create image with url in "--copy-from"
> when image is bigger than my "swift_store_large_object_size" because then
> glance is trying to split image to chunks with size given in
> "swift_store_large_object_chunk_size" and when try to upload first chunk
> to swift I have error:
>
> 2014-09-30 15:05:29.361 18023 ERROR glance.store.swift [-] Error during
> chunked upload to backend, deleting stale chunks
> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift Traceback (most
> recent call last):
> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File
> "/usr/lib/python2.7/dist-packages/glance/store/swift.py", line 384, in add
> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
> content_length=content_length)
> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File
> "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1234, in
> put_object
> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
> response_dict=response_dict)
> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File
> "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1143, in
> _retry
> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
> reset_func(func, *args, **kwargs)
> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File
> "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1215, in
> _default_reset
> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift % (container,
> obj))
> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift ClientException:
> put_object('glance', '9f56ccec-deeb-4020-95ba-ca7bf1170056-1', ...)
> failure and no ability to reset contents for reupload.
> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
> 2014-09-30 15:05:29.362 18023 ERROR glance.store.swift [-] Failed to add
> object to Swift.
> Got error from Swift: put_object('glance',
> '9f56ccec-deeb-4020-95ba-ca7bf1170056-1',
> ...) failure and no ability to reset contents for reupload.
> 2014-09-30 15:05:29.362 18023 ERROR glance.api.v1.upload_utils [-] Failed
> to upload image 9f56ccec-deeb-4020-95ba-ca7bf1170056
> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils Traceback
> (most recent call last):
> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils File
> "/usr/lib/python2.7/dist-packages/glance/api/v1/upload_utils.py", line
> 101, in upload_data_to_store
> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils store)
> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils File
> "/usr/lib/python2.7/dist-packages/glance/store/__init__.py", line 333, in
> store_add_to_backend
> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
> (location, size, checksum, metadata) = store.add(image_id, data, size)
> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils File
> "/usr/lib/python2.7/dist-packages/glance/store/swift.py", line 447, in add
> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils raise
> glance.store.BackendException(msg)
> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
> BackendException: Failed to add object to Swift.
> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils Got error
> from Swift: put_object('glance', '9f56ccec-deeb-4020-95ba-ca7bf1170056-1',
> ...) failure and no ability to reset contents for reupload.
>
>
> Does someone of You got same error and know what is solution of it? I was
> searching about that in google but I not found anything what could solve my
> problem.
>
> --
> Best regards
> Sławek Kapłoński
> sla...@kaplonski.pl
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> Post to : openst...@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Openstack] Glance on swift problem
Hello,
I'm using openstack havana release and glance with swift backend. Today
I found that I have problem when I create image with url in
"--copy-from" when image is bigger than my
"swift_store_large_object_size" because then glance is trying to split
image to chunks with size given in "swift_store_large_object_chunk_size"
and when try to upload first chunk to swift I have error:
2014-09-30 15:05:29.361 18023 ERROR glance.store.swift [-] Error during
chunked upload to backend, deleting stale chunks
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift Traceback (most
recent call last):
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File
"/usr/lib/python2.7/dist-packages/glance/store/swift.py", line 384, in
add
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
content_length=content_length)
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File
"/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1234, in
put_object
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
response_dict=response_dict)
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File
"/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1143, in
_retry
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
reset_func(func, *args, **kwargs)
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File
"/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1215, in
_default_reset
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift % (container,
obj))
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift ClientException:
put_object('glance', '9f56ccec-deeb-4020-95ba-ca7bf1170056-1', ...)
failure and no ability to reset contents for reupload.
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
2014-09-30 15:05:29.362 18023 ERROR glance.store.swift [-] Failed to add
object to Swift.
Got error from Swift: put_object('glance',
'9f56ccec-deeb-4020-95ba-ca7bf1170056-1', ...) failure and no
ability to reset contents for reupload.
2014-09-30 15:05:29.362 18023 ERROR glance.api.v1.upload_utils [-]
Failed to upload image 9f56ccec-deeb-4020-95ba-ca7bf1170056
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils Traceback
(most recent call last):
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.7/dist-packages/glance/api/v1/upload_utils.py", line
101, in upload_data_to_store
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
store)
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.7/dist-packages/glance/store/__init__.py", line 333,
in store_add_to_backend
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
(location, size, checksum, metadata) = store.add(image_id, data, size)
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils File
"/usr/lib/python2.7/dist-packages/glance/store/swift.py", line 447, in
add
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils raise
glance.store.BackendException(msg)
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
BackendException: Failed to add object to Swift.
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils Got error
from Swift: put_object('glance',
'9f56ccec-deeb-4020-95ba-ca7bf1170056-1', ...) failure and no
ability to reset contents for reupload.
Does someone of You got same error and know what is solution of it? I
was searching about that in google but I not found anything what could
solve my problem.
--
Best regards
Sławek Kapłoński
sla...@kaplonski.pl
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openst...@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Openstack] Glance on swift problem
Hello,
I can't find that upload from was previous logs but I now try to upload
same image once again. In glance there was exactly same error. In swift
logs I have:
Sep 30 17:35:10 127.0.0.1 proxy-server X.X.X.X Y.Y.Y.Y
30/Sep/2014/15/35/10 HEAD
/v1/AUTH_7ef5a7661ccd4c069e3ad387a6dceebd/glance HTTP/1.0 204
Sep 30 17:35:16 127.0.0.1 proxy-server X.X.X.X Y.Y.Y.Y
30/Sep/2014/15/35/16 PUT
/v1/AUTH_7ef5a7661ccd4c069e3ad387a6dceebd/glance/fa5dfe09-74f5-4287-9852-d2f1991eebc0-1 HTTP/1.0 201 - -
Best regards
Slawek Kaplonski
W dniu 2014-09-30 17:03, Kuo Hugo napisał(a):
Hi ,
Could you please post the log of related requests in Swift's log ???
Thanks // Hugo
2014-09-30 22:20 GMT+08:00 Sławek Kapłoński :
Hello,
I'm using openstack havana release and glance with swift backend.
Today I found that I have problem when I create image with url in
"--copy-from" when image is bigger than my
"swift_store_large_object_size" because then glance is trying to
split image to chunks with size given in
"swift_store_large_object_chunk_size" and when try to upload first
chunk to swift I have error:
2014-09-30 15:05:29.361 18023 ERROR glance.store.swift [-] Error
during chunked upload to backend, deleting stale chunks
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift Traceback
(most recent call last):
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File
"/usr/lib/python2.7/dist-packages/glance/store/swift.py", line 384,
in add
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
content_length=content_length)
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File
"/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1234,
in put_object
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
response_dict=response_dict)
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File
"/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1143,
in _retry
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
reset_func(func, *args, **kwargs)
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File
"/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1215,
in _default_reset
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift %
(container, obj))
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
ClientException: put_object('glance',
'9f56ccec-deeb-4020-95ba-ca7bf1170056-1', ...) failure and no
ability to reset contents for reupload.
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
2014-09-30 15:05:29.362 18023 ERROR glance.store.swift [-] Failed
to add object to Swift.
Got error from Swift: put_object('glance',
'9f56ccec-deeb-4020-95ba-ca7bf1170056-1', ...) failure and no
ability to reset contents for reupload.
2014-09-30 15:05:29.362 18023 ERROR glance.api.v1.upload_utils [-]
Failed to upload image 9f56ccec-deeb-4020-95ba-ca7bf1170056
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
Traceback (most recent call last):
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
File
"/usr/lib/python2.7/dist-packages/glance/api/v1/upload_utils.py",
line 101, in upload_data_to_store
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
store)
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
File "/usr/lib/python2.7/dist-packages/glance/store/__init__.py",
line 333, in store_add_to_backend
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
(location, size, checksum, metadata) = store.add(image_id, data,
size)
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
File "/usr/lib/python2.7/dist-packages/glance/store/swift.py",
line 447, in add
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
raise glance.store.BackendException(msg)
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
BackendException: Failed to add object to Swift.
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils Got
error from Swift: put_object('glance',
'9f56ccec-deeb-4020-95ba-ca7bf1170056-1', ...) failure and no
ability to reset contents for reupload.
Does someone of You got same error and know what is solution of it?
I was searching about that in google but I not found anything what
could solve my problem.
--
Best regards
Sławek Kapłoński
sla...@kaplonski.pl
___
Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [1]
Post to : openst...@lists.openstack.org
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [1]
Links:
--
[1] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
--
Pozdrawiam
Sławek Kapłoński
sla...@kaplonski.pl
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openst...@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
___
OpenStack-dev m
