Ok, I dug a bit more, and it seems to me that the problem is with the
manifest that was created. I was able to reproduce a similar issue (opened
ceph bug #11622), for which I also have a fix.
I created new tests to cover this issue, and we'll get those recent fixes
as soon as we can, after we test for any regressions.
Thanks,
Yehuda
----- Original Message -----
From: "Yehuda Sadeh-Weinraub" <yeh...@redhat.com>
To: "Sean Sullivan" <seapasu...@uchicago.edu>
Cc: ceph-users@lists.ceph.com
Sent: Wednesday, May 13, 2015 2:33:07 PM
Subject: Re: [ceph-users] RGW - Can't download complete object
That's another interesting issue. Note that for part 12_80 the manifest
specifies (I assume, by the messenger log) this part:
default.20283.1__shadow_b235040a-46b6-42b3-b134-962b1f8813d5/28357709e44fff211de63b1d2c437159.bam.tJ8UddmcCxe0lOsgfHR9Q-ZHXdlrM14.12_80
(note the 'tJ8UddmcCxe0lOsgfHR9Q-ZHXdlrM14')
whereas it seems that you do have the original part:
default.20283.1__shadow_b235040a-46b6-42b3-b134-962b1f8813d5/28357709e44fff211de63b1d2c437159.bam.2/-ztodNISNLlaNeV4kDmrQwmkECBP2mZ.12_80
(note the '2/...')
The part that the manifest specifies does not exist, which makes me think
that there is some weird upload sequence, something like:
- client uploads part, upload finishes but client does not get ack for
it
- client retries (second upload)
- client gets ack for the first upload and gives up on the second one
But I'm not sure if it would explain the manifest, I'll need to take a
look
at the code. Could such a sequence happen with the client that you're
using
to upload?
Yehuda
----- Original Message -----
From: "Sean Sullivan" <seapasu...@uchicago.edu>
To: "Yehuda Sadeh-Weinraub" <yeh...@redhat.com>
Cc: ceph-users@lists.ceph.com
Sent: Wednesday, May 13, 2015 2:07:22 PM
Subject: Re: [ceph-users] RGW - Can't download complete object
Sorry for the delay. It took me a while to figure out how to do a range
request and append the data to a single file. The good news is that the
end
file seems to be 14G in size which matches the files manifest size. The
bad
news is that the file is completely corrupt and the radosgw log has
errors.
I am using the following code to perform the download::
https://raw.githubusercontent.com/mumrah/s3-multipart/master/s3-mp-download.py
Here is a clip of the log file::
--
2015-05-11 15:28:52.313742 7f570db7d700 1 -- 10.64.64.126:0/1033338
<==
osd.11 10.64.64.101:6809/942707 5 ==== osd_op_reply(74566287
default.20283.1__shadow_b235040a-46b6-42b3-b134-962b1f8813d5/28357709e44fff211de63b1d2c437159.bam.2/-ztodNISNLlaNeV4kDmrQwmkECBP2mZ.13_12
[read 0~858004] v0'0 uv41308 ondisk = 0) v6 ==== 304+0+858004
(1180387808 0
2445559038) 0x7f53d005b1a0 con 0x7f56f8119240
2015-05-11 15:28:52.313797 7f57067fc700 20 get_obj_aio_completion_cb:
io
completion ofs=12934184960 len=858004
2015-05-11 15:28:52.372453 7f570db7d700 1 -- 10.64.64.126:0/1033338
<==
osd.45 10.64.64.101:6845/944590 2 ==== osd_op_reply(74566142
default.20283.1__shadow_b235040a-46b6-42b3-b134-962b1f8813d5/28357709e44fff211de63b1d2c437159.bam.tJ8UddmcCxe0lOsgfHR9Q-ZHXdlrM14.12_80
[read 0~4194304] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6
====
302+0+0 (3754425489 0 0) 0x7f53d005b1a0 con 0x7f56f81b1f30
2015-05-11 15:28:52.372494 7f57067fc700 20 get_obj_aio_completion_cb:
io
completion ofs=12145655808 len=4194304
2015-05-11 15:28:52.372501 7f57067fc700 0 ERROR: got unexpected error
when
trying to read object: -2
2015-05-11 15:28:52.426079 7f570db7d700 1 -- 10.64.64.126:0/1033338
<==
osd.21 10.64.64.102:6856/1133473 16 ==== osd_op_reply(74566144
default.20283.1__shadow_b235040a-46b6-42b3-b134-962b1f8813d5/28357709e44fff211de63b1d2c437159.bam.2/-ztodNISNLlaNeV4kDmrQwmkECBP2mZ.11_12
[read 0~3671316] v0'0 uv41395 ondisk = 0) v6 ==== 304+0+3671316
(1695485150
0 3933234139) 0x7f53d005b1a0 con 0x7f56f81e17d0
2015-05-11 15:28:52.426123 7f57067fc700 20 get_obj_aio_completion_cb:
io
completion ofs=10786701312 len=3671316
2015-05-11 15:28:52.504072 7f570db7d700 1 -- 10.64.64.126:0/1033338
<==
osd.82 10.64.64.103:6857/88524 2 ==== osd_op_reply(74566283
default.20283.1__shadow_b235040a-46b6-42b3-b134-962b1f8813d5/28357709e44fff211de63b1d2c437159.bam.2/-ztodNISNLlaNeV4kDmrQwmkECBP2mZ.13_8
[read 0~4194304] v0'0 uv41566 ondisk = 0) v6 ==== 303+0+4194304
(1474509283
0 3209869954) 0x7f53d005b1a0 con 0x7f56f81b1420
2015-05-11 15:28:52.504118 7f57067fc700 20 get_obj_aio_completion_cb:
io
completion ofs=12917407744 len=4194304
I couldn't really find any good documentation on how fragments/files
are
layed out on the object file system so I am not sure on where the file
will
be. How could the 4mb object have issues but the cluster be completely
health okay? I did do the rados stat of each object inside ceph and
they
all
appear to be there::
http://paste.ubuntu.com/11118561/
The sum of all of the objects :: 14584887282
The stat of the object inside ceph:: 14577056082
So for some reason I have more data in objects than the key manifest.
We
easiliy identified this object via the same method as the other thread
I
have::
for key in keys:
....: if ( key.name ==
'b235040a-46b6-42b3-b134-962b1f8813d5/28357709e44fff211de63b1d2c437159.bam'
):
....: implicit = key.size
....: explicit =
conn.get_bucket(bucket).get_key(key.name).size
....: absolute = abs(implicit - explicit)
....: print key.name
....: print implicit
....: print explicit
....:
b235040a-46b6-42b3-b134-962b1f8813d5/28357709e44fff211de63b1d2c437159.bam
14578628946
14577056082
So it looks like I have 3 different sizes. I figure this may be the
network
issue that was mentioned in the other thread but seeing as this is not
the
first 512k and the overalll size still matches as well as the errors I
am
seeing in the gateway I feel that this may be a bigger issue.
Has anyone seen this before? The only mention of the "got unexpected
error
when trying to read object" is here
(http://lists.ceph.com/pipermail/ceph-commit-ceph.com/2014-May/021688.html)
but my google skills are pretty poor.
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
