Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.

On Thu, Jan 21, 2016 at 4:02 PM, seapasu...@uchicago.edu wrote: > I haven't been able to reproduce the issue on my end but I do not fully > understand how the bug exists or why it is happening. I was finally given > the code they are using to upload the files:: > >

Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.

I haven't been able to reproduce the issue on my end but I do not fully understand how the bug exists or why it is happening. I was finally given the code they are using to upload the files:: http://pastebin.com/N0j86NQJ I don't know if this helps at all :-(. the other thing is that I have

Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.

On 1/19/16 4:00 PM, Yehuda Sadeh-Weinraub wrote: On Fri, Jan 15, 2016 at 5:04 PM, seapasu...@uchicago.edu wrote: I have looked all over and I do not see any explicit mention of "NWS_NEXRAD_NXL2DP_PAKC_2015010111_20150101115959" in the logs nor do I see a

Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.

On Wed, Jan 20, 2016 at 10:43 AM, seapasu...@uchicago.edu wrote: > > > On 1/19/16 4:00 PM, Yehuda Sadeh-Weinraub wrote: >> >> On Fri, Jan 15, 2016 at 5:04 PM, seapasu...@uchicago.edu >> wrote: >>> >>> I have looked all over and I do not see any

Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.

So is there any way to prevent this from happening going forward? I mean ideally this should never be possible, right? Even with a complete object that is 0 bytes it should be downloaded as 0 bytes and have a different md5sum and not report as 7mb? On 1/20/16 1:30 PM, Yehuda Sadeh-Weinraub

Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.

Keep in mind that if the problem is that the tail is being sent to garbage collection, you'll only see the 404 after a few hours. A shorter way to check it would be by listing the gc entries (with --include-all). Yehuda On Wed, Jan 20, 2016 at 1:52 PM, seapasu...@uchicago.edu

Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.

We'll need to confirm that this is the actual issue, and then have it fixed. It would be nice to have some kind of a unitest that reproduces it. Yehuda On Wed, Jan 20, 2016 at 1:34 PM, seapasu...@uchicago.edu wrote: > So is there any way to prevent this from happening

Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.

I'm working on getting the code they used and trying different timeouts in my multipart upload code. Right now I have not created any new 404 keys though :-( On 1/20/16 3:44 PM, Yehuda Sadeh-Weinraub wrote: We'll need to confirm that this is the actual issue, and then have it fixed. It would

Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.

I was wondering if there is anything else I could provide outside of the radosgw logs during the file upload? At this point I am uploading 7mb files repeatedly to try to reproduce this error but so far I do not have any missing keys in my test bucket. I can't see this happening if we were to

Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.

On Fri, Jan 15, 2016 at 5:04 PM, seapasu...@uchicago.edu wrote: > I have looked all over and I do not see any explicit mention of > "NWS_NEXRAD_NXL2DP_PAKC_2015010111_20150101115959" in the logs nor do I > see a timestamp from November 4th although I do see log

Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.

Hello Yehuda, Here it is:: radosgw-admin object stat --bucket="noaa-nexrad-l2" --object="2015/01/01/PAKC/NWS_NEXRAD_NXL2DP_PAKC_2015010111_20150101115959.tar" { "name": "2015\/01\/01\/PAKC\/NWS_NEXRAD_NXL2DP_PAKC_2015010111_20150101115959.tar", "size": 7147520, "policy":

Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.

On Thu, Jan 14, 2016 at 10:51 PM, seapasu...@uchicago.edu wrote: > It looks like the gateway is experiencing a similar race condition to what > we reported before. > > The rados object has a size of 0 bytes but the bucket index shows the object > listed and the object

Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.

On Fri, Jan 15, 2016 at 9:36 AM, seapasu...@uchicago.edu wrote: > Hello Yehuda, > > Here it is:: > > radosgw-admin object stat --bucket="noaa-nexrad-l2" > --object="2015/01/01/PAKC/NWS_NEXRAD_NXL2DP_PAKC_2015010111_20150101115959.tar" > { > "name": >

Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.

lacadmin@kh28-10:~$ rados -p .rgw.buckets ls | grep 'pcu5Hz6' lacadmin@kh28-10:~$ Nothing was found. That said when I run the command with another prefix snippet:: lacadmin@kh28-10:~$ rados -p .rgw.buckets ls | grep 'wksHvto'

Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.

That's interesting, and might point at the underlying issue that caused it. Could be a racing upload that somehow ended up with the wrong object head. The 'multipart' object should be 4M in size, and the 'shadow' one should have the remainder of the data. You can run 'rados stat -p .rgw.buckets '

Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.

Sorry I am a bit confused. The successful list that I provided is from a different object of the same size to show that I could indeed get a list. Are you saying to copy the working object to the missing object? Sorry for the confusion. On 1/15/16 3:20 PM, Yehuda Sadeh-Weinraub wrote: That's

Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.

Ah, I see. Misread that and the object names were very similar. No, don't copy it. You can try to grep for the specific object name and see if there are pieces of it lying around under a different upload id. Yehuda On Fri, Jan 15, 2016 at 1:44 PM, seapasu...@uchicago.edu

Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.

Sorry for the confusion:: When I grepped for the prefix of the missing object:: "2015\/01\/01\/PAKC\/NWS_NEXRAD_NXL2DP_PAKC_2015010111_20150101115959.tar.2~pcu5Hz6foFXjlSxBat22D8YMcHlQOBD" I am not able to find any chunks of the object:: lacadmin@kh28-10:~$ rados -p .rgw.buckets ls | grep

Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.

The head object of a multipart object has 0 size, so it's expected. What's missing is the tail of the object. I don't assume you have any logs from when the object was uploaded? Yehuda On Fri, Jan 15, 2016 at 2:12 PM, seapasu...@uchicago.edu wrote: > Sorry for the

Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.

I have looked all over and I do not see any explicit mention of "NWS_NEXRAD_NXL2DP_PAKC_2015010111_20150101115959" in the logs nor do I see a timestamp from November 4th although I do see log rotations dating back to october 15th. I don't think it's possible it wasn't logged so I am going

[ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.

I am not sure why this is happening someone used s3cmd to upload around 130,000 7mb objects to a single bucket. Now we are tearing down the cluster to rebuild it better, stronger, and hopefully faster. Before we destroy it we need to download all of the data. I am running through all of the

Re: [ceph-users] RGW -- 404 on keys in bucket.list() thousands of multipart ids listed as well.

It looks like the gateway is experiencing a similar race condition to what we reported before. The rados object has a size of 0 bytes but the bucket index shows the object listed and the object metadata shows a size of 7147520 bytes. I have a lot of logs but I don't think any of them have