Re: [ceph-users] S3 Bucket usage up 150% diference between rgw-admin and external metering tools.
On Tue, Jan 21, 2020 at 05:57:51PM +, Robin H. Johnson wrote: > On Mon, Jan 20, 2020 at 12:57:51PM +, EDH - Manuel Rios wrote: > > Hi Cephs > > > > Several nodes of our Ceph 14.2.5 are fully dedicated to host cold storage / > > backups information. > > > > Today checking the data usage with a customer found that rgw-admin is > > reporting: > ... > > That's near 5TB used space in CEPH, and the external tools are reporting > > just 1.42TB. > - What are the external tools? > - How many objects do the external tools report as existing? > - Do the external tools include incomplete multipart uploads in their > size data? > - If bucket versioning is enabled, do the tools include all versions in the > size data? > - Are there leftover multipart pieces without a multipart head? (this > is a Ceph bug that I think is fixed in your release, but old pieces > might still exist). This has been filed as https://tracker.ceph.com/issues/43756 It's definitely lots of leftover MPU bits, as well as inconsistency between MPU heads/parts in the index vs RADOS objects existing: esp. cases with leftover MPU part in RADOS+index but the rest of the MPU is gone. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: PGP signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] S3 Bucket usage up 150% diference between rgw-admin and external metering tools.
Hi Cbodley ,
As you requested by IRC we tested directly with AWS Cli.
Results:
aws --endpoint=http://XX --profile=ceph s3api list-multipart-uploads
--bucket Evol6
It reports near 170 uploads.
We used the last one:
{
"Initiator": {
"DisplayName": "x",
"ID": "xx"
},
"Initiated": "2019-12-03T01:23:06.007Z",
"UploadId": "2~r0BMPPs8CewVZ6Qheu1s9WzaBn7bBvU",
"StorageClass": "STANDARD",
"Key":
"MBS-da43656f-2b8c-464f-b341-03fdbdf446ae/CBB_SRV2K12/CBB_VM/192.168.0.197/SRV2K12/Hard
disk 1$/20191203010516/431.cbrevision",
"Owner": {
"DisplayName": "x",
"ID": ""
}
}
aws --endpoint=http://x --profile=ceph s3api abort-multipart-upload
--bucket Evol6 --key
'MBS-da43656f-2b8c-464f-b341-03fdbdf446ae/CBB_SRV2K12/CBB_VM/192.168.0.197/SRV2K12/Hard
disk 1$/20191203010516/431.cbrevision' --upload-id
2~r0BMPPs8CewVZ6Qheu1s9WzaBn7bBvU
Return: An error occurred (NoSuchUpload) when calling the AbortMultipartUpload
operation: Unknown
The same error is reported by S3CMD.
Maybe is there something wrong parsing the "1$" inside the key
Best Regards,
Regards
Manuel
-----Mensaje original-
De: ceph-users En nombre de EDH - Manuel
Rios
Enviado el: martes, 21 de enero de 2020 20:09
Para: Robin H. Johnson
CC: ceph-users@lists.ceph.com
Asunto: Re: [ceph-users] S3 Bucket usage up 150% diference between rgw-admin
and external metering tools.
Hi Robin,
- What are the external tools? CloudBerry S3 Explorer and S3 Browser
- How many objects do the external tools report as existing? Tool report 72142
keys (Aprox 6TB) vs CEPH num_objects 180981 (9TB)
- Do the external tools include incomplete multipart uploads in their size
data? I think no one external software include incomplete objects in the size,
due S3 api list recursive don't include it.
Checking for incomplete multiparts , I got a response 404 NoSuchKeys.
- If bucket versioning is enabled, do the tools include all versions in the
size data? Versioning is not enabled
- Are there leftover multipart pieces without a multipart head? How can we
check it?
Specific bucket information:
{
"bucket": "XX",
"tenant": "",
"zonegroup": "4d8c7c5f-ca40-4ee3-b5bb-b2cad90bd007",
"placement_rule": "default-placement",
"explicit_placement": {
"data_pool": "default.rgw.buckets.data",
"data_extra_pool": "default.rgw.buckets.non-ec",
"index_pool": "default.rgw.buckets.index"
},
"id": "48efb8c3-693c-4fe0-bbe4-fdc16f590a82.132873679.2",
"marker": "48efb8c3-693c-4fe0-bbe4-fdc16f590a82.3886182.52",
"index_type": "Normal",
"owner": "XXX",
"ver": "0#89789,1#60165,2#80652,3#76367",
"master_ver": "0#0,1#0,2#0,3#0",
"mtime": "2020-01-05 19:29:59.360574Z",
"max_marker": "0#,1#,2#,3#",
"usage": {
"rgw.main": {
"size": 9050249319344,
"size_actual": 9050421526528,
"size_utilized": 9050249319344,
"size_kb": 8838134101,
"size_kb_actual": 8838302272,
"size_kb_utilized": 8838134101,
"num_objects": 180981
},
"rgw.multimeta": {
"size": 0,
"size_actual": 0,
"size_utilized": 3861,
"size_kb": 0,
"size_kb_actual": 0,
"size_kb_utilized": 4,
"num_objects": 143
}
},
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1024,
"max_size_kb": 0,
"max_objects": -1
}
}
-Mensaje original-
De: ceph-users En nombre de Robin H.
Johnson Enviado el: martes, 21 de enero de 2020 18:58
CC: ceph-users@lists.ceph.com
Asunto: Re: [ceph-users] S3 Bucket usage up 150% diference between rgw-admin
and external metering tools.
On Mon, Jan 20, 2020 at 12:57:51PM +, EDH - Manuel Rios wrote:
> Hi Cephs
>
> Several nodes of our Ceph 14.2.5 are fully dedicated to host cold storage /
> backups information.
>
> Today checking the data usage with a customer found that rgw-admin is
>
Re: [ceph-users] S3 Bucket usage up 150% diference between rgw-admin and external metering tools.
Hi Robin,
- What are the external tools? CloudBerry S3 Explorer and S3 Browser
- How many objects do the external tools report as existing? Tool report 72142
keys (Aprox 6TB) vs CEPH num_objects 180981 (9TB)
- Do the external tools include incomplete multipart uploads in their size
data? I think no one external software include incomplete objects in the size,
due S3 api list recursive don't include it.
Checking for incomplete multiparts , I got a response 404 NoSuchKeys.
- If bucket versioning is enabled, do the tools include all versions in the
size data? Versioning is not enabled
- Are there leftover multipart pieces without a multipart head? How can we
check it?
Specific bucket information:
{
"bucket": "XX",
"tenant": "",
"zonegroup": "4d8c7c5f-ca40-4ee3-b5bb-b2cad90bd007",
"placement_rule": "default-placement",
"explicit_placement": {
"data_pool": "default.rgw.buckets.data",
"data_extra_pool": "default.rgw.buckets.non-ec",
"index_pool": "default.rgw.buckets.index"
},
"id": "48efb8c3-693c-4fe0-bbe4-fdc16f590a82.132873679.2",
"marker": "48efb8c3-693c-4fe0-bbe4-fdc16f590a82.3886182.52",
"index_type": "Normal",
"owner": "XXX",
"ver": "0#89789,1#60165,2#80652,3#76367",
"master_ver": "0#0,1#0,2#0,3#0",
"mtime": "2020-01-05 19:29:59.360574Z",
"max_marker": "0#,1#,2#,3#",
"usage": {
"rgw.main": {
"size": 9050249319344,
"size_actual": 9050421526528,
"size_utilized": 9050249319344,
"size_kb": 8838134101,
"size_kb_actual": 8838302272,
"size_kb_utilized": 8838134101,
"num_objects": 180981
},
"rgw.multimeta": {
"size": 0,
"size_actual": 0,
"size_utilized": 3861,
"size_kb": 0,
"size_kb_actual": 0,
"size_kb_utilized": 4,
"num_objects": 143
}
},
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1024,
"max_size_kb": 0,
"max_objects": -1
}
}
-Mensaje original-
De: ceph-users En nombre de Robin H. Johnson
Enviado el: martes, 21 de enero de 2020 18:58
CC: ceph-users@lists.ceph.com
Asunto: Re: [ceph-users] S3 Bucket usage up 150% diference between rgw-admin
and external metering tools.
On Mon, Jan 20, 2020 at 12:57:51PM +, EDH - Manuel Rios wrote:
> Hi Cephs
>
> Several nodes of our Ceph 14.2.5 are fully dedicated to host cold storage /
> backups information.
>
> Today checking the data usage with a customer found that rgw-admin is
> reporting:
...
> That's near 5TB used space in CEPH, and the external tools are reporting just
> 1.42TB.
- What are the external tools?
- How many objects do the external tools report as existing?
- Do the external tools include incomplete multipart uploads in their
size data?
- If bucket versioning is enabled, do the tools include all versions in the
size data?
- Are there leftover multipart pieces without a multipart head? (this
is a Ceph bug that I think is fixed in your release, but old pieces
might still exist).
--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail : robb...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB
E9B85B1F 825BCECF EE05E6F6 A48F6136
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] S3 Bucket usage up 150% diference between rgw-admin and external metering tools.
On Mon, Jan 20, 2020 at 12:57:51PM +, EDH - Manuel Rios wrote: > Hi Cephs > > Several nodes of our Ceph 14.2.5 are fully dedicated to host cold storage / > backups information. > > Today checking the data usage with a customer found that rgw-admin is > reporting: ... > That's near 5TB used space in CEPH, and the external tools are reporting just > 1.42TB. - What are the external tools? - How many objects do the external tools report as existing? - Do the external tools include incomplete multipart uploads in their size data? - If bucket versioning is enabled, do the tools include all versions in the size data? - Are there leftover multipart pieces without a multipart head? (this is a Ceph bug that I think is fixed in your release, but old pieces might still exist). -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
