Hi,
I have a large-ish S3QL filesystem stored on OVH Swift Storage.. it was
created using S3QL 2.22, and has been upgraded over time, so it is now
using S3QL 2.29
Every so often for various reasons the mount has crashed (ovh network
issues, space issues on my server, etc).. but running a fsck.s3ql on the
system to resolve the issues, and remounting and continuing use has been OK.
But a few days ago the mount went offline with the error:
*s3ql.backends.s3c.HTTPError:
413 Request Entity Too Large*
I started an fsck.s3ql - and it failed to complete with the (not
unexpected!) same error. *s3ql.backends.s3c.HTTPError: 413 Request Entity
Too Large*
I am now running* fsck.f3ql* with --debug in the hope it will allow me to
zoom in on the problem object or objects, as the swift CLI seems quite slow
for digging down unto this information by listing object sizes, I'll also
open a request with OVH support if they have changed any limits on their
swift object server recently.
Are there any other commands I can run that will help drill down to find
and resolve the issue? s3ql_verify might just run for a long time and hit
the same problem?
The file system was created with the default --max-obj-size so I cant see
this being an issue? when OVH swift storage allows much much larger objects
according to the capabilities command.
more details below......
*Details from mount.log of the mount error:*
2018-08-08 14:47:30.960 18159:Metadata-Upload-Thread
s3ql.metadata.upload_metadata: Compressing and uploading metadata...
2018-08-08 15:41:03.977 18159:Metadata-Upload-Thread root.excepthook:
Uncaught top-level exception:
Traceback (most recent call last):
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/mount.py",
line 64, in run_with_except_hook
run_old(*args, **kw)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/mount.py",
line 659, in run
upload_metadata(backend, fh, self.param)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/metadata.py",
line 322, in upload_metadata
metadata=param, is_compressed=True)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/backends/common.py",
line 108, in wrapped
return method(*a, **kw)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/backends/common.py",
line 340, in perform_write
return fn(fh)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/backends/comprenc.py",
line 536, in __exit__
self.close()
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/backends/comprenc.py",
line 530, in close
self.fh.close()
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/backends/common.py",
line 108, in wrapped
return method(*a, **kw)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/backends/s3c.py",
line 948, in close
headers=self.headers, body=self.fh)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/backends/swift.py",
line 268, in _do_request
raise HTTPError(resp.status, resp.reason, resp.headers)
s3ql.backends.s3c.HTTPError: 413 Request Entity Too Large
2018-08-08 15:41:40.912 18159:MainThread s3ql.mount.unmount: Unmounting
file system...
2018-08-08 15:41:41.312 18159:MainThread root.excepthook: Uncaught
top-level exception:
Traceback (most recent call last):
File "/usr/local/bin/mount.s3ql", line 11, in <module>
load_entry_point('s3ql==2.29', 'console_scripts', 'mount.s3ql')()
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/mount.py",
line 222, in main
raise RuntimeError('Received signal %d, terminating' % (ret,))
RuntimeError: Received signal 15, terminating
*Details from fsck.log of the fsck error...*
2018-08-11 13:32:24.138 2396:MainThread s3ql.metadata.dump_metadata:
..contents..
2018-08-11 13:48:30.702 2396:MainThread s3ql.metadata.dump_metadata:
..ext_attributes..
2018-08-11 13:48:30.710 2396:MainThread s3ql.metadata.upload_metadata:
Compressing and uploading metadata...
2018-08-11 14:40:03.096 2396:MainThread root.excepthook: Uncaught top-level
exception:
Traceback (most recent call last):
File "/usr/local/bin/fsck.s3ql", line 11, in <module>
load_entry_point('s3ql==2.29', 'console_scripts', 'fsck.s3ql')()
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/fsck.py",
line 1343, in main
dump_and_upload_metadata(backend, db, param)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/metadata.py",
line 313, in dump_and_upload_metadata
upload_metadata(backend, fh, param)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/metadata.py",
line 322, in upload_metadata
metadata=param, is_compressed=True)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/backends/common.py",
line 108, in wrapped
return method(*a, **kw)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/backends/common.py",
line 340, in perform_write
return fn(fh)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/backends/comprenc.py",
line 536, in __exit__
self.close()
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/backends/comprenc.py",
line 530, in close
self.fh.close()
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/backends/common.py",
line 108, in wrapped
return method(*a, **kw)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/backends/s3c.py",
line 948, in close
headers=self.headers, body=self.fh)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.29-py3.5-linux-x86_64.egg/s3ql/backends/swift.py",
line 268, in _do_request
raise HTTPError(resp.status, resp.reason, resp.headers)
s3ql.backends.s3c.HTTPError: 413 Request Entity Too Large
*Output of s3qlstat from a recent run of my backup script*
Directory entries: 439577485
Inodes: 439576371
Data blocks: 3933555
Total data size: 50.9 TB
After de-duplication: 1.62 TB (3.17% of total)
After compression: 812 GiB (1.56% of total, 49.04% of de-duplicated)
Database size: 52.0 GiB (uncompressed)
Cache size: 55.4 GiB, 338820 entries
Cache size (dirty): 0 bytes, 0 entries
Queued object removals: 0
*Output of "swift capabilities"*
Core: swift
Options:
account_autocreate: True
account_listing_limit: 10000
allow_account_management: True
container_listing_limit: 10000
extra_header_count: 0
max_account_name_length: 256
max_container_name_length: 256
max_file_size: 5368709122
max_header_size: 8192
max_meta_count: 90
max_meta_name_length: 128
max_meta_overall_size: 4096
max_meta_value_length: 256
max_object_name_length: 1024
policies: [{u'default': True, u'diskfile': u'egg:swift#replication.fs',
u'name': u'PCS', u'aliases': u'PCS'}, {u'diskfile':
u'egg:swift#erasure_coding.fs', u'name': u'PCA', u'aliases': u'PCA'}]
strict_cors_mode: True
valid_api_versions: [u'v1', u'v1.0']
version: 2.15.1.dev65
Additional middleware: account_quotas
Additional middleware: bulk_delete
Options:
max_deletes_per_request: 10000
max_failed_deletes: 1000
Additional middleware: bulk_upload
Options:
max_containers_per_extraction: 10000
max_failed_extractions: 1000
Additional middleware: cname_lookup
Options:
lookup_depth: 1
Additional middleware: container_quotas
Additional middleware: container_sync
Options:
realms: {u'OVH_PUBLIC_CLOUD': {u'clusters': {u'WAW1': {}, u'SERCO-DIAS1':
{}, u'GRA1': {}, u'GRA3': {}, u'GRA5': {}, u'UK1': {}, u'SBG5': {},
u'SBG3': {}, u'SBG1': {}, u'BHS3': {}, u'DE1': {}, u'BHS1': {u'current':
True}}}}
Additional middleware: crossdomain
Additional middleware: domain_remap
Options:
default_reseller_prefix: None
Additional middleware: formpost
Additional middleware: slo
Options:
max_manifest_segments: 1000
max_manifest_size: 2097152
min_segment_size: 1
Additional middleware: staticweb
Additional middleware: tempurl
Options:
incoming_allow_headers: []
incoming_remove_headers: [u'x-timestamp']
methods: [u'GET', u'HEAD', u'PUT', u'POST', u'DELETE']
outgoing_allow_headers: [u'x-object-meta-public-*']
outgoing_remove_headers: [u'x-object-meta-*']
--
You received this message because you are subscribed to the Google Groups
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to s3ql+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
