Hi,
Every night my mailserver does a backup of the mail store filesystem which
often (but not allways) results in a waiting fuse process (
/sys/fs/fuse/connections/25/waiting) leaving the system with a steady load
of 1, see attachment.
The s3ql logs for the mailstore mountpoint are littered with these messages:
2015-09-13 09:14:08.932 [878] Thread-6: [backend] Encountered BadStatusLine
exception (''), retrying call to Backend.delete...
2015-09-13 09:23:09.387 [878] Dummy-21: [backend] Encountered BadStatusLine
exception (''), retrying call to Backend.open_read...
2015-09-13 10:14:59.053 [878] Thread-4: [backend] Encountered BadStatusLine
exception (''), retrying call to ObjectW.close...
2015-09-13 10:17:39.960 [878] Metadata-Upload-Thread: [mount] Dumping
metadata...
2015-09-13 10:17:39.963 [878] Metadata-Upload-Thread: [metadata]
..objects..
2015-09-13 10:17:40.008 [878] Metadata-Upload-Thread: [metadata] ..blocks..
2015-09-13 10:17:40.173 [878] Metadata-Upload-Thread: [metadata] ..inodes..
2015-09-13 10:17:40.281 [878] Metadata-Upload-Thread: [metadata]
..inode_blocks..
2015-09-13 10:17:40.347 [878] Metadata-Upload-Thread: [metadata]
..symlink_targets..
2015-09-13 10:17:40.348 [878] Metadata-Upload-Thread: [metadata] ..names..
2015-09-13 10:17:40.391 [878] Metadata-Upload-Thread: [metadata]
..contents..
2015-09-13 10:17:40.450 [878] Metadata-Upload-Thread: [metadata]
..ext_attributes..
2015-09-13 10:17:40.690 [878] Metadata-Upload-Thread: [mount] Compressing
and uploading metadata...
2015-09-13 10:17:42.092 [878] Metadata-Upload-Thread: [mount] Wrote 2.59
MiB of compressed metadata.
2015-09-13 10:17:42.092 [878] Metadata-Upload-Thread: [mount] Cycling
metadata backups...
2015-09-13 10:17:42.092 [878] Metadata-Upload-Thread: [metadata] Backing up
old metadata...
2015-09-13 10:43:11.665 [878] Thread-3: [backend] Encountered BadStatusLine
exception (''), retrying call to ObjectW.close...
2015-09-13 10:43:11.675 [878] Thread-4: [backend] Encountered BadStatusLine
exception (''), retrying call to ObjectW.close...
2015-09-13 11:24:09.157 [878] Thread-13: [backend] Encountered
BadStatusLine exception (''), retrying call to Backend.delete...
2015-09-13 11:24:09.158 [878] Thread-8: [backend] Encountered BadStatusLine
exception (''), retrying call to Backend.delete...
2015-09-13 11:24:09.163 [878] Thread-11: [backend] Encountered
BadStatusLine exception (''), retrying call to Backend.delete...
2015-09-13 11:44:06.777 [878] Thread-3: [backend] Encountered BadStatusLine
exception (''), retrying call to ObjectW.close...
2015-09-13 12:31:55.690 [878] Thread-4: [backend] Encountered BadStatusLine
exception (''), retrying call to ObjectW.close...
2015-09-13 13:12:08.495 [878] Thread-3: [backend] Encountered BadStatusLine
exception (''), retrying call to ObjectW.close...
2015-09-13 13:23:02.650 [878] Thread-4: [backend] Encountered BadStatusLine
exception (''), retrying call to ObjectW.close...
2015-09-13 14:29:51.875 [878] Thread-3: [backend] Encountered BadStatusLine
exception (''), retrying call to ObjectW.close...
2015-09-13 14:34:09.503 [878] Thread-9: [backend] Encountered BadStatusLine
exception (''), retrying call to Backend.delete...
2015-09-13 14:49:16.173 [878] Thread-3: [backend] Encountered BadStatusLine
exception (''), retrying call to ObjectW.close...
2015-09-13 15:02:08.641 [878] Dummy-21: [backend] Encountered BadStatusLine
exception (''), retrying call to Backend.open_read...
2015-09-13 16:06:59.665 [878] Thread-4: [backend] Encountered BadStatusLine
exception (''), retrying call to ObjectW.close...
2015-09-13 16:09:11.700 [878] Thread-3: [backend] Encountered BadStatusLine
exception (''), retrying call to ObjectW.close...
2015-09-13 16:44:09.737 [878] Thread-5: [backend] Encountered BadStatusLine
exception (''), retrying call to Backend.delete...
2015-09-13 16:44:09.739 [878] Thread-14: [backend] Encountered
BadStatusLine exception (''), retrying call to Backend.delete...
The backup mountpoint log looks like this:
2015-09-13 05:38:13.129 [902] Thread-7: [backend] Encountered error
exception ([Errno 104] Connection reset by peer), retrying call to
Backend.delete...
2015-09-13 05:38:13.131 [902] Thread-8: [backend] Encountered error
exception ([Errno 104] Connection reset by peer), retrying call to
Backend.delete...
2015-09-13 05:40:53.660 [902] Thread-4: [backend] Encountered error
exception ([Errno 104] Connection reset by peer), retrying call to
ObjectW.close...
2015-09-13 10:17:39.670 [902] Metadata-Upload-Thread: [mount] Dumping
metadata...
2015-09-13 10:17:39.683 [902] Metadata-Upload-Thread: [metadata]
..objects..
2015-09-13 10:17:39.704 [902] Metadata-Upload-Thread: [metadata] ..blocks..
2015-09-13 10:17:39.706 [902] Metadata-Upload-Thread: [metadata] ..inodes..
2015-09-13 10:17:39.706 [902] Metadata-Upload-Thread: [metadata]
..inode_blocks..
2015-09-13 10:17:39.708 [902] Metadata-Upload-Thread: [metadata]
..symlink_targets..
2015-09-13 10:17:39.709 [902] Metadata-Upload-Thread: [metadata] ..names..
2015-09-13 10:17:39.710 [902] Metadata-Upload-Thread: [metadata]
..contents..
2015-09-13 10:17:39.710 [902] Metadata-Upload-Thread: [metadata]
..ext_attributes..
2015-09-13 10:17:39.713 [902] Metadata-Upload-Thread: [backends.s3c]
Encountered BadStatusLine exception (''), retrying call to
s3c.Backend.list()
2015-09-13 10:17:40.202 [902] Metadata-Upload-Thread: [mount] Compressing
and uploading metadata...
2015-09-13 10:17:40.325 [902] Metadata-Upload-Thread: [mount] Wrote 0.02
MiB of compressed metadata.
2015-09-13 10:17:40.325 [902] Metadata-Upload-Thread: [mount] Cycling
metadata backups...
2015-09-13 10:17:40.325 [902] Metadata-Upload-Thread: [metadata] Backing up
old metadata...
2015-09-14 03:15:12.782 [902] Thread-7: [backend] Encountered BadStatusLine
exception (''), retrying call to Backend.delete...
2015-09-14 03:17:27.757 [902] Thread-11: [backend] Encountered
BadStatusLine exception (''), retrying call to Backend.delete...
2015-09-14 04:00:43.794 [902] Thread-6: [backend] Encountered BadStatusLine
exception (''), retrying call to Backend.delete...
2015-09-14 04:00:43.796 [902] Thread-12: [backend] Encountered error
exception ([Errno 32] Broken pipe), retrying call to Backend.delete...
2015-09-14 04:00:43.797 [902] Thread-9: [backend] Encountered BadStatusLine
exception (''), retrying call to Backend.delete...
2015-09-14 04:00:43.797 [902] Thread-13: [backend] Encountered
BadStatusLine exception (''), retrying call to Backend.delete...
2015-09-14 04:00:43.797 [902] Thread-5: [backend] Encountered BadStatusLine
exception (''), retrying call to Backend.delete...
2015-09-14 04:00:43.798 [902] Thread-14: [backend] Encountered
BadStatusLine exception (''), retrying call to Backend.delete...
2015-09-14 04:00:43.805 [902] Thread-7: [backend] Encountered BadStatusLine
exception (''), retrying call to Backend.delete...
2015-09-14 04:00:43.806 [902] Thread-8: [backend] Encountered BadStatusLine
exception (''), retrying call to Backend.delete...
2015-09-14 04:00:43.806 [902] Thread-10: [backend] Encountered error
exception ([Errno 32] Broken pipe), retrying call to Backend.delete...
The backup process is a special mailstore command that creates a tar file
for every mailbox.
I'm using 1.16-1 because I still haven't found the time to investigate a
working upgrade path to 2.14, since there is no path from 1.16 to 2.14. But
that's another issue.
Anyway, I was wondering what could cause this stale process? I allways fsck
both filesystems before mount on reboot, but it feels like there's an
offending file or block somewhere in the mailstore that triggers this
problem after some time. Is there a way to find out which inode/file is
triggering the fuse waiting state in sysfs?
I can get away with the waiting process by stopping the mailserver and
echo'ing something to abort and remounting the fs, but I'm unsure about
what mail will be lost in space (if so?).
As I write this, I see there's a 1.19 release in bitbucket. I use ubuntu
14.04 packages from http://ppa.launchpad.net/nikratio/s3ql/ubuntu trusty
main, why is 1.19 not in that repo?
I see, you don't support 1.x in that repo, it's the ubuntu one I'm seeing
and that's not updated apparently?
Regards,
Martin
--
You received this message because you are subscribed to the Google Groups
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
