[SOLVED] Squatter "No space left on device"

2015-06-16 Thread Tecnologia Charne.Net 
El 15/06/15 a las 11:26, Anthony Prades escibió:
> On 06/15/2015 03:41 PM, Tecnologia Charne.Net wrote:
>> Hello!
>>
>> i'm using cyrus IMAPD since 11 years, on Debian VM. [0]
>>
>> Now, I found this problem and I can't find docs or comments about it
>> on web:
>>
>> # cyrus squatter -riv user/jgcha...@comunidad.unnoba.edu.ar
>> Indexing mailbox user/jgcha...@comunidad.unnoba.edu.ar... Writing index
>> update: No space left on device
>>
>>
>> [0]
>> localhost> ver
>> name   : Cyrus IMAPD
>> version: v2.4.17-caldav-beta10-Debian-2.4.17+caldav~beta10-18
>> 97aa1c46 2014-07-22
>> vendor : Project Cyrus
>> support-url: http://www.cyrusimap.org
>> os : Linux
>> os-version : 3.14-1-amd64
>> environment: Built w/Cyrus SASL 2.1.26
>>   Running w/Cyrus SASL 2.1.26
>>   Built w/Berkeley DB 5.3.28: (September  9, 2013)
>>   Running w/Berkeley DB 5.3.28: (September  9, 2013)
>>   Built w/OpenSSL 1.0.1k 8 Jan 2015
>>   Running w/OpenSSL 1.0.2a 19 Mar 2015
>>   Built w/zlib 1.2.8
>>   Running w/zlib 1.2.8
>>   CMU Sieve 2.4
>>   TCP Wrappers
>>   NET-SNMP
>>   mmap = shared
>>   lock = fcntl
>>   nonblock = fcntl
>>   idle = poll
>>
>>
>> * Partition has 1.3 TB and only 400 GB used.
> Hi,
>
> maybe all inodes are used. What's the result of: df -i ?
>
> Anthony
>>
>>
>> * User mailbox has correct permissions.
>>
>> mda:/var/spool/cyrus/mail/domain/c/comunidad.unnoba.edu.ar/j/user/jgcharne#
>>
>> ls -all
>> drwx-- 1 cyrus mail  150 jun 15 10:38 .
>> drwx-- 1 cyrus mail   16 abr 13 11:17 ..
>> -rw--- 1 cyrus mail 1105 jun 15 10:22 7.
>> -rw--- 1 cyrus mail 7412 jun 15 10:22 cyrus.cache
>> -rw--- 1 cyrus mail  227 jun 15 10:15 cyrus.header
>> -rw--- 1 cyrus mail  800 jun 15 10:26 cyrus.index
>> -rw--- 1 cyrus mail  112 jun 15 10:17 cyrus.squat
>> drwx-- 1 cyrus mail   90 jun 10 05:17 Drafts
>> drwx-- 1 cyrus mail  122 jun 15 10:00 Sent
>> drwx-- 1 cyrus mail   90 jun 10 05:17 Spam
>> drwx-- 1 cyrus mail   90 jun 10 05:17 Templates
>> drwx-- 1 cyrus mail   90 jun 15 10:16 Trash
>>
>>
>>
>> Someone had this problem?
>>
>> Thanks in advance.
>>
>> Javier.-
>>
>>

Problem was btrfs metadata:

  cyrus squatter user/jav...@unnoba.edu.ar
  Writing index update: No space left on device


Filesystem checks:

mda:~# btrfs filesystem df /var/spool/cyrus/
Data, single: total=1.26TiB, used=349.31GiB
System, DUP: total=8.00MiB, used=160.00KiB
System, single: total=4.00MiB, used=0.00B
*Metadata, DUP: total=3.50GiB, used=2.99GiB*
Metadata, single: total=8.00MiB, used=0.00B


btrfs filesystem usage /var/spool/cyrus
Overall:
Device size:   1.27TiB
Device allocated:   1.27TiB
Device unallocated:  60.00KiB
Device missing: 0.00B
Used: 355.30GiB
Free (estimated): 943.69GiB(min: 943.69GiB)
Data ratio:  1.00
Metadata ratio:  2.00
Global reserve: 0.00B(used: 0.00B)

Data,single: Size:1.26TiB, Used:349.31GiB
   /dev/xvdb1   1.26TiB

Metadata,single: Size:8.00MiB, Used:0.00B
   /dev/xvdb1   8.00MiB

*Metadata,DUP: Size:3.50GiB, Used:2.99GiB**
**   /dev/xvdb1   7.00GiB*

System,single: Size:4.00MiB, Used:0.00B
   /dev/xvdb1   4.00MiB

System,DUP: Size:8.00MiB, Used:160.00KiB
   /dev/xvdb1  16.00MiB

Unallocated:
   /dev/xvdb1  60.00KiB


# btrfs balance start -v -dusage=5 /var/spool/cyrus/
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=5
Done, had to relocate 213 out of 1302 chunks


Now, it's working.


Javier.-

Re: LinkChecker

2015-06-16 Thread Nic Bernstein 

Chris,
My apologies, I phrased my question wrong.  I meant to ask "is this 
checking http://www.cyrusimap.org/ only, or also 
https://*.cyrus.foundation/ ..."?


I understand that the old site is still putatively the "main" customer 
facing site, we are developing https://docs.cyrus.foundation/ so it 
would be nice to know if we're throwing bogons up there.


BTW: is there a good way to ignore the favicon.ico errors?

Cheers,
-nic

On 06/15/2015 08:43 PM, Chris Davies wrote:

Hi Nic,
There could be some confusion because we have two domains for the same 
project and two Jenkins installations.
The application LinkChecker  was 
installed on our new Jenkins server and is running at 
https://ci.cyrus.foundation. LinkChecker can scan any website for 
links regardless of where it's installed, follow those links and will 
report on any broken links or missing pages. Currently I've set it to 
check https://www.cyrusimap.org once a day. As I understand things, 
www.cyrusimap.org is the main customer facing website and 
cyrus,foundation is the development site.

Cheers,
Chris
On Mon, 15 Jun 2015, at 10:29 PM, Nic Bernstein wrote:

On 06/14/2015 10:59 PM, Chris Davies wrote:
I've just installed LinkChecker 
on the Jenkins server.
It's currently running once an hour against 
http://www.cyrusimap.org/ and will report any broken links to 
https://ci.cyrus.foundation/job/LinkChecker%20-%20CyrusImap.org/
If you click the most recent job (under build history), then click 
'Console Output' you can see the most recent report.
Over time it would be good if we could fix the broken links or I can 
add ignore rules for anything we don't plan on fixing.

Cheers,
Chris

Chris,
Just to clarify, is this installed on http://www.cyrusimap.org/ only, 
or also on https://*.cyrus.foundation/ as well?  I ask because Bron's 
meeting notes say the latter.

Cheers,
-nic
--
Nic bernstein...@onlight.com  
Onlight llc.www.onlight.com  
219 N. Milwaukee St., Ste. 2A v. 414.272.4477
Milwaukee, Wisconsin  53202   f. 414.290.0335


--
Nic Bernstein n...@onlight.com
Onlight, Inc. www.onlight.com
1442 N Farwell Ave., Suite 600v. 414.272.4477
Milwaukee, Wisconsin  53202

<>

Re: [SOLVED] Squatter "No space left on device"

2015-06-16 Thread Bron Gondwana 
On Tue, Jun 16, 2015, at 10:29 PM, Tecnologia Charne.Net wrote:
>
Problem was btrfs metadata:
>
>
cyrus squatter user/jav...@unnoba.edu.ar
>
Writing index update: No space left on device
>
>
>
Filesystem checks:
>
>
mda:~# btrfs filesystem df /var/spool/cyrus/
>
Data, single: total=1.26TiB, used=349.31GiB
>
System, DUP: total=8.00MiB, used=160.00KiB
>
System, single: total=4.00MiB, used=0.00B
> *Metadata, DUP: total=3.50GiB, used=2.99GiB*
>
Metadata, single: total=8.00MiB, used=0.00B
>
>
>
btrfs filesystem usage /var/spool/cyrus
>
Overall:
>
Device size:           1.27TiB
>
Device allocated:           1.27TiB
>
Device unallocated:          60.00KiB
>
Device missing:         0.00B
>
Used:             355.30GiB
>
Free (estimated):         943.69GiB    (min: 943.69GiB)
>
Data ratio:              1.00
>
Metadata ratio:          2.00
>
Global reserve:         0.00B    (used: 0.00B)
>
>
Data,single: Size:1.26TiB, Used:349.31GiB
>
/dev/xvdb1       1.26TiB
>
>
Metadata,single: Size:8.00MiB, Used:0.00B
>
/dev/xvdb1       8.00MiB
>
> *Metadata,DUP: Size:3.50GiB, Used:2.99GiB**
**   /dev/xvdb1       7.00GiB*
>
>
System,single: Size:4.00MiB, Used:0.00B
>
/dev/xvdb1       4.00MiB
>
>
System,DUP: Size:8.00MiB, Used:160.00KiB
>
/dev/xvdb1      16.00MiB
>
>
Unallocated:
>
/dev/xvdb1      60.00KiB
>
>
>
# btrfs balance start -v -dusage=5 /var/spool/cyrus/
>
Dumping filters: flags 0x1, state 0x0, force is off
>
DATA (flags 0x2): balancing, usage=5
>
Done, had to relocate 213 out of 1302 chunks
>
>
>
Now, it's working.

Why would anybody ever run production data on btrfs?

https://btrfs.wiki.kernel.org/index.php/Main_Page

"The Btrfs code base is under heavy development."

https://btrfs.wiki.kernel.org/index.php/Gotchas

"Files with a lot of random writes can become heavily fragmented (1+
extents) causing trashing on HDDs and excessive multi-second spikes of
CPU load on systems with an SSD or large amount a RAM."

Unfortunately, both cyrus.index files and skiplist/twoskip databases get
a lot of random writes - though both repack occasionally which reduces
fragmentation.

I've tried other filesystems a few times, and I keep coming back to ext4
for consistent performance and crash safety (at least on decent hardware
which doesn't lie about fsync) - XFS has lost data, btrfs has always
been shiny at first, but turned into a performance disaster over time.

--
Bron Gondwana
br...@fastmail.fm

Re: LinkChecker

2015-06-16 Thread Chris Davies 
On Wed, 17 Jun 2015, at 02:38 AM, Nic Bernstein wrote:
> Chris, My apologies, I phrased my question wrong.  I meant to ask
> "is this
checking http://www.cyrusimap.org/ only, or also
https://*.cyrus.foundation/ ..."?
>
> I understand that the old site is still putatively the "main"
customer facing site, we are developing
https://docs.cyrus.foundation/ so it would be nice to know if we're
throwing bogons up there. I've just created a second job that will
run once a day and check docs.cyrus.foundation for broken links.
It's called Project LinkChecker - docs.cyrus.foundation[1].

>
> BTW: is there a good way to ignore the favicon.ico errors?
Done. All future scans will ignore this.

Both jobs run once every 24 hours. Let me know if you need a Jenkins
login to trigger them more frequently or if you want me to add any more
sites/exceptions.

Cheers, Chris


Links:

  1. https://ci.cyrus.foundation/job/LinkChecker%20-%20docs.cyrus.foundation/