Re: Cyrus IMAP 2.4.14 released
On Wed, Mar 14, 2012 at 07:50:26PM +0100, Carlos Velasco wrote: > Looking into Imap util.c code I saw that names are stopped at a "." for > the hash function and this is not in the rehash utility, so I patched a > bit again the tool and now it works as expected. Great - thanks! I'm going to rewrite tools/rehash I think - it's pretty awful perl. Amongst other things, we need a unit test for the hash algorithm in BOTH the C and the Perl code. Honestly, the whole 32/64 bit thing shits me. The algorithm should never have been type size dependent. I really REALLY don't want to support both - I think the answer will be a fallback check for the 64 bit hash location if the 32 bit hash location doesn't exist, and then move the file(s). Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Cyrus IMAP 2.4.14 released
Looking into Imap util.c code I saw that names are stopped at a "." for
the hash function and this is not in the rehash utility, so I patched a
bit again the tool and now it works as expected.
Here is my patch:
--- cyrus-imapd-2.4.14/tools/rehash 2012-03-12 12:47:51.0 +0100
+++ cyrus-imapd-2.4.14.patched/tools/rehash 2012-03-14
19:41:51.076296523 +0100
@@ -127,10 +127,12 @@
my $name = shift;
my $uint32 = (2**32);
my ($h, $n);
+my $stname;
if ($tofull) {
+ ($stname) = split(/\./, $name);
$n = 0;
- foreach my $b (split(/ */, $name)) {
+ foreach my $b (split(//, $stname)) {
$n = ((($n << 3) ^ ($n >> 5)) % $uint32) ^ ord($b);
}
$h = chr(ord('A') + ($n % 23));
@@ -146,10 +148,12 @@
sub dir_hash_c_64 {
my $name = shift;
my ($h, $n);
+my $stname;
if ($tofull) {
+ ($stname) = split(/\./, $name);
$n = Math::BigInt->new(0);
- foreach my $b (split(/ */, $name)) {
+ foreach my $b (split(//, $stname)) {
$n = (($n << 3) ^ ($n >> 5)) ^ ord($b);
}
$h = chr(ord('A') + ($n % 23));
@@ -194,7 +198,7 @@
$sievedir = $1;
print "you are using $sievedir as your sieve directory.\n";
}
-if (/^virtdomains:\s+(1|t|yes|on)/) {
+if (/^virtdomains:\s+(1|t|yes|on|userid)/) {
$virtdomains = 1;
print "i will deal with virtual domains.\n";
}
@@ -344,8 +348,8 @@
}
unless ($tonone) {
foreach $i (@{$old}) {
- if (opendir SUB, $i) {
- while ($s = readdir SUB) {
+ if (opendir SUBMVDM, $i) {
+ while ($s = readdir SUBMVDM) {
if ($s =~ /^\./s) { next; }
chdir "$i/$s";
@@ -372,7 +376,7 @@
rename ("$h_32/$s", "$h_64/$s") or ouch
"couldn't move $s back!";
}
}
- closedir SUB;
+ closedir SUBMVDM;
rmdir "$i" or die "couldn't remove $i";
}
}
Original Message
Subject: Re: Cyrus IMAP 2.4.14 released
From: Carlos Velasco
To: info-cyrus@lists.andrew.cmu.edu
Cc: cyrus-de...@lists.andrew.cmu.edu
Date: Wed Mar 14 2012 19:13:43 GMT+0100
> In a Linux LFS Pure 64 (no 32 libs) upgrading from 2.4.13 to 2.4.14 with
> fulldirhash: 1 and hashimapspool: 1 trigger the "System I/O error".
>
> If I am right, previous to the long long patch this system was hashed in
> 32 and now in 2.4.14 it is looking for indexes in hash64.
>
> So I tried to use rehash to convert to basic and then to full to
> recreate in hash64.
> First I had to patch the rehash tool a bit as it was lacking virtdomains
> switch "userid" and also there was a reutilization of the SUB opendir
> handle leading to nasty "invalid dirhandle" errors.
>
> But the real problem is that hashing of "rehash" utility does not match
> the used in cyrus imap.
>
> Rehash:
> domain moving k/kaka.com to N/kaka.com
> moving n/newipnet.com to T/newipnet.com
> moving n/nimastelecom.com to G/nimastelecom.com
> moving p/prueba.com to T/prueba.com
>
> Cyrus imap 2.4.14:
> open("/imap/part1/domain/S/nimastelecom.com/C/user/carlos^velasco/cyrus.index",
> O_RDWR) = -1 ENOENT (No such file or directory)
> open("/imap/part1/domain/U/newipnet.com/C/user/carlos^velasco/cyrus.index",
> O_RDWR) = -1 ENOENT (No such file or directory)
>
> Using the formula in rehash utility:
> ===
> require Math::BigInt;
>
> my $name = "newipnet.com";
>
> $n = Math::BigInt->new(0);
> foreach my $b (split(/ */, $name)) {
> $n = (($n << 3) ^ ($n >> 5)) ^ ord($b);
> }
> $h = chr(ord('A') + ($n % 23));
> print $h;
> ===
> Output: T
>
> But Cyrus Imap 2.4.14 is trying to get it into U in my case.
>
> So, I think rehash utility is broken really.
>
>
> --- cyrus-imapd-2.4.14/tools/rehash 2012-03-12 12:47:51.0 +0100
> +++ cyrus-imapd-2.4.14.patched/tools/rehash 2012-03-14
> 18:59:20.001558956 +0100
> @@ -194,7 +194,7 @@
> $sievedir = $1;
> print "you are using $sievedir as your sieve directory.\n";
> }
> -if (/^virtdomains:\s+(1|t|yes|on)/) {
> +if (/^virtdomains:\s+(1|t|yes|on|userid)/) {
> $virtdomains = 1;
> print "i will deal with virtual domains.\n";
> }
> @@ -344,8 +344,8 @@
> }
> unless ($tonone) {
> foreach $i (@{$old}) {
> - if (opendir SUB, $i) {
> - while
Re: Cyrus IMAP 2.4.14 released
In a Linux LFS Pure 64 (no 32 libs) upgrading from 2.4.13 to 2.4.14 with
fulldirhash: 1 and hashimapspool: 1 trigger the "System I/O error".
If I am right, previous to the long long patch this system was hashed in
32 and now in 2.4.14 it is looking for indexes in hash64.
So I tried to use rehash to convert to basic and then to full to
recreate in hash64.
First I had to patch the rehash tool a bit as it was lacking virtdomains
switch "userid" and also there was a reutilization of the SUB opendir
handle leading to nasty "invalid dirhandle" errors.
But the real problem is that hashing of "rehash" utility does not match
the used in cyrus imap.
Rehash:
domain moving k/kaka.com to N/kaka.com
moving n/newipnet.com to T/newipnet.com
moving n/nimastelecom.com to G/nimastelecom.com
moving p/prueba.com to T/prueba.com
Cyrus imap 2.4.14:
open("/imap/part1/domain/S/nimastelecom.com/C/user/carlos^velasco/cyrus.index",
O_RDWR) = -1 ENOENT (No such file or directory)
open("/imap/part1/domain/U/newipnet.com/C/user/carlos^velasco/cyrus.index",
O_RDWR) = -1 ENOENT (No such file or directory)
Using the formula in rehash utility:
===
require Math::BigInt;
my $name = "newipnet.com";
$n = Math::BigInt->new(0);
foreach my $b (split(/ */, $name)) {
$n = (($n << 3) ^ ($n >> 5)) ^ ord($b);
}
$h = chr(ord('A') + ($n % 23));
print $h;
===
Output: T
But Cyrus Imap 2.4.14 is trying to get it into U in my case.
So, I think rehash utility is broken really.
--- cyrus-imapd-2.4.14/tools/rehash 2012-03-12 12:47:51.0 +0100
+++ cyrus-imapd-2.4.14.patched/tools/rehash 2012-03-14
18:59:20.001558956 +0100
@@ -194,7 +194,7 @@
$sievedir = $1;
print "you are using $sievedir as your sieve directory.\n";
}
-if (/^virtdomains:\s+(1|t|yes|on)/) {
+if (/^virtdomains:\s+(1|t|yes|on|userid)/) {
$virtdomains = 1;
print "i will deal with virtual domains.\n";
}
@@ -344,8 +344,8 @@
}
unless ($tonone) {
foreach $i (@{$old}) {
- if (opendir SUB, $i) {
- while ($s = readdir SUB) {
+ if (opendir SUBMVDM, $i) {
+ while ($s = readdir SUBMVDM) {
if ($s =~ /^\./s) { next; }
chdir "$i/$s";
@@ -372,7 +372,7 @@
rename ("$h_32/$s", "$h_64/$s") or ouch
"couldn't move $s back!";
}
}
- closedir SUB;
+ closedir SUBMVDM;
rmdir "$i" or die "couldn't remove $i";
}
}
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Cyrus IMAP 2.4.14 released
On Tue, 13 Mar 2012, Bron Gondwana wrote: > On Tue, Mar 13, 2012 at 10:31:08AM +0200, Leena Heino wrote: > > I think making fulldirhash to use 64bit by default is a right thing > > to do. But if we want a really fast fix then maybe the fulldirhash > > 64bit change should be backed out. > > Personally I think throwing fulldirhash out with the trash is the > right thing to do. We use directory hashing at FM, though I'm > tempted to think it's no longer required what with ext3's 32k > folder limit gone away now. > > I can't see why a 32 bit hash isn't big enough though - it's no > less random, and it's not expensive to calculate. > > > In order to make it really work in both pure or mixed 32bit and > > 64bit environments the fulldirhash code in cyrus should look for the > > data in both places. The tools/rehash script does that already. > > I think looking multiple places is the pragmatic fix. I'd rather it become pluggable, and you could just tell it to use basic, full32 or full64 hash modes. No weird IO penalties and decreased performance that way. -- "One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie." -- The Silicon Valley Tarot Henrique Holschuh Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Cyrus IMAP 2.4.14 released
On Tue, 13 Mar 2012, Bron Gondwana wrote: > On Tue, Mar 13, 2012 at 10:31:08AM +0200, Leena Heino wrote: >> I think making fulldirhash to use 64bit by default is a right thing >> to do. But if we want a really fast fix then maybe the fulldirhash >> 64bit change should be backed out. > > Personally I think throwing fulldirhash out with the trash is the > right thing to do. We use directory hashing at FM, though I'm > tempted to think it's no longer required what with ext3's 32k > folder limit gone away now. We have to use fulldirhash because this way we are able to take backups more efficiently. The way Networker assings it resources makes it more efficient to use multiple directories that each have more or less the same amount of data and files. > I can't see why a 32 bit hash isn't big enough though - it's no > less random, and it's not expensive to calculate. It is more about making the system work in a predictable way regardless the size of int in various tools and systems. The size of 32bit hash and the randomness it gives is most likely more than enough. >> In order to make it really work in both pure or mixed 32bit and >> 64bit environments the fulldirhash code in cyrus should look for the >> data in both places. The tools/rehash script does that already. > > I think looking multiple places is the pragmatic fix. I Agree. -- Leena Heino University of Tampere / Computer Centre ( liinu at uta.fi ) ( http://www.uta.fi/laitokset/tkk ) Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Cyrus IMAP 2.4.14 released
On 03/13/2012 03:00 AM, Leena Heino wrote:
On 13.3.2012 3:59, Patrick Boutilier wrote:
I just updated my replication server from 2.4.12 to 2.4.14 and then got
nothing but "System I/O error" errors in the logs. I took a look at
http://bugzilla.cyrusimap.org/show_bug.cgi?id=3381 and noticed this:
You can convert your mail directory hashing scheme from 32bit
fulldirhash to
64bit fulldirhash with following commands:
% tools/rehash basic
% tools/rehash full
Unfortunately I have almost 5TB of mail spread out on 23 different mount
points. All the letters in /imap/mail are individual mount points so
rehashing would mean lots of moves across the mount points which won't
be quick.
Is there a way to force Cyrus to use the old 32bit way, other than just
removing the patch?
Unfortunately there does not seem to be way to force either 32bit or
64bit fulldirhash.
You have to remove this patch from 2.4.14:
diff --git a/lib/util.c b/lib/util.c
index 8bcb00e..c96caa5 100644
--- a/lib/util.c
+++ b/lib/util.c
@@ -269,7 +269,7 @@ int dir_hash_c(const char *name, int full)
if (full) {
unsigned char *pt;
- unsigned int n;
+ unsigned long long int n;
enum {
DIR_X = 3,
DIR_Y = 5,
Removed the patch last night and all was fine.
<>
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Cyrus IMAP 2.4.14 released
On Tue, Mar 13, 2012 at 10:31:08AM +0200, Leena Heino wrote: > I think making fulldirhash to use 64bit by default is a right thing > to do. But if we want a really fast fix then maybe the fulldirhash > 64bit change should be backed out. Personally I think throwing fulldirhash out with the trash is the right thing to do. We use directory hashing at FM, though I'm tempted to think it's no longer required what with ext3's 32k folder limit gone away now. I can't see why a 32 bit hash isn't big enough though - it's no less random, and it's not expensive to calculate. > In order to make it really work in both pure or mixed 32bit and > 64bit environments the fulldirhash code in cyrus should look for the > data in both places. The tools/rehash script does that already. I think looking multiple places is the pragmatic fix. Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Cyrus IMAP 2.4.14 released
On Tue, 13 Mar 2012, Bron Gondwana wrote: > Leena - should we back this out and get 2.4.15 out fast (with a > changes.html!) and make the fulldirhash always be 32 bit hashing > by changing the perl that way instead? > > Tell you the truth, I'm super-tempted to just change the code to > look in BOTH locations always if you have fulldirhash. I think making fulldirhash to use 64bit by default is a right thing to do. But if we want a really fast fix then maybe the fulldirhash 64bit change should be backed out. In order to make it really work in both pure or mixed 32bit and 64bit environments the fulldirhash code in cyrus should look for the data in both places. The tools/rehash script does that already. -- Leena Heino University of Tampere / Computer Centre ( liinu at uta.fi ) ( http://www.uta.fi/laitokset/tkk ) Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Cyrus IMAP 2.4.14 released
On Tue, Mar 13, 2012 at 08:00:58AM +0200, Leena Heino wrote: > Unfortunately there does not seem to be way to force either 32bit or > 64bit fulldirhash. Oh man. I knew there was a reason why we don't run fulldirhash! That sucks. Leena - should we back this out and get 2.4.15 out fast (with a changes.html!) and make the fulldirhash always be 32 bit hashing by changing the perl that way instead? Tell you the truth, I'm super-tempted to just change the code to look in BOTH locations always if you have fulldirhash. Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Cyrus IMAP 2.4.14 released
On 13.3.2012 3:59, Patrick Boutilier wrote:
> I just updated my replication server from 2.4.12 to 2.4.14 and then got
> nothing but "System I/O error" errors in the logs. I took a look at
> http://bugzilla.cyrusimap.org/show_bug.cgi?id=3381 and noticed this:
>
> You can convert your mail directory hashing scheme from 32bit
> fulldirhash to
> 64bit fulldirhash with following commands:
> % tools/rehash basic
> % tools/rehash full
>
>
> Unfortunately I have almost 5TB of mail spread out on 23 different mount
> points. All the letters in /imap/mail are individual mount points so
> rehashing would mean lots of moves across the mount points which won't
> be quick.
>
> Is there a way to force Cyrus to use the old 32bit way, other than just
> removing the patch?
Unfortunately there does not seem to be way to force either 32bit or
64bit fulldirhash.
You have to remove this patch from 2.4.14:
diff --git a/lib/util.c b/lib/util.c
index 8bcb00e..c96caa5 100644
--- a/lib/util.c
+++ b/lib/util.c
@@ -269,7 +269,7 @@ int dir_hash_c(const char *name, int full)
if (full) {
unsigned char *pt;
- unsigned int n;
+ unsigned long long int n;
enum {
DIR_X = 3,
DIR_Y = 5,
--
Leena Heino University of Tampere / Computer Centre
( liinu at uta.fi ) ( http://www.uta.fi/laitokset/tkk )
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Cyrus IMAP 2.4.14 released
On 03/12/2012 09:22 AM, Jeroen van Meeuwen (Kolab Systems) wrote: We are pleased to announce the release of Cyrus IMAPd 2.4.14. This is a stable release in the 2.4.x series. The release mainly contains bug fixes, mostly small but significant. Please find an overview of all bugs resolved in this release at: http://www.cyrusimap.org/mediawiki/index.php/Bugs_Resolved_in_2.4.14 You can download via HTTP or FTP: http://cyrusimap.org/releases/cyrus-imapd-2.4.14.tar.gz ftp://ftp.cyrusimap.org/cyrus-imapd/cyrus-imapd-2.4.14.tar.gz Kind regards, Jeroen van Meeuwen Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ I just updated my replication server from 2.4.12 to 2.4.14 and then got nothing but "System I/O error" errors in the logs. I took a look at http://bugzilla.cyrusimap.org/show_bug.cgi?id=3381 and noticed this: You can convert your mail directory hashing scheme from 32bit fulldirhash to 64bit fulldirhash with following commands: % tools/rehash basic % tools/rehash full Unfortunately I have almost 5TB of mail spread out on 23 different mount points. All the letters in /imap/mail are individual mount points so rehashing would mean lots of moves across the mount points which won't be quick. Is there a way to force Cyrus to use the old 32bit way, other than just removing the patch? <> Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Cyrus IMAP 2.4.14 released
Hi, On Mon, 12 Mar 2012 22:01:48 +0900, Jeroen van Meeuwen (Kolab Systems) wrote: > On 2012-03-12 12:56, OBATA Akio wrote: >> doc/changes.html and doc/text/changes in released tarball are not >> updated to 2.4.14? >> (I don't know about other document files) >> > > Hi, > > You are right, these files have not been updated with 2.4.14 release > information - we're missing a piece in our release process. These docs > have not been updated for 2.4.13 either, now that I examine them a > little closer. I feel it is good for 2.4.13, because it contains "Changes to the Cyrus IMAP Server since 2.4.12". -- OBATA Akio / ob...@lins.jp Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Cyrus IMAP 2.4.14 released
On 2012-03-12 12:56, OBATA Akio wrote: > doc/changes.html and doc/text/changes in released tarball are not > updated to 2.4.14? > (I don't know about other document files) > Hi, You are right, these files have not been updated with 2.4.14 release information - we're missing a piece in our release process. These docs have not been updated for 2.4.13 either, now that I examine them a little closer. The information normally contained within those pages is available though; - http://www.cyrusimap.org/mediawiki/index.php/Bugs_Resolved_in_2.4.14 - http://www.cyrusimap.org/mediawiki/index.php/Bugs_Resolved_in_2.4.13 Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08 Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Cyrus IMAP 2.4.14 released
doc/changes.html and doc/text/changes in released tarball are not updated to 2.4.14? (I don't know about other document files) On Mon, 12 Mar 2012 21:22:37 +0900, Jeroen van Meeuwen (Kolab Systems) wrote: > We are pleased to announce the release of Cyrus IMAPd 2.4.14. > > This is a stable release in the 2.4.x series. The release mainly contains bug > fixes, mostly small but significant. > > Please find an overview of all bugs resolved in this release at: > > http://www.cyrusimap.org/mediawiki/index.php/Bugs_Resolved_in_2.4.14 > > You can download via HTTP or FTP: > > http://cyrusimap.org/releases/cyrus-imapd-2.4.14.tar.gz > > ftp://ftp.cyrusimap.org/cyrus-imapd/cyrus-imapd-2.4.14.tar.gz > > Kind regards, > > Jeroen van Meeuwen > -- OBATA Akio / ob...@lins.jp Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Cyrus IMAP 2.4.14 released
We are pleased to announce the release of Cyrus IMAPd 2.4.14. This is a stable release in the 2.4.x series. The release mainly contains bug fixes, mostly small but significant. Please find an overview of all bugs resolved in this release at: http://www.cyrusimap.org/mediawiki/index.php/Bugs_Resolved_in_2.4.14 You can download via HTTP or FTP: http://cyrusimap.org/releases/cyrus-imapd-2.4.14.tar.gz ftp://ftp.cyrusimap.org/cyrus-imapd/cyrus-imapd-2.4.14.tar.gz Kind regards, Jeroen van Meeuwen -- Systems Architect, Kolab Systems AG e: vanmeeuwen at kolabsys.com m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08 signature.asc Description: This is a digitally signed message part. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
