from:"Egoitz Aurrekoetxea"

A ti Javier :)



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia)
ego...@sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electrónico piense si es necesario hacerlo.

> El 18 jul 2019, a las 16:27, Infraestructura TIC - UNNOBA 
>  escribió:
> 
> Muchísimas gracias, Egoitz!
> 
> 
> El 18/7/19 a las 11:12, Egoitz Aurrekoetxea escribió:
>> Hi!!,
>> 
>> Fine! Very happy sharing then :) :) . It only handles email. For 
>> Calendars/Contacts we have been long time now, using Davical (to which we 
>> contributed in it’s day https://wiki.davical.org/index.php/DAViCal-cli 
>> <https://wiki.davical.org/index.php/DAViCal-cli>) . We don’t refuse to use 
>> Caldav with Cyrus, it’s just we did the system previous to Cyrus Caldav 
>> system.
>> 
>> I attach the code in this email. I explain how we use it. We have each 
>> mailbox server running this code as a cron job and we have too some servers 
>> with Cyrus IMAP for just storing removed content (without the cron 
>> obviously). Each user in the restore server (a normal mailbox server but 
>> just for storing deleted content) is something like : 
>> user_dom...@recuperaciones.saremail.com 
>> <mailto:user_dom...@recuperaciones.saremail.com>. All our servers have 
>> autocreate feature (although in our mailbox servers is not being used 
>> nowadays). So, we keep track of what has been removed in a mailbox server 
>> with two elements… the Cyrus log and cyradm command. With cyradm command we 
>> keep track of deleted “folders" in each user account. With the log, we know 
>> where expunges had been run. Later, we take the DELETED mailboxes (the 
>> folders of each user) and upload them to Saremail-Restore. After that, we 
>> check the log (from some hours before till the present moment). Then we ask 
>> unexpunge to see what has been removed in each place. We upload them. We 
>> keep track in a database of what exactly has dealed with and what is 
>> remaining to deal with, so in the case a fail over to a slave is produced 
>> unexpunges can then be run there, even if there’s nothing in the logs that 
>> say that (because it’s obviously a slave).
>> 
>> 
>> If you think it could be useful, perhaps could be uploaded to contrib 
>> directory…
>> 
>> Cheers!
> 
> -- 
> Lic. Javier Charne
> Responsable Infraestructura Tecnológica
> Prosecretaría de TIC | UNNOBA
> Junín, Buenos Aires, Argentina
> jav...@unnoba.edu.ar <mailto:jav...@unnoba.edu.ar>
> Tel: +54 (0236) 4407750 int 11712
> Cel: +5492364542182
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Folder subscription issue

Hi!!,Fine! Very happy sharing then :) :) . It only handles email. For Calendars/Contacts we have been long time now, using Davical (to which we contributed in it’s day https://wiki.davical.org/index.php/DAViCal-cli) . We don’t refuse to use Caldav with Cyrus, it’s just we did the system previous to Cyrus Caldav system.I attach the code in this email. I explain how we use it. We have each mailbox server running this code as a cron job and we have too some servers with Cyrus IMAP for just storing removed content (without the cron obviously). Each user in the restore server (a normal mailbox server but just for storing deleted content) is something like : user_dom...@recuperaciones.saremail.com. All our servers have autocreate feature (although in our mailbox servers is not being used nowadays). So, we keep track of what has been removed in a mailbox server with two elements… the Cyrus log and cyradm command. With cyradm command we keep track of deleted “folders" in each user account. With the log, we know where expunges had been run. Later, we take the DELETED mailboxes (the folders of each user) and upload them to Saremail-Restore. After that, we check the log (from some hours before till the present moment). Then we ask unexpunge to see what has been removed in each place. We upload them. We keep track in a database of what exactly has dealed with and what is remaining to deal with, so in the case a fail over to a slave is produced unexpunges can then be run there, even if there’s nothing in the logs that say that (because it’s obviously a slave).If you think it could be useful, perhaps could be uploaded to contrib directory…Cheers!<>

Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470Parque Tecnológico. Edificio 10348170 Zamudio (Bizkaia)
ego...@sarenet.es
www.sarenet.es

Antes de imprimir este correo electrónico piense si es necesario hacerlo.

El 18 jul 2019, a las 14:09, Bron Gondwana <br...@fastmail.fm> escribió:Awesome, I'll take a look. We're just talking about having some level of self-service restore in JMAP as well, so it'll be great to look at what you've done.Cheers,Bron.On Thu, Jul 18, 2019, at 20:30, Egoitz Aurrekoetxea wrote:Thanks :) :)Take a look please at what I told you too in a new thread about the contribution with our deleted mail restoring system… perhaps it’s interesting for Cyrus?. For us it’s pretty useful at least… it allows users to restore deleted mail by their own…Cheers!! Egoitz AurrekoetxeaDpto. de sistemas944 209 470Parque Tecnológico. Edificio 10348170 Zamudio (Bizkaia)ego...@sarenet.eswww.sarenet.esAntes de imprimir este correo electrónico piense si es necesario hacerlo.El 18 jul 2019, a las 12:22, Bron Gondwana <br...@fastmail.fm> escribió:Hmmm so sync_client isn't validating the user match properly. Fun. That's definitely a bug that we should fix.I'll ask ellie (CC'd) to play with the test on that, or guide you through the process of adding one to Cassandane!Cheers,Bron.On Thu, Jul 18, 2019, at 17:22, Egoitz Aurrekoetxea wrote:Hi!,Yes, I’m so sorry for the noise… but you know…. It’s an un modified script in more than 8 years, not done by me… which you assume it works… because indeed, it copies mailbox content (sync_client -u aaa^a...@bbb.es seems to copy mailbox content as aaa.a...@bbb.es but not subs) but not “properly” the subs, sieve… Basically taking a look at some stored consoles (ssh sessions) of the migration…. I noted that were in fact those USER ___^@ so I did… I’m doing it by me!! Then looked at that script and saw….Sorry for the time wasted Ellie.. same Bron,Cheers!Egoitz AurrekoetxeaDpto. de sistemas944 209 470Parque Tecnológico. Edificio 10348170 Zamudio (Bizkaia)ego...@sarenet.eswww.sarenet.esAntes de imprimir este correo electrónico piense si es necesario hacerlo.El 17 jul 2019, a las 23:58, Bron Gondwana <br...@fastmailteam.com> escribió:Oh awesome - thanks for letting us know :) This will save ellie spending more time on it, though it's good to have tests for it anyway.Bron.On Wed, Jul 17, 2019, at 23:56, Egoitz Aurrekoetxea wrote:Hi mates,Reproduced and properly located (not a Cyrus bug). We have to apologize for the generated noise. Cyrus replication works just fine. As I explained before, we fetch from Mysql the accounts each server has (instead of using -A) for no reason… just for historical reason… So, when we fetched the emails, as in very old (now) versions of Cyrus a conversion from . to ^ seem to be needed (the first Cyrus version was 2.0 o even older), we were doing the following in the Perl script :while (@fila = $query->fetchrow()) { $fila[0] =~ s/\./^/g; $email = $fila[0]."@".$fila[1]; system("/usr/local/bin/sudo -u cyrus /usr/local/cyrus/sbin/sync_client -S xx.sarenet.es -v -u ".$email);This explained the existence of .seen and .sub files with “^” and Sieve dirs with ‘^’ too. That didn’t in fact explain, why they had non

Re: Folder subscription issue

Thanks :) :)

Take a look please at what I told you too in a new thread about the 
contribution with our deleted mail restoring system… perhaps it’s interesting 
for Cyrus?. For us it’s pretty useful at least… it allows users to restore 
deleted mail by their own…

Cheers!! 



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia)
ego...@sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electrónico piense si es necesario hacerlo.

> El 18 jul 2019, a las 12:22, Bron Gondwana  escribió:
> 
> Hmmm so sync_client isn't validating the user match properly.  Fun.  
> That's definitely a bug that we should fix.
> 
> I'll ask ellie (CC'd) to play with the test on that, or guide you through the 
> process of adding one to Cassandane!
> 
> Cheers,
> 
> Bron.
> 
> On Thu, Jul 18, 2019, at 17:22, Egoitz Aurrekoetxea wrote:
>> Hi!,
>> 
>> Yes, I’m so sorry for the noise… but you know…. It’s an un modified script 
>> in more than 8 years, not done by me… which you assume it works… because 
>> indeed, it copies mailbox content (sync_client -u aaa^a...@bbb.es 
>> <mailto:aaa%5ea...@bbb.es> seems to copy mailbox content as aaa.a...@bbb.es 
>> <mailto:aaa.a...@bbb.es> but not subs) but not “properly” the subs, sieve… 
>> Basically taking a look at some stored consoles (ssh sessions) of the 
>> migration…. I noted that were in fact those USER ___^@ so I did… I’m 
>> doing it by me!! Then looked at that script and saw….
>> 
>> Sorry for the time wasted Ellie.. same Bron,
>> 
>> Cheers!
>> 
>> 
>> Egoitz Aurrekoetxea
>> Dpto. de sistemas
>> 944 209 470
>> Parque Tecnológico. Edificio 103
>> 48170 Zamudio (Bizkaia)
>> ego...@sarenet.es
>> www.sarenet.es <http://www.sarenet.es/>
>> 
>> Antes de imprimir este correo electrónico piense si es necesario hacerlo.
>> 
>>> El 17 jul 2019, a las 23:58, Bron Gondwana >> <mailto:br...@fastmailteam.com>> escribió:
>>> 
>>> Oh awesome - thanks for letting us know :)  This will save ellie spending 
>>> more time on it, though it's good to have tests for it anyway.
>>> 
>>> Bron.
>>> 
>>> On Wed, Jul 17, 2019, at 23:56, Egoitz Aurrekoetxea wrote:
>>>> Hi mates,
>>>> 
>>>> Reproduced and properly located (not a Cyrus bug). We have to apologize 
>>>> for the generated noise. Cyrus replication works just fine. 
>>>> 
>>>> As I explained before, we fetch from Mysql the accounts each server has 
>>>> (instead of using -A) for no reason… just for historical reason… 
>>>> 
>>>> So, when we fetched the emails, as in very old (now) versions of Cyrus a 
>>>> conversion from . to ^ seem to be needed (the first Cyrus version was 2.0 
>>>> o even older), we were doing the following in the Perl script :
>>>> 
>>>> while (@fila = $query->fetchrow()) {
>>>> $fila[0] =~ s/\./^/g;
>>>> $email = $fila[0]."@".$fila[1];
>>>> system("/usr/local/bin/sudo -u cyrus 
>>>> /usr/local/cyrus/sbin/sync_client -S xx.sarenet.es 
>>>> <http://xx.sarenet.es/> -v -u ".$email);
>>>> 
>>>> This explained the existence of .seen and .sub files with “^” and Sieve 
>>>> dirs with ‘^’ too. That didn’t in fact explain, why they had non zero 
>>>> size. The reason of the non zero size (of files with ^ and ending in .sub 
>>>> and .seen and Sieve dirs), is that as we come from Cyrus versions 2.3 (and 
>>>> probably 2.4 perhaps do it) used the ‘^’ in the seen and sub filenames and 
>>>> Sieve names. So if you do a sync_client of a user with ‘^’ , even in 3.0 
>>>> it copies you that file content of .sub and .seen from the master to the 
>>>> slave (because it exists! because it has copied with sync_client too but 
>>>> from a 2.3/2.4 version), even when they have not been used in a 3.0 
>>>> version. Apart the own mail account content is always copied and 
>>>> sync_client doesn’t complain about it (whether you use ^ or a dot for the 
>>>> user).
>>>> 
>>>> So mates, thanks a lot again for your time and I honestly think I’ll sleep 
>>>> better today :)
>>>> 
>>>> We have just discovered these, so we wanted to clarify it.
>>>> 
>>>> Thanks again,
>>>> Regards,
>>>> 
>>>>

Re: Issues with replication and folder/Sieve subscription

That thread is clarified. Was an issue from a script from Sarenet…. It has been 
hard to find (as described in the new thread) but Cyrus was just fine.



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia)
ego...@sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electrónico piense si es necesario hacerlo.

> El 10 jul 2019, a las 10:03, Egoitz Aurrekoetxea  escribió:
> 
> The subject of this email is not properly set… it should be . Issues in 
> replication with folder subscription and Sieve
> 
> As I think I discovered something I reopen a new thread with the title 
> properly
> 
>  
> 
> 
> 
> Egoitz Aurrekoetxea
> Dpto. de sistemas
> 944 209 470
> Parque Tecnológico. Edificio 103
> 48170 Zamudio (Bizkaia)
> ego...@sarenet.es <mailto:undefined>
> www.sarenet.es <http://www.sarenet.es/>
> Antes de imprimir este correo electrónico piense si es necesario hacerlo.
> 
>> El 10 jul 2019, a las 9:22, Albert Shih > <mailto:albert.s...@obspm.fr>> escribió:
>> 
>> Le 09/07/2019 à 22:49:01+0200, Egoitz Aurrekoetxea a écrit
>>> By the way, for your case I would recommend doing a script that does a get 
>>> from
>>> dovecot and a put to Cyrus instead of copying Sieve files directly… it’s a 
>>> much
>>> more cleaner way…
>> 
>> Yes, it is what I did, before I try de sync I event do a
>> /usr/local/cyrsus/sievec on each file to by absolute sure the sieve file
>> compile correctly
>> 
>> Regards.
>> 
>> 
>> --
>> Albert SHIH
>> DIO bâtiment 15
>> Observatoire de Paris
>> xmpp: j...@obspm.fr <mailto:j...@obspm.fr>
>> Heure local/Local time:
>> Wed 10 Jul 2019 09:20:09 AM CEST
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ 
> <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus 
> <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Suggested feature and contribution

Good morning,

When using delete_delayed if someone removes a big folder (that one with more 
than 20 subfolders anywhere below it) in mboxlist_delayed_deletemailbox() only 
last 20 are preserved. We think it could be a good idea to preserve all and to 
have a parameter for configuring it. The reason for that, is that we use 
delete_delayed for storing the removed content remotely with the customer hired 
retention period in slow disk space. Perhaps could be a good idea something 
like : 

In mboxlist_delayed_deletemailbox() : 

If (!preserve_delete_delayed_folders_always)
{
/* keep the last 19, so the new one is the 20th */
for (i = 0; i < (int)existing.count - 19; i++) {
const char *subname = strarray_nth(, i);
syslog(LOG_NOTICE, "too many subfolders for %s, deleting %s (%d / %d)",
   newname, subname, i+1, (int)existing.count);
r = mboxlist_deletemailbox(subname, 1, userid, auth_state, NULL, 0, 1, 
1,
   keep_intermediaries);
if (r) goto done;
}
}

In Cyrus imapd.conf : 

preserve_delete_delayed_folders_always: true or false

Obviously, the part of code implied in reading the config file for 
contemplating the possibility….

At Sarenet, we have done some scripts for managing the copy of removed content 
(using delete delayed) and uploading it to a remote imap server. We are using 
them successfully some years ago now. Is Cyrus project interested in that 
scripts (that manage removed content) for later allowing the possibility of 
restoring removed elements from another different IMAP store?. I have asked to 
my superiors and would be happy to contribute it to Cyrus project… What do you 
think Bron, Ellie?

Cheers!





Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia)
ego...@sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electrónico piense si es necesario hacerlo.


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Folder subscription issue

Hi!,

Yes, I’m so sorry for the noise… but you know…. It’s an un modified script in 
more than 8 years, not done by me… which you assume it works… because indeed, 
it copies mailbox content (sync_client -u aaa^a...@bbb.es 
<mailto:aaa%5ea...@bbb.es> seems to copy mailbox content as aaa.a...@bbb.es 
<mailto:aaa.a...@bbb.es> but not subs) but not “properly” the subs, sieve… 
Basically taking a look at some stored consoles (ssh sessions) of the 
migration…. I noted that were in fact those USER ___^@ so I did… I’m 
doing it by me!! Then looked at that script and saw….

Sorry for the time wasted Ellie.. same Bron,

Cheers!


Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia)
ego...@sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electrónico piense si es necesario hacerlo.

> El 17 jul 2019, a las 23:58, Bron Gondwana  escribió:
> 
> Oh awesome - thanks for letting us know :)  This will save ellie spending 
> more time on it, though it's good to have tests for it anyway.
> 
> Bron.
> 
> On Wed, Jul 17, 2019, at 23:56, Egoitz Aurrekoetxea wrote:
>> Hi mates,
>> 
>> Reproduced and properly located (not a Cyrus bug). We have to apologize for 
>> the generated noise. Cyrus replication works just fine. 
>> 
>> As I explained before, we fetch from Mysql the accounts each server has 
>> (instead of using -A) for no reason… just for historical reason… 
>> 
>> So, when we fetched the emails, as in very old (now) versions of Cyrus a 
>> conversion from . to ^ seem to be needed (the first Cyrus version was 2.0 o 
>> even older), we were doing the following in the Perl script :
>> 
>> while (@fila = $query->fetchrow()) {
>> $fila[0] =~ s/\./^/g;
>> $email = $fila[0]."@".$fila[1];
>> system("/usr/local/bin/sudo -u cyrus 
>> /usr/local/cyrus/sbin/sync_client -S xx.sarenet.es 
>> <http://xx.sarenet.es/> -v -u ".$email);
>> 
>> This explained the existence of .seen and .sub files with “^” and Sieve dirs 
>> with ‘^’ too. That didn’t in fact explain, why they had non zero size. The 
>> reason of the non zero size (of files with ^ and ending in .sub and .seen 
>> and Sieve dirs), is that as we come from Cyrus versions 2.3 (and probably 
>> 2.4 perhaps do it) used the ‘^’ in the seen and sub filenames and Sieve 
>> names. So if you do a sync_client of a user with ‘^’ , even in 3.0 it copies 
>> you that file content of .sub and .seen from the master to the slave 
>> (because it exists! because it has copied with sync_client too but from a 
>> 2.3/2.4 version), even when they have not been used in a 3.0 version. Apart 
>> the own mail account content is always copied and sync_client doesn’t 
>> complain about it (whether you use ^ or a dot for the user).
>> 
>> So mates, thanks a lot again for your time and I honestly think I’ll sleep 
>> better today :)
>> 
>> We have just discovered these, so we wanted to clarify it.
>> 
>> Thanks again,
>> Regards,
>> 
>> 
>> 
>> Egoitz Aurrekoetxea
>> Dpto. de sistemas
>> 944 209 470
>> Parque Tecnológico. Edificio 103
>> 48170 Zamudio (Bizkaia)
>> ego...@sarenet.es
>> www.sarenet.es <http://www.sarenet.es/>
>> 
>> Antes de imprimir este correo electrónico piense si es necesario hacerlo.
>> 
>>> El 17 jul 2019, a las 12:51, Egoitz Aurrekoetxea >> <mailto:ego...@sarenet.es>> escribió:
>>> 
>>> Anyway, not being able to reproduce it… working on reproducing….
>>> 
>>> 
>>> 
>>> Egoitz Aurrekoetxea
>>> Dpto. de sistemas
>>> 944 209 470
>>> Parque Tecnológico. Edificio 103
>>> 48170 Zamudio (Bizkaia)
>>> ego...@sarenet.es
>>> www.sarenet.es <http://www.sarenet.es/>
>>> 
>>> Antes de imprimir este correo electrónico piense si es necesario hacerlo.
>>> 
>>>> El 16 jul 2019, a las 20:23, Egoitz Aurrekoetxea >>> <mailto:ego...@sarenet.es>> escribió:
>>>> 
>>>> Perhaps mbname_userid it’s not the exact function… perhaps we could just 
>>>> call something like _append_extbuf() in this case but…. That’s what I was 
>>>> trying to explain… could that be?
>>>> 
>>>> 
>>>> 
>>>> Egoitz Aurrekoetxea
>>>> Dpto. de sistemas
>>>> 944 209 470
>>>> Parque Tecnológico. Edificio 103
>>>> 48170 Zamudio (Bizkaia)
>>>> ego...@sarenet.es
>>>

Re: Folder subscription issue

2019-07-17 Thread Egoitz Aurrekoetxea

Hi mates,

Reproduced and properly located (not a Cyrus bug). We have to apologize for the 
generated noise. Cyrus replication works just fine. 

As I explained before, we fetch from Mysql the accounts each server has 
(instead of using -A) for no reason… just for historical reason… 

So, when we fetched the emails, as in very old (now) versions of Cyrus a 
conversion from . to ^ seem to be needed (the first Cyrus version was 2.0 o 
even older), we were doing the following in the Perl script :

while (@fila = $query->fetchrow()) {
$fila[0] =~ s/\./^/g;
$email = $fila[0]."@".$fila[1];
system("/usr/local/bin/sudo -u cyrus /usr/local/cyrus/sbin/sync_client 
-S xx.sarenet.es -v -u ".$email);

This explained the existence of .seen and .sub files with “^” and Sieve dirs 
with ‘^’ too. That didn’t in fact explain, why they had non zero size. The 
reason of the non zero size (of files with ^ and ending in .sub and .seen and 
Sieve dirs), is that as we come from Cyrus versions 2.3 (and probably 2.4 
perhaps do it) used the ‘^’ in the seen and sub filenames and Sieve names. So 
if you do a sync_client of a user with ‘^’ , even in 3.0 it copies you that 
file content of .sub and .seen from the master to the slave (because it exists! 
because it has copied with sync_client too but from a 2.3/2.4 version), even 
when they have not been used in a 3.0 version. Apart the own mail account 
content is always copied and sync_client doesn’t complain about it (whether you 
use ^ or a dot for the user).

So mates, thanks a lot again for your time and I honestly think I’ll sleep 
better today :)

We have just discovered these, so we wanted to clarify it.

Thanks again,
Regards,



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia)
ego...@sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electrónico piense si es necesario hacerlo.

> El 17 jul 2019, a las 12:51, Egoitz Aurrekoetxea  escribió:
> 
> Anyway, not being able to reproduce it… working on reproducing….
> 
> 
> 
> Egoitz Aurrekoetxea
> Dpto. de sistemas
> 944 209 470
> Parque Tecnológico. Edificio 103
> 48170 Zamudio (Bizkaia)
> ego...@sarenet.es <mailto:undefined>
> www.sarenet.es <http://www.sarenet.es/>
> Antes de imprimir este correo electrónico piense si es necesario hacerlo.
> 
>> El 16 jul 2019, a las 20:23, Egoitz Aurrekoetxea > <mailto:ego...@sarenet.es>> escribió:
>> 
>> Perhaps mbname_userid it’s not the exact function… perhaps we could just 
>> call something like _append_extbuf() in this case but…. That’s what I was 
>> trying to explain… could that be?
>> 
>> 
>> 
>> Egoitz Aurrekoetxea
>> Dpto. de sistemas
>> 944 209 470
>> Parque Tecnológico. Edificio 103
>> 48170 Zamudio (Bizkaia)
>> ego...@sarenet.es <mailto:undefined>
>> www.sarenet.es <http://www.sarenet.es/>
>> Antes de imprimir este correo electrónico piense si es necesario hacerlo.
>> 
>>> El 16 jul 2019, a las 20:20, Egoitz Aurrekoetxea >> <mailto:ego...@sarenet.es>> escribió:
>>> 
>>> mbname_userid
>> 
>> 
>> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ 
>> <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus 
>> <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>
> 
> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ 
> <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus 
> <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Folder subscription issue

2019-07-17 Thread Egoitz Aurrekoetxea

Anyway, not being able to reproduce it… working on reproducing….



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia)
ego...@sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electrónico piense si es necesario hacerlo.

> El 16 jul 2019, a las 20:23, Egoitz Aurrekoetxea  escribió:
> 
> Perhaps mbname_userid it’s not the exact function… perhaps we could just call 
> something like _append_extbuf() in this case but…. That’s what I was trying 
> to explain… could that be?
> 
> 
> 
> Egoitz Aurrekoetxea
> Dpto. de sistemas
> 944 209 470
> Parque Tecnológico. Edificio 103
> 48170 Zamudio (Bizkaia)
> ego...@sarenet.es <mailto:undefined>
> www.sarenet.es <http://www.sarenet.es/>
> Antes de imprimir este correo electrónico piense si es necesario hacerlo.
> 
>> El 16 jul 2019, a las 20:20, Egoitz Aurrekoetxea > <mailto:ego...@sarenet.es>> escribió:
>> 
>> mbname_userid
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ 
> <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus 
> <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Folder subscription issue

Perhaps mbname_userid it’s not the exact function… perhaps we could just call 
something like _append_extbuf() in this case but…. That’s what I was trying to 
explain… could that be?



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia)
ego...@sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electrónico piense si es necesario hacerlo.

> El 16 jul 2019, a las 20:20, Egoitz Aurrekoetxea  escribió:
> 
> mbname_userid


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Folder subscription issue

Hi!

Mmm perhaps I’m telling this excessively sure but… 

In sync_do_user() function, when your scripts syncs user by user instead of 
using MODE_ALLUSER (which perhaps is more normal… but our scripts are done with 
a for loop with each user) shouldn’t the userid be converted : 

   /* we don't hold locks while sending commands */
mailbox_close();
r = do_user_main(userid, topart, replica_folders, replica_quota,
 sync_be, channelp, flags);
if (r) goto done;

  /* COMMENTED CHANGE */
  char *user_without_dot = mbname_userid(userid)
r = sync_do_user_sub(user_without_dot, replica_subs, sync_be, flags);
if (r) goto done;
r = sync_do_user_sieve(user_without_dot, replica_sieve, sync_be, flags);
if (r) goto done;
r = sync_do_user_seen(user_without_dot, replica_seen, sync_be, flags);
  /* END COMMENTED CHANGE */

I got tons of cases like this that could are explained this way. :

It’s like if sync_do_user wan’t giving the proper user for later creating the 
file and so to sync_do_user_sub , sync_do_user_seen… etc etc…

Perhaps I have not reproduced (I realized now, although not tested) this 
because I didn’t create folders in my testing env (the one for reproducing 
this)… not at least this way… 

I think fix this issue…. What do you think about it?. At least that, shouldn’t 
damage… I mean at least the fact of replacing the ^ with a dot… and 
subscripciones would work…. Seen seems to be handled now perhaps in 
conversations database or similar… but subscriptions… the problem I’m seeing is 
this “^” in the file name….

What do you think mates?




Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia)
ego...@sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electrónico piense si es necesario hacerlo.

> El 16 jul 2019, a las 18:41, Egoitz Aurrekoetxea  escribió:
> 
> Hi!
> 
> One question are this records in the mailbox database (seen with a 
> ctl_mboxlist -d) normal ?
> 
> xx.ccc!user.ama.COMUNICACIONES CIAS-OTROS.COMUNICACIONES CIAS 2017. 
> NOVIEMBRE 2017 16 (null) 
> xx.ccc!user.ama.COMUNICACIONES CIAS-OTROS.COMUNICACIONES CIAS 2017. 
> OCTUBRE 2017   16 (null) 
> xx.ccc!user.ama.COMUNICACIONES CIAS-OTROS.COMUNICACIONES CIAS 
> 2017.AGOSTO 2017 16 (null) 
> xx.ccc!user.ama.COMUNICACIONES CIAS-OTROS.COMUNICACIONES CIAS 
> 2017.DICIEMBRE 2017  16 (null) 
> xx.ccc!user.ama.COMUNICACIONES CIAS-OTROS.COMUNICACIONES CIAS 
> 2017.SEPTIEMBRE 2017 16 (null) 
> xx.ccc!user.ama.COMUNICACIONES CIAS-OTROS.COMUNICACIONES CIAS 
> 2017.SEPTIEMBRE 2017.JULIO 2017  16 (null) 
> xx.ccc!user.ama.COMUNICACIONES CIAS-OTROS.COMUNICACIONES CIAS 
> 2017.SEPTIEMBRE 2017.JUNIO 2017  16 (null) 
> xx.ccc!user.ama.COMUNICACIONES CIAS-OTROS.COMUNICACIONES CIAS 
> 2017.SEPTIEMBRE 2017.MAYO 2017   16 (null) 
> 
> Cheers!
> 
> 
> 
> Egoitz Aurrekoetxea
> Dpto. de sistemas
> 944 209 470
> Parque Tecnológico. Edificio 103
> 48170 Zamudio (Bizkaia)
> ego...@sarenet.es <mailto:undefined>
> www.sarenet.es <http://www.sarenet.es/>
> Antes de imprimir este correo electrónico piense si es necesario hacerlo.
> 
>> El 16 jul 2019, a las 10:10, Egoitz Aurrekoetxea > <mailto:ego...@sarenet.es>> escribió:
>> 
>> No problem Bron!!
>> 
>> Very thankful for your time and your help!!. 
>> 
>> I have some ideas/questions about the synchronous replication too for 
>> commenting you… I have never heard about Cassandane (only to Ellie in this 
>> list or Github commit comments) :) but will check it :) :) although I’d need 
>> to wait the holidays to come, in order to be more free for all these :)
>> 
>> Cheers!
>> 
>> 
>> 
>> Egoitz Aurrekoetxea
>> Dpto. de sistemas
>> 944 209 470
>> Parque Tecnológico. Edificio 103
>> 48170 Zamudio (Bizkaia)
>> ego...@sarenet.es <mailto:undefined>
>> www.sarenet.es <http://www.sarenet.es/>
>> Antes de imprimir este correo electrónico piense si es necesario hacerlo.
>> 
>>> El 16 jul 2019, a las 2:04, Bron Gondwana >> <mailto:br...@fastmail.fm>> escribió:
>>> 
>>> Sorry for not getting back to you yesterday.  I was on my way back from 
>>> vacation and had family commitments all last night.
>>> 
>>> Regarding contribution - the very best thing to do for a case like this is 
>>> to build Cassandane tests which isolate the issue :)  I'll see if I can get 
>>> that done today.
>>> 
>>> Cheers,
>>> 
>>> Bron.
>>> 
>>> On Tue, Jul 16, 2019, at 02:34, E

Re: Folder subscription issue

Hi!

One question are this records in the mailbox database (seen with a ctl_mboxlist 
-d) normal ?

xx.ccc!user.ama.COMUNICACIONES CIAS-OTROS.COMUNICACIONES CIAS 2017. 
NOVIEMBRE 2017 16 (null) 
xx.ccc!user.ama.COMUNICACIONES CIAS-OTROS.COMUNICACIONES CIAS 2017. 
OCTUBRE 2017   16 (null) 
xx.ccc!user.ama.COMUNICACIONES CIAS-OTROS.COMUNICACIONES CIAS 
2017.AGOSTO 2017 16 (null) 
xx.ccc!user.ama.COMUNICACIONES CIAS-OTROS.COMUNICACIONES CIAS 
2017.DICIEMBRE 2017  16 (null) 
xx.ccc!user.ama.COMUNICACIONES CIAS-OTROS.COMUNICACIONES CIAS 
2017.SEPTIEMBRE 2017 16 (null) 
xx.ccc!user.ama.COMUNICACIONES CIAS-OTROS.COMUNICACIONES CIAS 
2017.SEPTIEMBRE 2017.JULIO 2017  16 (null) 
xx.ccc!user.ama.COMUNICACIONES CIAS-OTROS.COMUNICACIONES CIAS 
2017.SEPTIEMBRE 2017.JUNIO 2017  16 (null) 
xx.ccc!user.ama.COMUNICACIONES CIAS-OTROS.COMUNICACIONES CIAS 
2017.SEPTIEMBRE 2017.MAYO 2017   16 (null) 

Cheers!



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia)
ego...@sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electrónico piense si es necesario hacerlo.

> El 16 jul 2019, a las 10:10, Egoitz Aurrekoetxea  escribió:
> 
> No problem Bron!!
> 
> Very thankful for your time and your help!!. 
> 
> I have some ideas/questions about the synchronous replication too for 
> commenting you… I have never heard about Cassandane (only to Ellie in this 
> list or Github commit comments) :) but will check it :) :) although I’d need 
> to wait the holidays to come, in order to be more free for all these :)
> 
> Cheers!
> 
> 
> 
> Egoitz Aurrekoetxea
> Dpto. de sistemas
> 944 209 470
> Parque Tecnológico. Edificio 103
> 48170 Zamudio (Bizkaia)
> ego...@sarenet.es <mailto:undefined>
> www.sarenet.es <http://www.sarenet.es/>
> Antes de imprimir este correo electrónico piense si es necesario hacerlo.
> 
>> El 16 jul 2019, a las 2:04, Bron Gondwana > <mailto:br...@fastmail.fm>> escribió:
>> 
>> Sorry for not getting back to you yesterday.  I was on my way back from 
>> vacation and had family commitments all last night.
>> 
>> Regarding contribution - the very best thing to do for a case like this is 
>> to build Cassandane tests which isolate the issue :)  I'll see if I can get 
>> that done today.
>> 
>> Cheers,
>> 
>> Bron.
>> 
>> On Tue, Jul 16, 2019, at 02:34, Egoitz Aurrekoetxea wrote:
>>> Hi Bron!
>>> 
>>> Sorry for answering so late I had the thought I answered you before…. I’m 
>>> slightly stressed these days...
>>> 
>>> I answer below in green bold for instance…..
>>> 
>>> 
>>> 
>>> Egoitz Aurrekoetxea
>>> Dpto. de sistemas
>>> 944 209 470
>>> Parque Tecnológico. Edificio 103
>>> 48170 Zamudio (Bizkaia)
>>> ego...@sarenet.es <mailto:ego...@sarenet.es>
>>> www.sarenet.es <http://www.sarenet.es/>
>>> 
>>> Antes de imprimir este correo electrónico piense si es necesario hacerlo.
>>> 
>>>> El 14 jul 2019, a las 14:36, Bron Gondwana >>> <mailto:br...@fastmailteam.com>> escribió:
>>>> 
>>>> On Wed, Jul 10, 2019, at 19:11, Egoitz Aurrekoetxea wrote:
>>>>> Good morning,
>>>>> 
>>>>> In previous thread (with the title slightly incorrect) I talk about an 
>>>>> issue suffered some day with Sieve script and folder subscriptions. The 
>>>>> Sieve part, was related to the migration, so for the moment let’s forget 
>>>>> about it (for me at least) as it has never reproduced again since then.
>>>> 
>>>> Sorry, I missed the previous thread.  Did you mention which version of 
>>>> Cyrus you are running?  There's clearly a bug and it would be good to know 
>>>> which version(s) it affects.
>>> 
>>> 
>>> I think I have specified all these details (including what the previous 
>>> thread was saying) in the answered mail this morning… I think it's all 
>>> there… the version was 3.0.8…. If is not all or you need something, please 
>>> tell me and I’ll try my bests for providing you all the info you need..
>>> 
>>> 
>>>> 
>>>>> About the folder subscription issue, I think I got something, at least a 
>>>>> close approximation... When a user causes in mua a rename mailbox (a 
>>>>> rename for a folder caused by a folder move in hierarchy), after the own 
>>>>> ren

Re: Folder subscription issue

No problem Bron!!

Very thankful for your time and your help!!. 

I have some ideas/questions about the synchronous replication too for 
commenting you… I have never heard about Cassandane (only to Ellie in this list 
or Github commit comments) :) but will check it :) :) although I’d need to wait 
the holidays to come, in order to be more free for all these :)

Cheers!



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia)
ego...@sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electrónico piense si es necesario hacerlo.

> El 16 jul 2019, a las 2:04, Bron Gondwana  escribió:
> 
> Sorry for not getting back to you yesterday.  I was on my way back from 
> vacation and had family commitments all last night.
> 
> Regarding contribution - the very best thing to do for a case like this is to 
> build Cassandane tests which isolate the issue :)  I'll see if I can get that 
> done today.
> 
> Cheers,
> 
> Bron.
> 
> On Tue, Jul 16, 2019, at 02:34, Egoitz Aurrekoetxea wrote:
>> Hi Bron!
>> 
>> Sorry for answering so late I had the thought I answered you before…. I’m 
>> slightly stressed these days...
>> 
>> I answer below in green bold for instance…..
>> 
>> 
>> 
>> Egoitz Aurrekoetxea
>> Dpto. de sistemas
>> 944 209 470
>> Parque Tecnológico. Edificio 103
>> 48170 Zamudio (Bizkaia)
>> ego...@sarenet.es
>> www.sarenet.es <http://www.sarenet.es/>
>> 
>> Antes de imprimir este correo electrónico piense si es necesario hacerlo.
>> 
>>> El 14 jul 2019, a las 14:36, Bron Gondwana >> <mailto:br...@fastmailteam.com>> escribió:
>>> 
>>> On Wed, Jul 10, 2019, at 19:11, Egoitz Aurrekoetxea wrote:
>>>> Good morning,
>>>> 
>>>> In previous thread (with the title slightly incorrect) I talk about an 
>>>> issue suffered some day with Sieve script and folder subscriptions. The 
>>>> Sieve part, was related to the migration, so for the moment let’s forget 
>>>> about it (for me at least) as it has never reproduced again since then.
>>> 
>>> Sorry, I missed the previous thread.  Did you mention which version of 
>>> Cyrus you are running?  There's clearly a bug and it would be good to know 
>>> which version(s) it affects.
>> 
>> 
>> I think I have specified all these details (including what the previous 
>> thread was saying) in the answered mail this morning… I think it's all 
>> there… the version was 3.0.8…. If is not all or you need something, please 
>> tell me and I’ll try my bests for providing you all the info you need..
>> 
>> 
>>> 
>>>> About the folder subscription issue, I think I got something, at least a 
>>>> close approximation... When a user causes in mua a rename mailbox (a 
>>>> rename for a folder caused by a folder move in hierarchy), after the own 
>>>> rename, if folders were subscribed “should” (for the "plain user" at 
>>>> least) become subscribed in the new path. It seems that after a user 
>>>> rename in Cyrus the new folder is automatically subscribed (even if no 
>>>> subscribe command is sent by the mua). But this, does not cause in the 
>>>> replica (in the slave, if SUB is not sent by the client) a 
>>>> sync_apply_changesub() or something like entering in the “ 
>>>> move_subscription” condition in mboxlist_renamemailbox(), and then, the 
>>>> folder is properly renamed but not subscribed in the slave. I think this 
>>>> is what I’m suffering. Obviously, if after a rename the mua sends a 
>>>> subscribe too, no issue is seen. I think the problem happens when a 
>>>> mailbox rename happens and a SUB is not send later.
>>> 
>>> Subscriptions aren't automatically renamed when a folder is renamed, but 
>>> they should be automatically renamed for a user rename.  Intermediate 
>>> replicated states can be bit messy due to race conditions with replication, 
>>> but I believe it should always wind up in the final correct state.  If not, 
>>> that's a bug for sure!
>> 
>> This tests are some days ago… I’m slightly confused now because I don’t 
>> remember it exactly… 
>> 
>>> 
>>>> An example : 
>>>> 
>>>> The folder domain.com 
>>>> <http://domain.com/>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG 
>>>> is moved (renamed) to trash.
>>>> 
>>>> May 16 16:16:50 mx6

Re: Folder subscription issue

2019-07-15 Thread Egoitz Aurrekoetxea

Hi Bron!

Sorry for answering so late I had the thought I answered you before…. I’m 
slightly stressed these days...

I answer below in green bold for instance…..



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia)
ego...@sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electrónico piense si es necesario hacerlo.

> El 14 jul 2019, a las 14:36, Bron Gondwana  escribió:
> 
> On Wed, Jul 10, 2019, at 19:11, Egoitz Aurrekoetxea wrote:
>> Good morning,
>> 
>> In previous thread (with the title slightly incorrect) I talk about an issue 
>> suffered some day with Sieve script and folder subscriptions. The Sieve 
>> part, was related to the migration, so for the moment let’s forget about it 
>> (for me at least) as it has never reproduced again since then.
> 
> Sorry, I missed the previous thread.  Did you mention which version of Cyrus 
> you are running?  There's clearly a bug and it would be good to know which 
> version(s) it affects.


I think I have specified all these details (including what the previous thread 
was saying) in the answered mail this morning… I think it's all there… the 
version was 3.0.8…. If is not all or you need something, please tell me and 
I’ll try my bests for providing you all the info you need..


> 
>> About the folder subscription issue, I think I got something, at least a 
>> close approximation... When a user causes in mua a rename mailbox (a rename 
>> for a folder caused by a folder move in hierarchy), after the own rename, if 
>> folders were subscribed “should” (for the "plain user" at least) become 
>> subscribed in the new path. It seems that after a user rename in Cyrus the 
>> new folder is automatically subscribed (even if no subscribe command is sent 
>> by the mua). But this, does not cause in the replica (in the slave, if SUB 
>> is not sent by the client) a sync_apply_changesub() or something like 
>> entering in the “ move_subscription” condition in mboxlist_renamemailbox(), 
>> and then, the folder is properly renamed but not subscribed in the slave. I 
>> think this is what I’m suffering. Obviously, if after a rename the mua sends 
>> a subscribe too, no issue is seen. I think the problem happens when a 
>> mailbox rename happens and a SUB is not send later.
> 
> Subscriptions aren't automatically renamed when a folder is renamed, but they 
> should be automatically renamed for a user rename.  Intermediate replicated 
> states can be bit messy due to race conditions with replication, but I 
> believe it should always wind up in the final correct state.  If not, that's 
> a bug for sure!

This tests are some days ago… I’m slightly confused now because I don’t 
remember it exactly… 

> 
>> An example : 
>> 
>> The folder domain.com 
>> <http://domain.com/>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG is 
>> moved (renamed) to trash.
>> 
>> May 16 16:16:50 mx6c imap[83976]: conversations_rename_folder: renamed 
>> domain.com 
>> <http://domain.com/>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG to 
>> domain.com <http://domain.com/>!user.parta^partb.Trash.XINGANG
>> May 16 16:16:50 mx6c imap[83976]: Rename: domain.com 
>> <http://domain.com/>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG -> 
>> domain.com <http://domain.com/>!user.parta^partb.Trash.XINGANG
>> May 16 16:16:50 mx6c imap[83976]: Deleted mailbox domain.com 
>> <http://domain.com/>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>> 
>> In the master (sync), the folder in Trash is subscribed but in the slave it 
>> is not, and remains subscribed the folder in the original location in the 
>> “.sub” file.
> 
> This might just be a matter of failing to sync_log the subscription in that 
> codepath.  Hmm...
> 
> But there is a question of whether we should even be renaming the 
> subscription - RFC3501 is a little silent on this issue, but it does say in a 
> couple of places that the server MUST NOT remove a subscription even if the 
> mailbox with that name doesn't exist, which makes renaming subscriptions a 
> bit unclear.  I'll check in with the authors of draft-ietf-extra-imap4rev2 
> and ask for this to be clarified next time around!


That’s it Bron..  it was strange… anyway… you know, when you try to debug an 
issue like the commented you do a closer inspection to all this behaviour, you 
know…. Although as said before, perhaps it’s better to talk about today's 
examples… I got them more recent….


> 
>> A diff between the master (replication client) .sub file and the slave’s one 
>>

Re: Folder subscription issue

2019-07-15 Thread Egoitz Aurrekoetxea

 Logistics -> xxx.com!user.gl-p^expooort.Sent.OFERTAS.Gabana LogisticsJul 12 10:13:26 mx13c imap[53014]: Deleted mailbox xxx.com!user.gl-p^expooort.Sent.OFERTAS.AGENTES VENDA.Gabana LogisticsGabana Logistics (xxx.com!user.gl-p^expooort.Sent.OFERTAS.AGENTES VENDA.Gabana Logistics) is still subscribed in the actual slave (and in the slave in that moment).My suspects are : - For the first commented issue (number 1), sync_client when populating dlists, seems to perform an incorrect name conversion (conversion to internal name when shouldn’t) and then when creating quota, sieve, seen and subs, quota is created properly but not he other ones…- For the second commented one (number 2)… I’d say a mboxlist_changesub is missing if RFC says the behavior of the mua is correct. I do attach the files of a master and slave in Cyrus 3.0.8. If you needed a cyrus.conf file too please tell me. I’m at your disposal for whatever you needed for fixing this. Best regards,

master-file
Description: Binary data


slave-file
Description: Binary data
El 14 jul 2019, a las 14:36, Bron Gondwana <br...@fastmailteam.com> escribió:On Wed, Jul 10, 2019, at 19:11, Egoitz Aurrekoetxea wrote:Good morning,In previous thread (with the title slightly incorrect) I talk about an issue suffered some day with Sieve script and folder subscriptions. The Sieve part, was related to the migration, so for the moment let’s forget about it (for me at least) as it has never reproduced again since then.Sorry, I missed the previous thread.  Did you mention which version of Cyrus you are running?  There's clearly a bug and it would be good to know which version(s) it affects.About the folder subscription issue, I think I got something, at least a close approximation... When a user causes in mua a rename mailbox (a rename for a folder caused by a folder move in hierarchy), after the own rename, if folders were subscribed “should” (for the "plain user" at least) become subscribed in the new path. It seems that after a user rename in Cyrus the new folder is automatically subscribed (even if no subscribe command is sent by the mua). But this, does not cause in the replica (in the slave, if SUB is not sent by the client) a sync_apply_changesub() or something like entering in the “ move_subscription” condition in mboxlist_renamemailbox(), and then, the folder is properly renamed but not subscribed in the slave. I think this is what I’m suffering. Obviously, if after a rename the mua sends a subscribe too, no issue is seen. I think the problem happens when a mailbox rename happens and a SUB is not send later.Subscriptions aren't automatically renamed when a folder is renamed, but they should be automatically renamed for a user rename.  Intermediate replicated states can be bit messy due to race conditions with replication, but I believe it should always wind up in the final correct state.  If not, that's a bug for sure!An example : The folder domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG is moved (renamed) to trash.May 16 16:16:50 mx6c imap[83976]: conversations_rename_folder: renamed domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG to domain.com!user.parta^partb.Trash.XINGANGMay 16 16:16:50 mx6c imap[83976]: Rename: domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG -> domain.com!user.parta^partb.Trash.XINGANGMay 16 16:16:50 mx6c imap[83976]: Deleted mailbox domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANGIn the master (sync), the folder in Trash is subscribed but in the slave it is not, and remains subscribed the folder in the original location in the “.sub” file.This might just be a matter of failing to sync_log the subscription in that codepath.  Hmm...But there is a question of whether we should even be renaming the subscription - RFC3501 is a little silent on this issue, but it does say in a couple of places that the server MUST NOT remove a subscription even if the mailbox with that name doesn't exist, which makes renaming subscriptions a bit unclear.  I'll check in with the authors of draft-ietf-extra-imap4rev2 and ask for this to be clarified next time around!A diff between the master (replication client) .sub file and the slave’s one is (mx5 is the master, the client and 6 the slave): --- subscripciones-mx5c/parta.partb.sub 2019-07-10 08:47:29.0 +0200+++ subscripciones-mx6c/parta.partb.sub 2019-07-10 08:48:08.0 +0200+domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG-domain.com!user.parta^partb.Trash.XINGANGPerhaps a move_subscription to one or sync_apply_changesub should be forced in order to fix it?. I have seen issues with Outlook 2016 and Thunderbird… both mua… perhaps by RFC they should send the SUB command but… it’s the only theory I could arrive to… I have a ton more cases like this one.. Data gets properly handled but subscriptions have this issue (perhaps we could say is a mua issue but….)..Bron.--  Bron Gondwana, CEO

Re: Folder subscription issue

2019-07-12 Thread Egoitz Aurrekoetxea

By the way I think I found some more…..

When a sync_client in user mode… creates a non existing user in the replica… if 
the user has a dot… the quota gets properly created, but seen, sub files are 
wrongly created… for instance….

-rw---   1 cyrus  cyrus  2576528 Jul 12 13:03 f.veiga.conversations
-rw---   1 cyrus  cyrus   88 Jul 12 13:03 f.veiga.counters
-rw---   1 cyrus  cyrus  451 Jul 12 10:43 f.veiga.sub
-rw---   1 cyrus  cyrus   16 Jul 12 01:32 f.veiga.xapianactive
-rw---   1 cyrus  cyrus 2584 Apr  3 01:48 f^veiga.seen
-rw---   1 cyrus  cyrus  316 Apr  3 01:48 f^veiga.sub

The Apr 3 files are from the initial sync when the box was empty… the first 
ones (with the dot and not ^) are the actual files being used by Cyrus…. And 
updated with the replication and so…

It seems this to be the reason because I didn’t see in the initial sync any 
subscribed folders… but later when set them from a mua where properly 
replicated…

With Sieve seemed to happen something similar… but in this case with the ‘^’ 
and ‘.’ In the directory name….

It’s like the name used for creating the subscribing and seen databases in a 
user mode replication was not properly handled by Cyrus… because it does the 
same as with quota database with indeed with ‘^’ is properly created….

Thanks! Bye!



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia)
ego...@sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electrónico piense si es necesario hacerlo.

> El 10 jul 2019, a las 11:39, Egoitz Aurrekoetxea  escribió:
> 
> Hi!
> 
> It happens with any folder… in fact Trash is not the folder we would announce 
> in special-use as Trash… it’s just a normal folder really here…. It’s a 
> generally happening thing with rename of folders inside the own hierarchy 
> inside the own user… (don’t really know if a rename mailbox for changing 
> partition would have the same issue). Not something related to the Trash 
> folder...
> 
> Cheers!
> 
> 
> 
> Egoitz Aurrekoetxea
> Dpto. de sistemas
> 944 209 470
> Parque Tecnológico. Edificio 103
> 48170 Zamudio (Bizkaia)
> ego...@sarenet.es <mailto:undefined>
> www.sarenet.es <http://www.sarenet.es/>
> Antes de imprimir este correo electrónico piense si es necesario hacerlo.
> 
>> El 10 jul 2019, a las 11:34, Sebastian Hagedorn > <mailto:haged...@uni-koeln.de>> escribió:
>> 
>> Hi,
>> 
>> I'm curious if this only happens for rename to trash, or for all renames
>> of subscribed folders. IMHO it makes no sense to automatically subscribe
>> to a folder in the trash. So perhaps the bug isn't in the replication
>> code but rather in the handling of rename to trash?
>> 
>> 
>> Am 10.07.19 um 11:11 Uhr schrieb Egoitz Aurrekoetxea:
>>> About the folder subscription issue, I think I got something, at least a
>>> close approximation... When a user causes in mua a rename mailbox (a
>>> rename for a folder caused by a folder move in hierarchy), after the own
>>> rename, if folders were subscribed “should” (for the "plain user" at
>>> least) become subscribed in the new path. It seems that after a user
>>> rename in Cyrus the new folder is automatically subscribed (even if no
>>> subscribe command is sent by the mua). But this, does not cause in the
>>> replica (in the slave, if SUB is not sent by the client)
>>> a sync_apply_changesub() or something like entering in the
>>> “ move_subscription” condition in mboxlist_renamemailbox(), and then,
>>> the folder is properly renamed but not subscribed in the slave. I think
>>> this is what I’m suffering. Obviously, if after a rename the mua sends a
>>> subscribe too, no issue is seen. I think the problem happens when a
>>> mailbox rename happens and a SUB is not send later.
>>> 
>>> An example : 
>>> 
>>> The folder domain.com <http://domain.com/>
>>> <http://domain.com 
>>> <http://domain.com/>>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>>> is moved (renamed) to trash.
>>> 
>>> May 16 16:16:50 mx6c imap[83976]: conversations_rename_folder: renamed
>>> domain.com <http://domain.com/>
>>> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>>> to domain.com <http://domain.com>!user.parta^partb.Trash.XINGANG
>>> May 16 16:16:50 mx6c imap[83976]: Rename: domain.com
>>> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>>> -> domain.com <http://domain.com>!user.parta^partb.Trash.XINGANG

Re: Folder subscription issue

Hi!

It happens with any folder… in fact Trash is not the folder we would announce 
in special-use as Trash… it’s just a normal folder really here…. It’s a 
generally happening thing with rename of folders inside the own hierarchy 
inside the own user… (don’t really know if a rename mailbox for changing 
partition would have the same issue). Not something related to the Trash 
folder...

Cheers!



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia)
ego...@sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electrónico piense si es necesario hacerlo.

> El 10 jul 2019, a las 11:34, Sebastian Hagedorn  
> escribió:
> 
> Hi,
> 
> I'm curious if this only happens for rename to trash, or for all renames
> of subscribed folders. IMHO it makes no sense to automatically subscribe
> to a folder in the trash. So perhaps the bug isn't in the replication
> code but rather in the handling of rename to trash?
> 
> 
> Am 10.07.19 um 11:11 Uhr schrieb Egoitz Aurrekoetxea:
>> About the folder subscription issue, I think I got something, at least a
>> close approximation... When a user causes in mua a rename mailbox (a
>> rename for a folder caused by a folder move in hierarchy), after the own
>> rename, if folders were subscribed “should” (for the "plain user" at
>> least) become subscribed in the new path. It seems that after a user
>> rename in Cyrus the new folder is automatically subscribed (even if no
>> subscribe command is sent by the mua). But this, does not cause in the
>> replica (in the slave, if SUB is not sent by the client)
>> a sync_apply_changesub() or something like entering in the
>> “ move_subscription” condition in mboxlist_renamemailbox(), and then,
>> the folder is properly renamed but not subscribed in the slave. I think
>> this is what I’m suffering. Obviously, if after a rename the mua sends a
>> subscribe too, no issue is seen. I think the problem happens when a
>> mailbox rename happens and a SUB is not send later.
>> 
>> An example : 
>> 
>> The folder domain.com
>> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>> is moved (renamed) to trash.
>> 
>> May 16 16:16:50 mx6c imap[83976]: conversations_rename_folder: renamed
>> domain.com
>> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>> to domain.com <http://domain.com>!user.parta^partb.Trash.XINGANG
>> May 16 16:16:50 mx6c imap[83976]: Rename: domain.com
>> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>> -> domain.com <http://domain.com>!user.parta^partb.Trash.XINGANG
>> May 16 16:16:50 mx6c imap[83976]: Deleted mailbox domain.com
>> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>> 
>> In the master (sync), the folder in Trash is subscribed but in the slave
>> it is not, and remains subscribed the folder in the original location in
>> the “.sub” file.
>> 
>> A diff between the master (replication client) .sub file and the slave’s
>> one is (mx5 is the master, the client and 6 the slave): 
>> 
>> --- subscripciones-mx5c/parta.partb.sub2019-07-10 08:47:29.0 +0200
>> +++ subscripciones-mx6c/parta.partb.sub2019-07-10 08:48:08.0 +0200
>> 
>> +domain.com
>> <http://domain.com>!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
>> -domain.com <http://domain.com>!user.parta^partb.Trash.XINGANG
>> 
>> Perhaps a move_subscription to one or sync_apply_changesub should be
>> forced in order to fix it?. I have seen issues with Outlook 2016 and
>> Thunderbird… both mua… perhaps by RFC they should send the SUB command
>> but… it’s the only theory I could arrive to… I have a ton more cases
>> like this one.. Data gets properly handled but subscriptions have this
>> issue (perhaps we could say is a mua issue but….)..
> <0x197B06994D105B45.asc>


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Folder subscription issue

Good morning,

In previous thread (with the title slightly incorrect) I talk about an issue 
suffered some day with Sieve script and folder subscriptions. The Sieve part, 
was related to the migration, so for the moment let’s forget about it (for me 
at least) as it has never reproduced again since then.

About the folder subscription issue, I think I got something, at least a close 
approximation... When a user causes in mua a rename mailbox (a rename for a 
folder caused by a folder move in hierarchy), after the own rename, if folders 
were subscribed “should” (for the "plain user" at least) become subscribed in 
the new path. It seems that after a user rename in Cyrus the new folder is 
automatically subscribed (even if no subscribe command is sent by the mua). But 
this, does not cause in the replica (in the slave, if SUB is not sent by the 
client) a sync_apply_changesub() or something like entering in the “ 
move_subscription” condition in mboxlist_renamemailbox(), and then, the folder 
is properly renamed but not subscribed in the slave. I think this is what I’m 
suffering. Obviously, if after a rename the mua sends a subscribe too, no issue 
is seen. I think the problem happens when a mailbox rename happens and a SUB is 
not send later.

An example : 

The folder domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG is 
moved (renamed) to trash.

May 16 16:16:50 mx6c imap[83976]: conversations_rename_folder: renamed 
domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG to 
domain.com!user.parta^partb.Trash.XINGANG
May 16 16:16:50 mx6c imap[83976]: Rename: 
domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG -> 
domain.com!user.parta^partb.Trash.XINGANG
May 16 16:16:50 mx6c imap[83976]: Deleted mailbox 
domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG

In the master (sync), the folder in Trash is subscribed but in the slave it is 
not, and remains subscribed the folder in the original location in the 
“.sub” file.

A diff between the master (replication client) .sub file and the slave’s one is 
(mx5 is the master, the client and 6 the slave): 

--- subscripciones-mx5c/parta.partb.sub 2019-07-10 08:47:29.0 +0200
+++ subscripciones-mx6c/parta.partb.sub 2019-07-10 08:48:08.0 +0200

+domain.com!user.parta^partb.Archives.2019.TRAFICO.CHINA.XINGANG
-domain.com!user.parta^partb.Trash.XINGANG  

Perhaps a move_subscription to one or sync_apply_changesub should be forced in 
order to fix it?. I have seen issues with Outlook 2016 and Thunderbird… both 
mua… perhaps by RFC they should send the SUB command but… it’s the only theory 
I could arrive to… I have a ton more cases like this one.. Data gets properly 
handled but subscriptions have this issue (perhaps we could say is a mua issue 
but….)..

What do you think?.

Best regards,



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia)
ego...@sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electrónico piense si es necesario hacerlo.


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Issues with replication and folder/Sieve subscription

It would be better to just talk daemons IMHO…. That should work… but sometimes… 
perhaps another things could be implied… so… the most clear way, should be to 
just put as a normal client with a expect script for instance all the source 
scripts to Cyrus… but using sieveshell… and not using files directly (not 
copying) anything to Cyrus partitions “by hand”…

As said IMHO…

Cheers!



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia)
ego...@sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electrónico piense si es necesario hacerlo.

> El 10 jul 2019, a las 9:22, Albert Shih  escribió:
> 
> Le 09/07/2019 à 22:49:01+0200, Egoitz Aurrekoetxea a écrit
>> By the way, for your case I would recommend doing a script that does a get 
>> from
>> dovecot and a put to Cyrus instead of copying Sieve files directly… it’s a 
>> much
>> more cleaner way…
> 
> Yes, it is what I did, before I try de sync I event do a
> /usr/local/cyrsus/sievec on each file to by absolute sure the sieve file
> compile correctly
> 
> Regards.
> 
> 
> --
> Albert SHIH
> DIO bâtiment 15
> Observatoire de Paris
> xmpp: j...@obspm.fr
> Heure local/Local time:
> Wed 10 Jul 2019 09:20:09 AM CEST


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Issues with replication and folder/Sieve subscription

The subject of this email is not properly set… it should be . Issues in 
replication with folder subscription and Sieve

As I think I discovered something I reopen a new thread with the title properly

 



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia)
ego...@sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electrónico piense si es necesario hacerlo.

> El 10 jul 2019, a las 9:22, Albert Shih  escribió:
> 
> Le 09/07/2019 à 22:49:01+0200, Egoitz Aurrekoetxea a écrit
>> By the way, for your case I would recommend doing a script that does a get 
>> from
>> dovecot and a put to Cyrus instead of copying Sieve files directly… it’s a 
>> much
>> more cleaner way…
> 
> Yes, it is what I did, before I try de sync I event do a
> /usr/local/cyrsus/sievec on each file to by absolute sure the sieve file
> compile correctly
> 
> Regards.
> 
> 
> --
> Albert SHIH
> DIO bâtiment 15
> Observatoire de Paris
> xmpp: j...@obspm.fr
> Heure local/Local time:
> Wed 10 Jul 2019 09:20:09 AM CEST


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Issues with replication and folder/Sieve subscription

By the way, for your case I would recommend doing a script that does a get from 
dovecot and a put to Cyrus instead of copying Sieve files directly… it’s a much 
more cleaner way…

Cheers!



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia)
ego...@sarenet.es <mailto:ego...@sarenet.es>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electrónico piense si es necesario hacerlo.

> El 9 jul 2019, a las 22:44, Egoitz Aurrekoetxea  escribió:
> 
> Hi Albert,
> 
> If instead of -A you used -u for each of your users did it worked? Or did it 
> crashed in the same user as with -A?. Which Cyrus version were you running?. 
> 
> Cheers,
> 
> 
> 
> Egoitz Aurrekoetxea
> Dpto. de sistemas
> 944 209 470
> Parque Tecnológico. Edificio 103
> 48170 Zamudio (Bizkaia)
> ego...@sarenet.es <mailto:ego...@sarenet.es>
> www.sarenet.es <http://www.sarenet.es/>
> Antes de imprimir este correo electrónico piense si es necesario hacerlo.
> 
>> El 9 jul 2019, a las 15:03, Albert Shih > <mailto:albert.s...@obspm.fr>> escribió:
>> 
>> Le 09/07/2019 à 14:10:49+0200, Egoitz Aurrekoetxea a écrit
>>> Good morning,
>>> 
>>> 
>>> After we upgraded to Cyrus 3.0.8, we saw that some users in the replicas 
>>> didn't
>>> have some folders (or all) subscribed the same way they had in previous env 
>>> in
>>> Cyrus 2.3. Same happened for some users with Sieve scripts. It seemed the
>>> content itself was perfectly copied. It was like, if the copy between 
>>> versions
>>> would not had fully succeed. We fixed it easily by creating some scripts for
>>> checking folder subscriptions and Sieve scripts existence. We though it 
>>> could
>>> perhaps had something to do with some issue replicating folders 
>>> subscriptions
>>> and sieve scripts from 2.3 to 3.0.8. After that, as we fixed it easily and 
>>> was
>>> nothing related to content, we just didn’t go in deep in this topic.
>> 
>> After my transfering all mail from my old server (dovecot) to the new one 
>> (under
>> cyrus), I try to initialize the sync, so I launch the synchronization, and
>> find out everytime the user got a sieve, the sync processus crash, 
>> butevent the
>> it crash it still create « something », so the next time I launch the sync
>> it pass (and email a actually synchronize). So for the first synchro I
>> juste launch a infinite loop in bash to synchronize all user.
>> 
>> I known it's not a very satisfying method but it work.
>> 
>> With new user I don't have any problem.
>> 
>> I already send a email here but don't get any solution
>> 
>>https://lists.andrew.cmu.edu/pipermail/info-cyrus/2018-May/040186.html 
>> <https://lists.andrew.cmu.edu/pipermail/info-cyrus/2018-May/040186.html>
>> 
>> Don't know if that help
>> 
>> Regards
>> --
>> Albert SHIH
>> DIO bâtiment 15
>> Observatoire de Paris
>> 5 Place Jules Janssen
>> 92195 Meudon Cedex
>> France
>> xmpp: j...@obspm.fr <mailto:j...@obspm.fr>
>> Heure local/Local time:
>> Tue 09 Jul 2019 02:57:46 PM CEST
>> 
>> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ 
>> <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus 
>> <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Issues with replication and folder/Sieve subscription

Hi Albert,

If instead of -A you used -u for each of your users did it worked? Or did it 
crashed in the same user as with -A?. Which Cyrus version were you running?. 

Cheers,



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia)
ego...@sarenet.es <mailto:ego...@sarenet.es>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electrónico piense si es necesario hacerlo.

> El 9 jul 2019, a las 15:03, Albert Shih  escribió:
> 
> Le 09/07/2019 à 14:10:49+0200, Egoitz Aurrekoetxea a écrit
>> Good morning,
>> 
>> 
>> After we upgraded to Cyrus 3.0.8, we saw that some users in the replicas 
>> didn't
>> have some folders (or all) subscribed the same way they had in previous env 
>> in
>> Cyrus 2.3. Same happened for some users with Sieve scripts. It seemed the
>> content itself was perfectly copied. It was like, if the copy between 
>> versions
>> would not had fully succeed. We fixed it easily by creating some scripts for
>> checking folder subscriptions and Sieve scripts existence. We though it could
>> perhaps had something to do with some issue replicating folders subscriptions
>> and sieve scripts from 2.3 to 3.0.8. After that, as we fixed it easily and 
>> was
>> nothing related to content, we just didn’t go in deep in this topic.
> 
> After my transfering all mail from my old server (dovecot) to the new one 
> (under
> cyrus), I try to initialize the sync, so I launch the synchronization, and
> find out everytime the user got a sieve, the sync processus crash, 
> butevent the
> it crash it still create « something », so the next time I launch the sync
> it pass (and email a actually synchronize). So for the first synchro I
> juste launch a infinite loop in bash to synchronize all user.
> 
> I known it's not a very satisfying method but it work.
> 
> With new user I don't have any problem.
> 
> I already send a email here but don't get any solution
> 
>https://lists.andrew.cmu.edu/pipermail/info-cyrus/2018-May/040186.html
> 
> Don't know if that help
> 
> Regards
> --
> Albert SHIH
> DIO bâtiment 15
> Observatoire de Paris
> 5 Place Jules Janssen
> 92195 Meudon Cedex
> France
> xmpp: j...@obspm.fr
> Heure local/Local time:
> Tue 09 Jul 2019 02:57:46 PM CEST
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Issues with replication and folder/Sieve subscription

Could perhaps, some of this something to do, with having intermediate folders 
subscribed/unsubscribed in the middle of the tree? And that to cause something 
like it that perhaps is not caused when the mailbox is being accessed by the 
user instead of being replicated (when the change is applied by 
synchronization)?.



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia)
ego...@sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electrónico piense si es necesario hacerlo.

> El 9 jul 2019, a las 14:10, Egoitz Aurrekoetxea  escribió:
> 
> Good morning,
> 
> 
> After we upgraded to Cyrus 3.0.8, we saw that some users in the replicas 
> didn't have some folders (or all) subscribed the same way they had in 
> previous env in Cyrus 2.3. Same happened for some users with Sieve scripts. 
> It seemed the content itself was perfectly copied. It was like, if the copy 
> between versions would not had fully succeed. We fixed it easily by creating 
> some scripts for checking folder subscriptions and Sieve scripts existence. 
> We though it could perhaps had something to do with some issue replicating 
> folders subscriptions and sieve scripts from 2.3 to 3.0.8. After that, as we 
> fixed it easily and was nothing related to content, we just didn’t go in deep 
> in this topic.
> 
> When we started running couples of servers in Cyrus 3.0.8 (upgraded and 
> apparently with all the change process finished) we have seen in some cases 
> that folder subscription was not properly replicated again. Both members of 
> the couple of servers where running same Cyrus version, the 3.0.8 and this 
> issue, was not seen in all users… just in some of them… Due to this reason I 
> have started checking the git repo logs… for trying to see some perhaps 
> related change or similar… some commit that could very clearly affect to this 
> issue (that could have caused it). I have had no success.
> 
> So after that, and after not being able to reproduce it, have taken a look at 
> the code. By the nature of the things seen, I supposed that perhaps a META 
> operation failed could be involved. I say it because in sync_do_meta() I can 
> read : 
> 
> r = sync_response_parse(sync_be->in, "META", NULL,
> replica_subs, replica_sieve, replica_seen, NULL);
> if (!r) r = sync_do_user_seen(userid, replica_seen, sync_be, flags);
> if (!r) r = sync_do_user_sub(userid, replica_subs, sync_be, flags);
> if (!r) r = sync_do_user_sieve(userid, replica_sieve, sync_be, flags);
> 
> Have been looking for some hours around it but have not been able to see 
> nothing strange… nothing that could have caused this...
> 
> Have you ever heard about this issue?.
> 
> 
> Best regards,
> 
> 
> 
> Egoitz Aurrekoetxea
> Dpto. de sistemas
> 944 209 470
> Parque Tecnológico. Edificio 103
> 48170 Zamudio (Bizkaia)
> ego...@sarenet.es <mailto:undefined>
> www.sarenet.es <http://www.sarenet.es/>
> Antes de imprimir este correo electrónico piense si es necesario hacerlo.
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/ <http://www.cyrusimap.org/>
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ 
> <http://lists.andrew.cmu.edu/pipermail/info-cyrus/>
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus 
> <https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus>

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Issues with replication and folder/Sieve subscription

Good morning,


After we upgraded to Cyrus 3.0.8, we saw that some users in the replicas didn't 
have some folders (or all) subscribed the same way they had in previous env in 
Cyrus 2.3. Same happened for some users with Sieve scripts. It seemed the 
content itself was perfectly copied. It was like, if the copy between versions 
would not had fully succeed. We fixed it easily by creating some scripts for 
checking folder subscriptions and Sieve scripts existence. We though it could 
perhaps had something to do with some issue replicating folders subscriptions 
and sieve scripts from 2.3 to 3.0.8. After that, as we fixed it easily and was 
nothing related to content, we just didn’t go in deep in this topic.

When we started running couples of servers in Cyrus 3.0.8 (upgraded and 
apparently with all the change process finished) we have seen in some cases 
that folder subscription was not properly replicated again. Both members of the 
couple of servers where running same Cyrus version, the 3.0.8 and this issue, 
was not seen in all users… just in some of them… Due to this reason I have 
started checking the git repo logs… for trying to see some perhaps related 
change or similar… some commit that could very clearly affect to this issue 
(that could have caused it). I have had no success.

So after that, and after not being able to reproduce it, have taken a look at 
the code. By the nature of the things seen, I supposed that perhaps a META 
operation failed could be involved. I say it because in sync_do_meta() I can 
read : 

r = sync_response_parse(sync_be->in, "META", NULL,
replica_subs, replica_sieve, replica_seen, NULL);
if (!r) r = sync_do_user_seen(userid, replica_seen, sync_be, flags);
if (!r) r = sync_do_user_sub(userid, replica_subs, sync_be, flags);
if (!r) r = sync_do_user_sieve(userid, replica_sieve, sync_be, flags);

Have been looking for some hours around it but have not been able to see 
nothing strange… nothing that could have caused this...

Have you ever heard about this issue?.


Best regards,



Egoitz Aurrekoetxea
Dpto. de sistemas
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia)
ego...@sarenet.es <mailto:undefined>
www.sarenet.es <http://www.sarenet.es/>
Antes de imprimir este correo electrónico piense si es necesario hacerlo.


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Questions about user mode sync_client (move mailboxes from one server to another) and mailbox moving from partition (moving from partition in same server, renaming) lock

2019-05-24 Thread Egoitz Aurrekoetxea via Info-cyrus

Hi Sebastian, 

Thanks a lot for your comments. But that, anyway, won't assure no
mailbox access would exist in the middle of the rename... 

I think there should not be problems due that the function
mboxlist_renamemailbox() does a mailbox_open_iwl() which finally checks
if the mailbox is locked and then locks it. Anyway, if none of the gurus
of Cyrus sais it... I would read more deeply the code (for ensuring) and
will do some checks in testing env 

Thanks a lot Sebastian :) 

Cheers 

El 2019-05-23 19:28, Sebastian Hagedorn escribió:

> Hi,
> 
>> Our Cyrus machines (Cyrus 3.0.8), usually have 3 mailbox partitions.
>> Sometimes, one of them becomes highly filled so we usually perform a
>> mailbox rename to another partition of the same server. For that
>> purpose, we normally lock at our proxy barrier any access to the mailbox
>> (we do play with Nginx authentication, Postfix hold and so). Is it
>> really needed to lock that way the mailbox, at some "external to Cyrus
>> level," in order to avoid mailbox corruption?. Or does Cyrus handle that
>> properly?. Does Cyrus exclusively lock and after done, unlock again?.
> 
> I can only answer that part of the question. We have been doing it like that 
> (without blocking access from the outside) for years, but we're still on 
> Cyrus 2.4. We only make sure there are no active processes by the user before 
> starting the RENAME, and we do it at night. There haven't been any problems 
> with that approach.
> --
> Sebastian Hagedorn - Weyertal 121, Zimmer 2.02
> Regionales Rechenzentrum (RRZK)
> Universität zu Köln / Cologne University - Tel. +49-221-470-89578
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Questions about user mode sync_client (move mailboxes from one server to another) and mailbox moving from partition (moving from partition in same server, renaming) lock

2019-05-23 Thread Egoitz Aurrekoetxea via Info-cyrus

Good afternoon, 

Our Cyrus machines (Cyrus 3.0.8), usually have 3 mailbox partitions.
Sometimes, one of them becomes highly filled so we usually perform a
mailbox rename to another partition of the same server. For that
purpose, we normally lock at our proxy barrier any access to the mailbox
(we do play with Nginx authentication, Postfix hold and so). Is it
really needed to lock that way the mailbox, at some "external to Cyrus
level," in order to avoid mailbox corruption?. Or does Cyrus handle that
properly?. Does Cyrus exclusively lock and after done, unlock again?.
Have been taking a look at mboxlist_renamemailbox() and seemed so. Have
noticed too, that it seems that partition rename operation from and to
the same server but different parition at least, is not being inserted
in the rolling mode lock.. is this a new security measure for avoiding
accidents with the rename?. Always I have done a mailbox rename
previously (Cyrus 2.3.X), have stopped the master/slave replication,
done the rename in the master and later if all ended fine... launched in
the slave a dm of the "in the master renamed mailbox" and a sync_client
-u from the master for the mailbox to be copied to the appropiate
partition in the slave. 

My other question is.. with the new replication method (imap based and
so...), can I do a user mode sync_client from a mailbox, to another
server acting as a master?. I mean, in the following scenario :  

Server A (master) => Server B (slave) 

Server C (master) => Server D (slave) 

The a...@bbb.net mailbox is in A server. I want to move the mailbox from
A=>B couple of master/slave server to C=>D couple of mater/slave. I
launch a "sync_client -v -u a...@bbb.net -S C -p partition3" in server A.
Server C, has sync_log_chain enabled. Would that mailbox be replicated
in C=>D couple (to both from A to C and from C to D) and been able to be
accesible in C?. If so, does any kind of drawback exist in having always
sync_log_chain enabled?... else for this kind of movement seems to be
useful.. 

But thinking about it... if C is master... is it really needed that
sync_log_chain config statement in that case or it would just be
necessary (as I think), for replicating in the following scenario only?.


Server 1 (master) -> Server 2 (slave) -> Server 3 (slave) 

So, not needed when (there's a master in the middle) : 

Server 1 (master) -> Server 2 (master) -> Server 3 (slave)   perhaps
as in
https://www.cyrusimap.org/imap/reference/admin/sop/replication.html can
be read?. 

Thank you so much for your time, 

Best regards,
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Various questions about databases (upgrade and migration)

2019-02-14 Thread Egoitz Aurrekoetxea

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 14-02-2019 15:46, Raphaël Halimi escribió:

> Hi Egoitz,
> 
> Thank you for your quick answer.
> 
> Le 14/02/2019 à 14:46, Egoitz Aurrekoetxea a écrit : Now for the databases 
> themselves. In /var/lib/cyrus the global databases
> were converted on-the-fly:
> 
> # file /var/lib/cyrus/*.db
> /var/lib/cyrus/annotations.db:  Cyrus twoskip DB
> /var/lib/cyrus/deliver.db:  Cyrus twoskip DB
> /var/lib/cyrus/mailboxes.db:Cyrus twoskip DB
> /var/lib/cyrus/statuscache.db:  Cyrus twoskip DB
> /var/lib/cyrus/tls_sessions.db: Cyrus twoskip DB
> /var/lib/cyrus/user_deny.db:empty
> 
> However, the user databases were not converted:
> 
> # file /var/lib/cyrus/user/*/*
> /var/lib/cyrus/user/u/user1.seen: Cyrus skiplist DB
> /var/lib/cyrus/user/u/user1.sub:  ASCII text
> /var/lib/cyrus/user/u/user2.seen:   Cyrus skiplist DB
> /var/lib/cyrus/user/u/user2.sub:ASCII text
> /var/lib/cyrus/user/u/user3.seen:   Cyrus skiplist DB
> /var/lib/cyrus/user/u/user3.sub:ASCII text
> *Cyrus 2.4 converted databases on the fly. Cyrus 2.5 and newer don't.
> You should launch a "reconstruct -r -V max" for that purpose.* 
> So my next questions are: why are the databases still in skiplist
> format, whereas according to /usr/lib/cyrus/cyrus-db-types.txt, they
> should be twoskip ? Why didn't Cyrus convert them on-the-fly like the
> global databases ? Do I have to manually do it myself ? And if I do
> convert them, will it change anything (performance, reliability, etc
> etc) ?
> 
> *As said perhaps is a Debian derived config for the package. Yes you
> should with the command above.* 
> Also, what about the various databases in the mail directories
> (cyrus.cache, cyrus.header, cyrus.index) ? For most of them, the "file"
> command only reports "data". What format are they actually in ? Do I
> have to convert them too ?
> 
> *Sure... Just launch a reconstruct -r -V max...*

I just ran this command as cyrus user on my server (after reading the
manual page). Unfortunately, the "seen" databases in /var/lib/cyrus/user
are still reported by "file" as skiplist, and the "cyrus.cache",
"cyrus.header" and "cyrus.index" in the various (sub)mailboxes, are
still reported as simply "data".

It did create "cyrus.annotations" databases in each subfolder, though
(in twoskip format).

Also,I'm a bit worried. I did see in the logs lines that said:

repacking mailbox user...

and

reconstructing user... 

THOSE TWO LINES ARE PRETTY NORMAL... 

...but also some more worrying lines that said:

uniqueid clash with user... for  - changing user... 

I HAVEN'T EVER SEEN IT CAN'T SAY... I SHOULD TAKE A LOT AT THE CODE
FOR ANSWERING.

Is it something I should worry about ?

Regarding the fact that they're still not in the twoskip format, should
I use cvt_cyrusdb instead ? That would be unfortunate, since I'll have
to create a script fed to the "find" command to mass-convert all
databases; plus, I still don't know what the input format (the "data"
that file talks about) is. 

RECONSTRUCT -R -V MAX SHOULD HANDLE ALL CONVERSIONS...

>> When I mill migrate, will I have to convert the databases through the
>> flat format and back, or can I blindly copy the whole contents of
>> /var/spool/cyrus and /var/lib/cyrus to the new server and expect it to
>> work out of the box ?
>> 
>> *When doing such a migration, it would be better to setup a
>> replication between the 2.4 and the new 2.5 in the hosting. You should
>> encrypt that communication. You could use the own cyrus encription for
>> replication or something like OpenVPN. Although it should work, I
>> wouldn't copy directly (with an rsync or scp) the files.*

Yes, both servers communicate through a VPN, but since both will have
the same Cyrus version, I thought I could just copy the files. Why is it
a bad idea ? 

IT'S NOT A HORRIBLE IDEA... BUT WITH THE REPLICATION YOU CREATE ALL
INDEXES AND SO FROM 0, WITH THE CORRECT VERSION, WITHOUT NEEDING
CONVERSIONS AND WITH THE PROPER VERSION AND, GIVES YOU CLEANER
DATABASES... FOR INSTANCE... AND YOU COULD EVEN COPY CONSTANTLY CONTENT
TO THE DATACENTER FOR JUST FAILING-OVER TO THE DATACENTER WHEN YOU ARE
READY... MOVING TO THE NEW SERVER IS CLEANER AND EASIER... THAT'S WHAT I
WOULD DO :) 

Regards, 

CHEERS! 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Various questions about databases (upgrade and migration)

2019-02-14 Thread Egoitz Aurrekoetxea

Hi Raphaël, 

Answering below in blue for instance..

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 14-02-2019 13:27, Raphaël Halimi escribió:

> Hi,
> 
> I have a small (3 users) Cyrus IMAP server that started as a very old
> Debian (Woody or Sarge, I don't remember exactly) and that I upgraded
> through the years, and yesterday I upgraded from Jessie to Stretch
> (Cyrus 2.4.17 without caldav, to 2.5.10 with caldav).
> 
> As I had some trouble in the past with databases migrations between
> BerkeleyDB versions during some upgrades, and although Cyrus started
> without any serious problem this time (after minor tuning of some
> options in the configuration files), I still wanted to check the state
> of the databases.
> 
> So, the contents of /usr/lib/cyrus/cyrus-db-types.txt is:
> 
> ANNOTATION twoskip
> DBENGINE BerkeleyDB5.3
> DUPLICATE twoskip
> MBOX twoskip
> PTS twoskip
> QUOTA quotalegacy
> SEEN twoskip
> STATUSCACHE twoskip
> SUBS flat
> TLS DEFAULT
> TLS twoskip
> USERDENY flat
> ZONEINFO twoskip
> 
> (note that in Debian, no format is forced in /etc/imapd.conf)
> 
> So my first questions is: I thought that Cyrus had abandoned BerkeleyDB
> a long time ago for skiplist, and then twoskip. Why is it still listed
> in this file ? Is there a part of Cyrus that still uses it ? 
> 
> I SUPPOSE IT COULD SOME DEBIAN STANDARD CONFIG?. WE RUN FREEBSD AND I HAVE 
> NEVER SEE THAT FILE. 
> 
> Now for the databases themselves. In /var/lib/cyrus the global databases
> were converted on-the-fly:
> 
> # file /var/lib/cyrus/*.db
> /var/lib/cyrus/annotations.db:  Cyrus twoskip DB
> /var/lib/cyrus/deliver.db:  Cyrus twoskip DB
> /var/lib/cyrus/mailboxes.db:Cyrus twoskip DB
> /var/lib/cyrus/statuscache.db:  Cyrus twoskip DB
> /var/lib/cyrus/tls_sessions.db: Cyrus twoskip DB
> /var/lib/cyrus/user_deny.db:empty
> 
> However, the user databases were not converted:
> 
> # file /var/lib/cyrus/user/*/*
> /var/lib/cyrus/user/u/user1.seen: Cyrus skiplist DB
> /var/lib/cyrus/user/u/user1.sub:  ASCII text
> /var/lib/cyrus/user/u/user2.seen:   Cyrus skiplist DB
> /var/lib/cyrus/user/u/user2.sub:ASCII text
> /var/lib/cyrus/user/u/user3.seen:   Cyrus skiplist DB
> /var/lib/cyrus/user/u/user3.sub:ASCII text 
> 
> CYRUS 2.4 CONVERTED DATABASES ON THE FLY. CYRUS 2.5 AND NEWER DON'T. YOU 
> SHOULD LAUNCH A "RECONSTRUCT -R -V MAX" FOR THAT PURPOSE.
> 
> So my next questions are: why are the databases still in skiplist
> format, whereas according to /usr/lib/cyrus/cyrus-db-types.txt, they
> should be twoskip ? Why didn't Cyrus convert them on-the-fly like the
> global databases ? Do I have to manually do it myself ? And if I do
> convert them, will it change anything (performance, reliability, etc etc) ? 
> 
> AS SAID PERHAPS IS A DEBIAN DERIVED CONFIG FOR THE PACKAGE. YES YOU SHOULD 
> WITH THE COMMAND ABOVE.
> 
> Also, what about the various databases in the mail directories
> (cyrus.cache, cyrus.header, cyrus.index) ? For most of them, the "file"
> command only reports "data". What format are they actually in ? Do I
> have to convert them too ? 
> 
> SURE... JUST LAUNCH A RECONSTRUCT -R -V MAX...
> 
> My last question is about a planned migration of this home server to a
> hosted private server. This old server and the new one are now both
> Debian Stretch and thus, have the same Cyrus version, but they're not
> the same architecture: the old one is i386, and the new one is amd64.
> 
> When I mill migrate, will I have to convert the databases through the
> flat format and back, or can I blindly copy the whole contents of
> /var/spool/cyrus and /var/lib/cyrus to the new server and expect it to
> work out of the box ? 
> 
> WHEN DOING SUCH A MIGRATION, IT WOULD BE BETTER TO SETUP A REPLICATION 
> BETWEEN THE 2.4 AND THE NEW 2.5 IN THE HOSTING. YOU SHOULD ENCRYPT THAT 
> COMMUNICATION. YOU COULD USE THE OWN CYRUS ENCRIPTION FOR REPLICATION OR 
> SOMETHING LIKE OPENVPN. ALTHOUGH IT SHOULD WORK, I WOULDN'T COPY DIRECTLY 
> (WITH AN RSYNC OR SCP) THE FILES.
> 
> Thanks a lot in advance for your answers.
> 
> Regards, 
> 
> CHEERS!
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus 3.0.8 running :) and feedback of the upgrade

Hi mates! 

I don't really know where the difference is, but in the dev environment,
searches are hugely faster. I know, in dev is not the same as production
in terms of muas traffic and so... but... we are talking in differences
like from 5 seconds to perhaps 40-50 seconds and obviously when saying
this... production is a non loaded env... I mean we have there 4000
accounts, but io is fine, cpu and mem are nice... os limits too
there's no any bottleneck... 

The real difference between my dev env and prod one... is basically the
way they were built. 

FOR DEV I created a copy of one production machine. A copy... for being
able to be the most close as we could, from the prod env in this testing
env. That prod env, was at 2.3. I then, connected to it 2.4-Cyrus root
fs (with Cyrus 2.4). Tested this version and converted to 12 version
some databases (just the ones for testing, the same ones as where I have
seen this speed effect). Later I disconnected the 2.4-Cyrus root disk
and connected a 3.0-Cyrus root disk. Did a reconstruct -r -V max,
ctl_conversationsdb -z on-testing-used-mailboxes and ctl_conversationsdb
-b on-testing-used-mailboxes. 

FOR PROD we connected to 2.3 server the 2.4-Cyrus root disk. Later we
setup an empty slave with 3.0 and started a user by user, user mode
replication there. Later a rolling one (with the file generated between
last "all users, user by user sync" and the moment we started the
rolling one) when almost all in the slave was very near to the present
state of the master in that moment. This replication was done with Csync
not IMAP. 

These too are the basic differences...  could this give you some kind of
clue?... I'll try to debug code too in order to see something 

Cheers!

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 13-02-2019 18:19, Egoitz Aurrekoetxea escribió:

> thanks a lot mate! 
> 
> I'm doing checks... for comparing the previous testing env and live 
> production :) 
> 
> Cheers!
> 
> ---
> 
> EGOITZ AURREKOETXEA 
> Departamento de sistemas 
> 944 209 470
> Parque Tecnológico. Edificio 103
> 48170 Zamudio (Bizkaia) 
> ego...@sarenet.es 
> www.sarenet.es [1] 
> Antes de imprimir este correo electrónico piense si es necesario hacerlo. 
> 
> El 13-02-2019 17:58, Robert Stepanek escribió: 
> On Wed, Feb 13, 2019, at 2:24 PM, Egoitz Aurrekoetxea wrote: 
> 
> Or.. if using in imapd.conf "search_fuzzy_always: 1" isn't it?. 
> 
> Yes, that will instruct imapd to always use FUZZY search for IMAP SEARCH 
> commands [1]. If you use JMAP, it always use fuzzy search (and hence the 
> Xapian backend). 
> 
> [1] https://github.com/cyrusimap/cyrus-imapd/blob/master/imap/imapd.c#L6059

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus 3.0.8 running :) and feedback of the upgrade

thanks a lot mate! 

I'm doing checks... for comparing the previous testing env and live
production :) 

Cheers!

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 13-02-2019 17:58, Robert Stepanek escribió:

> On Wed, Feb 13, 2019, at 2:24 PM, Egoitz Aurrekoetxea wrote: 
> 
>> Or.. if using in imapd.conf "search_fuzzy_always: 1" isn't it?.
> 
> Yes, that will instruct imapd to always use FUZZY search for IMAP SEARCH 
> commands [1]. If you use JMAP, it always use fuzzy search (and hence the 
> Xapian backend). 
> 
> [1] https://github.com/cyrusimap/cyrus-imapd/blob/master/imap/imapd.c#L6059

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus 3.0.8 running :) and feedback of the upgrade

Or.. if using in imapd.conf "search_fuzzy_always: 1" isn't it?. 

If a search returns results... I assume is going through it... isn't
it?. else.. I assume nothing would be found... I really attached too the
configs (in url format of pastebin), https://pastebin.com/CtCEedty  just
for... the cause you could see something missing  there... which by the
way... I don't think something could be missed but I'm new to Xapian :)
:) although have been running Cyrus more than ten years ago :) :) ... 

Cheers!

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 13-02-2019 13:21, Robert Stepanek escribió:

> Hi Egoitz, 
> 
> On Wed, Feb 13, 2019, at 1:10 PM, Egoitz Aurrekoetxea wrote: 
> 
>> Xapian version is xapian-core-1.4.9_1 (FreeBSD port). Could I see by some 
>> debbuging log Xapian is working in the searchs in order to know, that can't 
>> be faster and that all is ok?. At the moment I have tree tiers. But I'm just 
>> with the default one in almost all mailboxes
> 
> If you use FUZZY search in your search expression then it must go through 
> Xapian. 
> 
> Cheers, 
> Robert 
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus 3.0.8 running :) and feedback of the upgrade

Hi mates! 

This has been the great day. We have upgraded a server pair
(master/slave). Almost all has gone fine, apart from the fact that we
have two extrange issues : 

- Some users had found their folders unsubscribed and seems the same
too, lost them Sieve filters. Perhaps due to autocreate or xlist?? which
we weren't previously using¿? 

- I'm not really sure, if Xapian is "actively" working. 

We have upgraded from 2.4 Cyrus to 3.0.8 with the replication method.
First has been finally fixed with a couple of scripts. The second one...
I paste the master/slave config here : https://pastebin.com/CtCEedty I
think it's all ok... and by the way log is saying : 

Feb 13 13:07:19 masterserver squatter[13446]: Xapian committed 1 updates
in 0.146300 sec
Feb 13 13:07:19 masterserver squatter[13446]: Xapian committed 3 updates
in 0.075679 sec
Feb 13 13:07:21 masterserver squatter[13446]: Xapian committed 36
updates in 1.219806 sec
Feb 13 13:07:22 masterserver squatter[13446]: Xapian committed 1 updates
in 0.256899 sec
Feb 13 13:07:22 masterserver squatter[13446]: Xapian committed 5 updates
in 0.165767 sec
Feb 13 13:07:22 masterserver squatter[13446]: Xapian committed 1 updates
in 0.252658 sec
Feb 13 13:07:22 masterserver squatter[13446]: Xapian committed 4 updates
in 0.083638 sec
Feb 13 13:07:24 masterserver squatter[13446]: Xapian committed 4 updates
in 0.952737 sec 

Xapian version is xapian-core-1.4.9_1 (FreeBSD port). Could I see by
some debbuging log Xapian is working in the searchs in order to know,
that can't be faster and that all is ok?. At the moment I have tree
tiers. But I'm just with the default one in almost all mailboxes. 

Thank you so much mates :) :) (as always) 

Cheers! 

-- 

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Removing email from Xapian tier databases

2019-02-11 Thread Egoitz Aurrekoetxea

Many many many many thanks a lot Brong!! :) :) :) :) :) :) :)

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 11-02-2019 15:19, Bron Gondwana escribió:

> It's definitely safe to have one rolling mode writing and one repacking.  I 
> wouldn't run multiple repacks in parallel, as they can wind up doing 
> duplicate work (though the end result should always be correct and safe). 
> 
> Here's what we run: 
> 
> # Any time the disk gets over 50%, compress -o single down to data 
> 13 *  * * * [% INCLUDE cronjob c='/home/mod_perl/hm/scripts/xapian_compact.pl 
> -a -o -d 50 temp data' %] 
> # Copy the temporary search databases down to data during the week 
> 43 1  * * 1,2,3,4,5,6 [% INCLUDE cronjob 
> c='/home/mod_perl/hm/scripts/xapian_compact.pl -a temp,meta data' %] 
> # Sundays repack the entire data directory 
> 43 1  * * 0 [% INCLUDE cronjob c='/home/mod_perl/hm/scripts/xapian_compact.pl 
> -a temp,meta,data data' %] 
> # Late on Sundays, pack any oversized data directories down to archive 
> 0 15 * * 0 [% INCLUDE cronjob c='/home/mod_perl/hm/scripts/xapian_archive.pl 
> -a' %] 
> 
> And here's the interesting logic.  In xapian_compact.pl: 
> 
> if ($Opts{d}) {
> 
> my $Path = $Slot->SearchPath();
> 
> my $Usage = df($Path);
> 
> my $RunUsage = df("/run/cyrus");
> 
> return Process::Status->new(0) if ($Usage->{per} < $Opts{d} and 
> $RunUsage->{per} < $Opts{d});
> 
> }
> 
> my @args = (-z => $dest, -t => $src);
> 
> push @args, '-v' if $Opts{v};
> 
> push @args, '-o' if $Opts{o};
> 
> push @args, '-F' if $Opts{F};
> 
> push @args, '-X' if $Opts{X};
> 
> push @args, ('-T' => $Opts{T}) if $Opts{T};
> 
> push @args, ('-u' => $Opts{u}) if $Opts{u};
> 
> my %RunOpts = (
> 
> PrintOutput => 1,
> 
> );
> 
> $RunOpts{Nice} = 1 unless $Opts{N};
> 
> $RunOpts{Daemon} = 1 if $Opts{D};
> 
> $0 = "xapian_compact: $SN";
> 
> $Slot->RunCommand(\%RunOpts, 'squatter', @args);
> 
> And in xapian_archive.pl: 
> 
> my $Percent = $Opts{P} || 20; 
> [...]
> 
> foreach my $user (sort keys %$DataUsage) {
> 
> my $au = $ArchiveUsage->{$user} || 1;
> 
> my $du = $DataUsage->{$user} || 1;
> 
> if ($du < 5000) {
> 
> print "Too small $user ($du)\n";
> 
> next;
> 
> }
> 
> my $This = int($du * 100 / $au);
> 
> if ($This < $Percent) {
> 
> print "Not enough dirty $user: ($du, $au)\n";
> 
> next;
> 
> }
> 
> print "Recompacting $user: ($du, $au)\n";
> 
> my @args = (-z => 'archive', -t => 'data,archive'); 
> [...]
> 
> In summary, repack data down to archive if data is more than 1/5 size of 
> existing archive.  So each of these scripts is a wrapper around squatter to 
> help it run automatically. 
> 
> Bron. 
> 
> On Mon, Feb 11, 2019, at 21:55, Egoitz Aurrekoetxea wrote: 
> 
> Now I'm noticing for instance, for moving data between Xapian databases.. you 
> need to launch something like :
> 
> sudo -u cyrus /usr/cyrus/bin/squatter -C /usr/local/etc/imapd.conf -v -z 
> archive -t temp,meta,data,archive -u user/ego...@sarenet.es 
> 
> perhaps would be better to do : 
> sudo -u cyrus /usr/cyrus/bin/squatter -C /usr/local/etc/imapd.conf -F -v -z 
> archive -t temp,meta,data,archive -u user/ego...@sarenet.es 
> But then, having two Squatter processes running at same time, one for rolling 
> mode and one for moving/repacking data, should not be an issue?. 
> 
> Thanks mates!! 
> 
> --- 
> 
> EGOITZ AURREKOETXEA 
> Departamento de sistemas 
> 
> 944 209 470 
> Parque Tecnológico. Edificio 103 
> 48170 Zamudio (Bizkaia) 
> ego...@sarenet.es
> 
> www.sarenet.es [1] 
> 
> Antes de imprimir este correo electrónico piense si es necesario hacerlo. 
> 
> El 11-02-2019 11:22, Egoitz Aurrekoetxea escribió: 
> 
> Hi Bron, 
> 
> So, it would be interesting to run once a day... for instance in cyrus.conf 
> in events section : 
> 
> repack_xapian  cmd="squatter -F" at=0200 
> 
> Is it needed top stop the other rolling Squatter we run, in same cyrus.conf 
> as : 
> 
> START { 
> # do not delete this entry! 
> recover   cmd="ctl_cyrusdb -r" 
> 
> squatter cmd="squatter -R" 
> } 
> 
> Thank you so much for all the clarifications mate :) really :) 
> 
> Cheers!
> 
> --- 
> 
> EGOITZ AURREKOETXEA 
> Departamento de sistemas 
> 
> 944 209

Re: Removing email from Xapian tier databases

2019-02-11 Thread Egoitz Aurrekoetxea

Hi mates! 

Just for finishing this thread... Two squatter proccesses then... one in
rolling mode and another one for info movement between Xapian databases
and repacking databases as Brong said... can be running without known
issues?. I say for avoid damaging something... thanks a lot mates! 

Cheers!!

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 11-02-2019 15:12, Bron Gondwana escribió:

> Yep, it's fixed in git now, so the next release will automatically create G 
> keys for messages, even if they don't have a threadid! 
> 
> Bron. 
> 
> On Mon, Feb 11, 2019, at 21:30, Sebastian Hagedorn wrote: 
> 
>> So running ctl_conversationsdb -z followed by -b would assign thread ids to  
>> those messages? Because it works when I do that. Clearly this is an edge  
>> case, but IMO it should be handled somehow other than silently failing ;-) 
>> 
>> --On 11. Februar 2019 um 05:16:47 -0500 Bron Gondwana  
>>  wrote: 
>> 
>>> That sounds like the source messages have no thread id, and hence they 
>>> aren't being stored. 
>>> 
>>> This is an interesting question actually, should we still store G keys 
>>> for messages without thread identifier (CID)? 
>>> 
>>> Bron. 
>>> 
>>> On Mon, Feb 11, 2019, at 21:11, Sebastian Hagedorn wrote: 
>>>> Hi Bron, 
>>>> 
>>>> --On 11. Februar 2019 um 04:23:16 -0500 Bron Gondwana 
>>>>  wrote: 
>>>> 
>>>>> The data in conversations.db is added and removed in real time as 
>>>>> messages are appended and updated in the cyrus.index. 
>>>> 
>>>> do you know why that does not seem to happen when using the "old" sync 
>>>> protocol for replication? 
>>>> 
>>>> <https://github.com/cyrusimap/cyrus-imapd/issues/2376> 
>> --  
>> .:.Sebastian Hagedorn - Weyertal 121 (Gebäude 133), Zimmer 2.02.:. 
>> .:.Regionales Rechenzentrum (RRZK).:. 
>> .:.Universität zu Köln / Cologne University - ✆ +49-221-470-89578.:. 
>>  
>> Cyrus Home Page: http://www.cyrusimap.org/ 
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ 
>> To Unsubscribe: 
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 
> -- 
> Bron Gondwana, CEO, FastMail Pty Ltd 
> br...@fastmailteam.com 
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Removing email from Xapian tier databases

2019-02-11 Thread Egoitz Aurrekoetxea

Now I'm noticing for instance, for moving data between Xapian
databases.. you need to launch something like : 

sudo -u cyrus /usr/cyrus/bin/squatter -C /usr/local/etc/imapd.conf -v -z
archive -t temp,meta,data,archive -u user/ego...@sarenet.es

perhaps would be better to do :

sudo -u cyrus /usr/cyrus/bin/squatter -C /usr/local/etc/imapd.conf -F -v
-z archive -t temp,meta,data,archive -u user/ego...@sarenet.es

But then, having two Squatter processes running at same time, one for
rolling mode and one for moving/repacking data, should not be an issue?.

Thanks mates!!

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 11-02-2019 11:22, Egoitz Aurrekoetxea escribió:

> Hi Bron, 
> 
> So, it would be interesting to run once a day... for instance in cyrus.conf 
> in events section : 
> 
> repack_xapian  cmd="squatter -F" at=0200 
> 
> Is it needed top stop the other rolling Squatter we run, in same cyrus.conf 
> as : 
> 
> START {
> # do not delete this entry!
> recover   cmd="ctl_cyrusdb -r"
> 
> squatter cmd="squatter -R"
> } 
> 
> Thank you so much for all the clarifications mate :) really :) 
> 
> Cheers!
> 
> ---
> 
> EGOITZ AURREKOETXEA 
> Departamento de sistemas 
> 944 209 470
> Parque Tecnológico. Edificio 103
> 48170 Zamudio (Bizkaia) 
> ego...@sarenet.es 
> www.sarenet.es [1] 
> Antes de imprimir este correo electrónico piense si es necesario hacerlo. 
> 
> El 11-02-2019 10:23, Bron Gondwana escribió: 
> Conversations.db is an index over lots of interesting bits of the message, 
> but the key part that's used by Xapian is the mapping from G key (aka: GUID, 
> aka: sha1 of the message RFC822 data) to individual email.  It's used for 
> deduplication and for mapping from results to messages. 
> 
> The data in conversations.db is added and removed in real time as messages 
> are appended and updated in the cyrus.index. 
> 
> The data in the xapian databases on the other hand is append only - so you 
> can wind up with hits that no longer map to existing emails.  The way to 
> solve that is with a xapian repack that filters messages - which can be done 
> using the -F flag to squatter. 
> 
> Cheers, 
> 
> Bron. 
> 
> On Sat, Feb 9, 2019, at 23:04, Egoitz Aurrekoetxea wrote: 
> 
> Good morning, 
> 
> As far as I understood, for Xapian you first create it's conversation 
> database in order to work. Later you create database(s) for each mailbox 
> where Xapian can search in. You can move data between them, new mails become 
> indexed for instance Squatter in rolling mode... that's ok... and understood 
> I  think. I was wondering, what happens when mail indexed in the archive 
> database in removed and then does not exist any more in the database... does 
> Squatter rolling log manage that too?. 
> 
> By the way. I was wondering if mail gets indexed in the tier databases (for 
> instance in Fastmail in temp, meta, data, archine...) what's the role or 
> function of conversations databases you create with ctl_conversationsdb -b -r 
> ?. 
> 
> Cheers!
> 
> -- 
> 
> EGOITZ AURREKOETXEA 
> Departamento de sistemas 
> 
> 944 209 470 
> Parque Tecnológico. Edificio 103 
> 48170 Zamudio (Bizkaia) 
> ego...@sarenet.es
> 
> www.sarenet.es [1] 
> 
> Antes de imprimir este correo electrónico piense si es necesario hacerlo. 
>  
> Cyrus Home Page: http://www.cyrusimap.org/ 
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ 
> To Unsubscribe: 
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus 
> 
> -- 
> Bron Gondwana, CEO, FastMail Pty Ltd 
> br...@fastmailteam.com 
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Removing email from Xapian tier databases

2019-02-09 Thread Egoitz Aurrekoetxea

Good morning, 

As far as I understood, for Xapian you first create it's conversation
database in order to work. Later you create database(s) for each mailbox
where Xapian can search in. You can move data between them, new mails
become indexed for instance Squatter in rolling mode... that's ok... and
understood I  think. I was wondering, what happens when mail indexed in
the archive database in removed and then does not exist any more in the
database... does Squatter rolling log manage that too?. 

By the way. I was wondering if mail gets indexed in the tier databases
(for instance in Fastmail in temp, meta, data, archine...) what's the
role or function of conversations databases you create with
ctl_conversationsdb -b -r ?. 

Cheers!

-- 

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Question about conversationsdb

2019-02-08 Thread Egoitz Aurrekoetxea

In the replica I assume sync_log should be off. But If I enable there
sync_log_unsuppressable_channels: squatter then I assume all Xapian
searches needing elements would be generated when the slave received
data from the master?. Even when the slave starts empty?. 

Thanks mates :)

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 08-02-2019 14:46, Egoitz Aurrekoetxea escribió:

> Hi!, 
> 
> One little question. If I wanted to upgrade a Cyrus server older then 3.0 but 
> greater than 2.3 (so replication is compatible) the conversations database 
> would be created automatically in the slave if I used an empty slave 3.0 and 
> I replicated the whole content there from the master running 2.4 for instance 
> (and then no Xapian searches or conversations for instance)?. 
> 
> Thanks a lot for your time :) 
> 
> Cheers!
> 
> -- 
> 
> EGOITZ AURREKOETXEA 
> Departamento de sistemas 
> 944 209 470
> Parque Tecnológico. Edificio 103
> 48170 Zamudio (Bizkaia) 
> ego...@sarenet.es 
> www.sarenet.es [1] 
> Antes de imprimir este correo electrónico piense si es necesario hacerlo. 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Big problem with replication

2019-01-16 Thread Egoitz Aurrekoetxea

Good afternoon, 

I would try doing it user by user (with -u). This way you would have all
synced except the problematic mailbox. 

Cheers!

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 16-01-2019 16:15, Albert Shih escribió:

> Hi everyone.
> 
> I've got some big issue with replication.
> 
> I've 
> 
> master --- replica ---> slave_1 --- replica ---> slave_2
> 
> The replication between master and slave_1 work nice. 
> 
> Between slave_1 and slave_2 I've got some issue (log to big after network
> failure and work nagios_supervision). 
> 
> So now I'm trying to build a new slave_3 to replace slave_2. And I'm unable
> to launch sync_client. Each time I try to manually launch I got 
> 
> [root@imap-mirror-p /bals/DELETED]# /usr/local/cyrus/sbin/sync_client -S 
> slave_3 -A -v
> MAILBOXES DELETED.DIO.5AEAD6F9
> Error from do_user(DELETED.DIO.5AEAD6F9): bailing out!
> [root@imap-mirror-p /bals/DELETED]# 
> 
> and the DIO folder don't event exist
> 
> [root@imap-mirror-p /bals/DELETED]# ls DIO
> ls: DIO: No such file or directory
> [root@imap-mirror-p /bals/DELETED]# 
> 
> Any help would be *very* welcome ;-) 
> 
> Regards.
> 
> --
> Albert SHIH
> DIO bâtiment 15
> Observatoire de Paris
> Heure local/Local time:
> Wed Jan 16 16:08:47 CET 2019
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Help with few questions

2019-01-09 Thread Egoitz Aurrekoetxea

Hi Sebastian!! 

Thank you so much mate!!. Point 2 is the most uncertain for me. I answer
your comments below :) 

Point 1 -> It's clear... yes I think so too... I was planning to force
all searches as FUZZY with search_fuzzy_always: 1 as it seems really
nice :) 

Point 2 -> So, you do a df every two minutes to see free space in
tmpfs?... and when you see it starts growing you launch a "sudo -u cyrus
/usr/cyrus/bin/squatter -C /etc/cyrus/imapd-sloti30t01.conf -v -z
archive -t temp,meta,data,archive -u brong" equivalent command?. You
would launch it without -u I suppose, for doing Squatter for all users
(no sense of doing it for just one user... isn't it?)... I assume this
command would move all items from temp, meta and data to archive
database. So, for instance if you launch a "sudo -u cyrus
/usr/cyrus/bin/squatter -C /etc/cyrus/imapd-sloti30t01.conf -v -z data
-t temp,meta,data -u brong" it moves (and then empties temp and meta)
temp and meta to data and even compacts the own data database?. We
usually have 2000 imap concurrent connections, 150 concurrent pop3 and
receive 120 messages/min more or less... and SSD array of disks yes..
for each server... 

Point 3 -> Totally agree. Just for confirming. 

Point 4 -> If having a crash in a server... the Cyrus log file, would
throw which mailboxes need a ctl_conversations -z and -b or "it's
better" doing it for all accounts (with -r and without -u?)?. When
talking about Xapian databases (not conversations) I assume some error
would appear for them surely... the same way as for instance if
cyrus.seen would be corrupt... isn't it?. 

Thanks a lot again. Your help is really really important for at this
stage... 

 Cheers!!

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 09-01-2019 12:59, Sebastian Hagedorn escribió:

> Hi,
> 
> I will try to answer what I can ... see below.
> 
> --On 8. Januar 2019 um 18:40:17 +0100 Egoitz Aurrekoetxea  
> wrote:
> 
>> - search_fuzzy_always: 1 , causes all searches to go through Xapian
>> engine. Being so good as it seems (and the way it speeds in my testing
>> env search operations, it's nice!!), what could be the reason for not
>> having it enabled by default?. Can it have some kind of problem?. I
>> can't see them... Just for avoiding surprises.
> 
> With FUZZY you may get more matches than without. Look here for an 
> explanation:
> 
> <https://tools.ietf.org/html/rfc6203>
> 
> The example in 3. is a good one:
> 
> 3.  The FUZZY Search Key
> 
> The FUZZY search key takes another search key as its argument.  The
> server is allowed to perform all matching in an implementation-
> defined manner for this search key, including ignoring the active
> comparator as defined by [RFC5255].  Typically, this would be used to
> search for strings.  For example:
> 
> C: A1 SEARCH FUZZY (SUBJECT "IMAP break")
> S: * SEARCH 1 5 10
> S: A1 OK Search completed.
> 
> Besides matching messages with a subject of "IMAP break", the above
> search may also match messages with subjects "broken IMAP", "IMAP is
> broken", or anything else the server decides that might be a good
> match.
> 
> Note that the *server* decides what "might be a good match". When all 
> searches become FUZZY it might confuse users, but on the other hand I doubt 
> there is a single IMAP client that lets the user choose whether a particular 
> search should be FUZZY or not ...
> 
>> - Bron, in this post
>> https://fastmail.blog/2014/12/01/email-search-system/ told that Fastmail
>> was not able to handle with just one Xapian default database (even on
>> SSD disks) all traffic. So he said Fastmail was using a in-memory
>> filesystem for a database (called temp) for new email. Later another for
>> cleaning up that in memory filesystem. And later one more, for keeping
>> definitively the content. You seemed to use a Squatter command for
>> moving elements between databases. Concretely (sudo -u cyrus
>> /usr/cyrus/bin/squatter -C /etc/cyrus/imapd-sloti30t01.conf -v -z
>> archive -t temp,meta,data,archive -u brong). I assume that compacts all
>> elements from all databases to archive?. If I wanted to compact elements
>> from temp to data, the command should be "sudo -u cyrus
>> /usr/cyrus/bin/squatter -C /etc/cyrus/imapd-sloti30t01.conf -v -z data
>> -t temp -u brong" (in this example for user brong) ??. I assume you
>> launch something like it with a cron weekly and it's done?.
> 
> That depends entirely on your system. Without knowing the number of us

Help with few questions

2019-01-08 Thread Egoitz Aurrekoetxea

Good afternoon,

I would be very thankful if you could clarify me these questions. Have
not been able to find documentation for them and my testing env has not
been able too :) of clarifying me these doubts... Here I go :)

- Some friendly mate, has told me here that when you enable Xapian
engine in Cyrus 3.0 and have a master/slave replication, Squatter in
rolling mode should be enabled in both, the master and the slave. I
understand the master should be using it for indexing with Xapian the
new mail entering in mailboxes, mail movement... and in the slave for
applying the Xapian transactions being applied and sent from the master.
But, how can be that having the same line in events, depending in
whether it's master or slave does what just should do?. The sync_log: on
and sync_log_channels: squatter should be too enabled in both master or
slave?. Or just in one of them?. I haven't never run a Cyrus replication
of Xapian databases.. so this is new for me and don't understand how all
this param behave and if should be enabled in both master and slave. I
would like to use rolling Squatter in the master and that changes to be
sent to the slave. 

- search_fuzzy_always: 1 , causes all searches to go through Xapian
engine. Being so good as it seems (and the way it speeds in my testing
env search operations, it's nice!!), what could be the reason for not
having it enabled by default?. Can it have some kind of problem?. I
can't see them... Just for avoiding surprises.

- Bron, in this post
https://fastmail.blog/2014/12/01/email-search-system/ told that Fastmail
was not able to handle with just one Xapian default database (even on
SSD disks) all traffic. So he said Fastmail was using a in-memory
filesystem for a database (called temp) for new email. Later another for
cleaning up that in memory filesystem. And later one more, for keeping
definitively the content. You seemed to use a Squatter command for
moving elements between databases. Concretely (sudo -u cyrus
/usr/cyrus/bin/squatter -C /etc/cyrus/imapd-sloti30t01.conf -v -z
archive -t temp,meta,data,archive -u brong). I assume that compacts all
elements from all databases to archive?. If I wanted to compact elements
from temp to data, the command should be "sudo -u cyrus
/usr/cyrus/bin/squatter -C /etc/cyrus/imapd-sloti30t01.conf -v -z data
-t temp -u brong" (in this example for user brong) ??. I assume you
launch something like it with a cron weekly and it's done?. 

- If something went wrong when the upgrade proccess from 2.4 to 3.0,
could I setup 3.0 as master of the 2.4 and later make 2.4 master again?.
Could that cause info loosing?. I assume yes but just 
for knowing posibilities.

- When a Xapian or conversation index becomes broken, a reconstruct
could recover?. What could be the repairing procedure?. 

Thank you so much mates. Sorry for having so many questions... but
neither testing nor documentation or mailing lists have been able to
clarify me them. 

Best regards,

-- 

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Question about manual replication (-u )

2019-01-08 Thread Egoitz Aurrekoetxea

Thanks a lot Bron!!! :) :)

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 08-01-2019 12:02, Bron Gondwana escribió:

> Yep, that's totally safe. Even doing the same user twice at the same time 
> should be safe, though it may do extra work. 
> 
> Bron. 
> 
> On Tue, Jan 8, 2019, at 05:05, Egoitz Aurrekoetxea wrote: 
> 
> Good afternoon, 
> 
> I know it seems a pretty stupid question, but some time ago, you cannot have 
> a Cyrus server acting for instance as a master and as slave... it was not 
> supported... worked... but not supported... so having multiple sync_client 
> instances... perhaps could damage something (although I suppose not) I'll 
> probably have the same dount for ctl_conversations -z and -b multiple 
> parallel commands... 
> 
> I know it seems ridiculous... but I ask it because I prefer to try get some 
> knowledge from Cyrus gurus... :) :) 
> 
> Cheers!
> 
> --- 
> 
> EGOITZ AURREKOETXEA 
> Departamento de sistemas 
> 
> 944 209 470 
> Parque Tecnológico. Edificio 103 
> 48170 Zamudio (Bizkaia) 
> ego...@sarenet.es
> 
> www.sarenet.es [1] 
> 
> Antes de imprimir este correo electrónico piense si es necesario hacerlo. 
> 
> El 03-01-2019 16:32, Egoitz Aurrekoetxea escribió: 
> 
> Good afternoon, 
> 
> Is it possible to launch several instances of 
> "/usr/local/cyrus/bin/sync_client -S DEST-HOST -v -u EMAIL" in parallel?. 
> Doing it just one mailbox at a time takes ages It would help me a lot, 
> the fact of parallelizing and have no disk bottleneck issues 
> 
> I think it should be possible... isn't it?. Perhaps it just allowed between 
> same version in source and dest?. Or can be done for instance too, with a 2.4 
> as master to 3.0 slave?. 
> 
> Cheers!!
> 
> -- 
> 
> EGOITZ AURREKOETXEA 
> Departamento de sistemas 
> 
> 944 209 470 
> Parque Tecnológico. Edificio 103 
> 48170 Zamudio (Bizkaia) 
> ego...@sarenet.es
> 
> www.sarenet.es [1] 
> 
> Antes de imprimir este correo electrónico piense si es necesario hacerlo. 
>  
> Cyrus Home Page: http://www.cyrusimap.org/ 
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ 
> To Unsubscribe: 
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

-- 
  Bron Gondwana, CEO, FastMail Pty Ltd 
  br...@fastmailteam.com 


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Xapian searches of the body of an email

2019-01-08 Thread Egoitz Aurrekoetxea

Hi Robert! 

Thank you so much mate :) :) 

Best regards,

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 08-01-2019 09:26, Robert Stepanek escribió:

> Hi, 
> 
> On Mon, Jan 7, 2019, at 7:32 PM, Egoitz Aurrekoetxea wrote: 
> 
>> So, if I run Squatter in the master in rolling mode... then I assume there's 
>> no need to launch manually squatter command in the master... isn't it?.
> 
> If you indexed all messages in the mailbox before, then there shouldn't be a 
> need to manually run squatter afterwards. The rolling squatter should pick up 
> and index all newly created messages. 
> 
>> I'm planning to upgrade some 2.3 running machines to either 2.4 or 3.0 
>> and you know... is a big responsability... am doing tons of sandbox 
>> testing... is it known (as Michael stated) not to be working traditional 
>> squatter in 3.0 if you don't want to use now the Xapian engine?.
> 
> Sorry, I have no experience with the upgrades from version 2. 
> 
> Cheers, 
> Robert
 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Xapian searches of the body of an email

Hi Robert! 

I see! I though you only needed to run Squatter in rolling mode in the
slave.. I though the roling mode was just for slaves to take the
Squatter changes caused by a normal squatter command launched in the
master... So, if I run Squatter in the master in rolling mode... then I
assume there's no need to launch manually squatter command in the
master... isn't it?. 

I'm planning to upgrade some 2.3 running machines to either 2.4 or
3.0 and you know... is a big responsability... am doing tons of
sandbox testing... is it known (as Michael stated) not to be working
traditional squatter in 3.0 if you don't want to use now the Xapian
engine?. 

I'll read the post right now :) :) 

Thank you so much for all your help mate :) 

Cheers!

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 07-01-2019 19:24, Robert Stepanek escribió:

> Hi Egon, 
> 
> Yes, the slave should index in conversations.db automatically AFAIK.  
> 
> You should run squatter in rolling mode on the master, too.   
> 
> BTW: in 2014, Bron wrote a blog post about the search setup at FastMail: 
> https://fastmail.blog/2014/12/01/email-search-system/ 
> It's quite technical, but should give you a good idea at how it's set up for 
> fast indexing and search  
> 
> Cheers, Robert  
> 
> On Mon, Jan 7, 2019, at 5:54 PM, Egoitz Aurrekoetxea wrote: 
> 
> Hi Robert! 
> 
> Thank you so much for helping us (mainly which is the one boring the list 
> with questions :) although I promise I've checked the doc before asking :) :) 
>  ). 
> 
> When you have a master/slave config... in the slave one, when running 
> Squatter in rolling mode... does it update the conversations db too?. By the 
> way, Squatter in rolling mode only makes sense in slave machines isn't it?. 
> 
> Many thanks! 
> 
> --- 
> 
> EGOITZ AURREKOETXEA 
> Departamento de sistemas 
> 
> 944 209 470 
> Parque Tecnológico. Edificio 103 
> 48170 Zamudio (Bizkaia) 
> ego...@sarenet.es
> 
> www.sarenet.es [1] 
> 
> Antes de imprimir este correo electrónico piense si es necesario hacerlo. 
> 
> El 07-01-2019 16:42, Robert Stepanek escribió: 
> Hi, 
> 
> Sebastian is right: 
> 
> On Mon, Jan 7, 2019, at 3:57 PM, Sebastian Hagedorn wrote: 
> 
> squatter is nowadays a bit of a misnomer, because it uses whatever index 
> you have configured. In cyrus 2.4, squatter would always create a SQUAT 
> index. When you run squatter with Xapian, it will build the index, but for 
> the index to actually work, you also need the conversationsdb. 
> 
> conversations.db is indeed a misnomer now. The database was only used to keep 
> track of mail threads (hence the name), but its role expanded. One of the 
> indexes it stores is the SHA1 hashes of every message, and separate hashes 
> for each of that message MIME parts. Such a hash is named the GUID, and for 
> each GUID we store a list of all mailbox:UID[bodypart] pairs where this 
> content occurs in. 
> 
> For search, we keep track of the indexed messages by GUID, so we can avoid 
> reindexing duplicate mails. To return a search result, we can now map that 
> GUID back to its mailbox:message pairs. That's why we need conversations.db 
> for search. 
> 
> I can't help with upgrading from 2.4, unfortunately, but if you re-index your 
> mailboxes once in conversations.db, you should be all set. 
> 
> Cheers, 
> Robert 
> 
>  
> Cyrus Home Page: http://www.cyrusimap.org/ 
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ 
> To Unsubscribe: 
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

  

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Question about manual replication (-u )

Good afternoon, 

I know it seems a pretty stupid question, but some time ago, you cannot
have a Cyrus server acting for instance as a master and as slave... it
was not supported... worked... but not supported... so having multiple
sync_client instances... perhaps could damage something (although I
suppose not) I'll probably have the same dount for ctl_conversations
-z and -b multiple parallel commands... 

I know it seems ridiculous... but I ask it because I prefer to try get
some knowledge from Cyrus gurus... :) :) 

Cheers!

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 03-01-2019 16:32, Egoitz Aurrekoetxea escribió:

> Good afternoon, 
> 
> Is it possible to launch several instances of 
> "/usr/local/cyrus/bin/sync_client -S DEST-HOST -v -u EMAIL" in parallel?. 
> Doing it just one mailbox at a time takes ages It would help me a lot, 
> the fact of parallelizing and have no disk bottleneck issues 
> 
> I think it should be possible... isn't it?. Perhaps it just allowed between 
> same version in source and dest?. Or can be done for instance too, with a 2.4 
> as master to 3.0 slave?. 
> 
> Cheers!!
> 
> -- 
> 
> EGOITZ AURREKOETXEA 
> Departamento de sistemas 
> 944 209 470
> Parque Tecnológico. Edificio 103
> 48170 Zamudio (Bizkaia) 
> ego...@sarenet.es 
> www.sarenet.es [1] 
> Antes de imprimir este correo electrónico piense si es necesario hacerlo.
 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Xapian searches of the body of an email

Hi Robert! 

Thank you so much for helping us (mainly which is the one boring the
list with questions :) although I promise I've checked the doc before
asking :) :)  ). 

When you have a master/slave config... in the slave one, when running
Squatter in rolling mode... does it update the conversations db too?. By
the way, Squatter in rolling mode only makes sense in slave machines
isn't it?. 

Many thanks! 

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 07-01-2019 16:42, Robert Stepanek escribió:

> Hi, 
> 
> Sebastian is right: 
> 
> On Mon, Jan 7, 2019, at 3:57 PM, Sebastian Hagedorn wrote: 
> 
>> squatter is nowadays a bit of a misnomer, because it uses whatever index 
>> you have configured. In cyrus 2.4, squatter would always create a SQUAT 
>> index. When you run squatter with Xapian, it will build the index, but for 
>> the index to actually work, you also need the conversationsdb.
> 
> conversations.db is indeed a misnomer now. The database was only used to keep 
> track of mail threads (hence the name), but its role expanded. One of the 
> indexes it stores is the SHA1 hashes of every message, and separate hashes 
> for each of that message MIME parts. Such a hash is named the GUID, and for 
> each GUID we store a list of all mailbox:UID[bodypart] pairs where this 
> content occurs in. 
> 
> For search, we keep track of the indexed messages by GUID, so we can avoid 
> reindexing duplicate mails. To return a search result, we can now map that 
> GUID back to its mailbox:message pairs. That's why we need conversations.db 
> for search. 
> 
> I can't help with upgrading from 2.4, unfortunately, but if you re-index your 
> mailboxes once in conversations.db, you should be all set. 
> 
> Cheers, 
> Robert 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Xapian searches of the body of an email

Hi Sebastian, 

I'll answer below (and in green) your answers!! Thanks a lot for your
explanations mate :) :)

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 07-01-2019 15:57, Sebastian Hagedorn escribió:

> Hi,
> 
> --On 7. Januar 2019 um 14:51:02 +0100 Egoitz Aurrekoetxea  
> wrote:
> 
>> This seems to take ages...
> 
> why don't you run it for a single account first, to make sure that it 
> actually helps? 
> 
> ==
>  
> 
> YES :) IT HAS WORKED AS EXPECTED FOR A SINGLE ACCOUNT! HAVE LAUNCHED LATER 
> THE : 
> 
> /USR/LOCAL/CYRUS/SBIN/CTL_CONVERSATIONSDB -R -Z 
> 
> (STILL RUNNING) 
> 
> AND LATER THE : 
> 
> /USR/LOCAL/CYRUS/SBIN/CTL_CONVERSATIONSDB -R -B 
> 
> ==
>  
> 
>> I'm trying to figure the best way of
>> implementing this and of clarifying concepts I'm running Squatter in
>> rolling replication mode and exist the concept of conversations then.
>> What is the exact role of each of them?. Squatter seems to index the
>> mailbox but when something is not properly indexed instead of running
>> Squatter you use ctl_conversations for reindexing some part again or...
> 
> squatter is nowadays a bit of a misnomer, because it uses whatever index you 
> have configured. In cyrus 2.4, squatter would always create a SQUAT index. 
> When you run squatter with Xapian, it will build the index, but for the index 
> to actually work, you also need the conversationsdb. squatter does not touch 
> the conversationsdb! The index is only a pointer to the conversationsdb, not 
> to actual messages. 
> 
> ==
>  
> 
> I SEE!! THIS WAS THE KEY POINT I WAS LOOSING OR WHICH I WAS NOT ABLE TO SEE 
> HOW ALL OF THEM WHERE RELATED... 
> 
> ==
>  
> 
> Robert can probably explain this much better than me, but I think the problem 
> is the following:
> 
> * when you have conversations enabled in imapd.conf, normal deliveries to the 
> mailboxes (e.g. using lmtpd) will update the conversationsdb
> 
> * syncing (at least using the "old" mechanism, not sure about sync between 
> instances running 3.0)  does *not* update the conversationsdb 
> 
> ==
>  
> 
> ALTHOUGH I ASSUME, HERE IF YOU RUN THE SQUATTER IN ROLLING MODE WITHIN A 
> SLAVE SERVER SEEMS TO UPDATE THE INDEX... ALTHOUGH NOT SURE IF IT COULD DO 
> TOO WITH CONVERSATIONSDB.. HOW DOES A SLAVE BEHAVE IN THIS SENSE?. COULD 
> ANYONE HELP US PLEASE :) ?? 
> 
> ==
>  
> 
> Once you have a running 3.0 server, you probably won't need to run 
> ctl_conversationsdb ever again. But when you are at the stage of syncing mail 
> from 2.4 to 3.0, you *will* need to rebuild each user's conversationdb at 
> least once, after you have finished with syncing that user. 
> 
> ==
>  
> 
> THIS IS IMPORTANT YES... 
> 
> ==
>  
> 
> Again, this is all based on my understanding and not an official answer. 
> 
> ==
>  
> 
> ANY I THANK A LOT ALL YOUR HELP!! MANY MANY THANKS MATE!! 
> 
> CHEERS!! 
> 
> ==
> 
> El 07-01-2019 10:19, Sebastian Hagedorn escribió:
> 
> That sounds like the conversationsdb issue I was talking about. Have you
> tried these steps?
> 
> ctl_conversationsdb -z USER
> ctl_conversationsdb -b USER

Mit freundlichen Grüßen

Sebastian Hagedorn 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Xapian searches of the body of an email

Sorry for asking again.. 

ctl_conversationsdb -z and -b (with -r I assume for all users) should be
run only on new user accounts or... periodically for any user account?.
Or does squatter maintain too the conversations database?. 

Best regards,

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 07-01-2019 15:05, Egoitz Aurrekoetxea escribió:

> And, by the way 
> 
> when using Squatter instead of Xapian as a search engine what do we 
> really lost?. Just the fact of having a statistical worse results?. Is it 
> Xapian faster than squat engine?. 
> 
> Sorry for having so many questions but... I suppose I don't have the 
> implications of each one totally clear :) 
> 
> Cheers!
> 
> ---
> 
> EGOITZ AURREKOETXEA 
> Departamento de sistemas 
> 944 209 470
> Parque Tecnológico. Edificio 103
> 48170 Zamudio (Bizkaia) 
> ego...@sarenet.es 
> www.sarenet.es [1] 
> Antes de imprimir este correo electrónico piense si es necesario hacerlo. 
> 
> El 07-01-2019 14:51, Egoitz Aurrekoetxea escribió: 
> 
> Hi mate! 
> 
> This seems to take ages... I'm trying to figure the best way of implementing 
> this and of clarifying concepts I'm running Squatter in rolling 
> replication mode and exist the concept of conversations then. What is the 
> exact role of each of them?. Squatter seems to index the mailbox but when 
> something is not properly indexed instead of running Squatter you use 
> ctl_conversations for reindexing some part again or... 
> 
> Thanks a lot!!
> 
> ---
> 
> EGOITZ AURREKOETXEA 
> Departamento de sistemas 
> 944 209 470
> Parque Tecnológico. Edificio 103
> 48170 Zamudio (Bizkaia) 
> ego...@sarenet.es 
> www.sarenet.es [1] 
> Antes de imprimir este correo electrónico piense si es necesario hacerlo. 
> 
> El 07-01-2019 10:19, Sebastian Hagedorn escribió: 
> That sounds like the conversationsdb issue I was talking about. Have you 
> tried these steps?
> 
> ctl_conversationsdb -z USER
> ctl_conversationsdb -b USER
> 
> I have been testing Xapian searches. Have seen, it's not able to find
> strings inside the body of the email. If I set in imap.conf
> "search_fuzzy_always: 1" no content is displayed in the searches of a
> Roundcube stock webmail. If I remove that config value from imap.conf
> and restart services, then search results appear. Does Xapian not index
> the body of emails?. Does Xapian, just index the headers?. But this
> affirmation does not seem to be possible in my case too... as I have in
> the config "search_index_headers: no".
> 
> I'm using the following config :
> 
> conversations: 1
> search_engine: xapian
> search_index_headers: no
> search_batchsize: 8192
> defaultsearchtier: t1
> t1searchpartition-default: /expert/search
> t1searchpartition-expert2: /expert2/search
> t1searchpartition-expert3: /expert3/search
> 
> Could anyone help me mates?. 
> 
> --
> Sebastian Hagedorn - Weyertal 121, Zimmer 2.02
> Regionales Rechenzentrum (RRZK)
> Universität zu Köln / Cologne University - Tel. +49-221-470-89578
 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Xapian searches of the body of an email

Hi mate! 

This seems to take ages... I'm trying to figure the best way of
implementing this and of clarifying concepts I'm running Squatter in
rolling replication mode and exist the concept of conversations then.
What is the exact role of each of them?. Squatter seems to index the
mailbox but when something is not properly indexed instead of running
Squatter you use ctl_conversations for reindexing some part again or... 

Thanks a lot!!

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 07-01-2019 10:19, Sebastian Hagedorn escribió:

> That sounds like the conversationsdb issue I was talking about. Have you 
> tried these steps?
> 
> ctl_conversationsdb -z USER
> ctl_conversationsdb -b USER
> 
>> I have been testing Xapian searches. Have seen, it's not able to find
>> strings inside the body of the email. If I set in imap.conf
>> "search_fuzzy_always: 1" no content is displayed in the searches of a
>> Roundcube stock webmail. If I remove that config value from imap.conf
>> and restart services, then search results appear. Does Xapian not index
>> the body of emails?. Does Xapian, just index the headers?. But this
>> affirmation does not seem to be possible in my case too... as I have in
>> the config "search_index_headers: no".
>> 
>> I'm using the following config :
>> 
>> conversations: 1
>> search_engine: xapian
>> search_index_headers: no
>> search_batchsize: 8192
>> defaultsearchtier: t1
>> t1searchpartition-default: /expert/search
>> t1searchpartition-expert2: /expert2/search
>> t1searchpartition-expert3: /expert3/search
>> 
>> Could anyone help me mates?.
> 
> --
> Sebastian Hagedorn - Weyertal 121, Zimmer 2.02
> Regionales Rechenzentrum (RRZK)
> Universität zu Köln / Cologne University - Tel. +49-221-470-89578
 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Xapian searches of the body of an email

Good morning, 

I have been testing Xapian searches. Have seen, it's not able to find
strings inside the body of the email. If I set in imap.conf
"search_fuzzy_always: 1" no content is displayed in the searches of a
Roundcube stock webmail. If I remove that config value from imap.conf
and restart services, then search results appear. Does Xapian not index
the body of emails?. Does Xapian, just index the headers?. But this
affirmation does not seem to be possible in my case too... as I have in
the config "search_index_headers: no". 

I'm using the following config : 

conversations: 1
search_engine: xapian
search_index_headers: no
search_batchsize: 8192
defaultsearchtier: t1
t1searchpartition-default: /expert/search
t1searchpartition-expert2: /expert2/search
t1searchpartition-expert3: /expert3/search 

Could anyone help me mates?. 

Best regards,

-- 

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Xapian searches in Cyrus 3.0

Hi Sebastian! 

I suppose I won't have problems when upgrading, because I'll do a 2.4 in
place upgrade (with no xapian, no indexing) and later a replication to a
3.0 Cyrus with Xapian enabled, where all indexes due to this sync should
being generated (it seems logs say that at least...). Is Xapian search
slower than "previous way" in some situation?. When talking about a
fallback, I meant you know, running Squatter without being in rolling
mode and behaving and indexing the way before Xapian appeared... which I
suppose it would be used berkeley or similar, but have seen no berkeley
support is present in last Cyrus versions so... don't really know which
database backend where used... I suppose to some other database type

Thanks mate!!!

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 03-01-2019 16:37, Sebastian Hagedorn escribió:

> --On 3. Januar 2019 um 16:08:44 +0100 Robert Stepanek  
> wrote:
> 
> On Thu, Jan 3, 2019, at 3:55 PM, Egoitz Aurrekoetxea wrote: I was planning to 
> perform mailboxes squattering in rolling mode. Have
> you had some kind of expience on searching with Xapian and IMAP
> protocol?. Perhaps is more designed for JMAP?. Or perhaps, using it
> with IMAP is not so advantageous?. Both IMAP and JMAP search share the same 
> core search API in Cyrus, so
> all search features available in JMAP can also be used using IMAP. The
> key is to use FUZZY search for text search, and this can be an advantage
> especially for non-English search over the other search backends.

A few additional comments:

With the setting "search_fuzzy_always: 1" all searches use Xapian.
Otherwise searches from clients that don't use FUZZY run completely
without a search index. There is no fallback mechanism to squatter. You
also need to be careful with the conversationsdb. In my experience the
DB is not updated when syncing from a 2.4 server to a 3.0 server. That
means that you need ro rebuild each user's conversationdb manually using
ctl_conversationsdb. 

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Xapian searches in Cyrus 3.0

Thank you so much Robert Many many thanks!!! 

Cheers!

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 03-01-2019 16:08, Robert Stepanek escribió:

> Hello, 
> 
> On Thu, Jan 3, 2019, at 3:55 PM, Egoitz Aurrekoetxea wrote: 
> 
>> I was planning to perform mailboxes squattering in rolling mode. Have you 
>> had some kind of expience on searching with Xapian and IMAP protocol?. 
>> Perhaps is more designed for JMAP?. Or perhaps, using it with IMAP is not so 
>> advantageous?.
> 
> Both IMAP and JMAP search share the same core search API in Cyrus, so all 
> search features available in JMAP can also be used using IMAP. The key is to 
> use FUZZY search for text search, and this can be an advantage especially for 
> non-English search over the other search backends. 
> 
> The complete list of attributes to search is in this C file: 
> https://github.com/cyrusimap/cyrus-imapd/blob/master/imap/search_expr.c#L2235 
> Any attribute that is marked SEA_FUZZABLE  will get stored in Xapian and is 
> available for stem-based search. 
> 
> I'm not sure if there's better documentation on the non-standard fields, but 
> if you have any questions feel free to ask! 
> 
> Cheers, 
> Robert 
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Xapian searches in Cyrus 3.0

Good afternoon, 

I was planning to perform mailboxes squattering in rolling mode. Have
you had some kind of expience on searching with Xapian and IMAP
protocol?. Perhaps is more designed for JMAP?. Or perhaps, using it with
IMAP is not so advantageous?. How your experiences had been in this
sense, with this technology?. Is better to use a non Xapian system?.
Just, let's say the old way of Squatter without Xapian?. 

Best regards,

-- 

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Question for upgrading

Hi Javier!, 

Sorry for the delay and many many thanks for your ideas :) 

Cheers!

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 18-12-2018 13:50, Javier Angulo escribió:

> On 12/18/18 12:15 PM, Binarus wrote: On 15.12.2018 17:05, Nic Bernstein 
> wrote: On 12/13/18 9:52 AM, Egoitz Aurrekoetxea wrote: I was trying to 
> upgrade part of our Cyrus imap installation,
> concretely that one consisting in still 2.3. I was planning to set up
> Cyrus 3.0. I have seen all works properly except for the unexpunge
> command because as someone stated here, a reconstruct -V max was
> needed.The problem is that this reconstruct command, takes ages and
> I'm not able to keep the service offline so many time. So I have been
> thinking in the following scenario :
> Egoitz,
> As long as you've followed all of the various steps needed to account
> for version changes, as outlined in the Release Notes for /all/
> intermediary releases, then the last step should be the updating of the
> indexes.  Rather than running "reconstruct -V max" on all mailboxes at
> once, simply run it on subsets.  We've done this on large installations
> without ill effect.  You can have the service on line during the
> reconstruct, and, as you note, have most all functionality present.
 In my case, before the reconstruct had finished, I had several problems
which might be not acceptable for large organizations.

For example, users could not move messages between folders in their
mailbox. I would consider this quite basic functionality, because
deleting a message (with most clients) also means moving it (to trash).
Functionality was back not before the reconstruct had finished
completely. 
Apart from those we also had some weird problems with the message sort
order (using roundcube) before reconstruct was run.

We split the mailboxes reconstruction into 8 parallel jobs (IO on the
mailbox spool is the limit if you go from 2.3 -> 2.4/3.0). If you ran
the reconstruction online, once finished I would suggest to check again
all indexes version (some reconstruction jobs fail).

Cheers

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Question for upgrading

Thank you so much Binarus!!! And as said before, sorry for the big
delay!!

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 18-12-2018 12:15, Binarus escribió:

> On 15.12.2018 17:05, Nic Bernstein wrote: On 12/13/18 9:52 AM, Egoitz 
> Aurrekoetxea wrote: 
> I was trying to upgrade part of our Cyrus imap installation,
> concretely that one consisting in still 2.3. I was planning to set up
> Cyrus 3.0. I have seen all works properly except for the unexpunge
> command because as someone stated here, a reconstruct -V max was
> needed.The problem is that this reconstruct command, takes ages and
> I'm not able to keep the service offline so many time. So I have been
> thinking in the following scenario :
> 
> Egoitz,
> As long as you've followed all of the various steps needed to account
> for version changes, as outlined in the Release Notes for /all/
> intermediary releases, then the last step should be the updating of the
> indexes.  Rather than running "reconstruct -V max" on all mailboxes at
> once, simply run it on subsets.  We've done this on large installations
> without ill effect.  You can have the service on line during the
> reconstruct, and, as you note, have most all functionality present.

In my case, before the reconstruct had finished, I had several problems
which might be not acceptable for large organizations.

For example, users could not move messages between folders in their
mailbox. I would consider this quite basic functionality, because
deleting a message (with most clients) also means moving it (to trash).
Functionality was back not before the reconstruct had finished
completely.

If interested, please see the respective thread from yesterday / today
for details (I don't want to clutter the list by repeating all that
stuff here).

Regards,

Binarus

Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Question for upgrading

Hi Nic, 

Sorry for the delay answering... I have been ill and later on holiday,
with very very very few time to answer to nothing when coming from
illness. Sorry then... 

My idea really, was to upgrade to 3.0... So I'm planning: 

- Upgrade in place from 2.3 to 2.4. 

- Continue giving the service... 

- Later setup a 3.0 slave all of this WITHOUT doing a reconstruct -V
max. as it seems in 2.4 all works (unexpunge command and so) without
it... should say too, it seems the own sync client performs database
upgrade 

Jan  3 14:46:02 mx8c sync_client[17186]: Index upgrade:
mydomain.com!user.mytestuser.Sarenet-staff.Alberto (10 -> 12)
Jan  3 14:46:06 mx8c sync_client[17186]: Index upgrade:
mydomain.com!user.mytestuser.Sarenet-staff.Comerciales (10 -> 12)
Jan  3 14:46:15 mx8c sync_client[17186]: Index upgrade:
mydomain.com!user.mytestuser.Sarenet-staff.Instalaciones-staff (10 ->
12)
Jan  3 14:46:25 mx8c sync_client[17186]: Index upgrade:
mydomain.com!user.mytestuser.Sarenet-staff.RRHH (10 -> 12) 

It seems to anyway be happening 

- When the 3.0 slave is totally replicated from the 2.4 mater... turn
the 3.0 as mater, and take a copy of it becoming it slave. 

This way I should have all the platform upgraded without downtimes 

How do you see Nic?. I think it should work... I'm still doing checks
anyway. 

Cheers!! Thanks a lot for your time!! 

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [2] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 15-12-2018 17:05, Nic Bernstein escribió:

> On 12/13/18 9:52 AM, Egoitz Aurrekoetxea wrote:
> 
>> I was trying to upgrade part of our Cyrus imap installation, concretely that 
>> one consisting in still 2.3. I was planning to set up Cyrus 3.0. I have seen 
>> all works properly except for the unexpunge command because as someone 
>> stated here, a reconstruct -V max was needed.The problem is that this 
>> reconstruct command, takes ages and I'm not able to keep the service offline 
>> so many time. So I have been thinking in the following scenario :
> 
> Egoitz,
> As long as you've followed all of the various steps needed to account for 
> version changes, as outlined in the Release Notes for _all_ intermediary 
> releases, then the last step should be the updating of the indexes.  Rather 
> than running "reconstruct -V max" on all mailboxes at once, simply run it on 
> subsets.  We've done this on large installations without ill effect.  You can 
> have the service on line during the reconstruct, and, as you note, have most 
> all functionality present.
> 
> We build a list of users (mboxlist.txt), and then run a cron job, during 
> off-peak hours, using the 'parallel' command like so (from 'crontab -e -u 
> cyrus'):
> 
>> ###
>> ## Start a batch of recursive reconstruct jobs at 7PM and discontinue
>> ## at 6:53AM.
>> 0 19 * * * parallel -j 5 --resume --joblog /var/lib/imap/reconstruct.log 
>> cyrus reconstruct -G -V max -r -u {} < /var/lib/imap/mboxlist.txt
>> 53 06 * * * kill -TERM `ps ax | grep [p]arallel | awk '{print $1}'`
> Cheers,
> -nic
> 
> -- 
> Nic Bernstein n...@onlight.com
> Onlight Inc.  www.onlight.com [1]
> 6525 W Bluemound Rd., Ste 24  v. 414.272.4477
> Milwaukee, Wisconsin  53213-4073  f. 414.290.0335
 

Links:
--
[1] http://www.onlight.com
[2] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

www.cyrusimap.org down?

Good afternoon, 

I'm suffering connectivity issues to Cyrus imap web site. Perhaps some
kind of problem at hosting side?. It takes from yesterday... do we know
something about when will it be up again?. 

Cheers!

-- 

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Question for upgrading

2018-12-17 Thread Egoitz Aurrekoetxea

Thank you so much Ellie!! 

I would test, read code when needed and finally attempt the upgrade... I
think this coud work should say I'm in testing phase still, 

I'll keep you informed :) 

Cheers!

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 17-12-2018 22:46, ellie timoney escribió:

>> Would be fine if Bron, Ellie or someone at Fastmail could tell something 
>> about it to us :) :) 
> 
> Cheeky!  Thing is, at FM our production servers more or less track the master 
> branch, so we were running "3.0" from the moment 2.5 forked off, and so our 
> upgrade process only ever really needs to deal with iterative changes.  Our 
> accumulated experience is what informs the "upgrade notes" included with the 
> 2.5.0 and 3.0.0 releases (not sure about older versions, I wasn't around 
> then). 
> 
> This mailing list is the right place to find people with real experience 
> doing big-bang upgrades. :) 
> 
> Cheers, 
> 
> ellie 
> 
> On Mon, Dec 17, 2018, at 8:40 PM, Egoitz Aurrekoetxea wrote: 
> 
> Hi mate, 
> 
> I think (and say think :) )I finally found a method. Although I'm testing it 
> deeply... it seems (say seems too :) ) 2.4 is compatible with a mail spool in 
> 2.3 (at least with my config). So I'll try to upgrade first to 2.4 and later 
> to 3.0 setting up a replication from 2.4 to 3.0. Would be fine if Bron, Ellie 
> or someone at Fastmail could tell something about it to us :) :) 
> 
> Cheers!
> 
> --- 
> 
> EGOITZ AURREKOETXEA 
> Departamento de sistemas 
> 
> 944 209 470 
> Parque Tecnológico. Edificio 103 
> 48170 Zamudio (Bizkaia) 
> ego...@sarenet.es
> 
> www.sarenet.es [1] 
> 
> Antes de imprimir este correo electrónico piense si es necesario hacerlo. 
> 
> El 16-12-2018 22:24, John Capo escribió: 
> 
> On Thu, December 13, 2018 13:25, Egoitz Aurrekoetxea wrote: 
> Hi again! 
> 
> Else as a simplication way can you replicate any manner a 2.3 with some 
> newer version?. At least in manual mode (not rolling)?. 
> 
> The replication protocol in 2.4 is not compatible wit 2.3. 
> 
> There is no easy way.  I'm rsync'ing one account at a time between 2.3 and 
> 2.4 with IMAP/POP and LMTP access disabled, followed by a reconstruct, and 
> then enabling IMAP/POP and LMTP access. 
> 
> Many terrabytes to go. 
> 
> You could also look at imapsync but its slow. 
> 
> Cheers. 
> 
> --- 
> 
> EGOITZ AURREKOETXEA 
> Departamento de sistemas 
> 944 209 470 
> Parque TecnolÃ³gico. Edificio 103 
> 48170 Zamudio (Bizkaia) 
> ego...@sarenet.es www.sarenet.es [1] [1 [1]] Antes de imprimir este correo 
> electrÃ³nico piense si es 
> necesario hacerlo. 
> 
> El 13-12-2018 16:52, Egoitz Aurrekoetxea escribiÃ³: 
> 
> Good afternoon, 
> 
> I was trying to upgrade part of our Cyrus imap installation, concretely that 
> one consisting in 
> still 2.3. I was planning to set up Cyrus 3.0. I have seen all works properly 
> except for the 
> unexpunge command because as someone stated here, a reconstruct -V max was 
> needed.The problem 
> is that this reconstruct command, takes ages and I'm not able to keep the 
> service offline so 
> many time. So I have been thinking in the following scenario : 
> 
> - Cyrus 2.3 master -> Cyrus 2.4 slave 
> 
> Get this 2.4 slave ready and set it as master. But here comes my first doubt. 
> Does the 2.4 
> replication work with the 2.3 replication?. Can in this pair, both (the 2.3 
> and the 2.4) be 
> both master and slave?. I mean to switch roles in the pair. Make one become 
> master and the 
> other slave and  vice versa?. 
> 
> Let's think now Cyrus 2.4 is ready and working. 
> 
> - Now, I would set up a new 3.0 slave. I know 2.4 could replicate with 3.0. 
> So I would get the 
> 3.0 ready and then set 3.0 as master. Can in this pair both the 2.4 and 3.0 
> be master and 
> slave?. Meaning again to the same role switching commented before... to make 
> one to be master 
> and the other slave or vice versa 
> 
> I'l will end up with 2 3.0 master and slave... but I need to trace the 
> path... 
> 
> Does anyone see any other way?. 
> 
> Best regards, 
> 
> - 
> 
> -- 
> 
> EGOITZ AURREKOETXEA 
> Departamento de sistemas 
> 944 209 470 
> Parque TecnolÃ³gico. Edificio 103 
> 48170 Zamudio (Bizkaia) 
> ego...@sarenet.es www.sarenet.es [1] [1 [1]] Antes de imprimir este correo 
> electrÃ³nico piense si es 
> necesario hacerlo.  
> Cyrus Home Page: http://www.cyrusimap.org/ 
>

Re: Question for upgrading

2018-12-17 Thread Egoitz Aurrekoetxea

Hi! 

Yes, I assume there shouldn't be problems... in fact reconstruct in 2.4
does not have the option for upgrading the mailbox databases version
with a reconstruct -V max command... So... I assume it's compatible 

There I could then setup a slave replication... I hope... I'll tell you
:) 

Cheers!

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 17-12-2018 12:38, Adam Tauno Williams escribió:

> On Mon, 2018-12-17 at 10:40 +0100, Egoitz Aurrekoetxea wrote: 
> 
>> I think (and say think :) )I finally found a method. Although I'm
>> testing it deeply... it seems (say seems too :) ) 2.4 is compatible
>> with a mail spool in 2.3 (at least with my config). So I'll try to
>> upgrade first to 2.4 and later to 3.0 setting up a replication from
>> 2.4 to 3.0. Would be fine if Bron, Ellie or someone at Fastmail could
>> tell something about it to us :) :)
> 
> Yes, I have performed an in-place upgrade of 2.3 to 2.4.  Other than
> some delay due to index reconstruction it went very smoothly.
> 
> The only issues I recall is that some users got some deleted messages
> back after the reconstruct, and there were some mailboxes which lost
> Seen status;  I never figured out why [they were related to that
> handful of users which somehow always have problems with everything].

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Question for upgrading

2018-12-17 Thread Egoitz Aurrekoetxea

Hi mate, 

I think (and say think :) )I finally found a method. Although I'm
testing it deeply... it seems (say seems too :) ) 2.4 is compatible with
a mail spool in 2.3 (at least with my config). So I'll try to upgrade
first to 2.4 and later to 3.0 setting up a replication from 2.4 to 3.0.
Would be fine if Bron, Ellie or someone at Fastmail could tell something
about it to us :) :) 

Cheers!

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 16-12-2018 22:24, John Capo escribió:

> On Thu, December 13, 2018 13:25, Egoitz Aurrekoetxea wrote: 
> 
>> Hi again!
>> 
>> Else as a simplication way can you replicate any manner a 2.3 with some
>> newer version?. At least in manual mode (not rolling)?.
> 
> The replication protocol in 2.4 is not compatible wit 2.3.
> 
> There is no easy way.  I'm rsync'ing one account at a time between 2.3 and 
> 2.4 with IMAP/POP and LMTP access disabled, followed by a reconstruct, and 
> then enabling IMAP/POP and LMTP access.
> 
> Many terrabytes to go.
> 
> You could also look at imapsync but its slow.
> 
> Cheers.
> 
> ---
> 
> EGOITZ AURREKOETXEA
> Departamento de sistemas
> 944 209 470
> Parque TecnolÃ³gico. Edificio 103
> 48170 Zamudio (Bizkaia)
> ego...@sarenet.es www.sarenet.es [1] [1 [1]] Antes de imprimir este correo 
> electrÃ³nico piense si es
> necesario hacerlo.
> 
> El 13-12-2018 16:52, Egoitz Aurrekoetxea escribiÃ³:
> 
> Good afternoon,
> 
> I was trying to upgrade part of our Cyrus imap installation, concretely that 
> one consisting in
> still 2.3. I was planning to set up Cyrus 3.0. I have seen all works properly 
> except for the
> unexpunge command because as someone stated here, a reconstruct -V max was 
> needed.The problem
> is that this reconstruct command, takes ages and I'm not able to keep the 
> service offline so
> many time. So I have been thinking in the following scenario :
> 
> - Cyrus 2.3 master -> Cyrus 2.4 slave
> 
> Get this 2.4 slave ready and set it as master. But here comes my first doubt. 
> Does the 2.4
> replication work with the 2.3 replication?. Can in this pair, both (the 2.3 
> and the 2.4) be
> both master and slave?. I mean to switch roles in the pair. Make one become 
> master and the
> other slave and  vice versa?.
> 
> Let's think now Cyrus 2.4 is ready and working.
> 
> - Now, I would set up a new 3.0 slave. I know 2.4 could replicate with 3.0. 
> So I would get the
> 3.0 ready and then set 3.0 as master. Can in this pair both the 2.4 and 3.0 
> be master and
> slave?. Meaning again to the same role switching commented before... to make 
> one to be master
> and the other slave or vice versa
> 
> I'l will end up with 2 3.0 master and slave... but I need to trace the path...
> 
> Does anyone see any other way?.
> 
> Best regards,
> 
> -
> 
> --
> 
> EGOITZ AURREKOETXEA
> Departamento de sistemas
> 944 209 470
> Parque TecnolÃ³gico. Edificio 103
> 48170 Zamudio (Bizkaia)
> ego...@sarenet.es www.sarenet.es [1] [1 [1]] Antes de imprimir este correo 
> electrÃ³nico piense si es
> necesario hacerlo. 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 
> Links:
> --
> [1] http://www.sarenet.es

 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Question for upgrading

2018-12-14 Thread Egoitz Aurrekoetxea

Common Bron, Ellie, mates :) any advise? 

Cheers :) :)

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 13-12-2018 19:25, Egoitz Aurrekoetxea escribió:

> Hi again! 
> 
> Else as a simplication way can you replicate any manner a 2.3 with some newer 
> version?. At least in manual mode (not rolling)?. 
> 
> Cheers.
> 
> ---
> 
> EGOITZ AURREKOETXEA 
> Departamento de sistemas 
> 944 209 470
> Parque Tecnológico. Edificio 103
> 48170 Zamudio (Bizkaia) 
> ego...@sarenet.es 
> www.sarenet.es [1] 
> Antes de imprimir este correo electrónico piense si es necesario hacerlo. 
> 
> El 13-12-2018 16:52, Egoitz Aurrekoetxea escribió:
> 
>> Good afternoon, 
>> 
>> I was trying to upgrade part of our Cyrus imap installation, concretely that 
>> one consisting in still 2.3. I was planning to set up Cyrus 3.0. I have seen 
>> all works properly except for the unexpunge command because as someone 
>> stated here, a reconstruct -V max was needed.The problem is that this 
>> reconstruct command, takes ages and I'm not able to keep the service offline 
>> so many time. So I have been thinking in the following scenario : 
>> 
>> - Cyrus 2.3 master -> Cyrus 2.4 slave 
>> 
>> Get this 2.4 slave ready and set it as master. But here comes my first 
>> doubt. Does the 2.4 replication work with the 2.3 replication?. Can in this 
>> pair, both (the 2.3 and the 2.4) be both master and slave?. I mean to switch 
>> roles in the pair. Make one become master and the other slave and  vice 
>> versa?. 
>> 
>> Let's think now Cyrus 2.4 is ready and working. 
>> 
>> - Now, I would set up a new 3.0 slave. I know 2.4 could replicate with 3.0. 
>> So I would get the 3.0 ready and then set 3.0 as master. Can in this pair 
>> both the 2.4 and 3.0 be master and slave?. Meaning again to the same role 
>> switching commented before... to make one to be master and the other slave 
>> or vice versa 
>> 
>> I'l will end up with 2 3.0 master and slave... but I need to trace the 
>> path... 
>> 
>> Does anyone see any other way?. 
>> 
>> Best regards, 
>> 
>> -
>> 
>> -- 
>> 
>> EGOITZ AURREKOETXEA 
>> Departamento de sistemas 
>> 944 209 470
>> Parque Tecnológico. Edificio 103
>> 48170 Zamudio (Bizkaia) 
>> ego...@sarenet.es 
>> www.sarenet.es [1] 
>> Antes de imprimir este correo electrónico piense si es necesario hacerlo. 
>> 
>> Cyrus Home Page: http://www.cyrusimap.org/
>> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
>> To Unsubscribe:
>> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Question for upgrading

2018-12-13 Thread Egoitz Aurrekoetxea

Hi again! 

Else as a simplication way can you replicate any manner a 2.3 with some
newer version?. At least in manual mode (not rolling)?. 

Cheers.

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 13-12-2018 16:52, Egoitz Aurrekoetxea escribió:

> Good afternoon, 
> 
> I was trying to upgrade part of our Cyrus imap installation, concretely that 
> one consisting in still 2.3. I was planning to set up Cyrus 3.0. I have seen 
> all works properly except for the unexpunge command because as someone stated 
> here, a reconstruct -V max was needed.The problem is that this reconstruct 
> command, takes ages and I'm not able to keep the service offline so many 
> time. So I have been thinking in the following scenario : 
> 
> - Cyrus 2.3 master -> Cyrus 2.4 slave 
> 
> Get this 2.4 slave ready and set it as master. But here comes my first doubt. 
> Does the 2.4 replication work with the 2.3 replication?. Can in this pair, 
> both (the 2.3 and the 2.4) be both master and slave?. I mean to switch roles 
> in the pair. Make one become master and the other slave and  vice versa?. 
> 
> Let's think now Cyrus 2.4 is ready and working. 
> 
> - Now, I would set up a new 3.0 slave. I know 2.4 could replicate with 3.0. 
> So I would get the 3.0 ready and then set 3.0 as master. Can in this pair 
> both the 2.4 and 3.0 be master and slave?. Meaning again to the same role 
> switching commented before... to make one to be master and the other slave or 
> vice versa 
> 
> I'l will end up with 2 3.0 master and slave... but I need to trace the 
> path... 
> 
> Does anyone see any other way?. 
> 
> Best regards, 
> 
> -
> 
> -- 
> 
> EGOITZ AURREKOETXEA 
> Departamento de sistemas 
> 944 209 470
> Parque Tecnológico. Edificio 103
> 48170 Zamudio (Bizkaia) 
> ego...@sarenet.es 
> www.sarenet.es [1] 
> Antes de imprimir este correo electrónico piense si es necesario hacerlo. 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Question for upgrading

2018-12-13 Thread Egoitz Aurrekoetxea

Good afternoon, 

I was trying to upgrade part of our Cyrus imap installation, concretely
that one consisting in still 2.3. I was planning to set up Cyrus 3.0. I
have seen all works properly except for the unexpunge command because as
someone stated here, a reconstruct -V max was needed.The problem is that
this reconstruct command, takes ages and I'm not able to keep the
service offline so many time. So I have been thinking in the following
scenario : 

- Cyrus 2.3 master -> Cyrus 2.4 slave 

Get this 2.4 slave ready and set it as master. But here comes my first
doubt. Does the 2.4 replication work with the 2.3 replication?. Can in
this pair, both (the 2.3 and the 2.4) be both master and slave?. I mean
to switch roles in the pair. Make one become master and the other slave
and  vice versa?. 

Let's think now Cyrus 2.4 is ready and working. 

- Now, I would set up a new 3.0 slave. I know 2.4 could replicate with
3.0. So I would get the 3.0 ready and then set 3.0 as master. Can in
this pair both the 2.4 and 3.0 be master and slave?. Meaning again to
the same role switching commented before... to make one to be master and
the other slave or vice versa 

I'l will end up with 2 3.0 master and slave... but I need to trace the
path... 

Does anyone see any other way?. 

Best regards, 

-

-- 

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Cyrus 3.0.8 longlocks and unexpunge

2018-12-11 Thread Egoitz Aurrekoetxea

Hi mate! 

Thank you so much!. The reconstruct solved the unexpunge issue. Could
anyone know something about long locks?. 

Best regards,

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 05-12-2018 10:45, Javier Angulo escribió:

> On 12/5/18 8:51 AM, Egoitz Aurrekoetxea wrote: 
> 
>> Good morning,
>> 
>> I'm testing 3.0.8 Cyrus IMAP. I have seen that for instance, when you
>> remove a couple of messages (with a vanila Roundcube webmail so IMAP,
>> for instance) you cause a copy to the trash and a expunge in the INBOX.
>> This operation takes very long comparing with older versions of Cyrus. I
>> mean :
>> 
>> Dec  5 08:38:42 testenv imap[88161]: login: [172.22.0.1]
>> ego...@sarenet.es PLAIN User logged in
>> SESSIONID=
>> Dec  5 08:38:42 testenv squatter[20782]: indexing mailbox
>> user/egoitz/papel...@sarenet.es...
>> Dec  5 08:38:42 testenv imap[88160]: Expunged 2 messages from
>> sarenet.es!user.egoitz
>> Dec  5 08:38:42 testenv imap[88160]: Repacking mailbox
>> sarenet.es!user.egoitz version 10
>> Dec  5 08:38:42 testenv squatter[20782]: Xapian committed 2 updates in
>> 0.049130 sec
>> Dec  5 08:38:43 testenv imap[88160]: message_parse_charset: unknown
>> charset RCMAIL_CHARSET for text/PLAIN
>> Dec  5 08:38:43 testenv imap[88160]: message_parse_charset: unknown
>> charset RCMAIL_CHARSET for text/CALENDAR
>> Dec  5 08:38:43 testenv imap[88160]: message_parse_charset: unknown
>> charset RCMAIL_CHARSET for text/PLAIN
>> Dec  5 08:38:43 testenv imap[88160]: message_parse_charset: unknown
>> charset RCMAIL_CHARSET for text/CALENDAR
>> Dec  5 08:38:43 testenv imap[88160]: message_parse_charset: unknown
>> charset RCMAIL_CHARSET for text/PLAIN
>> Dec  5 08:38:43 testenv imap[88160]: message_parse_charset: unknown
>> charset RCMAIL_CHARSET for text/CALENDAR
>> Dec  5 08:38:43 testenv imap[88160]: message_parse_charset: unknown
>> charset RCMAIL_CHARSET for text/PLAIN
>> Dec  5 08:38:43 testenv imap[88160]: message_parse_charset: unknown
>> charset RCMAIL_CHARSET for text/CALENDAR
>> *Dec  5 08:39:02 testenv imap[88160]: mailbox: longlock
>> sarenet.es!user.egoitz for 19.9 seconds*
>> Dec  5 08:39:02 testenv imap[88160]: USAGE ego...@sarenet.es user:
>> 8.469829 sys: 0.584660
>> 
>> Is this a normal or expected behavior?.
>> 
>> By the way, I have configured :
>> 
>> expunge_mode: delayed
>> delete_mode: delayed
>> deletedprefix: DELETED
>> 
>> But now, if I do :
>> 
>> % /usr/local/cyrus/sbin/unexpunge -lv user/ego...@sarenet.es
>> %
>> 
>> So, no results... unexpunge gives an error for instance if the mailbox
>> does not exist
>> 
>> Why this last problem can happen?. Why it could take so long to perform
>> the removal of the last commented messages?.
> 
> Hi Egoitz,
> 
> from this line: 
> 
>> Dec  5 08:38:42 testenv imap[88160]: Repacking mailbox
>> sarenet.es!user.egoitz version 10
> 
> it seems that you have not run a reconstruct on the mailbox (should be
> version 13 on cyrus 3.0)
> 
> try:
> reconstruct -V max user/ego...@sarenet.es
> 
> As you are probably coming from cyrus version 2.3 reconstruct is going
> to take a long time ...
> 
> I guess the unexpunge problem is also related and you need to
> reconstruct the mailbox.
> 
> Cheers,
> Javier
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Help on reconstruct - Cyrus2.3.11

2018-12-06 Thread Egoitz Aurrekoetxea

Hi!, 

I'm an experienced user too... more than 10 too :) 

It's just an advice... I'd recommend using Cyrus replication as an HA
mech. We use an own made mech for restoring mailboxes... the snapshot
causes often mailboxes to need a reconstruction. Obviously if it has
worked for you perhaps something has now changed... but as a general
advise... I'll tell you to use replication as ha and deletion delay for
instance as a recovery method. 

Cheers!

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 06-12-2018 10:32, Ismaël Tanguy escribió:

> Hello, 
> 
> thanks for your answer.
> We have been using for more than 10 years Cyrus with NFS because of the 
> snapshot.
> Snapshot give a way to restore mail or mailbox.
> It has worked like a charm until the migration.
> Now we're stuck on daily mailbox corruption due to this storage. 
> 
> We're looking, first, to a way to automate safely the reconstruct of mailbox, 
> ideally keeping the Seen State of mails.
> In parrellel, we're studying the migration to Cyrus 2.4.17 without the use of 
> NFS. 
> 
> Cheers,
> Ismael 
> 
> Le 06/12/2018 à 08:21, ego...@sarenet.es a écrit : Hi! 
> 
> Mate nfs, is no tan appropiate storage for Cyrus. I'd recommend you using 
> machine local storage. Using that kind of config won't success. 
> 
> Cheers,
> 
> Egoitz, 
> 
> El 5 dic 2018, a las 12:13, Ismaël Tanguy  
> escribió:
> 
> Hello, this is a Cyrus 2.3.11 on Centos 5.
> About 5000 users for 10 To. 
> 
> Mail storage has been moved from NetApp NFS to FluidsFS (aka Dell Compellent 
> NFS).
> Since an update on FluidFS, Imap spool undergoes daily NFS timeouts which 
> leads to corrupt mailboxes.
> Typically, this begins with lines like this in /var/log/messages:
> 
> Dec  5 09:54:43 mailhost kernel: lockd: server 192.xxx.xx.xx not responding, 
> timed out
> 
> Which is followed by IOERROR for accessed mailboxes during NFS timeout:
> 
> Dec  5 09:54:47 mailhost lmtpunix[14542]: IOERROR: locking index for 
> user.: Input/output error
> Dec  5 09:54:47 mailhost imaps[21999]: IOERROR: locking header for 
> user..Sent: Input/output error
> Dec  5 09:54:47 mailhost imaps[26935]: IOERROR: locking index for user.: 
> Input/output error
> Dec  5 09:54:47 mailhost imaps[24013]: IOERROR: locking index for user.: 
> Input/output error
> Dec  5 09:54:47 mailhost imaps[15672]: IOERROR: locking index for user.: 
> Input/output error
> Dec  5 09:54:47 mailhost imaps[3999]: IOERROR: locking index for user.: 
> Input/output error
> Dec  5 09:54:47 mailhost imaps[30671]: IOERROR: locking index for user.: 
> Input/output error
> 
> ...
> Around 15 maiboxes are corrupted at each timeouts.
> Manually, we can repair this mailbox: 
> 
> * first, we have to delete all cyrus files in mailbox, if not the following 
> reconstruct can be blocked
> * then, we reconstruct the mailbox (reconstruct -s user..
> 
> The downside of this method is that all messages in the reconstructed folder 
> are marked 'Not seen'.
> To automate this, a Python script has been written, but sometimes not all 
> cyrus files (cyrus.index) are recreated:
> 
> Dec  5 01:03:53 mailhost lmtpunix[497]: IOERROR: opening 
> /var/spool/imap/x/user/xx/cyrus.index: No such file or directory
> 
> Timeouts happen about 3 times per day, and cyrus deliver process is blocked 
> when delivering to a corrupted mailbox.
> So my first question is : how can we reconstruct a mailbox without marking 
> mails as not seen?
> And my second question is : why cyrus files are not recreated everytime? Is 
> this due to the -s parameter with reconstruct? 
> 
> Any help will be appreciated. 
> 
> Thanks 
> 
> -- 
> 
> Ismael TANGUY
> 
> -- 
> 
> 
> Cyrus Home Page: http://www.cyrusimap.org/
> List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
> To Unsubscribe:
> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Cyrus 3.0.8 longlocks and unexpunge

Good morning, 

I'm testing 3.0.8 Cyrus IMAP. I have seen that for instance, when you
remove a couple of messages (with a vanila Roundcube webmail so IMAP,
for instance) you cause a copy to the trash and a expunge in the INBOX.
This operation takes very long comparing with older versions of Cyrus. I
mean : 

Dec  5 08:38:42 testenv imap[88161]: login: [172.22.0.1]
ego...@sarenet.es PLAIN User logged in
SESSIONID=
Dec  5 08:38:42 testenv squatter[20782]: indexing mailbox
user/egoitz/papel...@sarenet.es... 
Dec  5 08:38:42 testenv imap[88160]: Expunged 2 messages from
sarenet.es!user.egoitz
Dec  5 08:38:42 testenv imap[88160]: Repacking mailbox
sarenet.es!user.egoitz version 10
Dec  5 08:38:42 testenv squatter[20782]: Xapian committed 2 updates in
0.049130 sec
Dec  5 08:38:43 testenv imap[88160]: message_parse_charset: unknown
charset RCMAIL_CHARSET for text/PLAIN
Dec  5 08:38:43 testenv imap[88160]: message_parse_charset: unknown
charset RCMAIL_CHARSET for text/CALENDAR
Dec  5 08:38:43 testenv imap[88160]: message_parse_charset: unknown
charset RCMAIL_CHARSET for text/PLAIN
Dec  5 08:38:43 testenv imap[88160]: message_parse_charset: unknown
charset RCMAIL_CHARSET for text/CALENDAR
Dec  5 08:38:43 testenv imap[88160]: message_parse_charset: unknown
charset RCMAIL_CHARSET for text/PLAIN
Dec  5 08:38:43 testenv imap[88160]: message_parse_charset: unknown
charset RCMAIL_CHARSET for text/CALENDAR
Dec  5 08:38:43 testenv imap[88160]: message_parse_charset: unknown
charset RCMAIL_CHARSET for text/PLAIN
Dec  5 08:38:43 testenv imap[88160]: message_parse_charset: unknown
charset RCMAIL_CHARSET for text/CALENDAR
DEC  5 08:39:02 TESTENV IMAP[88160]: MAILBOX: LONGLOCK
SARENET.ES!USER.EGOITZ FOR 19.9 SECONDS
Dec  5 08:39:02 testenv imap[88160]: USAGE ego...@sarenet.es user:
8.469829 sys: 0.584660 

Is this a normal or expected behavior?. 

By the way, I have configured : 

expunge_mode: delayed
delete_mode: delayed
deletedprefix: DELETED 

But now, if I do : 

% /usr/local/cyrus/sbin/unexpunge -lv user/ego...@sarenet.es
% 

So, no results... unexpunge gives an error for instance if the mailbox
does not exist 

Why this last problem can happen?. Why it could take so long to perform
the removal of the last commented messages?. 

Best regards,

-- 

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: Missing Email & Folders

Can anyone provide some more details about this bug?. Have seen
something similar to this... although have not been able to deep on that
issue because have not had reproduced it... 

Cheers!

---

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

El 10-11-2018 22:22, Marcus Schopen escribió:

> Am Mittwoch, den 07.11.2018, 09:16 +0100 schrieb Michael Menge: Hi,
> 
> Quoting Robert Covell :
> 
> If you suspect this is due to a client related problem, you could
> enable telemetry logging to find out who/what is causeing the
> emails to go 
> missing. 
> https://www.cyrusimap.org/imap/reference/faqs/o-telemetry.html 
> Good idea will turn this on.
> 
> If the purpose is to (mostly) copy emails into the folder and
> rarely
> delete, you could restrict delete access to a specific account
> via ACL.
 https://www.cyrusimap.org/imap/reference/admin/access-control/rights 

> -
> reference.html 
> Believe it is setup like this, been awhile since I did it. Will
> confirm.
> 
> Also forgot to say the one account is ~90GB if that makes any
> difference...
> 
> -Bob
> 
> (sent twice)

We have seen the same kind of problems with Outlook with much
smaller  
mailboxes.
I didn't have the time to debug it. So fare I suspected that some
Data  
in the Outlook
profile got somehow corrupted. 
Isn't this the known bug that Outlook 2013 doesn't really work well if
it is only used via IMAP and not with Exchange? There have been similar
cases with IMAP accounts that have been completely compromised. On the
IMAP there were whole folders missing, but they still existed locally.


Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Operating system performance for Cyrus and one more question

Good afternoon, 

Is there any noticiable operating systems whose syscalls, filesystems...
etc... could make Cyrus run faster?. We usually run it on FreeBSD and
are pretty happy. Perhaps newer versions run better in any Linux like
os?. 

About Xapian, XCONV, Jmap... I have tried doing a search with my mailbox
in a new testing env, where I have connected a tesing mailbox imap
partition with messages and mailboxes, coming from an older version of
Cyrus. Even having Xapian, Conversations and so enabled, it does not
find messages newer than 3 month located in the Sent folder. Perhaps
Xapian, only works for new incoming messages?. Or should the mailboxes
be reconstructed in order to a search to find and show all messages from
a conversation existing in all mailboxes?. Or perhaps it's my mua's
fault and is not searching properly in the IMAP server, because is not
issuing some new commands to be used in order to make this kind of
search to be done by Cyrus?. 

Best regards,

-- 

EGOITZ AURREKOETXEA 
Departamento de sistemas 
944 209 470
Parque Tecnológico. Edificio 103
48170 Zamudio (Bizkaia) 
ego...@sarenet.es 
www.sarenet.es [1] 
Antes de imprimir este correo electrónico piense si es necesario
hacerlo. 

Links:
--
[1] http://www.sarenet.es
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

Re: From 2.3.16 to 3.0.8 - lost expunged messages