Re: cyrus-imapd upstreamed patches and improvements
On Fri, Dec 18, 2020 at 05:52:04PM -0300, Anatoli wrote:
> > Can you remind me quickly these 4 FLAVORs?
>
> 1. base (what it is now, but without murder, i.e. a bare IMAP/POP3 server)
> 2. http (base + http, i.e. *DAV (CalDAV, CardDAV, WebDAV))
Maybe we could call this FLAVOR "dav"?
> 3. replication (base + murder + replication + backup)
Does this bring any new dependencies?
If not, why not mere it into the default FLAVOR?
> 4. http + replication (the previous 3 combinations together)
>
> This is the relevant part from my other email (we have already removed nntp):
>
> > For the empty base flavor I'd turn off nntp (IMO should always be turned
> > off,
> > not sure if someone is still using it at all nowadays, it's an old code base
> > with a previous 7.5 score CVE), murder and squat as these are not the
> > essential features (but save murder for later).
> >
> > With this combination we have a base POP3/IMAP server that is fully
> > functional
> > and is suitable for use-cases like hosting providers that offer their
> > clients
> > basic POP3/IMAP email accounts without anything special, but they require
> > the
> > setup to be as simple and as solid as possible.
> >
> > This combination has a few direct dependencies: pcre sqlite3 cyrus-sasl
> > icu4c
> > e2fsprogs jansson + 2 indirect ones: libiconv and gettext-runtime. Quite a
> > clean install.
> >
> > The next use-case is a setup for small/mid-sized organizations that, apart
> > from basic IMAP, also want remote/shared calendaring support, reminders and
> > server-side scheduling with free/busy states (CalDAV), remote/shared
> > contacts
> > (CardDAV), remote/shared folders (WebDAV), etc.
> >
> > All this is enabled with --enable-http option and it brings some 12
> > additional
> > direct dependencies, but IMO together with the base IMAP, this is the most
> > typically used deployment: solid, easy to setup and works well with most
> > widely-used clients.
> >
> > Then the next use-case is for larger organizations that have enough
> > resources
> > (both monetary and human) to run cyrus-imapd on multiple servers with
> > duplication, triplication, .. of mailstore. For them there are the following
> > options:
> >
> > --enable-murder: server aggregation in an imap cluster - a separation of
> > front
> > and back-end servers, sharding of the mailstore, HA, etc.,
> > --enable-replication: replication of the mailstore between multiple
> > machines,
> > and --enable-backup: an experimental feature similar to the replication that
> > runs a backup instance on another server and duplicates there the mailstore
> > for non-realtime backups.
> >
> > I'd group these 3 features together under the "replication" combination as
> > they share the same concept, could be used together and all imply certain
> > resources and know-how to deploy these setups.
> >
> > So I'd provide the 3 combinations (base, http, replication) as pre-built
> > binaries. And possibly the 4th one that combines the http (*DAV) features
> > with
> > the 3 replication features: http + replication.
>
>
>
> On 17/12/20 10:36, Antoine Jacoutot wrote:
> > On Wed, Dec 16, 2020 at 06:50:35PM -0300, Anatoli wrote:
> >> Antoine,
> >>
> >> Thanks for your feedback.
> >>
> >> I'm working now on the flavored version of the port based on our
> >> conversation
> >> with Stuart, in particular what I described in my mail on 24 Nov 2020
> >> 09:40 UTC
> >> in this thread.
> >>
> >> Does that sound reasonable?
> >>
> >> I plan to first enable the 4 most popular flavors/combinations and then,
> >> in a
> >> second diff, to add additional optional flavors that won't be built
> >> automatically.
> >
> > Can you remind me quickly these 4 FLAVORs?
> >
> >
> >
> >>
> >> Regards,
> >> Anatoli
> >>
> >>
> >>
> >> On 4/12/20 09:34, Antoine Jacoutot wrote:
> >>> On Mon, Nov 30, 2020 at 04:19:16AM -0300, Anatoli wrote:
> Antoine, Stuart,
>
> cyrus-imapd 3.2.5 was just released.
>
> I'm attaching an updated patch that also includes the SHA256 and the
> version
> bump + everything else from my initial mail, which is needed to
> accommodate the
> cross-platform changes and upstreamed port's patches that are included
> in this
> release (and the explanations for each change are in my initial mail).
> >>>
> >>> Committed, thanks a lot for your work :-)
> >>>
> >>>
>
> Regards,
> Anatoli
>
>
>
> diff --git Makefile Makefile
> index bfee0b835b1..d738a1ca91b 100644
> --- Makefile
> +++ Makefile
> @@ -4,7 +4,7 @@ PORTROACH= limitw:1,even
>
> COMMENT=Cyrus IMAP server
>
> -V= 3.2.4
> +V= 3.2.5
> DISTNAME= cyrus-imapd-${V}
>
> SHARED_LIBS += cyrus 0.0 # 0.0
> diff --git distinfo distinfo
> index 2c825c1a02a..367870468fe 100644
>
Re: cyrus-imapd upstreamed patches and improvements
> Can you remind me quickly these 4 FLAVORs?
1. base (what it is now, but without murder, i.e. a bare IMAP/POP3 server)
2. http (base + http, i.e. *DAV (CalDAV, CardDAV, WebDAV))
3. replication (base + murder + replication + backup)
4. http + replication (the previous 3 combinations together)
This is the relevant part from my other email (we have already removed nntp):
> For the empty base flavor I'd turn off nntp (IMO should always be turned off,
> not sure if someone is still using it at all nowadays, it's an old code base
> with a previous 7.5 score CVE), murder and squat as these are not the
> essential features (but save murder for later).
>
> With this combination we have a base POP3/IMAP server that is fully functional
> and is suitable for use-cases like hosting providers that offer their clients
> basic POP3/IMAP email accounts without anything special, but they require the
> setup to be as simple and as solid as possible.
>
> This combination has a few direct dependencies: pcre sqlite3 cyrus-sasl icu4c
> e2fsprogs jansson + 2 indirect ones: libiconv and gettext-runtime. Quite a
> clean install.
>
> The next use-case is a setup for small/mid-sized organizations that, apart
> from basic IMAP, also want remote/shared calendaring support, reminders and
> server-side scheduling with free/busy states (CalDAV), remote/shared contacts
> (CardDAV), remote/shared folders (WebDAV), etc.
>
> All this is enabled with --enable-http option and it brings some 12 additional
> direct dependencies, but IMO together with the base IMAP, this is the most
> typically used deployment: solid, easy to setup and works well with most
> widely-used clients.
>
> Then the next use-case is for larger organizations that have enough resources
> (both monetary and human) to run cyrus-imapd on multiple servers with
> duplication, triplication, .. of mailstore. For them there are the following
> options:
>
> --enable-murder: server aggregation in an imap cluster - a separation of front
> and back-end servers, sharding of the mailstore, HA, etc.,
> --enable-replication: replication of the mailstore between multiple machines,
> and --enable-backup: an experimental feature similar to the replication that
> runs a backup instance on another server and duplicates there the mailstore
> for non-realtime backups.
>
> I'd group these 3 features together under the "replication" combination as
> they share the same concept, could be used together and all imply certain
> resources and know-how to deploy these setups.
>
> So I'd provide the 3 combinations (base, http, replication) as pre-built
> binaries. And possibly the 4th one that combines the http (*DAV) features with
> the 3 replication features: http + replication.
On 17/12/20 10:36, Antoine Jacoutot wrote:
> On Wed, Dec 16, 2020 at 06:50:35PM -0300, Anatoli wrote:
>> Antoine,
>>
>> Thanks for your feedback.
>>
>> I'm working now on the flavored version of the port based on our conversation
>> with Stuart, in particular what I described in my mail on 24 Nov 2020 09:40
>> UTC
>> in this thread.
>>
>> Does that sound reasonable?
>>
>> I plan to first enable the 4 most popular flavors/combinations and then, in a
>> second diff, to add additional optional flavors that won't be built
>> automatically.
>
> Can you remind me quickly these 4 FLAVORs?
>
>
>
>>
>> Regards,
>> Anatoli
>>
>>
>>
>> On 4/12/20 09:34, Antoine Jacoutot wrote:
>>> On Mon, Nov 30, 2020 at 04:19:16AM -0300, Anatoli wrote:
Antoine, Stuart,
cyrus-imapd 3.2.5 was just released.
I'm attaching an updated patch that also includes the SHA256 and the
version
bump + everything else from my initial mail, which is needed to
accommodate the
cross-platform changes and upstreamed port's patches that are included in
this
release (and the explanations for each change are in my initial mail).
>>>
>>> Committed, thanks a lot for your work :-)
>>>
>>>
Regards,
Anatoli
diff --git Makefile Makefile
index bfee0b835b1..d738a1ca91b 100644
--- Makefile
+++ Makefile
@@ -4,7 +4,7 @@ PORTROACH= limitw:1,even
COMMENT= Cyrus IMAP server
-V=3.2.4
+V=3.2.5
DISTNAME= cyrus-imapd-${V}
SHARED_LIBS +=cyrus 0.0 # 0.0
diff --git distinfo distinfo
index 2c825c1a02a..367870468fe 100644
--- distinfo
+++ distinfo
@@ -1,2 +1,2 @@
-SHA256 (cyrus-imapd-3.2.4.tar.gz) =
UWEmLDgqpaeMKLGk9eolRQne4eet6DH37JsUB4+0LyM=
-SIZE (cyrus-imapd-3.2.4.tar.gz) = 12270070
+SHA256 (cyrus-imapd-3.2.5.tar.gz) =
zDhqdU4kOJtSr4NzdrO7YFW+pwV/U+yHq8oGqOjWlWM=
+SIZE (cyrus-imapd-3.2.5.tar.gz) = 12237158
diff --git Makefile Makefile
index c7fb05ebcee..bfee0b835b1 100644
--- Makefile
+++ Makefile
@@ -39,8 +39,7 @@ LIB_DEPENDS=
Re: cyrus-imapd upstreamed patches and improvements
On Wed, Dec 16, 2020 at 06:50:35PM -0300, Anatoli wrote:
> Antoine,
>
> Thanks for your feedback.
>
> I'm working now on the flavored version of the port based on our conversation
> with Stuart, in particular what I described in my mail on 24 Nov 2020 09:40
> UTC
> in this thread.
>
> Does that sound reasonable?
>
> I plan to first enable the 4 most popular flavors/combinations and then, in a
> second diff, to add additional optional flavors that won't be built
> automatically.
Can you remind me quickly these 4 FLAVORs?
>
> Regards,
> Anatoli
>
>
>
> On 4/12/20 09:34, Antoine Jacoutot wrote:
> > On Mon, Nov 30, 2020 at 04:19:16AM -0300, Anatoli wrote:
> >> Antoine, Stuart,
> >>
> >> cyrus-imapd 3.2.5 was just released.
> >>
> >> I'm attaching an updated patch that also includes the SHA256 and the
> >> version
> >> bump + everything else from my initial mail, which is needed to
> >> accommodate the
> >> cross-platform changes and upstreamed port's patches that are included in
> >> this
> >> release (and the explanations for each change are in my initial mail).
> >
> > Committed, thanks a lot for your work :-)
> >
> >
> >>
> >> Regards,
> >> Anatoli
> >>
> >>
> >>
> >> diff --git Makefile Makefile
> >> index bfee0b835b1..d738a1ca91b 100644
> >> --- Makefile
> >> +++ Makefile
> >> @@ -4,7 +4,7 @@ PORTROACH= limitw:1,even
> >>
> >> COMMENT= Cyrus IMAP server
> >>
> >> -V=3.2.4
> >> +V=3.2.5
> >> DISTNAME= cyrus-imapd-${V}
> >>
> >> SHARED_LIBS +=cyrus 0.0 # 0.0
> >> diff --git distinfo distinfo
> >> index 2c825c1a02a..367870468fe 100644
> >> --- distinfo
> >> +++ distinfo
> >> @@ -1,2 +1,2 @@
> >> -SHA256 (cyrus-imapd-3.2.4.tar.gz) =
> >> UWEmLDgqpaeMKLGk9eolRQne4eet6DH37JsUB4+0LyM=
> >> -SIZE (cyrus-imapd-3.2.4.tar.gz) = 12270070
> >> +SHA256 (cyrus-imapd-3.2.5.tar.gz) =
> >> zDhqdU4kOJtSr4NzdrO7YFW+pwV/U+yHq8oGqOjWlWM=
> >> +SIZE (cyrus-imapd-3.2.5.tar.gz) = 12237158
> >> diff --git Makefile Makefile
> >> index c7fb05ebcee..bfee0b835b1 100644
> >> --- Makefile
> >> +++ Makefile
> >> @@ -39,8 +39,7 @@ LIB_DEPENDS= databases/sqlite3 \
> >>
> >> CONFIGURE_STYLE= gnu
> >> CONFIGURE_ENV=CPPFLAGS="-I${LOCALBASE}/include" \
> >> - LDFLAGS="-L${LOCALBASE}/lib" \
> >> - cyrus_cv_sse42=no
> >> + LDFLAGS="-L${LOCALBASE}/lib"
> >> CONFIGURE_ARGS= --bindir=${PREFIX}/cyrus/bin \
> >>--libexec=${PREFIX}/cyrus/libexec \
> >>--sbindir=${PREFIX}/cyrus/sbin \
> >> @@ -48,17 +47,12 @@ CONFIGURE_ARGS=
> >> --bindir=${PREFIX}/cyrus/bin \
> >>--with-cyrus-user=_cyrus \
> >>--with-syslogfacility=MAIL \
> >>--without-chardet \
> >> - --without-cld2 \
> >>--without-sphinx-build \
> >>--without-zeroskip \
> >>--disable-gssapi \
> >>--enable-autocreate \
> >>--enable-idled \
> >> - --enable-murder \
> >> - --enable-nntp
> >> -
> >> -# XXX FLAVOR
> >> -CONFIGURE_ARGS += --without-snmp
> >> + --enable-murder
> >>
> >> # XXX notyet; FLAVOR
> >> CONFIGURE_ARGS += --without-clamav \
> >> diff --git patches/patch-imap_conversations_c
> >> patches/patch-imap_conversations_c
> >> deleted file mode 100644
> >> index 9eab9396e0d..000
> >> --- patches/patch-imap_conversations_c
> >> +++ /dev/null
> >> @@ -1,16 +0,0 @@
> >> -$OpenBSD: patch-imap_conversations_c,v 1.3 2020/05/14 12:26:39 ajacoutot
> >> Exp $
> >> -
> >> -64 bit time_t
> >> -
> >> -Index: imap/conversations.c
> >> imap/conversations.c.orig
> >> -+++ imap/conversations.c
> >> -@@ -567,7 +567,7 @@ static int _conversations_set_key(struct conversations
> >> - if (i) buf_putc(, ',');
> >> - buf_printf(, CONV_FMT, cid);
> >> - }
> >> --buf_printf(, " %lu", stamp);
> >> -+buf_printf(, " %lld", stamp);
> >> -
> >> - r = cyrusdb_store(state->db,
> >> - key, keylen,
> >> diff --git patches/patch-imap_fud_c patches/patch-imap_fud_c
> >> deleted file mode 100644
> >> index cc6a8f8d327..000
> >> --- patches/patch-imap_fud_c
> >> +++ /dev/null
> >> @@ -1,17 +0,0 @@
> >> -$OpenBSD: patch-imap_fud_c,v 1.2 2020/08/28 09:53:04 ajacoutot Exp $
> >> -
> >> -Index: imap/fud.c
> >> imap/fud.c.orig
> >> -+++ imap/fud.c
> >> -@@ -96,8 +96,10 @@ static void send_reply(struct sockaddr *sfrom, socklen
> >> -
> >> - static int soc = 0; /* inetd (master) has handed us the port as stdin */
> >> -
> >> --#ifndef MAXDOMNAME
> >> -+#ifndef MAXLOGNAME
> >> - #define MAXLOGNAME 16 /* should find out for real */
> >> -+#endif
> >> -+#ifndef MAXDOMNAME
> >> - #define MAXDOMNAME 20 /* should find out for real */
> >> -
Re: cyrus-imapd upstreamed patches and improvements
Antoine,
Thanks for your feedback.
I'm working now on the flavored version of the port based on our conversation
with Stuart, in particular what I described in my mail on 24 Nov 2020 09:40 UTC
in this thread.
Does that sound reasonable?
I plan to first enable the 4 most popular flavors/combinations and then, in a
second diff, to add additional optional flavors that won't be built
automatically.
Regards,
Anatoli
On 4/12/20 09:34, Antoine Jacoutot wrote:
> On Mon, Nov 30, 2020 at 04:19:16AM -0300, Anatoli wrote:
>> Antoine, Stuart,
>>
>> cyrus-imapd 3.2.5 was just released.
>>
>> I'm attaching an updated patch that also includes the SHA256 and the version
>> bump + everything else from my initial mail, which is needed to accommodate
>> the
>> cross-platform changes and upstreamed port's patches that are included in
>> this
>> release (and the explanations for each change are in my initial mail).
>
> Committed, thanks a lot for your work :-)
>
>
>>
>> Regards,
>> Anatoli
>>
>>
>>
>> diff --git Makefile Makefile
>> index bfee0b835b1..d738a1ca91b 100644
>> --- Makefile
>> +++ Makefile
>> @@ -4,7 +4,7 @@ PORTROACH= limitw:1,even
>>
>> COMMENT=Cyrus IMAP server
>>
>> -V= 3.2.4
>> +V= 3.2.5
>> DISTNAME= cyrus-imapd-${V}
>>
>> SHARED_LIBS += cyrus 0.0 # 0.0
>> diff --git distinfo distinfo
>> index 2c825c1a02a..367870468fe 100644
>> --- distinfo
>> +++ distinfo
>> @@ -1,2 +1,2 @@
>> -SHA256 (cyrus-imapd-3.2.4.tar.gz) =
>> UWEmLDgqpaeMKLGk9eolRQne4eet6DH37JsUB4+0LyM=
>> -SIZE (cyrus-imapd-3.2.4.tar.gz) = 12270070
>> +SHA256 (cyrus-imapd-3.2.5.tar.gz) =
>> zDhqdU4kOJtSr4NzdrO7YFW+pwV/U+yHq8oGqOjWlWM=
>> +SIZE (cyrus-imapd-3.2.5.tar.gz) = 12237158
>> diff --git Makefile Makefile
>> index c7fb05ebcee..bfee0b835b1 100644
>> --- Makefile
>> +++ Makefile
>> @@ -39,8 +39,7 @@ LIB_DEPENDS= databases/sqlite3 \
>>
>> CONFIGURE_STYLE=gnu
>> CONFIGURE_ENV= CPPFLAGS="-I${LOCALBASE}/include" \
>> -LDFLAGS="-L${LOCALBASE}/lib" \
>> -cyrus_cv_sse42=no
>> +LDFLAGS="-L${LOCALBASE}/lib"
>> CONFIGURE_ARGS= --bindir=${PREFIX}/cyrus/bin \
>> --libexec=${PREFIX}/cyrus/libexec \
>> --sbindir=${PREFIX}/cyrus/sbin \
>> @@ -48,17 +47,12 @@ CONFIGURE_ARGS= --bindir=${PREFIX}/cyrus/bin \
>> --with-cyrus-user=_cyrus \
>> --with-syslogfacility=MAIL \
>> --without-chardet \
>> ---without-cld2 \
>> --without-sphinx-build \
>> --without-zeroskip \
>> --disable-gssapi \
>> --enable-autocreate \
>> --enable-idled \
>> ---enable-murder \
>> ---enable-nntp
>> -
>> -# XXX FLAVOR
>> -CONFIGURE_ARGS += --without-snmp
>> +--enable-murder
>>
>> # XXX notyet; FLAVOR
>> CONFIGURE_ARGS += --without-clamav \
>> diff --git patches/patch-imap_conversations_c
>> patches/patch-imap_conversations_c
>> deleted file mode 100644
>> index 9eab9396e0d..000
>> --- patches/patch-imap_conversations_c
>> +++ /dev/null
>> @@ -1,16 +0,0 @@
>> -$OpenBSD: patch-imap_conversations_c,v 1.3 2020/05/14 12:26:39 ajacoutot
>> Exp $
>> -
>> -64 bit time_t
>> -
>> -Index: imap/conversations.c
>> imap/conversations.c.orig
>> -+++ imap/conversations.c
>> -@@ -567,7 +567,7 @@ static int _conversations_set_key(struct conversations
>> - if (i) buf_putc(, ',');
>> - buf_printf(, CONV_FMT, cid);
>> - }
>> --buf_printf(, " %lu", stamp);
>> -+buf_printf(, " %lld", stamp);
>> -
>> - r = cyrusdb_store(state->db,
>> - key, keylen,
>> diff --git patches/patch-imap_fud_c patches/patch-imap_fud_c
>> deleted file mode 100644
>> index cc6a8f8d327..000
>> --- patches/patch-imap_fud_c
>> +++ /dev/null
>> @@ -1,17 +0,0 @@
>> -$OpenBSD: patch-imap_fud_c,v 1.2 2020/08/28 09:53:04 ajacoutot Exp $
>> -
>> -Index: imap/fud.c
>> imap/fud.c.orig
>> -+++ imap/fud.c
>> -@@ -96,8 +96,10 @@ static void send_reply(struct sockaddr *sfrom, socklen
>> -
>> - static int soc = 0; /* inetd (master) has handed us the port as stdin */
>> -
>> --#ifndef MAXDOMNAME
>> -+#ifndef MAXLOGNAME
>> - #define MAXLOGNAME 16 /* should find out for real */
>> -+#endif
>> -+#ifndef MAXDOMNAME
>> - #define MAXDOMNAME 20 /* should find out for real */
>> - #endif
>> -
>> diff --git patches/patch-imap_mailbox_c patches/patch-imap_mailbox_c
>> deleted file mode 100644
>> index 0793441fdfa..000
>> --- patches/patch-imap_mailbox_c
>> +++ /dev/null
>> @@ -1,34 +0,0 @@
>> -$OpenBSD: patch-imap_mailbox_c,v 1.19 2020/05/30 10:09:27 ajacoutot Exp $
>> -
>> -64 bit time_t
>> -
>> -Index: imap/mailbox.c
>>
Re: cyrus-imapd upstreamed patches and improvements
On Mon, Nov 30, 2020 at 04:19:16AM -0300, Anatoli wrote:
> Antoine, Stuart,
>
> cyrus-imapd 3.2.5 was just released.
>
> I'm attaching an updated patch that also includes the SHA256 and the version
> bump + everything else from my initial mail, which is needed to accommodate
> the
> cross-platform changes and upstreamed port's patches that are included in this
> release (and the explanations for each change are in my initial mail).
Committed, thanks a lot for your work :-)
>
> Regards,
> Anatoli
>
>
>
> diff --git Makefile Makefile
> index bfee0b835b1..d738a1ca91b 100644
> --- Makefile
> +++ Makefile
> @@ -4,7 +4,7 @@ PORTROACH=limitw:1,even
>
> COMMENT= Cyrus IMAP server
>
> -V= 3.2.4
> +V= 3.2.5
> DISTNAME=cyrus-imapd-${V}
>
> SHARED_LIBS += cyrus 0.0 # 0.0
> diff --git distinfo distinfo
> index 2c825c1a02a..367870468fe 100644
> --- distinfo
> +++ distinfo
> @@ -1,2 +1,2 @@
> -SHA256 (cyrus-imapd-3.2.4.tar.gz) =
> UWEmLDgqpaeMKLGk9eolRQne4eet6DH37JsUB4+0LyM=
> -SIZE (cyrus-imapd-3.2.4.tar.gz) = 12270070
> +SHA256 (cyrus-imapd-3.2.5.tar.gz) =
> zDhqdU4kOJtSr4NzdrO7YFW+pwV/U+yHq8oGqOjWlWM=
> +SIZE (cyrus-imapd-3.2.5.tar.gz) = 12237158
> diff --git Makefile Makefile
> index c7fb05ebcee..bfee0b835b1 100644
> --- Makefile
> +++ Makefile
> @@ -39,8 +39,7 @@ LIB_DEPENDS=databases/sqlite3 \
>
> CONFIGURE_STYLE= gnu
> CONFIGURE_ENV= CPPFLAGS="-I${LOCALBASE}/include" \
> - LDFLAGS="-L${LOCALBASE}/lib" \
> - cyrus_cv_sse42=no
> + LDFLAGS="-L${LOCALBASE}/lib"
> CONFIGURE_ARGS= --bindir=${PREFIX}/cyrus/bin \
> --libexec=${PREFIX}/cyrus/libexec \
> --sbindir=${PREFIX}/cyrus/sbin \
> @@ -48,17 +47,12 @@ CONFIGURE_ARGS= --bindir=${PREFIX}/cyrus/bin \
> --with-cyrus-user=_cyrus \
> --with-syslogfacility=MAIL \
> --without-chardet \
> - --without-cld2 \
> --without-sphinx-build \
> --without-zeroskip \
> --disable-gssapi \
> --enable-autocreate \
> --enable-idled \
> - --enable-murder \
> - --enable-nntp
> -
> -# XXX FLAVOR
> -CONFIGURE_ARGS +=--without-snmp
> + --enable-murder
>
> # XXX notyet; FLAVOR
> CONFIGURE_ARGS +=--without-clamav \
> diff --git patches/patch-imap_conversations_c
> patches/patch-imap_conversations_c
> deleted file mode 100644
> index 9eab9396e0d..000
> --- patches/patch-imap_conversations_c
> +++ /dev/null
> @@ -1,16 +0,0 @@
> -$OpenBSD: patch-imap_conversations_c,v 1.3 2020/05/14 12:26:39 ajacoutot Exp
> $
> -
> -64 bit time_t
> -
> -Index: imap/conversations.c
> imap/conversations.c.orig
> -+++ imap/conversations.c
> -@@ -567,7 +567,7 @@ static int _conversations_set_key(struct conversations
> - if (i) buf_putc(, ',');
> - buf_printf(, CONV_FMT, cid);
> - }
> --buf_printf(, " %lu", stamp);
> -+buf_printf(, " %lld", stamp);
> -
> - r = cyrusdb_store(state->db,
> - key, keylen,
> diff --git patches/patch-imap_fud_c patches/patch-imap_fud_c
> deleted file mode 100644
> index cc6a8f8d327..000
> --- patches/patch-imap_fud_c
> +++ /dev/null
> @@ -1,17 +0,0 @@
> -$OpenBSD: patch-imap_fud_c,v 1.2 2020/08/28 09:53:04 ajacoutot Exp $
> -
> -Index: imap/fud.c
> imap/fud.c.orig
> -+++ imap/fud.c
> -@@ -96,8 +96,10 @@ static void send_reply(struct sockaddr *sfrom, socklen
> -
> - static int soc = 0; /* inetd (master) has handed us the port as stdin */
> -
> --#ifndef MAXDOMNAME
> -+#ifndef MAXLOGNAME
> - #define MAXLOGNAME 16 /* should find out for real */
> -+#endif
> -+#ifndef MAXDOMNAME
> - #define MAXDOMNAME 20 /* should find out for real */
> - #endif
> -
> diff --git patches/patch-imap_mailbox_c patches/patch-imap_mailbox_c
> deleted file mode 100644
> index 0793441fdfa..000
> --- patches/patch-imap_mailbox_c
> +++ /dev/null
> @@ -1,34 +0,0 @@
> -$OpenBSD: patch-imap_mailbox_c,v 1.19 2020/05/30 10:09:27 ajacoutot Exp $
> -
> -64 bit time_t
> -
> -Index: imap/mailbox.c
> imap/mailbox.c.orig
> -+++ imap/mailbox.c
> -@@ -2899,7 +2899,7 @@ static uint32_t crc_basic(const struct mailbox *mailbo
> - flagcrc ^= crc32_cstring(buf);
> - }
> -
> --snprintf(buf, sizeof(buf), "%u " MODSEQ_FMT " %lu (%u) %lu %s",
> -+snprintf(buf, sizeof(buf), "%u " MODSEQ_FMT " %lld (%u) %lld %s",
> - record->uid, record->modseq, record->last_updated,
> - flagcrc,
> - record->internaldate,
> -@@ -2959,7 +2959,7 @@ static uint32_t crc_virtannot(struct mailbox *mailbox,
> - }
> -
> - if (record->savedate &&
Re: cyrus-imapd upstreamed patches and improvements
On Mon, Nov 30, 2020 at 04:19:16AM -0300, Anatoli wrote:
> Antoine, Stuart,
>
> cyrus-imapd 3.2.5 was just released.
>
> I'm attaching an updated patch that also includes the SHA256 and the version
> bump + everything else from my initial mail, which is needed to accommodate
> the
> cross-platform changes and upstreamed port's patches that are included in this
> release (and the explanations for each change are in my initial mail).
Thanks, I will look at this over the week-end.
>
> Regards,
> Anatoli
>
>
>
> diff --git Makefile Makefile
> index bfee0b835b1..d738a1ca91b 100644
> --- Makefile
> +++ Makefile
> @@ -4,7 +4,7 @@ PORTROACH=limitw:1,even
>
> COMMENT= Cyrus IMAP server
>
> -V= 3.2.4
> +V= 3.2.5
> DISTNAME=cyrus-imapd-${V}
>
> SHARED_LIBS += cyrus 0.0 # 0.0
> diff --git distinfo distinfo
> index 2c825c1a02a..367870468fe 100644
> --- distinfo
> +++ distinfo
> @@ -1,2 +1,2 @@
> -SHA256 (cyrus-imapd-3.2.4.tar.gz) =
> UWEmLDgqpaeMKLGk9eolRQne4eet6DH37JsUB4+0LyM=
> -SIZE (cyrus-imapd-3.2.4.tar.gz) = 12270070
> +SHA256 (cyrus-imapd-3.2.5.tar.gz) =
> zDhqdU4kOJtSr4NzdrO7YFW+pwV/U+yHq8oGqOjWlWM=
> +SIZE (cyrus-imapd-3.2.5.tar.gz) = 12237158
> diff --git Makefile Makefile
> index c7fb05ebcee..bfee0b835b1 100644
> --- Makefile
> +++ Makefile
> @@ -39,8 +39,7 @@ LIB_DEPENDS=databases/sqlite3 \
>
> CONFIGURE_STYLE= gnu
> CONFIGURE_ENV= CPPFLAGS="-I${LOCALBASE}/include" \
> - LDFLAGS="-L${LOCALBASE}/lib" \
> - cyrus_cv_sse42=no
> + LDFLAGS="-L${LOCALBASE}/lib"
> CONFIGURE_ARGS= --bindir=${PREFIX}/cyrus/bin \
> --libexec=${PREFIX}/cyrus/libexec \
> --sbindir=${PREFIX}/cyrus/sbin \
> @@ -48,17 +47,12 @@ CONFIGURE_ARGS= --bindir=${PREFIX}/cyrus/bin \
> --with-cyrus-user=_cyrus \
> --with-syslogfacility=MAIL \
> --without-chardet \
> - --without-cld2 \
> --without-sphinx-build \
> --without-zeroskip \
> --disable-gssapi \
> --enable-autocreate \
> --enable-idled \
> - --enable-murder \
> - --enable-nntp
> -
> -# XXX FLAVOR
> -CONFIGURE_ARGS +=--without-snmp
> + --enable-murder
>
> # XXX notyet; FLAVOR
> CONFIGURE_ARGS +=--without-clamav \
> diff --git patches/patch-imap_conversations_c
> patches/patch-imap_conversations_c
> deleted file mode 100644
> index 9eab9396e0d..000
> --- patches/patch-imap_conversations_c
> +++ /dev/null
> @@ -1,16 +0,0 @@
> -$OpenBSD: patch-imap_conversations_c,v 1.3 2020/05/14 12:26:39 ajacoutot Exp
> $
> -
> -64 bit time_t
> -
> -Index: imap/conversations.c
> imap/conversations.c.orig
> -+++ imap/conversations.c
> -@@ -567,7 +567,7 @@ static int _conversations_set_key(struct conversations
> - if (i) buf_putc(, ',');
> - buf_printf(, CONV_FMT, cid);
> - }
> --buf_printf(, " %lu", stamp);
> -+buf_printf(, " %lld", stamp);
> -
> - r = cyrusdb_store(state->db,
> - key, keylen,
> diff --git patches/patch-imap_fud_c patches/patch-imap_fud_c
> deleted file mode 100644
> index cc6a8f8d327..000
> --- patches/patch-imap_fud_c
> +++ /dev/null
> @@ -1,17 +0,0 @@
> -$OpenBSD: patch-imap_fud_c,v 1.2 2020/08/28 09:53:04 ajacoutot Exp $
> -
> -Index: imap/fud.c
> imap/fud.c.orig
> -+++ imap/fud.c
> -@@ -96,8 +96,10 @@ static void send_reply(struct sockaddr *sfrom, socklen
> -
> - static int soc = 0; /* inetd (master) has handed us the port as stdin */
> -
> --#ifndef MAXDOMNAME
> -+#ifndef MAXLOGNAME
> - #define MAXLOGNAME 16 /* should find out for real */
> -+#endif
> -+#ifndef MAXDOMNAME
> - #define MAXDOMNAME 20 /* should find out for real */
> - #endif
> -
> diff --git patches/patch-imap_mailbox_c patches/patch-imap_mailbox_c
> deleted file mode 100644
> index 0793441fdfa..000
> --- patches/patch-imap_mailbox_c
> +++ /dev/null
> @@ -1,34 +0,0 @@
> -$OpenBSD: patch-imap_mailbox_c,v 1.19 2020/05/30 10:09:27 ajacoutot Exp $
> -
> -64 bit time_t
> -
> -Index: imap/mailbox.c
> imap/mailbox.c.orig
> -+++ imap/mailbox.c
> -@@ -2899,7 +2899,7 @@ static uint32_t crc_basic(const struct mailbox *mailbo
> - flagcrc ^= crc32_cstring(buf);
> - }
> -
> --snprintf(buf, sizeof(buf), "%u " MODSEQ_FMT " %lu (%u) %lu %s",
> -+snprintf(buf, sizeof(buf), "%u " MODSEQ_FMT " %lld (%u) %lld %s",
> - record->uid, record->modseq, record->last_updated,
> - flagcrc,
> - record->internaldate,
> -@@ -2959,7 +2959,7 @@ static uint32_t crc_virtannot(struct mailbox *mailbox,
> - }
> -
> - if (record->savedate
Re: cyrus-imapd upstreamed patches and improvements
Antoine, Stuart,
cyrus-imapd 3.2.5 was just released.
I'm attaching an updated patch that also includes the SHA256 and the version
bump + everything else from my initial mail, which is needed to accommodate the
cross-platform changes and upstreamed port's patches that are included in this
release (and the explanations for each change are in my initial mail).
Regards,
Anatoli
diff --git Makefile Makefile
index bfee0b835b1..d738a1ca91b 100644
--- Makefile
+++ Makefile
@@ -4,7 +4,7 @@ PORTROACH= limitw:1,even
COMMENT= Cyrus IMAP server
-V= 3.2.4
+V= 3.2.5
DISTNAME= cyrus-imapd-${V}
SHARED_LIBS += cyrus 0.0 # 0.0
diff --git distinfo distinfo
index 2c825c1a02a..367870468fe 100644
--- distinfo
+++ distinfo
@@ -1,2 +1,2 @@
-SHA256 (cyrus-imapd-3.2.4.tar.gz) =
UWEmLDgqpaeMKLGk9eolRQne4eet6DH37JsUB4+0LyM=
-SIZE (cyrus-imapd-3.2.4.tar.gz) = 12270070
+SHA256 (cyrus-imapd-3.2.5.tar.gz) =
zDhqdU4kOJtSr4NzdrO7YFW+pwV/U+yHq8oGqOjWlWM=
+SIZE (cyrus-imapd-3.2.5.tar.gz) = 12237158
diff --git Makefile Makefile
index c7fb05ebcee..bfee0b835b1 100644
--- Makefile
+++ Makefile
@@ -39,8 +39,7 @@ LIB_DEPENDS= databases/sqlite3 \
CONFIGURE_STYLE= gnu
CONFIGURE_ENV= CPPFLAGS="-I${LOCALBASE}/include" \
- LDFLAGS="-L${LOCALBASE}/lib" \
- cyrus_cv_sse42=no
+ LDFLAGS="-L${LOCALBASE}/lib"
CONFIGURE_ARGS=--bindir=${PREFIX}/cyrus/bin \
--libexec=${PREFIX}/cyrus/libexec \
--sbindir=${PREFIX}/cyrus/sbin \
@@ -48,17 +47,12 @@ CONFIGURE_ARGS= --bindir=${PREFIX}/cyrus/bin \
--with-cyrus-user=_cyrus \
--with-syslogfacility=MAIL \
--without-chardet \
- --without-cld2 \
--without-sphinx-build \
--without-zeroskip \
--disable-gssapi \
--enable-autocreate \
--enable-idled \
- --enable-murder \
- --enable-nntp
-
-# XXX FLAVOR
-CONFIGURE_ARGS += --without-snmp
+ --enable-murder
# XXX notyet; FLAVOR
CONFIGURE_ARGS += --without-clamav \
diff --git patches/patch-imap_conversations_c patches/patch-imap_conversations_c
deleted file mode 100644
index 9eab9396e0d..000
--- patches/patch-imap_conversations_c
+++ /dev/null
@@ -1,16 +0,0 @@
-$OpenBSD: patch-imap_conversations_c,v 1.3 2020/05/14 12:26:39 ajacoutot Exp $
-
-64 bit time_t
-
-Index: imap/conversations.c
imap/conversations.c.orig
-+++ imap/conversations.c
-@@ -567,7 +567,7 @@ static int _conversations_set_key(struct conversations
- if (i) buf_putc(, ',');
- buf_printf(, CONV_FMT, cid);
- }
--buf_printf(, " %lu", stamp);
-+buf_printf(, " %lld", stamp);
-
- r = cyrusdb_store(state->db,
- key, keylen,
diff --git patches/patch-imap_fud_c patches/patch-imap_fud_c
deleted file mode 100644
index cc6a8f8d327..000
--- patches/patch-imap_fud_c
+++ /dev/null
@@ -1,17 +0,0 @@
-$OpenBSD: patch-imap_fud_c,v 1.2 2020/08/28 09:53:04 ajacoutot Exp $
-
-Index: imap/fud.c
imap/fud.c.orig
-+++ imap/fud.c
-@@ -96,8 +96,10 @@ static void send_reply(struct sockaddr *sfrom, socklen
-
- static int soc = 0; /* inetd (master) has handed us the port as stdin */
-
--#ifndef MAXDOMNAME
-+#ifndef MAXLOGNAME
- #define MAXLOGNAME 16 /* should find out for real */
-+#endif
-+#ifndef MAXDOMNAME
- #define MAXDOMNAME 20 /* should find out for real */
- #endif
-
diff --git patches/patch-imap_mailbox_c patches/patch-imap_mailbox_c
deleted file mode 100644
index 0793441fdfa..000
--- patches/patch-imap_mailbox_c
+++ /dev/null
@@ -1,34 +0,0 @@
-$OpenBSD: patch-imap_mailbox_c,v 1.19 2020/05/30 10:09:27 ajacoutot Exp $
-
-64 bit time_t
-
-Index: imap/mailbox.c
imap/mailbox.c.orig
-+++ imap/mailbox.c
-@@ -2899,7 +2899,7 @@ static uint32_t crc_basic(const struct mailbox *mailbo
- flagcrc ^= crc32_cstring(buf);
- }
-
--snprintf(buf, sizeof(buf), "%u " MODSEQ_FMT " %lu (%u) %lu %s",
-+snprintf(buf, sizeof(buf), "%u " MODSEQ_FMT " %lld (%u) %lld %s",
- record->uid, record->modseq, record->last_updated,
- flagcrc,
- record->internaldate,
-@@ -2959,7 +2959,7 @@ static uint32_t crc_virtannot(struct mailbox *mailbox,
- }
-
- if (record->savedate && mailbox->i.minor_version >= 15) {
--buf_printf(, "%lu", record->savedate);
-+buf_printf(, "%lld", record->savedate);
- crc ^= crc_annot(record->uid, IMAP_ANNOT_NS "savedate", "", );
- buf_reset();
- }
-@@ -4520,7 +4520,7 @@ static int mailbox_index_repack(struct mailbox *mailbo
- if (mailbox->i.minor_version >= 15
Re: cyrus-imapd upstreamed patches and improvements
Stuart,
Thanks again for all the explanations!
> Yes. Try www/lighttpd for a simple example of this (vim is a poor example to
> crib from as it has a combination of multipackages *and* flavours).
>
> games/cataclysm-dda is a more complex example (though still just flavours not
> multipackages) using PFRAG.no_x11 for files which are only in the no_x11
> flavour, PFRAG.no-no_x11 for files which are _not_ in the no_x11 flavour, and
> PLIST has the files common to both flavours.
OK, this part is completely clear now, thanks!
> With flavours, all of the individual choices should be built as part of a bulk
> build, to make sure that any breakage in non-default options is picked up,
> plus whatever combinations are useful for real-world use should be built too.
Do I understand it correctly that apart from the common use-case flavor
combinations, there should be built an all-on combination too to test that
everything builds as expected?
> There is also ellie's reply, which makes a lot of sense at least in
> cases where it doesn't pull in a bunch of extra dependencies:
>
>"I would simply provide one package with all the
> stable/non-experimental features enabled, and admins can
> just not use the features they don't need."
Ellie is the one who prepares cyrus-imapd releases. We worked with her a lot on
all the fixes for the past 6 months, she has a lot of experience and I tend to
agree with her on most things, we also recently updated together [1] the
"compiling cyrus-imapd" doc page [2] with all the latest options and
non-obvious dependencies, but in this case I have to disagree.
The situation with cyrus-imapd is something like this. Let's see all relevant
configure features available as returned by the port's current cyr_buildinfo
(I removed some irrelevant ones that are always on or don't apply):
{
"component": {
"gssapi": false,
"autocreate": true,
"idled": true,
"httpd": false,
"kerberos_v4": false,
"murder": true,
"nntpd": true,
"replication": false,
"sieve": true,
"calalarmd": false,
"jmap": false,
"objectstore": false,
"backup": false,
},
"dependency": {
"ldap": false,
"clamav": false,
"ucdsnmp": false,
"netsnmp": false,
"openio": false,
"nghttp2": false,
"wslay": false,
"brotli": false,
"xml2": false,
"ical": false,
"shapelib": false,
"chardet": false
},
"database": {
"mysql": false,
"pgsql": false,
"sqlite": false
},
"search": {
"squat": true,
"xapian": false
}
}
For the empty base flavor I'd turn off nntp (IMO should always be turned off,
not sure if someone is still using it at all nowadays, it's an old code base
with a previous 7.5 score CVE), murder and squat as these are not the essential
features (but save murder for later).
With this combination we have a base POP3/IMAP server that is fully functional
and is suitable for use-cases like hosting providers that offer their clients
basic POP3/IMAP email accounts without anything special, but they require the
setup to be as simple and as solid as possible.
This combination has a few direct dependencies: pcre sqlite3 cyrus-sasl icu4c
e2fsprogs jansson + 2 indirect ones: libiconv and gettext-runtime. Quite a clean
install.
Now, if we turn on all the available options, we'd have dozens of direct
dependencies and a totally unnecessarily bloated active code-base. So IMO the
starting point should be the trimmed down basic IMAP server with as few
dependencies as possible.
The next use-case is a setup for small/mid-sized organizations that, apart from
basic IMAP, also want remote/shared calendaring support, reminders and
server-side scheduling with free/busy states (CalDAV), remote/shared contacts
(CardDAV), remote/shared folders (WebDAV), etc.
All this is enabled with --enable-http option and it brings some 12 additional
direct dependencies, but IMO together with the base IMAP, this is the most
typically used deployment: solid, easy to setup and works well with most
widely-used clients.
Then the next use-case is for larger organizations that have enough resources
(both monetary and human) to run cyrus-imapd on multiple servers with
duplication, triplication, .. of mailstore. For them there are the following
options:
--enable-murder: server aggregation in an imap cluster - a separation of front
and back-end servers, sharding of the mailstore, HA, etc.,
--enable-replication: replication of the mailstore between multiple machines,
and
--enable-backup: an experimental feature similar to the replication that runs a
backup instance on another server and duplicates there the mailstore for
non-realtime backups.
I'd group these 3 features together under the "replication" combination as they
share the same concept, could be used together and all imply certain resources
and know-how to deploy these setups.
So I'd provide the 3 combinations (base, http, replication) as pre-built
binaries. And
Re: cyrus-imapd upstreamed patches and improvements
Stuart,
Thanks a lot for the explanation! Much appreciated!
cyrus-imapd project is mostly modular (e.g. if it's built with --enable-http,
there would be additional binary httpd), but there are also some elements that
hold the compilation information, like cyr_buildinfo [1] which returns the
details about the components, dependencies, etc. that were activated or not
during the build:
$ cyr_buildinfo
{
"component": {
"event_notification": false,
"gssapi": false,
"autocreate": true,
"idled": true,
"httpd": true,
"kerberos_v4": false,
"murder": false,
"nntpd": false,
"replication": false,
"sieve": true,
"calalarmd": false,
"objectstore": false,
"backup": false
},
"dependency": {
"ldap": false,
"openssl": true,
"pcre": true,
"clamav": false
},
"database": {
"mysql": false,
"pgsql": false,
"sqlite": true,
"lmdb": false
},
"search": {
"squat": false,
"sphinx": false,
"xapian": false,
"xapian_flavor": "none"
},
"hardware": {
"sse42": true
}
}
I've asked [2] the devs about the extent of the modificaions that the same files
experience with different options. They are all affected by config.h include
holding the configuration options.
There's also a libcyrus library that has all the API definitions for the
requested functionality that builds conditionally. Part of the reply from a dev
from [2]:
> If you've built it with everything included, then there is no particular
> advantage to packaging up subsets of that. You might think "we can avoid
> bloat by excluding the features we don't need", but in this case all the APIs
> to support those features would still be in libcyrus, you would just be
> missing binaries. You could have achieved the same effect by just not running
> those services in cyrus.conf
I guess it would be more appropriate to proceed with flavors.
I was thinking about actually defining a number of flavors (basically for most
components and dependencies as returned by cyr_buildinfo), but to build only 4
combinations as part of the build infrastructure (the most clear use-cases), so
the rest of possible combinations could be activated by the users in the ports.
Does that sound good?
And if I understood it correctly, I'd have to use PFRAGs and %%flavor%% in PLIST
to define the build differences for each flavor, right?
Regards,
Anatoli
[1]
https://www.cyrusimap.org/imap/reference/manpages/systemcommands/cyr_buildinfo.html
[2]
https://cyrus.topicbox.com/groups/devel/T5acbcb9536dc47c3/cyrus-imapd-packaging-flavors-variations
On 22/11/20 10:53, Stuart Henderson wrote:
> On 2020/11/22 00:27, Anatoli wrote:
>> Then, I'm working now on the flavored version of the port, and my idea is to
>> apply it as soon as the new minor version is published (or maybe even before
>> this so not to deal with the REVISION) but this is my first time working with
>> ports, so I have no experience with the process and I have a couple of
>> questions
>> about some aspects of flavors definition.
>
> Is the build "modular"? That is, do the build options just have it build
> additional features in separate files (.so modules etc), or do they change
> the main binaries too?
>
> For builds which are modular, generally we do a single build and split
> the various files into subpackages. See for example PHP, Asterisk,
> Dovecot. MULTI_PACKAGES is defined with a list of subpackage names,
> the port is built once, packages are created for each subpackage
> using the relevant "PLIST-sub" file.
>
> (Heading off something you will see in some of these ports - there are
> "no_*" pseudo-flavours in some of these which are an optimization to
> reduce build dependencies for people building themselves rather than
> using packages - ignore these initially, they would be something to add
> later, if at all).
>
>> 1: vim-8.2.1805p0-gtk2
>> 2: vim-8.2.1805p0-gtk2-lua
>> 3: vim-8.2.1805p0-gtk2-perl-python-ruby
>> 4: vim-8.2.1805p0-gtk2-perl-python3-ruby
>> 5: vim-8.2.1805p0-gtk3
>> 6: vim-8.2.1805p0-gtk3-lua
>> 7: vim-8.2.1805p0-gtk3-perl-python-ruby
>> 8: vim-8.2.1805p0-gtk3-perl-python3-ruby
>> 9: vim-8.2.1805p0-no_x11-lua
>> 10: vim-8.2.1805p0-no_x11
>> 11: vim-8.2.1805p0-no_x11-perl-python-ruby
>> 12: vim-8.2.1805p0-no_x11-perl-python3-ruby
>> 13: vim-8.2.1805p0-no_x11-python
>> 14: vim-8.2.1805p0-no_x11-python3
>> 15: vim-8.2.1805p0-no_x11-ruby
>
> Solène explained how this list is defined. vim (and some other things
> like mutt) are just "either/or", so there's no alternative to building
> it multiple times. But it's better to avoid building what is essentially
> the same port 15 times if possible :)
>
>> Then I have some other questions like what does 'M'(option1) and 'L' mean in
>> places like: .if ${FLAVOR:Moption1} or .if ${FLAVOR:L:Mcrypto}? I've seen
>> other
>> letters (F) being used in a similar way, but haven't seen anything in the
Re: cyrus-imapd upstreamed patches and improvements
On 2020/11/23 07:31, Anatoli wrote: > Stuart, > > Thanks a lot for the explanation! Much appreciated! > > cyrus-imapd project is mostly modular (e.g. if it's built with --enable-http, > there would be additional binary httpd), but there are also some elements that > hold the compilation information, like cyr_buildinfo [1] which returns the > details about the components, dependencies, etc. that were activated or not > during the build: ok, so subpackages won't work for this - seems it also links libcyrus against all the various libraries used by the various components. (last time I ran cyrus imapd it was in 2.2.something days so I'm not familiar with current development :-) > > If you've built it with everything included, then there is no particular > > advantage to packaging up subsets of that. You might think "we can avoid > > bloat by excluding the features we don't need", but in this case all the > > APIs > > to support those features would still be in libcyrus, you would just be > > missing binaries. You could have achieved the same effect by just not > > running > > those services in cyrus.conf > > I guess it would be more appropriate to proceed with flavors. There is also ellie's reply, which makes a lot of sense at least in cases where it doesn't pull in a bunch of extra dependencies: "I would simply provide one package with all the stable/non-experimental features enabled, and admins can just not use the features they don't need." > I was thinking about actually defining a number of flavors (basically for most > components and dependencies as returned by cyr_buildinfo), but to build only 4 > combinations as part of the build infrastructure (the most clear use-cases), > so > the rest of possible combinations could be activated by the users in the > ports. > > Does that sound good? With flavours, all of the individual choices should be built as part of a bulk build, to make sure that any breakage in non-default options is picked up, plus whatever combinations are useful for real-world use should be built too. When the port is updated later, the set of different "standard build" flavours should be tested. When deciding what flavours to support, this extra work for every update needs factoring in :-) Additionally consider the case where new users install the software; pkg_add will present the big list of options which can be a bit overwhelming if a flavour is provided for every build option. Generally my approach would be to add flavours for things which are fairly popular but that also some users will really not want to use. This could either be "positive" flavours to enable those features, or "negative" flavours i.e. "no_whatever" to disable them (for example many ports have a no_x11 flavour). I would tend towards "negative" for core features that some people may want to avoid (considering cyrus then http might be a candidate for this) - and "positive" for things pulling in heavier-weight extra dependencies. > And if I understood it correctly, I'd have to use PFRAGs and %%flavor%% in > PLIST > to define the build differences for each flavor, right? Yes. Try www/lighttpd for a simple example of this (vim is a poor example to crib from as it has a combination of multipackages *and* flavours). games/cataclysm-dda is a more complex example (though still just flavours not multipackages) using PFRAG.no_x11 for files which are only in the no_x11 flavour, PFRAG.no-no_x11 for files which are _not_ in the no_x11 flavour, and PLIST has the files common to both flavours. > Anatoli > > [1] > https://www.cyrusimap.org/imap/reference/manpages/systemcommands/cyr_buildinfo.html > [2] > https://cyrus.topicbox.com/groups/devel/T5acbcb9536dc47c3/cyrus-imapd-packaging-flavors-variations > > > On 22/11/20 10:53, Stuart Henderson wrote: > > On 2020/11/22 00:27, Anatoli wrote: > >> Then, I'm working now on the flavored version of the port, and my idea is > >> to > >> apply it as soon as the new minor version is published (or maybe even > >> before > >> this so not to deal with the REVISION) but this is my first time working > >> with > >> ports, so I have no experience with the process and I have a couple of > >> questions > >> about some aspects of flavors definition. > > > > Is the build "modular"? That is, do the build options just have it build > > additional features in separate files (.so modules etc), or do they change > > the main binaries too? > > > > For builds which are modular, generally we do a single build and split > > the various files into subpackages. See for example PHP, Asterisk, > > Dovecot. MULTI_PACKAGES is defined with a list of subpackage names, > > the port is built once, packages are created for each subpackage > > using the relevant "PLIST-sub" file. > > > > (Heading off something you will see in some of these ports - there are > > "no_*" pseudo-flavours in some of these which are an optimization to > > reduce build dependencies for people
Re: cyrus-imapd upstreamed patches and improvements
Solene, Thanks! Now everything is clear! :) On 22/11/20 07:43, Solene Rapenne wrote: > On Sun, 22 Nov 2020 00:27:42 -0300 > Anatoli : > >> Hi Antoine, Stuart, All, >> >> E.g. in the vim port there are these flavors defined: >> >> 35:FLAVORS+=gtk2 gtk3 no_x11 >> 37:FLAVORS+=lua perl python python3 ruby >> >> But I can't find how they end up being those flavor options available to the >> users for install via pkg_add: > > The various combinations are set in the Makefile of > the port's category, for vim it's in ports/editors/Makefile > > http://cvsweb.openbsd.org/ports/editors/Makefile?rev=1.142=text/x-cvsweb-markup > > SUBDIR += vim,gtk2 > SUBDIR += vim,gtk2,lua > SUBDIR += vim,gtk2,perl,python,ruby > SUBDIR += vim,gtk2,perl,python3,ruby > SUBDIR += vim,gtk3 > SUBDIR += vim,gtk3,lua > SUBDIR += vim,gtk3,perl,python,ruby > SUBDIR += vim,gtk3,perl,python3,ruby > SUBDIR += vim,no_x11 > SUBDIR += vim,no_x11,lua > SUBDIR += vim,no_x11,python > SUBDIR += vim,no_x11,python3 > SUBDIR += vim,no_x11,perl,python,ruby > SUBDIR += vim,no_x11,perl,python3,ruby >
Re: cyrus-imapd upstreamed patches and improvements
On Sun, 22 Nov 2020 00:27:42 -0300 Anatoli : > Hi Antoine, Stuart, All, > > E.g. in the vim port there are these flavors defined: > > 35:FLAVORS+=gtk2 gtk3 no_x11 > 37:FLAVORS+=lua perl python python3 ruby > > But I can't find how they end up being those flavor options available to the > users for install via pkg_add: The various combinations are set in the Makefile of the port's category, for vim it's in ports/editors/Makefile http://cvsweb.openbsd.org/ports/editors/Makefile?rev=1.142=text/x-cvsweb-markup SUBDIR += vim,gtk2 SUBDIR += vim,gtk2,lua SUBDIR += vim,gtk2,perl,python,ruby SUBDIR += vim,gtk2,perl,python3,ruby SUBDIR += vim,gtk3 SUBDIR += vim,gtk3,lua SUBDIR += vim,gtk3,perl,python,ruby SUBDIR += vim,gtk3,perl,python3,ruby SUBDIR += vim,no_x11 SUBDIR += vim,no_x11,lua SUBDIR += vim,no_x11,python SUBDIR += vim,no_x11,python3 SUBDIR += vim,no_x11,perl,python,ruby SUBDIR += vim,no_x11,perl,python3,ruby
Re: cyrus-imapd upstreamed patches and improvements
On 2020/11/22 00:27, Anatoli wrote:
> Then, I'm working now on the flavored version of the port, and my idea is to
> apply it as soon as the new minor version is published (or maybe even before
> this so not to deal with the REVISION) but this is my first time working with
> ports, so I have no experience with the process and I have a couple of
> questions
> about some aspects of flavors definition.
Is the build "modular"? That is, do the build options just have it build
additional features in separate files (.so modules etc), or do they change
the main binaries too?
For builds which are modular, generally we do a single build and split
the various files into subpackages. See for example PHP, Asterisk,
Dovecot. MULTI_PACKAGES is defined with a list of subpackage names,
the port is built once, packages are created for each subpackage
using the relevant "PLIST-sub" file.
(Heading off something you will see in some of these ports - there are
"no_*" pseudo-flavours in some of these which are an optimization to
reduce build dependencies for people building themselves rather than
using packages - ignore these initially, they would be something to add
later, if at all).
> 1: vim-8.2.1805p0-gtk2
> 2: vim-8.2.1805p0-gtk2-lua
> 3: vim-8.2.1805p0-gtk2-perl-python-ruby
> 4: vim-8.2.1805p0-gtk2-perl-python3-ruby
> 5: vim-8.2.1805p0-gtk3
> 6: vim-8.2.1805p0-gtk3-lua
> 7: vim-8.2.1805p0-gtk3-perl-python-ruby
> 8: vim-8.2.1805p0-gtk3-perl-python3-ruby
> 9: vim-8.2.1805p0-no_x11-lua
> 10: vim-8.2.1805p0-no_x11
> 11: vim-8.2.1805p0-no_x11-perl-python-ruby
> 12: vim-8.2.1805p0-no_x11-perl-python3-ruby
> 13: vim-8.2.1805p0-no_x11-python
> 14: vim-8.2.1805p0-no_x11-python3
> 15: vim-8.2.1805p0-no_x11-ruby
Solène explained how this list is defined. vim (and some other things
like mutt) are just "either/or", so there's no alternative to building
it multiple times. But it's better to avoid building what is essentially
the same port 15 times if possible :)
> Then I have some other questions like what does 'M'(option1) and 'L' mean in
> places like: .if ${FLAVOR:Moption1} or .if ${FLAVOR:L:Mcrypto}? I've seen
> other
> letters (F) being used in a similar way, but haven't seen anything in the docs
> explaining what they mean.
:L is lowercase, :M is match, these are documented in make(1), "Each
modifier begins with a colon and [...]"
F isn't in this list, is that just a variable defined in a Makefile
somewhere?
> How PLIST, DESCR files should be prepared and what's the purpose of the
> PFRAG.xx
> files? Looks like PLIST and DESCR could have versions for multi-packages, but
> not for the flavors, though PFRAG files appear to be flavorable, but not sure
> how exactly and what content/format they should have.
PFRAGs handle files which exist only in certain flavours of the port.
If a port has a flavour "foo" and PLIST has a %%foo%% entry, PFRAG.foo
is inserted into the PLIST at that point only when the build is done using
that flavour. Likewise if there is a !%%foo%%, PFRAG.no-foo is inserted
for all flavours *except* foo.
(This mechanism can also handle the case where there are differences
in the file list between different machine archs by defining a variable
for pkg_create - you won't need it here but for an example see how
PKG_ARGS is set in lang/go-bootstrap).
> So how should I update the PLIST in order to accommodate flavor differences if
> it's a single version for all flavors? And should I create other files
> (PFRAG)?
>
>
> And a side note: this port probably should NOT be built on 32-bit archs due to
> the devs supposition that all time-related structures are 64-bit and there
> could
> be security implications if this is not the case (signedness and int
> overflow).
> Here's a discussion about this without a definitive conclusion [6].
This only relates to the size of time_t, not the whole architecture.
NetBSD changed time_t to 64-bit on all architectures in 6.0 (2012),
OpenBSD did the same in 5.5 (2014). Linux now has support too (in 5.6,
this year) but maintained the old ABI and with the long term
binary/library compatibility I guess it will be quite some time before
software actually running on most people's machines will be rebuilt for
this. Looks like FreeBSD still uses 32-bit on i386 but 64-bit on others
(including non-i386 32-bit).
We still run into bugs in application software that assumes time_t is
"long", especially in format strings (the standard approach we take in
ports when patching things is to use long long format strings, and cast
the variable to long long so that the patch can be fed upstream without
breaking things on other OS). Presumably as Cyrus devs are aware of
this they will be taking reasonable care!
Re: cyrus-imapd upstreamed patches and improvements
Hi Antoine, Stuart, All,
Stuart, thanks for updating wslay.
Antoine, thanks for your feedback.
The diff from my first mail is to be applied to the port when the upstream
publishes the new minor release in the 3.2 branch, which will be 3.2.5 and I
guess it should be released in a week.
The diff adjusts the port to accommodate all the cross-platform improvements
that we incorporated since the last 3.2.4 release (it mostly removes the no more
needed patches and slightly adjusts the PLIST to accommodate some minor
CONFIGURE_ARGS changes).
Then, I'm working now on the flavored version of the port, and my idea is to
apply it as soon as the new minor version is published (or maybe even before
this so not to deal with the REVISION) but this is my first time working with
ports, so I have no experience with the process and I have a couple of questions
about some aspects of flavors definition.
I've read the docs [1], [2], [3], [4] and [5] (not sure if there are other
places documenting it), but I can't find where the final flavor combinations are
defined.
E.g. in the vim port there are these flavors defined:
35:FLAVORS+=gtk2 gtk3 no_x11
37:FLAVORS+=lua perl python python3 ruby
But I can't find how they end up being those flavor options available to the
users for install via pkg_add:
1: vim-8.2.1805p0-gtk2
2: vim-8.2.1805p0-gtk2-lua
3: vim-8.2.1805p0-gtk2-perl-python-ruby
4: vim-8.2.1805p0-gtk2-perl-python3-ruby
5: vim-8.2.1805p0-gtk3
6: vim-8.2.1805p0-gtk3-lua
7: vim-8.2.1805p0-gtk3-perl-python-ruby
8: vim-8.2.1805p0-gtk3-perl-python3-ruby
9: vim-8.2.1805p0-no_x11-lua
10: vim-8.2.1805p0-no_x11
11: vim-8.2.1805p0-no_x11-perl-python-ruby
12: vim-8.2.1805p0-no_x11-perl-python3-ruby
13: vim-8.2.1805p0-no_x11-python
14: vim-8.2.1805p0-no_x11-python3
15: vim-8.2.1805p0-no_x11-ruby
Why, for example, lua is never combined with perl-python-ruby, etc. I don't see
where this restriction is defined.
[3] says: The port maintainer will set FLAVORS to be the list of possible
options in the Makefile. When building the port, the package builder will set
FLAVOR='option1 option2...' to build a specific flavor of the port.
Does that mean that someone responsible for building the packages will set the
combinations manually and the maintainer doesn't have control over the versions
built? And if this is the case, what's the process of communicating to the build
team what flavor combinations make sense for the port?
I'm planning to define the following flavors for cyrus-imapd:
* an empty flavor (the default) with the options like it is now (with my
initial diff applied that makes minor adjustments to the CONFIGURE_ARGS);
* http: a http-enabled build which includes *DAV (WebDAV, CalDAV & CardDAV)
functionality as well as the new IMAP replacement protocol JMAP;
* replication: murder, replication and backup functionality;
They could be combined freely:
empty
http
replication
http-replication
I am using just the http flavor, so not sure if other combinations make sense to
the users (e.g. clamav integration, different storage options, databases, etc.).
Maybe we could enable just those listed above (e.g. the empty one, http,
replication and http-replication) and then ask users for feedback (I'm willing
to process all the feedback and adapt the port accordingly).
Then I have some other questions like what does 'M'(option1) and 'L' mean in
places like: .if ${FLAVOR:Moption1} or .if ${FLAVOR:L:Mcrypto}? I've seen other
letters (F) being used in a similar way, but haven't seen anything in the docs
explaining what they mean.
How PLIST, DESCR files should be prepared and what's the purpose of the PFRAG.xx
files? Looks like PLIST and DESCR could have versions for multi-packages, but
not for the flavors, though PFRAG files appear to be flavorable, but not sure
how exactly and what content/format they should have.
So how should I update the PLIST in order to accommodate flavor differences if
it's a single version for all flavors? And should I create other files (PFRAG)?
And a side note: this port probably should NOT be built on 32-bit archs due to
the devs supposition that all time-related structures are 64-bit and there could
be security implications if this is not the case (signedness and int overflow).
Here's a discussion about this without a definitive conclusion [6].
Regards,
Anatoli
[1] https://man.openbsd.org/ports.7#FLAVORS
[2] https://www.openbsd.org/faq/ports/ports.html#PortsFlavors
[3] https://man.openbsd.org/bsd.port.mk.5#FLAVORS_AND_MULTI_PACKAGES
[4] https://www.openbsd.org/faq/ports/guide.html
[5] https://man.openbsd.org/pkgpath.7
[6] https://github.com/cyrusimap/cyrus-imapd/pull/3262#issuecomment-723680233
On 19/11/20 09:16, Antoine Jacoutot wrote:
> On Wed, Nov 18, 2020 at 03:44:35PM -0300, Anatoli wrote:
>> Hi Antoine, all.
>>
>> For some months I was updating the cyrus-imapd port, upstreaming the
>> upstreamable patches and working
Re: cyrus-imapd upstreamed patches and improvements
On Wed, Nov 18, 2020 at 03:44:35PM -0300, Anatoli wrote: > Hi Antoine, all. > > For some months I was updating the cyrus-imapd port, upstreaming the > upstreamable patches and working with the upstream to improve additional > features of the port as follows. Hi Anatoli. That's awesome work, thanks for taking the time to work with upstream on this. I must say that I manage so many ports that I cannot find the time / motivation to work will all upstreams, all with a different contribution process etc. What's your next step on this? Wait until a new stable release is out and then we can update the port with your changes? Thanks again. -- Antoine
cyrus-imapd upstreamed patches and improvements
Hi Antoine, all.
For some months I was updating the cyrus-imapd port, upstreaming the
upstreamable patches and working with the upstream to improve additional
features of the port as follows.
Upstreamed patches [1]:
* patch-configure
* patch-lib_byteorder64_h
* patch-imap_fud_c
You already removed the first 2 patches in the latest port update and modified a
bit the 3rd one. I've also upstreamed that modification. [1.1]
A workaround for the lack of deflatePending in zlib was implemented [2][3] which
is needed for the http component of cyrus-imapd (not yet enabled in the port).
CLD2 dependency checks were removed [4].
SNMP was finally removed from cyrus-imapd [5] (metrics are implemented with
Prometheus now).
SSE4.2 implementation for CRC32 was removed [6].
A script that was part of the autoconf setup had a hardcoded #!/bin/bash which
was preventing the rebuild of configure & co on OpenBSD. Fixed [7].
I also disabled nntp. Not sure if you know someone still using it, but it had a
critical vuln some time ago, so if nobody is using it, I guess it's better to
disable it.
I'm including below a patch for all these changes in the port.
Then there are more changes already upstreamed, that didn't make it into the
3.2.4 release, though they are already in the 3.2 branch and should be included
in the next 3.2.5 release. These are related to the format specifier differences
on Linux and OpenBSD, i.e. format specifiers in printf/scanf for types like
time_t, off_t, etc. (%ld vs %ldd). [8]
The cross-platform fixes that we implemented with the upstream [9] remove the
following patches for non-http build:
patch-imap_conversations_c
patch-imap_mailbox_c
patch-imap_mbexamine_c
patch-imap_seen_db_c
patch-imap_sync_support_c
patch-imtest_imtest_c
patch-lib_auth_pts_c
patch-lib_prot_c
patch-master_master_c
patch-notifyd_notify_mailto_c
patch-sieve_sieved_c
and fix about 60 related additional issues and warnings for a http-enabled
build.
We have also determined the cause for the 37 "unknown attribute 'optimize'
ignored" warning and fixed it upstream. The fix will be in the 3.2.5. [10]
After all this, the only patches that would remain in the port would be the ones
that fix the paths specific to OpenBSD in man pages and some scripts (e.g.
/usr/bin vs /usr/local/bin), not sure if there's a way to upstream them too.
At this moment I'm also preparing a diff for the cyrus-imapd flavors, though I'd
probably wait for the 3.2.5 to have a clean base before sending the diff.
The JMAP subcomponent of the http component of cyrus-imapd has a dependency on
wslay-1.1.1. The wslay v1.1.0 was missing one change on which cyrus-imapd is
relying [11], so I asked [12] the wslay maintainer to release a new version,
which he has done about 4 months ago.
Could you please update it in the ports? The current version is still 1.1.0.
Regards,
Anatoli
[1] https://github.com/cyrusimap/cyrus-imapd/pull/3129
[1.1] https://github.com/cyrusimap/cyrus-imapd/pull/3239
[2] https://www.mail-archive.com/cyrus-devel@lists.andrew.cmu.edu/msg04440.html
[3] https://github.com/cyrusimap/cyrus-imapd/pull/3061
[4] https://github.com/cyrusimap/cyrus-imapd/pull/3112
[5] https://github.com/cyrusimap/cyrus-imapd/issues/1765
[6] https://github.com/cyrusimap/cyrus-imapd/pull/3113
[7] https://github.com/cyrusimap/cyrus-imapd/issues/3143
[8] https://github.com/cyrusimap/cyrus-imapd/issues/3128
[9] https://github.com/cyrusimap/cyrus-imapd/pull/3262
[10] https://github.com/cyrusimap/cyrus-imapd/issues/3265
[11] https://www.mail-archive.com/cyrus-devel@lists.andrew.cmu.edu/msg04445.html
[12] https://github.com/tatsuhiro-t/wslay/issues/57
diff --git Makefile Makefile
index c7fb05ebcee..bfee0b835b1 100644
--- Makefile
+++ Makefile
@@ -39,8 +39,7 @@ LIB_DEPENDS= databases/sqlite3 \
CONFIGURE_STYLE= gnu
CONFIGURE_ENV= CPPFLAGS="-I${LOCALBASE}/include" \
- LDFLAGS="-L${LOCALBASE}/lib" \
- cyrus_cv_sse42=no
+ LDFLAGS="-L${LOCALBASE}/lib"
CONFIGURE_ARGS=--bindir=${PREFIX}/cyrus/bin \
--libexec=${PREFIX}/cyrus/libexec \
--sbindir=${PREFIX}/cyrus/sbin \
@@ -48,17 +47,12 @@ CONFIGURE_ARGS= --bindir=${PREFIX}/cyrus/bin \
--with-cyrus-user=_cyrus \
--with-syslogfacility=MAIL \
--without-chardet \
- --without-cld2 \
--without-sphinx-build \
--without-zeroskip \
--disable-gssapi \
--enable-autocreate \
--enable-idled \
- --enable-murder \
- --enable-nntp
-
-# XXX FLAVOR
-CONFIGURE_ARGS += --without-snmp
+ --enable-murder
# XXX notyet; FLAVOR
Re: cyrus-imapd upstreamed patches and improvements
On 2020/11/18 15:44, Anatoli wrote: > The JMAP subcomponent of the http component of cyrus-imapd has a dependency on > wslay-1.1.1. The wslay v1.1.0 was missing one change on which cyrus-imapd is > relying [11], so I asked [12] the wslay maintainer to release a new version, > which he has done about 4 months ago. > > Could you please update it in the ports? The current version is still 1.1.0. i've updated wslay.
