Re: [zeromq-dev] ZeroMQ 4.2 release, planning

[Luca Boccassi]

Sorry, I meant if we go with (1), not (2), we might bump the size as
well, since we are already doing another ABI-breaking change.

I agree on the solution as well.

On Mon, 2016-08-29 at 17:12 +0200, Pieter Hintjens wrote:
> I'm confused between the (1) and (2) choices, and can't see where
> bumping the message size fits.
> 
> Nonetheless, I think bumping the size, fixing the alignment issues,
> and bumping the ABI version is the best solution here.
> 
> On Fri, Aug 26, 2016 at 12:33 PM, Luca Boccassi  
> wrote:
> > I've given some more thoughts and testing to the alignment issue. I can
> > reproduce the problem by enabling alignment checks on x86 too.
> >
> > But most importantly, I think we cannot get away from bumping the ABI
> > with this fix, however we rearrange it, simply because applications need
> > to be rebuilt against the new header to be fixed. A simple rebuild of
> > the libzmq.so is not enough. And the way to do this is to bump the ABI
> > so that distros can schedule transitions and rebuilds and so on.
> >
> > So the choice list is now restricted to:
> >
> > 1) Bump ABI
> > 2) Revert the fix and leave everything broken on sparc64 and some
> > aarch64 (rpi3 seems not to be affected, must depend on the SoC flavour)
> >
> > If we go with 2, we might as well get 2 birds with one stone and bump
> > the zmq_msg_t size to 128 as we have talked about in the past.
> >
> > Doron, this would help with the new UDP based socket types right?
> >
> > Pros of bumping msg size:
> >
> > - we can get rid of the malloc() in the lmsg type case as all the data
> > will fit
> >
> > Cons:
> >
> > - for the vsm/cmsg type cases (for most architectures anyway) it won't
> > fit anymore into a single cacheline
> >
> > Given all this, I'd say we should go for it.
> >
> > Opinions?
> >
> > On Sat, 2016-08-13 at 16:59 +0100, Luca Boccassi wrote:
> >> Hello,
> >>
> >> Trying to give some thoughts again on the libzmq 4.2 release. It's
> >> really long overdue!
> >>
> >> The main issue from my point of view is this change:
> >>
> >> https://github.com/zeromq/libzmq/commit/d9fb1d36ff2008966af538f722a1f4ab158dbf64
> >>
> >> -typedef struct zmq_msg_t {unsigned char _ [64];} zmq_msg_t;
> >>  +/* union here ensures correct alignment on architectures that require
> >> it, e.g.
> >>  + * SPARC
> >>  + */
> >>  +typedef union zmq_msg_t {unsigned char _ [64]; void *p; } zmq_msg_t;
> >>
> >>
> >> This is flagged by the common ABI checkers tools as an ABI breakage
> >> (see: http://abi-laboratory.pro/tracker/timeline/zeromq/ ). And it makes
> >> sense from this point of view: if some applications on some
> >> architectures are broken due to wrong alignment, they would need to be
> >> rebuilt, and the way to ensure that is to bump the ABI "current" digit
> >> to make sure maintainers do a rebuild.
> >>
> >> On the other hand, signaling an ABI breakage is a pain, and a cause of
> >> major churn for packagers and maintainers. It means for example a new
> >> package has to be created (eg: libzmq5 -> libzmq6), and a transition has
> >> to be started and all reverse dependencies need to be rebuilt. And if
> >> this is pointless for all save a few corner cases (eg SPARC64 as for
> >> above) it's all quite frustrating.
> >>
> >> So we have a choice to make before we release 4.2, four possibilities as
> >> far as I can see:
> >>
> >> 1) Ignore the ABI checkers and get yelled at by maintainers and
> >> packagers. Also the SPARC64 users will most likely NOT get their bug
> >> fixed
> >> 2) Bump ABI revision to 6 and get yelled at by maintainers and packagers
> >> 3) Revert the above change and postpone it to when we have a more
> >> generally useful reason to break ABI (bump zmq_msg_t from 64 to 128
> >> bytes for example, Doron?)
> >> 4) Try to be clever and revert the above change and use something like
> >> #pragma pack(8). This will fool the ABI checkers (I tried it), and given
> >> that typedef is only used externally to allocate the right size it
> >> shouldn't actually affect anything, apart from the users of SPARC64
> >> which should get the bugfix with this too. This is very sneaky :-)
> >>
> >> CC'ing Lazslo, the Debian maintainer, given what we choose to do might
> >> result in a lot of work for him :-)
> >>
> >> Opinions?
> >>
> >> Kind regards,
> >> Luca Boccassi
> >>
> >> On Tue, 2016-05-03 at 10:39 +0200, Pieter Hintjens wrote:
> >> > Hi all,
> >> >
> >> > I'm just throwing some ideas on the table. We have a good package of
> >> > work on master and it's probably time to make a 4.2 release.
> >> >
> >> > Luca has already back-ported the enable/disable draft design from
> >> > zproject (CZMQ et al). Yay! So we can now release stable master
> >> > safely, while continuing to refine and extend the draft API sections.
> >> >
> >> > I propose:
> >> >
> >> > - to end with the stable fork policy; this was needed years ago when
> >> > we had massively unstable masters. It's no longer a problem.
> >> > - to use 

Re: [zeromq-dev] ZeroMQ 4.2 release, planning

[Pieter Hintjens]

I'm confused between the (1) and (2) choices, and can't see where
bumping the message size fits.

Nonetheless, I think bumping the size, fixing the alignment issues,
and bumping the ABI version is the best solution here.

On Fri, Aug 26, 2016 at 12:33 PM, Luca Boccassi  wrote:
> I've given some more thoughts and testing to the alignment issue. I can
> reproduce the problem by enabling alignment checks on x86 too.
>
> But most importantly, I think we cannot get away from bumping the ABI
> with this fix, however we rearrange it, simply because applications need
> to be rebuilt against the new header to be fixed. A simple rebuild of
> the libzmq.so is not enough. And the way to do this is to bump the ABI
> so that distros can schedule transitions and rebuilds and so on.
>
> So the choice list is now restricted to:
>
> 1) Bump ABI
> 2) Revert the fix and leave everything broken on sparc64 and some
> aarch64 (rpi3 seems not to be affected, must depend on the SoC flavour)
>
> If we go with 2, we might as well get 2 birds with one stone and bump
> the zmq_msg_t size to 128 as we have talked about in the past.
>
> Doron, this would help with the new UDP based socket types right?
>
> Pros of bumping msg size:
>
> - we can get rid of the malloc() in the lmsg type case as all the data
> will fit
>
> Cons:
>
> - for the vsm/cmsg type cases (for most architectures anyway) it won't
> fit anymore into a single cacheline
>
> Given all this, I'd say we should go for it.
>
> Opinions?
>
> On Sat, 2016-08-13 at 16:59 +0100, Luca Boccassi wrote:
>> Hello,
>>
>> Trying to give some thoughts again on the libzmq 4.2 release. It's
>> really long overdue!
>>
>> The main issue from my point of view is this change:
>>
>> https://github.com/zeromq/libzmq/commit/d9fb1d36ff2008966af538f722a1f4ab158dbf64
>>
>> -typedef struct zmq_msg_t {unsigned char _ [64];} zmq_msg_t;
>>  +/* union here ensures correct alignment on architectures that require
>> it, e.g.
>>  + * SPARC
>>  + */
>>  +typedef union zmq_msg_t {unsigned char _ [64]; void *p; } zmq_msg_t;
>>
>>
>> This is flagged by the common ABI checkers tools as an ABI breakage
>> (see: http://abi-laboratory.pro/tracker/timeline/zeromq/ ). And it makes
>> sense from this point of view: if some applications on some
>> architectures are broken due to wrong alignment, they would need to be
>> rebuilt, and the way to ensure that is to bump the ABI "current" digit
>> to make sure maintainers do a rebuild.
>>
>> On the other hand, signaling an ABI breakage is a pain, and a cause of
>> major churn for packagers and maintainers. It means for example a new
>> package has to be created (eg: libzmq5 -> libzmq6), and a transition has
>> to be started and all reverse dependencies need to be rebuilt. And if
>> this is pointless for all save a few corner cases (eg SPARC64 as for
>> above) it's all quite frustrating.
>>
>> So we have a choice to make before we release 4.2, four possibilities as
>> far as I can see:
>>
>> 1) Ignore the ABI checkers and get yelled at by maintainers and
>> packagers. Also the SPARC64 users will most likely NOT get their bug
>> fixed
>> 2) Bump ABI revision to 6 and get yelled at by maintainers and packagers
>> 3) Revert the above change and postpone it to when we have a more
>> generally useful reason to break ABI (bump zmq_msg_t from 64 to 128
>> bytes for example, Doron?)
>> 4) Try to be clever and revert the above change and use something like
>> #pragma pack(8). This will fool the ABI checkers (I tried it), and given
>> that typedef is only used externally to allocate the right size it
>> shouldn't actually affect anything, apart from the users of SPARC64
>> which should get the bugfix with this too. This is very sneaky :-)
>>
>> CC'ing Lazslo, the Debian maintainer, given what we choose to do might
>> result in a lot of work for him :-)
>>
>> Opinions?
>>
>> Kind regards,
>> Luca Boccassi
>>
>> On Tue, 2016-05-03 at 10:39 +0200, Pieter Hintjens wrote:
>> > Hi all,
>> >
>> > I'm just throwing some ideas on the table. We have a good package of
>> > work on master and it's probably time to make a 4.2 release.
>> >
>> > Luca has already back-ported the enable/disable draft design from
>> > zproject (CZMQ et al). Yay! So we can now release stable master
>> > safely, while continuing to refine and extend the draft API sections.
>> >
>> > I propose:
>> >
>> > - to end with the stable fork policy; this was needed years ago when
>> > we had massively unstable masters. It's no longer a problem.
>> > - to use the github release function for libzmq releases and deprecate
>> > the separate delivery of tarballs.
>> > - we aim to make a 4.2.0 rc asap, then fix any issues we get, with
>> > patch releases as usual.
>> > - we backport the release function to older maintained releases (4.1,
>> > 3.2) so that their tarballs are provided by github instead of
>> > downloads.zeromq.org.
>> >
>> > Problems:
>> >
>> > - this will break a few things that depend on