Re: [zeromq-dev] ZeroMQ 4.2 release, planning
Cool!
I can take care of it if you like. Tentative plan:
Tomorrow push an RC1 for libzmq, then the pr to CZMQ to retire v2 APIs,
then the RC1 for CZMQ.
If it's all good then a couple days later the finals. I would really like
to make it for the debian 9 transition freeze which is Saturday.
On Oct 31, 2016 22:23, "Doron Somech" wrote:
> Sorry, yes, lets do it :)
>
> On Oct 31, 2016 11:44 PM, "Luca Boccassi" wrote:
>
>> Ping :-)
>>
>> On Oct 28, 2016 18:48, "Luca Boccassi" wrote:
>>
>>> I have sent a solution for the alignment problem that solves the sigbus
>>> problem without breaking ABI compat (plus follow-up for VC++ - sorry
>>> Windows guys https://github.com/zeromq/libzmq/pull/2179 ).
>>>
>>> I tested the alignment and sigbus problem on x86_64 by enabling
>>> alignment check with:
>>>
>>> __asm__("pushf\norl $0x4,(%rsp)\npopf");
>>>
>>> All was fine.
>>>
>>> I ran tests built from the zeromq4-1 repository against a shared lib
>>> from the head of libzmq repo, and they all run fine minus the
>>> ZMQ_REQ_CORRELATE one but that option was borken anyway.
>>>
>>> This allows us to do a release now, and then when we are ready we can do
>>> the ABI breakage, without blocking 4.2. Which is nice since it means it
>>> might make it for Debian 9!
>>>
>>> So, Doron et al, shall we do the bump this weekend?
>>>
>>> On Thu, 2016-10-20 at 17:12 -0500, Thomas Rodgers wrote:
>>> > I will have some time most likely the week of Nov6 (off for a week of
>>> C++
>>> > Committee 'fun') to test different message size alternatives. I'll
>>> follow
>>> > up with my results here for consideration the next time we are
>>> inclined to
>>> > break the ABI compatibility :)
>>> >
>>> > On Sunday, October 16, 2016, Brian Knox
>>> wrote:
>>> >
>>> > > A new stable version would definitely help me in my quest to get
>>> ZeroMQ
>>> > > support enabled by default in rsyslog in distros.
>>> > >
>>> > > On Sun, Oct 16, 2016 at 2:40 PM Doron Somech
>>> wrote:
>>> > >
>>> > >> I say lets bump.
>>> > >>
>>> > >> On Oct 15, 2016 20:32, "Luca Boccassi"
>>> wrote:
>>> > >>
>>> > >>> As Thomas said, false sharing would be a real issue with 96.
>>> > >>>
>>> > >>> So given a release is long due, at this point I'd say to drop this
>>> for
>>> > >>> the moment.
>>> > >>>
>>> > >>> What do we do for the change to union for zmq_msg_t? Bump ABI
>>> version or
>>> > >>> not?
>>> > >>>
>>> > >>> On Thu, 2016-10-06 at 09:53 +0300, Doron Somech wrote:
>>> > >>> > No new socket type, I worked at the time on binary message type,
>>> might
>>> > >>> > complete it sometime, but it is not urgent.
>>> > >>> >
>>> > >>> > If there is a lot of performance penalty we can give it up, I
>>> will
>>> > >>> > find another solution for the Radio-Dish.
>>> > >>> >
>>> > >>> > What about 96 bytes? same penalty?
>>> > >>> >
>>> > >>> > Regarding the binding, I'm not sure.
>>> > >>> >
>>> > >>> > On Sat, Oct 1, 2016 at 9:14 PM, Luca Boccassi <
>>> luca.bocca...@gmail.com>
>>> > >>> wrote:
>>> > >>> > > On Tue, 2016-09-27 at 09:41 +0300, Doron Somech wrote:
>>> > >>> > >> Sorry for the late response, increasing the msg_t structure
>>> will be
>>> > >>> > >> great, however this will require changing a lot of binding.
>>> > >>> > >
>>> > >>> > > I think I remember we need it for the new socket types, is that
>>> > >>> correct?
>>> > >>> > >
>>> > >>> > > There is a large performance penalty (intuitively due to not
>>> fitting
>>> > >>> > > into a single cache line anymore, but haven't ran
>>> perf/cachegrind),
>>> > >>> and
>>> > >>> > > the throughput with vsm type messages goes down by 4% (min)
>>> and 20%
>>> > >>> > > (max) for TCP, and 36% (min) 38 (max) for inproc, which is
>>> quite a
>>> > >>> lot,
>>> > >>> > > so we need to be sure it's worth it.
>>> > >>> > >
>>> > >>> > > Regarding the bindings, after a quick search on the Github
>>> org, I
>>> > >>> could
>>> > >>> > > only see:
>>> > >>> > >
>>> > >>> > > https://github.com/zeromq/lzmq/blob/master/src/lua/lzmq/
>>> > >>> ffi/api.lua#L144
>>> > >>> > > https://github.com/zeromq/clrzmq4/blob/master/lib/zmq.cs#L28
>>> > >>> > > https://github.com/zeromq/pyczmq/blob/master/pyczmq/zmq.py#L
>>> 177
>>> > >>> > >
>>> > >>> > > Other bindings just import zmq.h. Did I miss any?
>>> > >>> > >
>>> > >>> > >> Sorry for disappearing, baby and full time job is a lot :-),
>>> > >>> hopefully
>>> > >>> > >> I'm back...
>>> > >>> > >
>>> > >>> > > No worries, perfectly understandable :-)
>>> > >>> > >
>>> > >>> > >> On Mon, Aug 29, 2016 at 6:46 PM, Luca Boccassi <
>>> > >>> luca.bocca...@gmail.com> wrote:
>>> > >>> > >> > Sorry, I meant if we go with (1), not (2), we might bump
>>> the size
>>> > >>> as
>>> > >>> > >> > well, since we are already doing another ABI-breaking
>>> change.
>>> > >>> > >> >
>>> > >>> > >> > I agree on the solution as well.
>>> > >>> > >> >
>>> > >>> > >> > On Mon, 2016-08-29 at 17:12 +0200, Pieter Hintjens wrote:
>>> > >>> > >> >> I'm confused between the (1) and (2) choices, and can't
>>
Re: [zeromq-dev] ZeroMQ 4.2 release, planning
Sorry, yes, lets do it :)
On Oct 31, 2016 11:44 PM, "Luca Boccassi" wrote:
> Ping :-)
>
> On Oct 28, 2016 18:48, "Luca Boccassi" wrote:
>
>> I have sent a solution for the alignment problem that solves the sigbus
>> problem without breaking ABI compat (plus follow-up for VC++ - sorry
>> Windows guys https://github.com/zeromq/libzmq/pull/2179 ).
>>
>> I tested the alignment and sigbus problem on x86_64 by enabling
>> alignment check with:
>>
>> __asm__("pushf\norl $0x4,(%rsp)\npopf");
>>
>> All was fine.
>>
>> I ran tests built from the zeromq4-1 repository against a shared lib
>> from the head of libzmq repo, and they all run fine minus the
>> ZMQ_REQ_CORRELATE one but that option was borken anyway.
>>
>> This allows us to do a release now, and then when we are ready we can do
>> the ABI breakage, without blocking 4.2. Which is nice since it means it
>> might make it for Debian 9!
>>
>> So, Doron et al, shall we do the bump this weekend?
>>
>> On Thu, 2016-10-20 at 17:12 -0500, Thomas Rodgers wrote:
>> > I will have some time most likely the week of Nov6 (off for a week of
>> C++
>> > Committee 'fun') to test different message size alternatives. I'll
>> follow
>> > up with my results here for consideration the next time we are inclined
>> to
>> > break the ABI compatibility :)
>> >
>> > On Sunday, October 16, 2016, Brian Knox wrote:
>> >
>> > > A new stable version would definitely help me in my quest to get
>> ZeroMQ
>> > > support enabled by default in rsyslog in distros.
>> > >
>> > > On Sun, Oct 16, 2016 at 2:40 PM Doron Somech
>> wrote:
>> > >
>> > >> I say lets bump.
>> > >>
>> > >> On Oct 15, 2016 20:32, "Luca Boccassi"
>> wrote:
>> > >>
>> > >>> As Thomas said, false sharing would be a real issue with 96.
>> > >>>
>> > >>> So given a release is long due, at this point I'd say to drop this
>> for
>> > >>> the moment.
>> > >>>
>> > >>> What do we do for the change to union for zmq_msg_t? Bump ABI
>> version or
>> > >>> not?
>> > >>>
>> > >>> On Thu, 2016-10-06 at 09:53 +0300, Doron Somech wrote:
>> > >>> > No new socket type, I worked at the time on binary message type,
>> might
>> > >>> > complete it sometime, but it is not urgent.
>> > >>> >
>> > >>> > If there is a lot of performance penalty we can give it up, I will
>> > >>> > find another solution for the Radio-Dish.
>> > >>> >
>> > >>> > What about 96 bytes? same penalty?
>> > >>> >
>> > >>> > Regarding the binding, I'm not sure.
>> > >>> >
>> > >>> > On Sat, Oct 1, 2016 at 9:14 PM, Luca Boccassi <
>> luca.bocca...@gmail.com>
>> > >>> wrote:
>> > >>> > > On Tue, 2016-09-27 at 09:41 +0300, Doron Somech wrote:
>> > >>> > >> Sorry for the late response, increasing the msg_t structure
>> will be
>> > >>> > >> great, however this will require changing a lot of binding.
>> > >>> > >
>> > >>> > > I think I remember we need it for the new socket types, is that
>> > >>> correct?
>> > >>> > >
>> > >>> > > There is a large performance penalty (intuitively due to not
>> fitting
>> > >>> > > into a single cache line anymore, but haven't ran
>> perf/cachegrind),
>> > >>> and
>> > >>> > > the throughput with vsm type messages goes down by 4% (min) and
>> 20%
>> > >>> > > (max) for TCP, and 36% (min) 38 (max) for inproc, which is
>> quite a
>> > >>> lot,
>> > >>> > > so we need to be sure it's worth it.
>> > >>> > >
>> > >>> > > Regarding the bindings, after a quick search on the Github org,
>> I
>> > >>> could
>> > >>> > > only see:
>> > >>> > >
>> > >>> > > https://github.com/zeromq/lzmq/blob/master/src/lua/lzmq/
>> > >>> ffi/api.lua#L144
>> > >>> > > https://github.com/zeromq/clrzmq4/blob/master/lib/zmq.cs#L28
>> > >>> > > https://github.com/zeromq/pyczmq/blob/master/pyczmq/zmq.py#L177
>> > >>> > >
>> > >>> > > Other bindings just import zmq.h. Did I miss any?
>> > >>> > >
>> > >>> > >> Sorry for disappearing, baby and full time job is a lot :-),
>> > >>> hopefully
>> > >>> > >> I'm back...
>> > >>> > >
>> > >>> > > No worries, perfectly understandable :-)
>> > >>> > >
>> > >>> > >> On Mon, Aug 29, 2016 at 6:46 PM, Luca Boccassi <
>> > >>> luca.bocca...@gmail.com> wrote:
>> > >>> > >> > Sorry, I meant if we go with (1), not (2), we might bump the
>> size
>> > >>> as
>> > >>> > >> > well, since we are already doing another ABI-breaking change.
>> > >>> > >> >
>> > >>> > >> > I agree on the solution as well.
>> > >>> > >> >
>> > >>> > >> > On Mon, 2016-08-29 at 17:12 +0200, Pieter Hintjens wrote:
>> > >>> > >> >> I'm confused between the (1) and (2) choices, and can't see
>> where
>> > >>> > >> >> bumping the message size fits.
>> > >>> > >> >>
>> > >>> > >> >> Nonetheless, I think bumping the size, fixing the alignment
>> > >>> issues,
>> > >>> > >> >> and bumping the ABI version is the best solution here.
>> > >>> > >> >>
>> > >>> > >> >> On Fri, Aug 26, 2016 at 12:33 PM, Luca Boccassi <
>> > >>> luca.bocca...@gmail.com> wrote:
>> > >>> > >> >> > I've given some more thoughts and testing to the alignment
>> > >>> issue. I can
>> > >>> > >> >> > r
Re: [zeromq-dev] ZeroMQ 4.2 release, planning
Ping :-)
On Oct 28, 2016 18:48, "Luca Boccassi" wrote:
> I have sent a solution for the alignment problem that solves the sigbus
> problem without breaking ABI compat (plus follow-up for VC++ - sorry
> Windows guys https://github.com/zeromq/libzmq/pull/2179 ).
>
> I tested the alignment and sigbus problem on x86_64 by enabling
> alignment check with:
>
> __asm__("pushf\norl $0x4,(%rsp)\npopf");
>
> All was fine.
>
> I ran tests built from the zeromq4-1 repository against a shared lib
> from the head of libzmq repo, and they all run fine minus the
> ZMQ_REQ_CORRELATE one but that option was borken anyway.
>
> This allows us to do a release now, and then when we are ready we can do
> the ABI breakage, without blocking 4.2. Which is nice since it means it
> might make it for Debian 9!
>
> So, Doron et al, shall we do the bump this weekend?
>
> On Thu, 2016-10-20 at 17:12 -0500, Thomas Rodgers wrote:
> > I will have some time most likely the week of Nov6 (off for a week of C++
> > Committee 'fun') to test different message size alternatives. I'll follow
> > up with my results here for consideration the next time we are inclined
> to
> > break the ABI compatibility :)
> >
> > On Sunday, October 16, 2016, Brian Knox wrote:
> >
> > > A new stable version would definitely help me in my quest to get ZeroMQ
> > > support enabled by default in rsyslog in distros.
> > >
> > > On Sun, Oct 16, 2016 at 2:40 PM Doron Somech
> wrote:
> > >
> > >> I say lets bump.
> > >>
> > >> On Oct 15, 2016 20:32, "Luca Boccassi"
> wrote:
> > >>
> > >>> As Thomas said, false sharing would be a real issue with 96.
> > >>>
> > >>> So given a release is long due, at this point I'd say to drop this
> for
> > >>> the moment.
> > >>>
> > >>> What do we do for the change to union for zmq_msg_t? Bump ABI
> version or
> > >>> not?
> > >>>
> > >>> On Thu, 2016-10-06 at 09:53 +0300, Doron Somech wrote:
> > >>> > No new socket type, I worked at the time on binary message type,
> might
> > >>> > complete it sometime, but it is not urgent.
> > >>> >
> > >>> > If there is a lot of performance penalty we can give it up, I will
> > >>> > find another solution for the Radio-Dish.
> > >>> >
> > >>> > What about 96 bytes? same penalty?
> > >>> >
> > >>> > Regarding the binding, I'm not sure.
> > >>> >
> > >>> > On Sat, Oct 1, 2016 at 9:14 PM, Luca Boccassi <
> luca.bocca...@gmail.com>
> > >>> wrote:
> > >>> > > On Tue, 2016-09-27 at 09:41 +0300, Doron Somech wrote:
> > >>> > >> Sorry for the late response, increasing the msg_t structure
> will be
> > >>> > >> great, however this will require changing a lot of binding.
> > >>> > >
> > >>> > > I think I remember we need it for the new socket types, is that
> > >>> correct?
> > >>> > >
> > >>> > > There is a large performance penalty (intuitively due to not
> fitting
> > >>> > > into a single cache line anymore, but haven't ran
> perf/cachegrind),
> > >>> and
> > >>> > > the throughput with vsm type messages goes down by 4% (min) and
> 20%
> > >>> > > (max) for TCP, and 36% (min) 38 (max) for inproc, which is quite
> a
> > >>> lot,
> > >>> > > so we need to be sure it's worth it.
> > >>> > >
> > >>> > > Regarding the bindings, after a quick search on the Github org, I
> > >>> could
> > >>> > > only see:
> > >>> > >
> > >>> > > https://github.com/zeromq/lzmq/blob/master/src/lua/lzmq/
> > >>> ffi/api.lua#L144
> > >>> > > https://github.com/zeromq/clrzmq4/blob/master/lib/zmq.cs#L28
> > >>> > > https://github.com/zeromq/pyczmq/blob/master/pyczmq/zmq.py#L177
> > >>> > >
> > >>> > > Other bindings just import zmq.h. Did I miss any?
> > >>> > >
> > >>> > >> Sorry for disappearing, baby and full time job is a lot :-),
> > >>> hopefully
> > >>> > >> I'm back...
> > >>> > >
> > >>> > > No worries, perfectly understandable :-)
> > >>> > >
> > >>> > >> On Mon, Aug 29, 2016 at 6:46 PM, Luca Boccassi <
> > >>> luca.bocca...@gmail.com> wrote:
> > >>> > >> > Sorry, I meant if we go with (1), not (2), we might bump the
> size
> > >>> as
> > >>> > >> > well, since we are already doing another ABI-breaking change.
> > >>> > >> >
> > >>> > >> > I agree on the solution as well.
> > >>> > >> >
> > >>> > >> > On Mon, 2016-08-29 at 17:12 +0200, Pieter Hintjens wrote:
> > >>> > >> >> I'm confused between the (1) and (2) choices, and can't see
> where
> > >>> > >> >> bumping the message size fits.
> > >>> > >> >>
> > >>> > >> >> Nonetheless, I think bumping the size, fixing the alignment
> > >>> issues,
> > >>> > >> >> and bumping the ABI version is the best solution here.
> > >>> > >> >>
> > >>> > >> >> On Fri, Aug 26, 2016 at 12:33 PM, Luca Boccassi <
> > >>> luca.bocca...@gmail.com> wrote:
> > >>> > >> >> > I've given some more thoughts and testing to the alignment
> > >>> issue. I can
> > >>> > >> >> > reproduce the problem by enabling alignment checks on x86
> too.
> > >>> > >> >> >
> > >>> > >> >> > But most importantly, I think we cannot get away from
> bumping
> > >>> the ABI
> > >>> > >> >> > with this fix, however w
