Re: [zeromq-dev] ZeroMQ 4.2 release, planning

2016-11-01 Thread Brian Knox 
No objections here - been using czmq off various commits off head for over
a year anyway ;)

On Tue, Nov 1, 2016 at 5:33 PM Luca Boccassi 
wrote:

> Status update:
>
> libzmq 4.1.6, libzmq 4.2.0-rc1 and czmq 4.0.0-rc1 are out on Github:
>
> https://github.com/zeromq/zeromq4-1/releases/tag/v4.1.6
> https://github.com/zeromq/libzmq/releases/tag/v4.2.0-rc1
> https://github.com/zeromq/czmq/releases/tag/v4.0.0-rc1
>
> I'll send an email to the announce list shortly. As I wrote earlier
> I'll work to have proper release notes for the stable releases.
>
> Unless there are any objections, I'm aiming to push libzmq 4.2.0
> stable tomorrow by the end of the day, and czmq the day after.
>
> It's an aggressive schedule, but I would _really_ like to get CZMQ
> 4.0.0 in Debian and the transition freeze date is Saturday (ABI/API is
> borken so there needs to be a transition), and for that I need libzmq
> up before it.
>
> Any objections?
>
> I've also noticed that not all the libzmq socket options are available
> in CZMQ, so this gives me some time to fix that.
>
>
> On 1 November 2016 at 14:48, Doron Somech  wrote:
> > Great news!
> >
> > On Tue, Nov 1, 2016 at 4:07 PM, Luca Boccassi 
> > wrote:
> >>
> >> Status update:
> >>
> >> - v2 APIs are gone from CZMQ:
> >>   https://github.com/zeromq/czmq/pull/1531
> >>   https://github.com/zeromq/czmq/pull/1532
> >> - PR is out to bump the libtool version and changelog for libzmq:
> >>   https://github.com/zeromq/libzmq/pull/2184
> >> - PR is out to backport the zmq_msg_t fix to 4.1:
> >>   https://github.com/zeromq/zeromq4-1/pull/155
> >>
> >> Once it's all merged I will tag 4.2.0~rc1 first, then release 4.1.6 from
> >> zeromq4-1 since quite a few fixes have accumulated. Then I'll send PRs
> >> to prepare for CZMQ 4.0.0~rc1.
> >>
> >> After the RCs are out, I'll work on the changelogs/NEWS files (help is
> >> appreciated!) as they have fallen dramatically behind.
> >>
> >> I'll also prepare more formal release notes for the stable rels, the RCs
> >> will have just a quick note since they are RCs.
> >>
> >> On Mon, 2016-10-31 at 23:47 +, Luca Boccassi wrote:
> >> > Cool!
> >> >
> >> > I can take care of it if you like. Tentative plan:
> >> >
> >> > Tomorrow push an RC1 for libzmq, then the pr to CZMQ to retire v2
> APIs,
> >> > then the RC1 for CZMQ.
> >> >
> >> > If it's all good then a couple days later the finals. I would really
> >> > like
> >> > to make it for the debian 9 transition freeze which is Saturday.
> >> >
> >> > On Oct 31, 2016 22:23, "Doron Somech"  wrote:
> >> >
> >> > > Sorry, yes, lets do it :)
> >> > >
> >> > > On Oct 31, 2016 11:44 PM, "Luca Boccassi" 
> >> > > wrote:
> >> > >
> >> > >> Ping :-)
> >> > >>
> >> > >> On Oct 28, 2016 18:48, "Luca Boccassi" 
> >> > >> wrote:
> >> > >>
> >> > >>> I have sent a solution for the alignment problem that solves the
> >> > >>> sigbus
> >> > >>> problem without breaking ABI compat (plus follow-up for VC++ -
> sorry
> >> > >>> Windows guys https://github.com/zeromq/libzmq/pull/2179 ).
> >> > >>>
> >> > >>> I tested the alignment and sigbus problem on x86_64 by enabling
> >> > >>> alignment check with:
> >> > >>>
> >> > >>> __asm__("pushf\norl $0x4,(%rsp)\npopf");
> >> > >>>
> >> > >>> All was fine.
> >> > >>>
> >> > >>> I ran tests built from the zeromq4-1 repository against a shared
> lib
> >> > >>> from the head of libzmq repo, and they all run fine minus the
> >> > >>> ZMQ_REQ_CORRELATE one but that option was borken anyway.
> >> > >>>
> >> > >>> This allows us to do a release now, and then when we are ready we
> >> > >>> can do
> >> > >>> the ABI breakage, without blocking 4.2. Which is nice since it
> means
> >> > >>> it
> >> > >>> might make it for Debian 9!
> >> > >>>
> >> > >>> So, Doron et al, shall we do the bump this weekend?
> >> > >>>
> >> > >>> On Thu, 2016-10-20 at 17:12 -0500, Thomas Rodgers wrote:
> >> > >>> > I will have some time most likely the week of Nov6 (off for a
> week
> >> > >>> > of
> >> > >>> C++
> >> > >>> > Committee 'fun') to test different message size alternatives.
> I'll
> >> > >>> follow
> >> > >>> > up with my results here for consideration the next time we are
> >> > >>> inclined to
> >> > >>> > break the ABI compatibility :)
> >> > >>> >
> >> > >>> > On Sunday, October 16, 2016, Brian Knox  >
> >> > >>> wrote:
> >> > >>> >
> >> > >>> > > A new stable version would definitely help me in my quest to
> get
> >> > >>> ZeroMQ
> >> > >>> > > support enabled by default in rsyslog in distros.
> >> > >>> > >
> >> > >>> > > On Sun, Oct 16, 2016 at 2:40 PM Doron Somech
> >> > >>> > > 
> >> > >>> wrote:
> >> > >>> > >
> >> > >>> > >> I say lets bump.
> >> > >>> > >>
> >> > >>> > >> On Oct 15, 2016 20:32, "Luca Boccassi"
> >> > >>> > >> 
> >> > >>> wrote:
> >> > >>> > >>
> >> >

Re: [zeromq-dev] ZeroMQ 4.2 release, planning

2016-11-01 Thread Luca Boccassi 
Status update:

libzmq 4.1.6, libzmq 4.2.0-rc1 and czmq 4.0.0-rc1 are out on Github:

https://github.com/zeromq/zeromq4-1/releases/tag/v4.1.6
https://github.com/zeromq/libzmq/releases/tag/v4.2.0-rc1
https://github.com/zeromq/czmq/releases/tag/v4.0.0-rc1

I'll send an email to the announce list shortly. As I wrote earlier
I'll work to have proper release notes for the stable releases.

Unless there are any objections, I'm aiming to push libzmq 4.2.0
stable tomorrow by the end of the day, and czmq the day after.

It's an aggressive schedule, but I would _really_ like to get CZMQ
4.0.0 in Debian and the transition freeze date is Saturday (ABI/API is
borken so there needs to be a transition), and for that I need libzmq
up before it.

Any objections?

I've also noticed that not all the libzmq socket options are available
in CZMQ, so this gives me some time to fix that.


On 1 November 2016 at 14:48, Doron Somech  wrote:
> Great news!
>
> On Tue, Nov 1, 2016 at 4:07 PM, Luca Boccassi 
> wrote:
>>
>> Status update:
>>
>> - v2 APIs are gone from CZMQ:
>>   https://github.com/zeromq/czmq/pull/1531
>>   https://github.com/zeromq/czmq/pull/1532
>> - PR is out to bump the libtool version and changelog for libzmq:
>>   https://github.com/zeromq/libzmq/pull/2184
>> - PR is out to backport the zmq_msg_t fix to 4.1:
>>   https://github.com/zeromq/zeromq4-1/pull/155
>>
>> Once it's all merged I will tag 4.2.0~rc1 first, then release 4.1.6 from
>> zeromq4-1 since quite a few fixes have accumulated. Then I'll send PRs
>> to prepare for CZMQ 4.0.0~rc1.
>>
>> After the RCs are out, I'll work on the changelogs/NEWS files (help is
>> appreciated!) as they have fallen dramatically behind.
>>
>> I'll also prepare more formal release notes for the stable rels, the RCs
>> will have just a quick note since they are RCs.
>>
>> On Mon, 2016-10-31 at 23:47 +, Luca Boccassi wrote:
>> > Cool!
>> >
>> > I can take care of it if you like. Tentative plan:
>> >
>> > Tomorrow push an RC1 for libzmq, then the pr to CZMQ to retire v2 APIs,
>> > then the RC1 for CZMQ.
>> >
>> > If it's all good then a couple days later the finals. I would really
>> > like
>> > to make it for the debian 9 transition freeze which is Saturday.
>> >
>> > On Oct 31, 2016 22:23, "Doron Somech"  wrote:
>> >
>> > > Sorry, yes, lets do it :)
>> > >
>> > > On Oct 31, 2016 11:44 PM, "Luca Boccassi" 
>> > > wrote:
>> > >
>> > >> Ping :-)
>> > >>
>> > >> On Oct 28, 2016 18:48, "Luca Boccassi" 
>> > >> wrote:
>> > >>
>> > >>> I have sent a solution for the alignment problem that solves the
>> > >>> sigbus
>> > >>> problem without breaking ABI compat (plus follow-up for VC++ - sorry
>> > >>> Windows guys https://github.com/zeromq/libzmq/pull/2179 ).
>> > >>>
>> > >>> I tested the alignment and sigbus problem on x86_64 by enabling
>> > >>> alignment check with:
>> > >>>
>> > >>> __asm__("pushf\norl $0x4,(%rsp)\npopf");
>> > >>>
>> > >>> All was fine.
>> > >>>
>> > >>> I ran tests built from the zeromq4-1 repository against a shared lib
>> > >>> from the head of libzmq repo, and they all run fine minus the
>> > >>> ZMQ_REQ_CORRELATE one but that option was borken anyway.
>> > >>>
>> > >>> This allows us to do a release now, and then when we are ready we
>> > >>> can do
>> > >>> the ABI breakage, without blocking 4.2. Which is nice since it means
>> > >>> it
>> > >>> might make it for Debian 9!
>> > >>>
>> > >>> So, Doron et al, shall we do the bump this weekend?
>> > >>>
>> > >>> On Thu, 2016-10-20 at 17:12 -0500, Thomas Rodgers wrote:
>> > >>> > I will have some time most likely the week of Nov6 (off for a week
>> > >>> > of
>> > >>> C++
>> > >>> > Committee 'fun') to test different message size alternatives. I'll
>> > >>> follow
>> > >>> > up with my results here for consideration the next time we are
>> > >>> inclined to
>> > >>> > break the ABI compatibility :)
>> > >>> >
>> > >>> > On Sunday, October 16, 2016, Brian Knox 
>> > >>> wrote:
>> > >>> >
>> > >>> > > A new stable version would definitely help me in my quest to get
>> > >>> ZeroMQ
>> > >>> > > support enabled by default in rsyslog in distros.
>> > >>> > >
>> > >>> > > On Sun, Oct 16, 2016 at 2:40 PM Doron Somech
>> > >>> > > 
>> > >>> wrote:
>> > >>> > >
>> > >>> > >> I say lets bump.
>> > >>> > >>
>> > >>> > >> On Oct 15, 2016 20:32, "Luca Boccassi"
>> > >>> > >> 
>> > >>> wrote:
>> > >>> > >>
>> > >>> > >>> As Thomas said, false sharing would be a real issue with 96.
>> > >>> > >>>
>> > >>> > >>> So given a release is long due, at this point I'd say to drop
>> > >>> > >>> this
>> > >>> for
>> > >>> > >>> the moment.
>> > >>> > >>>
>> > >>> > >>> What do we do for the change to union for zmq_msg_t? Bump ABI
>> > >>> version or
>> > >>> > >>> not?
>> > >>> > >>>
>> > >>> > >>> On Thu, 2016-10-06 at 09:53 +0300, Doron Somech

Re: [zeromq-dev] ZeroMQ 4.2 release, planning

2016-11-01 Thread Doron Somech 
Great news!

On Tue, Nov 1, 2016 at 4:07 PM, Luca Boccassi 
wrote:

> Status update:
>
> - v2 APIs are gone from CZMQ:
>   https://github.com/zeromq/czmq/pull/1531
>   https://github.com/zeromq/czmq/pull/1532
> - PR is out to bump the libtool version and changelog for libzmq:
>   https://github.com/zeromq/libzmq/pull/2184
> - PR is out to backport the zmq_msg_t fix to 4.1:
>   https://github.com/zeromq/zeromq4-1/pull/155
>
> Once it's all merged I will tag 4.2.0~rc1 first, then release 4.1.6 from
> zeromq4-1 since quite a few fixes have accumulated. Then I'll send PRs
> to prepare for CZMQ 4.0.0~rc1.
>
> After the RCs are out, I'll work on the changelogs/NEWS files (help is
> appreciated!) as they have fallen dramatically behind.
>
> I'll also prepare more formal release notes for the stable rels, the RCs
> will have just a quick note since they are RCs.
>
> On Mon, 2016-10-31 at 23:47 +, Luca Boccassi wrote:
> > Cool!
> >
> > I can take care of it if you like. Tentative plan:
> >
> > Tomorrow push an RC1 for libzmq, then the pr to CZMQ to retire v2 APIs,
> > then the RC1 for CZMQ.
> >
> > If it's all good then a couple days later the finals. I would really like
> > to make it for the debian 9 transition freeze which is Saturday.
> >
> > On Oct 31, 2016 22:23, "Doron Somech"  wrote:
> >
> > > Sorry, yes, lets do it :)
> > >
> > > On Oct 31, 2016 11:44 PM, "Luca Boccassi" 
> wrote:
> > >
> > >> Ping :-)
> > >>
> > >> On Oct 28, 2016 18:48, "Luca Boccassi" 
> wrote:
> > >>
> > >>> I have sent a solution for the alignment problem that solves the
> sigbus
> > >>> problem without breaking ABI compat (plus follow-up for VC++ - sorry
> > >>> Windows guys https://github.com/zeromq/libzmq/pull/2179 ).
> > >>>
> > >>> I tested the alignment and sigbus problem on x86_64 by enabling
> > >>> alignment check with:
> > >>>
> > >>> __asm__("pushf\norl $0x4,(%rsp)\npopf");
> > >>>
> > >>> All was fine.
> > >>>
> > >>> I ran tests built from the zeromq4-1 repository against a shared lib
> > >>> from the head of libzmq repo, and they all run fine minus the
> > >>> ZMQ_REQ_CORRELATE one but that option was borken anyway.
> > >>>
> > >>> This allows us to do a release now, and then when we are ready we
> can do
> > >>> the ABI breakage, without blocking 4.2. Which is nice since it means
> it
> > >>> might make it for Debian 9!
> > >>>
> > >>> So, Doron et al, shall we do the bump this weekend?
> > >>>
> > >>> On Thu, 2016-10-20 at 17:12 -0500, Thomas Rodgers wrote:
> > >>> > I will have some time most likely the week of Nov6 (off for a week
> of
> > >>> C++
> > >>> > Committee 'fun') to test different message size alternatives. I'll
> > >>> follow
> > >>> > up with my results here for consideration the next time we are
> > >>> inclined to
> > >>> > break the ABI compatibility :)
> > >>> >
> > >>> > On Sunday, October 16, 2016, Brian Knox 
> > >>> wrote:
> > >>> >
> > >>> > > A new stable version would definitely help me in my quest to get
> > >>> ZeroMQ
> > >>> > > support enabled by default in rsyslog in distros.
> > >>> > >
> > >>> > > On Sun, Oct 16, 2016 at 2:40 PM Doron Somech  >
> > >>> wrote:
> > >>> > >
> > >>> > >> I say lets bump.
> > >>> > >>
> > >>> > >> On Oct 15, 2016 20:32, "Luca Boccassi"  >
> > >>> wrote:
> > >>> > >>
> > >>> > >>> As Thomas said, false sharing would be a real issue with 96.
> > >>> > >>>
> > >>> > >>> So given a release is long due, at this point I'd say to drop
> this
> > >>> for
> > >>> > >>> the moment.
> > >>> > >>>
> > >>> > >>> What do we do for the change to union for zmq_msg_t? Bump ABI
> > >>> version or
> > >>> > >>> not?
> > >>> > >>>
> > >>> > >>> On Thu, 2016-10-06 at 09:53 +0300, Doron Somech wrote:
> > >>> > >>> > No new socket type, I worked at the time on binary message
> type,
> > >>> might
> > >>> > >>> > complete it sometime, but it is not urgent.
> > >>> > >>> >
> > >>> > >>> > If there is a lot of performance penalty we can give it up, I
> > >>> will
> > >>> > >>> > find another solution for the Radio-Dish.
> > >>> > >>> >
> > >>> > >>> > What about 96 bytes? same penalty?
> > >>> > >>> >
> > >>> > >>> > Regarding the binding, I'm not sure.
> > >>> > >>> >
> > >>> > >>> > On Sat, Oct 1, 2016 at 9:14 PM, Luca Boccassi <
> > >>> luca.bocca...@gmail.com>
> > >>> > >>> wrote:
> > >>> > >>> > > On Tue, 2016-09-27 at 09:41 +0300, Doron Somech wrote:
> > >>> > >>> > >> Sorry for the late response, increasing the msg_t
> structure
> > >>> will be
> > >>> > >>> > >> great, however this will require changing a lot of
> binding.
> > >>> > >>> > >
> > >>> > >>> > > I think I remember we need it for the new socket types, is
> that
> > >>> > >>> correct?
> > >>> > >>> > >
> > >>> > >>> > > There is a large performance penalty (intuitively due to
> not
> > >>> fitting
> > >>> > >>> > > into a single cache line

Re: [zeromq-dev] ZeroMQ 4.2 release, planning

2016-11-01 Thread Luca Boccassi 
Status update:

- v2 APIs are gone from CZMQ:
  https://github.com/zeromq/czmq/pull/1531
  https://github.com/zeromq/czmq/pull/1532
- PR is out to bump the libtool version and changelog for libzmq:
  https://github.com/zeromq/libzmq/pull/2184
- PR is out to backport the zmq_msg_t fix to 4.1:
  https://github.com/zeromq/zeromq4-1/pull/155

Once it's all merged I will tag 4.2.0~rc1 first, then release 4.1.6 from
zeromq4-1 since quite a few fixes have accumulated. Then I'll send PRs
to prepare for CZMQ 4.0.0~rc1.

After the RCs are out, I'll work on the changelogs/NEWS files (help is
appreciated!) as they have fallen dramatically behind.

I'll also prepare more formal release notes for the stable rels, the RCs
will have just a quick note since they are RCs.

On Mon, 2016-10-31 at 23:47 +, Luca Boccassi wrote:
> Cool!
> 
> I can take care of it if you like. Tentative plan:
> 
> Tomorrow push an RC1 for libzmq, then the pr to CZMQ to retire v2 APIs,
> then the RC1 for CZMQ.
> 
> If it's all good then a couple days later the finals. I would really like
> to make it for the debian 9 transition freeze which is Saturday.
> 
> On Oct 31, 2016 22:23, "Doron Somech"  wrote:
> 
> > Sorry, yes, lets do it :)
> >
> > On Oct 31, 2016 11:44 PM, "Luca Boccassi"  wrote:
> >
> >> Ping :-)
> >>
> >> On Oct 28, 2016 18:48, "Luca Boccassi"  wrote:
> >>
> >>> I have sent a solution for the alignment problem that solves the sigbus
> >>> problem without breaking ABI compat (plus follow-up for VC++ - sorry
> >>> Windows guys https://github.com/zeromq/libzmq/pull/2179 ).
> >>>
> >>> I tested the alignment and sigbus problem on x86_64 by enabling
> >>> alignment check with:
> >>>
> >>> __asm__("pushf\norl $0x4,(%rsp)\npopf");
> >>>
> >>> All was fine.
> >>>
> >>> I ran tests built from the zeromq4-1 repository against a shared lib
> >>> from the head of libzmq repo, and they all run fine minus the
> >>> ZMQ_REQ_CORRELATE one but that option was borken anyway.
> >>>
> >>> This allows us to do a release now, and then when we are ready we can do
> >>> the ABI breakage, without blocking 4.2. Which is nice since it means it
> >>> might make it for Debian 9!
> >>>
> >>> So, Doron et al, shall we do the bump this weekend?
> >>>
> >>> On Thu, 2016-10-20 at 17:12 -0500, Thomas Rodgers wrote:
> >>> > I will have some time most likely the week of Nov6 (off for a week of
> >>> C++
> >>> > Committee 'fun') to test different message size alternatives. I'll
> >>> follow
> >>> > up with my results here for consideration the next time we are
> >>> inclined to
> >>> > break the ABI compatibility :)
> >>> >
> >>> > On Sunday, October 16, 2016, Brian Knox 
> >>> wrote:
> >>> >
> >>> > > A new stable version would definitely help me in my quest to get
> >>> ZeroMQ
> >>> > > support enabled by default in rsyslog in distros.
> >>> > >
> >>> > > On Sun, Oct 16, 2016 at 2:40 PM Doron Somech 
> >>> wrote:
> >>> > >
> >>> > >> I say lets bump.
> >>> > >>
> >>> > >> On Oct 15, 2016 20:32, "Luca Boccassi" 
> >>> wrote:
> >>> > >>
> >>> > >>> As Thomas said, false sharing would be a real issue with 96.
> >>> > >>>
> >>> > >>> So given a release is long due, at this point I'd say to drop this
> >>> for
> >>> > >>> the moment.
> >>> > >>>
> >>> > >>> What do we do for the change to union for zmq_msg_t? Bump ABI
> >>> version or
> >>> > >>> not?
> >>> > >>>
> >>> > >>> On Thu, 2016-10-06 at 09:53 +0300, Doron Somech wrote:
> >>> > >>> > No new socket type, I worked at the time on binary message type,
> >>> might
> >>> > >>> > complete it sometime, but it is not urgent.
> >>> > >>> >
> >>> > >>> > If there is a lot of performance penalty we can give it up, I
> >>> will
> >>> > >>> > find another solution for the Radio-Dish.
> >>> > >>> >
> >>> > >>> > What about 96 bytes? same penalty?
> >>> > >>> >
> >>> > >>> > Regarding the binding, I'm not sure.
> >>> > >>> >
> >>> > >>> > On Sat, Oct 1, 2016 at 9:14 PM, Luca Boccassi <
> >>> luca.bocca...@gmail.com>
> >>> > >>> wrote:
> >>> > >>> > > On Tue, 2016-09-27 at 09:41 +0300, Doron Somech wrote:
> >>> > >>> > >> Sorry for the late response, increasing the msg_t structure
> >>> will be
> >>> > >>> > >> great, however this will require changing a lot of binding.
> >>> > >>> > >
> >>> > >>> > > I think I remember we need it for the new socket types, is that
> >>> > >>> correct?
> >>> > >>> > >
> >>> > >>> > > There is a large performance penalty (intuitively due to not
> >>> fitting
> >>> > >>> > > into a single cache line anymore, but haven't ran
> >>> perf/cachegrind),
> >>> > >>> and
> >>> > >>> > > the throughput with vsm type messages goes down by 4% (min)
> >>> and 20%
> >>> > >>> > > (max) for TCP, and 36% (min) 38 (max) for inproc, which is
> >>> quite a
> >>> > >>> lot,
> >>> > >>> > > so we need to be sure it's worth it.
> >>> > >>> > >
> >>> > >>> > > Regarding the