Sorry, yes, lets do it :) On Oct 31, 2016 11:44 PM, "Luca Boccassi" <luca.bocca...@gmail.com> wrote:
> Ping :-) > > On Oct 28, 2016 18:48, "Luca Boccassi" <luca.bocca...@gmail.com> wrote: > >> I have sent a solution for the alignment problem that solves the sigbus >> problem without breaking ABI compat (plus follow-up for VC++ - sorry >> Windows guys https://github.com/zeromq/libzmq/pull/2179 ). >> >> I tested the alignment and sigbus problem on x86_64 by enabling >> alignment check with: >> >> __asm__("pushf\norl $0x40000,(%rsp)\npopf"); >> >> All was fine. >> >> I ran tests built from the zeromq4-1 repository against a shared lib >> from the head of libzmq repo, and they all run fine minus the >> ZMQ_REQ_CORRELATE one but that option was borken anyway. >> >> This allows us to do a release now, and then when we are ready we can do >> the ABI breakage, without blocking 4.2. Which is nice since it means it >> might make it for Debian 9! >> >> So, Doron et al, shall we do the bump this weekend? >> >> On Thu, 2016-10-20 at 17:12 -0500, Thomas Rodgers wrote: >> > I will have some time most likely the week of Nov6 (off for a week of >> C++ >> > Committee 'fun') to test different message size alternatives. I'll >> follow >> > up with my results here for consideration the next time we are inclined >> to >> > break the ABI compatibility :) >> > >> > On Sunday, October 16, 2016, Brian Knox <bk...@digitalocean.com> wrote: >> > >> > > A new stable version would definitely help me in my quest to get >> ZeroMQ >> > > support enabled by default in rsyslog in distros. >> > > >> > > On Sun, Oct 16, 2016 at 2:40 PM Doron Somech <somdo...@gmail.com> >> wrote: >> > > >> > >> I say lets bump. >> > >> >> > >> On Oct 15, 2016 20:32, "Luca Boccassi" <luca.bocca...@gmail.com> >> wrote: >> > >> >> > >>> As Thomas said, false sharing would be a real issue with 96. >> > >>> >> > >>> So given a release is long due, at this point I'd say to drop this >> for >> > >>> the moment. >> > >>> >> > >>> What do we do for the change to union for zmq_msg_t? Bump ABI >> version or >> > >>> not? >> > >>> >> > >>> On Thu, 2016-10-06 at 09:53 +0300, Doron Somech wrote: >> > >>> > No new socket type, I worked at the time on binary message type, >> might >> > >>> > complete it sometime, but it is not urgent. >> > >>> > >> > >>> > If there is a lot of performance penalty we can give it up, I will >> > >>> > find another solution for the Radio-Dish. >> > >>> > >> > >>> > What about 96 bytes? same penalty? >> > >>> > >> > >>> > Regarding the binding, I'm not sure. >> > >>> > >> > >>> > On Sat, Oct 1, 2016 at 9:14 PM, Luca Boccassi < >> luca.bocca...@gmail.com> >> > >>> wrote: >> > >>> > > On Tue, 2016-09-27 at 09:41 +0300, Doron Somech wrote: >> > >>> > >> Sorry for the late response, increasing the msg_t structure >> will be >> > >>> > >> great, however this will require changing a lot of binding. >> > >>> > > >> > >>> > > I think I remember we need it for the new socket types, is that >> > >>> correct? >> > >>> > > >> > >>> > > There is a large performance penalty (intuitively due to not >> fitting >> > >>> > > into a single cache line anymore, but haven't ran >> perf/cachegrind), >> > >>> and >> > >>> > > the throughput with vsm type messages goes down by 4% (min) and >> 20% >> > >>> > > (max) for TCP, and 36% (min) 38 (max) for inproc, which is >> quite a >> > >>> lot, >> > >>> > > so we need to be sure it's worth it. >> > >>> > > >> > >>> > > Regarding the bindings, after a quick search on the Github org, >> I >> > >>> could >> > >>> > > only see: >> > >>> > > >> > >>> > > https://github.com/zeromq/lzmq/blob/master/src/lua/lzmq/ >> > >>> ffi/api.lua#L144 >> > >>> > > https://github.com/zeromq/clrzmq4/blob/master/lib/zmq.cs#L28 >> > >>> > > https://github.com/zeromq/pyczmq/blob/master/pyczmq/zmq.py#L177 >> > >>> > > >> > >>> > > Other bindings just import zmq.h. Did I miss any? >> > >>> > > >> > >>> > >> Sorry for disappearing, baby and full time job is a lot :-), >> > >>> hopefully >> > >>> > >> I'm back... >> > >>> > > >> > >>> > > No worries, perfectly understandable :-) >> > >>> > > >> > >>> > >> On Mon, Aug 29, 2016 at 6:46 PM, Luca Boccassi < >> > >>> luca.bocca...@gmail.com> wrote: >> > >>> > >> > Sorry, I meant if we go with (1), not (2), we might bump the >> size >> > >>> as >> > >>> > >> > well, since we are already doing another ABI-breaking change. >> > >>> > >> > >> > >>> > >> > I agree on the solution as well. >> > >>> > >> > >> > >>> > >> > On Mon, 2016-08-29 at 17:12 +0200, Pieter Hintjens wrote: >> > >>> > >> >> I'm confused between the (1) and (2) choices, and can't see >> where >> > >>> > >> >> bumping the message size fits. >> > >>> > >> >> >> > >>> > >> >> Nonetheless, I think bumping the size, fixing the alignment >> > >>> issues, >> > >>> > >> >> and bumping the ABI version is the best solution here. >> > >>> > >> >> >> > >>> > >> >> On Fri, Aug 26, 2016 at 12:33 PM, Luca Boccassi < >> > >>> luca.bocca...@gmail.com> wrote: >> > >>> > >> >> > I've given some more thoughts and testing to the alignment >> > >>> issue. I can >> > >>> > >> >> > reproduce the problem by enabling alignment checks on x86 >> too. >> > >>> > >> >> > >> > >>> > >> >> > But most importantly, I think we cannot get away from >> bumping >> > >>> the ABI >> > >>> > >> >> > with this fix, however we rearrange it, simply because >> > >>> applications need >> > >>> > >> >> > to be rebuilt against the new header to be fixed. A simple >> > >>> rebuild of >> > >>> > >> >> > the libzmq.so is not enough. And the way to do this is to >> bump >> > >>> the ABI >> > >>> > >> >> > so that distros can schedule transitions and rebuilds and >> so >> > >>> on. >> > >>> > >> >> > >> > >>> > >> >> > So the choice list is now restricted to: >> > >>> > >> >> > >> > >>> > >> >> > 1) Bump ABI >> > >>> > >> >> > 2) Revert the fix and leave everything broken on sparc64 >> and >> > >>> some >> > >>> > >> >> > aarch64 (rpi3 seems not to be affected, must depend on >> the SoC >> > >>> flavour) >> > >>> > >> >> > >> > >>> > >> >> > If we go with 2, we might as well get 2 birds with one >> stone >> > >>> and bump >> > >>> > >> >> > the zmq_msg_t size to 128 as we have talked about in the >> past. >> > >>> > >> >> > >> > >>> > >> >> > Doron, this would help with the new UDP based socket types >> > >>> right? >> > >>> > >> >> > >> > >>> > >> >> > Pros of bumping msg size: >> > >>> > >> >> > >> > >>> > >> >> > - we can get rid of the malloc() in the lmsg type case as >> all >> > >>> the data >> > >>> > >> >> > will fit >> > >>> > >> >> > >> > >>> > >> >> > Cons: >> > >>> > >> >> > >> > >>> > >> >> > - for the vsm/cmsg type cases (for most architectures >> anyway) >> > >>> it won't >> > >>> > >> >> > fit anymore into a single cacheline >> > >>> > >> >> > >> > >>> > >> >> > Given all this, I'd say we should go for it. >> > >>> > >> >> > >> > >>> > >> >> > Opinions? >> > >>> > >> >> > >> > >>> > >> >> > On Sat, 2016-08-13 at 16:59 +0100, Luca Boccassi wrote: >> > >>> > >> >> >> Hello, >> > >>> > >> >> >> >> > >>> > >> >> >> Trying to give some thoughts again on the libzmq 4.2 >> release. >> > >>> It's >> > >>> > >> >> >> really long overdue! >> > >>> > >> >> >> >> > >>> > >> >> >> The main issue from my point of view is this change: >> > >>> > >> >> >> >> > >>> > >> >> >> https://github.com/zeromq/libzmq/commit/ >> > >>> d9fb1d36ff2008966af538f722a1f4ab158dbf64 >> > >>> > >> >> >> >> > >>> > >> >> >> -typedef struct zmq_msg_t {unsigned char _ [64];} >> zmq_msg_t; >> > >>> > >> >> >> +/* union here ensures correct alignment on >> architectures >> > >>> that require >> > >>> > >> >> >> it, e.g. >> > >>> > >> >> >> + * SPARC >> > >>> > >> >> >> + */ >> > >>> > >> >> >> +typedef union zmq_msg_t {unsigned char _ [64]; void >> *p; } >> > >>> zmq_msg_t; >> > >>> > >> >> >> >> > >>> > >> >> >> >> > >>> > >> >> >> This is flagged by the common ABI checkers tools as an >> ABI >> > >>> breakage >> > >>> > >> >> >> (see: http://abi-laboratory.pro/tracker/timeline/zeromq/ >> ). >> > >>> And it makes >> > >>> > >> >> >> sense from this point of view: if some applications on >> some >> > >>> > >> >> >> architectures are broken due to wrong alignment, they >> would >> > >>> need to be >> > >>> > >> >> >> rebuilt, and the way to ensure that is to bump the ABI >> > >>> "current" digit >> > >>> > >> >> >> to make sure maintainers do a rebuild. >> > >>> > >> >> >> >> > >>> > >> >> >> On the other hand, signaling an ABI breakage is a pain, >> and a >> > >>> cause of >> > >>> > >> >> >> major churn for packagers and maintainers. It means for >> > >>> example a new >> > >>> > >> >> >> package has to be created (eg: libzmq5 -> libzmq6), and a >> > >>> transition has >> > >>> > >> >> >> to be started and all reverse dependencies need to be >> > >>> rebuilt. And if >> > >>> > >> >> >> this is pointless for all save a few corner cases (eg >> SPARC64 >> > >>> as for >> > >>> > >> >> >> above) it's all quite frustrating. >> > >>> > >> >> >> >> > >>> > >> >> >> So we have a choice to make before we release 4.2, four >> > >>> possibilities as >> > >>> > >> >> >> far as I can see: >> > >>> > >> >> >> >> > >>> > >> >> >> 1) Ignore the ABI checkers and get yelled at by >> maintainers >> > >>> and >> > >>> > >> >> >> packagers. Also the SPARC64 users will most likely NOT >> get >> > >>> their bug >> > >>> > >> >> >> fixed >> > >>> > >> >> >> 2) Bump ABI revision to 6 and get yelled at by >> maintainers >> > >>> and packagers >> > >>> > >> >> >> 3) Revert the above change and postpone it to when we >> have a >> > >>> more >> > >>> > >> >> >> generally useful reason to break ABI (bump zmq_msg_t >> from 64 >> > >>> to 128 >> > >>> > >> >> >> bytes for example, Doron?) >> > >>> > >> >> >> 4) Try to be clever and revert the above change and use >> > >>> something like >> > >>> > >> >> >> #pragma pack(8). This will fool the ABI checkers (I tried >> > >>> it), and given >> > >>> > >> >> >> that typedef is only used externally to allocate the >> right >> > >>> size it >> > >>> > >> >> >> shouldn't actually affect anything, apart from the users >> of >> > >>> SPARC64 >> > >>> > >> >> >> which should get the bugfix with this too. This is very >> > >>> sneaky :-) >> > >>> > >> >> >> >> > >>> > >> >> >> CC'ing Lazslo, the Debian maintainer, given what we >> choose to >> > >>> do might >> > >>> > >> >> >> result in a lot of work for him :-) >> > >>> > >> >> >> >> > >>> > >> >> >> Opinions? >> > >>> > >> >> >> >> > >>> > >> >> >> Kind regards, >> > >>> > >> >> >> Luca Boccassi >> > >>> > >> >> >> >> > >>> > >> >> >> On Tue, 2016-05-03 at 10:39 +0200, Pieter Hintjens wrote: >> > >>> > >> >> >> > Hi all, >> > >>> > >> >> >> > >> > >>> > >> >> >> > I'm just throwing some ideas on the table. We have a >> good >> > >>> package of >> > >>> > >> >> >> > work on master and it's probably time to make a 4.2 >> release. >> > >>> > >> >> >> > >> > >>> > >> >> >> > Luca has already back-ported the enable/disable draft >> > >>> design from >> > >>> > >> >> >> > zproject (CZMQ et al). Yay! So we can now release >> stable >> > >>> master >> > >>> > >> >> >> > safely, while continuing to refine and extend the >> draft API >> > >>> sections. >> > >>> > >> >> >> > >> > >>> > >> >> >> > I propose: >> > >>> > >> >> >> > >> > >>> > >> >> >> > - to end with the stable fork policy; this was needed >> years >> > >>> ago when >> > >>> > >> >> >> > we had massively unstable masters. It's no longer a >> problem. >> > >>> > >> >> >> > - to use the github release function for libzmq >> releases >> > >>> and deprecate >> > >>> > >> >> >> > the separate delivery of tarballs. >> > >>> > >> >> >> > - we aim to make a 4.2.0 rc asap, then fix any issues >> we >> > >>> get, with >> > >>> > >> >> >> > patch releases as usual. >> > >>> > >> >> >> > - we backport the release function to older maintained >> > >>> releases (4.1, >> > >>> > >> >> >> > 3.2) so that their tarballs are provided by github >> instead >> > >>> of >> > >>> > >> >> >> > downloads.zeromq.org. >> > >>> > >> >> >> > >> > >>> > >> >> >> > Problems: >> > >>> > >> >> >> > >> > >>> > >> >> >> > - this will break a few things that depend on >> > >>> downloads.zeromq.org. To >> > >>> > >> >> >> > be fixed as we go. >> > >>> > >> >> >> > - github tarballs are not identical to source tarballs, >> > >>> particularly >> > >>> > >> >> >> > they lack `configure`. I propose changing our autotools >> > >>> build >> > >>> > >> >> >> > instructions so they always start with `./autogen,sh` >> no >> > >>> matter where >> > >>> > >> >> >> > the sources come from. >> > >>> > >> >> >> > >> > >>> > >> >> >> > I think this will work and also let us gracefully >> > >>> deprecate/switch off >> > >>> > >> >> >> > the downloads box. >> > >>> > >> >> >> > >> > >>> > >> >> >> > -Pieter >> > >>> > >> >> >> > _______________________________________________ >> > >>> > >> >> >> > zeromq-dev mailing list >> > >>> > >> >> >> > zeromq-dev@lists.zeromq.org >> > >>> > >> >> >> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> > >>> > >> >> >> >> > >>> > >> >> >> >> > >>> > >> >> > >> > >>> > >> >> > >> > >>> > >> >> > >> > >>> > >> >> > _______________________________________________ >> > >>> > >> >> > zeromq-dev mailing list >> > >>> > >> >> > zeromq-dev@lists.zeromq.org >> > >>> > >> >> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> > >>> > >> >> _______________________________________________ >> > >>> > >> >> zeromq-dev mailing list >> > >>> > >> >> zeromq-dev@lists.zeromq.org >> > >>> > >> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> > >>> > >> > >> > >>> > >> > >> > >>> > > >> > >>> >> > >>> _______________________________________________ >> > >> zeromq-dev mailing list >> > >> zeromq-dev@lists.zeromq.org >> > >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> > > >> > > >> > _______________________________________________ >> > zeromq-dev mailing list >> > zeromq-dev@lists.zeromq.org >> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >> >>
_______________________________________________ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
