Couple of replies inline.

On 06/02/15 13:19, Rafael Schloming wrote:
A couple of questions/comments inline, but first off, any reason this isn't on the list? (I don't mind, just curious.)

No good reason; Alan PMed me and I replied, then though I'd best copy you guys. It's slightly head explodingly long, so if conversation develops it might be best to split it up a bit - it was mainly a bit of a brain dump for Alan and I got carried away :-D I've moved to the Qpid user and proton lists now for maximum exposure.

On Fri, Feb 6, 2015 at 6:20 AM, Fraser Adams < <>> wrote:

    Hi Again Alan,
    Sorry this has turned out to be massively TL;DR I kind of got on a

    On the WebSocket thing it's worth pointing out that I've mainly
    been in the game of *using* them and have never had the pleasure
    of writing an implementation from scratch, but OTOH I've done a
    reasonable amount of"hacking around" - but that's mostly been in
    JavaScript and Python implementations so for C things are more

    I've copied Rafael so he's in the loop I've also copied Gordon,
    'cause I'd really quite like WebSocket support in qpidd at some
    point and Rob because he implemented the WebSocket transport in
    the Java Broker and I suspect he knows way more than me -
    especially about the trades around different implementation
    choices, he also knows the AMQP WebSocket spec intricacies way
    better than me so will be better placed to advise gotchas - and
    also I guess an approach that makes is easier for proton-j to
    introduce WebSocket support too.

    I guess first off I should say that when I suggested it might be a
    car crash if not thought through I was perhaps being a little
    parochial. As you know the JavaScript binding uses emscripten to
    compile proton-c to JavaScript - the way that works is the core
    grammar is compiled to LLVM bitcode and ends up as asm.js but
    library/system calls either wind up calling other compiled code or
    they break out into some pure JavaScript, that's the case for the
    network code and basically all the posix network calls get trapped
    and "faked" in a useful way; library.js and library_sockfs.js are
    the main places to look if you're interested.

    I guess that the key point is that all the *socket* calls in
    proton-c wind up as *WebSocket* stuff in the compiled JavaScript,
    for browsers it uses native WebSockets and for Node.js it uses the
    ws WebSocket library, which is I think the most popular one.

    The gnarly bit that I can think of is that clearly adding
    WebSocket support to proton-c risks accidentally causing the
    JavaScript stuff to implement WebSocket over WebSocket, which
    would be quite funny but probably not terribly useful :-D

It sounds like we quite probably would want to keep any web socket stuff compile-time optional and simply not build it at all with emscripten. That would a) avoid the possibility of nested websocketry, and b) keep useless code out of the already large proton.js file.

It's worth keeping an open mind. The proton-messenger.js stuff it indeed a bit big, most of that is actually down to the emscripten runtime stuff and there's a limit to what is easy to strip, there's some work ongoing on emscripten to make this a bit better. It's <500K minified and it could be compressed. I'm all for making it smaller but I doubt it'll make a huge difference I'm afraid. My main concern would be to make sure that any bits of code to add an API to select between TCP and WebSockets *weren't* removed I *want* to be able to pass that info to the emscripten code for when I get round to adding native TCP support.

    Another thing I worry a little about is how one implements the
    server side, there are two choices
    1. Have a separate WebSocket transport
    2. Note that WebSockets are actually a frame based protocol
    layered on top of TCP and initiated via an HTTP UPGRADE command.
    The spec is here:

    The Java Broker (currently) takes the former approach, it's a
    sensible and convenient approach because it uses Jetty as the Web
    Server for its UI and Jetty comes complete with a WebSocket
    implementation, but there's a down side - you have to stand up a
    separate port so basically AMQP over TCP connections connect to
    one port and AMQP over WebSocket connections connect over a
    different port.

    But as I say WebSockets are actually layered over TCP so it's
    possible if you have more control over the server to look at the
    data on the TCP buffers and detect that it's actually WebSocket,
    so from a user perspective it would be *really nice* and
    technically possible to have a single listen port.

When I suggested implementing it in proton I meant the latter, simply autodetecting and upgrading. Andrew has done some nice work around this sort of thing already with detecting SSL vs plain and detecting SASL vs just core AMQP, so autodetecting the HTTP header and doing the upgrade would be a natural and hopefully straightforward way to go. At that point you just configure on the transport whether you want to allow it or not.
That's cool, I thought that was what you had in mind when you mentioned Andrew's autodetect stuff but thought I should include Rob and mention the Java Broker approach to see if there were views on the relative merits of each.

    That's kind of a nice segway into another point
    1. Use a library
    2. Roll your own/copy paste reuse

    I'm torn on that, using a library is clearly good from a support
    perspective but the counter to that is illustrated by the point I
    make about using Jetty in the Java Broker. Of course you might
    *want* to force TCP and WebSocket connections to different ports
    but do we care more that it's AMQP or what the transport is
    (that's an open question BTW, I personally like the idea of just
    connecting to a single port but there may be good counter arguments).

    On a slightly library related note it's worth mentioning something
    on my TODO list. As you probably know the JavaScript binding is
    currently limited to using a WebSocket transport, that makes sense
    as a starter for ten 'cause obviously browsers have a number of
    constraints but for Node.js there's no real reason for that to be
    the case. You might have seen the TCP->WebSocket proxy stuff I
    wrote, if you have you'll see that it's actually technically
    pretty easy to implement at TCP version using the Node.js built in
    net library. The main reasons I've not done it yet are:
    1. Time - I only get to code at weekends and I'm juggling a couple
    of O/S projects at the moment #ifonlyididntneedtoworktolive :-)
    2. The emscripten network code really isn't as modular as it
    should be - I've done quite a lot of it, but I didn't originate it
    and TBH I'd quite like to pick it apart and make it a bit more
    layered, lack of time again.
    3. The biggest reason though is actually what approach to take. As
    above I'd really like to have a Node.js AMQP application that can
    just receive either a TCP or a WebSocket connection and *just
    work*. That's a bit awkward because I use ws as the WebSocket
    server. As it's so well established it's a bit daft to roll my own
    but I don't know, it's probably possible to Monkey Patch or
    otherwise expose the native TCP socket if a WebSocket connection
    fails, though probably the best approach would be to add a feature
    to ws and hope that the pull request gets accepted.

    Hopefully point three above illustrates one potentially awkward
    pinch-point, you'll be adding WebSocket server code in pure C and
    I'll be making the backend of the TCP path all WebSockety too :-)

    It *should* work out OK because  the emscripten stuff will
    establish a listener and ws will have removed all of the
    WebSockety stuff and just exposes the AMQP frames in the octet
    array that comes off the recv call, so the proton-c code *should*
    just interpret that as AMQP off a TCP socket and not send it to
    any of the proton-c WebSocket code, but depending on how its done
    it could go horribly wrong, which is why I figured I'd give a bit
    of background.

    All of the above has been from the perspective of a server
    receiving a WebSocket connection, in some respects though
    establishing a connection is actually more awkward....

    So it's not like it's technically hard to write the code to have
    Node.js establish an AMQP over TCP connection - the net library is
    pretty straightforward. The bit that I'm actually struggling with
    is how best to let applications choose between a TCP and WebSocket

    As I say the default position is that the proton-c network code
    maps to emscripten library_sockfs.js calls that push the data over
    WebSockets - so what API to use to tell it to use TCP :-) It's
    easy to set a compile time option but that sucks, I'd like an
    application to be able to connect using either TCP or WebSockets.

    I was wondering if the getprotoent /etc/protocols stuff might be
    the best place, that's certainly where you'd get the protocol
    number for UDP, TCP or SCTP to the sockets API but that's used
    pretty much for the underlying Operating System networking and
    WebSockets currently tend to be implemented as user level
    libraries over TCP and I'm not aware of an official protocol entry
    for WebSockets.

    That's the only place in the socket API that I can think of that's
    even close to being able to specify that sort of thing though :-/

    There are certainly other ways to pass that sort of info down to
    emscripten, but as I say I'm struggling for a clean approach.

    So that conundrum is kind of relevant to the native proton-c
    implementation too 'cause if you want to allow the ability to
    connect using either TCP or WebSockets there needs to be something
    in the API that allows users to choose. Is there anything in any
    of the AMQP specs that have actually covered this? I know that
    Qpid C++ has support for rdma but I *think* that's specified in
    the Connection config which offers a slightly richer set of
    connection options than the sort of amqp connection available in
    Proton (certainly from what I'm aware of with Messenger).

I think this should be fairly straightforward in proton-c. We already have a similar set of scenarios there with SSL and SASL autodetect. When you configure a Transport, you basically construct a stack of protocols. In server mode you can it autodetect whether each layer is present and fall through to the next. In client mode the stack is just what you have specified.

So I can see how the autodetect could be extended to handle this when acting as a server, but this last bit was really about what sort of "API" one might use for a client to establish a connection giving a runtime choice between a TCP or WebSocket transport. As I say for the qpid::messaging API the Connection allows quite a range of options but in Messenger at any rate I think the options are more limited and ISTR there are some restrictions even on SASL options, so as I say it's probably not "technically" hard I'm just not sure what the best "specify the transport" API might be for Proton - if you see what I mean?

    So I've covered some of the thought processes going through my
    mind as I've been thinking around WebSockets in native proton-c
    and most of them aren't really about the technical implementation
    of the WebSocket protocol, but I think are worth bearing in mind.

    In terms of actually implementing this stuff as Rafael said I
    think that options in C are fairly limited. As well as the library
    issues I mentioned earlier there's also the little matter of
    worrying about license incompatibilities, so my gut feeling is
    that the best approach might to get "inspiration" from other
    places and to grasp the nettle and implement something that
    integrates nicely with Proton.

    If it were me implementing this the first place I'd look would be

    WebSockify is actually a TCP->WebSocket proxy and TBH bits of it
    irritate me, but itQpid user list now for maximum exposure
    seems to be used quite a lot. The most significant things though
    is that:
    1. It implements a pretty complete WebSocket stack in Python,
    which you know so should help you figure things out
    2. It actually also has an implementation in C (under the "other"
    directory) so you might want to look at that
    3. I *think* that it accepts TCP or WebSocket connections on its
    listen port and just passes through TCP (I can't swear to that but
    it rings a bell)
    4. It's well used and well maintained so is likely to be a
    reliable starting point

    I'd also take a look at ws:

    This is clearly JavaScript, but it's one of the most commonly used
    and well maintained WebSocket libraries in any language, so again
    as a means to figure out how it actually works it might be a good
    place to look.

    BTW Rafael mentioned the annoying thing about there being lots of
    different WebSocket versions, that's certainly true, but is much
    less of an issue now. That stuff was generally a pain as the
    WebSocket protocol was evolving, but it has now been standardised
    in the IETF RFC 6455 I linked earlier and all of the main browsers
    primarily support that. Looking on the websockify page there is a
    comment "Starting with websockify 0.5.0, only the HyBi / IETF 6455
    WebSocket protocol is supported". Unless there's a compelling
    customer reason to support pre-standard versions I don't think
    it's unreasonable to only support RFC 6455. It's also worth
    bearing in mind that the emscripten based JavaScript binding
    requires binary support anyway and I think that's only supported
    on the final versions of the spec. so there's probably little
    actual value in supporting anything else.

    Sorry for the essay, hope it's useful.

    I'm off to focus on my other project now - I'm currently trying to
    write an implementation of webcl using webgl, cause I need to do
    some GPU programming and my platform doesn't have opencl


    On 04/02/15 19:30, Alan Conway wrote:

        On Wed, 2015-02-04 at 17:55 +0000, Fraser Adams wrote:

            Hi Alan, very quick placeholder response. I've a couple of
            commitments so won't be able to give a decent response for
            a day or so -
            don't want you to think I'm rude by not getting back.

            It's a topic that deserves a bit of thought 'cause it
            could be a bit of
            a car-crash if not thought through.

        Thanks for the warning! Andrew Stitcher is making changes to the
        transport layer that will affect this (make it easier) so I'm
        not moving
        on proton till that is done. When I do I will consult
        carefully with
        yourself and Andrew to avoid crashing any cars.

        I may try some short term hackery to tide me over but if I do
        it will be
        in the dispatch codebase and strictly limited to supporting
        JS management console. Longer term proton is the right place
        to do it
        and I will avail of all advice so I do it right ;)


            On 04/02/15 14:46, Alan Conway wrote:

                Hi Fraser,

                I need a websocket listener in dispatch. Rafi points
                out (rightly) that
                it would be of wider use to do it in proton-c than in
                dispatch. Quick
                JIRA search reveals the word websocket next to your
                name - is that
                Java-only? I know nada about websockets, and not much
                more about JS if
                you remember our prev. conversation ;) Actually on
                that - I now realize
                that (duh!) the java build does the same
                barf-in-the-source-dir thing so
                I was razzing you for nothing. git-ignore is my friend.

                Hoping to pick your brains for anything useful on
                websockets. I'm
                guessing my options are
                - find a C websockets library (looks like there are
                some but none
                packaged with popular OS distros (at least the ones
                popular with me)
                - implement the websocket spec in C. Not sure how big
                a job that is yet.
                The spec is full of non-normative waffle so I'm not
                sure just how much
                real action there is.
                - grab a python ws library and just stick with
                dispatch since it has a
                nice C+python implementation. That wouldn't help
                proton though.

                Any thoughts?


Reply via email to