Re: On vacation next week for 4 weeks.

[Fj]

Have fun on your vacation, but actually I have a question (with an implied
complaint): how do I serialize and deserialize `message.correlation_id()`?

Like, *naturally* when writing an AMQP bridge to some other application
server, you have to do it asynchronously, and for that you have to pass
through the `correlation_id` thing among other things. So that when the
bridge receives a response from some application it knows where and how to
send it. Which means either storing the original message in some std::map,
which is fraught with soft memory leaks and actual crashes in my
experience, or you serialize the required stuff, so that you get all you
need to send a response from some std::string field that was passed all the
way around.

So, are there an Industry Best Practices™ regarding passing through the
`correlation_id`?

What they should cover: the python class proton.utils.SyncRequestResponse
that I'm using in tests uses `uint64_t` as `correlation_id` and expects the
same `uint64_t` in the response. So in my actual C++ code I have to
serialize to a string *both* the type of the original `correlation_id` and
its value, and then deserialize when I'm sending a response with a correct
type so that the python unit test passes (proton.utils.SyncRequestResponse
checks the `correlation_id` against the original python int value). And, of
course, I'd like some other client to get an uuid or string or whatever
they originally sent as a correlation id.

Is there already some code that takes care of that, serializing a
`correlation_id` into a csv or json or anything that could be passed around
as a dumb string? Or do I have to write it myself?

Again, have fun on your vacation, but you said that you are here for three
more days, and I couldn't resist asking what's on my mind.

On Tue, Jul 4, 2017 at 7:14 PM, Alan Conway  wrote:

> If you have anything that needs my urgent attention, ahhhahahahahhaaha!
>
> But seriously, I have 4 days to commiserate with you before I do a
> runner.
>
> I am working on one urgent item before I go - add reconnect support to
> the python qmf.client we created for Satellite. That is on track.
> Otherwise I'm mostly involved in very new or very old projects that
> should all be here when I get back.
>
> Cheers,
> Alan.
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
> For additional commands, e-mail: users-h...@qpid.apache.org
>
>

Re: [qpid-proton-cpp] default_container does not implement thread safety at all?

[Fj]

By the way, if anyone is interested, as a stop-gap solution I ended up just
implementing a timer-based callback reenqueuing itself every 50ms and
pulling callbacks from a properly synchronized queue and executing them on
the worker thread. It's not even that bad performance-wise, it just adds
those 50/2 ms of latency on average.

A better solution would be to do the same thing the Python binding does,
expose a Selectable interface and use it to wake up the reactor thread.

And by the way there's a thing that you might want to consider: there's a
bunch of use cases that don't need inter-thread synchronization, for
example, single-threaded workers, or multi-threaded workers that only need
to know when to stop and they know it from the socket being closed. In
those cases exposing a thread-safe Container.inject(callback) is a total
waste, because it means that everything else must constantly take locks for
no reason.

Maybe in the spirit of "you don't pay for what you don't use" the way
Python binding does it is actually fundamentally better, with an
EventInjector thingie that you can add to your Container if you want and
then call injector.inject(callback) instead of container.inject?

On Fri, Mar 24, 2017 at 3:30 PM, Alan Conway  wrote:

> On Fri, 2017-03-24 at 14:06 +0530, Venkat B wrote:
> > Good day,
> >
> > -- Forwarded message --
> > > From: Alan Conway 
> > > To: users@qpid.apache.org
> > > Cc:
> > > Bcc:
> > > Date: Wed, 22 Mar 2017 14:09:00 -0400
> > > Subject: Re: [qpid-proton-cpp] default_container does not implement
> > > thread
> > > safety at all?
> > >
> >
> > [snip]
> > >
> > > And to be clear this is abuse to hack out of a short-term hole - we
> > > will have a properly behaved solution soon.
> > >
> > >
> >
> > Do you have something that I could test?
> > The code in examples/cpp/mt does not compile out of the box on 0.17.0
> > and
> > in case it is being redesigned anyway then I would prefer to work
> > with the
> > new API.
> >
>
> I defer to astitcher on the current state of the examples. The API
> isn't changing significantly, the code in mt/broker.cpp will still be
> correct with only minor changes if any.
>
> The code in epoll_container.cpp will also still be basically unchanged
> but writing a custom container will no longer be necessary for most
> cases: the default_container will be thread-safe and we will have
> epoll-native, iocp-native and libuv flavours to cover most needs.
>
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
> For additional commands, e-mail: users-h...@qpid.apache.org
>
>

[qpid-proton-cpp] default_container does not implement thread safety at all?

[Fj]

Hello, I'm mightily confused.

There's
https://qpid.apache.org/releases/qpid-proton-0.17.0/proton/cpp/api/md_mt.html
that says in no uncertain terms that I can use event_loop.inject() to
execute a callback on the thread that runs the event loop. There's an
examples/cpp/mt/broker.cpp linked from the above that's supposed to be a
thread safe implementation of a thread pool running a bunch of containers.

But after "grep inject" and carefully sifting through the results, it seems
that after some indirection it all ends up calling

// FIXME aconway 2016-06-07: this is not thread safe. It is sufficient for
using
// default_container::schedule() inside a handler but not for inject() from
// another thread.
bool event_loop::impl::inject(void_function0& f) {
try { f(); } catch(...) {}
return true;
}

What the hell? Blank exception suppression is just the icing on the cake,
nothing here even resembles doing the actual hard thing: telling the
reactor to inject a new custom event in a threadsafe manner.

Am I missing something obvious, or is the documentation there, and in other
places, and the example, chemically pure wishful thinking, like, it would
be pretty nice if we supported multithreading, and it would work in such
and such ways then, but unfortunately we don't, as a matter of fact?

If so, maybe you gals and guys should fix the documentation, and not just
with "experimental" but with "DOESN'T WORK AT ALL" prominent on the page?



On more constructive notes:

1) do people maybe use some custom containers, similar to the
epoll_container in the examples (which doesn't support schedule() though, I
have to point out)? If so, I much desire to see one.

2) why do you attempt to bolt on your event_loop thing to Connection and
below when the only thing that has the run() method is Container, therefore
that's the scope that any worker thread would have? It's the Container that
should have the inject() method. Underlying C API notwithstanding.

3) What's the best way forward if I really have to have a threadsafe
container that I can inject some event into threadsafely:

  3a) does the C API support thread safety? Like, the whole problem is
touching the reactor's job queue or whatever it has with taking an
appropriate lock, and it must take that same lock itself for it to work.

  3b) I checked out the Python API, they use a mock EventInjector pollable
thingie instead of reaching inside the reactor and telling it add an event.

  3c) epoll example does yet another thing apparently (though I'm not sure
it works because it's not covered by any tests): just tell the reactor to
wake up and process the queued job list in your handler.