Re: [systemd-devel] sd-bus: Enabling free-standing, bus-independent plain messages
On Do, 23.05.19 11:56, Stanislav Angelovič (angelovi...@gmail.com) wrote: > Hi Lennart, > > Sorry for a bit late reply, I've been quite busy recently. > > > So we could readd the ability to create a bus message with a NULL bus, > > but I am a bit concerned that people then always pass NULL which might > > ultimately result in constant remarshalling if the message is > > eventually enqueued on a real bus. When you put together a message it > > actually matters which bus connection you do that for... > > My proposal is that we would allow passing NULL bus ptr only for > `sd_bus_message_new` function. And such a message would be disallowed from > enqueuing. In functions that need real bus ptr we might place an assert > that bus != NULL. In a limited set of functions, which I mentioned in > previous e-mail (`sd_bus_message_seal` for example), we would allow bus == > NULL. Prepare a patch that does this. If it's minimal and doesn't litter the API with too many new calls, that should be OK. (Note that due to the demise of kdbus, gvariant-over-dbus never really materialized in the end, hence the logic in sd-bus that can generate either is kinda dead. There's hope though that dbus-broker might add this eventually, not sure where that went though. In general I think it would make a ton of sense to switch to gvariant for everything, it just needs somebody to hack it up in dbus-daemon/dbus-broker) > OK, I will try. So we don't introduce any new message factory function > without bus parameter, but make use of existing `sd_bus_message_new`, > right? We will modify it to allow NULL bus ptr. However, this function > needs three things from the bus when initializing message: > `message_version`, `can_fds`, and `allow_interactive_authorization`. What > values shall we use there when bus is NULL? there never was any other version than 1, and fds are off. the allow a_i_a stuff should probably left off, as it doesn't really matter for most cases. Lennart -- Lennart Poettering, Berlin ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] sd-bus: Enabling free-standing, bus-independent plain messages
Hi Lennart, Sorry for a bit late reply, I've been quite busy recently. > So we could readd the ability to create a bus message with a NULL bus, > but I am a bit concerned that people then always pass NULL which might > ultimately result in constant remarshalling if the message is > eventually enqueued on a real bus. When you put together a message it > actually matters which bus connection you do that for... My proposal is that we would allow passing NULL bus ptr only for `sd_bus_message_new` function. And such a message would be disallowed from enqueuing. In functions that need real bus ptr we might place an assert that bus != NULL. In a limited set of functions, which I mentioned in previous e-mail (`sd_bus_message_seal` for example), we would allow bus == NULL. > Note that message behaviour and so on depends in sometimes subtle and sometimes not > so subtle ways on the bus connection used, i.e. whether we are talking > to a real bus or not, and so on. I am not to keen of making this > completely independent I must admit... > > What's the usecase for using bus message as general marshalling > storage anyway? Can you elaborate? I provided a use case example a few years ago in https://lists.freedesktop.org/archives/systemd-devel/2016- November/037929.html, but I'll elaborate more. sdbus-c++ provides such a generic layer of abstraction that its clients don't work with messages but call D-Bus methods as-if it was a local operation, using native types. For that, all D-Bus types have their native representation. A D-Bus array is represented as std::vector. A D-Bus string is std::string. And for D-Bus variant there is custom sdbus::Variant class that needs to be able to store whatever D-Bus types. So for example, for a method `doSomething` that takes a D-Bus string and an int, and returns an array of variants, sdbus-c++ generates something like `std::vector doSomething(std::string str, int i);`. Clients call this method, and sdbus-c++ does all the job -- it creates a method call message, serializes data into it, sends the message, receives the reply message, and automatically deserializes its contents into std::vector. But variant is special. When deserializing, sdbus-c++ is unaware what real type is under each variant, but it still must be able to get a given variant out of the message and store it in the vector to be returned. This is done by `sd_bus_message_copy` of just that variant message part. It cannot do `sd_bus_message_enter_container` and `sd_bus_message_read` directly, because sd-bus API requires signature of underlying variant contents, and this is what sdbus-c++ has no idea about at this point. It's the client who knows, so the client then at a later point, in his own code, asks for an object of specific type from that Variant, like so `double d = variant.get()`. Here is when `sd_bus_message_enter_container` and `sd_bus_message_read` on variant message are called, so the deserialization from variant message takes place. It's something like 2-step deserialization -- but only with variants, they are special in that. Of course, our first idea was to use C++ type-erasure technique employed by e.g. std::any to implement the Variant class. This works fine when we construct Variant from a native type. But when we need to create Variant out of a part of incoming D-Bus message, like in the example of method reply message above, we are stuck. The code would be very complex, practically impossible to write, due to conversion of run-time combinations of D-Bus types to static, compile-time type representations (vectors, maps of various keys and values that could maps again etc...). Hence, using sd-bus message as an underlying implementation of Variant class is the simplest and most reasonable solution; we are simply using the concept that is implemented in sd-bus already. We can easily create Variant out of a native type (by serializing the value into underlying message), create Variant out of existing message (by copying the message part into underlying message), and read data back from Variant (by reading the underlying message). > A compromise might be that we readd a concept of allowing > bus-independent messages to be generated again (i.e. pass NULL as bus > object when creating a bus message), with the most reduced feature set > possible, and at the same time refuse to enqueue such messages on any > bus, thus forcing people to use sd_bus_message_copy() to explicitly > remarshal for the bus, instead of doing implicitly so. Fully agree, that is also what I meant. (The reduced feature set would be the functions I mentioned in the previous e-mail.) > if you want to prep such a patch I think we should merge it. OK, I will try. So we don't introduce any new message factory function without bus parameter, but make use of existing `sd_bus_message_new`, right? We will modify it to allow NULL bus ptr. However, this function needs three things from the bus when initializing message: `message_version`,
Re: [systemd-devel] sd-bus: Enabling free-standing, bus-independent plain messages
On Mon, 13 May 2019 at 10:38:38 +0200, Lennart Poettering wrote:
> On So, 12.05.19 13:06, Stanislav Angelovič (angelovi...@gmail.com) wrote:
> > Quick question: Would it be possible to extend sd-bus to also allow
> > creating messages without requiring a bus?
>
> This used to be available but we dropped it, since the serialization
> depends on the bus feature set, i.e. wheter fd passing is available,
> and whether gvariant serialization is available.
I think it might make sense to have a way to serialize data to a D-Bus
1.0 message, and/or to a GVariant. I agree that it doesn't make sense
to serialize to an unspecified D-Bus-adjacent binary format, unless you
already know which bus you are talking to, in which case you should use
the most natural format for that bus.
> What's the usecase for using bus message as general marshalling
> storage anyway? Can you elaborate?
Various GLib-related projects like gvdb (used in dconf) and libostree
use GVariant as a stable on-disk encoding for strongly-typed data,
reusing GVariant's D-Bus-inspired type model, in the same sorts of
places that a different project might have used (for example) JSON,
XDR or Protocol Buffers.
Again, this is a reasonable thing to do with data known to be serialized
in GVariant format, or known to be serialized in D-Bus 1.0 format, but
not a reasonable thing to do with a binary format that might be GVariant
or D-Bus 1.0 or some future message encoding.
If what you want is a "variant" type, note that the recursive type
systems of both GVariant and D-Bus 1.0 require out-of-band information
to encode/decode. For both formats, you need to know the type signature
(e.g. "a{sv}") and the byte-order (big- or little-endian), and for
GVariant you additionally need to know the message length in bytes. On
D-Bus this information is already available in the message header, but
if you're using GVariant or D-Bus 1.0 as a standalone serialization you
will need to provide or reinvent a subset of the message header.
smcv
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] sd-bus: Enabling free-standing, bus-independent plain messages
On So, 12.05.19 13:06, Stanislav Angelovič (angelovi...@gmail.com) wrote: > Hi (Lennart :) > > Quick question: Would it be possible to extend sd-bus to also allow > creating messages without requiring a bus? This used to be available but we dropped it, since the serialization depends on the bus feature set, i.e. wheter fd passing is available, and whether gvariant serialization is available. Some of that never materialized IRL due to the demise of kdbus, but the semantics still make sense: serialization depends on the feature set of the bus, and hence each message should be associated with a connection. > Let me explain: Currently, if we want to create any message in sd-bus, we > need a valid bus pointer. That might make perfect sense for messages that > come from or will eventually go to bus. But sd-bus also supports plain > messages. Do we also need that hard msg->bus dependency for plain messages > which we use only as a local, temporary storage of serialized data? > > Some years ago we had a little discussion about a few tweaks in sd-bus API > to allow modelling the concept of Variant type in higher-level sd-bus > bindings ( > https://lists.freedesktop.org/archives/systemd-devel/2016-November/037929.html). > A Variant is essentially implemented as a class around plain sd-bus message > which is only used a storage of the underlying type. To create a Variant > instance, we simply call `sd_bus_message_new` factory with type > _SD_BUS_MESSAGE_TYPE_INVALID. Nice and simple. But that factory requires > real bus ptr. This leads to a more complicated, less efficient and > not-that-nice solution that must take hold of some bus (if there is none it > must create one), must cache the bus (so that it's not created and > destroyed at every creation/deletion of Variant instance). Yes, we have > `sd_bus_default_system`, but even with that there is some effort that has > some pitfalls (e.g. we can't `std::move` a Variant to a different thread, > because it might potentially outlive the current one). All that > nomenclature is needed just to get a plain message to read/write > data from. So we could readd the ability to create a bus message with a NULL bus, but I am a bit concerned that people then always pass NULL which might ultimately result in constant remarshalling if the message is eventually enqueued on a real bus. When you put together a message it actually matters which bus connection you do that for... > Even today, vast majority of these functions doesn't seem to need the bus. > The exception are `sd_bus_message_seal` and `sd_bus_message_new` itself. > We'd need to either modify `sd_bus_message_new` to allow taking in NULL bus > ptr, or creating a new function for creating such plain local messages. > We'd need to modify `sd_bus_message_seal` to simply consider the fact that > m->bus can be NULL. And we'd probably need to add asserts for m->bus to > functions which truly require real bus ptr presence. (We might even provide > the option to link the message to a real bus later, and introduce new > `sd_bus_message_set_bus` function, but I'm not sure whether that is good.) > > To summarize: sd-bus already offers API for creating plain message. To make > it fully true, let's just not require the bus there. The proposed change > would IMHO make sd-bus message more flexible, and would make it easier, > more intuitive and more robust to work with and model Variants around such > free-standing, bus-independent sd-bus messages in higher-level languages. > > What do you think? Note that message behaviour and so on depends in sometimes subtle and sometimes not so subtle ways on the bus connection used, i.e. whether we are talking to a real bus or not, and so on. I am not to keen of making this completely independent I must admit... What's the usecase for using bus message as general marshalling storage anyway? Can you elaborate? A compromise might be that we readd a concept of allowing bus-independent messages to be generated again (i.e. pass NULL as bus object when creating a bus message), with the most reduced feature set possible, and at the same time refuse to enqueue such messages on any bus, thus forcing people to use sd_bus_message_copy() to explicitly remarshal for the bus, instead of doing implicitly so. if you want to prep such a patch I think we should merge it. Lennart -- Lennart Poettering, Berlin ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] sd-bus: Enabling free-standing, bus-independent plain messages
Hi (Lennart :) Quick question: Would it be possible to extend sd-bus to also allow creating messages without requiring a bus? Let me explain: Currently, if we want to create any message in sd-bus, we need a valid bus pointer. That might make perfect sense for messages that come from or will eventually go to bus. But sd-bus also supports plain messages. Do we also need that hard msg->bus dependency for plain messages which we use only as a local, temporary storage of serialized data? Some years ago we had a little discussion about a few tweaks in sd-bus API to allow modelling the concept of Variant type in higher-level sd-bus bindings ( https://lists.freedesktop.org/archives/systemd-devel/2016-November/037929.html). A Variant is essentially implemented as a class around plain sd-bus message which is only used a storage of the underlying type. To create a Variant instance, we simply call `sd_bus_message_new` factory with type _SD_BUS_MESSAGE_TYPE_INVALID. Nice and simple. But that factory requires real bus ptr. This leads to a more complicated, less efficient and not-that-nice solution that must take hold of some bus (if there is none it must create one), must cache the bus (so that it's not created and destroyed at every creation/deletion of Variant instance). Yes, we have `sd_bus_default_system`, but even with that there is some effort that has some pitfalls (e.g. we can't `std::move` a Variant to a different thread, because it might potentially outlive the current one). All that nomenclature is needed just to get a plain message to read/write data from. It looks like the bus is not really needed when the message is used as a local storage only, i.e. when only these API functions are used: * creating plain, empty message (`sd_bus_message_new`) * writing/reading to/from the message (`sd_bus_message_append*`, `sd_bus_message_read*`, ...) * sealing the message (`sd_bus_message_seal` * copying to/from the message (`sd_bus_message_copy`) * rewinding the message (`sd_bus_message_rewind`) * peeking message type (`sd_bus_message_peek_type`) * reffing/unreffing the message (sd_bus_message_ref/unref`) Even today, vast majority of these functions doesn't seem to need the bus. The exception are `sd_bus_message_seal` and `sd_bus_message_new` itself. We'd need to either modify `sd_bus_message_new` to allow taking in NULL bus ptr, or creating a new function for creating such plain local messages. We'd need to modify `sd_bus_message_seal` to simply consider the fact that m->bus can be NULL. And we'd probably need to add asserts for m->bus to functions which truly require real bus ptr presence. (We might even provide the option to link the message to a real bus later, and introduce new `sd_bus_message_set_bus` function, but I'm not sure whether that is good.) To summarize: sd-bus already offers API for creating plain message. To make it fully true, let's just not require the bus there. The proposed change would IMHO make sd-bus message more flexible, and would make it easier, more intuitive and more robust to work with and model Variants around such free-standing, bus-independent sd-bus messages in higher-level languages. What do you think? Thank you, and kind regards, Stanislav. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
