> a new Empty message type to avoid the overhead (and semantics) of > empty record batches
I like this! I wonder if the name is a bit misleading (the messages will contain *something*) but I'm happy with any name mapping to that concept. > a new application_data field in the Message table to pass arbitrary > opaque data with any kind of message I believe this could be done with the Empty message by putting the bytes in the body instead of in the header. Probably the only place this functionally makes a difference would be dissociated IPC where the body is transported separately. Perhaps both are useful. > It could be interesting to support multiplexing multiple IPC streams over the same socket I agree that there are some applications of the Empty where it would be tempting to have the payload of the Empty be serialized IPC (e.g., if used for statistics and the statistics are encoded in the lovely Arrow spec we have for that). Perhaps with Empty one could prototype that. -dewey On Wed, Feb 4, 2026 at 6:18 PM David Li <[email protected]> wrote: > custom_metadata requires strings, so adding a bytes field may be useful to > some applications, though. (This came up several years ago; you could > always base64 encode but that takes time.) > > On Thu, Feb 5, 2026, at 02:33, Rusty Conover wrote: > > Hi Antoine, > > > >> Ok, perhaps we can find a generic solution using two additions: > >> > >> 1) a new Empty message type to avoid the overhead (and semantics) of > >> empty record batches > >> 2) a new application_data field in the Message table to pass arbitrary > >> opaque data with any kind of message > >> > >> Something like: > >> https://gist.github.com/pitrou/363c4509706f56743f0ca0373f20949c > >> > >> What do you think? > > > > Thanks for making the gist. It looks good to me! > > > > One thought I've had about the `application_data` field is that it > > could be considered to be duplicative since you could accomplish the > > same thing with `custom_metadata`'s KeyValue. To do that you would need > > a coordinated value for the "key" field from the writer to reader. > > > > How can I help with this? > > > > Thank you, > > > > Rusty >
