On 2020/09/04 07:17:36, Alexander Sibiryakov <sibirya...@scrapinghub.com>
wrote:
> Hello,
>
> I would like to get your opinions on this KIP idea.
>
> In short it will allow to transfer messages of bigger size than allowed by
> the broker.
>
> https://docs.google.com/document/d/1cKBNxPkVVENly9YczYXsVDVWwrCdRq3G_cja5s2QDQg/edit?usp=sharing
>
> If all that makes sense, I'll create a full fledged KIP document and expand
> the idea.
>
> Thanks,
> A.
>
Hi Alexander,
It's an interesting proposal. I've worked on previous brokers which support
sending very large messages, up to 8 GiB with the server running in only 50 MB
of RAM.
https://activemq.apache.org/components/artemis/documentation/2.6.0/large-messages.html
There are generally two approaches that can be used to accomplish this:
1). Split the large message into smaller chunks at the producer and send it
over the same mechanism as normal messages. Reassemble the large message from
the chunks at the consumer.
2). The client streams the large message out of band from the client to the
disk on the server (not stored in the log with other messages), then send a
unique id for the message in the actual message. When the consumer receives the
small message with the unique id it initiates a streaming download out of band
from the server.
There are pros and cons for both approaches:
1) Means you don't need any special machinery for large messages on the broker,
things just work as is
2) Is more efficient as you don't have to encode everything in little chunks
and send through the machinery of the broker. You can open a socket to the
server, and the server can slam it do disk using sendfile so this can be very
fast and CPU efficient.
1) Can mean small messages can get lost in the log and slow to retrieve when
there are millions of fragments of large messages in the same log.
BTW the concept of streaming messages has been around at least since JMS (1998
iirc)!
