I'm pretty sure that with the C++ broker and Qpid Messaging the maximum
message size is limited only by available memory and ISTR int64_t types
are used internally. I've certainly on a few occasions send messages of
a few GB, though that was mostly for experimental purposes rather than
on a production system.
Whether you'd want to send such large messages depends on how
heterogeneous your messaging architecture is - that is to say if you
have lots of smaller messages and periodically a few larger ones it may
be fine, but OTOH there's a risk that when your larger messages are
being transported the QoS for the smaller messages might be adversely
affected - YMMV. One important thing to remember is the impact on the
broker because message queues tend to live in memory and unless you are
*very careful* wrt. max queue sizes, flow control and the like it's
really pretty easy to DoS the broker - when I was playing with this
stuff having everything grind to a halt due to swapping was very common.
You can have similar issues with clients too if you accidentally forget
to disable message prefetching.
On "If multiple sender publishing data to one queue chunk by chunk, how
consumer serialize the data of respective file data." well I don't
*think* that AMQP has anything built in to do this - it does support
transactions
http://docs.oasis-open.org/amqp/core/v1.0/amqp-core-transactions-v1.0.html
but although that is intended for reliable delivery of groups of
messages I'm not clear if there are mechanism to deal with sequencing,
though I might be wrong. I think that basically you need to treat your
messaging layer like a datagram layer and have your underlying
application perform the chunking and arrange for the chunks to be sent
as binary payloads with a sequence number sent as a message header.
TBH if I were doing this I'd also consider compressing the chunks - the
CPU overhead and compute latency will almost certainly be balanced out
by reduced network latency and costs.
So basically if you are thinking of transferring really large files then
writing a small library to chunk, compress and sequence is likely your
best option (I don't believe one exists off the shelf) - though there's
a larger question as to whether AMQP is really the best protocol for
that type of file transfer?
TL;DR For smaller files, well if you are talking a few 10s of MB then
there should be few issues, it'll deal with GB transfers, though there
may be implications and you might want to have a think about your use
case and whether you've hit the "if you have a hammer everything looks
like a nail" scenario https://en.wikipedia.org/wiki/Law_of_the_instrument
HTH
On 21/07/2020 07:40, umohank wrote:
Hi,
Can someone suggest me best method to transfer the file using QPID Messaging
C++(AMQP) API?
In general(TCP), we do copy (char buffer)data chunk by chunk and send data
until reaches end of the file, is that the same approach need to follow or
any other method to do the same.
Can we send the complete file in one shot?
If yes, what is the max size can be send?
If multiple sender publishing data to one queue chunk by chunk,
how consumer serialize the data of respective file data.
and please do share the snippet for file transfer example.
Thanks,
Mohan
--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org
