Are you using StreamMessage because you think it will enable 'steaming'? If so I wouldnt bother, the Stream name isnt related, its just a JMS message type name meaning there is a list/sequence of things in it. Previously you just seemed to want to send text/bytes.
For the Core client side you can see https://activemq.apache.org/components/artemis/documentation/latest/large-messages.html For the broker, it would ordinarily treat the large message as a chunked stream if doing e.g core->core, or amqp->amqp...but in this case having to do core->amqp means the broker will necessarily have to convert the message and in doing so it will load the entire message. (It seems like its that last step currently not handling your sent Core messages, because you are sending raw Core payloads that dont match the JMS style Core message structure the brokers core->AMQP converter is geared around, meaning it falls back to one that tries to convert it as a string, which presumably fails due to starting to read your bare payload as a size indicator) On Mon, 25 Jul 2022 at 06:40, Jan Šmucr <jan.sm...@aimtecglobal.com> wrote: > > Thank you for the feedback. > So what would this be the proper way of streaming files (if the code below is > correct)? > > final String body = "foo"; > > try (Connection connection = createConnection()) { > Session session = connection.createSession(false, > Session.AUTO_ACKNOWLEDGE); > MessageProducer producer = > session.createProducer(session.createQueue(getQueueName())); > StreamMessage message = session.createStreamMessage(); > message.writeBytes(body.getBytes()); > producer.send(message); > } > > AmqpClient client = createAmqpClient(); > AmqpConnection amqpConnection = addConnection(client.connect()); > AmqpSession amqpSession = amqpConnection.createSession(); > AmqpReceiver receiver = amqpSession.createReceiver(getQueueName()); > receiver.flow(100); > AmqpMessage amqpMessage = receiver.receive(1, TimeUnit.SECONDS); > Assert.assertNotNull(amqpMessage); > Assert.assertEquals(body, new String(((Binary) ((AmqpSequence) > amqpMessage.getWrappedMessage().getBody()).getValue().get(0)).getArray())); > > We want to stream the payloads due to the fact that these can take up to > hundreds of megabytes. > > Thank you. > Jan > > From: Clebert Suconic<mailto:clebert.suco...@gmail.com> > Sent: sobota 23. července 2022 20:50 > To: users@activemq.apache.org<mailto:users@activemq.apache.org> > Subject: Re: Artemis - Large Message - Core => AMQP > > We could look at enhancing the converters. But the best would be to change > your producer to have the exact format the converter would have. > > On Sat, Jul 23, 2022 at 2:41 PM Clebert Suconic <clebert.suco...@gmail.com> > wrote: > > > Change your message in a way is compatible ? > > > > On Fri, Jul 22, 2022 at 3:36 PM Jan Šmucr <jan.sm...@aimtecglobal.com> > > wrote: > > > >> The reason for this is that there's a whole infrastructure built using > >> the core protocol, and now we need to connect a Python-based Lambda capable > >> of receiving large messages. Is there any other, core-compatible method? > >> > >> Jan > >> > >> Dne 22. 7. 2022 21:21 napsal uživatel Clebert Suconic < > >> clebert.suco...@gmail.com>: > >> If you expect conversions to happen, you have to send messages from > >> qpid-ms or core-jms client. > >> > >> > >> On the case: > >> > >> > >> ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory(); > >> Connection conn = factory.createConnection(...); > >> Session session = conn.createSession(...) > >> Producer producer = session.createProducer(....); > >> producer.send(session.createTextMessage(YOUR-LARGE-BODY-GOES-HERE); > >> > >> > >> > >> Then the conversion from Core to AMQP should happen fine when you > >> receive the message on the other side at your Python client. > >> > >> > >> > >> If you really must use the Core-API, then you will need to hack your > >> way in and find what the JMS client would do.. but at that point it's > >> just a non standard way of doing this... You would need to have a very > >> good reason to not do the right thing on this case. > >> > >> On Fri, Jul 22, 2022 at 3:00 PM Clebert Suconic > >> <clebert.suco...@gmail.com> wrote: > >> > > >> > If you’re not using qpid JMS it will not work. > >> > > >> > On Fri, Jul 22, 2022 at 8:06 AM Andy Yar <andyya...@gmail.com> wrote: > >> >> > >> >> There is another attempt employing AMQP test structures. It produces > >> the same incorrect message body. > >> >> > >> >> On Fri, Jul 22, 2022 at 12:21 PM Andy Yar <andyya...@gmail.com> wrote: > >> >>> > >> >>> Hello, > >> >>> Using the LargeServerMessageImpl was just an attempt providing a > >> seemingly easy way to introduce a kind of Large Message into the test. I'm > >> not familiar with the code base at all - so I simply tried to somehow > >> provide the requested test replicating my "sending via Core -> receiving > >> via Qpid" issue. The same applies to the explicit AMQP conversion - the > >> encountered error simply pointed on it. Is there an intended "test way" to > >> model the message transfer among different clients/protocols? > >> >>> > >> >>> Anyway, I've changed the test to employ > >> ClientLargeMessageImpl/ClientMessageImpl instead of the server-related > >> LargeServerMessageImpl with a very similar result. Does that implementation > >> fit the simulated use case (sending via Core -> receiving via Qpid AMQP) > >> better or is the explicit AMQP conversion misused? > >> >>> > >> >>> Please, see the attached patch. > >> >>> > >> >>> On Mon, Jul 18, 2022 at 3:50 PM Clebert Suconic < > >> clebert.suco...@gmail.com> wrote: > >> >>>> > >> >>>> AMQP to Core and Core to AMQP conversions expect you using JMS > >> clients > >> >>>> to generate your message. There are certain caveats that both clients > >> >>>> will add to the messages. > >> >>>> > >> >>>> > >> >>>> Usually I have seen more users doing weird conversions in AMQP when > >> >>>> they use a C++ or non Java Client.. but on your case you used a Core > >> >>>> Client, using a non standard way to send a message (ServerMessage > >> from > >> >>>> the client)... and it's not clear the message is correctly set for a > >> >>>> conversion to work. > >> >>>> > >> >>>> > >> >>>> I need you to provide some explanation on what you're doing.. if this > >> >>>> was a way to hack a bug your saw or if this is just the way you're > >> >>>> using it. > >> >>>> > >> >>>> > >> >>>> if this is how you're actually using, I would suggest you either use > >> >>>> the proper APIs or if you're doing some hack for performance > >> >>>> consideration you have to set the message in a better way the > >> >>>> converters would understand... > >> >>>> > >> >>>> On Mon, Jul 18, 2022 at 9:38 AM Clebert Suconic > >> >>>> <clebert.suco...@gmail.com> wrote: > >> >>>> > > >> >>>> > Are you using the LargeServerMessageImpl on your client, or that > >> was > >> >>>> > just a "hack" to reproduce your issue. > >> >>>> > > >> >>>> > > >> >>>> > LargeServerMessageImpl was not meant to be used on the client. But > >> if > >> >>>> > you're doing that just to show something you faced in production > >> it's > >> >>>> > ok.. but using the LargeServerMessageImpl to send a Client Message > >> is > >> >>>> > a big not for me. > >> >>>> > > >> >>>> > The only reason we do that is for server to server transfer. > >> >>>> > > >> >>>> > On Mon, Jul 18, 2022 at 7:39 AM Andy Yar <andyya...@gmail.com> > >> wrote: > >> >>>> > > > >> >>>> > > Hello, > >> >>>> > > Yes, the 2.23.1 test instance was freshly created. > >> >>>> > > > >> >>>> > > I've attached a test as a part of ARTEMIS-3897 > >> >>>> > > <https://issues.apache.org/jira/browse/ARTEMIS-3897>. I hope it > >> helps. > >> >>>> > > > >> >>>> > > > >> >>>> > > On Sat, Jul 16, 2022 at 5:21 PM Clebert Suconic < > >> clebert.suco...@gmail.com> > >> >>>> > > wrote: > >> >>>> > > > >> >>>> > > > Did you start with fresh data on 2.23.1. > >> >>>> > > > > >> >>>> > > > If you did please provide a self enclosing test reproducing > >> your issue. > >> >>>> > > > > >> >>>> > > > On Fri, Jul 15, 2022 at 4:29 AM Andy Yar <andyya...@gmail.com> > >> wrote: > >> >>>> > > > > >> >>>> > > > > A quick test using 2.23.1 results in the same error payload > >> being > >> >>>> > > > received: > >> >>>> > > > > > >> >>>> > > > > "Message(address='test', durable=True, priority=4, > >> >>>> > > > > annotations=AnnotationDict({symbol('x-opt-jms-dest'): > >> byte(0), > >> >>>> > > > > symbol('x-opt-jms-msg-type'): byte(0)}), > >> >>>> > > > > properties={'JMSXDeliveryCount': None, '_AMQ_LARGE_SIZE': > >> 67}, > >> >>>> > > > > body='Conversion to AMQP error: Error reading in > >> simpleString, > >> >>>> > > > > length=1330464032 is greater than readableBytes=62')" > >> >>>> > > > > > >> >>>> > > > > Best regards > >> >>>> > > > > > >> >>>> > > > > On Thu, Jul 14, 2022 at 9:18 PM Clebert Suconic > >> >>>> > > > > <clebert.suco...@gmail.com> wrote: > >> >>>> > > > > > > >> >>>> > > > > > There's been a few fixes in AMQP Large message. > >> >>>> > > > > > > >> >>>> > > > > > > >> >>>> > > > > > More prominently a fix with the JDBC implementation > >> between 2.17 and > >> >>>> > > > > HEAD. > >> >>>> > > > > > > >> >>>> > > > > > > >> >>>> > > > > > I would definitely recommend you to upgrade. > >> >>>> > > > > > > >> >>>> > > > > > On Thu, Jul 14, 2022 at 10:26 AM Justin Bertram < > >> jbert...@apache.org> > >> >>>> > > > > wrote: > >> >>>> > > > > > > > >> >>>> > > > > > > I would expect this to work. Can you try this on the > >> latest release > >> >>>> > > > > (i.e. > >> >>>> > > > > > > 2.23.1) [1]? > >> >>>> > > > > > > > >> >>>> > > > > > > If it still doesn't work please open a Jira with all the > >> details > >> >>>> > > > > necessary > >> >>>> > > > > > > to reproduce the problem. An actual test-case would be > >> ideal. > >> >>>> > > > > > > > >> >>>> > > > > > > Thanks! > >> >>>> > > > > > > > >> >>>> > > > > > > > >> >>>> > > > > > > Justin > >> >>>> > > > > > > > >> >>>> > > > > > > [1] > >> https://activemq.apache.org/components/artemis/download/ > >> >>>> > > > > > > > >> >>>> > > > > > > On Thu, Jul 14, 2022 at 9:20 AM Andy Yar < > >> andyya...@gmail.com> > >> >>>> > > > wrote: > >> >>>> > > > > > > > >> >>>> > > > > > > > Hello, > >> >>>> > > > > > > > Sending a message as a Large Message via Core protocol > >> and > >> >>>> > > > receiving > >> >>>> > > > > > > > it via AMQP on Artemis 2.17.0 ends with receiving the > >> following > >> >>>> > > > > string > >> >>>> > > > > > > > as message body: > >> >>>> > > > > > > > > >> >>>> > > > > > > > "Conversion to AMQP error: Error reading in > >> simpleString, length=y > >> >>>> > > > is > >> >>>> > > > > > > > greater than readableBytes=x" > >> >>>> > > > > > > > > >> >>>> > > > > > > > Broker lists the message as Type: Default and Large: > >> True. > >> >>>> > > > > > > > > >> >>>> > > > > > > > The receiver is basically Qpid's: > >> >>>> > > > > > > > > >> >>>> > > > > > > > > >> >>>> > > > > > >> >>>> > > > > >> https://github.com/apache/qpid-proton/blob/main/python/examples/queue_browser.py > >> >>>> > > > > > > > > >> >>>> > > > > > > > The doc > >> >>>> > > > > > > > > >> >>>> > > > > > >> >>>> > > > > >> https://activemq.apache.org/components/artemis/documentation/2.17.0/large-messages.html > >> >>>> > > > > > > > states both Core and AMQP support Large Message. Is > >> this approach > >> >>>> > > > > > > > valid? > >> >>>> > > > > > > > > >> >>>> > > > > > > > Thanks > >> >>>> > > > > > > > > >> >>>> > > > > > > > > >> >>>> > > > > > > >> >>>> > > > > > > >> >>>> > > > > > > >> >>>> > > > > > -- > >> >>>> > > > > > Clebert Suconic > >> >>>> > > > > > >> >>>> > > > -- > >> >>>> > > > Clebert Suconic > >> >>>> > > > > >> >>>> > > >> >>>> > > >> >>>> > > >> >>>> > -- > >> >>>> > Clebert Suconic > >> >>>> > >> >>>> > >> >>>> > >> >>>> -- > >> >>>> Clebert Suconic > >> > > >> > -- > >> > Clebert Suconic > >> > >> > >> > >> -- > >> Clebert Suconic > >> > > -- > > Clebert Suconic > > > -- > Clebert Suconic >
