[
https://issues.apache.org/jira/browse/ARTEMIS-4024?focusedWorklogId=814098&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-814098
]
ASF GitHub Bot logged work on ARTEMIS-4024:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 05/Oct/22 20:06
Start Date: 05/Oct/22 20:06
Worklog Time Spent: 10m
Work Description: clebertsuconic commented on code in PR #4240:
URL: https://github.com/apache/activemq-artemis/pull/4240#discussion_r985901312
##########
artemis-protocols/artemis-openwire-protocol/src/main/java/org/apache/activemq/artemis/core/protocol/openwire/OpenWireConnection.java:
##########
@@ -558,6 +564,21 @@ public void physicalSend(Command command) throws
IOException {
}
+ private void chunkSend(final ByteSequence bytes, final int bufferSize,
final int maxPacketSize) {
+ if (logger.isTraceEnabled()) {
+ logger.trace("Sending a big packet sized as {} with smaller packets
of {}", bufferSize, maxPacketSize);
+ }
+ for (int posBuffer = 0; posBuffer < bufferSize; posBuffer +=
maxPacketSize) {
+ int chunkSize = Math.min(bufferSize - posBuffer, maxPacketSize);
+ if (logger.isTraceEnabled()) {
+ logger.trace("Sending a partial packet of {} bytes, starting at
{}", chunkSize, posBuffer);
+ }
+ final ActiveMQBuffer buffer =
transportConnection.createTransportBuffer(chunkSize);
Review Comment:
I'm splitting a big buffer chunk into smaller buffers
so, if you have a 200MB message, I'm calling multiple writes instead of just
one mega-write
##########
artemis-protocols/artemis-openwire-protocol/src/main/java/org/apache/activemq/artemis/core/protocol/openwire/OpenWireConnection.java:
##########
@@ -558,6 +564,21 @@ public void physicalSend(Command command) throws
IOException {
}
+ private void chunkSend(final ByteSequence bytes, final int bufferSize,
final int maxPacketSize) {
+ if (logger.isTraceEnabled()) {
+ logger.trace("Sending a big packet sized as {} with smaller packets
of {}", bufferSize, maxPacketSize);
+ }
+ for (int posBuffer = 0; posBuffer < bufferSize; posBuffer +=
maxPacketSize) {
+ int chunkSize = Math.min(bufferSize - posBuffer, maxPacketSize);
+ if (logger.isTraceEnabled()) {
+ logger.trace("Sending a partial packet of {} bytes, starting at
{}", chunkSize, posBuffer);
+ }
+ final ActiveMQBuffer buffer =
transportConnection.createTransportBuffer(chunkSize);
Review Comment:
ah... no.. it can't be reused.... netty will be holding it down the
pipeline...
##########
artemis-protocols/artemis-openwire-protocol/src/main/java/org/apache/activemq/artemis/core/protocol/openwire/OpenWireConnection.java:
##########
@@ -558,6 +564,21 @@ public void physicalSend(Command command) throws
IOException {
}
+ private void chunkSend(final ByteSequence bytes, final int bufferSize,
final int maxPacketSize) {
+ if (logger.isTraceEnabled()) {
+ logger.trace("Sending a big packet sized as {} with smaller packets
of {}", bufferSize, maxPacketSize);
+ }
+ for (int posBuffer = 0; posBuffer < bufferSize; posBuffer +=
maxPacketSize) {
+ int chunkSize = Math.min(bufferSize - posBuffer, maxPacketSize);
+ if (logger.isTraceEnabled()) {
+ logger.trace("Sending a partial packet of {} bytes, starting at
{}", chunkSize, posBuffer);
+ }
+ final ActiveMQBuffer buffer =
transportConnection.createTransportBuffer(chunkSize);
Review Comment:
although I'm calling flush... but I would prefer to keep it through the pool
like it's always done.
if I wanted to reuse it.. I would have to increase the the counter with a
refUp here.. I would prefer to not deal with that.
##########
artemis-protocols/artemis-openwire-protocol/src/main/java/org/apache/activemq/artemis/core/protocol/openwire/OpenWireConnection.java:
##########
@@ -558,6 +564,21 @@ public void physicalSend(Command command) throws
IOException {
}
+ private void chunkSend(final ByteSequence bytes, final int bufferSize,
final int maxPacketSize) {
+ if (logger.isTraceEnabled()) {
+ logger.trace("Sending a big packet sized as {} with smaller packets
of {}", bufferSize, maxPacketSize);
+ }
+ for (int posBuffer = 0; posBuffer < bufferSize; posBuffer +=
maxPacketSize) {
+ int chunkSize = Math.min(bufferSize - posBuffer, maxPacketSize);
+ if (logger.isTraceEnabled()) {
+ logger.trace("Sending a partial packet of {} bytes, starting at
{}", chunkSize, posBuffer);
+ }
+ final ActiveMQBuffer buffer =
transportConnection.createTransportBuffer(chunkSize);
Review Comment:
I tried to make a change with the same buffer, but I got some refCounts
issues... I would prefer to keep the way it is now.
Issue Time Tracking
-------------------
Worklog Id: (was: 814098)
Time Spent: 40m (was: 0.5h)
> Avoid excessive NativeMemory allocation when sending OpenWire Multi mega
> sized messages in openwire
> ---------------------------------------------------------------------------------------------------
>
> Key: ARTEMIS-4024
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4024
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Reporter: Clebert Suconic
> Assignee: Robbie Gemmell
> Priority: Major
> Fix For: 2.27.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> when sending a large message in openwire, we will read the entire file on the
> memory, make the conversion from core, and send it on net
> throughOpenWireProtocolManager::sendPhisical.
> Such allocation should be limited and be sent in chunks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
