wy96f commented on a change in pull request #2845: ARTEMIS-2336 Use zero copy
to replicate journal/page/large message file (AGAIN)
URL: https://github.com/apache/activemq-artemis/pull/2845#discussion_r328099516
##########
File path:
artemis-server/src/main/java/org/apache/activemq/artemis/core/replication/ReplicationManager.java
##########
@@ -560,49 +590,52 @@ private void
sendLargeFile(AbstractJournalStorageManager.JournalContent content,
if (!file.isOpen()) {
file.open();
}
- int size = 32 * 1024;
+ final int size = 1024 * 1024;
+ long fileSize = file.size();
int flowControlSize = 10;
int packetsSent = 0;
FlushAction action = new FlushAction();
+ long offset = 0;
+ RandomAccessFile raf = null;
+ FileChannel fileChannel = null;
try {
- try (FileInputStream fis = new FileInputStream(file.getJavaFile());
FileChannel channel = fis.getChannel()) {
-
- // We can afford having a single buffer here for this entire loop
- // because sendReplicatePacket will encode the packet as a
NettyBuffer
- // through ActiveMQBuffer class leaving this buffer free to be
reused on the next copy
- while (true) {
- final ByteBuf buffer =
PooledByteBufAllocator.DEFAULT.directBuffer(size, size);
- buffer.clear();
- ByteBuffer byteBuffer =
buffer.writerIndex(size).readerIndex(0).nioBuffer();
- final int bytesRead = channel.read(byteBuffer);
- int toSend = bytesRead;
- if (bytesRead > 0) {
- if (bytesRead >= maxBytesToSend) {
- toSend = (int) maxBytesToSend;
- maxBytesToSend = 0;
- } else {
- maxBytesToSend = maxBytesToSend - bytesRead;
- }
- }
- logger.debug("sending " + buffer.writerIndex() + " bytes on
file " + file.getFileName());
- // sending -1 or 0 bytes will close the file at the backup
- // We cannot simply send everything of a file through the
executor,
- // otherwise we would run out of memory.
- // so we don't use the executor here
- sendReplicatePacket(new ReplicationSyncFileMessage(content,
pageStore, id, toSend, buffer), true);
- packetsSent++;
-
- if (packetsSent % flowControlSize == 0) {
- flushReplicationStream(action);
+ raf = new RandomAccessFile(file.getJavaFile(), "r");
Review comment:
AFAIK 0 bytes file will be created in some cases:
1. In the beginning of `startReplication`, 0 bytes page file will be
created. If backup fails and connects to live again, 0 bytes page file will be
sent.
2. When all pages are consumed, `PageCursorProviderImpl::cleanupComplete`
will be called generating 0 bytes page file, and might be sent to backup later.
That works, but we'll add a new send method only used here? How about not
opening raf and `new ReplicationSyncFileMessage(content, pageStore, id, null,
null, offset, toSend)` when file size is 0 so we don't take care to release it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
