[
https://issues.apache.org/jira/browse/ARTEMIS-2336?focusedWorklogId=243994&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243994
]
ASF GitHub Bot logged work on ARTEMIS-2336:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 17/May/19 12:26
Start Date: 17/May/19 12:26
Worklog Time Spent: 10m
Work Description: franz1981 commented on pull request #2666: ARTEMIS-2336
Use zero copy to replicate journal/page/large message file
URL: https://github.com/apache/activemq-artemis/pull/2666#discussion_r285103297
##########
File path:
artemis-server/src/main/java/org/apache/activemq/artemis/core/replication/ReplicationManager.java
##########
@@ -551,49 +580,52 @@ private void
sendLargeFile(AbstractJournalStorageManager.JournalContent content,
if (!file.isOpen()) {
file.open();
}
- int size = 32 * 1024;
+ final int size = 1024 * 1024;
+ long fileSize = file.size();
int flowControlSize = 10;
int packetsSent = 0;
FlushAction action = new FlushAction();
+ long offset = 0;
+ RandomAccessFile raf;
+ FileChannel fileChannel = null;
try {
- try (FileInputStream fis = new FileInputStream(file.getJavaFile());
FileChannel channel = fis.getChannel()) {
-
- // We can afford having a single buffer here for this entire loop
- // because sendReplicatePacket will encode the packet as a
NettyBuffer
- // through ActiveMQBuffer class leaving this buffer free to be
reused on the next copy
- while (true) {
- final ByteBuf buffer =
PooledByteBufAllocator.DEFAULT.directBuffer(size, size);
- buffer.clear();
- ByteBuffer byteBuffer =
buffer.writerIndex(size).readerIndex(0).nioBuffer();
- final int bytesRead = channel.read(byteBuffer);
- int toSend = bytesRead;
- if (bytesRead > 0) {
- if (bytesRead >= maxBytesToSend) {
- toSend = (int) maxBytesToSend;
- maxBytesToSend = 0;
- } else {
- maxBytesToSend = maxBytesToSend - bytesRead;
- }
- }
- logger.debug("sending " + buffer.writerIndex() + " bytes on
file " + file.getFileName());
- // sending -1 or 0 bytes will close the file at the backup
- // We cannot simply send everything of a file through the
executor,
- // otherwise we would run out of memory.
- // so we don't use the executor here
- sendReplicatePacket(new ReplicationSyncFileMessage(content,
pageStore, id, toSend, buffer), true);
- packetsSent++;
-
- if (packetsSent % flowControlSize == 0) {
- flushReplicationStream(action);
+ raf = new RandomAccessFile(file.getJavaFile(), "r");
+ fileChannel = raf.getChannel();
+ while (true) {
+ long chunkSize = Math.min(size, fileSize - offset);
+ int toSend = (int) chunkSize;
+ if (chunkSize > 0) {
+ if (chunkSize >= maxBytesToSend) {
+ toSend = (int) maxBytesToSend;
+ maxBytesToSend = 0;
+ } else {
+ maxBytesToSend = maxBytesToSend - chunkSize;
}
- if (bytesRead == -1 || bytesRead == 0 || maxBytesToSend == 0)
- break;
}
+ logger.debug("sending " + toSend + " bytes on file " +
file.getFileName());
+ // sending -1 or 0 bytes will close the file at the backup
+ // We cannot simply send everything of a file through the executor,
+ // otherwise we would run out of memory.
+ // so we don't use the executor here
+ sendSyncFileMessage(new ReplicationSyncFileMessage(content,
pageStore, id, raf, fileChannel, offset, toSend), offset + toSend == fileSize);
+ packetsSent++;
+ offset += toSend;
+
+ if (packetsSent % flowControlSize == 0) {
+ flushReplicationStream(action);
+ }
+ if (toSend == 0 || maxBytesToSend == 0)
+ break;
}
flushReplicationStream(action);
+
+ } catch (Exception e) {
+ if (fileChannel != null)
+ fileChannel.close();
Review comment:
Yes, but just the supposed last packet need that listener, while the other
writes can just use a void promise instead
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 243994)
Time Spent: 6.5h (was: 6h 20m)
> Use zero copy to replicate journal/page/large message file
> ----------------------------------------------------------
>
> Key: ARTEMIS-2336
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2336
> Project: ActiveMQ Artemis
> Issue Type: Improvement
> Components: Broker
> Reporter: yangwei
> Priority: Major
> Time Spent: 6.5h
> Remaining Estimate: 0h
>
> Use sendfile to replicate files during backup sever bootstrap.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
