Greetings All,
I have a Netty client that sends two 5GB files to a Netty server. The
Netty client sends one 5GB file directly to the Netty Server through a 10Gb
link. The Netty Client sends the other 5GB file through two Netty Proxy
Servers to the Netty Server using 1Gb links. The client, proxy servers
and servers are directly connected to each other and do not have background
traffic. *I would like to know why the Netty Client only pushes KiloBytes
of data through the socket instead of the specified 100MB? *The network is
more than capable of pushing MB of data. In the code I pass in a
NioChunkedInput to the ChunkedWriteHandler that fetches 100MB chunks of
data to send through the socket, but only KiloBytes and a max of 1 or 2MB
of data is actually pushed through the NioSocketChannel at a time. Each of
the Channels were bootstrapped with a socket send buffer size of 100MB,
socket receive buffer size of 100MB and TCP_NODELAY was set to true to
disable Nagle's algorithm. I even set the NioEventLoopGroup's I/O ratio to
100 to spend 100% of the time processing I/O events. Each channel shares
the same NioEventLoopGroup. in addition the Netty proxy servers and the
Netty Server channels were configured with the same configuration. Please
see the below log file that shows amount of data being pushed through the
NioSocket. Note, I collected metrics from the
*io.netty.channel.socket.nio.NioSocketChannel
doWrite method, please see a portion of the log file below containing these
metrics* In addition*, *when the client sends the two files through both
paths as specified below the throughput of the transfer for both Paths
decreases by approximately 50% as opposed to a Single One 5GB file transfer
through only one path. For Example the *throughput rendered for each path
when sending one 5GB down path 1 and one 5GB down path 2 is: *
Path 1: Netty Client (WS5), Netty Server (WS7) -- 345 Mb/s
Path 2: Netty Client (WS5), Netty Proxy Server 1 (WS11), Netty Proxy
Server 2 (WS12), Netty Server (WS7) -- 318 Mb/s
*Throughput rendered from the client just sending one 5GB file down path 1
and nothing down path 2 is:*
Path 1: Netty Client (WS5), Netty Server (WS7) -- 842 Mb/s
*Throughput rendered from the client just sending one 5GB file down path 2
and nothing down path 1 is: *
Path 2: Netty Client (WS5), Netty Proxy Server 1 (WS11), Netty Proxy Server
2 (WS12), Netty Server (WS7) -- 724 Mb/s
*My question is why does the throughput decrease for each path when sending
data down 2 disjoint paths that share the same source and destination as
opposed to sending data down one single path? Am I missing something that
Netty is doing behind the scenes?*
*Netty Client = Machine WS5*
*Netty Server = Machine WS7*
*Netty Proxy Servers = Machines WS11 & WS12*
Path 1: WS5,WS7
----Channel: *Thread ID: 15*
Path 2 WS5,WS11,WS12,WS7
----Channel: *Thread ID: 12*
*Portion of the log file showing the number of bytes being transferred
through the socket at a given time*
nioEventLoopGroup-3-2, *Thread ID: 15 *NioSocketChannel:doWrite(), (Call #:
17, *Attempting to Write 104857600 Bytes* to the Socket, Remaining bytes in
Buffer = 104857600, Buffer Capacity = 104857600, Number of Flushed Msgs in
the ChannelOutboundBuffer = 1, current Time: 1494460673317
---- Wrote: 101360 Bytes to the Socket, remaining Bytes to Write = 104756240
---- Wrote: 0 Bytes to the Socket, remaining Bytes to Write = 104756240
**T*otal Bytes Wrote to Socket = 101360 Bytes,* Start Time 1494460673317,
The endTime = 1494460673317, Elapsed Time = 0, Total Elapsed Time Since
First Write: 509 ms
nioEventLoopGroup-2-2, *Thread ID: 12 *NioSocketChannel:doWrite(), (Call #:
17, *Attempting to Write 104857600 Bytes* to the Socket, Remaining bytes in
Buffer = 104857600, Buffer Capacity = 104857600, Number of Flushed Msgs in
the ChannelOutboundBuffer = 1, current Time: 1494460673334
---- Wrote: 101360 Bytes to the Socket, remaining Bytes to Write = 104756240
---- Wrote: 0 Bytes to the Socket, remaining Bytes to Write = 104756240
***Total Bytes Wrote to Socket = 101360 Bytes,* Start Time 1494460673334,
The endTime = 1494460673334, Elapsed Time = 0, Total Elapsed Time Since
First Write: 529 ms
*In addition, each machine running the client, proxy servers and servers
have the following TCP Setting: *
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 87389 67108864
#
net.core.rmem_max = 67108864
net.core.wmem_max = 67108864
*FileSenderHandler.java - **Sends the file header info - File Name, offset,
length and then the Actual File Contents*
@Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
try {
String fileRequest = "ftp WS5/home/5GB_File1.dat
WS7/tmp/5GB_File1_Copy.dat";
//Source File to send / transfer to the Destination Node
String theSrcFilePath = "/home/5GB_File1.dat";
//File Name to write on the destination node, once the file is received
String theDestFilePath = "/tmp/5GB_File1_Copy.dat";
//Get the source file to send
File theFile = new File(theSrcFilePath);
FileChannel theFileChannel = new RandomAccessFile(theFile, "r").getChannel
();
//Get the length of the file
long fileLength = theFileChannel.size();
//Get the offset
long offSet = 0;
//Copy the offset to the ByteBuf
ByteBuf offSetBuf = Unpooled.copyLong(offSet);
//Copy the file length to the ByteBuf
ByteBuf fileLengthBuf = Unpooled.copyLong(fileLength);
//Get the Destination Filename (including the file path) in Bytes
byte[] theDestFilePathInBytes = theDestFilePath.getBytes();
//Get the length of theFilePath
int theDestSize = theDestFilePathInBytes.length;
//Copy the Dest File Path length to the ByteBuf
ByteBuf theDestSizeBuf = Unpooled.copyInt(theDestSize);
//Copy the theDestFilePathInBytes to the Byte Buf
ByteBuf theDestFileBuf = Unpooled.copiedBuffer(theDestFilePathInBytes);
//Send the file Headers: FileName Length, the FileName, the Offset and the
file length
ctx.write(theDestSizeBuf);
ctx.write(theDestFileBuf);
ctx.write(offSetBuf);
ctx.write(fileLengthBuf);
ctx.flush();
//Send the 5GB File in 100MB chunks as specified by the following chunk
size (1024*1024*100)
ctx.write(new ChunkedNioFile(theFileChannel, offSet, fileLength, 1024 *
1024 * 100)); //Send File in 100MB Chunks
ctx.flush();
}catch(Exception e){
System.err.printf("FileSenderHandler: Channel Active: Error: "+e.getMessage
());
e.printStackTrace();
}
} //End channelActive
--
You received this message because you are subscribed to the Google Groups
"Netty discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/netty/36f9e439-cf16-4413-8497-ab16f49820c8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
