Github user caneGuy commented on the issue:
https://github.com/apache/spark/pull/18730
I mock some local test for two different api. From the simple test result,
we can see slice will not affect the performance of write bytes.Test result
below:
```
ãTest 10 chunks each with 30m for 1 loopã
Time cost with 1 loop for writeFully(): 83 ms
Time cost with 1 loop for writeWithSlice(): 76 ms
ãEndingã
ãTest 10 chunks each with 100m for 1 loopã
Time cost with 1 loop for writeFully(): 219 ms
Time cost with 1 loop for writeWithSlice(): 213 ms
ãEndingã
ãTest 10 chunks each with 30m for 10 loopã
Time cost with 10 loop for writeFully(): 982 ms
Time cost with 10 loop for writeWithSlice(): 1000 ms
ãEndingã
ãTest 10 chunks each with 100m for 10 loopã
Time cost with 10 loop for writeFully(): 3298 ms
Time cost with 10 loop for writeWithSlice(): 3454 ms
ãEndingã
ãTest 10 chunks each with 30m for 50 loopã
Time cost with 50 loop for writeFully(): 3444 ms
Time cost with 50 loop for writeWithSlice(): 3329 ms
ãEndingã
ãTest 10 chunks each with 100m for 50 loopã
Time cost with 50 loop for writeFully(): 21913 ms
Time cost with 50 loop for writeWithSlice(): 17574 ms
ãEndingã
```
Test code below:
```
test("benchmark testing") {
// scalastyle:off
val buffer100 = ByteBuffer.allocate(1024 * 1024 * 100)
val buffer30 = ByteBuffer.allocate(1024 * 1024 * 30)
testWithLoop(1, new ChunkedByteBuffer(Array.fill(10)(buffer30)), "Test
10 chunks each with 30m for 1 loop")
testWithLoop(1, new ChunkedByteBuffer(Array.fill(10)(buffer100)), "Test
10 chunks each with 100m for 1 loop")
testWithLoop(10, new ChunkedByteBuffer(Array.fill(10)(buffer30)), "Test
10 chunks each with 30m for 10 loop")
testWithLoop(10, new ChunkedByteBuffer(Array.fill(10)(buffer100)),
"Test 10 chunks each with 100m for 10 loop")
testWithLoop(50, new ChunkedByteBuffer(Array.fill(10)(buffer30)), "Test
10 chunks each with 30m for 50 loop")
testWithLoop(50, new ChunkedByteBuffer(Array.fill(10)(buffer100)),
"Test 10 chunks each with 100m for 50 loop")
}
// scalastyle:off
private def testWithLoop(loopTimes : Int, chunkedByteBuffer:
ChunkedByteBuffer, testString: String) {
System.out.println(s"ã$testStringã")
var starTime = System.currentTimeMillis()
for (i <- 1 to loopTimes) {
chunkedByteBuffer.writeFully(new
ByteArrayWritableChannel(chunkedByteBuffer.size.toInt))
}
System.out.println(s"Time cost with $loopTimes loop for
writeFully():${Utils.getUsedTimeMs(starTime)}")
starTime = System.currentTimeMillis()
for (i <- 1 to loopTimes) {
chunkedByteBuffer.writeWithSlice(new
ByteArrayWritableChannel(chunkedByteBuffer.size.toInt))
}
System.out.println(s"Time cost with $loopTimes loop for
writeWithSlice():${Utils.getUsedTimeMs(starTime)}")
System.out.println("ãEndingã")
System.out.println("")
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
