Github user etiennecarriere commented on a diff in the pull request:
https://github.com/apache/flink/pull/6149#discussion_r195687859
--- Diff:
flink-core/src/test/java/org/apache/flink/core/fs/LimitedConnectionsFileSystemTest.java
---
@@ -122,6 +122,42 @@ public void testLimitingInputStreams() throws
Exception {
}
}
+ @Test
+ public void testLimitingRateLimitingStream() throws Exception {
+ final LimitedConnectionsFileSystem limitedFs = new
LimitedConnectionsFileSystem(
+ LocalFileSystem.getSharedInstance(),
+ Integer.MAX_VALUE,
+ Integer.MAX_VALUE,
+ Integer.MAX_VALUE,
+ 0,
+ 0,
+ 10000, // Limit write to 10 kbytes/s
+ 10000); // Limit read to 10 kbytes/s
+ File file = tempFolder.newFile();
+ Path path = new Path(file.toURI());
+ long durationWrite = System.currentTimeMillis();
+ try (FSDataOutputStream stream = limitedFs.create(path,
WriteMode.OVERWRITE)) {
+ final Random rnd = new Random();
+ final byte[] data = new byte[100];
+ for (int i = 0; i < (1000 + 10); i++) {
+ rnd.nextBytes(data);
+ stream.write(data);
+ }
+ }
+ durationWrite = System.currentTimeMillis() - durationWrite;
+
+ long durationRead = System.currentTimeMillis();
+ final byte[] data = new byte[100];
+ try (FSDataInputStream stream = limitedFs.open(path)) {
+ //noinspection StatementWithEmptyBody
+ while (stream.read(data) != -1) {}
+ }
+ durationRead = System.currentTimeMillis() - durationRead;
+ file.delete();
+ assertTrue(durationWrite > 10000);
+ assertTrue(durationRead > 8000); // Less stability with read
limiter than write
--- End diff --
My initial need is only for the output Ratelimiter : we want to be able to
limit the rate of sending checkpoint to s3/hadoop because it is in concurrence
with flink exchange of data between task manager (we are on an infrastructure
with only gigabit)
I implemented input and output because some people could also want to limit
input . (even if I don't see real use cases for input ratelimiter)
I implemented separated limiter because on a full duplex network it is
possible you want to privilege one part of the traffic but I am also ok to have
a shared ratelimiter
---
