[
https://issues.apache.org/jira/browse/HDFS-9756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lin Yiqun updated HDFS-9756:
----------------------------
Status: Patch Available (was: Open)
Attach a initial patch. The patch make the period value be configurable. I
found a mistake in the existed test {{testThrottler}} when I plan to add my
testcase to this method. In this method, it use the variable {{totalBytes}} to
calculate the ended bandwidth. But this variable set to 0 instead of
TOTAL_BYTES.
{code}
@Test
public void testThrottler() throws IOException {
Configuration conf = new HdfsConfiguration();
FileSystem.setDefaultUri(conf, "hdfs://localhost:0");
long bandwidthPerSec = 1024*1024L;
final long TOTAL_BYTES =6*bandwidthPerSec;
long bytesToSend = TOTAL_BYTES;
long start = Time.monotonicNow();
DataTransferThrottler throttler = new
DataTransferThrottler(bandwidthPerSec);
long totalBytes = 0L;
long bytesSent = 1024*512L; // 0.5MB
throttler.throttle(bytesSent);
bytesToSend -= bytesSent;
bytesSent = 1024*768L; // 0.75MB
throttler.throttle(bytesSent);
bytesToSend -= bytesSent;
try {
Thread.sleep(1000);
} catch (InterruptedException ignored) {}
throttler.throttle(bytesToSend);
long end = Time.monotonicNow();
assertTrue(totalBytes*1000/(end-start)<=bandwidthPerSec);
}
{code}
So {{totalBytes*1000/(end-start)}} will be always 0 and the comparation will be
always true. In this patch, I also fix this error.
> Hard-Code value in DataTransferThrottler
> ----------------------------------------
>
> Key: HDFS-9756
> URL: https://issues.apache.org/jira/browse/HDFS-9756
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.7.1
> Reporter: Lin Yiqun
> Assignee: Lin Yiqun
>
> In DataTransferThrottler, the period time is hard-code for 500 ms. Even
> though it has other construction method,
> {code}
> /**
> * Constructor
> * @param period in milliseconds. Bandwidth is enforced over this
> * period.
> * @param bandwidthPerSec bandwidth allowed in bytes per second.
> */
> public DataTransferThrottler(long period, long bandwidthPerSec) {
> this.curPeriodStart = monotonicNow();
> this.period = period;
> this.curReserve = this.bytesPerPeriod = bandwidthPerSec*period/1000;
> this.periodExtension = period*3;
> }
> {code}
> but it was only invoked by this method
> {code}
> public DataTransferThrottler(long bandwidthPerSec) {
> this(500, bandwidthPerSec); // by default throttling period is 500ms
> }
> {code}
> So the period is a hard-code. This value can also influence the
> data-transfering. If period time value set small, the number of times to
> waitting for next period will be increased and the total waitting-time also
> be increased. So the average bandwidth will be decreased.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
