[
https://issues.apache.org/jira/browse/HDFS-14403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812543#comment-16812543
]
Yiqun Lin commented on HDFS-14403:
----------------------------------
Thanks for working on this, [~cgregori] and [~xkrogen]. I took some time to
read the design and implementation in patch. Mainly two comments:
Can we refactor LockCostProvider into COMMON not in HDFS module? Based on the
current approach, we maintain a new mapping in FSN lock but it's actual used by
FCQ not for HDFS itself. I mean for HDFS, it can do more easier, pass the rpc
lock time into FCQ. Then FCQ implement plugin cost provider. Isn't this a
better way?
{quote}
In DecayRpcScheduler, recordOperation is called in getPriorityLevel (once
before each call) and getAndResetCost is called upon each decay sweep (every
few seconds, typically).
{quote}
We can make getAndResetCost updated more faster, don't must each decay
interval. The original way should still make sense.
The improved way:
{code}
/**
- * Update the scheduleCache to match current conditions in callCounts.
+ * Queries the CostProvider for the cumulative cost since the last call for
+ * each user that currently has a record in callCosts.
+ */
+ void updateFromCostProvider(Object identity) {
+ long totalDecayedCostIncrease = 0;
+ long totalRawCostIncrease = 0;
+
+ String identity = entry.getKey().toString();
+ long newCost = costProvider.getAndResetCost(identity);
+
+ AtomicLong decayedCost = entry.getValue().get(0);
+ decayedCost.getAndAdd(newCost);
+ AtomicLong rawCost = entry.getValue().get(1);
+ rawCost.getAndAdd(newCost);
+
+ totalDecayedCostIncrease += newCost;
+ totalRawCostIncrease += newCost;
+
+ totalDecayedCallCost.getAndAdd(totalDecayedCostIncrease);
+ totalRawCallCost.getAndAdd(totalRawCostIncrease);
+ }
{code}
Do the update operation when each request is coming. Since every time we only
updated for specified user, the cost of this should be okay.
{code}
.. private int cachedOrComputedPriorityLevel(Object identity) {
try {
updateFromCostProvider(identity);
,,,
{code}
> Cost-Based RPC FairCallQueue
> ----------------------------
>
> Key: HDFS-14403
> URL: https://issues.apache.org/jira/browse/HDFS-14403
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ipc, namenode
> Reporter: Erik Krogen
> Assignee: Christopher Gregorian
> Priority: Major
> Labels: qos, rpc
> Attachments: CostBasedFairCallQueueDesign_v0.pdf, HDFS-14403.001.patch
>
>
> HADOOP-15016 initially described extensions to the Hadoop FairCallQueue
> encompassing both cost-based analysis of incoming RPCs, as well as support
> for reservations of RPC capacity for system/platform users. This JIRA intends
> to track the former, as HADOOP-15016 was repurposed to more specifically
> focus on the reservation portion of the work.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
