[
https://issues.apache.org/jira/browse/GOBBLIN-1728?focusedWorklogId=818926&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818926
]
ASF GitHub Bot logged work on GOBBLIN-1728:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 20/Oct/22 19:51
Start Date: 20/Oct/22 19:51
Worklog Time Spent: 10m
Work Description: homatthew commented on code in PR #3586:
URL: https://github.com/apache/gobblin/pull/3586#discussion_r1001052159
##########
gobblin-yarn/src/main/java/org/apache/gobblin/yarn/YarnService.java:
##########
@@ -845,8 +877,6 @@ public void onContainersAllocated(List<Container>
containers) {
instanceName = null;
}
}
- allocatedContainerCountMap.put(containerHelixTag,
Review Comment:
I re-read the API and I think you're correct.
https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-client/apidocs/org/apache/hadoop/yarn/client/api/async/AMRMClientAsync.CallbackHandler.html
Because this method is triggered by heartbeat and this method is called
after completion, maybe any implementation would work here since it's not even
concurrent lol. All writes to this map actually seem single threaded
Issue Time Tracking
-------------------
Worklog Id: (was: 818926)
Time Spent: 3h (was: 2h 50m)
> Yarn Service requests too many containers due to improper calculation
> ---------------------------------------------------------------------
>
> Key: GOBBLIN-1728
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1728
> Project: Apache Gobblin
> Issue Type: New Feature
> Reporter: Matthew Ho
> Priority: Major
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Yarn Service is responsible for calculating the number of instances based on
> the helix tasks. Yarn service tracks the number of instances by asking Yarn
> for the number of resource requests and the number of allocated containers.
>
> It uses this count to determine if it should ask for more containers or
> shrink the number of containers. This calculation is currently done
> improperly and we continue to request containers when we have enough
> requested.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
