joseluisll commented on code in PR #8770:
URL: https://github.com/apache/hadoop/pull/8770#discussion_r4204242885
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/timeline/TimelineUtils.java:
##########
@@ -60,6 +73,73 @@ public class TimelineUtils {
YarnJacksonJaxbJsonProvider.configObjectMapper(mapper);
}
+ @Private
+ public static Map<String, Map<String, Object>> getCustomResourceInfo(
+ Resource resource) {
+ Map<String, Map<String, Object>> resources = new HashMap<>();
+ for (ResourceInformation information : resource.getResources()) {
+ String name = information.getName();
+ if (!ResourceInformation.MEMORY_URI.equals(name)
+ && !ResourceInformation.VCORES_URI.equals(name)
+ && information.getValue() != 0) {
+ Map<String, Object> allocation = new HashMap<>();
+ allocation.put("value", information.getValue());
Review Comment:
nit, which I should have caught on the first pass: `"value"` and `"units"`
are spelled out separately by the writer (`getCustomResourceInfo`) and the
reader (`getContainerResource`). They're part of the stored format now, so
could they be constants next to `ALLOCATED_RESOURCES_INFO` in
`ContainerMetricsConstants`? That way writer and reader can't drift apart.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/timeline/TimelineUtils.java:
##########
@@ -60,6 +73,73 @@ public class TimelineUtils {
YarnJacksonJaxbJsonProvider.configObjectMapper(mapper);
}
+ @Private
+ public static Map<String, Map<String, Object>> getCustomResourceInfo(
+ Resource resource) {
+ Map<String, Map<String, Object>> resources = new HashMap<>();
+ for (ResourceInformation information : resource.getResources()) {
+ String name = information.getName();
+ if (!ResourceInformation.MEMORY_URI.equals(name)
+ && !ResourceInformation.VCORES_URI.equals(name)
+ && information.getValue() != 0) {
+ Map<String, Object> allocation = new HashMap<>();
+ allocation.put("value", information.getValue());
+ allocation.put("units", information.getUnits());
+ resources.put(name, allocation);
+ }
+ }
+ return resources;
+ }
+
+ @Private
+ public static Resource getContainerResource(Map<String, Object> entityInfo) {
+ if (entityInfo == null) {
+ return Resource.newInstance(0, 0);
+ }
+ long memory = ((Number) entityInfo.getOrDefault(
+ ContainerMetricsConstants.ALLOCATED_MEMORY_INFO, 0L)).longValue();
+ int vcores = ((Number) entityInfo.getOrDefault(
+ ContainerMetricsConstants.ALLOCATED_VCORE_INFO, 0)).intValue();
+ Resource resource = Resource.newInstance(memory, vcores);
+ Object allocationInfo = entityInfo.get(
+ ContainerMetricsConstants.ALLOCATED_RESOURCES_INFO);
+ if (allocationInfo instanceof Map) {
+ Map<?, ?> allocations = (Map<?, ?>) allocationInfo;
+ for (Map.Entry<?, ?> entry : allocations.entrySet()) {
+ try {
+ String name = (String) entry.getKey();
+ if (ResourceInformation.MEMORY_URI.equals(name)
+ || ResourceInformation.VCORES_URI.equals(name)) {
+ continue;
+ }
+ if (!ResourceUtils.getResourceTypes().containsKey(name)) {
+ LOG.debug("Skipping unknown resource type {} in container
history", name);
+ continue;
+ }
+ Map<?, ?> allocation = (Map<?, ?>) entry.getValue();
+ Number storedValue = (Number) allocation.get("value");
+ if (storedValue instanceof Float || storedValue instanceof Double) {
+ throw new IllegalArgumentException("Floating-point resource
value");
+ }
+ long value = storedValue instanceof Integer || storedValue
instanceof Long
+ ? storedValue.longValue()
+ : new BigDecimal(storedValue.toString()).longValueExact();
+ String units = (String) allocation.get("units");
+ String defaultUnits =
resource.getResourceInformation(name).getUnits();
+ resource.setResourceValue(name,
+ UnitsConversionUtil.convert(units, defaultUnits, value));
+ } catch (ClassCastException | NullPointerException
Review Comment:
optional: Following up on my earlier suggestion: listing the four exceptions
covers today's code paths. A single `catch (RuntimeException e)` would also
cover anything not listed, such as a `ResourceNotFoundException` from
`getResourceInformation`, which is what a per-entry skip guard is for. Feel
free to leave as is.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/timeline/TimelineUtils.java:
##########
@@ -60,6 +73,73 @@ public class TimelineUtils {
YarnJacksonJaxbJsonProvider.configObjectMapper(mapper);
}
+ @Private
+ public static Map<String, Map<String, Object>> getCustomResourceInfo(
Review Comment:
nit, also from the first pass: A short Javadoc on these two public helpers
would help. On `getCustomResourceInfo` it could show the map shape (`name ->
{value, units}`, leaving out memory, vcores and zero values). On
`getContainerResource` it could say that unknown or malformed entries are
skipped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
