[
https://issues.apache.org/jira/browse/YARN-11897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18056902#comment-18056902
]
ASF GitHub Bot commented on YARN-11897:
---------------------------------------
Hean-Chhinling commented on code in PR #8227:
URL: https://github.com/apache/hadoop/pull/8227#discussion_r2774062999
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/jsonprovider/NMJsonProvider.java:
##########
@@ -0,0 +1,67 @@
+package org.apache.hadoop.yarn.server.nodemanager.webapp.jsonprovider;
+
+import
org.apache.hadoop.yarn.server.nodemanager.webapp.dao.NMDeviceResourceInfo;
+import
org.apache.hadoop.yarn.server.nodemanager.webapp.dao.gpu.NMGpuResourceInfo;
+import org.apache.hadoop.yarn.server.webapp.dao.ContainerLogsInfoes;
+import org.eclipse.persistence.jaxb.MarshallerProperties;
+import org.eclipse.persistence.jaxb.rs.MOXyJsonProvider;
+
+import javax.ws.rs.Consumes;
+import javax.ws.rs.Produces;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.MultivaluedMap;
+import javax.ws.rs.ext.Provider;
+import javax.xml.bind.JAXBException;
+import javax.xml.bind.Marshaller;
+import javax.xml.bind.Unmarshaller;
+import java.lang.annotation.Annotation;
+import java.lang.reflect.Type;
+
+/**
+ * MOXy JSON provider for NodeManager WebService.
+ *
+ * <p>This class configures a MOXy JSON provider for the NodeManager REST API
endpoints.
+ * The endpoints should be able to provide two types of JSON responses:</p>
+ * <ul>
+ * <li>
+ * <b>Wrapped classes</b> – classes whose JSON representation includes a
root wrapper element.
+ * </li>
+ * <li>
+ * <b>Unwrapped classes</b> – classes whose JSON representation omits a
root wrapper element.
+ * </li>
+ * </ul>
+ *
+ * <p>This behaviour can be configured by the
MarshallerProperties.JSON_INCLUDE_ROOT property.
+ *
+ * By default NodeManager REST API endpoints should include the root wrapper
element in the
+ * responses, however there are some exceptions (e.g. ContainerLogsInfoes
class) which
+ * was introduced to provide backward-compatibility with the Jersey 1 response
format.</p>
+ */
+@Provider
+@Produces(MediaType.APPLICATION_JSON)
+@Consumes(MediaType.APPLICATION_JSON)
+public class NMJsonProvider extends MOXyJsonProvider {
+
+ private boolean isRootElementNeeded (Class<?> type) {
+ return !type.equals(ContainerLogsInfoes.class)
+ && !type.equals(NMGpuResourceInfo.class)
+ && !type.equals(NMDeviceResourceInfo.class);
+ }
+
+ @Override
+ protected void preReadFrom(Class<Object> type, Type genericType,
+ Annotation[] annotations, MediaType mediaType,
+ MultivaluedMap<String, String> httpHeaders,
+ Unmarshaller unmarshaller) throws JAXBException {
+ unmarshaller.setProperty(MarshallerProperties.JSON_INCLUDE_ROOT,
isRootElementNeeded(type));
+ }
+
+ @Override
+ protected void preWriteTo(Object object, Class<?> type, Type genericType,
+ Annotation[] annotations, MediaType mediaType,
+ MultivaluedMap<String, Object> httpHeaders,
Marshaller marshaller)
+ throws JAXBException {
+
marshaller.setProperty(MarshallerProperties.JSON_MARSHAL_EMPTY_COLLECTIONS,
false);
+ marshaller.setProperty(MarshallerProperties.JSON_INCLUDE_ROOT,
isRootElementNeeded(type));
Review Comment:
I tested in both cases with/without adding the JSON_REDUCE_ANY_ARRAYS
property.
And surprisingly in both cases the containers log is not available when the
application is in running state.
With JSON_REDUCE_ANY_ARRAYS:
<img width="2560" height="1440" alt="jsonreduce-no-container-log"
src="https://github.com/user-attachments/assets/bdc651c3-0f20-4524-b270-48b491f46285";
/>
Without JSON_REDUCE_ANY_ARRAYS:
<img width="2560" height="1440" alt="without-still-nolog"
src="https://github.com/user-attachments/assets/62d86bee-b8c8-4b7a-85bc-01ad02ccea49";
/>
I did the following:
- Run Mapreduce job
- While Application is running
- Access container log YARN UI2 -> Application -> logs
I will check with the NM endpoints that give this logs probably there is a
mismatch in the structure.
> NodeManager REST API backward compatibility with Jersey1
> --------------------------------------------------------
>
> Key: YARN-11897
> URL: https://issues.apache.org/jira/browse/YARN-11897
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
> Affects Versions: 3.5.0
> Reporter: Peter Szucs
> Assignee: chhinlinghean
> Priority: Major
> Labels: pull-request-available
>
> Jersey 2 upgrade broke NM REST API, so nodes applications page is not
> rendering in yarn UI v2 and also logs CLI cannot retrieve container logs.
> To fix this we need to:
> * introduce moxy in NM similar to YARN-11874
> * "/containers/\{containerid}/logs" endpoint returns a list, which will
> always be generated as a JSON array with Jersey 2. In Jersey 1 it returned a
> JSON object in case the list had a single element, and logs CLI expects a
> JSON object as well when calling this endpoint. That should be changed to
> keep backward compatibility. With returning a wrapper object instead of a
> list we could achieve the same behaviour.
> * Fix unit tests
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
