Looks like our REST endpoints return malformed JSON for any DAO that
includes a Map. That includes:
* the resourceSecondsMap and preemptedResourceSecondsMap entries in all
the GET /apps/* endpoints,
* the operationsInfo entry in the GET /scheduler endpoint for capacity
scheduler,
* the local_resources, environment, and acls entries in the POST /apps
endpoint, and
* the labelsToNodes entry in the GET /label-mappings endpoint.
The issue is that each entry in the map is included with a duplicate key
("entry"). Some JSON parsers will choke on the error, and some will
quietly drop the duplicates. I've filed YARN-7505 to address the issue.
The solution is to replace the Jersey JSON serializer with the Jackson
JSON serializer. This change fixes the issue, but it changes the
structure of the resulting JSON. For example, without YARN-7505,
hitting /apps might yield JSON that contains something like:
"resourceSecondsMap":{
"entry":{"key":"memory-mb","value":"11225"},
"entry":{"key":"vcores","value":"5"}
"entry":{"key":"test","value":"0"}
"entry":{"key":"test2","value":"0"}
}
With YARN-7505, we get:
"resourceSecondsMap": {
"test2":0,
"test":0,
"memory-mb":11225,
"vcores":5
}
The first example is obviously broken, so the second one is clearly
better, but it's structurally different.
For the GET /label-mappings endpoint, the keys of the map also have to
be changed to simple strings because JSON doesn't allow for complex map
keys. So this:
"labelsToNodes":{
"entry":{
"key":{"name":"label1","exclusivity":"true"},
"value":{"nodes":"localhost:63261"}
}
}
becomes this:
"labelsToNodes":{
"label1":{
"nodes":["dhcp-10-16-0-181.pa.cloudera.com:63261"]
}
}
The first one sucks and is invalid, but changing to the second one will
break clients that are parsing the first one, especially if they're
expecting to get the label exclusivity from this endpoint.
Before I try to get YARN-7505 committed, I want to give the community a
chance to voice any concerns about the change. It's too late to get
into 3.0.0, so we'd be looking at 3.0.1 and 3.1.0.
Feel free to comment here or on the JIRA directly.
Thanks,
Daniel
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
