Tao Yang created YARN-6259:
------------------------------
Summary: Support pagination and optimize data transfer with
zero-copy approach for containerlogs REST API in NMWebServices
Key: YARN-6259
URL: https://issues.apache.org/jira/browse/YARN-6259
Project: Hadoop YARN
Issue Type: Improvement
Components: nodemanager
Affects Versions: 2.8.1
Reporter: Tao Yang
Assignee: Tao Yang
Currently containerlogs REST API in NMWebServices will read and send the entire
content of container logs. Most of container logs are large and it's useful to
support pagination.
* Add pagesize and pageindex parameters for containerlogs REST API
{code}
URL: http://<nm_address>/ws/v1/node/containerlogs/<container_id>/<file_name>
QueryParams:
pagesize - max bytes of one page , default 1MB
pageindex - index of required page, default 0, can be nagative(set -1 will
get the last page content)
{code}
* Add containerlogs-info REST API since sometimes we need to know the
totalSize/pageSize/pageCount info of log
{code}
URL:
http://<nm_address>/ws/v1/node/containerlogs-info/<container_id>/<file_name>
QueryParams:
pagesize - max bytes of one page , default 1MB
Response example:
{"logInfo":{"totalSize":2497280,"pageSize":1048576,"pageCount":3}}
{code}
Moreover, the data transfer pipeline (disk --> read buffer --> NM buffer -->
socket buffer) can be optimized to pipeline(disk --> read buffer --> socket
buffer) with zero-copy approach.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
