[
https://issues.apache.org/jira/browse/YARN-6259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130303#comment-16130303
]
Tao Yang commented on YARN-6259:
--------------------------------
Thanks [~djp] for your suggestions. It makes sense to me. I have created
YARN-7037 to handle performance improvement and the patch of this issue will be
updated later. I noticed that there are many differences between 2.8 and
2.9/trunk, 2.9/trunk supports getting head or tail part of log file. It's close
to our requirements but still not enough to support pagination.
> Support pagination and optimize data transfer with zero-copy approach for
> containerlogs REST API in NMWebServices
> -----------------------------------------------------------------------------------------------------------------
>
> Key: YARN-6259
> URL: https://issues.apache.org/jira/browse/YARN-6259
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
> Affects Versions: 2.8.1
> Reporter: Tao Yang
> Assignee: Tao Yang
> Attachments: YARN-6259.001.patch
>
>
> Currently containerlogs REST API in NMWebServices will read and send the
> entire content of container logs. Most of container logs are large and it's
> useful to support pagination.
> * Add pagesize and pageindex parameters for containerlogs REST API
> {code}
> URL: http://<nm_address>/ws/v1/node/containerlogs/<container_id>/<file_name>
> QueryParams:
> pagesize - max bytes of one page , default 1MB
> pageindex - index of required page, default 0, can be nagative(set -1 will
> get the last page content)
> {code}
> * Add containerlogs-info REST API since sometimes we need to know the
> totalSize/pageSize/pageCount info of log
> {code}
> URL:
> http://<nm_address>/ws/v1/node/containerlogs-info/<container_id>/<file_name>
> QueryParams:
> pagesize - max bytes of one page , default 1MB
> Response example:
> {"logInfo":{"totalSize":2497280,"pageSize":1048576,"pageCount":3}}
> {code}
> Moreover, the data transfer pipeline (disk --> read buffer --> NM buffer -->
> socket buffer) can be optimized to pipeline(disk --> read buffer --> socket
> buffer) with zero-copy approach.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
