Tao Yang created YARN-6259: ------------------------------ Summary: Support pagination and optimize data transfer with zero-copy approach for containerlogs REST API in NMWebServices Key: YARN-6259 URL: https://issues.apache.org/jira/browse/YARN-6259 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Affects Versions: 2.8.1 Reporter: Tao Yang Assignee: Tao Yang
Currently containerlogs REST API in NMWebServices will read and send the entire content of container logs. Most of container logs are large and it's useful to support pagination. * Add pagesize and pageindex parameters for containerlogs REST API {code} URL: http://<nm_address>/ws/v1/node/containerlogs/<container_id>/<file_name> QueryParams: pagesize - max bytes of one page , default 1MB pageindex - index of required page, default 0, can be nagative(set -1 will get the last page content) {code} * Add containerlogs-info REST API since sometimes we need to know the totalSize/pageSize/pageCount info of log {code} URL: http://<nm_address>/ws/v1/node/containerlogs-info/<container_id>/<file_name> QueryParams: pagesize - max bytes of one page , default 1MB Response example: {"logInfo":{"totalSize":2497280,"pageSize":1048576,"pageCount":3}} {code} Moreover, the data transfer pipeline (disk --> read buffer --> NM buffer --> socket buffer) can be optimized to pipeline(disk --> read buffer --> socket buffer) with zero-copy approach. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org