[jira] [Commented] (YARN-6259) Support pagination and optimize data transfer with zero-copy approach for containerlogs REST API in NMWebServices
[
https://issues.apache.org/jira/browse/YARN-6259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130303#comment-16130303
]
Tao Yang commented on YARN-6259:
Thanks [~djp] for your suggestions. It makes sense to me. I have created
YARN-7037 to handle performance improvement and the patch of this issue will be
updated later. I noticed that there are many differences between 2.8 and
2.9/trunk, 2.9/trunk supports getting head or tail part of log file. It's close
to our requirements but still not enough to support pagination.
> Support pagination and optimize data transfer with zero-copy approach for
> containerlogs REST API in NMWebServices
> -
>
> Key: YARN-6259
> URL: https://issues.apache.org/jira/browse/YARN-6259
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Affects Versions: 2.8.1
>Reporter: Tao Yang
>Assignee: Tao Yang
> Attachments: YARN-6259.001.patch
>
>
> Currently containerlogs REST API in NMWebServices will read and send the
> entire content of container logs. Most of container logs are large and it's
> useful to support pagination.
> * Add pagesize and pageindex parameters for containerlogs REST API
> {code}
> URL: http:///ws/v1/node/containerlogs//
> QueryParams:
> pagesize - max bytes of one page , default 1MB
> pageindex - index of required page, default 0, can be nagative(set -1 will
> get the last page content)
> {code}
> * Add containerlogs-info REST API since sometimes we need to know the
> totalSize/pageSize/pageCount info of log
> {code}
> URL:
> http:///ws/v1/node/containerlogs-info//
> QueryParams:
> pagesize - max bytes of one page , default 1MB
> Response example:
> {"logInfo":{"totalSize":2497280,"pageSize":1048576,"pageCount":3}}
> {code}
> Moreover, the data transfer pipeline (disk --> read buffer --> NM buffer -->
> socket buffer) can be optimized to pipeline(disk --> read buffer --> socket
> buffer) with zero-copy approach.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6259) Support pagination and optimize data transfer with zero-copy approach for containerlogs REST API in NMWebServices
[
https://issues.apache.org/jira/browse/YARN-6259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121019#comment-16121019
]
Junping Du commented on YARN-6259:
--
Thanks for the patch, [~Tao Yang]! It looks like we have performance
improvement here with mixing a new requirement of pagination. Can we split the
patch into two different parts? I believe no argument on performance gains and
we can have separated discussion on pagination requirement. Make sense?
> Support pagination and optimize data transfer with zero-copy approach for
> containerlogs REST API in NMWebServices
> -
>
> Key: YARN-6259
> URL: https://issues.apache.org/jira/browse/YARN-6259
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Affects Versions: 2.8.1
>Reporter: Tao Yang
>Assignee: Tao Yang
> Attachments: YARN-6259.001.patch
>
>
> Currently containerlogs REST API in NMWebServices will read and send the
> entire content of container logs. Most of container logs are large and it's
> useful to support pagination.
> * Add pagesize and pageindex parameters for containerlogs REST API
> {code}
> URL: http:///ws/v1/node/containerlogs//
> QueryParams:
> pagesize - max bytes of one page , default 1MB
> pageindex - index of required page, default 0, can be nagative(set -1 will
> get the last page content)
> {code}
> * Add containerlogs-info REST API since sometimes we need to know the
> totalSize/pageSize/pageCount info of log
> {code}
> URL:
> http:///ws/v1/node/containerlogs-info//
> QueryParams:
> pagesize - max bytes of one page , default 1MB
> Response example:
> {"logInfo":{"totalSize":2497280,"pageSize":1048576,"pageCount":3}}
> {code}
> Moreover, the data transfer pipeline (disk --> read buffer --> NM buffer -->
> socket buffer) can be optimized to pipeline(disk --> read buffer --> socket
> buffer) with zero-copy approach.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6259) Support pagination and optimize data transfer with zero-copy approach for containerlogs REST API in NMWebServices
[
https://issues.apache.org/jira/browse/YARN-6259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044062#comment-16044062
]
Hadoop QA commented on YARN-6259:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} YARN-6259 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-6259 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12855348/YARN-6259.001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/16165/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Support pagination and optimize data transfer with zero-copy approach for
> containerlogs REST API in NMWebServices
> -
>
> Key: YARN-6259
> URL: https://issues.apache.org/jira/browse/YARN-6259
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Affects Versions: 2.8.1
>Reporter: Tao Yang
>Assignee: Tao Yang
> Attachments: YARN-6259.001.patch
>
>
> Currently containerlogs REST API in NMWebServices will read and send the
> entire content of container logs. Most of container logs are large and it's
> useful to support pagination.
> * Add pagesize and pageindex parameters for containerlogs REST API
> {code}
> URL: http:///ws/v1/node/containerlogs//
> QueryParams:
> pagesize - max bytes of one page , default 1MB
> pageindex - index of required page, default 0, can be nagative(set -1 will
> get the last page content)
> {code}
> * Add containerlogs-info REST API since sometimes we need to know the
> totalSize/pageSize/pageCount info of log
> {code}
> URL:
> http:///ws/v1/node/containerlogs-info//
> QueryParams:
> pagesize - max bytes of one page , default 1MB
> Response example:
> {"logInfo":{"totalSize":2497280,"pageSize":1048576,"pageCount":3}}
> {code}
> Moreover, the data transfer pipeline (disk --> read buffer --> NM buffer -->
> socket buffer) can be optimized to pipeline(disk --> read buffer --> socket
> buffer) with zero-copy approach.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6259) Support pagination and optimize data transfer with zero-copy approach for containerlogs REST API in NMWebServices
[
https://issues.apache.org/jira/browse/YARN-6259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891423#comment-15891423
]
Tao Yang commented on YARN-6259:
Hi, [~rohithsharma]. Thank you for looking into this issue.
{quote}
I am not sure about how use cases will be served
{quote}
One common use case is to request last part of log and easily skip to another
part for detecting problem, instead of loading the entire log, it perhaps can
save a lot of time. We have an outer system to track apps and show container
logs inside, meanwhile most of logs are very large, so that pagination function
is needed and the newly added containerlogs-info REST API is a part of it.
{quote}
Instead of adding new LogInfo file, there is ContainerLogInfo file which can be
used for pageSize and pageIndex.
{quote}
ContainerLogInfo seems not exist in branch-2.8, perhaps it's for higher version?
> Support pagination and optimize data transfer with zero-copy approach for
> containerlogs REST API in NMWebServices
> -
>
> Key: YARN-6259
> URL: https://issues.apache.org/jira/browse/YARN-6259
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Affects Versions: 2.8.1
>Reporter: Tao Yang
>Assignee: Tao Yang
> Attachments: YARN-6259.001.patch
>
>
> Currently containerlogs REST API in NMWebServices will read and send the
> entire content of container logs. Most of container logs are large and it's
> useful to support pagination.
> * Add pagesize and pageindex parameters for containerlogs REST API
> {code}
> URL: http:///ws/v1/node/containerlogs//
> QueryParams:
> pagesize - max bytes of one page , default 1MB
> pageindex - index of required page, default 0, can be nagative(set -1 will
> get the last page content)
> {code}
> * Add containerlogs-info REST API since sometimes we need to know the
> totalSize/pageSize/pageCount info of log
> {code}
> URL:
> http:///ws/v1/node/containerlogs-info//
> QueryParams:
> pagesize - max bytes of one page , default 1MB
> Response example:
> {"logInfo":{"totalSize":2497280,"pageSize":1048576,"pageCount":3}}
> {code}
> Moreover, the data transfer pipeline (disk --> read buffer --> NM buffer -->
> socket buffer) can be optimized to pipeline(disk --> read buffer --> socket
> buffer) with zero-copy approach.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6259) Support pagination and optimize data transfer with zero-copy approach for containerlogs REST API in NMWebServices
[
https://issues.apache.org/jira/browse/YARN-6259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15890159#comment-15890159
]
Rohith Sharma K S commented on YARN-6259:
-
I am not sure about how use cases will be served, but skimmed through the
patch.
bq. Add containerlogs-info REST API since sometimes we need to know the
totalSize/pageSize/pageCount info of log
Instead of adding new LogInfo file, there is ContainerLogInfo file which can be
used for pageSize and pageIndex.
> Support pagination and optimize data transfer with zero-copy approach for
> containerlogs REST API in NMWebServices
> -
>
> Key: YARN-6259
> URL: https://issues.apache.org/jira/browse/YARN-6259
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Affects Versions: 2.8.1
>Reporter: Tao Yang
>Assignee: Tao Yang
> Attachments: YARN-6259.001.patch
>
>
> Currently containerlogs REST API in NMWebServices will read and send the
> entire content of container logs. Most of container logs are large and it's
> useful to support pagination.
> * Add pagesize and pageindex parameters for containerlogs REST API
> {code}
> URL: http:///ws/v1/node/containerlogs//
> QueryParams:
> pagesize - max bytes of one page , default 1MB
> pageindex - index of required page, default 0, can be nagative(set -1 will
> get the last page content)
> {code}
> * Add containerlogs-info REST API since sometimes we need to know the
> totalSize/pageSize/pageCount info of log
> {code}
> URL:
> http:///ws/v1/node/containerlogs-info//
> QueryParams:
> pagesize - max bytes of one page , default 1MB
> Response example:
> {"logInfo":{"totalSize":2497280,"pageSize":1048576,"pageCount":3}}
> {code}
> Moreover, the data transfer pipeline (disk --> read buffer --> NM buffer -->
> socket buffer) can be optimized to pipeline(disk --> read buffer --> socket
> buffer) with zero-copy approach.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
