[
https://issues.apache.org/jira/browse/YARN-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16144398#comment-16144398
]
Junping Du commented on YARN-7037:
----------------------------------
bq. LogToolUtils#outputContainerLog was used for both local log which can be
optimized by FileInputStream and aggregated log which can't because it's
transferred by DataInputStream from remote.
I see. That make sense to me.
+1 on latest patch. Will commit it tomorrow if no further comments from others.
> Optimize data transfer with zero-copy approach for containerlogs REST API in
> NMWebServices
> ------------------------------------------------------------------------------------------
>
> Key: YARN-7037
> URL: https://issues.apache.org/jira/browse/YARN-7037
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
> Affects Versions: 2.8.0
> Reporter: Tao Yang
> Assignee: Tao Yang
> Attachments: YARN-7037.001.patch, YARN-7037.branch-2.8.001.patch
>
>
> Split this improvement from YARN-6259.
> It's useful to read container logs more efficiently. With zero-copy approach,
> data transfer pipeline (disk --> read buffer --> NM buffer --> socket buffer)
> can be optimized to pipeline(disk --> read buffer --> socket buffer) .
> In my local test, time cost of copying 256MB file with zero-copy can be
> reduced from 12 seconds to 2.5 seconds.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
