[
https://issues.apache.org/jira/browse/TEZ-4137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17115681#comment-17115681
]
Ashutosh Chauhan commented on TEZ-4137:
---------------------------------------
+1 LGTM
> Input/Output/Processor should merge payload to local conf
> ---------------------------------------------------------
>
> Key: TEZ-4137
> URL: https://issues.apache.org/jira/browse/TEZ-4137
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Mustafa Iman
> Assignee: Mustafa Iman
> Priority: Major
> Attachments: TEZ-4137.1.patch, TEZ-4137.2.patch, TEZ-4137.3.patch,
> TEZ-4137.4.patch, TEZ-4137.4.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> This patch introduces config merging to various Input and Output processors.
> As described in https://issues.apache.org/jira/browse/TEZ-4073 , we need to
> reduce the size of the configuration objects transferred over the wire. There
> are two improvements we are planning to do regarding to that:
> # Skip sending default configs and configuration coming from xml files in
> payload
> # Send dag, vertex and session configurations in layers instead of sending
> dag + vertex + session configs all together three times.
> In order to achieve these,
> * We need to expose local config on Task side through TaskContext.
> * Input/Output/Processors must merge the config from user payload to local
> config in their TaskContext
> Since runtime components did not have access to local config before, tez
> clients sent all config required at runtime in user payload. After this
> change, tez clients can reduce their payload size.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
