[
https://issues.apache.org/jira/browse/TEZ-4236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17216670#comment-17216670
]
László Bodor edited comment on TEZ-4236 at 10/19/20, 12:10 PM:
---------------------------------------------------------------
could you please take a look at this change [~ashutoshc], [~rajesh.balamohan]?
this only affects tez local mode, so it's not supposed to cause problems in
prod systems
about the patch, feel free to comment on the [latest
version|https://github.com/abstractdog/tez/commit/df5b1117e992833016c2b5ec8bc873fe75686b49]
the motivation hasn't changed since I wrote the jira's description above: we
should find a(n optional) way to disable DAGAppMaster's RPC server in case of
local mode, as LocalClient already has a DAGAppMaster embedded
1. [introduced new
config|https://github.com/abstractdog/tez/commit/df5b1117e992833016c2b5ec8bc873fe75686b49#diff-bedc4ac5f0df37e202aa39d270d75e32cad8a996b39741c6e267c355ece8738eR1760],
tez.local.mode.without.network, which is about to drive whether a dag rpc
server is started for DAGAppMaster, or not <-- I was struggling with the proper
config name, maybe it should be tez.am.dag.rpc.server=false to reflect the
behavior
2. Adapt to the setting in LocalClient: instantiate the new LocalDAGAppMaster,
which is a no-op for initClientRpcServer
3. Most of the change is done in TezClient/FrameworkClient: these changes are
what I didn't do for the first time, but then I needed and it was
worth...basically, TezClient is a base logic for communicating dag events,
wrapping a FrameworkClient, which is an abstract for every specific
implementation...now as LocalClient started to do more locally (without RPC), I
needed to refactor stuff from TezClient -> FrameworkClient, because TezClient
relied heavily on the fact that it should communicate with an RPC server...now,
all these moved to FrameworkClient/TezYarnClient (getAMStatus, submitDag,
shutdownSession)
4. adapting tests:
[TestLocalMode|https://github.com/abstractdog/tez/commit/df5b1117e992833016c2b5ec8bc873fe75686b49#diff-d1d1b79a331141bde965de33cf204b77f15094760f7bea52d56abf1fe9fff93eR80]
has a full coverage on the new config option
[TestTezClient|https://github.com/abstractdog/tez/commit/df5b1117e992833016c2b5ec8bc873fe75686b49#diff-e3ed5501876742e26dfcfffa2a50f39fa22c02e3039ec8a6868022dc837539fbR141]
has been changed in a way that it reflects that from new FrameworkClient is
the class which needs to be "mocked" -> TezYarnClientForTest
was (Author: abstractdog):
could you please take a look at this change [~ashutoshc], [~rajesh.balamohan]?
this only affects tez local mode, so it's not supposed to cause problems in
prod systems
about the patch, feel free to comment on the [latest
version|https://github.com/abstractdog/tez/commit/df5b1117e992833016c2b5ec8bc873fe75686b49]
1. [introduced new
config|https://github.com/abstractdog/tez/commit/df5b1117e992833016c2b5ec8bc873fe75686b49#diff-bedc4ac5f0df37e202aa39d270d75e32cad8a996b39741c6e267c355ece8738eR1760],
tez.local.mode.without.network, which is about to drive whether a dag rpc
server is started for DAGAppMaster, or not <-- I was struggling with the proper
config name, maybe it should be tez.am.dag.rpc.server=false to reflect the
behavior
2. Adapt to the setting in LocalClient: instantiate the new LocalDAGAppMaster,
which is a no-op for initClientRpcServer
3. Most of the change is done in TezClient/FrameworkClient: these changes are
what I didn't do for the first time, but then I needed and it was
worth...basically, TezClient is a base logic for communicating dag events,
wrapping a FrameworkClient, which is an abstract for every specific
implementation...now as LocalClient started to do more locally (without RPC), I
needed to refactor stuff from TezClient -> FrameworkClient, because TezClient
relied heavily on the fact that it should communicate with an RPC server...now,
all these moved to FrameworkClient/TezYarnClient (getAMStatus, submitDag,
shutdownSession)
4. adapting tests:
[TestLocalMode|https://github.com/abstractdog/tez/commit/df5b1117e992833016c2b5ec8bc873fe75686b49#diff-d1d1b79a331141bde965de33cf204b77f15094760f7bea52d56abf1fe9fff93eR80]
has a full coverage on the new config option
[TestTezClient|https://github.com/abstractdog/tez/commit/df5b1117e992833016c2b5ec8bc873fe75686b49#diff-e3ed5501876742e26dfcfffa2a50f39fa22c02e3039ec8a6868022dc837539fbR141]
has been changed in a way that it reflects that from new FrameworkClient is
the class which needs to be "mocked" -> TezYarnClientForTest
> Investigate if DAGAppMaster's DAGClientServer is really need to be
> started/used in local mode
> ---------------------------------------------------------------------------------------------
>
> Key: TEZ-4236
> URL: https://issues.apache.org/jira/browse/TEZ-4236
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Attachments: TEZ-4236.01.patch, TEZ-4236.02.patch, TEZ-4236.03.patch,
> TEZ-4236.04.patch
>
>
> For a customer, we found that they try to run hive on tez local mode tests,
> and due to strict firewall rules and security policy, the DagClientServer
> struggles to start or respond to connections (think about corporate laptops
> here). Instead of investigating the network-related workarounds, let's find
> another way and decide if DAGAppMaster really needs this server.
> LocalClient has an embedded
> [DAGAppMaster|https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/client/LocalClient.java#L65],
> so at the first sight, it doesn't make sense to rely on the network
> environment in terms of AM usage. Hopefully, AM can become more lightweight
> in the long term if we try to eliminate pointless network usage.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
