[
https://issues.apache.org/jira/browse/TEZ-1379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14086952#comment-14086952
]
Bikas Saha commented on TEZ-1379:
---------------------------------
We might be trying to do 2 things that are potentially at odds here. 1) Making
the API easy to use 2) Reducing the amount of junk that comes in via
configuration files.
If we simply accept Configuration then we make the API short and easy to use.
We can figure out how remove the junk from it later on.
Accepting a conf per comparator/partitioner/etc is making the API more
cumbersome to use without preventing the users from dumping that same junk in
each one of those configuration. If the user is motivated to write extra code
to filter out the params per entity then they would be motivated to do that for
the global conf too. For someone who is not motivated, there is nothing we can
do to avoid the junk being passed in every parameter. So might as well opt for
simplicity given the worst cast scenarios for getting junk are the same in both
cases.
I would prefer having "setComparator(String comparator), setComparator(String
comparator, Configuration conf)" etc. and recommend the non-configuration
method in javadoc. So that users dont dump configuration on us on every API
unless they hit an issue which makes them have to pass the configuration. At a
later point, if we decide to allow a filtered global config via
setConfiguration() then the individual set configurations can be easily
deprecated without affecting everyones code. Exposing the DAG configuration via
TezUserPayload may be an efficient way to ease the configuration pain at a
later point.
> EdgeConfigurers should accept a Partitioner configuration, accept parameters
> for compression and secure shuffle
> ---------------------------------------------------------------------------------------------------------------
>
> Key: TEZ-1379
> URL: https://issues.apache.org/jira/browse/TEZ-1379
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Blocker
> Attachments: TEZ-1379.1.txt, TEZ-1379.2.txt
>
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
