[
https://issues.apache.org/jira/browse/HIVE-15881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15861995#comment-15861995
]
Thomas Poepping commented on HIVE-15881:
----------------------------------------
>From what I can tell, saying
bq. These methods are Hive related
isn't exactly true.
I agree that the configuration value name is confusing, mainly because of the
inclusion of {{mapred}} (we support Tez too, right? ;) ). But it seems like the
dfsclient (HDFS, S3a, etc) should actually have some level of control over the
number of threads being used to access it. What if we have some {{FileSystem}}
implementation that, for some reason, is non-threadsafe? It should be able to
limit threads. Renaming the config to something like
{{hive.get.input.listing.num.threads}} can hide what's _really_ happening here,
which would be the dfsclient controlling its maximum allowable number of
parallel accessors on its own.
That being said, I can't find a link in open source Hadoop to this
configuration value. Can you link it here? If it is actually only used by Hive
across the entire Hadoop ecosystem, then I have no problem doing what you
suggest. I just have a hard time believing that's the case.
> Use new thread count variable name instead of mapred.dfsclient.parallelism.max
> ------------------------------------------------------------------------------
>
> Key: HIVE-15881
> URL: https://issues.apache.org/jira/browse/HIVE-15881
> Project: Hive
> Issue Type: Task
> Components: Query Planning
> Reporter: Sergio Peña
> Assignee: Sergio Peña
> Priority: Minor
>
> The Utilities class has two methods, {{getInputSummary}} and
> {{getInputPaths}}, that use the variable {{mapred.dfsclient.parallelism.max}}
> to get the summary of a list of input locations in parallel. These methods
> are Hive related, but the variable name does not look it is specific for Hive.
> Also, the above variable is not on HiveConf nor used anywhere else. I just
> found a reference on the Hadoop MR1 code.
> I'd like to propose the deprecation of {{mapred.dfsclient.parallelism.max}},
> and use a different variable name, such as
> {{hive.get.input.listing.num.threads}}, that reflects the intention of the
> variable. The removal of the old variable might happen on Hive 3.x
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)