[
https://issues.apache.org/jira/browse/SQOOP-2318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14506285#comment-14506285
]
Jarek Jarcec Cecho commented on SQOOP-2318:
-------------------------------------------
I'm slightly concerned that not cleaning up "dummy" configuration objects might
get us into trouble. In usual Hadoop process, Hadoop configuration files are
already on classpath and hence "correct" properties get's loaded up (properties
relevant to this particular cluster). We're however not running with proper
Hadoop classpath, so this won't be applicable for us and we get just default
values as are defined in the upstream project. That is the reason why we're
loading all the configuration manually and I'm worried those "defaults" might
get us into trouble by not being compatible with the "cluster" configurations.
Trying to solve the use case that [~szehon] had on mind. Right now we will
[throw an
exception|https://github.com/apache/sqoop/blob/sqoop2/submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java#L101]
if the configured directory doesn't exists at all. What about changing that
behavior to "if the configured property is empty, then use the default
{{Configuration}} object without clearing it (and print out big warning that
this is test mode not meant for production use). If the directory is
configured, then we can preserve current behavior.
What do you think [~szehon]?
> Use hadoop configuration defaults for MapReduceSubmissionEngine
> ---------------------------------------------------------------
>
> Key: SQOOP-2318
> URL: https://issues.apache.org/jira/browse/SQOOP-2318
> Project: Sqoop
> Issue Type: Bug
> Components: sqoop2-framework
> Reporter: Szehon Ho
> Assignee: Szehon Ho
> Attachments: SQOOP-2318.patch
>
>
> Found this while trying to run a local job runner out of the box for
> development testing.
> The MapReduceSubmissionEngine looks in /etc/hadoop/conf for hadoop
> configurations. When trying to run a local job runner without an actual
> hadoop install, this forces the user to create an /etc/hadoop/conf directory,
> and then put a lot of configs in there to get local job runner to work.
> Proposing a change to use hadoop defaults (comes with Hadoop Configuration
> object), which will make local job runner work out of the box. Though, empty
> /etc/hadoop/conf directory still needs to be created.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
