Ajay Chitre created SQOOP-1301:
----------------------------------
Summary: 'run' method in Sqoop.java is not thread-safe
Key: SQOOP-1301
URL: https://issues.apache.org/jira/browse/SQOOP-1301
Project: Sqoop
Issue Type: Bug
Affects Versions: 1.4.4
Reporter: Ajay Chitre
Fix For: 1.4.5
It seems the ‘run’ method in Sqoop is not thread-safe. When multiple Sqoop
jobs are triggered exactly at the same time, they end up stepping on each
other. Here’s what seems to be the problem: The ‘run’ method calls
tool.parseArguments which in turn calls ConfigurationHelper.parseGenericOptions
which in turn creates a new instance of GenericOptionsParser. The constructor
for GenericOptionsParser is not thread-safe because it ends up calling
‘buildGeneralOptions’ which uses OptionBuilder.withArgName. This method uses
instance variables thereby making it thread unsafe.
The way we’ve got around it is by creating a ‘Lock’ object. This seems to be
working. If there’s a better way, please let us know. If not, please consider
adding this feature. We can create a patch if there’s an interest. Thanks.
Ajay Chitre ([email protected])
Virendra Singh ([email protected])
Anyway, here’s what we’ve done:
1) Created a class that extends Sqoop.java
public class DlSqoop extends Sqoop {
2) Created a Lock object:
private static Lock monitor = new ReentrantLock();
3) Overridden ‘run’ method as follows:
public int run(String [] args) {
if (options.getConf() == null) {
// Configuration wasn't initialized until after the ToolRunner
// got us to this point. ToolRunner gave Sqoop itself a Conf
// though.
options.setConf(getConf());
}
try {
monitor.lock();
options = tool.parseArguments(args, null, options, false);
tool.appendArgs(this.childPrgmArgs);
tool.validateOptions(options);
} catch (Exception e) {
// Couldn't parse arguments.
// Log the stack trace for this exception
LOG.debug(e.getMessage(), e);
// Print exception message.
System.err.println(e.getMessage());
// Print the tool usage message and exit.
ToolOptions toolOpts = new ToolOptions();
tool.configureOptions(toolOpts);
tool.printHelp(toolOpts);
return 1; // Exit on exception here.
} finally {
monitor.unlock();
}
return tool.run(options);
}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
