[
https://issues.apache.org/jira/browse/OOZIE-3218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16732866#comment-16732866
]
Hadoop QA commented on OOZIE-3218:
----------------------------------
Testing JIRA OOZIE-3218
Cleaning local git workspace
----------------------------
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
. {color:green}+1{color} the patch does not introduce any @author tags
. {color:green}+1{color} the patch does not introduce any tabs
. {color:green}+1{color} the patch does not introduce any trailing spaces
. {color:green}+1{color} the patch does not introduce any star imports
. {color:green}+1{color} the patch does not introduce any line longer than
132
. {color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
. {color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
. {color:green}+1{color} Javadoc generation succeeded with the patch
. {color:green}+1{color} the patch does not seem to introduce new Javadoc
warning(s)
. {color:orange}WARNING{color}: the current HEAD has 100 Javadoc warning(s)
{color:green}+1 COMPILE{color}
. {color:green}+1{color} HEAD compiles
. {color:green}+1{color} patch compiles
. {color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/git].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in
[fluent-job/fluent-job-api].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [core].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
. {color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
. {color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
. Tests run : 3106
. Tests failed : 1
. Tests in error : 0
. Tests timed out : 0
{color:red}-1{color} [ERROR] There are [1] test failures in [sharelib]. Listing
only the first [5] ones
testSqoopActionWithRedundantPrefix:org.apache.oozie.action.hadoop.TestSqoopActionExecutor
Check console output for the full list of errors/failures
. {color:orange}Tests failed at first run:{color}
TestPurgeXCommand#testPurgeBundleWithCoordChildWithWFChild2
. For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
. {color:green}+1{color} distro tarball builds with the patch
----------------------------
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
{color:red}. There is at least one warning, please check{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/957/
> Oozie Sqoop action with command splits the select clause into multiple parts
> due to delimiter being space
> ---------------------------------------------------------------------------------------------------------
>
> Key: OOZIE-3218
> URL: https://issues.apache.org/jira/browse/OOZIE-3218
> Project: Oozie
> Issue Type: Bug
> Components: action, workflow
> Affects Versions: 3.3.2, 4.1.0, 4.2.0, 4.3.0, 5.0.0
> Environment: Hortonworks Hadoop HDP-2.6.4.x release
> oozie admin -version: Oozie server build version: 4.2.0.2.6.4.0-91
> Reporter: Mahesh Balakrishnan
> Assignee: Mahesh Balakrishnan
> Priority: Major
> Attachments: OOZIE-3218-2.patch, OOZIE-3218-3.patch, OOZIE-3218.patch
>
>
> When running a Oozie Sqoop action which has command with {{--query}} in place
> the query is split into multiple parts causing {{"Unrecognized argument:"}}
> and in-turn fails.
> {code:xml}
> <sqoop xmlns="uri:oozie:sqoop-action:0.4">
> <job-tracker>${resourceManager}</job-tracker>
> <name-node>${nameNode}</name-node>
> <command>import --verbose --connect jdbc:mysql://test.openstacklocal/db
> --query select * from abc where $CONDITIONS --username test --password test
> --driver com.mysql.jdbc.Driver -m 1 </command>
> <ok to="end"/>
> </sqoop>
> {code}
>
> Oozie Launcher logs:
> {noformat}
> Sqoop command arguments :
> import
> --verbose
> --connect
> jdbc:mysql://test.openstacklocal/db
> --query
> "select
> *
> from
> abc
> where
> $CONDITIONS"
> --username
> hive
> --password
> ********
> --driver
> com.mysql.jdbc.Driver
> -m
> 1
> Fetching child yarn jobs
> tag id : oozie-a1bbe03a0983b9e822d12ae7bb269ee3
> 2791 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to
> ResourceManager at hdp263-3.openstacklocal/172.26.105.248:8050
> Child yarn jobs are found -
> =================================================================
> >>> Invoking Sqoop command line now >>>
> 3172 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not
> been set in the environment. Cannot check for additional configuration.
> 3172 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not
> been set in the environment. Cannot check for additional configuration.
> 3218 [main] INFO org.apache.sqoop.Sqoop - Running Sqoop version:
> 1.4.6.2.6.4.0-91
> 3218 [main] INFO org.apache.sqoop.Sqoop - Running Sqoop version:
> 1.4.6.2.6.4.0-91
> 3287 [main] DEBUG org.apache.sqoop.tool.BaseSqoopTool - Enabled debug
> logging.
> 3287 [main] DEBUG org.apache.sqoop.tool.BaseSqoopTool - Enabled debug
> logging.
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Error parsing
> arguments for import:
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Error parsing
> arguments for import:
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: *
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: *
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: from
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: from
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: abc
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: abc
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: where
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: where
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: $CONDITIONS"
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: $CONDITIONS"
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: --username
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: --username
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: abc
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: abc
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: --password
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: --password
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: abc
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: abc
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: --driver
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: --driver
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: com.mysql.jdbc.Driver
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: com.mysql.jdbc.Driver
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: -m
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: -m
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: 1
> 3288 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized
> argument: 1
> 3289 [main] DEBUG org.apache.sqoop.Sqoop -
> Try --help for usage instructions.
> {noformat}
> The code piece which causes the issue is in
> [*{{SqoopActionExecutor.java}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java#L89-L117]:
> {code:java}
> // Build a list of arguments from either a tokenized <command> string
> or a list of <arg>
> if (actionXml.getChild("command", ns) != null) {
> String command = actionXml.getChild("command", ns).getTextTrim();
> StringTokenizer st = new StringTokenizer(command, " ");
> while (st.hasMoreTokens()) {
> argList.add(st.nextToken());
> }
> }
> else {
> @SuppressWarnings("unchecked")
> List<Element> eArgs = (List<Element>)
> actionXml.getChildren("arg", ns);
> for (Element elem : eArgs) {
> argList.add(elem.getTextTrim());
> }
> }
> // If the command is given accidentally as "sqoop import --option"
> // instead of "import --option" we can make a user's life easier
> // by removing away the unnecessary "sqoop" token.
> // However, we do not do this if the command looks like
> // "sqoop --option", as that's entirely invalid.
> if (argList.size() > 1 &&
> argList.get(0).equalsIgnoreCase(SQOOP) &&
> !argList.get(1).startsWith("-")) {
> XLog.getLog(getClass()).info(
> "Found a redundant 'sqoop' prefixing the command.
> Removing it.");
> argList.remove(0);
> }
> setSqoopCommand(actionConf, argList.toArray(new
> String[argList.size()]));
> return actionConf;
> {code}
> Since the delimiter is a space, the code splits the {{select * from table
> into select, *, from, table as nextToken}} and adds them separately into the
> array causing the issue.
>
> I have made a code change locally to address this issue and did some testing
> around this and it seem to work fine, hence submitting the code change for
> this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
