[
https://issues.apache.org/jira/browse/SQOOP-2496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15235326#comment-15235326
]
Alvin Ge commented on SQOOP-2496:
---------------------------------
Hi Jarek Jarcec Cecho
Thanks for your reply, Good news~! I have been fixed it, I put some codes in
MapreduceSubmissionEngine.java( near by #215) like this:
// Promote all required jars to the job
Set<String> jarPaths = request.getJars();
Set<String> fixedPath = new HashSet<String>(jarPaths.size());
for (String jarPath : jarPaths) {
//If found some jar path begin with jar:file, fixed it for hadoop.
if (jarPath.indexOf("jar:file://") == 0) {
//I can not sure the suffix string always are !/
jarPath = "file:" + jarPath.substring("jar:file://".length(),
jarPath.length() - "!/".length());
}
fixedPath.add(jarPath);
}
String requestJarPath = StringUtils.join(fixedPath, ",");
LOG.debug(requestJarPath);
configuration.set("tmpjars", requestJarPath);
then, I compile it and put it into sqoop/WEB-INF/lib, now, they are works for
me very well now.....:)
But I'm not sure it is suitable.
> Sqoop2: Provide a way to inject external connectors
> ---------------------------------------------------
>
> Key: SQOOP-2496
> URL: https://issues.apache.org/jira/browse/SQOOP-2496
> Project: Sqoop
> Issue Type: Bug
> Affects Versions: 1.99.6
> Reporter: Jarek Jarcec Cecho
> Fix For: 1.99.7
>
>
> On internal hackathon we we're hacking Sqoop 2 connector with [~singhashish]
> and we went through few troubles that we should address.
> We have a a [configuration
> property|https://github.com/apache/sqoop/blob/sqoop2/dist/src/main/server/conf/sqoop.properties#L173]
> for extra directory from which we will load jar files. We were able to use
> this configuration property to load our hacked connector to Server, but we
> were not able to get it working through job submission. Here is the exception
> that we hit:
> {code}
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative
> path in absolute URI:
> jar:file://var/lib/sqoop/connectors/Connector-1.0-SNAPSHOT.jar!
> at org.apache.hadoop.fs.Path.initialize(Path.java:206)
> at org.apache.hadoop.fs.Path.<init>(Path.java:172)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:215)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:390)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:483)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1306)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1303)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1303)
> at
> org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submitToCluster(MapreduceSubmissionEngine.java:274)
> at
> org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:255)
> at org.apache.sqoop.driver.JobManager.start(JobManager.java:288)
> at
> org.apache.sqoop.handler.JobRequestHandler.startJob(JobRequestHandler.java:380)
> at
> org.apache.sqoop.handler.JobRequestHandler.handleEvent(JobRequestHandler.java:116)
> at
> org.apache.sqoop.server.v1.JobServlet.handlePutRequest(JobServlet.java:96)
> at
> org.apache.sqoop.server.SqoopProtocolServlet.doPut(SqoopProtocolServlet.java:79)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
> {code}
> To put into nutshell,
> [ClassUtils.jarForClass|https://github.com/apache/sqoop/blob/sqoop2/common/src/main/java/org/apache/sqoop/utils/ClassUtils.java#L136]
> returns for external connectors path starting with prefix {{jar:file}} and
> suffix {{!}} that breaks mapreduce code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
