Jarek Jarcec Cecho created SQOOP-2496:
-----------------------------------------
Summary: Sqoop2: Provide a way to inject external connectors
Key: SQOOP-2496
URL: https://issues.apache.org/jira/browse/SQOOP-2496
Project: Sqoop
Issue Type: Bug
Affects Versions: 1.99.6
Reporter: Jarek Jarcec Cecho
Fix For: 1.99.7
On internal hackathon we we're hacking Sqoop 2 connector with [~singhashish]
and we went through few troubles that we should address.
We have a a [configuration
property|https://github.com/apache/sqoop/blob/sqoop2/dist/src/main/server/conf/sqoop.properties#L173]
for extra directory from which we will load jar files. We were able to use
this configuration property to load our hacked connector to Server, but we were
not able to get it working through job submission. Here is the exception that
we hit:
{code}
java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path
in absolute URI: jar:file://var/lib/sqoop/connectors/Connector-1.0-SNAPSHOT.jar!
at org.apache.hadoop.fs.Path.initialize(Path.java:206)
at org.apache.hadoop.fs.Path.<init>(Path.java:172)
at
org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:215)
at
org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:390)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:483)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1306)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1303)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1303)
at
org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submitToCluster(MapreduceSubmissionEngine.java:274)
at
org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:255)
at org.apache.sqoop.driver.JobManager.start(JobManager.java:288)
at
org.apache.sqoop.handler.JobRequestHandler.startJob(JobRequestHandler.java:380)
at
org.apache.sqoop.handler.JobRequestHandler.handleEvent(JobRequestHandler.java:116)
at
org.apache.sqoop.server.v1.JobServlet.handlePutRequest(JobServlet.java:96)
at
org.apache.sqoop.server.SqoopProtocolServlet.doPut(SqoopProtocolServlet.java:79)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
{code}
To put into nutshell,
[ClassUtils.jarForClass|https://github.com/apache/sqoop/blob/sqoop2/common/src/main/java/org/apache/sqoop/utils/ClassUtils.java#L136]
returns for external connectors path starting with prefix {{jar:file}} and
suffix {{!}} that breaks mapreduce code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
