[
https://issues.apache.org/jira/browse/SQOOP-2634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14996883#comment-14996883
]
Jarek Jarcec Cecho commented on SQOOP-2634:
-------------------------------------------
{quote}
At the first step we could put all a connector's dependencies into its
classpath. Then we could improve this as some jars are dependent by all the
connectors, like the jars listed by you: {{joda-time}}, {{connector-sdk}},
{{sqoop-common}}, etc. We could put these common dependencies into
configuration "tmpjars" before submitting jobs to make sure they could be
shared by all the connectors.
{quote}
I'm not so much concerned about shipping one jar to the execution engine twice.
I'm more concerned about how we load classes from shared jars inside the
running code - whether each connector will load it's own instance of classes
inside those shared jars or not.
The reason for that is that I see a possible problem with exchanging data from
one connector to the other. As far as I know, class equivalency doesn't cross
{{ClassLoader}} boundaries. E.g. {{ClassUtils.loadClass(A.class,
FirstClassLoader) != ClassUtils.loadClass(B.class, SecondClassLoader)}}, which
could be a potential problem here.
{quote}
For *HDFS Connector*, we should use HDFS libraries from the cluster as before.
{quote}
I'm open to this behavior. Can we include how this actually will be implemented
in the design doc? So far I don't see any provision to add dependency jars from
Server's own classpath to the connector's classpath.
> Sqoop2: Allow connectors to express jar dependencies
> ----------------------------------------------------
>
> Key: SQOOP-2634
> URL: https://issues.apache.org/jira/browse/SQOOP-2634
> Project: Sqoop
> Issue Type: Sub-task
> Reporter: Dian Fu
> Assignee: Dian Fu
> Fix For: 1.99.7
>
> Attachments: SQOOP-2634.001.patch, SQOOP-2634.002.patch,
> SQOOP-2634.003.patch, SQOOP-2634.004.patch, SQOOP-2634.005.patch,
> SQOOP-2634.006.patch, SQOOP-2634.007.patch, SQOOP-2634.008.patch,
> SQOOP-2634.009.patch, SQOOP-2634.010.patch, design-doc-v1.pdf,
> design-doc-v2.pdf, design-doc-v3.pdf
>
>
> Currently Sqoop 2 has already provided the ability to config jar dependencies
> with property "org.apache.sqoop.classpath.extra". The limitation of this
> property is that we have to put all the dependencies together. It can't
> express jar dependencies for a specified connector. This capacity is useful
> as some connectors may have conflict jar dependencies. Put all the
> dependencies from different connectors together may cause problems.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
