[
https://issues.apache.org/jira/browse/GORA-257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13709603#comment-13709603
]
Alfonso Nishikawa commented on GORA-257:
----------------------------------------
We are facing the "jar hell" problem [0] (dependency problems in general [1]).
I think the most "formal" solution is the one of OSGi [2], but that implies a
lot of reengineering and an adaptation of all dependences to that framework.
Better discard, I guess.
The solution shown for Nutch [3] seems a good approach, but I have to comment
in this issue that must be defined how will be faced the loading and sharing of
dependences in the following situations:
- Job run in Hadoop with dependences installed in the nodes.
- Job run in Hadoop with dependences in the job.
- Job run in Hadoop with dependences sent by Distributed Cache.
- Job run in Nutch with dependences installed in the nodes.
- Job run in Nutch with dependences in the job.
- Job run in Pig with dependences in another jars (registered).
- Job run in Pig with dependences in hadoop nodes.
I guess jar-jar will not be allowed, althought maybe could be interesting
(somewhere I read that is discouraged, but I find fine to send all dependencies
as a zip/jar).
About versions, I don't have clear if they should go in MANIFEST or not in our
case. Suggestions?
And I think we will need to create another gora-hbase module, one for 0.90.4-
and one for 0.92+ since they are not internally compatible because of that of
Avro dependence (check). How to exactly face this?
About Pig stuff, after defining the other things I could tell :)
[0] - http://en.wikipedia.org/wiki/JAR_hell#JAR_hell
[1] - http://en.wikipedia.org/wiki/Dependency_hell
[2] - https://en.wikipedia.org/wiki/OSGi
[3] - http://wiki.apache.org/nutch/WhatsTheProblemWithPluginsAndClass-loading
> Use separate class loaders for each module
> ------------------------------------------
>
> Key: GORA-257
> URL: https://issues.apache.org/jira/browse/GORA-257
> Project: Apache Gora
> Issue Type: Improvement
> Components: gora-core, maven, storage
> Reporter: Scott Stults
> Fix For: 0.5
>
>
> Some modules require conflicting versions of dependent libraries. For
> example, HBase and Solr require conflicting versions of the Zookeeper client
> library. In order to use both stores in the same VM they should each load and
> use their own version of the library. Decoupling the module dependencies from
> the core of Gora allows for independent versioning as well as multiple
> versions.
> To accomplish this, each module needs its own class loader. I think this
> could be done in one of two places: Either Gora core creates the class loader
> and assigns it to a module, or each module starts with a pared-down class
> with no external library dependencies and creates a class loader for the rest.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
