[
https://issues.apache.org/jira/browse/CAMEL-7249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13931564#comment-13931564
]
ASF GitHub Bot commented on CAMEL-7249:
---------------------------------------
GitHub user grgrzybek opened a pull request:
https://github.com/apache/camel/pull/110
[CAMEL-7249] camel-hdfs2 component
Finally I've managed to adapt camel-hdfs component to work with
hadoop-2.3.0 libraries.
Thanks to Jean-Baptiste Onofré for providing hadoop-client-2.3.0 OSGi
bundle. It'd be hard to use OSGi bundle per hadoop-*.jar artifact because of
split packages. So aggregate `hadoop-client` is ok.
Information:
* Generally camel-hdfs2 component is repackaged version of camel-hdfs
component with Maven dependencies switched to hadoop-common:2.3.0 and
hadoop-hdfs:2.3.0
* Some deprecated API usage was replaced by 2.3.0 equivalents
* This component works in OSGi environment (e.g., inside Apache Karaf) both
as classic bundle and blueprint definition
* There are some quirks related to OSGi:
* - Hadoop uses `java.util.ServiceLocator` to find filesystem
implementations, so some static code initialization must be performed to
replace this mechanism (works both in pure OSGi and Blueprint container)
* - `Import-Package` is not just a list generated by *maven-bundle-plugin*
- it must be adjusted to contain all packages reachable by reflection (e.g.
`org.apache.hadoop.hdfs*`)
* - I didn't touch some bundle versions in parent POM and instead I used
hardcoded versions for example in AVRO in
`platforms/karaf/features/src/main/resources/features.xml`, as the one used in
parent POM requires OSGi package `sun.misc`
* - I had to tweak `org.osgi.framework.system.packages.extra` property in
`camel-itest-osgi/**/itest/karaf/config.properties` to contain Xerces packages
* The documentation must be added for new component
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/grgrzybek/camel CAMEL-7249
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/camel/pull/110.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #110
----
commit af7661abb98b85658c35974bb914aae6602365a4
Author: Grzegorz Grzybek <[email protected]>
Date: 2014-03-10T14:05:43Z
[CAMEL-7249] Working version of camel-hdfs2 component
* All unit tests works fine.
* Usage of all deprecated APIs changed.
* hdfs://localhost tests work (@Ignored - they require external hdfs
system)
commit 0b9b3531e4a4bfe82492e3037eb36381ad0b1865
Author: Grzegorz Grzybek <[email protected]>
Date: 2014-03-11T18:29:30Z
[CAMEL-7249] Working OSGi tests for camel-hdfs2
There are some issues with this implementation:
* Import-Packages for `camel-hdfs2` are not just the ones generated by
maven-bundle plugin. They must contain some packages reachable only by
reflection (different FileSystem implementations)
* There is problem with using java.util.ServiceLoader inside OSGi and
Hadoop 2
uses ServiceLoader to discover different FileSystem implementations. For
OSGi
(especially blueprint container) we have to use a hack to populate STATIC
cache
* `camel-hdfs2` feature contains some hardcoded versions and for e.g., AVRO
a
downgraded one (1.7.5_1 requires `sun.misc` package - this changed however
since SMX4-1709)
* org.osgi.framework.system.packages.extra property in
camel-itest-osgi/**/itest/karaf/config.properties was added with xerces
packages
* documentation has to be added describing hadoop2 specific configuration
inside
OSGi - this changed since hadoop 1.2.1 (using java.util.ServiceLoader)
commit 2131dce4d8df10ec207d74f160e926dd74f068c5
Author: Grzegorz Grzybek <[email protected]>
Date: 2014-03-12T07:05:36Z
[CAMEL-7249] Working camel-itest-karaf for camel-hdfs2
commit cd101646c688206f73adac68352d14d264287b30
Author: Grzegorz Grzybek <[email protected]>
Date: 2014-03-12T08:54:29Z
[CAMEL-7249] Working blueprint tests
All itests now work both with LOCAL and HDFS filesystems
----
> Upgrade Camel-HDFS component to be compatible with Apache Hadoop 2.x
> --------------------------------------------------------------------
>
> Key: CAMEL-7249
> URL: https://issues.apache.org/jira/browse/CAMEL-7249
> Project: Camel
> Issue Type: New Feature
> Reporter: Grzegorz Grzybek
> Assignee: Grzegorz Grzybek
>
> Currently camel-hdfs depends on hadoop-core:1.2.1. Make (working name)
> {{camel-hdfs2}} component using newest (hadoop 2.3.0) libraries.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
