[
https://issues.apache.org/jira/browse/CAMEL-7249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13930278#comment-13930278
]
Grzegorz Grzybek commented on CAMEL-7249:
-----------------------------------------
The problem is this hadoop code:
{code:java}
private static void loadFileSystems() {
synchronized (FileSystem.class) {
if (!FILE_SYSTEMS_LOADED) {
ServiceLoader<FileSystem> serviceLoader =
ServiceLoader.load(FileSystem.class);
for (FileSystem fs : serviceLoader) {
SERVICE_FILE_SYSTEMS.put(fs.getScheme(), fs.getClass());
}
FILE_SYSTEMS_LOADED = true;
}
}
}
...
public static Class<? extends FileSystem> getFileSystemClass(String scheme,
Configuration conf) throws IOException {
if (!FILE_SYSTEMS_LOADED) {
loadFileSystems();
}
Class<? extends FileSystem> clazz = null;
if (conf != null) {
clazz = (Class<? extends FileSystem>) conf.getClass("fs." + scheme +
".impl", null);
}
if (clazz == null) {
clazz = SERVICE_FILE_SYSTEMS.get(scheme);
}
if (clazz == null) {
throw new IOException("No FileSystem for scheme: " + scheme);
}
return clazz;
}
{code}
which uses either {{java.util.ServiceLoader}} (doesn't work in OSGi) or
{{"fs.<scheme>.impl"}} configuration properties.
OK. If I create routes manually, I can do the following:
{code:java}
org.apache.hadoop.conf.Configuration conf = new
org.apache.hadoop.conf.Configuration();
conf.setClass("fs.file.impl", org.apache.hadoop.fs.LocalFileSystem.class,
FileSystem.class);
conf.setClass("fs.hdfs.impl",
org.apache.hadoop.hdfs.DistributedFileSystem.class, FileSystem.class);
FileSystem.get("file:///", conf)
{code}
which caches the mapping {{file: ->
org.apache.hadoop.fs.LocalFileSystem.class}} in current classloader (static
field {{org.apache.hadoop.fs.FileSystem.CACHE}}).
But it's not that easy with Blueprint based routes...
OK. I'll work more on ways to configure {{HdfsComponent}}.
> Upgrade Camel-HDFS component to be compatible with Apache Hadoop 2.x
> --------------------------------------------------------------------
>
> Key: CAMEL-7249
> URL: https://issues.apache.org/jira/browse/CAMEL-7249
> Project: Camel
> Issue Type: New Feature
> Reporter: Grzegorz Grzybek
> Assignee: Grzegorz Grzybek
>
> Currently camel-hdfs depends on hadoop-core:1.2.1. Make (working name)
> {{camel-hdfs2}} component using newest (hadoop 2.3.0) libraries.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
