Kengo Seki created CAMEL-18776:
----------------------------------
Summary: camel-hdfs - Fix HdfsNormalFileHandler to handle
temporary file path correctly
Key: CAMEL-18776
URL: https://issues.apache.org/jira/browse/CAMEL-18776
Project: Camel
Issue Type: Bug
Components: camel-hdfs
Reporter: Kengo Seki
Currently, HDFS consumer doesn't seem to work with normal files.
Given the following file on HDFS,
{code}
$ hdfs dfs -ls /tmp/input
Found 1 items
-rw-r--r-- 3 sekikn supergroup 4 2022-11-30 20:40 /tmp/input/foo.txt
$ hdfs dfs -cat /tmp/input/foo.txt
bar
{code}
I tried to run the following route so as to copy normal files from HDFS to
local filesystem.
{code}
$ cat put.java
import org.apache.camel.builder.RouteBuilder;
public class put extends RouteBuilder {
@Override
public void configure() throws Exception {
from("hdfs:192.168.2.2:9000/tmp/input/?repeatCount=1").log("${body}");
}
}
{code}
But I came across the following error.
{code}
$ camel run put.java
...
2022-11-30 20:46:25.151 WARN 2718863 --- [9000/tmp/input/]
apache.camel.component.hdfs.HdfsConsumer : Failed polling endpoint:
hdfs://192.168.2.2:9000/tmp/input/?repeatCount=1. Will try again at next poll.
Caused by: [org.apache.camel.RuntimeCamelException -
java.io.FileNotFoundException:
/tmp/foo.txt.opened5427239805734101292.hdfs/foo.txt.opened (Not a directory)]
org.apache.camel.RuntimeCamelException: java.io.FileNotFoundException:
/tmp/foo.txt.opened5427239805734101292.hdfs/foo.txt.opened (Not a directory)
at
org.apache.camel.component.hdfs.HdfsNormalFileHandler.createInputStream(HdfsNormalFileHandler.java:98)
~[camel-hdfs-3.19.0.jar:3.19.0]
at
org.apache.camel.component.hdfs.HdfsNormalFileHandler.createInputStream(HdfsNormalFileHandler.java:38)
~[camel-hdfs-3.19.0.jar:3.19.0]
at
org.apache.camel.component.hdfs.HdfsFileType.createInputStream(HdfsFileType.java:46)
~[camel-hdfs-3.19.0.jar:3.19.0]
at
org.apache.camel.component.hdfs.HdfsInputStream.createInputStream(HdfsInputStream.java:71)
~[camel-hdfs-3.19.0.jar:3.19.0]
at
org.apache.camel.component.hdfs.HdfsConsumer.asHdfsFile(HdfsConsumer.java:246)
~[camel-hdfs-3.19.0.jar:3.19.0]
...
Caused by: java.io.FileNotFoundException:
/tmp/foo.txt.opened5427239805734101292.hdfs/foo.txt.opened (Not a directory)
at java.io.FileInputStream.open0(Native Method) ~[?:?]
at java.io.FileInputStream.open(FileInputStream.java:219) ~[?:?]
at java.io.FileInputStream.<init>(FileInputStream.java:157) ~[?:?]
at
org.apache.camel.component.hdfs.HdfsNormalFileHandler.createInputStream(HdfsNormalFileHandler.java:94)
~[camel-hdfs-3.19.0.jar:3.19.0]
... 27 more
{code}
Temporary file was correctly created as follows, but HdfsNormalFileHandler
seems to have mistaken its path. The correct one is
"/tmp/foo.txt.opened5427239805734101292.hdfs", but there's unnecessary trailing
string "/foo.txt.opened" in the log above.
{code}
$ ls /tmp/foo.txt*
/tmp/foo.txt.opened5427239805734101292.hdfs
$ cat /tmp/foo.txt.opened5427239805734101292.hdfs
bar
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
