[
https://issues.apache.org/jira/browse/CAMEL-7318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348482#comment-14348482
]
Grzegorz Grzybek commented on CAMEL-7318:
-----------------------------------------
You're right - for "hdfs://" scheme, the rename just returns {{false}}.
> Concurrency on HDFS Consumer not working efficiently
> ----------------------------------------------------
>
> Key: CAMEL-7318
> URL: https://issues.apache.org/jira/browse/CAMEL-7318
> Project: Camel
> Issue Type: Bug
> Components: camel-hdfs
> Affects Versions: 2.11.2
> Reporter: Martha Obrinteschi
> Assignee: Grzegorz Grzybek
> Priority: Minor
> Labels: concurrency, parallel
>
> If we have two HDFS consumers the files are being processed twice (once by
> each consumer, waiting one after the other) so the consumers are not working
> in parallel. If we add this the consumers will work as a team and the
> transfer will go faster.
> This happens because there is no exception thrown (the rename method just
> returns true or false and everything carries on as nothing would have
> happened :).
> In order to fix this we could add in the HdfsInputStream: 49
> boolean tf = info.getFileSystem().rename(new Path(ret.actualPath), new
> Path(ret.suffixedPath));
> if (!tf) {
> throw new IOException("This exception is thrown because the rename did
> not succeded.");
> }
> And also in the HdfsConsumer: 150
> try {
> this.rwlock.writeLock().lock();
> this.istream =
> HdfsInputStream.createInputStream(fileStatuses[i].getPath().toString(),
> this.config);
> } catch (IOException ioe) {
> log.info(ioe.getMessage() + " If the rename fails we move on to
> the next file.");
> continue;
> }
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
