[
https://issues.apache.org/jira/browse/CAMEL-7318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348694#comment-14348694
]
Grzegorz Grzybek edited comment on CAMEL-7318 at 3/5/15 12:45 PM:
------------------------------------------------------------------
Thanks [~thadia] for investigating this issue. I've added [similar
check|https://github.com/apache/camel/commit/08eb8bf] to skip missing (failed
to rename) file.
was (Author: gzres):
Thanks [~thadia] for investigating this issue. I've added similar check to skip
missing (failed to rename) file.
> Concurrency on HDFS Consumer not working efficiently
> ----------------------------------------------------
>
> Key: CAMEL-7318
> URL: https://issues.apache.org/jira/browse/CAMEL-7318
> Project: Camel
> Issue Type: Bug
> Components: camel-hdfs
> Affects Versions: 2.11.2
> Reporter: Martha Obrinteschi
> Assignee: Grzegorz Grzybek
> Priority: Minor
> Labels: concurrency, parallel
> Fix For: 2.14.3, 2.15.0
>
>
> If we have two HDFS consumers the files are being processed twice (once by
> each consumer, waiting one after the other) so the consumers are not working
> in parallel. If we add this the consumers will work as a team and the
> transfer will go faster.
> This happens because there is no exception thrown (the rename method just
> returns true or false and everything carries on as nothing would have
> happened :).
> In order to fix this we could add in the HdfsInputStream: 49
> boolean tf = info.getFileSystem().rename(new Path(ret.actualPath), new
> Path(ret.suffixedPath));
> if (!tf) {
> throw new IOException("This exception is thrown because the rename did
> not succeded.");
> }
> And also in the HdfsConsumer: 150
> try {
> this.rwlock.writeLock().lock();
> this.istream =
> HdfsInputStream.createInputStream(fileStatuses[i].getPath().toString(),
> this.config);
> } catch (IOException ioe) {
> log.info(ioe.getMessage() + " If the rename fails we move on to
> the next file.");
> continue;
> }
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
