Martha Obrinteschi created CAMEL-7318:
-----------------------------------------
Summary: Concurrency on HDFS Consumer not working efficiently
Key: CAMEL-7318
URL: https://issues.apache.org/jira/browse/CAMEL-7318
Project: Camel
Issue Type: Bug
Components: camel-hdfs
Affects Versions: 2.11.2
Reporter: Martha Obrinteschi
Priority: Minor
If we have two HDFS consumers the files are being processed twice (once by each
consumer, waiting one after the other) so the consumers are not working in
parallel. If we add this the consumers will work as a team and the transfer
will go faster.
This happens because there is no exception thrown (the rename method just
returns true or false and everything carries on as nothing would have happened
:).
In order to fix this we could add in the HdfsInputStream: 49
boolean tf = info.getFileSystem().rename(new Path(ret.actualPath), new
Path(ret.suffixedPath));
if (!tf) {
throw new IOException("This exception is thrown because the rename did
not succeded.");
}
And also in the HdfsConsumer: 150
try {
this.rwlock.writeLock().lock();
this.istream =
HdfsInputStream.createInputStream(fileStatuses[i].getPath().toString(),
this.config);
} catch (IOException ioe) {
log.info(ioe.getMessage() + " If the rename fails we move on to the next
file.");
continue;
}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
