Sharath created CAMEL-13624:
-------------------------------
Summary: Pollenrich issue while consuming second file with the
same name
Key: CAMEL-13624
URL: https://issues.apache.org/jira/browse/CAMEL-13624
Project: Camel
Issue Type: Bug
Components: came-core, camel-sftp
Affects Versions: 2.16.3, 2.10.4
Reporter: Sharath
Pollenrich issue while consuming second file with the same name
We have implemented a logic where a checksum will be compared during poll
enrich using aggregationStrategy.
Below is the entire flow of this logic
1. Poll xls file from sftp location. Lets say the name of the file will be
sample.xls
ftp.poll.options=&delay=\{{ftp.bt_in.delay}}&initialDelay=\{{ftp.bt_in.delay}}\
&binary=true&readLock=change\
&preMove=.inprogress&moveFailed=../.failed&delete=true\
&throwExceptionOnConnectFailed=true&pollStrategy=#ftpPollStrategy\
&localWorkDirectory=\{{ftp.localWorkDirectory}}\
&strictHostKeyChecking=\{{sftp.strictHostKeyChecking}}&knownHostsFile=\{{sftp.knownHostsFile}}\
&runLoggingLevel=DEBUG&disconnect=true
2. Poll subsequent sha file where the name will be sample.xls.sha512
ftp.sha512.poll.options=&delay=2000&initialDelay=2000\
&binary=true&readLock=change\
&moveFailed=.failed&delete=true\
&throwExceptionOnConnectFailed=true&pollStrategy=#ftpPollStrategy\
&strictHostKeyChecking=\{{sftp.strictHostKeyChecking}}&knownHostsFile=\{{sftp.knownHostsFile}}\
&runLoggingLevel=DEBUG
3. Polling sha file using pollenrich is as below
from("direct:" + routeId)
.id(routeId)
.onCompletion()
.process(new ShutDownProcessor())
.end()
.log(LoggingLevel.INFO,
"Polling SHA512 checksum for transfered file '${in.header.CamelFileName}' with
timeout " + timeout + " ms")
.log(LoggingLevel.DEBUG,"SHA512 dynamic polling route details: enrichUri: " +
this.enrichUri)
.convertBodyTo(byte[].class, "UTF-8")
.pollEnrich(enrichUri, timeout, MY_AGGREGATION_STRATEGY)
.to(outUri);
}
Where
enrichUri - "\{{source.ftp.endpoint}}\{{ftp.sha512.poll.options}}" +
"&fileName=" + fileName + ".sha512"
timeout - 30000
MY_AGGREGATION_STRATEGY - Strategy to compare old checksum
4. Aggregator logic has been implemeted as below.
public Exchange aggregate(Exchange oldExchange, Exchange newExchange) {
if (newExchange == null) {
log.error("No sftp file found for the input file!");
throw new RuntimeException("No sftp file found for the input file!");
}
Message sha512Msg = newExchange.hasOut() ? newExchange.getOut() :
newExchange.getIn();
Message incomingMsg = oldExchange.hasOut() ? oldExchange.getOut() :
oldExchange.getIn();
try {
String incomingSHA512 = fetch sha of received file
String computedSHA512 = fetch sha of computed file
if(incomingSHA512.equals(computedSHA512)) {
log.info("SHA512 check for file '" +
oldExchange.getIn().getHeader("CamelFileName", String.class) + "' successful.");
return oldExchange;
} else {
throw new RuntimeException("Computed SHA512 checksum '" + computedSHA512 + "'
does not match with provided SHA512 checksum '" + incomingSHA512 + "'");
}
} catch (Exception e) {
log.error("Error while processing SHA512 checksum of the passing file: " +
e.getMessage());
throw new RuntimeException(e);
}
Now the issue which we are facing is
1. First time if we introduce sample.xls and corresponding sha file where
checksum doesnt match, then it will throw an exception and the file will be
moved to failed folder. This is an expected behavior and works fine.
2. If we introduce the same file again with the same name, then sample.xls
file will be consumed, but corresponding sha file will not be consumed. The
file is present in the location but still it will landeup in "No sftp file
found for the input file!" which we have logged in aggregate method.
3. Those file will consume if i restart the server.
We have tried below
1. We are using camel 2.10.4 and tried with 2.16.3.
2. Adding idempotent=false in ftp.sha512.poll.options.
3. Changing timer and keep it for quite a long time.
But still facing same issue.
Are we missing anything here?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
