[ https://issues.apache.org/jira/browse/NIFI-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16980183#comment-16980183 ]
ASF subversion and git services commented on NIFI-6724: ------------------------------------------------------- Commit 6507b789483f9de9faf8dabd8d668abab0b0db78 in nifi's branch refs/heads/master from Pierre Villard [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=6507b78 ] NIFI-6724 - Check for SQS API call result in case of failures This closes #3897. Signed-off-by: Peter Turcsanyi <turcsa...@apache.org> > PutSQS does not check for individual status in batch message > ------------------------------------------------------------ > > Key: NIFI-6724 > URL: https://issues.apache.org/jira/browse/NIFI-6724 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions > Affects Versions: 1.9.2 > Environment: Docker running in AWS > Reporter: Martin > Assignee: Pierre Villard > Priority: Critical > Labels: Bug, PutSQS, SQS > Original Estimate: 2h > Time Spent: 20m > Remaining Estimate: 1h 40m > > PutSQS does not check for individual status in batch message. This could lead > to PutSQS reporting success and dropping files even when individual files > actually failed in SQS. > As specified in the AWSSimpleQueueService API Reference for > [SendMessageBatch|https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessageBatch.html]: > {quote}The result of sending each message is reported individually in the > response. Because the batch request can result in a combination of successful > and unsuccessful actions, *you should check for batch errors even when the > call returns an HTTP status code of 200.* > {quote} > > We learned this as we did not Base64-encode compressed files (gzip) before > sending them to PutSQS. All the compressed files failed to queue in SQS, but > SQS acknowledged the batch message and reported status 200 (success). Thus > PutSQS also reported success and dropped the files, when they actually > failed. > I suggest a simple fix by implementing > [SendMessageBatchResult|https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/model/SendMessageBatchResult.html] > to check status on individual files by implementing something similar to this > {code:java} > SendMessageBatchResult sendMessageBatchResult = > sqsClient.sendMessageBatch(request); > if (!sendMessageBatchResult.getFailed().isEmpty()) { > // Error > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)