Martin created NIFI-6724:
----------------------------
Summary: PutSQS does not check for individual status in batch
message
Key: NIFI-6724
URL: https://issues.apache.org/jira/browse/NIFI-6724
Project: Apache NiFi
Issue Type: Bug
Components: Extensions
Affects Versions: 1.9.2
Environment: Docker running in AWS
Reporter: Martin
PutSQS does not check for batch errors. This could lead to PutSQS reporting
success and dropping files even while individual files fails.
As specified in the [AWSSimpleQueueService API Reference for
SendMessageBatch|[https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessageBatch.html]]:
{quote}The result of sending each message is reported individually in the
response. Because the batch request can result in a combination of successful
and unsuccessful actions, *you should check for batch errors even when the call
returns an HTTP status code of 200.*
{quote}
We learned this as we did not Base64-encode compressed files (gzip) before
sending them to PutSQS. All the compressed files failed to queue in SQS, but
SQS acknowledged the batch message and reported status 200 (success). Thus
PutSQS also reported success and dropped the files, when they actually failed.
I suggest a simple fix by implementing
[SendMessageBatchResult|[https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/model/SendMessageBatchResult.html]]
to check status on individual files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
