[
https://issues.apache.org/jira/browse/NIFI-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin updated NIFI-6724:
-------------------------
Description:
PutSQS does not check for individual status in batch message. This could lead
to PutSQS reporting success and dropping files even when individual files
actually failed in SQS.
As specified in the [AWSSimpleQueueService API Reference for
SendMessageBatch|https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessageBatch.html]:
{quote}The result of sending each message is reported individually in the
response. Because the batch request can result in a combination of successful
and unsuccessful actions, *you should check for batch errors even when the call
returns an HTTP status code of 200.*
{quote}
We learned this as we did not Base64-encode compressed files (gzip) before
sending them to PutSQS. All the compressed files failed to queue in SQS, but
SQS acknowledged the batch message and reported status 200 (success). Thus
PutSQS also reported success and dropped the files, when they actually failed.
I suggest a simple fix by implementing
[SendMessageBatchResult|https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/model/SendMessageBatchResult.html]
to check status on individual files.
was:
PutSQS does not check for individual status in batch message. This could lead
to PutSQS reporting success and dropping files even when individual files
actually failed in SQS.
As specified in the [AWSSimpleQueueService API Reference for
SendMessageBatch|[https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessageBatch.html]]:
{quote}The result of sending each message is reported individually in the
response. Because the batch request can result in a combination of successful
and unsuccessful actions, *you should check for batch errors even when the call
returns an HTTP status code of 200.*
{quote}
We learned this as we did not Base64-encode compressed files (gzip) before
sending them to PutSQS. All the compressed files failed to queue in SQS, but
SQS acknowledged the batch message and reported status 200 (success). Thus
PutSQS also reported success and dropped the files, when they actually failed.
I suggest a simple fix by implementing
[SendMessageBatchResult|[https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/model/SendMessageBatchResult.html]]
to check status on individual files.
> PutSQS does not check for individual status in batch message
> ------------------------------------------------------------
>
> Key: NIFI-6724
> URL: https://issues.apache.org/jira/browse/NIFI-6724
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Affects Versions: 1.9.2
> Environment: Docker running in AWS
> Reporter: Martin
> Priority: Critical
> Labels: Bug, PutSQS, SQS
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> PutSQS does not check for individual status in batch message. This could lead
> to PutSQS reporting success and dropping files even when individual files
> actually failed in SQS.
> As specified in the [AWSSimpleQueueService API Reference for
> SendMessageBatch|https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessageBatch.html]:
> {quote}The result of sending each message is reported individually in the
> response. Because the batch request can result in a combination of successful
> and unsuccessful actions, *you should check for batch errors even when the
> call returns an HTTP status code of 200.*
> {quote}
>
> We learned this as we did not Base64-encode compressed files (gzip) before
> sending them to PutSQS. All the compressed files failed to queue in SQS, but
> SQS acknowledged the batch message and reported status 200 (success). Thus
> PutSQS also reported success and dropped the files, when they actually
> failed.
>
> I suggest a simple fix by implementing
> [SendMessageBatchResult|https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/model/SendMessageBatchResult.html]
> to check status on individual files.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
