Adam,
Thanks for the reply!
Amazon supports (and recommends) long polling on SQS queues[1]. The
GetSQS code doesn't attempt long polling at all, but I wasn't sure if
this was intentional or if the option had just never been added. With a
20 second long poll, the processor would make 3 requests per minute
instead of 60, assuming the queue was empty during that time.
Another data point - even during high CPU usage, the GetSQS processor
was only making one request per second to SQS (verified via tcpdump).
While not ideal from a billing perspective, doesn't it seem wrong that 1
request a second is causing such high CPU?
Perhaps to muddy the waters a bit, I played with the run schedule
yesterday, and even now that I've turned it back to 1 second, CPU usage
is remaining low. Before I could start/stop GetSQS repeatedly and
observe the high CPU usage, but now I can't reproduce it. If I'm able to
consistently reproduce the issue in the future, I'll be sure to post again.
Cheers,
Adam
[1]
http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html
On 10/20/15 4:37 AM, Adam Estrada wrote:
Adam,
I suspect that getSQS is polling Amazon to check for data. It's not exactly
like your standard message broker in that you have to force the poll. Anyway,
throw a wait time in there and see if that fixes it. This will also help lower
your monthly Amazon bill...
Adam
On Oct 19, 2015, at 11:41 PM, Adam Lamar <adamond...@gmail.com> wrote:
Hi everybody!
I've been testing NiFi 0.3.0 with the GetSQS processor to fetch objects from an
AWS bucket as they're created. My flow looks like this:
GetSQS
SplitJson
ExtractText
FetchS3Object
PutFile
I noticed that GetSQS causes a high amount of CPU usage - about 90% of one
core. If I turn off GetSQS, CPU usage immediately drops to 2%. If I turn GetSQS
back on with the run schedule at 10, it stays at 2%.
Would it be worth using setWaitTimeSeconds [1] to make the SQS receive a
blocking call? Alternatively, should GetSQS default to a longer run schedule?
Cheers,
Adam
[1]
http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/model/ReceiveMessageRequest.html#setWaitTimeSeconds(java.lang.Integer)
