Hey Joe,
I think there are two possible JIRAs.
1) Add long polling support using setWaitTimeSeconds() - should be
really easy. I can take a crack at a pull request. Here's a JIRA:
https://issues.apache.org/jira/browse/NIFI-1103
2) Investigate the high CPU usage. I saw this initially for several
days, but it went away after I adjusted the run schedule (from 1 second
to 10 seconds back to 1 second). I have CPU charts showing the high
usage and corresponding drop, but I need to reproduce the issue.
I'll circle back in a few days when I get some time to work on it.
Cheers,
Adam
On 11/3/15 2:41 AM, Joe Witt wrote:
Adam,
Just wanted to follow up on this. Have you had any better results and
should we put a JIRA in behind what you're seeing?
Thanks
Joe
On Tue, Oct 20, 2015 at 7:58 PM, Adam Lamar <[email protected]> wrote:
Adam,
Thanks for the reply!
Amazon supports (and recommends) long polling on SQS queues[1]. The GetSQS
code doesn't attempt long polling at all, but I wasn't sure if this was
intentional or if the option had just never been added. With a 20 second
long poll, the processor would make 3 requests per minute instead of 60,
assuming the queue was empty during that time.
Another data point - even during high CPU usage, the GetSQS processor was
only making one request per second to SQS (verified via tcpdump). While not
ideal from a billing perspective, doesn't it seem wrong that 1 request a
second is causing such high CPU?
Perhaps to muddy the waters a bit, I played with the run schedule yesterday,
and even now that I've turned it back to 1 second, CPU usage is remaining
low. Before I could start/stop GetSQS repeatedly and observe the high CPU
usage, but now I can't reproduce it. If I'm able to consistently reproduce
the issue in the future, I'll be sure to post again.
Cheers,
Adam
[1]
http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html
On 10/20/15 4:37 AM, Adam Estrada wrote:
Adam,
I suspect that getSQS is polling Amazon to check for data. It's not
exactly like your standard message broker in that you have to force the
poll. Anyway, throw a wait time in there and see if that fixes it. This will
also help lower your monthly Amazon bill...
Adam
On Oct 19, 2015, at 11:41 PM, Adam Lamar <[email protected]> wrote:
Hi everybody!
I've been testing NiFi 0.3.0 with the GetSQS processor to fetch objects
from an AWS bucket as they're created. My flow looks like this:
GetSQS
SplitJson
ExtractText
FetchS3Object
PutFile
I noticed that GetSQS causes a high amount of CPU usage - about 90% of
one core. If I turn off GetSQS, CPU usage immediately drops to 2%. If I turn
GetSQS back on with the run schedule at 10, it stays at 2%.
Would it be worth using setWaitTimeSeconds [1] to make the SQS receive a
blocking call? Alternatively, should GetSQS default to a longer run
schedule?
Cheers,
Adam
[1]
http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/model/ReceiveMessageRequest.html#setWaitTimeSeconds(java.lang.Integer)
