Zsihovszki Krisztina created NIFI-13146:
-------------------------------------------
Summary: ConsumeSlack processor rate limited
Key: NIFI-13146
URL: https://issues.apache.org/jira/browse/NIFI-13146
Project: Apache NiFi
Issue Type: Improvement
Reporter: Zsihovszki Krisztina
ConsumeSlack processor was not able to start when it was running in a Slack
workplace with thousands of channels. It got stuck in initialization phase and
reported rate limit error continuously.
The processor fetches all available channels (conversation list) during its
setup and creates a channel id/name mapping.
Fetching the conversation list items is executed in batches, 1000
channels/batch. Since the fetch is done continously, after a while Slack API
returns rate limit error. (Rate limit settings were at default in Slack.)
Even if some delay was added after each API call, 30 seconds was not enough to
fetch all the channels (since it is in onScheduled, after 30 seconds the
initialization is re-attempted)
As a mitigation for the problem I'd like to add a logic which checks if
"Channels" property contains only IDs. In case no channel name is specified,
another Slack API call (
[conversations.info|https://api.slack.com/methods/conversations.info]) can be
used to fetch the channel names for the channel IDs and it is not necessary to
fetch the whole conversation list.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
