[
https://issues.apache.org/jira/browse/NIFI-13146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17845107#comment-17845107
]
ASF subversion and git services commented on NIFI-13146:
--------------------------------------------------------
Commit 5a3b47353e66d124dd09a1ab4cff75b661911adc in nifi's branch
refs/heads/main from krisztina-zsihovszki
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=5a3b47353e ]
NIFI-13146 ConsumeSlack rate limit error mitigation (#8748)
Co-authored-by: Krisztina Zsihovszki <[email protected]>
> ConsumeSlack processor rate limited
> -----------------------------------
>
> Key: NIFI-13146
> URL: https://issues.apache.org/jira/browse/NIFI-13146
> Project: Apache NiFi
> Issue Type: Improvement
> Reporter: Zsihovszki Krisztina
> Assignee: Zsihovszki Krisztina
> Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> ConsumeSlack processor was not able to start when it was running in a Slack
> workplace with thousands of channels. It got stuck in initialization phase
> and reported rate limit error continuously.
> The processor fetches all available channels (conversation list) during its
> setup and creates a channel id/name mapping.
> Fetching the conversation list items is executed in batches, 1000
> channels/batch. Since the fetch is done continously, after a while Slack API
> returns rate limit error. (Rate limit settings were at default in Slack.)
> Even if some delay was added after each API call, 30 seconds was not enough
> to fetch all the channels (since it is in onScheduled, after 30 seconds the
> initialization is re-attempted)
> As a mitigation for the problem I'd like to add a logic which checks if
> "Channels" property contains only IDs. In case no channel name is specified,
> another Slack API call (
> [conversations.info|https://api.slack.com/methods/conversations.info]) can be
> used to fetch the channel names for the channel IDs and it is not necessary
> to fetch the whole conversation list.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
