[GitHub] nifi issue #563: NIFI-2078: External state management.
Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/563 Hello @JPercivall @olegz @markap14 I've updated Cluster and External state behavior in a NiFi cluster, so that only a Primary node access Zookeeper or External data store. Tested, with the latest master branch. Would you be able to resume review process for this? Now this PR contains three related JIRA issues: - [NIFI-2078](https://issues.apache.org/jira/browse/NIFI-2078): State managed externally - [NIFI-2363](https://issues.apache.org/jira/browse/NIFI-2363): CLUSTER scope should be managed on primary node - [NIFI-2364](https://issues.apache.org/jira/browse/NIFI-2364): Avoid being disconnected by error related to external system Code change summaries for each topic is written in the commit message. For External state, it's also possible to make the state external local (per node). Please see following screenshots, fig-1 and 2. fig-1: External - CLUSTER  fig-2: External - LOCAL, GetKafka in this image is just an example. The actual code uses External - CLUSTER scope. There's no implementation using External - LOCAL as of now  Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #563: NIFI-2078: External state management.
Github user mcgilman commented on the issue: https://github.com/apache/nifi/pull/563 @ijokarumawak Awesome, thank you. There was just some confusion about whether we could resolve the PR for NIFI-2078 and just wanted to verify with you first. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #563: NIFI-2078: External state management.
Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/563 @mcgilman Yes, I noticed that 'Clear state' moved above the state table, so I removed z-index change. Thanks :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #563: NIFI-2078: External state management.
Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/563 Thanks @markap14 . I understand, I'll test to see if this works well with primary node failure and auto-elect scenario. It'd be the best if we could control nodes we replicate the HTTP requests to, however, getState and clearState are designed as per component request, meaning, local, cluster and external state are accessed all at once. So I will simply replicate HTTP requests to all nodes as it does now, and check the external state scope at DAO or service layer. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #563: NIFI-2078: External state management.
Github user olegz commented on the issue: https://github.com/apache/nifi/pull/563 @ijokarumawak I am not yet done, but good that you have rebased. I am just tied up with another thing at the moment, but will be reviewing later on --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #563: NIFI-2078: External state management.
Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/563 @JPercivall @olegz Thanks for reviewing this! I've rebased the branch, and addressed the feedback. However, while I was testing this again, I noticed that if it's running on a clustered NiFi environment, every node gets and clears offsets from Kafka. While it's working, I think these method should be executed on only the primary node, or if it's a standalone instance since offsets information is a global state among consumers. With `@OnPrimaryNodeStateChange` annotation, it's possible to know if it's a cluster and a primary node. But it's only called if a node get elected as a primary, or a node was primary but revoked. Since other node is not notified, I couldn't distinguish standalone or primary from a processor.. So, I left it as it is for now. Do you know If there's a way to know if NiFi is clustered from a processor? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
