[jira] [Commented] (NIFI-5083) Support for Expression Language in Group ID field of Kafka Consumer Processors
[ https://issues.apache.org/jira/browse/NIFI-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558887#comment-16558887 ] ASF GitHub Bot commented on NIFI-5083: -- Github user snagafritz closed the pull request at: https://github.com/apache/nifi/pull/2913 > Support for Expression Language in Group ID field of Kafka Consumer Processors > -- > > Key: NIFI-5083 > URL: https://issues.apache.org/jira/browse/NIFI-5083 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.6.0 >Reporter: Robert Bruno >Priority: Minor > Labels: kafka > Fix For: 1.8.0 > > Attachments: NIFI-5083.mp4 > > > I have the use case where I have two separate NiFi clusters reading data from > the same topics but need to have unique group ids so they each get separate > copies of the data. It would be useful if the group id field supported > expression language so I could use a processor group variable to define the > group id to use. I have a large enough number of kafka consumers to make > this tedious to manually change. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5083) Support for Expression Language in Group ID field of Kafka Consumer Processors
[ https://issues.apache.org/jira/browse/NIFI-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558550#comment-16558550 ] ASF GitHub Bot commented on NIFI-5083: -- Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2913 @snagafritz Hey.. I forgot to close this PR while merging. It is now merged to the master. Can you manually close this PR? > Support for Expression Language in Group ID field of Kafka Consumer Processors > -- > > Key: NIFI-5083 > URL: https://issues.apache.org/jira/browse/NIFI-5083 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.6.0 >Reporter: Robert Bruno >Priority: Minor > Labels: kafka > Attachments: NIFI-5083.mp4 > > > I have the use case where I have two separate NiFi clusters reading data from > the same topics but need to have unique group ids so they each get separate > copies of the data. It would be useful if the group id field supported > expression language so I could use a processor group variable to define the > group id to use. I have a large enough number of kafka consumers to make > this tedious to manually change. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5083) Support for Expression Language in Group ID field of Kafka Consumer Processors
[ https://issues.apache.org/jira/browse/NIFI-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558537#comment-16558537 ] ASF GitHub Bot commented on NIFI-5083: -- Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2913 +1 LGTM. Merged to master. Thanks @snagafritz > Support for Expression Language in Group ID field of Kafka Consumer Processors > -- > > Key: NIFI-5083 > URL: https://issues.apache.org/jira/browse/NIFI-5083 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.6.0 >Reporter: Robert Bruno >Priority: Minor > Labels: kafka > Attachments: NIFI-5083.mp4 > > > I have the use case where I have two separate NiFi clusters reading data from > the same topics but need to have unique group ids so they each get separate > copies of the data. It would be useful if the group id field supported > expression language so I could use a processor group variable to define the > group id to use. I have a large enough number of kafka consumers to make > this tedious to manually change. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5083) Support for Expression Language in Group ID field of Kafka Consumer Processors
[ https://issues.apache.org/jira/browse/NIFI-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558534#comment-16558534 ] ASF subversion and git services commented on NIFI-5083: --- Commit 4baf35db5ba296d246e3491e8d8c84408306f685 in nifi's branch refs/heads/master from [~snagafritz] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=4baf35d ] NIFI-5083: enabled EL support for processor configuration of Kafka consumer `group.id` property Signed-off-by: zenfenan > Support for Expression Language in Group ID field of Kafka Consumer Processors > -- > > Key: NIFI-5083 > URL: https://issues.apache.org/jira/browse/NIFI-5083 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.6.0 >Reporter: Robert Bruno >Priority: Minor > Labels: kafka > Attachments: NIFI-5083.mp4 > > > I have the use case where I have two separate NiFi clusters reading data from > the same topics but need to have unique group ids so they each get separate > copies of the data. It would be useful if the group id field supported > expression language so I could use a processor group variable to define the > group id to use. I have a large enough number of kafka consumers to make > this tedious to manually change. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5083) Support for Expression Language in Group ID field of Kafka Consumer Processors
[ https://issues.apache.org/jira/browse/NIFI-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558462#comment-16558462 ] ASF GitHub Bot commented on NIFI-5083: -- Github user zenfenan commented on the issue: https://github.com/apache/nifi/pull/2913 Reviewing.. > Support for Expression Language in Group ID field of Kafka Consumer Processors > -- > > Key: NIFI-5083 > URL: https://issues.apache.org/jira/browse/NIFI-5083 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.6.0 >Reporter: Robert Bruno >Priority: Minor > Labels: kafka > Attachments: NIFI-5083.mp4 > > > I have the use case where I have two separate NiFi clusters reading data from > the same topics but need to have unique group ids so they each get separate > copies of the data. It would be useful if the group id field supported > expression language so I could use a processor group variable to define the > group id to use. I have a large enough number of kafka consumers to make > this tedious to manually change. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5083) Support for Expression Language in Group ID field of Kafka Consumer Processors
[ https://issues.apache.org/jira/browse/NIFI-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555632#comment-16555632 ] ASF GitHub Bot commented on NIFI-5083: -- Github user bbende commented on the issue: https://github.com/apache/nifi/pull/2913 Actually, I take back my last comment, I now see where the value is obtained and it does have the logic: ``` String propertyValue = propertyDescriptor.isExpressionLanguageSupported() ? context.getProperty(propertyDescriptor).evaluateAttributeExpressions().getValue() : context.getProperty(propertyDescriptor).getValue(); ``` Sorry about that. > Support for Expression Language in Group ID field of Kafka Consumer Processors > -- > > Key: NIFI-5083 > URL: https://issues.apache.org/jira/browse/NIFI-5083 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.6.0 >Reporter: Robert Bruno >Priority: Minor > Labels: kafka > Attachments: NIFI-5083.mp4 > > > I have the use case where I have two separate NiFi clusters reading data from > the same topics but need to have unique group ids so they each get separate > copies of the data. It would be useful if the group id field supported > expression language so I could use a processor group variable to define the > group id to use. I have a large enough number of kafka consumers to make > this tedious to manually change. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5083) Support for Expression Language in Group ID field of Kafka Consumer Processors
[ https://issues.apache.org/jira/browse/NIFI-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555630#comment-16555630 ] ASF GitHub Bot commented on NIFI-5083: -- Github user bbende commented on the issue: https://github.com/apache/nifi/pull/2913 I think you will also need to find where the group id property is used and ensure that evaluateAttributeExpressions() is called when obtaining the value. > Support for Expression Language in Group ID field of Kafka Consumer Processors > -- > > Key: NIFI-5083 > URL: https://issues.apache.org/jira/browse/NIFI-5083 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.6.0 >Reporter: Robert Bruno >Priority: Minor > Labels: kafka > Attachments: NIFI-5083.mp4 > > > I have the use case where I have two separate NiFi clusters reading data from > the same topics but need to have unique group ids so they each get separate > copies of the data. It would be useful if the group id field supported > expression language so I could use a processor group variable to define the > group id to use. I have a large enough number of kafka consumers to make > this tedious to manually change. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5083) Support for Expression Language in Group ID field of Kafka Consumer Processors
[ https://issues.apache.org/jira/browse/NIFI-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553758#comment-16553758 ] Corey Fritz commented on NIFI-5083: --- Attached video of me verifying expected behavior from a build of my branch. > Support for Expression Language in Group ID field of Kafka Consumer Processors > -- > > Key: NIFI-5083 > URL: https://issues.apache.org/jira/browse/NIFI-5083 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.6.0 >Reporter: Robert Bruno >Priority: Minor > Labels: kafka > Attachments: NIFI-5083.mp4 > > > I have the use case where I have two separate NiFi clusters reading data from > the same topics but need to have unique group ids so they each get separate > copies of the data. It would be useful if the group id field supported > expression language so I could use a processor group variable to define the > group id to use. I have a large enough number of kafka consumers to make > this tedious to manually change. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5083) Support for Expression Language in Group ID field of Kafka Consumer Processors
[ https://issues.apache.org/jira/browse/NIFI-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16553753#comment-16553753 ] ASF GitHub Bot commented on NIFI-5083: -- GitHub user snagafritz opened a pull request: https://github.com/apache/nifi/pull/2913 NIFI-5083: enabled EL support for processor configuration of Kafka consumer group id Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? ``` I started to write unit tests for these changes then realized that the tests were really verifying the evaluation of the EL expressions, and I assume far better unit tests for this already exist. I will attach a video of me verifying expected behavior of a build of this branch to the JIRA ticket. ``` - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Snagajob/nifi NIFI-5083 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2913.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2913 commit 9d0a1ecd1d83ee2c6e54e3018db41280876a9629 Author: Corey Fritz Date: 2018-07-07T05:28:50Z NIFI-5083: enabled EL support for processor configuration of Kafka consumer `group.id` property > Support for Expression Language in Group ID field of Kafka Consumer Processors > -- > > Key: NIFI-5083 > URL: https://issues.apache.org/jira/browse/NIFI-5083 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.6.0 >Reporter: Robert Bruno >Priority: Minor > Labels: kafka > > I have the use case where I have two separate NiFi clusters reading data from > the same topics but need to have unique group ids so they each get separate > copies of the data. It would be useful if the group id field supported > expression language so I could use a processor group variable to define the > group id to use. I have a large enough number of kafka consumers to make > this tedious to manually change. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5083) Support for Expression Language in Group ID field of Kafka Consumer Processors
[ https://issues.apache.org/jira/browse/NIFI-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16535609#comment-16535609 ] Corey Fritz commented on NIFI-5083: --- I would also like to be able to use EL to set consumer group ids. I'm going to take a crack at this. > Support for Expression Language in Group ID field of Kafka Consumer Processors > -- > > Key: NIFI-5083 > URL: https://issues.apache.org/jira/browse/NIFI-5083 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.6.0 >Reporter: Robert Bruno >Priority: Minor > Labels: kafka > > I have the use case where I have two separate NiFi clusters reading data from > the same topics but need to have unique group ids so they each get separate > copies of the data. It would be useful if the group id field supported > expression language so I could use a processor group variable to define the > group id to use. I have a large enough number of kafka consumers to make > this tedious to manually change. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5083) Support for Expression Language in Group ID field of Kafka Consumer Processors
[ https://issues.apache.org/jira/browse/NIFI-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438649#comment-16438649 ] Robert Bruno commented on NIFI-5083: Currently I am using templates to sync between instances. I will likely for the time being do a find and replace in the template file, but this seems far from ideal. With that said FlowRegistry is on my short list of new things to give a try. Hopefully it is a better way to keep my 2 clusters in sync along with potentially dev clusters. > Support for Expression Language in Group ID field of Kafka Consumer Processors > -- > > Key: NIFI-5083 > URL: https://issues.apache.org/jira/browse/NIFI-5083 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.6.0 >Reporter: Robert Bruno >Priority: Minor > Labels: kafka > > I have the use case where I have two separate NiFi clusters reading data from > the same topics but need to have unique group ids so they each get separate > copies of the data. It would be useful if the group id field supported > expression language so I could use a processor group variable to define the > group id to use. I have a large enough number of kafka consumers to make > this tedious to manually change. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5083) Support for Expression Language in Group ID field of Kafka Consumer Processors
[ https://issues.apache.org/jira/browse/NIFI-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437779#comment-16437779 ] Joseph Witt commented on NIFI-5083: --- [~rbruno] does this mean you're using the FlowRegistry to ensure both clusters have a consistent flow and thus are ending up with the same group id being used? Adding support for EL to let variables be used is a very doable option and a good idea. Just wanted to understand how you're keeping the flows so in sync between clusters that this would be matching at this time. Thanks > Support for Expression Language in Group ID field of Kafka Consumer Processors > -- > > Key: NIFI-5083 > URL: https://issues.apache.org/jira/browse/NIFI-5083 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.6.0 >Reporter: Robert Bruno >Priority: Minor > Labels: kafka > > I have the use case where I have two separate NiFi clusters reading data from > the same topics but need to have unique group ids so they each get separate > copies of the data. It would be useful if the group id field supported > expression language so I could use a processor group variable to define the > group id to use. I have a large enough number of kafka consumers to make > this tedious to manually change. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
