[
https://issues.apache.org/jira/browse/NIFI-9863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17516398#comment-17516398
]
David Handermann commented on NIFI-9863:
----------------------------------------
Thanks for outlining possible paths for improving Grok configuration [~otto].
Understanding that the terminology can create some confusion, is the purpose of
this feature to support Expressions or Patterns? Based on the current
GrokReader property terminology, an Expression consists of multiple Patterns,
and the Reader uses the Expression to match against lines.
NIFI-9850 implemented supported for multiple Expressions, based on the Resource
Reference property definition. This approach allows multiple Expressions to
provided directly, or through a configured file or URL. Sharable Parameters
would enable sets of Expressions to be shared across processors, and across all
nodes within a cluster.
NIFI-3964 included a previous attempt to support multiple Patterns through a
Controller Service, but previous pull requests were not merged. It seems like
NIFI-3964 could be implemented using the same approach as multiple Expressions,
allowing Patterns to be provided through properties, Parameters, or through an
external file or URL.
The flexibility of Resource References, and support for shared configuration
through Parameters, seems like it could meet the goals described.
A Controller Service might be useful if different FlowFile attributes could
trigger the use of different Expressions, if that is the goal.
Some additional details on the intended capabilities, along with disambiguating
Expressions and Patterns, would be helpful
> Controller Service for managing custom Grok patterns
> ----------------------------------------------------
>
> Key: NIFI-9863
> URL: https://issues.apache.org/jira/browse/NIFI-9863
> Project: Apache NiFi
> Issue Type: New Feature
> Reporter: Otto Fowler
> Priority: Major
>
> Managing custom Grok expressions in properties for the Grok processors or
> Record readers is cumbersome and not ideal.
> Having a service that managed these expressions in a centralized and reusable
> way would be a benefit to those using Grok patterns.
> This service would allow the configuration of some number custom Grok
> patterns as the service configuration. The MVP would be manual entry, but
> loading patterns from File ( upload to configuration? ) or from some external
> location could be allowed as well down the line.
> In use, it could be argued that the patterns should be loaded from something
> like the schema registry.
> consumers of the service should then be able select the specific service
> instance and then using dependent properties select which patterns provided
> by the service to consume.
> To this end, it may be nice to have the service support pattern 'groups',
> such that you can select all patterns for a group at once. This would be the
> easy button version of the linked multiple expressions to grok reader issue.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
