[ https://issues.apache.org/jira/browse/NIFI-9863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17517143#comment-17517143 ]
Otto Fowler edited comment on NIFI-9863 at 4/4/22 10:46 PM: ------------------------------------------------------------ [I'm going to keep this Grok specific, but there is a generic class of operation here I *think*] On the controller and its configuration side: * Allow entering in of grok patterns with test and validation * Allow entering of grok expressions with test and validation * Allow grouping of expressions into logical sets * Allow schema discovery from expressions and association with the expression * the schema of a set needs to be the same for each expression in the set * possibly schema for a pattern as well So, let me manage my grok patterns in a reusable way, and help me to make sure that they do what I think they do, and that the schema is known so I don't mess that up. from the consumer? List<PatternObjectMayHaveSchema> getPatterns() List<String> getAvailablePatterns() PatternObjectMayHaveSchema getPattern(String name) List<ExpressionObjectMayHaveSchema> getExpressions() List<String> getAvailableExpressions() ExpressionObjectMayHaveSchema getExpression(String name) List<ExpressionSet> getExpressionSets() List<String> getAvailableExpressionSets() ExpressionSet getExpressionSet(String name) We can also have a cache for compiled expressions ( that times out ) associated was (Author: ottobackwards): [I'm going to keep this Grok specific, but there is a generic class of operation here I *think*] On the controller and its configuration side: * Allow entering in of grok patterns with test and validation * Allow entering of grok expressions with test and validation * Allow grouping of expressions into logical sets * Allow schema discovery from expressions and association with the expression * the schema of a set needs to be the same for each expression in the set * possibly schema for a pattern as well So, let me manage my grok patterns in a reusable way, and help me to make sure that they do what I think they do, and that the schema is know so I don't mess that up. from the consumer? List<PatternObjectMayHaveSchema> getPatterns() List<String> getAvailablePatterns() PatternObjectMayHaveSchema getPattern(String name) List<ExpressionObjectMayHaveSchema> getExpressions() List<String> getAvailableExpressions() ExpressionObjectMayHaveSchema getExpression(String name) List<ExpressionSet> getExpressionSets() List<String> getAvailableExpressionSets() ExpressionSet getExpressionSet(String name) We can also have a cache for compiled expressions ( that times out ) associated > Controller Service for managing custom Grok patterns > ---------------------------------------------------- > > Key: NIFI-9863 > URL: https://issues.apache.org/jira/browse/NIFI-9863 > Project: Apache NiFi > Issue Type: New Feature > Reporter: Otto Fowler > Priority: Major > > Managing custom Grok expressions in properties for the Grok processors or > Record readers is cumbersome and not ideal. > Having a service that managed these expressions in a centralized and reusable > way would be a benefit to those using Grok patterns. > This service would allow the configuration of some number custom Grok > patterns as the service configuration. The MVP would be manual entry, but > loading patterns from File ( upload to configuration? ) or from some external > location could be allowed as well down the line. > In use, it could be argued that the patterns should be loaded from something > like the schema registry. > consumers of the service should then be able select the specific service > instance and then using dependent properties select which patterns provided > by the service to consume. > To this end, it may be nice to have the service support pattern 'groups', > such that you can select all patterns for a group at once. This would be the > easy button version of the linked multiple expressions to grok reader issue. -- This message was sent by Atlassian Jira (v8.20.1#820001)