[jira] [Updated] (NIFI-7572) Add a ScriptedTransformRecord processor
[
https://issues.apache.org/jira/browse/NIFI-7572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andy LoPresto updated NIFI-7572:
Resolution: Fixed
Status: Resolved (was: Patch Available)
Closing again after Matt Burgess resolved the "recompile script on change"
issue.
> Add a ScriptedTransformRecord processor
> ---
>
> Key: NIFI-7572
> URL: https://issues.apache.org/jira/browse/NIFI-7572
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.12.0
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> NiFi has started to put a heavier emphasis on Record-oriented processors, as
> they provide many benefits including better performance and a better UX over
> their purely byte-oriented counterparts. It is common to see users wanting to
> transform a Record in some very specific way, but NiFi doesn't make this as
> easy as it should. There are methods using ExecuteScript,
> InvokedScriptedProcessor, ScriptedRecordWriter, and ScriptedRecordReader for
> instance.
> But each of these requires that the Script writer understand a lot about NiFi
> and how to expose properties, create Property Descriptors, etc. and for
> fairly simple transformation we end up with scripts where the logic takes
> fewer lines of code than the boilerplate.
> We should expose a Processor that allows a user to write a script that takes
> a Record and transforms that Record in some way. The processor should be
> configured with the following:
> * Record Reader (required)
> * Record Writer (required)
> * Script Language (required)
> * Script Body or Script File (one and only one of these required)
> The script should implement a single method along the lines of:
> {code:java}
> Record transform(Record input) throws Exception; {code}
> If the script returns null, the input Record should be dropped. Otherwise,
> whatever Record is returned should be written to the Record Writer.
> The processor should have two relationships: "success" and "failure."
> The script should not be allowed to expose any properties or define any
> relationships. The point is to keep the script focused purely on processing
> the record itself.
> It's not entirely clear to me how easy the Record API works with some of the
> scripting languages. The Record object does expose a method named toMap()
> that returns a Map containing the underlying key/value pairs.
> However, the values in that Map may themselves be Records. It might make
> sense to expose a new method toNormalizedMap() or something along those lines
> that would return a Map where the values have been
> recursively normalized, in much the same way that we do for
> JoltTransformRecord. This would perhaps allow for cleaner syntax, but I'm not
> a scripting expert so I can't say for sure whether such a method is necessary.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-7572) Add a ScriptedTransformRecord processor
[
https://issues.apache.org/jira/browse/NIFI-7572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-7572:
---
Status: Patch Available (was: Reopened)
> Add a ScriptedTransformRecord processor
> ---
>
> Key: NIFI-7572
> URL: https://issues.apache.org/jira/browse/NIFI-7572
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.12.0
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> NiFi has started to put a heavier emphasis on Record-oriented processors, as
> they provide many benefits including better performance and a better UX over
> their purely byte-oriented counterparts. It is common to see users wanting to
> transform a Record in some very specific way, but NiFi doesn't make this as
> easy as it should. There are methods using ExecuteScript,
> InvokedScriptedProcessor, ScriptedRecordWriter, and ScriptedRecordReader for
> instance.
> But each of these requires that the Script writer understand a lot about NiFi
> and how to expose properties, create Property Descriptors, etc. and for
> fairly simple transformation we end up with scripts where the logic takes
> fewer lines of code than the boilerplate.
> We should expose a Processor that allows a user to write a script that takes
> a Record and transforms that Record in some way. The processor should be
> configured with the following:
> * Record Reader (required)
> * Record Writer (required)
> * Script Language (required)
> * Script Body or Script File (one and only one of these required)
> The script should implement a single method along the lines of:
> {code:java}
> Record transform(Record input) throws Exception; {code}
> If the script returns null, the input Record should be dropped. Otherwise,
> whatever Record is returned should be written to the Record Writer.
> The processor should have two relationships: "success" and "failure."
> The script should not be allowed to expose any properties or define any
> relationships. The point is to keep the script focused purely on processing
> the record itself.
> It's not entirely clear to me how easy the Record API works with some of the
> scripting languages. The Record object does expose a method named toMap()
> that returns a Map containing the underlying key/value pairs.
> However, the values in that Map may themselves be Records. It might make
> sense to expose a new method toNormalizedMap() or something along those lines
> that would return a Map where the values have been
> recursively normalized, in much the same way that we do for
> JoltTransformRecord. This would perhaps allow for cleaner syntax, but I'm not
> a scripting expert so I can't say for sure whether such a method is necessary.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-7572) Add a ScriptedTransformRecord processor
[
https://issues.apache.org/jira/browse/NIFI-7572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Thomsen updated NIFI-7572:
---
Fix Version/s: 1.12.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Add a ScriptedTransformRecord processor
> ---
>
> Key: NIFI-7572
> URL: https://issues.apache.org/jira/browse/NIFI-7572
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.12.0
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> NiFi has started to put a heavier emphasis on Record-oriented processors, as
> they provide many benefits including better performance and a better UX over
> their purely byte-oriented counterparts. It is common to see users wanting to
> transform a Record in some very specific way, but NiFi doesn't make this as
> easy as it should. There are methods using ExecuteScript,
> InvokedScriptedProcessor, ScriptedRecordWriter, and ScriptedRecordReader for
> instance.
> But each of these requires that the Script writer understand a lot about NiFi
> and how to expose properties, create Property Descriptors, etc. and for
> fairly simple transformation we end up with scripts where the logic takes
> fewer lines of code than the boilerplate.
> We should expose a Processor that allows a user to write a script that takes
> a Record and transforms that Record in some way. The processor should be
> configured with the following:
> * Record Reader (required)
> * Record Writer (required)
> * Script Language (required)
> * Script Body or Script File (one and only one of these required)
> The script should implement a single method along the lines of:
> {code:java}
> Record transform(Record input) throws Exception; {code}
> If the script returns null, the input Record should be dropped. Otherwise,
> whatever Record is returned should be written to the Record Writer.
> The processor should have two relationships: "success" and "failure."
> The script should not be allowed to expose any properties or define any
> relationships. The point is to keep the script focused purely on processing
> the record itself.
> It's not entirely clear to me how easy the Record API works with some of the
> scripting languages. The Record object does expose a method named toMap()
> that returns a Map containing the underlying key/value pairs.
> However, the values in that Map may themselves be Records. It might make
> sense to expose a new method toNormalizedMap() or something along those lines
> that would return a Map where the values have been
> recursively normalized, in much the same way that we do for
> JoltTransformRecord. This would perhaps allow for cleaner syntax, but I'm not
> a scripting expert so I can't say for sure whether such a method is necessary.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-7572) Add a ScriptedTransformRecord processor
[
https://issues.apache.org/jira/browse/NIFI-7572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-7572:
-
Status: Patch Available (was: Open)
> Add a ScriptedTransformRecord processor
> ---
>
> Key: NIFI-7572
> URL: https://issues.apache.org/jira/browse/NIFI-7572
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
>
> NiFi has started to put a heavier emphasis on Record-oriented processors, as
> they provide many benefits including better performance and a better UX over
> their purely byte-oriented counterparts. It is common to see users wanting to
> transform a Record in some very specific way, but NiFi doesn't make this as
> easy as it should. There are methods using ExecuteScript,
> InvokedScriptedProcessor, ScriptedRecordWriter, and ScriptedRecordReader for
> instance.
> But each of these requires that the Script writer understand a lot about NiFi
> and how to expose properties, create Property Descriptors, etc. and for
> fairly simple transformation we end up with scripts where the logic takes
> fewer lines of code than the boilerplate.
> We should expose a Processor that allows a user to write a script that takes
> a Record and transforms that Record in some way. The processor should be
> configured with the following:
> * Record Reader (required)
> * Record Writer (required)
> * Script Language (required)
> * Script Body or Script File (one and only one of these required)
> The script should implement a single method along the lines of:
> {code:java}
> Record transform(Record input) throws Exception; {code}
> If the script returns null, the input Record should be dropped. Otherwise,
> whatever Record is returned should be written to the Record Writer.
> The processor should have two relationships: "success" and "failure."
> The script should not be allowed to expose any properties or define any
> relationships. The point is to keep the script focused purely on processing
> the record itself.
> It's not entirely clear to me how easy the Record API works with some of the
> scripting languages. The Record object does expose a method named toMap()
> that returns a Map containing the underlying key/value pairs.
> However, the values in that Map may themselves be Records. It might make
> sense to expose a new method toNormalizedMap() or something along those lines
> that would return a Map where the values have been
> recursively normalized, in much the same way that we do for
> JoltTransformRecord. This would perhaps allow for cleaner syntax, but I'm not
> a scripting expert so I can't say for sure whether such a method is necessary.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-7572) Add a ScriptedTransformRecord processor
[
https://issues.apache.org/jira/browse/NIFI-7572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-7572:
-
Description:
NiFi has started to put a heavier emphasis on Record-oriented processors, as
they provide many benefits including better performance and a better UX over
their purely byte-oriented counterparts. It is common to see users wanting to
transform a Record in some very specific way, but NiFi doesn't make this as
easy as it should. There are methods using ExecuteScript,
InvokedScriptedProcessor, ScriptedRecordWriter, and ScriptedRecordReader for
instance.
But each of these requires that the Script writer understand a lot about NiFi
and how to expose properties, create Property Descriptors, etc. and for fairly
simple transformation we end up with scripts where the logic takes fewer lines
of code than the boilerplate.
We should expose a Processor that allows a user to write a script that takes a
Record and transforms that Record in some way. The processor should be
configured with the following:
* Record Reader (required)
* Record Writer (required)
* Script Language (required)
* Script Body or Script File (one and only one of these required)
The script should implement a single method along the lines of:
{code:java}
Record transform(Record input) throws Exception; {code}
If the script returns null, the input Record should be dropped. Otherwise,
whatever Record is returned should be written to the Record Writer.
The processor should have two relationships: "success" and "failure."
The script should not be allowed to expose any properties or define any
relationships. The point is to keep the script focused purely on processing the
record itself.
It's not entirely clear to me how easy the Record API works with some of the
scripting languages. The Record object does expose a method named toMap() that
returns a Map containing the underlying key/value pairs.
However, the values in that Map may themselves be Records. It might make sense
to expose a new method toNormalizedMap() or something along those lines that
would return a Map where the values have been recursively
normalized, in much the same way that we do for JoltTransformRecord. This would
perhaps allow for cleaner syntax, but I'm not a scripting expert so I can't say
for sure whether such a method is necessary.
was:
NiFi has started to put a heavier emphasis on Record-oriented processors, as
they provide many benefits including better performance and a better UX over
their purely byte-oriented counterparts. It is common to see users wanting to
transform a Record in some very specific way, but NiFi doesn't make this as
easy as it should. There are methods using ExecuteScript,
InvokedScriptedProcessor, ScriptedRecordWriter, and ScriptedRecordReader for
instance.
But each of these requires that the Script writer understand a lot about NiFi
and how to expose properties, create Property Descriptors, etc. and for fairly
simple transformation we end up with scripts where the logic takes fewer lines
of code than the boilerplate.
We should expose a Processor that allows a user to write a script that takes a
Record and transforms that Record in some way. The processor should be
configured with the following:
* Record Reader (required)
* Record Writer (required)
* Script Language (required)
* Script Body or Script File (one and only one of these required)
The script should implement a single method along the lines of:
{code:java}
void transform(Record record) throws Exception; {code}
The processor should have two relationships: "success" and "failure."
The script should not be allowed to expose any properties or define any
relationships. The point is to keep the script focused purely on processing the
record itself.
It's not entirely clear to me how easy the Record API works with some of the
scripting languages. The Record object does expose a method named toMap() that
returns a Map containing the underlying key/value pairs.
However, the values in that Map may themselves be Records. It might make sense
to expose a new method toNormalizedMap() or something along those lines that
would return a Map where the values have been recursively
normalized, in much the same way that we do for JoltTransformRecord. This would
perhaps allow for cleaner syntax, but I'm not a scripting expert so I can't say
for sure whether such a method is necessary.
> Add a ScriptedTransformRecord processor
> ---
>
> Key: NIFI-7572
> URL: https://issues.apache.org/jira/browse/NIFI-7572
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
>Reporter: Mark Payne
>Priority: Major
>
> NiFi has started to put a heavier emphasis on Record-oriented processors, as
> they provide many benefits including
[jira] [Updated] (NIFI-7572) Add a ScriptedTransformRecord processor
[
https://issues.apache.org/jira/browse/NIFI-7572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-7572:
-
Description:
NiFi has started to put a heavier emphasis on Record-oriented processors, as
they provide many benefits including better performance and a better UX over
their purely byte-oriented counterparts. It is common to see users wanting to
transform a Record in some very specific way, but NiFi doesn't make this as
easy as it should. There are methods using ExecuteScript,
InvokedScriptedProcessor, ScriptedRecordWriter, and ScriptedRecordReader for
instance.
But each of these requires that the Script writer understand a lot about NiFi
and how to expose properties, create Property Descriptors, etc. and for fairly
simple transformation we end up with scripts where the logic takes fewer lines
of code than the boilerplate.
We should expose a Processor that allows a user to write a script that takes a
Record and transforms that Record in some way. The processor should be
configured with the following:
* Record Reader (required)
* Record Writer (required)
* Script Language (required)
* Script Body or Script File (one and only one of these required)
The script should implement a single method along the lines of:
{code:java}
void transform(Record record) throws Exception; {code}
The processor should have two relationships: "success" and "failure."
The script should not be allowed to expose any properties or define any
relationships. The point is to keep the script focused purely on processing the
record itself.
It's not entirely clear to me how easy the Record API works with some of the
scripting languages. The Record object does expose a method named toMap() that
returns a Map containing the underlying key/value pairs.
However, the values in that Map may themselves be Records. It might make sense
to expose a new method toNormalizedMap() or something along those lines that
would return a Map where the values have been recursively
normalized, in much the same way that we do for JoltTransformRecord. This would
perhaps allow for cleaner syntax, but I'm not a scripting expert so I can't say
for sure whether such a method is necessary.
was:
NiFi has started to put a heavier emphasis on Record-oriented processors, as
they provide many benefits including better performance and a better UX over
their purely byte-oriented counterparts. It is common to see users wanting to
transform a Record in some very specific way, but NiFi doesn't make this as
easy as it should. There are methods using ExecuteScript,
InvokedScriptedProcessor, ScriptedRecordWriter, and ScriptedRecordReader for
instance.
But each of these requires that the Script writer understand a lot about NiFi
and how to expose properties, create Property Descriptors, etc. and for fairly
simple transformation we end up with scripts where the logic takes fewer lines
of code than the boilerplate.
We should expose a Processor that allows a user to write a script that takes a
Record and transforms that Record in some way. The processor should be
configured with the following:
* Record Reader (required)
* Record Writer (required)
* Script Language (required)
* Script Body or Script File (one and only one of these required)
The script should implement a single method along the lines of:
{code:java}
void transform(Record record) throws Exception; {code}
The processor should have two relationships: "success" and "failure."
The script should not be allowed to expose any properties ore define any
relationships. The point is to keep the script focused purely on processing the
record itself.
It's not entirely clear to me how easy the Record API works with some of the
scripting languages. The Record object does expose a method named toMap() that
returns a Map containing the underlying key/value pairs.
However, the values in that Map may themselves be Records. It might make sense
to expose a new method toNormalizedMap() or something along those lines that
would return a Map where the values have been recursively
normalized, in much the same way that we do for JoltTransformRecord. This would
perhaps allow for cleaner syntax, but I'm not a scripting expert so I can't say
for sure whether such a method is necessary.
> Add a ScriptedTransformRecord processor
> ---
>
> Key: NIFI-7572
> URL: https://issues.apache.org/jira/browse/NIFI-7572
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Extensions
>Reporter: Mark Payne
>Priority: Major
>
> NiFi has started to put a heavier emphasis on Record-oriented processors, as
> they provide many benefits including better performance and a better UX over
> their purely byte-oriented counterparts. It is common to see users wanting to
> transform a Record in
