[
https://issues.apache.org/jira/browse/NIFI-11240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17783567#comment-17783567
]
Clement Law commented on NIFI-11240:
------------------------------------
Question: do we have any thoughts about how to monitor the resource usage of
the python servers? As it stands, I believe we can easily monitor things in the
JVM, but these "headless" python servers will be outside that world. If the
resource usage of the python processors can be bubbled up to the UI in the
conventional way, that would make this option a lot more appetising.
> Introduce Python API for building Processors
> --------------------------------------------
>
> Key: NIFI-11240
> URL: https://issues.apache.org/jira/browse/NIFI-11240
> Project: Apache NiFi
> Issue Type: Epic
> Components: Core Framework, Documentation & Website, Extensions
> Reporter: Mark Payne
> Assignee: Mark Payne
> Priority: Major
> Fix For: 2.0.0
>
>
> The scripting processors are very common for data transformation in NiFi. In
> particular, the Jython based scripts are quite heavily used. However, Jython
> is run on the JVM and does not support CPython libraries. As a result, it's
> syntax compatible but doesn't make use of the wealth of Python libraries. And
> the wealth of Python libraries are what make Python popular to begin with.
> Additionally, use of many script-based processors hurts the UX. They are
> cumbersome to configure, with script files and/or script bodies. They result
> in a dataflow that's difficult to understand because instead of nicely named
> processors like CompressContent the type and default name are
> "ExecuteScript." They're also difficult to share.
> I have been playing with Py4J for introduce a true Python-based API for
> developing Processors. This will introduce new APIs, new framework changes,
> and documentation. And this will likely take a while to stabilize. However,
> the sooner that we are able to land it into the hands of users, the better.
> Therefore, I pose that we introduce it in multiple milestones. We can create
> sub-tickets for different milestones, but in general it should follow:
> Milestone 1: Initial implementation. Provides the capability and an API for
> building processors. Includes sample code and some documentation. Includes
> tests to ensure proper operation. Should not be used in production. API will
> not be stable and may change frequently. Performance may be subpar. Get into
> the hands of developers to begin exploring and providing feedback /
> submitting PRs.
> Milestone 2: Bug fixes. API refinement. Improve performance.
> Milestone 3: Additional bug fixes and API refinement. API should become more
> stable.
> Milestone 4: Additional bug fixes. API becomes stable. Documentation is clear
> and sufficient. Recommend production use.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
