fgerlits commented on code in PR #2039:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2039#discussion_r2405496777
##########
extensions/python/ExecutePythonProcessor.cpp:
##########
@@ -129,39 +88,47 @@ void ExecutePythonProcessor::loadScriptFromFile() {
throw std::runtime_error("Failed to read Script File: " +
script_file_path_);
}
script_to_exec_ = std::string{
(std::istreambuf_iterator<char>(file_handle)),
(std::istreambuf_iterator<char>()) };
+ if (script_to_exec_.empty()) {
+ throw std::runtime_error("Script body to execute is empty");
+ }
}
-void ExecutePythonProcessor::loadScript() {
- if (script_file_path_.empty() && script_to_exec_.empty()) {
- throw std::runtime_error("Neither Script Body nor Script File is available
to execute");
- }
+std::unique_ptr<PythonScriptEngine>
ExecutePythonProcessor::createScriptEngine() {
+ auto engine = std::make_unique<PythonScriptEngine>();
+ engine->initialize(Success, Failure, Original, python_logger_);
+ return engine;
+}
- if (!script_file_path_.empty()) {
- if (!script_to_exec_.empty()) {
- throw std::runtime_error("Only one of Script File or Script Body may be
used");
+void ExecutePythonProcessor::initalizeThroughScriptEngine() {
Review Comment:
nitpicking, and old code, but should be
```suggestion
void ExecutePythonProcessor::initializeThroughScriptEngine() {
```
##########
extensions/python/PYTHON.md:
##########
@@ -75,10 +82,13 @@ export
LD_LIBRARY_PATH="${PYENV_ROOT}/versions/${PY_VERSION}/lib${LD_LIBRARY_PAT
## Description
-Python native processors can be updated at any time by simply adding a new
processor to the directory defined in
-the `minifi.properties` file in the `nifi.python.processor.dir` property. The
processor name, when provided to MiNiFi C++ and any C2 manifest will be that
+There are two types of python processors that can be used: MiNiFi C++ style
native python processors and NiFi style python processors. MiNiFi C++ style
native python processors are defined using a simple python API, while NiFi
python processors use the NiFi python API documented in the [Apache NiFi Python
Developer’s
Guide](https://nifi.apache.org/nifi-docs/python-developer-guide.html). This
readme describes the use of both types of processors.
+
+Python processors can be updated at any time by simply adding a new processor
to the minifi-python directory defined in
+the `minifi.properties` file in the `nifi.python.processor.dir` property. For
NiFi style python processors the
`${nifi.python.processor.dir}/nifi_python_processor` directory should be used.
Review Comment:
typo:
```suggestion
the `minifi.properties` file in the `nifi.python.processor.dir` property.
For NiFi style python processors the
`${nifi.python.processor.dir}/nifi_python_processors` directory should be used.
```
##########
extensions/python/PYTHON.md:
##########
@@ -190,9 +172,59 @@ The SentimentAnalysis processor will perform a Vader
Sentiment Analysis. This re
pip install VaderSentiment
+## Using MiNiFi C++ style native python processors
+
+In MiNiFi C++ style native python processors, methods that are enabled within
the processor are describe, onSchedule, onInitialize, and onTrigger.
+
+Describe is passed the processor and is a required function. You must set the
description like so:
+
+```python
+def describe(processor):
+ processor.setDescription("Adds an attribute to your flow files")
+```
+
+onInitialize is also passed the processor reference and can be where you set
properties. The first argument is the property display name,
+followed by the description, and default value. The next three arguments are
booleans describing if the property is required, support expression language,
and if it is a sensitive property.
+The seventh argument is the property type code. The property type code is an
integer that represents the type of the property. The supported property type
codes and their corresponding types:
+```
+INTEGER = 0
+LONG = 1
+BOOLEAN = 2
+DATA_SIZE = 3
+TIME_PERIOD = 4
+NON_BLANK = 5
+PORT = 6
+```
+
+The last parameter of addProperty is the controller service type. If the
property is a controller service, the controller service type should be
provided. It should be the non-qualified type name of the controller service.
Currently SSLContextService is the only controller service type supported.
+
+```python
+def onInitialize(processor):
+ processor.setSupportsDynamicProperties()
+ # arguments: property name, description, default value, is required,
expression language supported, is sensitive, property type code, controller
service type name
+ processor.addProperty("property name", "description", "default value", True,
False, False, 1, None)
+```
+
+The onSchedule function is passed the context and session factory. This should
be where your processor loads and reads properties via
+the getProperty function. onTrigger is executed and passed the processor
context and session. You may keep state within your processor.
+
+Much like C++ processors, callbacks may be defined for reading/writing streams
of data through the session. Those callback classes will
+have a process function that accepts the input stream. You may use codecs
getReader to read that data as in the example, below, from
Review Comment:
typos:
```suggestion
have a process function that accepts the input stream. You may use
`codecs.getreader` to read that data as in the example below, from
```
##########
extensions/python/tests/ExecutePythonProcessorTests.cpp:
##########
@@ -221,95 +186,29 @@ class SimplePythonFlowFileTransferTest : public
ExecutePythonProcessorTestBase {
//
TEST_CASE_METHOD(SimplePythonFlowFileTransferTest, "Simple file passthrough",
"[executePythonProcessorSimple]") {
// Expectations
- const auto OUTPUT_FILE_MATCHES_INPUT =
SimplePythonFlowFileTransferTest::Expectation::OUTPUT_FILE_MATCHES_INPUT;
- const auto RUNTIME_RELATIONSHIP_EXCEPTION =
SimplePythonFlowFileTransferTest::Expectation::RUNTIME_RELATIONSHIP_EXCEPTION;
+ const auto OUTPUT_FILE_MATCHES_INPUT =
SimplePythonFlowFileTransferTest::Expectation::OUTPUT_FILE_MATCHES_INPUT;
+ const auto RUNTIME_RELATIONSHIP_EXCEPTION =
SimplePythonFlowFileTransferTest::Expectation::RUNTIME_RELATIONSHIP_EXCEPTION;
+ const auto INITIALIZATION_EXCEPTION =
SimplePythonFlowFileTransferTest::Expectation::INITIALIZATION_EXCEPTION;
// ExecutePython outbound relationships
- const core::Relationship SUCCESS {"success", "description"};
+ const core::Relationship SUCCESS{"success", "description"};
const core::Relationship FAILURE{"failure", "description"};
- // For the tests "" is treated as none-provided since no optional
implementation was ported to the project yet
-
// 0. Neither valid script file nor script body provided
- // TEST EXPECTATION OUT_REL
USE_AS_SCRIPT_FILE USE_AS_SCRIPT_BODY
- testSimpleFilePassthrough(RUNTIME_RELATIONSHIP_EXCEPTION, SUCCESS,
"", ""); // NOLINT
- testSimpleFilePassthrough(RUNTIME_RELATIONSHIP_EXCEPTION, FAILURE,
"", ""); // NOLINT
- testSimpleFilePassthrough(RUNTIME_RELATIONSHIP_EXCEPTION, SUCCESS,
"non_existent_script.py", ""); // NOLINT
- testSimpleFilePassthrough(RUNTIME_RELATIONSHIP_EXCEPTION, FAILURE,
"non_existent_script.py", ""); // NOLINT
+ // TEST EXPECTATION OUT_REL
USE_AS_SCRIPT_FILE
+ testSimpleFilePassthrough(INITIALIZATION_EXCEPTION, SUCCESS,
"non_existent_script.py"); // NOLINT
+ testSimpleFilePassthrough(INITIALIZATION_EXCEPTION, FAILURE,
"non_existent_script.py"); // NOLINT
Review Comment:
very minor, but the header is out of alignment:
```suggestion
// TEST EXPECTATION OUT_REL
USE_AS_SCRIPT_FILE
testSimpleFilePassthrough(INITIALIZATION_EXCEPTION, SUCCESS,
"non_existent_script.py"); // NOLINT
testSimpleFilePassthrough(INITIALIZATION_EXCEPTION, FAILURE,
"non_existent_script.py"); // NOLINT
```
##########
extensions/python/tests/ExecutePythonProcessorTests.cpp:
##########
@@ -116,8 +122,15 @@ class SimplePythonFlowFileTransferTest : public
ExecutePythonProcessorTestBase {
reInitialize();
const auto output_dir = testController_->createTempDirectory();
- auto executePythonProcessor =
plan_->addProcessor("ExecutePythonProcessor", "executePythonProcessor");
- plan_->setProperty(executePythonProcessor,
minifi::extensions::python::processors::ExecutePythonProcessor::ScriptFile,
getScriptFullPath("stateful_processor.py").string());
+ auto uuid = utils::IdGenerator::getIdGenerator()->generate();
+ auto execute_python_processor =
std::make_unique<minifi::extensions::python::processors::ExecutePythonProcessor>(core::ProcessorMetadata{
+ .uuid = uuid,
+ .name = "ExecutePythonProcessor",
+ .logger =
logging::LoggerFactory<minifi::extensions::python::processors::ExecutePythonProcessor>::getLogger(uuid)
+ });
+
execute_python_processor->setScriptFilePath(getScriptFullPath("stateful_processor.py").string());
+ auto execute_python_processor_unique_ptr =
std::make_unique<core::Processor>(execute_python_processor->getName(),
execute_python_processor->getUUID(), std::move(execute_python_processor));
+ plan_->addProcessor(std::move(execute_python_processor_unique_ptr),
"executePythonProcessor");
Review Comment:
this looks very similar to `addExecutePythonProcessorToPlan`; could we call
that instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
