[jira] [Comment Edited] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)

[James Medel (Jira)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842449#comment-17842449
 ] 

James Medel edited comment on MINIFICPP-2346 at 4/30/24 4:58 PM:
-

I think it may be beneficial to leverage conan at the beginning for installing 
MiNiFi C++ external lib dependencies (conan supports Linux and Windows 
packages). For my medical robotics company, I created an ezconan python wrapper 
tool around conan for speeding up our C++ cmake build infrastructure for our 
medical robot. I can build that code on Linux and Windows. I wanted to try 
adding support for conan to the MiNiFi C++ project too since I am looking at 
using MiNiFi C++ to develop edge data pipelines programmatically as alternative 
to the yml approach. As for the vcpkg, CMake can generate those build files 
when we run CMake generate on Windows. Similar for Makefiles, where we run 
CMake generate on Linux and CMake generates them too.

I will follow up on this thread with updates to my progress on conan for 
speeding up MiNiFi C++ builds [~szaszm] . I will keep you posted.

 

I am also considering creating a MiNiFI C++ conan package that I can smoothly 
integrate MiNiFi C++ lib into other C++ projects like in my medical imaging 
robot.


was (Author: james.medel):
I think it may be beneficial to leverage conan at the beginning for installing 
MiNiFi C++ external lib dependencies (conan supports Linux and Windows 
packages). For my medical robotics company, I created an ezconan python wrapper 
tool around conan for speeding up our C++ cmake build infrastructure for our 
medical robot. I can build that code on Linux and Windows. I wanted to try 
adding support for conan to the MiNiFi C++ project too since I am looking at 
using MiNiFi C++ to develop edge data pipelines programmatically as alternative 
to the yml approach. As for the vcpkg, CMake can generate those build files 
when we run CMake generate on Windows.

I will follow up on this thread with updates to my progress on conan for 
speeding up MiNiFi C++ builds [~szaszm] . I will keep you posted.

 

I am also considering creating a MiNiFI C++ conan package that I can smoothly 
integrate MiNiFi C++ lib into other C++ projects like in my medical imaging 
robot.

> Speed up MiNiFi Build with Conan package manager (C++)
> --
>
> Key: MINIFICPP-2346
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2346
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Improvement
>Affects Versions: 0.15.0
> Environment: Ubuntu 22.04
>Reporter: James Medel
>Assignee: James Medel
>Priority: Minor
>  Labels: build, conan, ubuntu
>
> *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. 
> During the the build process, it clones all the external dependencies and 
> then builds those external dependencies while its building MiNiFi CPP. 
> Sometimes the MiNiFi CPP build fails too when external lib dependencies fail 
> to download or fail to build. It would be faster to build MiNiFi C++ with the 
> external dependencies already preinstalled.
>  
> *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to 
> enable support for using *conan install* to install all or most of MiNiFi's 
> external dependencies, using *conan install --build=missing* to build them as 
> prebuilt binary conan packages, upload them to conancenter with {*}conan 
> upload{*}, and then run cmake generate to generate the appropriate build 
> files for the OS, then run make to build MiNiFi CPP. Also conan has really 
> good support for cross platform compatibility for Linux and Windows.  At this 
> point because we already have most of MiNiFi CPP's external lib dependencies 
> installed with conan, our CMake build will just focus on building the MiNiFi 
> CPP code, enabling faster builds. Also I will need to account for backward 
> compatibility support with the previous way of installing MiNiFi CPP external 
> lib dependencies using CMake FetchContent_Declare(...) approach until 
> conanfile.py approach covers installing all these dependencies. A valid 
> verification of the MiNiFi C++ build would be to run GTESTs and if all the 
> expected test cases pass, then it may be fair to conclude we would be ready 
> to deprecate the slow CMake FetchContent_Declare(...) approach.
>  
> *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into 
> script(s)):*
>  # Install conan version 2 into MiNiFi C++ dev environment (my preference is 
> using docker container)
>  # Create a conanfile.py file at root of MiNiFi C++ project where we run 
> "conan install" or "conan install --build=missing" to install MiNiFi C++ 
> external lib dependencies first. Then we run CMake to build MiNiFi C+

[jira] [Commented] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)

[James Medel (Jira)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842449#comment-17842449
 ] 

James Medel commented on MINIFICPP-2346:


I think it may be beneficial to leverage conan at the beginning for installing 
MiNiFi C++ external lib dependencies (conan supports Linux and Windows 
packages). For my medical robotics company, I created an ezconan python wrapper 
tool around conan for speeding up our C++ cmake build infrastructure for our 
medical robot. I can build that code on Linux and Windows. I wanted to try 
adding support for conan to the MiNiFi C++ project too since I am looking at 
using MiNiFi C++ to develop edge data pipelines programmatically as alternative 
to the yml approach. As for the vcpkg, CMake can generate those build files 
when we run CMake generate on Windows.

I will follow up on this thread with updates to my progress on conan for 
speeding up MiNiFi C++ builds [~szaszm] . I will keep you posted.

 

I am also considering creating a MiNiFI C++ conan package that I can smoothly 
integrate MiNiFi C++ lib into other C++ projects like in my medical imaging 
robot.

> Speed up MiNiFi Build with Conan package manager (C++)
> --
>
> Key: MINIFICPP-2346
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2346
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Improvement
>Affects Versions: 0.15.0
> Environment: Ubuntu 22.04
>Reporter: James Medel
>Assignee: James Medel
>Priority: Minor
>  Labels: build, conan, ubuntu
>
> *Description of Issue:* Building MiNiFi C++ with just CMake is really slow. 
> During the the build process, it clones all the external dependencies and 
> then builds those external dependencies while its building MiNiFi CPP. 
> Sometimes the MiNiFi CPP build fails too when external lib dependencies fail 
> to download or fail to build. It would be faster to build MiNiFi C++ with the 
> external dependencies already preinstalled.
>  
> *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to 
> enable support for using *conan install* to install all or most of MiNiFi's 
> external dependencies, using *conan install --build=missing* to build them as 
> prebuilt binary conan packages, upload them to conancenter with {*}conan 
> upload{*}, and then run cmake generate to generate the appropriate build 
> files for the OS, then run make to build MiNiFi CPP. Also conan has really 
> good support for cross platform compatibility for Linux and Windows.  At this 
> point because we already have most of MiNiFi CPP's external lib dependencies 
> installed with conan, our CMake build will just focus on building the MiNiFi 
> CPP code, enabling faster builds. Also I will need to account for backward 
> compatibility support with the previous way of installing MiNiFi CPP external 
> lib dependencies using CMake FetchContent_Declare(...) approach until 
> conanfile.py approach covers installing all these dependencies. A valid 
> verification of the MiNiFi C++ build would be to run GTESTs and if all the 
> expected test cases pass, then it may be fair to conclude we would be ready 
> to deprecate the slow CMake FetchContent_Declare(...) approach.
>  
> *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into 
> script(s)):*
>  # Install conan version 2 into MiNiFi C++ dev environment (my preference is 
> using docker container)
>  # Create a conanfile.py file at root of MiNiFi C++ project where we run 
> "conan install" or "conan install --build=missing" to install MiNiFi C++ 
> external lib dependencies first. Then we run CMake to build MiNiFi C++.
>  # Find all MiNiFi C++ find_package(...) CMake function calls and check if 
> there is already supported conan packages on the conancenter, so we can 
> install those packages.
>  # Find all FetchContent_Declare(...) CMake function calls and check if there 
> is already supported conan packages on the conancenter, so we can install 
> those packages.
>  # At this point we will have used conan version 2 C++ package manager to 
> install almost all MiNiFi C++'s external dependencies.
>  ## With Conan, we will have installed these MiNiFi external lib dependencies 
> as prebuilt binary conan packages.
>  ## Or we will have instructed conan to first build each C++ external lib 
> dependency, upload each lib to conancenter.
>  # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so 
> we can then run CMake to build MiNiFi faster.
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)

[James Medel (Jira)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Medel updated MINIFICPP-2346:
---
Description: 
*Description of Issue:* Building MiNiFi C++ with just CMake is really slow. 
During the the build process, it clones all the external dependencies and then 
builds those external dependencies while its building MiNiFi CPP. Sometimes the 
MiNiFi CPP build fails too when external lib dependencies fail to download or 
fail to build. It would be faster to build MiNiFi C++ with the external 
dependencies already preinstalled.

 

*Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to 
enable support for using *conan install* to install all or most of MiNiFi's 
external dependencies, using *conan install --build=missing* to build them as 
prebuilt binary conan packages, upload them to conancenter with {*}conan 
upload{*}, and then run cmake generate to generate the appropriate build files 
for the OS, then run make to build MiNiFi CPP. Also conan has really good 
support for cross platform compatibility for Linux and Windows.  At this point 
because we already have most of MiNiFi CPP's external lib dependencies 
installed with conan, our CMake build will just focus on building the MiNiFi 
CPP code, enabling faster builds. Also I will need to account for backward 
compatibility support with the previous way of installing MiNiFi CPP external 
lib dependencies using CMake FetchContent_Declare(...) approach until 
conanfile.py approach covers installing all these dependencies. A valid 
verification of the MiNiFi C++ build would be to run GTESTs and if all the 
expected test cases pass, then it may be fair to conclude we would be ready to 
deprecate the slow CMake FetchContent_Declare(...) approach.

 

*Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into 
script(s)):*
 # Install conan version 2 into MiNiFi C++ dev environment (my preference is 
using docker container)
 # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan 
install" or "conan install --build=missing" to install MiNiFi C++ external lib 
dependencies first. Then we run CMake to build MiNiFi C++.
 # Find all MiNiFi C++ find_package(...) CMake function calls and check if 
there is already supported conan packages on the conancenter, so we can install 
those packages.
 # Find all FetchContent_Declare(...) CMake function calls and check if there 
is already supported conan packages on the conancenter, so we can install those 
packages.
 # At this point we will have used conan version 2 C++ package manager to 
install almost all MiNiFi C++'s external dependencies.
 ## With Conan, we will have installed these MiNiFi external lib dependencies 
as prebuilt binary conan packages.
 ## Or we will have instructed conan to first build each C++ external lib 
dependency, upload each lib to conancenter.
 # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we 
can then run CMake to build MiNiFi faster.

 

 

  was:
*Description of Issue:* Building MiNiFi C++ with just CMake is really slow. 
During the the build process, it clones all the external dependencies and then 
builds those external dependencies while its building MiNiFi C++. Sometimes the 
MiNiFi C++ build fails too when external lib dependencies fail to download or 
fail to build. It would be faster to build MiNiFi C++ with the external 
dependencies already preinstalled.

 

*Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to 
enable support for using *conan install* to install all or most of MiNiFi's 
external dependencies, using *conan install --build=missing* to build them as 
prebuilt binary conan packages, upload them to conancenter with {*}conan 
upload{*}, and then run cmake generate to generate the appropriate build files 
for the OS, then run make to build MiNiFi C++. Also conan has really good 
support for cross platform compatibility for Linux and Windows.  At this point 
because we already  most of MiNiFi C's external dependencies with conan, our 
CMake build will just focus on building the MiNiFi C++ code.

 

*Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into 
script(s)):*
 # Install conan version 2 into MiNiFi C++ dev environment (my preference is 
using docker container)
 # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan 
install" or "conan install --build=missing" to install MiNiFi C++ external lib 
dependencies first. Then we run CMake to build MiNiFi C++.
 # Find all MiNiFi C++ find_package(...) CMake function calls and check if 
there is already supported conan packages on the conancenter, so we can install 
those packages.
 # Find all FetchContent_Declare(...) CMake function calls and check if there 
is already supported conan packages on the conancenter, so we can install those 
packages.
 # At this point w

[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)

[James Medel (Jira)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Medel updated MINIFICPP-2346:
---
Description: 
*Description of Issue:* Building MiNiFi C++ with just CMake is really slow. 
During the the build process, it clones all the external dependencies and then 
builds those external dependencies while its building MiNiFi C++. Sometimes the 
MiNiFi C++ build fails too when external lib dependencies fail to download or 
fail to build. It would be faster to build MiNiFi C++ with the external 
dependencies already preinstalled.

 

*Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to 
enable support for using *conan install* to install all or most of MiNiFi's 
external dependencies, using *conan install --build=missing* to build them as 
prebuilt binary conan packages, upload them to conancenter with {*}conan 
upload{*}, and then run cmake generate to generate the appropriate build files 
for the OS, then run make to build MiNiFi C++. Also conan has really good 
support for cross platform compatibility for Linux and Windows.  At this point 
because we already  most of MiNiFi C's external dependencies with conan, our 
CMake build will just focus on building the MiNiFi C++ code.

 

*Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into 
script(s)):*
 # Install conan version 2 into MiNiFi C++ dev environment (my preference is 
using docker container)
 # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan 
install" or "conan install --build=missing" to install MiNiFi C++ external lib 
dependencies first. Then we run CMake to build MiNiFi C++.
 # Find all MiNiFi C++ find_package(...) CMake function calls and check if 
there is already supported conan packages on the conancenter, so we can install 
those packages.
 # Find all FetchContent_Declare(...) CMake function calls and check if there 
is already supported conan packages on the conancenter, so we can install those 
packages.
 # At this point we will have used conan version 2 C++ package manager to 
install almost all MiNiFi C++'s external dependencies.
 ## With Conan, we will have installed these MiNiFi external lib dependencies 
as prebuilt binary conan packages.
 ## Or we will have instructed conan to first build each C++ external lib 
dependency, upload each lib to conancenter.
 # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we 
can then run CMake to build MiNiFi faster.

 

 

  was:
*Description of Issue:* Building MiNiFi C++ with just CMake is really slow. 
During the the build process, it clones all the external dependencies and then 
builds those external dependencies while its building MiNiFi C+{+}. It would be 
faster to build MiNiFi C{+}+ with the external dependencies already 
preinstalled.

 

*Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to 
enable support for using *conan install* to install all or most of MiNiFi's 
external dependencies, using *conan install --build=missing* to build them as 
prebuilt binary conan packages, upload them to conancenter with {*}conan 
upload{*}, and then run cmake generate to generate the appropriate build files 
for the OS, then run make to build MiNiFi C++. Also conan has really good 
support for cross platform compatibility for Linux and Windows.  At this point 
because we already  most of MiNiFi C's external dependencies with conan, our 
CMake build will just focus on building the MiNiFi C++ code.

 

*Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into 
script(s)):*
 # Install conan version 2 into MiNiFi C++ dev environment (my preference is 
using docker container)
 # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan 
install" or "conan install --build=missing" to install MiNiFi C++ external lib 
dependencies first. Then we run CMake to build MiNiFi C++.
 # Find all MiNiFi C++ find_package(...) CMake function calls and check if 
there is already supported conan packages on the conancenter, so we can install 
those packages.
 # Find all FetchContent_Declare(...) CMake function calls and check if there 
is already supported conan packages on the conancenter, so we can install those 
packages.
 # At this point we will have used conan version 2 C++ package manager to 
install almost all MiNiFi C++'s external dependencies.
 ## With Conan, we will have installed these MiNiFi external lib dependencies 
as prebuilt binary conan packages.
 ## Or we will have instructed conan to first build each C++ external lib 
dependency, upload each lib to conancenter.
 # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we 
can then run CMake to build MiNiFi faster.

 

 


> Speed up MiNiFi Build with Conan package manager (C++)
> --
>
> Key: MINIFICPP-2346

[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)

[James Medel (Jira)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Medel updated MINIFICPP-2346:
---
Description: 
*Description of Issue:* Building MiNiFi C++ with just CMake is really slow. 
During the the build process, it clones all the external dependencies and then 
builds those external dependencies while its building MiNiFi C+{+}. It would be 
faster to build MiNiFi C{+}+ with the external dependencies already 
preinstalled.

 

*Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to 
enable support for using *conan install* to install all or most of MiNiFi's 
external dependencies, using *conan install --build=missing* to build them as 
prebuilt binary conan packages, upload them to conancenter with {*}conan 
upload{*}, and then run cmake generate to generate the appropriate build files 
for the OS, then run make to build MiNiFi C++. Also conan has really good 
support for cross platform compatibility for Linux and Windows.  At this point 
because we already  most of MiNiFi C's external dependencies with conan, our 
CMake build will just focus on building the MiNiFi C++ code.

 

*Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into 
script(s)):*
 # Install conan version 2 into MiNiFi C++ dev environment (my preference is 
using docker container)
 # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan 
install" or "conan install --build=missing" to install MiNiFi C++ external lib 
dependencies first. Then we run CMake to build MiNiFi C++.
 # Find all MiNiFi C++ find_package(...) CMake function calls and check if 
there is already supported conan packages on the conancenter, so we can install 
those packages.
 # Find all FetchContent_Declare(...) CMake function calls and check if there 
is already supported conan packages on the conancenter, so we can install those 
packages.
 # At this point we will have used conan version 2 C++ package manager to 
install almost all MiNiFi C++'s external dependencies.
 ## With Conan, we will have installed these MiNiFi external lib dependencies 
as prebuilt binary conan packages.
 ## Or we will have instructed conan to first build each C++ external lib 
dependency, upload each lib to conancenter.
 # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we 
can then run CMake to build MiNiFi faster.

 

 

  was:
*Description of Issue:* Building MiNiFi C++ with just CMake is really slow. 
During the the build process, it clones all the external dependencies and then 
builds those external dependencies while its building MiNiFi C+{+}. It would be 
faster to build MiNiFi C{+}+ with the external dependencies already 
preinstalled.

 

*Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to 
enable support for using *conan install* to install all or most of MiNiFi's 
external dependencies, using *conan install --build=missing* to build them as 
prebuilt binary conan packages, upload them to conancenter with {*}conan 
upload{*}, and then run cmake generate to generate the appropriate build files 
for the OS, then run make to build MiNiFi C+{+}. At this point because we 
already  most of MiNiFi C{+}{+}'s external dependencies with conan, our CMake 
build will just focus on building the MiNiFi C{+}+ code.

 

*Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into 
script(s)):*
 # Install conan version 2 into MiNiFi C++ dev environment (my preference is 
using docker container)
 # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan 
install" or "conan install --build=missing" to install MiNiFi C++ external lib 
dependencies first. Then we run CMake to build MiNiFi C++.
 # Find all MiNiFi C++ find_package(...) CMake function calls and check if 
there is already supported conan packages on the conancenter, so we can install 
those packages.
 # Find all FetchContent_Declare(...) CMake function calls and check if there 
is already supported conan packages on the conancenter, so we can install those 
packages.
 # At this point we will have used conan version 2 C++ package manager to 
install almost all MiNiFi C++'s external dependencies.
 ## With Conan, we will have installed these MiNiFi external lib dependencies 
as prebuilt binary conan packages.
 ## Or we will have instructed conan to first build each C++ external lib 
dependency, upload each lib to conancenter.
 # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we 
can then run CMake to build MiNiFi faster.

 

 


> Speed up MiNiFi Build with Conan package manager (C++)
> --
>
> Key: MINIFICPP-2346
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2346
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Improvement
>Affects Versions: 0.15.0

[jira] [Updated] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)

[James Medel (Jira)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Medel updated MINIFICPP-2346:
---
Description: 
*Description of Issue:* Building MiNiFi C++ with just CMake is really slow. 
During the the build process, it clones all the external dependencies and then 
builds those external dependencies while its building MiNiFi C+{+}. It would be 
faster to build MiNiFi C{+}+ with the external dependencies already 
preinstalled.

 

*Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to 
enable support for using *conan install* to install all or most of MiNiFi's 
external dependencies, using *conan install --build=missing* to build them as 
prebuilt binary conan packages, upload them to conancenter with {*}conan 
upload{*}, and then run cmake generate to generate the appropriate build files 
for the OS, then run make to build MiNiFi C+{+}. At this point because we 
already  most of MiNiFi C{+}{+}'s external dependencies with conan, our CMake 
build will just focus on building the MiNiFi C{+}+ code.

 

*Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into 
script(s)):*
 # Install conan version 2 into MiNiFi C++ dev environment (my preference is 
using docker container)
 # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan 
install" or "conan install --build=missing" to install MiNiFi C++ external lib 
dependencies first. Then we run CMake to build MiNiFi C++.
 # Find all MiNiFi C++ find_package(...) CMake function calls and check if 
there is already supported conan packages on the conancenter, so we can install 
those packages.
 # Find all FetchContent_Declare(...) CMake function calls and check if there 
is already supported conan packages on the conancenter, so we can install those 
packages.
 # At this point we will have used conan version 2 C++ package manager to 
install almost all MiNiFi C++'s external dependencies.
 ## With Conan, we will have installed these MiNiFi external lib dependencies 
as prebuilt binary conan packages.
 ## Or we will have instructed conan to first build each C++ external lib 
dependency, upload each lib to conancenter.
 # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we 
can then run CMake to build MiNiFi faster.

 

 

  was:
*Description of Issue:* Building MiNiFi C++ with just CMake is really slow. 
During the the build process, it clones all the external dependencies and then 
builds those external dependencies while its building MiNiFi C++. It would be 
faster to build MiNiFi C++ with the external dependencies already preinstalled.

 

*Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to 
enable support for using *conan install* to install all or most of MiNiFi's 
external dependencies, using *conan install --build=missing* to build them as 
prebuilt binary conan packages, upload them to conancenter with {*}conan 
upload{*}, and then run cmake generate to generate the appropriate build files 
for the OS, then run make to build MiNiFi C++. At this point because we already 
 most of MiNiFi C++'s external dependencies with conan, our CMake build will 
just focus on building the MiNiFi C++ code.

 

*Steps to Perform MiNiFi C++ Build Enhancement:*
 # Install conan version 2 into MiNiFi C++ dev environment (my preference is 
using docker container)
 # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan 
install" or "conan install --build=missing" to install MiNiFi C++ external lib 
dependencies first. Then we run CMake to build MiNiFi C++.
 # Find all MiNiFi C++ find_package(...) CMake function calls and check if 
there is already supported conan packages on the conancenter, so we can install 
those packages.
 # Find all FetchContent_Declare(...) CMake function calls and check if there 
is already supported conan packages on the conancenter, so we can install those 
packages.
 # At this point we will have used conan version 2 C++ package manager to 
install almost all MiNiFi C++'s external dependencies.
 ## With Conan, we will have installed these MiNiFi external lib dependencies 
as prebuilt binary conan packages.
 ## Or we will have instructed conan to first build each C++ external lib 
dependency, upload each lib to conancenter.
 # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we 
can then run CMake to build MiNiFi faster.

 

 


> Speed up MiNiFi Build with Conan package manager (C++)
> --
>
> Key: MINIFICPP-2346
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2346
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Improvement
>Affects Versions: 0.15.0
> Environment: Ubuntu 22.04
>Reporter: James Medel
>Assignee: James Medel
>Priority: Min

[jira] [Created] (MINIFICPP-2346) Speed up MiNiFi Build with Conan package manager (C++)

[James Medel (Jira)]

James Medel created MINIFICPP-2346:
--

 Summary: Speed up MiNiFi Build with Conan package manager (C++)
 Key: MINIFICPP-2346
 URL: https://issues.apache.org/jira/browse/MINIFICPP-2346
 Project: Apache NiFi MiNiFi C++
  Issue Type: Improvement
Affects Versions: 0.15.0
 Environment: Ubuntu 22.04
Reporter: James Medel
Assignee: James Medel


*Description of Issue:* Building MiNiFi C++ with just CMake is really slow. 
During the the build process, it clones all the external dependencies and then 
builds those external dependencies while its building MiNiFi C++. It would be 
faster to build MiNiFi C++ with the external dependencies already preinstalled.

 

*Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to 
enable support for using *conan install* to install all or most of MiNiFi's 
external dependencies, using *conan install --build=missing* to build them as 
prebuilt binary conan packages, upload them to conancenter with {*}conan 
upload{*}, and then run cmake generate to generate the appropriate build files 
for the OS, then run make to build MiNiFi C++. At this point because we already 
 most of MiNiFi C++'s external dependencies with conan, our CMake build will 
just focus on building the MiNiFi C++ code.

 

*Steps to Perform MiNiFi C++ Build Enhancement:*
 # Install conan version 2 into MiNiFi C++ dev environment (my preference is 
using docker container)
 # Create a conanfile.py file at root of MiNiFi C++ project where we run "conan 
install" or "conan install --build=missing" to install MiNiFi C++ external lib 
dependencies first. Then we run CMake to build MiNiFi C++.
 # Find all MiNiFi C++ find_package(...) CMake function calls and check if 
there is already supported conan packages on the conancenter, so we can install 
those packages.
 # Find all FetchContent_Declare(...) CMake function calls and check if there 
is already supported conan packages on the conancenter, so we can install those 
packages.
 # At this point we will have used conan version 2 C++ package manager to 
install almost all MiNiFi C++'s external dependencies.
 ## With Conan, we will have installed these MiNiFi external lib dependencies 
as prebuilt binary conan packages.
 ## Or we will have instructed conan to first build each C++ external lib 
dependency, upload each lib to conancenter.
 # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so we 
can then run CMake to build MiNiFi faster.

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (MINIFICPP-1259) Create Python C++ Bindings for MiNiFi C++ Testing Framework

[James Medel (Jira)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17232551#comment-17232551
 ] 

James Medel commented on MINIFICPP-1259:


Hi [~oceanfish81], so after reviewing this ticket and the other one related 
MINIFICPP-1233, I decided to write a C++ test program, so I can get a program 
out to test my h2o processors sooner. Once I finish development for the C++ 
test program, then I will return to this ticket. I added more info on the C++ 
test program and my progress in MINIFICPP-1233.

> Create Python C++ Bindings for MiNiFi C++ Testing Framework
> ---
>
> Key: MINIFICPP-1259
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1259
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: James Medel
>Assignee: James Medel
>Priority: Blocker
> Fix For: 0.8.0
>
>
> Per https://issues.apache.org/jira/browse/MINIFICPP-1233
> After thinking about different ways to test my MiNiFi H2O Python Processors, 
> I thought of two scenarios:
>  * I test them using the MiNiFi C++ testing framework
>  * I test them using a hybrid MiNiFi C++ Python bindings testing framework. 
> The first approach already exists. With this Jira ticket, I will address the 
> second approach by creating Python C++ bindings to the MiNiFi testing 
> framework, so then users can use MiNiFi C++'s testController in Python to 
> programmatically build their MiNiFi flow and test their C++ and Python 
> processors. Thus, users will have the flexibility to test their processors 
> using Python or C++.
> Files that I will be working with include TestBase.h/.cpp, BuildTests.cmake 
> and more.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (MINIFICPP-1233) Create Regression Tests for H2O Py Processors

[James Medel (Jira)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17232549#comment-17232549
 ] 

James Medel commented on MINIFICPP-1233:


[~oceanfish81] [~szaszm]  [~phrocker] I have written a new C++ test program 
that is meant to test the h2o python processors. I also updated the 
CMakeLists.txt file with the new .cpp file, which I am calling 
"ExecuteH2oPythonProcessorTests.cpp". I also created an 
"ExecutePythonProcessorTestBase.hpp" from the original .cpp version, so I can 
include that file in my h2o test cpp file. Next I need to test this new test 
program and verify the tests pass. There are two tests since there are two h2o 
processors that I am testing. The first one tests ConvertDsToCsv.py processor's 
ability to convert a Data source format, in our case JSON to Csv. The second 
one tests ExecuteH2oMojoScoring.py processor's ability to perform a prediction. 
I plan to finish testing my program this upcoming week into the weekend.

> Create Regression Tests for H2O Py Processors
> -
>
> Key: MINIFICPP-1233
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1233
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: James Medel
>Assignee: James Medel
>Priority: Blocker
> Fix For: 0.8.0
>
> Attachments: CMakeLists.txt
>
>
> Per https://issues.apache.org/jira/browse/MINIFICPP-1214
> We should resolve Marc's and Szasz's concerns about whether 
> [https://github.com/apache/nifi-minifi-cpp/pull/784] breaks these two new 
> processors by having regression tests. We will explore and plan to use 
> pytest-mock ([https://github.com/pytest-dev/pytest-mock]) testing framework 
> to implement the regression tests for the new processors. Therefore, these 
> new processors would be easier to merge.
>  
> regression test concerns



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (MINIFICPP-1259) Create Python C++ Bindings for MiNiFi C++ Testing Framework

[James Medel (Jira)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17230758#comment-17230758
 ] 

James Medel commented on MINIFICPP-1259:


Hi [~oceanfish81]. Not yet. I do plan on returning to this ticket this week and 
have an update on it by end of sunday cdt.

> Create Python C++ Bindings for MiNiFi C++ Testing Framework
> ---
>
> Key: MINIFICPP-1259
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1259
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: James Medel
>Assignee: James Medel
>Priority: Blocker
> Fix For: 0.8.0
>
>
> Per https://issues.apache.org/jira/browse/MINIFICPP-1233
> After thinking about different ways to test my MiNiFi H2O Python Processors, 
> I thought of two scenarios:
>  * I test them using the MiNiFi C++ testing framework
>  * I test them using a hybrid MiNiFi C++ Python bindings testing framework. 
> The first approach already exists. With this Jira ticket, I will address the 
> second approach by creating Python C++ bindings to the MiNiFi testing 
> framework, so then users can use MiNiFi C++'s testController in Python to 
> programmatically build their MiNiFi flow and test their C++ and Python 
> processors. Thus, users will have the flexibility to test their processors 
> using Python or C++.
> Files that I will be working with include TestBase.h/.cpp, BuildTests.cmake 
> and more.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (MINIFICPP-1259) Create Python C++ Bindings for MiNiFi C++ Testing Framework

[James Medel (Jira)]

James Medel created MINIFICPP-1259:
--

 Summary: Create Python C++ Bindings for MiNiFi C++ Testing 
Framework
 Key: MINIFICPP-1259
 URL: https://issues.apache.org/jira/browse/MINIFICPP-1259
 Project: Apache NiFi MiNiFi C++
  Issue Type: Bug
Reporter: James Medel
Assignee: James Medel
 Fix For: 0.8.0


Per https://issues.apache.org/jira/browse/MINIFICPP-1233

After thinking about different ways to test my MiNiFi H2O Python Processors, I 
thought of two scenarios:
 * I test them using the MiNiFi C++ testing framework
 * I test them using a hybrid MiNiFi C++ Python bindings testing framework. 

The first approach already exists. With this Jira ticket, I will address the 
second approach by creating Python C++ bindings to the MiNiFi testing 
framework, so then users can use MiNiFi C++'s testController in Python to 
programmatically build their MiNiFi flow and test their C++ and Python 
processors. Thus, users will have the flexibility to test their processors 
using Python or C++.

Files that I will be working with include TestBase.h/.cpp, BuildTests.cmake and 
more.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (MINIFICPP-1233) Create Regression Tests for H2O Py Processors

[James Medel (Jira)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17128469#comment-17128469
 ] 

James Medel commented on MINIFICPP-1233:


[~phrocker] I would like to learn more about the alternative approach to 
testing by having stubs that run a minifi environment. Yes if you get some 
time, can you help me with some of the CMAKE changes?

Here is the CMakeLists.txt file that I have been working on.

[^CMakeLists.txt]

I have been working on two C++ test programs: one that tests ConvertDsToCsv py 
processor in a data flow and the other tests ExecuteH2oMojoScoring py processor 
in a data flow. I would like to hear more about the stubs approach though 
before I proceed further with the previous approach I was doing.

Also, when I try to sign up for apache.slack.com, I don't have a gxyster.com 
email. Who do I contact to get access to the apache slack?

> Create Regression Tests for H2O Py Processors
> -
>
> Key: MINIFICPP-1233
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1233
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: James Medel
>Assignee: James Medel
>Priority: Blocker
> Fix For: 0.8.0
>
> Attachments: CMakeLists.txt
>
>
> Per https://issues.apache.org/jira/browse/MINIFICPP-1214
> We should resolve Marc's and Szasz's concerns about whether 
> [https://github.com/apache/nifi-minifi-cpp/pull/784] breaks these two new 
> processors by having regression tests. We will explore and plan to use 
> pytest-mock ([https://github.com/pytest-dev/pytest-mock]) testing framework 
> to implement the regression tests for the new processors. Therefore, these 
> new processors would be easier to merge.
>  
> regression test concerns



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (MINIFICPP-1233) Create Regression Tests for H2O Py Processors

[James Medel (Jira)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Medel updated MINIFICPP-1233:
---
Attachment: CMakeLists.txt

> Create Regression Tests for H2O Py Processors
> -
>
> Key: MINIFICPP-1233
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1233
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: James Medel
>Assignee: James Medel
>Priority: Blocker
> Fix For: 0.8.0
>
> Attachments: CMakeLists.txt
>
>
> Per https://issues.apache.org/jira/browse/MINIFICPP-1214
> We should resolve Marc's and Szasz's concerns about whether 
> [https://github.com/apache/nifi-minifi-cpp/pull/784] breaks these two new 
> processors by having regression tests. We will explore and plan to use 
> pytest-mock ([https://github.com/pytest-dev/pytest-mock]) testing framework 
> to implement the regression tests for the new processors. Therefore, these 
> new processors would be easier to merge.
>  
> regression test concerns



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (MINIFICPP-1233) Create Regression Tests for H2O Py Processors

[James Medel (Jira)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17123429#comment-17123429
 ] 

James Medel commented on MINIFICPP-1233:


I have written the CMakeLists.txt file for my h2o processor tests. I am working 
on the C++ code for testing both h2o python processors. I will try to complete 
the code tomorrow. I will follow up with you in a day or two on how testing 
goes.

Two MiNiFi test programs I am using as references for writing my C++ test code:

PythonExecuteScriptTests.cpp: 
[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/script-tests/PythonExecuteScriptTests.cpp]

TensorFlowTests.cpp: 
[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/tensorflow-tests/TensorFlowTests.cpp]

> Create Regression Tests for H2O Py Processors
> -
>
> Key: MINIFICPP-1233
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1233
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: James Medel
>Assignee: James Medel
>Priority: Blocker
> Fix For: 0.8.0
>
>
> Per https://issues.apache.org/jira/browse/MINIFICPP-1214
> We should resolve Marc's and Szasz's concerns about whether 
> [https://github.com/apache/nifi-minifi-cpp/pull/784] breaks these two new 
> processors by having regression tests. We will explore and plan to use 
> pytest-mock ([https://github.com/pytest-dev/pytest-mock]) testing framework 
> to implement the regression tests for the new processors. Therefore, these 
> new processors would be easier to merge.
>  
> regression test concerns



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (MINIFICPP-1233) Create Regression Tests for H2O Py Processors

[James Medel (Jira)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Medel updated MINIFICPP-1233:
---
Summary: Create Regression Tests for H2O Py Processors  (was: Create 
Regression Tests for H2O Processors using pytest-mock)

> Create Regression Tests for H2O Py Processors
> -
>
> Key: MINIFICPP-1233
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1233
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: James Medel
>Assignee: James Medel
>Priority: Blocker
> Fix For: 0.8.0
>
>
> Per https://issues.apache.org/jira/browse/MINIFICPP-1214
> We should resolve Marc's and Szasz's concerns about whether 
> [https://github.com/apache/nifi-minifi-cpp/pull/784] breaks these two new 
> processors by having regression tests. We will explore and plan to use 
> pytest-mock ([https://github.com/pytest-dev/pytest-mock]) testing framework 
> to implement the regression tests for the new processors. Therefore, these 
> new processors would be easier to merge.
>  
> regression test concerns



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (MINIFICPP-1233) Create Regression Tests for H2O Processors using pytest-mock

[James Medel (Jira)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17119854#comment-17119854
 ] 

James Medel commented on MINIFICPP-1233:


[~aboda] I don't plan to test my processors without the C++ code. I was asking 
for guidance. 

I see two scenarios for testing my processors:
 * I test them using the MiNiFi C++ testing framework
 * I test them using a hybrid MiNiFi C++ Python bindings testing framework. 

I think the approach that would be quickest right now is scenario 1 where I use 
C++ to test my processors. So, I will go with approach 1 for now.

In the case of scenario 2, I would create a follow on ticket to testing the 
MiNiFi C++ Python processors where we could look at creating C++ Python 
bindings for the MiNiFi testing framework. For instance, C++ Python bindings 
are used for the ExecutePythonProcessor, so users can write their processors in 
Python. I think we could do the same with the MiNiFi testing framework, we 
create C++ Python Bindings, so users can write in Python to test their MiNiFi 
C++ Python processors. For new developers who contribute MiNiFi C++ Python 
processors, I think it would be faster for them to write test programs in 
Python while still having the power of C++ on the backend. 

> Create Regression Tests for H2O Processors using pytest-mock
> 
>
> Key: MINIFICPP-1233
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1233
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: James Medel
>Assignee: James Medel
>Priority: Blocker
> Fix For: 0.8.0
>
>
> Per https://issues.apache.org/jira/browse/MINIFICPP-1214
> We should resolve Marc's and Szasz's concerns about whether 
> [https://github.com/apache/nifi-minifi-cpp/pull/784] breaks these two new 
> processors by having regression tests. We will explore and plan to use 
> pytest-mock ([https://github.com/pytest-dev/pytest-mock]) testing framework 
> to implement the regression tests for the new processors. Therefore, these 
> new processors would be easier to merge.
>  
> regression test concerns



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (MINIFICPP-1233) Create Regression Tests for H2O Processors using pytest-mock

[James Medel (Jira)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17119716#comment-17119716
 ] 

James Medel edited comment on MINIFICPP-1233 at 5/29/20, 3:50 PM:
--

[~phrocker] I have examined some of the Test programs in 
*nifi-minifi-cpp/libminifi/test/* folder. For example, *TensorFlowTests.cpp* 
uses the C++ Catch Test library to create test cases for each TensorFlow C++ 
processor. Let's look at the first test case: *TEST_CASE("TensorFlow: Apply 
Graph", "[tfApplyGraph]")*. A testController object of type TestController is 
created to then create a plan object by calling testController's createPlan. 
Then the input files and output files are defined using strings. Then plan 
object is used to build the MiNiFi data flow aka processing graph, which 
includes adding the processors and setting their properties. They then build 
test TensorFlow graph, read TensorFlow graph into TFApplyGraph, write test 
input tensor and read test output tensor.

 

It looks like they are testing the TFApplyGraph C++ processor by building a 
MiNiFi flow in C++ code, providing input data and analyzing the expected output 
data.

 

Should I follow a similar approach for the h2o Python processors? I was looking 
into building and testing a hybrid Python/C++ program: *Building and testing a 
hybrid Python/C++ package*

Or just focus on testing the two individual h2o Python processors without 
building the MiNiFi flow in Python/C++ code?

References:
 * nifi-minifi-cpp/libminifi/test/: 
[https://github.com/apache/nifi-minifi-cpp/tree/master/libminifi/test]
 * TEST_CASE("TensorFlow: Apply Graph", "[tfApplyGraph]"): 
[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/tensorflow-tests/TensorFlowTests.cpp]
 * TestController: 
[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/TestBase.h]
 * Building and testing a hybrid Python/C++ package: 
[https://www.benjack.io/2017/06/12/python-cpp-tests.html]


was (Author: james.medel):
[~phrocker] I have examined some of the Test programs in 
*nifi-minifi-cpp/libminifi/test/* folder. For example, *TensorFlowTests.cpp* 
uses the C++ Catch Test library to create test cases for each TensorFlow C++ 
processor. Let's look at the first test case: *TEST_CASE("TensorFlow: Apply 
Graph", "[tfApplyGraph]")*. A testController object of type TestController is 
created to then create a plan object by calling testController's createPlan. 
Then the input files and output files are defined using strings. Then plan 
object is used to build the MiNiFi data flow aka processing graph, which 
includes adding the processors and setting their properties. They then build 
test TensorFlow graph, read TensorFlow graph into TFApplyGraph, write test 
input tensor and read test output tensor.

 

It looks like they are testing the TFApplyGraph C++ processor by building a 
MiNiFi flow in C++ code, providing input data and analyzing the expected output 
data.

 

Should I follow a similar approach for the h2o Python processors? I was looking 
into building and testing a hybrid Python/C++ program: *Building and testing a 
hybrid Python/C++ package*



Or just focus on testing the two individual h2o Python processors without 
building the MiNiFi flow in Python/C++ code?

References:

- nifi-minifi-cpp/libminifi/test/: 
[https://github.com/apache/nifi-minifi-cpp/tree/master/libminifi/test]

- TEST_CASE("TensorFlow: Apply Graph", "[tfApplyGraph]"): 
[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/tensorflow-tests/TensorFlowTests.cpp]

- TestController: 
[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/TestBase.h]

- Building and testing a hybrid Python/C++ package: 
[https://www.benjack.io/2017/06/12/python-cpp-tests.html]

> Create Regression Tests for H2O Processors using pytest-mock
> 
>
> Key: MINIFICPP-1233
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1233
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: James Medel
>Assignee: James Medel
>Priority: Blocker
> Fix For: 0.8.0
>
>
> Per https://issues.apache.org/jira/browse/MINIFICPP-1214
> We should resolve Marc's and Szasz's concerns about whether 
> [https://github.com/apache/nifi-minifi-cpp/pull/784] breaks these two new 
> processors by having regression tests. We will explore and plan to use 
> pytest-mock ([https://github.com/pytest-dev/pytest-mock]) testing framework 
> to implement the regression tests for the new processors. Therefore, these 
> new processors would be easier to merge.
>  
> regression test concerns



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (MINIFICPP-1233) Create Regression Tests for H2O Processors using pytest-mock

[James Medel (Jira)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17119716#comment-17119716
 ] 

James Medel edited comment on MINIFICPP-1233 at 5/29/20, 3:49 PM:
--

[~phrocker] I have examined some of the Test programs in 
*nifi-minifi-cpp/libminifi/test/* folder. For example, *TensorFlowTests.cpp* 
uses the C++ Catch Test library to create test cases for each TensorFlow C++ 
processor. Let's look at the first test case: *TEST_CASE("TensorFlow: Apply 
Graph", "[tfApplyGraph]")*. A testController object of type TestController is 
created to then create a plan object by calling testController's createPlan. 
Then the input files and output files are defined using strings. Then plan 
object is used to build the MiNiFi data flow aka processing graph, which 
includes adding the processors and setting their properties. They then build 
test TensorFlow graph, read TensorFlow graph into TFApplyGraph, write test 
input tensor and read test output tensor.

 

It looks like they are testing the TFApplyGraph C++ processor by building a 
MiNiFi flow in C++ code, providing input data and analyzing the expected output 
data.

 

Should I follow a similar approach for the h2o Python processors? I was looking 
into building and testing a hybrid Python/C++ program: *Building and testing a 
hybrid Python/C++ package*



Or just focus on testing the two individual h2o Python processors without 
building the MiNiFi flow in Python/C++ code?

References:

- nifi-minifi-cpp/libminifi/test/: 
[https://github.com/apache/nifi-minifi-cpp/tree/master/libminifi/test]

- TEST_CASE("TensorFlow: Apply Graph", "[tfApplyGraph]"): 
[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/tensorflow-tests/TensorFlowTests.cpp]

- TestController: 
[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/TestBase.h]

- Building and testing a hybrid Python/C++ package: 
[https://www.benjack.io/2017/06/12/python-cpp-tests.html]


was (Author: james.medel):
[~phrocker] I have examined some of the Test programs in 
[minifi-tests|[https://github.com/apache/nifi-minifi-cpp/tree/master/libminifi/test]]
 folder. For example, TensorFlowTests.cpp uses the C++ Catch Test library to 
create test cases for each TensorFlow C++ processor. Let's look at the first 
test case: [TEST_CASE("TensorFlow: Apply Graph", 
"[tfApplyGraph]")|[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/tensorflow-tests/TensorFlowTests.cpp]].
 A testController object of type 
[TestController|[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/TestBase.h]]
 is created to then create a plan object by calling testController's 
createPlan. Then the input files and output files are defined using strings. 
Then plan object is used to build the MiNiFi data flow aka processing graph, 
which includes adding the processors and setting their properties. They then 
build test TensorFlow graph, read TensorFlow graph into TFApplyGraph, write 
test input tensor and read test output tensor.

 

It looks like they are testing the TFApplyGraph C++ processor by building a 
MiNiFi flow in C++ code, providing input data and analyzing the expected output 
data.

 

Should I follow a similar approach for the h2o Python processors? I was looking 
into building and testing a hybrid Python/C++ program: 
[https://www.benjack.io/2017/06/12/python-cpp-tests.html]

 

Or just focus on testing the two individual h2o Python processors without 
building the MiNiFi flow in Python/C++ code?

> Create Regression Tests for H2O Processors using pytest-mock
> 
>
> Key: MINIFICPP-1233
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1233
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: James Medel
>Assignee: James Medel
>Priority: Blocker
> Fix For: 0.8.0
>
>
> Per https://issues.apache.org/jira/browse/MINIFICPP-1214
> We should resolve Marc's and Szasz's concerns about whether 
> [https://github.com/apache/nifi-minifi-cpp/pull/784] breaks these two new 
> processors by having regression tests. We will explore and plan to use 
> pytest-mock ([https://github.com/pytest-dev/pytest-mock]) testing framework 
> to implement the regression tests for the new processors. Therefore, these 
> new processors would be easier to merge.
>  
> regression test concerns



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (MINIFICPP-1233) Create Regression Tests for H2O Processors using pytest-mock

[James Medel (Jira)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17119716#comment-17119716
 ] 

James Medel edited comment on MINIFICPP-1233 at 5/29/20, 3:44 PM:
--

[~phrocker] I have examined some of the Test programs in 
[minifi-tests|[https://github.com/apache/nifi-minifi-cpp/tree/master/libminifi/test]]
 folder. For example, TensorFlowTests.cpp uses the C++ Catch Test library to 
create test cases for each TensorFlow C++ processor. Let's look at the first 
test case: [TEST_CASE("TensorFlow: Apply Graph", 
"[tfApplyGraph]")|[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/tensorflow-tests/TensorFlowTests.cpp]].
 A testController object of type 
[TestController|[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/TestBase.h]]
 is created to then create a plan object by calling testController's 
createPlan. Then the input files and output files are defined using strings. 
Then plan object is used to build the MiNiFi data flow aka processing graph, 
which includes adding the processors and setting their properties. They then 
build test TensorFlow graph, read TensorFlow graph into TFApplyGraph, write 
test input tensor and read test output tensor.

 

It looks like they are testing the TFApplyGraph C++ processor by building a 
MiNiFi flow in C++ code, providing input data and analyzing the expected output 
data.

 

Should I follow a similar approach for the h2o Python processors? I was looking 
into building and testing a hybrid Python/C++ program: 
[https://www.benjack.io/2017/06/12/python-cpp-tests.html]

 

Or just focus on testing the two individual h2o Python processors without 
building the MiNiFi flow in Python/C++ code?


was (Author: james.medel):
[~phrocker] I have examined some of the Test programs in 
[nifi-minifi-cpp/libminifi/test|[https://github.com/apache/nifi-minifi-cpp/tree/master/libminifi/test]]
 folder. For example, TensorFlowTests.cpp uses the C++ Catch Test library to 
create test cases for each TensorFlow C++ processor. Let's look at the first 
test case: [TEST_CASE("TensorFlow: Apply Graph", 
"[tfApplyGraph]")|[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/tensorflow-tests/TensorFlowTests.cpp]].
 A testController object of type 
[TestController|[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/TestBase.h]]
 is created to then create a plan object by calling testController's 
createPlan. Then the input files and output files are defined using strings. 
Then plan object is used to build the MiNiFi data flow aka processing graph, 
which includes adding the processors and setting their properties. They then 
build test TensorFlow graph, read TensorFlow graph into TFApplyGraph, write 
test input tensor and read test output tensor.

 

It looks like they are testing the TFApplyGraph C++ processor by building a 
MiNiFi flow in C++ code, providing input data and analyzing the expected output 
data.

 

Should I follow a similar approach for the h2o Python processors? I was looking 
into building and testing a hybrid Python/C++ program: 
[https://www.benjack.io/2017/06/12/python-cpp-tests.html]

 

Or just focus on testing the two individual h2o Python processors without 
building the MiNiFi flow in Python/C++ code?

> Create Regression Tests for H2O Processors using pytest-mock
> 
>
> Key: MINIFICPP-1233
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1233
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: James Medel
>Assignee: James Medel
>Priority: Blocker
> Fix For: 0.8.0
>
>
> Per https://issues.apache.org/jira/browse/MINIFICPP-1214
> We should resolve Marc's and Szasz's concerns about whether 
> [https://github.com/apache/nifi-minifi-cpp/pull/784] breaks these two new 
> processors by having regression tests. We will explore and plan to use 
> pytest-mock ([https://github.com/pytest-dev/pytest-mock]) testing framework 
> to implement the regression tests for the new processors. Therefore, these 
> new processors would be easier to merge.
>  
> regression test concerns



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (MINIFICPP-1233) Create Regression Tests for H2O Processors using pytest-mock

[James Medel (Jira)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17119716#comment-17119716
 ] 

James Medel edited comment on MINIFICPP-1233 at 5/29/20, 3:42 PM:
--

[~phrocker] I have examined some of the Test programs in 
[nifi-minifi-cpp/libminifi/test|[https://github.com/apache/nifi-minifi-cpp/tree/master/libminifi/test]]
 folder. For example, TensorFlowTests.cpp uses the C++ Catch Test library to 
create test cases for each TensorFlow C++ processor. Let's look at the first 
test case: [TEST_CASE("TensorFlow: Apply Graph", 
"[tfApplyGraph]")|[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/tensorflow-tests/TensorFlowTests.cpp]].
 A testController object of type 
[TestController|[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/TestBase.h]]
 is created to then create a plan object by calling testController's 
createPlan. Then the input files and output files are defined using strings. 
Then plan object is used to build the MiNiFi data flow aka processing graph, 
which includes adding the processors and setting their properties. They then 
build test TensorFlow graph, read TensorFlow graph into TFApplyGraph, write 
test input tensor and read test output tensor.

 

It looks like they are testing the TFApplyGraph C++ processor by building a 
MiNiFi flow in C++ code, providing input data and analyzing the expected output 
data.

 

Should I follow a similar approach for the h2o Python processors? I was looking 
into building and testing a hybrid Python/C++ program: 
[https://www.benjack.io/2017/06/12/python-cpp-tests.html]

 

Or just focus on testing the two individual h2o Python processors without 
building the MiNiFi flow in Python/C++ code?


was (Author: james.medel):
[~phrocker] I have examined some of the Test programs in

 

[minifi 
tests|[https://github.com/apache/nifi-minifi-cpp/tree/master/libminifi/test]]

folder. For example, TensorFlowTests.cpp uses the C++ Catch Test library to 
create test cases for each TensorFlow C++ processor. Let's look at the first 
test case: [TEST_CASE("TensorFlow: Apply Graph", 
"[tfApplyGraph]")|[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/tensorflow-tests/TensorFlowTests.cpp]].
 A testController object of type 
[TestController|[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/TestBase.h]]
 is created to then create a plan object by calling testController's 
createPlan. Then the input files and output files are defined using strings. 
Then plan object is used to build the MiNiFi data flow aka processing graph, 
which includes adding the processors and setting their properties. They then 
build test TensorFlow graph, read TensorFlow graph into TFApplyGraph, write 
test input tensor and read test output tensor.

 

It looks like they are testing the TFApplyGraph C++ processor by building a 
MiNiFi flow in C++ code, providing input data and analyzing the expected output 
data.

 

Should I follow a similar approach for the h2o Python processors? I was looking 
into building and testing a hybrid Python/C++ program: 
[https://www.benjack.io/2017/06/12/python-cpp-tests.html]

 

Or just focus on testing the two individual h2o Python processors without 
building the MiNiFi flow in Python/C++ code?

> Create Regression Tests for H2O Processors using pytest-mock
> 
>
> Key: MINIFICPP-1233
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1233
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: James Medel
>Assignee: James Medel
>Priority: Blocker
> Fix For: 0.8.0
>
>
> Per https://issues.apache.org/jira/browse/MINIFICPP-1214
> We should resolve Marc's and Szasz's concerns about whether 
> [https://github.com/apache/nifi-minifi-cpp/pull/784] breaks these two new 
> processors by having regression tests. We will explore and plan to use 
> pytest-mock ([https://github.com/pytest-dev/pytest-mock]) testing framework 
> to implement the regression tests for the new processors. Therefore, these 
> new processors would be easier to merge.
>  
> regression test concerns



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (MINIFICPP-1233) Create Regression Tests for H2O Processors using pytest-mock

[James Medel (Jira)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17119716#comment-17119716
 ] 

James Medel edited comment on MINIFICPP-1233 at 5/29/20, 3:41 PM:
--

[~phrocker] I have examined some of the Test programs in

 

[minifi 
tests|[https://github.com/apache/nifi-minifi-cpp/tree/master/libminifi/test]]

folder. For example, TensorFlowTests.cpp uses the C++ Catch Test library to 
create test cases for each TensorFlow C++ processor. Let's look at the first 
test case: [TEST_CASE("TensorFlow: Apply Graph", 
"[tfApplyGraph]")|[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/tensorflow-tests/TensorFlowTests.cpp]].
 A testController object of type 
[TestController|[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/TestBase.h]]
 is created to then create a plan object by calling testController's 
createPlan. Then the input files and output files are defined using strings. 
Then plan object is used to build the MiNiFi data flow aka processing graph, 
which includes adding the processors and setting their properties. They then 
build test TensorFlow graph, read TensorFlow graph into TFApplyGraph, write 
test input tensor and read test output tensor.

 

It looks like they are testing the TFApplyGraph C++ processor by building a 
MiNiFi flow in C++ code, providing input data and analyzing the expected output 
data.

 

Should I follow a similar approach for the h2o Python processors? I was looking 
into building and testing a hybrid Python/C++ program: 
[https://www.benjack.io/2017/06/12/python-cpp-tests.html]

 

Or just focus on testing the two individual h2o Python processors without 
building the MiNiFi flow in Python/C++ code?


was (Author: james.medel):
[~phrocker] I have examined some of the Test programs in

 

["nifi-minifi-cpp/libminifi/test/"|[https://github.com/apache/nifi-minifi-cpp/tree/master/libminifi/test]]

folder. For example, TensorFlowTests.cpp uses the C++ Catch Test library to 
create test cases for each TensorFlow C++ processor. Let's look at the first 
test case: [TEST_CASE("TensorFlow: Apply Graph", 
"[tfApplyGraph]")|[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/tensorflow-tests/TensorFlowTests.cpp]].
 A testController object of type 
[TestController|[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/TestBase.h]]
 is created to then create a plan object by calling testController's 
createPlan. Then the input files and output files are defined using strings. 
Then plan object is used to build the MiNiFi data flow aka processing graph, 
which includes adding the processors and setting their properties. They then 
build test TensorFlow graph, read TensorFlow graph into TFApplyGraph, write 
test input tensor and read test output tensor.

 

It looks like they are testing the TFApplyGraph C++ processor by building a 
MiNiFi flow in C++ code, providing input data and analyzing the expected output 
data.

 

Should I follow a similar approach for the h2o Python processors? I was looking 
into building and testing a hybrid Python/C++ program: 
[https://www.benjack.io/2017/06/12/python-cpp-tests.html]

 

Or just focus on testing the two individual h2o Python processors without 
building the MiNiFi flow in Python/C++ code?

> Create Regression Tests for H2O Processors using pytest-mock
> 
>
> Key: MINIFICPP-1233
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1233
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: James Medel
>Assignee: James Medel
>Priority: Blocker
> Fix For: 0.8.0
>
>
> Per https://issues.apache.org/jira/browse/MINIFICPP-1214
> We should resolve Marc's and Szasz's concerns about whether 
> [https://github.com/apache/nifi-minifi-cpp/pull/784] breaks these two new 
> processors by having regression tests. We will explore and plan to use 
> pytest-mock ([https://github.com/pytest-dev/pytest-mock]) testing framework 
> to implement the regression tests for the new processors. Therefore, these 
> new processors would be easier to merge.
>  
> regression test concerns



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (MINIFICPP-1233) Create Regression Tests for H2O Processors using pytest-mock

[James Medel (Jira)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17119716#comment-17119716
 ] 

James Medel edited comment on MINIFICPP-1233 at 5/29/20, 3:40 PM:
--

[~phrocker] I have examined some of the Test programs in

 

["nifi-minifi-cpp/libminifi/test/"|[https://github.com/apache/nifi-minifi-cpp/tree/master/libminifi/test]]

folder. For example, TensorFlowTests.cpp uses the C++ Catch Test library to 
create test cases for each TensorFlow C++ processor. Let's look at the first 
test case: [TEST_CASE("TensorFlow: Apply Graph", 
"[tfApplyGraph]")|[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/tensorflow-tests/TensorFlowTests.cpp]].
 A testController object of type 
[TestController|[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/TestBase.h]]
 is created to then create a plan object by calling testController's 
createPlan. Then the input files and output files are defined using strings. 
Then plan object is used to build the MiNiFi data flow aka processing graph, 
which includes adding the processors and setting their properties. They then 
build test TensorFlow graph, read TensorFlow graph into TFApplyGraph, write 
test input tensor and read test output tensor.

 

It looks like they are testing the TFApplyGraph C++ processor by building a 
MiNiFi flow in C++ code, providing input data and analyzing the expected output 
data.

 

Should I follow a similar approach for the h2o Python processors? I was looking 
into building and testing a hybrid Python/C++ program: 
[https://www.benjack.io/2017/06/12/python-cpp-tests.html]

 

Or just focus on testing the two individual h2o Python processors without 
building the MiNiFi flow in Python/C++ code?


was (Author: james.medel):
[~phrocker] I have examined some of the Test programs in

[nifi-minifi-cpp/libminifi/test/|[https://github.com/apache/nifi-minifi-cpp/tree/master/libminifi/test]]

folder. For example, TensorFlowTests.cpp uses the C++ Catch Test library to 
create test cases for each TensorFlow C++ processor. Let's look at the first 
test case: [TEST_CASE("TensorFlow: Apply Graph", 
"[tfApplyGraph]")|[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/tensorflow-tests/TensorFlowTests.cpp]].
 A testController object of type 
[TestController|[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/TestBase.h]]
 is created to then create a plan object by calling testController's 
createPlan. Then the input files and output files are defined using strings. 
Then plan object is used to build the MiNiFi data flow aka processing graph, 
which includes adding the processors and setting their properties. They then 
build test TensorFlow graph, read TensorFlow graph into TFApplyGraph, write 
test input tensor and read test output tensor.

 

It looks like they are testing the TFApplyGraph C++ processor by building a 
MiNiFi flow in C++ code, providing input data and analyzing the expected output 
data.

 

Should I follow a similar approach for the h2o Python processors? I was looking 
into building and testing a hybrid Python/C++ program: 
[https://www.benjack.io/2017/06/12/python-cpp-tests.html]

 

Or just focus on testing the two individual h2o Python processors without 
building the MiNiFi flow in Python/C++ code?

> Create Regression Tests for H2O Processors using pytest-mock
> 
>
> Key: MINIFICPP-1233
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1233
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: James Medel
>Assignee: James Medel
>Priority: Blocker
> Fix For: 0.8.0
>
>
> Per https://issues.apache.org/jira/browse/MINIFICPP-1214
> We should resolve Marc's and Szasz's concerns about whether 
> [https://github.com/apache/nifi-minifi-cpp/pull/784] breaks these two new 
> processors by having regression tests. We will explore and plan to use 
> pytest-mock ([https://github.com/pytest-dev/pytest-mock]) testing framework 
> to implement the regression tests for the new processors. Therefore, these 
> new processors would be easier to merge.
>  
> regression test concerns



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (MINIFICPP-1233) Create Regression Tests for H2O Processors using pytest-mock

[James Medel (Jira)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17119716#comment-17119716
 ] 

James Medel edited comment on MINIFICPP-1233 at 5/29/20, 3:40 PM:
--

[~phrocker] I have examined some of the Test programs in

[nifi-minifi-cpp/libminifi/test/|[https://github.com/apache/nifi-minifi-cpp/tree/master/libminifi/test]]

folder. For example, TensorFlowTests.cpp uses the C++ Catch Test library to 
create test cases for each TensorFlow C++ processor. Let's look at the first 
test case: [TEST_CASE("TensorFlow: Apply Graph", 
"[tfApplyGraph]")|[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/tensorflow-tests/TensorFlowTests.cpp]].
 A testController object of type 
[TestController|[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/TestBase.h]]
 is created to then create a plan object by calling testController's 
createPlan. Then the input files and output files are defined using strings. 
Then plan object is used to build the MiNiFi data flow aka processing graph, 
which includes adding the processors and setting their properties. They then 
build test TensorFlow graph, read TensorFlow graph into TFApplyGraph, write 
test input tensor and read test output tensor.

 

It looks like they are testing the TFApplyGraph C++ processor by building a 
MiNiFi flow in C++ code, providing input data and analyzing the expected output 
data.

 

Should I follow a similar approach for the h2o Python processors? I was looking 
into building and testing a hybrid Python/C++ program: 
[https://www.benjack.io/2017/06/12/python-cpp-tests.html]

 

Or just focus on testing the two individual h2o Python processors without 
building the MiNiFi flow in Python/C++ code?


was (Author: james.medel):
[~phrocker] I have examined some of the Test programs in

[nifi-minifi-cpp/libminifi/test/ | 
[https://github.com/apache/nifi-minifi-cpp/tree/master/libminifi/test]]

folder. For example, TensorFlowTests.cpp uses the C++ Catch Test library to 
create test cases for each TensorFlow C++ processor. Let's look at the first 
test case: [TEST_CASE("TensorFlow: Apply Graph", 
"[tfApplyGraph]")|[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/tensorflow-tests/TensorFlowTests.cpp]].
 A testController object of type 
[TestController|[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/TestBase.h]]
 is created to then create a plan object by calling testController's 
createPlan. Then the input files and output files are defined using strings. 
Then plan object is used to build the MiNiFi data flow aka processing graph, 
which includes adding the processors and setting their properties. They then 
build test TensorFlow graph, read TensorFlow graph into TFApplyGraph, write 
test input tensor and read test output tensor.

 

It looks like they are testing the TFApplyGraph C++ processor by building a 
MiNiFi flow in C++ code, providing input data and analyzing the expected output 
data.

 

Should I follow a similar approach for the h2o Python processors? I was looking 
into building and testing a hybrid Python/C++ program: 
[https://www.benjack.io/2017/06/12/python-cpp-tests.html]

 

Or just focus on testing the two individual h2o Python processors without 
building the MiNiFi flow in Python/C++ code?

> Create Regression Tests for H2O Processors using pytest-mock
> 
>
> Key: MINIFICPP-1233
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1233
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: James Medel
>Assignee: James Medel
>Priority: Blocker
> Fix For: 0.8.0
>
>
> Per https://issues.apache.org/jira/browse/MINIFICPP-1214
> We should resolve Marc's and Szasz's concerns about whether 
> [https://github.com/apache/nifi-minifi-cpp/pull/784] breaks these two new 
> processors by having regression tests. We will explore and plan to use 
> pytest-mock ([https://github.com/pytest-dev/pytest-mock]) testing framework 
> to implement the regression tests for the new processors. Therefore, these 
> new processors would be easier to merge.
>  
> regression test concerns



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (MINIFICPP-1233) Create Regression Tests for H2O Processors using pytest-mock

[James Medel (Jira)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17119716#comment-17119716
 ] 

James Medel edited comment on MINIFICPP-1233 at 5/29/20, 3:39 PM:
--

[~phrocker] I have examined some of the Test programs in 
[nifi-minifi-cpp/libminifi/test/ | 
[https://github.com/apache/nifi-minifi-cpp/tree/master/libminifi/test]] folder. 
For example, TensorFlowTests.cpp uses the C++ Catch Test library to create test 
cases for each TensorFlow C++ processor. Let's look at the first test case: 
[TEST_CASE("TensorFlow: Apply Graph", 
"[tfApplyGraph]")|[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/tensorflow-tests/TensorFlowTests.cpp]].
 A testController object of type 
[TestController|[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/TestBase.h]]
 is created to then create a plan object by calling testController's 
createPlan. Then the input files and output files are defined using strings. 
Then plan object is used to build the MiNiFi data flow aka processing graph, 
which includes adding the processors and setting their properties. They then 
build test TensorFlow graph, read TensorFlow graph into TFApplyGraph, write 
test input tensor and read test output tensor.

 

It looks like they are testing the TFApplyGraph C++ processor by building a 
MiNiFi flow in C++ code, providing input data and analyzing the expected output 
data.

 

Should I follow a similar approach for the h2o Python processors? I was looking 
into building and testing a hybrid Python/C++ program: 
[https://www.benjack.io/2017/06/12/python-cpp-tests.html]

 

Or just focus on testing the two individual h2o Python processors without 
building the MiNiFi flow in Python/C++ code?


was (Author: james.medel):
[~phrocker] I have examined some of the Test programs in 
[nifi-minifi-cpp/libminifi/test/|[https://github.com/apache/nifi-minifi-cpp/tree/master/libminifi/test]]
 folder. For example, TensorFlowTests.cpp uses the C++ Catch Test library to 
create test cases for each TensorFlow C++ processor. Let's look at the first 
test case: [TEST_CASE("TensorFlow: Apply Graph", 
"[tfApplyGraph]")|[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/tensorflow-tests/TensorFlowTests.cpp]].
 A testController object of type 
[TestController|[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/TestBase.h]]
 is created to then create a plan object by calling testController's 
createPlan. Then the input files and output files are defined using strings. 
Then plan object is used to build the MiNiFi data flow aka processing graph, 
which includes adding the processors and setting their properties. They then 
build test TensorFlow graph, read TensorFlow graph into TFApplyGraph, write 
test input tensor and read test output tensor.

 

It looks like they are testing the TFApplyGraph C++ processor by building a 
MiNiFi flow in C++ code, providing input data and analyzing the expected output 
data.

 

Should I follow a similar approach for the h2o Python processors? I was looking 
into building and testing a hybrid Python/C++ program: 
[https://www.benjack.io/2017/06/12/python-cpp-tests.html]

 

Or just focus on testing the two individual h2o Python processors without 
building the MiNiFi flow in Python/C++ code?

> Create Regression Tests for H2O Processors using pytest-mock
> 
>
> Key: MINIFICPP-1233
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1233
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: James Medel
>Assignee: James Medel
>Priority: Blocker
> Fix For: 0.8.0
>
>
> Per https://issues.apache.org/jira/browse/MINIFICPP-1214
> We should resolve Marc's and Szasz's concerns about whether 
> [https://github.com/apache/nifi-minifi-cpp/pull/784] breaks these two new 
> processors by having regression tests. We will explore and plan to use 
> pytest-mock ([https://github.com/pytest-dev/pytest-mock]) testing framework 
> to implement the regression tests for the new processors. Therefore, these 
> new processors would be easier to merge.
>  
> regression test concerns



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (MINIFICPP-1233) Create Regression Tests for H2O Processors using pytest-mock

[James Medel (Jira)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17119716#comment-17119716
 ] 

James Medel commented on MINIFICPP-1233:


[~phrocker] I have examined some of the Test programs in 
[nifi-minifi-cpp/libminifi/test/|[https://github.com/apache/nifi-minifi-cpp/tree/master/libminifi/test]]
 folder. For example, TensorFlowTests.cpp uses the C++ Catch Test library to 
create test cases for each TensorFlow C++ processor. Let's look at the first 
test case: [TEST_CASE("TensorFlow: Apply Graph", 
"[tfApplyGraph]")|[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/tensorflow-tests/TensorFlowTests.cpp]].
 A testController object of type 
[TestController|[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/TestBase.h]]
 is created to then create a plan object by calling testController's 
createPlan. Then the input files and output files are defined using strings. 
Then plan object is used to build the MiNiFi data flow aka processing graph, 
which includes adding the processors and setting their properties. They then 
build test TensorFlow graph, read TensorFlow graph into TFApplyGraph, write 
test input tensor and read test output tensor.

 

It looks like they are testing the TFApplyGraph C++ processor by building a 
MiNiFi flow in C++ code, providing input data and analyzing the expected output 
data.

 

Should I follow a similar approach for the h2o Python processors? I was looking 
into building and testing a hybrid Python/C++ program: 
[https://www.benjack.io/2017/06/12/python-cpp-tests.html]

 

Or just focus on testing the two individual h2o Python processors without 
building the MiNiFi flow in Python/C++ code?

> Create Regression Tests for H2O Processors using pytest-mock
> 
>
> Key: MINIFICPP-1233
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1233
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: James Medel
>Assignee: James Medel
>Priority: Blocker
> Fix For: 0.8.0
>
>
> Per https://issues.apache.org/jira/browse/MINIFICPP-1214
> We should resolve Marc's and Szasz's concerns about whether 
> [https://github.com/apache/nifi-minifi-cpp/pull/784] breaks these two new 
> processors by having regression tests. We will explore and plan to use 
> pytest-mock ([https://github.com/pytest-dev/pytest-mock]) testing framework 
> to implement the regression tests for the new processors. Therefore, these 
> new processors would be easier to merge.
>  
> regression test concerns



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (MINIFICPP-1233) Create Regression Tests for H2O Processors using pytest-mock

[James Medel (Jira)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17119716#comment-17119716
 ] 

James Medel edited comment on MINIFICPP-1233 at 5/29/20, 3:39 PM:
--

[~phrocker] I have examined some of the Test programs in

[nifi-minifi-cpp/libminifi/test/ | 
[https://github.com/apache/nifi-minifi-cpp/tree/master/libminifi/test]]

folder. For example, TensorFlowTests.cpp uses the C++ Catch Test library to 
create test cases for each TensorFlow C++ processor. Let's look at the first 
test case: [TEST_CASE("TensorFlow: Apply Graph", 
"[tfApplyGraph]")|[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/tensorflow-tests/TensorFlowTests.cpp]].
 A testController object of type 
[TestController|[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/TestBase.h]]
 is created to then create a plan object by calling testController's 
createPlan. Then the input files and output files are defined using strings. 
Then plan object is used to build the MiNiFi data flow aka processing graph, 
which includes adding the processors and setting their properties. They then 
build test TensorFlow graph, read TensorFlow graph into TFApplyGraph, write 
test input tensor and read test output tensor.

 

It looks like they are testing the TFApplyGraph C++ processor by building a 
MiNiFi flow in C++ code, providing input data and analyzing the expected output 
data.

 

Should I follow a similar approach for the h2o Python processors? I was looking 
into building and testing a hybrid Python/C++ program: 
[https://www.benjack.io/2017/06/12/python-cpp-tests.html]

 

Or just focus on testing the two individual h2o Python processors without 
building the MiNiFi flow in Python/C++ code?


was (Author: james.medel):
[~phrocker] I have examined some of the Test programs in 
[nifi-minifi-cpp/libminifi/test/ | 
[https://github.com/apache/nifi-minifi-cpp/tree/master/libminifi/test]] folder. 
For example, TensorFlowTests.cpp uses the C++ Catch Test library to create test 
cases for each TensorFlow C++ processor. Let's look at the first test case: 
[TEST_CASE("TensorFlow: Apply Graph", 
"[tfApplyGraph]")|[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/tensorflow-tests/TensorFlowTests.cpp]].
 A testController object of type 
[TestController|[https://github.com/apache/nifi-minifi-cpp/blob/master/libminifi/test/TestBase.h]]
 is created to then create a plan object by calling testController's 
createPlan. Then the input files and output files are defined using strings. 
Then plan object is used to build the MiNiFi data flow aka processing graph, 
which includes adding the processors and setting their properties. They then 
build test TensorFlow graph, read TensorFlow graph into TFApplyGraph, write 
test input tensor and read test output tensor.

 

It looks like they are testing the TFApplyGraph C++ processor by building a 
MiNiFi flow in C++ code, providing input data and analyzing the expected output 
data.

 

Should I follow a similar approach for the h2o Python processors? I was looking 
into building and testing a hybrid Python/C++ program: 
[https://www.benjack.io/2017/06/12/python-cpp-tests.html]

 

Or just focus on testing the two individual h2o Python processors without 
building the MiNiFi flow in Python/C++ code?

> Create Regression Tests for H2O Processors using pytest-mock
> 
>
> Key: MINIFICPP-1233
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1233
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: James Medel
>Assignee: James Medel
>Priority: Blocker
> Fix For: 0.8.0
>
>
> Per https://issues.apache.org/jira/browse/MINIFICPP-1214
> We should resolve Marc's and Szasz's concerns about whether 
> [https://github.com/apache/nifi-minifi-cpp/pull/784] breaks these two new 
> processors by having regression tests. We will explore and plan to use 
> pytest-mock ([https://github.com/pytest-dev/pytest-mock]) testing framework 
> to implement the regression tests for the new processors. Therefore, these 
> new processors would be easier to merge.
>  
> regression test concerns



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (MINIFICPP-1233) Create Regression Tests for H2O Processors using pytest-mock

[James Medel (Jira)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17113520#comment-17113520
 ] 

James Medel commented on MINIFICPP-1233:


[~szaszm] Thank you for your input. I will look at the 
libminifi/test/integration (and a handful of extension tests) as I work on 
writing the test. I will add a comment on GitHub and here once I have a test 
for the two new H2O processors.

> Create Regression Tests for H2O Processors using pytest-mock
> 
>
> Key: MINIFICPP-1233
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1233
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: James Medel
>Assignee: James Medel
>Priority: Blocker
> Fix For: 0.8.0
>
>
> Per https://issues.apache.org/jira/browse/MINIFICPP-1214
> We should resolve Marc's and Szasz's concerns about whether 
> [https://github.com/apache/nifi-minifi-cpp/pull/784] breaks these two new 
> processors by having regression tests. We will explore and plan to use 
> pytest-mock ([https://github.com/pytest-dev/pytest-mock]) testing framework 
> to implement the regression tests for the new processors. Therefore, these 
> new processors would be easier to merge.
>  
> regression test concerns



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (MINIFICPP-1233) Create Regression Tests for H2O Processors using pytest-mock

[James Medel (Jira)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Medel updated MINIFICPP-1233:
---
Summary: Create Regression Tests for H2O Processors using pytest-mock  
(was: Create Regression Tests for H2O-3 Processors using pytest-mock)

> Create Regression Tests for H2O Processors using pytest-mock
> 
>
> Key: MINIFICPP-1233
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1233
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: James Medel
>Assignee: James Medel
>Priority: Blocker
> Fix For: 0.8.0
>
>
> Per https://issues.apache.org/jira/browse/MINIFICPP-1214
> We should resolve Marc's and Szasz's concerns about whether 
> [https://github.com/apache/nifi-minifi-cpp/pull/784] breaks these two new 
> processors by having regression tests. We will explore and plan to use 
> pytest-mock ([https://github.com/pytest-dev/pytest-mock]) testing framework 
> to implement the regression tests for the new processors. Therefore, these 
> new processors would be easier to merge.
>  
> regression test concerns



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (MINIFICPP-1233) Create Regression Tests for H2O-3 Processors using pytest-mock

[James Medel (Jira)]

James Medel created MINIFICPP-1233:
--

 Summary: Create Regression Tests for H2O-3 Processors using 
pytest-mock
 Key: MINIFICPP-1233
 URL: https://issues.apache.org/jira/browse/MINIFICPP-1233
 Project: Apache NiFi MiNiFi C++
  Issue Type: Bug
Reporter: James Medel
Assignee: James Medel
 Fix For: 0.8.0


Per https://issues.apache.org/jira/browse/MINIFICPP-1214

We should resolve Marc's and Szasz's concerns about whether 
[https://github.com/apache/nifi-minifi-cpp/pull/784] breaks these two new 
processors by having regression tests. We will explore and plan to use 
pytest-mock ([https://github.com/pytest-dev/pytest-mock]) testing framework to 
implement the regression tests for the new processors. Therefore, these new 
processors would be easier to merge.

 

regression test concerns



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (MINIFICPP-1214) Convert H2O Processors to use ALv2 compliant H20-3 library

[James Medel (Jira)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-1214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106008#comment-17106008
 ] 

James Medel edited comment on MINIFICPP-1214 at 5/13/20, 6:35 AM:
--

[~phrocker]

I submitted a pull request that addresses this ticket: 
[https://github.com/apache/nifi-minifi-cpp/pull/781]


was (Author: james.medel):
I submitted a pull request that addresses this ticket: 
[https://github.com/apache/nifi-minifi-cpp/pull/781]

> Convert H2O Processors to use ALv2 compliant H20-3 library
> --
>
> Key: MINIFICPP-1214
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1214
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: Marc Parisi
>Assignee: James Medel
>Priority: Blocker
> Fix For: 0.8.0
>
>
> Per https://issues.apache.org/jira/browse/MINIFICPP-1201 We should resolve 
> Joe's licensing concerns so we're on the up and up. We can either revert or 
> mitigate this via using H20-3 (https://github.com/h2oai/h2o-3)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (MINIFICPP-1214) Convert H2O Processors to use ALv2 compliant H20-3 library

[James Medel (Jira)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-1214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106008#comment-17106008
 ] 

James Medel commented on MINIFICPP-1214:


I submitted a pull request that addresses this ticket: 
[https://github.com/apache/nifi-minifi-cpp/pull/781]

> Convert H2O Processors to use ALv2 compliant H20-3 library
> --
>
> Key: MINIFICPP-1214
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1214
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: Marc Parisi
>Assignee: James Medel
>Priority: Blocker
> Fix For: 0.8.0
>
>
> Per https://issues.apache.org/jira/browse/MINIFICPP-1201 We should resolve 
> Joe's licensing concerns so we're on the up and up. We can either revert or 
> mitigate this via using H20-3 (https://github.com/h2oai/h2o-3)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (NIFI-7411) Integrates NiFi with H2O Driverless AI MOJO Scoring Pipeline (Java Runtime) To Do ML Inference

[James Medel (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-7411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Medel resolved NIFI-7411.
---
Resolution: Resolved

NiFi + H2O Driverless AI MOJO Processor will not be merged into the project. 
Instead it will be managed, maintained by H2O repo: 
[https://github.com/h2oai/dai-deployment-examples/pull/18]

> Integrates NiFi with H2O Driverless AI MOJO Scoring Pipeline (Java Runtime) 
> To Do ML Inference
> --
>
> Key: NIFI-7411
> URL: https://issues.apache.org/jira/browse/NIFI-7411
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Affects Versions: 1.12.0
> Environment: Mac OS X Mojave 10.14.6
>Reporter: James Medel
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> *NiFi and H2O Driverless AI Integration* via Custom NiFi Processor:
> Integrates NiFi with H2O Driverless AI by using Driverless AI's MOJO Scoring 
> Pipeline (in Java Runtime) and NiFi's Custom Processor. This processor 
> executes the MOJO Scoring Pipeline to do batch scoring or real-time scoring 
> for one or more predicted labels on tabular data in the incoming flow file 
> content. If the tabular data is one row, then the MOJO does real-time 
> scoring. If the tabular data is multiple rows, then the MOJO does batch 
> scoring. I would like to contribute my processor to NiFi as a new feature.
> *1 Custom Processor* created for NiFi:
> *ExecuteMojoScoringRecord* - Executes H2O Driverless AI's MOJO Scoring 
> Pipeline in Java Runtime to do batch scoring or real-time scoring on a frame 
> of data within each incoming flow file. It requires the user to add 
> *mojo2-runtime.jar* filepath into *MOJO2 Runtime JAR Directory* ** property 
> to dynamically modify the classpath. It also requires the user to add the 
> *pipeline.mojo* filepath into the *Pipeline MOJO Filepath* property. This 
> property is used in the onTrigger() method to get the pipeline.mojo filepath, 
> so we can pass it into the
> MojoPipeline.loadFrom(pipelineMojoPath) to instantiate our MojoPipeline 
> model. Then the record read in with Record Reader and the model are passed 
> into a predict() method to make predictions on the test data within the 
> record. Inside the predict() method, I use MojoFrameBuilder and 
> MojoRowBuilder with the recordMap to build an input MojoFrame. Then I use the 
> model's transform(input MojoFrame) method to make the predictions on the 
> input and store them into an output MojoFrame. I iterate through the 
> MojoFrame by row and column to store each key value pair prediction into the 
> predictedRecordMap. I then convert the predictedRecordMap to predictedRecord 
> and return the record back to onTrigger to write the record to the flow file 
> content using RecordSetWriter. We keep writing predicted Records to the flow 
> file content until there are no more records to write. Then we reach near end 
> of onTrigger() and the flow file is either transferred on relationship 
> failure, success or original to the next connection.
>  
> *Hydraulic System Condition Monitoring* Data used in NiFi Flow:
>  
> The sensor test data I used in this integration comes from UCI ML Repo: 
> Condition Monitoring for Hydraulic Systems. I was able to predict the 
> hydraulic cooling condition through NiFi and H2O Integration described above. 
> This use case is hydraulic system predictive maintenance.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (NIFI-7411) Integrates NiFi with H2O Driverless AI MOJO Scoring Pipeline (Java Runtime) To Do ML Inference

[James Medel (Jira)]

James Medel created NIFI-7411:
-

 Summary: Integrates NiFi with H2O Driverless AI MOJO Scoring 
Pipeline (Java Runtime) To Do ML Inference
 Key: NIFI-7411
 URL: https://issues.apache.org/jira/browse/NIFI-7411
 Project: Apache NiFi
  Issue Type: New Feature
  Components: Extensions
Affects Versions: 1.12.0
 Environment: Mac OS X Mojave 10.14.6
Reporter: James Medel


*NiFi and H2O Driverless AI Integration* via Custom NiFi Processor:

Integrates NiFi with H2O Driverless AI by using Driverless AI's MOJO Scoring 
Pipeline (in Java Runtime) and NiFi's Custom Processor. This processor executes 
the MOJO Scoring Pipeline to do batch scoring or real-time scoring for one or 
more predicted labels on tabular data in the incoming flow file content. If the 
tabular data is one row, then the MOJO does real-time scoring. If the tabular 
data is multiple rows, then the MOJO does batch scoring. I would like to 
contribute my processor to NiFi as a new feature.

*1 Custom Processor* created for NiFi:

*ExecuteMojoScoringRecord* - Executes H2O Driverless AI's MOJO Scoring Pipeline 
in Java Runtime to do batch scoring or real-time scoring on a frame of data 
within each incoming flow file. It requires the user to add *mojo2-runtime.jar* 
filepath into *MOJO2 Runtime JAR Directory* ** property to dynamically modify 
the classpath. It also requires the user to add the *pipeline.mojo* filepath 
into the *Pipeline MOJO Filepath* property. This property is used in the 
onTrigger() method to get the pipeline.mojo filepath, so we can pass it into the
MojoPipeline.loadFrom(pipelineMojoPath) to instantiate our MojoPipeline model. 
Then the record read in with Record Reader and the model are passed into a 
predict() method to make predictions on the test data within the record. Inside 
the predict() method, I use MojoFrameBuilder and MojoRowBuilder with the 
recordMap to build an input MojoFrame. Then I use the model's transform(input 
MojoFrame) method to make the predictions on the input and store them into an 
output MojoFrame. I iterate through the MojoFrame by row and column to store 
each key value pair prediction into the predictedRecordMap. I then convert the 
predictedRecordMap to predictedRecord and return the record back to onTrigger 
to write the record to the flow file content using RecordSetWriter. We keep 
writing predicted Records to the flow file content until there are no more 
records to write. Then we reach near end of onTrigger() and the flow file is 
either transferred on relationship failure, success or original to the next 
connection.
 
*Hydraulic System Condition Monitoring* Data used in NiFi Flow:
 
The sensor test data I used in this integration comes from UCI ML Repo: 
Condition Monitoring for Hydraulic Systems. I was able to predict the hydraulic 
cooling condition through NiFi and H2O Integration described above. This use 
case is hydraulic system predictive maintenance.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (MINIFICPP-1201) Integrates MiNiFi C++ with H2O Driverless AI MOJO Scoring Pipeline (C++ Runtime Python Wrapper) To Do ML Inference on Edge

[James Medel (Jira)]

James Medel created MINIFICPP-1201:
--

 Summary: Integrates MiNiFi C++ with H2O Driverless AI MOJO Scoring 
Pipeline (C++ Runtime Python Wrapper) To Do ML Inference on Edge
 Key: MINIFICPP-1201
 URL: https://issues.apache.org/jira/browse/MINIFICPP-1201
 Project: Apache NiFi MiNiFi C++
  Issue Type: New Feature
Affects Versions: master
 Environment: Ubuntu 18.04 in AWS EC2
MiNiFi C++ 0.7.0
Reporter: James Medel
 Fix For: master


*MiNiFi C++ and H2O Driverless AI Integration* via Custom Python Processors:

Integrates MiNiFi C++ with H2O Driverless AI by using Driverless AI's MOJO 
Scoring Pipeline (in C++ Runtime Python Wrapper) and MiNiFi's Custom Python 
Processor. Uses a Python Processor to execute the MOJO Scoring Pipeline to do 
batch scoring or real-time scoring for one or more predicted labels on tabular 
test data in the incoming flow file content. If the tabular data is one row, 
then the MOJO does real-time scoring. If the tabular data is multiple rows, 
then the MOJO does batch scoring. I would like to contribute my processors to 
MiNiFi C++ as a new feature.

*1 custom python processor* created for MiNiFi:

*H2oMojoPwScoring* - Executes H2O Driverless AI's MOJO Scoring Pipeline in C++ 
Runtime Python Wrapper to do batch scoring or real-time scoring on a frame of 
data within each incoming flow file. Requires the user to add the 
*pipeline.mojo* filepath into the "MOJO Pipeline Filepath" property. This 
property is used in the onTrigger(context, session) function to get the 
pipeline.mojo filepath, so we can *pass it into* the 
*daimojo.model(pipeline_mojo_filepath)* function to instantiate our 
*mojo_scorer*. MOJO creation time and uuid are added as individual flow file 
attributes. Then the *flow file content* is *loaded into Datatable* *frame* to 
hold the test data. Then a Python lambda function called compare is used to 
compare whether the datatable frame header column names equals the expected 
header column names from the mojo scorer. This check is done because the 
datatable frame could have a missing header, which is true when the header does 
not equal the expected header and so we update the datatable frame header with 
the mojo scorer's expected header. Having the correct header works nicely 
because the *mojo scorer's* *predict(datatable_frame)* function needs the 
header and then does the prediction returning a predictions datatable frame. 
The mojo scorer's predict function is *capable of doing real-time scoring or 
batch scoring*, it just depends on the amount of rows that the tabular data 
has. This predictions datatable frame is then converted to pandas dataframe, so 
we can use pandas' to_string(index=False) function to convert the dataframe to 
a string without the dataframe's index. Then *the prediction string is written 
to flow file content*. A flow file attribute is added for the number of rows 
scored. Another one or more flow file attributes are added for the predicted 
label name and its associated score. Finally, the flow file is transferred on a 
success relationship.

 

*Hydraulic System Condition Monitoring* Data used in MiNiFi Flow:

The sensor test data I used in this integration comes from Kaggle: Condition 
Monitoring of Hydraulic Systems. I was able to predict hydraulic system cooling 
efficiency through MiNiFi and H2O integration described above. This use case 
here is hydraulic system predictive maintenance.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (MINIFICPP-1199) Integrates MiNiFi C++ with H2O Driverless AI Python Scoring Pipeline To Do ML Inference on Edge

[James Medel (Jira)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Medel updated MINIFICPP-1199:
---
Summary: Integrates MiNiFi C++ with H2O Driverless AI Python Scoring 
Pipeline To Do ML Inference on Edge  (was: Integrates MiNiFi C++ with H2O's 
Driverless AI To Do ML Inference on Edge)

> Integrates MiNiFi C++ with H2O Driverless AI Python Scoring Pipeline To Do ML 
> Inference on Edge
> ---
>
> Key: MINIFICPP-1199
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1199
> Project: Apache NiFi MiNiFi C++
>  Issue Type: New Feature
>Affects Versions: master
> Environment: Ubuntu 18.04 in AWS EC2
> MiNiFi C++ 0.7.0
>Reporter: James Medel
>Priority: Major
> Fix For: master
>
>
> *MiNiFi C++ and H2O Driverless AI Integration* via Custom Python Processors:
> Integrates MiNiFi C++ with H2O's Driverless AI by Using Driverless AI's 
> Python Scoring Pipeline and MiNiFi's Custom Python Processors. Uses the 
> Python Processors to execute the Python Scoring Pipeline scorer to do batch 
> scoring and real-time scoring for one or more predicted labels on test data 
> in the incoming flow file content. I would like to contribute my processors 
> to MiNiFi C++ as a new feature.
>  
> *3 custom python processors* created for MiNiFi:
> *H2oPspScoreRealTime* - Executes H2O Driverless AI's Python Scoring Pipeline 
> to do interactive scoring (real-time) scoring on an individual row or list of 
> test data within each incoming flow file. Uses H2O's open-source Datatable 
> library to load test data into a frame, then converts it to pandas dataframe. 
> Pandas is used to convert the pandas dataframe rows to a list of lists, but 
> since each flow file passing through this processor should have only 1 row, 
> we extract the 1st list. Then that list is passed into the Driverless AI's 
> Python scorer.score() function to predict one or more predicted labels. The 
> prediction is returned to a list. The number of predicted labels is specified 
> when the user built the Python Scoring Pipeline in Driverless AI. With that 
> knowledge, there is a property for the user to pass in one or more predicted 
> label names that will be used as the predicted header. I create a comma 
> separated string using the predicted header and predicted value. The 
> predicted header(s) is on one line followed by a newline and the predicted 
> value(s) is on the next line followed by a newline. The string is written to 
> the flow file content. Flow File attributes are added to the flow file for 
> the number of lists scored and the predicted label name and its associated 
> score. Finally, the flow file is transferred on a success relationship.
>  
> *H2oPspScoreBatches* - Executes H2O Driverless AI's Python Scoring Pipeline 
> to do batch scoring on a frame of data within each incoming flow file. Uses 
> H2O's open-source Datatable library to load test data into a frame. Each 
> frame from the flow file passing through this processor should have multiple 
> rows. That frame is passed into the Driverless AI's Python 
> scorer.score_batch() function to predict one or more predicted labels. The 
> prediction is returned to a pandas dataframe, then that dataframe is 
> converted to a string, so it can be written to the flow file content. Flow 
> File attributes are added to the flow file for the number of rows scored. 
> There are also flow file attributes added for the predicted label name and 
> its associated score for the first row in the frame. Finally, the flow file 
> is transferred on a success relationship.
>  
> *ConvertDsToCsv* - Converts data source of incoming flow file to csv. Uses 
> H2O's open-source Datatable library to load data into a frame, then converts 
> it to pandas dataframe. Pandas is used to convert the pandas dataframe to a 
> csv and store it into in-memory text stream StringIO without pandas dataframe 
> index. The csv string data is grabbed using file read() function on the 
> StringIO object, so it can be written to the flow file content. The flow file 
> is transferred on a success relationship.
>  
> *Hydraulic System Condition Monitoring* Data used in MiNiFi Flow:
> The sensor test data I used in this integration comes from [Kaggle: Condition 
> Monitoring of Hydraulic 
> Systems|[https://www.kaggle.com/jjacostupa/condition-monitoring-of-hydraulic-systems#description.txt]].
>  I was able to predict hydraulic system cooling efficiency through MiNiFi and 
> H2O integration described above. This use case here is hydraulic system 
> predictive maintenance.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (MINIFICPP-1200) Add MiNiFi Logo Image Link to README.md

[James Medel (Jira)]

James Medel created MINIFICPP-1200:
--

 Summary: Add MiNiFi Logo Image Link to README.md
 Key: MINIFICPP-1200
 URL: https://issues.apache.org/jira/browse/MINIFICPP-1200
 Project: Apache NiFi MiNiFi C++
  Issue Type: Documentation
Affects Versions: master
Reporter: James Medel
 Fix For: master


Adds MiNiFi Logo to the README.md. Also includes a hyperlink to the MiNiFi 
documentation homepage: [https://nifi.apache.org/minifi/]

Update will be applied to nifi-minifi-cpp/README.md



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (MINIFICPP-1199) Integrates MiNiFi C++ with H2O's Driverless AI To Do ML Inference on Edge

[James Medel (Jira)]

James Medel created MINIFICPP-1199:
--

 Summary: Integrates MiNiFi C++ with H2O's Driverless AI To Do ML 
Inference on Edge
 Key: MINIFICPP-1199
 URL: https://issues.apache.org/jira/browse/MINIFICPP-1199
 Project: Apache NiFi MiNiFi C++
  Issue Type: New Feature
Affects Versions: master
 Environment: Ubuntu 18.04 in AWS EC2
MiNiFi C++ 0.7.0
Reporter: James Medel
 Fix For: master


*MiNiFi C++ and H2O Driverless AI Integration* via Custom Python Processors:

Integrates MiNiFi C++ with H2O's Driverless AI by Using Driverless AI's Python 
Scoring Pipeline and MiNiFi's Custom Python Processors. Uses the Python 
Processors to execute the Python Scoring Pipeline scorer to do batch scoring 
and real-time scoring for one or more predicted labels on test data in the 
incoming flow file content. I would like to contribute my processors to MiNiFi 
C++ as a new feature.

 

*3 custom python processors* created for MiNiFi:

*H2oPspScoreRealTime* - Executes H2O Driverless AI's Python Scoring Pipeline to 
do interactive scoring (real-time) scoring on an individual row or list of test 
data within each incoming flow file. Uses H2O's open-source Datatable library 
to load test data into a frame, then converts it to pandas dataframe. Pandas is 
used to convert the pandas dataframe rows to a list of lists, but since each 
flow file passing through this processor should have only 1 row, we extract the 
1st list. Then that list is passed into the Driverless AI's Python 
scorer.score() function to predict one or more predicted labels. The prediction 
is returned to a list. The number of predicted labels is specified when the 
user built the Python Scoring Pipeline in Driverless AI. With that knowledge, 
there is a property for the user to pass in one or more predicted label names 
that will be used as the predicted header. I create a comma separated string 
using the predicted header and predicted value. The predicted header(s) is on 
one line followed by a newline and the predicted value(s) is on the next line 
followed by a newline. The string is written to the flow file content. Flow 
File attributes are added to the flow file for the number of lists scored and 
the predicted label name and its associated score. Finally, the flow file is 
transferred on a success relationship.

 

*H2oPspScoreBatches* - Executes H2O Driverless AI's Python Scoring Pipeline to 
do batch scoring on a frame of data within each incoming flow file. Uses H2O's 
open-source Datatable library to load test data into a frame. Each frame from 
the flow file passing through this processor should have multiple rows. That 
frame is passed into the Driverless AI's Python scorer.score_batch() function 
to predict one or more predicted labels. The prediction is returned to a pandas 
dataframe, then that dataframe is converted to a string, so it can be written 
to the flow file content. Flow File attributes are added to the flow file for 
the number of rows scored. There are also flow file attributes added for the 
predicted label name and its associated score for the first row in the frame. 
Finally, the flow file is transferred on a success relationship.

 

*ConvertDsToCsv* - Converts data source of incoming flow file to csv. Uses 
H2O's open-source Datatable library to load data into a frame, then converts it 
to pandas dataframe. Pandas is used to convert the pandas dataframe to a csv 
and store it into in-memory text stream StringIO without pandas dataframe 
index. The csv string data is grabbed using file read() function on the 
StringIO object, so it can be written to the flow file content. The flow file 
is transferred on a success relationship.

 

*Hydraulic System Condition Monitoring* Data used in MiNiFi Flow:

The sensor test data I used in this integration comes from [Kaggle: Condition 
Monitoring of Hydraulic 
Systems|[https://www.kaggle.com/jjacostupa/condition-monitoring-of-hydraulic-systems#description.txt]].
 I was able to predict hydraulic system cooling efficiency through MiNiFi and 
H2O integration described above. This use case here is hydraulic system 
predictive maintenance.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (MINIFICPP-1194) MiNiFi-DOCS - Fix Table Formatting for 2 Execute.* Processors

[James Medel (Jira)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Medel resolved MINIFICPP-1194.

Resolution: Fixed

The table formatting issue was that the 2 tables were not rendering. I updated 
the markdown for the tables and now the tables render correctly. Arpad Boda 
approved the PR and merged.

> MiNiFi-DOCS - Fix Table Formatting for 2 Execute.* Processors
> -
>
> Key: MINIFICPP-1194
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1194
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Documentation
>Affects Versions: master
> Environment: Ubuntu 18.04 LTS in AWS EC2
>Reporter: James Medel
>Priority: Major
> Fix For: master
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Table formatting issues were making it difficult to read properties for 
> ExecutePythonProcessor and ExecuteScript. So, this change in markdown 
> resolves the formatting issues.
> Doc affected: 
> [https://github.com/apache/nifi-minifi-cpp/blob/master/PROCESSORS.md#executescript]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (MINIFICPP-1194) MiNiFi-DOCS - Fix Table Formatting for 2 Execute.* Processors

[James Medel (Jira)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Medel updated MINIFICPP-1194:
---
Fix Version/s: (was: 0.7.0)
   master

> MiNiFi-DOCS - Fix Table Formatting for 2 Execute.* Processors
> -
>
> Key: MINIFICPP-1194
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1194
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Documentation
>Affects Versions: master
> Environment: Ubuntu 18.04 LTS in AWS EC2
>Reporter: James Medel
>Priority: Major
> Fix For: master
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Table formatting issues were making it difficult to read properties for 
> ExecutePythonProcessor and ExecuteScript. So, this change in markdown 
> resolves the formatting issues.
> Doc affected: 
> [https://github.com/apache/nifi-minifi-cpp/blob/master/PROCESSORS.md#executescript]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (MINIFICPP-1194) MiNiFi-DOCS - Fix Table Formatting for 2 Execute.* Processors

[James Medel (Jira)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Medel updated MINIFICPP-1194:
---
Affects Version/s: (was: 0.7.0)
   master

> MiNiFi-DOCS - Fix Table Formatting for 2 Execute.* Processors
> -
>
> Key: MINIFICPP-1194
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1194
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Documentation
>Affects Versions: master
> Environment: Ubuntu 18.04 LTS in AWS EC2
>Reporter: James Medel
>Priority: Major
> Fix For: 0.7.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Table formatting issues were making it difficult to read properties for 
> ExecutePythonProcessor and ExecuteScript. So, this change in markdown 
> resolves the formatting issues.
> Doc affected: 
> [https://github.com/apache/nifi-minifi-cpp/blob/master/PROCESSORS.md#executescript]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (MINIFICPP-1195) MiNiFiCPP-DOCS: Updates README Outdated Bootstrapping Info

[James Medel (Jira)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-1195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Medel resolved MINIFICPP-1195.

Resolution: Fixed

I updated the Getting Started: Bootstrapping section with the latest menu 
guided bootstrap process text from the console. ArpadBoda approved the pull 
request and merged.

> MiNiFiCPP-DOCS: Updates README Outdated Bootstrapping Info
> --
>
> Key: MINIFICPP-1195
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1195
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Documentation
>Affects Versions: master
> Environment: Ubuntu 18.04 LTS on AWS EC2
>Reporter: James Medel
>Priority: Major
> Fix For: master
>
>
> The paragraph in the Getting Started: Bootstrapping section originally told 
> the user to enter N to continue the build process, but with the updated menu 
> guided bootstrap process displayed in the console, P is what one must enter 
> to continue the build process. I updated that info.
> The previous menu guided bootstrap process was outdated and did not show the 
> new features Bustache Support to SQL Support and the Build Options. So, I 
> updated it to show that information.
>  
> Here is my pull request with the updates: 
> [https://github.com/apache/nifi-minifi-cpp/pull/760] 
> Here is the doc that I am proposing updates for: 
> [https://github.com/apache/nifi-minifi-cpp/blob/master/README.md]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (MINIFICPP-1195) MiNiFiCPP-DOCS: Updates README Outdated Bootstrapping Info

[James Medel (Jira)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-1195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Medel updated MINIFICPP-1195:
---
Fix Version/s: (was: 0.7.0)
   master

> MiNiFiCPP-DOCS: Updates README Outdated Bootstrapping Info
> --
>
> Key: MINIFICPP-1195
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1195
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Documentation
>Affects Versions: master
> Environment: Ubuntu 18.04 LTS on AWS EC2
>Reporter: James Medel
>Priority: Major
> Fix For: master
>
>
> The paragraph in the Getting Started: Bootstrapping section originally told 
> the user to enter N to continue the build process, but with the updated menu 
> guided bootstrap process displayed in the console, P is what one must enter 
> to continue the build process. I updated that info.
> The previous menu guided bootstrap process was outdated and did not show the 
> new features Bustache Support to SQL Support and the Build Options. So, I 
> updated it to show that information.
>  
> Here is my pull request with the updates: 
> [https://github.com/apache/nifi-minifi-cpp/pull/760] 
> Here is the doc that I am proposing updates for: 
> [https://github.com/apache/nifi-minifi-cpp/blob/master/README.md]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (MINIFICPP-1195) MiNiFiCPP-DOCS: Updates README Outdated Bootstrapping Info

[James Medel (Jira)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-1195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Medel updated MINIFICPP-1195:
---
Affects Version/s: (was: 0.7.0)
   master

> MiNiFiCPP-DOCS: Updates README Outdated Bootstrapping Info
> --
>
> Key: MINIFICPP-1195
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1195
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Documentation
>Affects Versions: master
> Environment: Ubuntu 18.04 LTS on AWS EC2
>Reporter: James Medel
>Priority: Major
> Fix For: 0.7.0
>
>
> The paragraph in the Getting Started: Bootstrapping section originally told 
> the user to enter N to continue the build process, but with the updated menu 
> guided bootstrap process displayed in the console, P is what one must enter 
> to continue the build process. I updated that info.
> The previous menu guided bootstrap process was outdated and did not show the 
> new features Bustache Support to SQL Support and the Build Options. So, I 
> updated it to show that information.
>  
> Here is my pull request with the updates: 
> [https://github.com/apache/nifi-minifi-cpp/pull/760] 
> Here is the doc that I am proposing updates for: 
> [https://github.com/apache/nifi-minifi-cpp/blob/master/README.md]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (MINIFICPP-1198) Fix AWSCredentialsService Table Render Issue in CONTROLLERS.md

[James Medel (Jira)]

James Medel created MINIFICPP-1198:
--

 Summary: Fix AWSCredentialsService Table Render Issue in 
CONTROLLERS.md
 Key: MINIFICPP-1198
 URL: https://issues.apache.org/jira/browse/MINIFICPP-1198
 Project: Apache NiFi MiNiFi C++
  Issue Type: Documentation
Affects Versions: master
Reporter: James Medel
 Fix For: master


Under AWSCredentialsService section, the Properties markdown table was not 
rendering. I realized the markdown table was missing a hyphen '-' and pipe '|' 
for the last header. I added this missing markdown and now the table renders 
correctly.

Page affected: 
https://github.com/apache/nifi-minifi-cpp/blob/master/CONTROLLERS.md



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (MINIFICPP-1197) Update ExampleProcessor.py to Show More Features of MiNiFiCPP Python

[James Medel (Jira)]

James Medel created MINIFICPP-1197:
--

 Summary: Update ExampleProcessor.py to Show More Features of 
MiNiFiCPP Python
 Key: MINIFICPP-1197
 URL: https://issues.apache.org/jira/browse/MINIFICPP-1197
 Project: Apache NiFi MiNiFi C++
  Issue Type: Improvement
Affects Versions: master
Reporter: James Medel
 Fix For: master


I feel we could use the python *ExampleProcessor* as a processor template to 
introduce people to the various features that are available when you build 
custom Python Processors in MiNiFi CPP. MiNiFi CPP's Python processor feature 
is a great way to get started with developing processors and it can be used for 
rapid prototyping. Currently, the ExampleProcessor.py shows how to add a Flow 
File attribute. We could update the processor to include how to read content 
from incoming flow files using session.write(flow_file, input_stream_cb) and 
how to write the content using session.write(flow_file, output_stream_cb) and 
more features defined in MiNiFi CPP PythonBindings.h.

 

My suggestion is we update:  

- 
[nifi-minifi-cpp/extensions/script/ExampleProcessor.py|[https://github.com/apache/nifi-minifi-cpp/blob/master/extensions/script/ExampleProcessor.py]]

 

References:

- 
[nifi-minifi-cpp/extensions/script/python/PythonBindings.h|[https://github.com/apache/nifi-minifi-cpp/blob/master/extensions/script/python/PythonBindings.h]]

- 
[nifi-minifi-cpp/extensions/script/python/ExecutePythonProcessor.h|[https://github.com/apache/nifi-minifi-cpp/blob/master/extensions/script/python/ExecutePythonProcessor.h]]

- 
[nifi-minifi-cpp/extensions/script/python/PyProcessSession.h|[https://github.com/apache/nifi-minifi-cpp/blob/master/extensions/script/python/PyProcessSession.h]]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (MINIFICPP-1196) MiNiFiCPP-BUG: Failed to initialize and start ExampleProcessor.py

[James Medel (Jira)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-1196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Medel resolved MINIFICPP-1196.

Resolution: Fixed

I figured out why the ExampleProcessor.py was not starting. It turned out my 
script file path specified in the config.yml file was wrong. Once I corrected 
it, the python processor worked.

> MiNiFiCPP-BUG: Failed to initialize and start ExampleProcessor.py
> -
>
> Key: MINIFICPP-1196
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1196
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Documentation
>Affects Versions: 0.7.0
>Reporter: James Medel
>Priority: Major
> Fix For: 0.7.0
>
> Attachments: config.yml
>
>
> I am working on creating custom MiNiFiCPP Python Processors, my processors 
> were failing to start, so I tried the ExampleProcessor.py from the repo and 
> dropped it into my flow. When I run my flow, I receive the error message: 
> {color:#0747a6}[2020-04-17 18:29:10.807] 
> [org::apache::nifi::minifi::core::ProcessGroup] [error] Failed to start 
> processor 54d61268-80d9-11ea-a76f-022edbf4da95 (ExampleProcessor): Could not 
> correctly in initialize ExampleProcessor{color}
> I also receive warning message:
> {color:#0747a6}[2020-04-17 18:29:10.779] 
> [org::apache::nifi::minifi::core::ConfigurableComponent] [warning] Could not 
> find property Script File{color}
> I placed the ExampleProcessor.py into the minifi-python/ directory, then 
> specified the Script File path for ExecutePythonProcessor in my config.yml 
> file. 
> I followed the instructions at *Apache NiFi - MiNiFi - Python Processors 
> Readme*: 
> [https://github.com/apache/nifi-minifi-cpp/tree/master/extensions/script]
> I used *ExampleProcessor.py*: 
> [https://github.com/apache/nifi-minifi-cpp/blob/master/extensions/script/ExampleProcessor.py]
>  
> Any help with getting this Python Processor to work would be greatly 
> appreciated.
> [^config.yml]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (MINIFICPP-1196) MiNiFiCPP-BUG: Failed to initialize and start ExampleProcessor.py

[James Medel (Jira)]

James Medel created MINIFICPP-1196:
--

 Summary: MiNiFiCPP-BUG: Failed to initialize and start 
ExampleProcessor.py
 Key: MINIFICPP-1196
 URL: https://issues.apache.org/jira/browse/MINIFICPP-1196
 Project: Apache NiFi MiNiFi C++
  Issue Type: Documentation
Affects Versions: 0.7.0
Reporter: James Medel
 Fix For: 0.7.0
 Attachments: config.yml

I am working on creating custom MiNiFiCPP Python Processors, my processors were 
failing to start, so I tried the ExampleProcessor.py from the repo and dropped 
it into my flow. When I run my flow, I receive the error message: 

{color:#0747a6}[2020-04-17 18:29:10.807] 
[org::apache::nifi::minifi::core::ProcessGroup] [error] Failed to start 
processor 54d61268-80d9-11ea-a76f-022edbf4da95 (ExampleProcessor): Could not 
correctly in initialize ExampleProcessor{color}

I also receive warning message:

{color:#0747a6}[2020-04-17 18:29:10.779] 
[org::apache::nifi::minifi::core::ConfigurableComponent] [warning] Could not 
find property Script File{color}

I placed the ExampleProcessor.py into the minifi-python/ directory, then 
specified the Script File path for ExecutePythonProcessor in my config.yml 
file. 

I followed the instructions at *Apache NiFi - MiNiFi - Python Processors 
Readme*: 
[https://github.com/apache/nifi-minifi-cpp/tree/master/extensions/script]

I used *ExampleProcessor.py*: 
[https://github.com/apache/nifi-minifi-cpp/blob/master/extensions/script/ExampleProcessor.py]

 

Any help with getting this Python Processor to work would be greatly 
appreciated.

[^config.yml]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (MINIFICPP-1195) MiNiFiCPP-DOCS: Updates README Outdated Bootstrapping Info

[James Medel (Jira)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-1195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Medel updated MINIFICPP-1195:
---
Description: 
The paragraph in the Getting Started: Bootstrapping section originally told the 
user to enter N to continue the build process, but with the updated menu guided 
bootstrap process displayed in the console, P is what one must enter to 
continue the build process. I updated that info.

The previous menu guided bootstrap process was outdated and did not show the 
new features Bustache Support to SQL Support and the Build Options. So, I 
updated it to show that information.

 

Here is my pull request with the updates: 
[https://github.com/apache/nifi-minifi-cpp/pull/760] 

Here is the doc that I am proposing updates for: 
[https://github.com/apache/nifi-minifi-cpp/blob/master/README.md]

  was:
The paragraph in the Getting Started: Bootstrapping section originally told the 
user to enter N to continue the build process, but with the updated menu guided 
bootstrap process displayed in the console, P is what one must enter to 
continue the build process. I updated that info.

The previous menu guided bootstrap process was outdated and did not show the 
new features Bustache Support to SQL Support and the Build Options. So, I 
updated it to show that information.

 

Here is the doc that I am proposing updates for: 
[https://github.com/apache/nifi-minifi-cpp/blob/master/README.md]


> MiNiFiCPP-DOCS: Updates README Outdated Bootstrapping Info
> --
>
> Key: MINIFICPP-1195
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1195
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Documentation
>Affects Versions: 0.7.0
> Environment: Ubuntu 18.04 LTS on AWS EC2
>Reporter: James Medel
>Priority: Major
> Fix For: 0.7.0
>
>
> The paragraph in the Getting Started: Bootstrapping section originally told 
> the user to enter N to continue the build process, but with the updated menu 
> guided bootstrap process displayed in the console, P is what one must enter 
> to continue the build process. I updated that info.
> The previous menu guided bootstrap process was outdated and did not show the 
> new features Bustache Support to SQL Support and the Build Options. So, I 
> updated it to show that information.
>  
> Here is my pull request with the updates: 
> [https://github.com/apache/nifi-minifi-cpp/pull/760] 
> Here is the doc that I am proposing updates for: 
> [https://github.com/apache/nifi-minifi-cpp/blob/master/README.md]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (MINIFICPP-1195) MiNiFiCPP-DOCS: Updates README Outdated Bootstrapping Info

[James Medel (Jira)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-1195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Medel updated MINIFICPP-1195:
---
Summary: MiNiFiCPP-DOCS: Updates README Outdated Bootstrapping Info  (was: 
MiNiFiCPP-DOCS: Updated README Outdated Bootstrapping Info)

> MiNiFiCPP-DOCS: Updates README Outdated Bootstrapping Info
> --
>
> Key: MINIFICPP-1195
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1195
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Documentation
>Affects Versions: 0.7.0
> Environment: Ubuntu 18.04 LTS on AWS EC2
>Reporter: James Medel
>Priority: Major
> Fix For: 0.7.0
>
>
> The paragraph in the Getting Started: Bootstrapping section originally told 
> the user to enter N to continue the build process, but with the updated menu 
> guided bootstrap process displayed in the console, P is what one must enter 
> to continue the build process. I updated that info.
> The previous menu guided bootstrap process was outdated and did not show the 
> new features Bustache Support to SQL Support and the Build Options. So, I 
> updated it to show that information.
>  
> Here is the doc that I am proposing updates for: 
> [https://github.com/apache/nifi-minifi-cpp/blob/master/README.md]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (MINIFICPP-1195) MiNiFiCPP-DOCS: Updated README Outdated Bootstrapping Info

[James Medel (Jira)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-1195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Medel updated MINIFICPP-1195:
---
Summary: MiNiFiCPP-DOCS: Updated README Outdated Bootstrapping Info  (was: 
MiNiFiCPP-DOCS: Updated Outdated Bootstrapping Info in README.md)

> MiNiFiCPP-DOCS: Updated README Outdated Bootstrapping Info
> --
>
> Key: MINIFICPP-1195
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1195
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Documentation
>Affects Versions: 0.7.0
> Environment: Ubuntu 18.04 LTS on AWS EC2
>Reporter: James Medel
>Priority: Major
> Fix For: 0.7.0
>
>
> The paragraph in the Getting Started: Bootstrapping section originally told 
> the user to enter N to continue the build process, but with the updated menu 
> guided bootstrap process displayed in the console, P is what one must enter 
> to continue the build process. I updated that info.
> The previous menu guided bootstrap process was outdated and did not show the 
> new features Bustache Support to SQL Support and the Build Options. So, I 
> updated it to show that information.
>  
> Here is the doc that I am proposing updates for: 
> [https://github.com/apache/nifi-minifi-cpp/blob/master/README.md]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (MINIFICPP-1195) MiNiFiCPP-DOCS: Updated Outdated Bootstrapping Info in README.md

[James Medel (Jira)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-1195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Medel updated MINIFICPP-1195:
---
Summary: MiNiFiCPP-DOCS: Updated Outdated Bootstrapping Info in README.md  
(was: MiNiFiCPP-DOCS: Updated Outdated Bootstrapping Info)

> MiNiFiCPP-DOCS: Updated Outdated Bootstrapping Info in README.md
> 
>
> Key: MINIFICPP-1195
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1195
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Documentation
>Affects Versions: 0.7.0
> Environment: Ubuntu 18.04 LTS on AWS EC2
>Reporter: James Medel
>Priority: Major
> Fix For: 0.7.0
>
>
> The paragraph in the Getting Started: Bootstrapping section originally told 
> the user to enter N to continue the build process, but with the updated menu 
> guided bootstrap process displayed in the console, P is what one must enter 
> to continue the build process. I updated that info.
> The previous menu guided bootstrap process was outdated and did not show the 
> new features Bustache Support to SQL Support and the Build Options. So, I 
> updated it to show that information.
>  
> Here is the doc that I am proposing updates for: 
> [https://github.com/apache/nifi-minifi-cpp/blob/master/README.md]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (MINIFICPP-1195) MiNiFiCPP-DOCS: Updated Outdated Bootstrapping Info

[James Medel (Jira)]

James Medel created MINIFICPP-1195:
--

 Summary: MiNiFiCPP-DOCS: Updated Outdated Bootstrapping Info
 Key: MINIFICPP-1195
 URL: https://issues.apache.org/jira/browse/MINIFICPP-1195
 Project: Apache NiFi MiNiFi C++
  Issue Type: Documentation
Affects Versions: 0.7.0
 Environment: Ubuntu 18.04 LTS on AWS EC2
Reporter: James Medel
 Fix For: 0.7.0


The paragraph in the Getting Started: Bootstrapping section originally told the 
user to enter N to continue the build process, but with the updated menu guided 
bootstrap process displayed in the console, P is what one must enter to 
continue the build process. I updated that info.

The previous menu guided bootstrap process was outdated and did not show the 
new features Bustache Support to SQL Support and the Build Options. So, I 
updated it to show that information.

 

Here is the doc that I am proposing updates for: 
[https://github.com/apache/nifi-minifi-cpp/blob/master/README.md]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (MINIFICPP-1194) MiNiFi-DOCS - Fix Table Formatting for 2 Execute.* Processors

[James Medel (Jira)]

James Medel created MINIFICPP-1194:
--

 Summary: MiNiFi-DOCS - Fix Table Formatting for 2 Execute.* 
Processors
 Key: MINIFICPP-1194
 URL: https://issues.apache.org/jira/browse/MINIFICPP-1194
 Project: Apache NiFi MiNiFi C++
  Issue Type: Documentation
Affects Versions: 0.7.0
 Environment: Ubuntu 18.04 LTS in AWS EC2
Reporter: James Medel
 Fix For: 0.7.0


Table formatting issues were making it difficult to read properties for 
ExecutePythonProcessor and ExecuteScript. So, this change in markdown resolves 
the formatting issues.

Doc affected: 
[https://github.com/apache/nifi-minifi-cpp/blob/master/PROCESSORS.md#executescript]

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)