[
https://issues.apache.org/jira/browse/MINIFICPP-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Guzman (Medel) updated MINIFICPP-2346:
--------------------------------------------
Affects Version/s: 0.99.0
> Speed up MiNiFi Build with Conan package manager (C++)
> ------------------------------------------------------
>
> Key: MINIFICPP-2346
> URL: https://issues.apache.org/jira/browse/MINIFICPP-2346
> Project: Apache NiFi MiNiFi C++
> Issue Type: Improvement
> Affects Versions: 0.15.0, 0.99.0
> Environment: Ubuntu 22.04
> Reporter: James Guzman (Medel)
> Assignee: James Guzman (Medel)
> Priority: Minor
> Labels: build, conan, ubuntu
> Attachments: minifi_cpp_cmake_standalone_build_7eb8beb.log,
> minifi_cpp_conan_build_2d1b884.log, minifi_cpp_conan_build_7eb8beb.log
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> *NOTE:* I noticed you guys recently switched from 0.15 to 0.99, so at certain
> point, I will pull in lastest updates from MiNiFi C++ main branch on parent
> level and rebase or merge into my PR.
> *Description of Issue:* Building MiNiFi C++ with just CMake is really slow.
> During the the build process, it clones all the external dependencies and
> then builds those external dependencies while its building MiNiFi CPP.
> Sometimes the MiNiFi CPP build fails too when external lib dependencies fail
> to download or fail to build. It would be faster to build MiNiFi C++ with the
> external dependencies already preinstalled.
>
> *Potential Solution:* Integrate *conan version 2* into MiNiFi C++ project to
> enable support for using *conan install* to install all or most of MiNiFi's
> external dependencies, using *conan install --build=missing* to build them as
> prebuilt binary conan packages, upload them to conancenter with {*}conan
> upload{*}, and then run cmake generate to generate the appropriate build
> files for the OS, then run make to build MiNiFi CPP. Also conan has really
> good support for cross platform compatibility for Linux and Windows. At this
> point because we already have most of MiNiFi CPP's external lib dependencies
> installed with conan, our CMake build will just focus on building the MiNiFi
> CPP code, enabling faster builds. Also I will need to account for backward
> compatibility support with the previous way of installing MiNiFi CPP external
> lib dependencies using CMake FetchContent_Declare(...) approach until
> conanfile.py approach covers installing all these dependencies. A valid
> verification of the MiNiFi C++ build would be to run GTESTs and if all the
> expected test cases pass, then it may be fair to conclude we would be ready
> to deprecate the slow CMake FetchContent_Declare(...) approach.
>
> *Steps to Perform MiNiFi C++ Build Enhancement (Plan to Integrate into
> script(s)):*
> # Install conan version 2 into MiNiFi C++ dev environment (my preference is
> using docker container)
> # Create a conanfile.py file at root of MiNiFi C++ project where we run
> "conan install" or "conan install --build=missing" to install MiNiFi C++
> external lib dependencies first. Then we run CMake to build MiNiFi C++.
> # Find all MiNiFi C++ find_package(...) CMake function calls and check if
> there is already supported conan packages on the conancenter, so we can
> install those packages.
> # Find all FetchContent_Declare(...) CMake function calls and check if there
> is already supported conan packages on the conancenter, so we can install
> those packages.
> # At this point we will have used conan version 2 C++ package manager to
> install almost all MiNiFi C++'s external dependencies.
> ## With Conan, we will have installed these MiNiFi external lib dependencies
> as prebuilt binary conan packages.
> ## Or we will have instructed conan to first build each C++ external lib
> dependency, upload each lib to conancenter.
> # Thus, conan manages installing MiNiFi C++'s external lib dependencies, so
> we can then run CMake to build MiNiFi faster.
>
> Here is my draft PR for this Jira Ticket for your reference:
> [https://github.com/apache/nifi-minifi-cpp/pull/1775]
> Here are the commands that I run:
>
> {code:java}
> cd $HOME/src/pipeline/nifi-minifi-cpp
> # NOTE: I also plan to add smoother integration of conan commands into MiNiFI
> CPP too
> # For instance, I know you guys have your bootstrap scripts too
> # You also have your github actions workflow scripts too
> conan install . --output-folder=build
> -pr=$HOME/src/nifi-minifi-cpp/etc/build/conan/profiles/release-linux
> # install conan packages and build C++ for minifi using Conan & CMake, we run
> this way, so conan's tc.cache_variables
> # are like passing -D{minifi_option} upon running cmake generate, then
> make to build MiNiFi C++
> conan build . --output-folder=build
> -pr=/home/bizon/src/jam-repo/main/etc/build/conan/profiles/release-linux{code}
>
> *UPDATE(May 14, 2024):* Updated due date to May 19, 2024 since the
> refactoring of MiNiFi CPP build infra over to conan version 2 is taking
> longer than expected. The good news is that I believe I have more than half
> of MiNiFi external libraries now managed by conan with backward compatibility
> for standalone CMake approach.
>
> *UPDATE (May 13, 2024)* -
> *[22205c9|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/22205c98cef30ee1ebe252880ae113724589bd78]:*
> Fetch and Pulled latest upstream main branch of parent MiNiFi to my fork
> main branch of MiNiFi repo; thus, merged in the latest updates from main
> branch of parent MiNiFi to my MINIFICPP-2346 branch. Resolved conflicts
> between a few CMake files coming from Date.cmake and BundledRocksdb.cmake
> files. For standalone CMake build appraoch, ran GTESTs with 100% (238/238)
> tests passing. I did the conan build approach and ran GTESTs with 99%
> (237/238) passing.
> *UPDATE (May 13, 2024)* -
> [59ffefb|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/59ffefb1caf662874054f55f7171692c6b9d345f]:
> {*}Enabled LIBRDKAFKA & GRPC_FOR_LOKI MiNiFi C++ extension using Conan &
> CMake{*}, *Improved Running Python Maven Executable for JNI & SFTP Test
> Server for conan approach* and {*}Improved standalone CMake approach for
> building PcapPlusPlus with its dependency on LibPcap{*}. LIBRDKAFKA required
> creating a librdkafka conan package passing similar CMake definitions to it
> to build it like we did for standalone CMake approach. GRPC_FOR_LOKI required
> creating a grpc conan package that uses minifi's openssl and libsystemd conan
> packages. In both LIBRDKAFKA & GRPC_FOR_LOKI, I updated their .cmake Bundles
> to account for the condition of using their conan package, so minimal changes
> would be needed to their extension CMakeLists.txt file. In LIBRDKAFKA's
> extension CMakeLists.txt file, it didnt really change after integrating the
> conan package since I created a library alias for it similar to the
> standalone CMake approach. In GRPC_FOR_LOKI's extension CMakeLists.txt,
> minimal changes were made to that file mainly on the add_custom_command,
> target_include_directories and target_link_libraries commands. You'll see for
> Grpc.cmake for conan approach, I created similar variables to standalone
> CMake approach for GRPC_CPP_PLUGIN, INCLUDE_DIR, PROTOBUF_INCLUDE_DIR,
> PROTOBUF_COMPILER getting the conan related properties. For the conan side, I
> also made improvements to the Python Maven Executable Wrapper script since I
> noticed there were times at the beginning when running maven version test, it
> didn't always run it successfully, so I updated conan maven executable path
> to be explicit for passing to Python script. I then moved the location of
> running maven version check into a CMake function and ran it after we
> included JavaMaven and JavaOpenJDK conan packages in the MiNiFi root
> CMakeLists.txt file. Thus, it succesfully runs maven version check at
> beginning each time. Its important to make sure we have conan maven to
> successfully build NiFi JNI and Java SFTP Test Server. Finally, on the
> standalone CMake approach for building PcapPlusPlus, I noticed some issues
> with PCAP headers not being found when building PcapPlusPlus, so I switched
> from execute_process to run the configure scripts to using
> ExternalProject_Add and I also added a bundle for LibPcap since the system
> LibPcap version wasn't always installed on the system. This problem with
> building PcapPlusPlus dependency on LibPcap didn't happen with my conan
> approach. It took a lot of time to fix the standalone CMake backward
> compatibility approach, but I did it for now since we're keeping support that
> approach. 99% of GTESTs ( 237/238) passed for conan.
>
> *UPDATE (May 11, 2024)* -
> [c96d430|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/c96d430814a067a6a538f4324337b719cb75bf48]:
> Refactoring MiNiFi C++ build infrastructure adding Conan Packager support
> *Enabled JNI, SENSORS, USB_CAMERA, OPENCV, and SFTP,* which required creating
> *Maven 3.9.6 conan package* updating the conan recipe to expose the Maven
> Executable environment variable in CMake and *installing prebuilt Java
> OpenJDK 21.0.2 conan package* and creating a python wrapper run_maven.py
> script for calling the conan maven Java build tool to successfully build Java
> SFTP Test Server for SFTP extension testing and maven to build {*}Apache NiFi
> 1.9.0 JNI Assembly and Framework{*}; We had to go the python maven wrapper
> script approach since I ran into some issues calling conan maven executable
> from CMake directly; Also added prebuilt install for libssh2 conan package
> toward SFTP extensions; Thus, now the Java support for MiNiFi C++ is less
> reliant on the user’s system since we can handle building this part of MiNiFi
> C++ using conan; Also created an OpenCV 4.8.1 conan package updating the
> conan recipe to use MiNiFi C++ conan package dependencies from libtiff,
> ffmpeg and xz_utils (liblzma) and successfully build MiNiFi OpenCV extension;
> Also installed prebuilt libuvc conan package for enabling USB_CAMERA. For the
> conan build approach, 99% (235/236) of GTESTs passed. For the standalone
> CMake build approach, 99% (234/236) of GTESTs passed.
> * Also we could look at updating Apache NiFi 1.9.0 to latest 2.0 version. I
> noticed we use a Java SFTP Test Server now that NiFi has native python custom
> extensibility, maybe we could look at Python SFTP Test Server?
>
> *UPDATE (May 11, 2024)* -
> [64dd8ad|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/64dd8ad14d2ddcea5eaa12da14a0e1b8a7289bb8]:
> {*}Enabled Build LUA SCRIPTING, ENCRYPT CONFIG, SPLUNK, ELASTIC SEARCH, TEST
> PROCESSORS, GRANFANA LOKI and CONTROLLER by Conan & CMake{*}. For building
> these components with conan, the LUA SCRIPTING extension required me updating
> its Lua.cmake and Sol2.cmake files, so we can use the conan packages in CMake
> when USE_CONAN_PACKAGE condition is true. For lua and sol2 conan packages, I
> was able to just install the prebuilt binary conan packages from conancenter
> directly since conancenter provided the versions that match with MiNiFi C++'s
> standalone CMake approach's lua 5.4.6 and sol2 3.3.0 source built versions.
> MiNiFi controller being enabled required updating the ArgParse.cmake file, so
> we could also account for the condition when we build the controller using
> conan, thus I added back using the argparse conan package. After enabling
> these components, I was able to successfully build them for MiNiFi using
> conan. I also verified backward compatibility for building these components
> in standalone CMake worked too. We achieved similar results for GTESTs where
> out of 230 tests, 99% of them passed for conan build and 100% passed for
> standalone CMake build.
> *UPDATE (May 11, 2024)* -
> [f5cfb79|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/f5cfb792bed5d94bd3c55217ed8f32e777f741d1]:
> {*}Enabled Build MQTT & PCAP MiNiFi PCAP Extension by Conan & CMake{*}.
> Refactored the pcap extension CMake code moving the part that downloads and
> builds pcapplusplus over to its own PcapPlusPlus.cmake file. Also added conan
> build condition to that .cmake file, so we can build PcapPlusPlus using conan
> while still keeping backward compatibility using standalone CMake approach.
> While I was able to build MiNiFi C++ with PCAP Extension enabled using
> standalone CMake approach, when I built it using conan build approach, I
> found pcap extension complaining about libminifi core headers and realized
> include_directories(include) didnt share all the header directory paths from
> libminifi to pcap extension. Thus, I created a python script under
> libminifi/cmake/python to parse for the libminifi header directories
> generating a include_dirs.cmake file with the LIBMINIFI_INCLUDE_DIRS CMake
> variable list that gets stored in build folder and followed by
> libminifi/include_dirs.cmake. That way we keep our source libminifi directory
> and its subdirectories clean. Overall, I was able to build MiNiFi C++ with
> MQTT and PCAP extensions enabled using conan & cmake. conan build achieved
> 99% (222/223) of GTESTs passing. standalone cmake build achieved 100% of
> GTESTs passing.
>
> *UPDATE (May 9, 2024)* -
> {*}[63290db|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/63290db57b60054ba8ed32c0b3a7d1f87f315314]{*}:
> *Refactored IODBC and SOCI libraries, so they are now CMake Bundles for
> easier integration with Conan.* Thus, minifi-sql library now calls
> include(BundledIODBC), use_bundled_iodbc(...) and include(BundledSOCI) and
> use_bundled_soci(...). Verified I can still build MiNiFi C++ using standalone
> CMake. 100% (220/220) of GTESTS passed using standalone CMake. Integrated
> conan for building SOCI & ODBC for MiNiFi SQL extension.
> *[4232399|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/4232399657c9da42dff57ff2b6806aeea2485444]*
> *Added a SOCI conan recipe to create SOCI conan package with unixODBC conan
> package, so SOCI comes with targets SOCI::libsoci_core and
> SOCI::lib;soci_odbc.* Verified I can build MiNiFi C++'s minifi-sql extension
> using conan build. However, with the conan build approach for minifi-sql
> extension, I was only able to build minifi-sql as a STATIC library
> succesfully. When I tried building minifi-sql as a SHARED library, I ran into
> multiple undefined references to SOCI functions. I think to fix the
> minifi-sql SHARED lib with conan will require making further updates to the
> SOCI conanfile.py's package_info() method. After building MiNiFi with conan
> including minifi-sql, I ran GTESTs and 99% of tests passed. I will look
> further into minifi-sql SHARED library to bring that approach later. For now,
> at least we can build minifi-sql as a STATIC library.
>
> *UPDATE (May 9, 2024)* -
> [ca3d6df|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/ca3d6df318162e2ee9d3e76e173d9f83e17d3010]:
> {*}Enabled MiNiFi C++ extensions GPS and COAP{*}. I added conan recipe for
> libcoap keeping the windows cmake patches for minifi. Since the original
> libcoap conan recipe's conanfile.py on conan-center-index was tailored for
> CMake building libcoap versions > 4.3.0, I modified the recipe, so we can
> build libcoap 4.2.1, which didnt have CMake support yet and thus, I updated
> that conanfile.py to run 'autogen.sh' to create the configure script, I
> passed the configure args to the configure script and then I used conan's
> Autotools() to run the configure script and then ran the generated make file
> to build libcoap 4.2.1. I updated BundledLibCoAP to account for the conan
> way, but kept backward compatibility for standalone CMake way. For a later
> update when I test on Windows, I need to add support into the libcoap 4.2.1
> conanfile.py for adding and applying the windows cmake patch to build
> libcoap. For now, I am mainly testing this new conan build infra for MiNiFi
> on Ubuntu 22.04 docker container and backward compatibility for standalone
> CMake on Ubuntu too. 99% of GTESTs passed (215/216).
>
> *(UPDATE (May 8, 2024) -*
> *[93fe5d1|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/93fe5d1a712834b422d5c8e45829a3641852ca7a]:*
> {color:#172b4d}*Enabled NANOFI* lib and binary executable, {*}Enabled MiNiFi
> C++ extensions SYSTEMD, PROCFS, LIBARCHIVE and XY_UTILS (LZMA){*}. I added
> conan recipe for liblzma (xy_utils) and libarchive keeping the patches for
> minifi. One area I found an issue with was integrating openssl conan package
> with libarchive conan package and libarchive complained about not being able
> to find openssl::openssl target. I did try to solve the issue by updating the
> libarchive conan recipe, but wasn't able to solve that openssl integration
> issue. I think the problem comes from libarchive CMakeLists.txt and its
> integration with conan in the repo. I opened an issue on conan-center-index
> to help with resolving this issue in case I get around to solving it in time.
> Good news is that I was still able to create libarchive conan package with
> the configuration settings similar to how we do it the standalone CMake
> BundledLibArchive.cmake way just in libarchive conan recipe. I also verified
> I can build these minifi extensions (LibArchive, LZMA, Procfs, Systemd) and
> NANOFI using conan build and standalone CMake. The standalone CMake way still
> allows us to build LibArchive with OpenSSL enabled. 213/214 (99%) GTESTs
> passed after building MiNiFi C++ with conan.{color}
>
> *UPDATE (May 7, 2024) -
> [*2460e84*|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/2460e846832d693c991e1da632b86ba2d4e34516]:*
> *Enabled OPS and OPC* MiNiFi Options, so those *MiNiFi C++ extensions* are
> built using *Conan CMake* and their bundled CMake dependencies use their
> particular *{{open62541/1.3.3@minifi/dev}}* and
> {*}{{mbedtls/2.16.3@minifi/dev}} conan packages{*}. I had to fix the mbedtls
> conan package from conan center index since it wasnt compatible when
> open62541 tried to integrate with mbedtls. In the thirdparty folder, you will
> see on line 139 where I updated libembedtls to be libmbedtls and fixing that
> typo allowed open62541 conan package to be able to be created with mbedtls as
> the encryption. I plan to make a follow up PR on conan center index for
> mbedtls conan recipe. Currently, we are able to build MiNiFi's additional
> extensions for OPS and OPC using conan. Verified backward compatibility with
> standalone CMake works, so I can also just build MiNiFi using CMake only.
>
> *UPDATE (May 5, 2024) -
> [446ddb6|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/446ddb6db1a0c76ba8c10ce6b5aaa14d48e8bbcb]:*
> When building *MiNiFi C++ GTESTs for MiNiFi Core, MainExe* and
> *Standard-Processors* and the other relevant that needed to be activated for
> these 3 essential components, I also set SKIP_TESTS to OFF, I set
> ENABLED_EXPRESSION_LANGUAGE to ON and ENABLE_ROCKSDB to ON to build GTESTs
> using conan build. I updated Bundle Rocksdb and Find Rocksdb to account for
> conan approach while keeping backward compatibility with standalone CMake
> approach. I got the following error coming from extension
> robksdb-repos/database/RocksDbInstance.cpp on no match for operator, and I
> will add the fix for this issue:
> {code:java}
> /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/extensions/rocksdb-repos/database/RocksDbInstance.cpp:71:26:
> error: no match for ‘operator!=’ (operand types are ‘rocksdb::DBOptions’ and
> ‘rocksdb::DBOptions’)
> 71 | if (db_opts_copy != db_options_) {
> | ~~~~~~~~~~~~ ^~ ~~~~~~~~~~~
> | | |
> | | rocksdb::DBOptions
> | rocksdb::DBOptions {code}
> We ran into this issue since when we created rocksdb/8.10.2 conan package, we
> didn't apply the 2 patches like was done in the standalone CMake approach, so
> I updated our conan approach where I created a rocksdb conan recipe in the
> nifi-minifi-cpp/thirdparty/rocksdb/ folder and added the 2 patches and
> applied them to create the rocksdb conan package successfully. Also I do want
> to note that I referenced the rocksdb conan recipe from conan center index
> repo. After creating the rocksdb conan package, then I built MiNiFi C++
> GTESTs for MiNiFi Core, MainExe, Standard-Processors and the other relevant
> extensions that needed to be activated for these 3 essential components using
> conan build. Here we have the results after running ctest in the build folder:
> {code:java}
> 192/194 Test #192: ConfigFileEncryptorTests .................... Passed
> 0.02 sec
> Start 193: ConfigFileTests
> 193/194 Test #193: ConfigFileTests ............................. Passed
> 0.01 sec
> Start 194: ControllerTests
> 194/194 Test #194: ControllerTests .............................***Failed
> 0.52 sec
> 99% tests passed, 2 tests failed out of 194
> Total Test time (real) = 382.89 sec
> The following tests FAILED: 45 - LoggerTests (Failed) 194 -
> ControllerTests (Failed)Errors while running CTestOutput from these tests are
> in:
> /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/build_conan/Testing/Temporary/LastTest.logUse
> "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
> {code}
> *We can see that 99% of the tests passed. Thus, 192/194 tests passed. This
> was using conan to build MiNiFi C++.*
>
> I also verified I can build *MiNiFi C++ GTESTs for MiNiFi Core, MainExe* and
> *Standard-Processors* and the other relevant that needed to be activated for
> these 3 essential components, I also set SKIP_TESTS to OFF, I set
> ENABLED_EXPRESSION_LANGUAGE to ON, ENABLE_BZIP2 to ON, BUILD_ROCKSDB to ON,
> ENABLE_ROCKSDB to ON to {*}build GTESTs using standalone CMake{*}. Here we
> can see similar results with running the MiNiFi GESTs:
>
> {code:java}
> 192/194 Test #192: ConfigFileEncryptorTests .................... Passed
> 0.07 sec
> Start 193: ConfigFileTests
> 193/194 Test #193: ConfigFileTests ............................. Passed
> 0.08 sec
> Start 194: ControllerTests
> 194/194 Test #194: ControllerTests .............................***Failed
> 0.76 sec
> 99% tests passed, 1 tests failed out of 194
> Total Test time (real) = 236.38 sec
> The following tests FAILED: 194 - ControllerTests (Failed)Errors while
> running CTestOutput from these tests are in:
> /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/build_cmake2/Testing/Temporary/LastTest.logUse
> "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
> {code}
> We can see that 99% of the tests passed. Thus, 193/194 tests passed. This was
> using standalone CMake to build MiNiFi C++.
>
> {*}{color:#0747a6}UPDATE (May 4, 2024) -
> {color}[7eb8beb|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/7eb8bebf79f5d09c4e508402161668ef8bcd8c45]{*}:
> Verified I can build MiNiFi C++ with standalone CMake like how we currently
> do it even after integrating conan into the build infrastructure. CMake built
> *MiNiFi Core, Main* and *Standard Processors* successfully. Also double
> checked that conan build MiNiFi still works. (Feel free to reference the logs
> for the CMake standalone build and the conan build of MiNiFi CPP)
>
> {color:#0747a6}{*}UPDATE (May 4, 2024) -
> [2d1b884|https://github.com/apache/nifi-minifi-cpp/pull/1775/commits/2d1b8842c6809380f981f9857543805efbfd00cc]{*}:{color}
> Updated MiNiFi CPP code with Conan & CMake to build libMiNiFi *core-minifi*
> and MiNiFi Main's *minifiexe* and {*}minifi-{*}{*}standard-processors{*}
> extension with minimal set of external libraries needed. If you look at our
> conanfile.py, you'll see {*}minifi_core_external_libraries{*}, which is a
> tuple of the minimal external libraries needed to build these two essential
> building blocks of MiNFi CPP. Next when you look at conanfile.py
> *MiNiFiCppMain* class's generate(self) method, you'll see I have enabled
> USE_CONAN_PACKAGER to ON, SKIP_TESTS to ON, and then the essential external
> library dependencies that aren needed for the *core-minifi* library and
> *minifiexe* binary executable, which are {*}ENABLE_OPENWSMAN, ENABLE_OPENSSL,
> ENABLE_CIVET, ENABLE_CURL{*}. I notice most of these external libraries are
> in the extensions folder and add some dependencies on these custom extensions
> that are created. {color:#ff8b00}*I think we should consider creating an
> extensions subfolder for core-minifi related extensions.*{color}
> Here is the output I get after building the MiNiFi CPP code, you can see
> MiNiFi CPP libraries and binary executables built successfully using *CONAN
> version 2* and CMake (feel free to check the attached log for the full conan
> build output): [~szaszm]
>
> {code:java}
> Adding minifi-standard-processors to EXTENSIONS_ENABLED_BY_DEFAULT list
> Checking for enabled extensions to append to EXTENSIONS_ENABLED_BY_DEFAULT
> list
> Civet Enabled appending to EXTENSIONS_ENABLED_BY_DEFAULT
> -- Creating API Documentation using /usr/bin/doxygen
> -- Configuring done
> -- Generating done
> -- Build files have been written to:
> /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/buildconanfile.py
> (minifi-cpp-main/0.15.0): Running CMake.build()
> conanfile.py (minifi-cpp-main/0.15.0): RUN: cmake --build
> "/home/ubuntu/src/james/pipeline/nifi-minifi-cpp/build" -- -j20
> [ 3%] Built target ossp-uuid-external-build
> [ 6%] Built target openwsman-external
> [ 7%] Built target ossp-uuid-external
> Consolidate compiler generated dependencies of target core-minifi
> [ 8%] Building CXX object
> libminifi/CMakeFiles/core-minifi.dir/agent_version.cpp.o
> [ 8%] Linking CXX shared library ../bin/libcore-minifi.so
> [ 78%] Built target core-minifi
> Consolidate compiler generated dependencies of target minifi-civet-extensions
> Consolidate compiler generated dependencies of target minifi-openwsman
> Consolidate compiler generated dependencies of target minifi-http-curl
> Consolidate compiler generated dependencies of target minifiexe
> [ 78%] Linking CXX shared library ../../bin/libminifi-openwsman.so
> [ 79%] Linking CXX shared library ../../bin/libminifi-civet-extensions.so
> Consolidate compiler generated dependencies of target
> minifi-standard-processors
> [ 79%] Building CXX object
> minifi_main/CMakeFiles/minifiexe.dir/MiNiFiMain.cpp.o
> [ 79%] Building CXX object
> minifi_main/CMakeFiles/minifiexe.dir/MainHelper.cpp.o
> [ 79%] Building CXX object
> minifi_main/CMakeFiles/minifiexe.dir/AgentDocs.cpp.o
> [ 80%] Building CXX object
> minifi_main/CMakeFiles/minifiexe.dir/MiNiFiWindowsService.cpp.o
> [ 81%] Building CXX object
> minifi_main/CMakeFiles/minifiexe.dir/TableFormatter.cpp.o
> [ 82%] Linking CXX shared library ../../bin/libminifi-http-curl.so
> [ 83%] Linking CXX shared library ../../bin/libminifi-standard-processors.so
> [ 84%] Built target minifi-civet-extensions
> [ 86%] Built target minifi-http-curl
> [ 87%] Built target minifi-openwsman
> [100%] Built target minifi-standard-processors
> [100%] Linking CXX executable ../bin/minifi
> [100%] Built target minifiexe {code}
> NEXT: I will verify I can build these MiNiFi CPP libraries and binary
> executables using USE_CMAKE_FETCH_CONTENT option set to ON and
> USE_CONAN_PACKAGER option set to OFF for the for the follow up commit.
>
> Tracking Prebuilt Binary Conan Packages Available on Conancenter:
> * 'abseil/20230125.3',
> * 'argparse/3.0',
> * 'asio/1.30.2'
> Tracking Conan Packages Not Available on Conancenter, so we need to use an
> alternative hosting service to manage building package sources into prebuilt
> binary conan packages. Here are some of the free conan hosting services for
> open-source C/C++ projects:
> * {*}Conan Center Index (CCI){*}: official, free, centralized repo for conan
> packages maintained by conan team
> * Bincrafters Bintray Repo: community-driven repo that hosts large number of
> pre-built conan packages for open-source libraries for free
> * Artifactory by JFrog: typically used as a commercial hosting service, but
> also provides a free tier for open-source projects.
> * {*}Self-Hosting{*}: While there are multiple options, I will focus on
> *GitHub Packages* as its a free package hosting service, which can be used to
> host our own conan package repos, allowing us to store and distribute our
> pre-built conan packages without having to use 3rd party services like
> Bincrafters or pay for commercial solution.
> ** {*}Advantage with GitHub Packages{*}: its free, integrated with our
> existing GitHub workflow, allows us full control hosting our conan packages
> without relying on 3rd party services, useful if we need to have specific
> requirements or need flexibility in managing our package hosting.
> Tracking Non Prebuilt Binary Conan Packages, which later I will create a
> *GitHub Packages* repo for MiNiFi CPP prebuilt conan packages (also account
> for Linux, Windows, Mac, etc):
> * bustache: [https://github.com/jamboree/bustache]
>
> {*}FUTURE IMPROVEMENTS{*}: Some more build infra we can improve in MiNiFi CPP
> after I finish my work for MINFICPP-2346: While I am working on migrating
> MiNiFi CPP build infrastructure toward leveraging conan packager, for a later
> PR we could do the following:
> # Put the external libraries that MiNiFi CPP Core depends on into its own
> *BundledMiNiFiCoreExtLibs.cmake* (also feel free to suggest some other
> names), we can run this CMake code before adding *add_subdirectory(libminifi)*
> ## External libs that libminifi needs that are in MiNiFi CPP's root
> CMakeLists.txt (refactor for better organization):
> yaml-cpp, ZLIB::ZLIB, concurrentqueue, RapidJSON, spdlog, Threads::Threads,
> gsl-lite, libsodium, range-v3, expected-lite, date::date, date::tz, asio,
> magic_enum, RangeV3, Asio, MagicEnum
>
> ## We can also clean up the *thirdparty* folder as we migrate to conan
> packager for managing these depedencies, so in MiNiFi C+{+}, we mainly keep
> it MiNiFi C{+}+ code and its extension processors, etc.
> # I tried building MiNiFi CPP Core just with just
> *minifi-standard-processors* and the other extensions disabled, but it seems
> even some of MiNiFi CPP Core depends on some of the extensions to be built
> with it, so it doesn't fail building, such as Openwsman. If we don't have
> ENABLE_OPENWSMAN, then we get this error when building MiNiFi CPP Core, so we
> should look having a build config to build minimal MiNiFi CPP Core:
> {code:java}
> libminifi/src/controllers/SSLContextService.cpp:520:80: error:
> ‘X509_unique_ptr’ in namespace ‘org::apache::nifi::minifi::utils::tls{code}
>
> # I noticed that when building MiNiFi Core, MainExe, Standard-Processors and
> their external lib dependencies and then building the appropriate GTEST, I
> get a header error coming from rocksdb, but I haven't enabled rocksdb, so I
> think at a later point we should shift that test out, so when building these
> 3 core building blocks of MiNiFi, they dont rely on rocksdb unless MiNiFi
> Core and MainExe have a dependency for storing into that DB. Here is an
> example of the header error (I will ENABLE_ROCKSDB to ON):
> {code:java}
> [ 99%] Built target ControllerTests
> In file included from
> /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/libminifi/include/../../extensions/rocksdb-repos/DatabaseContentRepository.h:30,
> from
> /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/libminifi/test/flow-tests/SessionTests.cpp:24:
> /home/ubuntu/src/james/pipeline/nifi-minifi-cpp/libminifi/include/../../extensions/rocksdb-repos/database/RocksDatabase.h:26:10:
> fatal error: rocksdb/db.h: No such file or directory
> 26 | #include "rocksdb/db.h" {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
