[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #914: MINIFICPP-1323 Encrypt sensitive properties using libsodium
szaszm commented on a change in pull request #914:
URL: https://github.com/apache/nifi-minifi-cpp/pull/914#discussion_r498584923
##
File path: encrypt-config/ConfigFile.cpp
##
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ConfigFile.h"
+
+#include
+#include
+#include
+
+#include "utils/StringUtils.h"
+
+namespace {
+constexpr std::array
DEFAULT_SENSITIVE_PROPERTIES{"nifi.security.client.pass.phrase",
Review comment:
You can find all of the properties in
`libminifi/include/properties/Configure.h`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] alopresto commented on a change in pull request #914: MINIFICPP-1323 Encrypt sensitive properties using libsodium
alopresto commented on a change in pull request #914:
URL: https://github.com/apache/nifi-minifi-cpp/pull/914#discussion_r498583289
##
File path: encrypt-config/ConfigFile.cpp
##
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ConfigFile.h"
+
+#include
+#include
+#include
+
+#include "utils/StringUtils.h"
+
+namespace {
+constexpr std::array
DEFAULT_SENSITIVE_PROPERTIES{"nifi.security.client.pass.phrase",
Review comment:
Is there a canonical enumeration of the default properties somewhere?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] levilentz commented on pull request #4204: NIFI-7355 Added Gremlin bytecode client service.
levilentz commented on pull request #4204: URL: https://github.com/apache/nifi/pull/4204#issuecomment-702476413 > @levilentz can you take a look at this and get back to me on what improvements you made need to be incorporated? @MikeThomsen, the main changes that I have are the following: 1. Allow for custom Jar specification 2. Allow for custom Class import 3. Allow for specification of Yaml connection string or file 4. Logic to allow for more complex object returns from the groovy script (right now it only plays well with simple-ish objects) 1, 2, and 3 are really important for connecting to services such as Janusgraph that are "gremlin-compliant" but are more efficient when you use their encoders/decoders. 4 is a nice to have and might be outside the scope of this Ticket. Let me know what you think and if you want me to add that code into this branch. Thanks for all your work on this! This will be a really nice addition to the graph bundle. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-7844) substring should return empty instead of throwing IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/NIFI-7844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Thomsen updated NIFI-7844:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> substring should return empty instead of throwing IndexOutOfBoundsException
> ---
>
> Key: NIFI-7844
> URL: https://issues.apache.org/jira/browse/NIFI-7844
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> In the expression language, right now, for the substring function:
> {quote}If the _starting index_ or the _ending index_ is greater than the
> length of the Subject or has a value less than 0, this function call will
> result in an error.
> {quote}
> I suggest to change this behavior and return an empty string instead. Reason
> being that with the current behavior, an UpdateAttribute would fail and roll
> back the flow file in the incoming relationship, this could cause back
> pressure and completely block the whole flow to be running.
> It might be worth mentioning it in the migration guide as this changes the
> current behavior but I still think this is a reasonable change.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7844) substring should return empty instead of throwing IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/NIFI-7844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205885#comment-17205885
]
ASF subversion and git services commented on NIFI-7844:
---
Commit f32405ed16b7e07a0d445f1ed19032acaf33246d in nifi's branch
refs/heads/main from Pierre Villard
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f32405e ]
NIFI-7844 - substring should return empty instead of throwing
IndexOutOfBoundsException
This closes #4553
Signed-off-by: Mike Thomsen
> substring should return empty instead of throwing IndexOutOfBoundsException
> ---
>
> Key: NIFI-7844
> URL: https://issues.apache.org/jira/browse/NIFI-7844
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In the expression language, right now, for the substring function:
> {quote}If the _starting index_ or the _ending index_ is greater than the
> length of the Subject or has a value less than 0, this function call will
> result in an error.
> {quote}
> I suggest to change this behavior and return an empty string instead. Reason
> being that with the current behavior, an UpdateAttribute would fail and roll
> back the flow file in the incoming relationship, this could cause back
> pressure and completely block the whole flow to be running.
> It might be worth mentioning it in the migration guide as this changes the
> current behavior but I still think this is a reasonable change.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] asfgit closed pull request #4553: NIFI-7844 - substring should return empty instead of throwing IndexOu…
asfgit closed pull request #4553: URL: https://github.com/apache/nifi/pull/4553 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-7870) X-Content-Type missing for advanced UI resources
[
https://issues.apache.org/jira/browse/NIFI-7870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205871#comment-17205871
]
Nathan Gough commented on NIFI-7870:
Sounds like this may be more complicated than first anticipated. Upon
investigation, it appears that the advanced UI static assets are not accessible
due to another change in -NIFI-7170- which was intended to disable anonymous
access to API endpoints. This was not intended to block access to static
assets.--
When using LDAP, OIDC and perhaps other authentication mechanisms other than
X509, a 'HTTP 401 Unauthorized' is returned for advanced UI static assets,
which results in the Content-Type being returned as 'text/plain' (an error
message) instead of the expected 'application/javascript' Content-Type. The
server will show:
{code:java}
2020-10-01 19:30:38,159 INFO [NiFi Web Server-19]
o.a.n.w.s.NiFiAuthenticationFilter Attempting request for () GET
https://localhost:8443/nifi-jolt-transform-json-ui-1.12.1/app/transformjson/transformjson.service.js
(source ip: 172.17.0.1)
2020-10-01 19:30:38,159 WARN [NiFi Web Server-19]
o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Anonymous
authentication has not been configured.{code}
This is because the NiFi UI does not include the JWT for requests to static
assets, but authentication is being checked as anonymous authentication is now
disabled by default. So, a 401 is returned.
The current workaround is to enable anonymous authentication which will allow
access to static assets for advanced UI's (javascript etc):
nifi.security.allow.anonymous.authentication=true.
A fix for this will likely involve a combination of explicitly allowing access
to advanced UI static assets for UI's bundled with NiFi, and for external
custom UI's, anonymous authentication will need to be enabled.
We expect that this will be a temporary change, with the long term goal of
refactoring/redesigning some of these security features in a future release/a
NiFi 2.0 release.
Also related is NIFI-7489
> X-Content-Type missing for advanced UI resources
>
>
> Key: NIFI-7870
> URL: https://issues.apache.org/jira/browse/NIFI-7870
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
>Affects Versions: 1.12.0, 1.12.1
>Reporter: Nathan Gough
>Assignee: Nathan Gough
>Priority: Critical
> Labels: UI, content-type, header, security
>
> The X-Content-Type header was added in NiFi 1.12.0, which blocks resources in
> the browser if they do not have the content type added. It appears that some
> 'advanced UI' resources do not have the content type applied to their
> resources and are blocked from loading.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7685) Add UTF8 support for FetchFTP
[ https://issues.apache.org/jira/browse/NIFI-7685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205840#comment-17205840 ] ASF subversion and git services commented on NIFI-7685: --- Commit 6990f0d3a9962144c4ab9156780e633023f9 in nifi's branch refs/heads/main from Denes Arvay [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=6990f0d ] NIFI-7685: Add UTF8 support for FetchFTP Signed-off-by: Matthew Burgess This closes #4446 > Add UTF8 support for FetchFTP > - > > Key: NIFI-7685 > URL: https://issues.apache.org/jira/browse/NIFI-7685 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Luca Giovannini >Assignee: Denes Arvay >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > When FetchFTP tries to fetch a file with name containing non-ASCII characters > (e.g. “Xml_APE001_unit*à*immobiliareguarda.xml”), the process succeeds > but the corresponding flowfile is empty. > > The issue is the same as the one in [1] but it just wasn't fixed for > FetchFTP. > [1] > [https://issues.apache.org/jira/browse/NIFI-4137|https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FNIFI-4137=02%7C01%7CLuca.Giovannini%40dedagroup.it%7C35da53a2e1db4aadb69a08d82800df73%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637303331502181485=ETqdW2puaeZKwDl1cr4raz8oll29qGFSN6fqVKO9%2FI4%3D=0] > > Thank you! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7685) Add UTF8 support for FetchFTP
[ https://issues.apache.org/jira/browse/NIFI-7685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-7685: --- Fix Version/s: 1.13.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Add UTF8 support for FetchFTP > - > > Key: NIFI-7685 > URL: https://issues.apache.org/jira/browse/NIFI-7685 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Luca Giovannini >Assignee: Denes Arvay >Priority: Major > Fix For: 1.13.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > When FetchFTP tries to fetch a file with name containing non-ASCII characters > (e.g. “Xml_APE001_unit*à*immobiliareguarda.xml”), the process succeeds > but the corresponding flowfile is empty. > > The issue is the same as the one in [1] but it just wasn't fixed for > FetchFTP. > [1] > [https://issues.apache.org/jira/browse/NIFI-4137|https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FNIFI-4137=02%7C01%7CLuca.Giovannini%40dedagroup.it%7C35da53a2e1db4aadb69a08d82800df73%7Cbbf156d433fa4fee86f62cfcb1359ef0%7C0%7C0%7C637303331502181485=ETqdW2puaeZKwDl1cr4raz8oll29qGFSN6fqVKO9%2FI4%3D=0] > > Thank you! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] mattyb149 closed pull request #4446: NIFI-7685: Add UTF8 support for FetchFTP
mattyb149 closed pull request #4446: URL: https://github.com/apache/nifi/pull/4446 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] mattyb149 commented on pull request #4446: NIFI-7685: Add UTF8 support for FetchFTP
mattyb149 commented on pull request #4446: URL: https://github.com/apache/nifi/pull/4446#issuecomment-702411269 +1 LGTM, ran contrib-check and verified expected behavior on FTP servers with and without UTF8 support. Thanks for the improvement! Merging to main This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] jfrazee commented on pull request #4460: NIFI-7717 Updated main NiFi images to JRE 11.
jfrazee commented on pull request #4460: URL: https://github.com/apache/nifi/pull/4460#issuecomment-702386711 @MikeThomsen I'm going to nit on something. This being called `JAVA_VERSION` would make me think it'd be something like 11 or 1.8 or one of the many openjdk tags and not the base image name. That brings up another question. What's the tradeoff between having the base image be completely specified vs. just a tag for the openjdk image? My thinking is: 1. If it was just the tag for openjdk it's more likely the image will be as expected (since most but not all of the openjdk tags are Debian based). 2. Whereas allowing the entire base image to be changed would let people use the released source to build images of their own, let's say hardened or approved images, with the drawback of it not working if they provide a non-Debian based image. (That said, it's very light on Debian assumptions, so there might be even some room for some easy changes to make it work with other distros later.) I think (2) is reasonable motivation. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-6738) ListenBeats receives partial messages
[ https://issues.apache.org/jira/browse/NIFI-6738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205804#comment-17205804 ] Scott Nicholas commented on NIFI-6738: -- I'm seeing this too. Surprised more don't. > ListenBeats receives partial messages > - > > Key: NIFI-6738 > URL: https://issues.apache.org/jira/browse/NIFI-6738 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.9.2 > Environment: Ubuntu 16.04.6 LTS (GNU/Linux 4.4.0-159-generic x86_64) >Reporter: John Black >Priority: Major > Attachments: image-2019-10-02-22-17-04-236.png, > image-2019-10-02-22-19-45-813.png, image-2019-10-02-22-21-07-982.png > > > Hi! > I receive windows events in ListenBeats from winlogbeats, however some > messages (usually those which bigger than 3 KB) arrive truncated. I observe > it in Data Provenance. I checked the content of outgoing packets on > winlogbeat side - the messages are shipped properly. Sometimes one FlowFile > in Data Provenance has part of one message followed by non-printable symbols > (probably winlogbeat's header) combined with truncated itself. Please find > attached a few screenshots below. > !image-2019-10-02-22-17-04-236.png! > !image-2019-10-02-22-19-45-813.png! > !image-2019-10-02-22-21-07-982.png! > > !https://aws1.discourse-cdn.com/elastic/original/3X/9/f/9fd25b03f089e2c19e29b34244c18519a12b8b1f.jpeg! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] turcsanyip commented on a change in pull request #4510: NIFI-7549 Adding Hazelcast based DistributedMapCacheClient support
turcsanyip commented on a change in pull request #4510: URL: https://github.com/apache/nifi/pull/4510#discussion_r498473697 ## File path: nifi-nar-bundles/nifi-hazelcast-bundle/nifi-hazelcast-services/src/main/resources/org.apache.nifi.hazelcast.services.cacheclient.HazelcastMapCacheClient/additionalDetails.html ## @@ -0,0 +1,46 @@ + Review comment: Additional Details pages are not available on the UI. These files should be in `resources/docs/...` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-7871) Correct errors in documentation for UUID3, UUID5 and hash in Expression Language Guide
Andrew M. Lim created NIFI-7871: --- Summary: Correct errors in documentation for UUID3, UUID5 and hash in Expression Language Guide Key: NIFI-7871 URL: https://issues.apache.org/jira/browse/NIFI-7871 Project: Apache NiFi Issue Type: Bug Components: Documentation Website Reporter: Andrew M. Lim When the documentation for hash function was added, it was misplaced to interfere with existing UUID3 and UUID5 documentation ([https://github.com/apache/nifi/commit/0f4b79b55ec7e4a85334d4a0d3e7200021950d1a#diff-daac74ec3b89e26d99806d7a90254fe3).] The examples provided for UUID3 and UUID5 could also be improved. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (NIFI-7871) Correct errors in documentation for UUID3, UUID5 and hash in Expression Language Guide
[ https://issues.apache.org/jira/browse/NIFI-7871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew M. Lim reassigned NIFI-7871: --- Assignee: Andrew M. Lim > Correct errors in documentation for UUID3, UUID5 and hash in Expression > Language Guide > -- > > Key: NIFI-7871 > URL: https://issues.apache.org/jira/browse/NIFI-7871 > Project: Apache NiFi > Issue Type: Bug > Components: Documentation Website >Reporter: Andrew M. Lim >Assignee: Andrew M. Lim >Priority: Major > > When the documentation for hash function was added, it was misplaced to > interfere with existing UUID3 and UUID5 documentation > ([https://github.com/apache/nifi/commit/0f4b79b55ec7e4a85334d4a0d3e7200021950d1a#diff-daac74ec3b89e26d99806d7a90254fe3).] > The examples provided for UUID3 and UUID5 could also be improved. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] szaszm edited a comment on pull request #910: MINIFICPP-1375 Windows: Redistribute Universal CRT DLLs with our MSI
szaszm edited a comment on pull request #910: URL: https://github.com/apache/nifi-minifi-cpp/pull/910#issuecomment-702308647 Reopened because we can keep the MSI generation in our repo and make it package non-free DLLs, but we cannot redistribute the DLLs or the generated installer, so at this point, it becomes just a convenience script for our users. We will need to create a license prompt in the MSI for the visual c++ redist and ucrt redist dlls in a followup issue. https://issues.apache.org/jira/browse/LEGAL-540 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on pull request #910: MINIFICPP-1375 Windows: Redistribute Universal CRT DLLs with our MSI
szaszm commented on pull request #910: URL: https://github.com/apache/nifi-minifi-cpp/pull/910#issuecomment-702308647 Reopened because we can keep the script in our repo. We will need to create a license prompt in the MSI for the visual c++ redist and ucrt redist dlls in a followup issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm edited a comment on pull request #910: MINIFICPP-1375 Windows: Redistribute Universal CRT DLLs with our MSI
szaszm edited a comment on pull request #910: URL: https://github.com/apache/nifi-minifi-cpp/pull/910#issuecomment-702308647 Reopened because we can keep the script in our repo. We will need to create a license prompt in the MSI for the visual c++ redist and ucrt redist dlls in a followup issue. https://issues.apache.org/jira/browse/LEGAL-540 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-7835) SFTP processors (maybe other too) proxy doesn't use credentials
[ https://issues.apache.org/jira/browse/NIFI-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wiktor Kubicki updated NIFI-7835: - Summary: SFTP processors (maybe other too) proxy doesn't use credentials (was: SFTP processors (maybe other too) can't use proxy) > SFTP processors (maybe other too) proxy doesn't use credentials > --- > > Key: NIFI-7835 > URL: https://issues.apache.org/jira/browse/NIFI-7835 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.12.0, 1.11.4 > Environment: CentOS 7, Java 8 and 11, standalone Nifi >Reporter: Wiktor Kubicki >Priority: Major > > Hello, > we tried use socks proxy configuration (with authorization) for > listSFTP/fetchSFTP/getSFTP using processor configuration or Proxy > Configuration Service. Each time we has got Getting java.net.SocketException: > Malformed. > We did tcp dump and saw, that the user/password is not provided - user is > same as system user rather than this one from configuration. > I've easly recreated issue on clean CentOS with java 8 and 11. Same problem > was on NiFi 1.11.4 and 1.12.0. > I think the problem is due of using sshj, but can be wrong. Very similar > issue is https://issues.apache.org/jira/browse/NIFI-7749 - probably with the > same cause. > > My workaround for sftp is using fuse-sftp and mount remote server as local > share. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #914: MINIFICPP-1323 Encrypt sensitive properties using libsodium
szaszm commented on a change in pull request #914:
URL: https://github.com/apache/nifi-minifi-cpp/pull/914#discussion_r498366948
##
File path: cmake/BundledLibSodium.cmake
##
@@ -0,0 +1,95 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+function(use_bundled_libsodium SOURCE_DIR BINARY_DIR)
+message("Using bundled libsodium")
+
+# Define patch step
+if (WIN32)
+set(PC "${Patch_EXECUTABLE}" -p1 -i
"${SOURCE_DIR}/thirdparty/libsodium/libsodium.patch")
+endif()
+
+# Define byproduct
+if (WIN32)
+set(BYPRODUCT "lib/sodium.lib")
+else()
+set(BYPRODUCT "lib/libsodium.a")
+endif()
+
+# Set build options
+set(LIBSODIUM_BIN_DIR "${BINARY_DIR}/thirdparty/libsodium-install" CACHE
STRING "" FORCE)
+
+if (WIN32)
+set(LIBSODIUM_CMAKE_ARGS ${PASSTHROUGH_CMAKE_ARGS}
+"-DCMAKE_INSTALL_PREFIX=${LIBSODIUM_BIN_DIR}"
+"-DSODIUM_LIBRARY_MINIMAL=1")
+endif()
+
+# Build project
+set(LIBSODIUM_URL
https://download.libsodium.org/libsodium/releases/libsodium-1.0.18.tar.gz)
+set(LIBSODIUM_URL_HASH
"SHA256=6f504490b342a4f8a4c4a02fc9b866cbef8622d5df4e5452b46be121e46636c1")
+
+if (WIN32)
+ExternalProject_Add(
+libsodium-external
+URL ${LIBSODIUM_URL}
+URL_HASH ${LIBSODIUM_URL_HASH}
+SOURCE_DIR "${BINARY_DIR}/thirdparty/libsodium-src"
+LIST_SEPARATOR % # This is needed for passing
semicolon-separated lists
+CMAKE_ARGS ${LIBSODIUM_CMAKE_ARGS}
+PATCH_COMMAND ${PC}
+BUILD_BYPRODUCTS "${LIBSODIUM_BIN_DIR}/${BYPRODUCT}"
+EXCLUDE_FROM_ALL TRUE
+)
+else()
+set(CONFIGURE_COMMAND ./configure --disable-pie --enable-minimal
"--prefix=${LIBSODIUM_BIN_DIR}")
+
+ExternalProject_Add(
+libsodium-external
+URL ${LIBSODIUM_URL}
+URL_HASH ${LIBSODIUM_URL_HASH}
+BUILD_IN_SOURCE true
+SOURCE_DIR "${BINARY_DIR}/thirdparty/libsodium-src"
+BUILD_COMMAND make
+CMAKE_COMMAND ""
+UPDATE_COMMAND ""
+INSTALL_COMMAND make install
+BUILD_BYPRODUCTS "${LIBSODIUM_BIN_DIR}/${BYPRODUCT}"
+CONFIGURE_COMMAND "${CONFIGURE_COMMAND}"
+PATCH_COMMAND ""
+STEP_TARGETS build
+EXCLUDE_FROM_ALL TRUE
+)
+endif()
+
+# Set variables
+set(LIBSODIUM_FOUND "YES" CACHE STRING "" FORCE)
+set(LIBSODIUM_INCLUDE_DIRS "${LIBSODIUM_BIN_DIR}/include" CACHE STRING ""
FORCE)
+set(LIBSODIUM_LIBRARIES "${LIBSODIUM_BIN_DIR}/${BYPRODUCT}" CACHE STRING
"" FORCE)
+
+# Set exported variables for FindPackage.cmake
+set(PASSTHROUGH_VARIABLES ${PASSTHROUGH_VARIABLES}
"-DEXPORTED_LIBSODIUM_INCLUDE_DIRS=${LIBSODIUM_INCLUDE_DIRS}" CACHE STRING ""
FORCE)
+set(PASSTHROUGH_VARIABLES ${PASSTHROUGH_VARIABLES}
"-DEXPORTED_LIBSODIUM_LIBRARIES=${LIBSODIUM_LIBRARIES}" CACHE STRING "" FORCE)
+
+# Create imported targets
+file(MAKE_DIRECTORY ${LIBSODIUM_INCLUDE_DIRS})
+
+add_library(libsodium STATIC IMPORTED)
+set_target_properties(libsodium PROPERTIES IMPORTED_LOCATION
"${LIBSODIUM_LIBRARIES}")
+add_dependencies(libsodium libsodium-external)
+set_property(TARGET libsodium APPEND PROPERTY
INTERFACE_INCLUDE_DIRECTORIES "${LIBSODIUM_INCLUDE_DIRS}")
Review comment:
:disappointed:
No, it's not yet worth bumping the required cmake version IMO, thanks for
the references.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] taftster commented on pull request #4221: NIFI-6394 - frontend queue/connection size limit
taftster commented on pull request #4221: URL: https://github.com/apache/nifi/pull/4221#issuecomment-702239736 OK @anaylor - I'll follow your lead. If you come up with a better design, then I can grab your commits and add them here. Or you can grab my commits and we'll merge your branch. Either way, whatever is easiest. And if you end up taking on the comment above (to deprecate the original api method, not remove it), that would be great too. But if you don't get to that, I will do it. In short, I will follow your lead here. Just let me know how to support. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #914: MINIFICPP-1323 Encrypt sensitive properties using libsodium
fgerlits commented on a change in pull request #914:
URL: https://github.com/apache/nifi-minifi-cpp/pull/914#discussion_r498356767
##
File path: encrypt-config/CMakeLists.txt
##
@@ -0,0 +1,25 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+file(GLOB ENCRYPT_CONFIG_FILES "*.cpp")
+add_executable(encrypt-config "${ENCRYPT_CONFIG_FILES}")
+target_include_directories(encrypt-config PRIVATE ../libminifi/include
../thirdparty/cxxopts/include)
+target_wholearchive_library(encrypt-config minifi)
+target_link_libraries(encrypt-config cxxopts)
Review comment:
fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #914: MINIFICPP-1323 Encrypt sensitive properties using libsodium
fgerlits commented on a change in pull request #914:
URL: https://github.com/apache/nifi-minifi-cpp/pull/914#discussion_r498356448
##
File path: encrypt-config/EncryptConfig.cpp
##
@@ -0,0 +1,150 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "EncryptConfig.h"
+
+#include
+
+#include
+
+#include "ConfigFile.h"
+#include "ConfigFileEncryptor.h"
+#include "cxxopts.hpp"
+#include "utils/file/FileUtils.h"
+#include "utils/OptionalUtils.h"
+
+namespace {
+constexpr const char* CONF_DIRECTORY_NAME = "conf";
+constexpr const char* BOOTSTRAP_FILE_NAME = "bootstrap.conf";
+constexpr const char* MINIFI_PROPERTIES_FILE_NAME = "minifi.properties";
+constexpr const char* ENCRYPTION_KEY_PROPERTY_NAME =
"nifi.bootstrap.sensitive.key";
+} // namespace
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace encrypt_config {
+
+EncryptConfig::EncryptConfig(int argc, char* argv[]) :
minifi_home_(parseMinifiHomeFromTheOptions(argc, argv)) {
+ if (sodium_init() < 0) {
+throw std::runtime_error{"Could not initialize the libsodium library!"};
+ }
+}
+
+std::string EncryptConfig::parseMinifiHomeFromTheOptions(int argc, char*
argv[]) {
+ cxxopts::Options options("encrypt-config", "Encrypt sensitive minifi
properties");
+ options.add_options()
+ ("h,help", "Shows help")
+ ("m,minifi-home", "The MINIFI_HOME directory",
cxxopts::value());
+
+ auto parse_result = options.parse(argc, argv);
+
+ if (parse_result.count("help")) {
+std::cout << options.help() << '\n';
+std::exit(0);
+ }
+
+ if (parse_result.count("minifi-home")) {
+return parse_result["minifi-home"].as();
+ } else {
+throw std::runtime_error{"Required parameter missing: --minifi-home"};
+ }
+}
+
+void EncryptConfig::encryptSensitiveProperties() const {
+ utils::crypto::Bytes encryption_key = getEncryptionKey();
+ encryptSensitiveProperties(encryption_key);
+}
+
+std::string EncryptConfig::bootstrapFilePath() const {
+ return utils::file::FileUtils::concat_path(
+ utils::file::FileUtils::concat_path(minifi_home_, CONF_DIRECTORY_NAME),
+ BOOTSTRAP_FILE_NAME);
+}
+
+std::string EncryptConfig::propertiesFilePath() const {
+ return utils::file::FileUtils::concat_path(
+ utils::file::FileUtils::concat_path(minifi_home_, CONF_DIRECTORY_NAME),
+ MINIFI_PROPERTIES_FILE_NAME);
+}
+
+utils::crypto::Bytes EncryptConfig::getEncryptionKey() const {
+ encrypt_config::ConfigFile
bootstrap_file{std::ifstream{bootstrapFilePath()}};
+ utils::optional key_from_bootstrap_file =
bootstrap_file.getValue(ENCRYPTION_KEY_PROPERTY_NAME);
+
+ if (key_from_bootstrap_file && !key_from_bootstrap_file->empty()) {
+std::string binary_key = hexDecodeAndValidateKey(*key_from_bootstrap_file);
+std::cout << "Using the existing encryption key found in " <<
bootstrapFilePath() << '\n';
+return utils::crypto::stringToBytes(binary_key);
+ } else {
+std::cout << "Generating a new encryption key...\n";
+utils::crypto::Bytes encryption_key = utils::crypto::generateKey();
+writeEncryptionKeyToBootstrapFile(encryption_key);
+std::cout << "Wrote the new encryption key to " << bootstrapFilePath() <<
'\n';
+return encryption_key;
+ }
+}
+
+std::string EncryptConfig::hexDecodeAndValidateKey(const std::string& key)
const {
+ std::string binary_key = utils::StringUtils::from_hex(key);
Review comment:
I have added a comment.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] anaylor commented on pull request #4221: NIFI-6394 - frontend queue/connection size limit
anaylor commented on pull request #4221: URL: https://github.com/apache/nifi/pull/4221#issuecomment-702209932 Hey @taftster I was planning on pulling in your changes to my existing branch but it would probably make more sense just to smash my changes onto here instead of opening a new PR. Its still just a link for now but intend on making it look nice. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-7870) X-Content-Type missing for advanced UI resources
[ https://issues.apache.org/jira/browse/NIFI-7870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy LoPresto updated NIFI-7870: Component/s: Core UI > X-Content-Type missing for advanced UI resources > > > Key: NIFI-7870 > URL: https://issues.apache.org/jira/browse/NIFI-7870 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.12.0, 1.12.1 >Reporter: Nathan Gough >Assignee: Nathan Gough >Priority: Critical > Labels: UI, content-type, header, security > > The X-Content-Type header was added in NiFi 1.12.0, which blocks resources in > the browser if they do not have the content type added. It appears that some > 'advanced UI' resources do not have the content type applied to their > resources and are blocked from loading. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7870) X-Content-Type missing for advanced UI resources
[ https://issues.apache.org/jira/browse/NIFI-7870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy LoPresto updated NIFI-7870: Priority: Critical (was: Major) > X-Content-Type missing for advanced UI resources > > > Key: NIFI-7870 > URL: https://issues.apache.org/jira/browse/NIFI-7870 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.12.0 >Reporter: Nathan Gough >Assignee: Nathan Gough >Priority: Critical > Labels: UI, content-type, header, security > > The X-Content-Type header was added in NiFi 1.12.0, which blocks resources in > the browser if they do not have the content type added. It appears that some > 'advanced UI' resources do not have the content type applied to their > resources and are blocked from loading. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7870) X-Content-Type missing for advanced UI resources
[ https://issues.apache.org/jira/browse/NIFI-7870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy LoPresto updated NIFI-7870: Affects Version/s: 1.12.1 > X-Content-Type missing for advanced UI resources > > > Key: NIFI-7870 > URL: https://issues.apache.org/jira/browse/NIFI-7870 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.12.0, 1.12.1 >Reporter: Nathan Gough >Assignee: Nathan Gough >Priority: Critical > Labels: UI, content-type, header, security > > The X-Content-Type header was added in NiFi 1.12.0, which blocks resources in > the browser if they do not have the content type added. It appears that some > 'advanced UI' resources do not have the content type applied to their > resources and are blocked from loading. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-7870) X-Content-Type missing for advanced UI resources
Nathan Gough created NIFI-7870: -- Summary: X-Content-Type missing for advanced UI resources Key: NIFI-7870 URL: https://issues.apache.org/jira/browse/NIFI-7870 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.12.0 Reporter: Nathan Gough Assignee: Nathan Gough The X-Content-Type header was added in NiFi 1.12.0, which blocks resources in the browser if they do not have the content type added. It appears that some 'advanced UI' resources do not have the content type applied to their resources and are blocked from loading. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #914: MINIFICPP-1323 Encrypt sensitive properties using libsodium
fgerlits commented on a change in pull request #914:
URL: https://github.com/apache/nifi-minifi-cpp/pull/914#discussion_r498309792
##
File path: CMakeLists.txt
##
@@ -245,6 +245,12 @@ if (NOT OPENSSL_OFF)
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -DOPENSSL_SUPPORT")
endif()
+# libsodium
+include(BundledLibSodium)
+use_bundled_libsodium("${CMAKE_CURRENT_SOURCE_DIR}"
"${CMAKE_CURRENT_BINARY_DIR}")
+set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -DSODIUM_STATIC=1")
+set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -DSODIUM_STATIC=1")
Review comment:
requires CMake >= 3.11, see below
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Assigned] (NIFI-4890) OIDC Token Refresh is not done correctly
[ https://issues.apache.org/jira/browse/NIFI-4890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman reassigned NIFI-4890: - Assignee: Raz Dobkies > OIDC Token Refresh is not done correctly > > > Key: NIFI-4890 > URL: https://issues.apache.org/jira/browse/NIFI-4890 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 1.5.0 > Environment: Environment: > Browser: Chrome / Firefox > Configuration of NiFi: > - SSL certificate for the server (no client auth) > - OIDC configuration including end_session_endpoint (see the link > https://auth.s.orchestracities.com/auth/realms/default/.well-known/openid-configuration) > >Reporter: Federico Michele Facca >Assignee: Raz Dobkies >Priority: Major > > It looks like the NIFI UI is not refreshing the OIDC token in background, and > because of that, when the token expires, tells you that your session is > expired. and you need to refresh the page, to get a new token. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #914: MINIFICPP-1323 Encrypt sensitive properties using libsodium
fgerlits commented on a change in pull request #914:
URL: https://github.com/apache/nifi-minifi-cpp/pull/914#discussion_r498308693
##
File path: cmake/BundledLibSodium.cmake
##
@@ -0,0 +1,95 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+function(use_bundled_libsodium SOURCE_DIR BINARY_DIR)
+message("Using bundled libsodium")
+
+# Define patch step
+if (WIN32)
+set(PC "${Patch_EXECUTABLE}" -p1 -i
"${SOURCE_DIR}/thirdparty/libsodium/libsodium.patch")
+endif()
+
+# Define byproduct
+if (WIN32)
+set(BYPRODUCT "lib/sodium.lib")
+else()
+set(BYPRODUCT "lib/libsodium.a")
+endif()
+
+# Set build options
+set(LIBSODIUM_BIN_DIR "${BINARY_DIR}/thirdparty/libsodium-install" CACHE
STRING "" FORCE)
+
+if (WIN32)
+set(LIBSODIUM_CMAKE_ARGS ${PASSTHROUGH_CMAKE_ARGS}
+"-DCMAKE_INSTALL_PREFIX=${LIBSODIUM_BIN_DIR}"
+"-DSODIUM_LIBRARY_MINIMAL=1")
+endif()
+
+# Build project
+set(LIBSODIUM_URL
https://download.libsodium.org/libsodium/releases/libsodium-1.0.18.tar.gz)
+set(LIBSODIUM_URL_HASH
"SHA256=6f504490b342a4f8a4c4a02fc9b866cbef8622d5df4e5452b46be121e46636c1")
+
+if (WIN32)
+ExternalProject_Add(
+libsodium-external
+URL ${LIBSODIUM_URL}
+URL_HASH ${LIBSODIUM_URL_HASH}
+SOURCE_DIR "${BINARY_DIR}/thirdparty/libsodium-src"
+LIST_SEPARATOR % # This is needed for passing
semicolon-separated lists
+CMAKE_ARGS ${LIBSODIUM_CMAKE_ARGS}
+PATCH_COMMAND ${PC}
+BUILD_BYPRODUCTS "${LIBSODIUM_BIN_DIR}/${BYPRODUCT}"
+EXCLUDE_FROM_ALL TRUE
+)
+else()
+set(CONFIGURE_COMMAND ./configure --disable-pie --enable-minimal
"--prefix=${LIBSODIUM_BIN_DIR}")
+
+ExternalProject_Add(
+libsodium-external
+URL ${LIBSODIUM_URL}
+URL_HASH ${LIBSODIUM_URL_HASH}
+BUILD_IN_SOURCE true
+SOURCE_DIR "${BINARY_DIR}/thirdparty/libsodium-src"
+BUILD_COMMAND make
+CMAKE_COMMAND ""
+UPDATE_COMMAND ""
+INSTALL_COMMAND make install
+BUILD_BYPRODUCTS "${LIBSODIUM_BIN_DIR}/${BYPRODUCT}"
+CONFIGURE_COMMAND "${CONFIGURE_COMMAND}"
+PATCH_COMMAND ""
+STEP_TARGETS build
+EXCLUDE_FROM_ALL TRUE
+)
+endif()
+
+# Set variables
+set(LIBSODIUM_FOUND "YES" CACHE STRING "" FORCE)
+set(LIBSODIUM_INCLUDE_DIRS "${LIBSODIUM_BIN_DIR}/include" CACHE STRING ""
FORCE)
+set(LIBSODIUM_LIBRARIES "${LIBSODIUM_BIN_DIR}/${BYPRODUCT}" CACHE STRING
"" FORCE)
+
+# Set exported variables for FindPackage.cmake
+set(PASSTHROUGH_VARIABLES ${PASSTHROUGH_VARIABLES}
"-DEXPORTED_LIBSODIUM_INCLUDE_DIRS=${LIBSODIUM_INCLUDE_DIRS}" CACHE STRING ""
FORCE)
+set(PASSTHROUGH_VARIABLES ${PASSTHROUGH_VARIABLES}
"-DEXPORTED_LIBSODIUM_LIBRARIES=${LIBSODIUM_LIBRARIES}" CACHE STRING "" FORCE)
+
+# Create imported targets
+file(MAKE_DIRECTORY ${LIBSODIUM_INCLUDE_DIRS})
+
+add_library(libsodium STATIC IMPORTED)
+set_target_properties(libsodium PROPERTIES IMPORTED_LOCATION
"${LIBSODIUM_LIBRARIES}")
+add_dependencies(libsodium libsodium-external)
+set_property(TARGET libsodium APPEND PROPERTY
INTERFACE_INCLUDE_DIRECTORIES "${LIBSODIUM_INCLUDE_DIRS}")
Review comment:
Both of these require CMake >= 3.11 (ticket:
https://gitlab.kitware.com/cmake/cmake/-/issues/15689, PR:
https://gitlab.kitware.com/cmake/cmake/-/merge_requests/1264).
Do you think it's worth bumping the minimum CMake version for this? Ubuntu
18.04 comes with cmake 3.10 by default.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] turcsanyip commented on a change in pull request #4510: NIFI-7549 Adding Hazelcast based DistributedMapCacheClient support
turcsanyip commented on a change in pull request #4510: URL: https://github.com/apache/nifi/pull/4510#discussion_r498305509 ## File path: nifi-nar-bundles/nifi-hazelcast-bundle/nifi-hazelcast-services/pom.xml ## @@ -0,0 +1,74 @@ + + +http://maven.apache.org/POM/4.0.0; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd;> +4.0.0 + + +org.apache.nifi +nifi-hazelcast-bundle +1.13.0-SNAPSHOT + + +nifi-hazelcast-services +1.13.0-SNAPSHOT +jar + + + + + +org.apache.nifi +nifi-hazelcast-services-api +1.13.0-SNAPSHOT +provided + + + +org.apache.nifi +nifi-distributed-cache-client-service-api +1.13.0-SNAPSHOT Review comment: The version should come from the dependency management and it does not need to be specified here (the `provided` scope is coming from there). Also for `nifi-api` below and in `nifi-hazelcast-services-api`'s pom. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #914: MINIFICPP-1323 Encrypt sensitive properties using libsodium
lordgamez commented on a change in pull request #914:
URL: https://github.com/apache/nifi-minifi-cpp/pull/914#discussion_r498300651
##
File path: encrypt-config/ConfigFileEncryptor.cpp
##
@@ -0,0 +1,64 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ConfigFileEncryptor.h"
+
+#include
+#include
+
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace encrypt_config {
+
+int encryptSensitivePropertiesInFile(ConfigFile& config_file, const
utils::crypto::Bytes& encryption_key) {
Review comment:
Thanks! If a return value is an `int` it usually suggests to me that it
must be some kind of error code usually from legacy C code. An unsigned integer
type would suggest some count result, like the one we have here. Maybe this is
just my thinking.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #914: MINIFICPP-1323 Encrypt sensitive properties using libsodium
lordgamez commented on a change in pull request #914:
URL: https://github.com/apache/nifi-minifi-cpp/pull/914#discussion_r498302716
##
File path: encrypt-config/EncryptConfig.cpp
##
@@ -0,0 +1,150 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "EncryptConfig.h"
+
+#include
+
+#include
+
+#include "ConfigFile.h"
+#include "ConfigFileEncryptor.h"
+#include "cxxopts.hpp"
+#include "utils/file/FileUtils.h"
+#include "utils/OptionalUtils.h"
+
+namespace {
Review comment:
Thanks, I was referring to the last use case, where this could help with
the readability a bit.
`Blank lines immediately inside a declaration of a namespace or block of
namespaces may help readability by visually separating the load-bearing content
from the (largely non-semantic) organizational wrapper.`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #914: MINIFICPP-1323 Encrypt sensitive properties using libsodium
lordgamez commented on a change in pull request #914:
URL: https://github.com/apache/nifi-minifi-cpp/pull/914#discussion_r498300651
##
File path: encrypt-config/ConfigFileEncryptor.cpp
##
@@ -0,0 +1,64 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ConfigFileEncryptor.h"
+
+#include
+#include
+
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace encrypt_config {
+
+int encryptSensitivePropertiesInFile(ConfigFile& config_file, const
utils::crypto::Bytes& encryption_key) {
Review comment:
Thanks! If a return value is an `int` it usually suggests me that it
must be some kind of error code usually from legacy C code. An unsigned integer
type would suggest some count result, like the one we have here. Maybe this is
just my thinking.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #914: MINIFICPP-1323 Encrypt sensitive properties using libsodium
lordgamez commented on a change in pull request #914:
URL: https://github.com/apache/nifi-minifi-cpp/pull/914#discussion_r498296412
##
File path: encrypt-config/ConfigFile.cpp
##
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ConfigFile.h"
+
+#include
+#include
+#include
+
+#include "utils/StringUtils.h"
+
+namespace {
+constexpr std::array
DEFAULT_SENSITIVE_PROPERTIES{"nifi.security.client.pass.phrase",
+
"nifi.rest.api.password"};
+constexpr const char* ADDITIONAL_SENSITIVE_PROPS_PROPERTY_NAME =
"nifi.sensitive.props.additional.keys";
+} // namespace
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace encrypt_config {
+
+ConfigLine::ConfigLine(std::string line) : line_(line) {
+ line = utils::StringUtils::trim(line);
Review comment:
I understand, it definitely removes any clutter from the diff file.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-registry] pvillard31 opened a new pull request #305: NIFIREG-421 - adding GCS support to Ranger plugin
pvillard31 opened a new pull request #305:
URL: https://github.com/apache/nifi-registry/pull/305
Thank you for submitting a contribution to Apache NiFi Registry.
Please provide a short description of the PR here:
Description of PR
_Enables X functionality; fixes bug NIFIREG-._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with **NIFIREG-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [ ] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi-registry` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-registry-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-registry-assembly`?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NIFIREG-421) Add GCS support for Ranger plugin
Pierre Villard created NIFIREG-421: -- Summary: Add GCS support for Ranger plugin Key: NIFIREG-421 URL: https://issues.apache.org/jira/browse/NIFIREG-421 Project: NiFi Registry Issue Type: Improvement Reporter: Pierre Villard Assignee: Pierre Villard Adding a profile for the Ranger plugin to include the GCS connector for sending audit logs -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #914: MINIFICPP-1323 Encrypt sensitive properties using libsodium
fgerlits commented on a change in pull request #914:
URL: https://github.com/apache/nifi-minifi-cpp/pull/914#discussion_r498290578
##
File path: encrypt-config/ConfigFileEncryptor.cpp
##
@@ -0,0 +1,64 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ConfigFileEncryptor.h"
+
+#include
+#include
+
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace encrypt_config {
+
+int encryptSensitivePropertiesInFile(ConfigFile& config_file, const
utils::crypto::Bytes& encryption_key) {
Review comment:
I am not totally convinced: to me, `int` means "a smallish integer,
don't care too much which type", which is what I meant here; but I have changed
it to `uint32_t`.
##
File path: encrypt-config/EncryptConfig.cpp
##
@@ -0,0 +1,150 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "EncryptConfig.h"
+
+#include
+
+#include
+
+#include "ConfigFile.h"
+#include "ConfigFileEncryptor.h"
+#include "cxxopts.hpp"
+#include "utils/file/FileUtils.h"
+#include "utils/OptionalUtils.h"
+
+namespace {
Review comment:
That reference says "do _not_ use blank lines in general, but you can
use them in certain cases", and it's not clear this is one of those cases, but
OK.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] turcsanyip commented on pull request #4510: NIFI-7549 Adding Hazelcast based DistributedMapCacheClient support
turcsanyip commented on pull request #4510: URL: https://github.com/apache/nifi/pull/4510#issuecomment-702176664 Licensing info is missing from the nar modules. `LICENSE` / `NOTICE` files need to be added with entries to the referenced libraries (eg. `hazelcast`). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #914: MINIFICPP-1323 Encrypt sensitive properties using libsodium
fgerlits commented on a change in pull request #914:
URL: https://github.com/apache/nifi-minifi-cpp/pull/914#discussion_r498290976
##
File path: encrypt-config/EncryptConfig.cpp
##
@@ -0,0 +1,150 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "EncryptConfig.h"
+
+#include
+
+#include
+
+#include "ConfigFile.h"
+#include "ConfigFileEncryptor.h"
+#include "cxxopts.hpp"
+#include "utils/file/FileUtils.h"
+#include "utils/OptionalUtils.h"
+
+namespace {
+constexpr const char* CONF_DIRECTORY_NAME = "conf";
+constexpr const char* BOOTSTRAP_FILE_NAME = "bootstrap.conf";
+constexpr const char* MINIFI_PROPERTIES_FILE_NAME = "minifi.properties";
+constexpr const char* ENCRYPTION_KEY_PROPERTY_NAME =
"nifi.bootstrap.sensitive.key";
+} // namespace
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace encrypt_config {
+
+EncryptConfig::EncryptConfig(int argc, char* argv[]) :
minifi_home_(parseMinifiHomeFromTheOptions(argc, argv)) {
+ if (sodium_init() < 0) {
+throw std::runtime_error{"Could not initialize the libsodium library!"};
+ }
+}
+
+std::string EncryptConfig::parseMinifiHomeFromTheOptions(int argc, char*
argv[]) {
+ cxxopts::Options options("encrypt-config", "Encrypt sensitive minifi
properties");
+ options.add_options()
+ ("h,help", "Shows help")
+ ("m,minifi-home", "The MINIFI_HOME directory",
cxxopts::value());
+
+ auto parse_result = options.parse(argc, argv);
+
+ if (parse_result.count("help")) {
+std::cout << options.help() << '\n';
+std::exit(0);
+ }
+
+ if (parse_result.count("minifi-home")) {
+return parse_result["minifi-home"].as();
+ } else {
+throw std::runtime_error{"Required parameter missing: --minifi-home"};
+ }
+}
+
+void EncryptConfig::encryptSensitiveProperties() const {
+ utils::crypto::Bytes encryption_key = getEncryptionKey();
+ encryptSensitiveProperties(encryption_key);
Review comment:
I prefer one line to do one thing, and the compiler will optimize it
away in any case.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #914: MINIFICPP-1323 Encrypt sensitive properties using libsodium
fgerlits commented on a change in pull request #914:
URL: https://github.com/apache/nifi-minifi-cpp/pull/914#discussion_r498290405
##
File path: encrypt-config/ConfigFile.cpp
##
@@ -0,0 +1,166 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ConfigFile.h"
+
+#include
+#include
+#include
+
+#include "utils/StringUtils.h"
+
+namespace {
+constexpr std::array
DEFAULT_SENSITIVE_PROPERTIES{"nifi.security.client.pass.phrase",
+
"nifi.rest.api.password"};
+constexpr const char* ADDITIONAL_SENSITIVE_PROPS_PROPERTY_NAME =
"nifi.sensitive.props.additional.keys";
+} // namespace
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace encrypt_config {
+
+ConfigLine::ConfigLine(std::string line) : line_(line) {
+ line = utils::StringUtils::trim(line);
Review comment:
I feel fairly strongly that we should only update the properties we are
encrypting, and not touch the rest of the configuration file. If I run the
script and it says "encrypted 1 property", then the diff between the old and
new file should be something like
```
-nifi.rest.api.password=password123
+nifi.rest.api.password=iARpRfcyawNeEkEYLcLfatLUABuf9Nq3||OKHK6goZgGPskfKBow7CARSwdhyExPr1xEzE
+nifi.rest.api.password.protected=xsalsa20poly1305
```
and nothing else.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-registry] pvillard31 opened a new pull request #304: NIFIREG-420 - adding Ozone support to Ranger plugin
pvillard31 opened a new pull request #304:
URL: https://github.com/apache/nifi-registry/pull/304
Thank you for submitting a contribution to Apache NiFi Registry.
Please provide a short description of the PR here:
Description of PR
_Enables X functionality; fixes bug NIFIREG-._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with **NIFIREG-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [ ] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi-registry` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-registry-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-registry-assembly`?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm opened a new pull request #918: MINIFICPP-1383 fix indiv_ceil
szaszm opened a new pull request #918: URL: https://github.com/apache/nifi-minifi-cpp/pull/918 Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically main)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] If applicable, have you updated the LICENSE file? - [x] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFIREG-420) Add Ozone support for Ranger plugin
Pierre Villard created NIFIREG-420: -- Summary: Add Ozone support for Ranger plugin Key: NIFIREG-420 URL: https://issues.apache.org/jira/browse/NIFIREG-420 Project: NiFi Registry Issue Type: Improvement Reporter: Pierre Villard Assignee: Pierre Villard Adding a profile for Ozone support with the Ranger plugin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #914: MINIFICPP-1323 Encrypt sensitive properties using libsodium
szaszm commented on a change in pull request #914:
URL: https://github.com/apache/nifi-minifi-cpp/pull/914#discussion_r497392257
##
File path: CMakeLists.txt
##
@@ -245,6 +245,12 @@ if (NOT OPENSSL_OFF)
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -DOPENSSL_SUPPORT")
endif()
+# libsodium
+include(BundledLibSodium)
+use_bundled_libsodium("${CMAKE_CURRENT_SOURCE_DIR}"
"${CMAKE_CURRENT_BINARY_DIR}")
+set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -DSODIUM_STATIC=1")
+set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -DSODIUM_STATIC=1")
Review comment:
This should move to `use_bundled_libsodium` as
`target_compile_definitions`. If it's required on user code, then it should be
public or interface.
```suggestion
```
##
File path: cmake/BundledLibSodium.cmake
##
@@ -0,0 +1,95 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+function(use_bundled_libsodium SOURCE_DIR BINARY_DIR)
+message("Using bundled libsodium")
+
+# Define patch step
+if (WIN32)
+set(PC "${Patch_EXECUTABLE}" -p1 -i
"${SOURCE_DIR}/thirdparty/libsodium/libsodium.patch")
+endif()
+
+# Define byproduct
+if (WIN32)
+set(BYPRODUCT "lib/sodium.lib")
+else()
+set(BYPRODUCT "lib/libsodium.a")
+endif()
+
+# Set build options
+set(LIBSODIUM_BIN_DIR "${BINARY_DIR}/thirdparty/libsodium-install" CACHE
STRING "" FORCE)
+
+if (WIN32)
+set(LIBSODIUM_CMAKE_ARGS ${PASSTHROUGH_CMAKE_ARGS}
+"-DCMAKE_INSTALL_PREFIX=${LIBSODIUM_BIN_DIR}"
+"-DSODIUM_LIBRARY_MINIMAL=1")
+endif()
+
+# Build project
+set(LIBSODIUM_URL
https://download.libsodium.org/libsodium/releases/libsodium-1.0.18.tar.gz)
+set(LIBSODIUM_URL_HASH
"SHA256=6f504490b342a4f8a4c4a02fc9b866cbef8622d5df4e5452b46be121e46636c1")
+
+if (WIN32)
+ExternalProject_Add(
+libsodium-external
+URL ${LIBSODIUM_URL}
+URL_HASH ${LIBSODIUM_URL_HASH}
+SOURCE_DIR "${BINARY_DIR}/thirdparty/libsodium-src"
+LIST_SEPARATOR % # This is needed for passing
semicolon-separated lists
+CMAKE_ARGS ${LIBSODIUM_CMAKE_ARGS}
+PATCH_COMMAND ${PC}
+BUILD_BYPRODUCTS "${LIBSODIUM_BIN_DIR}/${BYPRODUCT}"
+EXCLUDE_FROM_ALL TRUE
+)
+else()
+set(CONFIGURE_COMMAND ./configure --disable-pie --enable-minimal
"--prefix=${LIBSODIUM_BIN_DIR}")
+
+ExternalProject_Add(
+libsodium-external
+URL ${LIBSODIUM_URL}
+URL_HASH ${LIBSODIUM_URL_HASH}
+BUILD_IN_SOURCE true
+SOURCE_DIR "${BINARY_DIR}/thirdparty/libsodium-src"
+BUILD_COMMAND make
+CMAKE_COMMAND ""
+UPDATE_COMMAND ""
+INSTALL_COMMAND make install
+BUILD_BYPRODUCTS "${LIBSODIUM_BIN_DIR}/${BYPRODUCT}"
+CONFIGURE_COMMAND "${CONFIGURE_COMMAND}"
+PATCH_COMMAND ""
+STEP_TARGETS build
+EXCLUDE_FROM_ALL TRUE
+)
+endif()
+
+# Set variables
+set(LIBSODIUM_FOUND "YES" CACHE STRING "" FORCE)
+set(LIBSODIUM_INCLUDE_DIRS "${LIBSODIUM_BIN_DIR}/include" CACHE STRING ""
FORCE)
+set(LIBSODIUM_LIBRARIES "${LIBSODIUM_BIN_DIR}/${BYPRODUCT}" CACHE STRING
"" FORCE)
+
+# Set exported variables for FindPackage.cmake
+set(PASSTHROUGH_VARIABLES ${PASSTHROUGH_VARIABLES}
"-DEXPORTED_LIBSODIUM_INCLUDE_DIRS=${LIBSODIUM_INCLUDE_DIRS}" CACHE STRING ""
FORCE)
+set(PASSTHROUGH_VARIABLES ${PASSTHROUGH_VARIABLES}
"-DEXPORTED_LIBSODIUM_LIBRARIES=${LIBSODIUM_LIBRARIES}" CACHE STRING "" FORCE)
+
+# Create imported targets
+file(MAKE_DIRECTORY ${LIBSODIUM_INCLUDE_DIRS})
+
+add_library(libsodium STATIC IMPORTED)
+set_target_properties(libsodium PROPERTIES IMPORTED_LOCATION
"${LIBSODIUM_LIBRARIES}")
+add_dependencies(libsodium libsodium-external)
+set_property(TARGET libsodium APPEND PROPERTY
INTERFACE_INCLUDE_DIRECTORIES "${LIBSODIUM_INCLUDE_DIRS}")
Review comment:
```suggestion
target_include_directories(libsodium INTERFACE
"${LIBSODIUM_INCLUDE_DIRS}")
[jira] [Created] (NIFI-7869) Add GCS for HDFS support in NiFi
Pierre Villard created NIFI-7869: Summary: Add GCS for HDFS support in NiFi Key: NIFI-7869 URL: https://issues.apache.org/jira/browse/NIFI-7869 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: Pierre Villard Assignee: Pierre Villard Add a specific profile to support the GCS connector for Hadoop File System on top of Google Cloud Storage -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7856) Provenance failed to be compressed after nifi upgrade to 1.12
[
https://issues.apache.org/jira/browse/NIFI-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205531#comment-17205531
]
Mengze Li commented on NIFI-7856:
-
Thanks for the reply.
Do you mind trying ReplaceText 1.12.0 which for us, doesn't show any data
provenance since the upgrade.
A few records in the 28th pop up randomly.
> Provenance failed to be compressed after nifi upgrade to 1.12
> -
>
> Key: NIFI-7856
> URL: https://issues.apache.org/jira/browse/NIFI-7856
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Mengze Li
>Priority: Major
> Attachments: 1683472.prov, ls.png, screenshot-1.png,
> screenshot-2.png, screenshot-3.png
>
>
> We upgraded our nifi cluster from 1.11.3 to 1.12.0.
> The nodes come up and everything looks to be functional. I can see 1.12.0 is
> running.
> Later on, we discovered that the data provenance is missing. From checking
> our logs, we see tons of errors compressing the logs.
> {code}
> 2020-09-28 03:38:35,205 ERROR [Compress Provenance Logs-1-thread-1]
> o.a.n.p.s.EventFileCompressor Failed to compress
> ./provenance_repository/2752821.prov on rollover
> {code}
> This didn't happen in 1.11.3.
> Is this a known issue? We are considering reverting back if there is no
> solution for this since we can't go prod with no/broken data provenance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-7856) Provenance failed to be compressed after nifi upgrade to 1.12
[
https://issues.apache.org/jira/browse/NIFI-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mengze Li updated NIFI-7856:
Attachment: screenshot-3.png
> Provenance failed to be compressed after nifi upgrade to 1.12
> -
>
> Key: NIFI-7856
> URL: https://issues.apache.org/jira/browse/NIFI-7856
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Mengze Li
>Priority: Major
> Attachments: 1683472.prov, ls.png, screenshot-1.png,
> screenshot-2.png, screenshot-3.png
>
>
> We upgraded our nifi cluster from 1.11.3 to 1.12.0.
> The nodes come up and everything looks to be functional. I can see 1.12.0 is
> running.
> Later on, we discovered that the data provenance is missing. From checking
> our logs, we see tons of errors compressing the logs.
> {code}
> 2020-09-28 03:38:35,205 ERROR [Compress Provenance Logs-1-thread-1]
> o.a.n.p.s.EventFileCompressor Failed to compress
> ./provenance_repository/2752821.prov on rollover
> {code}
> This didn't happen in 1.11.3.
> Is this a known issue? We are considering reverting back if there is no
> solution for this since we can't go prod with no/broken data provenance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] naddym commented on pull request #4560: NIFI-7859: Support for capturing execution duration of query run as a…
naddym commented on pull request #4560: URL: https://github.com/apache/nifi/pull/4560#issuecomment-702137249 Thanks @mattyb149 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] turcsanyip commented on a change in pull request #4540: NIFI-7825: Support native library loading via absolute path
turcsanyip commented on a change in pull request #4540:
URL: https://github.com/apache/nifi/pull/4540#discussion_r498244907
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/conf/bootstrap.conf
##
@@ -66,6 +66,15 @@ java.arg.16=-Djavax.security.auth.useSubjectCredsOnly=true
# Please see
https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_adminserver_config
for configuration options.
java.arg.17=-Dzookeeper.admin.enableServer=false
+# The following options configure a Java Agent to handle native library
loading.
+# It is needed when a custom jar (eg. JDBC driver) has been configured on a
component in the flow and this custom jar depends on a native library
+# and tries to load it by its absolute path (java.lang.System.load(String
filename) method call).
+# Use this Java Agent only if you get "Native Library ... already loaded in
another classloader" errors otherwise!
+#java.arg.18=-javaagent:./lib/aspectjweaver-${aspectj.version}.jar
Review comment:
It has been revealed that `aspectjrt` does not needed at runtime, only
`aspectjweaver` (which is loaded directly by the agent, not from classpath).
So I removed `aspectjrt.ajr` from `lib` and moved `aspectjweaver.jar` to
`lib/aspectj` subdirectory.
In this way, there are no AspectJ libraries on the system classpath by
default (when the agent is turned off).
When the agent is turned on, it is being loaded by the system classloader
(even if `aspectjweaver.jar` is not on the system classpath). So there might be
collision between different versions in this case. I believe it is acceptable
because only the "agent turned on" case affected.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (NIFI-7856) Provenance failed to be compressed after nifi upgrade to 1.12
[
https://issues.apache.org/jira/browse/NIFI-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205508#comment-17205508
]
Mark Payne commented on NIFI-7856:
--
I've tried replicating the issue but so far haven't been able to.
> Provenance failed to be compressed after nifi upgrade to 1.12
> -
>
> Key: NIFI-7856
> URL: https://issues.apache.org/jira/browse/NIFI-7856
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Mengze Li
>Priority: Major
> Attachments: 1683472.prov, ls.png, screenshot-1.png, screenshot-2.png
>
>
> We upgraded our nifi cluster from 1.11.3 to 1.12.0.
> The nodes come up and everything looks to be functional. I can see 1.12.0 is
> running.
> Later on, we discovered that the data provenance is missing. From checking
> our logs, we see tons of errors compressing the logs.
> {code}
> 2020-09-28 03:38:35,205 ERROR [Compress Provenance Logs-1-thread-1]
> o.a.n.p.s.EventFileCompressor Failed to compress
> ./provenance_repository/2752821.prov on rollover
> {code}
> This didn't happen in 1.11.3.
> Is this a known issue? We are considering reverting back if there is no
> solution for this since we can't go prod with no/broken data provenance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (MINIFICPP-1375) Ship Universal C Runtime DLLs
[ https://issues.apache.org/jira/browse/MINIFICPP-1375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz reopened MINIFICPP-1375: - > Ship Universal C Runtime DLLs > - > > Key: MINIFICPP-1375 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1375 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Marton Szasz >Assignee: Marton Szasz >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > MiNiFi C++ fails to start on older windows versions that don't have the > Universal C Runtime embedded or the respective update installed. Microsoft > allows for the redistribution of these DLLs and they only take up about 2 MB, > so we should ship them with the MSI for extra compatibility. > > [https://devblogs.microsoft.com/cppblog/introducing-the-universal-crt/] (see > point 6. near the bottom) > Redistributable files list: > [https://docs.microsoft.com/en-us/legal/windows-sdk/redist] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7856) Provenance failed to be compressed after nifi upgrade to 1.12
[
https://issues.apache.org/jira/browse/NIFI-7856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205501#comment-17205501
]
Mengze Li commented on NIFI-7856:
-
[~markap14] any chance that you have looked at this issue? Thanks
> Provenance failed to be compressed after nifi upgrade to 1.12
> -
>
> Key: NIFI-7856
> URL: https://issues.apache.org/jira/browse/NIFI-7856
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Mengze Li
>Priority: Major
> Attachments: 1683472.prov, ls.png, screenshot-1.png, screenshot-2.png
>
>
> We upgraded our nifi cluster from 1.11.3 to 1.12.0.
> The nodes come up and everything looks to be functional. I can see 1.12.0 is
> running.
> Later on, we discovered that the data provenance is missing. From checking
> our logs, we see tons of errors compressing the logs.
> {code}
> 2020-09-28 03:38:35,205 ERROR [Compress Provenance Logs-1-thread-1]
> o.a.n.p.s.EventFileCompressor Failed to compress
> ./provenance_repository/2752821.prov on rollover
> {code}
> This didn't happen in 1.11.3.
> Is this a known issue? We are considering reverting back if there is no
> solution for this since we can't go prod with no/broken data provenance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7859) Write execution duration attributes to SelectHive3QL output flow files
[ https://issues.apache.org/jira/browse/NIFI-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205496#comment-17205496 ] ASF subversion and git services commented on NIFI-7859: --- Commit e2ccfbbacfa7321e0bed2e7a23e4e26ca6e2b36c in nifi's branch refs/heads/main from Mohammed Nadeem [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=e2ccfbb ] NIFI-7859: Support for capturing execution duration of query run as attributes in SelectHiveQL processors Signed-off-by: Matthew Burgess This closes #4560 > Write execution duration attributes to SelectHive3QL output flow files > --- > > Key: NIFI-7859 > URL: https://issues.apache.org/jira/browse/NIFI-7859 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Rahul Soni >Assignee: Nadeem >Priority: Minor > Fix For: 1.13.0 > > Time Spent: 20m > Remaining Estimate: 0h > > SelectHive3QL & SelectHiveQL processors do not write attributes like > query.duration, query.executiontime, query.fetchtime etc. While the generic > ExecuteSQL processor does. These attributes can be useful in certain > scenarios where you want to measure the performance of the queries. > Can these attributes be added to the said processor? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] mattyb149 closed pull request #4560: NIFI-7859: Support for capturing execution duration of query run as a…
mattyb149 closed pull request #4560: URL: https://github.com/apache/nifi/pull/4560 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-7859) Write execution duration attributes to SelectHive3QL output flow files
[ https://issues.apache.org/jira/browse/NIFI-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-7859: --- Fix Version/s: 1.13.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Write execution duration attributes to SelectHive3QL output flow files > --- > > Key: NIFI-7859 > URL: https://issues.apache.org/jira/browse/NIFI-7859 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Rahul Soni >Assignee: Nadeem >Priority: Minor > Fix For: 1.13.0 > > Time Spent: 20m > Remaining Estimate: 0h > > SelectHive3QL & SelectHiveQL processors do not write attributes like > query.duration, query.executiontime, query.fetchtime etc. While the generic > ExecuteSQL processor does. These attributes can be useful in certain > scenarios where you want to measure the performance of the queries. > Can these attributes be added to the said processor? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] mattyb149 commented on pull request #4560: NIFI-7859: Support for capturing execution duration of query run as a…
mattyb149 commented on pull request #4560: URL: https://github.com/apache/nifi/pull/4560#issuecomment-702121443 +1 LGTM, ran tests and verified results were as expected. Thanks for the improvement! Merging to main This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-7863) PutSmbFile does not create missing folder parents folder
[
https://issues.apache.org/jira/browse/NIFI-7863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205463#comment-17205463
]
Si Sun commented on NIFI-7863:
--
hmm, yes this case wasn't considered ...
according to [smbj
docs|https://javadoc.io/static/com.hierynomus/smbj/0.10.0/com/hierynomus/smbj/share/DiskShare.html]
and [this response|https://github.com/hierynomus/smbj/issues/296] there is no
direct method to create the folders in a recursive manner. So I suppose we need
to split the folder path and recursively check with folderExists and then call
mkdir.
> PutSmbFile does not create missing folder parents folder
>
>
> Key: NIFI-7863
> URL: https://issues.apache.org/jira/browse/NIFI-7863
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.12.1
>Reporter: Jens M Kofoed
>Priority: Major
>
> I'm trying the new PutSmbFile, which I have been waiting for for years. So
> many many thanks to the code writer.
> In the properties it is possible to set "Create Missing Directories" to true.
> But it will only create the last subdir and not multiple dirs.
> So for instance I would like to save files in: \\server\share\dir1\dir2\dir3.
> Both Dir1 and Dir2 have to exists.
> It is not possible to save files in dynamically subpaths like:
> ${year}\${month}\${day}
> This i possible with PutFTP.
> Looking through the code, it use share.mkdir(directory) instead of .mkdir*s*
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #914: MINIFICPP-1323 Encrypt sensitive properties using libsodium
lordgamez commented on a change in pull request #914:
URL: https://github.com/apache/nifi-minifi-cpp/pull/914#discussion_r498117414
##
File path: encrypt-config/ConfigFileEncryptor.cpp
##
@@ -0,0 +1,64 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ConfigFileEncryptor.h"
+
+#include
+#include
+
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace encrypt_config {
+
+int encryptSensitivePropertiesInFile(ConfigFile& config_file, const
utils::crypto::Bytes& encryption_key) {
Review comment:
An unsigned type like std::size_t or uint32_t may be more appropriate in
this case.
##
File path: encrypt-config/EncryptConfig.cpp
##
@@ -0,0 +1,150 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "EncryptConfig.h"
+
+#include
+
+#include
+
+#include "ConfigFile.h"
+#include "ConfigFileEncryptor.h"
+#include "cxxopts.hpp"
+#include "utils/file/FileUtils.h"
+#include "utils/OptionalUtils.h"
+
+namespace {
Review comment:
Blank lines could be added after the start and before the end of
namespace https://google.github.io/styleguide/cppguide.html#Vertical_Whitespace
##
File path: encrypt-config/EncryptConfig.cpp
##
@@ -0,0 +1,150 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "EncryptConfig.h"
+
+#include
+
+#include
+
+#include "ConfigFile.h"
+#include "ConfigFileEncryptor.h"
+#include "cxxopts.hpp"
+#include "utils/file/FileUtils.h"
+#include "utils/OptionalUtils.h"
+
+namespace {
+constexpr const char* CONF_DIRECTORY_NAME = "conf";
+constexpr const char* BOOTSTRAP_FILE_NAME = "bootstrap.conf";
+constexpr const char* MINIFI_PROPERTIES_FILE_NAME = "minifi.properties";
+constexpr const char* ENCRYPTION_KEY_PROPERTY_NAME =
"nifi.bootstrap.sensitive.key";
+} // namespace
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace encrypt_config {
+
+EncryptConfig::EncryptConfig(int argc, char* argv[]) :
minifi_home_(parseMinifiHomeFromTheOptions(argc, argv)) {
+ if (sodium_init() < 0) {
+throw std::runtime_error{"Could not initialize the libsodium library!"};
+ }
+}
+
+std::string EncryptConfig::parseMinifiHomeFromTheOptions(int argc, char*
argv[]) {
+ cxxopts::Options options("encrypt-config", "Encrypt sensitive minifi
properties");
+ options.add_options()
+ ("h,help", "Shows help")
+ ("m,minifi-home", "The MINIFI_HOME directory",
cxxopts::value());
+
+ auto parse_result = options.parse(argc, argv);
+
+ if (parse_result.count("help")) {
+std::cout << options.help() << '\n';
+std::exit(0);
+ }
+
+ if
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #917: MINIFICPP-1380 - Batch behavior for CompressContent and MergeContent processors
adamdebreceni commented on a change in pull request #917:
URL: https://github.com/apache/nifi-minifi-cpp/pull/917#discussion_r498163298
##
File path: extensions/libarchive/CompressContent.cpp
##
@@ -171,7 +183,7 @@ void CompressContent::onTrigger(const
std::shared_ptr
std::shared_ptr processFlowFile = session->create(flowFile);
bool success = false;
if (encapsulateInTar_) {
-CompressContent::WriteCallback callback(compressMode_, compressLevel_,
compressFormat, flowFile, session);
+CompressContent::WriteCallback callback(compressMode_, compressLevel_,
toString(compressFormat), flowFile, session);
Review comment:
we should forward the enum itself instead of its string representation
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #917: MINIFICPP-1380 - Batch behavior for CompressContent and MergeContent processors
adamdebreceni commented on a change in pull request #917:
URL: https://github.com/apache/nifi-minifi-cpp/pull/917#discussion_r498162379
##
File path: extensions/libarchive/CompressContent.cpp
##
@@ -81,86 +101,78 @@ void CompressContent::initialize() {
}
void CompressContent::onSchedule(core::ProcessContext *context,
core::ProcessSessionFactory *sessionFactory) {
- std::string value;
context->getProperty(CompressLevel.getName(), compressLevel_);
context->getProperty(CompressMode.getName(), compressMode_);
- context->getProperty(CompressFormat.getName(), compressFormat_);
+
+ {
+std::string compressFormatStr;
+context->getProperty(CompressFormat.getName(), compressFormatStr);
+std::transform(compressFormatStr.begin(), compressFormatStr.end(),
compressFormatStr.begin(), ::tolower);
+compressFormat_ =
ExtendedCompressionFormat::parse(compressFormatStr.c_str());
+ }
+
context->getProperty(UpdateFileName.getName(), updateFileName_);
context->getProperty(EncapsulateInTar.getName(), encapsulateInTar_);
+ context->getProperty(BatchSize.getName(), batchSize_);
- logger_->log_info("Compress Content: Mode [%s] Format [%s] Level [%d]
UpdateFileName [%d] EncapsulateInTar [%d]",
- compressMode_, compressFormat_, compressLevel_, updateFileName_,
encapsulateInTar_);
-
- // update the mimeTypeMap
- compressionFormatMimeTypeMap_["application/gzip"] = COMPRESSION_FORMAT_GZIP;
- compressionFormatMimeTypeMap_["application/bzip2"] =
COMPRESSION_FORMAT_BZIP2;
- compressionFormatMimeTypeMap_["application/x-bzip2"] =
COMPRESSION_FORMAT_BZIP2;
- compressionFormatMimeTypeMap_["application/x-lzma"] =
COMPRESSION_FORMAT_LZMA;
- compressionFormatMimeTypeMap_["application/x-xz"] =
COMPRESSION_FORMAT_XZ_LZMA2;
- fileExtension_[COMPRESSION_FORMAT_GZIP] = ".gz";
- fileExtension_[COMPRESSION_FORMAT_LZMA] = ".lzma";
- fileExtension_[COMPRESSION_FORMAT_BZIP2] = ".bz2";
- fileExtension_[COMPRESSION_FORMAT_XZ_LZMA2] = ".xz";
+ logger_->log_info("Compress Content: Mode [%s] CompressionFormat [%s] Level
[%d] UpdateFileName [%d] EncapsulateInTar [%d]",
+ compressMode_, toString(compressFormat_), compressLevel_,
updateFileName_, encapsulateInTar_);
}
void CompressContent::onTrigger(const std::shared_ptr
, const std::shared_ptr ) {
+ for (size_t i = 0; i < batchSize_; ++i) {
+if (onTriggerImpl(context, session) != TriggerResult::CONTINUE) {
+ break;
+}
+ }
+}
+
+CompressContent::TriggerResult CompressContent::onTriggerImpl(const
std::shared_ptr , const
std::shared_ptr ) {
std::shared_ptr flowFile = session->get();
if (!flowFile) {
-return;
+return TriggerResult::BREAK;
Review comment:
since this is the only place we return a `BREAK` we could extract the
flowFile in the `onTrigger` and call this something like `processFlowFile` and
eliminate the `TriggerResult`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #917: MINIFICPP-1380 - Batch behavior for CompressContent and MergeContent processors
adamdebreceni commented on a change in pull request #917:
URL: https://github.com/apache/nifi-minifi-cpp/pull/917#discussion_r498162629
##
File path: extensions/libarchive/CompressContent.cpp
##
@@ -81,86 +101,78 @@ void CompressContent::initialize() {
}
void CompressContent::onSchedule(core::ProcessContext *context,
core::ProcessSessionFactory *sessionFactory) {
- std::string value;
context->getProperty(CompressLevel.getName(), compressLevel_);
context->getProperty(CompressMode.getName(), compressMode_);
- context->getProperty(CompressFormat.getName(), compressFormat_);
+
+ {
+std::string compressFormatStr;
+context->getProperty(CompressFormat.getName(), compressFormatStr);
+std::transform(compressFormatStr.begin(), compressFormatStr.end(),
compressFormatStr.begin(), ::tolower);
+compressFormat_ =
ExtendedCompressionFormat::parse(compressFormatStr.c_str());
+ }
+
context->getProperty(UpdateFileName.getName(), updateFileName_);
context->getProperty(EncapsulateInTar.getName(), encapsulateInTar_);
+ context->getProperty(BatchSize.getName(), batchSize_);
- logger_->log_info("Compress Content: Mode [%s] Format [%s] Level [%d]
UpdateFileName [%d] EncapsulateInTar [%d]",
- compressMode_, compressFormat_, compressLevel_, updateFileName_,
encapsulateInTar_);
-
- // update the mimeTypeMap
- compressionFormatMimeTypeMap_["application/gzip"] = COMPRESSION_FORMAT_GZIP;
- compressionFormatMimeTypeMap_["application/bzip2"] =
COMPRESSION_FORMAT_BZIP2;
- compressionFormatMimeTypeMap_["application/x-bzip2"] =
COMPRESSION_FORMAT_BZIP2;
- compressionFormatMimeTypeMap_["application/x-lzma"] =
COMPRESSION_FORMAT_LZMA;
- compressionFormatMimeTypeMap_["application/x-xz"] =
COMPRESSION_FORMAT_XZ_LZMA2;
- fileExtension_[COMPRESSION_FORMAT_GZIP] = ".gz";
- fileExtension_[COMPRESSION_FORMAT_LZMA] = ".lzma";
- fileExtension_[COMPRESSION_FORMAT_BZIP2] = ".bz2";
- fileExtension_[COMPRESSION_FORMAT_XZ_LZMA2] = ".xz";
+ logger_->log_info("Compress Content: Mode [%s] CompressionFormat [%s] Level
[%d] UpdateFileName [%d] EncapsulateInTar [%d]",
+ compressMode_, toString(compressFormat_), compressLevel_,
updateFileName_, encapsulateInTar_);
}
void CompressContent::onTrigger(const std::shared_ptr
, const std::shared_ptr ) {
+ for (size_t i = 0; i < batchSize_; ++i) {
+if (onTriggerImpl(context, session) != TriggerResult::CONTINUE) {
+ break;
+}
+ }
+}
+
+CompressContent::TriggerResult CompressContent::onTriggerImpl(const
std::shared_ptr , const
std::shared_ptr ) {
std::shared_ptr flowFile = session->get();
if (!flowFile) {
-return;
+return TriggerResult::BREAK;
}
session->remove(flowFile);
- std::string compressFormat = compressFormat_;
- if (compressFormat_ == COMPRESSION_FORMAT_ATTRIBUTE) {
+ CompressionFormat::Type compressFormat;
+ if (compressFormat_ == ExtendedCompressionFormat::USE_MIME_TYPE) {
std::string attr;
flowFile->getAttribute(core::SpecialFlowAttribute::MIME_TYPE, attr);
if (attr.empty()) {
logger_->log_error("No %s attribute existed for the flow, route to
failure", core::SpecialFlowAttribute::MIME_TYPE);
session->transfer(flowFile, Failure);
- return;
+ return TriggerResult::CONTINUE;
}
auto search = compressionFormatMimeTypeMap_.find(attr);
if (search != compressionFormatMimeTypeMap_.end()) {
compressFormat = search->second;
} else {
- logger_->log_info("Mime type of %s is not indicated a support format,
route to success", attr);
+ logger_->log_info("Mime type of %s is not indicated a support
CompressionFormat, route to success", attr);
Review comment:
Ctrl+R error
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #917: MINIFICPP-1380 - Batch behavior for CompressContent and MergeContent processors
adamdebreceni commented on a change in pull request #917:
URL: https://github.com/apache/nifi-minifi-cpp/pull/917#discussion_r498160741
##
File path: extensions/libarchive/CompressContent.cpp
##
@@ -42,14 +42,14 @@ core::Property CompressContent::CompressMode(
core::PropertyBuilder::createProperty("Mode")->withDescription("Indicates
whether the processor should compress content or decompress content.")
->isRequired(false)->withAllowableValues({MODE_COMPRESS,
MODE_DECOMPRESS})->withDefaultValue(MODE_COMPRESS)->build());
core::Property CompressContent::CompressFormat(
-core::PropertyBuilder::createProperty("Compression
Format")->withDescription("The compression format to use.")
+core::PropertyBuilder::createProperty("Compression
CompressionFormat")->withDescription("The compression CompressionFormat to
use.")
Review comment:
restore property name
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni opened a new pull request #917: MINIFICPP-1380 - Batch behavior for CompressContent and MergeContent processors
adamdebreceni opened a new pull request #917: URL: https://github.com/apache/nifi-minifi-cpp/pull/917 Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically main)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (MINIFICPP-1384) Fix intdiv_ceil
Adam Debreceni created MINIFICPP-1384:
-
Summary: Fix intdiv_ceil
Key: MINIFICPP-1384
URL: https://issues.apache.org/jira/browse/MINIFICPP-1384
Project: Apache NiFi MiNiFi C++
Issue Type: Bug
Reporter: Adam Debreceni
The current implementation goes like
{code:java}
return a / b + (a % b > 0){code}
this might give incorrect results on negative numbers:
{code:java}
intdiv_ceil(-5, -3) == 1
{code}
Either provide an implementation that behaves correctly for negative numbers as
well, or restrict the domain of the function to unsigned integral types.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (MINIFICPP-1383) intdiv_ceil gives incorrect results on negative input
Marton Szasz created MINIFICPP-1383: --- Summary: intdiv_ceil gives incorrect results on negative input Key: MINIFICPP-1383 URL: https://issues.apache.org/jira/browse/MINIFICPP-1383 Project: Apache NiFi MiNiFi C++ Issue Type: Bug Reporter: Marton Szasz Assignee: Marton Szasz -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi-minifi-cpp] arpadboda closed pull request #875: MINIFICPP-1332 Prevent errneous behavior by stopping FlowController in low disk space conditions
arpadboda closed pull request #875: URL: https://github.com/apache/nifi-minifi-cpp/pull/875 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #875: MINIFICPP-1332 Prevent errneous behavior by stopping FlowController in low disk space conditions
adamdebreceni commented on a change in pull request #875:
URL: https://github.com/apache/nifi-minifi-cpp/pull/875#discussion_r498148335
##
File path: libminifi/include/utils/GeneralUtils.h
##
@@ -43,6 +43,7 @@ using std::make_unique;
template::value>::type>
T intdiv_ceil(T numerator, T denominator) {
+ gsl_Expects(denominator != 0);
// note: division and remainder is 1 instruction on x86
return numerator / denominator + (numerator % denominator > 0);
Review comment:
I know this is old code, but unless we restrict the domain of this
function to non-negative arguments, it might give incorrect results, e.g.
`intdiv_ceil(-5, -3) == 1`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #914: MINIFICPP-1323 Encrypt sensitive properties using libsodium
fgerlits commented on a change in pull request #914:
URL: https://github.com/apache/nifi-minifi-cpp/pull/914#discussion_r498101379
##
File path: main/MiNiFiMain.cpp
##
@@ -208,6 +238,10 @@ int main(int argc, char **argv) {
configure->setHome(minifiHome);
configure->loadConfigureFile(DEFAULT_NIFI_PROPERTIES_FILE);
+ if (containsEncryptedProperties(*configure)) {
+decryptSensitiveProperties(*configure, minifiHome, *logger);
Review comment:
Ouch. Yes, that is a serious problem.
EDIT: as discussed, persisting the `Configure` object doesn't work at the
moment, due to a bug: new properties are added to the `minifi.properties` file,
but existing and modified properties are not updated. So the decrypted
sensitive properties cannot be leaked right now.
I think the best long-term solution would be not to update the sensitive
values in the `Configure` object, but store the key instead, and decrypt the
sensitive values on the fly in the getter function.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #875: MINIFICPP-1332 Prevent errneous behavior by stopping FlowController in low disk space conditions
adamdebreceni commented on a change in pull request #875:
URL: https://github.com/apache/nifi-minifi-cpp/pull/875#discussion_r498139911
##
File path: libminifi/include/DiskSpaceWatchdog.h
##
@@ -0,0 +1,60 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include
+#include
+#include
+#include
+
+#include "utils/IntervalSwitch.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+
+class Configure;
+namespace core {
+namespace logging {
+class Logger;
+} // namespace logging
+} // namespace core
+
+namespace disk_space_watchdog {
+struct Config {
+ std::chrono::milliseconds interval;
+ std::uintmax_t stop_threshold_bytes;
+ std::uintmax_t restart_threshold_bytes;
+};
+
+Config read_config(const Configure&);
+
+inline utils::IntervalSwitch disk_space_interval_switch(Config
config) {
Review comment:
I would be really surprised if it made any difference performance-wise,
this function is most probably going to get inlined either way with the
optimization level we are using
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #875: MINIFICPP-1332 Prevent errneous behavior by stopping FlowController in low disk space conditions
lordgamez commented on a change in pull request #875:
URL: https://github.com/apache/nifi-minifi-cpp/pull/875#discussion_r498129867
##
File path: libminifi/src/utils/file/PathUtils.cpp
##
@@ -84,13 +86,58 @@ std::string PathUtils::getFullPath(const std::string& path)
{
#endif
}
-std::string PathUtils::globToRegex(std::string glob) {
+std::string globToRegex(std::string glob) {
utils::StringUtils::replaceAll(glob, ".", "\\.");
utils::StringUtils::replaceAll(glob, "*", ".*");
utils::StringUtils::replaceAll(glob, "?", ".");
return glob;
}
+space_info space(const path p, std::error_code& ec) noexcept {
Review comment:
I'm fine with that as well.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #875: MINIFICPP-1332 Prevent errneous behavior by stopping FlowController in low disk space conditions
lordgamez commented on a change in pull request #875:
URL: https://github.com/apache/nifi-minifi-cpp/pull/875#discussion_r498129745
##
File path: libminifi/include/DiskSpaceWatchdog.h
##
@@ -0,0 +1,60 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include
+#include
+#include
+#include
+
+#include "utils/IntervalSwitch.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+
+class Configure;
+namespace core {
+namespace logging {
+class Logger;
+} // namespace logging
+} // namespace core
+
+namespace disk_space_watchdog {
+struct Config {
+ std::chrono::milliseconds interval;
+ std::uintmax_t stop_threshold_bytes;
+ std::uintmax_t restart_threshold_bytes;
+};
+
+Config read_config(const Configure&);
+
+inline utils::IntervalSwitch disk_space_interval_switch(Config
config) {
Review comment:
You can leave it as it is, I do not insist on it if it's still cheap to
copy. My only concern was if this config may be expanded in the future, but I
suppose the watchdog configuration will not really change.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #875: MINIFICPP-1332 Prevent errneous behavior by stopping FlowController in low disk space conditions
szaszm commented on a change in pull request #875:
URL: https://github.com/apache/nifi-minifi-cpp/pull/875#discussion_r498122682
##
File path: libminifi/src/utils/file/PathUtils.cpp
##
@@ -84,13 +86,58 @@ std::string PathUtils::getFullPath(const std::string& path)
{
#endif
}
-std::string PathUtils::globToRegex(std::string glob) {
+std::string globToRegex(std::string glob) {
utils::StringUtils::replaceAll(glob, ".", "\\.");
utils::StringUtils::replaceAll(glob, "*", ".*");
utils::StringUtils::replaceAll(glob, "?", ".");
return glob;
}
+space_info space(const path p, std::error_code& ec) noexcept {
Review comment:
I'm not a fan of trailing underscores unless really necessary, so I went
with `path` and let them clash. Let me know if you're fine with this approach.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #875: MINIFICPP-1332 Prevent errneous behavior by stopping FlowController in low disk space conditions
szaszm commented on a change in pull request #875:
URL: https://github.com/apache/nifi-minifi-cpp/pull/875#discussion_r498119240
##
File path: libminifi/include/DiskSpaceWatchdog.h
##
@@ -0,0 +1,60 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include
+#include
+#include
+#include
+
+#include "utils/IntervalSwitch.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+
+class Configure;
+namespace core {
+namespace logging {
+class Logger;
+} // namespace logging
+} // namespace core
+
+namespace disk_space_watchdog {
+struct Config {
+ std::chrono::milliseconds interval;
+ std::uintmax_t stop_threshold_bytes;
+ std::uintmax_t restart_threshold_bytes;
+};
+
+Config read_config(const Configure&);
+
+inline utils::IntervalSwitch disk_space_interval_switch(Config
config) {
Review comment:
It has the size of 3 doubles/pointers on my machine and is trivially
copiable, so I thought it's better to copy it than pass by const reference.
https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#Rf-in
The text says that 2-3 doubles/pointers is cheap to copy, yet the
enforcement section suggests flagging anything over `2*sizeof(void*)`. I'm
still leaning towards copy, so let me know if you insist on the change given
the above.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #875: MINIFICPP-1332 Prevent errneous behavior by stopping FlowController in low disk space conditions
arpadboda commented on a change in pull request #875:
URL: https://github.com/apache/nifi-minifi-cpp/pull/875#discussion_r498111891
##
File path: libminifi/include/properties/Properties.h
##
@@ -82,6 +84,16 @@ class Properties {
*/
int getInt(const std::string , int default_value) const;
+ utils::optional get(const std::string& key) const {
+std::string result;
+const bool found = get(key, result);
Review comment:
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #914: MINIFICPP-1323 Encrypt sensitive properties using libsodium
fgerlits commented on a change in pull request #914:
URL: https://github.com/apache/nifi-minifi-cpp/pull/914#discussion_r498101379
##
File path: main/MiNiFiMain.cpp
##
@@ -208,6 +238,10 @@ int main(int argc, char **argv) {
configure->setHome(minifiHome);
configure->loadConfigureFile(DEFAULT_NIFI_PROPERTIES_FILE);
+ if (containsEncryptedProperties(*configure)) {
+decryptSensitiveProperties(*configure, minifiHome, *logger);
Review comment:
Ouch. Yes, that is a serious problem.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #914: MINIFICPP-1323 Encrypt sensitive properties using libsodium
adamdebreceni commented on a change in pull request #914:
URL: https://github.com/apache/nifi-minifi-cpp/pull/914#discussion_r498090718
##
File path: main/MiNiFiMain.cpp
##
@@ -208,6 +238,10 @@ int main(int argc, char **argv) {
configure->setHome(minifiHome);
configure->loadConfigureFile(DEFAULT_NIFI_PROPERTIES_FILE);
+ if (containsEncryptedProperties(*configure)) {
+decryptSensitiveProperties(*configure, minifiHome, *logger);
Review comment:
as I see, we update the `configure` object in
`decryptSensitiveProperties `, one possible problem with this, is that on a
flow update (from a C2 agent) the configurations file is persisted by default
(to store the new flow url) this could inadvertently leak the decrypted values
see
[here](https://github.com/apache/nifi-minifi-cpp/blob/da90d98b79c5590844b28ea81ce80f08765f48c9/libminifi/src/c2/C2Agent.cpp#L720)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #914: MINIFICPP-1323 Encrypt sensitive properties using libsodium
adamdebreceni commented on a change in pull request #914:
URL: https://github.com/apache/nifi-minifi-cpp/pull/914#discussion_r498090718
##
File path: main/MiNiFiMain.cpp
##
@@ -208,6 +238,10 @@ int main(int argc, char **argv) {
configure->setHome(minifiHome);
configure->loadConfigureFile(DEFAULT_NIFI_PROPERTIES_FILE);
+ if (containsEncryptedProperties(*configure)) {
+decryptSensitiveProperties(*configure, minifiHome, *logger);
Review comment:
as I see, we update the `configure` object in
`decryptSensitiveProperties `, one possible problem with this, is that on a
flow update (from a C2 agent) the configurations file is persisted by default
(to store the new flow url) this could inadvertently leak the decrypted values
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #875: MINIFICPP-1332 Prevent errneous behavior by stopping FlowController in low disk space conditions
lordgamez commented on a change in pull request #875:
URL: https://github.com/apache/nifi-minifi-cpp/pull/875#discussion_r498035883
##
File path: libminifi/include/DiskSpaceWatchdog.h
##
@@ -0,0 +1,60 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include
+#include
+#include
+#include
+
+#include "utils/IntervalSwitch.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+
+class Configure;
+namespace core {
+namespace logging {
+class Logger;
+} // namespace logging
+} // namespace core
+
+namespace disk_space_watchdog {
+struct Config {
+ std::chrono::milliseconds interval;
+ std::uintmax_t stop_threshold_bytes;
+ std::uintmax_t restart_threshold_bytes;
+};
+
+Config read_config(const Configure&);
+
+inline utils::IntervalSwitch disk_space_interval_switch(Config
config) {
Review comment:
It would be better to pass config by const ref.
##
File path: libminifi/src/utils/file/PathUtils.cpp
##
@@ -84,13 +86,58 @@ std::string PathUtils::getFullPath(const std::string& path)
{
#endif
}
-std::string PathUtils::globToRegex(std::string glob) {
+std::string globToRegex(std::string glob) {
utils::StringUtils::replaceAll(glob, ".", "\\.");
utils::StringUtils::replaceAll(glob, "*", ".*");
utils::StringUtils::replaceAll(glob, "?", ".");
return glob;
}
+space_info space(const path p, std::error_code& ec) noexcept {
Review comment:
Please rename 'p' to something more searchable as it has a bit larger
scope. If it clashes with the type name it is usually recommended to use the
'_' suffix in these cases.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #914: MINIFICPP-1323 Encrypt sensitive properties using libsodium
adamdebreceni commented on a change in pull request #914:
URL: https://github.com/apache/nifi-minifi-cpp/pull/914#discussion_r498055943
##
File path: libminifi/include/properties/Properties.h
##
@@ -65,6 +67,13 @@ class Properties {
*/
bool get(const std::string , std::string ) const;
+ /**
+ * Returns the config value.
+ * @param key key to look up
+ * @returns the value if found, otherwise nullopt
+ */
+ utils::optional get(const std::string ) const;
Review comment:
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
