[GitHub] [nifi] timeabarna opened a new pull request #4824: NIFI-8176 adding ackChecked flag to QuerySplunkIndexingStatus.onTrigg…

[GitBox]

timeabarna opened a new pull request #4824:
URL: https://github.com/apache/nifi/pull/4824


   …er() method ensuring at least one acknoledgement check is happening with 
Splunk
   
   https://issues.apache.org/jira/browse/NIFI-8176#
   
    Description of PR
   
   adding flag to ensure at least one acknowledgment check with Splunk
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
in the commit message?
   
   - [ ] Does your PR title start with **NIFI-** where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   
   - [ ] Has your PR been rebased against the latest commit within the target 
branch (typically `main`)?
   
   - [ ] Is your initial contribution a single, squashed commit? _Additional 
commits in response to PR reviewer feedback should be made on this branch and 
pushed to allow change tracking. Do not `squash` or use `--force` when pushing 
to allow for clean monitoring of changes._
   
   ### For code changes:
   - [ ] Have you ensured that the full suite of tests is executed via `mvn 
-Pcontrib-check clean install` at the root `nifi` folder?
   - [ ] Have you written or updated unit tests to verify your changes?
   - [ ] Have you verified that the full build is successful on JDK 8?
   - [ ] Have you verified that the full build is successful on JDK 11?
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
   - [ ] If applicable, have you updated the `LICENSE` file, including the main 
`LICENSE` file under `nifi-assembly`?
   - [ ] If applicable, have you updated the `NOTICE` file, including the main 
`NOTICE` file found under `nifi-assembly`?
   - [ ] If adding new Properties, have you added `.displayName` in addition to 
.name (programmatic access) for each of the new properties?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check GitHub Actions CI for 
build issues and submit an update to your PR as soon as possible.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] bhaveshpatelh commented on pull request #4822: NIFI-2892 Implement AWS Kinesis Stream Get Processor

[GitBox]

bhaveshpatelh commented on pull request #4822:
URL: https://github.com/apache/nifi/pull/4822#issuecomment-779624802


   Thanks! Looking forward to this piece going out 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Resolved] (NIFI-4369) ConfigEncryptionTool should stream flow contents

[David Handermann (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Handermann resolved NIFI-4369.

Fix Version/s: 1.13.0
 Assignee: Nathan Gough
   Resolution: Fixed

This particular issue appears to be resolved by GitHub PR 4715 for NIFI-6999.

> ConfigEncryptionTool should stream flow contents
> 
>
> Key: NIFI-4369
> URL: https://issues.apache.org/jira/browse/NIFI-4369
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Tools and Build
>Affects Versions: 1.3.0
>Reporter: Jeff Storck
>Assignee: Nathan Gough
>Priority: Major
>  Labels: memory, performance, security, toolkit
> Fix For: 1.13.0
>
>
> ConfigEncryptionTool loads the entire flow xml into memory before operating 
> on it.  Rather, it should stream the flow contents to avoid requiring large 
> amounts of memory to process.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi] exceptionfactory commented on pull request #4788: NIFI-8132 Replaced framework uses of MD5 with SHA-256

[GitBox]

exceptionfactory commented on pull request #4788:
URL: https://github.com/apache/nifi/pull/4788#issuecomment-779475885


   > Per 
https://cwiki.apache.org/confluence/display/NIFI/Version+Scheme+and+API+Compatibility,
 there isn't a commitment for commons and utils. I personally tend toward being 
conservative, though.
   > 
   > I think `FileUtils` gets used by third parties, BUT, searching GitHub I 
couldn't find any users of `computeMd5Digest()` itself.
   > 
   > Thanks for the explanation.
   
   Thanks @jfrazee looks like we are on the same page!



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] jfrazee commented on pull request #4788: NIFI-8132 Replaced framework uses of MD5 with SHA-256

[GitBox]

jfrazee commented on pull request #4788:
URL: https://github.com/apache/nifi/pull/4788#issuecomment-779475432


   Per 
https://cwiki.apache.org/confluence/display/NIFI/Version+Scheme+and+API+Compatibility,
 there isn't a commitment for commons and utils. I personally tend toward being 
conservative, though.
   
   I think `FileUtils` gets used by third parties, BUT, searching GitHub I 
couldn't find any users of `computeMd5Digest()` itself.
   
   Thanks for the explanation.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] exceptionfactory commented on pull request #4788: NIFI-8132 Replaced framework uses of MD5 with SHA-256

[GitBox]

exceptionfactory commented on pull request #4788:
URL: https://github.com/apache/nifi/pull/4788#issuecomment-779475215


   @jfrazee The [Version Scheme and API 
Compatibility](https://cwiki.apache.org/confluence/display/NIFI/Version+Scheme+and+API+Compatibility)
 page indicates anything outside of `nifi-api`, aside from the REST API, 
extension components, and other elements mentioned, is not considered part of 
the public API:
   
   > Anything not listed above is not considered part of the API and is subject 
to change in a manner that may differ from the guidance below. There are 
substantial portions of the codebase which are intentionally considered private 
from an API point of view.
   
   So with that background, it does not seem necessary to maintain the MD5 
implementations noted.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] exceptionfactory commented on a change in pull request #4788: NIFI-8132 Replaced framework uses of MD5 with SHA-256

[GitBox]

exceptionfactory commented on a change in pull request #4788:
URL: https://github.com/apache/nifi/pull/4788#discussion_r576442657



##
File path: 
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/util/file/classloader/ClassLoaderUtils.java
##
@@ -137,25 +137,17 @@ private static boolean isNotBlank(final String value) {
 return additionalClasspath.toArray(new 
URL[additionalClasspath.size()]);
 }
 
-public static String generateAdditionalUrlsFingerprint(Set urls) {
-List listOfUrls = 
urls.stream().map(Object::toString).collect(Collectors.toList());
-StringBuffer urlBuffer = new StringBuffer();
+public static String generateAdditionalUrlsFingerprint(final Set 
urls) {
+final StringBuilder fingerprintBuilder = new StringBuilder();
 
 //Sorting so that the order is maintained for generating the 
fingerprint
-Collections.sort(listOfUrls);
-try {
-MessageDigest md = MessageDigest.getInstance("MD5");
-listOfUrls.forEach(url -> {
-
urlBuffer.append(url).append("-").append(getLastModified(url)).append(";");
-});
-byte[] bytesOfAdditionalUrls = 
urlBuffer.toString().getBytes(StandardCharsets.UTF_8);
-byte[] bytesOfDigest = md.digest(bytesOfAdditionalUrls);
-
-return DatatypeConverter.printHexBinary(bytesOfDigest);
-} catch (NoSuchAlgorithmException e) {
-LOGGER.error("Unable to generate fingerprint for the provided 
additional resources {}", new Object[]{urls, e});
-return null;
-}
+final List sortedUrls = 
urls.stream().map(Object::toString).collect(Collectors.toList());
+Collections.sort(sortedUrls);

Review comment:
   Thanks for the clarification, that makes sense, will make the change.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] exceptionfactory commented on a change in pull request #4788: NIFI-8132 Replaced framework uses of MD5 with SHA-256

[GitBox]

exceptionfactory commented on a change in pull request #4788:
URL: https://github.com/apache/nifi/pull/4788#discussion_r576441917



##
File path: 
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/util/file/FileUtils.java
##
@@ -541,57 +540,30 @@ public static void syncWithRestore(final File 
primaryFile, final File restoreFil
 }
 
 /**
- * Returns true if the given files are the same according to their MD5 
hash.
+ * Returns true if the given files are the same according to their hash.
  *
  * @param file1 a file
  * @param file2 a file
  * @return true if the files are the same; false otherwise
- * @throws IOException if the MD5 hash could not be computed
+ * @throws IOException if the hash could not be computed
  */
 public static boolean isSame(final File file1, final File file2) throws 
IOException {
-return Arrays.equals(computeMd5Digest(file1), computeMd5Digest(file2));
+return Arrays.equals(computeDigest(file1), computeDigest(file2));
 }
 
 /**
- * Returns the MD5 hash of the given file.
+ * Returns the hash of the given file using default digest algorithm
  *
  * @param file a file
- * @return the MD5 hash
- * @throws IOException if the MD5 hash could not be computed
+ * @return Digest Hash Bytes
+ * @throws IOException if the hash could not be computed
  */
-public static byte[] computeMd5Digest(final File file) throws IOException {
+public static byte[] computeDigest(final File file) throws IOException {
 try (final FileInputStream fis = new FileInputStream(file)) {
-return computeMd5Digest(fis);
+return MessageDigestUtils.getDigest(fis);
 }
 }
 
-/**
- * Returns the MD5 hash of the given stream.
- *
- * @param stream an input stream
- * @return the MD5 hash
- * @throws IOException if the MD5 hash could not be computed
- */
-public static byte[] computeMd5Digest(final InputStream stream) throws 
IOException {

Review comment:
   This is a good point to consider.  Should all public methods in classes 
outside of API modules be considered part of the semantically versioned API?  
Perhaps more specifically, should classes in utility modules be considered part 
of the public API?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] jfrazee commented on a change in pull request #4788: NIFI-8132 Replaced framework uses of MD5 with SHA-256

[GitBox]

jfrazee commented on a change in pull request #4788:
URL: https://github.com/apache/nifi/pull/4788#discussion_r576441402



##
File path: 
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/util/file/classloader/ClassLoaderUtils.java
##
@@ -137,25 +137,17 @@ private static boolean isNotBlank(final String value) {
 return additionalClasspath.toArray(new 
URL[additionalClasspath.size()]);
 }
 
-public static String generateAdditionalUrlsFingerprint(Set urls) {
-List listOfUrls = 
urls.stream().map(Object::toString).collect(Collectors.toList());
-StringBuffer urlBuffer = new StringBuffer();
+public static String generateAdditionalUrlsFingerprint(final Set 
urls) {
+final StringBuilder fingerprintBuilder = new StringBuilder();
 
 //Sorting so that the order is maintained for generating the 
fingerprint
-Collections.sort(listOfUrls);
-try {
-MessageDigest md = MessageDigest.getInstance("MD5");
-listOfUrls.forEach(url -> {
-
urlBuffer.append(url).append("-").append(getLastModified(url)).append(";");
-});
-byte[] bytesOfAdditionalUrls = 
urlBuffer.toString().getBytes(StandardCharsets.UTF_8);
-byte[] bytesOfDigest = md.digest(bytesOfAdditionalUrls);
-
-return DatatypeConverter.printHexBinary(bytesOfDigest);
-} catch (NoSuchAlgorithmException e) {
-LOGGER.error("Unable to generate fingerprint for the provided 
additional resources {}", new Object[]{urls, e});
-return null;
-}
+final List sortedUrls = 
urls.stream().map(Object::toString).collect(Collectors.toList());
+Collections.sort(sortedUrls);

Review comment:
   I'm suggestion adding `.sorted()` instead following with the 
`Collections.sort(sortedUrls);`.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] exceptionfactory commented on a change in pull request #4788: NIFI-8132 Replaced framework uses of MD5 with SHA-256

[GitBox]

exceptionfactory commented on a change in pull request #4788:
URL: https://github.com/apache/nifi/pull/4788#discussion_r576440903



##
File path: 
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/util/file/classloader/ClassLoaderUtils.java
##
@@ -137,25 +137,17 @@ private static boolean isNotBlank(final String value) {
 return additionalClasspath.toArray(new 
URL[additionalClasspath.size()]);
 }
 
-public static String generateAdditionalUrlsFingerprint(Set urls) {
-List listOfUrls = 
urls.stream().map(Object::toString).collect(Collectors.toList());
-StringBuffer urlBuffer = new StringBuffer();
+public static String generateAdditionalUrlsFingerprint(final Set 
urls) {

Review comment:
   As mentioned in the more lengthy comment, the return is still a string, 
although the content of the string is now based on SHA-256 instead of MD5.  
Since the method name does not imply MD5, I'm not sure if it constitutes a 
change in semantic behavior.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] exceptionfactory commented on a change in pull request #4788: NIFI-8132 Replaced framework uses of MD5 with SHA-256

[GitBox]

exceptionfactory commented on a change in pull request #4788:
URL: https://github.com/apache/nifi/pull/4788#discussion_r576440030



##
File path: 
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/util/file/classloader/ClassLoaderUtils.java
##
@@ -137,25 +137,17 @@ private static boolean isNotBlank(final String value) {
 return additionalClasspath.toArray(new 
URL[additionalClasspath.size()]);
 }
 
-public static String generateAdditionalUrlsFingerprint(Set urls) {
-List listOfUrls = 
urls.stream().map(Object::toString).collect(Collectors.toList());
-StringBuffer urlBuffer = new StringBuffer();
+public static String generateAdditionalUrlsFingerprint(final Set 
urls) {
+final StringBuilder fingerprintBuilder = new StringBuilder();
 
 //Sorting so that the order is maintained for generating the 
fingerprint
-Collections.sort(listOfUrls);
-try {
-MessageDigest md = MessageDigest.getInstance("MD5");
-listOfUrls.forEach(url -> {
-
urlBuffer.append(url).append("-").append(getLastModified(url)).append(";");
-});
-byte[] bytesOfAdditionalUrls = 
urlBuffer.toString().getBytes(StandardCharsets.UTF_8);
-byte[] bytesOfDigest = md.digest(bytesOfAdditionalUrls);
-
-return DatatypeConverter.printHexBinary(bytesOfDigest);
-} catch (NoSuchAlgorithmException e) {
-LOGGER.error("Unable to generate fingerprint for the provided 
additional resources {}", new Object[]{urls, e});
-return null;
-}
+final List sortedUrls = 
urls.stream().map(Object::toString).collect(Collectors.toList());
+Collections.sort(sortedUrls);

Review comment:
   This is a formatting change, but follows the same function.  The 
`map(Object::toString)` method is necessary to convert each URL to a string.  
Perhaps I am missing what you are suggesting.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] exceptionfactory commented on pull request #4788: NIFI-8132 Replaced framework uses of MD5 with SHA-256

[GitBox]

exceptionfactory commented on pull request #4788:
URL: https://github.com/apache/nifi/pull/4788#issuecomment-779460977


   > I think this is a good change, but the behavior of a couple of public 
methods in util classes has changed and a a class was renamed. These could be 
reasonably be being used by users, so I think it'd be useful to have these 
deprecated for a cycle with a call out in the release/upgrade notes. 
Specifically:
   > 
   > 1. I think it's probably ok to change `isSame()`. It's rather unlikely 
for a difference to surface.
   > 
   > 2. It'd be good to leave a `@Deprecated` `computeMd5Digest()` for the 
1.14 cycle.
   > 
   > 3. I'm torn on `generateAdditionalUrlsFingerprint` since it's intended 
use is pretty specific and I don't think it's as likely that there are users of 
it.
   > 
   > 4. I think we can just deprecate `MD5SumMonitor` and add your new 
`DigestUpdateMonitor`.
   > 
   > 
   > Since it's (2) and (4) don't require any work, I think we should leave 
them around for a bit. The tough one is `generateAdditionalUrlsFingerprint`.
   
   Thanks for the thoughtful feedback @jfrazee!  The points noted raise some 
good questions about the expectation of compatibility in various NiFi modules.  
For instance, `MD5SumMonitor` is used exclusively in unit tests within the code 
base.  Aside from the usage of `computeMd5Digest()` in the `isSame()` method, 
the only usage of that method was in tests.  For that method in particular, it 
seems that it would have been better to mark it as private from the beginning.  
Regarding `generateAdditionalUrlsFingerprint()`, the return value is now an 
SHA-256 digest instead of an MD5 digest, but the concept of computing a unique 
fingerprint from the provided values is unchanged.  Unfortunately the method 
was not documented and did not otherwise indicate that an MD5 digest would be 
returned.
   
   It would be helpful to get some input from additional maintainers, but if 
all public methods in any class within NiFi should be considered part of the 
official semantic version, then it seems that more rigor is necessary when 
creating things like utility classes.  I'm open to marking `computeMd5Digest` 
as deprecated if absolutely necessary, but would prefer to leave the other 
changes.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] jfrazee commented on pull request #4815: NIFI-8200: Modifying PutAzureDataLakeStorage to delete temp file if e…

[GitBox]

jfrazee commented on pull request #4815:
URL: https://github.com/apache/nifi/pull/4815#issuecomment-779444703


   @timeabarna What happens when an existing file is being overwritten with the 
Conflict Resolution Strategy set to replace?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] jfrazee commented on a change in pull request #4788: NIFI-8132 Replaced framework uses of MD5 with SHA-256

[GitBox]

jfrazee commented on a change in pull request #4788:
URL: https://github.com/apache/nifi/pull/4788#discussion_r576380776



##
File path: 
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/util/file/classloader/ClassLoaderUtils.java
##
@@ -137,25 +137,17 @@ private static boolean isNotBlank(final String value) {
 return additionalClasspath.toArray(new 
URL[additionalClasspath.size()]);
 }
 
-public static String generateAdditionalUrlsFingerprint(Set urls) {
-List listOfUrls = 
urls.stream().map(Object::toString).collect(Collectors.toList());
-StringBuffer urlBuffer = new StringBuffer();
+public static String generateAdditionalUrlsFingerprint(final Set 
urls) {

Review comment:
   Similar comment as above. This changes the behavior of a public method.

##
File path: 
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/util/file/FileUtils.java
##
@@ -541,57 +540,30 @@ public static void syncWithRestore(final File 
primaryFile, final File restoreFil
 }
 
 /**
- * Returns true if the given files are the same according to their MD5 
hash.
+ * Returns true if the given files are the same according to their hash.
  *
  * @param file1 a file
  * @param file2 a file
  * @return true if the files are the same; false otherwise
- * @throws IOException if the MD5 hash could not be computed
+ * @throws IOException if the hash could not be computed
  */
 public static boolean isSame(final File file1, final File file2) throws 
IOException {
-return Arrays.equals(computeMd5Digest(file1), computeMd5Digest(file2));
+return Arrays.equals(computeDigest(file1), computeDigest(file2));
 }
 
 /**
- * Returns the MD5 hash of the given file.
+ * Returns the hash of the given file using default digest algorithm
  *
  * @param file a file
- * @return the MD5 hash
- * @throws IOException if the MD5 hash could not be computed
+ * @return Digest Hash Bytes
+ * @throws IOException if the hash could not be computed
  */
-public static byte[] computeMd5Digest(final File file) throws IOException {
+public static byte[] computeDigest(final File file) throws IOException {
 try (final FileInputStream fis = new FileInputStream(file)) {
-return computeMd5Digest(fis);
+return MessageDigestUtils.getDigest(fis);
 }
 }
 
-/**
- * Returns the MD5 hash of the given stream.
- *
- * @param stream an input stream
- * @return the MD5 hash
- * @throws IOException if the MD5 hash could not be computed
- */
-public static byte[] computeMd5Digest(final InputStream stream) throws 
IOException {

Review comment:
   This is public in a package users could be using in custom code. It 
should probably be marked `@Deprecated` for a cycle or two before removing. 

##
File path: 
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/util/file/classloader/ClassLoaderUtils.java
##
@@ -137,25 +137,17 @@ private static boolean isNotBlank(final String value) {
 return additionalClasspath.toArray(new 
URL[additionalClasspath.size()]);
 }
 
-public static String generateAdditionalUrlsFingerprint(Set urls) {
-List listOfUrls = 
urls.stream().map(Object::toString).collect(Collectors.toList());
-StringBuffer urlBuffer = new StringBuffer();
+public static String generateAdditionalUrlsFingerprint(final Set 
urls) {
+final StringBuilder fingerprintBuilder = new StringBuilder();
 
 //Sorting so that the order is maintained for generating the 
fingerprint
-Collections.sort(listOfUrls);
-try {
-MessageDigest md = MessageDigest.getInstance("MD5");
-listOfUrls.forEach(url -> {
-
urlBuffer.append(url).append("-").append(getLastModified(url)).append(";");
-});
-byte[] bytesOfAdditionalUrls = 
urlBuffer.toString().getBytes(StandardCharsets.UTF_8);
-byte[] bytesOfDigest = md.digest(bytesOfAdditionalUrls);
-
-return DatatypeConverter.printHexBinary(bytesOfDigest);
-} catch (NoSuchAlgorithmException e) {
-LOGGER.error("Unable to generate fingerprint for the provided 
additional resources {}", new Object[]{urls, e});
-return null;
-}
+final List sortedUrls = 
urls.stream().map(Object::toString).collect(Collectors.toList());
+Collections.sort(sortedUrls);

Review comment:
   I think we can just use the `sorted()` on the stream.
   ```suggestion
   final List sortedUrls = urls.stream()
   .map(Object::toString)
   .sorted()
   .collect(Collectors.toList());
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
UR

[GitHub] [nifi] exceptionfactory commented on a change in pull request #4079: NIFI-7192 - nifi.sh install not working on systemd servers

[GitBox]

exceptionfactory commented on a change in pull request #4079:
URL: https://github.com/apache/nifi/pull/4079#discussion_r576395796



##
File path: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/bin/nifi.sh
##
@@ -245,6 +245,11 @@ SERVICEDESCRIPTOR
 ln -s "/etc/init.d/${SVC_NAME}" "/etc/rc2.d/K65${SVC_NAME}" || { echo 
"Could not create link /etc/rc2.d/K65${SVC_NAME}"; exit 1; }
 echo "Service ${SVC_NAME} installed"
 fi
+
+# systemd: generate nifi.service from init.d
+if [ "$(pidof systemd)" ]; then

Review comment:
   Thanks for the feedback @rmarturano!  The link from @jfrazee in an 
earlier comment is a helpful reference point for his recommendation, so I will 
defer to the approach that both of you think is best under the circumstances.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] rmarturano commented on a change in pull request #4079: NIFI-7192 - nifi.sh install not working on systemd servers

[GitBox]

rmarturano commented on a change in pull request #4079:
URL: https://github.com/apache/nifi/pull/4079#discussion_r576391843



##
File path: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/bin/nifi.sh
##
@@ -245,6 +245,11 @@ SERVICEDESCRIPTOR
 ln -s "/etc/init.d/${SVC_NAME}" "/etc/rc2.d/K65${SVC_NAME}" || { echo 
"Could not create link /etc/rc2.d/K65${SVC_NAME}"; exit 1; }
 echo "Service ${SVC_NAME} installed"
 fi
+
+# systemd: generate nifi.service from init.d
+if [ "$(pidof systemd)" ]; then

Review comment:
   Can I apply those changes and update the PR?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] rmarturano commented on a change in pull request #4079: NIFI-7192 - nifi.sh install not working on systemd servers

[GitBox]

rmarturano commented on a change in pull request #4079:
URL: https://github.com/apache/nifi/pull/4079#discussion_r576391664



##
File path: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/bin/nifi.sh
##
@@ -245,6 +245,11 @@ SERVICEDESCRIPTOR
 ln -s "/etc/init.d/${SVC_NAME}" "/etc/rc2.d/K65${SVC_NAME}" || { echo 
"Could not create link /etc/rc2.d/K65${SVC_NAME}"; exit 1; }
 echo "Service ${SVC_NAME} installed"
 fi
+
+# systemd: generate nifi.service from init.d
+if [ "$(pidof systemd)" ]; then

Review comment:
   I believe it's important to check if systemd is currently running.
   I'd opt for changes suggested by @jfrazee, even though @exceptionfactory 
option could already cover most of the cases.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] exceptionfactory commented on a change in pull request #4079: NIFI-7192 - nifi.sh install not working on systemd servers

[GitBox]

exceptionfactory commented on a change in pull request #4079:
URL: https://github.com/apache/nifi/pull/4079#discussion_r576387004



##
File path: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/bin/nifi.sh
##
@@ -245,6 +245,11 @@ SERVICEDESCRIPTOR
 ln -s "/etc/init.d/${SVC_NAME}" "/etc/rc2.d/K65${SVC_NAME}" || { echo 
"Could not create link /etc/rc2.d/K65${SVC_NAME}"; exit 1; }
 echo "Service ${SVC_NAME} installed"
 fi
+
+# systemd: generate nifi.service from init.d
+if [ "$(pidof systemd)" ]; then

Review comment:
   That's a good question.  It would be helpful to know how likely that 
scenario is to occur, but if systemd is not being used for initialization, then 
running `systemctl daemon-reload` shouldn't have an impact on the 
initialization configuration.  Ultimately, it would be better to handle service 
installation for systemd using an actual service configuration as opposed to 
relying on backward compatibility, but as it stands, this update seems like a 
good step forward.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Resolved] (NIFI-7873) Conduct 1.13.0 release

[Joe Witt (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-7873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Witt resolved NIFI-7873.

Resolution: Fixed

> Conduct 1.13.0 release
> --
>
> Key: NIFI-7873
> URL: https://issues.apache.org/jira/browse/NIFI-7873
> Project: Apache NiFi
>  Issue Type: Task
>Affects Versions: 1.12.1
>Reporter: Andrei Lopukhov
>Assignee: Joe Witt
>Priority: Trivial
> Fix For: 1.13.0
>
>
> Provide exactly same version of nars in distribution and in maven repo
> Currently nar artifcats in NiFi distribution differs from same artifacts in 
> maven.
> It looks like nars are packaged once again for making distribution assembly.
> Example difference, MANIFEST.MF from nifi-avro-nar-1.12.1:
> ||Maven||Archieve distribution||
> |Manifest-Version: 1.0
> Build-Branch: UNKNOWN
> Build-Timestamp: 2020-09-23T10:17:41Z
> Archiver-Version: Plexus Archiver
> Built-By: jwitt
> Nar-Id: nifi-avro-nar
> Clone-During-Instance-Class-Loading: false
> Nar-Version: 1.12.1
> Build-Tag: nifi-1.12.1-RC2
> Build-Revision: accfaa3
> Nar-Group: org.apache.nifi
> Created-By: Apache Maven 3.6.3
> Build-Jdk: 1.8.0_265|Manifest-Version: 1.0
> Build-Timestamp: 2020-09-23T14:15:53Z
> Clone-During-Instance-Class-Loading: false
> Archiver-Version: Plexus Archiver
> Built-By: jwitt
> Nar-Version: 1.12.1
> Build-Tag: nifi-1.12.1-RC2
> Nar-Id: nifi-avro-nar
> Nar-Group: org.apache.nifi
> Created-By: Apache Maven 3.6.3
> Build-Jdk: 1.8.0_265|
> So it is not possible to validate individual libraries from distribution 
> against libraries in maven cental.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi] jfrazee commented on a change in pull request #4079: NIFI-7192 - nifi.sh install not working on systemd servers

[GitBox]

jfrazee commented on a change in pull request #4079:
URL: https://github.com/apache/nifi/pull/4079#discussion_r576384160



##
File path: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/bin/nifi.sh
##
@@ -245,6 +245,11 @@ SERVICEDESCRIPTOR
 ln -s "/etc/init.d/${SVC_NAME}" "/etc/rc2.d/K65${SVC_NAME}" || { echo 
"Could not create link /etc/rc2.d/K65${SVC_NAME}"; exit 1; }
 echo "Service ${SVC_NAME} installed"
 fi
+
+# systemd: generate nifi.service from init.d
+if [ "$(pidof systemd)" ]; then

Review comment:
   So, there's this unlikely/weird scenario where some of the systemd 
components could be installed but systemd isn't actually the init. Should we 
care about that? I dunno.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (NIFI-7873) Conduct 1.13.0 release

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-7873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284919#comment-17284919
 ] 

ASF subversion and git services commented on NIFI-7873:
---

Commit a69313e34cd2f40e0ef88d53cdf410451f221dc1 in nifi's branch 
refs/heads/main from Joe Witt
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=a69313e ]

NIFI-7873 updating docker files


> Conduct 1.13.0 release
> --
>
> Key: NIFI-7873
> URL: https://issues.apache.org/jira/browse/NIFI-7873
> Project: Apache NiFi
>  Issue Type: Task
>Affects Versions: 1.12.1
>Reporter: Andrei Lopukhov
>Assignee: Joe Witt
>Priority: Trivial
> Fix For: 1.13.0
>
>
> Provide exactly same version of nars in distribution and in maven repo
> Currently nar artifcats in NiFi distribution differs from same artifacts in 
> maven.
> It looks like nars are packaged once again for making distribution assembly.
> Example difference, MANIFEST.MF from nifi-avro-nar-1.12.1:
> ||Maven||Archieve distribution||
> |Manifest-Version: 1.0
> Build-Branch: UNKNOWN
> Build-Timestamp: 2020-09-23T10:17:41Z
> Archiver-Version: Plexus Archiver
> Built-By: jwitt
> Nar-Id: nifi-avro-nar
> Clone-During-Instance-Class-Loading: false
> Nar-Version: 1.12.1
> Build-Tag: nifi-1.12.1-RC2
> Build-Revision: accfaa3
> Nar-Group: org.apache.nifi
> Created-By: Apache Maven 3.6.3
> Build-Jdk: 1.8.0_265|Manifest-Version: 1.0
> Build-Timestamp: 2020-09-23T14:15:53Z
> Clone-During-Instance-Class-Loading: false
> Archiver-Version: Plexus Archiver
> Built-By: jwitt
> Nar-Version: 1.12.1
> Build-Tag: nifi-1.12.1-RC2
> Nar-Id: nifi-avro-nar
> Nar-Group: org.apache.nifi
> Created-By: Apache Maven 3.6.3
> Build-Jdk: 1.8.0_265|
> So it is not possible to validate individual libraries from distribution 
> against libraries in maven cental.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (NIFI-7873) Conduct 1.13.0 release

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-7873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284918#comment-17284918
 ] 

ASF subversion and git services commented on NIFI-7873:
---

Commit 88fab00e299b3ff161e5a40a4326f93c054b2bf9 in nifi's branch 
refs/heads/main from Joe Witt
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=88fab00 ]

NIFI-7873 merging release branch to latest and updating to 1.14.0-SNAPSHOT


> Conduct 1.13.0 release
> --
>
> Key: NIFI-7873
> URL: https://issues.apache.org/jira/browse/NIFI-7873
> Project: Apache NiFi
>  Issue Type: Task
>Affects Versions: 1.12.1
>Reporter: Andrei Lopukhov
>Assignee: Joe Witt
>Priority: Trivial
> Fix For: 1.13.0
>
>
> Provide exactly same version of nars in distribution and in maven repo
> Currently nar artifcats in NiFi distribution differs from same artifacts in 
> maven.
> It looks like nars are packaged once again for making distribution assembly.
> Example difference, MANIFEST.MF from nifi-avro-nar-1.12.1:
> ||Maven||Archieve distribution||
> |Manifest-Version: 1.0
> Build-Branch: UNKNOWN
> Build-Timestamp: 2020-09-23T10:17:41Z
> Archiver-Version: Plexus Archiver
> Built-By: jwitt
> Nar-Id: nifi-avro-nar
> Clone-During-Instance-Class-Loading: false
> Nar-Version: 1.12.1
> Build-Tag: nifi-1.12.1-RC2
> Build-Revision: accfaa3
> Nar-Group: org.apache.nifi
> Created-By: Apache Maven 3.6.3
> Build-Jdk: 1.8.0_265|Manifest-Version: 1.0
> Build-Timestamp: 2020-09-23T14:15:53Z
> Clone-During-Instance-Class-Loading: false
> Archiver-Version: Plexus Archiver
> Built-By: jwitt
> Nar-Version: 1.12.1
> Build-Tag: nifi-1.12.1-RC2
> Nar-Id: nifi-avro-nar
> Nar-Group: org.apache.nifi
> Created-By: Apache Maven 3.6.3
> Build-Jdk: 1.8.0_265|
> So it is not possible to validate individual libraries from distribution 
> against libraries in maven cental.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (NIFI-7873) Conduct 1.13.0 release

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-7873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284917#comment-17284917
 ] 

ASF subversion and git services commented on NIFI-7873:
---

Commit 4afb2ba743d1c84875db9f09d5c2cc04af32055d in nifi's branch 
refs/heads/main from Joe Witt
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=4afb2ba ]

NIFI-7873-RC4 prepare for next development iteration


> Conduct 1.13.0 release
> --
>
> Key: NIFI-7873
> URL: https://issues.apache.org/jira/browse/NIFI-7873
> Project: Apache NiFi
>  Issue Type: Task
>Affects Versions: 1.12.1
>Reporter: Andrei Lopukhov
>Assignee: Joe Witt
>Priority: Trivial
> Fix For: 1.13.0
>
>
> Provide exactly same version of nars in distribution and in maven repo
> Currently nar artifcats in NiFi distribution differs from same artifacts in 
> maven.
> It looks like nars are packaged once again for making distribution assembly.
> Example difference, MANIFEST.MF from nifi-avro-nar-1.12.1:
> ||Maven||Archieve distribution||
> |Manifest-Version: 1.0
> Build-Branch: UNKNOWN
> Build-Timestamp: 2020-09-23T10:17:41Z
> Archiver-Version: Plexus Archiver
> Built-By: jwitt
> Nar-Id: nifi-avro-nar
> Clone-During-Instance-Class-Loading: false
> Nar-Version: 1.12.1
> Build-Tag: nifi-1.12.1-RC2
> Build-Revision: accfaa3
> Nar-Group: org.apache.nifi
> Created-By: Apache Maven 3.6.3
> Build-Jdk: 1.8.0_265|Manifest-Version: 1.0
> Build-Timestamp: 2020-09-23T14:15:53Z
> Clone-During-Instance-Class-Loading: false
> Archiver-Version: Plexus Archiver
> Built-By: jwitt
> Nar-Version: 1.12.1
> Build-Tag: nifi-1.12.1-RC2
> Nar-Id: nifi-avro-nar
> Nar-Group: org.apache.nifi
> Created-By: Apache Maven 3.6.3
> Build-Jdk: 1.8.0_265|
> So it is not possible to validate individual libraries from distribution 
> against libraries in maven cental.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (NIFI-7873) Conduct 1.13.0 release

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-7873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284916#comment-17284916
 ] 

ASF subversion and git services commented on NIFI-7873:
---

Commit 487280bee993ae07aa66a043999664e650af9b64 in nifi's branch 
refs/heads/main from Joe Witt
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=487280b ]

NIFI-7873-RC4 prepare release nifi-1.13.0-RC4


> Conduct 1.13.0 release
> --
>
> Key: NIFI-7873
> URL: https://issues.apache.org/jira/browse/NIFI-7873
> Project: Apache NiFi
>  Issue Type: Task
>Affects Versions: 1.12.1
>Reporter: Andrei Lopukhov
>Assignee: Joe Witt
>Priority: Trivial
> Fix For: 1.13.0
>
>
> Provide exactly same version of nars in distribution and in maven repo
> Currently nar artifcats in NiFi distribution differs from same artifacts in 
> maven.
> It looks like nars are packaged once again for making distribution assembly.
> Example difference, MANIFEST.MF from nifi-avro-nar-1.12.1:
> ||Maven||Archieve distribution||
> |Manifest-Version: 1.0
> Build-Branch: UNKNOWN
> Build-Timestamp: 2020-09-23T10:17:41Z
> Archiver-Version: Plexus Archiver
> Built-By: jwitt
> Nar-Id: nifi-avro-nar
> Clone-During-Instance-Class-Loading: false
> Nar-Version: 1.12.1
> Build-Tag: nifi-1.12.1-RC2
> Build-Revision: accfaa3
> Nar-Group: org.apache.nifi
> Created-By: Apache Maven 3.6.3
> Build-Jdk: 1.8.0_265|Manifest-Version: 1.0
> Build-Timestamp: 2020-09-23T14:15:53Z
> Clone-During-Instance-Class-Loading: false
> Archiver-Version: Plexus Archiver
> Built-By: jwitt
> Nar-Version: 1.12.1
> Build-Tag: nifi-1.12.1-RC2
> Nar-Id: nifi-avro-nar
> Nar-Group: org.apache.nifi
> Created-By: Apache Maven 3.6.3
> Build-Jdk: 1.8.0_265|
> So it is not possible to validate individual libraries from distribution 
> against libraries in maven cental.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi] exceptionfactory commented on a change in pull request #4079: NIFI-7192 - nifi.sh install not working on systemd servers

[GitBox]

exceptionfactory commented on a change in pull request #4079:
URL: https://github.com/apache/nifi/pull/4079#discussion_r576382210



##
File path: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/bin/nifi.sh
##
@@ -245,6 +245,11 @@ SERVICEDESCRIPTOR
 ln -s "/etc/init.d/${SVC_NAME}" "/etc/rc2.d/K65${SVC_NAME}" || { echo 
"Could not create link /etc/rc2.d/K65${SVC_NAME}"; exit 1; }
 echo "Service ${SVC_NAME} installed"
 fi
+
+# systemd: generate nifi.service from init.d
+if [ "$(pidof systemd)" ]; then

Review comment:
   As one additional alternative, since the conditional runs the 
`systemctl` command, what about checking for the command itself?
   ```suggestion
   if [ "$(which systemctl)" ]; then
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] jfrazee commented on a change in pull request #4460: NIFI-7717 Updated main NiFi images to JRE 11.

[GitBox]

jfrazee commented on a change in pull request #4460:
URL: https://github.com/apache/nifi/pull/4460#discussion_r576373673



##
File path: nifi-docker/dockerhub/Dockerfile
##
@@ -15,8 +15,8 @@
 # specific language governing permissions and limitations
 # under the License.
 #
-
-FROM openjdk:8-jre
+ARG JAVA_VERSION
+FROM ${JAVA_VERSION}

Review comment:
   ```suggestion
   ARG IMAGE_NAME=openjdk
   ARG IMAGE_TAG=8-jre
   ARG MAINTAINER="Apache NiFi "
   
   FROM ${IMAGE_NAME}:${IMAGE_TAG} AS artifactbase
   ```

##
File path: nifi-docker/dockermaven/Dockerfile
##
@@ -53,7 +54,7 @@ RUN mkdir -p ${NIFI_HOME}/conf \
 && mkdir -p ${NIFI_HOME}/state \
 && mkdir -p ${NIFI_LOG_DIR}
 
-FROM openjdk:8-jre
+FROM $JAVA_VERSION

Review comment:
   ```suggestion
   FROM ${IMAGE_NAME}:${IMAGE_TAG}
   ```

##
File path: nifi-docker/dockerhub/pom.xml
##
@@ -37,6 +41,7 @@
 
 
 
+
${docker.java.version}

Review comment:
   ```suggestion
   
${docker.image.name}
   
${docker.image.tag}
   
${docker.maintainer}  
  
   ```

##
File path: nifi-docker/dockerhub/pom.xml
##
@@ -20,6 +20,10 @@
 
 dockerhub
 
+
+openjdk:8-jre

Review comment:
   ```suggestion
   openjdk
   8-jre
   
   ```

##
File path: nifi-docker/dockerhub/Dockerfile
##
@@ -15,8 +15,8 @@
 # specific language governing permissions and limitations
 # under the License.
 #
-
-FROM openjdk:8-jre
+ARG JAVA_VERSION
+FROM ${JAVA_VERSION}
 LABEL maintainer="Apache NiFi "

Review comment:
   ```suggestion
   LABEL maintainer="${MAINTAINER}"
   ```

##
File path: nifi-docker/dockermaven/Dockerfile
##
@@ -16,7 +16,8 @@
 # under the License.
 #
 
-FROM openjdk:8-jre AS artifactbase
+ARG JAVA_VERSION
+FROM $JAVA_VERSION AS artifactbase

Review comment:
   ```suggestion
   ARG IMAGE_NAME=openjdk
   ARG IMAGE_TAG=8-jre
   ARG MAINTAINER="Apache NiFi "
   
   FROM ${IMAGE_NAME}:${IMAGE_TAG} AS artifactbase
   ```

##
File path: nifi-docker/dockermaven/pom.xml
##
@@ -20,6 +20,10 @@
 
 dockermaven
 
+
+openjdk:8-jre

Review comment:
   ```suggestion
   openjdk
   8-jre
   
   ```

##
File path: nifi-docker/dockermaven/pom.xml
##
@@ -37,6 +41,7 @@
 
 
 
+
${docker.java.version}

Review comment:
   ```suggestion
   
${docker.image.name}
   
${docker.image.tag}
   
${docker.maintainer}  
  
   ```

##
File path: nifi-docker/dockermaven/Dockerfile
##
@@ -53,7 +54,7 @@ RUN mkdir -p ${NIFI_HOME}/conf \
 && mkdir -p ${NIFI_HOME}/state \
 && mkdir -p ${NIFI_LOG_DIR}
 
-FROM openjdk:8-jre
+FROM $JAVA_VERSION
 LABEL maintainer="Apache NiFi "

Review comment:
   ```suggestion
   LABEL maintainer="${MAINTAINER}"
   ```

##
File path: nifi-docker/dockermaven/Dockerfile
##
@@ -16,7 +16,8 @@
 # under the License.
 #
 
-FROM openjdk:8-jre AS artifactbase
+ARG JAVA_VERSION
+FROM $JAVA_VERSION AS artifactbase
 LABEL maintainer="Apache NiFi "

Review comment:
   ```suggestion
   LABEL maintainer="${MAINTAINER}"
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] jfrazee commented on a change in pull request #4079: NIFI-7192 - nifi.sh install not working on systemd servers

[GitBox]

jfrazee commented on a change in pull request #4079:
URL: https://github.com/apache/nifi/pull/4079#discussion_r576372149



##
File path: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/bin/nifi.sh
##
@@ -245,6 +245,11 @@ SERVICEDESCRIPTOR
 ln -s "/etc/init.d/${SVC_NAME}" "/etc/rc2.d/K65${SVC_NAME}" || { echo 
"Could not create link /etc/rc2.d/K65${SVC_NAME}"; exit 1; }
 echo "Service ${SVC_NAME} installed"
 fi
+
+# systemd: generate nifi.service from init.d
+if [ "$(pidof systemd)" ]; then

Review comment:
   ```suggestion
   if [ -d "/run/systemd/system/" ] || [ ! -z "$(pidof systemd 
2>/dev/null)" ]; then
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] jfrazee commented on pull request #4460: NIFI-7717 Updated main NiFi images to JRE 11.

[GitBox]

jfrazee commented on pull request #4460:
URL: https://github.com/apache/nifi/pull/4460#issuecomment-779386944


   @MikeThomsen I wanted to come back to this. I've been using this for a while 
now and think it'd be helpful to split off the image/repository name, the image 
tag, and the maintainer as ARGs. For example:
   
   ```dockerfile
   ARG IMAGE_NAME=openjdk
   ARG IMAGE_TAG=8-jre
   ARG MAINTAINER="Apache NiFi "
   
   FROM ${IMAGE_NAME}:${IMAGE_TAG} AS artifactbase
   LABEL maintainer="${MAINTAINER}"
   
   ...
   
   FROM ${IMAGE_NAME}:${IMAGE_TAG}
   LABEL maintainer="${MAINTAINER}"
   ```
   
   And:
   
   ```xml
   
   openjdk
   8-jre
   
   
   ```
   
   What do you think?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (NIFIREG-439) Logout causes NullPointerException and continutes to display resources anonymous should not see

[Chris Sampson (Jira)]

Chris Sampson created NIFIREG-439:
-

 Summary: Logout causes NullPointerException and continutes to 
display resources anonymous should not see
 Key: NIFIREG-439
 URL: https://issues.apache.org/jira/browse/NIFIREG-439
 Project: NiFi Registry
  Issue Type: Bug
Affects Versions: 0.8.0
Reporter: Chris Sampson


After configuring OIDC login through NiFi Registry UI (which I note appears to 
need an explicit click of the {{Login}} button in the UI rather than 
automatically logging the user in like NiFi UI), I see the following behaviour:
* {{Login}} via OIDC (link in UI)
* Display list of buckets (to which {{anonymous}} users do not have access)
* {{Logout}} (link in UI)
* See the below log from NiFi Registry
* Note that the buckets are still displayed in the UI for the {{anonymous}} user

{code:java}
2021-02-15 17:10:48,374 ERROR [NiFi Registry Web Server-18] 
o.a.n.r.web.mapper.ThrowableMapper An unexpected error has occurred: 
java.lang.NullPointerException. Returning Internal Server Error response.
java.lang.NullPointerException: null
at java.util.regex.Matcher.getTextLength(Matcher.java:1283)
at java.util.regex.Matcher.reset(Matcher.java:309)
at java.util.regex.Matcher.(Matcher.java:229)
at java.util.regex.Pattern.matcher(Pattern.java:1093)
at 
org.apache.nifi.registry.web.security.authentication.jwt.JwtService.getTokenFromHeader(JwtService.java:238)
at 
org.apache.nifi.registry.web.security.authentication.jwt.JwtService.logOutUsingAuthHeader(JwtService.java:233)
at 
org.apache.nifi.registry.web.api.AccessResource.oidcLogout(AccessResource.java:708)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
at 
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:159)
at 
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)
at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)
at 
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)
at 
org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at 
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
at 
org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
at 
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
at 
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:392)
at 
org.glassfish.jersey.servlet.ServletContainer.serviceImpl(ServletContainer.java:385)
at 
org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:560)
at 
org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:501)
at 
org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:438)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:566)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)
at 

[jira] [Commented] (NIFI-7873) Conduct 1.13.0 release

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-7873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284849#comment-17284849
 ] 

ASF subversion and git services commented on NIFI-7873:
---

Commit 1886537 from Joe Witt in branch 'site/trunk'
[ https://svn.apache.org/r1886537 ]

NIFI-7873 Added 1.13.0 docs to NiFi site.

> Conduct 1.13.0 release
> --
>
> Key: NIFI-7873
> URL: https://issues.apache.org/jira/browse/NIFI-7873
> Project: Apache NiFi
>  Issue Type: Task
>Affects Versions: 1.12.1
>Reporter: Andrei Lopukhov
>Assignee: Joe Witt
>Priority: Trivial
> Fix For: 1.13.0
>
>
> Provide exactly same version of nars in distribution and in maven repo
> Currently nar artifcats in NiFi distribution differs from same artifacts in 
> maven.
> It looks like nars are packaged once again for making distribution assembly.
> Example difference, MANIFEST.MF from nifi-avro-nar-1.12.1:
> ||Maven||Archieve distribution||
> |Manifest-Version: 1.0
> Build-Branch: UNKNOWN
> Build-Timestamp: 2020-09-23T10:17:41Z
> Archiver-Version: Plexus Archiver
> Built-By: jwitt
> Nar-Id: nifi-avro-nar
> Clone-During-Instance-Class-Loading: false
> Nar-Version: 1.12.1
> Build-Tag: nifi-1.12.1-RC2
> Build-Revision: accfaa3
> Nar-Group: org.apache.nifi
> Created-By: Apache Maven 3.6.3
> Build-Jdk: 1.8.0_265|Manifest-Version: 1.0
> Build-Timestamp: 2020-09-23T14:15:53Z
> Clone-During-Instance-Class-Loading: false
> Archiver-Version: Plexus Archiver
> Built-By: jwitt
> Nar-Version: 1.12.1
> Build-Tag: nifi-1.12.1-RC2
> Nar-Id: nifi-avro-nar
> Nar-Group: org.apache.nifi
> Created-By: Apache Maven 3.6.3
> Build-Jdk: 1.8.0_265|
> So it is not possible to validate individual libraries from distribution 
> against libraries in maven cental.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (NIFI-7873) Conduct 1.13.0 release

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-7873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284842#comment-17284842
 ] 

ASF subversion and git services commented on NIFI-7873:
---

Commit 1886536 from Joe Witt in branch 'site/trunk'
[ https://svn.apache.org/r1886536 ]

NIFI-7873 1.13 released

> Conduct 1.13.0 release
> --
>
> Key: NIFI-7873
> URL: https://issues.apache.org/jira/browse/NIFI-7873
> Project: Apache NiFi
>  Issue Type: Task
>Affects Versions: 1.12.1
>Reporter: Andrei Lopukhov
>Assignee: Joe Witt
>Priority: Trivial
> Fix For: 1.13.0
>
>
> Provide exactly same version of nars in distribution and in maven repo
> Currently nar artifcats in NiFi distribution differs from same artifacts in 
> maven.
> It looks like nars are packaged once again for making distribution assembly.
> Example difference, MANIFEST.MF from nifi-avro-nar-1.12.1:
> ||Maven||Archieve distribution||
> |Manifest-Version: 1.0
> Build-Branch: UNKNOWN
> Build-Timestamp: 2020-09-23T10:17:41Z
> Archiver-Version: Plexus Archiver
> Built-By: jwitt
> Nar-Id: nifi-avro-nar
> Clone-During-Instance-Class-Loading: false
> Nar-Version: 1.12.1
> Build-Tag: nifi-1.12.1-RC2
> Build-Revision: accfaa3
> Nar-Group: org.apache.nifi
> Created-By: Apache Maven 3.6.3
> Build-Jdk: 1.8.0_265|Manifest-Version: 1.0
> Build-Timestamp: 2020-09-23T14:15:53Z
> Clone-During-Instance-Class-Loading: false
> Archiver-Version: Plexus Archiver
> Built-By: jwitt
> Nar-Version: 1.12.1
> Build-Tag: nifi-1.12.1-RC2
> Nar-Id: nifi-avro-nar
> Nar-Group: org.apache.nifi
> Created-By: Apache Maven 3.6.3
> Build-Jdk: 1.8.0_265|
> So it is not possible to validate individual libraries from distribution 
> against libraries in maven cental.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #979: MINIFICPP-1456 Introduce PutAzureBlobStorage processor

[GitBox]

lordgamez commented on a change in pull request #979:
URL: https://github.com/apache/nifi-minifi-cpp/pull/979#discussion_r576324164



##
File path: extensions/azure/processors/PutAzureBlobStorage.cpp
##
@@ -0,0 +1,225 @@
+/**
+ * @file PutAzureBlobStorage.cpp
+ * PutAzureBlobStorage class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "PutAzureBlobStorage.h"
+
+#include 
+#include 
+
+#include "storage/AzureBlobStorage.h"
+#include "controllerservices/AzureStorageCredentialsService.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace azure {
+namespace processors {
+
+const core::Property PutAzureBlobStorage::ContainerName(
+  core::PropertyBuilder::createProperty("Container Name")
+->withDescription("Name of the Azure storage container. In case of 
PutAzureBlobStorage processor, container can be created if it does not exist.")
+->supportsExpressionLanguage(true)
+->isRequired(true)
+->build());
+const core::Property PutAzureBlobStorage::AzureStorageCredentialsService(
+  core::PropertyBuilder::createProperty("Azure Storage Credentials Service")
+->withDescription("Name of the Azure Storage Credentials Service used to 
retrieve the connection string from.")
+->build());
+const core::Property PutAzureBlobStorage::StorageAccountName(
+core::PropertyBuilder::createProperty("Storage Account Name")
+  ->withDescription("The storage account name.")
+  ->supportsExpressionLanguage(true)
+  ->build());
+const core::Property PutAzureBlobStorage::StorageAccountKey(
+core::PropertyBuilder::createProperty("Storage Account Key")
+  ->withDescription("The storage account key. This is an admin-like 
password providing access to every container in this account. "
+"It is recommended one uses Shared Access Signature 
(SAS) token instead for fine-grained control with policies.")
+  ->supportsExpressionLanguage(true)
+  ->build());
+const core::Property PutAzureBlobStorage::SASToken(
+core::PropertyBuilder::createProperty("SAS Token")
+  ->withDescription("Shared Access Signature token. Specify either SAS 
Token (recommended) or Account Key.")
+  ->supportsExpressionLanguage(true)
+  ->build());
+const core::Property PutAzureBlobStorage::CommonStorageAccountEndpointSuffix(
+core::PropertyBuilder::createProperty("Common Storage Account Endpoint 
Suffix")
+  ->withDescription("Storage accounts in public Azure always use a common 
FQDN suffix. Override this endpoint suffix with a "
+"different suffix in certain circumstances (like Azure 
Stack or non-public Azure regions). ")
+  ->supportsExpressionLanguage(true)
+  ->build());
+const core::Property PutAzureBlobStorage::ConnectionString(
+  core::PropertyBuilder::createProperty("Connection String")
+->withDescription("Connection string used to connect to Azure Storage 
service. This overrides all other set credential properties.")
+->supportsExpressionLanguage(true)
+->build());
+const core::Property PutAzureBlobStorage::Blob(
+  core::PropertyBuilder::createProperty("Blob")
+->withDescription("The filename of the blob.")
+->supportsExpressionLanguage(true)
+->isRequired(true)
+->build());
+const core::Property PutAzureBlobStorage::CreateContainer(
+  core::PropertyBuilder::createProperty("Create Container")
+->withDescription("Specifies whether to check if the container exists and 
to automatically create it if it does not. "
+  "Permission to list containers is required. If false, 
this check is not made, but the Put operation will "
+  "fail if the container does not exist.")
+->isRequired(true)
+->withDefaultValue(false)
+->build());
+
+const core::Relationship PutAzureBlobStorage::Success("success", "All 
successfully processed FlowFiles are routed to this relationship");
+const core::Relationship PutAzureBlobStorage::Failure("failure", "Unsuccessful 
operations will be transferred to the failure relationship");
+
+void PutAzureBlobStorage::initialize() {
+  // Set the supported properties
+  setSupportedProperties({
+

[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #979: MINIFICPP-1456 Introduce PutAzureBlobStorage processor

[GitBox]

lordgamez commented on a change in pull request #979:
URL: https://github.com/apache/nifi-minifi-cpp/pull/979#discussion_r576323911



##
File path: extensions/azure/processors/PutAzureBlobStorage.h
##
@@ -0,0 +1,138 @@
+/**
+ * @file PutAzureBlobStorage.h
+ * PutAzureBlobStorage class declaration
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include 
+#include 
+#include 
+#include 
+
+#include "core/Property.h"
+#include "core/Processor.h"
+#include "core/logging/Logger.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "storage/BlobStorage.h"
+#include "utils/OptionalUtils.h"
+
+class PutAzureBlobStorageTestsFixture;
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace azure {
+namespace processors {
+
+class PutAzureBlobStorage : public core::Processor {
+ public:
+  static constexpr char const* ProcessorName = "PutAzureBlobStorage";
+
+  // Supported Properties
+  static const core::Property ContainerName;
+  static const core::Property AzureStorageCredentialsService;
+  static const core::Property StorageAccountName;
+  static const core::Property StorageAccountKey;
+  static const core::Property SASToken;
+  static const core::Property CommonStorageAccountEndpointSuffix;
+  static const core::Property ConnectionString;
+  static const core::Property Blob;
+  static const core::Property CreateContainer;
+
+  // Supported Relationships
+  static const core::Relationship Failure;
+  static const core::Relationship Success;
+
+  explicit PutAzureBlobStorage(std::string name, minifi::utils::Identifier 
uuid = minifi::utils::Identifier())
+: PutAzureBlobStorage(name, uuid, nullptr) {
+  }
+
+  ~PutAzureBlobStorage() override = default;
+
+  bool supportsDynamicProperties() override { return true; }
+  void initialize() override;
+  void onSchedule(const std::shared_ptr &context, const 
std::shared_ptr &sessionFactory) override;
+  void onTrigger(const std::shared_ptr &context, const 
std::shared_ptr &session) override;
+
+  class ReadCallback : public InputStreamCallback {
+   public:
+ReadCallback(uint64_t flow_size, azure::storage::BlobStorage& 
blob_storage_wrapper, const std::string &blob_name)
+  : flow_size_(flow_size)
+  , blob_storage_wrapper_(blob_storage_wrapper)
+  , blob_name_(blob_name) {
+}
+
+int64_t process(const std::shared_ptr& stream) override {
+  std::vector buffer;
+  buffer.reserve(flow_size_);
+  int read_ret = stream->read(buffer.data(), flow_size_);

Review comment:
   Fixed in 
[8853da](https://github.com/apache/nifi-minifi-cpp/pull/979/commits/8853da7346c55a08a326609e97ad09675443e7fb).
 Used read overload with vector parameter where resize is used already, where 
it was possible.

##
File path: extensions/azure/storage/AzureBlobStorage.cpp
##
@@ -0,0 +1,83 @@
+/**
+ * @file AzureBlobStorage.cpp
+ * AzureBlobStorage class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "AzureBlobStorage.h"
+#include "utils/GeneralUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace azure {
+namespace storage {
+
+AzureBlobStorage::AzureBlobStorage(const std::string &connection_string, const 
std::string &container_name)
+  : BlobStorage(connection_string, container_name)
+  , 
container_client_(minifi::utils::make_unique(
+  
Azure::Storage::Blobs::BlobContainerClient::C

[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #979: MINIFICPP-1456 Introduce PutAzureBlobStorage processor

[GitBox]

lordgamez commented on a change in pull request #979:
URL: https://github.com/apache/nifi-minifi-cpp/pull/979#discussion_r576323319



##
File path: extensions/azure/processors/PutAzureBlobStorage.cpp
##
@@ -0,0 +1,225 @@
+/**
+ * @file PutAzureBlobStorage.cpp
+ * PutAzureBlobStorage class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "PutAzureBlobStorage.h"
+
+#include 
+#include 
+
+#include "storage/AzureBlobStorage.h"
+#include "controllerservices/AzureStorageCredentialsService.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace azure {
+namespace processors {
+
+const core::Property PutAzureBlobStorage::ContainerName(
+  core::PropertyBuilder::createProperty("Container Name")
+->withDescription("Name of the Azure storage container. In case of 
PutAzureBlobStorage processor, container can be created if it does not exist.")
+->supportsExpressionLanguage(true)
+->isRequired(true)
+->build());
+const core::Property PutAzureBlobStorage::AzureStorageCredentialsService(
+  core::PropertyBuilder::createProperty("Azure Storage Credentials Service")
+->withDescription("Name of the Azure Storage Credentials Service used to 
retrieve the connection string from.")
+->build());
+const core::Property PutAzureBlobStorage::StorageAccountName(
+core::PropertyBuilder::createProperty("Storage Account Name")
+  ->withDescription("The storage account name.")
+  ->supportsExpressionLanguage(true)
+  ->build());
+const core::Property PutAzureBlobStorage::StorageAccountKey(
+core::PropertyBuilder::createProperty("Storage Account Key")
+  ->withDescription("The storage account key. This is an admin-like 
password providing access to every container in this account. "
+"It is recommended one uses Shared Access Signature 
(SAS) token instead for fine-grained control with policies.")
+  ->supportsExpressionLanguage(true)
+  ->build());
+const core::Property PutAzureBlobStorage::SASToken(
+core::PropertyBuilder::createProperty("SAS Token")
+  ->withDescription("Shared Access Signature token. Specify either SAS 
Token (recommended) or Account Key.")
+  ->supportsExpressionLanguage(true)
+  ->build());
+const core::Property PutAzureBlobStorage::CommonStorageAccountEndpointSuffix(
+core::PropertyBuilder::createProperty("Common Storage Account Endpoint 
Suffix")
+  ->withDescription("Storage accounts in public Azure always use a common 
FQDN suffix. Override this endpoint suffix with a "
+"different suffix in certain circumstances (like Azure 
Stack or non-public Azure regions). ")
+  ->supportsExpressionLanguage(true)
+  ->build());
+const core::Property PutAzureBlobStorage::ConnectionString(
+  core::PropertyBuilder::createProperty("Connection String")
+->withDescription("Connection string used to connect to Azure Storage 
service. This overrides all other set credential properties.")
+->supportsExpressionLanguage(true)
+->build());
+const core::Property PutAzureBlobStorage::Blob(
+  core::PropertyBuilder::createProperty("Blob")
+->withDescription("The filename of the blob.")
+->supportsExpressionLanguage(true)
+->isRequired(true)
+->build());
+const core::Property PutAzureBlobStorage::CreateContainer(
+  core::PropertyBuilder::createProperty("Create Container")
+->withDescription("Specifies whether to check if the container exists and 
to automatically create it if it does not. "
+  "Permission to list containers is required. If false, 
this check is not made, but the Put operation will "
+  "fail if the container does not exist.")
+->isRequired(true)
+->withDefaultValue(false)
+->build());
+
+const core::Relationship PutAzureBlobStorage::Success("success", "All 
successfully processed FlowFiles are routed to this relationship");
+const core::Relationship PutAzureBlobStorage::Failure("failure", "Unsuccessful 
operations will be transferred to the failure relationship");
+
+void PutAzureBlobStorage::initialize() {
+  // Set the supported properties
+  setSupportedProperties({
+

[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #979: MINIFICPP-1456 Introduce PutAzureBlobStorage processor

[GitBox]

lordgamez commented on a change in pull request #979:
URL: https://github.com/apache/nifi-minifi-cpp/pull/979#discussion_r576323135



##
File path: extensions/azure/controllerservices/AzureStorageCredentialsService.h
##
@@ -0,0 +1,85 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include 
+#include 
+
+#include "core/Resource.h"
+#include "core/controller/ControllerService.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "storage/AzureStorageCredentials.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace azure {
+namespace controllers {
+
+class AzureStorageCredentialsService : public 
core::controller::ControllerService {
+ public:
+  static const core::Property StorageAccountName;
+  static const core::Property StorageAccountKey;
+  static const core::Property SASToken;
+  static const core::Property CommonStorageAccountEndpointSuffix;
+  static const core::Property ConnectionString;
+
+  explicit AzureStorageCredentialsService(const std::string &name, const 
minifi::utils::Identifier& uuid = {})
+  : ControllerService(name, uuid),
+
logger_(logging::LoggerFactory::getLogger()) {
+  }
+
+  explicit AzureStorageCredentialsService(const std::string &name, const 
std::shared_ptr &configuration)
+  : ControllerService(name),
+
logger_(logging::LoggerFactory::getLogger()) {
+  }
+
+  void initialize() override;
+
+  void yield() override {
+  };

Review comment:
   Updated in 
[8853da](https://github.com/apache/nifi-minifi-cpp/pull/979/commits/8853da7346c55a08a326609e97ad09675443e7fb)





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (MINIFICPP-1494) InvokeHTTP should behave in the same way as NiFi

[Ferenc Gerlits (Jira)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-1494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ferenc Gerlits updated MINIFICPP-1494:
--
Description: 
There are some differences between the NiFi and MiNiFi implementations of 
InvokeHTTP.  I suspect these are not intentional, and we should change MiNiFi 
to behave in the same way as NiFi.
 #  When InvokeHTTP is timer-driven and the Method is GET, then NiFi does not 
require an incoming flow file, but MiNiFi does.
 # The NiFi version of InvokeHTTP has an output relationship called Original.  
This is called Success in MiNiFi; it should probably be renamed to Original.

TODO: any others?

*These changes could break some existing flow files; discuss whether this is 
acceptable.*

  was:
There are some differences between the NiFi and MiNiFi implementations of 
InvokeHTTP.  I suspect these are not intentional, and we should change MiNiFi 
to behave in the same way as NiFi.
 #  When InvokeHTTP is timer-driven and the Method is GET, then NiFi does not 
require an incoming flow file, but MiNiFi does.
 # The NiFi version of InvokeHTTP has an output relationship called Original.  
This is called Success in MiNiFi; it should probably be renamed to Original.

TODO: any others?

* These changes could break some existing flow files; discuss whether this is 
acceptable. *


> InvokeHTTP should behave in the same way as NiFi
> 
>
> Key: MINIFICPP-1494
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1494
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Ferenc Gerlits
>Priority: Minor
>
> There are some differences between the NiFi and MiNiFi implementations of 
> InvokeHTTP.  I suspect these are not intentional, and we should change MiNiFi 
> to behave in the same way as NiFi.
>  #  When InvokeHTTP is timer-driven and the Method is GET, then NiFi does not 
> require an incoming flow file, but MiNiFi does.
>  # The NiFi version of InvokeHTTP has an output relationship called Original. 
>  This is called Success in MiNiFi; it should probably be renamed to Original.
> TODO: any others?
> *These changes could break some existing flow files; discuss whether this is 
> acceptable.*



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (MINIFICPP-1494) InvokeHTTP should behave in the same way as NiFi

[Ferenc Gerlits (Jira)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-1494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ferenc Gerlits updated MINIFICPP-1494:
--
Description: 
There are some differences between the NiFi and MiNiFi implementations of 
InvokeHTTP.  I suspect these are not intentional, and we should change MiNiFi 
to behave in the same way as NiFi.
 #  When InvokeHTTP is timer-driven and the Method is GET, then NiFi does not 
require an incoming flow file, but MiNiFi does.
 # The NiFi version of InvokeHTTP has an output relationship called Original.  
This is called Success in MiNiFi; it should probably be renamed to Original.

TODO: any others?

*These changes could break some existing flows; discuss whether this is 
acceptable.*

  was:
There are some differences between the NiFi and MiNiFi implementations of 
InvokeHTTP.  I suspect these are not intentional, and we should change MiNiFi 
to behave in the same way as NiFi.
 #  When InvokeHTTP is timer-driven and the Method is GET, then NiFi does not 
require an incoming flow file, but MiNiFi does.
 # The NiFi version of InvokeHTTP has an output relationship called Original.  
This is called Success in MiNiFi; it should probably be renamed to Original.

TODO: any others?

*These changes could break some existing flow files; discuss whether this is 
acceptable.*


> InvokeHTTP should behave in the same way as NiFi
> 
>
> Key: MINIFICPP-1494
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1494
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Ferenc Gerlits
>Priority: Minor
>
> There are some differences between the NiFi and MiNiFi implementations of 
> InvokeHTTP.  I suspect these are not intentional, and we should change MiNiFi 
> to behave in the same way as NiFi.
>  #  When InvokeHTTP is timer-driven and the Method is GET, then NiFi does not 
> require an incoming flow file, but MiNiFi does.
>  # The NiFi version of InvokeHTTP has an output relationship called Original. 
>  This is called Success in MiNiFi; it should probably be renamed to Original.
> TODO: any others?
> *These changes could break some existing flows; discuss whether this is 
> acceptable.*



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (MINIFICPP-1494) InvokeHTTP should behave in the same way as NiFi

[Ferenc Gerlits (Jira)]

Ferenc Gerlits created MINIFICPP-1494:
-

 Summary: InvokeHTTP should behave in the same way as NiFi
 Key: MINIFICPP-1494
 URL: https://issues.apache.org/jira/browse/MINIFICPP-1494
 Project: Apache NiFi MiNiFi C++
  Issue Type: Improvement
Reporter: Ferenc Gerlits


There are some differences between the NiFi and MiNiFi implementations of 
InvokeHTTP.  I suspect these are not intentional, and we should change MiNiFi 
to behave in the same way as NiFi.
 #  When InvokeHTTP is timer-driven and the Method is GET, then NiFi does not 
require an incoming flow file, but MiNiFi does.
 # The NiFi version of InvokeHTTP has an output relationship called Original.  
This is called Success in MiNiFi; it should probably be renamed to Original.

TODO: any others?

* These changes could break some existing flow files; discuss whether this is 
acceptable. *



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (NIFI-7361) UI - policy icon not showing up correctly

[David Handermann (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-7361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Handermann updated NIFI-7361:
---
Fix Version/s: 1.14.0
   Resolution: Fixed
   Status: Resolved  (was: Patch Available)

> UI - policy icon not showing up correctly
> -
>
> Key: NIFI-7361
> URL: https://issues.apache.org/jira/browse/NIFI-7361
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Affects Versions: 1.13.0, 1.12.1
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Minor
> Fix For: 1.14.0
>
> Attachments: Screenshot 2020-04-14 at 21.13.27.png
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> In the UI, the policy icon is not always showing up correctly. For instance, 
> here is a screenshot for a stopped reporting task:
> !Screenshot 2020-04-14 at 21.13.27.png|width=518,height=83!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (NIFI-7361) UI - policy icon not showing up correctly

[David Handermann (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-7361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Handermann updated NIFI-7361:
---
Affects Version/s: 1.13.0
   1.12.1

> UI - policy icon not showing up correctly
> -
>
> Key: NIFI-7361
> URL: https://issues.apache.org/jira/browse/NIFI-7361
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Affects Versions: 1.13.0, 1.12.1
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Minor
> Attachments: Screenshot 2020-04-14 at 21.13.27.png
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> In the UI, the policy icon is not always showing up correctly. For instance, 
> here is a screenshot for a stopped reporting task:
> !Screenshot 2020-04-14 at 21.13.27.png|width=518,height=83!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (NIFI-7361) UI - policy icon not showing up correctly

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-7361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284829#comment-17284829
 ] 

ASF subversion and git services commented on NIFI-7361:
---

Commit a1c82a66a1705c12c7917b232a7f85dc93981143 in nifi's branch 
refs/heads/main from Pierre Villard
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=a1c82a6 ]

NIFI-7361 - increase actions column width for reporting tasks

This closes #4214

Signed-off-by: David Handermann 


> UI - policy icon not showing up correctly
> -
>
> Key: NIFI-7361
> URL: https://issues.apache.org/jira/browse/NIFI-7361
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Minor
> Attachments: Screenshot 2020-04-14 at 21.13.27.png
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> In the UI, the policy icon is not always showing up correctly. For instance, 
> here is a screenshot for a stopped reporting task:
> !Screenshot 2020-04-14 at 21.13.27.png|width=518,height=83!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi] asfgit closed pull request #4214: NIFI-7361 - increase actions column width for reporting tasks

[GitBox]

asfgit closed pull request #4214:
URL: https://github.com/apache/nifi/pull/4214


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] exceptionfactory commented on pull request #4214: NIFI-7361 - increase actions column width for reporting tasks

[GitBox]

exceptionfactory commented on pull request #4214:
URL: https://github.com/apache/nifi/pull/4214#issuecomment-779337149


   Confirmed changes, looks good. +1 Merging.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] lordgamez commented on pull request #1001: MINIFICPP-1491: Fixing build failures in Tensorflow extension

[GitBox]

lordgamez commented on pull request #1001:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1001#issuecomment-779307903


   I am okay with the change, but IMO we should also consider adding the 
compilation of the extension to one of our CI jobs as well to avoid regression 
in the future. That would unfortunately implicate further changes:
   
   - Bazel should be included in the bootstrap.sh in case Tensorflow is built
   - A CMake file for bundling Tensorflow should be added
   - Tensorflow then should be built locally which needs to be checked how long 
that may take in the CI environment



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] exceptionfactory commented on pull request #4214: NIFI-7361 - increase actions column width for reporting tasks

[GitBox]

exceptionfactory commented on pull request #4214:
URL: https://github.com/apache/nifi/pull/4214#issuecomment-779298217


   Thanks @pvillard31, I will confirm the changes and plan on merging unless I 
notice any issues.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] martinzink commented on a change in pull request #1002: MINIFICPP-1490: Add ARCH support to bootstrap process

[GitBox]

martinzink commented on a change in pull request #1002:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1002#discussion_r576271702



##
File path: arch.sh
##
@@ -0,0 +1,94 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+verify_enable_platform(){
+feature="$1"
+feature_status=${!1}
+verify_gcc_enable $feature
+}
+add_os_flags() {
+CMAKE_BUILD_COMMAND="${CMAKE_BUILD_COMMAND} -DFAIL_ON_WARNINGS= "
+}
+bootstrap_cmake(){
+sudo pacman -S --noconfirm cmake
+}
+build_deps(){
+COMMAND="sudo pacman -S --noconfirm --needed cmake gcc zlib openssl 
util-linux"
+
+INSTALLED=()
+INSTALLED+=("bzip2")
+for option in "${OPTIONS[@]}" ; do
+option_value="${!option}"
+if [ "$option_value" = "${TRUE}" ]; then
+# option is enabled
+FOUND_VALUE=""
+for cmake_opt in "${DEPENDENCIES[@]}" ; do
+KEY=${cmake_opt%%:*}
+VALUE=${cmake_opt#*:}
+if [ "$KEY" = "$option" ]; then
+FOUND_VALUE="$VALUE"
+if [ "$FOUND_VALUE" = "libcurl" ]; then
+INSTALLED+=("curl")
+elif [ "$FOUND_VALUE" = "libpcap" ]; then
+INSTALLED+=("libpcap")
+elif [ "$FOUND_VALUE" = "openssl" ]; then
+INSTALLED+=("openssl")
+elif [ "$FOUND_VALUE" = "libusb" ]; then
+INSTALLED+=("libusb")
+elif [ "$FOUND_VALUE" = "libpng" ]; then
+INSTALLED+=("libpng")
+elif [ "$FOUND_VALUE" = "bison" ]; then
+INSTALLED+=("bison")
+elif [ "$FOUND_VALUE" = "flex" ]; then
+INSTALLED+=("flex")
+elif [ "$FOUND_VALUE" = "automake" ]; then
+INSTALLED+=("automake")
+elif [ "$FOUND_VALUE" = "autoconf" ]; then
+INSTALLED+=("autoconf")
+elif [ "$FOUND_VALUE" = "libtool" ]; then
+INSTALLED+=("libtool")
+elif [ "$FOUND_VALUE" = "python" ]; then
+INSTALLED+=("python")
+elif [ "$FOUND_VALUE" = "jnibuild" ]; then
+INSTALLED+=("jdk8-openjdk")
+INSTALLED+=("maven")
+elif [ "$FOUND_VALUE" = "lua" ]; then
+INSTALLED+=("lua")
+elif [ "$FOUND_VALUE" = "gpsd" ]; then
+INSTALLED+=("gpsd")
+elif [ "$FOUND_VALUE" = "libarchive" ]; then
+INSTALLED+=("libarchive")
+elif [ "$FOUND_VALUE" = "tensorflow" ]; then
+INSTALLED+=("tensorflow")
+elif [ "$FOUND_VALUE" = "boost" ]; then
+INSTALLED+=("boost")
+fi
+fi
+done
+
+fi
+done
+
+for option in "${INSTALLED[@]}" ; do
+COMMAND="${COMMAND} $option"
+done
+
+echo "Ensuring you have all dependencies installed..."
+${COMMAND}
+
+}

Review comment:
   This file was created based on the aptitude.sh, replaced the apt calls 
with the proper pacman calls.
   Tested on arch (docker), and manjaro (native)
   Successfully installed with all extensions enabled, except 
   * TensorFlow Support (see 
[MINIFICPP-1491](https://issues.apache.org/jira/browse/MINIFICPP-1491))
   * PCAP support (see 
[MINIFICPP-1492](https://issues.apache.org/jira/browse/MINIFICPP-1492))
   * GPS support (see 
[MINIFICPP-1493](https://issues.apache.org/jira/browse/MINIFICPP-1493))





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] martinzink commented on a change in pull request #1002: MINIFICPP-1490: Add ARCH support to bootstrap process

[GitBox]

martinzink commented on a change in pull request #1002:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1002#discussion_r576271702



##
File path: arch.sh
##
@@ -0,0 +1,94 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+verify_enable_platform(){
+feature="$1"
+feature_status=${!1}
+verify_gcc_enable $feature
+}
+add_os_flags() {
+CMAKE_BUILD_COMMAND="${CMAKE_BUILD_COMMAND} -DFAIL_ON_WARNINGS= "
+}
+bootstrap_cmake(){
+sudo pacman -S --noconfirm cmake
+}
+build_deps(){
+COMMAND="sudo pacman -S --noconfirm --needed cmake gcc zlib openssl 
util-linux"
+
+INSTALLED=()
+INSTALLED+=("bzip2")
+for option in "${OPTIONS[@]}" ; do
+option_value="${!option}"
+if [ "$option_value" = "${TRUE}" ]; then
+# option is enabled
+FOUND_VALUE=""
+for cmake_opt in "${DEPENDENCIES[@]}" ; do
+KEY=${cmake_opt%%:*}
+VALUE=${cmake_opt#*:}
+if [ "$KEY" = "$option" ]; then
+FOUND_VALUE="$VALUE"
+if [ "$FOUND_VALUE" = "libcurl" ]; then
+INSTALLED+=("curl")
+elif [ "$FOUND_VALUE" = "libpcap" ]; then
+INSTALLED+=("libpcap")
+elif [ "$FOUND_VALUE" = "openssl" ]; then
+INSTALLED+=("openssl")
+elif [ "$FOUND_VALUE" = "libusb" ]; then
+INSTALLED+=("libusb")
+elif [ "$FOUND_VALUE" = "libpng" ]; then
+INSTALLED+=("libpng")
+elif [ "$FOUND_VALUE" = "bison" ]; then
+INSTALLED+=("bison")
+elif [ "$FOUND_VALUE" = "flex" ]; then
+INSTALLED+=("flex")
+elif [ "$FOUND_VALUE" = "automake" ]; then
+INSTALLED+=("automake")
+elif [ "$FOUND_VALUE" = "autoconf" ]; then
+INSTALLED+=("autoconf")
+elif [ "$FOUND_VALUE" = "libtool" ]; then
+INSTALLED+=("libtool")
+elif [ "$FOUND_VALUE" = "python" ]; then
+INSTALLED+=("python")
+elif [ "$FOUND_VALUE" = "jnibuild" ]; then
+INSTALLED+=("jdk8-openjdk")
+INSTALLED+=("maven")
+elif [ "$FOUND_VALUE" = "lua" ]; then
+INSTALLED+=("lua")
+elif [ "$FOUND_VALUE" = "gpsd" ]; then
+INSTALLED+=("gpsd")
+elif [ "$FOUND_VALUE" = "libarchive" ]; then
+INSTALLED+=("libarchive")
+elif [ "$FOUND_VALUE" = "tensorflow" ]; then
+INSTALLED+=("tensorflow")
+elif [ "$FOUND_VALUE" = "boost" ]; then
+INSTALLED+=("boost")
+fi
+fi
+done
+
+fi
+done
+
+for option in "${INSTALLED[@]}" ; do
+COMMAND="${COMMAND} $option"
+done
+
+echo "Ensuring you have all dependencies installed..."
+${COMMAND}
+
+}

Review comment:
   This file was created based on the aptitude.sh, replaced the apt calls 
with the proper pacman calls.
   Tested on arch (docker), and manjaro (native)
   Successfully installed with all extensions enabled, except 
   * TensorFlow Support (see MINIFICPP-1491)
   * PCAP support (see MINIFICPP-1492)
   * GPS support (see MINIFICPP-1493)





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (MINIFICPP-1493) GPS extension incompatible with newer gpsd versions (3.21 or higher)

[Martin Zink (Jira)]

Martin Zink created MINIFICPP-1493:
--

 Summary: GPS extension incompatible with newer gpsd versions (3.21 
or higher)
 Key: MINIFICPP-1493
 URL: https://issues.apache.org/jira/browse/MINIFICPP-1493
 Project: Apache NiFi MiNiFi C++
  Issue Type: Bug
Reporter: Martin Zink


GPS extension is incompatible with the latest gpsd library releases.

Causing build error
{code:java}
[ 47%] Building CXX object extensions/gps/CMakeFiles/minifi-gps.dir/GetGPS.cpp.o
./nifi-minifi-cpp/extensions/gps/GetGPS.cpp: In member function ‘virtual void 
org::apache::nifi::minifi::processors::GetGPS::onTrigger(const 
std::shared_ptr&, const 
std::shared_ptr&)’:
./nifi-minifi-cpp/extensions/gps/GetGPS.cpp:109:22: error: ‘struct gps_data_t’ 
has no member named ‘status’{code}
This is due to that gpsd 3.21 (released in 2020-08-04)  changed gps_data_t 
struct.

https://gitlab.com/gpsd/gpsd/commit/29991d6ffeb41ecfc8297db68bb68be0128c8514



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (MINIFICPP-1492) The bundled PCAP++ is not compatible with linux kernel 5.5(or higher)

[Martin Zink (Jira)]

Martin Zink created MINIFICPP-1492:
--

 Summary: The bundled PCAP++ is not compatible with linux kernel 
5.5(or higher)
 Key: MINIFICPP-1492
 URL: https://issues.apache.org/jira/browse/MINIFICPP-1492
 Project: Apache NiFi MiNiFi C++
  Issue Type: Bug
Reporter: Martin Zink


The bundled pcap++ is not compatible with linux kernel 5.5 (or higher), due to 
the removal of sys/sysctl.h 
[https://man7.org/linux/man-pages/man2/sysctl.2.html]

Causing a build error:
{code:java}
src/PcapLiveDevice.cpp:25:10: fatal error: sys/sysctl.h: No such file or 
directory 25 | #include 
{code}
This is fixed in newer PCAP++ versions, it also pretty easy to backport to our 
version, 
[https://github.com/seladb/PcapPlusPlus/commit/f474316b2fcdcd3b2d6ab6143ef4c3b2b5ec9ea8]
 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi-minifi-cpp] martinzink opened a new pull request #1002: MINIFICPP-1490: Add ARCH support to bootstrap process

[GitBox]

martinzink opened a new pull request #1002:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1002


   Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
   
   - [ ] Does your PR title start with MINIFICPP- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   
   - [ ] Has your PR been rebased against the latest commit within the target 
branch (typically main)?
   
   - [ ] Is your initial contribution a single, squashed commit?
   
   ### For code changes:
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the LICENSE file?
   - [ ] If applicable, have you updated the NOTICE file?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check GitHub Actions CI 
results for build issues and submit an update to your PR as soon as possible.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] martinzink commented on a change in pull request #1001: MINIFICPP-1491: Fixing build failures in Tensorflow extension

[GitBox]

martinzink commented on a change in pull request #1001:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1001#discussion_r576234257



##
File path: extensions/tensorflow/TFExtractTopLabels.cpp
##
@@ -133,11 +133,11 @@ int64_t 
TFExtractTopLabels::LabelsReadCallback::process(const std::shared_ptrgetSize()) {
+  while (total_read < stream->size()) {

Review comment:
   This is a leftover from the stream refactor 3a1a35b





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] martinzink commented on a change in pull request #1001: MINIFICPP-1491: Fixing build failures in Tensorflow extension

[GitBox]

martinzink commented on a change in pull request #1001:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1001#discussion_r576233353



##
File path: libminifi/test/tensorflow-tests/TensorFlowTests.cpp
##
@@ -45,8 +45,8 @@ TEST_CASE("TensorFlow: Apply Graph", "[tfApplyGraph]") { // 
NOLINT
   auto repo = std::make_shared();
 
   // Define directory for input protocol buffers
-  std::string in_dir("/tmp/gt.XX");
-  REQUIRE(testController.createTempDirectory(&in_dir[0]) != nullptr);
+  char in_dir_format[] = "/tmp/gt.XX";
+  std::string in_dir = testController.createTempDirectory(in_dir_format);

Review comment:
   The createTempDirectory-s behavior was changed in a6876db





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] martinzink opened a new pull request #1001: MINIFICPP-1491: Fixing build failures in Tensorflow extension

[GitBox]

martinzink opened a new pull request #1001:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1001


   Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
   
   - [ ] Does your PR title start with MINIFICPP- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   
   - [ ] Has your PR been rebased against the latest commit within the target 
branch (typically main)?
   
   - [ ] Is your initial contribution a single, squashed commit?
   
   ### For code changes:
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the LICENSE file?
   - [ ] If applicable, have you updated the NOTICE file?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check GitHub Actions CI 
results for build issues and submit an update to your PR as soon as possible.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #979: MINIFICPP-1456 Introduce PutAzureBlobStorage processor

[GitBox]

adamdebreceni commented on a change in pull request #979:
URL: https://github.com/apache/nifi-minifi-cpp/pull/979#discussion_r576216074



##
File path: extensions/azure/processors/PutAzureBlobStorage.h
##
@@ -0,0 +1,138 @@
+/**
+ * @file PutAzureBlobStorage.h
+ * PutAzureBlobStorage class declaration
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include 
+#include 
+#include 
+#include 
+
+#include "core/Property.h"
+#include "core/Processor.h"
+#include "core/logging/Logger.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "storage/BlobStorage.h"
+#include "utils/OptionalUtils.h"
+
+class PutAzureBlobStorageTestsFixture;
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace azure {
+namespace processors {
+
+class PutAzureBlobStorage : public core::Processor {
+ public:
+  static constexpr char const* ProcessorName = "PutAzureBlobStorage";
+
+  // Supported Properties
+  static const core::Property ContainerName;
+  static const core::Property AzureStorageCredentialsService;
+  static const core::Property StorageAccountName;
+  static const core::Property StorageAccountKey;
+  static const core::Property SASToken;
+  static const core::Property CommonStorageAccountEndpointSuffix;
+  static const core::Property ConnectionString;
+  static const core::Property Blob;
+  static const core::Property CreateContainer;
+
+  // Supported Relationships
+  static const core::Relationship Failure;
+  static const core::Relationship Success;
+
+  explicit PutAzureBlobStorage(std::string name, minifi::utils::Identifier 
uuid = minifi::utils::Identifier())
+: PutAzureBlobStorage(name, uuid, nullptr) {
+  }
+
+  ~PutAzureBlobStorage() override = default;
+
+  bool supportsDynamicProperties() override { return true; }
+  void initialize() override;
+  void onSchedule(const std::shared_ptr &context, const 
std::shared_ptr &sessionFactory) override;
+  void onTrigger(const std::shared_ptr &context, const 
std::shared_ptr &session) override;
+
+  class ReadCallback : public InputStreamCallback {
+   public:
+ReadCallback(uint64_t flow_size, azure::storage::BlobStorage& 
blob_storage_wrapper, const std::string &blob_name)
+  : flow_size_(flow_size)
+  , blob_storage_wrapper_(blob_storage_wrapper)
+  , blob_name_(blob_name) {
+}
+
+int64_t process(const std::shared_ptr& stream) override {
+  std::vector buffer;
+  buffer.reserve(flow_size_);
+  int read_ret = stream->read(buffer.data(), flow_size_);

Review comment:
   this is also an issue in `S3WrapperBase.cpp:147`, `PutS3Object.h:94` and 
`PublishMQTT.h:82`





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (MINIFICPP-1491) Build failure when building Tensorflow extension

[Martin Zink (Jira)]

Martin Zink created MINIFICPP-1491:
--

 Summary: Build failure when building Tensorflow extension
 Key: MINIFICPP-1491
 URL: https://issues.apache.org/jira/browse/MINIFICPP-1491
 Project: Apache NiFi MiNiFi C++
  Issue Type: Bug
Reporter: Martin Zink
Assignee: Martin Zink


 

 

I've come across this build error while trying to build various extensions
{code:java}
[ 26%] Building CXX object 
extensions/tensorflow/CMakeFiles/minifi-tensorflow-extensions.dir/TFExtractTopLabels.cpp.o
./nifi-minifi-cpp/extensions/tensorflow/TFExtractTopLabels.cpp: In member 
function ‘virtual int64_t 
org::apache::nifi::minifi::processors::TFExtractTopLabels::LabelsReadCallback::process(const
 std::shared_ptr&)’:
./nifi-minifi-cpp/extensions/tensorflow/TFExtractTopLabels.cpp:136:31: error: 
‘using element_type = class org::apache::nifi::minifi::io::BaseStream’ {aka 
‘class org::apache::nifi::minifi::io::BaseStream’} has no member named ‘getSize’
 136 | while (total_read < stream->getSize()) {
 | ^~~
./nifi-minifi-cpp/extensions/tensorflow/TFExtractTopLabels.cpp:140:55: error: 
‘using element_type = class org::apache::nifi::minifi::io::BaseStream’ {aka 
‘class org::apache::nifi::minifi::io::BaseStream’} has no member named ‘getSize’
 140 | if (buf[i] == '\n' || total_read + i == stream->getSize()) {
 | ^~~
./nifi-minifi-cpp/extensions/tensorflow/TFExtractTopLabels.cpp: In member 
function ‘virtual void 
org::apache::nifi::minifi::processors::TFExtractTopLabels::onTrigger(const 
std::shared_ptr&, const 
std::shared_ptr&)’:
./nifi-minifi-cpp/extensions/tensorflow/TFExtractTopLabels.cpp:90:20: warning: 
ignoring return value of ‘bool tensorflow::Tensor::FromProto(const 
tensorflow::TensorProto&)’ declared with attribute ‘warn_unused_result’ 
[-Wunused-result]
 90 | input.FromProto(*input_tensor_proto);
 | ~~~^
make[2]: *** 
[extensions/tensorflow/CMakeFiles/minifi-tensorflow-extensions.dir/build.make:108:
 
extensions/tensorflow/CMakeFiles/minifi-tensorflow-extensions.dir/TFExtractTopLabels.cpp.o]
 Error 1
make[1]: *** [CMakeFiles/Makefile2:4365: 
extensions/tensorflow/CMakeFiles/minifi-tensorflow-extensions.dir/all] Error 2
make: *** [Makefile:182: all] Error 2{code}
Seems like it was missed during [MINIFICPP-1319 Stream 
refactor|https://github.com/apache/nifi-minifi-cpp/commit/3a1a35b89e8d930dde43b3b18a1bdf990d10cc01]
 and since there is no CI for this extension, it went unnoticed.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #979: MINIFICPP-1456 Introduce PutAzureBlobStorage processor

[GitBox]

adamdebreceni commented on a change in pull request #979:
URL: https://github.com/apache/nifi-minifi-cpp/pull/979#discussion_r576207532



##
File path: extensions/azure/processors/PutAzureBlobStorage.cpp
##
@@ -0,0 +1,225 @@
+/**
+ * @file PutAzureBlobStorage.cpp
+ * PutAzureBlobStorage class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "PutAzureBlobStorage.h"
+
+#include 
+#include 
+
+#include "storage/AzureBlobStorage.h"
+#include "controllerservices/AzureStorageCredentialsService.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace azure {
+namespace processors {
+
+const core::Property PutAzureBlobStorage::ContainerName(
+  core::PropertyBuilder::createProperty("Container Name")
+->withDescription("Name of the Azure storage container. In case of 
PutAzureBlobStorage processor, container can be created if it does not exist.")
+->supportsExpressionLanguage(true)
+->isRequired(true)
+->build());
+const core::Property PutAzureBlobStorage::AzureStorageCredentialsService(
+  core::PropertyBuilder::createProperty("Azure Storage Credentials Service")
+->withDescription("Name of the Azure Storage Credentials Service used to 
retrieve the connection string from.")
+->build());
+const core::Property PutAzureBlobStorage::StorageAccountName(
+core::PropertyBuilder::createProperty("Storage Account Name")
+  ->withDescription("The storage account name.")
+  ->supportsExpressionLanguage(true)
+  ->build());
+const core::Property PutAzureBlobStorage::StorageAccountKey(
+core::PropertyBuilder::createProperty("Storage Account Key")
+  ->withDescription("The storage account key. This is an admin-like 
password providing access to every container in this account. "
+"It is recommended one uses Shared Access Signature 
(SAS) token instead for fine-grained control with policies.")
+  ->supportsExpressionLanguage(true)
+  ->build());
+const core::Property PutAzureBlobStorage::SASToken(
+core::PropertyBuilder::createProperty("SAS Token")
+  ->withDescription("Shared Access Signature token. Specify either SAS 
Token (recommended) or Account Key.")
+  ->supportsExpressionLanguage(true)
+  ->build());
+const core::Property PutAzureBlobStorage::CommonStorageAccountEndpointSuffix(
+core::PropertyBuilder::createProperty("Common Storage Account Endpoint 
Suffix")
+  ->withDescription("Storage accounts in public Azure always use a common 
FQDN suffix. Override this endpoint suffix with a "
+"different suffix in certain circumstances (like Azure 
Stack or non-public Azure regions). ")
+  ->supportsExpressionLanguage(true)
+  ->build());
+const core::Property PutAzureBlobStorage::ConnectionString(
+  core::PropertyBuilder::createProperty("Connection String")
+->withDescription("Connection string used to connect to Azure Storage 
service. This overrides all other set credential properties.")
+->supportsExpressionLanguage(true)
+->build());
+const core::Property PutAzureBlobStorage::Blob(
+  core::PropertyBuilder::createProperty("Blob")
+->withDescription("The filename of the blob.")
+->supportsExpressionLanguage(true)
+->isRequired(true)
+->build());
+const core::Property PutAzureBlobStorage::CreateContainer(
+  core::PropertyBuilder::createProperty("Create Container")
+->withDescription("Specifies whether to check if the container exists and 
to automatically create it if it does not. "
+  "Permission to list containers is required. If false, 
this check is not made, but the Put operation will "
+  "fail if the container does not exist.")
+->isRequired(true)
+->withDefaultValue(false)
+->build());
+
+const core::Relationship PutAzureBlobStorage::Success("success", "All 
successfully processed FlowFiles are routed to this relationship");
+const core::Relationship PutAzureBlobStorage::Failure("failure", "Unsuccessful 
operations will be transferred to the failure relationship");
+
+void PutAzureBlobStorage::initialize() {
+  // Set the supported properties
+  setSupportedProperties

[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #979: MINIFICPP-1456 Introduce PutAzureBlobStorage processor

[GitBox]

adamdebreceni commented on a change in pull request #979:
URL: https://github.com/apache/nifi-minifi-cpp/pull/979#discussion_r576201597



##
File path: extensions/azure/storage/AzureBlobStorage.cpp
##
@@ -0,0 +1,83 @@
+/**
+ * @file AzureBlobStorage.cpp
+ * AzureBlobStorage class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "AzureBlobStorage.h"
+#include "utils/GeneralUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace azure {
+namespace storage {
+
+AzureBlobStorage::AzureBlobStorage(const std::string &connection_string, const 
std::string &container_name)
+  : BlobStorage(connection_string, container_name)
+  , 
container_client_(minifi::utils::make_unique(
+  
Azure::Storage::Blobs::BlobContainerClient::CreateFromConnectionString(connection_string,
 container_name))) {
+}
+
+void AzureBlobStorage::resetClientIfNeeded(const std::string 
&connection_string, const std::string &container_name) {
+  if (connection_string == connection_string_ && container_name_ == 
container_name) {
+logger_->log_debug("Client credentials have not changed, no need to reset 
client");
+return;
+  }
+  connection_string_ = connection_string;
+  container_name_ = container_name;
+  logger_->log_debug("Client has been reset with new credentials");
+  container_client_ = 
minifi::utils::make_unique(Azure::Storage::Blobs::BlobContainerClient::CreateFromConnectionString(connection_string,
 container_name));
+}
+
+void AzureBlobStorage::createContainer() {
+  try {
+auto blob_client = container_client_->Create();
+logger_->log_debug("Container created");
+  } catch (const std::runtime_error&) {
+logger_->log_debug("Container creation failed, it already exists.");
+  }
+}
+
+utils::optional AzureBlobStorage::uploadBlob(const 
std::string &blob_name, const uint8_t* buffer, std::size_t buffer_size) {
+  try {
+auto blob_client = container_client_->GetBlockBlobClient(blob_name);
+auto response = blob_client.UploadFrom(buffer, buffer_size);
+if (!response.HasValue()) {
+  return utils::nullopt;
+}
+
+UploadBlobResult result;
+result.length = buffer_size;
+result.primary_uri = container_client_->GetUrl();
+if (response->ETag.HasValue()) {
+  result.etag = response->ETag.ToString();
+}
+result.timestamp = 
response->LastModified.GetString(Azure::Core::DateTime::DateFormat::Rfc1123);
+return result;
+  } catch (const std::runtime_error&) {
+return utils::nullopt;

Review comment:
   should we log here?

##
File path: extensions/azure/storage/AzureBlobStorage.cpp
##
@@ -0,0 +1,83 @@
+/**
+ * @file AzureBlobStorage.cpp
+ * AzureBlobStorage class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "AzureBlobStorage.h"
+#include "utils/GeneralUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace azure {
+namespace storage {
+
+AzureBlobStorage::AzureBlobStorage(const std::string &connection_string, const 
std::string &container_name)
+  : BlobStorage(connection_string, container_name)
+  , 
container_client_(minifi::utils::make_unique(
+  
Azure::Storage::Blobs::BlobContainerClient::CreateFromConnectionString(connection_string,
 container_name))) {
+}
+
+void AzureBlobStorage::resetClientIfNeeded(const std::string 
&connection_string, const std::string &container_name) {
+  i

[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #979: MINIFICPP-1456 Introduce PutAzureBlobStorage processor

[GitBox]

adamdebreceni commented on a change in pull request #979:
URL: https://github.com/apache/nifi-minifi-cpp/pull/979#discussion_r576198119



##
File path: extensions/azure/processors/PutAzureBlobStorage.h
##
@@ -0,0 +1,138 @@
+/**
+ * @file PutAzureBlobStorage.h
+ * PutAzureBlobStorage class declaration
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include 
+#include 
+#include 
+#include 
+
+#include "core/Property.h"
+#include "core/Processor.h"
+#include "core/logging/Logger.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "storage/BlobStorage.h"
+#include "utils/OptionalUtils.h"
+
+class PutAzureBlobStorageTestsFixture;
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace azure {
+namespace processors {
+
+class PutAzureBlobStorage : public core::Processor {
+ public:
+  static constexpr char const* ProcessorName = "PutAzureBlobStorage";
+
+  // Supported Properties
+  static const core::Property ContainerName;
+  static const core::Property AzureStorageCredentialsService;
+  static const core::Property StorageAccountName;
+  static const core::Property StorageAccountKey;
+  static const core::Property SASToken;
+  static const core::Property CommonStorageAccountEndpointSuffix;
+  static const core::Property ConnectionString;
+  static const core::Property Blob;
+  static const core::Property CreateContainer;
+
+  // Supported Relationships
+  static const core::Relationship Failure;
+  static const core::Relationship Success;
+
+  explicit PutAzureBlobStorage(std::string name, minifi::utils::Identifier 
uuid = minifi::utils::Identifier())
+: PutAzureBlobStorage(name, uuid, nullptr) {
+  }
+
+  ~PutAzureBlobStorage() override = default;
+
+  bool supportsDynamicProperties() override { return true; }
+  void initialize() override;
+  void onSchedule(const std::shared_ptr &context, const 
std::shared_ptr &sessionFactory) override;
+  void onTrigger(const std::shared_ptr &context, const 
std::shared_ptr &session) override;
+
+  class ReadCallback : public InputStreamCallback {
+   public:
+ReadCallback(uint64_t flow_size, azure::storage::BlobStorage& 
blob_storage_wrapper, const std::string &blob_name)
+  : flow_size_(flow_size)
+  , blob_storage_wrapper_(blob_storage_wrapper)
+  , blob_name_(blob_name) {
+}
+
+int64_t process(const std::shared_ptr& stream) override {
+  std::vector buffer;
+  buffer.reserve(flow_size_);
+  int read_ret = stream->read(buffer.data(), flow_size_);

Review comment:
   note that `std::vector::data` is only guaranteed to return a pointer 
such that `[data(), data() + size())` is a valid range, and for empty vectors 
it might return whatever, we should use `resize` 
[[1]](https://stackoverflow.com/questions/38972443/use-stdvectordata-after-reserve)





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #979: MINIFICPP-1456 Introduce PutAzureBlobStorage processor

[GitBox]

adamdebreceni commented on a change in pull request #979:
URL: https://github.com/apache/nifi-minifi-cpp/pull/979#discussion_r576198119



##
File path: extensions/azure/processors/PutAzureBlobStorage.h
##
@@ -0,0 +1,138 @@
+/**
+ * @file PutAzureBlobStorage.h
+ * PutAzureBlobStorage class declaration
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include 
+#include 
+#include 
+#include 
+
+#include "core/Property.h"
+#include "core/Processor.h"
+#include "core/logging/Logger.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "storage/BlobStorage.h"
+#include "utils/OptionalUtils.h"
+
+class PutAzureBlobStorageTestsFixture;
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace azure {
+namespace processors {
+
+class PutAzureBlobStorage : public core::Processor {
+ public:
+  static constexpr char const* ProcessorName = "PutAzureBlobStorage";
+
+  // Supported Properties
+  static const core::Property ContainerName;
+  static const core::Property AzureStorageCredentialsService;
+  static const core::Property StorageAccountName;
+  static const core::Property StorageAccountKey;
+  static const core::Property SASToken;
+  static const core::Property CommonStorageAccountEndpointSuffix;
+  static const core::Property ConnectionString;
+  static const core::Property Blob;
+  static const core::Property CreateContainer;
+
+  // Supported Relationships
+  static const core::Relationship Failure;
+  static const core::Relationship Success;
+
+  explicit PutAzureBlobStorage(std::string name, minifi::utils::Identifier 
uuid = minifi::utils::Identifier())
+: PutAzureBlobStorage(name, uuid, nullptr) {
+  }
+
+  ~PutAzureBlobStorage() override = default;
+
+  bool supportsDynamicProperties() override { return true; }
+  void initialize() override;
+  void onSchedule(const std::shared_ptr &context, const 
std::shared_ptr &sessionFactory) override;
+  void onTrigger(const std::shared_ptr &context, const 
std::shared_ptr &session) override;
+
+  class ReadCallback : public InputStreamCallback {
+   public:
+ReadCallback(uint64_t flow_size, azure::storage::BlobStorage& 
blob_storage_wrapper, const std::string &blob_name)
+  : flow_size_(flow_size)
+  , blob_storage_wrapper_(blob_storage_wrapper)
+  , blob_name_(blob_name) {
+}
+
+int64_t process(const std::shared_ptr& stream) override {
+  std::vector buffer;
+  buffer.reserve(flow_size_);
+  int read_ret = stream->read(buffer.data(), flow_size_);

Review comment:
   note that `std::vector::data` is only guaranteed to return a pointer 
such that `[data(), data() + size())` is a valid range, and for empty vectors 
it might return whatever, we should use `resize` 
[1](https://stackoverflow.com/questions/38972443/use-stdvectordata-after-reserve)





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #979: MINIFICPP-1456 Introduce PutAzureBlobStorage processor

[GitBox]

adamdebreceni commented on a change in pull request #979:
URL: https://github.com/apache/nifi-minifi-cpp/pull/979#discussion_r576191515



##
File path: extensions/azure/processors/PutAzureBlobStorage.cpp
##
@@ -0,0 +1,225 @@
+/**
+ * @file PutAzureBlobStorage.cpp
+ * PutAzureBlobStorage class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "PutAzureBlobStorage.h"
+
+#include 
+#include 
+
+#include "storage/AzureBlobStorage.h"
+#include "controllerservices/AzureStorageCredentialsService.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace azure {
+namespace processors {
+
+const core::Property PutAzureBlobStorage::ContainerName(
+  core::PropertyBuilder::createProperty("Container Name")
+->withDescription("Name of the Azure storage container. In case of 
PutAzureBlobStorage processor, container can be created if it does not exist.")
+->supportsExpressionLanguage(true)
+->isRequired(true)
+->build());
+const core::Property PutAzureBlobStorage::AzureStorageCredentialsService(
+  core::PropertyBuilder::createProperty("Azure Storage Credentials Service")
+->withDescription("Name of the Azure Storage Credentials Service used to 
retrieve the connection string from.")
+->build());
+const core::Property PutAzureBlobStorage::StorageAccountName(
+core::PropertyBuilder::createProperty("Storage Account Name")
+  ->withDescription("The storage account name.")
+  ->supportsExpressionLanguage(true)
+  ->build());
+const core::Property PutAzureBlobStorage::StorageAccountKey(
+core::PropertyBuilder::createProperty("Storage Account Key")
+  ->withDescription("The storage account key. This is an admin-like 
password providing access to every container in this account. "
+"It is recommended one uses Shared Access Signature 
(SAS) token instead for fine-grained control with policies.")
+  ->supportsExpressionLanguage(true)
+  ->build());
+const core::Property PutAzureBlobStorage::SASToken(
+core::PropertyBuilder::createProperty("SAS Token")
+  ->withDescription("Shared Access Signature token. Specify either SAS 
Token (recommended) or Account Key.")
+  ->supportsExpressionLanguage(true)
+  ->build());
+const core::Property PutAzureBlobStorage::CommonStorageAccountEndpointSuffix(
+core::PropertyBuilder::createProperty("Common Storage Account Endpoint 
Suffix")
+  ->withDescription("Storage accounts in public Azure always use a common 
FQDN suffix. Override this endpoint suffix with a "
+"different suffix in certain circumstances (like Azure 
Stack or non-public Azure regions). ")
+  ->supportsExpressionLanguage(true)
+  ->build());
+const core::Property PutAzureBlobStorage::ConnectionString(
+  core::PropertyBuilder::createProperty("Connection String")
+->withDescription("Connection string used to connect to Azure Storage 
service. This overrides all other set credential properties.")
+->supportsExpressionLanguage(true)
+->build());
+const core::Property PutAzureBlobStorage::Blob(
+  core::PropertyBuilder::createProperty("Blob")
+->withDescription("The filename of the blob.")
+->supportsExpressionLanguage(true)
+->isRequired(true)
+->build());
+const core::Property PutAzureBlobStorage::CreateContainer(
+  core::PropertyBuilder::createProperty("Create Container")
+->withDescription("Specifies whether to check if the container exists and 
to automatically create it if it does not. "
+  "Permission to list containers is required. If false, 
this check is not made, but the Put operation will "
+  "fail if the container does not exist.")
+->isRequired(true)
+->withDefaultValue(false)
+->build());
+
+const core::Relationship PutAzureBlobStorage::Success("success", "All 
successfully processed FlowFiles are routed to this relationship");
+const core::Relationship PutAzureBlobStorage::Failure("failure", "Unsuccessful 
operations will be transferred to the failure relationship");
+
+void PutAzureBlobStorage::initialize() {
+  // Set the supported properties
+  setSupportedProperties

[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #979: MINIFICPP-1456 Introduce PutAzureBlobStorage processor

[GitBox]

adamdebreceni commented on a change in pull request #979:
URL: https://github.com/apache/nifi-minifi-cpp/pull/979#discussion_r576187105



##
File path: extensions/azure/controllerservices/AzureStorageCredentialsService.h
##
@@ -0,0 +1,85 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include 
+#include 
+
+#include "core/Resource.h"
+#include "core/controller/ControllerService.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "storage/AzureStorageCredentials.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace azure {
+namespace controllers {
+
+class AzureStorageCredentialsService : public 
core::controller::ControllerService {
+ public:
+  static const core::Property StorageAccountName;
+  static const core::Property StorageAccountKey;
+  static const core::Property SASToken;
+  static const core::Property CommonStorageAccountEndpointSuffix;
+  static const core::Property ConnectionString;
+
+  explicit AzureStorageCredentialsService(const std::string &name, const 
minifi::utils::Identifier& uuid = {})
+  : ControllerService(name, uuid),
+
logger_(logging::LoggerFactory::getLogger()) {
+  }
+
+  explicit AzureStorageCredentialsService(const std::string &name, const 
std::shared_ptr &configuration)
+  : ControllerService(name),
+
logger_(logging::LoggerFactory::getLogger()) {
+  }
+
+  void initialize() override;
+
+  void yield() override {
+  };

Review comment:
   `;` accidental empty declaration?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] hunyadi-dev commented on a change in pull request #940: MINIFICPP-1373 - Implement ConsumeKafka

[GitBox]

hunyadi-dev commented on a change in pull request #940:
URL: https://github.com/apache/nifi-minifi-cpp/pull/940#discussion_r576144662



##
File path: extensions/librdkafka/rdkafka_utils.h
##
@@ -0,0 +1,104 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/OptionalUtils.h"
+#include "rdkafka.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+
+enum class KafkaEncoding {
+  UTF8,
+  HEX
+};
+
+struct rd_kafka_conf_deleter {
+  void operator()(rd_kafka_conf_t* ptr) const noexcept { 
rd_kafka_conf_destroy(ptr); }
+};
+
+struct rd_kafka_producer_deleter {
+  void operator()(rd_kafka_t* ptr) const noexcept {
+rd_kafka_resp_err_t flush_ret = rd_kafka_flush(ptr, 1 /* ms */);  // 
Matching the wait time of KafkaConnection.cpp
+// If concerned, we could log potential errors here:
+// if (RD_KAFKA_RESP_ERR__TIMED_OUT == flush_ret) {
+//   std::cerr << "Deleting producer failed: time-out while trying to 
flush" << std::endl;
+// }
+rd_kafka_destroy(ptr);
+  }
+};
+
+struct rd_kafka_consumer_deleter {
+  void operator()(rd_kafka_t* ptr) const noexcept {
+rd_kafka_consumer_close(ptr);
+rd_kafka_destroy(ptr);
+  }
+};
+
+struct rd_kafka_topic_partition_list_deleter {
+  void operator()(rd_kafka_topic_partition_list_t* ptr) const noexcept { 
rd_kafka_topic_partition_list_destroy(ptr); }
+};
+
+struct rd_kafka_topic_conf_deleter {
+  void operator()(rd_kafka_topic_conf_t* ptr) const noexcept { 
rd_kafka_topic_conf_destroy(ptr); }
+};
+struct rd_kafka_topic_deleter {
+  void operator()(rd_kafka_topic_t* ptr) const noexcept { 
rd_kafka_topic_destroy(ptr); }
+};
+
+struct rd_kafka_message_deleter {
+  void operator()(rd_kafka_message_t* ptr) const noexcept { 
rd_kafka_message_destroy(ptr); }
+};
+
+struct rd_kafka_headers_deleter {
+  void operator()(rd_kafka_headers_t* ptr) const noexcept { 
rd_kafka_headers_destroy(ptr); }
+};
+
+template 
+void kafka_headers_for_each(const rd_kafka_headers_t* headers, T 
key_value_handle) {
+  const char *key;  // Null terminated, not to be freed
+  const void *value;
+  std::size_t size;
+  for (std::size_t i = 0; RD_KAFKA_RESP_ERR_NO_ERROR == 
rd_kafka_header_get_all(headers, i, &key, &value, &size); ++i) {
+key_value_handle(std::string(key), std::string(static_cast(value), size));

Review comment:
   I replaced this with a `gsl::span`, but there is still a copy - now 
performed by the caller, and the function signature looks less clean.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] hunyadi-dev commented on a change in pull request #940: MINIFICPP-1373 - Implement ConsumeKafka

[GitBox]

hunyadi-dev commented on a change in pull request #940:
URL: https://github.com/apache/nifi-minifi-cpp/pull/940#discussion_r576122753



##
File path: libminifi/test/TestBase.cpp
##
@@ -62,78 +63,51 @@ TestPlan::~TestPlan() {
   for (auto& processor : configured_processors_) {
 processor->setScheduledState(core::ScheduledState::STOPPED);
   }
+  for (auto& connection : relationships_) {
+// This is a patch solving circular references between processors and 
connections
+connection->setSource(nullptr);
+connection->setDestination(nullptr);
+  }
   controller_services_provider_->clearControllerServices();
 }
 
 std::shared_ptr TestPlan::addProcessor(const 
std::shared_ptr &processor, const std::string &name, const 
std::initializer_list& relationships,
-bool linkToPrevious) {
+bool linkToPrevious) {
   if (finalized) {
 return nullptr;
   }
   std::lock_guard guard(mutex);
-
   utils::Identifier uuid = utils::IdGenerator::getIdGenerator()->generate();
-
   processor->setStreamFactory(stream_factory);
   // initialize the processor
   processor->initialize();
   processor->setFlowIdentifier(flow_version_->getFlowIdentifier());
-
   processor_mapping_[processor->getUUID()] = processor;
-
   if (!linkToPrevious) {
 termination_ = *(relationships.begin());
   } else {
 std::shared_ptr last = processor_queue_.back();
-
 if (last == nullptr) {
   last = processor;
   termination_ = *(relationships.begin());
 }
-
-std::stringstream connection_name;
-connection_name << last->getUUIDStr() << "-to-" << processor->getUUIDStr();
-logger_->log_info("Creating %s connection for proc %d", 
connection_name.str(), processor_queue_.size() + 1);
-std::shared_ptr connection = 
std::make_shared(flow_repo_, content_repo_, 
connection_name.str());
-
 for (const auto& relationship : relationships) {
-  connection->addRelationship(relationship);
-}
-
-// link the connections so that we can test results at the end for this
-connection->setSource(last);
-connection->setDestination(processor);
-
-connection->setSourceUUID(last->getUUID());
-connection->setDestinationUUID(processor->getUUID());
-last->addConnection(connection);
-if (last != processor) {
-  processor->addConnection(connection);
+  addConnection(last, relationship, processor);

Review comment:
   Why do you expect behavioural differences between the two? If you want 
to have more ellaborate test-setups, you need to have the flexibility of 
separately linking up relationships on processors.
   
   From a more pragmatical standpoint, this change is also tested through all 
the unit tests ran.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi-minifi-cpp] hunyadi-dev commented on a change in pull request #995: MINIFICPP-1445 - Move docker integration tests to python behave

[GitBox]

hunyadi-dev commented on a change in pull request #995:
URL: https://github.com/apache/nifi-minifi-cpp/pull/995#discussion_r576101288



##
File path: docker/test/integration/MiNiFi_integration_test_driver.py
##
@@ -0,0 +1,208 @@
+from subprocess import Popen, PIPE, STDOUT
+
+import docker
+import logging
+import os
+import shutil
+import threading
+import time
+import uuid
+
+from pydoc import locate
+
+from minifi.core.InputPort import InputPort
+
+from minifi.core.DockerTestCluster import DockerTestCluster
+from minifi.core.SingleNodeDockerCluster import SingleNodeDockerCluster
+from minifi.core.DockerTestDirectoryBindings import DockerTestDirectoryBindings
+
+from minifi.validators.EmptyFilesOutPutValidator import 
EmptyFilesOutPutValidator
+from minifi.validators.NoFileOutPutValidator import NoFileOutPutValidator
+from minifi.validators.SingleFileOutputValidator import 
SingleFileOutputValidator
+
+class MiNiFi_integration_test():
+def __init__(self, context):
+logging.info("MiNiFi_integration_test init")
+self.test_id = str(uuid.uuid4())
+self.clusters = {}
+
+self.connectable_nodes = []
+# Remote process groups are not connectables
+self.remote_process_groups = []
+self.file_system_observer = None
+
+self.docker_network = None
+
+self.docker_directory_bindings = DockerTestDirectoryBindings()
+
self.docker_directory_bindings.create_new_data_directories(self.test_id)
+
+def __del__(self):
+logging.info("MiNiFi_integration_test cleanup")
+
+# Clean up network, for some reason only this order of events work for 
cleanup
+if self.docker_network is not None:
+logging.info('Cleaning up network network: %s', 
self.docker_network.name)
+while len(self.docker_network.containers) != 0:
+for container in self.docker_network.containers:
+self.docker_network.disconnect(container, force=True)
+self.docker_network.reload()
+self.docker_network.remove()
+
+container_ids = []
+for cluster in self.clusters.values():
+for container in cluster.containers.values():
+container_ids.append(container.id)
+del cluster
+
+# Backup for cleaning up containers as the cluster deleter is not 
reliable
+docker_client = docker.from_env()
+for container_id in container_ids:
+wait_start_time = time.perf_counter()
+while (time.perf_counter() - wait_start_time) < 35:
+# There is no clean way to check for container existence
+try:
+container = docker_client.containers.get(container_id)
+logging.error("Failure when trying to clean up containers. 
Attempting secondary cleanup.")
+container.kill()
+time.sleep(5)
+container.remove(v=True, force=True)
+time.sleep(5)
+except docker.errors.NotFound:
+break
+try:
+container = docker_client.containers.get(container_id)
+logging.error("All attempts to clean up docker containers were 
unsuccessful.")
+except docker.errors.NotFound:
+logging.info("Docker container secondary cleanup successful.")
+pass

Review comment:
   Updated





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (NIFI-8227) Enable extend of AbstractDatabaseFetchProcessor without adding the whole nifi-standard processors

[Jorge Machado (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-8227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jorge Machado updated NIFI-8227:

Affects Version/s: 1.12.1

> Enable extend of AbstractDatabaseFetchProcessor without adding the whole 
> nifi-standard processors
> -
>
> Key: NIFI-8227
> URL: https://issues.apache.org/jira/browse/NIFI-8227
> Project: Apache NiFi
>  Issue Type: Wish
>Affects Versions: 1.12.1
>Reporter: Jorge Machado
>Priority: Minor
>
> Hi, 
>  
> I'm trying to extend AbstractDatabaseFetchProcessor, which ends up to be 
> difficult. The only way is to pull the standard processors as dependency and 
> this will duplicate all processors on the canvas. We should refactor so that 
> AbstractDatabaseFetchProcessor is moved to a place where can be extended 
> properlly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (NIFI-8227) Enable extend of AbstractDatabaseFetchProcessor without adding the whole nifi-standard processors

[Jorge Machado (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-8227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jorge Machado updated NIFI-8227:

Priority: Minor  (was: Major)

> Enable extend of AbstractDatabaseFetchProcessor without adding the whole 
> nifi-standard processors
> -
>
> Key: NIFI-8227
> URL: https://issues.apache.org/jira/browse/NIFI-8227
> Project: Apache NiFi
>  Issue Type: Wish
>Reporter: Jorge Machado
>Priority: Minor
>
> Hi, 
>  
> I'm trying to extend AbstractDatabaseFetchProcessor, which ends up to be 
> difficult. The only way is to pull the standard processors as dependency and 
> this will duplicate all processors on the canvas. We should refactor so that 
> AbstractDatabaseFetchProcessor is moved to a place where can be extended 
> properlly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (NIFI-8227) Enable extend of AbstractDatabaseFetchProcessor without adding the whole nifi-standard processors

[Jorge Machado (Jira)]

Jorge Machado created NIFI-8227:
---

 Summary: Enable extend of AbstractDatabaseFetchProcessor without 
adding the whole nifi-standard processors
 Key: NIFI-8227
 URL: https://issues.apache.org/jira/browse/NIFI-8227
 Project: Apache NiFi
  Issue Type: Wish
Reporter: Jorge Machado


Hi, 

 

I'm trying to extend AbstractDatabaseFetchProcessor, which ends up to be 
difficult. The only way is to pull the standard processors as dependency and 
this will duplicate all processors on the canvas. We should refactor so that 
AbstractDatabaseFetchProcessor is moved to a place where can be extended 
properlly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi] timeabarna commented on a change in pull request #4815: NIFI-8200: Modifying PutAzureDataLakeStorage to delete temp file if e…

[GitBox]

timeabarna commented on a change in pull request #4815:
URL: https://github.com/apache/nifi/pull/4815#discussion_r576091826



##
File path: 
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/PutAzureDataLakeStorage.java
##
@@ -126,6 +126,9 @@ public void onTrigger(final ProcessContext context, final 
ProcessSession session
 if (length > 0) {
 try (final InputStream rawIn = session.read(flowFile); 
final BufferedInputStream bufferedIn = new BufferedInputStream(rawIn)) {
 uploadContent(fileClient, bufferedIn, length);
+} catch (Exception e) {
+fileClient.delete();
+throw e;

Review comment:
   Thanks for your help, corrected in the next commit





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (MINIFICPP-1490) Add ARCH support to bootstrap process.

[Martin Zink (Jira)]

Martin Zink created MINIFICPP-1490:
--

 Summary: Add ARCH support to bootstrap process.
 Key: MINIFICPP-1490
 URL: https://issues.apache.org/jira/browse/MINIFICPP-1490
 Project: Apache NiFi MiNiFi C++
  Issue Type: Improvement
Reporter: Martin Zink
Assignee: Martin Zink


Bootstrap process currently doesn't support Arch (or its derivatives), which 
makes building the project on them cumbersome.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (MINIFICPP-1473) E2P-2. Migrate hand-written class loader to Poco::ClassLoader

[Ferenc Gerlits (Jira)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-1473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ferenc Gerlits updated MINIFICPP-1473:
--
Description: 
Our current class loading infrastructure is quite confusing, and relies on 
static self-registry of classes.

Poco::ClassLoader looks like a robust 3rd-party implementation of the same 
thing, which
 * would be easier to adapt to the shared-library-plugin world;
 * and already supports both Windows and Linux/Mac, so we can hide OS-specific 
differences (which are handled by our hand-crafted code, but it's not clear how 
well).

As part of this, we should get rid of the static self-registry, which won't 
work with plugins.

TODO: may need to be broken down further
 * bring in the Poco library: Foundation only, strip out everything else
 * rewrite the classloader to use Poco::ClassLoader
 * remove static self-registry and load the classes explicitly when needed
 * change the STATIC extension libraries to SHARED, and --whole-archive should 
no longer be necessary
 * the main executable and the integration tests should not link with any of 
the extension libraries, and instead load them dynamically using (our wrapper 
of) Poco::ClassLoader
 * unit tests should link with the extensions, so they have access to all their 
public methods, and can test the utility classes contained in the extension

 

TODO: I expect complications; add them here when we find them.

 

{{AgentDocs}} relies on static self-registration of extensions, so if we remove 
that, we'll need to find a different way of generating the documentation.

  was:
Our current class loading infrastructure is quite confusing, and relies on 
static self-registry of classes.

Poco::ClassLoader looks like a robust 3rd-party implementation of the same 
thing, which
 * would be easier to adapt to the shared-library-plugin world;
 * and already supports both Windows and Linux/Mac, so we can hide OS-specific 
differences (which are handled by our hand-crafted code, but it's not clear how 
well).

As part of this, we should get rid of the static self-registry, which won't 
work with plugins.

TODO: may need to be broken down further
 * bring in the Poco library: Foundation only, strip out everything else
 * rewrite the classloader to use Poco::ClassLoader
 * remove static self-registry and load the classes explicitly when needed
 * change the STATIC extension libraries to SHARED, and --whole-archive should 
no longer be necessary
 * the main executable and the integration tests should not link with any of 
the extension libraries, and instead load them dynamically using (our wrapper 
of) Poco::ClassLoader
 * unit tests should link with the extensions, so they have access to all their 
public methods, and can test the utility classes contained in the extension

 

TODO: I expect complications; add them here when we find them.


> E2P-2. Migrate hand-written class loader to Poco::ClassLoader
> -
>
> Key: MINIFICPP-1473
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1473
> Project: Apache NiFi MiNiFi C++
>  Issue Type: New Feature
>Reporter: Ferenc Gerlits
>Priority: Minor
>
> Our current class loading infrastructure is quite confusing, and relies on 
> static self-registry of classes.
> Poco::ClassLoader looks like a robust 3rd-party implementation of the same 
> thing, which
>  * would be easier to adapt to the shared-library-plugin world;
>  * and already supports both Windows and Linux/Mac, so we can hide 
> OS-specific differences (which are handled by our hand-crafted code, but it's 
> not clear how well).
> As part of this, we should get rid of the static self-registry, which won't 
> work with plugins.
> TODO: may need to be broken down further
>  * bring in the Poco library: Foundation only, strip out everything else
>  * rewrite the classloader to use Poco::ClassLoader
>  * remove static self-registry and load the classes explicitly when needed
>  * change the STATIC extension libraries to SHARED, and --whole-archive 
> should no longer be necessary
>  * the main executable and the integration tests should not link with any of 
> the extension libraries, and instead load them dynamically using (our wrapper 
> of) Poco::ClassLoader
>  * unit tests should link with the extensions, so they have access to all 
> their public methods, and can test the utility classes contained in the 
> extension
>  
> TODO: I expect complications; add them here when we find them.
>  
> {{AgentDocs}} relies on static self-registration of extensions, so if we 
> remove that, we'll need to find a different way of generating the 
> documentation.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (MINIFICPP-1489) Introduce caching to Mac and Windows CI builds

[Gabor Gyimesi (Jira)]

Gabor Gyimesi created MINIFICPP-1489:


 Summary: Introduce caching to Mac and Windows CI builds
 Key: MINIFICPP-1489
 URL: https://issues.apache.org/jira/browse/MINIFICPP-1489
 Project: Apache NiFi MiNiFi C++
  Issue Type: Improvement
Reporter: Gabor Gyimesi
Assignee: Gabor Gyimesi






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (MINIFICPP-1287) Make test configuration files build artifacts

[Adam Hunyadi (Jira)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-1287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Hunyadi resolved MINIFICPP-1287.
-
Resolution: Won't Fix

> Make test configuration files build artifacts
> -
>
> Key: MINIFICPP-1287
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1287
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Improvement
>Affects Versions: 1.0.0
>Reporter: Adam Hunyadi
>Priority: Minor
>  Labels: MiNiFi-CPP-Hygiene
> Fix For: 0.7.0
>
>
> *Background:*
> Currently built tests refer back to configuration files placed in the source 
> directory of the project. These configurations are passed by command line 
> arguments.
> This causes multiple problems: combining integration tests with functionality 
> provided by the catch2 testing framework is difficult, and finding failing 
> tests by their name assumes understanding on where the test is defined in the 
> cmake file.
> *Proposal:*
> Ideally, we should copy configuration files required for testing to the build 
> directory as build artifacts, so that they could be relatively referenced by 
> test files. This would make the configuration <-> test definition 
> associations.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (MINIFICPP-1300) FlowController circularly referenced through ScheduleAgents

[Adam Hunyadi (Jira)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-1300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Hunyadi resolved MINIFICPP-1300.
-
Resolution: Fixed

> FlowController circularly referenced through ScheduleAgents
> ---
>
> Key: MINIFICPP-1300
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1300
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: Adam Debreceni
>Assignee: Adam Hunyadi
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> FlowController holds shared_ptr-s to the different schedulers, which in turn 
> hold shared_ptr-s to the FlowController instance, keeping it alive forever.
>  
> ScheduleAgents should hold either a gsl::not_null or at most 
> a weak_ptr.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (MINIFICPP-1389) Upgrade librdkafka version

[Adam Hunyadi (Jira)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-1389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Hunyadi resolved MINIFICPP-1389.
-
Resolution: Fixed

> Upgrade librdkafka version
> --
>
> Key: MINIFICPP-1389
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1389
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Task
>Affects Versions: 0.9.0
>Reporter: Adam Hunyadi
>Assignee: Adam Hunyadi
>Priority: Minor
> Fix For: 1.0.0
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> *Background:*
> The current version of librdkafka we use has cherry-picked patches applied to 
> it and also does not support transactions required for the implementation of 
> ConsumeKafka processor.
> *Proposal:*
> We should update the librdkafka version to 1.5.0



--
This message was sent by Atlassian Jira
(v8.3.4#803005)