[GitHub] [nifi] natural commented on a change in pull request #4164: NIFI-5481 - Add new providers of protected sensitive configuration values

[GitBox]

natural commented on a change in pull request #4164: NIFI-5481 - Add new 
providers of protected sensitive configuration values
URL: https://github.com/apache/nifi/pull/4164#discussion_r409985580
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/ProtectedNiFiProperties.java
 ##
 @@ -359,40 +373,22 @@ public NiFiProperties getUnprotectedProperties() throws 
SensitivePropertyProtect
 }
 }
 
-NiFiProperties unprotected = new 
StandardNiFiProperties(rawProperties);
-
-return unprotected;
+return new StandardNiFiProperties(rawProperties);
 } else {
 logger.debug("No protected properties");
 return getInternalNiFiProperties();
 }
 }
 
-/**
- * Registers a new {@link SensitivePropertyProvider}. This method will 
throw a {@link UnsupportedOperationException} if a provider is already 
registered for the protection scheme.
- *
- * @param sensitivePropertyProvider the provider
- */
-void addSensitivePropertyProvider(SensitivePropertyProvider 
sensitivePropertyProvider) {
-if (sensitivePropertyProvider == null) {
-throw new IllegalArgumentException("Cannot add null 
SensitivePropertyProvider");
-}
-
-if 
(getSensitivePropertyProviders().containsKey(sensitivePropertyProvider.getIdentifierKey()))
 {
-throw new UnsupportedOperationException("Cannot overwrite existing 
sensitive property provider registered for " + 
sensitivePropertyProvider.getIdentifierKey());
-}
-
-
getSensitivePropertyProviders().put(sensitivePropertyProvider.getIdentifierKey(),
 sensitivePropertyProvider);
-}
-
-private String getDefaultProtectionScheme() {
-if (!getSensitivePropertyProviders().isEmpty()) {
-List schemes = new 
ArrayList<>(getSensitivePropertyProviders().keySet());
-Collections.sort(schemes);
-return schemes.get(0);
-} else {
-throw new IllegalStateException("No registered protection 
schemes");
-}
+@Override
+public String toString() {
 
 Review comment:
   Super helpful in logs.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] natural commented on a change in pull request #4164: NIFI-5481 - Add new providers of protected sensitive configuration values

[GitBox]

natural commented on a change in pull request #4164: NIFI-5481 - Add new 
providers of protected sensitive configuration values
URL: https://github.com/apache/nifi/pull/4164#discussion_r409984829
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/property/provider/keystore/KeyStoreWrappedSensitivePropertyProvider.java
 ##
 @@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties.sensitive.property.provider.keystore;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.properties.sensitive.ExternalProperties;
+import 
org.apache.nifi.properties.sensitive.property.provider.aes.AESSensitivePropertyProvider;
+import 
org.apache.nifi.properties.sensitive.SensitivePropertyConfigurationException;
+import 
org.apache.nifi.properties.sensitive.SensitivePropertyProtectionException;
+import org.apache.nifi.properties.sensitive.SensitivePropertyProvider;
+import org.apache.nifi.properties.sensitive.StandardExternalPropertyLookup;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.security.Key;
+import java.security.KeyStore;
+import java.security.KeyStoreException;
+import java.security.NoSuchAlgorithmException;
+import java.security.UnrecoverableKeyException;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Sensitive properties using KeyStore keys with an inner AES SPP.
+ */
+public class KeyStoreWrappedSensitivePropertyProvider implements 
SensitivePropertyProvider {
+private static final Logger logger = 
LoggerFactory.getLogger(KeyStoreWrappedSensitivePropertyProvider.class);
+
+private static final String PROVIDER_NAME = "KeyStore Sensitive Property 
Provider";
+private static final String MATERIAL_PREFIX = "keystore";
+private static final String MATERIAL_SEPARATOR = "/";
+
+public static final String KEYSTORE_TYPE_JCECKS = "jceks";
+private static final String KEYSTORE_TYPE_PKCS12 = "pkcs12";
+private static final String KEYSTORE_TYPE_BKS = "bks";
+
+static final Set KEYSTORE_TYPES = new HashSet<>(Arrays.asList(
+KEYSTORE_TYPE_JCECKS,
+KEYSTORE_TYPE_PKCS12,
+KEYSTORE_TYPE_BKS));
+
+private final ExternalProperties externalProperties;
+private final SensitivePropertyProvider wrappedSensitivePropertyProvider;
+private final String storeType;
+private final String keyAlias;
+
+/**
+ * Constructor, as expected by the standard sensitive property provider 
implementation.
+ *
+ * @param keyId string in the form "keystore/jcecks/user-key-alias"
+ */
+public KeyStoreWrappedSensitivePropertyProvider(String keyId)  {
+this(keyId, null, null);
+}
+
+public KeyStoreWrappedSensitivePropertyProvider(String keyId, 
KeyStoreProvider keyStoreProvider, ExternalProperties externalProperties)  {
+if (externalProperties == null) {
+externalProperties = new StandardExternalPropertyLookup(null, 
getKeyStorePropertiesMapping() );
+}
+this.externalProperties = externalProperties;
+
+if (StringUtils.isBlank(keyId))
+throw new SensitivePropertyConfigurationException("The key cannot 
be empty");
+
+String storeType;
+String keyAlias;
+try {
+String[] parts = keyId.split(MATERIAL_SEPARATOR);
+storeType = parts.length > 0 ? parts[1] : "";
+keyAlias = parts.length > 1 ? parts[2] : "";
+} catch (final ArrayIndexOutOfBoundsException e) {
+throw new SensitivePropertyConfigurationException("Invalid Key 
Store key", e);
+}
+
+this.storeType = storeType;
+this.keyAlias = keyAlias;
+
+if (keyStoreProvider == null ){
+keyStoreProvider = new StandardKeyStoreProvider(getStoreUri(), 
this.storeType, getStorePassword());
+}
+
+try {
+KeyStore store = keyStoreProvider.getKeyStore();
+Key secretKey = store.getKey(keyAlias, 
getKeyPassword().toCharAr

[GitHub] [nifi] natural commented on a change in pull request #4164: NIFI-5481 - Add new providers of protected sensitive configuration values

[GitBox]

natural commented on a change in pull request #4164: NIFI-5481 - Add new 
providers of protected sensitive configuration values
URL: https://github.com/apache/nifi/pull/4164#discussion_r409985312
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/main/java/org/apache/nifi/properties/sensitive/property/provider/keystore/KeyStoreWrappedSensitivePropertyProvider.java
 ##
 @@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.properties.sensitive.property.provider.keystore;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.properties.sensitive.ExternalProperties;
+import 
org.apache.nifi.properties.sensitive.property.provider.aes.AESSensitivePropertyProvider;
+import 
org.apache.nifi.properties.sensitive.SensitivePropertyConfigurationException;
+import 
org.apache.nifi.properties.sensitive.SensitivePropertyProtectionException;
+import org.apache.nifi.properties.sensitive.SensitivePropertyProvider;
+import org.apache.nifi.properties.sensitive.StandardExternalPropertyLookup;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.security.Key;
+import java.security.KeyStore;
+import java.security.KeyStoreException;
+import java.security.NoSuchAlgorithmException;
+import java.security.UnrecoverableKeyException;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Sensitive properties using KeyStore keys with an inner AES SPP.
+ */
+public class KeyStoreWrappedSensitivePropertyProvider implements 
SensitivePropertyProvider {
+private static final Logger logger = 
LoggerFactory.getLogger(KeyStoreWrappedSensitivePropertyProvider.class);
+
+private static final String PROVIDER_NAME = "KeyStore Sensitive Property 
Provider";
+private static final String MATERIAL_PREFIX = "keystore";
+private static final String MATERIAL_SEPARATOR = "/";
+
+public static final String KEYSTORE_TYPE_JCECKS = "jceks";
+private static final String KEYSTORE_TYPE_PKCS12 = "pkcs12";
+private static final String KEYSTORE_TYPE_BKS = "bks";
+
+static final Set KEYSTORE_TYPES = new HashSet<>(Arrays.asList(
+KEYSTORE_TYPE_JCECKS,
+KEYSTORE_TYPE_PKCS12,
+KEYSTORE_TYPE_BKS));
+
+private final ExternalProperties externalProperties;
+private final SensitivePropertyProvider wrappedSensitivePropertyProvider;
+private final String storeType;
+private final String keyAlias;
+
+/**
+ * Constructor, as expected by the standard sensitive property provider 
implementation.
+ *
+ * @param keyId string in the form "keystore/jcecks/user-key-alias"
+ */
+public KeyStoreWrappedSensitivePropertyProvider(String keyId)  {
+this(keyId, null, null);
+}
+
+public KeyStoreWrappedSensitivePropertyProvider(String keyId, 
KeyStoreProvider keyStoreProvider, ExternalProperties externalProperties)  {
+if (externalProperties == null) {
+externalProperties = new StandardExternalPropertyLookup(null, 
getKeyStorePropertiesMapping() );
+}
+this.externalProperties = externalProperties;
+
+if (StringUtils.isBlank(keyId))
+throw new SensitivePropertyConfigurationException("The key cannot 
be empty");
+
+String storeType;
+String keyAlias;
+try {
+String[] parts = keyId.split(MATERIAL_SEPARATOR);
+storeType = parts.length > 0 ? parts[1] : "";
+keyAlias = parts.length > 1 ? parts[2] : "";
+} catch (final ArrayIndexOutOfBoundsException e) {
+throw new SensitivePropertyConfigurationException("Invalid Key 
Store key", e);
+}
+
+this.storeType = storeType;
+this.keyAlias = keyAlias;
+
+if (keyStoreProvider == null ){
+keyStoreProvider = new StandardKeyStoreProvider(getStoreUri(), 
this.storeType, getStorePassword());
+}
+
+try {
+KeyStore store = keyStoreProvider.getKeyStore();
+Key secretKey = store.getKey(keyAlias, 
getKeyPassword().toCharAr

[GitHub] [nifi] natural removed a comment on issue #4164: NIFI-5481 - Add new providers of protected sensitive configuration values

[GitBox]

natural removed a comment on issue #4164: NIFI-5481 - Add new providers of 
protected sensitive configuration values
URL: https://github.com/apache/nifi/pull/4164#issuecomment-615032288
 
 
   > Hi @thenatog,
   > I am planning to add an Azure Key Vault (AKV) secret provider to Nifi. Can 
you please give me some idea on how it will fit with this change? If you are 
available in nifi slack channel, we can also discuss it there.
   
   Hi @bahlulh 
   There's an Azure provider already written as part of this effort, it's just 
not in the PR.  We're trying to split the changes into this PR (refactoring) 
and follow-on PRs for the various third-party providers.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] natural commented on issue #4164: NIFI-5481 - Add new providers of protected sensitive configuration values

[GitBox]

natural commented on issue #4164: NIFI-5481 - Add new providers of protected 
sensitive configuration values
URL: https://github.com/apache/nifi/pull/4164#issuecomment-615032288
 
 
   > Hi @thenatog,
   > I am planning to add an Azure Key Vault (AKV) secret provider to Nifi. Can 
you please give me some idea on how it will fit with this change? If you are 
available in nifi slack channel, we can also discuss it there.
   
   Hi @bahlulh 
   There's an Azure provider already written as part of this effort, it's just 
not in the PR.  We're trying to split the changes into this PR (refactoring) 
and follow-on PRs for the various third-party providers.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] natural closed pull request #3441: NIFI-5556 Adds `NIFI_DEBUG` env var to admin scripts.

[GitBox]

natural closed pull request #3441: NIFI-5556 Adds `NIFI_DEBUG` env var to admin 
scripts.
URL: https://github.com/apache/nifi/pull/3441
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] natural closed pull request #3861: NIFI-6825 Adds Hashicorp Vault Controller Service

[GitBox]

natural closed pull request #3861: NIFI-6825 Adds Hashicorp Vault Controller 
Service
URL: https://github.com/apache/nifi/pull/3861
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] natural closed pull request #3594: NIFI-3833 Support for Encrypted Flow File Repositories

[GitBox]

natural closed pull request #3594: NIFI-3833 Support for Encrypted Flow File 
Repositories
URL: https://github.com/apache/nifi/pull/3594
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] natural opened a new pull request #4216: NIFI-7356 Enable TLS for embedded Zookeeper when NiFi has TLS enabled

[GitBox]

natural opened a new pull request #4216: NIFI-7356 Enable TLS for embedded 
Zookeeper when NiFi has TLS enabled
URL: https://github.com/apache/nifi/pull/4216
 
 
    Config TLS for Embedded ZooKeeper when NiFi TLS Enabled
   
   _Enables automatic ZooKeeper TLS configuration; implements behaviors 
outlined in NIFI-7356 and NIFI-7203._
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [X] Is there a JIRA ticket associated with this PR? Is it referenced 
in the commit message?
   
   - [X] Does your PR title start with **NIFI-** where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   
   - [X] Has your PR been rebased against the latest commit within the target 
branch (typically `master`)?
   
   - [X] Is your initial contribution a single, squashed commit? _Additional 
commits in response to PR reviewer feedback should be made on this branch and 
pushed to allow change tracking. Do not `squash` or use `--force` when pushing 
to allow for clean monitoring of changes._
   
   ### For code changes:
   - [X] Have you ensured that the full suite of tests is executed via `mvn 
-Pcontrib-check clean install` at the root `nifi` folder?
   - [X] Have you written or updated unit tests to verify your changes?
   - [ ] Have you verified that the full build is successful on both JDK 8 and 
JDK 11?
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
   - [ ] If applicable, have you updated the `LICENSE` file, including the main 
`LICENSE` file under `nifi-assembly`?
   - [ ] If applicable, have you updated the `NOTICE` file, including the main 
`NOTICE` file found under `nifi-assembly`?
   - [ ] If adding new Properties, have you added `.displayName` in addition to 
.name (programmatic access) for each of the new properties?
   
   ### For documentation related changes:
   - [X] Have you ensured that format looks appropriate for the output in which 
it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] msharee9 commented on a change in pull request #759: MINIFICPP-1193 - heartbeat interval config doesn't work properly

[GitBox]

msharee9 commented on a change in pull request #759: MINIFICPP-1193 - heartbeat 
interval config doesn't work properly
URL: https://github.com/apache/nifi-minifi-cpp/pull/759#discussion_r409913141
 
 

 ##
 File path: libminifi/src/c2/C2Agent.cpp
 ##
 @@ -90,22 +87,16 @@ C2Agent::C2Agent(const 
std::shared_ptr heart_beat_period_ ) {
-last_run_ = now;
-try {
-  performHeartBeat();
-}
-catch(const std::exception &e) {
-  logger_->log_error("Exception occurred while performing heartbeat. 
error: %s", e.what());
-}
-catch(...) {
-  logger_->log_error("Unknonwn exception occurred while performing 
heartbeat.");
-}
+  try {
+performHeartBeat();
+  } catch(const std::exception &e) {
+logger_->log_error("Exception occurred while performing heartbeat. 
error: %s", e.what());
+  } catch(...) {
+logger_->log_error("Unknonwn exception occurred while performing 
heartbeat.");
   }
 
   checkTriggers();
 
-  std::this_thread::sleep_for(std::chrono::milliseconds(heart_beat_period_ 
> 500 ? 500 : heart_beat_period_));
 
 Review comment:
   Do we honor the heartbeat period here? I mean doesn't the threadpool 
schedule this task immediately or at a different delay interval than the 
heartbeat period.
   Also, I believe this PR will conflict with MINFICPP-1169. In that PR we are 
using the new theadpool changes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] msharee9 commented on a change in pull request #759: MINIFICPP-1193 - heartbeat interval config doesn't work properly

[GitBox]

msharee9 commented on a change in pull request #759: MINIFICPP-1193 - heartbeat 
interval config doesn't work properly
URL: https://github.com/apache/nifi-minifi-cpp/pull/759#discussion_r409913141
 
 

 ##
 File path: libminifi/src/c2/C2Agent.cpp
 ##
 @@ -90,22 +87,16 @@ C2Agent::C2Agent(const 
std::shared_ptr heart_beat_period_ ) {
-last_run_ = now;
-try {
-  performHeartBeat();
-}
-catch(const std::exception &e) {
-  logger_->log_error("Exception occurred while performing heartbeat. 
error: %s", e.what());
-}
-catch(...) {
-  logger_->log_error("Unknonwn exception occurred while performing 
heartbeat.");
-}
+  try {
+performHeartBeat();
+  } catch(const std::exception &e) {
+logger_->log_error("Exception occurred while performing heartbeat. 
error: %s", e.what());
+  } catch(...) {
+logger_->log_error("Unknonwn exception occurred while performing 
heartbeat.");
   }
 
   checkTriggers();
 
-  std::this_thread::sleep_for(std::chrono::milliseconds(heart_beat_period_ 
> 500 ? 500 : heart_beat_period_));
 
 Review comment:
   Do we honor the heartbeat period here? I mean doesn't the threadpool 
schedule this task immediately or at a different delay interval that the 
heartbeat period.
   Also, I believe this PR will conflict with MINFICPP-1169. In that PR we are 
using the new theadpool changes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] msharee9 opened a new pull request #761: Minificpp 1008 - Chunkio integration into nanofi

[GitBox]

msharee9 opened a new pull request #761: Minificpp 1008 - Chunkio integration 
into nanofi
URL: https://github.com/apache/nifi-minifi-cpp/pull/761
 
 
   Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
   
   - [ ] Does your PR title start with MINIFICPP- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   
   - [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?
   
   - [ ] Is your initial contribution a single, squashed commit?
   
   ### For code changes:
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the LICENSE file?
   - [ ] If applicable, have you updated the NOTICE file?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
   
   Broken down MINIFICPP-1007 into ECU core and C2 separately. This PR has ECU 
core changes with chunkio and libyaml integration and deleted the prior in 
memory buffer implementation.
   Some more improvements has to be made:
   1. Task execution improvements in threadpool (not major).
   2. Configurable flush interval and coordinate output flush tasks with input. 
(This will improve overall cpu and memory usage performance) and will scale to 
number of ecus.
   3. Improve tailfile processing itself to handle regex based multiple files 
tailing


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (NIFI-7371) Improve error handling of S3 processors

[endzeit (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-7371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

endzeit updated NIFI-7371:
--
Description: 
Currently the S3 processors, such as FetchS3Object, only expose the 
relationsships "success" and "failure". However the are a multitute of reasons 
why an interaction with an S3 storage might fail.

As of now there is no easy way of knowing why a flow file was directed to the 
"failure" relationsship just by the flow file itself.

A way of finding out the reason might be to search for a corresponing log / 
bulletin entry.
 This seems rather complicated.

A use case where having this information is useful could be when deciding 
whether
 * the action should be retried, e.g. on a timeout,
 * or the failure be handled, e.g. when the object for the specified key does 
not exists.

I haven't looked much into the underlying AWS library yet as I wanted to 
discuss whether such an improvement is desired at all first?
 If so, should the information be exposed via
 * additional / other relationsships, such as in the FetchSFTP processor,
 * or rather an attribute added to the flow file, such as in the ValidateXml 
processor?

This might also apply to other AWS processors but we just have come across the 
S3 processors as we use them quite regulary.

Any thoughts or comments would be highly appreciated!

  was:
Currently the S3 processors, such as FetchS3Object, only expose the 
relationsships "success" and "failure". However the are a multitute of reasons 
why an interaction with an S3 storage might fail.

As of now there is no easy way of knowing why a flow file was directed to the 
"failure" relationsship just by the flow file itself.

A way of finding out the reason might be to search for a corresponing log / 
bulletin entry.
This seems rather complicated.

A use case where having this information is useful could be when deciding 
whether 
 * the action should be retried, e.g. on a timeout,
 * or the failure be handled, e.g. when the object for the specified key does 
not exists.

I haven't looked much into the underlying AWS library yet as I wanted to 
discuss whether such an improvement is desired at all first?
If so, should the information be exposed via 
 * additional / other relationsships, such as in the FetchSFTP processor, 
 * or rather an attribute added to the flow file, such as in the ValidateXml 
processor?

This might also apply to other AWS processors but we jut have come across the 
S3 processors as we use them quite regulary.

Any thoughts or comments would be highly appreciated!


> Improve error handling of S3 processors
> ---
>
> Key: NIFI-7371
> URL: https://issues.apache.org/jira/browse/NIFI-7371
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.11.4
>Reporter: endzeit
>Priority: Major
>
> Currently the S3 processors, such as FetchS3Object, only expose the 
> relationsships "success" and "failure". However the are a multitute of 
> reasons why an interaction with an S3 storage might fail.
> As of now there is no easy way of knowing why a flow file was directed to the 
> "failure" relationsship just by the flow file itself.
> A way of finding out the reason might be to search for a corresponing log / 
> bulletin entry.
>  This seems rather complicated.
> A use case where having this information is useful could be when deciding 
> whether
>  * the action should be retried, e.g. on a timeout,
>  * or the failure be handled, e.g. when the object for the specified key does 
> not exists.
> I haven't looked much into the underlying AWS library yet as I wanted to 
> discuss whether such an improvement is desired at all first?
>  If so, should the information be exposed via
>  * additional / other relationsships, such as in the FetchSFTP processor,
>  * or rather an attribute added to the flow file, such as in the ValidateXml 
> processor?
> This might also apply to other AWS processors but we just have come across 
> the S3 processors as we use them quite regulary.
> Any thoughts or comments would be highly appreciated!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (NIFI-7371) Improve error handling of S3 processors

[endzeit (Jira)]

endzeit created NIFI-7371:
-

 Summary: Improve error handling of S3 processors
 Key: NIFI-7371
 URL: https://issues.apache.org/jira/browse/NIFI-7371
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Extensions
Affects Versions: 1.11.4
Reporter: endzeit


Currently the S3 processors, such as FetchS3Object, only expose the 
relationsships "success" and "failure". However the are a multitute of reasons 
why an interaction with an S3 storage might fail.

As of now there is no easy way of knowing why a flow file was directed to the 
"failure" relationsship just by the flow file itself.

A way of finding out the reason might be to search for a corresponing log / 
bulletin entry.
This seems rather complicated.

A use case where having this information is useful could be when deciding 
whether 
 * the action should be retried, e.g. on a timeout,
 * or the failure be handled, e.g. when the object for the specified key does 
not exists.

I haven't looked much into the underlying AWS library yet as I wanted to 
discuss whether such an improvement is desired at all first?
If so, should the information be exposed via 
 * additional / other relationsships, such as in the FetchSFTP processor, 
 * or rather an attribute added to the flow file, such as in the ValidateXml 
processor?

This might also apply to other AWS processors but we jut have come across the 
S3 processors as we use them quite regulary.

Any thoughts or comments would be highly appreciated!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi-minifi-cpp] nghiaxlee commented on issue #697: MINIFICPP-1104 - SSL docker tests for PublishKafka

[GitBox]

nghiaxlee commented on issue #697: MINIFICPP-1104 - SSL docker tests for 
PublishKafka
URL: https://github.com/apache/nifi-minifi-cpp/pull/697#issuecomment-614934518
 
 
   In the master branch, I figured out that, since you use Linux, 
`host.docker.internal` will not work, thus lead to 2/3 kafka tests failing (the 
success one was no_broker of course :) ). That part is a blameworthy 
misunderstanding from my side, and is removed in this PR.
   
   For the current PR, still not sure what is happening, I am wondering what 
cause the `docker.errors.BuildError: failed to get destination image...` error, 
I ran those tests on a virtual Ubuntu machine and they worked fine.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] nghiaxlee commented on a change in pull request #697: MINIFICPP-1104 - SSL docker tests for PublishKafka

[GitBox]

nghiaxlee commented on a change in pull request #697: MINIFICPP-1104 - SSL 
docker tests for PublishKafka
URL: https://github.com/apache/nifi-minifi-cpp/pull/697#discussion_r409887035
 
 

 ##
 File path: docker/test/integration/resources/kafka_broker/Dockerfile
 ##
 @@ -0,0 +1,3 @@
+FROM teivah/kafka:latest
 
 Review comment:
   So previously I chose `spotify/kafka` but it was pretty outdated (3 years 
ago) so I could not try secure connection, the current choice is not much 
better (2 years ago). Those popular images often need docker-compose, and I 
guess at that moment, I just wanted an image which could run both zookeeper and 
kafka. I still want that now, but I would love to hear your suggestion for 
better quality.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] jerem991 opened a new pull request #4215: NIFI-7366 - ConsumeEWS Processor parse EML

[GitBox]

jerem991 opened a new pull request #4215:  NIFI-7366 - ConsumeEWS Processor 
parse EML
URL: https://github.com/apache/nifi/pull/4215
 
 
   Fix ConsumeEWS to support ItemAttachement (EML for example)
   This commit permit to fix the good retrieval of the complete attachment, but 
it requires some tricks to extract properly the ItemAttachment...
   
   Thank you for submitting a contribution to Apache NiFi.
   
   Please provide a short description of the PR here:
   https://issues.apache.org/jira/browse/NIFI-7366
   
    Description of PR
   
   _Enables retrieval of EML attachement functionality; fixes bug NIFI-7366._
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [X] Is there a JIRA ticket associated with this PR? Is it referenced 
in the commit message?
   
   - [X] Does your PR title start with **NIFI-** where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   
   - [X] Has your PR been rebased against the latest commit within the target 
branch (typically `master`)?
   
   - [X] Is your initial contribution a single, squashed commit? _Additional 
commits in response to PR reviewer feedback should be made on this branch and 
pushed to allow change tracking. Do not `squash` or use `--force` when pushing 
to allow for clean monitoring of changes._
   
   ### For code changes:
   - [X] Have you ensured that the full suite of tests is executed via `mvn 
-Pcontrib-check clean install` at the root `nifi` folder?
   - [ ] Have you written or updated unit tests to verify your changes?
   - [X] Have you verified that the full build is successful on both JDK 8 and 
JDK 11?
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
   - [ ] If applicable, have you updated the `LICENSE` file, including the main 
`LICENSE` file under `nifi-assembly`?
   - [ ] If applicable, have you updated the `NOTICE` file, including the main 
`NOTICE` file found under `nifi-assembly`?
   - [ ] If adding new Properties, have you added `.displayName` in addition to 
.name (programmatic access) for each of the new properties?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (NIFI-7366) ConsumeEWS Processor - not able to parse ItemAttachement (EML)

[Jeremie (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-7366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeremie updated NIFI-7366:
--
Priority: Major  (was: Minor)

> ConsumeEWS Processor - not able to parse ItemAttachement (EML)
> --
>
> Key: NIFI-7366
> URL: https://issues.apache.org/jira/browse/NIFI-7366
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.11.4
>Reporter: Jeremie
>Priority: Major
> Attachments: image-2020-04-15-15-37-11-310.png
>
>
> Context: We try to ingest phishing reported by our user, and it's compiled 
> with [EML |[https://fileinfo.com/extension/eml]] as an attachment.
>  
> When the ConsumeEWS try to get an email with an EML attachment, the 
> consumeEWS is not able to parse it to extract the content of the email.
> !image-2020-04-15-15-37-11-310.png!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi] MikeThomsen commented on issue #4204: NIFI-7355 Added Gremlin bytecode client service.

[GitBox]

MikeThomsen commented on issue #4204: NIFI-7355 Added Gremlin bytecode client 
service.
URL: https://github.com/apache/nifi/pull/4204#issuecomment-614912115
 
 
   @mattyb149 changes pushed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] sjyang18 commented on a change in pull request #4200: NIFI-7335: PutAzureCosmosDBDocument to provide the cosmos sql api support for Azure Cosmos DB

[GitBox]

sjyang18 commented on a change in pull request #4200: NIFI-7335: 
PutAzureCosmosDBDocument to provide the cosmos sql api support for Azure Cosmos 
DB
URL: https://github.com/apache/nifi/pull/4200#discussion_r409846155
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/cosmos/document/PutAzureCosmosDBDocument.java
 ##
 @@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.cosmos.document;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SystemResourceConsideration;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SystemResource;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+
+@EventDriven
+@Tags({ "azure", "cosmos", "document", "insert", "update", "write", "put" })
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Writes the contents of a FlowFile to Azure Cosmos DB 
with Core SQL API. For CosmosDB with Mongo API, use PutMongo instead.")
+@SystemResourceConsideration(resource = SystemResource.MEMORY)
+public class PutAzureCosmosDBDocument extends AbstractAzureCosmosDBProcessor {
+
+private final static Set relationships;
+private final static List propertyDescriptors;
+
+private final static ObjectMapper mapper;
+
+static {
+List _propertyDescriptors = new ArrayList<>();
+_propertyDescriptors.addAll(descriptors);
+_propertyDescriptors.add(CHARACTER_SET);
+propertyDescriptors = 
Collections.unmodifiableList(_propertyDescriptors);
+
+final Set _relationships = new HashSet<>();
+_relationships.add(REL_SUCCESS);
+_relationships.add(REL_FAILURE);
+relationships = Collections.unmodifiableSet(_relationships);
+mapper = new ObjectMapper();
+}
+
+@Override
+public Set getRelationships() {
+return relationships;
+}
+
+@Override
+public List getSupportedPropertyDescriptors() {
+return propertyDescriptors;
+}
+
+
+@Override
+public void onTrigger(final ProcessContext context, final ProcessSession 
session) throws ProcessException {
+final FlowFile flowFile = session.get();
+if (flowFile == null) {
+return;
+}
+
+final ComponentLog logger = getLogger();
+
+final Charset charset = 
Charset.forName(context.getProperty(CHARACTER_SET).getValue());
+final String partitionKeyField = 
context.getProperty(PARTITION_KEY).getValue();
+JsonNode doc = null;
+
+try {
+// Read the contents of the FlowFile into a byte array
+final byte[] content = new byte[(int) flowFile.getSize()];
+session.read(flowFile, in -> StreamUtils.fillBuffer(in, content, 
true));
+
+// parse conten into JsonNode object
+doc = mapper.readTree(new String(content, charset));
+
+// make sure type of id is String type if exists
+
+if(doc.has("id")) {
+JsonNode idNode = doc.get("id");
+if(idNode.isNumber()) {
+logger.debug("coverting number id into string...");
+((ObjectNode) doc).put("id", doc.get("id

[GitHub] [nifi] MikeThomsen commented on a change in pull request #4204: NIFI-7355 Added Gremlin bytecode client service.

[GitBox]

MikeThomsen commented on a change in pull request #4204: NIFI-7355 Added 
Gremlin bytecode client service.
URL: https://github.com/apache/nifi/pull/4204#discussion_r409839496
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-graph-bundle/nifi-other-graph-services/src/test/groovy/org/apache/nifi/graph/GremlinBytecodeClientServiceIT.groovy
 ##
 @@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.graph
+
+import org.apache.nifi.processor.AbstractProcessor
+import org.apache.nifi.processor.ProcessContext
+import org.apache.nifi.processor.ProcessSession
+import org.apache.nifi.processor.exception.ProcessException
+import org.apache.nifi.util.TestRunner
+import org.apache.nifi.util.TestRunners
+import org.junit.Before
+import org.junit.Test
+
+class GremlinBytecodeClientServiceIT {
+TestRunner runner
+GraphClientService service
+
+@Before
+void before() {
+runner = TestRunners.newTestRunner(new AbstractProcessor() {
+@Override
+void onTrigger(ProcessContext processContext, ProcessSession 
processSession) throws ProcessException {
+
+}
+})
+
+service = new GremlinBytecodeClientService()
+runner.addControllerService("service", service);
+runner.setProperty(service, 
GremlinBytecodeClientService.CONTACT_POINTS, "localhost")
+runner.setProperty(service, GremlinBytecodeClientService.PORT, "8182")
+runner.setProperty(service, GremlinBytecodeClientService.PATH, 
"/gremlin")
+runner.enableControllerService(service)
+runner.assertValid()
+}
+
+@Test
+void testQuery() {
+def query = """
+assert param
+g.V().hasLabel("nada").count().next()
+"""
+
+service.executeQuery(query, [ param: "Hi!" ], { result, more ->
+assert result
+assert result["result"] == 0
+} as GraphQueryResultCallback)
+}
+
+@Test
+void testMassGenerate() {
+def query = """
+assert param
+1.upto(100) {
+def trav = g.addV("it_test_node")
+2.upto(250) {
+trav.addV("it_test_node").property("uuid", 
UUID.randomUUID().toString())
+}
+trav.next()
+}
+def count = g.V().hasLabel("it_test_node").count().next()
+g.V().hasLabel("it_test_node").drop().iterate()
+count
+""".trim()
+
+service.executeQuery(query, [ param: "Hi!" ], { result, more ->
 
 Review comment:
   I am updating the test and use it in setting a property.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (NIFI-7363) Deleting/Modifying Parameters with Nifi Registry

[Bryan Bende (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-7363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17085230#comment-17085230
 ] 

Bryan Bende commented on NIFI-7363:
---

The way it currently works is the following...

On the NiFi side, parameter contexts are stored outside the flow so that 
multiple process groups can bind to the same context, and also so that access 
policies can be setup for the parameter context, different from what may be 
setup on the flow.

When version control is started on a process group in NiFI, or when a new 
version is committed, a snapshot of all the parameter contexts used by that 
versioned flow is taken, the sensitive values are removed, and those contexts 
are stored with the flow snapshot sent to registry.

When a versioned flow is imported to NiFi, or an existing versioned flow is 
upgraded to a new version, NiFi retrieves the flow snapshot from registry and 
goes through each parameter context that was stored with that snapshot.
 * If NiFi has no context with the same name, then a new context is created 
using the parameters and values that were stored in registry.
 * If a context with the same name already exists, then NiFi updates that 
context, adding any parameters that are in the flow snapshot from registry, but 
not in the context in NiFi, and not touching any existing parameters.

When modifying a parameter context, such as adding or removing a parameter, it 
does not cause the versioned process groups referencing it to become dirty 
(i.e. requiring a commit). This is because nothing has changed in the versioned 
flow yet.
 * If you added a new parameter, you would then have to go into a flow and 
modify a component to use that parameter which then needs to be committed and 
would capture the new state of the context.
 * If you deleted a parameter that was being used in versioned flows, those 
components are now invalid for referencing a parameter that doesn't exist, so 
you would have to remove the references to those parameters, which triggers 
changes that need to be committed which would in turn capture the new state of 
the context.

As a test, lets say we create param context with two parameters, param1 and 
param2. We then make flow that references both of these parameters and start 
version control. The snapshot file in registry contains the context with the 
two params:
{code:java}
"parameterContexts" : {
"Param Context 1" : {
  "name" : "Param Context 1",
  "parameters" : [ {
"description" : "",
"name" : "param1",
"sensitive" : false,
"value" : "val1"
  }, {
"description" : "",
"name" : "param2",
"sensitive" : false,
"value" : "val2"
  } ]
}
  } {code}
Now we go into the context in NiFi and delete param2. At this point nothing 
happens, but now the GenerateFlowFile processor referencing #\{param2} is 
invalid, so we remove that reference, and now the flow is dirty so we commit 
version 2. The version 2 snapshot in registry now contains only param1:
{code:java}
"parameterContexts" : {
"Param Context 1" : {
  "name" : "Param Context 1",
  "parameters" : [ {
"description" : "",
"name" : "param1",
"sensitive" : false,
"value" : "val1"
  } ]
}
  } {code}
Now lets say we blow away everything on the NiFi side, so no param contexts or 
versioned flows.

If we import v1 of the flow, it creates "Param Context 1" with param1 and 
param2. We then upgrade to v2 of the flow, and nothing happens to the param 
context because v2 has only param1 coming from registry, which already exists 
in the context in NiFi. We also don't change the value of param1 even if it was 
different coming from registry, because parameter values are meant to be local 
to the NiFi instance and we can't know if we should override what someone may 
have changed it to locally. In fact, it most likely was changed locally on 
purpose to set it to the correct URL/password/etc for the given environment.

Instead, lets say we started by importing v2 of the flow... it creates "Param 
Context 1" with only param1. We then import another versioned flow using v1 of 
the flow, it sees an existing context with the same name, and sees that param2 
isn't in that context, so it adds param2, and does not touch param1.

So the order of imports/upgrades can impact whether there are leftover 
parameters that are unused. The ordering also impacts the initial value used 
when the parameter is first added to the context in NiFi, but parameters are 
primarily environmental and meant to be a mechanism to run/promote a flow 
through environments and swap in the appropriate configuration, so I would 
think that as soon as the flow is imported, the initial values would be 
reconfigured to whatever is needed fo

[GitHub] [nifi] pvillard31 commented on a change in pull request #4200: NIFI-7335: PutAzureCosmosDBDocument to provide the cosmos sql api support for Azure Cosmos DB

[GitBox]

pvillard31 commented on a change in pull request #4200: NIFI-7335: 
PutAzureCosmosDBDocument to provide the cosmos sql api support for Azure Cosmos 
DB
URL: https://github.com/apache/nifi/pull/4200#discussion_r409810795
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/cosmos/document/PutAzureCosmosDBDocument.java
 ##
 @@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.cosmos.document;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SystemResourceConsideration;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SystemResource;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+
+@EventDriven
+@Tags({ "azure", "cosmos", "document", "insert", "update", "write", "put" })
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Writes the contents of a FlowFile to Azure Cosmos DB 
with Core SQL API. For CosmosDB with Mongo API, use PutMongo instead.")
+@SystemResourceConsideration(resource = SystemResource.MEMORY)
+public class PutAzureCosmosDBDocument extends AbstractAzureCosmosDBProcessor {
+
+private final static Set relationships;
+private final static List propertyDescriptors;
+
+private final static ObjectMapper mapper;
+
+static {
+List _propertyDescriptors = new ArrayList<>();
+_propertyDescriptors.addAll(descriptors);
+_propertyDescriptors.add(CHARACTER_SET);
+propertyDescriptors = 
Collections.unmodifiableList(_propertyDescriptors);
+
+final Set _relationships = new HashSet<>();
+_relationships.add(REL_SUCCESS);
+_relationships.add(REL_FAILURE);
+relationships = Collections.unmodifiableSet(_relationships);
+mapper = new ObjectMapper();
+}
+
+@Override
+public Set getRelationships() {
+return relationships;
+}
+
+@Override
+public List getSupportedPropertyDescriptors() {
+return propertyDescriptors;
+}
+
+
+@Override
+public void onTrigger(final ProcessContext context, final ProcessSession 
session) throws ProcessException {
+final FlowFile flowFile = session.get();
+if (flowFile == null) {
+return;
+}
+
+final ComponentLog logger = getLogger();
+
+final Charset charset = 
Charset.forName(context.getProperty(CHARACTER_SET).getValue());
+final String partitionKeyField = 
context.getProperty(PARTITION_KEY).getValue();
+JsonNode doc = null;
+
+try {
+// Read the contents of the FlowFile into a byte array
+final byte[] content = new byte[(int) flowFile.getSize()];
+session.read(flowFile, in -> StreamUtils.fillBuffer(in, content, 
true));
+
+// parse conten into JsonNode object
+doc = mapper.readTree(new String(content, charset));
+
+// make sure type of id is String type if exists
+
+if(doc.has("id")) {
+JsonNode idNode = doc.get("id");
+if(idNode.isNumber()) {
+logger.debug("coverting number id into string...");
+((ObjectNode) doc).put("id", doc.get("

[GitHub] [nifi] sjyang18 commented on a change in pull request #4200: NIFI-7335: PutAzureCosmosDBDocument to provide the cosmos sql api support for Azure Cosmos DB

[GitBox]

sjyang18 commented on a change in pull request #4200: NIFI-7335: 
PutAzureCosmosDBDocument to provide the cosmos sql api support for Azure Cosmos 
DB
URL: https://github.com/apache/nifi/pull/4200#discussion_r409796414
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/cosmos/document/AbstractAzureCosmosDBProcessor.java
 ##
 @@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.nifi.processors.azure.cosmos.document;
+
+import java.io.ByteArrayInputStream;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+import com.azure.cosmos.ConnectionPolicy;
+import com.azure.cosmos.ConsistencyLevel;
+import com.azure.cosmos.CosmosClient;
+import com.azure.cosmos.CosmosClientBuilder;
+import com.azure.cosmos.CosmosClientException;
+import com.azure.cosmos.CosmosContainer;
+import com.azure.cosmos.CosmosDatabase;
+import com.azure.cosmos.models.CosmosContainerProperties;
+
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.cosmos.AzureCosmosDBConnectionService;
+
+public abstract class AbstractAzureCosmosDBProcessor extends AbstractProcessor 
{
+static final String CONSISTENCY_STRONG = "STRONG";
+static final String CONSISTENCY_BOUNDED_STALENESS= "BOUNDED_STALENESS";
+static final String CONSISTENCY_SESSION = "SESSION";
+static final String CONSISTENCY_CONSISTENT_PREFIX = "CONSISTENT_PREFIX";
+static final String CONSISTENCY_EVENTUAL = "EVENTUAL";
+
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to Azure Cosmos DB are routed 
to this relationship").build();
+
+static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to Azure Cosmos DB are 
routed to this relationship").build();
+
+static final Relationship REL_ORIGINAL = new Relationship.Builder()
+.name("original")
+.description("All input FlowFiles that are part of a successful 
query execution go here.")
+.build();
+
+static final PropertyDescriptor CONNECTION_SERVICE = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-connection-service")
+.displayName("Azure Cosmos DB Connection Service")
+.description("If configured, this property will use the assigned for 
retrieving connection string info.")
+.required(false)
+.identifiesControllerService(AzureCosmosDBConnectionService.class)
+.build();
+
+
+static final PropertyDescriptor URI = new PropertyDescriptor.Builder()
+.name("azure-cosmos-db-uri")
+.displayName("Azure Cosmos DB URI")
+.description("Azure Cosmos DB URI, typically of the form: 
https://{databaseaccount}.documents.azure.com:443/. Note this host URL is for 
Azure Cosmos DB with CORE SQL API")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+static final PropertyDescriptor DB_ACCESS_KEY = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-key")
+.displayName("Azrue Cosmos DB Access Key")
+.description("Azure Cosmos DB Access Key from Azure Portal 
(Settings->Keys)")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+
+static final PropertyDescriptor DATABASE_NAME = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-d

[GitHub] [nifi] sjyang18 commented on a change in pull request #4200: NIFI-7335: PutAzureCosmosDBDocument to provide the cosmos sql api support for Azure Cosmos DB

[GitBox]

sjyang18 commented on a change in pull request #4200: NIFI-7335: 
PutAzureCosmosDBDocument to provide the cosmos sql api support for Azure Cosmos 
DB
URL: https://github.com/apache/nifi/pull/4200#discussion_r409793723
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/cosmos/document/PutAzureCosmosDBDocument.java
 ##
 @@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.cosmos.document;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SystemResourceConsideration;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SystemResource;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+
+@EventDriven
+@Tags({ "azure", "cosmos", "document", "insert", "update", "write", "put" })
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Writes the contents of a FlowFile to Azure Cosmos DB 
with Core SQL API. For CosmosDB with Mongo API, use PutMongo instead.")
+@SystemResourceConsideration(resource = SystemResource.MEMORY)
+public class PutAzureCosmosDBDocument extends AbstractAzureCosmosDBProcessor {
+
+private final static Set relationships;
+private final static List propertyDescriptors;
+
+private final static ObjectMapper mapper;
+
+static {
+List _propertyDescriptors = new ArrayList<>();
+_propertyDescriptors.addAll(descriptors);
+_propertyDescriptors.add(CHARACTER_SET);
+propertyDescriptors = 
Collections.unmodifiableList(_propertyDescriptors);
+
+final Set _relationships = new HashSet<>();
+_relationships.add(REL_SUCCESS);
+_relationships.add(REL_FAILURE);
+relationships = Collections.unmodifiableSet(_relationships);
+mapper = new ObjectMapper();
+}
+
+@Override
+public Set getRelationships() {
+return relationships;
+}
+
+@Override
+public List getSupportedPropertyDescriptors() {
+return propertyDescriptors;
+}
+
+
+@Override
+public void onTrigger(final ProcessContext context, final ProcessSession 
session) throws ProcessException {
+final FlowFile flowFile = session.get();
+if (flowFile == null) {
+return;
+}
+
+final ComponentLog logger = getLogger();
+
+final Charset charset = 
Charset.forName(context.getProperty(CHARACTER_SET).getValue());
+final String partitionKeyField = 
context.getProperty(PARTITION_KEY).getValue();
+JsonNode doc = null;
+
+try {
+// Read the contents of the FlowFile into a byte array
+final byte[] content = new byte[(int) flowFile.getSize()];
+session.read(flowFile, in -> StreamUtils.fillBuffer(in, content, 
true));
+
+// parse conten into JsonNode object
+doc = mapper.readTree(new String(content, charset));
+
+// make sure type of id is String type if exists
+
+if(doc.has("id")) {
+JsonNode idNode = doc.get("id");
+if(idNode.isNumber()) {
+logger.debug("coverting number id into string...");
+((ObjectNode) doc).put("id", doc.get("id

[GitHub] [nifi] sjyang18 commented on a change in pull request #4200: NIFI-7335: PutAzureCosmosDBDocument to provide the cosmos sql api support for Azure Cosmos DB

[GitBox]

sjyang18 commented on a change in pull request #4200: NIFI-7335: 
PutAzureCosmosDBDocument to provide the cosmos sql api support for Azure Cosmos 
DB
URL: https://github.com/apache/nifi/pull/4200#discussion_r409792671
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/cosmos/document/AbstractAzureCosmosDBProcessor.java
 ##
 @@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.nifi.processors.azure.cosmos.document;
+
+import java.io.ByteArrayInputStream;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+import com.azure.cosmos.ConnectionPolicy;
+import com.azure.cosmos.ConsistencyLevel;
+import com.azure.cosmos.CosmosClient;
+import com.azure.cosmos.CosmosClientBuilder;
+import com.azure.cosmos.CosmosClientException;
+import com.azure.cosmos.CosmosContainer;
+import com.azure.cosmos.CosmosDatabase;
+import com.azure.cosmos.models.CosmosContainerProperties;
+
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.cosmos.AzureCosmosDBConnectionService;
+
+public abstract class AbstractAzureCosmosDBProcessor extends AbstractProcessor 
{
+static final String CONSISTENCY_STRONG = "STRONG";
+static final String CONSISTENCY_BOUNDED_STALENESS= "BOUNDED_STALENESS";
+static final String CONSISTENCY_SESSION = "SESSION";
+static final String CONSISTENCY_CONSISTENT_PREFIX = "CONSISTENT_PREFIX";
+static final String CONSISTENCY_EVENTUAL = "EVENTUAL";
+
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to Azure Cosmos DB are routed 
to this relationship").build();
+
+static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to Azure Cosmos DB are 
routed to this relationship").build();
+
+static final Relationship REL_ORIGINAL = new Relationship.Builder()
+.name("original")
+.description("All input FlowFiles that are part of a successful 
query execution go here.")
+.build();
+
+static final PropertyDescriptor CONNECTION_SERVICE = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-connection-service")
+.displayName("Azure Cosmos DB Connection Service")
+.description("If configured, this property will use the assigned for 
retrieving connection string info.")
+.required(false)
+.identifiesControllerService(AzureCosmosDBConnectionService.class)
+.build();
+
+
+static final PropertyDescriptor URI = new PropertyDescriptor.Builder()
+.name("azure-cosmos-db-uri")
+.displayName("Azure Cosmos DB URI")
+.description("Azure Cosmos DB URI, typically of the form: 
https://{databaseaccount}.documents.azure.com:443/. Note this host URL is for 
Azure Cosmos DB with CORE SQL API")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+static final PropertyDescriptor DB_ACCESS_KEY = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-key")
+.displayName("Azrue Cosmos DB Access Key")
+.description("Azure Cosmos DB Access Key from Azure Portal 
(Settings->Keys)")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+
+static final PropertyDescriptor DATABASE_NAME = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-d

[GitHub] [nifi] sjyang18 commented on a change in pull request #4200: NIFI-7335: PutAzureCosmosDBDocument to provide the cosmos sql api support for Azure Cosmos DB

[GitBox]

sjyang18 commented on a change in pull request #4200: NIFI-7335: 
PutAzureCosmosDBDocument to provide the cosmos sql api support for Azure Cosmos 
DB
URL: https://github.com/apache/nifi/pull/4200#discussion_r409792451
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/cosmos/document/AbstractAzureCosmosDBProcessor.java
 ##
 @@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.nifi.processors.azure.cosmos.document;
+
+import java.io.ByteArrayInputStream;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+import com.azure.cosmos.ConnectionPolicy;
+import com.azure.cosmos.ConsistencyLevel;
+import com.azure.cosmos.CosmosClient;
+import com.azure.cosmos.CosmosClientBuilder;
+import com.azure.cosmos.CosmosClientException;
+import com.azure.cosmos.CosmosContainer;
+import com.azure.cosmos.CosmosDatabase;
+import com.azure.cosmos.models.CosmosContainerProperties;
+
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.cosmos.AzureCosmosDBConnectionService;
+
+public abstract class AbstractAzureCosmosDBProcessor extends AbstractProcessor 
{
+static final String CONSISTENCY_STRONG = "STRONG";
+static final String CONSISTENCY_BOUNDED_STALENESS= "BOUNDED_STALENESS";
+static final String CONSISTENCY_SESSION = "SESSION";
+static final String CONSISTENCY_CONSISTENT_PREFIX = "CONSISTENT_PREFIX";
+static final String CONSISTENCY_EVENTUAL = "EVENTUAL";
+
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to Azure Cosmos DB are routed 
to this relationship").build();
+
+static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to Azure Cosmos DB are 
routed to this relationship").build();
+
+static final Relationship REL_ORIGINAL = new Relationship.Builder()
+.name("original")
+.description("All input FlowFiles that are part of a successful 
query execution go here.")
+.build();
+
+static final PropertyDescriptor CONNECTION_SERVICE = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-connection-service")
+.displayName("Azure Cosmos DB Connection Service")
+.description("If configured, this property will use the assigned for 
retrieving connection string info.")
+.required(false)
+.identifiesControllerService(AzureCosmosDBConnectionService.class)
+.build();
+
+
+static final PropertyDescriptor URI = new PropertyDescriptor.Builder()
+.name("azure-cosmos-db-uri")
+.displayName("Azure Cosmos DB URI")
+.description("Azure Cosmos DB URI, typically of the form: 
https://{databaseaccount}.documents.azure.com:443/. Note this host URL is for 
Azure Cosmos DB with CORE SQL API")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+static final PropertyDescriptor DB_ACCESS_KEY = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-key")
+.displayName("Azrue Cosmos DB Access Key")
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache

[GitHub] [nifi] sjyang18 commented on a change in pull request #4200: NIFI-7335: PutAzureCosmosDBDocument to provide the cosmos sql api support for Azure Cosmos DB

[GitBox]

sjyang18 commented on a change in pull request #4200: NIFI-7335: 
PutAzureCosmosDBDocument to provide the cosmos sql api support for Azure Cosmos 
DB
URL: https://github.com/apache/nifi/pull/4200#discussion_r409789767
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/cosmos/document/AbstractAzureCosmosDBProcessor.java
 ##
 @@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.nifi.processors.azure.cosmos.document;
+
+import java.io.ByteArrayInputStream;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+import com.azure.cosmos.ConnectionPolicy;
+import com.azure.cosmos.ConsistencyLevel;
+import com.azure.cosmos.CosmosClient;
+import com.azure.cosmos.CosmosClientBuilder;
+import com.azure.cosmos.CosmosClientException;
+import com.azure.cosmos.CosmosContainer;
+import com.azure.cosmos.CosmosDatabase;
+import com.azure.cosmos.models.CosmosContainerProperties;
+
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.cosmos.AzureCosmosDBConnectionService;
+
+public abstract class AbstractAzureCosmosDBProcessor extends AbstractProcessor 
{
+static final String CONSISTENCY_STRONG = "STRONG";
+static final String CONSISTENCY_BOUNDED_STALENESS= "BOUNDED_STALENESS";
+static final String CONSISTENCY_SESSION = "SESSION";
+static final String CONSISTENCY_CONSISTENT_PREFIX = "CONSISTENT_PREFIX";
+static final String CONSISTENCY_EVENTUAL = "EVENTUAL";
+
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to Azure Cosmos DB are routed 
to this relationship").build();
+
+static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to Azure Cosmos DB are 
routed to this relationship").build();
+
+static final Relationship REL_ORIGINAL = new Relationship.Builder()
+.name("original")
+.description("All input FlowFiles that are part of a successful 
query execution go here.")
+.build();
+
+static final PropertyDescriptor CONNECTION_SERVICE = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-connection-service")
+.displayName("Azure Cosmos DB Connection Service")
+.description("If configured, this property will use the assigned for 
retrieving connection string info.")
+.required(false)
+.identifiesControllerService(AzureCosmosDBConnectionService.class)
+.build();
+
+
+static final PropertyDescriptor URI = new PropertyDescriptor.Builder()
+.name("azure-cosmos-db-uri")
+.displayName("Azure Cosmos DB URI")
+.description("Azure Cosmos DB URI, typically of the form: 
https://{databaseaccount}.documents.azure.com:443/. Note this host URL is for 
Azure Cosmos DB with CORE SQL API")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+static final PropertyDescriptor DB_ACCESS_KEY = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-key")
+.displayName("Azrue Cosmos DB Access Key")
+.description("Azure Cosmos DB Access Key from Azure Portal 
(Settings->Keys)")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+
+static final PropertyDescriptor DATABASE_NAME = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-d

[jira] [Updated] (MINIFICPP-1195) MiNiFiCPP-DOCS: Updates README Outdated Bootstrapping Info

[James Medel (Jira)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-1195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Medel updated MINIFICPP-1195:
---
Description: 
The paragraph in the Getting Started: Bootstrapping section originally told the 
user to enter N to continue the build process, but with the updated menu guided 
bootstrap process displayed in the console, P is what one must enter to 
continue the build process. I updated that info.

The previous menu guided bootstrap process was outdated and did not show the 
new features Bustache Support to SQL Support and the Build Options. So, I 
updated it to show that information.

 

Here is my pull request with the updates: 
[https://github.com/apache/nifi-minifi-cpp/pull/760] 

Here is the doc that I am proposing updates for: 
[https://github.com/apache/nifi-minifi-cpp/blob/master/README.md]

  was:
The paragraph in the Getting Started: Bootstrapping section originally told the 
user to enter N to continue the build process, but with the updated menu guided 
bootstrap process displayed in the console, P is what one must enter to 
continue the build process. I updated that info.

The previous menu guided bootstrap process was outdated and did not show the 
new features Bustache Support to SQL Support and the Build Options. So, I 
updated it to show that information.

 

Here is the doc that I am proposing updates for: 
[https://github.com/apache/nifi-minifi-cpp/blob/master/README.md]


> MiNiFiCPP-DOCS: Updates README Outdated Bootstrapping Info
> --
>
> Key: MINIFICPP-1195
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1195
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Documentation
>Affects Versions: 0.7.0
> Environment: Ubuntu 18.04 LTS on AWS EC2
>Reporter: James Medel
>Priority: Major
> Fix For: 0.7.0
>
>
> The paragraph in the Getting Started: Bootstrapping section originally told 
> the user to enter N to continue the build process, but with the updated menu 
> guided bootstrap process displayed in the console, P is what one must enter 
> to continue the build process. I updated that info.
> The previous menu guided bootstrap process was outdated and did not show the 
> new features Bustache Support to SQL Support and the Build Options. So, I 
> updated it to show that information.
>  
> Here is my pull request with the updates: 
> [https://github.com/apache/nifi-minifi-cpp/pull/760] 
> Here is the doc that I am proposing updates for: 
> [https://github.com/apache/nifi-minifi-cpp/blob/master/README.md]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (MINIFICPP-1195) MiNiFiCPP-DOCS: Updates README Outdated Bootstrapping Info

[James Medel (Jira)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-1195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Medel updated MINIFICPP-1195:
---
Summary: MiNiFiCPP-DOCS: Updates README Outdated Bootstrapping Info  (was: 
MiNiFiCPP-DOCS: Updated README Outdated Bootstrapping Info)

> MiNiFiCPP-DOCS: Updates README Outdated Bootstrapping Info
> --
>
> Key: MINIFICPP-1195
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1195
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Documentation
>Affects Versions: 0.7.0
> Environment: Ubuntu 18.04 LTS on AWS EC2
>Reporter: James Medel
>Priority: Major
> Fix For: 0.7.0
>
>
> The paragraph in the Getting Started: Bootstrapping section originally told 
> the user to enter N to continue the build process, but with the updated menu 
> guided bootstrap process displayed in the console, P is what one must enter 
> to continue the build process. I updated that info.
> The previous menu guided bootstrap process was outdated and did not show the 
> new features Bustache Support to SQL Support and the Build Options. So, I 
> updated it to show that information.
>  
> Here is the doc that I am proposing updates for: 
> [https://github.com/apache/nifi-minifi-cpp/blob/master/README.md]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (MINIFICPP-1195) MiNiFiCPP-DOCS: Updated README Outdated Bootstrapping Info

[James Medel (Jira)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-1195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Medel updated MINIFICPP-1195:
---
Summary: MiNiFiCPP-DOCS: Updated README Outdated Bootstrapping Info  (was: 
MiNiFiCPP-DOCS: Updated Outdated Bootstrapping Info in README.md)

> MiNiFiCPP-DOCS: Updated README Outdated Bootstrapping Info
> --
>
> Key: MINIFICPP-1195
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1195
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Documentation
>Affects Versions: 0.7.0
> Environment: Ubuntu 18.04 LTS on AWS EC2
>Reporter: James Medel
>Priority: Major
> Fix For: 0.7.0
>
>
> The paragraph in the Getting Started: Bootstrapping section originally told 
> the user to enter N to continue the build process, but with the updated menu 
> guided bootstrap process displayed in the console, P is what one must enter 
> to continue the build process. I updated that info.
> The previous menu guided bootstrap process was outdated and did not show the 
> new features Bustache Support to SQL Support and the Build Options. So, I 
> updated it to show that information.
>  
> Here is the doc that I am proposing updates for: 
> [https://github.com/apache/nifi-minifi-cpp/blob/master/README.md]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi-minifi-cpp] james94 opened a new pull request #760: MiNiFCPP-1195: Updated Outdated README Bootstrapping Info

[GitBox]

james94 opened a new pull request #760: MiNiFCPP-1195: Updated Outdated README 
Bootstrapping Info
URL: https://github.com/apache/nifi-minifi-cpp/pull/760
 
 
   The paragraph in the Getting Started: Bootstrapping section originally told 
the user to enter N to continue the build process, but with the updated menu 
guided bootstrap process displayed in the console, P is what one must enter to 
continue the build process. I updated that info. 
   
   The previous menu guided bootstrap process was outdated and did not show the 
new features Bustache Support to SQL Support and the Build Options. So, I 
updated it to show that information.
   
   Here is the Jira Ticket: https://issues.apache.org/jira/browse/MINIFICPP-1195
   
   Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
   
   - [ ] Does your PR title start with MINIFICPP- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   
   - [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?
   
   - [ ] Is your initial contribution a single, squashed commit?
   
   ### For code changes:
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the LICENSE file?
   - [ ] If applicable, have you updated the NOTICE file?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (MINIFICPP-1195) MiNiFiCPP-DOCS: Updated Outdated Bootstrapping Info in README.md

[James Medel (Jira)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-1195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Medel updated MINIFICPP-1195:
---
Summary: MiNiFiCPP-DOCS: Updated Outdated Bootstrapping Info in README.md  
(was: MiNiFiCPP-DOCS: Updated Outdated Bootstrapping Info)

> MiNiFiCPP-DOCS: Updated Outdated Bootstrapping Info in README.md
> 
>
> Key: MINIFICPP-1195
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1195
> Project: Apache NiFi MiNiFi C++
>  Issue Type: Documentation
>Affects Versions: 0.7.0
> Environment: Ubuntu 18.04 LTS on AWS EC2
>Reporter: James Medel
>Priority: Major
> Fix For: 0.7.0
>
>
> The paragraph in the Getting Started: Bootstrapping section originally told 
> the user to enter N to continue the build process, but with the updated menu 
> guided bootstrap process displayed in the console, P is what one must enter 
> to continue the build process. I updated that info.
> The previous menu guided bootstrap process was outdated and did not show the 
> new features Bustache Support to SQL Support and the Build Options. So, I 
> updated it to show that information.
>  
> Here is the doc that I am proposing updates for: 
> [https://github.com/apache/nifi-minifi-cpp/blob/master/README.md]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (MINIFICPP-1195) MiNiFiCPP-DOCS: Updated Outdated Bootstrapping Info

[James Medel (Jira)]

James Medel created MINIFICPP-1195:
--

 Summary: MiNiFiCPP-DOCS: Updated Outdated Bootstrapping Info
 Key: MINIFICPP-1195
 URL: https://issues.apache.org/jira/browse/MINIFICPP-1195
 Project: Apache NiFi MiNiFi C++
  Issue Type: Documentation
Affects Versions: 0.7.0
 Environment: Ubuntu 18.04 LTS on AWS EC2
Reporter: James Medel
 Fix For: 0.7.0


The paragraph in the Getting Started: Bootstrapping section originally told the 
user to enter N to continue the build process, but with the updated menu guided 
bootstrap process displayed in the console, P is what one must enter to 
continue the build process. I updated that info.

The previous menu guided bootstrap process was outdated and did not show the 
new features Bustache Support to SQL Support and the Build Options. So, I 
updated it to show that information.

 

Here is the doc that I am proposing updates for: 
[https://github.com/apache/nifi-minifi-cpp/blob/master/README.md]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi-minifi-cpp] james94 commented on issue #755: MINIFICPP-1194 - Fix Table Formatting for 2 Execute.* Processors

[GitBox]

james94 commented on issue #755: MINIFICPP-1194 - Fix Table Formatting for 2 
Execute.* Processors
URL: https://github.com/apache/nifi-minifi-cpp/pull/755#issuecomment-614826537
 
 
   I created a matching Jira ticket for my pull request. 
   
   https://issues.apache.org/jira/browse/MINIFICPP-1194
   
   Please let me know if there is anything else I need to do.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (MINIFICPP-1194) MiNiFi-DOCS - Fix Table Formatting for 2 Execute.* Processors

[James Medel (Jira)]

James Medel created MINIFICPP-1194:
--

 Summary: MiNiFi-DOCS - Fix Table Formatting for 2 Execute.* 
Processors
 Key: MINIFICPP-1194
 URL: https://issues.apache.org/jira/browse/MINIFICPP-1194
 Project: Apache NiFi MiNiFi C++
  Issue Type: Documentation
Affects Versions: 0.7.0
 Environment: Ubuntu 18.04 LTS in AWS EC2
Reporter: James Medel
 Fix For: 0.7.0


Table formatting issues were making it difficult to read properties for 
ExecutePythonProcessor and ExecuteScript. So, this change in markdown resolves 
the formatting issues.

Doc affected: 
[https://github.com/apache/nifi-minifi-cpp/blob/master/PROCESSORS.md#executescript]

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (NIFI-7363) Deleting/Modifying Parameters with Nifi Registry

[Pierre Villard (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-7363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17085160#comment-17085160
 ] 

Pierre Villard commented on NIFI-7363:
--

Deleting parameters should not happen very frequently. Could another option be 
to un-version the flow and version it as a new flow? I know this is not 
ideal... but it could be another option to mitigate the issue in the meantime.

> Deleting/Modifying Parameters with Nifi Registry
> 
>
> Key: NIFI-7363
> URL: https://issues.apache.org/jira/browse/NIFI-7363
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.11.1, 1.11.4
> Environment: NIFIREG 0.5.0
> NIFIREG 0.6.0
>Reporter: Eric Secules
>Priority: Major
>
> I am noticing that when I delete a parameter from a parameter context in NiFi 
> a number of strange things result.
>  * It doesn't register as a change that I can commit to the registry (the 
> process group still has the green check mark)
>  * When I do make a change and commit to the registry, the deleted parameter 
> remains there and will be downloaded by anyone downloading the latest version 
> of the flow.
>  * When I look into the registry's file system, at the bottom of the 
> .snapshot files I notice that each individual process group has a listing of 
> parameters and their values each with slight differences from eachother. This 
> is weird because the documentation says that parameter contexts are global so 
> I was expecting a single parameter context entity somewhere.
> What I expect to happen:
>  * When I delete or modify the value of a parameter I should be able to check 
> that in in a versioned way so that newer process group versions see the 
> changes but older versions do not.
>  * I expected that since parameter contexts are independent of process 
> groups, that they would be versioned independently of process groups.
>  
>  I have noticed that the parameter context gets bloated with a bunch of 
> parameters that have no users and no way to delete them after doing a big 
> renaming effort.
>  
> Initial Investigation:
>  * When I create and versioned a single process group which is the only user 
> of a parameter context everything behaves as expected. Deleted parameters 
> stay deleted and latest values arrive when re-importing the latest version of 
> the process group on a blank canvas of a freshly restarted docker container.
>  * When there are *two versioned process groups that share a single parameter 
> context* what actually ends up happening is the parameter context on the NiFi 
> instance is patched with the parameter contexts from the registry in the 
> order that the process groups are imported. So if you delete a parameter and 
> create a new version of a process gorup you also need to create new versions 
> of all the process gorups that use that parameter. This wouldn't happen if 
> parameters were versioned as their own entity rather than something that is 
> tacked onto a process group.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (NIFI-7363) Deleting/Modifying Parameters with Nifi Registry

[Joe Witt (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-7363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17085142#comment-17085142
 ] 

Joe Witt commented on NIFI-7363:


Nope I think you explained it quite well.  Just a matter of folks having time 
to focus on and dive into this.  But good report for sure

> Deleting/Modifying Parameters with Nifi Registry
> 
>
> Key: NIFI-7363
> URL: https://issues.apache.org/jira/browse/NIFI-7363
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.11.1, 1.11.4
> Environment: NIFIREG 0.5.0
> NIFIREG 0.6.0
>Reporter: Eric Secules
>Priority: Major
>
> I am noticing that when I delete a parameter from a parameter context in NiFi 
> a number of strange things result.
>  * It doesn't register as a change that I can commit to the registry (the 
> process group still has the green check mark)
>  * When I do make a change and commit to the registry, the deleted parameter 
> remains there and will be downloaded by anyone downloading the latest version 
> of the flow.
>  * When I look into the registry's file system, at the bottom of the 
> .snapshot files I notice that each individual process group has a listing of 
> parameters and their values each with slight differences from eachother. This 
> is weird because the documentation says that parameter contexts are global so 
> I was expecting a single parameter context entity somewhere.
> What I expect to happen:
>  * When I delete or modify the value of a parameter I should be able to check 
> that in in a versioned way so that newer process group versions see the 
> changes but older versions do not.
>  * I expected that since parameter contexts are independent of process 
> groups, that they would be versioned independently of process groups.
>  
>  I have noticed that the parameter context gets bloated with a bunch of 
> parameters that have no users and no way to delete them after doing a big 
> renaming effort.
>  
> Initial Investigation:
>  * When I create and versioned a single process group which is the only user 
> of a parameter context everything behaves as expected. Deleted parameters 
> stay deleted and latest values arrive when re-importing the latest version of 
> the process group on a blank canvas of a freshly restarted docker container.
>  * When there are *two versioned process groups that share a single parameter 
> context* what actually ends up happening is the parameter context on the NiFi 
> instance is patched with the parameter contexts from the registry in the 
> order that the process groups are imported. So if you delete a parameter and 
> create a new version of a process gorup you also need to create new versions 
> of all the process gorups that use that parameter. This wouldn't happen if 
> parameters were versioned as their own entity rather than something that is 
> tacked onto a process group.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (NIFI-7363) Deleting/Modifying Parameters with Nifi Registry

[Eric Secules (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-7363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17085140#comment-17085140
 ] 

Eric Secules edited comment on NIFI-7363 at 4/16/20, 6:04 PM:
--

[~joewitt] Is there anything else I can do to help groom this issue so that it 
is ready for someone to work on and when can we expect a fix? It seems to me 
like a design flaw in how parameter contexts are versioned. This is a big issue 
for production deployment because the value of a parameter is based on the 
order that process groups are imported. My current workaround is to version 
parameter values in externally in a file and manually set them with the REST 
API.


was (Author: esecules):
[~joewitt] Is there anything else I can do to help groom this issue so that it 
is ready for someone to work on? It seems to me like a design flaw in how 
parameter contexts are versioned. This is a big issue for production deployment 
because the value of a parameter is based on the order that process groups are 
imported. My current workaround is to version parameter values in externally in 
a file and manually set them with the REST API.

> Deleting/Modifying Parameters with Nifi Registry
> 
>
> Key: NIFI-7363
> URL: https://issues.apache.org/jira/browse/NIFI-7363
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.11.1, 1.11.4
> Environment: NIFIREG 0.5.0
> NIFIREG 0.6.0
>Reporter: Eric Secules
>Priority: Major
>
> I am noticing that when I delete a parameter from a parameter context in NiFi 
> a number of strange things result.
>  * It doesn't register as a change that I can commit to the registry (the 
> process group still has the green check mark)
>  * When I do make a change and commit to the registry, the deleted parameter 
> remains there and will be downloaded by anyone downloading the latest version 
> of the flow.
>  * When I look into the registry's file system, at the bottom of the 
> .snapshot files I notice that each individual process group has a listing of 
> parameters and their values each with slight differences from eachother. This 
> is weird because the documentation says that parameter contexts are global so 
> I was expecting a single parameter context entity somewhere.
> What I expect to happen:
>  * When I delete or modify the value of a parameter I should be able to check 
> that in in a versioned way so that newer process group versions see the 
> changes but older versions do not.
>  * I expected that since parameter contexts are independent of process 
> groups, that they would be versioned independently of process groups.
>  
>  I have noticed that the parameter context gets bloated with a bunch of 
> parameters that have no users and no way to delete them after doing a big 
> renaming effort.
>  
> Initial Investigation:
>  * When I create and versioned a single process group which is the only user 
> of a parameter context everything behaves as expected. Deleted parameters 
> stay deleted and latest values arrive when re-importing the latest version of 
> the process group on a blank canvas of a freshly restarted docker container.
>  * When there are *two versioned process groups that share a single parameter 
> context* what actually ends up happening is the parameter context on the NiFi 
> instance is patched with the parameter contexts from the registry in the 
> order that the process groups are imported. So if you delete a parameter and 
> create a new version of a process gorup you also need to create new versions 
> of all the process gorups that use that parameter. This wouldn't happen if 
> parameters were versioned as their own entity rather than something that is 
> tacked onto a process group.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (NIFI-7363) Deleting/Modifying Parameters with Nifi Registry

[Eric Secules (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-7363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17085140#comment-17085140
 ] 

Eric Secules commented on NIFI-7363:


[~joewitt] Is there anything else I can do to help groom this issue so that it 
is ready for someone to work on? It seems to me like a design flaw in how 
parameter contexts are versioned. This is a big issue for production deployment 
because the value of a parameter is based on the order that process groups are 
imported. My current workaround is to version parameter values in externally in 
a file and manually set them with the REST API.

> Deleting/Modifying Parameters with Nifi Registry
> 
>
> Key: NIFI-7363
> URL: https://issues.apache.org/jira/browse/NIFI-7363
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.11.1, 1.11.4
> Environment: NIFIREG 0.5.0
> NIFIREG 0.6.0
>Reporter: Eric Secules
>Priority: Major
>
> I am noticing that when I delete a parameter from a parameter context in NiFi 
> a number of strange things result.
>  * It doesn't register as a change that I can commit to the registry (the 
> process group still has the green check mark)
>  * When I do make a change and commit to the registry, the deleted parameter 
> remains there and will be downloaded by anyone downloading the latest version 
> of the flow.
>  * When I look into the registry's file system, at the bottom of the 
> .snapshot files I notice that each individual process group has a listing of 
> parameters and their values each with slight differences from eachother. This 
> is weird because the documentation says that parameter contexts are global so 
> I was expecting a single parameter context entity somewhere.
> What I expect to happen:
>  * When I delete or modify the value of a parameter I should be able to check 
> that in in a versioned way so that newer process group versions see the 
> changes but older versions do not.
>  * I expected that since parameter contexts are independent of process 
> groups, that they would be versioned independently of process groups.
>  
>  I have noticed that the parameter context gets bloated with a bunch of 
> parameters that have no users and no way to delete them after doing a big 
> renaming effort.
>  
> Initial Investigation:
>  * When I create and versioned a single process group which is the only user 
> of a parameter context everything behaves as expected. Deleted parameters 
> stay deleted and latest values arrive when re-importing the latest version of 
> the process group on a blank canvas of a freshly restarted docker container.
>  * When there are *two versioned process groups that share a single parameter 
> context* what actually ends up happening is the parameter context on the NiFi 
> instance is patched with the parameter contexts from the registry in the 
> order that the process groups are imported. So if you delete a parameter and 
> create a new version of a process gorup you also need to create new versions 
> of all the process gorups that use that parameter. This wouldn't happen if 
> parameters were versioned as their own entity rather than something that is 
> tacked onto a process group.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (NIFI-7370) Update scripted component documentation about Jython issues

[Matt Burgess (Jira)]

Matt Burgess created NIFI-7370:
--

 Summary: Update scripted component documentation about Jython 
issues
 Key: NIFI-7370
 URL: https://issues.apache.org/jira/browse/NIFI-7370
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Extensions
Reporter: Matt Burgess


NIFI-4004 introduced a new default interface method for RecordReaderFactory, in 
order
to avoid breaking API compatibility for those with custom
record readers. This exposes the bug in [1] for the Jython script
engine. At a major release point (NiFi 2.0) we could refactor the NiFi
codebase to remove the default interface method, update all internal
implementations, and announce that the LookupService API has changed
and thus custom implementations would have to be updated. Not sure if
we can get away with that for minor releases or not, usually breaking
API compatibility is a no-no except for major releases.

As was done in NIFI-5995 for ScriptedLookupService, this Jira proposes to 
update the documentation for any other scripted component for which its 
underlying interface has default methods, and also to remove the Jython script 
engine from the allowable values for the Script Engine property. One such 
component is ScriptedReader, but there may be others as well.

[1] [https://bugs.jython.org/issue2403|https://bugs.jython.org/issue2403]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi-minifi-cpp] arpadboda opened a new pull request #759: MINIFICPP-1193 - heartbeat interval config doesn't work properly

[GitBox]

arpadboda opened a new pull request #759: MINIFICPP-1193 - heartbeat interval 
config doesn't work properly
URL: https://github.com/apache/nifi-minifi-cpp/pull/759
 
 
   Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
   
   - [ ] Does your PR title start with MINIFICPP- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   
   - [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?
   
   - [ ] Is your initial contribution a single, squashed commit?
   
   ### For code changes:
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the LICENSE file?
   - [ ] If applicable, have you updated the NOTICE file?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (MINIFICPP-1193) heartbeat interval config doesn't work properly

[Arpad Boda (Jira)]

Arpad Boda created MINIFICPP-1193:
-

 Summary: heartbeat interval config doesn't work properly
 Key: MINIFICPP-1193
 URL: https://issues.apache.org/jira/browse/MINIFICPP-1193
 Project: Apache NiFi MiNiFi C++
  Issue Type: Bug
Affects Versions: 0.7.0
Reporter: Arpad Boda
Assignee: Arpad Boda
 Fix For: 0.8.0


As there is a misimplemented check in heartbeat callback, sometimes only every 
second call results in actually sending a heartbeat. This can cause 
communication issues in case the heartbeat rate is relatively low. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi] EndzeitBegins commented on issue #4201: NIFI-7348 Wait - Removes WAIT_START_TIMESTAMP after expiration

[GitBox]

EndzeitBegins commented on issue #4201: NIFI-7348 Wait - Removes 
WAIT_START_TIMESTAMP after expiration
URL: https://github.com/apache/nifi/pull/4201#issuecomment-614765450
 
 
   Thanks @ottobackwards and @ijokarumawak for the reviews. Have a nice week.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] MikeThomsen commented on issue #4204: NIFI-7355 Added Gremlin bytecode client service.

[GitBox]

MikeThomsen commented on issue #4204: NIFI-7355 Added Gremlin bytecode client 
service.
URL: https://github.com/apache/nifi/pull/4204#issuecomment-614760704
 
 
   @mattyb149 thanks for the feedback. I should be able to knock all of this 
out this afternoon.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] jfrazee commented on a change in pull request #4200: NIFI-7335: PutAzureCosmosDBDocument to provide the cosmos sql api support for Azure Cosmos DB

[GitBox]

jfrazee commented on a change in pull request #4200: NIFI-7335: 
PutAzureCosmosDBDocument to provide the cosmos sql api support for Azure Cosmos 
DB
URL: https://github.com/apache/nifi/pull/4200#discussion_r409690446
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/cosmos/document/PutAzureCosmosDBDocument.java
 ##
 @@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.cosmos.document;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SystemResourceConsideration;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SystemResource;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.stream.io.StreamUtils;
+
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+
+@EventDriven
+@Tags({ "azure", "cosmos", "document", "insert", "update", "write", "put" })
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Writes the contents of a FlowFile to Azure Cosmos DB 
with Core SQL API. For CosmosDB with Mongo API, use PutMongo instead.")
+@SystemResourceConsideration(resource = SystemResource.MEMORY)
+public class PutAzureCosmosDBDocument extends AbstractAzureCosmosDBProcessor {
+
+private final static Set relationships;
+private final static List propertyDescriptors;
+
+private final static ObjectMapper mapper;
+
+static {
+List _propertyDescriptors = new ArrayList<>();
+_propertyDescriptors.addAll(descriptors);
+_propertyDescriptors.add(CHARACTER_SET);
+propertyDescriptors = 
Collections.unmodifiableList(_propertyDescriptors);
+
+final Set _relationships = new HashSet<>();
+_relationships.add(REL_SUCCESS);
+_relationships.add(REL_FAILURE);
+relationships = Collections.unmodifiableSet(_relationships);
+mapper = new ObjectMapper();
+}
+
+@Override
+public Set getRelationships() {
+return relationships;
+}
+
+@Override
+public List getSupportedPropertyDescriptors() {
+return propertyDescriptors;
+}
+
+
+@Override
+public void onTrigger(final ProcessContext context, final ProcessSession 
session) throws ProcessException {
+final FlowFile flowFile = session.get();
+if (flowFile == null) {
+return;
+}
+
+final ComponentLog logger = getLogger();
+
+final Charset charset = 
Charset.forName(context.getProperty(CHARACTER_SET).getValue());
+final String partitionKeyField = 
context.getProperty(PARTITION_KEY).getValue();
+JsonNode doc = null;
+
+try {
+// Read the contents of the FlowFile into a byte array
+final byte[] content = new byte[(int) flowFile.getSize()];
+session.read(flowFile, in -> StreamUtils.fillBuffer(in, content, 
true));
+
+// parse conten into JsonNode object
+doc = mapper.readTree(new String(content, charset));
+
+// make sure type of id is String type if exists
+
+if(doc.has("id")) {
+JsonNode idNode = doc.get("id");
+if(idNode.isNumber()) {
+logger.debug("coverting number id into string...");
+((ObjectNode) doc).put("id", doc.get("id"

[GitHub] [nifi] jfrazee commented on a change in pull request #4200: NIFI-7335: PutAzureCosmosDBDocument to provide the cosmos sql api support for Azure Cosmos DB

[GitBox]

jfrazee commented on a change in pull request #4200: NIFI-7335: 
PutAzureCosmosDBDocument to provide the cosmos sql api support for Azure Cosmos 
DB
URL: https://github.com/apache/nifi/pull/4200#discussion_r409683267
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/cosmos/document/AbstractAzureCosmosDBProcessor.java
 ##
 @@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.nifi.processors.azure.cosmos.document;
+
+import java.io.ByteArrayInputStream;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+import com.azure.cosmos.ConnectionPolicy;
+import com.azure.cosmos.ConsistencyLevel;
+import com.azure.cosmos.CosmosClient;
+import com.azure.cosmos.CosmosClientBuilder;
+import com.azure.cosmos.CosmosClientException;
+import com.azure.cosmos.CosmosContainer;
+import com.azure.cosmos.CosmosDatabase;
+import com.azure.cosmos.models.CosmosContainerProperties;
+
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.cosmos.AzureCosmosDBConnectionService;
+
+public abstract class AbstractAzureCosmosDBProcessor extends AbstractProcessor 
{
+static final String CONSISTENCY_STRONG = "STRONG";
+static final String CONSISTENCY_BOUNDED_STALENESS= "BOUNDED_STALENESS";
+static final String CONSISTENCY_SESSION = "SESSION";
+static final String CONSISTENCY_CONSISTENT_PREFIX = "CONSISTENT_PREFIX";
+static final String CONSISTENCY_EVENTUAL = "EVENTUAL";
+
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to Azure Cosmos DB are routed 
to this relationship").build();
+
+static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to Azure Cosmos DB are 
routed to this relationship").build();
+
+static final Relationship REL_ORIGINAL = new Relationship.Builder()
+.name("original")
+.description("All input FlowFiles that are part of a successful 
query execution go here.")
+.build();
+
+static final PropertyDescriptor CONNECTION_SERVICE = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-connection-service")
+.displayName("Azure Cosmos DB Connection Service")
+.description("If configured, this property will use the assigned for 
retrieving connection string info.")
+.required(false)
+.identifiesControllerService(AzureCosmosDBConnectionService.class)
+.build();
+
+
+static final PropertyDescriptor URI = new PropertyDescriptor.Builder()
+.name("azure-cosmos-db-uri")
+.displayName("Azure Cosmos DB URI")
+.description("Azure Cosmos DB URI, typically of the form: 
https://{databaseaccount}.documents.azure.com:443/. Note this host URL is for 
Azure Cosmos DB with CORE SQL API")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+static final PropertyDescriptor DB_ACCESS_KEY = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-key")
+.displayName("Azrue Cosmos DB Access Key")
+.description("Azure Cosmos DB Access Key from Azure Portal 
(Settings->Keys)")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+
+static final PropertyDescriptor DATABASE_NAME = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db

[GitHub] [nifi] jfrazee commented on a change in pull request #4200: NIFI-7335: PutAzureCosmosDBDocument to provide the cosmos sql api support for Azure Cosmos DB

[GitBox]

jfrazee commented on a change in pull request #4200: NIFI-7335: 
PutAzureCosmosDBDocument to provide the cosmos sql api support for Azure Cosmos 
DB
URL: https://github.com/apache/nifi/pull/4200#discussion_r409680984
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/cosmos/document/AbstractAzureCosmosDBProcessor.java
 ##
 @@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.nifi.processors.azure.cosmos.document;
+
+import java.io.ByteArrayInputStream;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+import com.azure.cosmos.ConnectionPolicy;
+import com.azure.cosmos.ConsistencyLevel;
+import com.azure.cosmos.CosmosClient;
+import com.azure.cosmos.CosmosClientBuilder;
+import com.azure.cosmos.CosmosClientException;
+import com.azure.cosmos.CosmosContainer;
+import com.azure.cosmos.CosmosDatabase;
+import com.azure.cosmos.models.CosmosContainerProperties;
+
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.cosmos.AzureCosmosDBConnectionService;
+
+public abstract class AbstractAzureCosmosDBProcessor extends AbstractProcessor 
{
+static final String CONSISTENCY_STRONG = "STRONG";
+static final String CONSISTENCY_BOUNDED_STALENESS= "BOUNDED_STALENESS";
+static final String CONSISTENCY_SESSION = "SESSION";
+static final String CONSISTENCY_CONSISTENT_PREFIX = "CONSISTENT_PREFIX";
+static final String CONSISTENCY_EVENTUAL = "EVENTUAL";
+
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to Azure Cosmos DB are routed 
to this relationship").build();
+
+static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to Azure Cosmos DB are 
routed to this relationship").build();
+
+static final Relationship REL_ORIGINAL = new Relationship.Builder()
+.name("original")
+.description("All input FlowFiles that are part of a successful 
query execution go here.")
+.build();
+
+static final PropertyDescriptor CONNECTION_SERVICE = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-connection-service")
+.displayName("Azure Cosmos DB Connection Service")
+.description("If configured, this property will use the assigned for 
retrieving connection string info.")
+.required(false)
+.identifiesControllerService(AzureCosmosDBConnectionService.class)
+.build();
+
+
+static final PropertyDescriptor URI = new PropertyDescriptor.Builder()
+.name("azure-cosmos-db-uri")
+.displayName("Azure Cosmos DB URI")
+.description("Azure Cosmos DB URI, typically of the form: 
https://{databaseaccount}.documents.azure.com:443/. Note this host URL is for 
Azure Cosmos DB with CORE SQL API")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+static final PropertyDescriptor DB_ACCESS_KEY = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-key")
+.displayName("Azrue Cosmos DB Access Key")
+.description("Azure Cosmos DB Access Key from Azure Portal 
(Settings->Keys)")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+
+static final PropertyDescriptor DATABASE_NAME = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db

[GitHub] [nifi] jfrazee commented on a change in pull request #4200: NIFI-7335: PutAzureCosmosDBDocument to provide the cosmos sql api support for Azure Cosmos DB

[GitBox]

jfrazee commented on a change in pull request #4200: NIFI-7335: 
PutAzureCosmosDBDocument to provide the cosmos sql api support for Azure Cosmos 
DB
URL: https://github.com/apache/nifi/pull/4200#discussion_r409680984
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/cosmos/document/AbstractAzureCosmosDBProcessor.java
 ##
 @@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.nifi.processors.azure.cosmos.document;
+
+import java.io.ByteArrayInputStream;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+import com.azure.cosmos.ConnectionPolicy;
+import com.azure.cosmos.ConsistencyLevel;
+import com.azure.cosmos.CosmosClient;
+import com.azure.cosmos.CosmosClientBuilder;
+import com.azure.cosmos.CosmosClientException;
+import com.azure.cosmos.CosmosContainer;
+import com.azure.cosmos.CosmosDatabase;
+import com.azure.cosmos.models.CosmosContainerProperties;
+
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.cosmos.AzureCosmosDBConnectionService;
+
+public abstract class AbstractAzureCosmosDBProcessor extends AbstractProcessor 
{
+static final String CONSISTENCY_STRONG = "STRONG";
+static final String CONSISTENCY_BOUNDED_STALENESS= "BOUNDED_STALENESS";
+static final String CONSISTENCY_SESSION = "SESSION";
+static final String CONSISTENCY_CONSISTENT_PREFIX = "CONSISTENT_PREFIX";
+static final String CONSISTENCY_EVENTUAL = "EVENTUAL";
+
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to Azure Cosmos DB are routed 
to this relationship").build();
+
+static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to Azure Cosmos DB are 
routed to this relationship").build();
+
+static final Relationship REL_ORIGINAL = new Relationship.Builder()
+.name("original")
+.description("All input FlowFiles that are part of a successful 
query execution go here.")
+.build();
+
+static final PropertyDescriptor CONNECTION_SERVICE = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-connection-service")
+.displayName("Azure Cosmos DB Connection Service")
+.description("If configured, this property will use the assigned for 
retrieving connection string info.")
+.required(false)
+.identifiesControllerService(AzureCosmosDBConnectionService.class)
+.build();
+
+
+static final PropertyDescriptor URI = new PropertyDescriptor.Builder()
+.name("azure-cosmos-db-uri")
+.displayName("Azure Cosmos DB URI")
+.description("Azure Cosmos DB URI, typically of the form: 
https://{databaseaccount}.documents.azure.com:443/. Note this host URL is for 
Azure Cosmos DB with CORE SQL API")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+static final PropertyDescriptor DB_ACCESS_KEY = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-key")
+.displayName("Azrue Cosmos DB Access Key")
+.description("Azure Cosmos DB Access Key from Azure Portal 
(Settings->Keys)")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+
+static final PropertyDescriptor DATABASE_NAME = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db

[GitHub] [nifi] jfrazee commented on a change in pull request #4200: NIFI-7335: PutAzureCosmosDBDocument to provide the cosmos sql api support for Azure Cosmos DB

[GitBox]

jfrazee commented on a change in pull request #4200: NIFI-7335: 
PutAzureCosmosDBDocument to provide the cosmos sql api support for Azure Cosmos 
DB
URL: https://github.com/apache/nifi/pull/4200#discussion_r409679464
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/cosmos/document/AbstractAzureCosmosDBProcessor.java
 ##
 @@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.nifi.processors.azure.cosmos.document;
+
+import java.io.ByteArrayInputStream;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+import com.azure.cosmos.ConnectionPolicy;
+import com.azure.cosmos.ConsistencyLevel;
+import com.azure.cosmos.CosmosClient;
+import com.azure.cosmos.CosmosClientBuilder;
+import com.azure.cosmos.CosmosClientException;
+import com.azure.cosmos.CosmosContainer;
+import com.azure.cosmos.CosmosDatabase;
+import com.azure.cosmos.models.CosmosContainerProperties;
+
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.cosmos.AzureCosmosDBConnectionService;
+
+public abstract class AbstractAzureCosmosDBProcessor extends AbstractProcessor 
{
+static final String CONSISTENCY_STRONG = "STRONG";
+static final String CONSISTENCY_BOUNDED_STALENESS= "BOUNDED_STALENESS";
+static final String CONSISTENCY_SESSION = "SESSION";
+static final String CONSISTENCY_CONSISTENT_PREFIX = "CONSISTENT_PREFIX";
+static final String CONSISTENCY_EVENTUAL = "EVENTUAL";
+
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to Azure Cosmos DB are routed 
to this relationship").build();
+
+static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to Azure Cosmos DB are 
routed to this relationship").build();
+
+static final Relationship REL_ORIGINAL = new Relationship.Builder()
+.name("original")
+.description("All input FlowFiles that are part of a successful 
query execution go here.")
+.build();
+
+static final PropertyDescriptor CONNECTION_SERVICE = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-connection-service")
+.displayName("Azure Cosmos DB Connection Service")
+.description("If configured, this property will use the assigned for 
retrieving connection string info.")
+.required(false)
+.identifiesControllerService(AzureCosmosDBConnectionService.class)
+.build();
+
+
+static final PropertyDescriptor URI = new PropertyDescriptor.Builder()
+.name("azure-cosmos-db-uri")
+.displayName("Azure Cosmos DB URI")
+.description("Azure Cosmos DB URI, typically of the form: 
https://{databaseaccount}.documents.azure.com:443/. Note this host URL is for 
Azure Cosmos DB with CORE SQL API")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+static final PropertyDescriptor DB_ACCESS_KEY = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-key")
+.displayName("Azrue Cosmos DB Access Key")
+.description("Azure Cosmos DB Access Key from Azure Portal 
(Settings->Keys)")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+
+static final PropertyDescriptor DATABASE_NAME = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db

[GitHub] [nifi] mattyb149 commented on a change in pull request #4204: NIFI-7355 Added Gremlin bytecode client service.

[GitBox]

mattyb149 commented on a change in pull request #4204: NIFI-7355 Added Gremlin 
bytecode client service.
URL: https://github.com/apache/nifi/pull/4204#discussion_r409673719
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-graph-bundle/nifi-other-graph-services/src/main/resources/docs/org.apache.nifi.graph.GremlinBytecodeClientService/additionalDetails.html
 ##
 @@ -0,0 +1,38 @@
+
+
+
+
+
+GremlinClientService
 
 Review comment:
   Should be GremlinBytecodeClientService


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] mattyb149 commented on a change in pull request #4204: NIFI-7355 Added Gremlin bytecode client service.

[GitBox]

mattyb149 commented on a change in pull request #4204: NIFI-7355 Added Gremlin 
bytecode client service.
URL: https://github.com/apache/nifi/pull/4204#discussion_r409678391
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-graph-bundle/nifi-other-graph-services/src/main/java/org/apache/nifi/graph/GremlinBytecodeClientService.java
 ##
 @@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.graph;
+
+
+import org.apache.commons.codec.digest.DigestUtils;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.util.StringUtils;
+import org.apache.tinkerpop.gremlin.driver.Cluster;
+import org.apache.tinkerpop.gremlin.driver.remote.DriverRemoteConnection;
+import org.apache.tinkerpop.gremlin.process.traversal.AnonymousTraversalSource;
+import 
org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource;
+
+import javax.script.Bindings;
+import javax.script.Compilable;
+import javax.script.CompiledScript;
+import javax.script.ScriptEngine;
+import javax.script.ScriptEngineManager;
+import javax.script.ScriptException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+@CapabilityDescription("A client service that provides a scriptable interface 
to open a remote connection/travseral " +
+"against a Gremlin Server and execute operations against it.")
+@Tags({ "graph", "database", "gremlin", "tinkerpop" })
+public class GremlinBytecodeClientService extends 
AbstractTinkerpopClientService implements GraphClientService {
+private static final List NEW_DESCRIPTORS;
+
+public static final PropertyDescriptor TRAVERSAL_SOURCE_NAME = new 
PropertyDescriptor.Builder()
+.name("gremlin-traversal-source-name")
+.displayName("Traversal Source Name")
+.description("An optional property that lets you set the name of 
the remote traversal instance. " +
+"This can be really important when working with databases 
like JanusGraph that support " +
+"multiple backend traversal configurations 
simultaneously.")
+
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+.addValidator(Validator.VALID)
+.build();
+
+static {
+List _temp = new ArrayList<>();
+_temp.addAll(DESCRIPTORS);
+_temp.add(TRAVERSAL_SOURCE_NAME);
+NEW_DESCRIPTORS = Collections.unmodifiableList(_temp);
+}
+
+
+@Override
+public List getSupportedPropertyDescriptors() {
+return NEW_DESCRIPTORS;
+}
+
+private ScriptEngineManager MANAGER = new ScriptEngineManager();
+private ScriptEngine engine;
+private Map compiledCode;
+private Cluster cluster;
+private String traversalSourceName;
+
+/**
+ * @param context
+ *the configuration context
+ * @throws InitializationException
+ * if unable to create a database connection
+ */
+@OnEnabled
+public void onEnabled(final ConfigurationContext context) throws 
InitializationException {
+cluster = buildCluster(context);
+
+compiledCode = new ConcurrentHashMap<>();
+engine = MANAGER.getEngineByName("groovy");
+
+if (context.getProperty(TRAVERSAL_SOURCE_NAME).isSet()) {
+traversalSourceName = 
context.getProperty(TRAVERSAL_SOURCE_NAME).evaluateAttributeExpressions()
+.getValue();
+}
+}
+
+@OnDisabled
+public void shutdown() {
+try {
+compiledCode = null;
+engi

[GitHub] [nifi] mattyb149 commented on a change in pull request #4204: NIFI-7355 Added Gremlin bytecode client service.

[GitBox]

mattyb149 commented on a change in pull request #4204: NIFI-7355 Added Gremlin 
bytecode client service.
URL: https://github.com/apache/nifi/pull/4204#discussion_r407469988
 
 

 ##
 File path: nifi-nar-bundles/nifi-graph-bundle/nifi-other-graph-services/pom.xml
 ##
 @@ -91,6 +91,17 @@
 gremlin-driver
 3.3.5
 
+
+
+org.codehaus.groovy
+groovy
+2.5.10
 
 Review comment:
   I recommend bringing in the dateutils as well, as they are no longer 
included in the Groovy artifact as of 2.5.10 (see 
https://issues.apache.org/jira/browse/NIFI-7069 for details)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] mattyb149 commented on a change in pull request #4204: NIFI-7355 Added Gremlin bytecode client service.

[GitBox]

mattyb149 commented on a change in pull request #4204: NIFI-7355 Added Gremlin 
bytecode client service.
URL: https://github.com/apache/nifi/pull/4204#discussion_r409673719
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-graph-bundle/nifi-other-graph-services/src/main/resources/docs/org.apache.nifi.graph.GremlinBytecodeClientService/additionalDetails.html
 ##
 @@ -0,0 +1,38 @@
+
+
+
+
+
+GremlinClientService
 
 Review comment:
   Should be GremlineBytecodeClientService


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] mattyb149 commented on a change in pull request #4204: NIFI-7355 Added Gremlin bytecode client service.

[GitBox]

mattyb149 commented on a change in pull request #4204: NIFI-7355 Added Gremlin 
bytecode client service.
URL: https://github.com/apache/nifi/pull/4204#discussion_r409675718
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-graph-bundle/nifi-other-graph-services/src/test/groovy/org/apache/nifi/graph/GremlinBytecodeClientServiceIT.groovy
 ##
 @@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.graph
+
+import org.apache.nifi.processor.AbstractProcessor
+import org.apache.nifi.processor.ProcessContext
+import org.apache.nifi.processor.ProcessSession
+import org.apache.nifi.processor.exception.ProcessException
+import org.apache.nifi.util.TestRunner
+import org.apache.nifi.util.TestRunners
+import org.junit.Before
+import org.junit.Test
+
+class GremlinBytecodeClientServiceIT {
+TestRunner runner
+GraphClientService service
+
+@Before
+void before() {
+runner = TestRunners.newTestRunner(new AbstractProcessor() {
+@Override
+void onTrigger(ProcessContext processContext, ProcessSession 
processSession) throws ProcessException {
+
+}
+})
+
+service = new GremlinBytecodeClientService()
+runner.addControllerService("service", service);
+runner.setProperty(service, 
GremlinBytecodeClientService.CONTACT_POINTS, "localhost")
+runner.setProperty(service, GremlinBytecodeClientService.PORT, "8182")
+runner.setProperty(service, GremlinBytecodeClientService.PATH, 
"/gremlin")
+runner.enableControllerService(service)
+runner.assertValid()
+}
+
+@Test
+void testQuery() {
+def query = """
+assert param
+g.V().hasLabel("nada").count().next()
+"""
+
+service.executeQuery(query, [ param: "Hi!" ], { result, more ->
+assert result
+assert result["result"] == 0
+} as GraphQueryResultCallback)
+}
+
+@Test
+void testMassGenerate() {
+def query = """
+assert param
+1.upto(100) {
+def trav = g.addV("it_test_node")
+2.upto(250) {
+trav.addV("it_test_node").property("uuid", 
UUID.randomUUID().toString())
+}
+trav.next()
+}
+def count = g.V().hasLabel("it_test_node").count().next()
+g.V().hasLabel("it_test_node").drop().iterate()
+count
+""".trim()
+
+service.executeQuery(query, [ param: "Hi!" ], { result, more ->
 
 Review comment:
   I don't see the param value being used anywhere except the assert. Would it 
be good to add an example where you use the param as a property on the vertices 
and then test for its presence? Or am I misunderstanding how param should be 
used?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] jfrazee commented on a change in pull request #4200: NIFI-7335: PutAzureCosmosDBDocument to provide the cosmos sql api support for Azure Cosmos DB

[GitBox]

jfrazee commented on a change in pull request #4200: NIFI-7335: 
PutAzureCosmosDBDocument to provide the cosmos sql api support for Azure Cosmos 
DB
URL: https://github.com/apache/nifi/pull/4200#discussion_r409675676
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/cosmos/document/AbstractAzureCosmosDBProcessor.java
 ##
 @@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.nifi.processors.azure.cosmos.document;
+
+import java.io.ByteArrayInputStream;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+import com.azure.cosmos.ConnectionPolicy;
+import com.azure.cosmos.ConsistencyLevel;
+import com.azure.cosmos.CosmosClient;
+import com.azure.cosmos.CosmosClientBuilder;
+import com.azure.cosmos.CosmosClientException;
+import com.azure.cosmos.CosmosContainer;
+import com.azure.cosmos.CosmosDatabase;
+import com.azure.cosmos.models.CosmosContainerProperties;
+
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.cosmos.AzureCosmosDBConnectionService;
+
+public abstract class AbstractAzureCosmosDBProcessor extends AbstractProcessor 
{
+static final String CONSISTENCY_STRONG = "STRONG";
+static final String CONSISTENCY_BOUNDED_STALENESS= "BOUNDED_STALENESS";
+static final String CONSISTENCY_SESSION = "SESSION";
+static final String CONSISTENCY_CONSISTENT_PREFIX = "CONSISTENT_PREFIX";
+static final String CONSISTENCY_EVENTUAL = "EVENTUAL";
+
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to Azure Cosmos DB are routed 
to this relationship").build();
+
+static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to Azure Cosmos DB are 
routed to this relationship").build();
+
+static final Relationship REL_ORIGINAL = new Relationship.Builder()
+.name("original")
+.description("All input FlowFiles that are part of a successful 
query execution go here.")
+.build();
+
+static final PropertyDescriptor CONNECTION_SERVICE = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-connection-service")
+.displayName("Azure Cosmos DB Connection Service")
+.description("If configured, this property will use the assigned for 
retrieving connection string info.")
+.required(false)
+.identifiesControllerService(AzureCosmosDBConnectionService.class)
+.build();
+
+
+static final PropertyDescriptor URI = new PropertyDescriptor.Builder()
+.name("azure-cosmos-db-uri")
+.displayName("Azure Cosmos DB URI")
+.description("Azure Cosmos DB URI, typically of the form: 
https://{databaseaccount}.documents.azure.com:443/. Note this host URL is for 
Azure Cosmos DB with CORE SQL API")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+static final PropertyDescriptor DB_ACCESS_KEY = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-key")
+.displayName("Azrue Cosmos DB Access Key")
 
 Review comment:
   Typo here. Azrue -> Azure.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure a

[GitHub] [nifi] turcsanyip commented on a change in pull request #4213: NIFI-7280 Use Atlas metadata namespace in Atlas Reporting Task

[GitBox]

turcsanyip commented on a change in pull request #4213: NIFI-7280 Use Atlas 
metadata namespace in Atlas Reporting Task
URL: https://github.com/apache/nifi/pull/4213#discussion_r40969
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task/src/main/java/org/apache/nifi/atlas/reporting/ReportLineageToAtlas.java
 ##
 @@ -585,12 +594,58 @@ private void initAtlasProperties(ConfigurationContext 
context) throws IOExceptio
 if (confDir != null) {
 // If atlasConfDir is not set, atlas-application.properties will 
be searched under classpath.
 Properties props = System.getProperties();
-final String atlasConfProp = "atlas.conf";
+final String atlasConfProp = 
ApplicationProperties.ATLAS_CONFIGURATION_DIRECTORY_PROPERTY;
 props.setProperty(atlasConfProp, confDir.getAbsolutePath());
 getLogger().debug("{} has been set to: {}", new 
Object[]{atlasConfProp, props.getProperty(atlasConfProp)});
 }
 }
 
+private void parseAtlasUrls(final PropertyValue atlasUrlsProp, final 
Consumer urlStrConsumer) {
 
 Review comment:
   `parseAtlasUrls()` is now called with only a single type of consumers: 
`List::add`
   It could be simplified by omitting the consumer parameter, creating the list 
instance here and returning it to the caller.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] turcsanyip commented on a change in pull request #4213: NIFI-7280 Use Atlas metadata namespace in Atlas Reporting Task

[GitBox]

turcsanyip commented on a change in pull request #4213: NIFI-7280 Use Atlas 
metadata namespace in Atlas Reporting Task
URL: https://github.com/apache/nifi/pull/4213#discussion_r409533808
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task/src/main/java/org/apache/nifi/atlas/reporting/ReportLineageToAtlas.java
 ##
 @@ -192,11 +196,11 @@
 
 public static final PropertyDescriptor ATLAS_DEFAULT_CLUSTER_NAME = new 
PropertyDescriptor.Builder()
 .name("atlas-default-cluster-name")
-.displayName("Atlas Default Cluster Name")
-.description("Cluster name for Atlas entities reported by this 
ReportingTask." +
-" If not specified, 'atlas.cluster.name' in Atlas 
Configuration File is used." +
-" Cluster name mappings can be configured by user defined 
properties." +
-" See additional detail for detail.")
+.displayName("Atlas Default Metadata Namespace")
+.description("Namespace for Atlas entities reported by this 
ReportingTask." +
+" If not specified, 'atlas.metadata.namespace' or 
'atlas.cluster.name' (the former having priority) in Atlas Configuration File 
is used." +
+" Multiple mappings can be configured by user defined 
properties." +
+" See additional detail for more.")
 
 Review comment:
   Additional Details


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] turcsanyip commented on a change in pull request #4213: NIFI-7280 Use Atlas metadata namespace in Atlas Reporting Task

[GitBox]

turcsanyip commented on a change in pull request #4213: NIFI-7280 Use Atlas 
metadata namespace in Atlas Reporting Task
URL: https://github.com/apache/nifi/pull/4213#discussion_r409530555
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task/src/main/java/org/apache/nifi/atlas/reporting/ReportLineageToAtlas.java
 ##
 @@ -333,30 +339,34 @@
 @Override
 protected List getSupportedPropertyDescriptors() {
 final List properties = new ArrayList<>();
+// Basic atlas config
 properties.add(ATLAS_URLS);
-properties.add(ATLAS_CONNECT_TIMEOUT);
-properties.add(ATLAS_READ_TIMEOUT);
-properties.add(ATLAS_AUTHN_METHOD);
-properties.add(ATLAS_USER);
-properties.add(ATLAS_PASSWORD);
-properties.add(ATLAS_CONF_DIR);
-properties.add(ATLAS_NIFI_URL);
 properties.add(ATLAS_DEFAULT_CLUSTER_NAME);
-properties.add(NIFI_LINEAGE_STRATEGY);
-properties.add(PROVENANCE_START_POSITION);
-properties.add(PROVENANCE_BATCH_SIZE);
-properties.add(SSL_CONTEXT_SERVICE);
+properties.add(ATLAS_CONF_DIR);
+properties.add(ATLAS_CONF_CREATE);
 
 // Following properties are required if ATLAS_CONF_CREATE is enabled.
 // Otherwise should be left blank.
-properties.add(ATLAS_CONF_CREATE);
+// Will be used by the atlas client by reading the values from the 
atlas config file
+properties.add(ATLAS_CONNECT_TIMEOUT);
+properties.add(ATLAS_READ_TIMEOUT);
+properties.add(SSL_CONTEXT_SERVICE);
 properties.add(KERBEROS_CREDENTIALS_SERVICE);
 properties.add(NIFI_KERBEROS_PRINCIPAL);
 properties.add(NIFI_KERBEROS_KEYTAB);
 properties.add(KAFKA_KERBEROS_SERVICE_NAME);
 properties.add(KAFKA_BOOTSTRAP_SERVERS);
 properties.add(KAFKA_SECURITY_PROTOCOL);
 
+// General config used by the processor
+properties.add(NIFI_LINEAGE_STRATEGY);
 
 Review comment:
   Similar to the Kerberos properties: the NiFi prefix could be removed from 
the display name.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] turcsanyip commented on a change in pull request #4213: NIFI-7280 Use Atlas metadata namespace in Atlas Reporting Task

[GitBox]

turcsanyip commented on a change in pull request #4213: NIFI-7280 Use Atlas 
metadata namespace in Atlas Reporting Task
URL: https://github.com/apache/nifi/pull/4213#discussion_r409546060
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task/src/main/resources/docs/org.apache.nifi.atlas.reporting.ReportLineageToAtlas/additionalDetails.html
 ##
 @@ -27,7 +27,7 @@ ReportLineageToAtlas
 
 Information reported to Atlas
 NiFi Atlas Types
-Cluster Name Resolution
+Namespaces (Formerly Cluster Name 
Resolution)
 
 Review comment:
   formerly


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] turcsanyip commented on a change in pull request #4213: NIFI-7280 Use Atlas metadata namespace in Atlas Reporting Task

[GitBox]

turcsanyip commented on a change in pull request #4213: NIFI-7280 Use Atlas 
metadata namespace in Atlas Reporting Task
URL: https://github.com/apache/nifi/pull/4213#discussion_r409536696
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task/src/main/java/org/apache/nifi/atlas/reporting/ReportLineageToAtlas.java
 ##
 @@ -585,12 +594,58 @@ private void initAtlasProperties(ConfigurationContext 
context) throws IOExceptio
 if (confDir != null) {
 // If atlasConfDir is not set, atlas-application.properties will 
be searched under classpath.
 Properties props = System.getProperties();
-final String atlasConfProp = "atlas.conf";
+final String atlasConfProp = 
ApplicationProperties.ATLAS_CONFIGURATION_DIRECTORY_PROPERTY;
 props.setProperty(atlasConfProp, confDir.getAbsolutePath());
 getLogger().debug("{} has been set to: {}", new 
Object[]{atlasConfProp, props.getProperty(atlasConfProp)});
 }
 }
 
+private void parseAtlasUrls(final PropertyValue atlasUrlsProp, final 
Consumer urlStrConsumer) {
+setValue(
+value -> Arrays.stream(value.split(ATLAS_URL_DELIMITER))
+.map(String::trim)
+.forEach(urlStrConsumer),
+() -> {},
+atlasUrlsProp,
+atlasProperties.getProperty(ATLAS_PROPERTY_REST_ADDRESS)
+);
+}
+
+private void setValue(Consumer setter, Runnable emptyHandler, 
PropertyValue elPropertyValue, String... properties) {
+String value = StringSelector
+.of(elPropertyValue.evaluateAttributeExpressions().getValue())
+.or(properties)
+.toString();
+
+if (value == null || value.isEmpty()) {
+emptyHandler.run();
+} else {
+setter.accept(value);
+}
+}
+
+private void checkAtlasUrls(List urlStrings, ConfigurationContext 
context) {
+if (urlStrings.isEmpty()) {
+throw new ProcessException("No Atlas URL has been specified! Set 
either the '" + ATLAS_URLS.getDisplayName() + "' " +
+"property on the processor or the 'atlas.rest.address' 
porperty in the atlas configuration file.");
+}
+
+final boolean isSSLContextServiceSet = 
context.getProperty(SSL_CONTEXT_SERVICE).isSet();
 
 Review comment:
   Despite its description, the SSL Context Service property is used only for 
Kafka at the moment. This validation is not applicable for SSL Context Service 
/ HTTPS.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] turcsanyip commented on a change in pull request #4213: NIFI-7280 Use Atlas metadata namespace in Atlas Reporting Task

[GitBox]

turcsanyip commented on a change in pull request #4213: NIFI-7280 Use Atlas 
metadata namespace in Atlas Reporting Task
URL: https://github.com/apache/nifi/pull/4213#discussion_r409636601
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task/src/main/resources/docs/org.apache.nifi.atlas.reporting.ReportLineageToAtlas/additionalDetails.html
 ##
 @@ -121,40 +121,41 @@ NiFi Atlas Types
 
 
 
-Cluster Name Resolution
+Namespaces
 
 An entity in Atlas can be identified by its GUID for any existing 
objects, or type name and unique attribute can be used if GUID is not known. 
Qualified name is commonly used as the unique attribute.
-Since one Atlas instance can be used to manage multiple 
environments, i.e clusters, Atlas has to manage objects in different clusters 
those may have the same name. For example, a Hive table 'request_logs' in a 
'cluster-A' and 'cluster-B'. In such case, cluster name embedded in qualified 
names are crucial.
+One Atlas instance can be used to manage multiple environments and 
objects in different environments may have the same name. For example, a Hive 
table 'request_logs' in two different clusters, 'cluster-A' and 'cluster-B'. 
For this reason the qualified names contain a so-called metadata namespace.
+It's common practice to provide the cluster name as the namespace, 
but it can be any arbitrary string.
 
-For these requirements, a qualified name has 
'componentId@clusterName' format. E.g. A Hive table qualified name would be 
dbName.tableName@clusterName (default.request_logs@cluster-A).
+With this, a qualified name has 'componentId@namespace' format. 
E.g. A Hive table qualified name would be dbName.tableName@namespace 
(default.request_logs@cluster-A).
 
-From this NiFi reporting task standpoint, a cluster name is need to 
be resolved at following situations:
+From this NiFi reporting task standpoint, a namespace is needed to 
be resolved at following situations:
 
-To register NiFi component entities. Which cluster name 
should be used to represent the current NiFi cluster?
-To create lineages from NiFi component to other DataSets. 
Which cluster does the DataSet resides?
+To register NiFi component entities. Which namespace 
should be used to represent the current NiFi environment?
+To create lineages from NiFi component to other DataSets. 
Which environment does the DataSet resides in?
 
 
 
-To answer such questions, ReportLineageToAtlas reporting task 
provides a way to define mappings from ip address or hostname to a cluster name.
-The mapping can be defined by Dynamic Properties with a name in 
'hostnamePattern.ClusterName' format, having its value as a set of Regular 
Expression Patterns to match ip addresses or host names to a particular cluster 
name.
+To answer such questions, ReportLineageToAtlas reporting task 
provides a way to define mappings from ip address or hostname to a namespace.
+The mapping can be defined by Dynamic Properties with a name in 
'hostnamePattern.namespace' format, having its value as a set of Regular 
Expression Patterns to match ip addresses or host names to a particular cluster 
name.
 
 Review comment:
   "to a particular namespace." ?
   
   Could you please also fix this old typo: ip address => IP address (several 
times around this section)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] turcsanyip commented on a change in pull request #4213: NIFI-7280 Use Atlas metadata namespace in Atlas Reporting Task

[GitBox]

turcsanyip commented on a change in pull request #4213: NIFI-7280 Use Atlas 
metadata namespace in Atlas Reporting Task
URL: https://github.com/apache/nifi/pull/4213#discussion_r409524153
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task/src/main/java/org/apache/nifi/atlas/reporting/ReportLineageToAtlas.java
 ##
 @@ -333,30 +339,34 @@
 @Override
 protected List getSupportedPropertyDescriptors() {
 final List properties = new ArrayList<>();
+// Basic atlas config
 properties.add(ATLAS_URLS);
-properties.add(ATLAS_CONNECT_TIMEOUT);
-properties.add(ATLAS_READ_TIMEOUT);
-properties.add(ATLAS_AUTHN_METHOD);
-properties.add(ATLAS_USER);
-properties.add(ATLAS_PASSWORD);
-properties.add(ATLAS_CONF_DIR);
-properties.add(ATLAS_NIFI_URL);
 properties.add(ATLAS_DEFAULT_CLUSTER_NAME);
-properties.add(NIFI_LINEAGE_STRATEGY);
-properties.add(PROVENANCE_START_POSITION);
-properties.add(PROVENANCE_BATCH_SIZE);
-properties.add(SSL_CONTEXT_SERVICE);
+properties.add(ATLAS_CONF_DIR);
+properties.add(ATLAS_CONF_CREATE);
 
 // Following properties are required if ATLAS_CONF_CREATE is enabled.
 // Otherwise should be left blank.
-properties.add(ATLAS_CONF_CREATE);
+// Will be used by the atlas client by reading the values from the 
atlas config file
+properties.add(ATLAS_CONNECT_TIMEOUT);
+properties.add(ATLAS_READ_TIMEOUT);
+properties.add(SSL_CONTEXT_SERVICE);
 
 Review comment:
   It is only used for Kafka at the moment. I would move it down, after Kafka 
Security Protocol. Could you please also rename the property on the UI to 
"Kafka SSL Configuration" or something like that? Please also fix the 
description and remove Atlas from it. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] turcsanyip commented on a change in pull request #4213: NIFI-7280 Use Atlas metadata namespace in Atlas Reporting Task

[GitBox]

turcsanyip commented on a change in pull request #4213: NIFI-7280 Use Atlas 
metadata namespace in Atlas Reporting Task
URL: https://github.com/apache/nifi/pull/4213#discussion_r409539113
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task/src/main/java/org/apache/nifi/atlas/reporting/ReportLineageToAtlas.java
 ##
 @@ -585,12 +594,58 @@ private void initAtlasProperties(ConfigurationContext 
context) throws IOExceptio
 if (confDir != null) {
 // If atlasConfDir is not set, atlas-application.properties will 
be searched under classpath.
 Properties props = System.getProperties();
-final String atlasConfProp = "atlas.conf";
+final String atlasConfProp = 
ApplicationProperties.ATLAS_CONFIGURATION_DIRECTORY_PROPERTY;
 props.setProperty(atlasConfProp, confDir.getAbsolutePath());
 getLogger().debug("{} has been set to: {}", new 
Object[]{atlasConfProp, props.getProperty(atlasConfProp)});
 }
 }
 
+private void parseAtlasUrls(final PropertyValue atlasUrlsProp, final 
Consumer urlStrConsumer) {
+setValue(
+value -> Arrays.stream(value.split(ATLAS_URL_DELIMITER))
+.map(String::trim)
+.forEach(urlStrConsumer),
+() -> {},
+atlasUrlsProp,
+atlasProperties.getProperty(ATLAS_PROPERTY_REST_ADDRESS)
+);
+}
+
+private void setValue(Consumer setter, Runnable emptyHandler, 
PropertyValue elPropertyValue, String... properties) {
 
 Review comment:
   Seems to me a bit complicated too, but not insisted on changing it to 
something simpler.
   
   Minor: in case of the PropertyValue, it does not really matter whether it 
has EL or not, so it could be simply called `propertyValue` I think.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] turcsanyip commented on a change in pull request #4213: NIFI-7280 Use Atlas metadata namespace in Atlas Reporting Task

[GitBox]

turcsanyip commented on a change in pull request #4213: NIFI-7280 Use Atlas 
metadata namespace in Atlas Reporting Task
URL: https://github.com/apache/nifi/pull/4213#discussion_r409537271
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task/src/main/java/org/apache/nifi/atlas/reporting/ReportLineageToAtlas.java
 ##
 @@ -585,12 +594,58 @@ private void initAtlasProperties(ConfigurationContext 
context) throws IOExceptio
 if (confDir != null) {
 // If atlasConfDir is not set, atlas-application.properties will 
be searched under classpath.
 Properties props = System.getProperties();
-final String atlasConfProp = "atlas.conf";
+final String atlasConfProp = 
ApplicationProperties.ATLAS_CONFIGURATION_DIRECTORY_PROPERTY;
 props.setProperty(atlasConfProp, confDir.getAbsolutePath());
 getLogger().debug("{} has been set to: {}", new 
Object[]{atlasConfProp, props.getProperty(atlasConfProp)});
 }
 }
 
+private void parseAtlasUrls(final PropertyValue atlasUrlsProp, final 
Consumer urlStrConsumer) {
+setValue(
+value -> Arrays.stream(value.split(ATLAS_URL_DELIMITER))
+.map(String::trim)
+.forEach(urlStrConsumer),
+() -> {},
+atlasUrlsProp,
+atlasProperties.getProperty(ATLAS_PROPERTY_REST_ADDRESS)
+);
+}
+
+private void setValue(Consumer setter, Runnable emptyHandler, 
PropertyValue elPropertyValue, String... properties) {
+String value = StringSelector
+.of(elPropertyValue.evaluateAttributeExpressions().getValue())
+.or(properties)
+.toString();
+
+if (value == null || value.isEmpty()) {
+emptyHandler.run();
+} else {
+setter.accept(value);
+}
+}
+
+private void checkAtlasUrls(List urlStrings, ConfigurationContext 
context) {
+if (urlStrings.isEmpty()) {
+throw new ProcessException("No Atlas URL has been specified! Set 
either the '" + ATLAS_URLS.getDisplayName() + "' " +
+"property on the processor or the 'atlas.rest.address' 
porperty in the atlas configuration file.");
 
 Review comment:
   This could be moved to the emptyHandler of the property.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] turcsanyip commented on a change in pull request #4213: NIFI-7280 Use Atlas metadata namespace in Atlas Reporting Task

[GitBox]

turcsanyip commented on a change in pull request #4213: NIFI-7280 Use Atlas 
metadata namespace in Atlas Reporting Task
URL: https://github.com/apache/nifi/pull/4213#discussion_r409545352
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task/src/main/java/org/apache/nifi/atlas/reporting/ReportLineageToAtlas.java
 ##
 @@ -585,12 +594,58 @@ private void initAtlasProperties(ConfigurationContext 
context) throws IOExceptio
 if (confDir != null) {
 // If atlasConfDir is not set, atlas-application.properties will 
be searched under classpath.
 Properties props = System.getProperties();
-final String atlasConfProp = "atlas.conf";
+final String atlasConfProp = 
ApplicationProperties.ATLAS_CONFIGURATION_DIRECTORY_PROPERTY;
 props.setProperty(atlasConfProp, confDir.getAbsolutePath());
 getLogger().debug("{} has been set to: {}", new 
Object[]{atlasConfProp, props.getProperty(atlasConfProp)});
 }
 }
 
+private void parseAtlasUrls(final PropertyValue atlasUrlsProp, final 
Consumer urlStrConsumer) {
+setValue(
+value -> Arrays.stream(value.split(ATLAS_URL_DELIMITER))
+.map(String::trim)
+.forEach(urlStrConsumer),
+() -> {},
+atlasUrlsProp,
+atlasProperties.getProperty(ATLAS_PROPERTY_REST_ADDRESS)
+);
+}
+
+private void setValue(Consumer setter, Runnable emptyHandler, 
PropertyValue elPropertyValue, String... properties) {
+String value = StringSelector
+.of(elPropertyValue.evaluateAttributeExpressions().getValue())
+.or(properties)
+.toString();
+
+if (value == null || value.isEmpty()) {
 
 Review comment:
   Having a reference to the StringSelector and using `if (selector.found()) { 
... }` would be more elegant I think.
   Otherwise, it is less reasonable to use the StringSelector over 
`StringUtils.firstNonEmpty()` and then null check.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] turcsanyip commented on a change in pull request #4213: NIFI-7280 Use Atlas metadata namespace in Atlas Reporting Task

[GitBox]

turcsanyip commented on a change in pull request #4213: NIFI-7280 Use Atlas 
metadata namespace in Atlas Reporting Task
URL: https://github.com/apache/nifi/pull/4213#discussion_r409589604
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task/src/main/java/org/apache/nifi/atlas/reporting/ReportLineageToAtlas.java
 ##
 @@ -371,35 +381,32 @@ protected PropertyDescriptor 
getSupportedDynamicPropertyDescriptor(String proper
 return null;
 }
 
-private void parseAtlasUrls(final PropertyValue atlasUrlsProp, final 
Consumer urlStrConsumer) {
-final String atlasUrlsStr = 
atlasUrlsProp.evaluateAttributeExpressions().getValue();
-if (atlasUrlsStr != null && !atlasUrlsStr.isEmpty()) {
-Arrays.stream(atlasUrlsStr.split(","))
-.map(String::trim)
-.forEach(urlStrConsumer);
-}
-}
-
 @Override
 protected Collection customValidate(ValidationContext 
context) {
 final Collection results = new ArrayList<>();
 
 final boolean isSSLContextServiceSet = 
context.getProperty(SSL_CONTEXT_SERVICE).isSet();
 final ValidationResult.Builder invalidSSLService = new 
ValidationResult.Builder()
 .subject(SSL_CONTEXT_SERVICE.getDisplayName()).valid(false);
-parseAtlasUrls(context.getProperty(ATLAS_URLS), input -> {
-final ValidationResult.Builder builder = new 
ValidationResult.Builder().subject(ATLAS_URLS.getDisplayName()).input(input);
-try {
-final URL url = new URL(input);
-if ("https".equalsIgnoreCase(url.getProtocol()) && 
!isSSLContextServiceSet) {
-results.add(invalidSSLService.explanation("required by 
HTTPS Atlas access").build());
-} else {
-results.add(builder.explanation("Valid 
URI").valid(true).build());
-}
-} catch (Exception e) {
-results.add(builder.explanation("Contains invalid URI: " + 
e).valid(false).build());
-}
-});
+
+String atlasUrls = 
context.getProperty(ATLAS_URLS).evaluateAttributeExpressions().getValue();
+if (!StringUtils.isEmpty(atlasUrls)) {
+Arrays.stream(atlasUrls.split(ATLAS_URL_DELIMITER))
+.map(String::trim)
+.forEach(input -> {
+final ValidationResult.Builder builder = new 
ValidationResult.Builder().subject(ATLAS_URLS.getDisplayName()).input(input);
+try {
+final URL url = new URL(input);
+if ("https".equalsIgnoreCase(url.getProtocol()) && 
!isSSLContextServiceSet) {
 
 Review comment:
   SSL Context Service is not used for Atlas connection so this validation 
check can be removed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] turcsanyip commented on a change in pull request #4213: NIFI-7280 Use Atlas metadata namespace in Atlas Reporting Task

[GitBox]

turcsanyip commented on a change in pull request #4213: NIFI-7280 Use Atlas 
metadata namespace in Atlas Reporting Task
URL: https://github.com/apache/nifi/pull/4213#discussion_r409530073
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task/src/main/java/org/apache/nifi/atlas/reporting/ReportLineageToAtlas.java
 ##
 @@ -333,30 +339,34 @@
 @Override
 protected List getSupportedPropertyDescriptors() {
 final List properties = new ArrayList<>();
+// Basic atlas config
 properties.add(ATLAS_URLS);
-properties.add(ATLAS_CONNECT_TIMEOUT);
-properties.add(ATLAS_READ_TIMEOUT);
-properties.add(ATLAS_AUTHN_METHOD);
-properties.add(ATLAS_USER);
-properties.add(ATLAS_PASSWORD);
-properties.add(ATLAS_CONF_DIR);
-properties.add(ATLAS_NIFI_URL);
 properties.add(ATLAS_DEFAULT_CLUSTER_NAME);
-properties.add(NIFI_LINEAGE_STRATEGY);
-properties.add(PROVENANCE_START_POSITION);
-properties.add(PROVENANCE_BATCH_SIZE);
-properties.add(SSL_CONTEXT_SERVICE);
+properties.add(ATLAS_CONF_DIR);
+properties.add(ATLAS_CONF_CREATE);
 
 // Following properties are required if ATLAS_CONF_CREATE is enabled.
 // Otherwise should be left blank.
-properties.add(ATLAS_CONF_CREATE);
+// Will be used by the atlas client by reading the values from the 
atlas config file
+properties.add(ATLAS_CONNECT_TIMEOUT);
+properties.add(ATLAS_READ_TIMEOUT);
+properties.add(SSL_CONTEXT_SERVICE);
 properties.add(KERBEROS_CREDENTIALS_SERVICE);
 properties.add(NIFI_KERBEROS_PRINCIPAL);
 properties.add(NIFI_KERBEROS_KEYTAB);
 
 Review comment:
   Not sure why these properties have NIFI prefix while the Kerberos Service 
does not. Could you please remove the prefix on the UI? (I mean displayName 
only, and further code changes are not necessary)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] turcsanyip commented on a change in pull request #4213: NIFI-7280 Use Atlas metadata namespace in Atlas Reporting Task

[GitBox]

turcsanyip commented on a change in pull request #4213: NIFI-7280 Use Atlas 
metadata namespace in Atlas Reporting Task
URL: https://github.com/apache/nifi/pull/4213#discussion_r409506936
 
 

 ##
 File path: 
nifi-commons/nifi-utils/src/main/java/org/apache/nifi/util/StringSelector.java
 ##
 @@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.util;
+
+/**
+ * Fluent api for checking one or more strings and selecting the first 
non-empty one.
+ * 
+ * {@link #toString()} returns the first encountered non-empty string or "".
+ * 
+ * Optimized so that no intermediary objects are created, only one, once the 
first non-empty string is found.
+ */
+public interface StringSelector {
 
 Review comment:
   Apache Commons Lang3 `StringUtils.firstNonEmpty(T... values) / 
firstNonBlank(T... values)` would be fair enough for the current use case.
   It is an interesting approach nevertheless. So we can keep it as an 
alternative solution.
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] turcsanyip commented on a change in pull request #4213: NIFI-7280 Use Atlas metadata namespace in Atlas Reporting Task

[GitBox]

turcsanyip commented on a change in pull request #4213: NIFI-7280 Use Atlas 
metadata namespace in Atlas Reporting Task
URL: https://github.com/apache/nifi/pull/4213#discussion_r409531548
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task/src/main/java/org/apache/nifi/atlas/reporting/ReportLineageToAtlas.java
 ##
 @@ -333,30 +339,34 @@
 @Override
 protected List getSupportedPropertyDescriptors() {
 final List properties = new ArrayList<>();
+// Basic atlas config
 properties.add(ATLAS_URLS);
-properties.add(ATLAS_CONNECT_TIMEOUT);
-properties.add(ATLAS_READ_TIMEOUT);
-properties.add(ATLAS_AUTHN_METHOD);
-properties.add(ATLAS_USER);
-properties.add(ATLAS_PASSWORD);
-properties.add(ATLAS_CONF_DIR);
-properties.add(ATLAS_NIFI_URL);
 properties.add(ATLAS_DEFAULT_CLUSTER_NAME);
-properties.add(NIFI_LINEAGE_STRATEGY);
-properties.add(PROVENANCE_START_POSITION);
-properties.add(PROVENANCE_BATCH_SIZE);
-properties.add(SSL_CONTEXT_SERVICE);
+properties.add(ATLAS_CONF_DIR);
+properties.add(ATLAS_CONF_CREATE);
 
 // Following properties are required if ATLAS_CONF_CREATE is enabled.
 // Otherwise should be left blank.
-properties.add(ATLAS_CONF_CREATE);
+// Will be used by the atlas client by reading the values from the 
atlas config file
+properties.add(ATLAS_CONNECT_TIMEOUT);
+properties.add(ATLAS_READ_TIMEOUT);
+properties.add(SSL_CONTEXT_SERVICE);
 properties.add(KERBEROS_CREDENTIALS_SERVICE);
 properties.add(NIFI_KERBEROS_PRINCIPAL);
 properties.add(NIFI_KERBEROS_KEYTAB);
 properties.add(KAFKA_KERBEROS_SERVICE_NAME);
 
 Review comment:
   This property depends on KAFKA_SECURITY_PROTOCOL property so I would rather 
move it after that one.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] turcsanyip commented on a change in pull request #4213: NIFI-7280 Use Atlas metadata namespace in Atlas Reporting Task

[GitBox]

turcsanyip commented on a change in pull request #4213: NIFI-7280 Use Atlas 
metadata namespace in Atlas Reporting Task
URL: https://github.com/apache/nifi/pull/4213#discussion_r409521875
 
 

 ##
 File path: 
nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task/src/main/java/org/apache/nifi/atlas/reporting/ReportLineageToAtlas.java
 ##
 @@ -333,30 +339,34 @@
 @Override
 protected List getSupportedPropertyDescriptors() {
 final List properties = new ArrayList<>();
+// Basic atlas config
 properties.add(ATLAS_URLS);
-properties.add(ATLAS_CONNECT_TIMEOUT);
-properties.add(ATLAS_READ_TIMEOUT);
-properties.add(ATLAS_AUTHN_METHOD);
-properties.add(ATLAS_USER);
-properties.add(ATLAS_PASSWORD);
-properties.add(ATLAS_CONF_DIR);
-properties.add(ATLAS_NIFI_URL);
 properties.add(ATLAS_DEFAULT_CLUSTER_NAME);
-properties.add(NIFI_LINEAGE_STRATEGY);
-properties.add(PROVENANCE_START_POSITION);
-properties.add(PROVENANCE_BATCH_SIZE);
-properties.add(SSL_CONTEXT_SERVICE);
+properties.add(ATLAS_CONF_DIR);
+properties.add(ATLAS_CONF_CREATE);
 
 // Following properties are required if ATLAS_CONF_CREATE is enabled.
 // Otherwise should be left blank.
-properties.add(ATLAS_CONF_CREATE);
+// Will be used by the atlas client by reading the values from the 
atlas config file
+properties.add(ATLAS_CONNECT_TIMEOUT);
+properties.add(ATLAS_READ_TIMEOUT);
+properties.add(SSL_CONTEXT_SERVICE);
 properties.add(KERBEROS_CREDENTIALS_SERVICE);
 properties.add(NIFI_KERBEROS_PRINCIPAL);
 properties.add(NIFI_KERBEROS_KEYTAB);
 properties.add(KAFKA_KERBEROS_SERVICE_NAME);
 properties.add(KAFKA_BOOTSTRAP_SERVERS);
 properties.add(KAFKA_SECURITY_PROTOCOL);
 
+// General config used by the processor
+properties.add(NIFI_LINEAGE_STRATEGY);
+properties.add(PROVENANCE_START_POSITION);
+properties.add(PROVENANCE_BATCH_SIZE);
+properties.add(ATLAS_AUTHN_METHOD);
+properties.add(ATLAS_USER);
+properties.add(ATLAS_PASSWORD);
 
 Review comment:
   The auth parameters are quite important / always need to be set.
   I'd suggest moving them up, after the Atlas URL or just before the Kerberos 
params. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (NIFI-7369) Fixing precision of floating point numbers when reading Avro record

[Simon Bence (Jira)]

Simon Bence created NIFI-7369:
-

 Summary: Fixing precision of floating point numbers when reading 
Avro record
 Key: NIFI-7369
 URL: https://issues.apache.org/jira/browse/NIFI-7369
 Project: Apache NiFi
  Issue Type: Bug
  Components: Extensions
Affects Versions: 1.9.2
Reporter: Simon Bence
Assignee: Simon Bence
 Fix For: 1.12.0


h3. Summary:

When ConvertRecord reads in decimal (logical) data type from Avro record, it 
converts it into Java Double, which depending on the decimal precision, might 
cause precision loss.
h3. In details:

AbstractRecordProcessor (parent of ConvertRecord) works in a way it reads in 
record with RecordReader and writes it out with RecordSetWriter. Between the 
two steps, we have an internal representation uses RecordFieldTypes. In this 
example, the following transformations are happening: AVRO into internal 
Record. Internal Record into the format defined by the writer. The specific 
reader where the issue comes up is the AvroReaderWithEmbeddedSchema. This 
(based on the embedded schema) tries to figure out what could be the proper 
internal representation for the given fields, using 
AvroTypeUtil#determineDataType. This util, with the type definition specifies 
decimal [ends 
up|https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/main/java/org/apache/nifi/avro/AvroTypeUtil.java#L343-L346]
 using double (the originally read data with the reader is BigDecimal and it is 
converted). Double is not capable to correctly represent the incoming data when 
it is high precision so in further steps, a less accurate number will be used 
and written into the output.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi-minifi-cpp] szaszm commented on issue #697: MINIFICPP-1104 - SSL docker tests for PublishKafka

[GitBox]

szaszm commented on issue #697: MINIFICPP-1104 - SSL docker tests for 
PublishKafka
URL: https://github.com/apache/nifi-minifi-cpp/pull/697#issuecomment-614629091
 
 
   No, but I may have done something wrong. The tests left containers and 
networks behind that I had to manually delete to be able to rerun them.
   The result on master:
   ```
   

 short test summary info 

   FAILED ../docker/test/integration/test_rdkafka.py::test_publish_kafka - 
assert False
   FAILED ../docker/test/integration/test_rdkafka.py::test_broker_on_off - 
AssertionError: assert False
   

 2 failed, 8 passed in 326.46s (0:05:26) 

   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (NIFI-7368) MoveHDFS - flow-files disapper when Concurrent Tasks > 1

[doron (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-7368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

doron updated NIFI-7368:

Environment: CDH 5.11.0

> MoveHDFS - flow-files disapper when Concurrent Tasks > 1
> 
>
> Key: NIFI-7368
> URL: https://issues.apache.org/jira/browse/NIFI-7368
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.11.4
> Environment: CDH 5.11.0
>Reporter: doron
>Priority: Major
>
> Hi, 
> I encounter a bug when using MoveHDFS with Concurrent Tasks > 1.
> it seems that some flow files disappear (Not routed to failure).
>  
> +*Test:*+
> Create an HDFS folder FILES_A in with 10 files
> ListHDFS with path to FILES_A
> MoveHDFS with Concurrent Tasks = 5 Source: ${path}/${filename} destination: 
> path to FILES_B
> In my test only 3 files moved from FILES_A to FILES_B
> Routes: Failures = 0, sucessfull = 3
> MoveHDFS - In = 10, out = 3
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (NIFI-7368) MoveHDFS - flow-files disapper when Concurrent Tasks > 1

[doron (Jira)]

doron created NIFI-7368:
---

 Summary: MoveHDFS - flow-files disapper when Concurrent Tasks > 1
 Key: NIFI-7368
 URL: https://issues.apache.org/jira/browse/NIFI-7368
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core Framework
Affects Versions: 1.11.4
Reporter: doron


Hi, 

I encounter a bug when using MoveHDFS with Concurrent Tasks > 1.

it seems that some flow files disappear (Not routed to failure).

 

+*Test:*+

Create an HDFS folder FILES_A in with 10 files

ListHDFS with path to FILES_A

MoveHDFS with Concurrent Tasks = 5 Source: ${path}/${filename} destination: 
path to FILES_B
In my test only 3 files moved from FILES_A to FILES_B

Routes: Failures = 0, sucessfull = 3

MoveHDFS - In = 10, out = 3

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi] ottobackwards commented on issue #4201: NIFI-7348 Wait - Removes WAIT_START_TIMESTAMP after expiration

[GitBox]

ottobackwards commented on issue #4201: NIFI-7348 Wait - Removes 
WAIT_START_TIMESTAMP after expiration
URL: https://github.com/apache/nifi/pull/4201#issuecomment-614604897
 
 
   Thanks @ijokarumawak!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] nghiaxlee commented on issue #697: MINIFICPP-1104 - SSL docker tests for PublishKafka

[GitBox]

nghiaxlee commented on issue #697: MINIFICPP-1104 - SSL docker tests for 
PublishKafka
URL: https://github.com/apache/nifi-minifi-cpp/pull/697#issuecomment-614596853
 
 
   I see, will dig into this, were the warnings/errors in master the same as 
this PR?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] szaszm edited a comment on issue #697: MINIFICPP-1104 - SSL docker tests for PublishKafka

[GitBox]

szaszm edited a comment on issue #697: MINIFICPP-1104 - SSL docker tests for 
PublishKafka
URL: https://github.com/apache/nifi-minifi-cpp/pull/697#issuecomment-614573840
 
 
   > @szaszm : I don't think docker tests have ever worked on Ubuntu, they 
surely didn't work for me on Debian when I last tried.
   > What's the result if you run them on master without this commit?
   
   Rdkafka docker tests fail on master, too. The rest of the docker tests work 
fine for me. Tried on Gentoo Linux ~amd64 (testing branch).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] szaszm commented on issue #697: MINIFICPP-1104 - SSL docker tests for PublishKafka

[GitBox]

szaszm commented on issue #697: MINIFICPP-1104 - SSL docker tests for 
PublishKafka
URL: https://github.com/apache/nifi-minifi-cpp/pull/697#issuecomment-614573840
 
 
   > @szaszm : I don't think docker tests have ever worked on Ubuntu, they 
surely didn't work for me on Debian when I last tried.
   > What's the result if you run them on master without this commit?
   
   The rdkafka docker tests fail on master, too. The rest of the docker tests 
work fine for me. Tried on Gentoo Linux ~amd64 (testing branch).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] szaszm edited a comment on issue #697: MINIFICPP-1104 - SSL docker tests for PublishKafka

[GitBox]

szaszm edited a comment on issue #697: MINIFICPP-1104 - SSL docker tests for 
PublishKafka
URL: https://github.com/apache/nifi-minifi-cpp/pull/697#issuecomment-614573840
 
 
   > @szaszm : I don't think docker tests have ever worked on Ubuntu, they 
surely didn't work for me on Debian when I last tried.
   > What's the result if you run them on master without this commit?
   
   The rest of the rdkafka docker tests fail on master, too. The rest of the 
docker tests work fine for me. Tried on Gentoo Linux ~amd64 (testing branch).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (NIFI-7334) FetchAzureDataLakeStorage processor to provide native fetch support for Azure Data lake Gen 2 Storage

[Peter Turcsanyi (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-7334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Turcsanyi resolved NIFI-7334.
---
Resolution: Fixed

> FetchAzureDataLakeStorage processor to provide native fetch support for Azure 
> Data lake Gen 2 Storage
> -
>
> Key: NIFI-7334
> URL: https://issues.apache.org/jira/browse/NIFI-7334
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.12.0
>Reporter: Muazma Zahid
>Assignee: Muazma Zahid
>Priority: Major
>  Labels: azure
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> [Azure Data Lake Storage 
> Gen2|[https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction]]
>  makes Azure Storage the foundation for building enterprise data lakes on 
> Azure. NiFi supports Azure Blob storage but has no native support for ADLS 
> Gen 2
> This processor will be followed by List to provide native support for ADLS 
> Gen 2



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (NIFI-7334) FetchAzureDataLakeStorage processor to provide native fetch support for Azure Data lake Gen 2 Storage

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-7334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17084721#comment-17084721
 ] 

ASF subversion and git services commented on NIFI-7334:
---

Commit 58118cf904cfb58ea119e5507a9a9549cda53bd9 in nifi's branch 
refs/heads/master from muazmaz
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=58118cf ]

NIFI-7334 Adding FetchDataLakeStorage Processor to provide native support for 
Azure Data lake Gen 2 Storage.

NIFI-7334 Update to FetchDataLakeStorage Processor

This closes #4212.

Signed-off-by: Peter Turcsanyi 


> FetchAzureDataLakeStorage processor to provide native fetch support for Azure 
> Data lake Gen 2 Storage
> -
>
> Key: NIFI-7334
> URL: https://issues.apache.org/jira/browse/NIFI-7334
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.12.0
>Reporter: Muazma Zahid
>Assignee: Muazma Zahid
>Priority: Major
>  Labels: azure
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> [Azure Data Lake Storage 
> Gen2|[https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction]]
>  makes Azure Storage the foundation for building enterprise data lakes on 
> Azure. NiFi supports Azure Blob storage but has no native support for ADLS 
> Gen 2
> This processor will be followed by List to provide native support for ADLS 
> Gen 2



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (NIFI-7334) FetchAzureDataLakeStorage processor to provide native fetch support for Azure Data lake Gen 2 Storage

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-7334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17084720#comment-17084720
 ] 

ASF subversion and git services commented on NIFI-7334:
---

Commit 58118cf904cfb58ea119e5507a9a9549cda53bd9 in nifi's branch 
refs/heads/master from muazmaz
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=58118cf ]

NIFI-7334 Adding FetchDataLakeStorage Processor to provide native support for 
Azure Data lake Gen 2 Storage.

NIFI-7334 Update to FetchDataLakeStorage Processor

This closes #4212.

Signed-off-by: Peter Turcsanyi 


> FetchAzureDataLakeStorage processor to provide native fetch support for Azure 
> Data lake Gen 2 Storage
> -
>
> Key: NIFI-7334
> URL: https://issues.apache.org/jira/browse/NIFI-7334
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.12.0
>Reporter: Muazma Zahid
>Assignee: Muazma Zahid
>Priority: Major
>  Labels: azure
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> [Azure Data Lake Storage 
> Gen2|[https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction]]
>  makes Azure Storage the foundation for building enterprise data lakes on 
> Azure. NiFi supports Azure Blob storage but has no native support for ADLS 
> Gen 2
> This processor will be followed by List to provide native support for ADLS 
> Gen 2



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi] asfgit closed pull request #4212: NIFI-7334 Adding FetchDataLakeStorage Processor to provide native support for Azure Data lake Gen 2 Storage

[GitBox]

asfgit closed pull request #4212: NIFI-7334 Adding FetchDataLakeStorage 
Processor to provide native support for Azure Data lake Gen 2 Storage
URL: https://github.com/apache/nifi/pull/4212
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (NIFI-7367) Add tests for FetchAzureDataLakeStorage

[Peter Turcsanyi (Jira)]

Peter Turcsanyi created NIFI-7367:
-

 Summary: Add tests for FetchAzureDataLakeStorage
 Key: NIFI-7367
 URL: https://issues.apache.org/jira/browse/NIFI-7367
 Project: Apache NiFi
  Issue Type: Test
  Components: Extensions
Reporter: Peter Turcsanyi






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #743: Minificpp 1169 - Simplify C2 metrics collection and reporting

[GitBox]

arpadboda commented on a change in pull request #743: Minificpp 1169 - Simplify 
C2 metrics collection and reporting
URL: https://github.com/apache/nifi-minifi-cpp/pull/743#discussion_r409396297
 
 

 ##
 File path: libminifi/src/c2/C2Agent.cpp
 ##
 @@ -89,44 +93,68 @@ C2Agent::C2Agent(const 
std::shared_ptr heart_beat_period_ ) {
-last_run_ = now;
-try {
-  performHeartBeat();
-}
-catch(const std::exception &e) {
-  logger_->log_error("Exception occurred while performing heartbeat. 
error: %s", e.what());
-}
-catch(...) {
-  logger_->log_error("Unknonwn exception occurred while performing 
heartbeat.");
-}
+  try {
+performHeartBeat();
+  }
+  catch(const std::exception &e) {
+logger_->log_error("Exception occurred while performing heartbeat. 
error: %s", e.what());
+  }
+  catch(...) {
+logger_->log_error("Unknonwn exception occurred while performing 
heartbeat.");
   }
 
   checkTriggers();
 
-  std::this_thread::sleep_for(std::chrono::milliseconds(heart_beat_period_ 
> 500 ? 500 : heart_beat_period_));
-  return 
state::Update(state::UpdateStatus(state::UpdateState::READ_COMPLETE, false));
+  return 
utils::TaskRescheduleInfo::RetryIn(std::chrono::milliseconds(heart_beat_period_));
 };
-
   functions_.push_back(c2_producer_);
 
   c2_consumer_ = [&]() {
 auto now = std::chrono::steady_clock::now();
 if ( queue_mutex.try_lock_until(now + std::chrono::seconds(1)) ) {
-  if (responses.size() > 0) {
-const C2Payload payload(std::move(responses.back()));
-responses.pop_back();
-extractPayload(std::move(payload));
+  if (responses.empty()) {
+queue_mutex.unlock();
+return utils::TaskRescheduleInfo::RetryImmediately();
 
 Review comment:
   RetryImmediately was introduced for a reason: when something takes quite 
long (an ontrigger call of a processor for eg), it might happen that the next 
schedule is already in the past. Although we cannot continue executing it as 
that would result in starvation. 
   In this case it's fair to reschedule immediately. 
   This means that the given task is put back to the end of the queue, so in 
case there are tasks already waiting for CPU, those are executed first, 
otherwise this one can be continued immediately without unnecessary wait. 
   
   In this current aspect, you are right, it shouldn't be used for polling 
something, but otherwise it's something we need.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #743: Minificpp 1169 - Simplify C2 metrics collection and reporting

[GitBox]

arpadboda commented on a change in pull request #743: Minificpp 1169 - Simplify 
C2 metrics collection and reporting
URL: https://github.com/apache/nifi-minifi-cpp/pull/743#discussion_r409396297
 
 

 ##
 File path: libminifi/src/c2/C2Agent.cpp
 ##
 @@ -89,44 +93,68 @@ C2Agent::C2Agent(const 
std::shared_ptr heart_beat_period_ ) {
-last_run_ = now;
-try {
-  performHeartBeat();
-}
-catch(const std::exception &e) {
-  logger_->log_error("Exception occurred while performing heartbeat. 
error: %s", e.what());
-}
-catch(...) {
-  logger_->log_error("Unknonwn exception occurred while performing 
heartbeat.");
-}
+  try {
+performHeartBeat();
+  }
+  catch(const std::exception &e) {
+logger_->log_error("Exception occurred while performing heartbeat. 
error: %s", e.what());
+  }
+  catch(...) {
+logger_->log_error("Unknonwn exception occurred while performing 
heartbeat.");
   }
 
   checkTriggers();
 
-  std::this_thread::sleep_for(std::chrono::milliseconds(heart_beat_period_ 
> 500 ? 500 : heart_beat_period_));
-  return 
state::Update(state::UpdateStatus(state::UpdateState::READ_COMPLETE, false));
+  return 
utils::TaskRescheduleInfo::RetryIn(std::chrono::milliseconds(heart_beat_period_));
 };
-
   functions_.push_back(c2_producer_);
 
   c2_consumer_ = [&]() {
 auto now = std::chrono::steady_clock::now();
 if ( queue_mutex.try_lock_until(now + std::chrono::seconds(1)) ) {
-  if (responses.size() > 0) {
-const C2Payload payload(std::move(responses.back()));
-responses.pop_back();
-extractPayload(std::move(payload));
+  if (responses.empty()) {
+queue_mutex.unlock();
+return utils::TaskRescheduleInfo::RetryImmediately();
 
 Review comment:
   RetryImmediately was introduced for a reason: when something takes quite 
long (an ontrigger call of a processor for eg), it might happen that the next 
schedule is already in the past. Although we cannot continue executing it as 
that would result in starvation. 
   In this case it's fair to reschedule immediately. 
   This means that the given task is put back to the end of the queue, so in 
case there are tasks already waiting for CPU, those are executed first, 
otherwise this one can be continued immediately without unnecessary wait. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [nifi] ijokarumawak commented on issue #4072: NIFI-7182

[GitBox]

ijokarumawak commented on issue #4072: NIFI-7182
URL: https://github.com/apache/nifi/pull/4072#issuecomment-614473511
 
 
   @MikeThomsen @joewitt 
   
   The test is using Solr range facetting which depends on system date. 
Specifically the test uses `facet.range.start=NOW/MINUTE` and 
`facet.range.end=NOW/MINUTE+1MINUTE`.
   
   That range condition can be fail if the minutes when the processor executes 
the query, and the date field value are different. That can be happen by a long 
GC pause or in an extremely bad timing.
   For example, if `createSolrClient()` method creates the doc at 
`10:23:59.999`, and the query is executed at `10:24:001`, then the facet range 
would be `10:24 to 10:25` which does not query documents created at `10:23`.
   
   I was able to reproduce the assertion error by addint `Thread.sleep(60_000)` 
at the `testAllFacet Categories` method between it creates `solorClient` and 
`runner.run`.
   
   Since the test does not have to filter documents by date range, we can make 
the range wider (cannot remove the range as Solr requires it). I suggest 
changing the facet range as follows:
   ```
   runner.setProperty("facet.range.start", "NOW/MINUTE-3MINUTE");
   runner.setProperty("facet.range.end", "NOW/MINUTE+3MINUTE");
   ```
   
   This makes the test passing even with sleeping 1 minute between doc creation 
and query execution.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Resolved] (NIFI-7348) FlowFiles re-entering a Wait-processor after they've expired expire immediatelly

[Koji Kawamura (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-7348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Koji Kawamura resolved NIFI-7348.
-
Fix Version/s: 1.12.0
   Resolution: Fixed

> FlowFiles re-entering a Wait-processor after they've expired expire 
> immediatelly
> 
>
> Key: NIFI-7348
> URL: https://issues.apache.org/jira/browse/NIFI-7348
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.11.4
> Environment: Windows 10 / Ubuntu
>Reporter: endzeit
>Assignee: endzeit
>Priority: Major
>  Labels: easyfix
> Fix For: 1.12.0
>
> Attachments: Wait_processor_expiration_issue.xml
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> We recently noticed a behaviour of the Wait processor that we thought of to 
> be a bug.
>  
> As the attribute WAIT_START_TIMESTAMP is only removed once the FlowFile 
> leaves the processor successfully or failing, it affects FlowFiles that 
> expire the EXPIRATION_DURATION and re-enter the processor.
> In case the FlowFile enters the same processor again - after expiring 
> beforehand - it is transported to the expired output immediately, without 
> waiting for the EXPIRATION_DURATION again.
> Is this desired behaviour? 
>  
> I'll attach a very simple demonstration. Just let it run a minute or two and 
> look at the FlowFile attribute "counter" afterwards.
>  
> There has been a pull-request addressing a similar issue (NIFI-5892), which 
> resulted in the attribute being removed after success and failure. This case 
> just seems to haven't been thought about back then. Or was there a reason to 
> not clear the attribute after expiration? I couldn't find a mention regarding 
> expiration in the issue.
>  
> As this should be a very easy fix I would love to contribute, once you 
> confirm this is not intentional. 
>  
> *Current workaround:*
> simply remove the attribute WAIT_START_TIMESTAMP after the FlowFile leaves 
> the Wait processor, e.g. using an UpdateAttribute processor 
>  
> *Edit 2020-04-13:*
> Also this seems to have the side effect of NOT documenting the repeated 
> processing. There is no provenance entry added when re-entering the processor 
> and expiring immediately, leading to the error being harder to trace.
> Because of this I reset the priority to "Major", which seems to be the 
> default anyway.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (NIFI-7348) FlowFiles re-entering a Wait-processor after they've expired expire immediatelly

[Koji Kawamura (Jira)]

[ 
https://issues.apache.org/jira/browse/NIFI-7348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17084594#comment-17084594
 ] 

Koji Kawamura commented on NIFI-7348:
-

Thanks [~EndzeitBegins] for the detailed explanation and fixing this! I've 
reviewed the PR and merged it to master.

> FlowFiles re-entering a Wait-processor after they've expired expire 
> immediatelly
> 
>
> Key: NIFI-7348
> URL: https://issues.apache.org/jira/browse/NIFI-7348
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.11.4
> Environment: Windows 10 / Ubuntu
>Reporter: endzeit
>Assignee: endzeit
>Priority: Major
>  Labels: easyfix
> Attachments: Wait_processor_expiration_issue.xml
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> We recently noticed a behaviour of the Wait processor that we thought of to 
> be a bug.
>  
> As the attribute WAIT_START_TIMESTAMP is only removed once the FlowFile 
> leaves the processor successfully or failing, it affects FlowFiles that 
> expire the EXPIRATION_DURATION and re-enter the processor.
> In case the FlowFile enters the same processor again - after expiring 
> beforehand - it is transported to the expired output immediately, without 
> waiting for the EXPIRATION_DURATION again.
> Is this desired behaviour? 
>  
> I'll attach a very simple demonstration. Just let it run a minute or two and 
> look at the FlowFile attribute "counter" afterwards.
>  
> There has been a pull-request addressing a similar issue (NIFI-5892), which 
> resulted in the attribute being removed after success and failure. This case 
> just seems to haven't been thought about back then. Or was there a reason to 
> not clear the attribute after expiration? I couldn't find a mention regarding 
> expiration in the issue.
>  
> As this should be a very easy fix I would love to contribute, once you 
> confirm this is not intentional. 
>  
> *Current workaround:*
> simply remove the attribute WAIT_START_TIMESTAMP after the FlowFile leaves 
> the Wait processor, e.g. using an UpdateAttribute processor 
>  
> *Edit 2020-04-13:*
> Also this seems to have the side effect of NOT documenting the repeated 
> processing. There is no provenance entry added when re-entering the processor 
> and expiring immediately, leading to the error being harder to trace.
> Because of this I reset the priority to "Major", which seems to be the 
> default anyway.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)