subject:"\[GitHub\] \[nifi\] jfrazee commented on a change in pull request #4253\: NIFI\-7406\: PutAzureCosmosRecord Processor"

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

2020-11-03 Thread GitBox



jfrazee commented on a change in pull request #4253:
URL: https://github.com/apache/nifi/pull/4253#discussion_r516238042



##
File path: 
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/cosmos/document/PutAzureCosmosDBRecord.java
##
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.cosmos.document;
+
+import com.azure.cosmos.CosmosContainer;
+import com.azure.cosmos.CosmosException;
+import com.azure.cosmos.implementation.ConflictException;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SystemResource;
+import org.apache.nifi.annotation.behavior.SystemResourceConsideration;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.serialization.MalformedRecordException;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.util.DataTypeUtils;
+
+@EventDriven
+@Tags({ "azure", "cosmos", "insert", "record", "put" })
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@CapabilityDescription("This processor is a record-aware processor for 
inserting data into Cosmos DB with Core SQL API. It uses a configured record 
reader and " +
+"schema to read an incoming record set from the body of a Flowfile and 
then inserts those records into " +
+"a configured Cosmos DB Container.")
+@SystemResourceConsideration(resource = SystemResource.MEMORY)
+public class PutAzureCosmosDBRecord extends AbstractAzureCosmosDBProcessor {
+
+private String conflictHandlingStrategy;
+static final AllowableValue IGNORE_CONFLICT = new AllowableValue("ignore", 
"Ignore", "Conflicting records will not be inserted, and FlowFile will not be 
routed to failure");
+static final AllowableValue UPSERT_CONFLICT = new AllowableValue("upsert", 
"Upsert", "Conflicting records will be upserted, and FlowFile will not be 
routed to failure");
+
+static final PropertyDescriptor RECORD_READER_FACTORY = new 
PropertyDescriptor.Builder()
+.name("record-reader")
+.displayName("Record Reader")
+.description("Specifies the Controller Service to use for parsing 
incoming data and determining the data's schema")
+.identifiesControllerService(RecordReaderFactory.class)
+.required(true)
+.build();
+
+static final PropertyDescriptor INSERT_BATCH_SIZE = new 
PropertyDescriptor.Builder()
+.name("insert-batch-size")
+.displayName("Insert Batch Size")
+.description("The number of records to group together for one single 
insert operation against Cosmos DB")
+.defaultValue("20")
+.required(false)
+.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+.build();
+
+static final PropertyDescriptor CONFLICT_HANDLE_STRATEGY = new 
PropertyDescriptor.Builder()
+

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

2020-10-23 Thread GitBox



jfrazee commented on a change in pull request #4253:
URL: https://github.com/apache/nifi/pull/4253#discussion_r510490268



##
File path: 
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/cosmos/document/PutAzureCosmosDBRecord.java
##
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.cosmos.document;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+
+import com.azure.cosmos.CosmosContainer;
+import com.azure.cosmos.CosmosException;
+import com.azure.cosmos.implementation.ConflictException;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SystemResource;
+import org.apache.nifi.annotation.behavior.SystemResourceConsideration;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.serialization.MalformedRecordException;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.util.DataTypeUtils;
+
+@EventDriven
+@Tags({ "azure", "cosmos", "insert", "record", "put" })
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@CapabilityDescription("This processor is a record-aware processor for 
inserting data into Cosmos DB with Core SQL API. It uses a configured record 
reader and " +
+"schema to read an incoming record set from the body of a Flowfile and 
then inserts those records into " +
+"a configured Cosmos DB Container.")
+@SystemResourceConsideration(resource = SystemResource.MEMORY)
+public class PutAzureCosmosDBRecord extends AbstractAzureCosmosDBProcessor {
+
+private String conflictHandlingStrategy;
+static final AllowableValue IGNORE_CONFLICT = new AllowableValue("ignore", 
"Ignore", "Conflicting records will not be inserted, and FlowFile will not be 
routed to failure");
+static final AllowableValue UPSERT_CONFLICT = new AllowableValue("upsert", 
"Upsert", "Conflicting records will be upserted, and FlowFile will not be 
routed to failure");
+
+static final PropertyDescriptor RECORD_READER_FACTORY = new 
PropertyDescriptor.Builder()
+.name("record-reader")
+.displayName("Record Reader")
+.description("Specifies the Controller Service to use for parsing 
incoming data and determining the data's schema")
+.identifiesControllerService(RecordReaderFactory.class)
+.required(true)
+.build();
+
+static final PropertyDescriptor INSERT_BATCH_SIZE = new 
PropertyDescriptor.Builder()
+.name("insert-batch-size")
+.displayName("Insert Batch Size")
+.description("The number of records to group together for one single 
insert operation against Cosmos DB")
+.defaultValue("20")
+.required(false)
+.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+.build();
+
+static final PropertyDescriptor CONFLICT_HANDLE_STRATEGY = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-conflict-handling-strategy")
+

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

2020-05-06 Thread GitBox



jfrazee commented on a change in pull request #4253:
URL: https://github.com/apache/nifi/pull/4253#discussion_r421010643



##
File path: 
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/cosmos/document/ITAbstractAzureCosmosDBDocument.java
##
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.cosmos.document;
+
+import com.azure.cosmos.ConnectionPolicy;
+import com.azure.cosmos.CosmosClient;
+import com.azure.cosmos.CosmosClientBuilder;
+import com.azure.cosmos.CosmosClientException;
+import com.azure.cosmos.CosmosContainer;
+import com.azure.cosmos.CosmosDatabase;
+
+import com.azure.cosmos.models.CosmosContainerProperties;
+import com.azure.cosmos.models.CosmosItemRequestOptions;
+import com.azure.cosmos.models.FeedOptions;
+import com.azure.cosmos.models.PartitionKey;
+import com.azure.cosmos.util.CosmosPagedIterable;
+import com.fasterxml.jackson.databind.JsonNode;
+
+import org.apache.nifi.processor.Processor;
+import 
org.apache.nifi.services.azure.cosmos.document.AzureCosmosDBConnectionControllerService;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.apache.nifi.util.file.FileUtils;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+
+import java.io.FileInputStream;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.Properties;
+import java.util.logging.Logger;
+
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.fail;
+
+public abstract class ITAbstractAzureCosmosDBDocument {
+static Logger logger = 
Logger.getLogger(ITAbstractAzureCosmosDBDocument.class.getName());
+
+private static final Properties CONFIG;
+
+private static final String CREDENTIALS_FILE = 
System.getProperty("user.home") + "/azure-credentials.PROPERTIES";
+protected static final String TEST_COSMOS_DB_NAME = "nifi-test-db";
+protected static final String TEST_COSMOS_CONTAINER_NAME = 
"nifi-test-container";
+protected static final String TEST_COSMOS_PARTITION_KEY_FIELD_NAME = 
"category";
+protected static CosmosClient client;
+protected static CosmosDatabase cdb;
+protected static CosmosContainer container;
+
+static {
+final FileInputStream fis;
+CONFIG = new Properties();
+try {
+fis = new FileInputStream(CREDENTIALS_FILE);
+try {
+CONFIG.load(fis);
+} catch (IOException e) {
+fail("Could not open credentials file " + CREDENTIALS_FILE + 
": " + e.getLocalizedMessage());
+} finally {
+FileUtils.closeQuietly(fis);
+}
+} catch (FileNotFoundException e) {
+fail("Could not open credentials file " + CREDENTIALS_FILE + ": " 
+ e.getLocalizedMessage());
+}
+}
+
+protected static String getComosURI() {
+return CONFIG.getProperty("cosmosURI");
+}
+
+protected static String getCosmosKey() {
+return CONFIG.getProperty("cosmosKey");
+}
+
+protected TestRunner runner;
+
+@BeforeClass
+public static void createTestDBContainerIfNeeded() throws 
CosmosClientException {
+final String testDBURI =  getComosURI();
+final String testDBContainer = getCosmosKey();
+client = new CosmosClientBuilder()
+.endpoint(testDBURI)
+.key(testDBContainer)
+.connectionPolicy(ConnectionPolicy.getDefaultPolicy())
+.buildClient();
+cdb = 
client.createDatabaseIfNotExists(TEST_COSMOS_DB_NAME).getDatabase();
+CosmosContainerProperties containerProperties =
+new CosmosContainerProperties(TEST_COSMOS_CONTAINER_NAME, 
"/"+TEST_COSMOS_PARTITION_KEY_FIELD_NAME);
+container = cdb.createContainerIfNotExists(containerProperties, 
400).getContainer();
+assertNotNull(container);
+}
+
+@AfterClass
+public static void dropTestDBAndContainer() throws CosmosClientException {
+resetTestCosmosConnection();
+if(container != null) {
+

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

2020-05-06 Thread GitBox



jfrazee commented on a change in pull request #4253:
URL: https://github.com/apache/nifi/pull/4253#discussion_r421005976



##
File path: 
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/cosmos/document/ITAbstractAzureCosmosDBDocument.java
##
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.cosmos.document;
+
+import com.azure.cosmos.ConnectionPolicy;
+import com.azure.cosmos.CosmosClient;
+import com.azure.cosmos.CosmosClientBuilder;
+import com.azure.cosmos.CosmosClientException;
+import com.azure.cosmos.CosmosContainer;
+import com.azure.cosmos.CosmosDatabase;
+
+import com.azure.cosmos.models.CosmosContainerProperties;
+import com.azure.cosmos.models.CosmosItemRequestOptions;
+import com.azure.cosmos.models.FeedOptions;
+import com.azure.cosmos.models.PartitionKey;
+import com.azure.cosmos.util.CosmosPagedIterable;
+import com.fasterxml.jackson.databind.JsonNode;
+
+import org.apache.nifi.processor.Processor;
+import 
org.apache.nifi.services.azure.cosmos.document.AzureCosmosDBConnectionControllerService;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.apache.nifi.util.file.FileUtils;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+
+import java.io.FileInputStream;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.Properties;
+import java.util.logging.Logger;
+
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.fail;
+
+public abstract class ITAbstractAzureCosmosDBDocument {
+static Logger logger = 
Logger.getLogger(ITAbstractAzureCosmosDBDocument.class.getName());
+
+private static final Properties CONFIG;
+
+private static final String CREDENTIALS_FILE = 
System.getProperty("user.home") + "/azure-credentials.PROPERTIES";
+protected static final String TEST_COSMOS_DB_NAME = "nifi-test-db";
+protected static final String TEST_COSMOS_CONTAINER_NAME = 
"nifi-test-container";
+protected static final String TEST_COSMOS_PARTITION_KEY_FIELD_NAME = 
"category";
+protected static CosmosClient client;
+protected static CosmosDatabase cdb;
+protected static CosmosContainer container;
+
+static {
+final FileInputStream fis;
+CONFIG = new Properties();
+try {
+fis = new FileInputStream(CREDENTIALS_FILE);
+try {
+CONFIG.load(fis);
+} catch (IOException e) {
+fail("Could not open credentials file " + CREDENTIALS_FILE + 
": " + e.getLocalizedMessage());
+} finally {
+FileUtils.closeQuietly(fis);
+}
+} catch (FileNotFoundException e) {
+fail("Could not open credentials file " + CREDENTIALS_FILE + ": " 
+ e.getLocalizedMessage());
+}
+}
+
+protected static String getComosURI() {
+return CONFIG.getProperty("cosmosURI");
+}
+
+protected static String getCosmosKey() {
+return CONFIG.getProperty("cosmosKey");
+}
+
+protected TestRunner runner;
+
+@BeforeClass
+public static void createTestDBContainerIfNeeded() throws 
CosmosClientException {
+final String testDBURI =  getComosURI();
+final String testDBContainer = getCosmosKey();
+client = new CosmosClientBuilder()
+.endpoint(testDBURI)
+.key(testDBContainer)
+.connectionPolicy(ConnectionPolicy.getDefaultPolicy())
+.buildClient();
+cdb = 
client.createDatabaseIfNotExists(TEST_COSMOS_DB_NAME).getDatabase();
+CosmosContainerProperties containerProperties =
+new CosmosContainerProperties(TEST_COSMOS_CONTAINER_NAME, 
"/"+TEST_COSMOS_PARTITION_KEY_FIELD_NAME);
+container = cdb.createContainerIfNotExists(containerProperties, 
400).getContainer();
+assertNotNull(container);
+}
+
+@AfterClass
+public static void dropTestDBAndContainer() throws CosmosClientException {
+resetTestCosmosConnection();
+if(container != null) {
+

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

2020-05-06 Thread GitBox



jfrazee commented on a change in pull request #4253:
URL: https://github.com/apache/nifi/pull/4253#discussion_r421005293



##
File path: 
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/cosmos/document/ITAbstractAzureCosmosDBDocument.java
##
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.cosmos.document;
+
+import com.azure.cosmos.ConnectionPolicy;
+import com.azure.cosmos.CosmosClient;
+import com.azure.cosmos.CosmosClientBuilder;
+import com.azure.cosmos.CosmosClientException;
+import com.azure.cosmos.CosmosContainer;
+import com.azure.cosmos.CosmosDatabase;
+
+import com.azure.cosmos.models.CosmosContainerProperties;
+import com.azure.cosmos.models.CosmosItemRequestOptions;
+import com.azure.cosmos.models.FeedOptions;
+import com.azure.cosmos.models.PartitionKey;
+import com.azure.cosmos.util.CosmosPagedIterable;
+import com.fasterxml.jackson.databind.JsonNode;
+
+import org.apache.nifi.processor.Processor;
+import 
org.apache.nifi.services.azure.cosmos.document.AzureCosmosDBConnectionControllerService;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.apache.nifi.util.file.FileUtils;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+
+import java.io.FileInputStream;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.Properties;
+import java.util.logging.Logger;
+
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.fail;
+
+public abstract class ITAbstractAzureCosmosDBDocument {
+static Logger logger = 
Logger.getLogger(ITAbstractAzureCosmosDBDocument.class.getName());
+
+private static final Properties CONFIG;
+
+private static final String CREDENTIALS_FILE = 
System.getProperty("user.home") + "/azure-credentials.PROPERTIES";

Review comment:
   Can we just assume the `cosmosURI` and `cosmosKey` for the IT are being 
passed in on the command line or a properties file specified on it?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

2020-05-06 Thread GitBox



jfrazee commented on a change in pull request #4253:
URL: https://github.com/apache/nifi/pull/4253#discussion_r420989837



##
File path: 
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/cosmos/document/AzureCosmosDBConnectionControllerService.java
##
@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.services.azure.cosmos.document;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.cosmos.AzureCosmosDBConnectionService;
+import org.apache.nifi.util.StringUtils;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+
+import com.azure.cosmos.ConnectionPolicy;
+import com.azure.cosmos.ConsistencyLevel;
+import com.azure.cosmos.CosmosClient;
+import com.azure.cosmos.CosmosClientBuilder;
+
+@Tags({"azure", "cosmos", "document", "service"})
+@CapabilityDescription(
+"Provides a controller service that configures a connection to Azure 
Cosmos DB (Document API or Core SQL API recently renamed) " +
+" and provides access to that connection to other 
AzureCosmosDB-related components."
+)
+public class AzureCosmosDBConnectionControllerService extends 
AbstractControllerService implements AzureCosmosDBConnectionService {
+static final String CONSISTENCY_STRONG = "STRONG";
+static final String CONSISTENCY_BOUNDED_STALENESS= "BOUNDED_STALENESS";
+static final String CONSISTENCY_SESSION = "SESSION";
+static final String CONSISTENCY_CONSISTENT_PREFIX = "CONSISTENT_PREFIX";
+static final String CONSISTENCY_EVENTUAL = "EVENTUAL";
+private String uri;
+private String accessKey;
+protected CosmosClient cosmosClient;
+
+public static final PropertyDescriptor URI = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-uri")
+.displayName("Azure Cosmos DB URI")
+.description("CosmosURI, typically of the form: 
https://{databaseaccount}.documents.azure.com:443/. Note this URI is for Azure 
Cosmos DB with SQL API")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor DB_ACCESS_KEY = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-key")
+.displayName("Azure Cosmos DB Access Key")
+.description("Azure Cosmos DB Access Key from Azure Portal 
(Settings->Keys)")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+
+public static final PropertyDescriptor CONSISTENCY = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-consistency-level")
+.displayName("Azure Cosmos DB Consistency Level")
+.description("Azure Cosmos DB Consistency Level to use")
+.required(false)
+.allowableValues(CONSISTENCY_STRONG, CONSISTENCY_BOUNDED_STALENESS, 
CONSISTENCY_SESSION,
+CONSISTENCY_CONSISTENT_PREFIX, CONSISTENCY_EVENTUAL)
+.defaultValue(CONSISTENCY_SESSION)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+@OnEnabled
+public void onEnabled(final ConfigurationContext context) {
+this.uri = context.getProperty(URI).getValue();
+this.accessKey = context.getProperty(DB_ACCESS_KEY).getValue();
+final ConsistencyLevel clevel;
+final String selectedConsistency = 
context.getProperty(CONSISTENCY).getValue();
+
+switch(selectedConsistency) {
+case CONSISTENCY_STRONG:
+clevel =  ConsistencyLevel.STRONG;
+break;

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

2020-05-06 Thread GitBox



jfrazee commented on a change in pull request #4253:
URL: https://github.com/apache/nifi/pull/4253#discussion_r420984605



##
File path: 
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/cosmos/document/AzureCosmosDBConnectionControllerService.java
##
@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.services.azure.cosmos.document;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.cosmos.AzureCosmosDBConnectionService;
+import org.apache.nifi.util.StringUtils;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+
+import com.azure.cosmos.ConnectionPolicy;
+import com.azure.cosmos.ConsistencyLevel;
+import com.azure.cosmos.CosmosClient;
+import com.azure.cosmos.CosmosClientBuilder;
+
+@Tags({"azure", "cosmos", "document", "service"})
+@CapabilityDescription(
+"Provides a controller service that configures a connection to Azure 
Cosmos DB (Document API or Core SQL API recently renamed) " +
+" and provides access to that connection to other 
AzureCosmosDB-related components."
+)
+public class AzureCosmosDBConnectionControllerService extends 
AbstractControllerService implements AzureCosmosDBConnectionService {
+static final String CONSISTENCY_STRONG = "STRONG";
+static final String CONSISTENCY_BOUNDED_STALENESS= "BOUNDED_STALENESS";
+static final String CONSISTENCY_SESSION = "SESSION";
+static final String CONSISTENCY_CONSISTENT_PREFIX = "CONSISTENT_PREFIX";
+static final String CONSISTENCY_EVENTUAL = "EVENTUAL";

Review comment:
   We have these constants in two places. Maybe pull out into another class 
or enum or something.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

2020-05-06 Thread GitBox



jfrazee commented on a change in pull request #4253:
URL: https://github.com/apache/nifi/pull/4253#discussion_r420982697



##
File path: 
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/cosmos/document/PutAzureCosmosDBRecord.java
##
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.cosmos.document;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SystemResourceConsideration;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SystemResource;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.serialization.MalformedRecordException;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.util.DataTypeUtils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import com.azure.cosmos.CosmosClientException;
+import com.azure.cosmos.implementation.ConflictException;
+
+@EventDriven
+@Tags({ "azure", "cosmos", "insert", "record", "put" })
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@CapabilityDescription("This processor is a record-aware processor for 
inserting data into Azure CosmosDB with Core SQL API. It uses a configured 
record reader and " +
+"schema to read an incoming record set from the body of a flowfile and 
then inserts those records into " +
+"a configured Cosmos Container.")
+@SystemResourceConsideration(resource = SystemResource.MEMORY)
+public class PutAzureCosmosDBRecord extends AbstractAzureCosmosDBProcessor {
+
+static final PropertyDescriptor RECORD_READER_FACTORY = new 
PropertyDescriptor.Builder()
+.name("record-reader")
+.displayName("Record Reader")
+.description("Specifies the Controller Service to use for parsing 
incoming data and determining the data's schema")
+.identifiesControllerService(RecordReaderFactory.class)
+.required(true)
+.build();
+
+static final PropertyDescriptor INSERT_BATCH_SIZE = new 
PropertyDescriptor.Builder()
+.name("insert-batch-size")
+.displayName("Insert Batch Size")
+.description("The number of records to group together for one single 
insert operation against Azure CosmosDB.")
+.defaultValue("20")
+.required(false)
+.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+.build();
+
+private final static Set relationships;
+private final static List propertyDescriptors;
+
+static {
+List _propertyDescriptors = new ArrayList<>();
+_propertyDescriptors.addAll(descriptors);
+_propertyDescriptors.add(RECORD_READER_FACTORY);
+_propertyDescriptors.add(INSERT_BATCH_SIZE);
+propertyDescriptors = 
Collections.unmodifiableList(_propertyDescriptors);
+
+final Set _relationships = new HashSet<>();
+_relationships.add(REL_SUCCESS);
+_relationships.add(REL_FAILURE);
+relationships = Collections.unmodifiableSet(_relationships);
+}
+
+@Override
+public Set getRelationships() {
+

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

2020-05-06 Thread GitBox



jfrazee commented on a change in pull request #4253:
URL: https://github.com/apache/nifi/pull/4253#discussion_r420978605



##
File path: 
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/cosmos/document/PutAzureCosmosDBRecord.java
##
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.cosmos.document;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.SystemResourceConsideration;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SystemResource;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.schema.access.SchemaNotFoundException;
+import org.apache.nifi.serialization.MalformedRecordException;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.serialization.record.util.DataTypeUtils;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import com.azure.cosmos.CosmosClientException;
+import com.azure.cosmos.implementation.ConflictException;
+
+@EventDriven
+@Tags({ "azure", "cosmos", "insert", "record", "put" })
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@CapabilityDescription("This processor is a record-aware processor for 
inserting data into Azure CosmosDB with Core SQL API. It uses a configured 
record reader and " +
+"schema to read an incoming record set from the body of a flowfile and 
then inserts those records into " +
+"a configured Cosmos Container.")
+@SystemResourceConsideration(resource = SystemResource.MEMORY)
+public class PutAzureCosmosDBRecord extends AbstractAzureCosmosDBProcessor {
+
+static final PropertyDescriptor RECORD_READER_FACTORY = new 
PropertyDescriptor.Builder()
+.name("record-reader")
+.displayName("Record Reader")
+.description("Specifies the Controller Service to use for parsing 
incoming data and determining the data's schema")
+.identifiesControllerService(RecordReaderFactory.class)
+.required(true)
+.build();
+
+static final PropertyDescriptor INSERT_BATCH_SIZE = new 
PropertyDescriptor.Builder()
+.name("insert-batch-size")
+.displayName("Insert Batch Size")
+.description("The number of records to group together for one single 
insert operation against Azure CosmosDB.")
+.defaultValue("20")
+.required(false)
+.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+.build();
+
+private final static Set relationships;
+private final static List propertyDescriptors;
+
+static {
+List _propertyDescriptors = new ArrayList<>();
+_propertyDescriptors.addAll(descriptors);
+_propertyDescriptors.add(RECORD_READER_FACTORY);
+_propertyDescriptors.add(INSERT_BATCH_SIZE);
+propertyDescriptors = 
Collections.unmodifiableList(_propertyDescriptors);
+
+final Set _relationships = new HashSet<>();
+_relationships.add(REL_SUCCESS);
+_relationships.add(REL_FAILURE);
+relationships = Collections.unmodifiableSet(_relationships);
+}
+
+@Override
+public Set getRelationships() {
+

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

2020-05-06 Thread GitBox



jfrazee commented on a change in pull request #4253:
URL: https://github.com/apache/nifi/pull/4253#discussion_r420908809



##
File path: 
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/cosmos/document/AbstractAzureCosmosDBProcessor.java
##
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.nifi.processors.azure.cosmos.document;
+
+import java.io.ByteArrayInputStream;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+import com.azure.cosmos.ConnectionPolicy;
+import com.azure.cosmos.ConsistencyLevel;
+import com.azure.cosmos.CosmosClient;
+import com.azure.cosmos.CosmosClientBuilder;
+import com.azure.cosmos.CosmosClientException;
+import com.azure.cosmos.CosmosContainer;
+import com.azure.cosmos.CosmosDatabase;
+import com.azure.cosmos.models.CosmosContainerProperties;
+
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.logging.ComponentLog;
+
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.cosmos.AzureCosmosDBConnectionService;
+
+public abstract class AbstractAzureCosmosDBProcessor extends AbstractProcessor 
{
+static final String CONSISTENCY_STRONG = "STRONG";
+static final String CONSISTENCY_BOUNDED_STALENESS= "BOUNDED_STALENESS";
+static final String CONSISTENCY_SESSION = "SESSION";
+static final String CONSISTENCY_CONSISTENT_PREFIX = "CONSISTENT_PREFIX";
+static final String CONSISTENCY_EVENTUAL = "EVENTUAL";
+
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to Azure Cosmos DB are routed 
to this relationship").build();
+
+static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to Azure Cosmos DB are 
routed to this relationship").build();
+
+static final Relationship REL_ORIGINAL = new Relationship.Builder()
+.name("original")
+.description("All input FlowFiles that are part of a successful 
query execution go here.")
+.build();
+
+static final PropertyDescriptor CONNECTION_SERVICE = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-connection-service")
+.displayName("Azure Cosmos DB Connection Service")
+.description("If configured, this property will use the assigned for 
retrieving connection string info.")
+.required(false)
+.identifiesControllerService(AzureCosmosDBConnectionService.class)
+.build();
+
+
+static final PropertyDescriptor URI = new PropertyDescriptor.Builder()
+.name("azure-cosmos-db-uri")
+.displayName("Azure Cosmos DB URI")
+.description("Azure Cosmos DB URI, typically of the form: 
https://{databaseaccount}.documents.azure.com:443/. Note this host URL is for 
Azure Cosmos DB with CORE SQL API")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+static final PropertyDescriptor DB_ACCESS_KEY = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-key")
+.displayName("Azure Cosmos DB Access Key")
+.description("Azure Cosmos DB Access Key from Azure Portal 
(Settings->Keys)")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+
+static final PropertyDescriptor DATABASE_NAME = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-name")
+.displayName("Azure Cosmos DB

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

2020-05-06 Thread GitBox



jfrazee commented on a change in pull request #4253:
URL: https://github.com/apache/nifi/pull/4253#discussion_r420908809



##
File path: 
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/cosmos/document/AbstractAzureCosmosDBProcessor.java
##
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.nifi.processors.azure.cosmos.document;
+
+import java.io.ByteArrayInputStream;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+import com.azure.cosmos.ConnectionPolicy;
+import com.azure.cosmos.ConsistencyLevel;
+import com.azure.cosmos.CosmosClient;
+import com.azure.cosmos.CosmosClientBuilder;
+import com.azure.cosmos.CosmosClientException;
+import com.azure.cosmos.CosmosContainer;
+import com.azure.cosmos.CosmosDatabase;
+import com.azure.cosmos.models.CosmosContainerProperties;
+
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.logging.ComponentLog;
+
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.cosmos.AzureCosmosDBConnectionService;
+
+public abstract class AbstractAzureCosmosDBProcessor extends AbstractProcessor 
{
+static final String CONSISTENCY_STRONG = "STRONG";
+static final String CONSISTENCY_BOUNDED_STALENESS= "BOUNDED_STALENESS";
+static final String CONSISTENCY_SESSION = "SESSION";
+static final String CONSISTENCY_CONSISTENT_PREFIX = "CONSISTENT_PREFIX";
+static final String CONSISTENCY_EVENTUAL = "EVENTUAL";
+
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to Azure Cosmos DB are routed 
to this relationship").build();
+
+static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to Azure Cosmos DB are 
routed to this relationship").build();
+
+static final Relationship REL_ORIGINAL = new Relationship.Builder()
+.name("original")
+.description("All input FlowFiles that are part of a successful 
query execution go here.")
+.build();
+
+static final PropertyDescriptor CONNECTION_SERVICE = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-connection-service")
+.displayName("Azure Cosmos DB Connection Service")
+.description("If configured, this property will use the assigned for 
retrieving connection string info.")
+.required(false)
+.identifiesControllerService(AzureCosmosDBConnectionService.class)
+.build();
+
+
+static final PropertyDescriptor URI = new PropertyDescriptor.Builder()
+.name("azure-cosmos-db-uri")
+.displayName("Azure Cosmos DB URI")
+.description("Azure Cosmos DB URI, typically of the form: 
https://{databaseaccount}.documents.azure.com:443/. Note this host URL is for 
Azure Cosmos DB with CORE SQL API")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+static final PropertyDescriptor DB_ACCESS_KEY = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-key")
+.displayName("Azure Cosmos DB Access Key")
+.description("Azure Cosmos DB Access Key from Azure Portal 
(Settings->Keys)")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+
+static final PropertyDescriptor DATABASE_NAME = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-name")
+.displayName("Azure Cosmos DB

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

2020-05-06 Thread GitBox



jfrazee commented on a change in pull request #4253:
URL: https://github.com/apache/nifi/pull/4253#discussion_r420901023



##
File path: 
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/cosmos/document/AbstractAzureCosmosDBProcessor.java
##
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.nifi.processors.azure.cosmos.document;
+
+import java.io.ByteArrayInputStream;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+import com.azure.cosmos.ConnectionPolicy;
+import com.azure.cosmos.ConsistencyLevel;
+import com.azure.cosmos.CosmosClient;
+import com.azure.cosmos.CosmosClientBuilder;
+import com.azure.cosmos.CosmosClientException;
+import com.azure.cosmos.CosmosContainer;
+import com.azure.cosmos.CosmosDatabase;
+import com.azure.cosmos.models.CosmosContainerProperties;
+
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.logging.ComponentLog;
+
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.cosmos.AzureCosmosDBConnectionService;
+
+public abstract class AbstractAzureCosmosDBProcessor extends AbstractProcessor 
{
+static final String CONSISTENCY_STRONG = "STRONG";
+static final String CONSISTENCY_BOUNDED_STALENESS= "BOUNDED_STALENESS";
+static final String CONSISTENCY_SESSION = "SESSION";
+static final String CONSISTENCY_CONSISTENT_PREFIX = "CONSISTENT_PREFIX";
+static final String CONSISTENCY_EVENTUAL = "EVENTUAL";
+
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to Azure Cosmos DB are routed 
to this relationship").build();
+
+static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to Azure Cosmos DB are 
routed to this relationship").build();
+
+static final Relationship REL_ORIGINAL = new Relationship.Builder()
+.name("original")
+.description("All input FlowFiles that are part of a successful 
query execution go here.")
+.build();
+
+static final PropertyDescriptor CONNECTION_SERVICE = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-connection-service")
+.displayName("Azure Cosmos DB Connection Service")
+.description("If configured, this property will use the assigned for 
retrieving connection string info.")
+.required(false)
+.identifiesControllerService(AzureCosmosDBConnectionService.class)
+.build();
+
+
+static final PropertyDescriptor URI = new PropertyDescriptor.Builder()
+.name("azure-cosmos-db-uri")
+.displayName("Azure Cosmos DB URI")
+.description("Azure Cosmos DB URI, typically of the form: 
https://{databaseaccount}.documents.azure.com:443/. Note this host URL is for 
Azure Cosmos DB with CORE SQL API")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+static final PropertyDescriptor DB_ACCESS_KEY = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-key")
+.displayName("Azure Cosmos DB Access Key")
+.description("Azure Cosmos DB Access Key from Azure Portal 
(Settings->Keys)")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+
+static final PropertyDescriptor DATABASE_NAME = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-name")
+.displayName("Azure Cosmos DB

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

2020-05-06 Thread GitBox



jfrazee commented on a change in pull request #4253:
URL: https://github.com/apache/nifi/pull/4253#discussion_r420890146



##
File path: 
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/cosmos/document/AbstractAzureCosmosDBProcessor.java
##
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.nifi.processors.azure.cosmos.document;
+
+import java.io.ByteArrayInputStream;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+import com.azure.cosmos.ConnectionPolicy;
+import com.azure.cosmos.ConsistencyLevel;
+import com.azure.cosmos.CosmosClient;
+import com.azure.cosmos.CosmosClientBuilder;
+import com.azure.cosmos.CosmosClientException;
+import com.azure.cosmos.CosmosContainer;
+import com.azure.cosmos.CosmosDatabase;
+import com.azure.cosmos.models.CosmosContainerProperties;
+
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.logging.ComponentLog;
+
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.cosmos.AzureCosmosDBConnectionService;
+
+public abstract class AbstractAzureCosmosDBProcessor extends AbstractProcessor 
{
+static final String CONSISTENCY_STRONG = "STRONG";
+static final String CONSISTENCY_BOUNDED_STALENESS= "BOUNDED_STALENESS";
+static final String CONSISTENCY_SESSION = "SESSION";
+static final String CONSISTENCY_CONSISTENT_PREFIX = "CONSISTENT_PREFIX";
+static final String CONSISTENCY_EVENTUAL = "EVENTUAL";
+
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to Azure Cosmos DB are routed 
to this relationship").build();
+
+static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to Azure Cosmos DB are 
routed to this relationship").build();
+
+static final Relationship REL_ORIGINAL = new Relationship.Builder()
+.name("original")
+.description("All input FlowFiles that are part of a successful 
query execution go here.")
+.build();
+
+static final PropertyDescriptor CONNECTION_SERVICE = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-connection-service")
+.displayName("Azure Cosmos DB Connection Service")
+.description("If configured, this property will use the assigned for 
retrieving connection string info.")
+.required(false)
+.identifiesControllerService(AzureCosmosDBConnectionService.class)
+.build();
+
+
+static final PropertyDescriptor URI = new PropertyDescriptor.Builder()
+.name("azure-cosmos-db-uri")
+.displayName("Azure Cosmos DB URI")
+.description("Azure Cosmos DB URI, typically of the form: 
https://{databaseaccount}.documents.azure.com:443/. Note this host URL is for 
Azure Cosmos DB with CORE SQL API")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+static final PropertyDescriptor DB_ACCESS_KEY = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-key")
+.displayName("Azure Cosmos DB Access Key")
+.description("Azure Cosmos DB Access Key from Azure Portal 
(Settings->Keys)")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+
+static final PropertyDescriptor DATABASE_NAME = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-name")
+.displayName("Azure Cosmos DB

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

2020-05-06 Thread GitBox



jfrazee commented on a change in pull request #4253:
URL: https://github.com/apache/nifi/pull/4253#discussion_r420886112



##
File path: 
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/cosmos/document/AbstractAzureCosmosDBProcessor.java
##
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.nifi.processors.azure.cosmos.document;
+
+import java.io.ByteArrayInputStream;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+import com.azure.cosmos.ConnectionPolicy;
+import com.azure.cosmos.ConsistencyLevel;
+import com.azure.cosmos.CosmosClient;
+import com.azure.cosmos.CosmosClientBuilder;
+import com.azure.cosmos.CosmosClientException;
+import com.azure.cosmos.CosmosContainer;
+import com.azure.cosmos.CosmosDatabase;
+import com.azure.cosmos.models.CosmosContainerProperties;
+
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.logging.ComponentLog;
+
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.cosmos.AzureCosmosDBConnectionService;
+
+public abstract class AbstractAzureCosmosDBProcessor extends AbstractProcessor 
{
+static final String CONSISTENCY_STRONG = "STRONG";
+static final String CONSISTENCY_BOUNDED_STALENESS= "BOUNDED_STALENESS";
+static final String CONSISTENCY_SESSION = "SESSION";
+static final String CONSISTENCY_CONSISTENT_PREFIX = "CONSISTENT_PREFIX";
+static final String CONSISTENCY_EVENTUAL = "EVENTUAL";
+
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to Azure Cosmos DB are routed 
to this relationship").build();
+
+static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to Azure Cosmos DB are 
routed to this relationship").build();
+
+static final Relationship REL_ORIGINAL = new Relationship.Builder()
+.name("original")
+.description("All input FlowFiles that are part of a successful 
query execution go here.")
+.build();
+
+static final PropertyDescriptor CONNECTION_SERVICE = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-connection-service")
+.displayName("Azure Cosmos DB Connection Service")
+.description("If configured, this property will use the assigned for 
retrieving connection string info.")
+.required(false)
+.identifiesControllerService(AzureCosmosDBConnectionService.class)
+.build();
+
+
+static final PropertyDescriptor URI = new PropertyDescriptor.Builder()
+.name("azure-cosmos-db-uri")
+.displayName("Azure Cosmos DB URI")
+.description("Azure Cosmos DB URI, typically of the form: 
https://{databaseaccount}.documents.azure.com:443/. Note this host URL is for 
Azure Cosmos DB with CORE SQL API")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+static final PropertyDescriptor DB_ACCESS_KEY = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-key")
+.displayName("Azure Cosmos DB Access Key")
+.description("Azure Cosmos DB Access Key from Azure Portal 
(Settings->Keys)")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+
+static final PropertyDescriptor DATABASE_NAME = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-name")
+.displayName("Azure Cosmos DB

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

2020-05-06 Thread GitBox



jfrazee commented on a change in pull request #4253:
URL: https://github.com/apache/nifi/pull/4253#discussion_r420885481



##
File path: 
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/cosmos/document/AbstractAzureCosmosDBProcessor.java
##
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.nifi.processors.azure.cosmos.document;
+
+import java.io.ByteArrayInputStream;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+import com.azure.cosmos.ConnectionPolicy;
+import com.azure.cosmos.ConsistencyLevel;
+import com.azure.cosmos.CosmosClient;
+import com.azure.cosmos.CosmosClientBuilder;
+import com.azure.cosmos.CosmosClientException;
+import com.azure.cosmos.CosmosContainer;
+import com.azure.cosmos.CosmosDatabase;
+import com.azure.cosmos.models.CosmosContainerProperties;
+
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.logging.ComponentLog;
+
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.cosmos.AzureCosmosDBConnectionService;
+
+public abstract class AbstractAzureCosmosDBProcessor extends AbstractProcessor 
{
+static final String CONSISTENCY_STRONG = "STRONG";
+static final String CONSISTENCY_BOUNDED_STALENESS= "BOUNDED_STALENESS";
+static final String CONSISTENCY_SESSION = "SESSION";
+static final String CONSISTENCY_CONSISTENT_PREFIX = "CONSISTENT_PREFIX";
+static final String CONSISTENCY_EVENTUAL = "EVENTUAL";
+
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to Azure Cosmos DB are routed 
to this relationship").build();
+
+static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to Azure Cosmos DB are 
routed to this relationship").build();
+
+static final Relationship REL_ORIGINAL = new Relationship.Builder()
+.name("original")
+.description("All input FlowFiles that are part of a successful 
query execution go here.")
+.build();
+
+static final PropertyDescriptor CONNECTION_SERVICE = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-connection-service")
+.displayName("Azure Cosmos DB Connection Service")
+.description("If configured, this property will use the assigned for 
retrieving connection string info.")
+.required(false)
+.identifiesControllerService(AzureCosmosDBConnectionService.class)
+.build();
+
+
+static final PropertyDescriptor URI = new PropertyDescriptor.Builder()
+.name("azure-cosmos-db-uri")
+.displayName("Azure Cosmos DB URI")
+.description("Azure Cosmos DB URI, typically of the form: 
https://{databaseaccount}.documents.azure.com:443/. Note this host URL is for 
Azure Cosmos DB with CORE SQL API")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+static final PropertyDescriptor DB_ACCESS_KEY = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-key")
+.displayName("Azure Cosmos DB Access Key")
+.description("Azure Cosmos DB Access Key from Azure Portal 
(Settings->Keys)")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.sensitive(true)
+.build();
+
+static final PropertyDescriptor DATABASE_NAME = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-name")
+.displayName("Azure Cosmos DB

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

2020-05-06 Thread GitBox



jfrazee commented on a change in pull request #4253:
URL: https://github.com/apache/nifi/pull/4253#discussion_r420881234



##
File path: 
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/cosmos/document/AbstractAzureCosmosDBProcessor.java
##
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.nifi.processors.azure.cosmos.document;
+
+import java.io.ByteArrayInputStream;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+import com.azure.cosmos.ConnectionPolicy;
+import com.azure.cosmos.ConsistencyLevel;
+import com.azure.cosmos.CosmosClient;
+import com.azure.cosmos.CosmosClientBuilder;
+import com.azure.cosmos.CosmosClientException;
+import com.azure.cosmos.CosmosContainer;
+import com.azure.cosmos.CosmosDatabase;
+import com.azure.cosmos.models.CosmosContainerProperties;
+
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.logging.ComponentLog;
+
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.cosmos.AzureCosmosDBConnectionService;
+
+public abstract class AbstractAzureCosmosDBProcessor extends AbstractProcessor 
{
+static final String CONSISTENCY_STRONG = "STRONG";
+static final String CONSISTENCY_BOUNDED_STALENESS= "BOUNDED_STALENESS";
+static final String CONSISTENCY_SESSION = "SESSION";
+static final String CONSISTENCY_CONSISTENT_PREFIX = "CONSISTENT_PREFIX";
+static final String CONSISTENCY_EVENTUAL = "EVENTUAL";
+
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to Azure Cosmos DB are routed 
to this relationship").build();
+
+static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to Azure Cosmos DB are 
routed to this relationship").build();
+
+static final Relationship REL_ORIGINAL = new Relationship.Builder()
+.name("original")
+.description("All input FlowFiles that are part of a successful 
query execution go here.")
+.build();
+
+static final PropertyDescriptor CONNECTION_SERVICE = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-connection-service")
+.displayName("Azure Cosmos DB Connection Service")
+.description("If configured, this property will use the assigned for 
retrieving connection string info.")
+.required(false)
+.identifiesControllerService(AzureCosmosDBConnectionService.class)
+.build();
+
+
+static final PropertyDescriptor URI = new PropertyDescriptor.Builder()
+.name("azure-cosmos-db-uri")
+.displayName("Azure Cosmos DB URI")
+.description("Azure Cosmos DB URI, typically of the form: 
https://{databaseaccount}.documents.azure.com:443/. Note this host URL is for 
Azure Cosmos DB with CORE SQL API")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+static final PropertyDescriptor DB_ACCESS_KEY = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-key")
+.displayName("Azure Cosmos DB Access Key")
+.description("Azure Cosmos DB Access Key from Azure Portal 
(Settings->Keys)")

Review comment:
   I do like the concept of pointing to where to get the info. Do you have 
a sense of how stable that will be though? If it makes sense here I think it 
would end up making sense for the other properties too.





This

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

2020-05-06 Thread GitBox



jfrazee commented on a change in pull request #4253:
URL: https://github.com/apache/nifi/pull/4253#discussion_r420877750



##
File path: 
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/cosmos/document/AbstractAzureCosmosDBProcessor.java
##
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.nifi.processors.azure.cosmos.document;
+
+import java.io.ByteArrayInputStream;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+import com.azure.cosmos.ConnectionPolicy;
+import com.azure.cosmos.ConsistencyLevel;
+import com.azure.cosmos.CosmosClient;
+import com.azure.cosmos.CosmosClientBuilder;
+import com.azure.cosmos.CosmosClientException;
+import com.azure.cosmos.CosmosContainer;
+import com.azure.cosmos.CosmosDatabase;
+import com.azure.cosmos.models.CosmosContainerProperties;
+
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.logging.ComponentLog;
+
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.cosmos.AzureCosmosDBConnectionService;
+
+public abstract class AbstractAzureCosmosDBProcessor extends AbstractProcessor 
{
+static final String CONSISTENCY_STRONG = "STRONG";
+static final String CONSISTENCY_BOUNDED_STALENESS= "BOUNDED_STALENESS";
+static final String CONSISTENCY_SESSION = "SESSION";
+static final String CONSISTENCY_CONSISTENT_PREFIX = "CONSISTENT_PREFIX";
+static final String CONSISTENCY_EVENTUAL = "EVENTUAL";
+
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to Azure Cosmos DB are routed 
to this relationship").build();
+
+static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to Azure Cosmos DB are 
routed to this relationship").build();
+
+static final Relationship REL_ORIGINAL = new Relationship.Builder()
+.name("original")
+.description("All input FlowFiles that are part of a successful 
query execution go here.")
+.build();
+
+static final PropertyDescriptor CONNECTION_SERVICE = new 
PropertyDescriptor.Builder()
+.name("azure-cosmos-db-connection-service")
+.displayName("Azure Cosmos DB Connection Service")
+.description("If configured, this property will use the assigned for 
retrieving connection string info.")
+.required(false)
+.identifiesControllerService(AzureCosmosDBConnectionService.class)
+.build();
+
+
+static final PropertyDescriptor URI = new PropertyDescriptor.Builder()
+.name("azure-cosmos-db-uri")
+.displayName("Azure Cosmos DB URI")
+.description("Azure Cosmos DB URI, typically of the form: 
https://{databaseaccount}.documents.azure.com:443/. Note this host URL is for 
Azure Cosmos DB with CORE SQL API")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)

Review comment:
   Would it make sense to use `URL_VALIDATOR` or `URI_VALIDATOR` here?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

2020-05-06 Thread GitBox



jfrazee commented on a change in pull request #4253:
URL: https://github.com/apache/nifi/pull/4253#discussion_r420875649



##
File path: 
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/cosmos/document/AbstractAzureCosmosDBProcessor.java
##
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.nifi.processors.azure.cosmos.document;
+
+import java.io.ByteArrayInputStream;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+import com.azure.cosmos.ConnectionPolicy;
+import com.azure.cosmos.ConsistencyLevel;
+import com.azure.cosmos.CosmosClient;
+import com.azure.cosmos.CosmosClientBuilder;
+import com.azure.cosmos.CosmosClientException;
+import com.azure.cosmos.CosmosContainer;
+import com.azure.cosmos.CosmosDatabase;
+import com.azure.cosmos.models.CosmosContainerProperties;
+
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.logging.ComponentLog;
+
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.cosmos.AzureCosmosDBConnectionService;
+
+public abstract class AbstractAzureCosmosDBProcessor extends AbstractProcessor 
{
+static final String CONSISTENCY_STRONG = "STRONG";
+static final String CONSISTENCY_BOUNDED_STALENESS= "BOUNDED_STALENESS";
+static final String CONSISTENCY_SESSION = "SESSION";
+static final String CONSISTENCY_CONSISTENT_PREFIX = "CONSISTENT_PREFIX";
+static final String CONSISTENCY_EVENTUAL = "EVENTUAL";
+
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to Azure Cosmos DB are routed 
to this relationship").build();
+
+static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to Azure Cosmos DB are 
routed to this relationship").build();

Review comment:
   Can you style these like on L65.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

2020-05-06 Thread GitBox



jfrazee commented on a change in pull request #4253:
URL: https://github.com/apache/nifi/pull/4253#discussion_r420874522



##
File path: nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/pom.xml
##
@@ -65,21 +87,74 @@
 com.microsoft.azure
 azure-eventhubs-eph
 ${azure-eventhubs-eph.version}
+
+
+
+com.fasterxml.jackson.core
+jackson-core
+
+
+com.fasterxml.jackson.core
+jackson-databind
+
+
 
 
 com.microsoft.azure
 azure-storage
-
+
+
+com.fasterxml.jackson.core
+jackson-core
+
+
+
+
+com.azure
+azure-cosmos
+4.0.1-beta.2
+
+
+com.azure
+azure-core
+
+
+com.google.guava
+guava
+
+
+

Review comment:
   Looks like this is indented too much. Should be 4 spaces.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

[GitHub] [nifi] jfrazee commented on a change in pull request #4253: NIFI-7406: PutAzureCosmosRecord Processor

19 matches

Site Navigation

Mail list logo

Footer information