Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
exceptionfactory closed pull request #7624: NIFI-11958 Azure Data Explorer as sink URL: https://github.com/apache/nifi/pull/7624 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
exceptionfactory commented on PR #7624: URL: https://github.com/apache/nifi/pull/7624#issuecomment-1894538604 Rebased to resolve merge conflicts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
tanmaya-panda1 commented on code in PR #7624:
URL: https://github.com/apache/nifi/pull/7624#discussion_r1430938458
##
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestQueryResponse;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.time.Duration;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector
which sends flowFiles using the ADX-Service to the provided Azure Data" +
+"Explorer Ingest Endpoint. The data can be sent through queued
ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+public static final String FETCH_TABLE_COMMAND = "%s | count";
+public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s
policy streamingingestion";
+public static final String DATABASE = "database";
+public static final AllowableValue IGNORE_FIRST_RECORD_YES = new
AllowableValue(
+"YES", "YES",
+"Ignore first record during ingestion");
+
+public static final AllowableValue IGNORE_FIRST_RECORD_NO = new
AllowableValue(
+"NO", "NO",
+"Do not ignore first record during ingestion");
+
+public static final PropertyDescriptor DATABASE_NAME = new
PropertyDescriptor.Builder()
+.name("Database Name")
+.displayName("Database Name")
+.description("Azure Data Explorer Database Name for querying")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.build();
+public static final PropertyDescriptor TABLE_NAME = new
PropertyDescriptor.Builder()
+.name("Table Name")
+.displayName("Table Name")
+.description("Azure Data Explorer Table Name for ingesting data")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.build();
+public static final PropertyDescriptor MAPPING_NAME = new
PropertyDescriptor
+.Builder().name("Ingest Mapping name")
+.displayName("Ingest Mapping name")
+.description("The name of the mapping responsible for storing the
data in the appropriate columns.")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+public static final PropertyDescriptor
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
exceptionfactory commented on code in PR #7624:
URL: https://github.com/apache/nifi/pull/7624#discussion_r1428874499
##
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/services/azure/data/explorer/StandardKustoIngestService.java:
##
@@ -0,0 +1,409 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.Client;
+import com.microsoft.azure.kusto.data.ClientFactory;
+import com.microsoft.azure.kusto.data.KustoResultColumn;
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.data.auth.ConnectionStringBuilder;
+import com.microsoft.azure.kusto.data.exceptions.DataClientException;
+import com.microsoft.azure.kusto.data.exceptions.DataServiceException;
+import com.microsoft.azure.kusto.ingest.IngestClientFactory;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import com.microsoft.azure.kusto.ingest.ManagedStreamingIngestClient;
+import com.microsoft.azure.kusto.ingest.QueuedIngestClient;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionClientException;
+import com.microsoft.azure.kusto.ingest.exceptions.IngestionServiceException;
+import com.microsoft.azure.kusto.ingest.result.IngestionResult;
+import com.microsoft.azure.kusto.ingest.result.IngestionStatus;
+import com.microsoft.azure.kusto.ingest.result.OperationStatus;
+import com.microsoft.azure.kusto.ingest.source.StreamSourceInfo;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+@Tags({"Azure", "ADX", "Kusto", "ingest", "azure"})
+@CapabilityDescription("Sends batches of flowfile content or stream flowfile
content to an Azure ADX cluster.")
+public class StandardKustoIngestService extends AbstractControllerService
implements KustoIngestService {
+
+public static final PropertyDescriptor AUTHENTICATION_STRATEGY = new
PropertyDescriptor
+.Builder().name("Authentication Strategy")
+.displayName("Authentication Strategy")
+.description("Authentication method for access to Azure Data
Explorer")
+.required(true)
+
.defaultValue(KustoAuthenticationStrategy.MANAGED_IDENTITY.getValue())
+.allowableValues(KustoAuthenticationStrategy.class)
+.build();
+
+public static final PropertyDescriptor APPLICATION_CLIENT_ID = new
PropertyDescriptor
+.Builder().name("Application Client ID")
+.displayName("Application Client ID")
+.description("Azure Data Explorer Application Client Identifier
for Authentication")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor APPLICATION_KEY = new
PropertyDescriptor
+.Builder().name("Application Key")
+.displayName("Application Key")
+.description("Azure Data Explorer Application Key for
Authentication")
+.required(true)
+.sensitive(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.dependsOn(AUTHENTICATION_STRATEGY,
KustoAuthenticationStrategy.APPLICATION_CREDENTIALS.getValue())
+.build();
+
+public static final PropertyDescriptor
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
exceptionfactory commented on PR #7624: URL: https://github.com/apache/nifi/pull/7624#issuecomment-1840798813 > @exceptionfactory : Hope you are doing good. Can you please take a look at this PR. This is long pending item, we wanted to close this before the end of this year. We would appreciate your review on this item whenever you have some time. Thanks for the follow-up @tanmaya-panda1. I am in the process of reviewing several pull requests, and I plan to return to this one soon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
tanmaya-panda1 commented on PR #7624: URL: https://github.com/apache/nifi/pull/7624#issuecomment-1837866306 @exceptionfactory : Can you please take a look at this PR. This is long pending item, we wanted to close this before the end of this year. We would appreciate your review on this item whenever you have some time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
exceptionfactory commented on code in PR #7624:
URL: https://github.com/apache/nifi/pull/7624#discussion_r1394354709
##
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestQueryResponse;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector
which sends flowFiles using the ADX-Service to the provided Azure Data" +
+"Explorer Ingest Endpoint. The data can be sent through queued
ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+public static final String FETCH_TABLE_COMMAND = "%s | count";
+public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s
policy streamingingestion";
+public static final String DATABASE = "database";
+public static final AllowableValue IGNORE_FIRST_RECORD_YES = new
AllowableValue(
+"YES", "YES",
+"Ignore first record during ingestion");
+
+public static final AllowableValue IGNORE_FIRST_RECORD_NO = new
AllowableValue(
+"NO", "NO",
+"Do not ignore first record during ingestion");
+
+public static final PropertyDescriptor DATABASE_NAME = new
PropertyDescriptor.Builder()
+.name("Database Name")
+.displayName("Database Name")
+.description("Azure Data Explorer Database Name for querying")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.build();
+public static final PropertyDescriptor TABLE_NAME = new
PropertyDescriptor.Builder()
+.name("Table Name")
+.displayName("Table Name")
+.description("Azure Data Explorer Table Name for ingesting data")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.build();
+public static final PropertyDescriptor MAPPING_NAME = new
PropertyDescriptor
+.Builder().name("Ingest Mapping name")
+.displayName("Ingest Mapping name")
+.description("The name of the mapping responsible for storing the
data in the appropriate columns.")
+.required(false)
+
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
tanmaya-panda1 commented on code in PR #7624:
URL: https://github.com/apache/nifi/pull/7624#discussion_r1381215021
##
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector
which sends flowFiles using the ADX-Service to the provided Azure Data" +
+"Explorer Ingest Endpoint. The data can be sent through queued
ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+public static final String FETCH_TABLE_COMMAND = "%s | count";
+public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s
policy streamingingestion";
+public static final String DATABASE = "database";
+
+private List descriptors;
+private Set relationships;
+private transient KustoIngestService service;
+private boolean streamingEnabled;
+private boolean pollOnIngestionStatus;
+
+public static final AllowableValue IGNORE_FIRST_RECORD_YES = new
AllowableValue(
+"YES", "YES",
+"Ignore first record during ingestion");
+
+public static final AllowableValue IGNORE_FIRST_RECORD_NO = new
AllowableValue(
+"NO", "NO",
+"Do not ignore first record during ingestion");
+
+public static final PropertyDescriptor DATABASE_NAME = new
PropertyDescriptor.Builder()
+.name("Database Name")
+.displayName("Database Name")
+.description("Azure Data Explorer Database Name for querying")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.build();
+
+public static final PropertyDescriptor TABLE_NAME = new
PropertyDescriptor.Builder()
+.name("Table Name")
+.displayName("Table Name")
+.description("Azure Data Explorer Table Name for ingesting data")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.build();
+
+public static final PropertyDescriptor MAPPING_NAME = new
PropertyDescriptor
+.Builder().name("Ingest Mapping name")
+.displayName("Ingest Mapping name")
+.description("The name of the mapping responsible for storing the
data in the
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
tanmaya-panda1 commented on code in PR #7624:
URL: https://github.com/apache/nifi/pull/7624#discussion_r1381171499
##
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##
@@ -0,0 +1,309 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector
which sends flowFiles using the ADX-Service to the provided Azure Data" +
+"Explorer Ingest Endpoint. The data can be sent through queued
ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+public static final String FETCH_TABLE_COMMAND = "%s | count";
+public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s
policy streamingingestion";
+public static final String DATABASE = "database";
+
+private List descriptors;
+private Set relationships;
+private transient KustoIngestService service;
+private boolean streamingEnabled;
+private boolean pollOnIngestionStatus;
+
+public static final AllowableValue IGNORE_FIRST_RECORD_YES = new
AllowableValue(
+"YES", "YES",
+"Ignore first record during ingestion");
+
+public static final AllowableValue IGNORE_FIRST_RECORD_NO = new
AllowableValue(
+"NO", "NO",
+"Do not ignore first record during ingestion");
+
+public static final PropertyDescriptor DATABASE_NAME = new
PropertyDescriptor.Builder()
+.name("Database Name")
+.displayName("Database Name")
+.description("Azure Data Explorer Database Name for querying")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.build();
+
+public static final PropertyDescriptor TABLE_NAME = new
PropertyDescriptor.Builder()
+.name("Table Name")
+.displayName("Table Name")
+.description("Azure Data Explorer Table Name for ingesting data")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.build();
+
+public static final PropertyDescriptor MAPPING_NAME = new
PropertyDescriptor
+.Builder().name("Ingest Mapping name")
+.displayName("Ingest Mapping name")
+.description("The name of the mapping responsible for storing the
data in the
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
exceptionfactory commented on code in PR #7624:
URL: https://github.com/apache/nifi/pull/7624#discussion_r1366908724
##
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-services-api/src/main/java/org/apache/nifi/services/azure/data/explorer/KustoIngestService.java:
##
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+
+import java.net.URISyntaxException;
+
+@Tags({"azure", "adx", "explorer", "kusto", "ingest"})
+@CapabilityDescription("Connection-Service for Azure ADX (Kusto) ingestion
cluster to ingest data.")
+public interface KustoIngestService extends KustoQueryService {
Review Comment:
Thanks for clarifying, that makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
tanmaya-panda1 commented on PR #7624: URL: https://github.com/apache/nifi/pull/7624#issuecomment-1772420811 @exceptionfactory : Thanks for the detailed review. I have addressed most of your comments, please feel free to review and comment if anything is missing. Thanks once again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
tanmaya-panda1 commented on code in PR #7624:
URL: https://github.com/apache/nifi/pull/7624#discussion_r1366722167
##
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-services-api/src/main/java/org/apache/nifi/services/azure/data/explorer/KustoIngestService.java:
##
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.services.azure.data.explorer;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+
+import java.net.URISyntaxException;
+
+@Tags({"azure", "adx", "explorer", "kusto", "ingest"})
+@CapabilityDescription("Connection-Service for Azure ADX (Kusto) ingestion
cluster to ingest data.")
+public interface KustoIngestService extends KustoQueryService {
Review Comment:
While using KustoIngestService, we will need to query for ingestor
privileges and then start ingestion hence we need executeQuery during the
ingest service. We didn't want to duplicate executeQuery in KustoIngestService,
so extended KustoQueryService.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
exceptionfactory commented on code in PR #7624:
URL: https://github.com/apache/nifi/pull/7624#discussion_r1364558334
##
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector
which sends flowFiles using the ADX-Service to the provided Azure Data" +
+"Explorer Ingest Endpoint. The data can be sent through queued
ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@ReadsAttributes({
+@ReadsAttribute(attribute = "DB_NAME", description = "Specifies the
name of the ADX database where the data needs to be stored."),
+@ReadsAttribute(attribute = "TABLE_NAME", description = "Specifies the
name of the ADX table where the data needs to be stored."),
+@ReadsAttribute(attribute = "MAPPING_NAME", description = "Specifies
the name of the mapping responsible for storing the data in appropriate
columns."),
+@ReadsAttribute(attribute = "FLUSH_IMMEDIATE", description = "In case
of queued ingestion, this property determines whether the data should be
flushed immediately to the ingest endpoint."),
+@ReadsAttribute(attribute = "DATA_FORMAT", description = "Specifies
the format of data that is send to Azure Data Explorer."),
+@ReadsAttribute(attribute = "IGNORE_FIRST_RECORD", description =
"Specifies whether we want to ignore ingestion of first record. " +
+"This is primarily applicable for csv files. Default is set to
NO"),
+@ReadsAttribute(attribute = "POLL_ON_INGESTION_STATUS", description =
"Specifies whether we want to poll on ingestion result during ingestion into
ADX." +
+"In case of applications that need high throughput it is
recommended to keep the default value as false. Default is set to false." +
+"This property should be set to true during Queued Ingestion
for near realtime micro-batches of data that require acknowledgement of
ingestion status.")
+})
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+public static final String FETCH_TABLE_COMMAND = "%s | count";
+public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s
policy streamingingestion";
+public static final String
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
tanmaya-panda1 commented on code in PR #7624:
URL: https://github.com/apache/nifi/pull/7624#discussion_r1364001567
##
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##
@@ -0,0 +1,399 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import com.microsoft.azure.kusto.data.KustoResultSetTable;
+import com.microsoft.azure.kusto.ingest.IngestionMapping;
+import com.microsoft.azure.kusto.ingest.IngestionProperties;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+import org.apache.nifi.services.azure.data.explorer.KustoQueryResponse;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector
which sends flowFiles using the ADX-Service to the provided Azure Data" +
+"Explorer Ingest Endpoint. The data can be sent through queued
ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@ReadsAttributes({
+@ReadsAttribute(attribute = "DB_NAME", description = "Specifies the
name of the ADX database where the data needs to be stored."),
+@ReadsAttribute(attribute = "TABLE_NAME", description = "Specifies the
name of the ADX table where the data needs to be stored."),
+@ReadsAttribute(attribute = "MAPPING_NAME", description = "Specifies
the name of the mapping responsible for storing the data in appropriate
columns."),
+@ReadsAttribute(attribute = "FLUSH_IMMEDIATE", description = "In case
of queued ingestion, this property determines whether the data should be
flushed immediately to the ingest endpoint."),
+@ReadsAttribute(attribute = "DATA_FORMAT", description = "Specifies
the format of data that is send to Azure Data Explorer."),
+@ReadsAttribute(attribute = "IGNORE_FIRST_RECORD", description =
"Specifies whether we want to ignore ingestion of first record. " +
+"This is primarily applicable for csv files. Default is set to
NO"),
+@ReadsAttribute(attribute = "POLL_ON_INGESTION_STATUS", description =
"Specifies whether we want to poll on ingestion result during ingestion into
ADX." +
+"In case of applications that need high throughput it is
recommended to keep the default value as false. Default is set to false." +
+"This property should be set to true during Queued Ingestion
for near realtime micro-batches of data that require acknowledgement of
ingestion status.")
+})
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+public static final String FETCH_TABLE_COMMAND = "%s | count";
+public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s
policy streamingingestion";
+public static final String
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
kustonaut commented on PR #7624: URL: https://github.com/apache/nifi/pull/7624#issuecomment-1766312186 @joewitt & @exceptionfactory - Thank you so much for your review and comments. @tanmaya-panda1 has fixed the issues. Could you please take a look when feasible? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
tanmaya-panda1 commented on PR #7624: URL: https://github.com/apache/nifi/pull/7624#issuecomment-1748661908 @exceptionfactory , Hope you are doing well. Can you please take a look the PR again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
