Re: [PR] NIFI-12368 Clear versionedComponentId on top-level components of popu… [nifi]
exceptionfactory commented on code in PR #8025: URL: https://github.com/apache/nifi/pull/8025#discussion_r1395098136 ## nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/util/SnippetUtils.java: ## @@ -404,9 +404,26 @@ private Set getControllerServices(final Map
Re: [PR] NIFI-8932: Add capability to skip first N rows in CSVReader [nifi]
exceptionfactory closed pull request #7952: NIFI-8932: Add capability to skip first N rows in CSVReader URL: https://github.com/apache/nifi/pull/7952 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-8932: Add capability to skip first N rows in CSVReader [nifi]
exceptionfactory commented on PR #7952: URL: https://github.com/apache/nifi/pull/7952#issuecomment-1813707051 @mattyb149 Just to streamline the reviews, I am closing this pull request for now given the test failures across the board, and the implementation concerns. Feel free to reopen whenever you are ready. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12351 Improve MiNiFi Service Scripts for Windows [nifi]
exceptionfactory commented on PR #8015: URL: https://github.com/apache/nifi/pull/8015#issuecomment-1813705379 Concurring with @bejancsaba, it would be helpful to elaborate on the goal of these changes. I updated the pull request title to reflect that fact that the changes related to service scripts for MiNiFi on Windows, but it would be helpful to summarize the intended improvements. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12379) Refactor nifi-uuid5 to Remove External Dependencies
[ https://issues.apache.org/jira/browse/NIFI-12379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12379: Status: Patch Available (was: Open) > Refactor nifi-uuid5 to Remove External Dependencies > --- > > Key: NIFI-12379 > URL: https://issues.apache.org/jira/browse/NIFI-12379 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: David Handermann >Assignee: David Handermann >Priority: Minor > Fix For: 2.latest > > Time Spent: 10m > Remaining Estimate: 0h > > The nifi-uuid5 module provides the Uuid5Util class for generating a UUID > string according to RFC 4122 Section 4.3 with SHA-1 hashing for version 5 > generation. The initial version used external libraries for hexadecimal > encoding and hashing, but these libraries can be removed in favor of standard > JDK HexFormat and MessageDigest classes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] NIFI-12379 Refactor nifi-uuid5 using standard classes [nifi]
exceptionfactory opened a new pull request, #8035: URL: https://github.com/apache/nifi/pull/8035 # Summary [NIFI-12379](https://issues.apache.org/jira/browse/NIFI-12379) Refactors the `Uuid5Util` class in `nifi-uuid5` using standard Java classes to remove dependencies on Apache Commons Lang3 and Apache Commons Codec. The Java [HexFormat](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/util/HexFormat.html) class provides hexadecimal encoding and the Java `MessageDigest` supports SHA-1. # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X] Pull Request based on current revision of the `main` branch - [X] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [X] Build completed using `mvn clean install -P contrib-check` - [X] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-12379) Refactor nifi-uuid5 to Remove External Dependencies
David Handermann created NIFI-12379: --- Summary: Refactor nifi-uuid5 to Remove External Dependencies Key: NIFI-12379 URL: https://issues.apache.org/jira/browse/NIFI-12379 Project: Apache NiFi Issue Type: Improvement Components: Core Framework Reporter: David Handermann Assignee: David Handermann Fix For: 2.latest The nifi-uuid5 module provides the Uuid5Util class for generating a UUID string according to RFC 4122 Section 4.3 with SHA-1 hashing for version 5 generation. The initial version used external libraries for hexadecimal encoding and hashing, but these libraries can be removed in favor of standard JDK HexFormat and MessageDigest classes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11171) Reorganize Standard Components for 2.0.0
[
https://issues.apache.org/jira/browse/NIFI-11171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-11171:
Fix Version/s: 2.latest
> Reorganize Standard Components for 2.0.0
>
>
> Key: NIFI-11171
> URL: https://issues.apache.org/jira/browse/NIFI-11171
> Project: Apache NiFi
> Issue Type: Epic
>Reporter: David Handermann
>Assignee: David Handermann
>Priority: Major
> Fix For: 2.latest
>
>
> The {{nifi-standard-processors}} and {{nifi-standard-services}} modules
> include a large number of Processors and Controller Services supporting an
> array of capabilities. Some of these capabilities require specialized
> libraries that apply to a limited number of components.
> Moving Processors and Controller Services changes class names and bundle
> coordinates which will break existing flow configurations. For this reason,
> the selection of components for reorganization should be limited and focused.
> Components with less frequent updates or usage and components with large
> dependencies trees should be considered.
> The following items should be considered as described in the [NiFi 2.0
> Release
> Goals|https://cwiki.apache.org/confluence/display/NIFI/NiFi+2.0+Release+Goals]:
> * SFTP Processors
> * Jolt Transform Processors
> * Jetty HTTP Processors
> * JSON Processors
> * Netty-based Processors
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] NIFI-11129 - adding PutMongoBulk processor to use the bulkWrite API [nifi]
exceptionfactory commented on code in PR #6918:
URL: https://github.com/apache/nifi/pull/6918#discussion_r1395071266
##
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/PutMongoBulkOperations.java:
##
@@ -0,0 +1,320 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.mongodb;
+
+import com.mongodb.WriteConcern;
+import com.mongodb.client.ClientSession;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.client.model.BulkWriteOptions;
+import com.mongodb.client.model.Collation;
+import com.mongodb.client.model.CollationAlternate;
+import com.mongodb.client.model.CollationCaseFirst;
+import com.mongodb.client.model.CollationMaxVariable;
+import com.mongodb.client.model.CollationStrength;
+import com.mongodb.client.model.DeleteManyModel;
+import com.mongodb.client.model.DeleteOneModel;
+import com.mongodb.client.model.DeleteOptions;
+import com.mongodb.client.model.InsertOneModel;
+import com.mongodb.client.model.ReplaceOneModel;
+import com.mongodb.client.model.ReplaceOptions;
+import com.mongodb.client.model.UpdateManyModel;
+import com.mongodb.client.model.UpdateOneModel;
+import com.mongodb.client.model.UpdateOptions;
+import com.mongodb.client.model.WriteModel;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SystemResource;
+import org.apache.nifi.annotation.behavior.SystemResourceConsideration;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.bson.BsonArray;
+import org.bson.BsonDocument;
+import org.bson.Document;
+import org.bson.codecs.BsonArrayCodec;
+import org.bson.codecs.DecoderContext;
+import org.bson.conversions.Bson;
+import org.bson.json.JsonReader;
+
+import java.io.ByteArrayOutputStream;
+import java.io.InputStreamReader;
+import java.io.Reader;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@Tags({ "mongodb", "insert", "update", "write", "put", "bulk" })
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Writes the contents of a FlowFile to MongoDB as
bulk-update")
+@SystemResourceConsideration(resource = SystemResource.MEMORY)
+public class PutMongoBulkOperations extends AbstractMongoProcessor {
+static final Relationship REL_SUCCESS = new
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to MongoDB are routed
to this relationship").build();
+static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to MongoDB are
routed to this relationship").build();
+
+static final PropertyDescriptor ORDERED = new PropertyDescriptor.Builder()
+.name("Ordered")
+.expressionLanguageSupported(ExpressionLanguageScope.NONE)
+.description("Ordered execution of bulk-writes & break on error -
otherwise arbitrary order & continue on error")
+.required(false)
+.addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+.allowableValues("true", "false")
+.defaultValue("true")
+.build();
+
+static final PropertyDescriptor TRANSACTIONS_ENABLED = new
PropertyDescriptor.Builder()
Review Comment:
Thanks for the reply. If you don't think it is necessary to support
transactions for now, it seems like it would be better to remove this property,
and the method from MongoDBClientService. Removing the option would
Re: [PR] NIFI-11958 Azure Data Explorer as sink [nifi]
exceptionfactory commented on code in PR #7624:
URL: https://github.com/apache/nifi/pull/7624#discussion_r1394354709
##
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/data/explorer/PutAzureDataExplorer.java:
##
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.data.explorer;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestDataFormat;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestQueryResponse;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestService;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionRequest;
+import org.apache.nifi.services.azure.data.explorer.KustoIngestionResult;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+@Tags({"Azure", "Kusto", "ADX", "Explorer", "Data"})
+@CapabilityDescription("The PutAzureDataExplorer acts as a ADX sink connector
which sends flowFiles using the ADX-Service to the provided Azure Data" +
+"Explorer Ingest Endpoint. The data can be sent through queued
ingestion or streaming ingestion to the Azure Data Explorer cluster.")
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+public class PutAzureDataExplorer extends AbstractProcessor {
+
+public static final String FETCH_TABLE_COMMAND = "%s | count";
+public static final String STREAMING_POLICY_SHOW_COMMAND = ".show %s %s
policy streamingingestion";
+public static final String DATABASE = "database";
+public static final AllowableValue IGNORE_FIRST_RECORD_YES = new
AllowableValue(
+"YES", "YES",
+"Ignore first record during ingestion");
+
+public static final AllowableValue IGNORE_FIRST_RECORD_NO = new
AllowableValue(
+"NO", "NO",
+"Do not ignore first record during ingestion");
+
+public static final PropertyDescriptor DATABASE_NAME = new
PropertyDescriptor.Builder()
+.name("Database Name")
+.displayName("Database Name")
+.description("Azure Data Explorer Database Name for querying")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.build();
+public static final PropertyDescriptor TABLE_NAME = new
PropertyDescriptor.Builder()
+.name("Table Name")
+.displayName("Table Name")
+.description("Azure Data Explorer Table Name for ingesting data")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.build();
+public static final PropertyDescriptor MAPPING_NAME = new
PropertyDescriptor
+.Builder().name("Ingest Mapping name")
+.displayName("Ingest Mapping name")
+.description("The name of the mapping responsible for storing the
data in the appropriate columns.")
+.required(false)
+
Re: [PR] NIFI-12372 [MiNiFi] Encrpyt raw flow sensitive properties [nifi]
exceptionfactory commented on code in PR #8028:
URL: https://github.com/apache/nifi/pull/8028#discussion_r1395048995
##
minifi/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/test/java/org/apache/nifi/minifi/c2/command/FlowPropertyEncryptorTest.java:
##
@@ -0,0 +1,338 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.minifi.c2.command;
+
+import static java.util.Map.entry;
+import static java.util.UUID.randomUUID;
+import static java.util.stream.Collectors.toMap;
+import static org.apache.commons.lang3.RandomStringUtils.randomAlphabetic;
+import static
org.apache.nifi.controller.serialization.FlowSerializer.ENC_PREFIX;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.stream.Stream;
+import org.apache.nifi.c2.protocol.component.api.Bundle;
+import org.apache.nifi.c2.protocol.component.api.ComponentManifest;
+import org.apache.nifi.c2.protocol.component.api.ControllerServiceDefinition;
+import org.apache.nifi.c2.protocol.component.api.ProcessorDefinition;
+import org.apache.nifi.c2.protocol.component.api.PropertyDescriptor;
+import org.apache.nifi.c2.protocol.component.api.RuntimeManifest;
+import org.apache.nifi.controller.flow.VersionedDataflow;
+import org.apache.nifi.encrypt.PropertyEncryptor;
+import org.apache.nifi.flow.VersionedConfigurableExtension;
+import org.apache.nifi.flow.VersionedControllerService;
+import org.apache.nifi.flow.VersionedParameter;
+import org.apache.nifi.flow.VersionedParameterContext;
+import org.apache.nifi.flow.VersionedProcessGroup;
+import org.apache.nifi.flow.VersionedProcessor;
+import org.apache.nifi.flow.VersionedPropertyDescriptor;
+import org.apache.nifi.minifi.commons.service.FlowSerDeService;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.mockito.Mock;
+import org.mockito.junit.jupiter.MockitoExtension;
+
+@ExtendWith(MockitoExtension.class)
+public class FlowPropertyEncryptorTest {
+
+private static final String PROCESSOR_TYPE_1 = "processor_type_1";
+private static final String PROCESSOR_TYPE_2 = "processor_type_2";
+private static final String PROCESSOR_TYPE_3 = "processor_type_3";
+private static final String CONTROLLER_SERVICE_TYPE_1 =
"controller_service_type_1";
+private static final String CONTROLLER_SERVICE_TYPE_2 =
"controller_service_type_2";
+private static final String CONTROLLER_SERVICE_TYPE_3 =
"controller_service_type_3";
+
+private static final String SENSITIVE_PROPERTY_NAME_PREFIX = "sensitive";
+
+private static final String NON_SENSITIVE_1 = "non-sensitive-1";
+private static final String SENSITIVE_1 = SENSITIVE_PROPERTY_NAME_PREFIX +
"-1";
+private static final String NON_SENSITIVE_2 = "non-sensitive-2";
+private static final String SENSITIVE_3 = SENSITIVE_PROPERTY_NAME_PREFIX +
"-3";
+
+private static final Map PARAMETERS1 = Map.of(
+NON_SENSITIVE_1, NON_SENSITIVE_1,
+SENSITIVE_1, SENSITIVE_1
+);
+private static final Map PARAMETERS2 = Map.of(
+NON_SENSITIVE_2, NON_SENSITIVE_2
+);
+
+private static final Map PARAMETERS3 = Map.of(
+SENSITIVE_3, SENSITIVE_3
+);
+private static final Map DESCRIPTORS1
= Map.of(
+NON_SENSITIVE_1, versionedPropertyDescriptor(NON_SENSITIVE_1, false),
+SENSITIVE_1, versionedPropertyDescriptor(SENSITIVE_1, true)
+);
+private static final Map DESCRIPTORS2
= Map.of(
+NON_SENSITIVE_2, versionedPropertyDescriptor(NON_SENSITIVE_2, false)
+);
+private static final Map DESCRIPTORS3
= Map.of(
+SENSITIVE_3, versionedPropertyDescriptor(SENSITIVE_3, true)
+);
+
+
+@Mock
+private PropertyEncryptor mockPropertyEncryptor;
+@Mock
[jira] [Updated] (NIFI-12374) Add Use Case based documentation for performing full/incremental loads
[ https://issues.apache.org/jira/browse/NIFI-12374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12374: Fix Version/s: 2.0.0-M1 (was: 2.latest) Resolution: Fixed Status: Resolved (was: Patch Available) > Add Use Case based documentation for performing full/incremental loads > -- > > Key: NIFI-12374 > URL: https://issues.apache.org/jira/browse/NIFI-12374 > Project: Apache NiFi > Issue Type: Task > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Minor > Fix For: 2.0.0-M1 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12374) Add Use Case based documentation for performing full/incremental loads
[ https://issues.apache.org/jira/browse/NIFI-12374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786568#comment-17786568 ] ASF subversion and git services commented on NIFI-12374: Commit e1942a8346ee2d4133c9ee92425eef599e883082 in nifi's branch refs/heads/main from Mark Payne [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=e1942a8346 ] NIFI-12374: Added use case based documentation to explain how to perform incremental / full loads from databases This closes #8030 Signed-off-by: David Handermann > Add Use Case based documentation for performing full/incremental loads > -- > > Key: NIFI-12374 > URL: https://issues.apache.org/jira/browse/NIFI-12374 > Project: Apache NiFi > Issue Type: Task > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Minor > Fix For: 2.latest > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12374: Added use case based documentation to explain how to perf… [nifi]
exceptionfactory closed pull request #8030: NIFI-12374: Added use case based documentation to explain how to perf… URL: https://github.com/apache/nifi/pull/8030 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-12358) NPE when configured network interfaces do not exist
[
https://issues.apache.org/jira/browse/NIFI-12358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786542#comment-17786542
]
ASF subversion and git services commented on NIFI-12358:
Commit 0a67620c0cd429c7c420e7405457d5defa0f678c in nifi's branch
refs/heads/support/nifi-1.x from David Handermann
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=0a67620c0c ]
NIFI-12358 Corrected syntax for Java 8
Signed-off-by: David Handermann
> NPE when configured network interfaces do not exist
> ---
>
> Key: NIFI-12358
> URL: https://issues.apache.org/jira/browse/NIFI-12358
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.20.0
>Reporter: Guillaume Lhermenier
>Assignee: Mark Payne
>Priority: Major
> Fix For: 2.0.0-M1, 1.24.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I recently had to switch our NiFi base AMIs in AWS from amazonlinux 2 to
> amazonlinux 2023.
> This went pretty smoothly but I an issue about network interfaces.
> For some reasons, I had the following configured in my nifi.properties :
> {code:java}
> nifi.web.https.host=nifi1.emea.qa.domain.io
> nifi.web.https.port=8443
> nifi.web.https.network.interface.eth0=eth0
> nifi.web.https.network.interface.eth1=eth1{code}
> And this worked for many years.
> However, in amazon Linux, networks seems to have changed and naming too.
> Instead of eth0/eth1, I had my network interfaces named ens5/ens6.
> Of course, NiFi wasn't able to find them.
> However, the log could be clearer than a NullPointerException
> {code:java}
> 2023-11-13 14:35:28,644 WARN [main] o.a.nifi.web.server.HostHeaderHandler
> Failed to determine custom network interfaces.
> java.lang.NullPointerException: null
> at
> org.apache.nifi.web.server.HostHeaderHandler.extractIPsFromNetworkInterfaces(HostHeaderHandler.java:335)
> at
> org.apache.nifi.web.server.HostHeaderHandler.generateDefaultHostnames(HostHeaderHandler.java:276)
> at
> org.apache.nifi.web.server.HostHeaderHandler.(HostHeaderHandler.java:100)
> at org.apache.nifi.web.server.JettyServer.init(JettyServer.java:217)
> at
> org.apache.nifi.web.server.JettyServer.initialize(JettyServer.java:1074)
> at org.apache.nifi.NiFi.(NiFi.java:164)
> at org.apache.nifi.NiFi.(NiFi.java:83)
> at org.apache.nifi.NiFi.main(NiFi.java:332)
> 2023-11-13 14:35:28,649 INFO [main] o.a.nifi.web.server.HostHeaderHandler
> Determined 14 valid hostnames and IP addresses for incoming headers:
> 127.0.0.1, 127.0.0.1:8443, localhost, localhost:8443, [::1], [::1]:8443,
> ip-172-30-xx-xx.eu-west-1.compute.internal,
> ip-172-30-xx-xx.eu-west-1.compute.internal:8443, 172.30.xx.xx,
> 172.30.xx.xx:8443, nifi1.emea.qa.domain.io, nifi1.emea.qa.domain.io:8443,
> nifi.emea.qa.domain.io, {code}
>
> NB : I hadn't tested this on newer versions than 1.20 and won't have time to
> in the coming weeks.
> However, our migration to 1.23 should be done in the next months, I'll update
> the ticket if needed at that time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12358) NPE when configured network interfaces do not exist
[
https://issues.apache.org/jira/browse/NIFI-12358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-12358:
Fix Version/s: 2.0.0-M1
1.24.0
(was: 2.latest)
Resolution: Fixed
Status: Resolved (was: Patch Available)
> NPE when configured network interfaces do not exist
> ---
>
> Key: NIFI-12358
> URL: https://issues.apache.org/jira/browse/NIFI-12358
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.20.0
>Reporter: Guillaume Lhermenier
>Assignee: Mark Payne
>Priority: Major
> Fix For: 2.0.0-M1, 1.24.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I recently had to switch our NiFi base AMIs in AWS from amazonlinux 2 to
> amazonlinux 2023.
> This went pretty smoothly but I an issue about network interfaces.
> For some reasons, I had the following configured in my nifi.properties :
> {code:java}
> nifi.web.https.host=nifi1.emea.qa.domain.io
> nifi.web.https.port=8443
> nifi.web.https.network.interface.eth0=eth0
> nifi.web.https.network.interface.eth1=eth1{code}
> And this worked for many years.
> However, in amazon Linux, networks seems to have changed and naming too.
> Instead of eth0/eth1, I had my network interfaces named ens5/ens6.
> Of course, NiFi wasn't able to find them.
> However, the log could be clearer than a NullPointerException
> {code:java}
> 2023-11-13 14:35:28,644 WARN [main] o.a.nifi.web.server.HostHeaderHandler
> Failed to determine custom network interfaces.
> java.lang.NullPointerException: null
> at
> org.apache.nifi.web.server.HostHeaderHandler.extractIPsFromNetworkInterfaces(HostHeaderHandler.java:335)
> at
> org.apache.nifi.web.server.HostHeaderHandler.generateDefaultHostnames(HostHeaderHandler.java:276)
> at
> org.apache.nifi.web.server.HostHeaderHandler.(HostHeaderHandler.java:100)
> at org.apache.nifi.web.server.JettyServer.init(JettyServer.java:217)
> at
> org.apache.nifi.web.server.JettyServer.initialize(JettyServer.java:1074)
> at org.apache.nifi.NiFi.(NiFi.java:164)
> at org.apache.nifi.NiFi.(NiFi.java:83)
> at org.apache.nifi.NiFi.main(NiFi.java:332)
> 2023-11-13 14:35:28,649 INFO [main] o.a.nifi.web.server.HostHeaderHandler
> Determined 14 valid hostnames and IP addresses for incoming headers:
> 127.0.0.1, 127.0.0.1:8443, localhost, localhost:8443, [::1], [::1]:8443,
> ip-172-30-xx-xx.eu-west-1.compute.internal,
> ip-172-30-xx-xx.eu-west-1.compute.internal:8443, 172.30.xx.xx,
> 172.30.xx.xx:8443, nifi1.emea.qa.domain.io, nifi1.emea.qa.domain.io:8443,
> nifi.emea.qa.domain.io, {code}
>
> NB : I hadn't tested this on newer versions than 1.20 and won't have time to
> in the coming weeks.
> However, our migration to 1.23 should be done in the next months, I'll update
> the ticket if needed at that time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12358) NPE when configured network interfaces do not exist
[
https://issues.apache.org/jira/browse/NIFI-12358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786541#comment-17786541
]
ASF subversion and git services commented on NIFI-12358:
Commit a78c478721a27cbb2f6e03c72f739de5a232824e in nifi's branch
refs/heads/support/nifi-1.x from Mark Payne
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=a78c478721 ]
NIFI-12358 Fixed NPE in HostHeaderHandler
Fixed NPE in ThreadDumpTask
This closes #8032
Signed-off-by: David Handermann
(cherry picked from commit b5c3e6f9aae9791b749ba3d908ce1b6672ee2124)
> NPE when configured network interfaces do not exist
> ---
>
> Key: NIFI-12358
> URL: https://issues.apache.org/jira/browse/NIFI-12358
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.20.0
>Reporter: Guillaume Lhermenier
>Assignee: Mark Payne
>Priority: Major
> Fix For: 2.latest
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I recently had to switch our NiFi base AMIs in AWS from amazonlinux 2 to
> amazonlinux 2023.
> This went pretty smoothly but I an issue about network interfaces.
> For some reasons, I had the following configured in my nifi.properties :
> {code:java}
> nifi.web.https.host=nifi1.emea.qa.domain.io
> nifi.web.https.port=8443
> nifi.web.https.network.interface.eth0=eth0
> nifi.web.https.network.interface.eth1=eth1{code}
> And this worked for many years.
> However, in amazon Linux, networks seems to have changed and naming too.
> Instead of eth0/eth1, I had my network interfaces named ens5/ens6.
> Of course, NiFi wasn't able to find them.
> However, the log could be clearer than a NullPointerException
> {code:java}
> 2023-11-13 14:35:28,644 WARN [main] o.a.nifi.web.server.HostHeaderHandler
> Failed to determine custom network interfaces.
> java.lang.NullPointerException: null
> at
> org.apache.nifi.web.server.HostHeaderHandler.extractIPsFromNetworkInterfaces(HostHeaderHandler.java:335)
> at
> org.apache.nifi.web.server.HostHeaderHandler.generateDefaultHostnames(HostHeaderHandler.java:276)
> at
> org.apache.nifi.web.server.HostHeaderHandler.(HostHeaderHandler.java:100)
> at org.apache.nifi.web.server.JettyServer.init(JettyServer.java:217)
> at
> org.apache.nifi.web.server.JettyServer.initialize(JettyServer.java:1074)
> at org.apache.nifi.NiFi.(NiFi.java:164)
> at org.apache.nifi.NiFi.(NiFi.java:83)
> at org.apache.nifi.NiFi.main(NiFi.java:332)
> 2023-11-13 14:35:28,649 INFO [main] o.a.nifi.web.server.HostHeaderHandler
> Determined 14 valid hostnames and IP addresses for incoming headers:
> 127.0.0.1, 127.0.0.1:8443, localhost, localhost:8443, [::1], [::1]:8443,
> ip-172-30-xx-xx.eu-west-1.compute.internal,
> ip-172-30-xx-xx.eu-west-1.compute.internal:8443, 172.30.xx.xx,
> 172.30.xx.xx:8443, nifi1.emea.qa.domain.io, nifi1.emea.qa.domain.io:8443,
> nifi.emea.qa.domain.io, {code}
>
> NB : I hadn't tested this on newer versions than 1.20 and won't have time to
> in the coming weeks.
> However, our migration to 1.23 should be done in the next months, I'll update
> the ticket if needed at that time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12358) NPE when configured network interfaces do not exist
[
https://issues.apache.org/jira/browse/NIFI-12358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786540#comment-17786540
]
ASF subversion and git services commented on NIFI-12358:
Commit b5c3e6f9aae9791b749ba3d908ce1b6672ee2124 in nifi's branch
refs/heads/main from Mark Payne
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=b5c3e6f9aa ]
NIFI-12358 Fixed NPE in HostHeaderHandler
Fixed NPE in ThreadDumpTask and TrackedLeaderElectionManager
This closes #8032
Signed-off-by: David Handermann
> NPE when configured network interfaces do not exist
> ---
>
> Key: NIFI-12358
> URL: https://issues.apache.org/jira/browse/NIFI-12358
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.20.0
>Reporter: Guillaume Lhermenier
>Assignee: Mark Payne
>Priority: Major
> Fix For: 2.latest
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I recently had to switch our NiFi base AMIs in AWS from amazonlinux 2 to
> amazonlinux 2023.
> This went pretty smoothly but I an issue about network interfaces.
> For some reasons, I had the following configured in my nifi.properties :
> {code:java}
> nifi.web.https.host=nifi1.emea.qa.domain.io
> nifi.web.https.port=8443
> nifi.web.https.network.interface.eth0=eth0
> nifi.web.https.network.interface.eth1=eth1{code}
> And this worked for many years.
> However, in amazon Linux, networks seems to have changed and naming too.
> Instead of eth0/eth1, I had my network interfaces named ens5/ens6.
> Of course, NiFi wasn't able to find them.
> However, the log could be clearer than a NullPointerException
> {code:java}
> 2023-11-13 14:35:28,644 WARN [main] o.a.nifi.web.server.HostHeaderHandler
> Failed to determine custom network interfaces.
> java.lang.NullPointerException: null
> at
> org.apache.nifi.web.server.HostHeaderHandler.extractIPsFromNetworkInterfaces(HostHeaderHandler.java:335)
> at
> org.apache.nifi.web.server.HostHeaderHandler.generateDefaultHostnames(HostHeaderHandler.java:276)
> at
> org.apache.nifi.web.server.HostHeaderHandler.(HostHeaderHandler.java:100)
> at org.apache.nifi.web.server.JettyServer.init(JettyServer.java:217)
> at
> org.apache.nifi.web.server.JettyServer.initialize(JettyServer.java:1074)
> at org.apache.nifi.NiFi.(NiFi.java:164)
> at org.apache.nifi.NiFi.(NiFi.java:83)
> at org.apache.nifi.NiFi.main(NiFi.java:332)
> 2023-11-13 14:35:28,649 INFO [main] o.a.nifi.web.server.HostHeaderHandler
> Determined 14 valid hostnames and IP addresses for incoming headers:
> 127.0.0.1, 127.0.0.1:8443, localhost, localhost:8443, [::1], [::1]:8443,
> ip-172-30-xx-xx.eu-west-1.compute.internal,
> ip-172-30-xx-xx.eu-west-1.compute.internal:8443, 172.30.xx.xx,
> 172.30.xx.xx:8443, nifi1.emea.qa.domain.io, nifi1.emea.qa.domain.io:8443,
> nifi.emea.qa.domain.io, {code}
>
> NB : I hadn't tested this on newer versions than 1.20 and won't have time to
> in the coming weeks.
> However, our migration to 1.23 should be done in the next months, I'll update
> the ticket if needed at that time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] NIFI-12358: Fixed NPE identified in Jira, as well as another that was… [nifi]
exceptionfactory closed pull request #8032: NIFI-12358: Fixed NPE identified in Jira, as well as another that was… URL: https://github.com/apache/nifi/pull/8032 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-12376) Logic error in AvroReader
[ https://issues.apache.org/jira/browse/NIFI-12376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786512#comment-17786512 ] ASF subversion and git services commented on NIFI-12376: Commit 75341cc5face0d48d22baaf4bd9398a9fd114f2c in nifi's branch refs/heads/support/nifi-1.x from Matt Burgess [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=75341cc5fa ] NIFI-12376: Fix logic error with bitwise operator in AvroReader Signed-off-by: Pierre Villard This closes #8034. > Logic error in AvroReader > - > > Key: NIFI-12376 > URL: https://issues.apache.org/jira/browse/NIFI-12376 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.latest, 2.latest > > Time Spent: 20m > Remaining Estimate: 0h > > There is a logic error in AvroReader.createRecordReader() where a "bitwise > and" operator is used where a "logical and" operator should be used instead. > It's an edge case to get it to surface, you have to be using AvroReader with > a schema registry that does not return a Schema Format (rather than the usual > configuration of using the schema embedded in the Avro file). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12376) Logic error in AvroReader
[ https://issues.apache.org/jira/browse/NIFI-12376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786511#comment-17786511 ] ASF subversion and git services commented on NIFI-12376: Commit 685c6861b3aa23fff56607bc4732c11e3a5680cc in nifi's branch refs/heads/main from Matt Burgess [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=685c6861b3 ] NIFI-12376: Fix logic error with bitwise operator in AvroReader Signed-off-by: Pierre Villard This closes #8034. > Logic error in AvroReader > - > > Key: NIFI-12376 > URL: https://issues.apache.org/jira/browse/NIFI-12376 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.latest, 2.latest > > Time Spent: 10m > Remaining Estimate: 0h > > There is a logic error in AvroReader.createRecordReader() where a "bitwise > and" operator is used where a "logical and" operator should be used instead. > It's an edge case to get it to surface, you have to be using AvroReader with > a schema registry that does not return a Schema Format (rather than the usual > configuration of using the schema embedded in the Avro file). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12376) Logic error in AvroReader
[ https://issues.apache.org/jira/browse/NIFI-12376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-12376: -- Fix Version/s: 2.0.0-M1 1.24.0 (was: 1.latest) (was: 2.latest) Resolution: Fixed Status: Resolved (was: Patch Available) > Logic error in AvroReader > - > > Key: NIFI-12376 > URL: https://issues.apache.org/jira/browse/NIFI-12376 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 2.0.0-M1, 1.24.0 > > Time Spent: 20m > Remaining Estimate: 0h > > There is a logic error in AvroReader.createRecordReader() where a "bitwise > and" operator is used where a "logical and" operator should be used instead. > It's an edge case to get it to surface, you have to be using AvroReader with > a schema registry that does not return a Schema Format (rather than the usual > configuration of using the schema embedded in the Avro file). -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12376: Fix logic error with bitwise operator in AvroReader [nifi]
asfgit closed pull request #8034: NIFI-12376: Fix logic error with bitwise operator in AvroReader URL: https://github.com/apache/nifi/pull/8034 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12375) Remove maxFileSize from non-applicable Appenders
[ https://issues.apache.org/jira/browse/NIFI-12375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-12375: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Remove maxFileSize from non-applicable Appenders > > > Key: NIFI-12375 > URL: https://issues.apache.org/jira/browse/NIFI-12375 > Project: Apache NiFi > Issue Type: Bug >Reporter: David Handermann >Assignee: David Handermann >Priority: Trivial > Fix For: 2.0.0-M1 > > Time Spent: 20m > Remaining Estimate: 0h > > Recent updates on the main branch to the default Logback configuration > included a maxFileSize setting for several appenders. This does not cause any > behavioral change, but Logback writes warning messages on shutdown due to the > non-applicable setting. Removing the non-applicable maxFileSize property from > TimeBasedAppenders resolves the warning. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12375) Remove maxFileSize from non-applicable Appenders
[ https://issues.apache.org/jira/browse/NIFI-12375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-12375: -- Fix Version/s: 2.0.0-M1 (was: 2.latest) > Remove maxFileSize from non-applicable Appenders > > > Key: NIFI-12375 > URL: https://issues.apache.org/jira/browse/NIFI-12375 > Project: Apache NiFi > Issue Type: Bug >Reporter: David Handermann >Assignee: David Handermann >Priority: Trivial > Fix For: 2.0.0-M1 > > Time Spent: 20m > Remaining Estimate: 0h > > Recent updates on the main branch to the default Logback configuration > included a maxFileSize setting for several appenders. This does not cause any > behavioral change, but Logback writes warning messages on shutdown due to the > non-applicable setting. Removing the non-applicable maxFileSize property from > TimeBasedAppenders resolves the warning. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12375) Remove maxFileSize from non-applicable Appenders
[ https://issues.apache.org/jira/browse/NIFI-12375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786505#comment-17786505 ] ASF subversion and git services commented on NIFI-12375: Commit 76a27b0ad89b9fee380299d5c3b8833762a885ed in nifi's branch refs/heads/main from David Handermann [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=76a27b0ad8 ] NIFI-12375 Removed maxFileSize from non-applicable policies in Logback - Removed the non-applicable maxFileSize from TimeBasedRollingPolicy configurations in default logback.xml files to avoid warnings on shutdown Signed-off-by: Pierre Villard This closes #8033. > Remove maxFileSize from non-applicable Appenders > > > Key: NIFI-12375 > URL: https://issues.apache.org/jira/browse/NIFI-12375 > Project: Apache NiFi > Issue Type: Bug >Reporter: David Handermann >Assignee: David Handermann >Priority: Trivial > Fix For: 2.latest > > Time Spent: 10m > Remaining Estimate: 0h > > Recent updates on the main branch to the default Logback configuration > included a maxFileSize setting for several appenders. This does not cause any > behavioral change, but Logback writes warning messages on shutdown due to the > non-applicable setting. Removing the non-applicable maxFileSize property from > TimeBasedAppenders resolves the warning. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12375 Remove maxFileSize from non-applicable policies in Logback [nifi]
asfgit closed pull request #8033: NIFI-12375 Remove maxFileSize from non-applicable policies in Logback URL: https://github.com/apache/nifi/pull/8033 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-12378) Remove Jython support in NiFi 2.x given real Python support in 2.x
Joe Witt created NIFI-12378: --- Summary: Remove Jython support in NiFi 2.x given real Python support in 2.x Key: NIFI-12378 URL: https://issues.apache.org/jira/browse/NIFI-12378 Project: Apache NiFi Issue Type: Task Reporter: Joe Witt Assignee: Joe Witt -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-4947) Bind dynamic properties in ScriptedLookupService as variables in script
[ https://issues.apache.org/jira/browse/NIFI-4947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Witt resolved NIFI-4947. Resolution: Won't Fix > Bind dynamic properties in ScriptedLookupService as variables in script > --- > > Key: NIFI-4947 > URL: https://issues.apache.org/jira/browse/NIFI-4947 > Project: Apache NiFi > Issue Type: Wish > Components: Extensions >Affects Versions: 1.5.0 >Reporter: Mohd Izhar Firdaus Bin Ismail >Priority: Minor > > It would be good to have dynamic properties declared on the > ScriptedLookupService to be made available as variables in the script, like > in the ExecuteScript processor. It would help with use-cases where user want > to build a configurable LookupService using Jython etc. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-7370) Update scripted component documentation about Jython issues
[ https://issues.apache.org/jira/browse/NIFI-7370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Witt resolved NIFI-7370. Resolution: Won't Fix > Update scripted component documentation about Jython issues > --- > > Key: NIFI-7370 > URL: https://issues.apache.org/jira/browse/NIFI-7370 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Priority: Minor > > NIFI-4004 introduced a new default interface method for RecordReaderFactory, > in order > to avoid breaking API compatibility for those with custom > record readers. This exposes the bug in [1] for the Jython script > engine. At a major release point (NiFi 2.0) we could refactor the NiFi > codebase to remove the default interface method, update all internal > implementations, and announce that the LookupService API has changed > and thus custom implementations would have to be updated. Not sure if > we can get away with that for minor releases or not, usually breaking > API compatibility is a no-no except for major releases. > As was done in NIFI-5995 for ScriptedLookupService, this Jira proposes to > update the documentation for any other scripted component for which its > underlying interface has default methods, and also to remove the Jython > script engine from the allowable values for the Script Engine property. One > such component is ScriptedReader, but there may be others as well. > [1] [https://bugs.jython.org/issue2403|https://bugs.jython.org/issue2403] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-5434) ExecuteScript/Jython imports fail randomly
[
https://issues.apache.org/jira/browse/NIFI-5434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe Witt resolved NIFI-5434.
Resolution: Won't Fix
> ExecuteScript/Jython imports fail randomly
> --
>
> Key: NIFI-5434
> URL: https://issues.apache.org/jira/browse/NIFI-5434
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.6.0, 1.7.0, 1.7.1
> Environment: Debian Stretch (9.5)
>Reporter: Wietze B
>Priority: Major
> Attachments: ibm.png, java_thread.png, log_extract.log
>
>
> I have a number of ExecuteScript processors running Jython that import
> libraries such as {{threading}}, {{urllib3}}, {{requests}}, etc.
> When I boot up my machine or initially start the NiFi service, the
> ExecuteScript processors relying on external libraries usually to don't work
> straight away, throwing {{AttributeError}} s or {{ImportError}} s on
> libraries that definitely exist and are accessible. Restarting the machine or
> the NiFi service again usually fixes the problem, but it is unclear to me why
> the imports sometimes fail, and sometimes work.
> See attached two screenshots and an extract from the logs.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (NIFI-8615) ExecuteScript with python when use module directory
[
https://issues.apache.org/jira/browse/NIFI-8615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe Witt resolved NIFI-8615.
Resolution: Won't Fix
> ExecuteScript with python when use module directory
> ---
>
> Key: NIFI-8615
> URL: https://issues.apache.org/jira/browse/NIFI-8615
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.13.2
> Environment: debian linux
>Reporter: Aldric DUPONT
>Priority: Major
> Labels: Debian
> Attachments: Capture2.PNG, Capture3.PNG, test_bug.xml
>
>
>
> *I use additional module "pytz" in python, this module directory is locate in
> "/usr/local/lib/python3.9/site-packages"*
> *I use sample1 in "Script Body" : (no comment)*
> _from org.python.core.util.FileUtil import wrap_
> _from org.apache.nifi.processors.script import ExecuteScript_
> _from datetime import datetime, tzinfo_
> _flow_file = session.get()_
> _import pytz_
> _from pytz import timezone_
> _utc = pytz.utc_
> _eastern = timezone('US/Eastern')_
> _flow_file = session.putAttribute(flow_file, 'timezone', eastern.zone)_
> _flow_file = session.putAttribute(flow_file, 'utc', utc.zone)_
> _flow_file = session.putAttribute(flow_file, 'test', 'salut')_
> _session.transfer(flow_file, ExecuteScript.REL_SUCCESS)_
>
> *and sample2 with comment :* __
> _from org.python.core.util.FileUtil import wrap_
> _from org.apache.nifi.processors.script import ExecuteScript_
> _from datetime import datetime, tzinfo_
> _flow_file = session.get()_
> _"""_
> _import pytz_
> _from pytz import timezone_
> _utc = pytz.utc_
> _eastern = timezone('US/Eastern')_
> _flow_file = session.putAttribute(flow_file, 'timezone', eastern.zone)_
> _flow_file = session.putAttribute(flow_file, 'utc', utc.zone)_
> _"""_
> _flow_file = session.putAttribute(flow_file, 'test', 'salut')_
> _session.transfer(flow_file, ExecuteScript.REL_SUCCESS)_
>
> +*Try 1*+
> *When use sample1 in version 1.13.2 ExecuteScript make error " :*
> {color:#ff8b00}_ERROR [Timer-Driven Process Thread-4]
> o.a.nifi.processors.script.ExecuteScript
> ExecuteScript[id=790ea9a2-0179-1000-a662-30042349b329] Failed to process
> session due to org.apache.nifi.processor.exception.ProcessException:
> javax.script.ScriptException: ImportError: No module named pytz in
[jira] [Created] (NIFI-12377) Deprecate support for Jython in NiFi 1.x line
Joe Witt created NIFI-12377: --- Summary: Deprecate support for Jython in NiFi 1.x line Key: NIFI-12377 URL: https://issues.apache.org/jira/browse/NIFI-12377 Project: Apache NiFi Issue Type: Task Reporter: Joe Witt Assignee: Joe Witt Fix For: 1.latest Jython is not being actively maintained. There are a number of vulnerability reports tied to it. Python 2 is EOL. The NiFi 2.x line has real integration with real Python and is a better go forward approach. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12376) Logic error in AvroReader
[ https://issues.apache.org/jira/browse/NIFI-12376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-12376: Status: Patch Available (was: In Progress) > Logic error in AvroReader > - > > Key: NIFI-12376 > URL: https://issues.apache.org/jira/browse/NIFI-12376 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.latest, 2.latest > > Time Spent: 10m > Remaining Estimate: 0h > > There is a logic error in AvroReader.createRecordReader() where a "bitwise > and" operator is used where a "logical and" operator should be used instead. > It's an edge case to get it to surface, you have to be using AvroReader with > a schema registry that does not return a Schema Format (rather than the usual > configuration of using the schema embedded in the Avro file). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] NIFI-12376: Fix logic error with bitwise operator in AvroReader [nifi]
mattyb149 opened a new pull request, #8034: URL: https://github.com/apache/nifi/pull/8034 # Summary [NIFI-12376](https://issues.apache.org/jira/browse/NIFI-12376) This PR simply replaces a bitwise AND with a logical AND. There is no unit test at the time of PR creation because the logic is easily verified and cannot regress, as bitwise operators do not "short-circuit" and logical operators do. # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [x] Pull Request based on current revision of the `main` branch - [x] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [x] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (NIFI-12376) Logic error in AvroReader
[ https://issues.apache.org/jira/browse/NIFI-12376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-12376: --- Assignee: Matt Burgess > Logic error in AvroReader > - > > Key: NIFI-12376 > URL: https://issues.apache.org/jira/browse/NIFI-12376 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > > There is a logic error in AvroReader.createRecordReader() where a "bitwise > and" operator is used where a "logical and" operator should be used instead. > It's an edge case to get it to surface, you have to be using AvroReader with > a schema registry that does not return a Schema Format (rather than the usual > configuration of using the schema embedded in the Avro file). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12376) Logic error in AvroReader
[ https://issues.apache.org/jira/browse/NIFI-12376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-12376: Fix Version/s: 1.latest 2.latest > Logic error in AvroReader > - > > Key: NIFI-12376 > URL: https://issues.apache.org/jira/browse/NIFI-12376 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.latest, 2.latest > > > There is a logic error in AvroReader.createRecordReader() where a "bitwise > and" operator is used where a "logical and" operator should be used instead. > It's an edge case to get it to surface, you have to be using AvroReader with > a schema registry that does not return a Schema Format (rather than the usual > configuration of using the schema embedded in the Avro file). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12376) Logic error in AvroReader
Matt Burgess created NIFI-12376: --- Summary: Logic error in AvroReader Key: NIFI-12376 URL: https://issues.apache.org/jira/browse/NIFI-12376 Project: Apache NiFi Issue Type: Bug Components: Extensions Reporter: Matt Burgess There is a logic error in AvroReader.createRecordReader() where a "bitwise and" operator is used where a "logical and" operator should be used instead. It's an edge case to get it to surface, you have to be using AvroReader with a schema registry that does not return a Schema Format (rather than the usual configuration of using the schema embedded in the Avro file). -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-1874 Added Character Set Detection to IdentifyMimeType [nifi]
EndzeitBegins commented on PR #8031: URL: https://github.com/apache/nifi/pull/8031#issuecomment-1813195065 Perfect, thanks @exceptionfactory. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-1874) Improve IdentifyMimeType to detect character encoding in text data
[ https://issues.apache.org/jira/browse/NIFI-1874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786490#comment-17786490 ] ASF subversion and git services commented on NIFI-1874: --- Commit f94e65214bb800ef34ce3ff2e2223c1cc0837399 in nifi's branch refs/heads/support/nifi-1.x from EndzeitBegins [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f94e65214b ] NIFI-1874 Added Character Set Detection to IdentifyMimeType - Added mime.charset FlowFile attribute when not null for text MIME types This closes #8031 Signed-off-by: David Handermann > Improve IdentifyMimeType to detect character encoding in text data > -- > > Key: NIFI-1874 > URL: https://issues.apache.org/jira/browse/NIFI-1874 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Michael W Moser >Assignee: endzeit >Priority: Minor > Fix For: 2.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Leverage Apache Tika > [EncodingDetector|https://tika.apache.org/1.12/api/org/apache/tika/detect/EncodingDetector.html] > to additionally detect the character encoding of text data. > See > [AutoDetectReader|https://tika.apache.org/1.12/api/org/apache/tika/detect/AutoDetectReader.html] > too. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-1874) Improve IdentifyMimeType to detect character encoding in text data
[ https://issues.apache.org/jira/browse/NIFI-1874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-1874: --- Fix Version/s: 1.24.0 > Improve IdentifyMimeType to detect character encoding in text data > -- > > Key: NIFI-1874 > URL: https://issues.apache.org/jira/browse/NIFI-1874 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Michael W Moser >Assignee: endzeit >Priority: Minor > Fix For: 1.24.0, 2.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Leverage Apache Tika > [EncodingDetector|https://tika.apache.org/1.12/api/org/apache/tika/detect/EncodingDetector.html] > to additionally detect the character encoding of text data. > See > [AutoDetectReader|https://tika.apache.org/1.12/api/org/apache/tika/detect/AutoDetectReader.html] > too. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-1874 Added Character Set Detection to IdentifyMimeType [nifi]
exceptionfactory closed pull request #8031: NIFI-1874 Added Character Set Detection to IdentifyMimeType URL: https://github.com/apache/nifi/pull/8031 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-1874 Added Character Set Detection to IdentifyMimeType [nifi]
exceptionfactory commented on PR #8031: URL: https://github.com/apache/nifi/pull/8031#issuecomment-1813190005 Merged in https://github.com/apache/nifi/commit/f94e65214bb800ef34ce3ff2e2223c1cc0837399 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12375) Remove maxFileSize from non-applicable Appenders
[ https://issues.apache.org/jira/browse/NIFI-12375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12375: Status: Patch Available (was: Open) > Remove maxFileSize from non-applicable Appenders > > > Key: NIFI-12375 > URL: https://issues.apache.org/jira/browse/NIFI-12375 > Project: Apache NiFi > Issue Type: Bug >Reporter: David Handermann >Assignee: David Handermann >Priority: Trivial > Fix For: 2.latest > > Time Spent: 10m > Remaining Estimate: 0h > > Recent updates on the main branch to the default Logback configuration > included a maxFileSize setting for several appenders. This does not cause any > behavioral change, but Logback writes warning messages on shutdown due to the > non-applicable setting. Removing the non-applicable maxFileSize property from > TimeBasedAppenders resolves the warning. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] NIFI-12375 Remove maxFileSize from non-applicable policies in Logback [nifi]
exceptionfactory opened a new pull request, #8033: URL: https://github.com/apache/nifi/pull/8033 # Summary [NIFI-12375](https://issues.apache.org/jira/browse/NIFI-12375) Removes the `maxFileSize` settings from several `TimeBasedRollingPolicy` definitions in default logback.xml files. This settings does not impact behavior, but causes warnings on application shutdown because it does not apply to TimeBasedRollingPolicy configurations. # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X] Pull Request based on current revision of the `main` branch - [X] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (NIFI-12373) Add License and Notice to nifi-standard-shared-nar
[ https://issues.apache.org/jira/browse/NIFI-12373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard resolved NIFI-12373. --- Resolution: Fixed > Add License and Notice to nifi-standard-shared-nar > -- > > Key: NIFI-12373 > URL: https://issues.apache.org/jira/browse/NIFI-12373 > Project: Apache NiFi > Issue Type: Bug >Reporter: David Handermann >Assignee: David Handermann >Priority: Blocker > Fix For: 2.0.0-M1, 1.24.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Recent changes introduced the nifi-standard-shared-bundle module with a new > nifi-standard-shared-nar containing a limited set of dependencies that apply > to a number of extension components. The initial implementation did not > include the License and Notice files required for distribution of the NAR as > a separate package. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-4550 Detect charset for FlowFile with MIME type text/* [nifi]
EndzeitBegins commented on PR #8011: URL: https://github.com/apache/nifi/pull/8011#issuecomment-1813151208 The backport was more straightforward than I've expected. I've opened #8031. I'd highly appreciate if you'd take a look at it @exceptionfactory, as you're familiar with this original pull-request already. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-12373) Add License and Notice to nifi-standard-shared-nar
[ https://issues.apache.org/jira/browse/NIFI-12373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786478#comment-17786478 ] ASF subversion and git services commented on NIFI-12373: Commit bbe7bff4b302a7873f53f5bc5e75135b8166c7a1 in nifi's branch refs/heads/support/nifi-1.x from David Handermann [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=bbe7bff4b3 ] NIFI-12373 Added LICENSE and NOTICE for nifi-standard-shared-nar Signed-off-by: Pierre Villard This closes #8029. > Add License and Notice to nifi-standard-shared-nar > -- > > Key: NIFI-12373 > URL: https://issues.apache.org/jira/browse/NIFI-12373 > Project: Apache NiFi > Issue Type: Bug >Reporter: David Handermann >Assignee: David Handermann >Priority: Blocker > Fix For: 2.0.0-M1, 1.24.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Recent changes introduced the nifi-standard-shared-bundle module with a new > nifi-standard-shared-nar containing a limited set of dependencies that apply > to a number of extension components. The initial implementation did not > include the License and Notice files required for distribution of the NAR as > a separate package. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12373) Add License and Notice to nifi-standard-shared-nar
[ https://issues.apache.org/jira/browse/NIFI-12373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786476#comment-17786476 ] ASF subversion and git services commented on NIFI-12373: Commit 9b6120187a9592fd120d5ac6a3da3f66484ed8aa in nifi's branch refs/heads/main from David Handermann [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=9b6120187a ] NIFI-12373 Added LICENSE and NOTICE for nifi-standard-shared-nar Signed-off-by: Pierre Villard This closes #8029. > Add License and Notice to nifi-standard-shared-nar > -- > > Key: NIFI-12373 > URL: https://issues.apache.org/jira/browse/NIFI-12373 > Project: Apache NiFi > Issue Type: Bug >Reporter: David Handermann >Assignee: David Handermann >Priority: Blocker > Fix For: 2.0.0-M1, 1.24.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Recent changes introduced the nifi-standard-shared-bundle module with a new > nifi-standard-shared-nar containing a limited set of dependencies that apply > to a number of extension components. The initial implementation did not > include the License and Notice files required for distribution of the NAR as > a separate package. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12373 Add LICENSE and NOTICE for nifi-standard-shared-nar [nifi]
asfgit closed pull request #8029: NIFI-12373 Add LICENSE and NOTICE for nifi-standard-shared-nar URL: https://github.com/apache/nifi/pull/8029 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12358) NPE when configured network interfaces do not exist
[
https://issues.apache.org/jira/browse/NIFI-12358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-12358:
--
Fix Version/s: 2.latest
Assignee: Mark Payne
Status: Patch Available (was: Open)
> NPE when configured network interfaces do not exist
> ---
>
> Key: NIFI-12358
> URL: https://issues.apache.org/jira/browse/NIFI-12358
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.20.0
>Reporter: Guillaume Lhermenier
>Assignee: Mark Payne
>Priority: Major
> Fix For: 2.latest
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I recently had to switch our NiFi base AMIs in AWS from amazonlinux 2 to
> amazonlinux 2023.
> This went pretty smoothly but I an issue about network interfaces.
> For some reasons, I had the following configured in my nifi.properties :
> {code:java}
> nifi.web.https.host=nifi1.emea.qa.domain.io
> nifi.web.https.port=8443
> nifi.web.https.network.interface.eth0=eth0
> nifi.web.https.network.interface.eth1=eth1{code}
> And this worked for many years.
> However, in amazon Linux, networks seems to have changed and naming too.
> Instead of eth0/eth1, I had my network interfaces named ens5/ens6.
> Of course, NiFi wasn't able to find them.
> However, the log could be clearer than a NullPointerException
> {code:java}
> 2023-11-13 14:35:28,644 WARN [main] o.a.nifi.web.server.HostHeaderHandler
> Failed to determine custom network interfaces.
> java.lang.NullPointerException: null
> at
> org.apache.nifi.web.server.HostHeaderHandler.extractIPsFromNetworkInterfaces(HostHeaderHandler.java:335)
> at
> org.apache.nifi.web.server.HostHeaderHandler.generateDefaultHostnames(HostHeaderHandler.java:276)
> at
> org.apache.nifi.web.server.HostHeaderHandler.(HostHeaderHandler.java:100)
> at org.apache.nifi.web.server.JettyServer.init(JettyServer.java:217)
> at
> org.apache.nifi.web.server.JettyServer.initialize(JettyServer.java:1074)
> at org.apache.nifi.NiFi.(NiFi.java:164)
> at org.apache.nifi.NiFi.(NiFi.java:83)
> at org.apache.nifi.NiFi.main(NiFi.java:332)
> 2023-11-13 14:35:28,649 INFO [main] o.a.nifi.web.server.HostHeaderHandler
> Determined 14 valid hostnames and IP addresses for incoming headers:
> 127.0.0.1, 127.0.0.1:8443, localhost, localhost:8443, [::1], [::1]:8443,
> ip-172-30-xx-xx.eu-west-1.compute.internal,
> ip-172-30-xx-xx.eu-west-1.compute.internal:8443, 172.30.xx.xx,
> 172.30.xx.xx:8443, nifi1.emea.qa.domain.io, nifi1.emea.qa.domain.io:8443,
> nifi.emea.qa.domain.io, {code}
>
> NB : I hadn't tested this on newer versions than 1.20 and won't have time to
> in the coming weeks.
> However, our migration to 1.23 should be done in the next months, I'll update
> the ticket if needed at that time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] NIFI-12358: Fixed NPE identified in Jira, as well as another that was… [nifi]
markap14 opened a new pull request, #8032: URL: https://github.com/apache/nifi/pull/8032 … identified in the ThreadDumpTask and a third in TrackedLeaderElectionManager # Summary [NIFI-0](https://issues.apache.org/jira/browse/NIFI-0) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [ ] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-12375) Remove maxFileSize from non-applicable Appenders
David Handermann created NIFI-12375: --- Summary: Remove maxFileSize from non-applicable Appenders Key: NIFI-12375 URL: https://issues.apache.org/jira/browse/NIFI-12375 Project: Apache NiFi Issue Type: Bug Reporter: David Handermann Assignee: David Handermann Fix For: 2.latest Recent updates on the main branch to the default Logback configuration included a maxFileSize setting for several appenders. This does not cause any behavioral change, but Logback writes warning messages on shutdown due to the non-applicable setting. Removing the non-applicable maxFileSize property from TimeBasedAppenders resolves the warning. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12313 - Update PutDatabaseRecord.java add property use database table column datatype [nifi]
avasquezkudaw commented on PR #7981: URL: https://github.com/apache/nifi/pull/7981#issuecomment-1813137155 Yes, I agree that fixing the millisecond truncation issue is the best option. Regarding your question, we have tested both functionalities, inferschema and setting a schema, and the result is the same in both cases. I am attaching both flows for versions 1.12 and 1.23. [NiFi_Flow_112.json](https://github.com/apache/nifi/files/13368857/NiFi_Flow_112.json) [NiFi_Flow_123.json](https://github.com/apache/nifi/files/13368858/NiFi_Flow_123.json) Another test we conducted is to transform the record into Avro format, but the result is the same as well.   -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12373 Add LICENSE and NOTICE for nifi-standard-shared-nar [nifi]
joewitt commented on PR #8029: URL: https://github.com/apache/nifi/pull/8029#issuecomment-1813134116 Looks good +1. Probably worth in another effort going through task removing these from any nars that actually no longer package them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12374) Add Use Case based documentation for performing full/incremental loads
[ https://issues.apache.org/jira/browse/NIFI-12374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-12374: -- Status: Patch Available (was: Open) > Add Use Case based documentation for performing full/incremental loads > -- > > Key: NIFI-12374 > URL: https://issues.apache.org/jira/browse/NIFI-12374 > Project: Apache NiFi > Issue Type: Task > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Minor > Fix For: 2.latest > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] NIFI-1874 Added Character Set Detection to IdentifyMimeType [nifi]
EndzeitBegins opened a new pull request, #8031: URL: https://github.com/apache/nifi/pull/8031 - Added mime.charset FlowFile attribute when not null for text MIME types # Summary [NIFI-1874](https://issues.apache.org/jira/browse/NIFI-1874) [NIFI-4550](https://issues.apache.org/jira/browse/NIFI-4550) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [x] Pull Request based on current revision of the ~~`main`~~`support/nifi-1.x` branch - [x] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [x] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 21 ### Licensing - [x] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [x] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [x] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] NIFI-12374: Added use case based documentation to explain how to perf… [nifi]
markap14 opened a new pull request, #8030: URL: https://github.com/apache/nifi/pull/8030 …orm incremental / full loads from databases # Summary [NIFI-0](https://issues.apache.org/jira/browse/NIFI-0) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [ ] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] NIFI-12373 Add LICENSE and NOTICE for nifi-standard-shared-nar [nifi]
exceptionfactory opened a new pull request, #8029: URL: https://github.com/apache/nifi/pull/8029 # Summary [NIFI-12373](https://issues.apache.org/jira/browse/NIFI-12373) Adds `LICENSE` and `NOTICE` files to `nifi-standard-shared-nar` which were not present in the initial version. These files follow the recommendations from the [Assembling LICENSE and NOTICE files](https://infra.apache.org/licensing-howto.html) on Apache Infrastructure documentation. The `LICENSE` includes the standard Apache Software License Version 2 and the MIT License for the Bouncy Castle and Checker Framework libraries. The `NOTICE` enumerates all dependencies. Most of the dependencies are licensed under ASLv2 and do not have explicit notices, but are listed for completeness. This addition should be applied to both the main and support branches. The current contents of the `nifi-standard-shared-nar` is as follows: ``` annotations-13.0.jar bcpkix-jdk18on-1.76.jar bcprov-jdk18on-1.76.jar bcutil-jdk18on-1.76.jar caffeine-3.1.8.jar checker-qual-3.37.0.jar commons-codec-1.16.0.jar commons-compress-1.24.0.jar commons-io-2.15.0.jar commons-lang3-3.13.0.jar commons-net-3.10.0.jar commons-text-1.11.0.jar error_prone_annotations-2.21.1.jar jackson-annotations-2.15.3.jar jackson-core-2.15.3.jar jackson-databind-2.15.3.jar kotlin-stdlib-1.9.10.jar kotlin-stdlib-common-1.9.10.jar kotlin-stdlib-jdk7-1.9.10.jar kotlin-stdlib-jdk8-1.9.10.jar netty-buffer-4.1.100.Final.jar netty-codec-4.1.100.Final.jar netty-common-4.1.100.Final.jar netty-handler-4.1.100.Final.jar netty-resolver-4.1.100.Final.jar netty-transport-4.1.100.Final.jar netty-transport-native-unix-common-4.1.100.Final.jar okhttp-4.12.0.jar okio-3.6.0.jar okio-jvm-3.6.0.jar ``` # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X] Pull Request based on current revision of the `main` branch - [X] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-12374) Add Use Case based documentation for performing full/incremental loads
Mark Payne created NIFI-12374: - Summary: Add Use Case based documentation for performing full/incremental loads Key: NIFI-12374 URL: https://issues.apache.org/jira/browse/NIFI-12374 Project: Apache NiFi Issue Type: Task Components: Extensions Reporter: Mark Payne Assignee: Mark Payne Fix For: 2.latest -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12373) Add License and Notice to nifi-standard-shared-nar
David Handermann created NIFI-12373: --- Summary: Add License and Notice to nifi-standard-shared-nar Key: NIFI-12373 URL: https://issues.apache.org/jira/browse/NIFI-12373 Project: Apache NiFi Issue Type: Bug Reporter: David Handermann Assignee: David Handermann Fix For: 2.0.0-M1, 1.24.0 Recent changes introduced the nifi-standard-shared-bundle module with a new nifi-standard-shared-nar containing a limited set of dependencies that apply to a number of extension components. The initial implementation did not include the License and Notice files required for distribution of the NAR as a separate package. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12373) Add License and Notice to nifi-standard-shared-nar
[ https://issues.apache.org/jira/browse/NIFI-12373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12373: Priority: Blocker (was: Major) > Add License and Notice to nifi-standard-shared-nar > -- > > Key: NIFI-12373 > URL: https://issues.apache.org/jira/browse/NIFI-12373 > Project: Apache NiFi > Issue Type: Bug >Reporter: David Handermann >Assignee: David Handermann >Priority: Blocker > Fix For: 2.0.0-M1, 1.24.0 > > > Recent changes introduced the nifi-standard-shared-bundle module with a new > nifi-standard-shared-nar containing a limited set of dependencies that apply > to a number of extension components. The initial implementation did not > include the License and Notice files required for distribution of the NAR as > a separate package. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12313 - Update PutDatabaseRecord.java add property use database table column datatype [nifi]
mattyb149 commented on PR #7981: URL: https://github.com/apache/nifi/pull/7981#issuecomment-1812950274 Is your CSVReader using an explicit schema or Infer Schema? If the latter, I believe this issue is actually related to NiFi's handling of microseconds rather than the fact that it is using the target DB's column datatype (I have seen the same issue with JSON). If it worked before, I think it was "lucky" in the sense that the value was always treated like a string and in the SQL the driver accepts the string as a timestamp literal and preserves the microseconds. I'm not sure that if the value had been converted to a NiFi timestamp that the microseconds would be retained (and perhaps not the milliseconds either as you are seeing). I am still investigating. I'm not strongly opposed to having a property to configure this but we might be better served by fixing the code that truncates the fractional seconds, if that's what's going on here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-62 Add SplitText processor [nifi-minifi-cpp]
fgerlits commented on code in PR #1682:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1682#discussion_r1394163260
##
extensions/standard-processors/tests/unit/SplitTextTests.cpp:
##
@@ -0,0 +1,860 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "FlowFileRecord.h"
+#include "TestBase.h"
+#include "Catch.h"
+#include "processors/SplitText.h"
+#include "SingleProcessorTestController.h"
+#include "io/BufferStream.h"
+
+namespace org::apache::nifi::minifi::test {
+
+TEST_CASE("Test LineReader with nullptr") {
+ processors::detail::LineReader reader{nullptr};
+ CHECK(reader.readNextLine() == std::nullopt);
+ CHECK(reader.getState() == processors::detail::StreamReadState::EndOfStream);
+}
+
+TEST_CASE("Test LineReader with empty stream") {
+ auto stream = std::make_shared();
+ processors::detail::LineReader reader{nullptr};
+ CHECK(reader.readNextLine() == std::nullopt);
+ CHECK(reader.getState() == processors::detail::StreamReadState::EndOfStream);
+}
+
+TEST_CASE("Test LineReader with trailing endline") {
+ auto stream = std::make_shared();
+ std::string input = "this is a new line\nand another line\r\nthirdline\n";
+ stream->write(reinterpret_cast(input.data()), input.size());
+ processors::detail::LineReader reader{stream};
+ CHECK(*reader.readNextLine() ==
processors::detail::LineReader::LineInfo{.offset = 0, .size = 19, .endline_size
= 1});
+ CHECK(*reader.readNextLine() ==
processors::detail::LineReader::LineInfo{.offset = 19, .size = 18,
.endline_size = 2});
+ CHECK(*reader.readNextLine() ==
processors::detail::LineReader::LineInfo{.offset = 37, .size = 10,
.endline_size = 1});
+ CHECK(reader.readNextLine() == std::nullopt);
+ CHECK(reader.getState() == processors::detail::StreamReadState::EndOfStream);
+}
+
+TEST_CASE("Test LineReader without trailing endlines") {
+ auto stream = std::make_shared();
+ std::string input = "this is a new line\nand another line\r\nthirdline";
+ stream->write(reinterpret_cast(input.data()), input.size());
+ processors::detail::LineReader reader{stream};
+ CHECK(*reader.readNextLine() ==
processors::detail::LineReader::LineInfo{.offset = 0, .size = 19, .endline_size
= 1});
+ CHECK(*reader.readNextLine() ==
processors::detail::LineReader::LineInfo{.offset = 19, .size = 18,
.endline_size = 2});
+ CHECK(*reader.readNextLine() ==
processors::detail::LineReader::LineInfo{.offset = 37, .size = 9, .endline_size
= 0});
+ CHECK(reader.readNextLine() == std::nullopt);
+ CHECK(reader.getState() == processors::detail::StreamReadState::EndOfStream);
+}
+
+TEST_CASE("Test LineReader with input larger than buffer length") {
+ auto stream = std::make_shared();
+ const auto first_line_size =
static_cast(processors::detail::SPLIT_TEXT_BUFFER_SIZE * 1.5);
+ const auto second_line_size =
static_cast(processors::detail::SPLIT_TEXT_BUFFER_SIZE * 1.7);
+ std::string input = std::string(first_line_size, 'a') + "\n" +
std::string(second_line_size, 'b') + "\n";
+ stream->write(reinterpret_cast(input.data()), input.size());
+ processors::detail::LineReader reader{stream};
+ CHECK(*reader.readNextLine() ==
processors::detail::LineReader::LineInfo{.offset = 0, .size = first_line_size +
1, .endline_size = 1});
+ CHECK(*reader.readNextLine() ==
processors::detail::LineReader::LineInfo{.offset = first_line_size +1 , .size =
second_line_size + 1, .endline_size = 1});
+ CHECK(reader.readNextLine() == std::nullopt);
+ CHECK(reader.getState() == processors::detail::StreamReadState::EndOfStream);
+}
+
+TEST_CASE("Test LineReader with input of same size as buffer length") {
+ auto stream = std::make_shared();
+ std::string input = std::string(processors::detail::SPLIT_TEXT_BUFFER_SIZE -
1, 'a') + "\n" + std::string(processors::detail::SPLIT_TEXT_BUFFER_SIZE * 2 -
1, 'b') + "\n";
+ stream->write(reinterpret_cast(input.data()), input.size());
+ processors::detail::LineReader reader{stream};
+ CHECK(*reader.readNextLine() ==
processors::detail::LineReader::LineInfo{.offset = 0, .size =
processors::detail::SPLIT_TEXT_BUFFER_SIZE, .endline_size = 1});
+ CHECK(*reader.readNextLine() ==
+processors::detail::LineReader::LineInfo{.offset =
Re: [PR] MINIFICPP-1415 Pass references to onTrigger and onSchedule instead of… [nifi-minifi-cpp]
szaszm commented on code in PR #1693:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1693#discussion_r1394504470
##
extensions/http-curl/processors/InvokeHTTP.cpp:
##
Review Comment:
Same here: flow file attributes are no longer usable in the expression
language. While InvokeHTTP doesn't document any EL support, I'd rather not
break behavior, and possibly document the properties that actually support EL
with flow file context.
##
extensions/splunk/PutSplunkHTTP.cpp:
##
Review Comment:
The changes to this processor kill EL support: referencing flow file
attributes is no longer possible if we evaluate the changed properties in
`onSchedule`. I'd prefer to keep the old behavior.
##
extensions/standard-processors/processors/ListenTCP.h:
##
@@ -86,7 +86,7 @@ class ListenTCP : public NetworkListenerProcessor {
EXTENSIONAPI static constexpr auto OutputAttributes =
std::array{PortOutputAttribute, Sender};
void initialize() override;
- void onSchedule(const std::shared_ptr& context, const
std::shared_ptr& sessionFactory) override;
+ void onSchedule(core::ProcessContext& context, core::ProcessSessionFactory&
session_Factory) override;
Review Comment:
```suggestion
void onSchedule(core::ProcessContext& context,
core::ProcessSessionFactory& sessionFactory) override;
```
or
```suggestion
void onSchedule(core::ProcessContext& context,
core::ProcessSessionFactory& session_factory) override;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-4550 Detect charset for FlowFile with MIME type text/* [nifi]
EndzeitBegins commented on PR #8011: URL: https://github.com/apache/nifi/pull/8011#issuecomment-1812907291 Thank you for your reviews and the fast merge @exceptionfactory. I'll take a look at what's required for the backport in the next days presumably and open a new pull-request if the work required is manageable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-12362) Conduct Apache NiFi 1.24.0 Release
[ https://issues.apache.org/jira/browse/NIFI-12362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786407#comment-17786407 ] ASF subversion and git services commented on NIFI-12362: Commit 51c9537306cc34e3e466f549c38222f58a564cd9 in nifi's branch refs/heads/NIFI-12362-RC2 from Pierre Villard [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=51c9537306 ] NIFI-12362-RC2 prepare for next development iteration > Conduct Apache NiFi 1.24.0 Release > -- > > Key: NIFI-12362 > URL: https://issues.apache.org/jira/browse/NIFI-12362 > Project: Apache NiFi > Issue Type: Task >Reporter: David Handermann >Assignee: Pierre Villard >Priority: Major > Fix For: 1.24.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12362) Conduct Apache NiFi 1.24.0 Release
[ https://issues.apache.org/jira/browse/NIFI-12362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786406#comment-17786406 ] ASF subversion and git services commented on NIFI-12362: Commit b67f9add3f9cfe77146b8339c044f72f89684d51 in nifi's branch refs/heads/NIFI-12362-RC2 from Pierre Villard [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=b67f9add3f ] NIFI-12362-RC2 prepare release nifi-1.24.0-RC2 > Conduct Apache NiFi 1.24.0 Release > -- > > Key: NIFI-12362 > URL: https://issues.apache.org/jira/browse/NIFI-12362 > Project: Apache NiFi > Issue Type: Task >Reporter: David Handermann >Assignee: Pierre Villard >Priority: Major > Fix For: 1.24.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-4550) Add an InferCharacterSet processor
[ https://issues.apache.org/jira/browse/NIFI-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-4550: --- Resolution: Duplicate Status: Resolved (was: Patch Available) > Add an InferCharacterSet processor > -- > > Key: NIFI-4550 > URL: https://issues.apache.org/jira/browse/NIFI-4550 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: endzeit >Priority: Minor > Time Spent: 2h 10m > Remaining Estimate: 0h > > Sometimes in a NiFi flow it is not known what character set an incoming flow > file is using. This can make it difficult for downstream processing if the > processors expect a particular charset (whether the user can configure it or > not). There is a ConvertCharacterSet processor, but it expects an explicit > value for Input Character Set, when this might not be known. > I propose an InferCharacterSet processor, which would presumably use some > license-friendly third-party library (there is a discussion > [here|https://stackoverflow.com/questions/499010/java-how-to-determine-the-correct-charset-encoding-of-a-stream]) > to guess the character set, perhaps adding it as an attribute for use > downstream in ConvertCharacterSet. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-4550 Detect charset for FlowFile with MIME type text/* [nifi]
exceptionfactory closed pull request #8011: NIFI-4550 Detect charset for FlowFile with MIME type text/* URL: https://github.com/apache/nifi/pull/8011 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-1874) Improve IdentifyMimeType to detect character encoding in text data
[ https://issues.apache.org/jira/browse/NIFI-1874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-1874: --- Fix Version/s: 2.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Improve IdentifyMimeType to detect character encoding in text data > -- > > Key: NIFI-1874 > URL: https://issues.apache.org/jira/browse/NIFI-1874 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Michael W Moser >Assignee: endzeit >Priority: Minor > Fix For: 2.0.0 > > > Leverage Apache Tika > [EncodingDetector|https://tika.apache.org/1.12/api/org/apache/tika/detect/EncodingDetector.html] > to additionally detect the character encoding of text data. > See > [AutoDetectReader|https://tika.apache.org/1.12/api/org/apache/tika/detect/AutoDetectReader.html] > too. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-1874) Improve IdentifyMimeType to detect character encoding in text data
[ https://issues.apache.org/jira/browse/NIFI-1874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786399#comment-17786399 ] ASF subversion and git services commented on NIFI-1874: --- Commit 6c333cdb7e29c53a79a49179862e60b8fe0a3697 in nifi's branch refs/heads/main from EndzeitBegins [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=6c333cdb7e ] NIFI-1874 Added Character Set Detection to IdentifyMimeType NIFI-4550 New Processor not required based on improvements to IdentifyMimeType - Added mime.charset FlowFile attribute when not null for text MIME types This closes #8011 Signed-off-by: David Handermann > Improve IdentifyMimeType to detect character encoding in text data > -- > > Key: NIFI-1874 > URL: https://issues.apache.org/jira/browse/NIFI-1874 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Michael W Moser >Assignee: endzeit >Priority: Minor > > Leverage Apache Tika > [EncodingDetector|https://tika.apache.org/1.12/api/org/apache/tika/detect/EncodingDetector.html] > to additionally detect the character encoding of text data. > See > [AutoDetectReader|https://tika.apache.org/1.12/api/org/apache/tika/detect/AutoDetectReader.html] > too. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-4550) Add an InferCharacterSet processor
[ https://issues.apache.org/jira/browse/NIFI-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786400#comment-17786400 ] ASF subversion and git services commented on NIFI-4550: --- Commit 6c333cdb7e29c53a79a49179862e60b8fe0a3697 in nifi's branch refs/heads/main from EndzeitBegins [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=6c333cdb7e ] NIFI-1874 Added Character Set Detection to IdentifyMimeType NIFI-4550 New Processor not required based on improvements to IdentifyMimeType - Added mime.charset FlowFile attribute when not null for text MIME types This closes #8011 Signed-off-by: David Handermann > Add an InferCharacterSet processor > -- > > Key: NIFI-4550 > URL: https://issues.apache.org/jira/browse/NIFI-4550 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: endzeit >Priority: Minor > Time Spent: 2h 10m > Remaining Estimate: 0h > > Sometimes in a NiFi flow it is not known what character set an incoming flow > file is using. This can make it difficult for downstream processing if the > processors expect a particular charset (whether the user can configure it or > not). There is a ConvertCharacterSet processor, but it expects an explicit > value for Input Character Set, when this might not be known. > I propose an InferCharacterSet processor, which would presumably use some > license-friendly third-party library (there is a discussion > [here|https://stackoverflow.com/questions/499010/java-how-to-determine-the-correct-charset-encoding-of-a-stream]) > to guess the character set, perhaps adding it as an attribute for use > downstream in ConvertCharacterSet. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-4550) Add an InferCharacterSet processor
[ https://issues.apache.org/jira/browse/NIFI-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann reassigned NIFI-4550: -- Assignee: endzeit > Add an InferCharacterSet processor > -- > > Key: NIFI-4550 > URL: https://issues.apache.org/jira/browse/NIFI-4550 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: endzeit >Priority: Minor > Time Spent: 2h > Remaining Estimate: 0h > > Sometimes in a NiFi flow it is not known what character set an incoming flow > file is using. This can make it difficult for downstream processing if the > processors expect a particular charset (whether the user can configure it or > not). There is a ConvertCharacterSet processor, but it expects an explicit > value for Input Character Set, when this might not be known. > I propose an InferCharacterSet processor, which would presumably use some > license-friendly third-party library (there is a discussion > [here|https://stackoverflow.com/questions/499010/java-how-to-determine-the-correct-charset-encoding-of-a-stream]) > to guess the character set, perhaps adding it as an attribute for use > downstream in ConvertCharacterSet. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-4550) Add an InferCharacterSet processor
[ https://issues.apache.org/jira/browse/NIFI-4550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-4550: --- Status: Patch Available (was: Open) > Add an InferCharacterSet processor > -- > > Key: NIFI-4550 > URL: https://issues.apache.org/jira/browse/NIFI-4550 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: endzeit >Priority: Minor > Time Spent: 2h > Remaining Estimate: 0h > > Sometimes in a NiFi flow it is not known what character set an incoming flow > file is using. This can make it difficult for downstream processing if the > processors expect a particular charset (whether the user can configure it or > not). There is a ConvertCharacterSet processor, but it expects an explicit > value for Input Character Set, when this might not be known. > I propose an InferCharacterSet processor, which would presumably use some > license-friendly third-party library (there is a discussion > [here|https://stackoverflow.com/questions/499010/java-how-to-determine-the-correct-charset-encoding-of-a-stream]) > to guess the character set, perhaps adding it as an attribute for use > downstream in ConvertCharacterSet. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-1874) Improve IdentifyMimeType to detect character encoding in text data
[ https://issues.apache.org/jira/browse/NIFI-1874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-1874: --- Status: Patch Available (was: Open) > Improve IdentifyMimeType to detect character encoding in text data > -- > > Key: NIFI-1874 > URL: https://issues.apache.org/jira/browse/NIFI-1874 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Michael W Moser >Assignee: endzeit >Priority: Minor > > Leverage Apache Tika > [EncodingDetector|https://tika.apache.org/1.12/api/org/apache/tika/detect/EncodingDetector.html] > to additionally detect the character encoding of text data. > See > [AutoDetectReader|https://tika.apache.org/1.12/api/org/apache/tika/detect/AutoDetectReader.html] > too. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-1874) Improve IdentifyMimeType to detect character encoding in text data
[ https://issues.apache.org/jira/browse/NIFI-1874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann reassigned NIFI-1874: -- Assignee: endzeit > Improve IdentifyMimeType to detect character encoding in text data > -- > > Key: NIFI-1874 > URL: https://issues.apache.org/jira/browse/NIFI-1874 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Michael W Moser >Assignee: endzeit >Priority: Minor > > Leverage Apache Tika > [EncodingDetector|https://tika.apache.org/1.12/api/org/apache/tika/detect/EncodingDetector.html] > to additionally detect the character encoding of text data. > See > [AutoDetectReader|https://tika.apache.org/1.12/api/org/apache/tika/detect/AutoDetectReader.html] > too. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12372 [MiNiFi] Encrpyt raw flow sensitive properties [nifi]
briansolo1985 commented on PR #8028: URL: https://github.com/apache/nifi/pull/8028#issuecomment-1812566214 A flow incoming via C2 may contain unencrypted sensitive properties as the agents can have there our encryption key. The goal is to check the properties in the incoming flow, and if there are unencrypted sensitive properties there, use the configured encryption key to encrypt them before persisting the flow.json.raw to disk. The logic first tries to utilise the property descriptors sent with the flow to detect the sensitive property names. However the property descriptors can omitted. In this case the logic uses the runtime manifest to get the list of sensitive properties. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] NIFI-12372 [MiNiFi] Encrpyt raw flow sensitive properties [nifi]
briansolo1985 opened a new pull request, #8028: URL: https://github.com/apache/nifi/pull/8028 # Summary [NIFI-12372](https://issues.apache.org/jira/browse/NIFI-12372) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [x] Pull Request based on current revision of the `main` branch - [x] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [x] Build completed using `mvn clean install -P contrib-check` - [x] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-12372) [MiNiFi] Encrypt raw flow’s sensitive properties before storing it to the disk
Ferenc Kis created NIFI-12372: - Summary: [MiNiFi] Encrypt raw flow’s sensitive properties before storing it to the disk Key: NIFI-12372 URL: https://issues.apache.org/jira/browse/NIFI-12372 Project: Apache NiFi Issue Type: Improvement Components: MiNiFi Reporter: Ferenc Kis Assignee: Ferenc Kis Sensitive parameters in the downloaded (C2) raw flow json should be encrypted (if there are unencrypted ones) with the provided encryption key in bootstrap (props.sensitive.key). Make sure the same is used in the final flow.json as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12371) Support tombstone messages in non-record Kafka processors
[ https://issues.apache.org/jira/browse/NIFI-12371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-12371: -- Description: In some scenarios (compaction, transactions, etc), you may have Kafka tombstone messages (ie. the message has null for the content) and it may be important to have NiFi properly consume and publish those tombstone messages. As of now, when consuming Kafka messages, there is no way to differentiate a message that would be null (tombstone) and a message with an empty string). Also, as of now, when publishing a Kafka message, there is no way to send a null message, it would always be an empty string (empty byte array). This JIRA is about adding support for tombstone messages in ConsumeKafka_2_6 and PublishKafka_2_6 when not using a message demarcator. Follow up JIRAs could be filed to add support in the record based processors. was: In some scenarios (compaction, transactions, etc), you may have Kafka tombstone messages (ie. the message has null for the content) and it may be important to have NiFi properly consume and publish those tombstone messages. As of now, when consuming Kafka messages, there is no way to differentiate a message that would be null (tombstone) and a message with an empty string). Also, as of now, when publishing a Kafka message, there is no way to send a null message, it would always be an empty string (empty byte array). This JIRA is about adding support for tombstone messages in ConsumeKafka_2_6 and PublishKafka_2_6 when not using a delimiter. Follow up JIRAs could be filed to add support in the record based processors. > Support tombstone messages in non-record Kafka processors > - > > Key: NIFI-12371 > URL: https://issues.apache.org/jira/browse/NIFI-12371 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Minor > > In some scenarios (compaction, transactions, etc), you may have Kafka > tombstone messages (ie. the message has null for the content) and it may be > important to have NiFi properly consume and publish those tombstone messages. > As of now, when consuming Kafka messages, there is no way to differentiate a > message that would be null (tombstone) and a message with an empty string). > Also, as of now, when publishing a Kafka message, there is no way to send a > null message, it would always be an empty string (empty byte array). > This JIRA is about adding support for tombstone messages in ConsumeKafka_2_6 > and PublishKafka_2_6 when not using a message demarcator. Follow up JIRAs > could be filed to add support in the record based processors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12371) Support tombstone messages in non-record Kafka processors
Pierre Villard created NIFI-12371: - Summary: Support tombstone messages in non-record Kafka processors Key: NIFI-12371 URL: https://issues.apache.org/jira/browse/NIFI-12371 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: Pierre Villard Assignee: Pierre Villard In some scenarios (compaction, transactions, etc), you may have Kafka tombstone messages (ie. the message has null for the content) and it may be important to have NiFi properly consume and publish those tombstone messages. As of now, when consuming Kafka messages, there is no way to differentiate a message that would be null (tombstone) and a message with an empty string). Also, as of now, when publishing a Kafka message, there is no way to send a null message, it would always be an empty string (empty byte array). This JIRA is about adding support for tombstone messages in ConsumeKafka_2_6 and PublishKafka_2_6 when not using a delimiter. Follow up JIRAs could be filed to add support in the record based processors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12370) DistributedMapCacheClientService leaks threads
[
https://issues.apache.org/jira/browse/NIFI-12370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786307#comment-17786307
]
ASF subversion and git services commented on NIFI-12370:
Commit 8fda8f2ca57e0a0c746ca1614a0939a41e18375a in nifi's branch
refs/heads/support/nifi-1.x from David Handermann
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=8fda8f2ca5 ]
NIFI-12370 Switched from close() to shutdown() for Java 8
- ExecutorService implements AutoCloseable in Java 21 but not on Java 17 and
earlier
Signed-off-by: David Handermann
> DistributedMapCacheClientService leaks threads
> --
>
> Key: NIFI-12370
> URL: https://issues.apache.org/jira/browse/NIFI-12370
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.21.0, 1.22.0, 1.23.2
>Reporter: Eric Secules
>Assignee: David Handermann
>Priority: Major
> Fix For: 1.24.0, 2.0.0
>
> Attachments: Test_NIFI-12370.xml
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> # Using the attached flow template first enable and run the flow. This should
> exercise the cache.
> # run `nifi.sh diagnostics --verbose` and save the file
> # Disable and delete the controller services
> # run `nifi.sh diagnostics --verbose` and save the file with a different name
> # compare the thread dumps. The second file should not have any threads
> related to the DistributedMapCacheClientService but it will.
> # create and enable new controller services and link them to the Wait and
> Notify processors
> # run `nifi.sh diagnostics --verbose` and save the file with a different name
> # There should be threads from two separate DistributedMapCacheClientServices
> like this:
> {code:java}
> "NettyDistributedMapCacheClient[d06c8ea0-018b-1000-2904-26aa7612632a]-3-1"
> "NettyDistributedMapCacheClient[d06c8ea0-018b-1000-2904-26aa7612632a]-3-2"
> "NettyDistributedMapCacheClient[d06c8ea0-018b-1000-2904-26aa7612632a]-3-3"
> "NettyDistributedMapCacheClient[d06c8ea0-018b-1000-2904-26aa7612632a]-3-4"
> "NettyDistributedMapCacheClient[d0774323-018b-1000-5e78-07a7455a2bd2]-5-1"
> "NettyDistributedMapCacheClient[d0774323-018b-1000-5e78-07a7455a2bd2]-5-2"
> {code}
> Slack Thread:
> https://apachenifi.slack.com/archives/C0L9VCD47/p179443394479
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-11240) Introduce Python API for building Processors
[ https://issues.apache.org/jira/browse/NIFI-11240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786236#comment-17786236 ] Denis Jakupovic commented on NIFI-11240: Is the Milestone 4 reached and the python api therefore production ready or are we now in Milestone 1 with the nifi 2.0.0 ML1 release? > Introduce Python API for building Processors > > > Key: NIFI-11240 > URL: https://issues.apache.org/jira/browse/NIFI-11240 > Project: Apache NiFi > Issue Type: Epic > Components: Core Framework, Documentation Website, Extensions >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 2.0.0-M1 > > > The scripting processors are very common for data transformation in NiFi. In > particular, the Jython based scripts are quite heavily used. However, Jython > is run on the JVM and does not support CPython libraries. As a result, it's > syntax compatible but doesn't make use of the wealth of Python libraries. And > the wealth of Python libraries are what make Python popular to begin with. > Additionally, use of many script-based processors hurts the UX. They are > cumbersome to configure, with script files and/or script bodies. They result > in a dataflow that's difficult to understand because instead of nicely named > processors like CompressContent the type and default name are > "ExecuteScript." They're also difficult to share. > I have been playing with Py4J for introduce a true Python-based API for > developing Processors. This will introduce new APIs, new framework changes, > and documentation. And this will likely take a while to stabilize. However, > the sooner that we are able to land it into the hands of users, the better. > Therefore, I pose that we introduce it in multiple milestones. We can create > sub-tickets for different milestones, but in general it should follow: > Milestone 1: Initial implementation. Provides the capability and an API for > building processors. Includes sample code and some documentation. Includes > tests to ensure proper operation. Should not be used in production. API will > not be stable and may change frequently. Performance may be subpar. Get into > the hands of developers to begin exploring and providing feedback / > submitting PRs. > Milestone 2: Bug fixes. API refinement. Improve performance. > Milestone 3: Additional bug fixes and API refinement. API should become more > stable. > Milestone 4: Additional bug fixes. API becomes stable. Documentation is clear > and sufficient. Recommend production use. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
