[GitHub] [nifi] Snorlaxa commented on pull request #4760: NIFI-8142 Add "on conflict do nothing" feature to PutDatabaseRecord
Snorlaxa commented on pull request #4760: URL: https://github.com/apache/nifi/pull/4760#issuecomment-765178568 I don't know why my push action cannot trigger any check. Maybe I should close this PR and create another one? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] asfgit closed pull request #4777: NIFI-8166 - Upgraded jackson-databind to 2.9.10.8
asfgit closed pull request #4777: URL: https://github.com/apache/nifi/pull/4777 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] asfgit closed pull request #4738: NIFI-7890 - Added record support to ConsumeMQTT processor
asfgit closed pull request #4738: URL: https://github.com/apache/nifi/pull/4738 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] bbende commented on pull request #4766: NIFI-8150 Change Download flow to Download flow definition for proces…
bbende commented on pull request #4766: URL: https://github.com/apache/nifi/pull/4766#issuecomment-763942429 +1 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #976: MINIFICPP-1448 - CWEL JSON output
adamdebreceni commented on a change in pull request #976:
URL: https://github.com/apache/nifi-minifi-cpp/pull/976#discussion_r561988161
##
File path: extensions/windows-event-log/wel/JSONUtils.cpp
##
@@ -0,0 +1,170 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "JSONUtils.h"
+
+#include
+#include
+#include
+
+#include
+
+#include "rapidjson/document.h"
+#include "rapidjson/writer.h"
+#include "rapidjson/stringbuffer.h"
+#include "rapidjson/prettywriter.h"
+
+#include "gsl/gsl-lite.hpp";
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace wel {
+
+namespace {
+
+rapidjson::Value xmlElementToJSON(const pugi::xml_node& node,
rapidjson::Document& doc) {
+ gsl_Expects(node.type() == pugi::xml_node_type::node_element);
+ rapidjson::Value object(rapidjson::kObjectType);
+ object.AddMember("name", rapidjson::StringRef(node.name()),
doc.GetAllocator());
+ auto& attributes = object.AddMember("attributes", rapidjson::kObjectType,
doc.GetAllocator())["attributes"];
+ for (const auto& attr : node.attributes()) {
+attributes.AddMember(rapidjson::StringRef(attr.name()),
rapidjson::StringRef(attr.value()), doc.GetAllocator());
+ }
+ auto& children = object.AddMember("children", rapidjson::kArrayType,
doc.GetAllocator())["children"];
+ for (const auto& child : node.children()) {
+if (child.type() == pugi::xml_node_type::node_element) {
+ children.PushBack(xmlElementToJSON(child, doc), doc.GetAllocator());
+}
+ }
+ object.AddMember("text", rapidjson::StringRef(node.text().get()),
doc.GetAllocator());
+ return object;
+}
+
+rapidjson::Value xmlDocumentToJSON(const pugi::xml_node& node,
rapidjson::Document& doc) {
+ gsl_Expects(node.type() == pugi::xml_node_type::node_document);
+ rapidjson::Value children(rapidjson::kArrayType);
+ for (const auto& child : node.children()) {
+if (child.type() == pugi::xml_node_type::node_element) {
+ children.PushBack(xmlElementToJSON(child, doc), doc.GetAllocator());
+}
+ }
+ return children;
+}
+
+rapidjson::Document toJSONImpl(const pugi::xml_node& root, bool flatten) {
+ rapidjson::Document doc{rapidjson::kObjectType};
+
+ auto event_xml = root.child("Event");
+
+ {
+auto system_xml = event_xml.child("System");
+auto& system = flatten ? doc : doc.AddMember("System",
rapidjson::kObjectType, doc.GetAllocator())["System"];
+
+{
+ auto provider_xml = system_xml.child("Provider");
+ auto& provider = flatten ? doc : system.AddMember("Provider",
rapidjson::kObjectType, doc.GetAllocator())["Provider"];
+ provider.AddMember("Name",
rapidjson::StringRef(provider_xml.attribute("Name").value()),
doc.GetAllocator());
+ provider.AddMember("Guid",
rapidjson::StringRef(provider_xml.attribute("Guid").value()),
doc.GetAllocator());
+}
+
+system.AddMember("EventID",
rapidjson::StringRef(system_xml.child("EventID").text().get()),
doc.GetAllocator());
+system.AddMember("Version",
rapidjson::StringRef(system_xml.child("Version").text().get()),
doc.GetAllocator());
+system.AddMember("Level",
rapidjson::StringRef(system_xml.child("Level").text().get()),
doc.GetAllocator());
+system.AddMember("Task",
rapidjson::StringRef(system_xml.child("Task").text().get()),
doc.GetAllocator());
+system.AddMember("Opcode",
rapidjson::StringRef(system_xml.child("Opcode").text().get()),
doc.GetAllocator());
+system.AddMember("Keywords",
rapidjson::StringRef(system_xml.child("Keywords").text().get()),
doc.GetAllocator());
+
+{
+ auto timeCreated_xml = system_xml.child("TimeCreated");
+ auto& timeCreated = flatten ? doc : system.AddMember("TimeCreated",
rapidjson::kObjectType, doc.GetAllocator())["TimeCreated"];
+ timeCreated.AddMember("SystemTime",
rapidjson::StringRef(timeCreated_xml.attribute("SystemTime").value()),
doc.GetAllocator());
+}
+
+system.AddMember("EventRecordID",
rapidjson::StringRef(system_xml.child("EventRecordID").text().get()),
doc.GetAllocator());
+
+{
+ auto correlation_xml = system_xml.child("Correlation");
+ auto& correlation = flatten ? doc : system.AddMember("Correlation",
rapidjson::kObjectType,
[GitHub] [nifi] arkadiyvleonov commented on a change in pull request #4776: Add "Mailbox" property to ConsumeEWS
arkadiyvleonov commented on a change in pull request #4776:
URL: https://github.com/apache/nifi/pull/4776#discussion_r562370297
##
File path:
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ConsumeEWS.java
##
@@ -107,6 +109,14 @@
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.sensitive(true)
.build();
+public static final PropertyDescriptor MAILBOX = new
PropertyDescriptor.Builder()
+.name("mailbox")
+.displayName("Mailbox")
+.description("Mailbox")
+.required(true)
Review comment:
made some changes to in below commits
##
File path:
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ConsumeEWS.java
##
@@ -107,6 +109,14 @@
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.sensitive(true)
.build();
+public static final PropertyDescriptor MAILBOX = new
PropertyDescriptor.Builder()
+.name("mailbox")
+.displayName("Mailbox")
+.description("Mailbox")
+.required(true)
Review comment:
made some changes in below commits
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] thenatog commented on pull request #4715: NIFI-6999 - Made changes to load flow.xml files using streams. Update…
thenatog commented on pull request #4715: URL: https://github.com/apache/nifi/pull/4715#issuecomment-764911205 Will merge. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] Snorlaxa commented on a change in pull request #4760: NIFI-8142 Add "on conflict do nothing" feature to PutDatabaseRecord
Snorlaxa commented on a change in pull request #4760:
URL: https://github.com/apache/nifi/pull/4760#discussion_r561746509
##
File path:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutDatabaseRecord.java
##
@@ -301,13 +301,23 @@
.name("put-db-record-max-batch-size")
.displayName("Maximum Batch Size")
.description("Specifies maximum batch size for INSERT and UPDATE
statements. This parameter has no effect for other statements specified in
'Statement Type'."
-+ " Zero means the batch size is not limited.")
++ " Zero means the batch size is not limited.")
.defaultValue("0")
.required(false)
.addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
.build();
+static final PropertyDescriptor UPSERT_DO_NOTHING = new
PropertyDescriptor.Builder()
Review comment:
Thanks for your reply and sorry for making that misleading.
"INSERT_IGNORE" is a good idea. I will try it later :D
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] urbandan commented on a change in pull request #4730: NIFI-8095: Created StatelessNiFi Sink Connector and Source Connector.…
urbandan commented on a change in pull request #4730:
URL: https://github.com/apache/nifi/pull/4730#discussion_r561648596
##
File path:
nifi-external/nifi-kafka-connect/nifi-kafka-connector/src/main/java/org/apache/nifi/kafka/connect/StatelessNiFiSourceConnector.java
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.kafka.connect;
+
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.connect.connector.Task;
+import org.apache.kafka.connect.source.SourceConnector;
+import
org.apache.nifi.kafka.connect.validators.ConnectRegularExpressionValidator;
+import org.apache.nifi.stateless.flow.StatelessDataflow;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class StatelessNiFiSourceConnector extends SourceConnector {
+static final String OUTPUT_PORT_NAME = "output.port";
+static final String TOPIC_NAME = "topics";
+
+static final String TOPIC_NAME_ATTRIBUTE = "topic.name.attribute";
+static final String KEY_ATTRIBUTE = "key.attribute";
+static final String HEADER_REGEX = "header.attribute.regex";
+
+private Map properties;
+private boolean primaryNodeOnly;
+
+@Override
+public void start(final Map properties) {
+this.properties = new HashMap<>(properties);
+
+final StatelessDataflow dataflow =
StatelessKafkaConnectorUtil.createDataflow(properties);
+primaryNodeOnly = dataflow.isSourcePrimaryNodeOnly();
+dataflow.shutdown();
+}
+
+@Override
+public void reconfigure(final Map properties) {
Review comment:
I wouldn't override reconfigure - the default implementation calls stop
and start on the connector.
If you do override it, you need to apply the new configurations - so
everything done in start needs to be repeated here.
##
File path:
nifi-external/nifi-kafka-connect/nifi-kafka-connector/src/main/java/org/apache/nifi/kafka/connect/StatelessNiFiSourceTask.java
##
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.kafka.connect;
+
+import org.apache.kafka.clients.producer.RecordMetadata;
+import org.apache.kafka.common.config.ConfigException;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.errors.RetriableException;
+import org.apache.kafka.connect.header.ConnectHeaders;
+import org.apache.kafka.connect.source.SourceRecord;
+import org.apache.kafka.connect.source.SourceTask;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.stateless.flow.DataflowTrigger;
+import org.apache.nifi.stateless.flow.StatelessDataflow;
+import org.apache.nifi.stateless.flow.TriggerResult;
+import org.apache.nifi.util.FormatUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.regex.Pattern;
+
+public class StatelessNiFiSourceTask extends SourceTask {
+public static final String STATE_MAP_KEY = "task.index";
+private static final Logger logger =
LoggerFactory.getLogger(StatelessNiFiSourceTask.class);
+
+private
[GitHub] [nifi] turcsanyip commented on a change in pull request #4738: NIFI-7890 - Added record support to ConsumeMQTT processor
turcsanyip commented on a change in pull request #4738:
URL: https://github.com/apache/nifi/pull/4738#discussion_r561332768
##
File path:
nifi-nar-bundles/nifi-mqtt-bundle/nifi-mqtt-processors/src/main/java/org/apache/nifi/processors/mqtt/ConsumeMQTT.java
##
@@ -334,14 +443,210 @@ public void process(final OutputStream out) throws
IOException {
if (!mqttQueue.remove(mqttMessage) && logger.isWarnEnabled()) {
logger.warn(new StringBuilder("FlowFile ")
.append(messageFlowfile.getAttribute(CoreAttributes.UUID.key()))
-.append(" for Mqtt message ")
+.append(" for MQTT message ")
.append(mqttMessage)
.append(" had already been removed from queue,
possible duplication of flow files")
.toString());
}
}
}
+private void transferQueueDemarcator(final ProcessContext context, final
ProcessSession session){
+final byte[] demarcator =
context.getProperty(MESSAGE_DEMARCATOR).evaluateAttributeExpressions().getValue().getBytes(StandardCharsets.UTF_8);
+
+FlowFile messageFlowfile = session.create();
+session.putAttribute(messageFlowfile, BROKER_ATTRIBUTE_KEY, broker);
+
+
+messageFlowfile = session.append(messageFlowfile, out -> {
+while (!mqttQueue.isEmpty()) {
+final MQTTQueueMessage mqttMessage = mqttQueue.poll();
+out.write(mqttMessage.getPayload());
+out.write(demarcator);
+session.adjustCounter(COUNTER_RECORDS_RECEIVED, 1L, false);
+}
+});
+
+session.getProvenanceReporter().receive(messageFlowfile, new
StringBuilder(broker).append(topicPrefix).append(topicFilter).toString());
Review comment:
There is no separator character between the broker and the topic prefix
(eg.: `tcp://myhost:1883mytopic`).
`'/'` cloud be added before topic prefix.
It could be changed in the existing `transferQueue()` method too.
##
File path:
nifi-nar-bundles/nifi-mqtt-bundle/nifi-mqtt-processors/src/main/java/org/apache/nifi/processors/mqtt/ConsumeMQTT.java
##
@@ -334,14 +443,210 @@ public void process(final OutputStream out) throws
IOException {
if (!mqttQueue.remove(mqttMessage) && logger.isWarnEnabled()) {
logger.warn(new StringBuilder("FlowFile ")
.append(messageFlowfile.getAttribute(CoreAttributes.UUID.key()))
-.append(" for Mqtt message ")
+.append(" for MQTT message ")
.append(mqttMessage)
.append(" had already been removed from queue,
possible duplication of flow files")
.toString());
}
}
}
+private void transferQueueDemarcator(final ProcessContext context, final
ProcessSession session){
+final byte[] demarcator =
context.getProperty(MESSAGE_DEMARCATOR).evaluateAttributeExpressions().getValue().getBytes(StandardCharsets.UTF_8);
+
+FlowFile messageFlowfile = session.create();
+session.putAttribute(messageFlowfile, BROKER_ATTRIBUTE_KEY, broker);
+
+
+messageFlowfile = session.append(messageFlowfile, out -> {
+while (!mqttQueue.isEmpty()) {
Review comment:
Emptying the queue seems to me a bit non-deterministic behaviour because
the queue is being written at the same time by the receiver thread.
Would not it be useful to define a max. size that may be fetched in one go?
(a magic number or a processor property)
##
File path:
nifi-nar-bundles/nifi-mqtt-bundle/nifi-mqtt-processors/src/main/java/org/apache/nifi/processors/mqtt/ConsumeMQTT.java
##
@@ -334,14 +443,210 @@ public void process(final OutputStream out) throws
IOException {
if (!mqttQueue.remove(mqttMessage) && logger.isWarnEnabled()) {
logger.warn(new StringBuilder("FlowFile ")
.append(messageFlowfile.getAttribute(CoreAttributes.UUID.key()))
-.append(" for Mqtt message ")
+.append(" for MQTT message ")
.append(mqttMessage)
.append(" had already been removed from queue,
possible duplication of flow files")
.toString());
}
}
}
+private void transferQueueDemarcator(final ProcessContext context, final
ProcessSession session){
+final byte[] demarcator =
context.getProperty(MESSAGE_DEMARCATOR).evaluateAttributeExpressions().getValue().getBytes(StandardCharsets.UTF_8);
+
+FlowFile messageFlowfile = session.create();
+session.putAttribute(messageFlowfile, BROKER_ATTRIBUTE_KEY, broker);
+
+
+messageFlowfile = session.append(messageFlowfile, out -> {
+
[GitHub] [nifi] thenatog closed pull request #4715: NIFI-6999 - Made changes to load flow.xml files using streams. Update…
thenatog closed pull request #4715: URL: https://github.com/apache/nifi/pull/4715 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] thenatog commented on pull request #4775: NIFI-8163: When counting number of components, we traverse into all P…
thenatog commented on pull request #4775: URL: https://github.com/apache/nifi/pull/4775#issuecomment-764893877 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] asfgit closed pull request #4772: NIFI-8043: Quote update key column names in PutDatabaseRecord
asfgit closed pull request #4772: URL: https://github.com/apache/nifi/pull/4772 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #955: MINIFICPP-1414 Create in-memory compressed logs
arpadboda commented on a change in pull request #955:
URL: https://github.com/apache/nifi-minifi-cpp/pull/955#discussion_r561787147
##
File path: libminifi/include/utils/ValueParser.h
##
@@ -126,6 +126,10 @@ class ValueParser {
}
}
+ std::string rest() const noexcept {
+return {str.c_str() + offset};
Review comment:
I think substring is a bit easier to read:
```
return str.substr(offset);
```
##
File path: libminifi/include/utils/Literals.h
##
@@ -0,0 +1,59 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+constexpr unsigned long long operator "" _KiB(unsigned long long n) { //
NOLINT
Review comment:
What would be reported by linter here?
##
File path: libminifi/include/utils/StagingQueue.h
##
@@ -0,0 +1,168 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include
+#include
+#include
+#include "MinifiConcurrentQueue.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+
+namespace internal {
+template
+struct default_allocator {
+ T operator()(size_t max_size) const {
+return T::allocate(max_size);
+ }
+};
+} // namespace internal
+
+template>
+class StagingQueue {
Review comment:
Please add a comment that tells the purpose of this class!
##
File path: libminifi/src/io/ZlibStream.cpp
##
@@ -127,7 +143,12 @@
ZlibDecompressStream::ZlibDecompressStream(gsl::not_null output,
ZlibDecompressStream::~ZlibDecompressStream() {
if (state_ != ZlibStreamState::UNINITIALIZED) {
-inflateEnd(_);
+int result = inflateEnd(_);
+if (result == Z_STREAM_ERROR) {
+ logger_->log_debug("Stream state was inconsistent");
+} else if (result != Z_OK) {
+ logger_->log_debug("Unknown error while finishing decompression %d",
result);
Review comment:
Is it only a debug level information? Seems like a quite bad state for
me.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] bbende merged pull request #4766: NIFI-8150 Change Download flow to Download flow definition for proces…
bbende merged pull request #4766: URL: https://github.com/apache/nifi/pull/4766 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] pvillard31 edited a comment on pull request #4768: NIFI-8155 - add banner text in page title
pvillard31 edited a comment on pull request #4768: URL: https://github.com/apache/nifi/pull/4768#issuecomment-764453581 Thanks for looking into this @exceptionfactory. As far as I can tell the other places you mentioned are not "really" opening new windows and this not impacting the overall page title: https://user-images.githubusercontent.com/11541012/105320361-801ab280-5bdf-11eb-8fc0-969aac536b0c.png;> The only edge case I can think of is when looking at the content of a flow file: this will open a new tab/window and the title won't contain the info. But I believe this a very special edge case and I think this is best to leave it outside of the change. https://user-images.githubusercontent.com/11541012/105320689-e8699400-5bdf-11eb-84a7-dff5269efa25.png;> This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] asfgit closed pull request #4771: NIFI-8156 Fixed byte handling bug in cassandra.
asfgit closed pull request #4771: URL: https://github.com/apache/nifi/pull/4771 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] joewitt commented on pull request #4730: NIFI-8095: Created StatelessNiFi Sink Connector and Source Connector.…
joewitt commented on pull request #4730: URL: https://github.com/apache/nifi/pull/4730#issuecomment-764705859 awesome work and discussion and review here. this is going to be awesome. I'm a +1 based on watching this engagement and reviewing the changes. mark please self merge as I cannot get to that part at this time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #975: MINIFICPP-1400 Create ListS3 processor
adamdebreceni commented on a change in pull request #975:
URL: https://github.com/apache/nifi-minifi-cpp/pull/975#discussion_r561703605
##
File path: extensions/aws/s3/S3Wrapper.h
##
@@ -31,11 +47,190 @@ namespace minifi {
namespace aws {
namespace s3 {
-class S3Wrapper : public S3WrapperBase {
+static const std::unordered_map
STORAGE_CLASS_MAP {
+ {"Standard", Aws::S3::Model::StorageClass::STANDARD},
+ {"ReducedRedundancy", Aws::S3::Model::StorageClass::REDUCED_REDUNDANCY},
+ {"StandardIA", Aws::S3::Model::StorageClass::STANDARD_IA},
+ {"OnezoneIA", Aws::S3::Model::StorageClass::ONEZONE_IA},
+ {"IntelligentTiering", Aws::S3::Model::StorageClass::INTELLIGENT_TIERING},
+ {"Glacier", Aws::S3::Model::StorageClass::GLACIER},
+ {"DeepArchive", Aws::S3::Model::StorageClass::DEEP_ARCHIVE}
+};
+
+static const std::unordered_map OBJECT_STORAGE_CLASS_MAP {
+ {Aws::S3::Model::ObjectStorageClass::STANDARD, "Standard"},
+ {Aws::S3::Model::ObjectStorageClass::REDUCED_REDUNDANCY,
"ReducedRedundancy"},
+ {Aws::S3::Model::ObjectStorageClass::STANDARD_IA, "StandardIA"},
+ {Aws::S3::Model::ObjectStorageClass::ONEZONE_IA, "OnezoneIA"},
+ {Aws::S3::Model::ObjectStorageClass::INTELLIGENT_TIERING,
"IntelligentTiering"},
+ {Aws::S3::Model::ObjectStorageClass::GLACIER, "Glacier"},
+ {Aws::S3::Model::ObjectStorageClass::DEEP_ARCHIVE, "DeepArchive"}
+};
+
+static const std::unordered_map VERSION_STORAGE_CLASS_MAP {
+ {Aws::S3::Model::ObjectVersionStorageClass::STANDARD, "Standard"}
+};
+
+static const std::unordered_map SERVER_SIDE_ENCRYPTION_MAP {
+ {"None", Aws::S3::Model::ServerSideEncryption::NOT_SET},
+ {"AES256", Aws::S3::Model::ServerSideEncryption::AES256},
+ {"aws_kms", Aws::S3::Model::ServerSideEncryption::aws_kms},
+};
+
+static const std::unordered_map
CANNED_ACL_MAP {
+ {"BucketOwnerFullControl",
Aws::S3::Model::ObjectCannedACL::bucket_owner_full_control},
+ {"BucketOwnerRead", Aws::S3::Model::ObjectCannedACL::bucket_owner_read},
+ {"AuthenticatedRead", Aws::S3::Model::ObjectCannedACL::authenticated_read},
+ {"PublicReadWrite", Aws::S3::Model::ObjectCannedACL::public_read_write},
+ {"PublicRead", Aws::S3::Model::ObjectCannedACL::public_read},
+ {"Private", Aws::S3::Model::ObjectCannedACL::private_},
+ {"AwsExecRead", Aws::S3::Model::ObjectCannedACL::aws_exec_read},
+};
+
+struct Expiration {
+ std::string expiration_time;
+ std::string expiration_time_rule_id;
+};
+
+struct PutObjectResult {
+ std::string version;
+ std::string etag;
+ std::string expiration;
+ std::string ssealgorithm;
+};
+
+struct PutObjectRequestParameters {
+ std::string bucket;
+ std::string object_key;
+ std::string storage_class;
+ std::string server_side_encryption;
+ std::string content_type;
+ std::map user_metadata_map;
+ std::string fullcontrol_user_list;
+ std::string read_permission_user_list;
+ std::string read_acl_user_list;
+ std::string write_acl_user_list;
+ std::string canned_acl;
+};
+
+struct GetObjectRequestParameters {
+ std::string bucket;
+ std::string object_key;
+ std::string version;
+ bool requester_pays = false;
+};
+
+struct HeadObjectResult {
+ std::string path;
+ std::string absolute_path;
+ std::string filename;
+ std::string mime_type;
+ std::string etag;
+ Expiration expiration;
+ std::string ssealgorithm;
+ std::string version;
+ std::map user_metadata_map;
+
+ void setFilePaths(const std::string& key);
+};
+
+struct GetObjectResult : public HeadObjectResult {
+ int64_t write_size = 0;
+};
+
+struct ListRequestParameters {
+ std::string bucket;
+ std::string delimiter;
+ std::string prefix;
+ bool use_versions = false;
+ uint64_t min_object_age = 0;
+};
+
+struct ListedObjectAttributes {
+ std::string filename;
+ std::string etag;
+ bool is_latest = false;
+ int64_t last_modified = 0;
+ int length = 0;
+ std::string store_class;
+ std::string version;
Review comment:
it seems like `version` is an optional field and we treat the empty
string as the `null`, now there are many examples of this throughout our
codebase so it is not unheard of, but do you think using `optional` to make
this explicit could be beneficial?
##
File path: extensions/aws/processors/ListS3.cpp
##
@@ -0,0 +1,294 @@
+/**
+ * @file ListS3.cpp
+ * ListS3 class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR
[GitHub] [nifi] markap14 closed pull request #4746: NIFI-8034: Fixed PropertyValue.isExpressionLanguagePresent always ret…
markap14 closed pull request #4746: URL: https://github.com/apache/nifi/pull/4746 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (NIFI-8164) ConsumeMQTT validation bug
[
https://issues.apache.org/jira/browse/NIFI-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Villard resolved NIFI-8164.
--
Resolution: Duplicate
> ConsumeMQTT validation bug
> --
>
> Key: NIFI-8164
> URL: https://issues.apache.org/jira/browse/NIFI-8164
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.12.1
>Reporter: Kay-Uwe Moosheimer
>Priority: Major
>
> In ConsumeMQTT.java the following lines (196-200)
> final boolean clientIDSet = context.getProperty(PROP_CLIENTID).isSet();
> final boolean groupIDSet = context.getProperty(PROP_GROUPID).isSet();
> if (clientIDSet && groupIDSet) {
> results.add(new ValidationResult.Builder().subject("Client ID and Group
> ID").valid(false).explanation("if client ID is not unique, multiple nodes
> cannot join the consumer group").build());
> }
> are counterproductive.
> It is true that a client ID must be unique for multiple mqtt clients to join
> a consumer group.
> If you enter the client ID as ${hostname()}-client, then the client ID is
> unique (if you use this only once per node).
> But this is not possible with the validation.
> On the other hand it makes no sense if the client ID is random and you set
> the session state to "resume session". A random client ID can never resume a
> previous session.
> In general it is also not advisable but only an add-on for lazy typists if
> you don't use a unique client ID to auto generate a UUID. Because then only
> random client IDs appear on the broker, which cannot be assigned to a certain
> processor/node easily or not at all (monitoring nightmare).
> Therefore please delete these lines, because ConsumeMQTT is applicable in
> this way but not really in a proper way.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] exceptionfactory commented on a change in pull request #4767: NIFI-1355 Implemented new methods in KeyStoreUtils to programmatical…
exceptionfactory commented on a change in pull request #4767:
URL: https://github.com/apache/nifi/pull/4767#discussion_r561195875
##
File path:
nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/security/util/KeyStoreUtils.java
##
@@ -125,6 +146,63 @@ public static KeyStore loadKeyStore(String keystorePath,
char[] keystorePassword
}
}
+/**
+ * Creates a temporary default Keystore and Truststore and returns it
wrapped in a TLS configuration.
+ *
+ * @return a {@link org.apache.nifi.security.util.TlsConfiguration}
+ */
+public static TlsConfiguration createTlsConfigAndNewKeystoreTruststore()
throws IOException, GeneralSecurityException {
+return createTlsConfigAndNewKeystoreTruststore(new
StandardTlsConfiguration());
+}
+
+/**
+ * Creates a temporary Keystore and Truststore and returns it wrapped in a
new TLS configuration with the given values.
+ *
+ * @param tlsConfiguration a {@link
org.apache.nifi.security.util.TlsConfiguration}
+ * @return a {@link org.apache.nifi.security.util.TlsConfiguration}
+ */
+public static TlsConfiguration
createTlsConfigAndNewKeystoreTruststore(final TlsConfiguration
tlsConfiguration) throws IOException, GeneralSecurityException {
+final Path keyStorePath;
+final String keystorePassword =
StringUtils.isNotBlank(tlsConfiguration.getKeystorePassword()) ?
tlsConfiguration.getKeystorePassword() : generatePassword();
+final String keyPassword =
StringUtils.isNotBlank(tlsConfiguration.getKeyPassword())?
tlsConfiguration.getKeyPassword() : keystorePassword;
+final KeystoreType keystoreType = tlsConfiguration.getKeystoreType()
!= null ? tlsConfiguration.getKeystoreType() : KeystoreType.PKCS12;
+final Path trustStorePath;
+final String truststorePassword =
StringUtils.isNotBlank(tlsConfiguration.getTruststorePassword()) ?
tlsConfiguration.getTruststorePassword() : "";
+final KeystoreType truststoreType =
tlsConfiguration.getTruststoreType() != null ?
tlsConfiguration.getTruststoreType() : KeystoreType.PKCS12;
+
+// Create temporary Keystore file
+try {
+keyStorePath = generateTempKeystorePath(keystoreType);
+} catch (IOException e) {
+logger.error(KEYSTORE_ERROR_MSG);
+throw new UncheckedIOException(KEYSTORE_ERROR_MSG, e);
+}
+
+// Create temporary Truststore file
+try {
+trustStorePath = generateTempTruststorePath(truststoreType);
+} catch (IOException e) {
+logger.error(TRUSTSTORE_ERROR_MSG);
+throw new UncheckedIOException(TRUSTSTORE_ERROR_MSG, e);
+}
+
+// Create X509 Certificate
+final X509Certificate clientCert =
createKeyStoreAndGetX509Certificate(KEY_ALIAS, keystorePassword, keyPassword,
keyStorePath.toString(), keystoreType);
+
+// Create Truststore
+createTrustStore(clientCert, CERT_ALIAS, truststorePassword,
trustStorePath.toString(), getKeystoreType(truststoreType.toString()));
+
+return new StandardTlsConfiguration(
+keyStorePath.toString(),
+keystorePassword,
+keyPassword,
+getKeystoreType(keystoreType.toString()),
+trustStorePath.toString(),
+truststorePassword,
+getKeystoreType(truststoreType.toString()),
Review comment:
Are these calls to `getKeystoreType()` necessary? It looks like the
values are already of the `KeystoreType` enum.
```suggestion
truststoreType,
```
##
File path:
nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/security/util/KeyStoreUtils.java
##
@@ -125,6 +146,63 @@ public static KeyStore loadKeyStore(String keystorePath,
char[] keystorePassword
}
}
+/**
+ * Creates a temporary default Keystore and Truststore and returns it
wrapped in a TLS configuration.
+ *
+ * @return a {@link org.apache.nifi.security.util.TlsConfiguration}
+ */
+public static TlsConfiguration createTlsConfigAndNewKeystoreTruststore()
throws IOException, GeneralSecurityException {
+return createTlsConfigAndNewKeystoreTruststore(new
StandardTlsConfiguration());
+}
+
+/**
+ * Creates a temporary Keystore and Truststore and returns it wrapped in a
new TLS configuration with the given values.
+ *
+ * @param tlsConfiguration a {@link
org.apache.nifi.security.util.TlsConfiguration}
+ * @return a {@link org.apache.nifi.security.util.TlsConfiguration}
+ */
+public static TlsConfiguration
createTlsConfigAndNewKeystoreTruststore(final TlsConfiguration
tlsConfiguration) throws IOException, GeneralSecurityException {
+final Path keyStorePath;
+final String keystorePassword =
StringUtils.isNotBlank(tlsConfiguration.getKeystorePassword()) ?
[jira] [Commented] (NIFI-8164) ConsumeMQTT validation bug
[
https://issues.apache.org/jira/browse/NIFI-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269861#comment-17269861
]
Pierre Villard commented on NIFI-8164:
--
A lot of changes have been done in this processor over the last few weeks and
will be available in 1.13. I believe what you are asking for is already
implemented with NIFI-7894. Closing as duplicate. Please re-open if you think
this is not correct.
> ConsumeMQTT validation bug
> --
>
> Key: NIFI-8164
> URL: https://issues.apache.org/jira/browse/NIFI-8164
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.12.1
>Reporter: Kay-Uwe Moosheimer
>Priority: Major
>
> In ConsumeMQTT.java the following lines (196-200)
> final boolean clientIDSet = context.getProperty(PROP_CLIENTID).isSet();
> final boolean groupIDSet = context.getProperty(PROP_GROUPID).isSet();
> if (clientIDSet && groupIDSet) {
> results.add(new ValidationResult.Builder().subject("Client ID and Group
> ID").valid(false).explanation("if client ID is not unique, multiple nodes
> cannot join the consumer group").build());
> }
> are counterproductive.
> It is true that a client ID must be unique for multiple mqtt clients to join
> a consumer group.
> If you enter the client ID as ${hostname()}-client, then the client ID is
> unique (if you use this only once per node).
> But this is not possible with the validation.
> On the other hand it makes no sense if the client ID is random and you set
> the session state to "resume session". A random client ID can never resume a
> previous session.
> In general it is also not advisable but only an add-on for lazy typists if
> you don't use a unique client ID to auto generate a UUID. Because then only
> random client IDs appear on the broker, which cannot be assigned to a certain
> processor/node easily or not at all (monitoring nightmare).
> Therefore please delete these lines, because ConsumeMQTT is applicable in
> this way but not really in a proper way.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] markap14 commented on a change in pull request #4730: NIFI-8095: Created StatelessNiFi Sink Connector and Source Connector.…
markap14 commented on a change in pull request #4730:
URL: https://github.com/apache/nifi/pull/4730#discussion_r561919864
##
File path:
nifi-external/nifi-kafka-connect/nifi-kafka-connector/src/main/java/org/apache/nifi/kafka/connect/StatelessNiFiSourceTask.java
##
@@ -0,0 +1,317 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.kafka.connect;
+
+import org.apache.kafka.clients.producer.RecordMetadata;
+import org.apache.kafka.common.config.ConfigException;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.errors.RetriableException;
+import org.apache.kafka.connect.header.ConnectHeaders;
+import org.apache.kafka.connect.source.SourceRecord;
+import org.apache.kafka.connect.source.SourceTask;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.stateless.flow.DataflowTrigger;
+import org.apache.nifi.stateless.flow.StatelessDataflow;
+import org.apache.nifi.stateless.flow.TriggerResult;
+import org.apache.nifi.util.FormatUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.regex.Pattern;
+
+public class StatelessNiFiSourceTask extends SourceTask {
+public static final String STATE_MAP_KEY = "task.index";
+private static final Logger logger =
LoggerFactory.getLogger(StatelessNiFiSourceTask.class);
+
+private StatelessDataflow dataflow;
+private String outputPortName;
+private String topicName;
+private String topicNameAttribute;
+private TriggerResult triggerResult;
+private String keyAttributeName;
+private Pattern headerAttributeNamePattern;
+private long timeoutMillis;
+private String dataflowName;
+private long failureYieldExpiration = 0L;
+
+private final Map clusterStatePartitionMap =
Collections.singletonMap(STATE_MAP_KEY, "CLUSTER");
+private Map localStatePartitionMap = new HashMap<>();
+
+private Map previousBatchComponentStates =
Collections.emptyMap();
+
+private final AtomicLong unacknowledgedRecords = new AtomicLong(0L);
+
+@Override
+public String version() {
+return StatelessKafkaConnectorUtil.getVersion();
+}
+
+@Override
+public void start(final Map properties) {
+logger.info("Starting Source Task with properties {}",
StatelessKafkaConnectorUtil.getLoggableProperties(properties));
+
+final String timeout =
properties.getOrDefault(StatelessKafkaConnectorUtil.DATAFLOW_TIMEOUT,
StatelessKafkaConnectorUtil.DEFAULT_DATAFLOW_TIMEOUT);
+timeoutMillis = (long) FormatUtils.getPreciseTimeDuration(timeout,
TimeUnit.MILLISECONDS);
+
+topicName = properties.get(StatelessNiFiSourceConnector.TOPIC_NAME);
+topicNameAttribute =
properties.get(StatelessNiFiSourceConnector.TOPIC_NAME_ATTRIBUTE);
+keyAttributeName =
properties.get(StatelessNiFiSourceConnector.KEY_ATTRIBUTE);
+
+if (topicName == null && topicNameAttribute == null) {
+throw new ConfigException("Either the topic.name or
topic.name.attribute configuration must be specified");
+}
+
+final String headerRegex =
properties.get(StatelessNiFiSourceConnector.HEADER_REGEX);
+headerAttributeNamePattern = headerRegex == null ? null :
Pattern.compile(headerRegex);
+
+dataflow = StatelessKafkaConnectorUtil.createDataflow(properties);
+
+// Determine the name of the Output Port to retrieve data from
+dataflowName =
properties.get(StatelessKafkaConnectorUtil.DATAFLOW_NAME);
+outputPortName =
properties.get(StatelessNiFiSourceConnector.OUTPUT_PORT_NAME);
+if (outputPortName == null) {
+final Set outputPorts = dataflow.getOutputPortNames();
+if (outputPorts.isEmpty()) {
+throw new ConfigException("The dataflow specified for <" +
dataflowName + "> does not have an Output Port at the root level.
[jira] [Resolved] (NIFI-8166) Upgrade jackson-databind to 2.9.10.8
[ https://issues.apache.org/jira/browse/NIFI-8166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard resolved NIFI-8166. -- Fix Version/s: 1.13.0 Resolution: Fixed > Upgrade jackson-databind to 2.9.10.8 > - > > Key: NIFI-8166 > URL: https://issues.apache.org/jira/browse/NIFI-8166 > Project: Apache NiFi > Issue Type: Task > Components: Core Framework >Affects Versions: 1.12.1 >Reporter: Nathan Gough >Assignee: Nathan Gough >Priority: Major > Labels: dependencies > Fix For: 1.13.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Upgrade the jackson-databind version from 2.9.10.5 to 2.9.10.8. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8166) Upgrade jackson-databind to 2.9.10.8
[ https://issues.apache.org/jira/browse/NIFI-8166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269849#comment-17269849 ] ASF subversion and git services commented on NIFI-8166: --- Commit 0ff43677819f4c556215dd5218577c8b31c0350f in nifi's branch refs/heads/main from Nathan Gough [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=0ff4367 ] NIFI-8166 - Upgraded jackson-databind to 2.9.10.8 Signed-off-by: Pierre Villard This closes #4777. > Upgrade jackson-databind to 2.9.10.8 > - > > Key: NIFI-8166 > URL: https://issues.apache.org/jira/browse/NIFI-8166 > Project: Apache NiFi > Issue Type: Task > Components: Core Framework >Affects Versions: 1.12.1 >Reporter: Nathan Gough >Assignee: Nathan Gough >Priority: Major > Labels: dependencies > Time Spent: 10m > Remaining Estimate: 0h > > Upgrade the jackson-databind version from 2.9.10.5 to 2.9.10.8. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] markap14 commented on pull request #4746: NIFI-8034: Fixed PropertyValue.isExpressionLanguagePresent always ret…
markap14 commented on pull request #4746: URL: https://github.com/apache/nifi/pull/4746#issuecomment-764814499 Thanks @turcsanyip! All looks good to me. +1 merged to main. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] markap14 commented on pull request #4730: NIFI-8095: Created StatelessNiFi Sink Connector and Source Connector.…
markap14 commented on pull request #4730: URL: https://github.com/apache/nifi/pull/4730#issuecomment-764692558 Thanks for sticking with me as I learn the connect API @urbandan and for all of the fantastic feedback! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] asfgit closed pull request #4774: NIFI-7243 Upgraded com.hierynomus.sshj to 0.30.0
asfgit closed pull request #4774: URL: https://github.com/apache/nifi/pull/4774 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] pvillard31 commented on pull request #4774: NIFI-7243 Upgraded com.hierynomus.sshj to 0.30.0
pvillard31 commented on pull request #4774: URL: https://github.com/apache/nifi/pull/4774#issuecomment-764808076 Merged, thanks @exceptionfactory ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] thenatog commented on pull request #4753: NIFI-7356 - Enable TLS for embedded Zookeeper when NiFi has TLS enabled
thenatog commented on pull request #4753: URL: https://github.com/apache/nifi/pull/4753#issuecomment-763911925 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] thenatog closed pull request #4753: NIFI-7356 - Enable TLS for embedded Zookeeper when NiFi has TLS enabled
thenatog closed pull request #4753: URL: https://github.com/apache/nifi/pull/4753 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a change in pull request #4594: NIFI-3669 Add SSL Support to CaptureChangeMySQL
exceptionfactory commented on a change in pull request #4594:
URL: https://github.com/apache/nifi/pull/4594#discussion_r561903819
##
File path:
nifi-nar-bundles/nifi-cdc/nifi-cdc-mysql-bundle/nifi-cdc-mysql-processors/src/main/java/org/apache/nifi/cdc/mysql/processors/CaptureChangeMySQL.java
##
@@ -368,6 +393,23 @@
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
.build();
+public static final PropertyDescriptor SSL_CONTEXT_SERVICE = new
PropertyDescriptor.Builder()
+.name("SSL Context Service")
+.displayName("SSL Context Service")
+.description("SSL Context Service supporting encrypted socket
communication")
+.required(false)
+.identifiesControllerService(SSLContextService.class)
+.build();
+
+public static final PropertyDescriptor SSL_MODE = new
PropertyDescriptor.Builder()
Review comment:
@pvillard31 Thanks for following up with the recommendation. I updated
the `SSL Context Service` property descriptor to depend on the relevant values
from the `SSL Mode` property.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] markap14 closed pull request #4730: NIFI-8095: Created StatelessNiFi Sink Connector and Source Connector.…
markap14 closed pull request #4730: URL: https://github.com/apache/nifi/pull/4730 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] arpadboda commented on pull request #974: MINIFICPP-1449 Add pause and resume command to C2
arpadboda commented on pull request #974: URL: https://github.com/apache/nifi-minifi-cpp/pull/974#issuecomment-764799268 I think we should investigate the CI failure here: ``` C2PauseResumeTest: /home/runner/work/nifi-minifi-cpp/nifi-minifi-cpp/extensions/http-curl/tests/HTTPHandlers.h:432: void HeartbeatHandler::verify(mg_connection*): Assertion `ok' failed. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] thenatog closed pull request #4775: NIFI-8163: When counting number of components, we traverse into all P…
thenatog closed pull request #4775: URL: https://github.com/apache/nifi/pull/4775 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] pvillard31 commented on a change in pull request #4594: NIFI-3669 Add SSL Support to CaptureChangeMySQL
pvillard31 commented on a change in pull request #4594:
URL: https://github.com/apache/nifi/pull/4594#discussion_r561646469
##
File path:
nifi-nar-bundles/nifi-cdc/nifi-cdc-mysql-bundle/nifi-cdc-mysql-processors/src/main/java/org/apache/nifi/cdc/mysql/processors/CaptureChangeMySQL.java
##
@@ -368,6 +393,23 @@
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
.build();
+public static final PropertyDescriptor SSL_CONTEXT_SERVICE = new
PropertyDescriptor.Builder()
+.name("SSL Context Service")
+.displayName("SSL Context Service")
+.description("SSL Context Service supporting encrypted socket
communication")
+.required(false)
+.identifiesControllerService(SSLContextService.class)
+.build();
+
+public static final PropertyDescriptor SSL_MODE = new
PropertyDescriptor.Builder()
Review comment:
Do we want to use the new .dependsOn functionality to display this
property only if the SSL Context Service is set?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #976: MINIFICPP-1448 - CWEL JSON output
szaszm commented on a change in pull request #976:
URL: https://github.com/apache/nifi-minifi-cpp/pull/976#discussion_r561927345
##
File path: extensions/windows-event-log/wel/JSONUtils.cpp
##
@@ -0,0 +1,170 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "JSONUtils.h"
+
+#include
+#include
+#include
+
+#include
+
+#include "rapidjson/document.h"
+#include "rapidjson/writer.h"
+#include "rapidjson/stringbuffer.h"
+#include "rapidjson/prettywriter.h"
+
+#include "gsl/gsl-lite.hpp";
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace wel {
+
+namespace {
+
+rapidjson::Value xmlElementToJSON(const pugi::xml_node& node,
rapidjson::Document& doc) {
+ gsl_Expects(node.type() == pugi::xml_node_type::node_element);
+ rapidjson::Value object(rapidjson::kObjectType);
+ object.AddMember("name", rapidjson::StringRef(node.name()),
doc.GetAllocator());
+ auto& attributes = object.AddMember("attributes", rapidjson::kObjectType,
doc.GetAllocator())["attributes"];
+ for (const auto& attr : node.attributes()) {
+attributes.AddMember(rapidjson::StringRef(attr.name()),
rapidjson::StringRef(attr.value()), doc.GetAllocator());
+ }
+ auto& children = object.AddMember("children", rapidjson::kArrayType,
doc.GetAllocator())["children"];
+ for (const auto& child : node.children()) {
+if (child.type() == pugi::xml_node_type::node_element) {
+ children.PushBack(xmlElementToJSON(child, doc), doc.GetAllocator());
+}
+ }
+ object.AddMember("text", rapidjson::StringRef(node.text().get()),
doc.GetAllocator());
+ return object;
+}
+
+rapidjson::Value xmlDocumentToJSON(const pugi::xml_node& node,
rapidjson::Document& doc) {
+ gsl_Expects(node.type() == pugi::xml_node_type::node_document);
+ rapidjson::Value children(rapidjson::kArrayType);
+ for (const auto& child : node.children()) {
+if (child.type() == pugi::xml_node_type::node_element) {
+ children.PushBack(xmlElementToJSON(child, doc), doc.GetAllocator());
+}
+ }
+ return children;
+}
+
+rapidjson::Document toJSONImpl(const pugi::xml_node& root, bool flatten) {
+ rapidjson::Document doc{rapidjson::kObjectType};
+
+ auto event_xml = root.child("Event");
+
+ {
+auto system_xml = event_xml.child("System");
+auto& system = flatten ? doc : doc.AddMember("System",
rapidjson::kObjectType, doc.GetAllocator())["System"];
+
+{
+ auto provider_xml = system_xml.child("Provider");
+ auto& provider = flatten ? doc : system.AddMember("Provider",
rapidjson::kObjectType, doc.GetAllocator())["Provider"];
+ provider.AddMember("Name",
rapidjson::StringRef(provider_xml.attribute("Name").value()),
doc.GetAllocator());
+ provider.AddMember("Guid",
rapidjson::StringRef(provider_xml.attribute("Guid").value()),
doc.GetAllocator());
+}
+
+system.AddMember("EventID",
rapidjson::StringRef(system_xml.child("EventID").text().get()),
doc.GetAllocator());
+system.AddMember("Version",
rapidjson::StringRef(system_xml.child("Version").text().get()),
doc.GetAllocator());
+system.AddMember("Level",
rapidjson::StringRef(system_xml.child("Level").text().get()),
doc.GetAllocator());
+system.AddMember("Task",
rapidjson::StringRef(system_xml.child("Task").text().get()),
doc.GetAllocator());
+system.AddMember("Opcode",
rapidjson::StringRef(system_xml.child("Opcode").text().get()),
doc.GetAllocator());
+system.AddMember("Keywords",
rapidjson::StringRef(system_xml.child("Keywords").text().get()),
doc.GetAllocator());
+
+{
+ auto timeCreated_xml = system_xml.child("TimeCreated");
+ auto& timeCreated = flatten ? doc : system.AddMember("TimeCreated",
rapidjson::kObjectType, doc.GetAllocator())["TimeCreated"];
+ timeCreated.AddMember("SystemTime",
rapidjson::StringRef(timeCreated_xml.attribute("SystemTime").value()),
doc.GetAllocator());
+}
+
+system.AddMember("EventRecordID",
rapidjson::StringRef(system_xml.child("EventRecordID").text().get()),
doc.GetAllocator());
+
+{
+ auto correlation_xml = system_xml.child("Correlation");
+ auto& correlation = flatten ? doc : system.AddMember("Correlation",
rapidjson::kObjectType,
[GitHub] [nifi] mtien-apache commented on a change in pull request #4767: NIFI-1355 Implemented new methods in KeyStoreUtils to programmatical…
mtien-apache commented on a change in pull request #4767:
URL: https://github.com/apache/nifi/pull/4767#discussion_r561247262
##
File path:
nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/security/util/KeyStoreUtils.java
##
@@ -125,6 +146,63 @@ public static KeyStore loadKeyStore(String keystorePath,
char[] keystorePassword
}
}
+/**
+ * Creates a temporary default Keystore and Truststore and returns it
wrapped in a TLS configuration.
+ *
+ * @return a {@link org.apache.nifi.security.util.TlsConfiguration}
+ */
+public static TlsConfiguration createTlsConfigAndNewKeystoreTruststore()
throws IOException, GeneralSecurityException {
+return createTlsConfigAndNewKeystoreTruststore(new
StandardTlsConfiguration());
+}
+
+/**
+ * Creates a temporary Keystore and Truststore and returns it wrapped in a
new TLS configuration with the given values.
+ *
+ * @param tlsConfiguration a {@link
org.apache.nifi.security.util.TlsConfiguration}
+ * @return a {@link org.apache.nifi.security.util.TlsConfiguration}
+ */
+public static TlsConfiguration
createTlsConfigAndNewKeystoreTruststore(final TlsConfiguration
tlsConfiguration) throws IOException, GeneralSecurityException {
+final Path keyStorePath;
+final String keystorePassword =
StringUtils.isNotBlank(tlsConfiguration.getKeystorePassword()) ?
tlsConfiguration.getKeystorePassword() : generatePassword();
+final String keyPassword =
StringUtils.isNotBlank(tlsConfiguration.getKeyPassword())?
tlsConfiguration.getKeyPassword() : keystorePassword;
+final KeystoreType keystoreType = tlsConfiguration.getKeystoreType()
!= null ? tlsConfiguration.getKeystoreType() : KeystoreType.PKCS12;
+final Path trustStorePath;
+final String truststorePassword =
StringUtils.isNotBlank(tlsConfiguration.getTruststorePassword()) ?
tlsConfiguration.getTruststorePassword() : "";
+final KeystoreType truststoreType =
tlsConfiguration.getTruststoreType() != null ?
tlsConfiguration.getTruststoreType() : KeystoreType.PKCS12;
+
+// Create temporary Keystore file
+try {
+keyStorePath = generateTempKeystorePath(keystoreType);
+} catch (IOException e) {
+logger.error(KEYSTORE_ERROR_MSG);
+throw new UncheckedIOException(KEYSTORE_ERROR_MSG, e);
+}
+
+// Create temporary Truststore file
+try {
+trustStorePath = generateTempTruststorePath(truststoreType);
+} catch (IOException e) {
+logger.error(TRUSTSTORE_ERROR_MSG);
+throw new UncheckedIOException(TRUSTSTORE_ERROR_MSG, e);
+}
+
+// Create X509 Certificate
+final X509Certificate clientCert =
createKeyStoreAndGetX509Certificate(KEY_ALIAS, keystorePassword, keyPassword,
keyStorePath.toString(), keystoreType);
+
+// Create Truststore
+createTrustStore(clientCert, CERT_ALIAS, truststorePassword,
trustStorePath.toString(), getKeystoreType(truststoreType.toString()));
+
+return new StandardTlsConfiguration(
+keyStorePath.toString(),
+keystorePassword,
+keyPassword,
+getKeystoreType(keystoreType.toString()),
+trustStorePath.toString(),
+truststorePassword,
+getKeystoreType(truststoreType.toString()),
Review comment:
The call is not necessary.
##
File path:
nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/security/util/KeyStoreUtils.java
##
@@ -245,7 +322,7 @@ public static TrustManagerFactory
loadTrustManagerFactory(TlsConfiguration tlsCo
*/
public static TrustManagerFactory loadTrustManagerFactory(String
truststorePath, String truststorePassword, String truststoreType) throws
TlsException {
// Legacy truststore passwords can be empty
-final char[] truststorePasswordChars =
StringUtils.isNotBlank(truststorePassword) ? truststorePassword.toCharArray() :
null;
+final char[] truststorePasswordChars =
StringUtils.isNotBlank(truststorePassword) ? truststorePassword.toCharArray() :
"".toCharArray();
Review comment:
@exceptionfactory I received a Null Pointer Exception for an empty
password when the truststore type is PKCS12, so I changed it to an empty
string. But after some investigation, I found that the Bouncy Castle PKCS12
store type does not allow empty passwords.
Since we allow passwordless truststores, I'll add a check for the truststore
type. If it's PKCS12, then I'll throw an Illegal Argument Exception, otherwise
I'll set it back to `null`.
##
File path:
nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/security/util/KeyStoreUtils.java
##
@@ -245,7 +322,7 @@ public static TrustManagerFactory
loadTrustManagerFactory(TlsConfiguration tlsCo
*/
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #975: MINIFICPP-1400 Create ListS3 processor
lordgamez commented on a change in pull request #975:
URL: https://github.com/apache/nifi-minifi-cpp/pull/975#discussion_r562005670
##
File path: extensions/aws/processors/ListS3.cpp
##
@@ -0,0 +1,294 @@
+/**
+ * @file ListS3.cpp
+ * ListS3 class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ListS3.h"
+
+#include
+#include
+#include
+#include
+#include
+
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace aws {
+namespace processors {
+
+const std::string ListS3::LATEST_LISTED_KEY_PREFIX = "listed_key.";
+const std::string ListS3::LATEST_LISTED_KEY_TIMESTAMP = "listed_key.timestamp";
+
+const core::Property ListS3::Delimiter(
+ core::PropertyBuilder::createProperty("Delimiter")
+->withDescription("The string used to delimit directories within the
bucket. Please consult the AWS documentation for the correct use of this
field.")
+->build());
+const core::Property ListS3::Prefix(
+ core::PropertyBuilder::createProperty("Prefix")
+->withDescription("The prefix used to filter the object list. In most
cases, it should end with a forward slash ('/').")
+->build());
+const core::Property ListS3::UseVersions(
+ core::PropertyBuilder::createProperty("Use Versions")
+->isRequired(true)
+->withDefaultValue(false)
+->withDescription("Specifies whether to use S3 versions, if applicable. If
false, only the latest version of each object will be returned.")
+->build());
+const core::Property ListS3::MinimumObjectAge(
+ core::PropertyBuilder::createProperty("Minimum Object Age")
+->isRequired(true)
+->withDefaultValue("0 sec")
+->withDescription("The minimum age that an S3 object must be in order to
be considered; any object younger than this amount of time (according to last
modification date) will be ignored.")
+->build());
+const core::Property ListS3::WriteObjectTags(
+ core::PropertyBuilder::createProperty("Write Object Tags")
+->isRequired(true)
+->withDefaultValue(false)
+->withDescription("If set to 'true', the tags associated with the S3
object will be written as FlowFile attributes.")
+->build());
+const core::Property ListS3::WriteUserMetadata(
+ core::PropertyBuilder::createProperty("Write User Metadata")
+->isRequired(true)
+->withDefaultValue(false)
+->withDescription("If set to 'true', the user defined metadata associated
with the S3 object will be added to FlowFile attributes/records.")
+->build());
+const core::Property ListS3::RequesterPays(
+ core::PropertyBuilder::createProperty("Requester Pays")
+->isRequired(true)
+->withDefaultValue(false)
+->withDescription("If true, indicates that the requester consents to pay
any charges associated with listing the S3 bucket. This sets the
'x-amz-request-payer' header to 'requester'. "
+ "Note that this setting is only used if Write User
Metadata is true.")
+->build());
+
+const core::Relationship ListS3::Success("success", "FlowFiles are routed to
success relationship");
+
+void ListS3::initialize() {
+ // Add new supported properties
+ updateSupportedProperties({Delimiter, Prefix, UseVersions, MinimumObjectAge,
WriteObjectTags, WriteUserMetadata, RequesterPays});
+ // Set the supported relationships
+ setSupportedRelationships({Success});
+}
+
+void ListS3::onSchedule(const std::shared_ptr ,
const std::shared_ptr ) {
+ S3Processor::onSchedule(context, sessionFactory);
+
+ state_manager_ = context->getStateManager();
+ if (state_manager_ == nullptr) {
+throw Exception(PROCESSOR_EXCEPTION, "Failed to get StateManager");
+ }
+
+ auto common_properties = getCommonELSupportedProperties(context, nullptr);
+ if (!common_properties) {
+throw Exception(PROCESS_SCHEDULE_EXCEPTION, "Required property is not set
or invalid");
+ }
+ configureS3Wrapper(common_properties.value());
+ list_request_params_.bucket = common_properties->bucket;
+
+ context->getProperty(Delimiter.getName(), list_request_params_.delimiter);
+ logger_->log_debug("ListS3: Delimiter [%s]", list_request_params_.delimiter);
+
+ context->getProperty(Prefix.getName(),
[GitHub] [nifi] pvillard31 commented on pull request #4768: NIFI-8155 - add banner text in page title
pvillard31 commented on pull request #4768: URL: https://github.com/apache/nifi/pull/4768#issuecomment-764453581 Thanks for looking into this @exceptionfactory. As far as I can tell the other places you mentioned are not "really" opening new windows and this not impacting the overall page title: https://user-images.githubusercontent.com/11541012/105320361-801ab280-5bdf-11eb-8fc0-969aac536b0c.png;> The only edge case I can think of is when looking at the content of a flow file: this will open a new tab/window and the title won't contain the info. But I believe this a very special edge case and I think this is best to leave it outside of the change. https://user-images.githubusercontent.com/11541012/105320626-d687f100-5bdf-11eb-9785-7dc48cb1da51.png;> This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] patricker commented on a change in pull request #4776: Add "Mailbox" property to ConsumeEWS
patricker commented on a change in pull request #4776:
URL: https://github.com/apache/nifi/pull/4776#discussion_r562273064
##
File path:
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ConsumeEWS.java
##
@@ -107,6 +109,14 @@
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.sensitive(true)
.build();
+public static final PropertyDescriptor MAILBOX = new
PropertyDescriptor.Builder()
+.name("mailbox")
+.displayName("Mailbox")
+.description("Mailbox")
+.required(true)
Review comment:
If your going to make MAILBOX required, then you'll need to include a
default value so that it doesn't break existing users. Or make it not
required, and adjust the code so that it works the way it used to if no MAILBOX
is provided.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] pvillard31 commented on pull request #4738: NIFI-7890 - Added record support to ConsumeMQTT processor
pvillard31 commented on pull request #4738: URL: https://github.com/apache/nifi/pull/4738#issuecomment-764515288 Thanks for the review @turcsanyip - I think I addressed your comments, let me know if there is something else. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] arpadboda closed pull request #957: MINIFICPP-1407 - Fetch flow configuration file from C2 if not found locally
arpadboda closed pull request #957: URL: https://github.com/apache/nifi-minifi-cpp/pull/957 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] arkadiyvleonov commented on a change in pull request #4776: Add "Mailbox" property to ConsumeEWS
arkadiyvleonov commented on a change in pull request #4776:
URL: https://github.com/apache/nifi/pull/4776#discussion_r562370297
##
File path:
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ConsumeEWS.java
##
@@ -107,6 +109,14 @@
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.sensitive(true)
.build();
+public static final PropertyDescriptor MAILBOX = new
PropertyDescriptor.Builder()
+.name("mailbox")
+.displayName("Mailbox")
+.description("Mailbox")
+.required(true)
Review comment:
made some changes in below commits
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] arkadiyvleonov commented on a change in pull request #4776: Add "Mailbox" property to ConsumeEWS
arkadiyvleonov commented on a change in pull request #4776:
URL: https://github.com/apache/nifi/pull/4776#discussion_r562370297
##
File path:
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ConsumeEWS.java
##
@@ -107,6 +109,14 @@
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.sensitive(true)
.build();
+public static final PropertyDescriptor MAILBOX = new
PropertyDescriptor.Builder()
+.name("mailbox")
+.displayName("Mailbox")
+.description("Mailbox")
+.required(true)
Review comment:
made some changes to in below commits
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] thenatog opened a new pull request #4777: NIFI-8166 - Upgraded jackson-databind to 2.9.10.8
thenatog opened a new pull request #4777:
URL: https://github.com/apache/nifi/pull/4777
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
_Enables X functionality; fixes bug NIFI-._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [x] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [x] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [x] Have you written or updated unit tests to verify your changes?
- [x] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NIFI-8166) Upgrade jackson-databind to 2.9.10.8
Nathan Gough created NIFI-8166: -- Summary: Upgrade jackson-databind to 2.9.10.8 Key: NIFI-8166 URL: https://issues.apache.org/jira/browse/NIFI-8166 Project: Apache NiFi Issue Type: Task Components: Core Framework Affects Versions: 1.12.1 Reporter: Nathan Gough Assignee: Nathan Gough Upgrade the jackson-databind version from 2.9.10.5 to 2.9.10.8. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] patricker commented on a change in pull request #4776: Add "Mailbox" property to ConsumeEWS
patricker commented on a change in pull request #4776:
URL: https://github.com/apache/nifi/pull/4776#discussion_r562273064
##
File path:
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ConsumeEWS.java
##
@@ -107,6 +109,14 @@
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.sensitive(true)
.build();
+public static final PropertyDescriptor MAILBOX = new
PropertyDescriptor.Builder()
+.name("mailbox")
+.displayName("Mailbox")
+.description("Mailbox")
+.required(true)
Review comment:
If your going to make MAILBOX required, then you'll need to include a
default value so that it doesn't break existing users. Or make it not
required, and adjust the code so that it works the way it used to if no MAILBOX
is provided.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] arkadiyvleonov opened a new pull request #4776: Add "Mailbox" property to ConsumeEWS
arkadiyvleonov opened a new pull request #4776:
URL: https://github.com/apache/nifi/pull/4776
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
Sometimes I need to consume EWS mailbox by service account that has access
to multiple mailboxes.
for example user "svc_ews_consu...@example.com" has access to two mailboxes
m...@example.com and m...@example.com
Maybe my code is not so good but I build it and it works for me.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #976: MINIFICPP-1448 - CWEL JSON output
szaszm commented on a change in pull request #976:
URL: https://github.com/apache/nifi-minifi-cpp/pull/976#discussion_r562252602
##
File path: extensions/windows-event-log/wel/JSONUtils.h
##
@@ -0,0 +1,80 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#undef RAPIDJSON_ASSERT
+#define RAPIDJSON_ASSERT(x) if (!(x)) throw std::logic_error("rapidjson
exception"); // NOLINT
+
+#include
+
+#include // for std::logic_error
+#include "rapidjson/document.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace wel {
+
+/**
+ * !!WARNING!! the json document must not outlive the xml argument
+ *
+ * Converts each xml element node to a json object of
+ * the form {name: String, attributes: Object, children: Array, text: String}
+ * Aims to preserve most of the input xml structure.
+ */
+rapidjson::Document toRawJSON(const pugi::xml_node& root);
+
+/**
+ * !!WARNING!! the json document must not outlive the xml argument
Review comment:
I would also explain why:
```suggestion
* !!WARNING!! The json document must not outlive the xml argument. For
better performance,
* the created json document stores references to values in the xml node.
Accessing the
* json document after the xml node has been changed or destroyed results in
undefined behavior.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #976: MINIFICPP-1448 - CWEL JSON output
szaszm commented on a change in pull request #976:
URL: https://github.com/apache/nifi-minifi-cpp/pull/976#discussion_r562252602
##
File path: extensions/windows-event-log/wel/JSONUtils.h
##
@@ -0,0 +1,80 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#undef RAPIDJSON_ASSERT
+#define RAPIDJSON_ASSERT(x) if (!(x)) throw std::logic_error("rapidjson
exception"); // NOLINT
+
+#include
+
+#include // for std::logic_error
+#include "rapidjson/document.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace wel {
+
+/**
+ * !!WARNING!! the json document must not outlive the xml argument
+ *
+ * Converts each xml element node to a json object of
+ * the form {name: String, attributes: Object, children: Array, text: String}
+ * Aims to preserve most of the input xml structure.
+ */
+rapidjson::Document toRawJSON(const pugi::xml_node& root);
+
+/**
+ * !!WARNING!! the json document must not outlive the xml argument
Review comment:
I would also explain why:
```suggestion
* !!WARNING!! The json document must not outlive the xml argument. For
better performance,
* the created json document stores references to values in the xml node.
Accessing the
* json document after the xml node has been destroyed results in undefined
behavior.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NIFI-8165) Add record path support to PublishKafkaRecord processor
Dave Lowe created NIFI-8165: --- Summary: Add record path support to PublishKafkaRecord processor Key: NIFI-8165 URL: https://issues.apache.org/jira/browse/NIFI-8165 Project: Apache NiFi Issue Type: New Feature Components: Extensions Affects Versions: 1.11.4 Reporter: Dave Lowe NiFi RecordPath's are not supported within the PublishKafkaRecord processor. Users are unable to leverage RecordPaths within properties such as the message key. Without support of RecordPaths, the message key must be within the root of the record being published - which is often not the case with complex real-world workloads. In such circumstances, the record must be adjusted prior to publishing to Kafka via the UpdateRecord procoessor to copy the nested element to the root so that it can be leveraged as the message key. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] mtien-apache commented on a change in pull request #4767: NIFI-1355 Implemented new methods in KeyStoreUtils to programmatical…
mtien-apache commented on a change in pull request #4767:
URL: https://github.com/apache/nifi/pull/4767#discussion_r562243813
##
File path:
nifi-commons/nifi-security-utils/src/test/groovy/org/apache/nifi/security/util/KeyStoreUtilsGroovyTest.groovy
##
@@ -141,4 +173,181 @@ class KeyStoreUtilsGroovyTest extends GroovyTestCase {
FileOutputStream fos = new
FileOutputStream("/Users/alopresto/Workspace/nifi/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/truststore.no-password.jks")
truststore.store(fos, "".chars)
}
+
+@Test
+void testShouldValidateTempKeystorePath() {
+// Act
+Path testKeystorePath =
KeyStoreUtils.generateTempKeystorePath(JKS_STORE_TYPE)
+deletePath(testKeystorePath)
+
+// Assert
+logger.info("Keystore path: ${testKeystorePath.toString()}")
+assert testKeystorePath
+}
+
+@Test
+void testShouldValidateTempTruststorePath() {
+// Act
+Path truststorePath =
KeyStoreUtils.generateTempTruststorePath(JKS_STORE_TYPE)
+deletePath(truststorePath)
+
+// Assert
+logger.info("Truststore path: ${truststorePath.toString()}")
+assert truststorePath
+}
+
+@Test
+void testShouldValidateTlsConfigAndNewKeystoreTruststoreWithParams() {
+// Arrange
+TlsConfiguration tlsConfigParam = new StandardTlsConfiguration(null,
TEST_KEYSTORE_PASSWORD, TEST_KEY_PASSWORD, JKS_STORE_TYPE, null,
TEST_TRUSTSTORE_PASSWORD, JKS_STORE_TYPE)
+
+// Act
+final TlsConfiguration tlsConfig =
KeyStoreUtils.createTlsConfigAndNewKeystoreTruststore(tlsConfigParam)
+deleteKeystoreTruststore(tlsConfig)
+
+// Assert
+assert tlsConfig.getKeystorePath()
+assert tlsConfig.getTruststorePath()
+assert tlsConfig.getKeystoreType() == KeystoreType.JKS
+assert tlsConfig.getTruststoreType() == KeystoreType.JKS
+assert tlsConfig.getKeystorePassword() == TEST_KEYSTORE_PASSWORD
+}
+
+@Test
+void testShouldValidateTlsConfigAndNewKeystoreTruststoreWithoutParams() {
+// Act
+TlsConfiguration tlsConfig =
KeyStoreUtils.createTlsConfigAndNewKeystoreTruststore()
+deleteKeystoreTruststore(tlsConfig)
+
+// Assert
+assert tlsConfig.getKeystorePath()
+assert tlsConfig.getKeyPassword() == tlsConfig.getKeystorePassword()
+assert tlsConfig.getKeystoreType() == KeystoreType.PKCS12
+assert tlsConfig.getTruststoreType() == KeystoreType.PKCS12
+}
+
+@Test
+void testShouldValidateTlsConfigWithoutKeyPasswordParam() {
+// Arrange
+TlsConfiguration tlsConfigParam = new StandardTlsConfiguration(null,
TEST_KEYSTORE_PASSWORD, null, JKS_STORE_TYPE, null, TEST_TRUSTSTORE_PASSWORD,
JKS_STORE_TYPE)
+
+// Act
+final TlsConfiguration tlsConfig =
KeyStoreUtils.createTlsConfigAndNewKeystoreTruststore(tlsConfigParam)
+deleteKeystoreTruststore(tlsConfig)
+
+// Assert
+assert tlsConfig.getKeyPassword() == tlsConfig.getKeystorePassword()
+}
+
+@Test
+void testShouldReturnX509CertWithNewKeystore() {
+// Arrange
+Path keystorePath =
KeyStoreUtils.generateTempKeystorePath(JKS_STORE_TYPE)
+
+// Act
+X509Certificate x509Cert =
KeyStoreUtils.createKeyStoreAndGetX509Certificate(KEY_ALIAS,
TEST_KEYSTORE_PASSWORD, TEST_KEYSTORE_PASSWORD, keystorePath.toString(),
JKS_STORE_TYPE)
+
+boolean isKeystoreValid =
KeyStoreUtils.isStoreValid(keystorePath.toUri().toURL(), JKS_STORE_TYPE,
TEST_KEYSTORE_PASSWORD.toCharArray())
+deletePath(keystorePath)
+
+// Assert
+final String certDN = x509Cert.getIssuerDN().toString()
+logger.info("Certificate DN: ${certDN}")
+assert certDN == "CN=localhost,OU=NiFi,O=Apache"
Review comment:
I changed the check to assert the string contains `CN=`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] mtien-apache commented on a change in pull request #4767: NIFI-1355 Implemented new methods in KeyStoreUtils to programmatical…
mtien-apache commented on a change in pull request #4767:
URL: https://github.com/apache/nifi/pull/4767#discussion_r562242893
##
File path:
nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/security/util/KeyStoreUtils.java
##
@@ -245,7 +322,7 @@ public static TrustManagerFactory
loadTrustManagerFactory(TlsConfiguration tlsCo
*/
public static TrustManagerFactory loadTrustManagerFactory(String
truststorePath, String truststorePassword, String truststoreType) throws
TlsException {
// Legacy truststore passwords can be empty
-final char[] truststorePasswordChars =
StringUtils.isNotBlank(truststorePassword) ? truststorePassword.toCharArray() :
null;
+final char[] truststorePasswordChars =
StringUtils.isNotBlank(truststorePassword) ? truststorePassword.toCharArray() :
"".toCharArray();
Review comment:
@exceptionfactory After making these changes, I found out the real issue
was that I received an NPE when loading a PKCS12 truststore and passing a
`null` password. For loading a JKS or BCFKS store type, a `null` password is
allowed.
After our discussion and finding out the behavior among keystore/truststore
types are different, we concluded to allow loading a store without a password,
but not persisting a store without a password. These changes affect the methods
I added to programmatically generate certificates, keystores, and truststores.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NIFI-8164) ConsumeMQTT validation bug
Kay-Uwe Moosheimer created NIFI-8164:
Summary: ConsumeMQTT validation bug
Key: NIFI-8164
URL: https://issues.apache.org/jira/browse/NIFI-8164
Project: Apache NiFi
Issue Type: Improvement
Affects Versions: 1.12.1
Reporter: Kay-Uwe Moosheimer
In ConsumeMQTT.java the following lines (196-200)
final boolean clientIDSet = context.getProperty(PROP_CLIENTID).isSet();
final boolean groupIDSet = context.getProperty(PROP_GROUPID).isSet();
if (clientIDSet && groupIDSet) {
results.add(new ValidationResult.Builder().subject("Client ID and Group
ID").valid(false).explanation("if client ID is not unique, multiple nodes
cannot join the consumer group").build());
}
are counterproductive.
It is true that a client ID must be unique for multiple mqtt clients to join a
consumer group.
If you enter the client ID as ${hostname()}-client, then the client ID is
unique (if you use this only once per node).
But this is not possible with the validation.
On the other hand it makes no sense if the client ID is random and you set the
session state to "resume session". A random client ID can never resume a
previous session.
In general it is also not advisable but only an add-on for lazy typists if you
don't use a unique client ID to auto generate a UUID. Because then only random
client IDs appear on the broker, which cannot be assigned to a certain
processor/node easily or not at all (monitoring nightmare).
Therefore please delete these lines, because ConsumeMQTT is applicable in this
way but not really in a proper way.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (NIFI-6999) Encrypt Config Toolkit fails on very large flow.xml.gz files
[
https://issues.apache.org/jira/browse/NIFI-6999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nathan Gough reopened NIFI-6999:
> Encrypt Config Toolkit fails on very large flow.xml.gz files
>
>
> Key: NIFI-6999
> URL: https://issues.apache.org/jira/browse/NIFI-6999
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Tools and Build
>Affects Versions: 1.2.0, 1.10.0
>Reporter: Andy LoPresto
>Assignee: Nathan Gough
>Priority: Critical
> Labels: documentation, encryption, heap, security, streaming,
> toolkit
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> A user reported failure when using the encrypt config toolkit to process
> (encrypt) a large {{flow.xml.gz}}. The compressed file was 49 MB, but was 687
> MB uncompressed. It contained 545 encrypted values, and approximately 90
> templates. This caused the toolkit to fail during {{loadFlowXml()}} unless
> the toolkit invocation set the heap to 8 GB via {{-Xms2g -Xmx8g}}. Even with
> the expanded heap, the serialization of the newly-encrypted flow XML to the
> file system fails with the following exception:
> {code}
> Exception in thread "main" java.lang.OutOfMemoryError: Requested array size
> exceeds VM limit
> at java.lang.StringCoding.encode(StringCoding.java:350)
> at java.lang.String.getBytes(String.java:941)
> at org.apache.commons.io.IOUtils.write(IOUtils.java:1857)
> at org.apache.commons.io.IOUtils$write$0.call(Unknown Source)
> at
> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:141)
> at
> org.apache.nifi.properties.ConfigEncryptionTool$_writeFlowXmlToFile_closure5$_closure20.doCall(ConfigEncryptionTool.groovy:692)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
> at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
> at
> org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
> at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019)
> at groovy.lang.Closure.call(Closure.java:426)
> at groovy.lang.Closure.call(Closure.java:442)
> at
> org.codehaus.groovy.runtime.IOGroovyMethods.withCloseable(IOGroovyMethods.java:1622)
> at
> org.codehaus.groovy.runtime.NioGroovyMethods.withCloseable(NioGroovyMethods.java:1754)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:54)
> at
> org.codehaus.groovy.runtime.metaclass.NewInstanceMetaMethod.invoke(NewInstanceMetaMethod.java:56)
> at
> org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
> at
> org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
> at
> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
> at
> org.apache.nifi.properties.ConfigEncryptionTool$_writeFlowXmlToFile_closure5.doCall(ConfigEncryptionTool.groovy:691)
> {code}
> The immediate fix was to remove the duplicated template definitions in the
> flow definition, returning the file to a reasonable size. However, if run as
> an inline replacement, this can cause the {{flow.xml.gz}} to be overwritten
> with an empty file, potentially leading to data loss. The following steps
> should be taken:
> # Guard against loading/operating on/serializing large files (log statements,
> simple conditional checks)
> # Handle large files internally (change from direct {{String}} access to
> {{BufferedInputStream}}, etc.)
> # Document the internal memory usage of the toolkit in the toolkit guide
> # Document best practices and steps to resolve issue in the toolkit guide
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (NIFI-6999) Encrypt Config Toolkit fails on very large flow.xml.gz files
[
https://issues.apache.org/jira/browse/NIFI-6999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nathan Gough resolved NIFI-6999.
Resolution: Implemented
Added streaming for the flow.xml.gz file specifically. A larger refactor may
need to be done to allow for streaming any file that is read if that were
required in future.
> Encrypt Config Toolkit fails on very large flow.xml.gz files
>
>
> Key: NIFI-6999
> URL: https://issues.apache.org/jira/browse/NIFI-6999
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Tools and Build
>Affects Versions: 1.2.0, 1.10.0
>Reporter: Andy LoPresto
>Assignee: Nathan Gough
>Priority: Critical
> Labels: documentation, encryption, heap, security, streaming,
> toolkit
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> A user reported failure when using the encrypt config toolkit to process
> (encrypt) a large {{flow.xml.gz}}. The compressed file was 49 MB, but was 687
> MB uncompressed. It contained 545 encrypted values, and approximately 90
> templates. This caused the toolkit to fail during {{loadFlowXml()}} unless
> the toolkit invocation set the heap to 8 GB via {{-Xms2g -Xmx8g}}. Even with
> the expanded heap, the serialization of the newly-encrypted flow XML to the
> file system fails with the following exception:
> {code}
> Exception in thread "main" java.lang.OutOfMemoryError: Requested array size
> exceeds VM limit
> at java.lang.StringCoding.encode(StringCoding.java:350)
> at java.lang.String.getBytes(String.java:941)
> at org.apache.commons.io.IOUtils.write(IOUtils.java:1857)
> at org.apache.commons.io.IOUtils$write$0.call(Unknown Source)
> at
> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:141)
> at
> org.apache.nifi.properties.ConfigEncryptionTool$_writeFlowXmlToFile_closure5$_closure20.doCall(ConfigEncryptionTool.groovy:692)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
> at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
> at
> org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
> at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019)
> at groovy.lang.Closure.call(Closure.java:426)
> at groovy.lang.Closure.call(Closure.java:442)
> at
> org.codehaus.groovy.runtime.IOGroovyMethods.withCloseable(IOGroovyMethods.java:1622)
> at
> org.codehaus.groovy.runtime.NioGroovyMethods.withCloseable(NioGroovyMethods.java:1754)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:54)
> at
> org.codehaus.groovy.runtime.metaclass.NewInstanceMetaMethod.invoke(NewInstanceMetaMethod.java:56)
> at
> org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
> at
> org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
> at
> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
> at
> org.apache.nifi.properties.ConfigEncryptionTool$_writeFlowXmlToFile_closure5.doCall(ConfigEncryptionTool.groovy:691)
> {code}
> The immediate fix was to remove the duplicated template definitions in the
> flow definition, returning the file to a reasonable size. However, if run as
> an inline replacement, this can cause the {{flow.xml.gz}} to be overwritten
> with an empty file, potentially leading to data loss. The following steps
> should be taken:
> # Guard against loading/operating on/serializing large files (log statements,
> simple conditional checks)
> # Handle large files internally (change from direct {{String}} access to
> {{BufferedInputStream}}, etc.)
> # Document the internal memory usage of the toolkit in the toolkit guide
> # Document
[jira] [Commented] (NIFI-6999) Encrypt Config Toolkit fails on very large flow.xml.gz files
[
https://issues.apache.org/jira/browse/NIFI-6999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269623#comment-17269623
]
ASF subversion and git services commented on NIFI-6999:
---
Commit 1c361d45ae94f155b6e2def7bd4430b1c9ca8b3b in nifi's branch
refs/heads/main from Nathan Gough
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=1c361d4 ]
NIFI-6999 - Made changes to load flow.xml files using streams. Updated tests.
NIFI-6999 - Slight change to test to check for WARN message.
NIFI-6999 - Removed very large flow file and test that uses it. This test ran
for about 2 minutes so was excessive to keep in. The other changed tests to
handle streams proves the functionality. A large file can be used on the
command line to manually test large flow files. Some other cleanup.
NIFI-6999 - Removed comments and altered the code a little bit for readability
as per code review.
NIFI-6999 - Removed commented code
NIFI-6999 - Renamed variable and removed assert comment.
Signed-off-by: Nathan Gough
This closes #4715.
> Encrypt Config Toolkit fails on very large flow.xml.gz files
>
>
> Key: NIFI-6999
> URL: https://issues.apache.org/jira/browse/NIFI-6999
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Tools and Build
>Affects Versions: 1.2.0, 1.10.0
>Reporter: Andy LoPresto
>Assignee: Nathan Gough
>Priority: Critical
> Labels: documentation, encryption, heap, security, streaming,
> toolkit
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> A user reported failure when using the encrypt config toolkit to process
> (encrypt) a large {{flow.xml.gz}}. The compressed file was 49 MB, but was 687
> MB uncompressed. It contained 545 encrypted values, and approximately 90
> templates. This caused the toolkit to fail during {{loadFlowXml()}} unless
> the toolkit invocation set the heap to 8 GB via {{-Xms2g -Xmx8g}}. Even with
> the expanded heap, the serialization of the newly-encrypted flow XML to the
> file system fails with the following exception:
> {code}
> Exception in thread "main" java.lang.OutOfMemoryError: Requested array size
> exceeds VM limit
> at java.lang.StringCoding.encode(StringCoding.java:350)
> at java.lang.String.getBytes(String.java:941)
> at org.apache.commons.io.IOUtils.write(IOUtils.java:1857)
> at org.apache.commons.io.IOUtils$write$0.call(Unknown Source)
> at
> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:141)
> at
> org.apache.nifi.properties.ConfigEncryptionTool$_writeFlowXmlToFile_closure5$_closure20.doCall(ConfigEncryptionTool.groovy:692)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
> at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
> at
> org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
> at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019)
> at groovy.lang.Closure.call(Closure.java:426)
> at groovy.lang.Closure.call(Closure.java:442)
> at
> org.codehaus.groovy.runtime.IOGroovyMethods.withCloseable(IOGroovyMethods.java:1622)
> at
> org.codehaus.groovy.runtime.NioGroovyMethods.withCloseable(NioGroovyMethods.java:1754)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:54)
> at
> org.codehaus.groovy.runtime.metaclass.NewInstanceMetaMethod.invoke(NewInstanceMetaMethod.java:56)
> at
> org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
> at
> org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
> at
> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
> at
>
[jira] [Commented] (NIFI-6999) Encrypt Config Toolkit fails on very large flow.xml.gz files
[
https://issues.apache.org/jira/browse/NIFI-6999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269622#comment-17269622
]
ASF subversion and git services commented on NIFI-6999:
---
Commit 1c361d45ae94f155b6e2def7bd4430b1c9ca8b3b in nifi's branch
refs/heads/main from Nathan Gough
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=1c361d4 ]
NIFI-6999 - Made changes to load flow.xml files using streams. Updated tests.
NIFI-6999 - Slight change to test to check for WARN message.
NIFI-6999 - Removed very large flow file and test that uses it. This test ran
for about 2 minutes so was excessive to keep in. The other changed tests to
handle streams proves the functionality. A large file can be used on the
command line to manually test large flow files. Some other cleanup.
NIFI-6999 - Removed comments and altered the code a little bit for readability
as per code review.
NIFI-6999 - Removed commented code
NIFI-6999 - Renamed variable and removed assert comment.
Signed-off-by: Nathan Gough
This closes #4715.
> Encrypt Config Toolkit fails on very large flow.xml.gz files
>
>
> Key: NIFI-6999
> URL: https://issues.apache.org/jira/browse/NIFI-6999
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Tools and Build
>Affects Versions: 1.2.0, 1.10.0
>Reporter: Andy LoPresto
>Assignee: Nathan Gough
>Priority: Critical
> Labels: documentation, encryption, heap, security, streaming,
> toolkit
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> A user reported failure when using the encrypt config toolkit to process
> (encrypt) a large {{flow.xml.gz}}. The compressed file was 49 MB, but was 687
> MB uncompressed. It contained 545 encrypted values, and approximately 90
> templates. This caused the toolkit to fail during {{loadFlowXml()}} unless
> the toolkit invocation set the heap to 8 GB via {{-Xms2g -Xmx8g}}. Even with
> the expanded heap, the serialization of the newly-encrypted flow XML to the
> file system fails with the following exception:
> {code}
> Exception in thread "main" java.lang.OutOfMemoryError: Requested array size
> exceeds VM limit
> at java.lang.StringCoding.encode(StringCoding.java:350)
> at java.lang.String.getBytes(String.java:941)
> at org.apache.commons.io.IOUtils.write(IOUtils.java:1857)
> at org.apache.commons.io.IOUtils$write$0.call(Unknown Source)
> at
> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:141)
> at
> org.apache.nifi.properties.ConfigEncryptionTool$_writeFlowXmlToFile_closure5$_closure20.doCall(ConfigEncryptionTool.groovy:692)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
> at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
> at
> org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
> at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019)
> at groovy.lang.Closure.call(Closure.java:426)
> at groovy.lang.Closure.call(Closure.java:442)
> at
> org.codehaus.groovy.runtime.IOGroovyMethods.withCloseable(IOGroovyMethods.java:1622)
> at
> org.codehaus.groovy.runtime.NioGroovyMethods.withCloseable(NioGroovyMethods.java:1754)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:54)
> at
> org.codehaus.groovy.runtime.metaclass.NewInstanceMetaMethod.invoke(NewInstanceMetaMethod.java:56)
> at
> org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
> at
> org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
> at
> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
> at
>
[jira] [Commented] (NIFI-6999) Encrypt Config Toolkit fails on very large flow.xml.gz files
[
https://issues.apache.org/jira/browse/NIFI-6999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269620#comment-17269620
]
ASF subversion and git services commented on NIFI-6999:
---
Commit 1c361d45ae94f155b6e2def7bd4430b1c9ca8b3b in nifi's branch
refs/heads/main from Nathan Gough
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=1c361d4 ]
NIFI-6999 - Made changes to load flow.xml files using streams. Updated tests.
NIFI-6999 - Slight change to test to check for WARN message.
NIFI-6999 - Removed very large flow file and test that uses it. This test ran
for about 2 minutes so was excessive to keep in. The other changed tests to
handle streams proves the functionality. A large file can be used on the
command line to manually test large flow files. Some other cleanup.
NIFI-6999 - Removed comments and altered the code a little bit for readability
as per code review.
NIFI-6999 - Removed commented code
NIFI-6999 - Renamed variable and removed assert comment.
Signed-off-by: Nathan Gough
This closes #4715.
> Encrypt Config Toolkit fails on very large flow.xml.gz files
>
>
> Key: NIFI-6999
> URL: https://issues.apache.org/jira/browse/NIFI-6999
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Tools and Build
>Affects Versions: 1.2.0, 1.10.0
>Reporter: Andy LoPresto
>Assignee: Nathan Gough
>Priority: Critical
> Labels: documentation, encryption, heap, security, streaming,
> toolkit
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> A user reported failure when using the encrypt config toolkit to process
> (encrypt) a large {{flow.xml.gz}}. The compressed file was 49 MB, but was 687
> MB uncompressed. It contained 545 encrypted values, and approximately 90
> templates. This caused the toolkit to fail during {{loadFlowXml()}} unless
> the toolkit invocation set the heap to 8 GB via {{-Xms2g -Xmx8g}}. Even with
> the expanded heap, the serialization of the newly-encrypted flow XML to the
> file system fails with the following exception:
> {code}
> Exception in thread "main" java.lang.OutOfMemoryError: Requested array size
> exceeds VM limit
> at java.lang.StringCoding.encode(StringCoding.java:350)
> at java.lang.String.getBytes(String.java:941)
> at org.apache.commons.io.IOUtils.write(IOUtils.java:1857)
> at org.apache.commons.io.IOUtils$write$0.call(Unknown Source)
> at
> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:141)
> at
> org.apache.nifi.properties.ConfigEncryptionTool$_writeFlowXmlToFile_closure5$_closure20.doCall(ConfigEncryptionTool.groovy:692)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
> at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
> at
> org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
> at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019)
> at groovy.lang.Closure.call(Closure.java:426)
> at groovy.lang.Closure.call(Closure.java:442)
> at
> org.codehaus.groovy.runtime.IOGroovyMethods.withCloseable(IOGroovyMethods.java:1622)
> at
> org.codehaus.groovy.runtime.NioGroovyMethods.withCloseable(NioGroovyMethods.java:1754)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:54)
> at
> org.codehaus.groovy.runtime.metaclass.NewInstanceMetaMethod.invoke(NewInstanceMetaMethod.java:56)
> at
> org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
> at
> org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
> at
> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
> at
>
[jira] [Commented] (NIFI-6999) Encrypt Config Toolkit fails on very large flow.xml.gz files
[
https://issues.apache.org/jira/browse/NIFI-6999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269621#comment-17269621
]
ASF subversion and git services commented on NIFI-6999:
---
Commit 1c361d45ae94f155b6e2def7bd4430b1c9ca8b3b in nifi's branch
refs/heads/main from Nathan Gough
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=1c361d4 ]
NIFI-6999 - Made changes to load flow.xml files using streams. Updated tests.
NIFI-6999 - Slight change to test to check for WARN message.
NIFI-6999 - Removed very large flow file and test that uses it. This test ran
for about 2 minutes so was excessive to keep in. The other changed tests to
handle streams proves the functionality. A large file can be used on the
command line to manually test large flow files. Some other cleanup.
NIFI-6999 - Removed comments and altered the code a little bit for readability
as per code review.
NIFI-6999 - Removed commented code
NIFI-6999 - Renamed variable and removed assert comment.
Signed-off-by: Nathan Gough
This closes #4715.
> Encrypt Config Toolkit fails on very large flow.xml.gz files
>
>
> Key: NIFI-6999
> URL: https://issues.apache.org/jira/browse/NIFI-6999
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Tools and Build
>Affects Versions: 1.2.0, 1.10.0
>Reporter: Andy LoPresto
>Assignee: Nathan Gough
>Priority: Critical
> Labels: documentation, encryption, heap, security, streaming,
> toolkit
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> A user reported failure when using the encrypt config toolkit to process
> (encrypt) a large {{flow.xml.gz}}. The compressed file was 49 MB, but was 687
> MB uncompressed. It contained 545 encrypted values, and approximately 90
> templates. This caused the toolkit to fail during {{loadFlowXml()}} unless
> the toolkit invocation set the heap to 8 GB via {{-Xms2g -Xmx8g}}. Even with
> the expanded heap, the serialization of the newly-encrypted flow XML to the
> file system fails with the following exception:
> {code}
> Exception in thread "main" java.lang.OutOfMemoryError: Requested array size
> exceeds VM limit
> at java.lang.StringCoding.encode(StringCoding.java:350)
> at java.lang.String.getBytes(String.java:941)
> at org.apache.commons.io.IOUtils.write(IOUtils.java:1857)
> at org.apache.commons.io.IOUtils$write$0.call(Unknown Source)
> at
> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:141)
> at
> org.apache.nifi.properties.ConfigEncryptionTool$_writeFlowXmlToFile_closure5$_closure20.doCall(ConfigEncryptionTool.groovy:692)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
> at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
> at
> org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
> at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019)
> at groovy.lang.Closure.call(Closure.java:426)
> at groovy.lang.Closure.call(Closure.java:442)
> at
> org.codehaus.groovy.runtime.IOGroovyMethods.withCloseable(IOGroovyMethods.java:1622)
> at
> org.codehaus.groovy.runtime.NioGroovyMethods.withCloseable(NioGroovyMethods.java:1754)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:54)
> at
> org.codehaus.groovy.runtime.metaclass.NewInstanceMetaMethod.invoke(NewInstanceMetaMethod.java:56)
> at
> org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
> at
> org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
> at
> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
> at
>
[jira] [Commented] (NIFI-6999) Encrypt Config Toolkit fails on very large flow.xml.gz files
[
https://issues.apache.org/jira/browse/NIFI-6999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269619#comment-17269619
]
ASF subversion and git services commented on NIFI-6999:
---
Commit 1c361d45ae94f155b6e2def7bd4430b1c9ca8b3b in nifi's branch
refs/heads/main from Nathan Gough
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=1c361d4 ]
NIFI-6999 - Made changes to load flow.xml files using streams. Updated tests.
NIFI-6999 - Slight change to test to check for WARN message.
NIFI-6999 - Removed very large flow file and test that uses it. This test ran
for about 2 minutes so was excessive to keep in. The other changed tests to
handle streams proves the functionality. A large file can be used on the
command line to manually test large flow files. Some other cleanup.
NIFI-6999 - Removed comments and altered the code a little bit for readability
as per code review.
NIFI-6999 - Removed commented code
NIFI-6999 - Renamed variable and removed assert comment.
Signed-off-by: Nathan Gough
This closes #4715.
> Encrypt Config Toolkit fails on very large flow.xml.gz files
>
>
> Key: NIFI-6999
> URL: https://issues.apache.org/jira/browse/NIFI-6999
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Tools and Build
>Affects Versions: 1.2.0, 1.10.0
>Reporter: Andy LoPresto
>Assignee: Nathan Gough
>Priority: Critical
> Labels: documentation, encryption, heap, security, streaming,
> toolkit
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> A user reported failure when using the encrypt config toolkit to process
> (encrypt) a large {{flow.xml.gz}}. The compressed file was 49 MB, but was 687
> MB uncompressed. It contained 545 encrypted values, and approximately 90
> templates. This caused the toolkit to fail during {{loadFlowXml()}} unless
> the toolkit invocation set the heap to 8 GB via {{-Xms2g -Xmx8g}}. Even with
> the expanded heap, the serialization of the newly-encrypted flow XML to the
> file system fails with the following exception:
> {code}
> Exception in thread "main" java.lang.OutOfMemoryError: Requested array size
> exceeds VM limit
> at java.lang.StringCoding.encode(StringCoding.java:350)
> at java.lang.String.getBytes(String.java:941)
> at org.apache.commons.io.IOUtils.write(IOUtils.java:1857)
> at org.apache.commons.io.IOUtils$write$0.call(Unknown Source)
> at
> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:141)
> at
> org.apache.nifi.properties.ConfigEncryptionTool$_writeFlowXmlToFile_closure5$_closure20.doCall(ConfigEncryptionTool.groovy:692)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
> at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
> at
> org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
> at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019)
> at groovy.lang.Closure.call(Closure.java:426)
> at groovy.lang.Closure.call(Closure.java:442)
> at
> org.codehaus.groovy.runtime.IOGroovyMethods.withCloseable(IOGroovyMethods.java:1622)
> at
> org.codehaus.groovy.runtime.NioGroovyMethods.withCloseable(NioGroovyMethods.java:1754)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:54)
> at
> org.codehaus.groovy.runtime.metaclass.NewInstanceMetaMethod.invoke(NewInstanceMetaMethod.java:56)
> at
> org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
> at
> org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
> at
> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
> at
>
[jira] [Commented] (NIFI-6999) Encrypt Config Toolkit fails on very large flow.xml.gz files
[
https://issues.apache.org/jira/browse/NIFI-6999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269618#comment-17269618
]
ASF subversion and git services commented on NIFI-6999:
---
Commit 1c361d45ae94f155b6e2def7bd4430b1c9ca8b3b in nifi's branch
refs/heads/main from Nathan Gough
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=1c361d4 ]
NIFI-6999 - Made changes to load flow.xml files using streams. Updated tests.
NIFI-6999 - Slight change to test to check for WARN message.
NIFI-6999 - Removed very large flow file and test that uses it. This test ran
for about 2 minutes so was excessive to keep in. The other changed tests to
handle streams proves the functionality. A large file can be used on the
command line to manually test large flow files. Some other cleanup.
NIFI-6999 - Removed comments and altered the code a little bit for readability
as per code review.
NIFI-6999 - Removed commented code
NIFI-6999 - Renamed variable and removed assert comment.
Signed-off-by: Nathan Gough
This closes #4715.
> Encrypt Config Toolkit fails on very large flow.xml.gz files
>
>
> Key: NIFI-6999
> URL: https://issues.apache.org/jira/browse/NIFI-6999
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Tools and Build
>Affects Versions: 1.2.0, 1.10.0
>Reporter: Andy LoPresto
>Assignee: Nathan Gough
>Priority: Critical
> Labels: documentation, encryption, heap, security, streaming,
> toolkit
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> A user reported failure when using the encrypt config toolkit to process
> (encrypt) a large {{flow.xml.gz}}. The compressed file was 49 MB, but was 687
> MB uncompressed. It contained 545 encrypted values, and approximately 90
> templates. This caused the toolkit to fail during {{loadFlowXml()}} unless
> the toolkit invocation set the heap to 8 GB via {{-Xms2g -Xmx8g}}. Even with
> the expanded heap, the serialization of the newly-encrypted flow XML to the
> file system fails with the following exception:
> {code}
> Exception in thread "main" java.lang.OutOfMemoryError: Requested array size
> exceeds VM limit
> at java.lang.StringCoding.encode(StringCoding.java:350)
> at java.lang.String.getBytes(String.java:941)
> at org.apache.commons.io.IOUtils.write(IOUtils.java:1857)
> at org.apache.commons.io.IOUtils$write$0.call(Unknown Source)
> at
> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:141)
> at
> org.apache.nifi.properties.ConfigEncryptionTool$_writeFlowXmlToFile_closure5$_closure20.doCall(ConfigEncryptionTool.groovy:692)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
> at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
> at
> org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
> at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019)
> at groovy.lang.Closure.call(Closure.java:426)
> at groovy.lang.Closure.call(Closure.java:442)
> at
> org.codehaus.groovy.runtime.IOGroovyMethods.withCloseable(IOGroovyMethods.java:1622)
> at
> org.codehaus.groovy.runtime.NioGroovyMethods.withCloseable(NioGroovyMethods.java:1754)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:54)
> at
> org.codehaus.groovy.runtime.metaclass.NewInstanceMetaMethod.invoke(NewInstanceMetaMethod.java:56)
> at
> org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
> at
> org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
> at
> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
> at
>
[GitHub] [nifi] thenatog closed pull request #4715: NIFI-6999 - Made changes to load flow.xml files using streams. Update…
thenatog closed pull request #4715: URL: https://github.com/apache/nifi/pull/4715 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] thenatog commented on pull request #4715: NIFI-6999 - Made changes to load flow.xml files using streams. Update…
thenatog commented on pull request #4715: URL: https://github.com/apache/nifi/pull/4715#issuecomment-764911205 Will merge. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (NIFI-7356) Enable TLS for embedded Zookeeper when NiFi has TLS enabled
[
https://issues.apache.org/jira/browse/NIFI-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nathan Gough resolved NIFI-7356.
Resolution: Implemented
> Enable TLS for embedded Zookeeper when NiFi has TLS enabled
> ---
>
> Key: NIFI-7356
> URL: https://issues.apache.org/jira/browse/NIFI-7356
> Project: Apache NiFi
> Issue Type: Sub-task
> Components: Configuration, Configuration Management, Security
>Reporter: Troy Melhase
>Assignee: Nathan Gough
>Priority: Major
> Fix For: 1.13.0
>
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> If embedded ZK has TLS properties in the {{zookeeper.properties}} file, these
> will be used. If however, this file does not populate those properties, and
> NiFi does have TLS properties configured ({{nifi.security.keyStore}}, etc.),
> these values will be used to override the ZK plaintext connection listener to
> create a TLS connection listener.
> If the {{zookeeper.properties}} file has an incomplete configuration (i.e.
> keystore password set but no keystore path), startup should fail with a clear
> error message indicating the missing properties and how to resolve.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (NIFI-8163) When RPG is invalid, process groups show incorrect count for 'invalid'
[ https://issues.apache.org/jira/browse/NIFI-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Gough resolved NIFI-8163. Resolution: Fixed > When RPG is invalid, process groups show incorrect count for 'invalid' > -- > > Key: NIFI-8163 > URL: https://issues.apache.org/jira/browse/NIFI-8163 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.13.0 > > Time Spent: 40m > Remaining Estimate: 0h > > When a Remote Process Group is created and is invalid (due to invalid > hostname, for example), the parent Process Group should have an 'invalid' > count of 1. Which it does. Its parent should also show an invalid count of 1, > but it shows 2. And its parent shows 3 instead of 1, and so on. So if the RPG > exists 6 levels deep, it will show an invalid count of 7 instead of 1. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8163) When RPG is invalid, process groups show incorrect count for 'invalid'
[ https://issues.apache.org/jira/browse/NIFI-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269599#comment-17269599 ] ASF subversion and git services commented on NIFI-8163: --- Commit f2a16cd02e80734f3f5e7c66c8d82211a659e34b in nifi's branch refs/heads/main from Mark Payne [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f2a16cd ] NIFI-8163: When counting number of components, we traverse into all Process Groups, but then call findAllRemoteProcessGroups, which is a recursive call, instead of calling getRemoteProcessGroups(). This results in counting the Process Groups many times. So fixed that. Signed-off-by: Nathan Gough This closes #4775. > When RPG is invalid, process groups show incorrect count for 'invalid' > -- > > Key: NIFI-8163 > URL: https://issues.apache.org/jira/browse/NIFI-8163 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.13.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > When a Remote Process Group is created and is invalid (due to invalid > hostname, for example), the parent Process Group should have an 'invalid' > count of 1. Which it does. Its parent should also show an invalid count of 1, > but it shows 2. And its parent shows 3 instead of 1, and so on. So if the RPG > exists 6 levels deep, it will show an invalid count of 7 instead of 1. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] thenatog closed pull request #4775: NIFI-8163: When counting number of components, we traverse into all P…
thenatog closed pull request #4775: URL: https://github.com/apache/nifi/pull/4775 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] thenatog commented on pull request #4775: NIFI-8163: When counting number of components, we traverse into all P…
thenatog commented on pull request #4775: URL: https://github.com/apache/nifi/pull/4775#issuecomment-764901659 Tested this out and could see the count issue without the fix. Rebuilt with fix and the counts are coming up correct now. Cool, looks good, +1, will merge. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] thenatog commented on pull request #4775: NIFI-8163: When counting number of components, we traverse into all P…
thenatog commented on pull request #4775: URL: https://github.com/apache/nifi/pull/4775#issuecomment-764893877 Reviewing... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-7356) Enable TLS for embedded Zookeeper when NiFi has TLS enabled
[
https://issues.apache.org/jira/browse/NIFI-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269551#comment-17269551
]
ASF subversion and git services commented on NIFI-7356:
---
Commit 76648bdc0b2a077bed6073d30345e9d1af876920 in nifi's branch
refs/heads/main from Troy Melhase
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=76648bd ]
NIFI-7356 - Config TLS for embedded ZooKeeper when NiFi TLS enabled.
NIFI-7356 - Addresses PR feedback.
NIFI-7356 - Additional changes from PR feedback.
NIFI-7356 - Adding integration tests for ZooKeeperStateServer for TLS.
NIFI-7356 - TLS + Zookeeper now working with single and quorum. Needs code
cleanup, need to fix IT tests and docs.
NIFI-7356 - Fixed up tests and removed some irrelevant ones. Refactored some of
ZooKeeperStateServer. Tested successfully with a secure and insecure 3 node
NiFi + Quorum.
NIFI-7356 - Checkstyle fixes.
NIFI-7356 - Updated administration guide with embedded ZooKeeper TLS
configuration.
NIFI-7356 - Updated the way ZooKeeper TLS properties are set/mapped from NiFi
properties.
NIFI-7356 - Updated per review, using NiFiProperties keystore strings,
classname for ocnnection factory, adjusted TLS configuration checks in
NiFiProperties.
NIFI-7356 - Updated configuration validation logic and added tests.
NIFI-7356 - Codestyle check fixes.
NIFI-7356 - Updated some of the log messages.
NIFI-7356 - Updated as per code review.
NIFI-7356 - Fixed max port number.
NIFI-7356 - Updated admin guide and updated small code issues as per code
review.
Signed-off-by: Nathan Gough
This closes #4753.
> Enable TLS for embedded Zookeeper when NiFi has TLS enabled
> ---
>
> Key: NIFI-7356
> URL: https://issues.apache.org/jira/browse/NIFI-7356
> Project: Apache NiFi
> Issue Type: Sub-task
> Components: Configuration, Configuration Management, Security
>Reporter: Troy Melhase
>Assignee: Nathan Gough
>Priority: Major
> Fix For: 1.13.0
>
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> If embedded ZK has TLS properties in the {{zookeeper.properties}} file, these
> will be used. If however, this file does not populate those properties, and
> NiFi does have TLS properties configured ({{nifi.security.keyStore}}, etc.),
> these values will be used to override the ZK plaintext connection listener to
> create a TLS connection listener.
> If the {{zookeeper.properties}} file has an incomplete configuration (i.e.
> keystore password set but no keystore path), startup should fail with a clear
> error message indicating the missing properties and how to resolve.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7356) Enable TLS for embedded Zookeeper when NiFi has TLS enabled
[
https://issues.apache.org/jira/browse/NIFI-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269549#comment-17269549
]
ASF subversion and git services commented on NIFI-7356:
---
Commit 76648bdc0b2a077bed6073d30345e9d1af876920 in nifi's branch
refs/heads/main from Troy Melhase
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=76648bd ]
NIFI-7356 - Config TLS for embedded ZooKeeper when NiFi TLS enabled.
NIFI-7356 - Addresses PR feedback.
NIFI-7356 - Additional changes from PR feedback.
NIFI-7356 - Adding integration tests for ZooKeeperStateServer for TLS.
NIFI-7356 - TLS + Zookeeper now working with single and quorum. Needs code
cleanup, need to fix IT tests and docs.
NIFI-7356 - Fixed up tests and removed some irrelevant ones. Refactored some of
ZooKeeperStateServer. Tested successfully with a secure and insecure 3 node
NiFi + Quorum.
NIFI-7356 - Checkstyle fixes.
NIFI-7356 - Updated administration guide with embedded ZooKeeper TLS
configuration.
NIFI-7356 - Updated the way ZooKeeper TLS properties are set/mapped from NiFi
properties.
NIFI-7356 - Updated per review, using NiFiProperties keystore strings,
classname for ocnnection factory, adjusted TLS configuration checks in
NiFiProperties.
NIFI-7356 - Updated configuration validation logic and added tests.
NIFI-7356 - Codestyle check fixes.
NIFI-7356 - Updated some of the log messages.
NIFI-7356 - Updated as per code review.
NIFI-7356 - Fixed max port number.
NIFI-7356 - Updated admin guide and updated small code issues as per code
review.
Signed-off-by: Nathan Gough
This closes #4753.
> Enable TLS for embedded Zookeeper when NiFi has TLS enabled
> ---
>
> Key: NIFI-7356
> URL: https://issues.apache.org/jira/browse/NIFI-7356
> Project: Apache NiFi
> Issue Type: Sub-task
> Components: Configuration, Configuration Management, Security
>Reporter: Troy Melhase
>Assignee: Nathan Gough
>Priority: Major
> Fix For: 1.13.0
>
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> If embedded ZK has TLS properties in the {{zookeeper.properties}} file, these
> will be used. If however, this file does not populate those properties, and
> NiFi does have TLS properties configured ({{nifi.security.keyStore}}, etc.),
> these values will be used to override the ZK plaintext connection listener to
> create a TLS connection listener.
> If the {{zookeeper.properties}} file has an incomplete configuration (i.e.
> keystore password set but no keystore path), startup should fail with a clear
> error message indicating the missing properties and how to resolve.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7356) Enable TLS for embedded Zookeeper when NiFi has TLS enabled
[
https://issues.apache.org/jira/browse/NIFI-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269548#comment-17269548
]
ASF subversion and git services commented on NIFI-7356:
---
Commit 76648bdc0b2a077bed6073d30345e9d1af876920 in nifi's branch
refs/heads/main from Troy Melhase
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=76648bd ]
NIFI-7356 - Config TLS for embedded ZooKeeper when NiFi TLS enabled.
NIFI-7356 - Addresses PR feedback.
NIFI-7356 - Additional changes from PR feedback.
NIFI-7356 - Adding integration tests for ZooKeeperStateServer for TLS.
NIFI-7356 - TLS + Zookeeper now working with single and quorum. Needs code
cleanup, need to fix IT tests and docs.
NIFI-7356 - Fixed up tests and removed some irrelevant ones. Refactored some of
ZooKeeperStateServer. Tested successfully with a secure and insecure 3 node
NiFi + Quorum.
NIFI-7356 - Checkstyle fixes.
NIFI-7356 - Updated administration guide with embedded ZooKeeper TLS
configuration.
NIFI-7356 - Updated the way ZooKeeper TLS properties are set/mapped from NiFi
properties.
NIFI-7356 - Updated per review, using NiFiProperties keystore strings,
classname for ocnnection factory, adjusted TLS configuration checks in
NiFiProperties.
NIFI-7356 - Updated configuration validation logic and added tests.
NIFI-7356 - Codestyle check fixes.
NIFI-7356 - Updated some of the log messages.
NIFI-7356 - Updated as per code review.
NIFI-7356 - Fixed max port number.
NIFI-7356 - Updated admin guide and updated small code issues as per code
review.
Signed-off-by: Nathan Gough
This closes #4753.
> Enable TLS for embedded Zookeeper when NiFi has TLS enabled
> ---
>
> Key: NIFI-7356
> URL: https://issues.apache.org/jira/browse/NIFI-7356
> Project: Apache NiFi
> Issue Type: Sub-task
> Components: Configuration, Configuration Management, Security
>Reporter: Troy Melhase
>Assignee: Nathan Gough
>Priority: Major
> Fix For: 1.13.0
>
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> If embedded ZK has TLS properties in the {{zookeeper.properties}} file, these
> will be used. If however, this file does not populate those properties, and
> NiFi does have TLS properties configured ({{nifi.security.keyStore}}, etc.),
> these values will be used to override the ZK plaintext connection listener to
> create a TLS connection listener.
> If the {{zookeeper.properties}} file has an incomplete configuration (i.e.
> keystore password set but no keystore path), startup should fail with a clear
> error message indicating the missing properties and how to resolve.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7356) Enable TLS for embedded Zookeeper when NiFi has TLS enabled
[
https://issues.apache.org/jira/browse/NIFI-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269550#comment-17269550
]
ASF subversion and git services commented on NIFI-7356:
---
Commit 76648bdc0b2a077bed6073d30345e9d1af876920 in nifi's branch
refs/heads/main from Troy Melhase
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=76648bd ]
NIFI-7356 - Config TLS for embedded ZooKeeper when NiFi TLS enabled.
NIFI-7356 - Addresses PR feedback.
NIFI-7356 - Additional changes from PR feedback.
NIFI-7356 - Adding integration tests for ZooKeeperStateServer for TLS.
NIFI-7356 - TLS + Zookeeper now working with single and quorum. Needs code
cleanup, need to fix IT tests and docs.
NIFI-7356 - Fixed up tests and removed some irrelevant ones. Refactored some of
ZooKeeperStateServer. Tested successfully with a secure and insecure 3 node
NiFi + Quorum.
NIFI-7356 - Checkstyle fixes.
NIFI-7356 - Updated administration guide with embedded ZooKeeper TLS
configuration.
NIFI-7356 - Updated the way ZooKeeper TLS properties are set/mapped from NiFi
properties.
NIFI-7356 - Updated per review, using NiFiProperties keystore strings,
classname for ocnnection factory, adjusted TLS configuration checks in
NiFiProperties.
NIFI-7356 - Updated configuration validation logic and added tests.
NIFI-7356 - Codestyle check fixes.
NIFI-7356 - Updated some of the log messages.
NIFI-7356 - Updated as per code review.
NIFI-7356 - Fixed max port number.
NIFI-7356 - Updated admin guide and updated small code issues as per code
review.
Signed-off-by: Nathan Gough
This closes #4753.
> Enable TLS for embedded Zookeeper when NiFi has TLS enabled
> ---
>
> Key: NIFI-7356
> URL: https://issues.apache.org/jira/browse/NIFI-7356
> Project: Apache NiFi
> Issue Type: Sub-task
> Components: Configuration, Configuration Management, Security
>Reporter: Troy Melhase
>Assignee: Nathan Gough
>Priority: Major
> Fix For: 1.13.0
>
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> If embedded ZK has TLS properties in the {{zookeeper.properties}} file, these
> will be used. If however, this file does not populate those properties, and
> NiFi does have TLS properties configured ({{nifi.security.keyStore}}, etc.),
> these values will be used to override the ZK plaintext connection listener to
> create a TLS connection listener.
> If the {{zookeeper.properties}} file has an incomplete configuration (i.e.
> keystore password set but no keystore path), startup should fail with a clear
> error message indicating the missing properties and how to resolve.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7356) Enable TLS for embedded Zookeeper when NiFi has TLS enabled
[
https://issues.apache.org/jira/browse/NIFI-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269546#comment-17269546
]
ASF subversion and git services commented on NIFI-7356:
---
Commit 76648bdc0b2a077bed6073d30345e9d1af876920 in nifi's branch
refs/heads/main from Troy Melhase
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=76648bd ]
NIFI-7356 - Config TLS for embedded ZooKeeper when NiFi TLS enabled.
NIFI-7356 - Addresses PR feedback.
NIFI-7356 - Additional changes from PR feedback.
NIFI-7356 - Adding integration tests for ZooKeeperStateServer for TLS.
NIFI-7356 - TLS + Zookeeper now working with single and quorum. Needs code
cleanup, need to fix IT tests and docs.
NIFI-7356 - Fixed up tests and removed some irrelevant ones. Refactored some of
ZooKeeperStateServer. Tested successfully with a secure and insecure 3 node
NiFi + Quorum.
NIFI-7356 - Checkstyle fixes.
NIFI-7356 - Updated administration guide with embedded ZooKeeper TLS
configuration.
NIFI-7356 - Updated the way ZooKeeper TLS properties are set/mapped from NiFi
properties.
NIFI-7356 - Updated per review, using NiFiProperties keystore strings,
classname for ocnnection factory, adjusted TLS configuration checks in
NiFiProperties.
NIFI-7356 - Updated configuration validation logic and added tests.
NIFI-7356 - Codestyle check fixes.
NIFI-7356 - Updated some of the log messages.
NIFI-7356 - Updated as per code review.
NIFI-7356 - Fixed max port number.
NIFI-7356 - Updated admin guide and updated small code issues as per code
review.
Signed-off-by: Nathan Gough
This closes #4753.
> Enable TLS for embedded Zookeeper when NiFi has TLS enabled
> ---
>
> Key: NIFI-7356
> URL: https://issues.apache.org/jira/browse/NIFI-7356
> Project: Apache NiFi
> Issue Type: Sub-task
> Components: Configuration, Configuration Management, Security
>Reporter: Troy Melhase
>Assignee: Nathan Gough
>Priority: Major
> Fix For: 1.13.0
>
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> If embedded ZK has TLS properties in the {{zookeeper.properties}} file, these
> will be used. If however, this file does not populate those properties, and
> NiFi does have TLS properties configured ({{nifi.security.keyStore}}, etc.),
> these values will be used to override the ZK plaintext connection listener to
> create a TLS connection listener.
> If the {{zookeeper.properties}} file has an incomplete configuration (i.e.
> keystore password set but no keystore path), startup should fail with a clear
> error message indicating the missing properties and how to resolve.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7356) Enable TLS for embedded Zookeeper when NiFi has TLS enabled
[
https://issues.apache.org/jira/browse/NIFI-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269547#comment-17269547
]
ASF subversion and git services commented on NIFI-7356:
---
Commit 76648bdc0b2a077bed6073d30345e9d1af876920 in nifi's branch
refs/heads/main from Troy Melhase
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=76648bd ]
NIFI-7356 - Config TLS for embedded ZooKeeper when NiFi TLS enabled.
NIFI-7356 - Addresses PR feedback.
NIFI-7356 - Additional changes from PR feedback.
NIFI-7356 - Adding integration tests for ZooKeeperStateServer for TLS.
NIFI-7356 - TLS + Zookeeper now working with single and quorum. Needs code
cleanup, need to fix IT tests and docs.
NIFI-7356 - Fixed up tests and removed some irrelevant ones. Refactored some of
ZooKeeperStateServer. Tested successfully with a secure and insecure 3 node
NiFi + Quorum.
NIFI-7356 - Checkstyle fixes.
NIFI-7356 - Updated administration guide with embedded ZooKeeper TLS
configuration.
NIFI-7356 - Updated the way ZooKeeper TLS properties are set/mapped from NiFi
properties.
NIFI-7356 - Updated per review, using NiFiProperties keystore strings,
classname for ocnnection factory, adjusted TLS configuration checks in
NiFiProperties.
NIFI-7356 - Updated configuration validation logic and added tests.
NIFI-7356 - Codestyle check fixes.
NIFI-7356 - Updated some of the log messages.
NIFI-7356 - Updated as per code review.
NIFI-7356 - Fixed max port number.
NIFI-7356 - Updated admin guide and updated small code issues as per code
review.
Signed-off-by: Nathan Gough
This closes #4753.
> Enable TLS for embedded Zookeeper when NiFi has TLS enabled
> ---
>
> Key: NIFI-7356
> URL: https://issues.apache.org/jira/browse/NIFI-7356
> Project: Apache NiFi
> Issue Type: Sub-task
> Components: Configuration, Configuration Management, Security
>Reporter: Troy Melhase
>Assignee: Nathan Gough
>Priority: Major
> Fix For: 1.13.0
>
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> If embedded ZK has TLS properties in the {{zookeeper.properties}} file, these
> will be used. If however, this file does not populate those properties, and
> NiFi does have TLS properties configured ({{nifi.security.keyStore}}, etc.),
> these values will be used to override the ZK plaintext connection listener to
> create a TLS connection listener.
> If the {{zookeeper.properties}} file has an incomplete configuration (i.e.
> keystore password set but no keystore path), startup should fail with a clear
> error message indicating the missing properties and how to resolve.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7356) Enable TLS for embedded Zookeeper when NiFi has TLS enabled
[
https://issues.apache.org/jira/browse/NIFI-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269543#comment-17269543
]
ASF subversion and git services commented on NIFI-7356:
---
Commit 76648bdc0b2a077bed6073d30345e9d1af876920 in nifi's branch
refs/heads/main from Troy Melhase
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=76648bd ]
NIFI-7356 - Config TLS for embedded ZooKeeper when NiFi TLS enabled.
NIFI-7356 - Addresses PR feedback.
NIFI-7356 - Additional changes from PR feedback.
NIFI-7356 - Adding integration tests for ZooKeeperStateServer for TLS.
NIFI-7356 - TLS + Zookeeper now working with single and quorum. Needs code
cleanup, need to fix IT tests and docs.
NIFI-7356 - Fixed up tests and removed some irrelevant ones. Refactored some of
ZooKeeperStateServer. Tested successfully with a secure and insecure 3 node
NiFi + Quorum.
NIFI-7356 - Checkstyle fixes.
NIFI-7356 - Updated administration guide with embedded ZooKeeper TLS
configuration.
NIFI-7356 - Updated the way ZooKeeper TLS properties are set/mapped from NiFi
properties.
NIFI-7356 - Updated per review, using NiFiProperties keystore strings,
classname for ocnnection factory, adjusted TLS configuration checks in
NiFiProperties.
NIFI-7356 - Updated configuration validation logic and added tests.
NIFI-7356 - Codestyle check fixes.
NIFI-7356 - Updated some of the log messages.
NIFI-7356 - Updated as per code review.
NIFI-7356 - Fixed max port number.
NIFI-7356 - Updated admin guide and updated small code issues as per code
review.
Signed-off-by: Nathan Gough
This closes #4753.
> Enable TLS for embedded Zookeeper when NiFi has TLS enabled
> ---
>
> Key: NIFI-7356
> URL: https://issues.apache.org/jira/browse/NIFI-7356
> Project: Apache NiFi
> Issue Type: Sub-task
> Components: Configuration, Configuration Management, Security
>Reporter: Troy Melhase
>Assignee: Nathan Gough
>Priority: Major
> Fix For: 1.13.0
>
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> If embedded ZK has TLS properties in the {{zookeeper.properties}} file, these
> will be used. If however, this file does not populate those properties, and
> NiFi does have TLS properties configured ({{nifi.security.keyStore}}, etc.),
> these values will be used to override the ZK plaintext connection listener to
> create a TLS connection listener.
> If the {{zookeeper.properties}} file has an incomplete configuration (i.e.
> keystore password set but no keystore path), startup should fail with a clear
> error message indicating the missing properties and how to resolve.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7356) Enable TLS for embedded Zookeeper when NiFi has TLS enabled
[
https://issues.apache.org/jira/browse/NIFI-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269544#comment-17269544
]
ASF subversion and git services commented on NIFI-7356:
---
Commit 76648bdc0b2a077bed6073d30345e9d1af876920 in nifi's branch
refs/heads/main from Troy Melhase
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=76648bd ]
NIFI-7356 - Config TLS for embedded ZooKeeper when NiFi TLS enabled.
NIFI-7356 - Addresses PR feedback.
NIFI-7356 - Additional changes from PR feedback.
NIFI-7356 - Adding integration tests for ZooKeeperStateServer for TLS.
NIFI-7356 - TLS + Zookeeper now working with single and quorum. Needs code
cleanup, need to fix IT tests and docs.
NIFI-7356 - Fixed up tests and removed some irrelevant ones. Refactored some of
ZooKeeperStateServer. Tested successfully with a secure and insecure 3 node
NiFi + Quorum.
NIFI-7356 - Checkstyle fixes.
NIFI-7356 - Updated administration guide with embedded ZooKeeper TLS
configuration.
NIFI-7356 - Updated the way ZooKeeper TLS properties are set/mapped from NiFi
properties.
NIFI-7356 - Updated per review, using NiFiProperties keystore strings,
classname for ocnnection factory, adjusted TLS configuration checks in
NiFiProperties.
NIFI-7356 - Updated configuration validation logic and added tests.
NIFI-7356 - Codestyle check fixes.
NIFI-7356 - Updated some of the log messages.
NIFI-7356 - Updated as per code review.
NIFI-7356 - Fixed max port number.
NIFI-7356 - Updated admin guide and updated small code issues as per code
review.
Signed-off-by: Nathan Gough
This closes #4753.
> Enable TLS for embedded Zookeeper when NiFi has TLS enabled
> ---
>
> Key: NIFI-7356
> URL: https://issues.apache.org/jira/browse/NIFI-7356
> Project: Apache NiFi
> Issue Type: Sub-task
> Components: Configuration, Configuration Management, Security
>Reporter: Troy Melhase
>Assignee: Nathan Gough
>Priority: Major
> Fix For: 1.13.0
>
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> If embedded ZK has TLS properties in the {{zookeeper.properties}} file, these
> will be used. If however, this file does not populate those properties, and
> NiFi does have TLS properties configured ({{nifi.security.keyStore}}, etc.),
> these values will be used to override the ZK plaintext connection listener to
> create a TLS connection listener.
> If the {{zookeeper.properties}} file has an incomplete configuration (i.e.
> keystore password set but no keystore path), startup should fail with a clear
> error message indicating the missing properties and how to resolve.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7356) Enable TLS for embedded Zookeeper when NiFi has TLS enabled
[
https://issues.apache.org/jira/browse/NIFI-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269542#comment-17269542
]
ASF subversion and git services commented on NIFI-7356:
---
Commit 76648bdc0b2a077bed6073d30345e9d1af876920 in nifi's branch
refs/heads/main from Troy Melhase
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=76648bd ]
NIFI-7356 - Config TLS for embedded ZooKeeper when NiFi TLS enabled.
NIFI-7356 - Addresses PR feedback.
NIFI-7356 - Additional changes from PR feedback.
NIFI-7356 - Adding integration tests for ZooKeeperStateServer for TLS.
NIFI-7356 - TLS + Zookeeper now working with single and quorum. Needs code
cleanup, need to fix IT tests and docs.
NIFI-7356 - Fixed up tests and removed some irrelevant ones. Refactored some of
ZooKeeperStateServer. Tested successfully with a secure and insecure 3 node
NiFi + Quorum.
NIFI-7356 - Checkstyle fixes.
NIFI-7356 - Updated administration guide with embedded ZooKeeper TLS
configuration.
NIFI-7356 - Updated the way ZooKeeper TLS properties are set/mapped from NiFi
properties.
NIFI-7356 - Updated per review, using NiFiProperties keystore strings,
classname for ocnnection factory, adjusted TLS configuration checks in
NiFiProperties.
NIFI-7356 - Updated configuration validation logic and added tests.
NIFI-7356 - Codestyle check fixes.
NIFI-7356 - Updated some of the log messages.
NIFI-7356 - Updated as per code review.
NIFI-7356 - Fixed max port number.
NIFI-7356 - Updated admin guide and updated small code issues as per code
review.
Signed-off-by: Nathan Gough
This closes #4753.
> Enable TLS for embedded Zookeeper when NiFi has TLS enabled
> ---
>
> Key: NIFI-7356
> URL: https://issues.apache.org/jira/browse/NIFI-7356
> Project: Apache NiFi
> Issue Type: Sub-task
> Components: Configuration, Configuration Management, Security
>Reporter: Troy Melhase
>Assignee: Nathan Gough
>Priority: Major
> Fix For: 1.13.0
>
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> If embedded ZK has TLS properties in the {{zookeeper.properties}} file, these
> will be used. If however, this file does not populate those properties, and
> NiFi does have TLS properties configured ({{nifi.security.keyStore}}, etc.),
> these values will be used to override the ZK plaintext connection listener to
> create a TLS connection listener.
> If the {{zookeeper.properties}} file has an incomplete configuration (i.e.
> keystore password set but no keystore path), startup should fail with a clear
> error message indicating the missing properties and how to resolve.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7356) Enable TLS for embedded Zookeeper when NiFi has TLS enabled
[
https://issues.apache.org/jira/browse/NIFI-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269539#comment-17269539
]
ASF subversion and git services commented on NIFI-7356:
---
Commit 76648bdc0b2a077bed6073d30345e9d1af876920 in nifi's branch
refs/heads/main from Troy Melhase
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=76648bd ]
NIFI-7356 - Config TLS for embedded ZooKeeper when NiFi TLS enabled.
NIFI-7356 - Addresses PR feedback.
NIFI-7356 - Additional changes from PR feedback.
NIFI-7356 - Adding integration tests for ZooKeeperStateServer for TLS.
NIFI-7356 - TLS + Zookeeper now working with single and quorum. Needs code
cleanup, need to fix IT tests and docs.
NIFI-7356 - Fixed up tests and removed some irrelevant ones. Refactored some of
ZooKeeperStateServer. Tested successfully with a secure and insecure 3 node
NiFi + Quorum.
NIFI-7356 - Checkstyle fixes.
NIFI-7356 - Updated administration guide with embedded ZooKeeper TLS
configuration.
NIFI-7356 - Updated the way ZooKeeper TLS properties are set/mapped from NiFi
properties.
NIFI-7356 - Updated per review, using NiFiProperties keystore strings,
classname for ocnnection factory, adjusted TLS configuration checks in
NiFiProperties.
NIFI-7356 - Updated configuration validation logic and added tests.
NIFI-7356 - Codestyle check fixes.
NIFI-7356 - Updated some of the log messages.
NIFI-7356 - Updated as per code review.
NIFI-7356 - Fixed max port number.
NIFI-7356 - Updated admin guide and updated small code issues as per code
review.
Signed-off-by: Nathan Gough
This closes #4753.
> Enable TLS for embedded Zookeeper when NiFi has TLS enabled
> ---
>
> Key: NIFI-7356
> URL: https://issues.apache.org/jira/browse/NIFI-7356
> Project: Apache NiFi
> Issue Type: Sub-task
> Components: Configuration, Configuration Management, Security
>Reporter: Troy Melhase
>Assignee: Nathan Gough
>Priority: Major
> Fix For: 1.13.0
>
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> If embedded ZK has TLS properties in the {{zookeeper.properties}} file, these
> will be used. If however, this file does not populate those properties, and
> NiFi does have TLS properties configured ({{nifi.security.keyStore}}, etc.),
> these values will be used to override the ZK plaintext connection listener to
> create a TLS connection listener.
> If the {{zookeeper.properties}} file has an incomplete configuration (i.e.
> keystore password set but no keystore path), startup should fail with a clear
> error message indicating the missing properties and how to resolve.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7356) Enable TLS for embedded Zookeeper when NiFi has TLS enabled
[
https://issues.apache.org/jira/browse/NIFI-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269545#comment-17269545
]
ASF subversion and git services commented on NIFI-7356:
---
Commit 76648bdc0b2a077bed6073d30345e9d1af876920 in nifi's branch
refs/heads/main from Troy Melhase
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=76648bd ]
NIFI-7356 - Config TLS for embedded ZooKeeper when NiFi TLS enabled.
NIFI-7356 - Addresses PR feedback.
NIFI-7356 - Additional changes from PR feedback.
NIFI-7356 - Adding integration tests for ZooKeeperStateServer for TLS.
NIFI-7356 - TLS + Zookeeper now working with single and quorum. Needs code
cleanup, need to fix IT tests and docs.
NIFI-7356 - Fixed up tests and removed some irrelevant ones. Refactored some of
ZooKeeperStateServer. Tested successfully with a secure and insecure 3 node
NiFi + Quorum.
NIFI-7356 - Checkstyle fixes.
NIFI-7356 - Updated administration guide with embedded ZooKeeper TLS
configuration.
NIFI-7356 - Updated the way ZooKeeper TLS properties are set/mapped from NiFi
properties.
NIFI-7356 - Updated per review, using NiFiProperties keystore strings,
classname for ocnnection factory, adjusted TLS configuration checks in
NiFiProperties.
NIFI-7356 - Updated configuration validation logic and added tests.
NIFI-7356 - Codestyle check fixes.
NIFI-7356 - Updated some of the log messages.
NIFI-7356 - Updated as per code review.
NIFI-7356 - Fixed max port number.
NIFI-7356 - Updated admin guide and updated small code issues as per code
review.
Signed-off-by: Nathan Gough
This closes #4753.
> Enable TLS for embedded Zookeeper when NiFi has TLS enabled
> ---
>
> Key: NIFI-7356
> URL: https://issues.apache.org/jira/browse/NIFI-7356
> Project: Apache NiFi
> Issue Type: Sub-task
> Components: Configuration, Configuration Management, Security
>Reporter: Troy Melhase
>Assignee: Nathan Gough
>Priority: Major
> Fix For: 1.13.0
>
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> If embedded ZK has TLS properties in the {{zookeeper.properties}} file, these
> will be used. If however, this file does not populate those properties, and
> NiFi does have TLS properties configured ({{nifi.security.keyStore}}, etc.),
> these values will be used to override the ZK plaintext connection listener to
> create a TLS connection listener.
> If the {{zookeeper.properties}} file has an incomplete configuration (i.e.
> keystore password set but no keystore path), startup should fail with a clear
> error message indicating the missing properties and how to resolve.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7356) Enable TLS for embedded Zookeeper when NiFi has TLS enabled
[
https://issues.apache.org/jira/browse/NIFI-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269538#comment-17269538
]
ASF subversion and git services commented on NIFI-7356:
---
Commit 76648bdc0b2a077bed6073d30345e9d1af876920 in nifi's branch
refs/heads/main from Troy Melhase
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=76648bd ]
NIFI-7356 - Config TLS for embedded ZooKeeper when NiFi TLS enabled.
NIFI-7356 - Addresses PR feedback.
NIFI-7356 - Additional changes from PR feedback.
NIFI-7356 - Adding integration tests for ZooKeeperStateServer for TLS.
NIFI-7356 - TLS + Zookeeper now working with single and quorum. Needs code
cleanup, need to fix IT tests and docs.
NIFI-7356 - Fixed up tests and removed some irrelevant ones. Refactored some of
ZooKeeperStateServer. Tested successfully with a secure and insecure 3 node
NiFi + Quorum.
NIFI-7356 - Checkstyle fixes.
NIFI-7356 - Updated administration guide with embedded ZooKeeper TLS
configuration.
NIFI-7356 - Updated the way ZooKeeper TLS properties are set/mapped from NiFi
properties.
NIFI-7356 - Updated per review, using NiFiProperties keystore strings,
classname for ocnnection factory, adjusted TLS configuration checks in
NiFiProperties.
NIFI-7356 - Updated configuration validation logic and added tests.
NIFI-7356 - Codestyle check fixes.
NIFI-7356 - Updated some of the log messages.
NIFI-7356 - Updated as per code review.
NIFI-7356 - Fixed max port number.
NIFI-7356 - Updated admin guide and updated small code issues as per code
review.
Signed-off-by: Nathan Gough
This closes #4753.
> Enable TLS for embedded Zookeeper when NiFi has TLS enabled
> ---
>
> Key: NIFI-7356
> URL: https://issues.apache.org/jira/browse/NIFI-7356
> Project: Apache NiFi
> Issue Type: Sub-task
> Components: Configuration, Configuration Management, Security
>Reporter: Troy Melhase
>Assignee: Nathan Gough
>Priority: Major
> Fix For: 1.13.0
>
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> If embedded ZK has TLS properties in the {{zookeeper.properties}} file, these
> will be used. If however, this file does not populate those properties, and
> NiFi does have TLS properties configured ({{nifi.security.keyStore}}, etc.),
> these values will be used to override the ZK plaintext connection listener to
> create a TLS connection listener.
> If the {{zookeeper.properties}} file has an incomplete configuration (i.e.
> keystore password set but no keystore path), startup should fail with a clear
> error message indicating the missing properties and how to resolve.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7356) Enable TLS for embedded Zookeeper when NiFi has TLS enabled
[
https://issues.apache.org/jira/browse/NIFI-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269537#comment-17269537
]
ASF subversion and git services commented on NIFI-7356:
---
Commit 76648bdc0b2a077bed6073d30345e9d1af876920 in nifi's branch
refs/heads/main from Troy Melhase
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=76648bd ]
NIFI-7356 - Config TLS for embedded ZooKeeper when NiFi TLS enabled.
NIFI-7356 - Addresses PR feedback.
NIFI-7356 - Additional changes from PR feedback.
NIFI-7356 - Adding integration tests for ZooKeeperStateServer for TLS.
NIFI-7356 - TLS + Zookeeper now working with single and quorum. Needs code
cleanup, need to fix IT tests and docs.
NIFI-7356 - Fixed up tests and removed some irrelevant ones. Refactored some of
ZooKeeperStateServer. Tested successfully with a secure and insecure 3 node
NiFi + Quorum.
NIFI-7356 - Checkstyle fixes.
NIFI-7356 - Updated administration guide with embedded ZooKeeper TLS
configuration.
NIFI-7356 - Updated the way ZooKeeper TLS properties are set/mapped from NiFi
properties.
NIFI-7356 - Updated per review, using NiFiProperties keystore strings,
classname for ocnnection factory, adjusted TLS configuration checks in
NiFiProperties.
NIFI-7356 - Updated configuration validation logic and added tests.
NIFI-7356 - Codestyle check fixes.
NIFI-7356 - Updated some of the log messages.
NIFI-7356 - Updated as per code review.
NIFI-7356 - Fixed max port number.
NIFI-7356 - Updated admin guide and updated small code issues as per code
review.
Signed-off-by: Nathan Gough
This closes #4753.
> Enable TLS for embedded Zookeeper when NiFi has TLS enabled
> ---
>
> Key: NIFI-7356
> URL: https://issues.apache.org/jira/browse/NIFI-7356
> Project: Apache NiFi
> Issue Type: Sub-task
> Components: Configuration, Configuration Management, Security
>Reporter: Troy Melhase
>Assignee: Nathan Gough
>Priority: Major
> Fix For: 1.13.0
>
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> If embedded ZK has TLS properties in the {{zookeeper.properties}} file, these
> will be used. If however, this file does not populate those properties, and
> NiFi does have TLS properties configured ({{nifi.security.keyStore}}, etc.),
> these values will be used to override the ZK plaintext connection listener to
> create a TLS connection listener.
> If the {{zookeeper.properties}} file has an incomplete configuration (i.e.
> keystore password set but no keystore path), startup should fail with a clear
> error message indicating the missing properties and how to resolve.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7356) Enable TLS for embedded Zookeeper when NiFi has TLS enabled
[
https://issues.apache.org/jira/browse/NIFI-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269535#comment-17269535
]
ASF subversion and git services commented on NIFI-7356:
---
Commit 76648bdc0b2a077bed6073d30345e9d1af876920 in nifi's branch
refs/heads/main from Troy Melhase
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=76648bd ]
NIFI-7356 - Config TLS for embedded ZooKeeper when NiFi TLS enabled.
NIFI-7356 - Addresses PR feedback.
NIFI-7356 - Additional changes from PR feedback.
NIFI-7356 - Adding integration tests for ZooKeeperStateServer for TLS.
NIFI-7356 - TLS + Zookeeper now working with single and quorum. Needs code
cleanup, need to fix IT tests and docs.
NIFI-7356 - Fixed up tests and removed some irrelevant ones. Refactored some of
ZooKeeperStateServer. Tested successfully with a secure and insecure 3 node
NiFi + Quorum.
NIFI-7356 - Checkstyle fixes.
NIFI-7356 - Updated administration guide with embedded ZooKeeper TLS
configuration.
NIFI-7356 - Updated the way ZooKeeper TLS properties are set/mapped from NiFi
properties.
NIFI-7356 - Updated per review, using NiFiProperties keystore strings,
classname for ocnnection factory, adjusted TLS configuration checks in
NiFiProperties.
NIFI-7356 - Updated configuration validation logic and added tests.
NIFI-7356 - Codestyle check fixes.
NIFI-7356 - Updated some of the log messages.
NIFI-7356 - Updated as per code review.
NIFI-7356 - Fixed max port number.
NIFI-7356 - Updated admin guide and updated small code issues as per code
review.
Signed-off-by: Nathan Gough
This closes #4753.
> Enable TLS for embedded Zookeeper when NiFi has TLS enabled
> ---
>
> Key: NIFI-7356
> URL: https://issues.apache.org/jira/browse/NIFI-7356
> Project: Apache NiFi
> Issue Type: Sub-task
> Components: Configuration, Configuration Management, Security
>Reporter: Troy Melhase
>Assignee: Nathan Gough
>Priority: Major
> Fix For: 1.13.0
>
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> If embedded ZK has TLS properties in the {{zookeeper.properties}} file, these
> will be used. If however, this file does not populate those properties, and
> NiFi does have TLS properties configured ({{nifi.security.keyStore}}, etc.),
> these values will be used to override the ZK plaintext connection listener to
> create a TLS connection listener.
> If the {{zookeeper.properties}} file has an incomplete configuration (i.e.
> keystore password set but no keystore path), startup should fail with a clear
> error message indicating the missing properties and how to resolve.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7356) Enable TLS for embedded Zookeeper when NiFi has TLS enabled
[
https://issues.apache.org/jira/browse/NIFI-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269541#comment-17269541
]
ASF subversion and git services commented on NIFI-7356:
---
Commit 76648bdc0b2a077bed6073d30345e9d1af876920 in nifi's branch
refs/heads/main from Troy Melhase
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=76648bd ]
NIFI-7356 - Config TLS for embedded ZooKeeper when NiFi TLS enabled.
NIFI-7356 - Addresses PR feedback.
NIFI-7356 - Additional changes from PR feedback.
NIFI-7356 - Adding integration tests for ZooKeeperStateServer for TLS.
NIFI-7356 - TLS + Zookeeper now working with single and quorum. Needs code
cleanup, need to fix IT tests and docs.
NIFI-7356 - Fixed up tests and removed some irrelevant ones. Refactored some of
ZooKeeperStateServer. Tested successfully with a secure and insecure 3 node
NiFi + Quorum.
NIFI-7356 - Checkstyle fixes.
NIFI-7356 - Updated administration guide with embedded ZooKeeper TLS
configuration.
NIFI-7356 - Updated the way ZooKeeper TLS properties are set/mapped from NiFi
properties.
NIFI-7356 - Updated per review, using NiFiProperties keystore strings,
classname for ocnnection factory, adjusted TLS configuration checks in
NiFiProperties.
NIFI-7356 - Updated configuration validation logic and added tests.
NIFI-7356 - Codestyle check fixes.
NIFI-7356 - Updated some of the log messages.
NIFI-7356 - Updated as per code review.
NIFI-7356 - Fixed max port number.
NIFI-7356 - Updated admin guide and updated small code issues as per code
review.
Signed-off-by: Nathan Gough
This closes #4753.
> Enable TLS for embedded Zookeeper when NiFi has TLS enabled
> ---
>
> Key: NIFI-7356
> URL: https://issues.apache.org/jira/browse/NIFI-7356
> Project: Apache NiFi
> Issue Type: Sub-task
> Components: Configuration, Configuration Management, Security
>Reporter: Troy Melhase
>Assignee: Nathan Gough
>Priority: Major
> Fix For: 1.13.0
>
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> If embedded ZK has TLS properties in the {{zookeeper.properties}} file, these
> will be used. If however, this file does not populate those properties, and
> NiFi does have TLS properties configured ({{nifi.security.keyStore}}, etc.),
> these values will be used to override the ZK plaintext connection listener to
> create a TLS connection listener.
> If the {{zookeeper.properties}} file has an incomplete configuration (i.e.
> keystore password set but no keystore path), startup should fail with a clear
> error message indicating the missing properties and how to resolve.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NIFI-7356) Enable TLS for embedded Zookeeper when NiFi has TLS enabled
[
https://issues.apache.org/jira/browse/NIFI-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269536#comment-17269536
]
ASF subversion and git services commented on NIFI-7356:
---
Commit 76648bdc0b2a077bed6073d30345e9d1af876920 in nifi's branch
refs/heads/main from Troy Melhase
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=76648bd ]
NIFI-7356 - Config TLS for embedded ZooKeeper when NiFi TLS enabled.
NIFI-7356 - Addresses PR feedback.
NIFI-7356 - Additional changes from PR feedback.
NIFI-7356 - Adding integration tests for ZooKeeperStateServer for TLS.
NIFI-7356 - TLS + Zookeeper now working with single and quorum. Needs code
cleanup, need to fix IT tests and docs.
NIFI-7356 - Fixed up tests and removed some irrelevant ones. Refactored some of
ZooKeeperStateServer. Tested successfully with a secure and insecure 3 node
NiFi + Quorum.
NIFI-7356 - Checkstyle fixes.
NIFI-7356 - Updated administration guide with embedded ZooKeeper TLS
configuration.
NIFI-7356 - Updated the way ZooKeeper TLS properties are set/mapped from NiFi
properties.
NIFI-7356 - Updated per review, using NiFiProperties keystore strings,
classname for ocnnection factory, adjusted TLS configuration checks in
NiFiProperties.
NIFI-7356 - Updated configuration validation logic and added tests.
NIFI-7356 - Codestyle check fixes.
NIFI-7356 - Updated some of the log messages.
NIFI-7356 - Updated as per code review.
NIFI-7356 - Fixed max port number.
NIFI-7356 - Updated admin guide and updated small code issues as per code
review.
Signed-off-by: Nathan Gough
This closes #4753.
> Enable TLS for embedded Zookeeper when NiFi has TLS enabled
> ---
>
> Key: NIFI-7356
> URL: https://issues.apache.org/jira/browse/NIFI-7356
> Project: Apache NiFi
> Issue Type: Sub-task
> Components: Configuration, Configuration Management, Security
>Reporter: Troy Melhase
>Assignee: Nathan Gough
>Priority: Major
> Fix For: 1.13.0
>
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> If embedded ZK has TLS properties in the {{zookeeper.properties}} file, these
> will be used. If however, this file does not populate those properties, and
> NiFi does have TLS properties configured ({{nifi.security.keyStore}}, etc.),
> these values will be used to override the ZK plaintext connection listener to
> create a TLS connection listener.
> If the {{zookeeper.properties}} file has an incomplete configuration (i.e.
> keystore password set but no keystore path), startup should fail with a clear
> error message indicating the missing properties and how to resolve.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] thenatog closed pull request #4753: NIFI-7356 - Enable TLS for embedded Zookeeper when NiFi has TLS enabled
thenatog closed pull request #4753: URL: https://github.com/apache/nifi/pull/4753 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] thenatog commented on pull request #4753: NIFI-7356 - Enable TLS for embedded Zookeeper when NiFi has TLS enabled
thenatog commented on pull request #4753: URL: https://github.com/apache/nifi/pull/4753#issuecomment-764857170 Great, thanks Joey, I'll merge now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-5571) PutDatabaseRecord raise error while handle byte array.
[
https://issues.apache.org/jira/browse/NIFI-5571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-5571:
---
Summary: PutDatabaseRecord raise error while handle byte array. (was:
org.apache.nifi.processors.standard.PutDatabaseRecord raise error while handle
byte array.)
> PutDatabaseRecord raise error while handle byte array.
> --
>
> Key: NIFI-5571
> URL: https://issues.apache.org/jira/browse/NIFI-5571
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.7.1
>Reporter: musarona
>Priority: Major
>
> line no:656
> when record has some values,the code uses setObject for any value. that will
> raise error when the value is a Byte Array.
> {code:java}
> ps.setObject(i + 1, values[fieldIndexes.get(i)]); {code}
>
> to fix the problem ,use AvroTypeUtil.convertByteArray to convert the value to
> byte[] ,then setBytes will work OK.
>
> {code:java}
> if(field.getDataType().getFieldType() == RecordFieldType.ARRAY){
> // byte need to be converted to bytearray
> ByteBuffer byteBufferValue =
> AvroTypeUtil.convertByteArray((Object[])values[fieldIndexes.get(i)]);
> ps.setBytes(i + 1, byteBufferValue.array());
> }else {
> ps.setObject(i + 1, values[fieldIndexes.get(i)]);
> }
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] markap14 opened a new pull request #4775: NIFI-8163: When counting number of components, we traverse into all P…
markap14 opened a new pull request #4775:
URL: https://github.com/apache/nifi/pull/4775
…rocess Groups, but then call findAllRemoteProcessGroups, which is a
recursive call, instead of calling getRemoteProcessGroups(). This results in
counting the Process Groups many times. So fixed that.
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
_Enables X functionality; fixes bug NIFI-._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [ ] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NIFI-8163) When RPG is invalid, process groups show incorrect count for 'invalid'
Mark Payne created NIFI-8163: Summary: When RPG is invalid, process groups show incorrect count for 'invalid' Key: NIFI-8163 URL: https://issues.apache.org/jira/browse/NIFI-8163 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Mark Payne Assignee: Mark Payne Fix For: 1.13.0 When a Remote Process Group is created and is invalid (due to invalid hostname, for example), the parent Process Group should have an 'invalid' count of 1. Which it does. Its parent should also show an invalid count of 1, but it shows 2. And its parent shows 3 instead of 1, and so on. So if the RPG exists 6 levels deep, it will show an invalid count of 7 instead of 1. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7873) Conduct 1.13.0 release
[ https://issues.apache.org/jira/browse/NIFI-7873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Witt updated NIFI-7873: --- Summary: Conduct 1.13.0 release (was: Provide exactly same version of nars in distribution and in maven repo) > Conduct 1.13.0 release > -- > > Key: NIFI-7873 > URL: https://issues.apache.org/jira/browse/NIFI-7873 > Project: Apache NiFi > Issue Type: Wish >Affects Versions: 1.12.1 >Reporter: Andrei Lopukhov >Assignee: Joe Witt >Priority: Blocker > Fix For: 1.13.0 > > > Currently nar artifcats in NiFi distribution differs from same artifacts in > maven. > It looks like nars are packaged once again for making distribution assembly. > Example difference, MANIFEST.MF from nifi-avro-nar-1.12.1: > ||Maven||Archieve distribution|| > |Manifest-Version: 1.0 > Build-Branch: UNKNOWN > Build-Timestamp: 2020-09-23T10:17:41Z > Archiver-Version: Plexus Archiver > Built-By: jwitt > Nar-Id: nifi-avro-nar > Clone-During-Instance-Class-Loading: false > Nar-Version: 1.12.1 > Build-Tag: nifi-1.12.1-RC2 > Build-Revision: accfaa3 > Nar-Group: org.apache.nifi > Created-By: Apache Maven 3.6.3 > Build-Jdk: 1.8.0_265|Manifest-Version: 1.0 > Build-Timestamp: 2020-09-23T14:15:53Z > Clone-During-Instance-Class-Loading: false > Archiver-Version: Plexus Archiver > Built-By: jwitt > Nar-Version: 1.12.1 > Build-Tag: nifi-1.12.1-RC2 > Nar-Id: nifi-avro-nar > Nar-Group: org.apache.nifi > Created-By: Apache Maven 3.6.3 > Build-Jdk: 1.8.0_265| > So it is not possible to validate individual libraries from distribution > against libraries in maven cental. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7873) Conduct 1.13.0 release
[ https://issues.apache.org/jira/browse/NIFI-7873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Witt updated NIFI-7873: --- Priority: Trivial (was: Blocker) > Conduct 1.13.0 release > -- > > Key: NIFI-7873 > URL: https://issues.apache.org/jira/browse/NIFI-7873 > Project: Apache NiFi > Issue Type: Task >Affects Versions: 1.12.1 >Reporter: Andrei Lopukhov >Assignee: Joe Witt >Priority: Trivial > Fix For: 1.13.0 > > > Provide exactly same version of nars in distribution and in maven repo > Currently nar artifcats in NiFi distribution differs from same artifacts in > maven. > It looks like nars are packaged once again for making distribution assembly. > Example difference, MANIFEST.MF from nifi-avro-nar-1.12.1: > ||Maven||Archieve distribution|| > |Manifest-Version: 1.0 > Build-Branch: UNKNOWN > Build-Timestamp: 2020-09-23T10:17:41Z > Archiver-Version: Plexus Archiver > Built-By: jwitt > Nar-Id: nifi-avro-nar > Clone-During-Instance-Class-Loading: false > Nar-Version: 1.12.1 > Build-Tag: nifi-1.12.1-RC2 > Build-Revision: accfaa3 > Nar-Group: org.apache.nifi > Created-By: Apache Maven 3.6.3 > Build-Jdk: 1.8.0_265|Manifest-Version: 1.0 > Build-Timestamp: 2020-09-23T14:15:53Z > Clone-During-Instance-Class-Loading: false > Archiver-Version: Plexus Archiver > Built-By: jwitt > Nar-Version: 1.12.1 > Build-Tag: nifi-1.12.1-RC2 > Nar-Id: nifi-avro-nar > Nar-Group: org.apache.nifi > Created-By: Apache Maven 3.6.3 > Build-Jdk: 1.8.0_265| > So it is not possible to validate individual libraries from distribution > against libraries in maven cental. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8162) Docs for CSV Reader are outdating, stating that the first line is required to be a header line
[ https://issues.apache.org/jira/browse/NIFI-8162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269486#comment-17269486 ] ASF subversion and git services commented on NIFI-8162: --- Commit 6741317cc45d333cce81c3341bebd9643bde90cd in nifi's branch refs/heads/main from Mark Payne [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=6741317 ] NIFI-8162: Fixed outdated docs. Self-merging as this is a trivial docs fix. Signed-off-by: Mark Payne > Docs for CSV Reader are outdating, stating that the first line is required to > be a header line > -- > > Key: NIFI-8162 > URL: https://issues.apache.org/jira/browse/NIFI-8162 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.13.0 > > > The CSV Reader allows several different ways for specifying the schema. > However, the docs (@CapabilityDescription, as well as additionalDetails.html) > indicate that the first line must be a header line. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7873) Conduct 1.13.0 release
[ https://issues.apache.org/jira/browse/NIFI-7873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Witt updated NIFI-7873: --- Description: Provide exactly same version of nars in distribution and in maven repo Currently nar artifcats in NiFi distribution differs from same artifacts in maven. It looks like nars are packaged once again for making distribution assembly. Example difference, MANIFEST.MF from nifi-avro-nar-1.12.1: ||Maven||Archieve distribution|| |Manifest-Version: 1.0 Build-Branch: UNKNOWN Build-Timestamp: 2020-09-23T10:17:41Z Archiver-Version: Plexus Archiver Built-By: jwitt Nar-Id: nifi-avro-nar Clone-During-Instance-Class-Loading: false Nar-Version: 1.12.1 Build-Tag: nifi-1.12.1-RC2 Build-Revision: accfaa3 Nar-Group: org.apache.nifi Created-By: Apache Maven 3.6.3 Build-Jdk: 1.8.0_265|Manifest-Version: 1.0 Build-Timestamp: 2020-09-23T14:15:53Z Clone-During-Instance-Class-Loading: false Archiver-Version: Plexus Archiver Built-By: jwitt Nar-Version: 1.12.1 Build-Tag: nifi-1.12.1-RC2 Nar-Id: nifi-avro-nar Nar-Group: org.apache.nifi Created-By: Apache Maven 3.6.3 Build-Jdk: 1.8.0_265| So it is not possible to validate individual libraries from distribution against libraries in maven cental. was: Currently nar artifcats in NiFi distribution differs from same artifacts in maven. It looks like nars are packaged once again for making distribution assembly. Example difference, MANIFEST.MF from nifi-avro-nar-1.12.1: ||Maven||Archieve distribution|| |Manifest-Version: 1.0 Build-Branch: UNKNOWN Build-Timestamp: 2020-09-23T10:17:41Z Archiver-Version: Plexus Archiver Built-By: jwitt Nar-Id: nifi-avro-nar Clone-During-Instance-Class-Loading: false Nar-Version: 1.12.1 Build-Tag: nifi-1.12.1-RC2 Build-Revision: accfaa3 Nar-Group: org.apache.nifi Created-By: Apache Maven 3.6.3 Build-Jdk: 1.8.0_265|Manifest-Version: 1.0 Build-Timestamp: 2020-09-23T14:15:53Z Clone-During-Instance-Class-Loading: false Archiver-Version: Plexus Archiver Built-By: jwitt Nar-Version: 1.12.1 Build-Tag: nifi-1.12.1-RC2 Nar-Id: nifi-avro-nar Nar-Group: org.apache.nifi Created-By: Apache Maven 3.6.3 Build-Jdk: 1.8.0_265| So it is not possible to validate individual libraries from distribution against libraries in maven cental. > Conduct 1.13.0 release > -- > > Key: NIFI-7873 > URL: https://issues.apache.org/jira/browse/NIFI-7873 > Project: Apache NiFi > Issue Type: Wish >Affects Versions: 1.12.1 >Reporter: Andrei Lopukhov >Assignee: Joe Witt >Priority: Blocker > Fix For: 1.13.0 > > > Provide exactly same version of nars in distribution and in maven repo > Currently nar artifcats in NiFi distribution differs from same artifacts in > maven. > It looks like nars are packaged once again for making distribution assembly. > Example difference, MANIFEST.MF from nifi-avro-nar-1.12.1: > ||Maven||Archieve distribution|| > |Manifest-Version: 1.0 > Build-Branch: UNKNOWN > Build-Timestamp: 2020-09-23T10:17:41Z > Archiver-Version: Plexus Archiver > Built-By: jwitt > Nar-Id: nifi-avro-nar > Clone-During-Instance-Class-Loading: false > Nar-Version: 1.12.1 > Build-Tag: nifi-1.12.1-RC2 > Build-Revision: accfaa3 > Nar-Group: org.apache.nifi > Created-By: Apache Maven 3.6.3 > Build-Jdk: 1.8.0_265|Manifest-Version: 1.0 > Build-Timestamp: 2020-09-23T14:15:53Z > Clone-During-Instance-Class-Loading: false > Archiver-Version: Plexus Archiver > Built-By: jwitt > Nar-Version: 1.12.1 > Build-Tag: nifi-1.12.1-RC2 > Nar-Id: nifi-avro-nar > Nar-Group: org.apache.nifi > Created-By: Apache Maven 3.6.3 > Build-Jdk: 1.8.0_265| > So it is not possible to validate individual libraries from distribution > against libraries in maven cental. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7873) Conduct 1.13.0 release
[ https://issues.apache.org/jira/browse/NIFI-7873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Witt updated NIFI-7873: --- Issue Type: Task (was: Wish) > Conduct 1.13.0 release > -- > > Key: NIFI-7873 > URL: https://issues.apache.org/jira/browse/NIFI-7873 > Project: Apache NiFi > Issue Type: Task >Affects Versions: 1.12.1 >Reporter: Andrei Lopukhov >Assignee: Joe Witt >Priority: Blocker > Fix For: 1.13.0 > > > Provide exactly same version of nars in distribution and in maven repo > Currently nar artifcats in NiFi distribution differs from same artifacts in > maven. > It looks like nars are packaged once again for making distribution assembly. > Example difference, MANIFEST.MF from nifi-avro-nar-1.12.1: > ||Maven||Archieve distribution|| > |Manifest-Version: 1.0 > Build-Branch: UNKNOWN > Build-Timestamp: 2020-09-23T10:17:41Z > Archiver-Version: Plexus Archiver > Built-By: jwitt > Nar-Id: nifi-avro-nar > Clone-During-Instance-Class-Loading: false > Nar-Version: 1.12.1 > Build-Tag: nifi-1.12.1-RC2 > Build-Revision: accfaa3 > Nar-Group: org.apache.nifi > Created-By: Apache Maven 3.6.3 > Build-Jdk: 1.8.0_265|Manifest-Version: 1.0 > Build-Timestamp: 2020-09-23T14:15:53Z > Clone-During-Instance-Class-Loading: false > Archiver-Version: Plexus Archiver > Built-By: jwitt > Nar-Version: 1.12.1 > Build-Tag: nifi-1.12.1-RC2 > Nar-Id: nifi-avro-nar > Nar-Group: org.apache.nifi > Created-By: Apache Maven 3.6.3 > Build-Jdk: 1.8.0_265| > So it is not possible to validate individual libraries from distribution > against libraries in maven cental. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-8162) Docs for CSV Reader are outdating, stating that the first line is required to be a header line
[ https://issues.apache.org/jira/browse/NIFI-8162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne resolved NIFI-8162. -- Resolution: Fixed > Docs for CSV Reader are outdating, stating that the first line is required to > be a header line > -- > > Key: NIFI-8162 > URL: https://issues.apache.org/jira/browse/NIFI-8162 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.13.0 > > > The CSV Reader allows several different ways for specifying the schema. > However, the docs (@CapabilityDescription, as well as additionalDetails.html) > indicate that the first line must be a header line. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-8162) Docs for CSV Reader are outdating, stating that the first line is required to be a header line
Mark Payne created NIFI-8162: Summary: Docs for CSV Reader are outdating, stating that the first line is required to be a header line Key: NIFI-8162 URL: https://issues.apache.org/jira/browse/NIFI-8162 Project: Apache NiFi Issue Type: Bug Components: Extensions Reporter: Mark Payne Assignee: Mark Payne Fix For: 1.13.0 The CSV Reader allows several different ways for specifying the schema. However, the docs (@CapabilityDescription, as well as additionalDetails.html) indicate that the first line must be a header line. -- This message was sent by Atlassian Jira (v8.3.4#803005)
