[jira] [Assigned] (MINIFI-498) C2 Server Error - Too Many Open Files
[ https://issues.apache.org/jira/browse/MINIFI-498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri reassigned MINIFI-498: -- Assignee: Corinne Jukes (was: Aldrin Piri) > C2 Server Error - Too Many Open Files > - > > Key: MINIFI-498 > URL: https://issues.apache.org/jira/browse/MINIFI-498 > Project: Apache NiFi MiNiFi > Issue Type: Bug > Components: Command and Control >Affects Versions: 0.5.0 > Environment: OS: Amazon Linux 2 (4.14.77-86.82.amzn2.x86_64) > Java: 1.8.0_191-b12 > ulimits: > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) unlimited > pending signals (-i) 31769 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 5 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 1 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited >Reporter: Michael >Assignee: Corinne Jukes >Priority: Blocker > Fix For: 0.6.0 > > > After several days of running C2 the errors are being logged about having > "Too Many Open Files". At this point, C2 is not responsive to clients trying > to check for new configurations. > _33e79e: Too many open files_ > _Apr 29 14:33:22 ec2.internal c2.sh[20851]: Caused by: > java.nio.file.FileSystemException: > /opt/minifi-c2/./files/5233730D-AC58-4853-9C0E-273D0E_ > > Running _lsof_ shows an increasingly larger number of open files. Our MiNiFi > instances are checking for new files every few seconds (probably too short), > but it seems like maybe some file handles are not being cleaned up. > Restarting C2 clears up this condition and C2 becomes responsive again. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (MINIFI-498) C2 Server Error - Too Many Open Files
[ https://issues.apache.org/jira/browse/MINIFI-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17161297#comment-17161297 ] Aldrin Piri commented on MINIFI-498: [~Jukes] I was not able to get it easily reproduced at the time. I've added you to the contributors and will make you the assignee on this ticket. Thanks for digging in! > C2 Server Error - Too Many Open Files > - > > Key: MINIFI-498 > URL: https://issues.apache.org/jira/browse/MINIFI-498 > Project: Apache NiFi MiNiFi > Issue Type: Bug > Components: Command and Control >Affects Versions: 0.5.0 > Environment: OS: Amazon Linux 2 (4.14.77-86.82.amzn2.x86_64) > Java: 1.8.0_191-b12 > ulimits: > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) unlimited > pending signals (-i) 31769 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 5 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 1 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited >Reporter: Michael >Assignee: Aldrin Piri >Priority: Blocker > Fix For: 0.6.0 > > > After several days of running C2 the errors are being logged about having > "Too Many Open Files". At this point, C2 is not responsive to clients trying > to check for new configurations. > _33e79e: Too many open files_ > _Apr 29 14:33:22 ec2.internal c2.sh[20851]: Caused by: > java.nio.file.FileSystemException: > /opt/minifi-c2/./files/5233730D-AC58-4853-9C0E-273D0E_ > > Running _lsof_ shows an increasingly larger number of open files. Our MiNiFi > instances are checking for new files every few seconds (probably too short), > but it seems like maybe some file handles are not being cleaned up. > Restarting C2 clears up this condition and C2 becomes responsive again. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[nifi-minifi] branch master updated: MINIFI-245: Expose content repository implementation configuration
This is an automated email from the ASF dual-hosted git repository.
aldrin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi.git
The following commit(s) were added to refs/heads/master by this push:
new b016378 MINIFI-245: Expose content repository implementation
configuration
new b6ec05d Merge pull request #193 from mattyb149/MINIFI-245
b016378 is described below
commit b016378b577d644b06e405e3f7e9398046c0aed8
Author: Matthew Burgess
AuthorDate: Wed Jul 1 15:56:48 2020 -0400
MINIFI-245: Expose content repository implementation configuration
---
.../minifi/bootstrap/util/ConfigTransformer.java | 2 +-
.../bootstrap/util/ConfigTransformerTest.java | 27 -
.../src/test/resources/MINIFI-245/config.yml | 96 +++-
.../resources/MINIFI-245/nifi.properties.before| 121 +
.../commons/schema/ContentRepositorySchema.java| 10 ++
.../src/test/resources/1.5_RPG_Handling.yml| 1 +
.../src/test/resources/CsvToJson.yml | 1 +
.../test/resources/DecompressionCircularFlow.yml | 1 +
.../resources/InvokeHttpMiNiFiTemplateTest.yml | 1 +
.../resources/MINIFI-521_1.3_TemplateEncoding.yml | 1 +
.../src/test/resources/MultipleRelationships.yml | 1 +
.../src/test/resources/MultipleUriRPG.yml | 1 +
.../test/resources/NestedControllerServices.yml| 1 +
.../test/resources/NoTemplateEncodingVersion.yml | 1 +
.../ProcessGroupsAndRemoteProcessGroups.yml| 1 +
...eplaceTextExpressionLanguageCSVReformatting.yml | 1 +
.../test/resources/SimpleRPGToLogAttributes.yml| 1 +
.../src/test/resources/SimpleTailFileToRPG.yml | 1 +
.../src/test/resources/StressTestFramework.yml | 1 +
.../test/resources/StressTestFrameworkFunnel.yml | 1 +
.../resources/VersionedFlowSnapshot-Simple.yml | 1 +
21 files changed, 215 insertions(+), 57 deletions(-)
diff --git
a/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/util/ConfigTransformer.java
b/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/util/ConfigTransformer.java
index f42d6ce..3609426 100644
---
a/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/util/ConfigTransformer.java
+++
b/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/util/ConfigTransformer.java
@@ -223,7 +223,7 @@ public final class ConfigTransformer {
orderedProperties.setProperty("nifi.swap.out.period",
swapProperties.getOutPeriod());
orderedProperties.setProperty("nifi.swap.out.threads",
String.valueOf(swapProperties.getOutThreads()));
-
orderedProperties.setProperty("nifi.content.repository.implementation",
"org.apache.nifi.controller.repository.FileSystemRepository",
System.lineSeparator() + "# Content Repository");
+
orderedProperties.setProperty("nifi.content.repository.implementation",
contentRepoProperties.getContentRepository(), System.lineSeparator() + "#
Content Repository");
orderedProperties.setProperty("nifi.content.claim.max.appendable.size",
contentRepoProperties.getContentClaimMaxAppendableSize());
orderedProperties.setProperty("nifi.content.claim.max.flow.files",
String.valueOf(contentRepoProperties.getContentClaimMaxFlowFiles()));
orderedProperties.setProperty("nifi.content.repository.archive.max.retention.period",
contentRepoProperties.getContentRepoArchiveMaxRetentionPeriod());
diff --git
a/minifi-bootstrap/src/test/java/org/apache/nifi/minifi/bootstrap/util/ConfigTransformerTest.java
b/minifi-bootstrap/src/test/java/org/apache/nifi/minifi/bootstrap/util/ConfigTransformerTest.java
index 88bc318..ae7ffa7 100644
---
a/minifi-bootstrap/src/test/java/org/apache/nifi/minifi/bootstrap/util/ConfigTransformerTest.java
+++
b/minifi-bootstrap/src/test/java/org/apache/nifi/minifi/bootstrap/util/ConfigTransformerTest.java
@@ -68,8 +68,8 @@ import java.util.Set;
import java.util.stream.Collectors;
import java.util.zip.GZIPInputStream;
-import static
org.apache.nifi.minifi.bootstrap.RunMiNiFiTest.getTestBootstrapProperties;
import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
import static org.junit.Assert.assertNull;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
@@ -693,6 +693,31 @@ public class ConfigTransformerTest {
assertEquals(expected.entrySet().stream().collect(Collectors.toMap(Map.Entry::getKey, e -> nullToEmpty(e.getValue(, properties);
}
+@Test
+public void testContentRepoOverride() throws IOException,
ConfigurationChangeException, SchemaLoaderException {
+Properties pre216Properties = new Properties();
+try (InputStream pre216PropertiesStream =
ConfigTransformerTest.class.getClassL
[jira] [Resolved] (MINIFI-528) Add Documentation for Minifi Arm64 Docker Image
[ https://issues.apache.org/jira/browse/MINIFI-528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri resolved MINIFI-528. Fix Version/s: 0.6.0 Resolution: Fixed > Add Documentation for Minifi Arm64 Docker Image > --- > > Key: MINIFI-528 > URL: https://issues.apache.org/jira/browse/MINIFI-528 > Project: Apache NiFi MiNiFi > Issue Type: Improvement >Reporter: kaustav mukherjee >Priority: Major > Fix For: 0.6.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (MINIFI-528) Add Documentation for Minifi Arm64 Docker Image
[ https://issues.apache.org/jira/browse/MINIFI-528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri reassigned MINIFI-528: -- Assignee: kaustav mukherjee > Add Documentation for Minifi Arm64 Docker Image > --- > > Key: MINIFI-528 > URL: https://issues.apache.org/jira/browse/MINIFI-528 > Project: Apache NiFi MiNiFi > Issue Type: Improvement >Reporter: kaustav mukherjee >Assignee: kaustav mukherjee >Priority: Major > Fix For: 0.6.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[nifi-minifi] branch master updated: MINIFI-528 - Added the instructions for runing the minifi docker on Arm64 based systems like RaspberryPI 4
This is an automated email from the ASF dual-hosted git repository.
aldrin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi.git
The following commit(s) were added to refs/heads/master by this push:
new e37426b MINIFI-528 - Added the instructions for runing the minifi
docker on Arm64 based systems like RaspberryPI 4
e37426b is described below
commit e37426b5931355efad0c23023f1dac8488bebe0b
Author: MATRIX4284
AuthorDate: Tue Jun 2 10:03:04 2020 +
MINIFI-528 - Added the instructions for runing the minifi docker on Arm64
based systems like RaspberryPI 4
This closes #189.
Signed-off-by: Aldrin Piri
---
minifi-docker/dockerhub_ARM64/README.md | 66 +
1 file changed, 66 insertions(+)
diff --git a/minifi-docker/dockerhub_ARM64/README.md
b/minifi-docker/dockerhub_ARM64/README.md
new file mode 100644
index 000..ad54504
--- /dev/null
+++ b/minifi-docker/dockerhub_ARM64/README.md
@@ -0,0 +1,66 @@
+
+# Docker Image Quickstart
+
+
+## Building
+The Docker image can be built using the following command:
+
+docker build -t apache/nifi-minifi-arm64:latest .
+
+This build will result in an image tagged apache/nifi:latest
+
+# user @ puter in
~/Development/code/apache/nifi-minifi/minifi-docker/dockerhub_ARM64
+$ docker images
+REPOSITORY TAG IMAGE IDCREATED
SIZE
+apache/nifi-minifi-arm64 latest ddc9dce6019eAbout
a minute ago 734MB
+**Note**: The default version of NiFi specified by the Dockerfile is typically
that of one that is unreleased if working from source.
+To build an image for a prior released version, one can override the
`NIFI_VERSION` build-arg with the following command:
+
+docker build --build-arg=MINIFI_VERSION={Desired MiNiFi Version} -t
apache/nifi-minifi-arm64:latest .
+
+## Running a container
+
+### Supplying configuration to a container
+The primary means by which a MiNiFi instance is configured is via the
`config.yml` or the `bootstrap.conf`.
+
+This can be accomplished through:
+ * the use of volumes, and
+ * overlaying the base image
+
+ Using volumes to provide configuration
+The following example shows the usage of two volumes to provide both a
`config.yml` and a `bootstrap.conf` to the container instance. This makes use
of configuration files on the host and maps them to be used by the MiNiFi
instance. This is helpful in scenarios where a single image is used for a
variety of configurations.
+
+docker run -d \
+-v ~/minifi-conf/config.yml:/opt/minifi/minifi-0.5.0/conf/config.yml \
+-v
~/minifi-conf/bootstrap.conf:/opt/minifi/minifi-0.5.0/conf/bootstrap.conf \
+apache/nifi-minifi-arm64:latest
+
+ Using volumes to provide configuration
+Alternatively, it is possible to create a custom image inheriting from the
published image. Creating a `Dockerfile` extending from the Apache NiFi MiNiFi
base image allows users to overlay the configuration permanently into a newly
built and custom image. A simple example follows:
+
+FROM apache/nifi-minifi-arm64
+
+ADD config.yml /opt/minifi/minifi-0.5.0/conf/config.yml
+ADD bootstrap.conf /opt/minifi/minifi-0.5.0/conf/bootstrap.conf
+
+Building this `Dockerfile` will result in a custom image with the specified
configuration files incorporated into the new image. This is best for
applications where configuration is well defined and relatively static.
+
+For more information, please consult [Dockerfile Reference:
FROM](https://docs.docker.com/engine/reference/builder/#from)
+
+
+
[jira] [Assigned] (MINIFI-490) Generically support reporting tasks
[ https://issues.apache.org/jira/browse/MINIFI-490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri reassigned MINIFI-490: -- Assignee: Nghia Le > Generically support reporting tasks > --- > > Key: MINIFI-490 > URL: https://issues.apache.org/jira/browse/MINIFI-490 > Project: Apache NiFi MiNiFi > Issue Type: Improvement >Reporter: Aldrin Piri >Assignee: Nghia Le >Priority: Major > Fix For: 0.6.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (MINIFI-490) Generically support reporting tasks
[ https://issues.apache.org/jira/browse/MINIFI-490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134531#comment-17134531 ] Aldrin Piri commented on MINIFI-490: Commit accidentally used MINIFI-492 and history is on that issue. > Generically support reporting tasks > --- > > Key: MINIFI-490 > URL: https://issues.apache.org/jira/browse/MINIFI-490 > Project: Apache NiFi MiNiFi > Issue Type: Improvement >Reporter: Aldrin Piri >Assignee: Nghia Le >Priority: Major > Fix For: 0.6.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (MINIFI-490) Generically support reporting tasks
[ https://issues.apache.org/jira/browse/MINIFI-490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri resolved MINIFI-490. Fix Version/s: 0.6.0 Resolution: Fixed > Generically support reporting tasks > --- > > Key: MINIFI-490 > URL: https://issues.apache.org/jira/browse/MINIFI-490 > Project: Apache NiFi MiNiFi > Issue Type: Improvement >Reporter: Aldrin Piri >Priority: Major > Fix For: 0.6.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[nifi-minifi] branch master updated (2e0970b -> 5a95f60)
This is an automated email from the ASF dual-hosted git repository. aldrin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi.git. from 2e0970b MINIFI-521 Change logic around how updated RPG logic is applied to handle all subsequent versions new e44da56 MINIFI-492 - Generic reporting tasks new 5a95f60 MINIFI-521 Adding missing license headers for test resources. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../minifi/bootstrap/util/ConfigTransformer.java | 68 +++ .../bootstrap/util/ConfigTransformerTest.java | 26 +++ .../src/test/resources/config-reporting-task.yml | 218 + .../nifi/minifi/commons/schema/ConfigSchema.java | 15 ++ .../commons/schema/ProvenanceReportingSchema.java | 42 +++- .../minifi/commons/schema/ReportingSchema.java | 98 + .../commons/schema/common/CommonPropertyKeys.java | 1 + .../resources/MINIFI-521_1.3_TemplateEncoding.xml | 14 ++ .../resources/MINIFI-521_1.3_TemplateEncoding.yml | 14 ++ 9 files changed, 454 insertions(+), 42 deletions(-) create mode 100644 minifi-bootstrap/src/test/resources/config-reporting-task.yml create mode 100644 minifi-commons/minifi-commons-schema/src/main/java/org/apache/nifi/minifi/commons/schema/ReportingSchema.java
[nifi-minifi] 02/02: MINIFI-521 Adding missing license headers for test resources.
This is an automated email from the ASF dual-hosted git repository. aldrin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi.git commit 5a95f60088c0d868b2db6ef186a7c31cfa1b536d Author: Aldrin Piri AuthorDate: Fri Jun 12 16:45:06 2020 -0400 MINIFI-521 Adding missing license headers for test resources. Signed-off-by: Aldrin Piri --- .../src/test/resources/MINIFI-521_1.3_TemplateEncoding.xml | 14 ++ .../src/test/resources/MINIFI-521_1.3_TemplateEncoding.yml | 14 ++ 2 files changed, 28 insertions(+) diff --git a/minifi-toolkit/minifi-toolkit-configuration/src/test/resources/MINIFI-521_1.3_TemplateEncoding.xml b/minifi-toolkit/minifi-toolkit-configuration/src/test/resources/MINIFI-521_1.3_TemplateEncoding.xml index 1ae143b..38c41e8 100644 --- a/minifi-toolkit/minifi-toolkit-configuration/src/test/resources/MINIFI-521_1.3_TemplateEncoding.xml +++ b/minifi-toolkit/minifi-toolkit-configuration/src/test/resources/MINIFI-521_1.3_TemplateEncoding.xml @@ -1,4 +1,18 @@ + dacdd7b8-016e-1000-3bfe-7b025a1f4a65 diff --git a/minifi-toolkit/minifi-toolkit-configuration/src/test/resources/MINIFI-521_1.3_TemplateEncoding.yml b/minifi-toolkit/minifi-toolkit-configuration/src/test/resources/MINIFI-521_1.3_TemplateEncoding.yml index 379216c..366846e 100644 --- a/minifi-toolkit/minifi-toolkit-configuration/src/test/resources/MINIFI-521_1.3_TemplateEncoding.yml +++ b/minifi-toolkit/minifi-toolkit-configuration/src/test/resources/MINIFI-521_1.3_TemplateEncoding.yml @@ -1,3 +1,17 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the \"License\"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an \"AS IS\" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. MiNiFi Config Version: 3 Flow Controller: name: MiNiFi2
[nifi-minifi] 01/02: MINIFI-492 - Generic reporting tasks
This is an automated email from the ASF dual-hosted git repository.
aldrin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi.git
commit e44da568d8c099cda7b7fa4db6722d8b66aa9562
Author: Nghia Le
AuthorDate: Mon May 25 22:46:14 2020 +0200
MINIFI-492 - Generic reporting tasks
This closes #187.
---
.../minifi/bootstrap/util/ConfigTransformer.java | 68 +++
.../bootstrap/util/ConfigTransformerTest.java | 26 +++
.../src/test/resources/config-reporting-task.yml | 218 +
.../nifi/minifi/commons/schema/ConfigSchema.java | 15 ++
.../commons/schema/ProvenanceReportingSchema.java | 42 +++-
.../minifi/commons/schema/ReportingSchema.java | 98 +
.../commons/schema/common/CommonPropertyKeys.java | 1 +
7 files changed, 426 insertions(+), 42 deletions(-)
diff --git
a/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/util/ConfigTransformer.java
b/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/util/ConfigTransformer.java
index da4af51..677080c 100644
---
a/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/util/ConfigTransformer.java
+++
b/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/util/ConfigTransformer.java
@@ -38,6 +38,7 @@ import
org.apache.nifi.minifi.commons.schema.ProvenanceReportingSchema;
import org.apache.nifi.minifi.commons.schema.ProvenanceRepositorySchema;
import org.apache.nifi.minifi.commons.schema.RemotePortSchema;
import org.apache.nifi.minifi.commons.schema.RemoteProcessGroupSchema;
+import org.apache.nifi.minifi.commons.schema.ReportingSchema;
import org.apache.nifi.minifi.commons.schema.SecurityPropertiesSchema;
import org.apache.nifi.minifi.commons.schema.SensitivePropsSchema;
import org.apache.nifi.minifi.commons.schema.SwapSchema;
@@ -81,7 +82,6 @@ public final class ConfigTransformer {
// Underlying version of NIFI will be using
public static final String NIFI_VERSION = "1.8.0";
public static final String ROOT_GROUP = "Root-Group";
-public static final String DEFAULT_PROV_REPORTING_TASK_CLASS =
"org.apache.nifi.reporting.SiteToSiteProvenanceReportingTask";
public static final String NIFI_VERSION_KEY = "nifi.version";
public static final Logger logger =
LoggerFactory.getLogger(ConfigTransformer.class);
@@ -178,7 +178,7 @@ public final class ConfigTransformer {
ProvenanceRepositorySchema provenanceRepositorySchema =
configSchema.getProvenanceRepositorySchema();
OrderedProperties orderedProperties = new OrderedProperties();
-orderedProperties.setProperty(NIFI_VERSION_KEY, NIFI_VERSION,"#
Core Properties #" + System.lineSeparator());
+orderedProperties.setProperty(NIFI_VERSION_KEY, NIFI_VERSION, "#
Core Properties #" + System.lineSeparator());
orderedProperties.setProperty("nifi.flow.configuration.file",
"./conf/flow.xml.gz");
orderedProperties.setProperty("nifi.flow.configuration.archive.enabled",
"false");
orderedProperties.setProperty("nifi.flow.configuration.archive.dir",
"./conf/archive/");
@@ -288,7 +288,7 @@ public final class ConfigTransformer {
}
}
-protected static DOMSource createFlowXml(ConfigSchema configSchema) throws
IOException, ConfigurationChangeException, ConfigTransformerException{
+protected static DOMSource createFlowXml(ConfigSchema configSchema) throws
IOException, ConfigurationChangeException, ConfigTransformerException {
try {
// create a new, empty document
final DocumentBuilderFactory docFactory =
DocumentBuilderFactory.newInstance();
@@ -317,9 +317,10 @@ public final class ConfigTransformer {
addProcessGroup(doc, element, processGroupSchema, new
ParentGroupIdResolver(processGroupSchema));
SecurityPropertiesSchema securityProperties =
configSchema.getSecurityProperties();
-if (securityProperties.useSSL()) {
+boolean useSSL = securityProperties.useSSL();
+if (useSSL) {
Element controllerServicesNode =
doc.getElementById("controllerServices");
-if(controllerServicesNode == null) {
+if (controllerServicesNode == null) {
controllerServicesNode =
doc.createElement("controllerServices");
}
@@ -327,17 +328,27 @@ public final class ConfigTransformer {
addSSLControllerService(controllerServicesNode,
securityProperties);
}
+List reportingTasks =
configSchema.getReportingTasksSchema();
ProvenanceReportingSchema provenanceProperties =
configSchema.getProvenanceReportingProperties();
if (provenanceProperties
[nifi-minifi] branch master updated: MINIFI-521 Change logic around how updated RPG logic is applied to handle all subsequent versions
This is an automated email from the ASF dual-hosted git repository.
aldrin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi.git
The following commit(s) were added to refs/heads/master by this push:
new 2e0970b MINIFI-521 Change logic around how updated RPG logic is
applied to handle all subsequent versions
2e0970b is described below
commit 2e0970b8dd29797b1f8d5d710ca29d0d0486113d
Author: Aldrin Piri
AuthorDate: Thu Mar 19 09:38:16 2020 -0400
MINIFI-521 Change logic around how updated RPG logic is applied to handle
all subsequent versions
Add test for supplied sample file
This closes #188 and closes #185.
Signed-off-by: Aldrin Piri
---
.../configuration/dto/FlowSnippetDTOEnricher.java | 9 +-
.../toolkit/configuration/ConfigMainTest.java | 5 +
.../resources/MINIFI-521_1.3_TemplateEncoding.xml | 205 +
.../resources/MINIFI-521_1.3_TemplateEncoding.yml | 112 +++
4 files changed, 328 insertions(+), 3 deletions(-)
diff --git
a/minifi-toolkit/minifi-toolkit-configuration/src/main/java/org/apache/nifi/minifi/toolkit/configuration/dto/FlowSnippetDTOEnricher.java
b/minifi-toolkit/minifi-toolkit-configuration/src/main/java/org/apache/nifi/minifi/toolkit/configuration/dto/FlowSnippetDTOEnricher.java
index a197925..4795e12 100644
---
a/minifi-toolkit/minifi-toolkit-configuration/src/main/java/org/apache/nifi/minifi/toolkit/configuration/dto/FlowSnippetDTOEnricher.java
+++
b/minifi-toolkit/minifi-toolkit-configuration/src/main/java/org/apache/nifi/minifi/toolkit/configuration/dto/FlowSnippetDTOEnricher.java
@@ -65,14 +65,17 @@ public class FlowSnippetDTOEnricher {
*/
if (encodingVersion != null) {
switch (encodingVersion) {
-case "1.2":
+// Perform a no-op for the first two versions of templates
+case "1.0":
+case "1.1":
+break;
+// Starting in 1.2, the way in which RPGs are mapped has
changed
+default:
// Map all port DTOs to their respective targetIds
rpgIdToTargetIdMap.putAll(
Stream.concat(rpgInputPortDtos.stream(),
rpgOutputPortDtos.stream())
.collect(Collectors.toMap(RemoteProcessGroupPortDTO::getId,
RemoteProcessGroupPortDTO::getTargetId)));
break;
-default:
-break;
}
}
diff --git
a/minifi-toolkit/minifi-toolkit-configuration/src/test/java/org/apache/nifi/minifi/toolkit/configuration/ConfigMainTest.java
b/minifi-toolkit/minifi-toolkit-configuration/src/test/java/org/apache/nifi/minifi/toolkit/configuration/ConfigMainTest.java
index fd8b56b..66accea 100644
---
a/minifi-toolkit/minifi-toolkit-configuration/src/test/java/org/apache/nifi/minifi/toolkit/configuration/ConfigMainTest.java
+++
b/minifi-toolkit/minifi-toolkit-configuration/src/test/java/org/apache/nifi/minifi/toolkit/configuration/ConfigMainTest.java
@@ -179,6 +179,11 @@ public class ConfigMainTest {
}
@Test
+public void testTransformRoundTrip13TemplateEncoding() throws IOException,
JAXBException, SchemaLoaderException {
+transformRoundTrip("MINIFI-521_1.3_TemplateEncoding");
+}
+
+@Test
public void testTransformRoundTripDecompression() throws IOException,
JAXBException, SchemaLoaderException {
transformRoundTrip("DecompressionCircularFlow");
}
diff --git
a/minifi-toolkit/minifi-toolkit-configuration/src/test/resources/MINIFI-521_1.3_TemplateEncoding.xml
b/minifi-toolkit/minifi-toolkit-configuration/src/test/resources/MINIFI-521_1.3_TemplateEncoding.xml
new file mode 100644
index 000..1ae143b
--- /dev/null
+++
b/minifi-toolkit/minifi-toolkit-configuration/src/test/resources/MINIFI-521_1.3_TemplateEncoding.xml
@@ -0,0 +1,205 @@
+
+
+
+dacdd7b8-016e-1000-3bfe-7b025a1f4a65
+MiNiFi2
+
+
+e544a343-e2f0-3112--
+f73cd914-561d-3ade--
+
+0.0
+0.0
+
+
+
+
+10de7342-41ef-37a5--
+
e544a343-e2f0-3112--
+1
GB
+
1
+
+1e18bdb9-7df3-3fba--
+9b02eb45-3c2d-3fb4-91cf-880d5038c0cd
+REMOTE_INPUT_PORT
+
+0 sec
+1
+
DO_NOT_COMPRESS
+
+
LOAD_BALANCE_NOT_CO
[jira] [Assigned] (MINIFI-524) Adding ARM 64 V8 Docker for Minifi Java for Execution on Raspberry Pi
[ https://issues.apache.org/jira/browse/MINIFI-524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri reassigned MINIFI-524: -- Assignee: Aldrin Piri > Adding ARM 64 V8 Docker for Minifi Java for Execution on Raspberry Pi > - > > Key: MINIFI-524 > URL: https://issues.apache.org/jira/browse/MINIFI-524 > Project: Apache NiFi MiNiFi > Issue Type: New Feature > Components: Docker >Affects Versions: 0.6.0 > Environment: UBUNTU 19.10 ARM64_V8 for Raspberry Pi 4 >Reporter: kaustav mukherjee >Assignee: Aldrin Piri >Priority: Major > Labels: ARM, Docker > Fix For: 0.6.0 > > Attachments: dockerhub_ARM_64_V8.tar.gz > > Time Spent: 2h > Remaining Estimate: 0h > > Adding ARM_V8 Docker Image for minifi java so that it can execute on Rapberry > Pi 4 abd any other edge Devices That needs to the Minifi Flow -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (MINIFI-524) Adding ARM 64 V8 Docker for Minifi Java for Execution on Raspberry Pi
[ https://issues.apache.org/jira/browse/MINIFI-524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri reassigned MINIFI-524: -- Assignee: kaustav mukherjee (was: Aldrin Piri) > Adding ARM 64 V8 Docker for Minifi Java for Execution on Raspberry Pi > - > > Key: MINIFI-524 > URL: https://issues.apache.org/jira/browse/MINIFI-524 > Project: Apache NiFi MiNiFi > Issue Type: New Feature > Components: Docker >Affects Versions: 0.6.0 > Environment: UBUNTU 19.10 ARM64_V8 for Raspberry Pi 4 >Reporter: kaustav mukherjee >Assignee: kaustav mukherjee >Priority: Major > Labels: ARM, Docker > Fix For: 0.6.0 > > Attachments: dockerhub_ARM_64_V8.tar.gz > > Time Spent: 2h > Remaining Estimate: 0h > > Adding ARM_V8 Docker Image for minifi java so that it can execute on Rapberry > Pi 4 abd any other edge Devices That needs to the Minifi Flow -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (MINIFI-524) Adding ARM 64 V8 Docker for Minifi Java for Execution on Raspberry Pi
[ https://issues.apache.org/jira/browse/MINIFI-524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri resolved MINIFI-524. Resolution: Fixed > Adding ARM 64 V8 Docker for Minifi Java for Execution on Raspberry Pi > - > > Key: MINIFI-524 > URL: https://issues.apache.org/jira/browse/MINIFI-524 > Project: Apache NiFi MiNiFi > Issue Type: New Feature > Components: Docker >Affects Versions: 0.6.0 > Environment: UBUNTU 19.10 ARM64_V8 for Raspberry Pi 4 >Reporter: kaustav mukherjee >Priority: Major > Labels: ARM, Docker > Fix For: 0.6.0 > > Attachments: dockerhub_ARM_64_V8.tar.gz > > Time Spent: 2h > Remaining Estimate: 0h > > Adding ARM_V8 Docker Image for minifi java so that it can execute on Rapberry > Pi 4 abd any other edge Devices That needs to the Minifi Flow -- This message was sent by Atlassian Jira (v8.3.4#803005)
[nifi-minifi] branch master updated: MINIFI-524 Added ARM64 Docker for MiNiFi
This is an automated email from the ASF dual-hosted git repository. aldrin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi.git The following commit(s) were added to refs/heads/master by this push: new 70d2e35 MINIFI-524 Added ARM64 Docker for MiNiFi 70d2e35 is described below commit 70d2e35758090728eb852817fc4c5f044206c2e5 Author: Data Philosopher AuthorDate: Sun Feb 2 01:46:02 2020 + MINIFI-524 Added ARM64 Docker for MiNiFi This closes #184. Signed-off-by: Aldrin Piri --- minifi-docker/dockerhub_ARM64/DockerBuild.sh | 31 +++ minifi-docker/dockerhub_ARM64/DockerImage.txt | 16 minifi-docker/dockerhub_ARM64/DockerRun.sh| 19 ++ minifi-docker/dockerhub_ARM64/Dockerfile | 54 +++ minifi-docker/dockerhub_ARM64/sh/start.sh | 26 + 5 files changed, 146 insertions(+) diff --git a/minifi-docker/dockerhub_ARM64/DockerBuild.sh b/minifi-docker/dockerhub_ARM64/DockerBuild.sh new file mode 100755 index 000..69d0459 --- /dev/null +++ b/minifi-docker/dockerhub_ARM64/DockerBuild.sh @@ -0,0 +1,31 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +#!/bin/sh + +DOCKER_UID=1000 +if [ -n "$1" ]; then + DOCKER_UID="$1" +fi + +DOCKER_GID=50 +if [ -n "$2" ]; then + DOCKER_GID="$2" +fi + +DOCKER_IMAGE="$(egrep -v '(^#|^\s*$|^\s*\t*#)' DockerImage.txt)" +MINIFI_IMAGE_VERSION="$(echo $DOCKER_IMAGE | cut -d : -f 2)" +echo "Building MiNiFi Image: '$DOCKER_IMAGE' Version: $MINIFI_IMAGE_VERSION" +docker build --build-arg UID="$DOCKER_UID" --build-arg GID="$DOCKER_GID" --build-arg MINIFI_VERSION="$MINIFI_IMAGE_VERSION" -t $DOCKER_IMAGE . diff --git a/minifi-docker/dockerhub_ARM64/DockerImage.txt b/minifi-docker/dockerhub_ARM64/DockerImage.txt new file mode 100644 index 000..6d1141d --- /dev/null +++ b/minifi-docker/dockerhub_ARM64/DockerImage.txt @@ -0,0 +1,16 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +apache/nifi-minifi:0.5.0 diff --git a/minifi-docker/dockerhub_ARM64/DockerRun.sh b/minifi-docker/dockerhub_ARM64/DockerRun.sh new file mode 100755 index 000..1f5b3f0 --- /dev/null +++ b/minifi-docker/dockerhub_ARM64/DockerRun.sh @@ -0,0 +1,19 @@ +#!/bin/sh +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +DOCKER_IMAGE="$(egrep -v '(^#|^\s*$|^\s*\t*#)' DockerImage.txt)" +echo "Running Docker Image: $DOCKER_IMAGE" +docker run -it -d $DOCKER_IMAGE \ No newline at end of file diff --git a/minifi-docker/dockerhub_ARM64/Dockerfile b/minifi-docker/dockerhub_ARM64/Dockerfile
[jira] [Resolved] (MINIFI-526) Add anchors to MiNiFi admin guide
[ https://issues.apache.org/jira/browse/MINIFI-526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri resolved MINIFI-526. Resolution: Fixed > Add anchors to MiNiFi admin guide > - > > Key: MINIFI-526 > URL: https://issues.apache.org/jira/browse/MINIFI-526 > Project: Apache NiFi MiNiFi > Issue Type: Improvement > Components: Documentation >Reporter: Andrew M. Lim >Assignee: Andrew M. Lim >Priority: Minor > Fix For: 0.6.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Add HTML anchors to admin guide so different sections are easy to reference > and link to as needed. > [https://nifi.apache.org/minifi/system-admin-guide.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (MINIFI-526) Add anchors to MiNiFi admin guide
[ https://issues.apache.org/jira/browse/MINIFI-526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri updated MINIFI-526: --- Fix Version/s: 0.6.0 > Add anchors to MiNiFi admin guide > - > > Key: MINIFI-526 > URL: https://issues.apache.org/jira/browse/MINIFI-526 > Project: Apache NiFi MiNiFi > Issue Type: Improvement > Components: Documentation >Reporter: Andrew M. Lim >Assignee: Andrew M. Lim >Priority: Minor > Fix For: 0.6.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Add HTML anchors to admin guide so different sections are easy to reference > and link to as needed. > [https://nifi.apache.org/minifi/system-admin-guide.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[nifi-minifi] branch master updated: MINIFI-526 Added Table of Contents and HTML anchors to titles in Admin Guide
This is an automated email from the ASF dual-hosted git repository.
aldrin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi.git
The following commit(s) were added to refs/heads/master by this push:
new 959e6c7 MINIFI-526 Added Table of Contents and HTML anchors to titles
in Admin Guide
959e6c7 is described below
commit 959e6c7ca53b22e78e94e8c2f06e7ca421136983
Author: Andrew Lim
AuthorDate: Sun Mar 29 23:36:02 2020 -0400
MINIFI-526 Added Table of Contents and HTML anchors to titles in Admin Guide
This closes #186.
Signed-off-by: Aldrin Piri
---
.../src/main/markdown/System_Admin_Guide.md| 169 +
1 file changed, 109 insertions(+), 60 deletions(-)
diff --git a/minifi-docs/src/main/markdown/System_Admin_Guide.md
b/minifi-docs/src/main/markdown/System_Admin_Guide.md
index b2e4b18..c3d0ed3 100644
--- a/minifi-docs/src/main/markdown/System_Admin_Guide.md
+++ b/minifi-docs/src/main/markdown/System_Admin_Guide.md
@@ -23,7 +23,62 @@
This documentation is for MiNiFi ${project.version}.
-# Automatic Warm-Redeploy
+## Table of Contents
+
+- [Automatic Warm-Redeploy](#automatic-warm-redeploy)
+ * [FileChangeIngestor](#filechangeingestor)
+ * [RestChangeIngestor](#restchangeingestor)
+ * [PullHttpChangeIngestor](#pullhttpchangeingestor)
+- [Status Reporting and Querying](#status-reporting-and-querying)
+ * [FlowStatus Script Query](#flowstatus-script-query)
+ * [Periodic Status Reporters](#periodic-status-reporters)
++ [StatusLogger](#statuslogger)
+ * [FlowStatus Query Options](#flowstatus-query-options)
++ [Processors](#processors)
++ [Connections](#connections)
++ [Remote Process Groups](#remote-process-groups)
++ [Controller Services](#controller-services)
++ [Provenance Reporting](#provenance-reporting)
++ [Instance](#instance)
++ [System Diagnostics](#system-diagnostics)
++ [Example Query](#example-query)
+- [Config File](#config-file)
+ * [Versioning](#versioning)
++ [Version 1 -> Version 2 Changes](#version-1-version-2-changes)
++ [Version 2 -> Version 3 Changes](#version-2-version-3-changes)
+ * [Flow Controller](#flow-controller)
+ * [Core Properties](#core-properties)
+ * [FlowFile Repository](#flowfile-repository)
+ - [Swap Subsection](#swap-subsection)
+ * [Content Repository](#content-repository)
+ * [Provenance Repository](#provenance-repository)
+ * [Component Status Repository](#component-status-repository)
+ * [Security Properties](#security-properties)
+ - [Sensitive Properties Subsection](#sensitive-properties-subsection)
+ * [Processors](#processors-2)
+ - [Processor Properties](#processor-properties)
+ * [Controller Services](#controller-services-2)
+ - [Controller Service Properties](#controller-service-properties)
+ * [Process Groups](#process-groups)
+ * [Input Ports](#input-ports)
+ * [Output Ports](#output-ports)
+ * [Funnels](#funnels)
+ * [Connections](#connections-2)
+ * [Remote Process Groups](#remote-process-groups-2)
+ - [Input/Output Ports Subsection](#input-output-ports-subsection)
+ * [Provenance Reporting](#provenance-reporting-2)
+ * [NiFi Properties Overrides](#nifi-properties-overrides)
++ [Example NiFi Properties Overrides](#example-nifi-properties-overrides)
+- [Security Configuration](#security-configuration)
+ * [Security Properties in
bootstrap.conf](#security-properties-in-bootstrapconf)
+ * [Sensitive Property Configuration in
bootstrap.conf](#sensitive-property-configuration-in-bootstrapconf)
+- [Running as a Windows Service](#running-as-a-windows-service)
+- [Flow Overriding with bootstrap.conf](#flow-overriding-with-bootstrapconf)
+- [Example Config File](#example-config-file)
+- [Recommended Antivirus Exclusions](#recommended-antivirus-exclusions)
++ [MiNiFi Java](#minifi-java)
+
+# Automatic
Warm-Redeploy
When many MiNiFi agents running on the edge, it may not be possible to
manually stop, edit the *config.yml* and then restart every one every time
their configuration needs to change. The Config Change Coordinator and its
Ingestors were designed to automatically redeploy in response to a
configuration update.
@@ -49,7 +104,7 @@
nifi.minifi.notifier.ingestors=org.apache.nifi.minifi.bootstrap.configuration.in
Ingestor specific configuration is also necessary and done in the
bootstrap.conf as well. Specifics for each are detailed below.
-## FileChangeIngestor
+## FileChangeIngestor
class name:
`org.apache.nifi.minifi.bootstrap.configuration.ingestors.FileChangeIngestor`
@@ -65,7 +120,7 @@ Option | Description
`nifi.minifi.notifier.ingestors.file.polling.period.seconds` | How frequently
the file specified by `nifi.minifi.notifier.file.config.path` should be
evaluated for changes. If not set then a default polling period of 15 seconds
will be used.
`nifi.minifi.notifier.ingestors.file.differentiator` | Which differen
[nifi-registry] branch CEM-1.2.0.0 created (now df00b46)
This is an automated email from the ASF dual-hosted git repository. aldrin pushed a change to branch CEM-1.2.0.0 in repository https://gitbox.apache.org/repos/asf/nifi-registry.git. at df00b46 NIFIREG-390: Add .asf.yaml file to GitHub repo No new revisions were added by this update.
[jira] [Commented] (MINIFI-527) minifi agent crashing frequently with java.lang.IllegalAccessError: tried to access method org.apache.nifi.provenance.StandardProvenanceEventRecord.setEventId(J)V from
[
https://issues.apache.org/jira/browse/MINIFI-527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101981#comment-17101981
]
Aldrin Piri commented on MINIFI-527:
[~Sasidhar.Jasti] could you provide some more information regarding environment
and configuration you were running?
> minifi agent crashing frequently with java.lang.IllegalAccessError: tried to
> access method
> org.apache.nifi.provenance.StandardProvenanceEventRecord.setEventId(J)V from
> class org.apache.nifi.provenance.MiNiFiPersistentProvenanceRepositoryexception
> --
>
> Key: MINIFI-527
> URL: https://issues.apache.org/jira/browse/MINIFI-527
> Project: Apache NiFi MiNiFi
> Issue Type: Bug
> Components: Processing Configuration
>Affects Versions: 0.5.0
>Reporter: Sasidhar Jasti
>Priority: Major
>
> Minfi agents are crashing with below exceptions frequently.
> Kindly assist here.
> 2020-05-05 23:35:03,151 INFO [main] o.a.n.c.r.WriteAheadFlowFileRepository
> Initialized FlowFile Repository using 256 partitions
> 2020-05-05 23:35:03,208 WARN [main] o.a.n.p.PersistentProvenanceRepository
> Merged Journal File {} already exists; however, all partial journal files
> also exist so assuming that the merge did not finish. Repeating procedure in
> order to ensure consistency.
> 2020-05-05 23:35:05,820 INFO [main] o.a.n.p.lucene.SimpleIndexManager Index
> Writer for provenance_repository/index-1588703552000 has been returned to
> Index Manager and is no longer in use. Closing Index Writer
> 2020-05-05 23:35:05,824 ERROR [main] org.apache.nifi.minifi.MiNiFi Failure to
> launch MiNiFi due to java.lang.IllegalAccessError: tried to access method
> org.apache.nifi.provenance.StandardProvenanceEventRecord.setEventId(J)V from
> class org.apache.nifi.provenance.MiNiFiPersistentProvenanceRepository
> java.lang.IllegalAccessError: tried to access method
> org.apache.nifi.provenance.StandardProvenanceEventRecord.setEventId(J)V from
> class org.apache.nifi.provenance.MiNiFiPersistentProvenanceRepository
> at
> org.apache.nifi.provenance.MiNiFiPersistentProvenanceRepository.truncateAttributes(MiNiFiPersistentProvenanceRepository.java:1880)
> at
> org.apache.nifi.provenance.MiNiFiPersistentProvenanceRepository.mergeJournals(MiNiFiPersistentProvenanceRepository.java:1731)
> at
> org.apache.nifi.provenance.MiNiFiPersistentProvenanceRepository.recoverJournalFiles(MiNiFiPersistentProvenanceRepository.java:1427)
> at
> org.apache.nifi.provenance.MiNiFiPersistentProvenanceRepository.recover(MiNiFiPersistentProvenanceRepository.java:587)
> at
> org.apache.nifi.provenance.MiNiFiPersistentProvenanceRepository.initialize(MiNiFiPersistentProvenanceRepository.java:254)
> at org.apache.nifi.controller.FlowController.(FlowController.java:509)
> at
> org.apache.nifi.controller.FlowController.createStandaloneInstance(FlowController.java:422)
> at org.apache.nifi.minifi.MiNiFiServer.start(MiNiFiServer.java:102)
> at org.apache.nifi.minifi.MiNiFi.(MiNiFi.java:148)
> at org.apache.nifi.minifi.MiNiFi.main(MiNiFi.java:247)
> 2020-05-05 23:35:05,830 INFO [Thread-1] org.apache.nifi.minifi.MiNiFi
> Initiating shutdown of MiNiFi server...
> 2020-05-05 23:35:05,830 WARN [Thread-1] org.apache.nifi.minifi.MiNiFiServer
> Problem occurred ensuring flow controller or repository was properly
> terminated due to java.lang.NullPointerException
> 2020-05-05 23:35:05,833 ERROR [Thread-1] org.apache.nifi.BootstrapListener
> Failed to communicate with Bootstrap. Bootstrap may be unable to issue or
> receive commands from MiNiFi
> 2020-05-05 23:35:05,833 INFO [Thread-1] org.apache.nifi.minifi.MiNiFi MiNiFi
> server shutdown completed (nicely or otherwise).
>
> Thanks,
> Sasi
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[nifi] branch master updated: NIFI-7354: Allow analytics properties to be set via environment variables in docker scripts
This is an automated email from the ASF dual-hosted git repository.
aldrin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/master by this push:
new c9f48e3 NIFI-7354: Allow analytics properties to be set via
environment variables in docker scripts
c9f48e3 is described below
commit c9f48e378837a67a4d0c05e4d1b3bf5ade9cab18
Author: Matthew Burgess
AuthorDate: Mon Apr 13 09:02:05 2020 -0400
NIFI-7354: Allow analytics properties to be set via environment variables
in docker scripts
This closes #4203.
Signed-off-by: Aldrin Piri
---
nifi-docker/dockerhub/sh/start.sh | 8
nifi-docker/dockermaven/sh/start.sh | 8
2 files changed, 16 insertions(+)
diff --git a/nifi-docker/dockerhub/sh/start.sh
b/nifi-docker/dockerhub/sh/start.sh
index 1049e85..bd67c27 100755
--- a/nifi-docker/dockerhub/sh/start.sh
+++ b/nifi-docker/dockerhub/sh/start.sh
@@ -55,6 +55,14 @@ prop_replace 'nifi.cluster.flow.election.max.wait.time'
"${NIFI_ELECTION_MAX
prop_replace 'nifi.cluster.flow.election.max.candidates'
"${NIFI_ELECTION_MAX_CANDIDATES:-}"
prop_replace 'nifi.web.proxy.context.path'
"${NIFI_WEB_PROXY_CONTEXT_PATH:-}"
+# Set analytics properties
+prop_replace 'nifi.analytics.predict.enabled'
"${NIFI_ANALYTICS_PREDICT_ENABLED:-false}"
+prop_replace 'nifi.analytics.predict.interval'
"${NIFI_ANALYTICS_PREDICT_INTERVAL:-3 mins}"
+prop_replace 'nifi.analytics.query.interval'
"${NIFI_ANALYTICS_QUERY_INTERVAL:-5 mins}"
+prop_replace 'nifi.analytics.connection.model.implementation'
"${NIFI_ANALYTICS_MODEL_IMPLEMENTATION:-org.apache.nifi.controller.status.analytics.models.OrdinaryLeastSquares}"
+prop_replace 'nifi.analytics.connection.model.score.name'
"${NIFI_ANALYTICS_MODEL_SCORE_NAME:-rSquared}"
+prop_replace 'nifi.analytics.connection.model.score.threshold'
"${NIFI_ANALYTICS_MODEL_SCORE_THRESHOLD:-.90}"
+
. "${scripts_dir}/update_cluster_state_management.sh"
# Check if we are secured or unsecured
diff --git a/nifi-docker/dockermaven/sh/start.sh
b/nifi-docker/dockermaven/sh/start.sh
index 1049e85..bd67c27 100755
--- a/nifi-docker/dockermaven/sh/start.sh
+++ b/nifi-docker/dockermaven/sh/start.sh
@@ -55,6 +55,14 @@ prop_replace 'nifi.cluster.flow.election.max.wait.time'
"${NIFI_ELECTION_MAX
prop_replace 'nifi.cluster.flow.election.max.candidates'
"${NIFI_ELECTION_MAX_CANDIDATES:-}"
prop_replace 'nifi.web.proxy.context.path'
"${NIFI_WEB_PROXY_CONTEXT_PATH:-}"
+# Set analytics properties
+prop_replace 'nifi.analytics.predict.enabled'
"${NIFI_ANALYTICS_PREDICT_ENABLED:-false}"
+prop_replace 'nifi.analytics.predict.interval'
"${NIFI_ANALYTICS_PREDICT_INTERVAL:-3 mins}"
+prop_replace 'nifi.analytics.query.interval'
"${NIFI_ANALYTICS_QUERY_INTERVAL:-5 mins}"
+prop_replace 'nifi.analytics.connection.model.implementation'
"${NIFI_ANALYTICS_MODEL_IMPLEMENTATION:-org.apache.nifi.controller.status.analytics.models.OrdinaryLeastSquares}"
+prop_replace 'nifi.analytics.connection.model.score.name'
"${NIFI_ANALYTICS_MODEL_SCORE_NAME:-rSquared}"
+prop_replace 'nifi.analytics.connection.model.score.threshold'
"${NIFI_ANALYTICS_MODEL_SCORE_THRESHOLD:-.90}"
+
. "${scripts_dir}/update_cluster_state_management.sh"
# Check if we are secured or unsecured
svn commit: r38857 - /release/nifi/nifi-registry/nifi-registry-0.6.0/
Author: aldrin Date: Wed Apr 8 13:47:32 2020 New Revision: 38857 Log: NIFIREG-371 Updating shasums to reflect expected format Modified: release/nifi/nifi-registry/nifi-registry-0.6.0/nifi-registry-0.6.0-bin.tar.gz.sha256 release/nifi/nifi-registry/nifi-registry-0.6.0/nifi-registry-0.6.0-bin.tar.gz.sha512 release/nifi/nifi-registry/nifi-registry-0.6.0/nifi-registry-0.6.0-bin.zip.sha256 release/nifi/nifi-registry/nifi-registry-0.6.0/nifi-registry-0.6.0-bin.zip.sha512 release/nifi/nifi-registry/nifi-registry-0.6.0/nifi-registry-0.6.0-source-release.zip.sha256 release/nifi/nifi-registry/nifi-registry-0.6.0/nifi-registry-0.6.0-source-release.zip.sha512 Modified: release/nifi/nifi-registry/nifi-registry-0.6.0/nifi-registry-0.6.0-bin.tar.gz.sha256 == --- release/nifi/nifi-registry/nifi-registry-0.6.0/nifi-registry-0.6.0-bin.tar.gz.sha256 (original) +++ release/nifi/nifi-registry/nifi-registry-0.6.0/nifi-registry-0.6.0-bin.tar.gz.sha256 Wed Apr 8 13:47:32 2020 @@ -1 +1 @@ -e52d07e5689dbab5a04fa6afb4e89b316980daac781144c9651aa81a68c38068 ./nifi-registry-0.6.0-bin.tar.gz +e52d07e5689dbab5a04fa6afb4e89b316980daac781144c9651aa81a68c38068 Modified: release/nifi/nifi-registry/nifi-registry-0.6.0/nifi-registry-0.6.0-bin.tar.gz.sha512 == --- release/nifi/nifi-registry/nifi-registry-0.6.0/nifi-registry-0.6.0-bin.tar.gz.sha512 (original) +++ release/nifi/nifi-registry/nifi-registry-0.6.0/nifi-registry-0.6.0-bin.tar.gz.sha512 Wed Apr 8 13:47:32 2020 @@ -1 +1 @@ -6ade0a94e629ac8068b624b2b7e3e954e5722e6cbc60e9d42897140f81e10f24480ceb76e6f7bff5f1a46e1c74c58ea3e6771dc14a2ad7ce48a5532f9ac9daf9 ./nifi-registry-0.6.0-bin.tar.gz +6ade0a94e629ac8068b624b2b7e3e954e5722e6cbc60e9d42897140f81e10f24480ceb76e6f7bff5f1a46e1c74c58ea3e6771dc14a2ad7ce48a5532f9ac9daf9 Modified: release/nifi/nifi-registry/nifi-registry-0.6.0/nifi-registry-0.6.0-bin.zip.sha256 == --- release/nifi/nifi-registry/nifi-registry-0.6.0/nifi-registry-0.6.0-bin.zip.sha256 (original) +++ release/nifi/nifi-registry/nifi-registry-0.6.0/nifi-registry-0.6.0-bin.zip.sha256 Wed Apr 8 13:47:32 2020 @@ -1 +1 @@ -f026da8ccadd072d5a8dba51a40af7fd8c6af15e428f5d4aad34a9440f8d9405 ./nifi-registry-0.6.0-bin.zip +f026da8ccadd072d5a8dba51a40af7fd8c6af15e428f5d4aad34a9440f8d9405 Modified: release/nifi/nifi-registry/nifi-registry-0.6.0/nifi-registry-0.6.0-bin.zip.sha512 == --- release/nifi/nifi-registry/nifi-registry-0.6.0/nifi-registry-0.6.0-bin.zip.sha512 (original) +++ release/nifi/nifi-registry/nifi-registry-0.6.0/nifi-registry-0.6.0-bin.zip.sha512 Wed Apr 8 13:47:32 2020 @@ -1 +1 @@ -975489edff1d4ac72cc65677ada5be00e583bb708d3e79178c5a72ebec8e525c645fef79cb4a33b831da8ae3e6b329cad964646d59903b31c15145db2e9227a2 ./nifi-registry-0.6.0-bin.zip +975489edff1d4ac72cc65677ada5be00e583bb708d3e79178c5a72ebec8e525c645fef79cb4a33b831da8ae3e6b329cad964646d59903b31c15145db2e9227a2 Modified: release/nifi/nifi-registry/nifi-registry-0.6.0/nifi-registry-0.6.0-source-release.zip.sha256 == --- release/nifi/nifi-registry/nifi-registry-0.6.0/nifi-registry-0.6.0-source-release.zip.sha256 (original) +++ release/nifi/nifi-registry/nifi-registry-0.6.0/nifi-registry-0.6.0-source-release.zip.sha256 Wed Apr 8 13:47:32 2020 @@ -1 +1 @@ -0ed06ef762588be0154207932d2d1ebd1266c45aed299d0fb170cbeabd4c6e2b ./nifi-registry-0.6.0-source-release.zip +0ed06ef762588be0154207932d2d1ebd1266c45aed299d0fb170cbeabd4c6e2b Modified: release/nifi/nifi-registry/nifi-registry-0.6.0/nifi-registry-0.6.0-source-release.zip.sha512 == --- release/nifi/nifi-registry/nifi-registry-0.6.0/nifi-registry-0.6.0-source-release.zip.sha512 (original) +++ release/nifi/nifi-registry/nifi-registry-0.6.0/nifi-registry-0.6.0-source-release.zip.sha512 Wed Apr 8 13:47:32 2020 @@ -1 +1 @@ -aa1f03c20902bf9cd94574e45cbd57526763ab135421e9d1df489cac7c487cb9c6d20ba3f6a74d7ae37b97a7de71d12edbc1c9fabf6d6b29f29397164011e097 ./nifi-registry-0.6.0-source-release.zip +aa1f03c20902bf9cd94574e45cbd57526763ab135421e9d1df489cac7c487cb9c6d20ba3f6a74d7ae37b97a7de71d12edbc1c9fabf6d6b29f29397164011e097
svn commit: r38812 - /dev/nifi/nifi-registry/nifi-registry-0.6.0/ /release/nifi/nifi-registry/nifi-registry-0.6.0/
Author: aldrin Date: Mon Apr 6 00:53:32 2020 New Revision: 38812 Log: NiFi Registry 0.6.0 release Added: release/nifi/nifi-registry/nifi-registry-0.6.0/ - copied from r38811, dev/nifi/nifi-registry/nifi-registry-0.6.0/ Removed: dev/nifi/nifi-registry/nifi-registry-0.6.0/
[jira] [Commented] (MINIFI-521) minifi-toolkit-0.5.0: "Connection with id ... has invalid destination id" confused by targetId?
[
https://issues.apache.org/jira/browse/MINIFI-521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062592#comment-17062592
]
Aldrin Piri commented on MINIFI-521:
Confirmed the expected config.yml is not quite right. Used the buggy xml with
the changes applied and was able to successfully transmit. Will update the
files and get the associated tests enabled.
> minifi-toolkit-0.5.0: "Connection with id ... has invalid destination id"
> confused by targetId?
> ---
>
> Key: MINIFI-521
> URL: https://issues.apache.org/jira/browse/MINIFI-521
> Project: Apache NiFi MiNiFi
> Issue Type: Bug
> Components: Agent Configuration/Installation
>Affects Versions: 0.5.0
> Environment: version: '3'
> services:
> nifi:
> image: apache/nifi:1:10.0
> minifi:
> image: apache/nifi-minifi:0.5.0
> minifi-toolkit-0.5.0, Windows
> Reporter: Sebastian Napiorkowski
>Assignee: Aldrin Piri
>Priority: Minor
> Attachments: MiNiFi-buggy.xml, MiNiFi-expected.xml, config-buggy.yml,
> config-expected.yml, config.yml, minifi-flow.xml
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I followed this tutorial: [Getting started with
> MiNiFi|https://nifi.apache.org/minifi/getting-started.html]
> And the problem was that, while transforming the Template to the yml-file, it
> failed with: "Connection with id 10de7342-41ef-37a5-- has
> invalid destination id 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd"
> {code:java}
> minifi-toolkit-0.5.0\bin\config.bat transform MiNiFi-buggy.xml
> config-buggy.yml
> There are validation errors with the template, still outputting YAML but it
> will need to be edited.
> Connection with id 10de7342-41ef-37a5-- has invalid
> destination id 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd
> {code}
> By the help of brosander@github (Thank you, mate!) we discovered that the
> tool is confused by the targetId property:
> {code:xml}
>
>
>
>
>
> 1
> true
> true
>
> db4d2e20-016e-1000-d3c7-73a67e2b1d67
> 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd
> input_minifi
>
> dadd7f29-016e-1000-3c91-89495c813674
> false
> false
> false
>
>
> {code}
> just by removing the {{targetId}}, the toolkit runs through:
> {code:java}
> minifi-toolkit-0.5.0\bin\config.bat transform MiNiFi-expected.xml
> config-expected.yml
> No validation errors found in converted configuration.
> {code}
> The produced files differ:
> {code:java}
> diff config-buggy.yml config-expected.yml
> 101c101
> < - id: dadd7f29-016e-1000-3c91-89495c813674
> ---
> > - id: 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd
> {code}
> -I tested it and NiFi discovers MiNiFi, everything seems fine.-
> EDIT:
> I celebrated to early. Removing the {{targetId}} is *not* a working
> workaround. The easiest way to get around this issue is to downgrade NiFi
> from 1.10 to 1.9.2.
> Originally I thought this is a MiNiFi issue, but now I'm thinking that
> something changed in NiFi from 1.9.2 to 1.10.0 when exporting Templates. The
> ids are mixed up. This could be a much more critical bug, than I initially
> thought.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (MINIFI-521) minifi-toolkit-0.5.0: "Connection with id ... has invalid destination id" confused by targetId?
[
https://issues.apache.org/jira/browse/MINIFI-521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062566#comment-17062566
]
Aldrin Piri commented on MINIFI-521:
[~tjfleming] [~tugenhao]
I have a WIP branch (as I need to make some tests) available with PR
[https://github.com/apache/nifi-minifi/pull/185]
The file supplied for the expected yml is not correct, however, using the code
change I was able to confirm successful transform and transmission for a local
instance. This transform was from minifi-flow.xml -> config.yml
> minifi-toolkit-0.5.0: "Connection with id ... has invalid destination id"
> confused by targetId?
> ---
>
> Key: MINIFI-521
> URL: https://issues.apache.org/jira/browse/MINIFI-521
> Project: Apache NiFi MiNiFi
> Issue Type: Bug
> Components: Agent Configuration/Installation
>Affects Versions: 0.5.0
> Environment: version: '3'
> services:
> nifi:
> image: apache/nifi:1:10.0
> minifi:
> image: apache/nifi-minifi:0.5.0
> minifi-toolkit-0.5.0, Windows
> Reporter: Sebastian Napiorkowski
>Assignee: Aldrin Piri
>Priority: Minor
> Attachments: MiNiFi-buggy.xml, MiNiFi-expected.xml, config-buggy.yml,
> config-expected.yml, config.yml, minifi-flow.xml
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I followed this tutorial: [Getting started with
> MiNiFi|https://nifi.apache.org/minifi/getting-started.html]
> And the problem was that, while transforming the Template to the yml-file, it
> failed with: "Connection with id 10de7342-41ef-37a5-- has
> invalid destination id 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd"
> {code:java}
> minifi-toolkit-0.5.0\bin\config.bat transform MiNiFi-buggy.xml
> config-buggy.yml
> There are validation errors with the template, still outputting YAML but it
> will need to be edited.
> Connection with id 10de7342-41ef-37a5-- has invalid
> destination id 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd
> {code}
> By the help of brosander@github (Thank you, mate!) we discovered that the
> tool is confused by the targetId property:
> {code:xml}
>
>
>
>
>
> 1
> true
> true
>
> db4d2e20-016e-1000-d3c7-73a67e2b1d67
> 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd
> input_minifi
>
> dadd7f29-016e-1000-3c91-89495c813674
> false
> false
> false
>
>
> {code}
> just by removing the {{targetId}}, the toolkit runs through:
> {code:java}
> minifi-toolkit-0.5.0\bin\config.bat transform MiNiFi-expected.xml
> config-expected.yml
> No validation errors found in converted configuration.
> {code}
> The produced files differ:
> {code:java}
> diff config-buggy.yml config-expected.yml
> 101c101
> < - id: dadd7f29-016e-1000-3c91-89495c813674
> ---
> > - id: 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd
> {code}
> -I tested it and NiFi discovers MiNiFi, everything seems fine.-
> EDIT:
> I celebrated to early. Removing the {{targetId}} is *not* a working
> workaround. The easiest way to get around this issue is to downgrade NiFi
> from 1.10 to 1.9.2.
> Originally I thought this is a MiNiFi issue, but now I'm thinking that
> something changed in NiFi from 1.9.2 to 1.10.0 when exporting Templates. The
> ids are mixed up. This could be a much more critical bug, than I initially
> thought.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (MINIFI-521) minifi-toolkit-0.5.0: "Connection with id ... has invalid destination id" confused by targetId?
[
https://issues.apache.org/jira/browse/MINIFI-521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aldrin Piri updated MINIFI-521:
---
Attachment: config.yml
> minifi-toolkit-0.5.0: "Connection with id ... has invalid destination id"
> confused by targetId?
> ---
>
> Key: MINIFI-521
> URL: https://issues.apache.org/jira/browse/MINIFI-521
> Project: Apache NiFi MiNiFi
> Issue Type: Bug
> Components: Agent Configuration/Installation
>Affects Versions: 0.5.0
> Environment: version: '3'
> services:
> nifi:
> image: apache/nifi:1:10.0
> minifi:
> image: apache/nifi-minifi:0.5.0
> minifi-toolkit-0.5.0, Windows
> Reporter: Sebastian Napiorkowski
>Assignee: Aldrin Piri
>Priority: Minor
> Attachments: MiNiFi-buggy.xml, MiNiFi-expected.xml, config-buggy.yml,
> config-expected.yml, config.yml, minifi-flow.xml
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I followed this tutorial: [Getting started with
> MiNiFi|https://nifi.apache.org/minifi/getting-started.html]
> And the problem was that, while transforming the Template to the yml-file, it
> failed with: "Connection with id 10de7342-41ef-37a5-- has
> invalid destination id 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd"
> {code:java}
> minifi-toolkit-0.5.0\bin\config.bat transform MiNiFi-buggy.xml
> config-buggy.yml
> There are validation errors with the template, still outputting YAML but it
> will need to be edited.
> Connection with id 10de7342-41ef-37a5-- has invalid
> destination id 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd
> {code}
> By the help of brosander@github (Thank you, mate!) we discovered that the
> tool is confused by the targetId property:
> {code:xml}
>
>
>
>
>
> 1
> true
> true
>
> db4d2e20-016e-1000-d3c7-73a67e2b1d67
> 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd
> input_minifi
>
> dadd7f29-016e-1000-3c91-89495c813674
> false
> false
> false
>
>
> {code}
> just by removing the {{targetId}}, the toolkit runs through:
> {code:java}
> minifi-toolkit-0.5.0\bin\config.bat transform MiNiFi-expected.xml
> config-expected.yml
> No validation errors found in converted configuration.
> {code}
> The produced files differ:
> {code:java}
> diff config-buggy.yml config-expected.yml
> 101c101
> < - id: dadd7f29-016e-1000-3c91-89495c813674
> ---
> > - id: 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd
> {code}
> -I tested it and NiFi discovers MiNiFi, everything seems fine.-
> EDIT:
> I celebrated to early. Removing the {{targetId}} is *not* a working
> workaround. The easiest way to get around this issue is to downgrade NiFi
> from 1.10 to 1.9.2.
> Originally I thought this is a MiNiFi issue, but now I'm thinking that
> something changed in NiFi from 1.9.2 to 1.10.0 when exporting Templates. The
> ids are mixed up. This could be a much more critical bug, than I initially
> thought.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (MINIFI-521) minifi-toolkit-0.5.0: "Connection with id ... has invalid destination id" confused by targetId?
[
https://issues.apache.org/jira/browse/MINIFI-521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aldrin Piri updated MINIFI-521:
---
Attachment: minifi-flow.xml
> minifi-toolkit-0.5.0: "Connection with id ... has invalid destination id"
> confused by targetId?
> ---
>
> Key: MINIFI-521
> URL: https://issues.apache.org/jira/browse/MINIFI-521
> Project: Apache NiFi MiNiFi
> Issue Type: Bug
> Components: Agent Configuration/Installation
>Affects Versions: 0.5.0
> Environment: version: '3'
> services:
> nifi:
> image: apache/nifi:1:10.0
> minifi:
> image: apache/nifi-minifi:0.5.0
> minifi-toolkit-0.5.0, Windows
> Reporter: Sebastian Napiorkowski
>Assignee: Aldrin Piri
>Priority: Minor
> Attachments: MiNiFi-buggy.xml, MiNiFi-expected.xml, config-buggy.yml,
> config-expected.yml, minifi-flow.xml
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I followed this tutorial: [Getting started with
> MiNiFi|https://nifi.apache.org/minifi/getting-started.html]
> And the problem was that, while transforming the Template to the yml-file, it
> failed with: "Connection with id 10de7342-41ef-37a5-- has
> invalid destination id 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd"
> {code:java}
> minifi-toolkit-0.5.0\bin\config.bat transform MiNiFi-buggy.xml
> config-buggy.yml
> There are validation errors with the template, still outputting YAML but it
> will need to be edited.
> Connection with id 10de7342-41ef-37a5-- has invalid
> destination id 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd
> {code}
> By the help of brosander@github (Thank you, mate!) we discovered that the
> tool is confused by the targetId property:
> {code:xml}
>
>
>
>
>
> 1
> true
> true
>
> db4d2e20-016e-1000-d3c7-73a67e2b1d67
> 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd
> input_minifi
>
> dadd7f29-016e-1000-3c91-89495c813674
> false
> false
> false
>
>
> {code}
> just by removing the {{targetId}}, the toolkit runs through:
> {code:java}
> minifi-toolkit-0.5.0\bin\config.bat transform MiNiFi-expected.xml
> config-expected.yml
> No validation errors found in converted configuration.
> {code}
> The produced files differ:
> {code:java}
> diff config-buggy.yml config-expected.yml
> 101c101
> < - id: dadd7f29-016e-1000-3c91-89495c813674
> ---
> > - id: 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd
> {code}
> -I tested it and NiFi discovers MiNiFi, everything seems fine.-
> EDIT:
> I celebrated to early. Removing the {{targetId}} is *not* a working
> workaround. The easiest way to get around this issue is to downgrade NiFi
> from 1.10 to 1.9.2.
> Originally I thought this is a MiNiFi issue, but now I'm thinking that
> something changed in NiFi from 1.9.2 to 1.10.0 when exporting Templates. The
> ids are mixed up. This could be a much more critical bug, than I initially
> thought.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (MINIFI-521) minifi-toolkit-0.5.0: "Connection with id ... has invalid destination id" confused by targetId?
[
https://issues.apache.org/jira/browse/MINIFI-521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aldrin Piri reassigned MINIFI-521:
--
Assignee: Aldrin Piri
> minifi-toolkit-0.5.0: "Connection with id ... has invalid destination id"
> confused by targetId?
> ---
>
> Key: MINIFI-521
> URL: https://issues.apache.org/jira/browse/MINIFI-521
> Project: Apache NiFi MiNiFi
> Issue Type: Bug
> Components: Agent Configuration/Installation
>Affects Versions: 0.5.0
> Environment: version: '3'
> services:
> nifi:
> image: apache/nifi:1:10.0
> minifi:
> image: apache/nifi-minifi:0.5.0
> minifi-toolkit-0.5.0, Windows
> Reporter: Sebastian Napiorkowski
>Assignee: Aldrin Piri
>Priority: Minor
> Attachments: MiNiFi-buggy.xml, MiNiFi-expected.xml, config-buggy.yml,
> config-expected.yml
>
>
> I followed this tutorial: [Getting started with
> MiNiFi|https://nifi.apache.org/minifi/getting-started.html]
> And the problem was that, while transforming the Template to the yml-file, it
> failed with: "Connection with id 10de7342-41ef-37a5-- has
> invalid destination id 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd"
> {code:java}
> minifi-toolkit-0.5.0\bin\config.bat transform MiNiFi-buggy.xml
> config-buggy.yml
> There are validation errors with the template, still outputting YAML but it
> will need to be edited.
> Connection with id 10de7342-41ef-37a5-- has invalid
> destination id 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd
> {code}
> By the help of brosander@github (Thank you, mate!) we discovered that the
> tool is confused by the targetId property:
> {code:xml}
>
>
>
>
>
> 1
> true
> true
>
> db4d2e20-016e-1000-d3c7-73a67e2b1d67
> 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd
> input_minifi
>
> dadd7f29-016e-1000-3c91-89495c813674
> false
> false
> false
>
>
> {code}
> just by removing the {{targetId}}, the toolkit runs through:
> {code:java}
> minifi-toolkit-0.5.0\bin\config.bat transform MiNiFi-expected.xml
> config-expected.yml
> No validation errors found in converted configuration.
> {code}
> The produced files differ:
> {code:java}
> diff config-buggy.yml config-expected.yml
> 101c101
> < - id: dadd7f29-016e-1000-3c91-89495c813674
> ---
> > - id: 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd
> {code}
> -I tested it and NiFi discovers MiNiFi, everything seems fine.-
> EDIT:
> I celebrated to early. Removing the {{targetId}} is *not* a working
> workaround. The easiest way to get around this issue is to downgrade NiFi
> from 1.10 to 1.9.2.
> Originally I thought this is a MiNiFi issue, but now I'm thinking that
> something changed in NiFi from 1.9.2 to 1.10.0 when exporting Templates. The
> ids are mixed up. This could be a much more critical bug, than I initially
> thought.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
svn commit: r1874911 - in /nifi/site/trunk: developer-guide.html docs.html download.html minifi/download.html registry.html
Author: aldrin Date: Fri Mar 6 16:26:26 2020 New Revision: 1874911 URL: http://svn.apache.org/viewvc?rev=1874911=rev Log: Update all subprojects to make use of new downloads structure. Modified: nifi/site/trunk/developer-guide.html nifi/site/trunk/docs.html nifi/site/trunk/download.html nifi/site/trunk/minifi/download.html nifi/site/trunk/registry.html Modified: nifi/site/trunk/developer-guide.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/developer-guide.html?rev=1874911=1874910=1874911=diff == --- nifi/site/trunk/developer-guide.html (original) +++ nifi/site/trunk/developer-guide.html Fri Mar 6 16:26:26 2020 @@ -109,6 +109,7 @@ + https://www.apache.org;> Modified: nifi/site/trunk/docs.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/docs.html?rev=1874911=1874910=1874911=diff == --- nifi/site/trunk/docs.html (original) +++ nifi/site/trunk/docs.html Fri Mar 6 16:26:26 2020 @@ -109,6 +109,7 @@ + https://www.apache.org;> Modified: nifi/site/trunk/download.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/download.html?rev=1874911=1874910=1874911=diff == --- nifi/site/trunk/download.html (original) +++ nifi/site/trunk/download.html Fri Mar 6 16:26:26 2020 @@ -156,7 +156,7 @@ 1.10.0 Released November 4, 2019 -Note: The download links pull from downloads.apache.org. Apache limits how much can be downloaded from this site per day so please avoid automated/continuous downloads from the archives +Note: The download links pull from archive.apache.org. Apache limits how much can be downloaded from this site per day so please avoid automated/continuous downloads from the archives Sources: Modified: nifi/site/trunk/minifi/download.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/minifi/download.html?rev=1874911=1874910=1874911=diff == --- nifi/site/trunk/minifi/download.html (original) +++ nifi/site/trunk/minifi/download.html Fri Mar 6 16:26:26 2020 @@ -114,7 +114,7 @@ To verify the downloads please follow these https://www.apache.org/info/verification.html;>procedures -using these https://www.apache.org/dist/nifi/KEYS;>KEYS +using these https://downloads.apache.org/nifi/KEYS;>KEYS If a download is not found please allow up to 24 hours for the mirrors to sync. @@ -129,9 +129,9 @@ Sources: https://www.apache.org/dyn/closer.lua?path=/nifi/minifi/0.5.0/minifi-0.5.0-source-release.zip;>minifi-0.5.0-source-release.zip -( https://www.apache.org/dist/nifi/minifi/0.5.0/minifi-0.5.0-source-release.zip.asc;>asc, - https://www.apache.org/dist/nifi/minifi/0.5.0/minifi-0.5.0-source-release.zip.sha1;>sha1, - https://www.apache.org/dist/nifi/minifi/0.5.0/minifi-0.5.0-source-release.zip.sha256;>sha256 ) +( https://downloads.apache.org/nifi/minifi/0.5.0/minifi-0.5.0-source-release.zip.asc;>asc, + https://downloads.apache.org/nifi/minifi/0.5.0/minifi-0.5.0-source-release.zip.sha1;>sha1, + https://downloads.apache.org/nifi/minifi/0.5.0/minifi-0.5.0-source-release.zip.sha256;>sha256 ) @@ -139,14 +139,14 @@ Binaries https://www.apache.org/dyn/closer.lua?path=/nifi/minifi/0.5.0/minifi-0.5.0-bin.tar.gz;>minifi-0.5.0-bin.tar.gz -( https://www.apache.org/dist/nifi/minifi/0.5.0/minifi-0.5.0-bin.tar.gz.asc;>asc, - https://www.apache.org/dist/nifi/minifi/0.5.0/minifi-0.5.0-bin.tar.gz.sha1;>sha1, - https://www.apache.org/dist/nifi/minifi/0.5.0/minifi-0.5.0-bin.tar.gz.sha256;>sha256 ) +( https://downloads.apache.org/nifi/minifi/0.5.0/minifi-0.5.0-bin.tar.gz.asc;>asc, + https://downloads.apache.org/nifi/minifi/0.5.0/minifi-0.5.0-bin.tar.gz.sha1;>sha1, + https://downloads.apache.org/nifi/minifi/0.5.0/minifi-0.5.0-bin.tar.gz.sha256;>sha256 ) https://www.apache.org/dyn/closer.lua?path=/nif
[nifi-site] branch master updated: Update all subprojects to make use of new downloads structure. Reverting the archive.apache.org link for NiFi archived downloads caveat
This is an automated email from the ASF dual-hosted git repository.
aldrin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-site.git
The following commit(s) were added to refs/heads/master by this push:
new 3094c45 Update all subprojects to make use of new downloads
structure. Reverting the archive.apache.org link for NiFi archived downloads
caveat
3094c45 is described below
commit 3094c451d269212edb1a064b0b732c612d5199a4
Author: Aldrin Piri
AuthorDate: Fri Mar 6 10:49:20 2020 -0500
Update all subprojects to make use of new downloads structure. Reverting
the archive.apache.org link for NiFi archived downloads caveat
---
src/layouts/markdown.hbs | 2 +-
src/pages/html/developer-guide.hbs | 2 +-
src/pages/html/docs.hbs| 2 +-
src/pages/html/download.hbs| 2 +-
src/pages/html/minifi/download.hbs | 190 ++---
src/pages/html/registry.hbs| 18 ++--
6 files changed, 108 insertions(+), 108 deletions(-)
diff --git a/src/layouts/markdown.hbs b/src/layouts/markdown.hbs
index a68c5a8..eff2d6a 100644
--- a/src/layouts/markdown.hbs
+++ b/src/layouts/markdown.hbs
@@ -9,4 +9,4 @@
-{{> footer}}
\ No newline at end of file
+{{> footer}}
diff --git a/src/pages/html/developer-guide.hbs
b/src/pages/html/developer-guide.hbs
index 6a2e1c1..08c3f09 100644
--- a/src/pages/html/developer-guide.hbs
+++ b/src/pages/html/developer-guide.hbs
@@ -4,4 +4,4 @@ title: Apache NiFi Developer Guide
-
\ No newline at end of file
+
diff --git a/src/pages/html/docs.hbs b/src/pages/html/docs.hbs
index 3dd95aa..0314fd5 100644
--- a/src/pages/html/docs.hbs
+++ b/src/pages/html/docs.hbs
@@ -4,4 +4,4 @@ title: Apache NiFi Documentation
-
\ No newline at end of file
+
diff --git a/src/pages/html/download.hbs b/src/pages/html/download.hbs
index 772eeb7..03f16c1 100644
--- a/src/pages/html/download.hbs
+++ b/src/pages/html/download.hbs
@@ -52,7 +52,7 @@ title: Apache NiFi Downloads
1.10.0
Released November 4, 2019
-Note: The download links pull from
downloads.apache.org. Apache limits how much can be downloaded from this site
per day so please avoid automated/continuous downloads from the
archives
+Note: The download links pull from
archive.apache.org. Apache limits how much can be downloaded from this site
per day so please avoid automated/continuous downloads from the
archives
Sources:
diff --git a/src/pages/html/minifi/download.hbs
b/src/pages/html/minifi/download.hbs
index 22788fa..14ce7db 100644
--- a/src/pages/html/minifi/download.hbs
+++ b/src/pages/html/minifi/download.hbs
@@ -15,7 +15,7 @@ title: Apache NiFi - MiNiFi Downloads
To verify the downloads please follow these https://www.apache.org/info/verification.html;>procedures
-using these https://www.apache.org/dist/nifi/KEYS;>KEYS
+using these https://downloads.apache.org/nifi/KEYS;>KEYS
If a download is not found please allow up to 24 hours for the
mirrors to sync.
@@ -30,9 +30,9 @@ title: Apache NiFi - MiNiFi Downloads
Sources:
https://www.apache.org/dyn/closer.lua?path=/nifi/minifi/0.5.0/minifi-0.5.0-source-release.zip;>minifi-0.5.0-source-release.zip
-( https://www.apache.org/dist/nifi/minifi/0.5.0/minifi-0.5.0-source-release.zip.asc;>asc,
- https://www.apache.org/dist/nifi/minifi/0.5.0/minifi-0.5.0-source-release.zip.sha1;>sha1,
- https://www.apache.org/dist/nifi/minifi/0.5.0/minifi-0.5.0-source-release.zip.sha256;>sha256
)
+( https://downloads.apache.org/nifi/minifi/0.5.0/minifi-0.5.0-source-release.zip.asc;>asc,
+ https://downloads.apache.org/nifi/minifi/0.5.0/minifi-0.5.0-source-release.zip.sha1;>sha1,
+ https://downloads.apache.org/nifi/minifi/0.5.0/minifi-0.5.0-source-release.zip.sha256;>sha256
)
@@ -40,14 +40,14 @@ title: Apache NiFi - MiNiFi Downloads
Binaries
https://www.apache.org/dyn/closer.lua?path=/nifi/minifi/0.5.0/minifi-0.5.0-bin.tar.gz;>minifi-0.5.0-bin.tar.gz
-( https://www.apache.org/dist/nifi/minifi/0.5.0/minifi-0.5.0-bin.tar.gz.asc;>asc,
- https://www.apache.org/dist/nifi/minifi/0.5.0/minifi-0.5.0-bin.tar.gz.sha1;>sha1,
- https://www.apache.org/dist/nifi/minifi/0.5.0/minifi-0.5.0-bin.tar.gz.sha256;>sha256
)
+( https://dow
[jira] [Resolved] (MINIFI-522) Fixed Access denied to: http://jcenter.bintray.com which caused Failed to execute goal on project minifi-standard-nar
[
https://issues.apache.org/jira/browse/MINIFI-522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aldrin Piri resolved MINIFI-522.
Resolution: Fixed
> Fixed Access denied to: http://jcenter.bintray.com which caused Failed to
> execute goal on project minifi-standard-nar
> -
>
> Key: MINIFI-522
> URL: https://issues.apache.org/jira/browse/MINIFI-522
> Project: Apache NiFi MiNiFi
> Issue Type: Bug
>Reporter: sapinkumar amin
>Assignee: sapinkumar amin
>Priority: Blocker
> Fix For: 0.6.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {code:java}
> mvn clean install {code}
> throws below error
> {code:java}
> [INFO] minifi-update-attribute-nar SUCCESS [ 0.326
> s]
> [INFO] minifi-standard-nar FAILURE [ 1.198
> s]
> [INFO] minifi-ssl-context-service-nar . SKIPPED
> [INFO] minifi-docs SKIPPED
> [INFO] minifi-assembly SKIPPED
> [INFO] minifi-toolkit . SKIPPED
> [INFO] minifi-toolkit-configuration ... SKIPPED
> [INFO] minifi-toolkit-assembly SKIPPED
> [INFO] minifi-docker .. SKIPPED
> [INFO] minifi-c2 .. SKIPPED
> [INFO] minifi-c2-api .. SKIPPED
> [INFO] minifi-c2-cache SKIPPED
> [INFO] minifi-c2-cache-filesystem . SKIPPED
> [INFO] minifi-c2-cache-s3 . SKIPPED
> [INFO] minifi-c2-provider . SKIPPED
> [INFO] minifi-c2-provider-util SKIPPED
> [INFO] minifi-c2-provider-cache ... SKIPPED
> [INFO] minifi-c2-provider-delegating .. SKIPPED
> [INFO] minifi-c2-provider-nifi-rest ... SKIPPED
> [INFO] minifi-c2-service .. SKIPPED
> [INFO] minifi-c2-jetty SKIPPED
> [INFO] minifi-c2-assembly . SKIPPED
> [INFO] minifi-c2-docker ... SKIPPED
> [INFO] minifi-c2-integration-tests SKIPPED
> [INFO] minifi-integration-tests ... SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 29.209 s
> [INFO] Finished at: 2020-01-14T14:02:00-08:00
> [INFO]
>
> [ERROR] Failed to execute goal on project minifi-standard-nar: Could not
> resolve dependencies for project
> org.apache.nifi.minifi:minifi-standard-nar:nar:0.6.0-SNAPSHOT: Failed to
> collect dependencies at org.apache.nifi:nifi-standard-processors:jar:1.8.0 ->
> org.apache.nifi:nifi-syslog-utils:jar:1.8.0 ->
> com.github.palindromicity:simple-syslog-5424:jar:0.0.8: Failed to read
> artifact descriptor for
> com.github.palindromicity:simple-syslog-5424:jar:0.0.8: Could not transfer
> artifact com.github.palindromicity:simple-syslog-5424:pom:0.0.8 from/to
> jcenter (http://jcenter.bintray.com): Access denied to:
> http://jcenter.bintray.com/com/github/palindromicity/simple-syslog-5424/0.0.8/simple-syslog-5424-0.0.8.pom
> -> [Help 1]
> {code}
>
> From Jan13th : 2020 , access to jcenter.bintray.com is only allowed over
> https protocol (previously it use to support http and https).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (MINIFI-522) Fixed Access denied to: http://jcenter.bintray.com which caused Failed to execute goal on project minifi-standard-nar
[
https://issues.apache.org/jira/browse/MINIFI-522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aldrin Piri reassigned MINIFI-522:
--
Assignee: sapinkumar amin
> Fixed Access denied to: http://jcenter.bintray.com which caused Failed to
> execute goal on project minifi-standard-nar
> -
>
> Key: MINIFI-522
> URL: https://issues.apache.org/jira/browse/MINIFI-522
> Project: Apache NiFi MiNiFi
> Issue Type: Bug
>Reporter: sapinkumar amin
>Assignee: sapinkumar amin
>Priority: Blocker
> Fix For: 0.6.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {code:java}
> mvn clean install {code}
> throws below error
> {code:java}
> [INFO] minifi-update-attribute-nar SUCCESS [ 0.326
> s]
> [INFO] minifi-standard-nar FAILURE [ 1.198
> s]
> [INFO] minifi-ssl-context-service-nar . SKIPPED
> [INFO] minifi-docs SKIPPED
> [INFO] minifi-assembly SKIPPED
> [INFO] minifi-toolkit . SKIPPED
> [INFO] minifi-toolkit-configuration ... SKIPPED
> [INFO] minifi-toolkit-assembly SKIPPED
> [INFO] minifi-docker .. SKIPPED
> [INFO] minifi-c2 .. SKIPPED
> [INFO] minifi-c2-api .. SKIPPED
> [INFO] minifi-c2-cache SKIPPED
> [INFO] minifi-c2-cache-filesystem . SKIPPED
> [INFO] minifi-c2-cache-s3 . SKIPPED
> [INFO] minifi-c2-provider . SKIPPED
> [INFO] minifi-c2-provider-util SKIPPED
> [INFO] minifi-c2-provider-cache ... SKIPPED
> [INFO] minifi-c2-provider-delegating .. SKIPPED
> [INFO] minifi-c2-provider-nifi-rest ... SKIPPED
> [INFO] minifi-c2-service .. SKIPPED
> [INFO] minifi-c2-jetty SKIPPED
> [INFO] minifi-c2-assembly . SKIPPED
> [INFO] minifi-c2-docker ... SKIPPED
> [INFO] minifi-c2-integration-tests SKIPPED
> [INFO] minifi-integration-tests ... SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 29.209 s
> [INFO] Finished at: 2020-01-14T14:02:00-08:00
> [INFO]
>
> [ERROR] Failed to execute goal on project minifi-standard-nar: Could not
> resolve dependencies for project
> org.apache.nifi.minifi:minifi-standard-nar:nar:0.6.0-SNAPSHOT: Failed to
> collect dependencies at org.apache.nifi:nifi-standard-processors:jar:1.8.0 ->
> org.apache.nifi:nifi-syslog-utils:jar:1.8.0 ->
> com.github.palindromicity:simple-syslog-5424:jar:0.0.8: Failed to read
> artifact descriptor for
> com.github.palindromicity:simple-syslog-5424:jar:0.0.8: Could not transfer
> artifact com.github.palindromicity:simple-syslog-5424:pom:0.0.8 from/to
> jcenter (http://jcenter.bintray.com): Access denied to:
> http://jcenter.bintray.com/com/github/palindromicity/simple-syslog-5424/0.0.8/simple-syslog-5424-0.0.8.pom
> -> [Help 1]
> {code}
>
> From Jan13th : 2020 , access to jcenter.bintray.com is only allowed over
> https protocol (previously it use to support http and https).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (MINIFI-516) Override processor SSLs with parent SSL
[ https://issues.apache.org/jira/browse/MINIFI-516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri resolved MINIFI-516. Fix Version/s: 0.6.0 Resolution: Fixed > Override processor SSLs with parent SSL > --- > > Key: MINIFI-516 > URL: https://issues.apache.org/jira/browse/MINIFI-516 > Project: Apache NiFi MiNiFi > Issue Type: Improvement >Reporter: GCHQ NiFi >Assignee: r65535 >Priority: Minor > Fix For: 0.6.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > When using a ingestor to deploy flows to MiNiFi, processors using SSL > contexts don't start up as the sensitive properties are not passed along in > the NiFi templates. > A user should be able to set a boolean in the bootstrap.conf to tell MiNiFi > to ignore any custom SSL contexts in the config.yml, and repoint to the > parent SSL called "SSL-Context-Service" instead. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[nifi-minifi] branch master updated: MINIFI-516: Added bootstrap option to override processor SSLs with parent SSL context
This is an automated email from the ASF dual-hosted git repository.
aldrin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi.git
The following commit(s) were added to refs/heads/master by this push:
new 4e60ec2 MINIFI-516: Added bootstrap option to override processor SSLs
with parent SSL context
4e60ec2 is described below
commit 4e60ec2585cd6813ac5b0ba2d89fcae135e6b37d
Author: r65535
AuthorDate: Thu Oct 10 08:56:39 2019 +0100
MINIFI-516: Added bootstrap option to override processor SSLs with parent
SSL context
This closes #177.
Signed-off-by: Aldrin Piri
---
.../bootstrap/util/BootstrapTransformer.java | 11 +++
.../minifi/bootstrap/util/ConfigTransformer.java | 14 +++
.../bootstrap/util/ConfigTransformerTest.java | 57
.../src/test/resources/MINIFI-516}/bootstrap.conf | 11 ++-
.../src/test/resources/MINIFI-516/config.yml | 102 +
.../schema/common/BootstrapPropertyKeys.java | 2 +
.../src/main/markdown/System_Admin_Guide.md| 8 ++
.../src/main/resources/conf/bootstrap.conf | 3 +
8 files changed, 204 insertions(+), 4 deletions(-)
diff --git
a/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/util/BootstrapTransformer.java
b/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/util/BootstrapTransformer.java
index dc57990..eb861b7 100644
---
a/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/util/BootstrapTransformer.java
+++
b/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/util/BootstrapTransformer.java
@@ -31,6 +31,7 @@ import static
org.apache.nifi.minifi.commons.schema.common.BootstrapPropertyKeys
import static
org.apache.nifi.minifi.commons.schema.common.BootstrapPropertyKeys.BOOTSTRAP_PROVENANCE_REPORTING_KEYS;
import static
org.apache.nifi.minifi.commons.schema.common.BootstrapPropertyKeys.BOOTSTRAP_SECURITY_PROPERTY_KEYS;
import static
org.apache.nifi.minifi.commons.schema.common.BootstrapPropertyKeys.BOOTSTRAP_SENSITIVE_PROPERTY_KEYS;
+import static
org.apache.nifi.minifi.commons.schema.common.BootstrapPropertyKeys.USE_PARENT_SSL;
public class BootstrapTransformer {
@@ -88,4 +89,14 @@ public class BootstrapTransformer {
return provenanceReportingPropsOptional;
}
+public static boolean processorSSLOverride(final Properties
bootstrapProperties) {
+boolean shouldOverride = false;
+
+if (bootstrapProperties != null) {
+shouldOverride =
Boolean.parseBoolean(bootstrapProperties.getProperty(USE_PARENT_SSL));
+}
+
+return shouldOverride;
+}
+
}
diff --git
a/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/util/ConfigTransformer.java
b/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/util/ConfigTransformer.java
index 0e92ece..da4af51 100644
---
a/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/util/ConfigTransformer.java
+++
b/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/util/ConfigTransformer.java
@@ -45,6 +45,8 @@ import
org.apache.nifi.minifi.commons.schema.common.ConvertableSchema;
import org.apache.nifi.minifi.commons.schema.common.Schema;
import org.apache.nifi.minifi.commons.schema.common.StringUtil;
import org.apache.nifi.minifi.commons.schema.serialization.SchemaLoader;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import org.w3c.dom.DOMException;
import org.w3c.dom.Document;
import org.w3c.dom.Element;
@@ -82,6 +84,8 @@ public final class ConfigTransformer {
public static final String DEFAULT_PROV_REPORTING_TASK_CLASS =
"org.apache.nifi.reporting.SiteToSiteProvenanceReportingTask";
public static final String NIFI_VERSION_KEY = "nifi.version";
+public static final Logger logger =
LoggerFactory.getLogger(ConfigTransformer.class);
+
// Final util classes should have private constructor
private ConfigTransformer() {
}
@@ -96,9 +100,19 @@ public final class ConfigTransformer {
// See if we are providing defined properties from the filesystem
configurations and use those as the definitive values
if (securityProperties != null) {
configSchema.setSecurityProperties(securityProperties);
+logger.info("Bootstrap flow override: Replaced security
properties");
}
if (provenanceReportingProperties != null) {
configSchema.setProvenanceReportingProperties(provenanceReportingProperties);
+logger.info("Bootstrap flow override: Replaced provenance
reporting properties");
+}
+
+// Replace all processor SSL controller services with MiNiFi parent,
if bootstrap boolean is set to true
+if (BootstrapTransformer.processorSSLOverride(bootstrapProperties)) {
+for (ProcessorSchema processorConfig :
configSche
svn commit: r37569 - /dev/nifi/nifi-minifi-cpp/0.7.0/ /release/nifi/nifi-minifi-cpp/0.7.0/
Author: aldrin Date: Sat Jan 11 19:14:37 2020 New Revision: 37569 Log: MINIFICPP-1115 Added: release/nifi/nifi-minifi-cpp/0.7.0/ - copied from r37568, dev/nifi/nifi-minifi-cpp/0.7.0/ Removed: dev/nifi/nifi-minifi-cpp/0.7.0/
[nifi-minifi] branch master updated: Added /opt/minifi/minifi-current symlink
This is an automated email from the ASF dual-hosted git repository.
aldrin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi.git
The following commit(s) were added to refs/heads/master by this push:
new c1bde30 Added /opt/minifi/minifi-current symlink
c1bde30 is described below
commit c1bde30306475651aaebf160b0141fc5ee89d0a0
Author: r65535 <56300962+r65...@users.noreply.github.com>
AuthorDate: Fri Jan 3 12:16:46 2020 +
Added /opt/minifi/minifi-current symlink
This closes #181
Signed-off-by: Aldrin Piri
---
minifi-docker/dockerhub/Dockerfile | 9 +
minifi-docker/dockermaven/Dockerfile | 7 ---
2 files changed, 9 insertions(+), 7 deletions(-)
diff --git a/minifi-docker/dockerhub/Dockerfile
b/minifi-docker/dockerhub/Dockerfile
index 484a2c7..98ea27a 100644
--- a/minifi-docker/dockerhub/Dockerfile
+++ b/minifi-docker/dockerhub/Dockerfile
@@ -24,13 +24,13 @@ ARG GID=1000
ARG MINIFI_VERSION=0.5.0
ENV MINIFI_BASE_DIR /opt/minifi
-ENV MINIFI_HOME $MINIFI_BASE_DIR/minifi-$MINIFI_VERSION
+ENV MINIFI_HOME $MINIFI_BASE_DIR/minifi-current
ENV MINIFI_BINARY_URL
https://archive.apache.org/dist/nifi/minifi/$MINIFI_VERSION/minifi-$MINIFI_VERSION-bin.tar.gz
# Setup MiNiFi user
RUN addgroup -g $GID minifi || groupmod -n minifi `getent group $GID | cut -d:
-f1`
RUN adduser -S -H -G minifi minifi
-RUN mkdir -p $MINIFI_HOME
+RUN mkdir -p $MINIFI_BASE_DIR
RUN apk --no-cache add curl
@@ -40,9 +40,10 @@ ADD sh/ ${MINIFI_BASE_DIR}/scripts/
RUN curl -fSL $MINIFI_BINARY_URL -o
$MINIFI_BASE_DIR/minifi-$MINIFI_VERSION-bin.tar.gz \
&& echo "$(curl $MINIFI_BINARY_URL.sha256)
*$MINIFI_BASE_DIR/minifi-$MINIFI_VERSION-bin.tar.gz" | sha256sum -c - \
&& tar -xvzf $MINIFI_BASE_DIR/minifi-$MINIFI_VERSION-bin.tar.gz -C
$MINIFI_BASE_DIR \
- && rm $MINIFI_BASE_DIR/minifi-$MINIFI_VERSION-bin.tar.gz
+ && rm $MINIFI_BASE_DIR/minifi-$MINIFI_VERSION-bin.tar.gz \
+ && ln -s $MINIFI_BASE_DIR/minifi-$MINIFI_VERSION $MINIFI_HOME
-RUN chown -R minifi:minifi $MINIFI_HOME
+RUN chown -R -L minifi:minifi $MINIFI_HOME
USER minifi
diff --git a/minifi-docker/dockermaven/Dockerfile
b/minifi-docker/dockermaven/Dockerfile
index 21eea54..9f0491c 100644
--- a/minifi-docker/dockermaven/Dockerfile
+++ b/minifi-docker/dockermaven/Dockerfile
@@ -26,15 +26,16 @@ ARG MINIFI_VERSION
ARG MINIFI_BINARY
ENV MINIFI_BASE_DIR /opt/minifi
-ENV MINIFI_HOME $MINIFI_BASE_DIR/minifi-$MINIFI_VERSION
+ENV MINIFI_HOME $MINIFI_BASE_DIR/minifi-current
# Setup MiNiFi user
RUN addgroup -g $GID minifi || groupmod -n minifi `getent group $GID | cut -d:
-f1`
RUN adduser -S -H -G minifi minifi
-RUN mkdir -p $MINIFI_HOME
+RUN mkdir -p $MINIFI_BASE_DIR
ADD $MINIFI_BINARY $MINIFI_BASE_DIR
-RUN chown -R minifi:minifi $MINIFI_HOME
+RUN ln -s $MINIFI_BASE_DIR/minifi-$MINIFI_VERSION $MINIFI_HOME
+RUN chown -R -L minifi:minifi $MINIFI_HOME
USER minifi
svn commit: r37541 - /dev/nifi/KEYS
Author: aldrin Date: Thu Jan 9 16:47:01 2020 New Revision: 37541 Log: Adding Arpad Boda and Dániel Bakai to KEYS file. Modified: dev/nifi/KEYS Modified: dev/nifi/KEYS == --- dev/nifi/KEYS (original) +++ dev/nifi/KEYS Thu Jan 9 16:47:01 2020 @@ -1382,3 +1382,74 @@ TPZj7cC30y/MUdi0zJI= =uY8D -END PGP PUBLIC KEY BLOCK- +2048 2019-10-24 [SC] [expires: 2021-10-23] + EE78BB8A82A7851477AAAD0A390C1B5ADE978835 +uid [ultimate] Arpad Boda +sig 3390C1B5ADE978835 2019-10-24 Arpad Boda +sub rsa2048 2019-10-24 [E] [expires: 2021-10-23] +sig 390C1B5ADE978835 2019-10-24 Arpad Boda + +-BEGIN PGP PUBLIC KEY BLOCK- + +mQENBF2xedcBCADhvZxpFnZ37r3LuC4ZBNJu5OX4ORkXSWOgo7JwGb0pDGWrQlcq +ZU57fjeGvyciLPF7ICAOKDGCF2YYrRfCG2IhI4S6B+pYpIs9H9XnfoVLUOYgCQAl +VsugMlMXAnLb5MNNFEV6XLQooqvFkTtsbRcEsES6DogBeYlMoF2koJkY5W/Evpcs +96nwVH9cTtdahOx0Er3U5vNoflSsWqMXQPK+IRviaJP9f3e+pLnV2179e2TX59PV +bBwXKGwXAoKq19TFwmLAUjuHDKccynwwWenEzQ5CUiUTwtwxzp85hiVZkG07WWST +aOyGYIUnYUGRQiis4/ElXoBpS6mmld4zlbV7ABEBAAG0HUFycGFkIEJvZGEgPGFi +b2RhQGFwYWNoZS5vcmc+iQFUBBMBCgA+FiEE7ni7ioKnhRR3qq0KOQwbWt6XiDUF +Al2xedcCGwMFCQPCZwAFCwkIBwMFFQoJCAsFFgIDAQACHgECF4AACgkQOQwbWt6X +iDXu/Qf8CS1g3ErUBwUkRPegJTjwTAxTEwyxMz2h8rYZMW2Tx55KWyRIqt0FxVqm +T7NtEeSoJbkIOcokaaOX673UmM5GJo2SaKqiBOzmYWPDiFYLHnvpfaDV9kDOSJG5 +ohhfSPGQX4x2L2GNO75h9bVSd6gITCSqq9FlyLAoaNZj5nS+1Px9ltAHpli4CQiP ++uCe3alFF7/WsYipUI5FyJ+2aUrHyua7MxNRx1SI0vky+ywqcb6O7+2BSOCkJsZS +5elSN43RfWq2+RABtSBv6kKtjxaC+FW6oCTCp6vJJvVEOA7I+rGUpP5mQHRJMtwW +2MfYBa0Byw+M+1ASE6orIjgyZPTn47kBDQRdsXnXAQgAz+nnewzYZcPEBVUQqObD +91g3sKO2PNrmW+Xql0V5li1D3s9JeufFHlkekcH6EtuvEskHrXBsqLRrCL+TlRf+ +uS+S6/+TPW8Bt0NX2UfEfV9DEKDzSnpxuzt6jDGYM6xLtDrgrJYFo/O/WGx69bnM +829Mg5DCBwLlMZ5x1gqEwFm6AKbsrqILLQSAnGP3N/FVI100aZSLe9dlTEvx/6uh +pRAUdJVNbwNr4GtQkc8PnwJkPNP8hU1yxngo613asQXSLcGOxs1TH5KPJIZq1oan +GTpyCgNYIDSSIjW6SvYEd8I6o3zbKhOGEuuRL3gJxoBWe2Y/vYO7dODXosGSk1tH +owARAQABiQE8BBgBCgAmFiEE7ni7ioKnhRR3qq0KOQwbWt6XiDUFAl2xedcCGwwF +CQPCZwAACgkQOQwbWt6XiDXhAAf/S7RThbngay8xsqtsURpVBDEpzb0a9S3HtzgM +tH0FRvqJdgGUimXoYKI6aF9cBu6d3UQx7i24pAYaMWHYXfsUg9GKXFA5UeKNfm5I +XvAljxB/dize3lAXm09DWeZR/x6eFfqixsVAAenxZz0+r3EM+H14k1HbmzZTHNtE +SPTTDAUii7UKZPuzOj0gjM9QSqY04KCyBoKzi4KXNjQM7yznpfVd0H/Gq2YZt8pd +a57suDGU8qrQdTI01RhizHCNnJWmIVLe4OcNOHQF4XnEWX1Meg24cbGjMocqLJkf +9tVXeGYnwosk65MO6kBnE98Bdsey7T1rH+PyN1ZLYP1Yiq/rbg== +=tbc4 +-END PGP PUBLIC KEY BLOCK- +pub rsa4096 2019-11-11 [SC] [expires: 2021-11-10] + ABD83C1B02498F0BF91B77B7FC73625597C9B1F2 +uid [ unknown] Daniel Bakai +sig 3FC73625597C9B1F2 2019-11-11 Daniel Bakai + +-BEGIN PGP PUBLIC KEY BLOCK- + +mQINBF3JXX0BEACz2j/XmlQaU4sxqimt/9QzaIk51BFbweddLG7FIGidvu+RcVdC +5vuMpz4z1/SXBcroEyI8HatQKUZ0EYwUsFlET3k6ScU7X9ttL8Q0YYwU0YQo3+gY +qx0vl2f5v+hxObwNfpoAWMi5e+fOMKnzj9pqarDfUyh9zouv4OR3Yx18I+YI/mNA +wUs2yq+BhU8DmEYKemrEGqi6fGGbuG7kf3Ckx0OdMLvoqAi6QIfY96v8v2Ov7whu +xJYoZSGXNAGzrMfJQ3I5WOI5XD2x1gPH0AAD+hPyrmZWPEhWM4KfdWCBEBnqV6d7 +S87LuLr33lCDIdJ+1E/hkAFNbvwTRo7XBuID1Fa29JgcEUc+dHNk3hMJve8gldK8 +N2Ba1Wd0OdsyNJY32U7dh6pyZnN4VMmj5AjSc17SNtmehgwt8OWK7X9DZFdA4Xsr +GT6UMsGHF21/RU6TyTlvhiRQas7F6hB6B2Go/QoiNyIj0Gtz5hcOjcz1VgZNa/2y +K5GEhfs+4IhNV78nsh4A68Ab6ThRwhavFIW+GXltRDu5dlv+haX3XmiLBryQgNqL +b7FJtGlVLc1CCf/+J+IvF0Lk8q/0xArxdiGa4fpyh7XVAlCCZAfPtkHjU0Nplm6g +TcqONX1B4dAb/BZijG3sOQ86moDuAkQO7jGwqqXJ2uu4OV6ywUfvEvjloQARAQAB +tCBEYW5pZWwgQmFrYWkgPGJha2FpZEBhcGFjaGUub3JnPokCVAQTAQgAPhYhBKvY +PBsCSY8L+Rt3t/xzYlWXybHyBQJdyV19AhsDBQkDwmcABQsJCAcCBhUKCQgLAgQW +AgMBAh4BAheAAAoJEPxzYlWXybHyMKMP/RCZYYHz7kTFNgjNfINfq7WwfRGRDKZK +QTMoxkv0qg2cC4Ilyc22qXtTAMBoApzI4BKUXKC5vmnh6woxUa1cSK46/V7fG7px +GFhmUBV8EYpKyAPy+Vd11VhVc1XOwPfFRf5RdQvJjoR3rIvgTka5ZWFTMHeQaZCp +31+rNfxiRxG1AeysZjRrph68ylME5YNSLWmShxa025Whqns+FbY/HY04n9V6vgEZ +IDIStvNSGnFNUgTy+qxPba/Dcdn3ZGECCo/WPWz8rsXnGx4nSflSWrXyLK9jN9Np +xc9TQMH5TxCxeFyAuuWIsPk4ksZ/EXVBe5IWIHr05h0r81SelGSFWKkkPURGgpLy +y5xIkFc9QP4aKvHw7nWv3XNrcx7HhibeORNN9Udb5r3qRKIRIFNcdPRvKv8Vdfm8 +a7IMA+ZltuB3zcpu9wPlczutyfZiHILjSG47BV4ZCQ0vN8crAn7rUrzY1h74Qaw/ +VHuDvSQTXSB74ND1z+ESeIgE5S904kz5Ha+vN8WxErOQOl5AfItrS5FMvFvWCc1r +8zD4EkpSuc9wK9zrhUko0G6nmN1pgWUyWumJp8ICEM5kIYP6kz4rSdevRJBqnOeo +htWUxbnSlV9LWyhhyRp8X6OrgMvWWKx/7KX791D71WdC5/FNdtWIFWINHWVAQ4bI +NoiIphYP57Bp +=dl0g +-END PGP PUBLIC KEY BLOCK-
[jira] [Assigned] (MINIFI-521) minifi-toolkit-0.5.0: "Connection with id ... has invalid destination id" confused by targetId?
[
https://issues.apache.org/jira/browse/MINIFI-521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aldrin Piri reassigned MINIFI-521:
--
Assignee: (was: Aldrin Piri)
> minifi-toolkit-0.5.0: "Connection with id ... has invalid destination id"
> confused by targetId?
> ---
>
> Key: MINIFI-521
> URL: https://issues.apache.org/jira/browse/MINIFI-521
> Project: Apache NiFi MiNiFi
> Issue Type: Bug
> Components: Agent Configuration/Installation
>Affects Versions: 0.5.0
> Environment: version: '3'
> services:
> nifi:
> image: apache/nifi:1:10.0
> minifi:
> image: apache/nifi-minifi:0.5.0
> minifi-toolkit-0.5.0, Windows
>Reporter: Sebastian Napiorkowski
>Priority: Minor
> Attachments: MiNiFi-buggy.xml, MiNiFi-expected.xml, config-buggy.yml,
> config-expected.yml
>
>
> I followed this tutorial: [Getting started with
> MiNiFi|https://nifi.apache.org/minifi/getting-started.html]
> And the problem was that, while transforming the Template to the yml-file, it
> failed with: "Connection with id 10de7342-41ef-37a5-- has
> invalid destination id 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd"
> {code:java}
> minifi-toolkit-0.5.0\bin\config.bat transform MiNiFi-buggy.xml
> config-buggy.yml
> There are validation errors with the template, still outputting YAML but it
> will need to be edited.
> Connection with id 10de7342-41ef-37a5-- has invalid
> destination id 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd
> {code}
> By the help of brosander@github (Thank you, mate!) we discovered that the
> tool is confused by the targetId property:
> {code:xml}
>
>
>
>
>
> 1
> true
> true
>
> db4d2e20-016e-1000-d3c7-73a67e2b1d67
> 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd
> input_minifi
>
> dadd7f29-016e-1000-3c91-89495c813674
> false
> false
> false
>
>
> {code}
> just by removing the {{targetId}}, the toolkit runs through:
> {code:java}
> minifi-toolkit-0.5.0\bin\config.bat transform MiNiFi-expected.xml
> config-expected.yml
> No validation errors found in converted configuration.
> {code}
> The produced files differ:
> {code:java}
> diff config-buggy.yml config-expected.yml
> 101c101
> < - id: dadd7f29-016e-1000-3c91-89495c813674
> ---
> > - id: 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd
> {code}
> -I tested it and NiFi discovers MiNiFi, everything seems fine.-
> EDIT:
> I celebrated to early. Removing the {{targetId}} is *not* a working
> workaround. The easiest way to get around this issue is to downgrade NiFi
> from 1.10 to 1.9.2.
> Originally I thought this is a MiNiFi issue, but now I'm thinking that
> something changed in NiFi from 1.9.2 to 1.10.0 when exporting Templates. The
> ids are mixed up. This could be a much more critical bug, than I initially
> thought.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
svn commit: r37526 - /release/nifi/KEYS
Author: aldrin Date: Wed Jan 8 15:27:17 2020 New Revision: 37526 Log: Adding Arpad Boda and Dániel Bakai to KEYS file. Modified: release/nifi/KEYS Modified: release/nifi/KEYS == --- release/nifi/KEYS (original) +++ release/nifi/KEYS Wed Jan 8 15:27:17 2020 @@ -1382,3 +1382,74 @@ TPZj7cC30y/MUdi0zJI= =uY8D -END PGP PUBLIC KEY BLOCK- +pub rsa2048 2019-10-24 [SC] [expires: 2021-10-23] + EE78BB8A82A7851477AAAD0A390C1B5ADE978835 +uid [ultimate] Arpad Boda +sig 3390C1B5ADE978835 2019-10-24 Arpad Boda +sub rsa2048 2019-10-24 [E] [expires: 2021-10-23] +sig 390C1B5ADE978835 2019-10-24 Arpad Boda + +-BEGIN PGP PUBLIC KEY BLOCK- + +mQENBF2xedcBCADhvZxpFnZ37r3LuC4ZBNJu5OX4ORkXSWOgo7JwGb0pDGWrQlcq +ZU57fjeGvyciLPF7ICAOKDGCF2YYrRfCG2IhI4S6B+pYpIs9H9XnfoVLUOYgCQAl +VsugMlMXAnLb5MNNFEV6XLQooqvFkTtsbRcEsES6DogBeYlMoF2koJkY5W/Evpcs +96nwVH9cTtdahOx0Er3U5vNoflSsWqMXQPK+IRviaJP9f3e+pLnV2179e2TX59PV +bBwXKGwXAoKq19TFwmLAUjuHDKccynwwWenEzQ5CUiUTwtwxzp85hiVZkG07WWST +aOyGYIUnYUGRQiis4/ElXoBpS6mmld4zlbV7ABEBAAG0HUFycGFkIEJvZGEgPGFi +b2RhQGFwYWNoZS5vcmc+iQFUBBMBCgA+FiEE7ni7ioKnhRR3qq0KOQwbWt6XiDUF +Al2xedcCGwMFCQPCZwAFCwkIBwMFFQoJCAsFFgIDAQACHgECF4AACgkQOQwbWt6X +iDXu/Qf8CS1g3ErUBwUkRPegJTjwTAxTEwyxMz2h8rYZMW2Tx55KWyRIqt0FxVqm +T7NtEeSoJbkIOcokaaOX673UmM5GJo2SaKqiBOzmYWPDiFYLHnvpfaDV9kDOSJG5 +ohhfSPGQX4x2L2GNO75h9bVSd6gITCSqq9FlyLAoaNZj5nS+1Px9ltAHpli4CQiP ++uCe3alFF7/WsYipUI5FyJ+2aUrHyua7MxNRx1SI0vky+ywqcb6O7+2BSOCkJsZS +5elSN43RfWq2+RABtSBv6kKtjxaC+FW6oCTCp6vJJvVEOA7I+rGUpP5mQHRJMtwW +2MfYBa0Byw+M+1ASE6orIjgyZPTn47kBDQRdsXnXAQgAz+nnewzYZcPEBVUQqObD +91g3sKO2PNrmW+Xql0V5li1D3s9JeufFHlkekcH6EtuvEskHrXBsqLRrCL+TlRf+ +uS+S6/+TPW8Bt0NX2UfEfV9DEKDzSnpxuzt6jDGYM6xLtDrgrJYFo/O/WGx69bnM +829Mg5DCBwLlMZ5x1gqEwFm6AKbsrqILLQSAnGP3N/FVI100aZSLe9dlTEvx/6uh +pRAUdJVNbwNr4GtQkc8PnwJkPNP8hU1yxngo613asQXSLcGOxs1TH5KPJIZq1oan +GTpyCgNYIDSSIjW6SvYEd8I6o3zbKhOGEuuRL3gJxoBWe2Y/vYO7dODXosGSk1tH +owARAQABiQE8BBgBCgAmFiEE7ni7ioKnhRR3qq0KOQwbWt6XiDUFAl2xedcCGwwF +CQPCZwAACgkQOQwbWt6XiDXhAAf/S7RThbngay8xsqtsURpVBDEpzb0a9S3HtzgM +tH0FRvqJdgGUimXoYKI6aF9cBu6d3UQx7i24pAYaMWHYXfsUg9GKXFA5UeKNfm5I +XvAljxB/dize3lAXm09DWeZR/x6eFfqixsVAAenxZz0+r3EM+H14k1HbmzZTHNtE +SPTTDAUii7UKZPuzOj0gjM9QSqY04KCyBoKzi4KXNjQM7yznpfVd0H/Gq2YZt8pd +a57suDGU8qrQdTI01RhizHCNnJWmIVLe4OcNOHQF4XnEWX1Meg24cbGjMocqLJkf +9tVXeGYnwosk65MO6kBnE98Bdsey7T1rH+PyN1ZLYP1Yiq/rbg== +=tbc4 +-END PGP PUBLIC KEY BLOCK- +pub rsa4096 2019-11-11 [SC] [expires: 2021-11-10] + ABD83C1B02498F0BF91B77B7FC73625597C9B1F2 +uid [ unknown] Daniel Bakai +sig 3FC73625597C9B1F2 2019-11-11 Daniel Bakai + +-BEGIN PGP PUBLIC KEY BLOCK- + +mQINBF3JXX0BEACz2j/XmlQaU4sxqimt/9QzaIk51BFbweddLG7FIGidvu+RcVdC +5vuMpz4z1/SXBcroEyI8HatQKUZ0EYwUsFlET3k6ScU7X9ttL8Q0YYwU0YQo3+gY +qx0vl2f5v+hxObwNfpoAWMi5e+fOMKnzj9pqarDfUyh9zouv4OR3Yx18I+YI/mNA +wUs2yq+BhU8DmEYKemrEGqi6fGGbuG7kf3Ckx0OdMLvoqAi6QIfY96v8v2Ov7whu +xJYoZSGXNAGzrMfJQ3I5WOI5XD2x1gPH0AAD+hPyrmZWPEhWM4KfdWCBEBnqV6d7 +S87LuLr33lCDIdJ+1E/hkAFNbvwTRo7XBuID1Fa29JgcEUc+dHNk3hMJve8gldK8 +N2Ba1Wd0OdsyNJY32U7dh6pyZnN4VMmj5AjSc17SNtmehgwt8OWK7X9DZFdA4Xsr +GT6UMsGHF21/RU6TyTlvhiRQas7F6hB6B2Go/QoiNyIj0Gtz5hcOjcz1VgZNa/2y +K5GEhfs+4IhNV78nsh4A68Ab6ThRwhavFIW+GXltRDu5dlv+haX3XmiLBryQgNqL +b7FJtGlVLc1CCf/+J+IvF0Lk8q/0xArxdiGa4fpyh7XVAlCCZAfPtkHjU0Nplm6g +TcqONX1B4dAb/BZijG3sOQ86moDuAkQO7jGwqqXJ2uu4OV6ywUfvEvjloQARAQAB +tCBEYW5pZWwgQmFrYWkgPGJha2FpZEBhcGFjaGUub3JnPokCVAQTAQgAPhYhBKvY +PBsCSY8L+Rt3t/xzYlWXybHyBQJdyV19AhsDBQkDwmcABQsJCAcCBhUKCQgLAgQW +AgMBAh4BAheAAAoJEPxzYlWXybHyMKMP/RCZYYHz7kTFNgjNfINfq7WwfRGRDKZK +QTMoxkv0qg2cC4Ilyc22qXtTAMBoApzI4BKUXKC5vmnh6woxUa1cSK46/V7fG7px +GFhmUBV8EYpKyAPy+Vd11VhVc1XOwPfFRf5RdQvJjoR3rIvgTka5ZWFTMHeQaZCp +31+rNfxiRxG1AeysZjRrph68ylME5YNSLWmShxa025Whqns+FbY/HY04n9V6vgEZ +IDIStvNSGnFNUgTy+qxPba/Dcdn3ZGECCo/WPWz8rsXnGx4nSflSWrXyLK9jN9Np +xc9TQMH5TxCxeFyAuuWIsPk4ksZ/EXVBe5IWIHr05h0r81SelGSFWKkkPURGgpLy +y5xIkFc9QP4aKvHw7nWv3XNrcx7HhibeORNN9Udb5r3qRKIRIFNcdPRvKv8Vdfm8 +a7IMA+ZltuB3zcpu9wPlczutyfZiHILjSG47BV4ZCQ0vN8crAn7rUrzY1h74Qaw/ +VHuDvSQTXSB74ND1z+ESeIgE5S904kz5Ha+vN8WxErOQOl5AfItrS5FMvFvWCc1r +8zD4EkpSuc9wK9zrhUko0G6nmN1pgWUyWumJp8ICEM5kIYP6kz4rSdevRJBqnOeo +htWUxbnSlV9LWyhhyRp8X6OrgMvWWKx/7KX791D71WdC5/FNdtWIFWINHWVAQ4bI +NoiIphYP57Bp +=dl0g +-END PGP PUBLIC KEY BLOCK-
[nifi-minifi-cpp] branch master updated: MINIFICPP-1091 - Update LICENSE and NOTICE
This is an automated email from the ASF dual-hosted git repository. aldrin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/master by this push: new aa42957 MINIFICPP-1091 - Update LICENSE and NOTICE aa42957 is described below commit aa42957a2e227df41510047cece3cd606dc1cb6a Author: Daniel Bakai AuthorDate: Mon Dec 16 17:20:53 2019 +0100 MINIFICPP-1091 - Update LICENSE and NOTICE This closes #702. Signed-off-by: Aldrin Piri --- LICENSE | 1206 -- NOTICE | 29 +- README.md|1 + extensions/opencv/CMakeLists.txt | 10 +- 4 files changed, 1052 insertions(+), 194 deletions(-) diff --git a/LICENSE b/LICENSE index 813c4d0..c51aa71 100644 --- a/LICENSE +++ b/LICENSE @@ -201,9 +201,15 @@ See the License for the specific language governing permissions and limitations under the License. -APACHE NIFI - MINIFI SUBCOMPONENTS: +APACHE NIFI - MINIFI C++ SUBCOMPONENTS: -The Apache NiFi - MiNiFi project contains subcomponents with separate copyright +The Apache NiFi - MiNiFi C++ project contains subcomponents licensed under the Apache License, Version 2.0: +This product bundles 'Simple-Windows-Posix-Semaphore' which is available under an ALv2 license +This project bundles 'mbedTLS' which is available under an ALv2 license +This project bundles 'RocksDB' which is available under an ALv2 license +This project bundles 'AWS SDK for C++' which is available under an ALv2 license + +The Apache NiFi - MiNiFi C++ project contains subcomponents with separate copyright notices and license terms. Your use of the source code for the these subcomponents is subject to the terms and conditions of the following licenses. @@ -1422,185 +1428,6 @@ This product bundles 'bsdiff' which is available under a "2-clause BSD" license. IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - This product bundles 'Simple-Windows-Posix-Semaphore' which is available under an Apache v2 License. - -Apache License - Version 2.0, January 2004 -http://www.apache.org/licenses/ - - TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - - 1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner -
[nifi-minifi-cpp] branch master updated: MINIFICPP-1108 - Automate the generation of processors.md
This is an automated email from the ASF dual-hosted git repository.
aldrin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 60da477 MINIFICPP-1108 - Automate the generation of processors.md
60da477 is described below
commit 60da4771a271d618ec253ed7ba546718f5de1ca1
Author: Marc Parisi
AuthorDate: Wed Dec 18 15:24:35 2019 +0100
MINIFICPP-1108 - Automate the generation of processors.md
This closes #700.
Signed-off-by: Aldrin Piri
---
main/AgentDocs.cpp | 175
main/AgentDocs.h| 44 +
main/CMakeLists.txt | 2 +-
main/MiNiFiMain.cpp | 35 +++
4 files changed, 255 insertions(+), 1 deletion(-)
diff --git a/main/AgentDocs.cpp b/main/AgentDocs.cpp
new file mode 100644
index 000..3d2eb76
--- /dev/null
+++ b/main/AgentDocs.cpp
@@ -0,0 +1,175 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "AgentDocs.h"
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include "core/ClassLoader.h"
+#include "core/ConfigurableComponent.h"
+#include "core/Core.h"
+#include "core/Processor.h"
+#include "core/Property.h"
+#include "core/PropertyValidation.h"
+#include "core/Relationship.h"
+#include "io/validation.h"
+#include "utils/file/FileUtils.h"
+#include "agent/build_description.h"
+#include "agent/agent_version.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace docs {
+
+std::string AgentDocs::extractClassName(const std::string ) const {
+ auto lastOfIdx = processor.find_last_of(".");
+ if (lastOfIdx != std::string::npos) {
+lastOfIdx++; // if a value is found, increment to move beyond the .
+int nameLength = processor.length() - lastOfIdx;
+std::string processorName = processor.substr(lastOfIdx, nameLength);
+return processorName;
+ }
+ return processor;
+
+}
+
+void AgentDocs::generate(const std::string , std::ostream ) {
+ std::map processorSet;
+ for (const auto : minifi::AgentBuild::getExtensions()) {
+
+struct Components descriptions =
BuildDescription::getClassDescriptions(group);
+for (const auto : descriptions.processors_) {
+
processorSet.insert(std::make_pair(extractClassName(processorName.class_name_),
processorName));
+}
+ }
+ for (const auto : processorSet) {
+const std::string = docsdir +
utils::file::FileUtils::get_separator() + processor.first;
+std::ofstream outfile(filename);
+
+std::string description;
+
+bool foundDescription = minifi::AgentDocs::getDescription(processor.first,
description);
+
+if (!foundDescription) {
+ foundDescription =
minifi::AgentDocs::getDescription(processor.second.class_name_, description);
+}
+
+outfile << "## " << processor.first << std::endl << std::endl;
+if (foundDescription) {
+ outfile << "### Description " << std::endl << std::endl;
+ outfile << description << std::endl;
+}
+
+outfile << "### Properties " << std::endl << std::endl;
+outfile
+<< "In the list below, the names of required properties appear in
bold. Any other properties (not in bold) are considered optional. The table
also indicates any default values, and whether a property supports the NiFi
Expression Language."
+<< std::endl<< std::endl;
+
+outfile << "| Name | Default Value | Allowable Values | Description | " <<
std::endl << "| - | - | - | - | " << std::endl;
+for (const auto : processor.second.class_properties_) {
+ bool supportsEl = prop.second.supportsExpressionLangauge();
+ outfile << "|";
+ if (prop.second.getRequired()) {
+outfile << "**";
+ }
+ outfile << prop.first;
[nifi-minifi-cpp] branch master updated: MINIFICPP-1109 - Update PROCESSORS.md to be alligned with latest changes
This is an automated email from the ASF dual-hosted git repository. aldrin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git The following commit(s) were added to refs/heads/master by this push: new cba266f MINIFICPP-1109 - Update PROCESSORS.md to be alligned with latest changes cba266f is described below commit cba266f9356b43f8f6e72cc07a7bb136bc7011f2 Author: Arpad Boda AuthorDate: Wed Dec 18 17:09:13 2019 +0100 MINIFICPP-1109 - Update PROCESSORS.md to be alligned with latest changes This closes #701. Signed-off-by: Aldrin Piri --- PROCESSORS.md | 378 -- 1 file changed, 105 insertions(+), 273 deletions(-) diff --git a/PROCESSORS.md b/PROCESSORS.md index 3f0a1af..0718f6b 100644 --- a/PROCESSORS.md +++ b/PROCESSORS.md @@ -10,14 +10,12 @@ - [CaptureRTSPFrame](#capturertspframe) - [CompressContent](#compresscontent) - [ConsumeMQTT](#consumemqtt) -- [ConvertHeartBeat](#convertheartbeat) -- [ConvertJSONAck](#convertjsonack) -- [ConvertUpdate](#convertupdate) - [ExecuteProcess](#executeprocess) - [ExecutePythonProcessor](#executepythonprocessor) - [ExecuteSQL](#executesql) - [ExecuteScript](#executescript) - [ExtractText](#extracttext) +- [FetchOPCProcessor](#fetchopcprocessor) - [FetchSFTP](#fetchsftp) - [FocusArchiveEntry](#focusarchiveentry) - [GenerateFlowFile](#generateflowfile) @@ -33,15 +31,14 @@ - [LogAttribute](#logattribute) - [ManipulateArchive](#manipulatearchive) - [MergeContent](#mergecontent) +- [MotionDetector](#motiondetector) - [PublishKafka](#publishkafka) - [PublishMQTT](#publishmqtt) - [PutFile](#putfile) +- [PutOPCProcessor](#putopcprocessor) - [PutSFTP](#putsftp) - [PutSQL](#putsql) - [RouteOnAttribute](#routeonattribute) -- [TFApplyGraph](#tfapplygraph) -- [TFConvertImageToTensor](#tfconvertimagetotensor) -- [TFExtractTopLabels](#tfextracttoplabels) - [TailFile](#tailfile) - [UnfocusArchiveEntry](#unfocusarchiveentry) - [UpdateAttribute](#updateattribute) @@ -124,7 +121,7 @@ In the list below, the names of required properties appear in bold. Any other pr |Base Directory|/tmp/||Scratch directory for PCAP files| |Batch Size|50||The number of packets to combine within a given PCAP| |Capture Bluetooth|false||True indicates that we support bluetooth interfaces| -|Network Controller|.*||Regular expression of the network controller(s) to which we will attach| +|Network Controllers|.*||Regular expression of the network controller(s) to which we will attach| ### Properties | Name | Description | @@ -209,67 +206,6 @@ In the list below, the names of required properties appear in bold. Any other pr |success|FlowFiles that are sent successfully to the destination are transferred to this relationship| -## ConvertHeartBeat - -### Description - -ConvertHeartBeat converts heatrbeats into MQTT messages. -### Properties - -In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. - -| Name | Default Value | Allowable Values | Description | -| - | - | - | - | -|Listening Topic|||Name of topic to listen to| -|MQTT Controller Service|||Name of controller service that will be used for MQTT interactivity| -### Properties - -| Name | Description | -| - | - | -|success|All files are routed to success| - - -## ConvertJSONAck - -### Description - -Converts JSON acks into an MQTT consumable by MQTTC2Protocol -### Properties - -In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. - -| Name | Default Value | Allowable Values | Description | -| - | - | - | - | -|Listening Topic|||Name of topic to listen to| -|MQTT Controller Service|||Name of controller service that will be used for MQTT interactivity| -### Properties - -| Name | Description | -| - | - | -|success|All files are routed to success| - - -## ConvertUpdate - -### Description - -onverts update messages into the appropriate RESTFul call -### Properties - -In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language. - -| Name | Default Value | Allowable Values | Description | -| - | - | - | - | -|Listening Topic|||Name of topic to listen to| -|MQTT Controller Service|||Name of controller service that will be used for MQTT interactivity| -|SSL Context Service|||The SSL Context Service used to provide client certificate information for TLS/SSL (https) connections.| -### Properties - -| Name | Description | -| - | - | -|success
[jira] [Commented] (MINIFI-521) minifi-toolkit-0.5.0: "Connection with id ... has invalid destination id" confused by targetId?
[
https://issues.apache.org/jira/browse/MINIFI-521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998455#comment-16998455
]
Aldrin Piri commented on MINIFI-521:
Definitely was a change, but thankfully the associated template encoding
version was also bumped. Will need to reason about the changes but basing
handling on that determination should be able to get us there.
> minifi-toolkit-0.5.0: "Connection with id ... has invalid destination id"
> confused by targetId?
> ---
>
> Key: MINIFI-521
> URL: https://issues.apache.org/jira/browse/MINIFI-521
> Project: Apache NiFi MiNiFi
> Issue Type: Bug
> Components: Agent Configuration/Installation
>Affects Versions: 0.5.0
> Environment: version: '3'
> services:
> nifi:
> image: apache/nifi:1:10.0
> minifi:
> image: apache/nifi-minifi:0.5.0
> minifi-toolkit-0.5.0, Windows
> Reporter: Sebastian Napiorkowski
>Assignee: Aldrin Piri
>Priority: Minor
> Attachments: MiNiFi-buggy.xml, MiNiFi-expected.xml, config-buggy.yml,
> config-expected.yml
>
>
> I followed this tutorial: [Getting started with
> MiNiFi|https://nifi.apache.org/minifi/getting-started.html]
> And the problem was that, while transforming the Template to the yml-file, it
> failed with: "Connection with id 10de7342-41ef-37a5-- has
> invalid destination id 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd"
> {code:java}
> minifi-toolkit-0.5.0\bin\config.bat transform MiNiFi-buggy.xml
> config-buggy.yml
> There are validation errors with the template, still outputting YAML but it
> will need to be edited.
> Connection with id 10de7342-41ef-37a5-- has invalid
> destination id 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd
> {code}
> By the help of brosander@github (Thank you, mate!) we discovered that the
> tool is confused by the targetId property:
> {code:xml}
>
>
>
>
>
> 1
> true
> true
>
> db4d2e20-016e-1000-d3c7-73a67e2b1d67
> 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd
> input_minifi
>
> dadd7f29-016e-1000-3c91-89495c813674
> false
> false
> false
>
>
> {code}
> just by removing the {{targetId}}, the toolkit runs through:
> {code:java}
> minifi-toolkit-0.5.0\bin\config.bat transform MiNiFi-expected.xml
> config-expected.yml
> No validation errors found in converted configuration.
> {code}
> The produced files differ:
> {code:java}
> diff config-buggy.yml config-expected.yml
> 101c101
> < - id: dadd7f29-016e-1000-3c91-89495c813674
> ---
> > - id: 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd
> {code}
> -I tested it and NiFi discovers MiNiFi, everything seems fine.-
> EDIT:
> I celebrated to early. Removing the {{targetId}} is *not* a working
> workaround. The easiest way to get around this issue is to downgrade NiFi
> from 1.10 to 1.9.2.
> Originally I thought this is a MiNiFi issue, but now I'm thinking that
> something changed in NiFi from 1.9.2 to 1.10.0 when exporting Templates. The
> ids are mixed up. This could be a much more critical bug, than I initially
> thought.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (MINIFI-521) minifi-toolkit-0.5.0: "Connection with id ... has invalid destination id" confused by targetId?
[
https://issues.apache.org/jira/browse/MINIFI-521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aldrin Piri reassigned MINIFI-521:
--
Assignee: Aldrin Piri
> minifi-toolkit-0.5.0: "Connection with id ... has invalid destination id"
> confused by targetId?
> ---
>
> Key: MINIFI-521
> URL: https://issues.apache.org/jira/browse/MINIFI-521
> Project: Apache NiFi MiNiFi
> Issue Type: Bug
> Components: Agent Configuration/Installation
>Affects Versions: 0.5.0
> Environment: version: '3'
> services:
> nifi:
> image: apache/nifi:1:10.0
> minifi:
> image: apache/nifi-minifi:0.5.0
> minifi-toolkit-0.5.0, Windows
> Reporter: Sebastian Napiorkowski
>Assignee: Aldrin Piri
>Priority: Minor
> Attachments: MiNiFi-buggy.xml, MiNiFi-expected.xml, config-buggy.yml,
> config-expected.yml
>
>
> I followed this tutorial: [Getting started with
> MiNiFi|https://nifi.apache.org/minifi/getting-started.html]
> And the problem was that, while transforming the Template to the yml-file, it
> failed with: "Connection with id 10de7342-41ef-37a5-- has
> invalid destination id 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd"
> {code:java}
> minifi-toolkit-0.5.0\bin\config.bat transform MiNiFi-buggy.xml
> config-buggy.yml
> There are validation errors with the template, still outputting YAML but it
> will need to be edited.
> Connection with id 10de7342-41ef-37a5-- has invalid
> destination id 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd
> {code}
> By the help of brosander@github (Thank you, mate!) we discovered that the
> tool is confused by the targetId property:
> {code:xml}
>
>
>
>
>
> 1
> true
> true
>
> db4d2e20-016e-1000-d3c7-73a67e2b1d67
> 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd
> input_minifi
>
> dadd7f29-016e-1000-3c91-89495c813674
> false
> false
> false
>
>
> {code}
> just by removing the {{targetId}}, the toolkit runs through:
> {code:java}
> minifi-toolkit-0.5.0\bin\config.bat transform MiNiFi-expected.xml
> config-expected.yml
> No validation errors found in converted configuration.
> {code}
> The produced files differ:
> {code:java}
> diff config-buggy.yml config-expected.yml
> 101c101
> < - id: dadd7f29-016e-1000-3c91-89495c813674
> ---
> > - id: 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd
> {code}
> -I tested it and NiFi discovers MiNiFi, everything seems fine.-
> EDIT:
> I celebrated to early. Removing the {{targetId}} is *not* a working
> workaround. The easiest way to get around this issue is to downgrade NiFi
> from 1.10 to 1.9.2.
> Originally I thought this is a MiNiFi issue, but now I'm thinking that
> something changed in NiFi from 1.9.2 to 1.10.0 when exporting Templates. The
> ids are mixed up. This could be a much more critical bug, than I initially
> thought.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
svn commit: r1868893 - /nifi/site/trunk/mailing_lists.html
Author: aldrin Date: Fri Oct 25 01:18:51 2019 New Revision: 1868893 URL: http://svn.apache.org/viewvc?rev=1868893=rev Log: Make use of ASF shortened URL to redirect to community Slack invite Modified: nifi/site/trunk/mailing_lists.html Modified: nifi/site/trunk/mailing_lists.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/mailing_lists.html?rev=1868893=1868892=1868893=diff == --- nifi/site/trunk/mailing_lists.html (original) +++ nifi/site/trunk/mailing_lists.html Fri Oct 25 01:18:51 2019 @@ -225,7 +225,7 @@ Apache NiFi Slack Workspace: https://apachenifi.slack.com/;>https://apachenifi.slack.com/ -New users can join the workspace using the following https://join.slack.com/t/apachenifi/shared_invite/enQtNDI2NDMyMTY3MTA5LTQ0YmQwMjljZDQ4NjkzODI0ZWU1YmU0YmU0YjJiNjVjMjhmMWQwZjY0Njk0NDY1OTlkMjM5NmNiODg4MGFmYWU;>invite link. +New users can join the workspace using the following https://s.apache.org/nifi-community-slack;>invite link. IRC: #nifi on http://webchat.freenode.net/?channels=#nifi;>irc.freenode.net
[nifi-site] branch master updated: Make use of ASF shortened URL for redirecting to community slack channel invite link
This is an automated email from the ASF dual-hosted git repository. aldrin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi-site.git The following commit(s) were added to refs/heads/master by this push: new 540f87a Make use of ASF shortened URL for redirecting to community slack channel invite link 540f87a is described below commit 540f87abc3e00668993450dde136be59235fe7b5 Author: Aldrin Piri AuthorDate: Thu Oct 24 21:12:53 2019 -0400 Make use of ASF shortened URL for redirecting to community slack channel invite link --- src/pages/html/mailing_lists.hbs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/pages/html/mailing_lists.hbs b/src/pages/html/mailing_lists.hbs index 4232757..8fbf90b 100644 --- a/src/pages/html/mailing_lists.hbs +++ b/src/pages/html/mailing_lists.hbs @@ -121,7 +121,7 @@ title: Apache NiFi Mailing Lists Chat Apache NiFi Slack Workspace: https://apachenifi.slack.com/;>https://apachenifi.slack.com/ -New users can join the workspace using the following https://join.slack.com/t/apachenifi/shared_invite/enQtNDI2NDMyMTY3MTA5LTQ0YmQwMjljZDQ4NjkzODI0ZWU1YmU0YmU0YjJiNjVjMjhmMWQwZjY0Njk0NDY1OTlkMjM5NmNiODg4MGFmYWU;>invite link. +New users can join the workspace using the following https://s.apache.org/nifi-community-slack;>invite link. IRC: #nifi on http://webchat.freenode.net/?channels=#nifi;>irc.freenode.net
svn commit: r1868808 - /nifi/site/trunk/mailing_lists.html
Author: aldrin Date: Wed Oct 23 15:20:19 2019 New Revision: 1868808 URL: http://svn.apache.org/viewvc?rev=1868808=rev Log: Update Slack invite link for community managed channel Modified: nifi/site/trunk/mailing_lists.html Modified: nifi/site/trunk/mailing_lists.html URL: http://svn.apache.org/viewvc/nifi/site/trunk/mailing_lists.html?rev=1868808=1868807=1868808=diff == --- nifi/site/trunk/mailing_lists.html (original) +++ nifi/site/trunk/mailing_lists.html Wed Oct 23 15:20:19 2019 @@ -225,7 +225,7 @@ Apache NiFi Slack Workspace: https://apachenifi.slack.com/;>https://apachenifi.slack.com/ -New users can join the workspace using the following https://join.slack.com/t/apachenifi/shared_invite/enQtNDI2NDMyMTY3MTA5LWJmZDI3MmM1ZmYyODQwZDYwM2MyMDY5ZjkyMDkxY2JmOGMyNmEzYTE0MTRkZTYwYzZlYTJkY2JhZTYyMzcyZGI;>invite link. +New users can join the workspace using the following https://join.slack.com/t/apachenifi/shared_invite/enQtNDI2NDMyMTY3MTA5LTQ0YmQwMjljZDQ4NjkzODI0ZWU1YmU0YmU0YjJiNjVjMjhmMWQwZjY0Njk0NDY1OTlkMjM5NmNiODg4MGFmYWU;>invite link. IRC: #nifi on http://webchat.freenode.net/?channels=#nifi;>irc.freenode.net
[nifi-site] branch master updated: Refresh invite link for community managed Slack channel.
This is an automated email from the ASF dual-hosted git repository. aldrin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi-site.git The following commit(s) were added to refs/heads/master by this push: new 052763d Refresh invite link for community managed Slack channel. 052763d is described below commit 052763de49f2b0f1b317c38270a4817ec9c4629f Author: Aldrin Piri AuthorDate: Wed Oct 23 11:00:33 2019 -0400 Refresh invite link for community managed Slack channel. --- src/pages/html/mailing_lists.hbs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/pages/html/mailing_lists.hbs b/src/pages/html/mailing_lists.hbs index 7ebf36c..4232757 100644 --- a/src/pages/html/mailing_lists.hbs +++ b/src/pages/html/mailing_lists.hbs @@ -121,7 +121,7 @@ title: Apache NiFi Mailing Lists Chat Apache NiFi Slack Workspace: https://apachenifi.slack.com/;>https://apachenifi.slack.com/ -New users can join the workspace using the following https://join.slack.com/t/apachenifi/shared_invite/enQtNDI2NDMyMTY3MTA5LWJmZDI3MmM1ZmYyODQwZDYwM2MyMDY5ZjkyMDkxY2JmOGMyNmEzYTE0MTRkZTYwYzZlYTJkY2JhZTYyMzcyZGI;>invite link. +New users can join the workspace using the following https://join.slack.com/t/apachenifi/shared_invite/enQtNDI2NDMyMTY3MTA5LTQ0YmQwMjljZDQ4NjkzODI0ZWU1YmU0YmU0YjJiNjVjMjhmMWQwZjY0Njk0NDY1OTlkMjM5NmNiODg4MGFmYWU;>invite link. IRC: #nifi on http://webchat.freenode.net/?channels=#nifi;>irc.freenode.net
[jira] [Resolved] (MINIFI-519) Add recommendations for antivirus exclusions to Admin guide
[ https://issues.apache.org/jira/browse/MINIFI-519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri resolved MINIFI-519. Resolution: Fixed > Add recommendations for antivirus exclusions to Admin guide > --- > > Key: MINIFI-519 > URL: https://issues.apache.org/jira/browse/MINIFI-519 > Project: Apache NiFi MiNiFi > Issue Type: Improvement >Reporter: Andrew M. Lim >Assignee: Andrew M. Lim >Priority: Minor > Fix For: 0.6.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Add similar section regarding antivirus exclusions to MiNiFi Admin Guide that > was added to NiFi Admin Guide via NIFI-6553 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (MINIFI-520) Migrate Docker image names to mirror those of the public repos
Aldrin Piri created MINIFI-520: -- Summary: Migrate Docker image names to mirror those of the public repos Key: MINIFI-520 URL: https://issues.apache.org/jira/browse/MINIFI-520 Project: Apache NiFi MiNiFi Issue Type: Improvement Reporter: Aldrin Piri Assignee: Aldrin Piri For consistency, it would be helpful to have the docker images reflect the image tags used in Docker Hub. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (MINIFI-519) Add recommendations for antivirus exclusions to Admin guide
[ https://issues.apache.org/jira/browse/MINIFI-519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri updated MINIFI-519: --- Fix Version/s: 0.6.0 > Add recommendations for antivirus exclusions to Admin guide > --- > > Key: MINIFI-519 > URL: https://issues.apache.org/jira/browse/MINIFI-519 > Project: Apache NiFi MiNiFi > Issue Type: Improvement >Reporter: Andrew M. Lim >Assignee: Andrew M. Lim >Priority: Minor > Fix For: 0.6.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Add similar section regarding antivirus exclusions to MiNiFi Admin Guide that > was added to NiFi Admin Guide via NIFI-6553 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[nifi-minifi] branch master updated: MINIFI-519 Added recommended antivirus exclusions section to Admin Guide
This is an automated email from the ASF dual-hosted git repository. aldrin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi.git The following commit(s) were added to refs/heads/master by this push: new c203c97 MINIFI-519 Added recommended antivirus exclusions section to Admin Guide c203c97 is described below commit c203c97e293791a7044be9fe299906e80fc7bb38 Author: Andrew Lim AuthorDate: Wed Oct 9 12:05:47 2019 -0400 MINIFI-519 Added recommended antivirus exclusions section to Admin Guide This closes #176. Signed-off-by: Aldrin Piri --- minifi-docs/src/main/markdown/System_Admin_Guide.md | 18 +++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/minifi-docs/src/main/markdown/System_Admin_Guide.md b/minifi-docs/src/main/markdown/System_Admin_Guide.md index 9546e97..b554615 100644 --- a/minifi-docs/src/main/markdown/System_Admin_Guide.md +++ b/minifi-docs/src/main/markdown/System_Admin_Guide.md @@ -618,7 +618,7 @@ When connecting via Site to Site, MiNiFi needs to know which input or output por ## Provenance Reporting -MiNiFi is currently designed only to report provenance data using the Site to Site protocol. +MiNiFi is currently designed only to report provenance data using the Site to Site protocol. **Note:** In order to send via HTTPS, the "Security Properties" must be fully configured. A StandardSSLContextService will be made automatically with the ID "SSL-Context-Service" and used by the Provenance Reporting. @@ -656,8 +656,8 @@ NiFi Properties Overrides: Currently, it is possible to specify keystore and truststore information to allow mutual TLS communication across the Site to Site protocol as well as provisioning an SSL Context for components in the config.yml Security can be configured in two ways for instances via: - -* **config.yml**: These properties are specified as outlined in the 'Security Properties' and 'Sensitive Properties' in the config.yml sections above. These allow the specification of security properties to be versioned with the flow and consumed via Change Ingestors. + +* **config.yml**: These properties are specified as outlined in the 'Security Properties' and 'Sensitive Properties' in the config.yml sections above. These allow the specification of security properties to be versioned with the flow and consumed via Change Ingestors. * **bootstrap.conf**: This is an alternative means of configuration that takes precedence over the config.yml configuration and is a way of separating security concerns from the processing flow. The following properties should be defined: ## Security Properties in bootstrap.conf @@ -951,3 +951,15 @@ Provenance Reporting: timeout: 30 secs batch size: 1000 ``` + +# Recommended Antivirus Exclusions + +Antivirus software can take a long time to scan directories and the files within them. Additionally, if the antivirus software locks files or directories during a scan, those resources are unavailable to MiNiFi processes, causing latency or unavailability of these resources in a MiNiFi instance. To prevent these performance and reliability issues from occurring, it is highly recommended to configure your antivirus software to skip scans on the following MiNiFi directories: + +### MiNiFi Java: + + - content_repository + - flowfile_repository + - logs + - provenance_repository + - state
[jira] [Resolved] (MINIFI-518) Version issue with httpcomponents
[ https://issues.apache.org/jira/browse/MINIFI-518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri resolved MINIFI-518. Resolution: Fixed > Version issue with httpcomponents > - > > Key: MINIFI-518 > URL: https://issues.apache.org/jira/browse/MINIFI-518 > Project: Apache NiFi MiNiFi > Issue Type: Bug > Components: Build >Affects Versions: 0.6.0 >Reporter: James >Assignee: James >Priority: Minor > Fix For: 0.6.0 > > Time Spent: 40m > Remaining Estimate: 0h > > The version of httpcomponents in the latest 0.6.0 pom causes and issue with > using a postHTTP with a security context defined (although I think its > generic to anything trying to use the SSLContextBuilder class). > The pom currently references; > > org.apache.httpcomponents > httpcore > 4.4.4 > > > But need to reference a later version, 4.4.12 works and is the latest. > I'll create a pull request so you can test the updated version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (MINIFI-518) Version issue with httpcomponents
[ https://issues.apache.org/jira/browse/MINIFI-518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri reassigned MINIFI-518: -- Assignee: James > Version issue with httpcomponents > - > > Key: MINIFI-518 > URL: https://issues.apache.org/jira/browse/MINIFI-518 > Project: Apache NiFi MiNiFi > Issue Type: Bug > Components: Build >Affects Versions: 0.6.0 >Reporter: James >Assignee: James >Priority: Minor > Time Spent: 40m > Remaining Estimate: 0h > > The version of httpcomponents in the latest 0.6.0 pom causes and issue with > using a postHTTP with a security context defined (although I think its > generic to anything trying to use the SSLContextBuilder class). > The pom currently references; > > org.apache.httpcomponents > httpcore > 4.4.4 > > > But need to reference a later version, 4.4.12 works and is the latest. > I'll create a pull request so you can test the updated version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (MINIFI-518) Version issue with httpcomponents
[ https://issues.apache.org/jira/browse/MINIFI-518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri updated MINIFI-518: --- Fix Version/s: 0.6.0 > Version issue with httpcomponents > - > > Key: MINIFI-518 > URL: https://issues.apache.org/jira/browse/MINIFI-518 > Project: Apache NiFi MiNiFi > Issue Type: Bug > Components: Build >Affects Versions: 0.6.0 >Reporter: James >Assignee: James >Priority: Minor > Fix For: 0.6.0 > > Time Spent: 40m > Remaining Estimate: 0h > > The version of httpcomponents in the latest 0.6.0 pom causes and issue with > using a postHTTP with a security context defined (although I think its > generic to anything trying to use the SSLContextBuilder class). > The pom currently references; > > org.apache.httpcomponents > httpcore > 4.4.4 > > > But need to reference a later version, 4.4.12 works and is the latest. > I'll create a pull request so you can test the updated version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[nifi-minifi] 01/03: updated httpcomponents version
This is an automated email from the ASF dual-hosted git repository. aldrin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi.git commit 49115f03cc480b891ad4e404f86dce709cb9d6a2 Author: jamesb4887 AuthorDate: Tue Oct 8 16:54:50 2019 +0100 updated httpcomponents version --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index 21a1e5e..544409f 100644 --- a/pom.xml +++ b/pom.xml @@ -598,7 +598,7 @@ limitations under the License. org.apache.httpcomponents httpcore -4.4.4 +4.4.12 org.apache.httpcomponents
[nifi-minifi] 03/03: MINIFI-518 Update httpcomponents httpcore to 4.4.12 to resolve issues with those items using the SSLContextBuilder class.
This is an automated email from the ASF dual-hosted git repository. aldrin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi.git commit f94cffbc6bcd845a85ff43c520ece43de7d17f26 Author: jamesb4887 AuthorDate: Tue Oct 8 17:00:04 2019 +0100 MINIFI-518 Update httpcomponents httpcore to 4.4.12 to resolve issues with those items using the SSLContextBuilder class. Signed-off-by: Aldrin Piri --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index 4a869f1..544409f 100644 --- a/pom.xml +++ b/pom.xml @@ -598,7 +598,7 @@ limitations under the License. org.apache.httpcomponents httpcore -4.4.11 +4.4.12 org.apache.httpcomponents
[nifi-minifi] branch master updated (8d26588 -> f94cffb)
This is an automated email from the ASF dual-hosted git repository. aldrin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi.git. from 8d26588 MINIFI-515 Use a repository location with long lived versions for the commons-daemon Windows binary files. new 49115f0 updated httpcomponents version new edb259f updated version new f94cffb MINIFI-518 Update httpcomponents httpcore to 4.4.12 to resolve issues with those items using the SSLContextBuilder class. The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[nifi-minifi] 02/03: updated version
This is an automated email from the ASF dual-hosted git repository. aldrin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi.git commit edb259f3627d267859b4ea9928c90e6ebae62750 Author: jamesb4887 AuthorDate: Tue Oct 8 16:59:09 2019 +0100 updated version --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index 544409f..4a869f1 100644 --- a/pom.xml +++ b/pom.xml @@ -598,7 +598,7 @@ limitations under the License. org.apache.httpcomponents httpcore -4.4.12 +4.4.11 org.apache.httpcomponents
[nifi] branch master updated: NIFI-6733 updating key apache commons dependencies and apache base dependency for build
This is an automated email from the ASF dual-hosted git repository. aldrin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/master by this push: new 0f02de6 NIFI-6733 updating key apache commons dependencies and apache base dependency for build 0f02de6 is described below commit 0f02de60024b7b8db58ef31482307ac1f0134543 Author: Joe Witt AuthorDate: Mon Oct 7 13:26:38 2019 -0400 NIFI-6733 updating key apache commons dependencies and apache base dependency for build This closes #3791. Signed-off-by: Aldrin Piri --- nifi-commons/nifi-expression-language/pom.xml | 2 +- nifi-commons/nifi-flowfile-packager/pom.xml | 6 +++--- nifi-commons/nifi-site-to-site-client/pom.xml | 2 +- nifi-commons/nifi-web-utils/pom.xml | 4 ++-- nifi-external/nifi-storm-spout/pom.xml | 2 +- .../nifi-atlas-bundle/nifi-atlas-reporting-task/pom.xml | 2 +- nifi-nar-bundles/nifi-avro-bundle/nifi-avro-processors/pom.xml | 4 ++-- .../nifi-aws-bundle/nifi-aws-abstract-processors/pom.xml| 2 +- .../nifi-cassandra-bundle/nifi-cassandra-processors/pom.xml | 2 +- nifi-nar-bundles/nifi-cdc/nifi-cdc-api/pom.xml | 2 +- .../nifi-couchbase-bundle/nifi-couchbase-processors/pom.xml | 2 +- .../nifi-elasticsearch-client-service/pom.xml | 2 +- .../nifi-elasticsearch-bundle/nifi-elasticsearch-processors/pom.xml | 2 +- nifi-nar-bundles/nifi-extension-utils/nifi-database-utils/pom.xml | 2 +- nifi-nar-bundles/nifi-extension-utils/nifi-hadoop-utils/pom.xml | 2 +- nifi-nar-bundles/nifi-framework-bundle/pom.xml | 2 +- nifi-nar-bundles/nifi-standard-bundle/pom.xml | 2 +- .../nifi-dbcp-service-bundle/nifi-dbcp-service/pom.xml | 2 +- nifi-toolkit/nifi-toolkit-admin/pom.xml | 2 +- nifi-toolkit/nifi-toolkit-cli/pom.xml | 2 +- nifi-toolkit/nifi-toolkit-encrypt-config/pom.xml| 2 +- pom.xml | 2 +- 22 files changed, 26 insertions(+), 26 deletions(-) diff --git a/nifi-commons/nifi-expression-language/pom.xml b/nifi-commons/nifi-expression-language/pom.xml index 606bd86..9e52f21 100644 --- a/nifi-commons/nifi-expression-language/pom.xml +++ b/nifi-commons/nifi-expression-language/pom.xml @@ -91,7 +91,7 @@ org.apache.commons commons-text -1.6 +1.8 diff --git a/nifi-commons/nifi-flowfile-packager/pom.xml b/nifi-commons/nifi-flowfile-packager/pom.xml index a1b4d5a..9e572b2 100644 --- a/nifi-commons/nifi-flowfile-packager/pom.xml +++ b/nifi-commons/nifi-flowfile-packager/pom.xml @@ -25,17 +25,17 @@ org.apache.commons commons-compress -1.18 +1.19 org.apache.commons commons-lang3 -3.8.1 +3.9 org.apache.commons commons-text -1.6 +1.8 diff --git a/nifi-commons/nifi-site-to-site-client/pom.xml b/nifi-commons/nifi-site-to-site-client/pom.xml index 38399e5..79452a2 100644 --- a/nifi-commons/nifi-site-to-site-client/pom.xml +++ b/nifi-commons/nifi-site-to-site-client/pom.xml @@ -54,7 +54,7 @@ org.apache.commons commons-lang3 -3.8.1 +3.9 com.fasterxml.jackson.core diff --git a/nifi-commons/nifi-web-utils/pom.xml b/nifi-commons/nifi-web-utils/pom.xml index 0d4c260..90fee20 100644 --- a/nifi-commons/nifi-web-utils/pom.xml +++ b/nifi-commons/nifi-web-utils/pom.xml @@ -34,12 +34,12 @@ commons-codec commons-codec -1.12 +1.13 org.apache.commons commons-lang3 -3.8.1 +3.9 org.glassfish.jersey.core diff --git a/nifi-external/nifi-storm-spout/pom.xml b/nifi-external/nifi-storm-spout/pom.xml index 59107da..8b85581 100644 --- a/nifi-external/nifi-storm-spout/pom.xml +++ b/nifi-external/nifi-storm-spout/pom.xml @@ -32,7 +32,7 @@ org.apache.commons commons-lang3 -3.8.1 +3.9 org.apache.nifi diff --git a/nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task/pom.xml b/nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task/pom.xml index 1fcf6ea..8b84ea6 100644 --- a/nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task/pom.xml +++ b/nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting
[nifi-minifi] branch master updated: MINIFI-515 Use a repository location with long lived versions for the commons-daemon Windows binary files.
This is an automated email from the ASF dual-hosted git repository.
aldrin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi.git
The following commit(s) were added to refs/heads/master by this push:
new 8d26588 MINIFI-515 Use a repository location with long lived versions
for the commons-daemon Windows binary files.
8d26588 is described below
commit 8d26588caefacc072a1c86bb0a1ab09fbed56072
Author: Aldrin Piri
AuthorDate: Mon Sep 30 16:06:36 2019 -0400
MINIFI-515 Use a repository location with long lived versions for the
commons-daemon Windows binary files.
---
minifi-assembly/pom.xml | 80 -
1 file changed, 39 insertions(+), 41 deletions(-)
diff --git a/minifi-assembly/pom.xml b/minifi-assembly/pom.xml
index cc90adc..6d5e826 100644
--- a/minifi-assembly/pom.xml
+++ b/minifi-assembly/pom.xml
@@ -281,53 +281,14 @@ limitations under the License.
1.2.1
62777d2424361162736b4a9b39b9fb4155760d4d7c4a3489fe1868e0912158ae126ef5f42abbd1bcca1dbf5aa6204d8d0029be7c49f93d6e1421a5173dd7
+
https://repo1.maven.org/maven2/commons-daemon/commons-daemon/${commons.daemon.version}
-default-assembly
-
-false
-
-
-
-
-maven-assembly-plugin
-
-minifi-${project.version}
-false
-
-
-
-make shared resource
-
-single
-
-package
-
-
-
0775
-0775
-0664
-
-
-
src/main/assembly/dependencies.xml
-
-posix
-
-
-
-
-
-
-
-
windows-service-assembly
true
-
-!default-assembly
-
@@ -359,7 +320,7 @@ limitations under the License.
-https://apache.claz.org/commons/daemon/binaries/windows/${commons.daemon.file};
dest="${java.io.tmpdir}/${commons.daemon.file}" skipexisting="true"/>
+
@@ -407,6 +368,43 @@ limitations under the License.
+
+minimal-assembly
+
+false
+
+
+
+
+maven-assembly-plugin
+
+minifi-${project.version}
+false
+
+
+
+make shared resource
+
+single
+
+package
+
+
+
0775
+0775
+0664
+
+
+
src/main/assembly/dependencies.xml
+
+posix
+
+
+
+
+
+
+
[nifi-minifi] branch master updated: MINIFI-508 Providing a profile, active by default, to download/use the Commons Daemon for running MiNiFi as a Windows Service. Those that wish to exclude these dep
This is an automated email from the ASF dual-hosted git repository.
aldrin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi.git
The following commit(s) were added to refs/heads/master by this push:
new fd99a6c MINIFI-508 Providing a profile, active by default, to
download/use the Commons Daemon for running MiNiFi as a Windows Service. Those
that wish to exclude these dependencies can use the default-assembly profile to
not require external resources.
fd99a6c is described below
commit fd99a6c72fe830b2aee5eb565e33d7c4b762d1c4
Author: Aldrin Piri
AuthorDate: Thu Sep 26 15:39:10 2019 -0400
MINIFI-508 Providing a profile, active by default, to download/use the
Commons Daemon for running MiNiFi as a Windows Service. Those that wish to
exclude these dependencies can use the default-assembly profile to not require
external resources.
---
minifi-assembly/pom.xml| 214 ++---
...encies.xml => dependencies-windows-service.xml} | 0
minifi-assembly/src/main/assembly/dependencies.xml | 10 -
pom.xml| 1 -
4 files changed, 141 insertions(+), 84 deletions(-)
diff --git a/minifi-assembly/pom.xml b/minifi-assembly/pom.xml
index 778e350..cc90adc 100644
--- a/minifi-assembly/pom.xml
+++ b/minifi-assembly/pom.xml
@@ -25,7 +25,21 @@ limitations under the License.
minifi-assembly
pom
This is the assembly of Apache NiFi - MiNiFi
+
+
+
+
+
+
+org.apache.maven.plugins
+maven-assembly-plugin
+3.1.1
+
+
+
org.codehaus.mojo
@@ -50,81 +64,9 @@ limitations under the License.
buildBranch
-
-org.apache.maven.plugins
-maven-antrun-plugin
-
-
-ant-contrib
-ant-contrib
-20020829
-
-
-
-
-download-commons-daemon
-prepare-package
-
-run
-
-
-
-
-
-
-
-
-
-
-
-
-https://apache.claz.org/commons/daemon/binaries/windows/${commons.daemon.file};
dest="${java.io.tmpdir}/${commons.daemon.file}" skipexisting="true"/>
-
-
-
-
-
-
-Checksum error
-
-
-
-
-
-
-
-
-
-
-
-maven-assembly-plugin
-
-minifi-${project.version}
-false
-
-
-
-make shared resource
-
-single
-
-package
-
-
-
0775
-0775
-0664
-
-
-
src/main/assembly/dependencies.xml
-
-posix
-
-
-
-
+
@@ -341,4 +283,130 @@ limitations under the License.
62777d2424361162736b4a9b39b9fb4155760d4d7c4a3489fe1868e0912158ae126ef5f42abbd1bcca1dbf5aa6204d8d0029be7c49f93d6e1421a5173dd7
+
+
+default-assembly
+
+false
+
+
+
+
+maven-assembly-plugin
+
+minifi-${project.version}
+
[nifi-minifi] branch master updated: MINIFI-424 Restructured ConfigTransformer to accept bootstrap properties and created BootstrapTransformer to consolidate future bootstrap use cases
This is an automated email from the ASF dual-hosted git repository. aldrin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi.git The following commit(s) were added to refs/heads/master by this push: new c6f2340 MINIFI-424 Restructured ConfigTransformer to accept bootstrap properties and created BootstrapTransformer to consolidate future bootstrap use cases c6f2340 is described below commit c6f2340a839efde6fadb68e704c3ee962457bc0a Author: GCHQ-NiFi AuthorDate: Mon Aug 19 19:23:25 2019 +0100 MINIFI-424 Restructured ConfigTransformer to accept bootstrap properties and created BootstrapTransformer to consolidate future bootstrap use cases This closes #167 Signed-off-by: Aldrin Piri --- .../apache/nifi/minifi/bootstrap/RunMiNiFi.java| 156 + .../bootstrap/status/reporters/StatusLogger.java | 2 +- .../bootstrap/util/BootstrapTransformer.java | 91 .../minifi/bootstrap/util/ConfigTransformer.java | 35 + .../nifi/minifi/bootstrap/RunMiNiFiTest.java | 11 +- .../bootstrap/util/ConfigTransformerTest.java | 44 -- .../schema/common/BootstrapPropertyKeys.java | 128 + 7 files changed, 262 insertions(+), 205 deletions(-) diff --git a/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/RunMiNiFi.java b/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/RunMiNiFi.java index 93d5f05..5d0951a 100644 --- a/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/RunMiNiFi.java +++ b/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/RunMiNiFi.java @@ -23,10 +23,6 @@ import org.apache.nifi.minifi.bootstrap.configuration.ConfigurationChangeExcepti import org.apache.nifi.minifi.bootstrap.configuration.ConfigurationChangeListener; import org.apache.nifi.minifi.bootstrap.status.PeriodicStatusReporter; import org.apache.nifi.minifi.bootstrap.util.ConfigTransformer; -import org.apache.nifi.minifi.commons.schema.ProvenanceReportingSchema; -import org.apache.nifi.minifi.commons.schema.SecurityPropertiesSchema; -import org.apache.nifi.minifi.commons.schema.SensitivePropsSchema; -import org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys; import org.apache.nifi.minifi.commons.status.FlowStatusReport; import org.apache.nifi.util.Tuple; import org.apache.nifi.util.file.FileUtils; @@ -69,7 +65,6 @@ import java.util.LinkedList; import java.util.List; import java.util.Map; import java.util.Map.Entry; -import java.util.Optional; import java.util.Properties; import java.util.Set; import java.util.concurrent.ExecutorService; @@ -84,11 +79,7 @@ import java.util.concurrent.locks.Lock; import java.util.concurrent.locks.ReentrantLock; import static java.nio.file.StandardCopyOption.REPLACE_EXISTING; -import static org.apache.nifi.minifi.commons.schema.RemoteProcessGroupSchema.TIMEOUT_KEY; -import static org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys.COMMENT_KEY; -import static org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys.SCHEDULING_PERIOD_KEY; -import static org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys.SCHEDULING_STRATEGY_KEY; -import static org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys.USE_COMPRESSION_KEY; +import static org.apache.nifi.minifi.commons.schema.common.BootstrapPropertyKeys.STATUS_REPORTER_COMPONENTS_KEY; /** * @@ -137,96 +128,6 @@ public class RunMiNiFi implements QueryableStatusAggregator, ConfigurationFileHo public static final String DUMP_CMD = "DUMP"; public static final String FLOW_STATUS_REPORT_CMD = "FLOW_STATUS_REPORT"; -public static final String NOTIFIER_PROPERTY_PREFIX = "nifi.minifi.notifier"; -public static final String NOTIFIER_COMPONENTS_KEY = NOTIFIER_PROPERTY_PREFIX + ".components"; - -public static final String STATUS_REPORTER_PROPERTY_PREFIX = "nifi.minifi.status.reporter"; -public static final String STATUS_REPORTER_COMPONENTS_KEY = STATUS_REPORTER_PROPERTY_PREFIX + ".components"; - - -public static final String SECURITY_KEYSTORE_KEY = "nifi.minifi.security.keystore"; -public static final String SECURITY_KEYSTORE_TYPE_KEY = "nifi.minifi.security.keystoreType"; -public static final String SECURITY_KEYSTORE_PASSWORD_KEY = "nifi.minifi.security.keystorePasswd"; -public static final String SECURITY_KEY_PASSWORD_KEY = "nifi.minifi.security.keyPasswd"; -public static final String SECURITY_TRUSTSTORE_KEY = "nifi.minifi.security.truststore"; -public static final String SECURITY_TRUSTSTORE_TYPE_KEY = "nifi.minifi.security.truststoreType"; -public static final String SECURITY_TRUSTSTORE_PASSWORD_KEY = "nifi.minifi.security.truststorePasswd"; -public static final
[nifi-minifi] branch master updated: MINIFI-513 Clear secret key when auto restarting in order to obtain new secret key from the MiNiFi process
This is an automated email from the ASF dual-hosted git repository. aldrin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi.git The following commit(s) were added to refs/heads/master by this push: new 6ba6ce4 MINIFI-513 Clear secret key when auto restarting in order to obtain new secret key from the MiNiFi process 6ba6ce4 is described below commit 6ba6ce474ff6a93b4a337ea5ec1684313c7c3f59 Author: Bryan Bende AuthorDate: Mon Sep 30 15:09:08 2019 -0400 MINIFI-513 Clear secret key when auto restarting in order to obtain new secret key from the MiNiFi process This closes #172. Signed-off-by: Aldrin Piri --- .../src/main/java/org/apache/nifi/minifi/bootstrap/RunMiNiFi.java| 1 + 1 file changed, 1 insertion(+) diff --git a/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/RunMiNiFi.java b/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/RunMiNiFi.java index 2fb4e0b..93d5f05 100644 --- a/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/RunMiNiFi.java +++ b/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/RunMiNiFi.java @@ -1388,6 +1388,7 @@ public class RunMiNiFi implements QueryableStatusAggregator, ConfigurationFileHo setNiFiStarted(false); } +secretKey = null; process = builder.start(); handleLogging(process);
[jira] [Created] (MINIFI-515) Make use of new location for commons-daemon
Aldrin Piri created MINIFI-515: -- Summary: Make use of new location for commons-daemon Key: MINIFI-515 URL: https://issues.apache.org/jira/browse/MINIFI-515 Project: Apache NiFi MiNiFi Issue Type: Improvement Reporter: Aldrin Piri Assignee: Aldrin Piri Fix For: 0.6.0 Related to MINIFI-508 and MINIFI-514. Plan to make use of a mirrored repository to have a consistent location and not make builds so fragile when releases occur. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[nifi-minifi-cpp] branch master updated: MINIFICPP-1041: Update JSON header for the connection queues
This is an automated email from the ASF dual-hosted git repository.
aldrin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new ac45201 MINIFICPP-1041: Update JSON header for the connection queues
ac45201 is described below
commit ac45201b68fd1c16ee2395b5faa5f39c44242a92
Author: Marc Parisi
AuthorDate: Mon Sep 23 12:59:35 2019 -0400
MINIFICPP-1041: Update JSON header for the connection queues
This closes #652.
Signed-off-by: Aldrin Piri
---
libminifi/include/c2/protocols/RESTProtocol.h | 11 ---
libminifi/src/c2/protocols/RESTProtocol.cpp | 43 +++
2 files changed, 45 insertions(+), 9 deletions(-)
diff --git a/libminifi/include/c2/protocols/RESTProtocol.h
b/libminifi/include/c2/protocols/RESTProtocol.h
index 6d848ec..d54e8f0 100644
--- a/libminifi/include/c2/protocols/RESTProtocol.h
+++ b/libminifi/include/c2/protocols/RESTProtocol.h
@@ -18,11 +18,8 @@
#ifndef LIBMINIFI_INCLUDE_C2_PROTOCOLS_RESTPROTOCOL_H_
#define LIBMINIFI_INCLUDE_C2_PROTOCOLS_RESTPROTOCOL_H_
-
-
#include
-
#ifdef RAPIDJSON_ASSERT
#undef RAPIDJSON_ASSERT
#endif
@@ -58,7 +55,7 @@ namespace c2 {
*
*/
-struct ValueObject{
+struct ValueObject {
std::string name;
std::vector values;
};
@@ -80,6 +77,12 @@ class RESTProtocol {
virtual rapidjson::Value serializeJsonPayload(const C2Payload ,
rapidjson::Document::AllocatorType );
+ /**
+ * connection queues should have the uuid as the object name; however since
we have an internal AST and don't want to
+ * impact backwards copmatibility ( where the object root is the name ),
then we should serialize the queues differently.
+ */
+ virtual rapidjson::Value serializeConnectionQueues(const C2Payload ,
std::string , rapidjson::Document::AllocatorType );
+
virtual std::string serializeJsonRootPayload(const C2Payload& payload);
virtual void mergePayloadContent(rapidjson::Value , const C2Payload
, rapidjson::Document::AllocatorType );
diff --git a/libminifi/src/c2/protocols/RESTProtocol.cpp
b/libminifi/src/c2/protocols/RESTProtocol.cpp
index fad4d67..37a53e6 100644
--- a/libminifi/src/c2/protocols/RESTProtocol.cpp
+++ b/libminifi/src/c2/protocols/RESTProtocol.cpp
@@ -311,19 +311,52 @@ bool RESTProtocol::containsPayload(const C2Payload ) {
return false;
}
+rapidjson::Value RESTProtocol::serializeConnectionQueues(const C2Payload
, std::string , rapidjson::Document::AllocatorType ) {
+ rapidjson::Value json_payload(payload.isContainer() ? rapidjson::kArrayType
: rapidjson::kObjectType);
+
+ C2Payload adjusted(payload.getOperation(), payload.getIdentifier(), false,
payload.isRaw());
+
+ auto name = payload.getLabel();
+ std::string uuid;
+ C2ContentResponse updatedContent(payload.getOperation());
+ for (const C2ContentResponse : payload.getContent()) {
+for (const auto& op_arg : content.operation_arguments) {
+ if (op_arg.first == "uuid") {
+uuid = op_arg.second.to_string();
+ }
+ updatedContent.operation_arguments.insert(op_arg);
+}
+ }
+ updatedContent.name = uuid;
+ adjusted.setLabel(uuid);
+ adjusted.setIdentifier(uuid);
+ state::response::ValueNode nd;
+ // name should be what was previously the TLN ( top level node )
+ nd = name;
+ updatedContent.operation_arguments.insert(std::make_pair("name", nd));
+ // the rvalue reference is an unfortunate side effect of the underlying API
decision.
+ adjusted.addContent(std::move(updatedContent), true);
+ mergePayloadContent(json_payload, adjusted, alloc);
+ label = uuid;
+ return json_payload;
+}
+
rapidjson::Value RESTProtocol::serializeJsonPayload(const C2Payload ,
rapidjson::Document::AllocatorType ) {
-// get the name from the content
+ // get the name from the content
rapidjson::Value json_payload(payload.isContainer() ? rapidjson::kArrayType
: rapidjson::kObjectType);
std::vector children;
+ bool isQueue = payload.getLabel() == "queues";
+
for (const auto _payload : payload.getNestedPayloads()) {
-rapidjson::Value* child_payload = new
rapidjson::Value(serializeJsonPayload(nested_payload, alloc));
+std::string label = nested_payload.getLabel();
+rapidjson::Value* child_payload = new rapidjson::Value(isQueue ?
serializeConnectionQueues(nested_payload, label, alloc) :
serializeJsonPayload(nested_payload, alloc));
if (nested_payload.isCollapsible()) {
bool combine = false;
for (auto : children) {
-if (subordinate.name == nested_payload.getLabel()) {
+if (subordinate.name == label) {
subordinate.values.push_back(child_payload);
combine = true;
break;
@@ -331,13 +364,13 @@ rapidjson::Value RESTProtocol::serializeJsonPayload(const
C2Payload , ra
}
if (!combine) {
ValueObject obj;
-
[jira] [Updated] (MINIFI-507) Audit MiNiFi dependencies for upgrade
[ https://issues.apache.org/jira/browse/MINIFI-507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri updated MINIFI-507: --- Fix Version/s: 0.6.0 > Audit MiNiFi dependencies for upgrade > - > > Key: MINIFI-507 > URL: https://issues.apache.org/jira/browse/MINIFI-507 > Project: Apache NiFi MiNiFi > Issue Type: Improvement >Reporter: Nathan Gough >Priority: Major > Fix For: 0.6.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Check the MiNiFi project for dependencies that need to be upgraded. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (MINIFI-507) Audit MiNiFi dependencies for upgrade
[ https://issues.apache.org/jira/browse/MINIFI-507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri resolved MINIFI-507. Assignee: Nathan Gough Resolution: Fixed > Audit MiNiFi dependencies for upgrade > - > > Key: MINIFI-507 > URL: https://issues.apache.org/jira/browse/MINIFI-507 > Project: Apache NiFi MiNiFi > Issue Type: Improvement >Reporter: Nathan Gough >Assignee: Nathan Gough >Priority: Major > Fix For: 0.6.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Check the MiNiFi project for dependencies that need to be upgraded. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[nifi-minifi] branch master updated: MINIFI-507 - Upgraded jetty, jackson-databind, zookeeper, h2database, spring and spring-security
This is an automated email from the ASF dual-hosted git repository.
aldrin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi.git
The following commit(s) were added to refs/heads/master by this push:
new bb40bc9 MINIFI-507 - Upgraded jetty, jackson-databind, zookeeper,
h2database, spring and spring-security
bb40bc9 is described below
commit bb40bc9dc23e864849ca015e8155d83dd0708336
Author: thenatog
AuthorDate: Wed Aug 28 14:57:46 2019 -0400
MINIFI-507 - Upgraded jetty, jackson-databind, zookeeper, h2database,
spring and spring-security
MINIFI-507 - Upgrading jersey
MINIFI-507 - Updated okhttp, and in turn had to update the keystores and
truststores to use a 'DNS:localhost' SAN in order for tests to pass.
This closes #169.
Signed-off-by: Aldrin Piri
---
.../ingestors/PullHttpChangeIngestorSSLTest.java | 2 +-
minifi-bootstrap/src/test/resources/localhost-ks.jks | Bin 3512 -> 3517 bytes
minifi-bootstrap/src/test/resources/localhost-ts.jks | Bin 1816 -> 1820 bytes
.../minifi-framework/minifi-framework-core/pom.xml| 14 +++---
pom.xml | 16
5 files changed, 16 insertions(+), 16 deletions(-)
diff --git
a/minifi-bootstrap/src/test/java/org/apache/nifi/minifi/bootstrap/configuration/ingestors/PullHttpChangeIngestorSSLTest.java
b/minifi-bootstrap/src/test/java/org/apache/nifi/minifi/bootstrap/configuration/ingestors/PullHttpChangeIngestorSSLTest.java
index 470b380..8fe9004 100644
---
a/minifi-bootstrap/src/test/java/org/apache/nifi/minifi/bootstrap/configuration/ingestors/PullHttpChangeIngestorSSLTest.java
+++
b/minifi-bootstrap/src/test/java/org/apache/nifi/minifi/bootstrap/configuration/ingestors/PullHttpChangeIngestorSSLTest.java
@@ -32,7 +32,7 @@ public class PullHttpChangeIngestorSSLTest extends
PullHttpChangeIngestorCommonT
public static void setUp() throws Exception {
PullHttpChangeIngestorCommonTest.init();
-SslContextFactory ssl = new SslContextFactory();
+SslContextFactory.Server ssl = new SslContextFactory.Server();
ssl.setKeyStorePath("./src/test/resources/localhost-ks.jks");
ssl.setKeyStorePassword("localtest");
diff --git a/minifi-bootstrap/src/test/resources/localhost-ks.jks
b/minifi-bootstrap/src/test/resources/localhost-ks.jks
index df36197..1e3989f 100755
Binary files a/minifi-bootstrap/src/test/resources/localhost-ks.jks and
b/minifi-bootstrap/src/test/resources/localhost-ks.jks differ
diff --git a/minifi-bootstrap/src/test/resources/localhost-ts.jks
b/minifi-bootstrap/src/test/resources/localhost-ts.jks
index 7824378..04d2a6f 100755
Binary files a/minifi-bootstrap/src/test/resources/localhost-ts.jks and
b/minifi-bootstrap/src/test/resources/localhost-ts.jks differ
diff --git
a/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/pom.xml
b/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/pom.xml
index efa3c67..202ae29 100644
---
a/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/pom.xml
+++
b/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/pom.xml
@@ -29,7 +29,7 @@ limitations under the License.
org.eclipse.jetty
jetty-server
-9.3.9.v20160517
+${jetty.version}
@@ -40,37 +40,37 @@ limitations under the License.
org.eclipse.jetty
jetty-servlet
-9.3.9.v20160517
+${jetty.version}
org.eclipse.jetty
jetty-webapp
-9.3.9.v20160517
+${jetty.version}
org.eclipse.jetty
jetty-servlets
-9.3.9.v20160517
+${jetty.version}
org.eclipse.jetty
jetty-annotations
-9.3.9.v20160517
+${jetty.version}
org.eclipse.jetty
apache-jsp
-9.3.9.v20160517
+${jetty.version}
org.eclipse.jetty
apache-jstl
-9.3.9.v20160517
+${jetty.version}
diff --git a/pom.xml b/pom.xml
index a6b769e..32b9582 100644
--- a/pom.xml
+++ b/pom.xml
@@ -100,11 +100,11 @@ limitations under the License.
1.7.25
1.8.0
1.2.3
-9.4.11.v20180605
-2.26
+9.4.19.v20190610
+2.29
2.2.0
-4.3.10.RELEASE
-4.2.4.RELEASE
+4.3.25.RELEASE
+4.2.13.RELEASE
1.16.1
1.11.172
@@ -636,7 +636,7 @@ limitations under the License.
[jira] [Commented] (MINIFI-508) Make Windows daemon inclusion a profile
[ https://issues.apache.org/jira/browse/MINIFI-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938782#comment-16938782 ] Aldrin Piri commented on MINIFI-508: As a result of MINIFI-514 it is time to address this. > Make Windows daemon inclusion a profile > --- > > Key: MINIFI-508 > URL: https://issues.apache.org/jira/browse/MINIFI-508 > Project: Apache NiFi MiNiFi > Issue Type: Improvement > Components: Build >Affects Versions: 0.5.0 >Reporter: Aldrin Piri >Assignee: Aldrin Piri >Priority: Major > Fix For: 0.6.0 > > > As the Windows service executable relies on an outside source it could be > beneficial to have this process driven by a profile to mitigate those areas > where users do not care about having that support. The intent would be for > this to still remain in released convenience binaries as part of the RM > process but make it disabled by default -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (MINIFI-514) Update commons-daemon to 1.2.1
Aldrin Piri created MINIFI-514: -- Summary: Update commons-daemon to 1.2.1 Key: MINIFI-514 URL: https://issues.apache.org/jira/browse/MINIFI-514 Project: Apache NiFi MiNiFi Issue Type: Task Affects Versions: 0.5.0 Reporter: Aldrin Piri Assignee: Aldrin Piri Fix For: 0.6.0 Commons-daemon had a bugfix release and we will need to perform another update as the prior version has rolled off many mirrors. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (MINIFI-508) Make Windows daemon inclusion a profile
[ https://issues.apache.org/jira/browse/MINIFI-508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri updated MINIFI-508: --- Fix Version/s: 0.6.0 > Make Windows daemon inclusion a profile > --- > > Key: MINIFI-508 > URL: https://issues.apache.org/jira/browse/MINIFI-508 > Project: Apache NiFi MiNiFi > Issue Type: Improvement > Components: Build >Affects Versions: 0.5.0 >Reporter: Aldrin Piri >Assignee: Aldrin Piri >Priority: Major > Fix For: 0.6.0 > > > As the Windows service executable relies on an outside source it could be > beneficial to have this process driven by a profile to mitigate those areas > where users do not care about having that support. The intent would be for > this to still remain in released convenience binaries as part of the RM > process but make it disabled by default -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (MINIFI-508) Make Windows daemon inclusion a profile
[ https://issues.apache.org/jira/browse/MINIFI-508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri reassigned MINIFI-508: -- Assignee: Aldrin Piri > Make Windows daemon inclusion a profile > --- > > Key: MINIFI-508 > URL: https://issues.apache.org/jira/browse/MINIFI-508 > Project: Apache NiFi MiNiFi > Issue Type: Improvement > Components: Build >Affects Versions: 0.5.0 >Reporter: Aldrin Piri >Assignee: Aldrin Piri >Priority: Major > > As the Windows service executable relies on an outside source it could be > beneficial to have this process driven by a profile to mitigate those areas > where users do not care about having that support. The intent would be for > this to still remain in released convenience binaries as part of the RM > process but make it disabled by default -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (MINIFI-513) CLONE - Updated bootstrap port handling causes restarts to fail
[
https://issues.apache.org/jira/browse/MINIFI-513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938083#comment-16938083
]
Aldrin Piri commented on MINIFI-513:
Introduced into this codebase by MINIFI-512
> CLONE - Updated bootstrap port handling causes restarts to fail
> ---
>
> Key: MINIFI-513
> URL: https://issues.apache.org/jira/browse/MINIFI-513
> Project: Apache NiFi MiNiFi
> Issue Type: Bug
>Reporter: Aldrin Piri
> Assignee: Aldrin Piri
>Priority: Blocker
> Fix For: 0.6.0
>
>
> NIFI-6653 introduced a way to avoid the port being changed to prevent a
> possible hijacking of that port to provide other commands than those
> explicitly between bootstrap and the nifi process. This causes issues when
> the NiFi process dies and precludes restart (the default functionality
> enabled out of the box).
> To recreate, build/run NiFi off of current master and kill the NiFi process.
> This will result in the following (when an additional nifi.sh status is
> carried out).
> {quote}2019-09-25 17:10:55,601 WARN [main] org.apache.nifi.bootstrap.RunNiFi
> Apache NiFi appears to have died. Restarting...
> 2019-09-25 17:10:55,620 INFO [main] org.apache.nifi.bootstrap.Command
> Launched Apache NiFi with Process ID 2088
> 2019-09-25 17:10:55,621 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Successfully started Apache NiFi with PID 2088
> 2019-09-25 17:10:56,174 WARN [NiFi Bootstrap Command Listener]
> org.apache.nifi.bootstrap.RunNiFi Blocking attempt to change NiFi command
> port and secret after they have already been initialized. requestedPort=37871
> 2019-09-25 17:11:50,783 INFO [main] o.a.n.b.NotificationServiceManager
> Successfully loaded the following 0 services: []
> 2019-09-25 17:11:50,785 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services for Notification Type NIFI_STARTED
> 2019-09-25 17:11:50,786 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services for Notification Type NIFI_STOPPED
> 2019-09-25 17:11:50,786 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services for Notification Type NIFI_DIED
> 2019-09-25 17:11:50,809 INFO [main] org.apache.nifi.bootstrap.Command Apache
> NiFi is running at PID 2088 but is not responding to ping requests{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Moved] (MINIFI-513) CLONE - Updated bootstrap port handling causes restarts to fail
[
https://issues.apache.org/jira/browse/MINIFI-513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aldrin Piri moved NIFI-6713 to MINIFI-513:
--
Fix Version/s: (was: 1.10.0)
0.6.0
Key: MINIFI-513 (was: NIFI-6713)
Project: Apache NiFi MiNiFi (was: Apache NiFi)
> CLONE - Updated bootstrap port handling causes restarts to fail
> ---
>
> Key: MINIFI-513
> URL: https://issues.apache.org/jira/browse/MINIFI-513
> Project: Apache NiFi MiNiFi
> Issue Type: Bug
>Reporter: Aldrin Piri
> Assignee: Aldrin Piri
>Priority: Blocker
> Fix For: 0.6.0
>
>
> NIFI-6653 introduced a way to avoid the port being changed to prevent a
> possible hijacking of that port to provide other commands than those
> explicitly between bootstrap and the nifi process. This causes issues when
> the NiFi process dies and precludes restart (the default functionality
> enabled out of the box).
> To recreate, build/run NiFi off of current master and kill the NiFi process.
> This will result in the following (when an additional nifi.sh status is
> carried out).
> {quote}2019-09-25 17:10:55,601 WARN [main] org.apache.nifi.bootstrap.RunNiFi
> Apache NiFi appears to have died. Restarting...
> 2019-09-25 17:10:55,620 INFO [main] org.apache.nifi.bootstrap.Command
> Launched Apache NiFi with Process ID 2088
> 2019-09-25 17:10:55,621 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Successfully started Apache NiFi with PID 2088
> 2019-09-25 17:10:56,174 WARN [NiFi Bootstrap Command Listener]
> org.apache.nifi.bootstrap.RunNiFi Blocking attempt to change NiFi command
> port and secret after they have already been initialized. requestedPort=37871
> 2019-09-25 17:11:50,783 INFO [main] o.a.n.b.NotificationServiceManager
> Successfully loaded the following 0 services: []
> 2019-09-25 17:11:50,785 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services for Notification Type NIFI_STARTED
> 2019-09-25 17:11:50,786 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services for Notification Type NIFI_STOPPED
> 2019-09-25 17:11:50,786 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services for Notification Type NIFI_DIED
> 2019-09-25 17:11:50,809 INFO [main] org.apache.nifi.bootstrap.Command Apache
> NiFi is running at PID 2088 but is not responding to ping requests{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[nifi-minifi] branch master updated: MINIFI-478 Fix broken OrderedProperties and OrderedPropertiesTest
This is an automated email from the ASF dual-hosted git repository.
aldrin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi.git
The following commit(s) were added to refs/heads/master by this push:
new 11e6395 MINIFI-478 Fix broken OrderedProperties and
OrderedPropertiesTest
11e6395 is described below
commit 11e6395dee978609ec88e017f7bbb1632101a8af
Author: Nenad V. Nikolić
AuthorDate: Tue Jul 9 08:34:56 2019 +0200
MINIFI-478 Fix broken OrderedProperties and OrderedPropertiesTest
In addition:
- a more self-descriptive private variable name was used
- the test data set was made bigger (and more "robust").
This closes #158.
Signed-off-by: Aldrin Piri
---
.../nifi/minifi/bootstrap/util/OrderedProperties.java | 15 ---
.../nifi/minifi/bootstrap/util/OrderedPropertiesTest.java | 11 ---
.../test/resources/orderedPropertiesExpected.properties | 5 -
3 files changed, 24 insertions(+), 7 deletions(-)
diff --git
a/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/util/OrderedProperties.java
b/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/util/OrderedProperties.java
index d7496fa..5d5fe51 100644
---
a/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/util/OrderedProperties.java
+++
b/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/util/OrderedProperties.java
@@ -25,27 +25,36 @@ import java.io.IOException;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.io.OutputStreamWriter;
+import java.util.AbstractMap;
import java.util.Collections;
import java.util.Enumeration;
import java.util.HashMap;
import java.util.LinkedHashSet;
import java.util.Map;
import java.util.Properties;
+import java.util.Set;
import java.util.stream.Collectors;
public class OrderedProperties extends Properties {
private final Map textBeforeMap = new HashMap<>();
-private final LinkedHashSet linkedHashSet = new LinkedHashSet<>();
+private final LinkedHashSet orderedKeys = new LinkedHashSet<>();
@Override
public synchronized Object put(Object key, Object value) {
-linkedHashSet.add(key);
+orderedKeys.add(key);
return super.put(key, value);
}
@Override
public synchronized Enumeration keys() {
-return
Collections.enumeration(linkedHashSet.stream().filter(this::containsKey).collect(Collectors.toList()));
+return
Collections.enumeration(orderedKeys.stream().filter(this::containsKey).collect(Collectors.toList()));
+}
+
+@Override
+public synchronized Set> entrySet() {
+return orderedKeys.stream()
+.map(k -> new AbstractMap.SimpleImmutableEntry<>(k,
this.get(k)))
+.collect(Collectors.toCollection(LinkedHashSet::new));
}
public synchronized Object setProperty(String key, String value, String
textBefore) {
diff --git
a/minifi-bootstrap/src/test/java/org/apache/nifi/minifi/bootstrap/util/OrderedPropertiesTest.java
b/minifi-bootstrap/src/test/java/org/apache/nifi/minifi/bootstrap/util/OrderedPropertiesTest.java
index bc9c4bc..11e9de6 100644
---
a/minifi-bootstrap/src/test/java/org/apache/nifi/minifi/bootstrap/util/OrderedPropertiesTest.java
+++
b/minifi-bootstrap/src/test/java/org/apache/nifi/minifi/bootstrap/util/OrderedPropertiesTest.java
@@ -36,20 +36,25 @@ public class OrderedPropertiesTest {
orderedProperties.setProperty("prop1", "origVal1");
orderedProperties.setProperty("prop2", "val2", "#this is property 2");
orderedProperties.setProperty("prop3", "val3");
+orderedProperties.setProperty("prop4", "val4");
+orderedProperties.setProperty("prop5", "val5", "#this is property 5");
+orderedProperties.setProperty("prop3", "newVal3");
orderedProperties.setProperty("prop1", "newVal1");
ByteArrayOutputStream byteArrayOutputStream = new
ByteArrayOutputStream();
orderedProperties.store(byteArrayOutputStream,
PROPERTIES_FILE_APACHE_2_0_LICENSE);
-try (BufferedReader bufferedReader = new BufferedReader(new
InputStreamReader(new
ByteArrayInputStream(byteArrayOutputStream.toByteArray(;
+try (BufferedReader actualReader = new BufferedReader(new
InputStreamReader(new
ByteArrayInputStream(byteArrayOutputStream.toByteArray(;
BufferedReader expectedReader = new BufferedReader(new
InputStreamReader(OrderedPropertiesTest.class.getClassLoader().getResourceAsStream("orderedPropertiesExpected.properties"
{
String expectedLine;
while((expectedLine = expectedReader.readLine()) != null) {
-String actualLine = buf
[nifi-minifi] branch master updated: Provenance reporting config can now be overridden using values set in the bootstrap.conf
This is an automated email from the ASF dual-hosted git repository. aldrin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi.git The following commit(s) were added to refs/heads/master by this push: new 2d6cdca Provenance reporting config can now be overridden using values set in the bootstrap.conf 2d6cdca is described below commit 2d6cdca142b15f83f144116f1ac5744dafc488a4 Author: GCHQ-NiFi AuthorDate: Fri Aug 16 09:44:09 2019 +0100 Provenance reporting config can now be overridden using values set in the bootstrap.conf This closes #166. Signed-off-by: Aldrin Piri --- .../apache/nifi/minifi/bootstrap/RunMiNiFi.java| 62 +- .../minifi/bootstrap/util/ConfigTransformer.java | 19 - .../nifi/minifi/bootstrap/RunMiNiFiTest.java | 27 ++ .../bootstrap.conf.configured | 44 ++ .../bootstrap.conf.default | 18 +++- .../bootstrap-provenance-reporting/config.yml | 97 ++ .../nifi/minifi/commons/schema/ConfigSchema.java | 4 + .../src/main/markdown/System_Admin_Guide.md| 31 --- .../src/main/resources/conf/bootstrap.conf | 10 +++ 9 files changed, 273 insertions(+), 39 deletions(-) diff --git a/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/RunMiNiFi.java b/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/RunMiNiFi.java index 26fbf53..cc59f68 100644 --- a/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/RunMiNiFi.java +++ b/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/RunMiNiFi.java @@ -23,6 +23,7 @@ import org.apache.nifi.minifi.bootstrap.configuration.ConfigurationChangeExcepti import org.apache.nifi.minifi.bootstrap.configuration.ConfigurationChangeListener; import org.apache.nifi.minifi.bootstrap.status.PeriodicStatusReporter; import org.apache.nifi.minifi.bootstrap.util.ConfigTransformer; +import org.apache.nifi.minifi.commons.schema.ProvenanceReportingSchema; import org.apache.nifi.minifi.commons.schema.SecurityPropertiesSchema; import org.apache.nifi.minifi.commons.schema.SensitivePropsSchema; import org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys; @@ -83,6 +84,11 @@ import java.util.concurrent.locks.Lock; import java.util.concurrent.locks.ReentrantLock; import static java.nio.file.StandardCopyOption.REPLACE_EXISTING; +import static org.apache.nifi.minifi.commons.schema.RemoteProcessGroupSchema.TIMEOUT_KEY; +import static org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys.COMMENT_KEY; +import static org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys.SCHEDULING_PERIOD_KEY; +import static org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys.SCHEDULING_STRATEGY_KEY; +import static org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys.USE_COMPRESSION_KEY; /** * @@ -167,6 +173,27 @@ public class RunMiNiFi implements QueryableStatusAggregator, ConfigurationFileHo SENSITIVE_PROPS_ALGORITHM_KEY, SENSITIVE_PROPS_PROVIDER_KEY)); +public static final String PROVENANCE_REPORTING_COMMENT_KEY = "nifi.minifi.provenance.reporting.comment"; +public static final String PROVENANCE_REPORTING_SCHEDULING_STRATEGY_KEY = "nifi.minifi.provenance.reporting.scheduling.strategy"; +public static final String PROVENANCE_REPORTING_SCHEDULING_PERIOD_KEY = "nifi.minifi.provenance.reporting.scheduling.period"; +public static final String PROVENANCE_REPORTING_DESTINATION_URL_KEY = "nifi.minifi.provenance.reporting.destination.url"; +public static final String PROVENANCE_REPORTING_INPUT_PORT_NAME_KEY = "nifi.minifi.provenance.reporting.input.port.name"; +public static final String PROVENANCE_REPORTING_INSTANCE_URL_KEY = "nifi.minifi.provenance.reporting.instance.url"; +public static final String PROVENANCE_REPORTING_COMPRESS_EVENTS_KEY = "nifi.minifi.provenance.reporting.compress.events"; +public static final String PROVENANCE_REPORTING_BATCH_SIZE_KEY = "nifi.minifi.provenance.reporting.batch.size"; +public static final String PROVENANCE_REPORTING_COMMUNICATIONS_TIMEOUT_KEY = "nifi.minifi.provenance.reporting.communications.timeout"; + +public static final Set BOOTSTRAP_PROVENANCE_REPORTING_KEYS = new HashSet<>( +Arrays.asList(PROVENANCE_REPORTING_COMMENT_KEY, +PROVENANCE_REPORTING_SCHEDULING_STRATEGY_KEY, +PROVENANCE_REPORTING_SCHEDULING_PERIOD_KEY, +PROVENANCE_REPORTING_DESTINATION_URL_KEY, +PROVENANCE_REPORTING_INPUT_PORT_NAME_KEY, +PROVENANCE_REPORTING_INSTANCE_URL_KEY, +PROVENANCE_REPORTING_COMPRESS_EVENTS_KEY, +PROVENANC
[jira] [Assigned] (MINIFI-511) Allow Provenance Reporting to be configured outside of the config.yml
[ https://issues.apache.org/jira/browse/MINIFI-511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri reassigned MINIFI-511: -- Assignee: GCHQ NiFi > Allow Provenance Reporting to be configured outside of the config.yml > - > > Key: MINIFI-511 > URL: https://issues.apache.org/jira/browse/MINIFI-511 > Project: Apache NiFi MiNiFi > Issue Type: Improvement > Components: Agent Configuration/Installation >Reporter: GCHQ NiFi >Assignee: GCHQ NiFi >Priority: Major > Fix For: 0.6.0 > > Time Spent: 20m > Remaining Estimate: 0h > > When using the MiNiFi config change notifiers, it's not currently possible to > set the provenance reporting properties. > MINIFI-501 allowed for the SSL context properties to be set in the bootstrap, > so it makes sense for the provenance reporting to be set in the same way. > I've created a fork with the changes completed already, so will link this > ticket to a pull request. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[nifi-minifi] branch master updated: MINIFI-509 S2S security properties are picked up from bootstrap.conf
This is an automated email from the ASF dual-hosted git repository.
aldrin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi.git
The following commit(s) were added to refs/heads/master by this push:
new 3528af2 MINIFI-509 S2S security properties are picked up from
bootstrap.conf
3528af2 is described below
commit 3528af24b622f7f9a911d6f06921461d1a8dda8d
Author: Ferenc Kis
AuthorDate: Tue Aug 13 15:29:11 2019 +0200
MINIFI-509 S2S security properties are picked up from bootstrap.conf
This closes #164.
Signed-off-by: Aldrin Piri
---
.../java/org/apache/nifi/minifi/bootstrap/RunMiNiFi.java | 15 ++-
.../org/apache/nifi/minifi/bootstrap/RunMiNiFiTest.java | 9 ++---
2 files changed, 16 insertions(+), 8 deletions(-)
diff --git
a/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/RunMiNiFi.java
b/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/RunMiNiFi.java
index e7ac3ed..26fbf53 100644
---
a/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/RunMiNiFi.java
+++
b/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/RunMiNiFi.java
@@ -1687,7 +1687,7 @@ public class RunMiNiFi implements
QueryableStatusAggregator, ConfigurationFileHo
newConfigBais.reset();
logger.info("Performing transformation for input
and saving outputs to {}", confDir);
-ByteBuffer tempConfigFile =
performTransformation(newConfigBais, confDir);
+ByteBuffer tempConfigFile =
runner.performTransformation(newConfigBais, confDir);
runner.currentConfigFileReference.set(tempConfigFile.asReadOnlyBuffer());
try {
@@ -1695,7 +1695,7 @@ public class RunMiNiFi implements
QueryableStatusAggregator, ConfigurationFileHo
restartInstance();
} catch (Exception e) {
logger.debug("Transformation of new config
file failed after transformation into Flow.xml and nifi.properties,
reverting.");
-ByteBuffer resetConfigFile =
performTransformation(new FileInputStream(swapConfigFile), confDir);
+ByteBuffer resetConfigFile =
runner.performTransformation(new FileInputStream(swapConfigFile), confDir);
runner.currentConfigFileReference.set(resetConfigFile.asReadOnlyBuffer());
throw e;
}
@@ -1758,11 +1758,15 @@ public class RunMiNiFi implements
QueryableStatusAggregator, ConfigurationFileHo
}
}
-private static ByteBuffer performTransformation(InputStream configIs,
String configDestinationPath) throws ConfigurationChangeException, IOException {
+private ByteBuffer performTransformation(InputStream configIs, String
configDestinationPath) throws ConfigurationChangeException, IOException {
try (ByteArrayOutputStream byteArrayOutputStream = new
ByteArrayOutputStream();
TeeInputStream teeInputStream = new TeeInputStream(configIs,
byteArrayOutputStream)) {
-ConfigTransformer.transformConfigFile(teeInputStream,
configDestinationPath);
+ConfigTransformer.transformConfigFile(
+teeInputStream,
+configDestinationPath,
+
buildSecurityPropertiesFromBootstrap(getBootstrapProperties()).orElse(null)
+);
return ByteBuffer.wrap(byteArrayOutputStream.toByteArray());
} catch (ConfigurationChangeException e){
@@ -1772,7 +1776,8 @@ public class RunMiNiFi implements
QueryableStatusAggregator, ConfigurationFileHo
}
}
-public static Optional
buildSecurityPropertiesFromBootstrap(final Properties bootstrapProperties) {
+// TODO extract this to separate class BootstrapTransformer, and make
private
+public Optional
buildSecurityPropertiesFromBootstrap(final Properties bootstrapProperties) {
Optional securityPropsOptional =
Optional.empty();
diff --git
a/minifi-bootstrap/src/test/java/org/apache/nifi/minifi/bootstrap/RunMiNiFiTest.java
b/minifi-bootstrap/src/test/java/org/apache/nifi/minifi/bootstrap/RunMiNiFiTest.java
index 055bab2..f6e0cb6 100644
---
a/minifi-bootstrap/src/test/java/org/apache/nifi/minifi/bootstrap/RunMiNiFiTest.java
+++
b/minifi-bootstrap/src/test/java/org/apache/nifi/minifi/bootstrap/RunMiNiFiTest.java
@@ -30,15 +30,17 @@ public class RunMiNiFiTest {
@Test
public void buildSecurityPropertiesNotDefined() throws Exception {
+final RunMiNiFi testMiNiFi = new RunMiNiFi(null);
final Properties bootstrapProperties =
getTestBootstrapProperties("bootstrap-ssl-ctx/bootstrap.conf.default");
-final
[jira] [Updated] (MINIFI-509) SSL Context defined in bootstrap Not Working?
[
https://issues.apache.org/jira/browse/MINIFI-509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aldrin Piri updated MINIFI-509:
---
Fix Version/s: 0.6.0
> SSL Context defined in bootstrap Not Working?
> -
>
> Key: MINIFI-509
> URL: https://issues.apache.org/jira/browse/MINIFI-509
> Project: Apache NiFi MiNiFi
> Issue Type: Bug
>Reporter: Ryan Whittington
>Assignee: Ferenc Kis
>Priority: Major
> Fix For: 0.6.0
>
>
> I'm trying to set my SSL-Context-Service properties using the bootstrap.conf,
> but it doesn't seem to propagate through to the flow.xml.gz when minifi
> starts/reloads - and isn't available to use in my flow?
> As this feature was recently added in MINIFI-501 by [~aldrin], I'm building
> minifi from the master branch on github.
> bootstrap.conf:
>
> {code:java}
> # Security Properties #
> # These properties take precedence over any equivalent properties specified
> in config.yml #
> nifi.minifi.security.keystore=/my/test/keystore.jks
> nifi.minifi.security.keystoreType=JKS
> nifi.minifi.security.keystorePasswd=mykeystorepassword
> nifi.minifi.security.keyPasswd=mykeypassword
> nifi.minifi.security.truststore=/my/test/truststore.jks
> nifi.minifi.security.truststoreType=JKS
> nifi.minifi.security.truststorePasswd=mytruststorepassword
> nifi.minifi.security.ssl.protocol=TLS
> {code}
>
> flow.xml.gz once running:
> {noformat}
>
>
> 1
> 1
>
>
> Root-Group
> MiNiFi Flow
>
>
>
>
>
> {noformat}
> Am I doing something wrong, or is this not fully implemented yet?
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Assigned] (MINIFI-509) SSL Context defined in bootstrap Not Working?
[
https://issues.apache.org/jira/browse/MINIFI-509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aldrin Piri reassigned MINIFI-509:
--
Assignee: Ferenc Kis (was: Aldrin Piri)
> SSL Context defined in bootstrap Not Working?
> -
>
> Key: MINIFI-509
> URL: https://issues.apache.org/jira/browse/MINIFI-509
> Project: Apache NiFi MiNiFi
> Issue Type: Bug
>Reporter: Ryan Whittington
>Assignee: Ferenc Kis
>Priority: Major
>
> I'm trying to set my SSL-Context-Service properties using the bootstrap.conf,
> but it doesn't seem to propagate through to the flow.xml.gz when minifi
> starts/reloads - and isn't available to use in my flow?
> As this feature was recently added in MINIFI-501 by [~aldrin], I'm building
> minifi from the master branch on github.
> bootstrap.conf:
>
> {code:java}
> # Security Properties #
> # These properties take precedence over any equivalent properties specified
> in config.yml #
> nifi.minifi.security.keystore=/my/test/keystore.jks
> nifi.minifi.security.keystoreType=JKS
> nifi.minifi.security.keystorePasswd=mykeystorepassword
> nifi.minifi.security.keyPasswd=mykeypassword
> nifi.minifi.security.truststore=/my/test/truststore.jks
> nifi.minifi.security.truststoreType=JKS
> nifi.minifi.security.truststorePasswd=mytruststorepassword
> nifi.minifi.security.ssl.protocol=TLS
> {code}
>
> flow.xml.gz once running:
> {noformat}
>
>
> 1
> 1
>
>
> Root-Group
> MiNiFi Flow
>
>
>
>
>
> {noformat}
> Am I doing something wrong, or is this not fully implemented yet?
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (MINIFI-510) Migrate Travis away from OracleJDK8 which seems to be problematic
[ https://issues.apache.org/jira/browse/MINIFI-510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906366#comment-16906366 ] Aldrin Piri commented on MINIFI-510: Example build issue: https://travis-ci.org/apache/nifi-minifi/jobs/571358468 > Migrate Travis away from OracleJDK8 which seems to be problematic > - > > Key: MINIFI-510 > URL: https://issues.apache.org/jira/browse/MINIFI-510 > Project: Apache NiFi MiNiFi > Issue Type: Task >Reporter: Aldrin Piri > Assignee: Aldrin Piri >Priority: Major > Fix For: 0.6.0 > > > It seems configuration of Travis has changed such that Oracle JDK 8 is no > longer available. We should migrate to supported versions of 8. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (MINIFI-510) Migrate Travis away from OracleJDK8 which seems to be problematic
Aldrin Piri created MINIFI-510: -- Summary: Migrate Travis away from OracleJDK8 which seems to be problematic Key: MINIFI-510 URL: https://issues.apache.org/jira/browse/MINIFI-510 Project: Apache NiFi MiNiFi Issue Type: Task Reporter: Aldrin Piri Assignee: Aldrin Piri Fix For: 0.6.0 It seems configuration of Travis has changed such that Oracle JDK 8 is no longer available. We should migrate to supported versions of 8. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (MINIFI-509) SSL Context defined in bootstrap Not Working?
[
https://issues.apache.org/jira/browse/MINIFI-509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906324#comment-16906324
]
Aldrin Piri commented on MINIFI-509:
https://github.com/apache/nifi-minifi/pull/164
> SSL Context defined in bootstrap Not Working?
> -
>
> Key: MINIFI-509
> URL: https://issues.apache.org/jira/browse/MINIFI-509
> Project: Apache NiFi MiNiFi
> Issue Type: Bug
>Reporter: Ryan Whittington
> Assignee: Aldrin Piri
>Priority: Major
>
> I'm trying to set my SSL-Context-Service properties using the bootstrap.conf,
> but it doesn't seem to propagate through to the flow.xml.gz when minifi
> starts/reloads - and isn't available to use in my flow?
> As this feature was recently added in MINIFI-501 by [~aldrin], I'm building
> minifi from the master branch on github.
> bootstrap.conf:
>
> {code:java}
> # Security Properties #
> # These properties take precedence over any equivalent properties specified
> in config.yml #
> nifi.minifi.security.keystore=/my/test/keystore.jks
> nifi.minifi.security.keystoreType=JKS
> nifi.minifi.security.keystorePasswd=mykeystorepassword
> nifi.minifi.security.keyPasswd=mykeypassword
> nifi.minifi.security.truststore=/my/test/truststore.jks
> nifi.minifi.security.truststoreType=JKS
> nifi.minifi.security.truststorePasswd=mytruststorepassword
> nifi.minifi.security.ssl.protocol=TLS
> {code}
>
> flow.xml.gz once running:
> {noformat}
>
>
> 1
> 1
>
>
> Root-Group
> MiNiFi Flow
>
>
>
>
>
> {noformat}
> Am I doing something wrong, or is this not fully implemented yet?
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[nifi] branch master updated: NIFI-6519 GrokReader does not evaluate EL while validating "Grok Pattern File" property
This is an automated email from the ASF dual-hosted git repository. aldrin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/master by this push: new f851a9e NIFI-6519 GrokReader does not evaluate EL while validating "Grok Pattern File" property f851a9e is described below commit f851a9e96fa72cb7fe8a5a2bcb177e3e59ec1f61 Author: Ferenc Szabó AuthorDate: Fri Aug 2 14:03:38 2019 +0200 NIFI-6519 GrokReader does not evaluate EL while validating "Grok Pattern File" property added evaluateAttributeExpressions call where it was missing. tested manually This closes #3627 Signed-off-by: Aldrin Piri --- .../src/main/java/org/apache/nifi/grok/GrokReader.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/grok/GrokReader.java b/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/grok/GrokReader.java index a462c72..099da2a 100644 --- a/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/grok/GrokReader.java +++ b/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/grok/GrokReader.java @@ -175,7 +175,7 @@ public class GrokReader extends SchemaRegistryService implements RecordReaderFac .build()); } -validator = new GrokExpressionValidator(validationContext.getProperty(PATTERN_FILE).getValue(),grokCompiler); +validator = new GrokExpressionValidator(validationContext.getProperty(PATTERN_FILE).evaluateAttributeExpressions().getValue(),grokCompiler); results.add(validator.validate(subject,input,validationContext)); return results; }
[jira] [Commented] (MINIFI-509) SSL Context defined in bootstrap Not Working?
[
https://issues.apache.org/jira/browse/MINIFI-509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16898838#comment-16898838
]
Aldrin Piri commented on MINIFI-509:
Hi there,
Thanks for reporting this. It is certainly not you and believe something may
have been blundered with the merge.
> SSL Context defined in bootstrap Not Working?
> -
>
> Key: MINIFI-509
> URL: https://issues.apache.org/jira/browse/MINIFI-509
> Project: Apache NiFi MiNiFi
> Issue Type: Bug
>Reporter: Ryan Whittington
> Assignee: Aldrin Piri
>Priority: Major
>
> I'm trying to set my SSL-Context-Service properties using the bootstrap.conf,
> but it doesn't seem to propagate through to the flow.xml.gz when minifi
> starts/reloads - and isn't available to use in my flow?
> As this feature was recently added in MINIFI-501 by [~aldrin], I'm building
> minifi from the master branch on github.
> bootstrap.conf:
>
> {code:java}
> # Security Properties #
> # These properties take precedence over any equivalent properties specified
> in config.yml #
> nifi.minifi.security.keystore=/my/test/keystore.jks
> nifi.minifi.security.keystoreType=JKS
> nifi.minifi.security.keystorePasswd=mykeystorepassword
> nifi.minifi.security.keyPasswd=mykeypassword
> nifi.minifi.security.truststore=/my/test/truststore.jks
> nifi.minifi.security.truststoreType=JKS
> nifi.minifi.security.truststorePasswd=mytruststorepassword
> nifi.minifi.security.ssl.protocol=TLS
> {code}
>
> flow.xml.gz once running:
> {noformat}
>
>
> 1
> 1
>
>
> Root-Group
> MiNiFi Flow
>
>
>
>
>
> {noformat}
> Am I doing something wrong, or is this not fully implemented yet?
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Assigned] (MINIFI-509) SSL Context defined in bootstrap Not Working?
[
https://issues.apache.org/jira/browse/MINIFI-509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aldrin Piri reassigned MINIFI-509:
--
Assignee: Aldrin Piri
> SSL Context defined in bootstrap Not Working?
> -
>
> Key: MINIFI-509
> URL: https://issues.apache.org/jira/browse/MINIFI-509
> Project: Apache NiFi MiNiFi
> Issue Type: Bug
>Reporter: Ryan Whittington
> Assignee: Aldrin Piri
>Priority: Major
>
> I'm trying to set my SSL-Context-Service properties using the bootstrap.conf,
> but it doesn't seem to propagate through to the flow.xml.gz when minifi
> starts/reloads - and isn't available to use in my flow?
> As this feature was recently added in MINIFI-501 by [~aldrin], I'm building
> minifi from the master branch on github.
> bootstrap.conf:
>
> {code:java}
> # Security Properties #
> # These properties take precedence over any equivalent properties specified
> in config.yml #
> nifi.minifi.security.keystore=/my/test/keystore.jks
> nifi.minifi.security.keystoreType=JKS
> nifi.minifi.security.keystorePasswd=mykeystorepassword
> nifi.minifi.security.keyPasswd=mykeypassword
> nifi.minifi.security.truststore=/my/test/truststore.jks
> nifi.minifi.security.truststoreType=JKS
> nifi.minifi.security.truststorePasswd=mytruststorepassword
> nifi.minifi.security.ssl.protocol=TLS
> {code}
>
> flow.xml.gz once running:
> {noformat}
>
>
> 1
> 1
>
>
> Root-Group
> MiNiFi Flow
>
>
>
>
>
> {noformat}
> Am I doing something wrong, or is this not fully implemented yet?
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[nifi-minifi-cpp] branch master updated: MINIFICPP-793: Allow SSL Context to be defined from properties by default without specifying manually. This will ease C2 usage of SSL RESTFul services. Note th
This is an automated email from the ASF dual-hosted git repository.
aldrin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 6789201 MINIFICPP-793: Allow SSL Context to be defined from
properties by default without specifying manually. This will ease C2 usage of
SSL RESTFul services. Note that Civetweb needed some changes to support SSL
contexts in testing.
6789201 is described below
commit 678920114a18890879c8069f7da4a18dedc2460d
Author: Marc Parisi
AuthorDate: Wed Mar 27 13:26:11 2019 -0400
MINIFICPP-793: Allow SSL Context to be defined from properties
by default without specifying manually. This will ease C2 usage
of SSL RESTFul services. Note that Civetweb needed some changes
to support SSL contexts in testing.
Change appveyor to use provided libressl
MINIFICPP-793: Fix appveyor by bringing fixes from windows test enablement
This closes #528.
Signed-off-by: Aldrin Piri
---
CMakeLists.txt | 21 ---
appveyor.yml | 8 +--
cmake/LibreSSL.cmake | 6 +-
darwin.sh | 2 +-
extensions/http-curl/protocols/RESTSender.cpp | 15 -
extensions/http-curl/protocols/RESTSender.h| 8 +++
.../http-curl/tests/C2VerifyHeartbeatAndStop.cpp | 7 ++-
.../http-curl/tests/C2VerifyServeResults.cpp | 9 ++-
extensions/http-curl/tests/CMakeLists.txt | 5 +-
extensions/http-curl/tests/HTTPIntegrationBase.h | 11 ++--
.../http-curl/tests/HttpGetIntegrationTest.cpp | 7 ++-
extensions/http-curl/tests/TestServer.h| 20 --
.../resources/C2VerifyHeartbeatAndStopSecure.yml | 73 ++
.../test/resources/C2VerifyServeResultsSecure.yml | 73 ++
14 files changed, 226 insertions(+), 39 deletions(-)
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 006107a..d37ebf4 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -170,7 +170,7 @@ if (WIN32)
endif()
# Provide custom modules for the project
-list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake")
+list(APPEND CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/cmake")
if (NOT OPENSSL_OFF)
if(USE_SYSTEM_OPENSSL)
@@ -185,7 +185,7 @@ if (NOT OPENSSL_OFF)
else()
include(LibreSSL)
use_libre_ssl(${CMAKE_CURRENT_SOURCE_DIR}
${CMAKE_CURRENT_BINARY_DIR})
- list(APPEND CMAKE_MODULE_PATH
"${CMAKE_CURRENT_SOURCE_DIR}/cmake/ssl")
+ list(APPEND CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/cmake/ssl")
endif()
find_package (OpenSSL REQUIRED)
@@ -200,7 +200,7 @@ if (NOT OPENSSL_OFF)
endif (OPENSSL_FOUND)
else()
- list(APPEND CMAKE_MODULE_PATH
"${CMAKE_CURRENT_SOURCE_DIR}/cmake/winssl")
+ list(APPEND CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/cmake/winssl")
endif()
if (OPENSSL_ROOT_DIR )
@@ -244,7 +244,7 @@ SET(TEST_DIR ${CMAKE_SOURCE_DIR}/libminifi/test)
include(Extensions)
-list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake/uuid/dummy")
+list(APPEND CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/cmake/uuid/dummy")
if (WIN32)
include_directories("thirdparty/uuid/include/win32/")
else()
@@ -312,7 +312,7 @@ endif()
-DHAVE_POLL_FINE_EXITCODE=0
-DHAVE_FSETXATTR_5=0
-DHAVE_FSETXATTR_5__TRYRUN_OUTPUT=""
- "-DCMAKE_MODULE_PATH=${CMAKE_CURRENT_SOURCE_DIR}/cmake/ssl"
+ "-DCMAKE_MODULE_PATH=${CMAKE_SOURCE_DIR}/cmake/ssl"
"-DCMAKE_C_FLAGS=${CURL_C_FLAGS}"
"-DCMAKE_CXX_FLAGS=${CURL_CXX_FLAGS}"
${PC}
@@ -328,7 +328,7 @@ endif()
set(CURL_BYPRODUCT_DIR "${BYPRODUCT}" CACHE STRING "" FORCE)
- list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake/curl/dummy")
+ list(APPEND CMAKE_MODULE_PATH "${CMAKE_SOURCE_DIR}/cmake/curl/dummy")
add_library(curl STATIC IMPORTED)
set_target_properties(curl PROPERTIES IMPORTED_LOCATION
"${CURL_BIN_DIR}${BYPRODUCT}")
@@ -357,7 +357,10 @@ include(ExternalProject)
set(CIVETWEB_ENABLE_SSL_DYNAMIC_LOADING OFF CACHE BOOL "Disable dynamic SSL
library loading")
set(CIVETWEB_ENABLE_CXX ON CACHE BOOL "Enable civet C++ library")
-set(CIVETWEB_ENABLE_SSL OFF CACHE BOOL "DISABLE SSL")
+set(CIVETWEB_SSL_SSL_LIB "${OPENSSL_SSL_LIBRARY}" CACHE STRING "Enable civet
C++ library")
+set(CIVETWEB_SSL_CRYPTO_LIB "${OPENSSL_CRYPTO_LIBRARY}" CACHE STRING "Enable
civet C++ library")
+set(CIVETWEB_ENABLE_CXX ON
[jira] [Created] (MINIFI-508) Make Windows daemon inclusion a profile
Aldrin Piri created MINIFI-508: -- Summary: Make Windows daemon inclusion a profile Key: MINIFI-508 URL: https://issues.apache.org/jira/browse/MINIFI-508 Project: Apache NiFi MiNiFi Issue Type: Improvement Components: Build Affects Versions: 0.5.0 Reporter: Aldrin Piri As the Windows service executable relies on an outside source it could be beneficial to have this process driven by a profile to mitigate those areas where users do not care about having that support. The intent would be for this to still remain in released convenience binaries as part of the RM process but make it disabled by default -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (MINIFI-505) Change digest algorithm to SHA-512
[ https://issues.apache.org/jira/browse/MINIFI-505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri resolved MINIFI-505. Resolution: Fixed Assignee: Nenad V. Nikolić Fix Version/s: 0.6.0 > Change digest algorithm to SHA-512 > -- > > Key: MINIFI-505 > URL: https://issues.apache.org/jira/browse/MINIFI-505 > Project: Apache NiFi MiNiFi > Issue Type: Improvement >Reporter: Nenad V. Nikolić >Assignee: Nenad V. Nikolić >Priority: Major > Fix For: 0.6.0 > > Time Spent: 20m > Remaining Estimate: 0h > > In the light of a digest algorithm change in the [PR > #157|https://github.com/apache/nifi-minifi/pull/157/commits/c13e286fdf380f315a7951573edd7d475b74372f#diff-081c5543514deb23ccd8543465113b8dR70] > from [~aldrin] and our agreement in PR #161 to keep that change separate > from the rest, the digest algorithm should be changed separately. > PR is coming shortly after PR #161 is approved and merged to master. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[nifi-minifi] branch master updated (d2a5544 -> 64eb5b0)
This is an automated email from the ASF dual-hosted git repository. aldrin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi.git. from d2a5544 MINIFI-506 Update deprecated 'tasks' tags to 'target' in assembly antrun plugin. new 2fe3897 MINIFI-505 Change digest algorithm to SHA-512 new 64eb5b0 Remove errant import in ConfigSchemaTest which was causing contrib-check to fail. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: minifi-assembly/pom.xml | 4 ++-- .../java/org/apache/nifi/minifi/commons/schema/ConfigSchemaTest.java | 1 - 2 files changed, 2 insertions(+), 3 deletions(-)
[nifi-minifi] 02/02: Remove errant import in ConfigSchemaTest which was causing contrib-check to fail.
This is an automated email from the ASF dual-hosted git repository.
aldrin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi.git
commit 64eb5b04e8623537d439face01ef73f04b0a7f28
Author: Aldrin Piri
AuthorDate: Mon Jul 22 09:17:08 2019 -0400
Remove errant import in ConfigSchemaTest which was causing contrib-check to
fail.
---
.../java/org/apache/nifi/minifi/commons/schema/ConfigSchemaTest.java | 1 -
1 file changed, 1 deletion(-)
diff --git
a/minifi-commons/minifi-commons-schema/src/test/java/org/apache/nifi/minifi/commons/schema/ConfigSchemaTest.java
b/minifi-commons/minifi-commons-schema/src/test/java/org/apache/nifi/minifi/commons/schema/ConfigSchemaTest.java
index a68467a..a585f57 100644
---
a/minifi-commons/minifi-commons-schema/src/test/java/org/apache/nifi/minifi/commons/schema/ConfigSchemaTest.java
+++
b/minifi-commons/minifi-commons-schema/src/test/java/org/apache/nifi/minifi/commons/schema/ConfigSchemaTest.java
@@ -36,7 +36,6 @@ import static
org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys.ID
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertNotEquals;
-import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
public class ConfigSchemaTest {
[nifi-minifi] 01/02: MINIFI-505 Change digest algorithm to SHA-512
This is an automated email from the ASF dual-hosted git repository.
aldrin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi.git
commit 2fe3897619baa83864468cb224eef4044280b4e1
Author: Nenad V. Nikolić
AuthorDate: Fri Jul 19 13:51:47 2019 +0200
MINIFI-505 Change digest algorithm to SHA-512
The reason for this change is that SHA-512 its 50% faster than SHA-256
64-bit platforms that are common nowadays.
https://crypto.stackexchange.com/questions/26336/sha512-faster-than-sha256
This closes #163.
Signed-off-by: Aldrin Piri
---
minifi-assembly/pom.xml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/minifi-assembly/pom.xml b/minifi-assembly/pom.xml
index 18b3760..557cd7f 100644
--- a/minifi-assembly/pom.xml
+++ b/minifi-assembly/pom.xml
@@ -80,8 +80,8 @@ limitations under the License.
https://apache.claz.org/commons/daemon/binaries/windows/${commons.daemon.file};
dest="${java.io.tmpdir}/${commons.daemon.file}" skipexisting="true" />
-
-
+
+
[nifi-minifi] branch master updated: MINIFI-506 Update deprecated 'tasks' tags to 'target' in assembly antrun plugin.
This is an automated email from the ASF dual-hosted git repository. aldrin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi.git The following commit(s) were added to refs/heads/master by this push: new d2a5544 MINIFI-506 Update deprecated 'tasks' tags to 'target' in assembly antrun plugin. d2a5544 is described below commit d2a55440a6c99f3ef40a4bc4e69a5dc97381aea9 Author: Aldrin Piri AuthorDate: Thu Jul 11 09:36:05 2019 -0400 MINIFI-506 Update deprecated 'tasks' tags to 'target' in assembly antrun plugin. --- minifi-assembly/pom.xml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/minifi-assembly/pom.xml b/minifi-assembly/pom.xml index e7b8928..18b3760 100644 --- a/minifi-assembly/pom.xml +++ b/minifi-assembly/pom.xml @@ -68,7 +68,7 @@ limitations under the License. run - + @@ -92,7 +92,7 @@ limitations under the License. - +
[jira] [Resolved] (MINIFI-502) Update the Windows commons daemon download URL
[ https://issues.apache.org/jira/browse/MINIFI-502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri resolved MINIFI-502. Resolution: Done > Update the Windows commons daemon download URL > -- > > Key: MINIFI-502 > URL: https://issues.apache.org/jira/browse/MINIFI-502 > Project: Apache NiFi MiNiFi > Issue Type: Improvement > Components: Build >Reporter: Jeff Zemerick >Priority: Major > > Update the URL of the Windows commons daemon zip in minifi-assembly/pom.xml. > It is currently set to > [https://apache.claz.org/commons/daemon/binaries/windows/commons-daemon-1.1.0-bin-windows.zip] > and it is not 100% reliable and often causes the build on Windows to fail. > Change the URL to > [https://www.apache.org/dist/commons/daemon/binaries/windows/commons-daemon-1.1.0-bin-windows.zip.] > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (MINIFI-502) Update the Windows commons daemon download URL
[ https://issues.apache.org/jira/browse/MINIFI-502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883275#comment-16883275 ] Aldrin Piri commented on MINIFI-502: Didn't see this one when reviewing the other items but we have addressed the versioning issue. As highlighted in the one PR comment (https://github.com/apache/nifi-minifi/pull/157#discussion_r301612041) I don't think using the dist offering of things for wide consumption would be suggested by INFRA. I am going to close this for those reasons but please let me know if you have a different perspective on those matters. > Update the Windows commons daemon download URL > -- > > Key: MINIFI-502 > URL: https://issues.apache.org/jira/browse/MINIFI-502 > Project: Apache NiFi MiNiFi > Issue Type: Improvement > Components: Build >Reporter: Jeff Zemerick >Priority: Major > > Update the URL of the Windows commons daemon zip in minifi-assembly/pom.xml. > It is currently set to > [https://apache.claz.org/commons/daemon/binaries/windows/commons-daemon-1.1.0-bin-windows.zip] > and it is not 100% reliable and often causes the build on Windows to fail. > Change the URL to > [https://www.apache.org/dist/commons/daemon/binaries/windows/commons-daemon-1.1.0-bin-windows.zip.] > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[nifi-minifi] branch master updated: MINIFI-501 Provide the ability to specify security and sensitive properties in bootstrap.conf instead of the config.yml
This is an automated email from the ASF dual-hosted git repository. aldrin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi.git The following commit(s) were added to refs/heads/master by this push: new f50d3f7 MINIFI-501 Provide the ability to specify security and sensitive properties in bootstrap.conf instead of the config.yml f50d3f7 is described below commit f50d3f7c36e18d19eba76a7524ac2dffec61ce68 Author: Aldrin Piri AuthorDate: Thu Jun 20 08:16:20 2019 -0400 MINIFI-501 Provide the ability to specify security and sensitive properties in bootstrap.conf instead of the config.yml Correct wrong usage of keyPasswd instead of keystorePasswd when creating the statically defined SSL-Context-Service Update nifi version This closes #155. --- .../apache/nifi/minifi/bootstrap/RunMiNiFi.java| 90 - .../minifi/bootstrap/util/ConfigTransformer.java | 25 - .../nifi/minifi/bootstrap/RunMiNiFiTest.java | 102 .../bootstrap/util/ConfigTransformerTest.java | 66 +++-- .../bootstrap-ssl-ctx/bootstrap.conf.configured| 24 - .../bootstrap.conf.configured.invalid | 24 - .../bootstrap-ssl-ctx/bootstrap.conf.default | 24 - .../test/resources/bootstrap-ssl-ctx/config.yml| 107 + .../src/main/markdown/System_Admin_Guide.md| 30 ++ .../src/main/resources/conf/bootstrap.conf | 16 +++ 10 files changed, 459 insertions(+), 49 deletions(-) diff --git a/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/RunMiNiFi.java b/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/RunMiNiFi.java index edc1f22..e7ac3ed 100644 --- a/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/RunMiNiFi.java +++ b/minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/RunMiNiFi.java @@ -23,6 +23,9 @@ import org.apache.nifi.minifi.bootstrap.configuration.ConfigurationChangeExcepti import org.apache.nifi.minifi.bootstrap.configuration.ConfigurationChangeListener; import org.apache.nifi.minifi.bootstrap.status.PeriodicStatusReporter; import org.apache.nifi.minifi.bootstrap.util.ConfigTransformer; +import org.apache.nifi.minifi.commons.schema.SecurityPropertiesSchema; +import org.apache.nifi.minifi.commons.schema.SensitivePropsSchema; +import org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys; import org.apache.nifi.minifi.commons.status.FlowStatusReport; import org.apache.nifi.util.Tuple; import org.apache.nifi.util.file.FileUtils; @@ -65,6 +68,7 @@ import java.util.LinkedList; import java.util.List; import java.util.Map; import java.util.Map.Entry; +import java.util.Optional; import java.util.Properties; import java.util.Set; import java.util.concurrent.ExecutorService; @@ -133,6 +137,59 @@ public class RunMiNiFi implements QueryableStatusAggregator, ConfigurationFileHo public static final String STATUS_REPORTER_PROPERTY_PREFIX = "nifi.minifi.status.reporter"; public static final String STATUS_REPORTER_COMPONENTS_KEY = STATUS_REPORTER_PROPERTY_PREFIX + ".components"; + +public static final String SECURITY_KEYSTORE_KEY = "nifi.minifi.security.keystore"; +public static final String SECURITY_KEYSTORE_TYPE_KEY = "nifi.minifi.security.keystoreType"; +public static final String SECURITY_KEYSTORE_PASSWORD_KEY = "nifi.minifi.security.keystorePasswd"; +public static final String SECURITY_KEY_PASSWORD_KEY = "nifi.minifi.security.keyPasswd"; +public static final String SECURITY_TRUSTSTORE_KEY = "nifi.minifi.security.truststore"; +public static final String SECURITY_TRUSTSTORE_TYPE_KEY = "nifi.minifi.security.truststoreType"; +public static final String SECURITY_TRUSTSTORE_PASSWORD_KEY = "nifi.minifi.security.truststorePasswd"; +public static final String SECURITY_SSL_PROTOCOL_KEY = "nifi.minifi.security.ssl.protocol"; + +public static final String SENSITIVE_PROPS_KEY_KEY = "nifi.minifi.sensitive.props.key"; +public static final String SENSITIVE_PROPS_ALGORITHM_KEY = "nifi.minifi.sensitive.props.algorithm"; +public static final String SENSITIVE_PROPS_PROVIDER_KEY = "nifi.minifi.sensitive.props.provider"; + +public static final Set BOOTSTRAP_SECURITY_PROPERTY_KEYS = new HashSet<>( +Arrays.asList(SECURITY_KEYSTORE_KEY, +SECURITY_KEYSTORE_TYPE_KEY, +SECURITY_KEYSTORE_PASSWORD_KEY, +SECURITY_KEY_PASSWORD_KEY, +SECURITY_TRUSTSTORE_KEY, +SECURITY_TRUSTSTORE_TYPE_KEY, +SECURITY_TRUSTSTORE_PASSWORD_KEY, +SECURITY_SSL_PROTOCOL_KEY)); + +public stati
[nifi-minifi] branch master updated: MINIFI-162 Capture build properties in a build.properties located in conf
This is an automated email from the ASF dual-hosted git repository.
aldrin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi.git
The following commit(s) were added to refs/heads/master by this push:
new 9bf85cf MINIFI-162 Capture build properties in a build.properties
located in conf
9bf85cf is described below
commit 9bf85cf40bd38d5ed0a20fcb5a9130b3e1062d7e
Author: Aldrin Piri
AuthorDate: Fri May 31 10:50:19 2019 -0400
MINIFI-162 Capture build properties in a build.properties located in conf
This closes #153.
---
minifi-assembly/pom.xml| 23
minifi-assembly/src/main/assembly/dependencies.xml | 7 +
.../src/main/resources/build.properties.template | 31 ++
3 files changed, 61 insertions(+)
diff --git a/minifi-assembly/pom.xml b/minifi-assembly/pom.xml
index 2008865..e7b8928 100644
--- a/minifi-assembly/pom.xml
+++ b/minifi-assembly/pom.xml
@@ -28,6 +28,29 @@ limitations under the License.
+org.codehaus.mojo
+buildnumber-maven-plugin
+1.4
+true
+
+
+validate
+
+create
+
+
+
+
+false
+false
+7
+true
+
+
buildRevision
+buildBranch
+
+
+
org.apache.maven.plugins
maven-antrun-plugin
diff --git a/minifi-assembly/src/main/assembly/dependencies.xml
b/minifi-assembly/src/main/assembly/dependencies.xml
index e588b8a..6ae5411 100644
--- a/minifi-assembly/src/main/assembly/dependencies.xml
+++ b/minifi-assembly/src/main/assembly/dependencies.xml
@@ -136,6 +136,13 @@
+src/main/resources/build.properties.template
+./
+build.properties
+0444
+true
+
+
./target/minifi.exe
./bin
false
diff --git a/minifi-assembly/src/main/resources/build.properties.template
b/minifi-assembly/src/main/resources/build.properties.template
new file mode 100644
index 000..6197a6e
--- /dev/null
+++ b/minifi-assembly/src/main/resources/build.properties.template
@@ -0,0 +1,31 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# NOTE: This file is autogenerated at build and is not intended to be edited.
Changes could cause inconsistent behavior.
+
+Build-Branch:${buildBranch}
+Build-Timestamp:${timestamp}
+Built-By:${user.name}
+MiNiFi-Version:${project.version}
+Build-Revision:${buildRevision}
+Maven-Home:${maven.home}
+Maven-Version:${maven.version}
+Created-By:${maven.build.version}
+Build-Java-Home:${java.home}
+Build-Jdk:${java.version}
+Build-Jdk-Vendor:${java.vendor}
+Build-Arch:${os.arch}
+Build-Os:${os.name}
+Build-Os-Version:${os.version}
