[nifi] branch master updated: NIFI-5176 NiFI builds on Java 11 Added "jigsaw" profile to multiple modules, which auto-activates when building with Java 11 and adds several dependencies: jaxb, activati
This is an automated email from the ASF dual-hosted git repository. thenatog pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi.git The following commit(s) were added to refs/heads/master by this push: new 33187f8 NIFI-5176 NiFI builds on Java 11 Added "jigsaw" profile to multiple modules, which auto-activates when building with Java 11 and adds several dependencies: jaxb, activation, and annotation-api. Updated SslContextFactory to return tuple of socket factory with trust manager for issue with okhttp api changes in java 9+ Updated TestGet/PostHTTPGroovy to use default SSL context to get default cipher suites Updated StandardTemplateDAOSpec.groovy, was using a 37-character UUID, [...] 33187f8 is described below commit 33187f8058b0f0a5808059795c5982dac24e5300 Author: Jeff Storck AuthorDate: Fri Nov 30 13:14:31 2018 -0500 NIFI-5176 NiFI builds on Java 11 Added "jigsaw" profile to multiple modules, which auto-activates when building with Java 11 and adds several dependencies: jaxb, activation, and annotation-api. Updated SslContextFactory to return tuple of socket factory with trust manager for issue with okhttp api changes in java 9+ Updated TestGet/PostHTTPGroovy to use default SSL context to get default cipher suites Updated StandardTemplateDAOSpec.groovy, was using a 37-character UUID, shortened to 36 characters. Multiple tests updated to specifically use TLSv1.2, since two-way TLSv1.3 for some component tests fail during the Java 11 build. Needs more investigation. Replaced GString with String concatenation for map key's value in ScriptedReportingTaskGroovyTest to avoid runtime error of casting GString to String when retrieved from a map that is storing the GString Removed nifi-toolkit-api plugin configuration of maven-compiler-plugin, it is inherited Resolved deprecation errors during Groovy compile for bouncycastle, Extension should be used in place of X509Extension Fixed JNAOverridingJUnitRunner's creation of the classpath for the custom classloader created to be able to mock jna classes Removed import of IOUtils (from the wrong package) from InferenceSchemaStrategy Updated maven-compiler-plugin version to 3.8.1 TLS (default in Java 11 is TLSv1.3) working for Site-to-Site client tests after upgrading JDK installation to JDK 11.0.3, httpclient5 5.0-beta4, and httpasyncclient 4.1.4 HttpNotificationService updated to find and use the first found X509TrustManager rather than casting directly from the array of TrustManagers returned from SslContextFactory Removed unnecessary throws declaration from getSslSocketFactory method Replaced DnsContextFactory.class.getName() with a string to avoid having to export/open the jdk.naming.dns module Updated TestGetIgniteCache and TestPutIgniteCache.java to skip tests on Java 11+ (via Assume), and noted the check should be removed once a version of Ignite is released that supports Java 11 Updated SpringContextProcessor to create proper ClassLoader and uses URLClassloader.getResource() instead of URLClassloader.findResource() in isConfigResolvable. Due to the application classloader no longer being a URLClassLoader in Java 9+, the URLClassLoader created in isConfigResolvable no longer has explicit access to the parent's resources as URLs. URLClassLoader.getResource() searches the parent classloaders, and must be used instead of URLClassLoader.findResource() which only searches the URLs in the URLClassLoader and does not search the parent classloaders. NIFI-5176 Moved exclusion of jdk.tools to the jigsaw profile in the POMs of nifi-hbase_1_1_2-client-service and nifi-hbase_2-client-service NIFI-5176 Updated site-to-site-client's POM to use properties to establish the correct httpclient dependency for when building with Java 11 This closes #3404. --- .../notification/http/HttpNotificationService.java | 25 +++-- nifi-commons/nifi-security-utils/pom.xml | 33 +++ .../nifi/security/util/SslContextFactory.java | 95 +-- nifi-commons/nifi-site-to-site-client/pom.xml | 45 - nifi-commons/nifi-utils/pom.xml| 24 + .../nifi-beats-processors/pom.xml | 25 + .../nifi-enrich-processors/pom.xml | 1 + .../apache/nifi/processors/enrich/QueryDNS.java| 2 +- .../nifi-database-utils/pom.xml| 24 + .../nifi-processor-utils/pom.xml | 24 + .../nifi-framework/nifi-authorizer/pom.xml | 57 +++ .../nifi-framework/nifi-client-dto/pom.xml | 24 + .../nifi-framework/nifi-file-authorizer/pom.xml| 104 + .../nifi-framework-cluster-protocol/pom.xml| 24 + .../nifi-framework/nifi-framework-cluster/pom.xml | 24 + .../nifi-framework/nifi-framework-core-api/pom.xml
[nifi] branch master updated: NIFI-6468: TestListS3 assertion fix
This is an automated email from the ASF dual-hosted git repository.
pvillard pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/master by this push:
new b99cecd NIFI-6468: TestListS3 assertion fix
b99cecd is described below
commit b99cecd4e7da9e37d8a8d634f6673d7fd81b55bd
Author: Peter Turcsanyi
AuthorDate: Tue Aug 13 21:46:09 2019 +0200
NIFI-6468: TestListS3 assertion fix
Signed-off-by: Pierre Villard
This closes #3649.
---
.../test/java/org/apache/nifi/processors/aws/s3/TestFetchS3Object.java | 2 +-
.../src/test/java/org/apache/nifi/processors/aws/s3/TestListS3.java | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/s3/TestFetchS3Object.java
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/s3/TestFetchS3Object.java
index 88ccfb0..6416d53 100644
---
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/s3/TestFetchS3Object.java
+++
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/s3/TestFetchS3Object.java
@@ -238,7 +238,7 @@ public class TestFetchS3Object {
public void testGetPropertyDescriptors() throws Exception {
FetchS3Object processor = new FetchS3Object();
List pd =
processor.getSupportedPropertyDescriptors();
-assertEquals("size should be eq", 18, pd.size());
+assertEquals("size should be eq", 19, pd.size());
assertTrue(pd.contains(FetchS3Object.ACCESS_KEY));
assertTrue(pd.contains(FetchS3Object.AWS_CREDENTIALS_PROVIDER_SERVICE));
assertTrue(pd.contains(FetchS3Object.BUCKET));
diff --git
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/s3/TestListS3.java
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/s3/TestListS3.java
index 86af835..679ca7b 100644
---
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/s3/TestListS3.java
+++
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/s3/TestListS3.java
@@ -458,7 +458,7 @@ public class TestListS3 {
public void testGetPropertyDescriptors() throws Exception {
ListS3 processor = new ListS3();
List pd =
processor.getSupportedPropertyDescriptors();
-assertEquals("size should be eq", 22, pd.size());
+assertEquals("size should be eq", 23, pd.size());
assertTrue(pd.contains(ListS3.ACCESS_KEY));
assertTrue(pd.contains(ListS3.AWS_CREDENTIALS_PROVIDER_SERVICE));
assertTrue(pd.contains(ListS3.BUCKET));
[nifi] branch master updated: NIFI-4256 Adds AWS Encryption Controller Service. NIFI-4256 Adds AWS S3 FlowFile encryption attributes, more javadocs, better names.
This is an automated email from the ASF dual-hosted git repository.
alopresto pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/master by this push:
new e841f4d NIFI-4256 Adds AWS Encryption Controller Service. NIFI-4256
Adds AWS S3 FlowFile encryption attributes, more javadocs, better names.
e841f4d is described below
commit e841f4d5b7da7cf88cbb1805d351bf75ca8a3280
Author: Troy Melhase
AuthorDate: Sat Jul 6 11:59:44 2019 -0800
NIFI-4256 Adds AWS Encryption Controller Service.
NIFI-4256 Adds AWS S3 FlowFile encryption attributes, more javadocs,
better names.
This closes #3574.
Signed-off-by: Andy LoPresto
---
.../nifi/processors/aws/AbstractAWSProcessor.java | 4 +-
.../processors/aws/s3/AbstractS3Processor.java | 31 ++--
.../aws/s3/AmazonS3EncryptionService.java | 80 +
.../nifi-aws-bundle/nifi-aws-processors/pom.xml| 5 +
.../nifi/processors/aws/s3/FetchS3Object.java | 14 +-
.../apache/nifi/processors/aws/s3/PutS3Object.java | 37 +++-
.../ClientSideCMKEncryptionStrategy.java | 95 ++
.../ClientSideKMSEncryptionStrategy.java | 65 +++
.../aws/s3/encryption/NoOpEncryptionStrategy.java | 20 +++
.../aws/s3/encryption/S3EncryptionStrategy.java| 92 ++
.../ServerSideCEKEncryptionStrategy.java | 75
.../ServerSideKMSEncryptionStrategy.java | 43 +
.../encryption/ServerSideS3EncryptionStrategy.java | 36
.../s3/encryption/StandardS3EncryptionService.java | 189
.../org.apache.nifi.controller.ControllerService | 4 +-
.../additionalDetails.html | 71
.../nifi/processors/aws/s3/AbstractS3IT.java | 27 +++
.../nifi/processors/aws/s3/ITPutS3Object.java | 198 -
.../nifi/processors/aws/s3/TestFetchS3Object.java | 4 +-
.../nifi/processors/aws/s3/TestPutS3Object.java| 7 +-
.../s3/encryption/TestS3EncryptionStrategies.java | 180 +++
.../TestStandardS3EncryptionService.java | 102 +++
22 files changed, 1347 insertions(+), 32 deletions(-)
diff --git
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java
index 06f4a16..10d19ff 100644
---
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java
+++
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java
@@ -157,11 +157,11 @@ public abstract class AbstractAWSProcessor values = new ArrayList<>();
for (final Regions region : Regions.values()) {
values.add(createAllowableValue(region));
diff --git
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/s3/AbstractS3Processor.java
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/s3/AbstractS3Processor.java
index 5cc24e9..5e8ff32 100644
---
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/s3/AbstractS3Processor.java
+++
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/s3/AbstractS3Processor.java
@@ -20,6 +20,7 @@ import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
+import com.amazonaws.auth.AWSStaticCredentialsProvider;
import org.apache.commons.lang3.StringUtils;
import org.apache.nifi.components.AllowableValue;
import org.apache.nifi.components.PropertyDescriptor;
@@ -126,19 +127,34 @@ public abstract class AbstractS3Processor extends
AbstractAWSCredentialsProvider
new AllowableValue("S3SignerType", "Signature v2"))
.defaultValue("Default Signature")
.build();
+public static final PropertyDescriptor ENCRYPTION_SERVICE = new
PropertyDescriptor.Builder()
+.name("encryption-service")
+.displayName("Encryption Service")
+.description("Specifies the Encryption Service Controller used
configure requests. "
++ "For backward compatibility, this value is ignored when
'Server Side Encryption' is set.")
+.required(false)
+.identifiesControllerService(AmazonS3EncryptionService.class)
+.build();
+
/**
* Create client using credentials provider. This is the preferred way for
creating clients
*/
@Override
protected AmazonS3Client createClient(final ProcessContext context, final
AWSCredentialsProvider
[nifi-minifi] branch master updated: MINIFI-510 Update Travis to use supported JDKs
This is an automated email from the ASF dual-hosted git repository. kdoran pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi.git The following commit(s) were added to refs/heads/master by this push: new b01d0fe MINIFI-510 Update Travis to use supported JDKs b01d0fe is described below commit b01d0fe8ec2b6735f9a9681f874deb01ef0a7361 Author: Aldrin Piri AuthorDate: Tue Aug 13 12:45:30 2019 -0400 MINIFI-510 Update Travis to use supported JDKs --- .travis.yml | 31 +++ 1 file changed, 19 insertions(+), 12 deletions(-) diff --git a/.travis.yml b/.travis.yml index e5aba1a..90db7c1 100644 --- a/.travis.yml +++ b/.travis.yml @@ -1,19 +1,26 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. language: java matrix: include: -- os: linux +- dist: trusty jdk: oraclejdk8 - # before_install aids in a couple workarounds for issues within the Travis-CI environment - # 1. Workaround for buffer overflow issues with OpenJDK versions of java as per https://github.com/travis-ci/travis-ci/issues/5227#issuecomment-165135711 - # 2. Workaround for non-existent Maven repository as per: https://github.com/travis-ci/travis-ci/issues/4629#issuecomment-131214465 - before_install: -- sudo hostname "$(hostname | cut -c1-63)" -- sed -e "s/^\\(127\\.0\\.0\\.1.*\\)/\\1 $(hostname | cut -c1-63)/" /etc/hosts | sudo tee /etc/hosts -- sed -i.bak -e 's|https://nexus.codehaus.org/snapshots/|https://oss.sonatype.org/content/repositories/codehaus-snapshots/|g' ~/.m2/settings.xml +- dist: trusty + jdk: openjdk8 - os: osx - osx_image: xcode9.2 + osx_image: xcode9.3 - - -script: mvn clean install -Pcontrib-check +script: mvn clean install -Pcontrib-check \ No newline at end of file
[nifi-registry] branch master updated: NIFIREG-301 Added description to VersionedParameterContext
This is an automated email from the ASF dual-hosted git repository.
kdoran pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-registry.git
The following commit(s) were added to refs/heads/master by this push:
new 5fbe0a3 NIFIREG-301 Added description to VersionedParameterContext
5fbe0a3 is described below
commit 5fbe0a380b9b0c4b01f822c4c80e58857f1acea5
Author: Bryan Bende
AuthorDate: Fri Aug 9 10:02:30 2019 -0400
NIFIREG-301 Added description to VersionedParameterContext
---
.../apache/nifi/registry/flow/VersionedParameterContext.java | 10 ++
1 file changed, 10 insertions(+)
diff --git
a/nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/flow/VersionedParameterContext.java
b/nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/flow/VersionedParameterContext.java
index 5294c61..bb0b837 100644
---
a/nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/flow/VersionedParameterContext.java
+++
b/nifi-registry-core/nifi-registry-data-model/src/main/java/org/apache/nifi/registry/flow/VersionedParameterContext.java
@@ -23,6 +23,7 @@ import java.util.Set;
public class VersionedParameterContext {
private String name;
+private String description;
private Set parameters;
@ApiModelProperty("The name of the context")
@@ -34,6 +35,15 @@ public class VersionedParameterContext {
this.name = name;
}
+@ApiModelProperty("The description of the parameter context")
+public String getDescription() {
+return description;
+}
+
+public void setDescription(String description) {
+this.description = description;
+}
+
@ApiModelProperty("The parameters in the context")
public Set getParameters() {
return parameters;
[jira] [Updated] (MINIFI-509) SSL Context defined in bootstrap Not Working?
[
https://issues.apache.org/jira/browse/MINIFI-509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aldrin Piri updated MINIFI-509:
---
Fix Version/s: 0.6.0
> SSL Context defined in bootstrap Not Working?
> -
>
> Key: MINIFI-509
> URL: https://issues.apache.org/jira/browse/MINIFI-509
> Project: Apache NiFi MiNiFi
> Issue Type: Bug
>Reporter: Ryan Whittington
>Assignee: Ferenc Kis
>Priority: Major
> Fix For: 0.6.0
>
>
> I'm trying to set my SSL-Context-Service properties using the bootstrap.conf,
> but it doesn't seem to propagate through to the flow.xml.gz when minifi
> starts/reloads - and isn't available to use in my flow?
> As this feature was recently added in MINIFI-501 by [~aldrin], I'm building
> minifi from the master branch on github.
> bootstrap.conf:
>
> {code:java}
> # Security Properties #
> # These properties take precedence over any equivalent properties specified
> in config.yml #
> nifi.minifi.security.keystore=/my/test/keystore.jks
> nifi.minifi.security.keystoreType=JKS
> nifi.minifi.security.keystorePasswd=mykeystorepassword
> nifi.minifi.security.keyPasswd=mykeypassword
> nifi.minifi.security.truststore=/my/test/truststore.jks
> nifi.minifi.security.truststoreType=JKS
> nifi.minifi.security.truststorePasswd=mytruststorepassword
> nifi.minifi.security.ssl.protocol=TLS
> {code}
>
> flow.xml.gz once running:
> {noformat}
>
>
> 1
> 1
>
>
> Root-Group
> MiNiFi Flow
>
>
>
>
>
> {noformat}
> Am I doing something wrong, or is this not fully implemented yet?
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Assigned] (MINIFI-509) SSL Context defined in bootstrap Not Working?
[
https://issues.apache.org/jira/browse/MINIFI-509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aldrin Piri reassigned MINIFI-509:
--
Assignee: Ferenc Kis (was: Aldrin Piri)
> SSL Context defined in bootstrap Not Working?
> -
>
> Key: MINIFI-509
> URL: https://issues.apache.org/jira/browse/MINIFI-509
> Project: Apache NiFi MiNiFi
> Issue Type: Bug
>Reporter: Ryan Whittington
>Assignee: Ferenc Kis
>Priority: Major
>
> I'm trying to set my SSL-Context-Service properties using the bootstrap.conf,
> but it doesn't seem to propagate through to the flow.xml.gz when minifi
> starts/reloads - and isn't available to use in my flow?
> As this feature was recently added in MINIFI-501 by [~aldrin], I'm building
> minifi from the master branch on github.
> bootstrap.conf:
>
> {code:java}
> # Security Properties #
> # These properties take precedence over any equivalent properties specified
> in config.yml #
> nifi.minifi.security.keystore=/my/test/keystore.jks
> nifi.minifi.security.keystoreType=JKS
> nifi.minifi.security.keystorePasswd=mykeystorepassword
> nifi.minifi.security.keyPasswd=mykeypassword
> nifi.minifi.security.truststore=/my/test/truststore.jks
> nifi.minifi.security.truststoreType=JKS
> nifi.minifi.security.truststorePasswd=mytruststorepassword
> nifi.minifi.security.ssl.protocol=TLS
> {code}
>
> flow.xml.gz once running:
> {noformat}
>
>
> 1
> 1
>
>
> Root-Group
> MiNiFi Flow
>
>
>
>
>
> {noformat}
> Am I doing something wrong, or is this not fully implemented yet?
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (MINIFI-510) Migrate Travis away from OracleJDK8 which seems to be problematic
[ https://issues.apache.org/jira/browse/MINIFI-510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906366#comment-16906366 ] Aldrin Piri commented on MINIFI-510: Example build issue: https://travis-ci.org/apache/nifi-minifi/jobs/571358468 > Migrate Travis away from OracleJDK8 which seems to be problematic > - > > Key: MINIFI-510 > URL: https://issues.apache.org/jira/browse/MINIFI-510 > Project: Apache NiFi MiNiFi > Issue Type: Task >Reporter: Aldrin Piri >Assignee: Aldrin Piri >Priority: Major > Fix For: 0.6.0 > > > It seems configuration of Travis has changed such that Oracle JDK 8 is no > longer available. We should migrate to supported versions of 8. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (MINIFI-510) Migrate Travis away from OracleJDK8 which seems to be problematic
Aldrin Piri created MINIFI-510: -- Summary: Migrate Travis away from OracleJDK8 which seems to be problematic Key: MINIFI-510 URL: https://issues.apache.org/jira/browse/MINIFI-510 Project: Apache NiFi MiNiFi Issue Type: Task Reporter: Aldrin Piri Assignee: Aldrin Piri Fix For: 0.6.0 It seems configuration of Travis has changed such that Oracle JDK 8 is no longer available. We should migrate to supported versions of 8. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (MINIFI-509) SSL Context defined in bootstrap Not Working?
[
https://issues.apache.org/jira/browse/MINIFI-509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906324#comment-16906324
]
Aldrin Piri commented on MINIFI-509:
https://github.com/apache/nifi-minifi/pull/164
> SSL Context defined in bootstrap Not Working?
> -
>
> Key: MINIFI-509
> URL: https://issues.apache.org/jira/browse/MINIFI-509
> Project: Apache NiFi MiNiFi
> Issue Type: Bug
>Reporter: Ryan Whittington
>Assignee: Aldrin Piri
>Priority: Major
>
> I'm trying to set my SSL-Context-Service properties using the bootstrap.conf,
> but it doesn't seem to propagate through to the flow.xml.gz when minifi
> starts/reloads - and isn't available to use in my flow?
> As this feature was recently added in MINIFI-501 by [~aldrin], I'm building
> minifi from the master branch on github.
> bootstrap.conf:
>
> {code:java}
> # Security Properties #
> # These properties take precedence over any equivalent properties specified
> in config.yml #
> nifi.minifi.security.keystore=/my/test/keystore.jks
> nifi.minifi.security.keystoreType=JKS
> nifi.minifi.security.keystorePasswd=mykeystorepassword
> nifi.minifi.security.keyPasswd=mykeypassword
> nifi.minifi.security.truststore=/my/test/truststore.jks
> nifi.minifi.security.truststoreType=JKS
> nifi.minifi.security.truststorePasswd=mytruststorepassword
> nifi.minifi.security.ssl.protocol=TLS
> {code}
>
> flow.xml.gz once running:
> {noformat}
>
>
> 1
> 1
>
>
> Root-Group
> MiNiFi Flow
>
>
>
>
>
> {noformat}
> Am I doing something wrong, or is this not fully implemented yet?
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[nifi-minifi-cpp] branch master updated: minificpp-34 Implemented.
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 11cefc8 minificpp-34 Implemented.
11cefc8 is described below
commit 11cefc8342aefcd843d48faeef01e025db8c5b9f
Author: amarmer
AuthorDate: Wed Aug 7 13:43:07 2019 -0700
minificpp-34 Implemented.
minificpp-34 Formatting.
minificpp-34 Removed unused variable.
minificpp-34 1. Increased wait time for exe termination to 5sec. 2. Small
refactoring.
minificpp-34 Small refactoring.
minificpp-34 Small refactoring.
minificpp-34 Refactoring.
minificpp-34 No changes, to re-trigger appveyor.
This closes #622.
Signed-off-by: Marc Parisi
---
CMakeLists.txt| 4 +
main/CMakeLists.txt | 3 +-
main/MiNiFiMain.cpp | 17 ++
main/MiNiFiWindowsService.cpp | 372 ++
main/MiNiFiWindowsService.h | 11 ++
5 files changed, 406 insertions(+), 1 deletion(-)
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 4965605..f87f074 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -149,6 +149,10 @@ endif()
endif()
+if (WIN32)
+ add_definitions(-DSERVICE_NAME="MiNiFi")
+endif()
+
#set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall")
if (NOT PORTABLE)
diff --git a/main/CMakeLists.txt b/main/CMakeLists.txt
index 062449b..ecd5f6b 100644
--- a/main/CMakeLists.txt
+++ b/main/CMakeLists.txt
@@ -55,7 +55,8 @@ endif()
endif()
-add_executable(minifiexe MiNiFiMain.cpp)
+add_executable(minifiexe MiNiFiMain.cpp MiNiFiWindowsService.cpp)
+
if (NOT USE_SHARED_LIBS)
if (LIBC_STATIC)
set_target_properties(minifiexe PROPERTIES
LINK_SEARCH_START_STATIC 1)
diff --git a/main/MiNiFiMain.cpp b/main/MiNiFiMain.cpp
index 6c39b5e..5028eb2 100644
--- a/main/MiNiFiMain.cpp
+++ b/main/MiNiFiMain.cpp
@@ -27,6 +27,7 @@
#include
#include
#include
+#include "MiNiFiWindowsService.h"
#pragma comment(lib, "Ws2_32.lib")
#pragma comment(lib, "legacy_stdio_definitions.lib")
#ifdef ENABLE_JNI
@@ -53,6 +54,7 @@
#include "core/RepositoryFactory.h"
#include "FlowController.h"
#include "Main.h"
+
// Variables that allow us to avoid a timed wait.
sem_t *running;
//! Flow Controller
@@ -79,7 +81,12 @@ BOOL WINAPI consoleSignalHandler(DWORD signal) {
return TRUE;
}
+
+void SignalExitProcess() {
+ sem_post(running);
+}
#endif
+
void sigHandler(int signal) {
if (signal == SIGINT || signal == SIGTERM) {
// avoid stopping the controller here.
@@ -88,8 +95,18 @@ void sigHandler(int signal) {
}
int main(int argc, char **argv) {
+#ifdef WIN32
+ CheckRunAsService();
+#endif
+
std::shared_ptr logger =
logging::LoggerConfiguration::getConfiguration().getLogger("main");
+#ifdef WIN32
+ if (!CreateServiceTerminationThread(logger)) {
+return -1;
+ }
+#endif
+
uint16_t stop_wait_time = STOP_WAIT_TIME_MS;
// initialize static functions that were defined apriori
diff --git a/main/MiNiFiWindowsService.cpp b/main/MiNiFiWindowsService.cpp
new file mode 100644
index 000..be0e8db
--- /dev/null
+++ b/main/MiNiFiWindowsService.cpp
@@ -0,0 +1,372 @@
+#ifdef WIN32
+
+#include "MiNiFiWindowsService.h"
+
+#include
+#include
+#include
+#include
+
+#include "core/FlowConfiguration.h"
+
+//#define DEBUG_SERVICE
+
+#ifdef DEBUG_SERVICE
+ #define LOG_INFO(...) OutputDebug(__VA_ARGS__)
+ #define LOG_ERROR(...) OutputDebug(__VA_ARGS__)
+ #define LOG_LASTERROR(str) OutputDebug(str " lastError %x", GetLastError())
+#else
+ #define LOG_INFO(...) Log()->log_info(__VA_ARGS__)
+ #define LOG_ERROR(...) Log()->log_error(__VA_ARGS__)
+ #define LOG_LASTERROR(str) Log()->log_error(str " lastError %x",
GetLastError())
+#endif
+
+#undef DEBUG_SERVICE
+
+// Implemented in MiNiFiMain.cpp
+void SignalExitProcess();
+
+static char* SERVICE_TERMINATION_EVENT_NAME =
"Global\\MiNiFiServiceTermination";
+
+static void OutputDebug(const char* format, ...) {
+ va_list args;
+ va_start(args, format);
+
+ char buf[256];
+ sprintf_s(buf, _countof(buf), "%s: %s", SERVICE_NAME, format);
+
+ char out[1024];
+ StringCbVPrintfA(out, sizeof(out), buf, args);
+
+ OutputDebugStringA(out);
+
+ va_end(args);
+};
+
+void CheckRunAsService() {
+ static const int WAIT_TIME_EXE_TERMINATION = 5000;
+ static const int WAIT_TIME_EXE_RESTART = 6;
+
+ static SERVICE_STATUS s_serviceStatus;
+ static SERVICE_STATUS_HANDLE s_statusHandle;
+ static HANDLE s_hProcess;
+ static HANDLE s_hEvent;
+
+ static auto Log = []() {
+static std::shared_ptr s_logger =
logging::LoggerConfiguration::getConfiguration().getLogger("service");
+return s_logger;
+ };
+
+ SERVICE_TABLE_ENTRY serviceTable[] =
+ {
+{
+ SERVICE_NAME,
+ [](DWORD
[nifi-minifi-cpp] 01/01: Merge pull request #619 from apache/phrocker-patch-1
This is an automated email from the ASF dual-hosted git repository. phrocker pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git commit 4f13e523fb40ff709e7641495d7d20eac19dbd9f Merge: fb92490 6e217bc Author: Marc AuthorDate: Tue Aug 13 08:39:16 2019 -0400 Merge pull request #619 from apache/phrocker-patch-1 Ran gen-processors-docs PROCESSORS.md | 1517 + 1 file changed, 781 insertions(+), 736 deletions(-)
[nifi-minifi-cpp] branch master updated (fb92490 -> 4f13e52)
This is an automated email from the ASF dual-hosted git repository. phrocker pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from fb92490 MINIFICPP-1000 - Add proper logging for configuration loading add 6e217bc Ran gen-processors-docs new 4f13e52 Merge pull request #619 from apache/phrocker-patch-1 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: PROCESSORS.md | 1517 + 1 file changed, 781 insertions(+), 736 deletions(-)
[nifi-minifi-cpp] branch master updated: MINIFICPP-1000 - Add proper logging for configuration loading
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new fb92490 MINIFICPP-1000 - Add proper logging for configuration loading
fb92490 is described below
commit fb9249086d4a9512ccbb129b493d8acd6f93580a
Author: Arpad Boda
AuthorDate: Thu Aug 8 15:18:32 2019 -0400
MINIFICPP-1000 - Add proper logging for configuration loading
This closes #623.
Signed-off-by: Marc Parisi
---
.../include/core/logging/LoggerConfiguration.h | 1 +
libminifi/include/properties/Configure.h | 1 +
libminifi/include/properties/Properties.h | 10 -
libminifi/src/Properties.cpp | 45 ++
main/MiNiFiMain.cpp| 2 +-
5 files changed, 31 insertions(+), 28 deletions(-)
diff --git a/libminifi/include/core/logging/LoggerConfiguration.h
b/libminifi/include/core/logging/LoggerConfiguration.h
index d015862..7f9fdd2 100644
--- a/libminifi/include/core/logging/LoggerConfiguration.h
+++ b/libminifi/include/core/logging/LoggerConfiguration.h
@@ -57,6 +57,7 @@ struct LoggerNamespace {
class LoggerProperties : public Properties {
public:
+ LoggerProperties() : Properties("Logger properties") {}
/**
* Gets all keys that start with the given prefix and do not have a "."
after the prefix and "." separator.
*
diff --git a/libminifi/include/properties/Configure.h
b/libminifi/include/properties/Configure.h
index 9fc639f..f4a3a4e 100644
--- a/libminifi/include/properties/Configure.h
+++ b/libminifi/include/properties/Configure.h
@@ -30,6 +30,7 @@ namespace minifi {
class Configure : public Properties {
public:
+ Configure() : Properties("MiNiFi configuration") {}
void setAgentIdentifier(const std::string ) {
std::lock_guard lock(mutex_);
diff --git a/libminifi/include/properties/Properties.h
b/libminifi/include/properties/Properties.h
index 9047d11..420f26c 100644
--- a/libminifi/include/properties/Properties.h
+++ b/libminifi/include/properties/Properties.h
@@ -45,12 +45,16 @@ namespace minifi {
class Properties {
public:
- Properties();
+ Properties(const std::string& name = "");
virtual ~Properties() {
}
+ virtual const std::string& getName() {
+return name_;
+ }
+
// Clear the load config
void clear() {
std::lock_guard lock(mutex_);
@@ -110,7 +114,7 @@ class Properties {
}
// Get the determined MINIFI_HOME
- std::string getHome() {
+ std::string getHome() const {
return minifi_home_;
}
// Parse Command Line
@@ -137,6 +141,8 @@ class Properties {
std::shared_ptr logger_;
// Home location for this executable
std::string minifi_home_;
+
+ std::string name_;
};
} /* namespace minifi */
diff --git a/libminifi/src/Properties.cpp b/libminifi/src/Properties.cpp
index 0e23b45..95cb18e 100644
--- a/libminifi/src/Properties.cpp
+++ b/libminifi/src/Properties.cpp
@@ -22,14 +22,6 @@
#include "core/Core.h"
#include "core/logging/LoggerConfiguration.h"
-#ifndef FILE_SEPARATOR_C
-#ifdef WIN32
-#define FILE_SEPARATOR_C '\\'
-#else
-#define FILE_SEPARATOR_C '/'
-#endif
-#endif
-
namespace org {
namespace apache {
namespace nifi {
@@ -37,8 +29,9 @@ namespace minifi {
#define TRACE_BUFFER_SIZE 512
-Properties::Properties()
-: logger_(logging::LoggerFactory::getLogger()) {
+Properties::Properties(const std::string& name)
+: logger_(logging::LoggerFactory::getLogger()),
+name_(name) {
}
// Get the config value
@@ -122,26 +115,27 @@ bool Properties::parseConfigureFileLine(char *buf,
std::string _key, std::s
// Load Configure File
void Properties::loadConfigureFile(const char *fileName) {
- std::string adjustedFilename;
- if (fileName) {
-// perform a naive determination if this is a relative path
-if (fileName[0] != FILE_SEPARATOR_C) {
- adjustedFilename = adjustedFilename + getHome() + FILE_SEPARATOR_C +
fileName;
-} else {
- if (adjustedFilename.empty()) {
-adjustedFilename = getHome();
- }
- adjustedFilename += fileName;
-}
+ if (NULL == fileName) {
+logger_->log_error("Configuration file path for %s is a nullptr!",
getName().c_str());
+return;
+ }
+
+ std::string adjustedFilename = getHome();
+ // perform a naive determination if this is a relative path
+ if (fileName[0] != utils::file::FileUtils::get_separator()) {
+adjustedFilename += utils::file::FileUtils::get_separator();
}
- char *path = NULL;
+
+ adjustedFilename += fileName;
+
+ const char *path = NULL;
#ifndef WIN32
char full_path[PATH_MAX];
path = realpath(adjustedFilename.c_str(), full_path);
#else
- path = const_cast(adjustedFilename.c_str());
+ path = adjustedFilename.c_str();
#endif
- logger_->log_info("Using configuration file located at %s, from %s", path,
[nifi-minifi-cpp] branch master updated: MINIFICPP-984 - Add "Hexencode Payload" and "Maximum Payload Line Length" options to LogAttribute with defaults to emulate the previous behaviour
This is an automated email from the ASF dual-hosted git repository.
phrocker pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
The following commit(s) were added to refs/heads/master by this push:
new 2cde150 MINIFICPP-984 - Add "Hexencode Payload" and "Maximum Payload
Line Length" options to LogAttribute with defaults to emulate the previous
behaviour
2cde150 is described below
commit 2cde15078778bbe73d35a666a74f8e0f65c869a8
Author: Daniel Bakai
AuthorDate: Mon Aug 12 18:42:47 2019 +0200
MINIFICPP-984 - Add "Hexencode Payload" and "Maximum Payload Line Length"
options to LogAttribute with defaults to emulate the previous behaviour
This closes #625.
Signed-off-by: Marc Parisi
---
.../processors/LogAttribute.cpp| 30 ++---
.../standard-processors/processors/LogAttribute.h | 6 +
.../tests/integration/TailFileCronTest.cpp | 2 +-
.../tests/integration/TailFileTest.cpp | 2 +-
.../tests/unit/TailFileTests.cpp | 31 +++---
5 files changed, 62 insertions(+), 9 deletions(-)
diff --git a/extensions/standard-processors/processors/LogAttribute.cpp
b/extensions/standard-processors/processors/LogAttribute.cpp
index cc44f25..1dbaf4a 100644
--- a/extensions/standard-processors/processors/LogAttribute.cpp
+++ b/extensions/standard-processors/processors/LogAttribute.cpp
@@ -56,6 +56,10 @@ core::Property LogAttribute::AttributesToIgnore(
core::Property
LogAttribute::LogPayload(core::PropertyBuilder::createProperty("Log
Payload")->withDescription("If true, the FlowFile's payload will be logged, in
addition to its attributes."
"otherwise, just the Attributes will be
logged")->withDefaultValue(false)->build());
+core::Property
LogAttribute::HexencodePayload(core::PropertyBuilder::createProperty("Hexencode
Payload")->withDescription("If true, the FlowFile's payload will be logged in a
hexencoded format")->withDefaultValue(false)->build());
+
+core::Property
LogAttribute::MaxPayloadLineLength(core::PropertyBuilder::createProperty("Maximum
Payload Line Length")->withDescription("The logged payload will be broken into
lines this long. 0 means no newlines will be
added.")->withDefaultValue(80U)->build());
+
core::Property LogAttribute::LogPrefix(
core::PropertyBuilder::createProperty("Log Prefix")->withDescription("Log
prefix appended to the log lines. It helps to distinguish the output of
multiple LogAttribute processors.")->build());
@@ -68,6 +72,8 @@ void LogAttribute::initialize() {
properties.insert(AttributesToLog);
properties.insert(AttributesToIgnore);
properties.insert(LogPayload);
+ properties.insert(HexencodePayload);
+ properties.insert(MaxPayloadLineLength);
properties.insert(FlowFilesToLog);
properties.insert(LogPrefix);
setSupportedProperties(properties);
@@ -87,6 +93,14 @@ void LogAttribute::onSchedule(const
std::shared_ptr
throw Exception(PROCESS_SCHEDULE_EXCEPTION, "Invalid value for flowfiles
to log: " + flowsToLog.getValue().to_string());
flowfiles_to_log_ = flowsToLog.getValue();
}
+
+ std::string value;
+ if (context->getProperty(HexencodePayload.getName(), value)) {
+utils::StringUtils::StringToBool(value, hexencode_);
+ }
+ if (context->getProperty(MaxPayloadLineLength.getName(), value)) {
+core::Property::StringToInt(value, max_line_length_);
+ }
}
// OnTrigger method, implemented by NiFi LogAttribute
void LogAttribute::onTrigger(const std::shared_ptr
, const std::shared_ptr ) {
@@ -138,9 +152,19 @@ void LogAttribute::onTrigger(const
std::shared_ptr
ReadCallback callback(logger_, flow->getSize());
session->read(flow, );
- auto payload_hex = utils::StringUtils::to_hex(callback.buffer_.data(),
callback.buffer_.size());
- for (size_t i = 0; i < payload_hex.size(); i += 80) {
-message << payload_hex.substr(i, 80) << '\n';
+ std::string printable_payload;
+ if (hexencode_) {
+printable_payload =
utils::StringUtils::to_hex(callback.buffer_.data(), callback.buffer_.size());
+ } else {
+printable_payload =
std::string(reinterpret_cast(callback.buffer_.data()),
callback.buffer_.size());
+ }
+
+ if (max_line_length_ == 0U) {
+message << printable_payload << "\n";
+ } else {
+for (size_t i = 0; i < printable_payload.size(); i +=
max_line_length_) {
+ message << printable_payload.substr(i, max_line_length_) << '\n';
+}
}
} else {
message << "\n";
diff --git a/extensions/standard-processors/processors/LogAttribute.h
b/extensions/standard-processors/processors/LogAttribute.h
index a8f00eb..99f53d4 100644
--- a/extensions/standard-processors/processors/LogAttribute.h
+++
[nifi] branch master updated: NIFI-6468: Adding AWS S3 'requester pays' to Fetch and List processors.
This is an automated email from the ASF dual-hosted git repository.
pvillard pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/master by this push:
new 32c46f0 NIFI-6468: Adding AWS S3 'requester pays' to Fetch and List
processors.
32c46f0 is described below
commit 32c46f0bdd49f039cabebe7c6652693e6948081f
Author: Joe Gresock
AuthorDate: Tue Jul 30 15:11:59 2019 +
NIFI-6468: Adding AWS S3 'requester pays' to Fetch and List processors.
Signed-off-by: Pierre Villard
This closes #3601.
---
.../nifi/processors/aws/s3/FetchS3Object.java | 18 +++-
.../org/apache/nifi/processors/aws/s3/ListS3.java | 53 +-
.../nifi/processors/aws/s3/TestFetchS3Object.java | 63 +++-
.../apache/nifi/processors/aws/s3/TestListS3.java | 107 +
4 files changed, 238 insertions(+), 3 deletions(-)
diff --git
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/FetchS3Object.java
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/FetchS3Object.java
index 122231a..b66468a 100644
---
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/FetchS3Object.java
+++
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/FetchS3Object.java
@@ -32,6 +32,7 @@ import org.apache.nifi.annotation.behavior.WritesAttributes;
import org.apache.nifi.annotation.documentation.CapabilityDescription;
import org.apache.nifi.annotation.documentation.SeeAlso;
import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.AllowableValue;
import org.apache.nifi.components.PropertyDescriptor;
import org.apache.nifi.expression.ExpressionLanguageScope;
import org.apache.nifi.flowfile.FlowFile;
@@ -73,10 +74,23 @@ public class FetchS3Object extends AbstractS3Processor {
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
.required(false)
.build();
+public static final PropertyDescriptor REQUESTER_PAYS = new
PropertyDescriptor.Builder()
+.name("requester-pays")
+.displayName("Requester Pays")
+.required(true)
+.description("If true, indicates that the requester consents to
pay any charges associated with retrieving objects from "
++ "the S3 bucket. This sets the 'x-amz-request-payer'
header to 'requester'.")
+.addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+.allowableValues(new AllowableValue("true", "True", "Indicates
that the requester consents to pay any charges associated "
++ "with retrieving objects from the S3 bucket."), new
AllowableValue("false", "False", "Does not consent to pay "
++ "requester charges for retrieving objects from
the S3 bucket."))
+.defaultValue("false")
+.build();
public static final List properties =
Collections.unmodifiableList(
Arrays.asList(BUCKET, KEY, REGION, ACCESS_KEY, SECRET_KEY,
CREDENTIALS_FILE, AWS_CREDENTIALS_PROVIDER_SERVICE, TIMEOUT, VERSION_ID,
-SSL_CONTEXT_SERVICE, ENDPOINT_OVERRIDE, SIGNER_OVERRIDE,
PROXY_CONFIGURATION_SERVICE, PROXY_HOST, PROXY_HOST_PORT, PROXY_USERNAME,
PROXY_PASSWORD));
+SSL_CONTEXT_SERVICE, ENDPOINT_OVERRIDE, SIGNER_OVERRIDE,
PROXY_CONFIGURATION_SERVICE, PROXY_HOST, PROXY_HOST_PORT, PROXY_USERNAME,
PROXY_PASSWORD,
+REQUESTER_PAYS));
@Override
protected List getSupportedPropertyDescriptors() {
@@ -94,6 +108,7 @@ public class FetchS3Object extends AbstractS3Processor {
final String bucket =
context.getProperty(BUCKET).evaluateAttributeExpressions(flowFile).getValue();
final String key =
context.getProperty(KEY).evaluateAttributeExpressions(flowFile).getValue();
final String versionId =
context.getProperty(VERSION_ID).evaluateAttributeExpressions(flowFile).getValue();
+final boolean requesterPays =
context.getProperty(REQUESTER_PAYS).asBoolean();
final AmazonS3 client = getClient();
final GetObjectRequest request;
@@ -102,6 +117,7 @@ public class FetchS3Object extends AbstractS3Processor {
} else {
request = new GetObjectRequest(bucket, key, versionId);
}
+request.setRequesterPays(requesterPays);
final Map attributes = new HashMap<>();
try (final S3Object s3Object = client.getObject(request)) {
diff --git
a/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/ListS3.java
b/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/ListS3.java
index fb4e49f..34c1dec 100644
