[nifi] branch support/nifi-1.11.x updated: NIFI-7080
This is an automated email from the ASF dual-hosted git repository.
kdoran pushed a commit to branch support/nifi-1.11.x
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/support/nifi-1.11.x by this
push:
new e62841f NIFI-7080
e62841f is described below
commit e62841f6432d618ba9e07d111a9d4a7cdebde9a2
Author: Aldrin Piri
AuthorDate: Thu Jan 30 14:46:16 2020 -0500
NIFI-7080
* Ensure script files in Docker builds are treated as executable.
* Updating git permissions to reflect executable for both secure.sh scripts.
Signed-off-by: Aldrin Piri
Signed-off-by: Kevin Doran
This closes #4027.
Signed-off-by: Kevin Doran
---
nifi-docker/dockerhub/Dockerfile | 1 +
nifi-docker/dockerhub/sh/secure.sh | 0
nifi-docker/dockermaven/Dockerfile | 1 +
nifi-docker/dockermaven/sh/secure.sh | 0
4 files changed, 2 insertions(+)
diff --git a/nifi-docker/dockerhub/Dockerfile b/nifi-docker/dockerhub/Dockerfile
index 60aad2c..373389a 100644
--- a/nifi-docker/dockerhub/Dockerfile
+++ b/nifi-docker/dockerhub/Dockerfile
@@ -36,6 +36,7 @@ ENV NIFI_PID_DIR=${NIFI_HOME}/run
ENV NIFI_LOG_DIR=${NIFI_HOME}/logs
ADD sh/ ${NIFI_BASE_DIR}/scripts/
+RUN chmod -R +x ${NIFI_BASE_DIR}/scripts/*.sh
# Setup NiFi user and create necessary directories
RUN groupadd -g ${GID} nifi || groupmod -n nifi `getent group ${GID} | cut -d:
-f1` \
diff --git a/nifi-docker/dockerhub/sh/secure.sh
b/nifi-docker/dockerhub/sh/secure.sh
old mode 100644
new mode 100755
diff --git a/nifi-docker/dockermaven/Dockerfile
b/nifi-docker/dockermaven/Dockerfile
index cb73c74..36262e9 100644
--- a/nifi-docker/dockermaven/Dockerfile
+++ b/nifi-docker/dockermaven/Dockerfile
@@ -30,6 +30,7 @@ ENV NIFI_PID_DIR=${NIFI_HOME}/run
ENV NIFI_LOG_DIR=${NIFI_HOME}/logs
ADD sh/ ${NIFI_BASE_DIR}/scripts/
+RUN chmod -R +x ${NIFI_BASE_DIR}/scripts/*.sh
COPY $NIFI_BINARY $NIFI_BASE_DIR
RUN unzip ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.zip -d ${NIFI_BASE_DIR} \
diff --git a/nifi-docker/dockermaven/sh/secure.sh
b/nifi-docker/dockermaven/sh/secure.sh
old mode 100644
new mode 100755
[nifi] branch master updated: NIFI-7080
This is an automated email from the ASF dual-hosted git repository.
kdoran pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/master by this push:
new c8af8ea NIFI-7080
c8af8ea is described below
commit c8af8eaa766407cb137d0074f824981b5bc07d93
Author: Aldrin Piri
AuthorDate: Thu Jan 30 14:46:16 2020 -0500
NIFI-7080
* Ensure script files in Docker builds are treated as executable.
* Updating git permissions to reflect executable for both secure.sh scripts.
Signed-off-by: Aldrin Piri
Signed-off-by: Kevin Doran
This closes #4027.
Signed-off-by: Kevin Doran
---
nifi-docker/dockerhub/Dockerfile | 1 +
nifi-docker/dockerhub/sh/secure.sh | 0
nifi-docker/dockermaven/Dockerfile | 1 +
nifi-docker/dockermaven/sh/secure.sh | 0
4 files changed, 2 insertions(+)
diff --git a/nifi-docker/dockerhub/Dockerfile b/nifi-docker/dockerhub/Dockerfile
index 9cfdb16..41ebe42 100644
--- a/nifi-docker/dockerhub/Dockerfile
+++ b/nifi-docker/dockerhub/Dockerfile
@@ -36,6 +36,7 @@ ENV NIFI_PID_DIR=${NIFI_HOME}/run
ENV NIFI_LOG_DIR=${NIFI_HOME}/logs
ADD sh/ ${NIFI_BASE_DIR}/scripts/
+RUN chmod -R +x ${NIFI_BASE_DIR}/scripts/*.sh
# Setup NiFi user and create necessary directories
RUN groupadd -g ${GID} nifi || groupmod -n nifi `getent group ${GID} | cut -d:
-f1` \
diff --git a/nifi-docker/dockerhub/sh/secure.sh
b/nifi-docker/dockerhub/sh/secure.sh
old mode 100644
new mode 100755
diff --git a/nifi-docker/dockermaven/Dockerfile
b/nifi-docker/dockermaven/Dockerfile
index cb73c74..36262e9 100644
--- a/nifi-docker/dockermaven/Dockerfile
+++ b/nifi-docker/dockermaven/Dockerfile
@@ -30,6 +30,7 @@ ENV NIFI_PID_DIR=${NIFI_HOME}/run
ENV NIFI_LOG_DIR=${NIFI_HOME}/logs
ADD sh/ ${NIFI_BASE_DIR}/scripts/
+RUN chmod -R +x ${NIFI_BASE_DIR}/scripts/*.sh
COPY $NIFI_BINARY $NIFI_BASE_DIR
RUN unzip ${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}-bin.zip -d ${NIFI_BASE_DIR} \
diff --git a/nifi-docker/dockermaven/sh/secure.sh
b/nifi-docker/dockermaven/sh/secure.sh
old mode 100644
new mode 100755
[nifi] branch master updated: NIFI-7073: This closes #4025. Route to failure when error on PutHDFS file system close
This is an automated email from the ASF dual-hosted git repository.
joewitt pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/master by this push:
new 76e8c51 NIFI-7073: This closes #4025. Route to failure when error on
PutHDFS file system close
76e8c51 is described below
commit 76e8c51e11ce42fc99d48722507e101a05bec868
Author: Matthew Burgess
AuthorDate: Thu Jan 30 10:50:29 2020 -0500
NIFI-7073: This closes #4025. Route to failure when error on PutHDFS file
system close
Signed-off-by: Joe Witt
---
.../org/apache/nifi/processors/hadoop/PutHDFS.java | 6 +--
.../apache/nifi/processors/hadoop/PutHDFSTest.java | 48 +-
2 files changed, 49 insertions(+), 5 deletions(-)
diff --git
a/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java
b/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java
index 37c306c..b6c4328 100644
---
a/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java
+++
b/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java
@@ -23,7 +23,6 @@ import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.permission.FsCreateModes;
import org.apache.hadoop.fs.permission.FsPermission;
import org.apache.hadoop.io.compress.CompressionCodec;
-import org.apache.hadoop.ipc.RemoteException;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.nifi.annotation.behavior.InputRequirement;
import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
@@ -353,7 +352,7 @@ public class PutHDFS extends AbstractHadoopProcessor {
if (fos != null) {
fos.close();
}
-} catch (RemoteException re) {
+} catch (Throwable t) {
// when talking to remote HDFS clusters,
we don't notice problems until fos.close()
if (createdFile != null) {
try {
@@ -361,8 +360,7 @@ public class PutHDFS extends AbstractHadoopProcessor {
} catch (Throwable ignore) {
}
}
-throw re;
-} catch (Throwable ignore) {
+throw t;
}
fos = null;
}
diff --git
a/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/test/java/org/apache/nifi/processors/hadoop/PutHDFSTest.java
b/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/test/java/org/apache/nifi/processors/hadoop/PutHDFSTest.java
index 9c51f34..9413018 100644
---
a/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/test/java/org/apache/nifi/processors/hadoop/PutHDFSTest.java
+++
b/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/test/java/org/apache/nifi/processors/hadoop/PutHDFSTest.java
@@ -432,6 +432,33 @@ public class PutHDFSTest {
fileSystem.getFileStatus(new
Path("target/test-classes/randombytes-1")).getPermission());
}
+@Test
+public void testPutFileWithCloseException() throws IOException {
+mockFileSystem = new MockFileSystem(true);
+String dirName = "target/testPutFileCloseException";
+File file = new File(dirName);
+file.mkdirs();
+Path p = new Path(dirName).makeQualified(mockFileSystem.getUri(),
mockFileSystem.getWorkingDirectory());
+
+TestRunner runner = TestRunners.newTestRunner(new
TestablePutHDFS(kerberosProperties, mockFileSystem));
+runner.setProperty(PutHDFS.DIRECTORY, dirName);
+runner.setProperty(PutHDFS.CONFLICT_RESOLUTION, "replace");
+
+try (FileInputStream fis = new
FileInputStream("src/test/resources/testdata/randombytes-1")) {
+Map attributes = new HashMap<>();
+attributes.put(CoreAttributes.FILENAME.key(), "randombytes-1");
+runner.enqueue(fis, attributes);
+runner.run();
+}
+
+List failedFlowFiles = runner
+.getFlowFilesForRelationship(PutHDFS.REL_FAILURE);
+assertFalse(failedFlowFiles.isEmpty());
+assertTrue(failedFlowFiles.get(0).isPenalized());
+
+mockFileSystem.delete(p, true);
+}
+
private class TestablePutHDFS extends PutHDFS {
private KerberosProperties testKerberosProperties;
@@ -461,6 +488,15 @@ public class PutHDFSTest {
private class MockFileSystem extends File
[nifi] branch support/nifi-1.11.x updated: NIFI-7073: This closes #4025. Route to failure when error on PutHDFS file system close
This is an automated email from the ASF dual-hosted git repository.
joewitt pushed a commit to branch support/nifi-1.11.x
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/support/nifi-1.11.x by this
push:
new add2146 NIFI-7073: This closes #4025. Route to failure when error on
PutHDFS file system close
add2146 is described below
commit add2146992e91d50a71eb67f41d336272825430e
Author: Matthew Burgess
AuthorDate: Thu Jan 30 10:50:29 2020 -0500
NIFI-7073: This closes #4025. Route to failure when error on PutHDFS file
system close
Signed-off-by: Joe Witt
---
.../org/apache/nifi/processors/hadoop/PutHDFS.java | 6 +--
.../apache/nifi/processors/hadoop/PutHDFSTest.java | 48 +-
2 files changed, 49 insertions(+), 5 deletions(-)
diff --git
a/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java
b/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java
index 37c306c..b6c4328 100644
---
a/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java
+++
b/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java
@@ -23,7 +23,6 @@ import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.permission.FsCreateModes;
import org.apache.hadoop.fs.permission.FsPermission;
import org.apache.hadoop.io.compress.CompressionCodec;
-import org.apache.hadoop.ipc.RemoteException;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.nifi.annotation.behavior.InputRequirement;
import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
@@ -353,7 +352,7 @@ public class PutHDFS extends AbstractHadoopProcessor {
if (fos != null) {
fos.close();
}
-} catch (RemoteException re) {
+} catch (Throwable t) {
// when talking to remote HDFS clusters,
we don't notice problems until fos.close()
if (createdFile != null) {
try {
@@ -361,8 +360,7 @@ public class PutHDFS extends AbstractHadoopProcessor {
} catch (Throwable ignore) {
}
}
-throw re;
-} catch (Throwable ignore) {
+throw t;
}
fos = null;
}
diff --git
a/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/test/java/org/apache/nifi/processors/hadoop/PutHDFSTest.java
b/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/test/java/org/apache/nifi/processors/hadoop/PutHDFSTest.java
index 9c51f34..9413018 100644
---
a/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/test/java/org/apache/nifi/processors/hadoop/PutHDFSTest.java
+++
b/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/test/java/org/apache/nifi/processors/hadoop/PutHDFSTest.java
@@ -432,6 +432,33 @@ public class PutHDFSTest {
fileSystem.getFileStatus(new
Path("target/test-classes/randombytes-1")).getPermission());
}
+@Test
+public void testPutFileWithCloseException() throws IOException {
+mockFileSystem = new MockFileSystem(true);
+String dirName = "target/testPutFileCloseException";
+File file = new File(dirName);
+file.mkdirs();
+Path p = new Path(dirName).makeQualified(mockFileSystem.getUri(),
mockFileSystem.getWorkingDirectory());
+
+TestRunner runner = TestRunners.newTestRunner(new
TestablePutHDFS(kerberosProperties, mockFileSystem));
+runner.setProperty(PutHDFS.DIRECTORY, dirName);
+runner.setProperty(PutHDFS.CONFLICT_RESOLUTION, "replace");
+
+try (FileInputStream fis = new
FileInputStream("src/test/resources/testdata/randombytes-1")) {
+Map attributes = new HashMap<>();
+attributes.put(CoreAttributes.FILENAME.key(), "randombytes-1");
+runner.enqueue(fis, attributes);
+runner.run();
+}
+
+List failedFlowFiles = runner
+.getFlowFilesForRelationship(PutHDFS.REL_FAILURE);
+assertFalse(failedFlowFiles.isEmpty());
+assertTrue(failedFlowFiles.get(0).isPenalized());
+
+mockFileSystem.delete(p, true);
+}
+
private class TestablePutHDFS extends PutHDFS {
private KerberosProperties testKerberosProperties;
@@ -461,6 +488,15 @@ public class PutHDFSTest {
private class
[nifi-minifi-cpp] branch master updated (405b9de -> 14aa970)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from 405b9de MINIFICPP-1131 fix syntax error in install script add 14aa970 MINIFICPP-1136 - Fix win_build_vs.bat /D option No new revisions were added by this update. Summary of changes: win_build_vs.bat | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
[nifi-minifi-cpp] branch master updated (205fbfa -> 405b9de)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git. from 205fbfa MINIFICPP-1124 - Fix target_wholearchive_library on Windows with older CMake versions add 405b9de MINIFICPP-1131 fix syntax error in install script No new revisions were added by this update. Summary of changes: bin/minifi.sh | 18 +- 1 file changed, 9 insertions(+), 9 deletions(-)
[nifi] branch support/nifi-1.11.x updated: NIFI-7069: Add groovy-dateutil module to scripting bundles
This is an automated email from the ASF dual-hosted git repository.
joewitt pushed a commit to branch support/nifi-1.11.x
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/support/nifi-1.11.x by this
push:
new c255156 NIFI-7069: Add groovy-dateutil module to scripting bundles
c255156 is described below
commit c255156f39a9bef6c54601295d3ec9856ecca759
Author: Matthew Burgess
AuthorDate: Mon Jan 27 15:14:42 2020 -0500
NIFI-7069: Add groovy-dateutil module to scripting bundles
Signed-off-by: Pierre Villard
This closes #4021.
---
.../nifi-groovyx-nar/src/main/resources/META-INF/NOTICE | 15 +++
.../nifi-groovyx-bundle/nifi-groovyx-processors/pom.xml | 6 ++
.../src/test/resources/groovy/test_onTrigger.groovy | 3 +++
.../nifi-scripting-nar/src/main/resources/META-INF/NOTICE | 15 +++
.../nifi-scripting-processors/pom.xml | 8 +++-
.../src/test/resources/groovy/test_reader.groovy | 3 +++
6 files changed, 49 insertions(+), 1 deletion(-)
diff --git
a/nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-nar/src/main/resources/META-INF/NOTICE
b/nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-nar/src/main/resources/META-INF/NOTICE
index 5ea8698..805dee5 100644
---
a/nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-nar/src/main/resources/META-INF/NOTICE
+++
b/nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-nar/src/main/resources/META-INF/NOTICE
@@ -47,6 +47,21 @@ The following binary components are provided under the
Apache Software License v
Licensed under the Creative Commons Attribution Licence v2.5
http://creativecommons.org/licenses/by/2.5/
+ (ASLv2) Groovy Dateutil
+The following NOTICE information applies:
+ Groovy Language
+ Copyright 2003-2018 The respective authors and developers
+ Developers and Contributors are listed in the project POM file
+ and Gradle build file
+
+ This product includes software developed by
+ The Groovy community (http://groovy.codehaus.org/).
+
+ This product bundles icons from the famfamfam.com silk icons set
+ http://www.famfamfam.com/lab/icons/silk/
+ Licensed under the Creative Commons Attribution Licence v2.5
+ http://creativecommons.org/licenses/by/2.5/
+
(ASLv2) Apache Ant
The following NOTICE information applies:
Copyright 1999-2018 The Apache Software Foundation
diff --git
a/nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/pom.xml
b/nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/pom.xml
index c42c0de..18f69ea 100644
--- a/nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/pom.xml
+++ b/nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/pom.xml
@@ -53,6 +53,12 @@
${nifi.groovy.version}
provided
+
+
+org.codehaus.groovy
+groovy-dateutil
+${nifi.groovy.version}
+
org.apache.ivy
ivy
diff --git
a/nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/test/resources/groovy/test_onTrigger.groovy
b/nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/test/resources/groovy/test_onTrigger.groovy
index e78307d..827f148 100644
---
a/nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/test/resources/groovy/test_onTrigger.groovy
+++
b/nifi-nar-bundles/nifi-groovyx-bundle/nifi-groovyx-processors/src/test/resources/groovy/test_onTrigger.groovy
@@ -19,6 +19,9 @@ if (flowFile == null) {
context?.yield();
}
else {
+// Try to parse a date here, will fail after Groovy 2.5.0 if
groovy-dateutil is not included
+Date.parse('MMdd', '20190630')
+
flowFile = session.putAttribute(flowFile, "from-content", "test content")
session.transfer(flowFile, REL_SUCCESS)
}
\ No newline at end of file
diff --git
a/nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-nar/src/main/resources/META-INF/NOTICE
b/nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-nar/src/main/resources/META-INF/NOTICE
index 4f3dc76..aa0c33c 100644
---
a/nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-nar/src/main/resources/META-INF/NOTICE
+++
b/nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-nar/src/main/resources/META-INF/NOTICE
@@ -47,6 +47,21 @@ The following binary components are provided under the
Apache Software License v
Licensed under the Creative Commons Attribution Licence v2.5
http://creativecommons.org/licenses/by/2.5/
+ (ASLv2) Groovy Dateutil
+The following NOTICE information applies:
+ Groovy Language
+ Copyright 2003-2018 The respective authors and developers
+ Developers and Contributors are listed in the project POM file
+ and Gradle build file
+
+ This product includes software developed by
+
[nifi] branch master updated (04fae9c -> 850869c)
This is an automated email from the ASF dual-hosted git repository. aboda pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/nifi.git. from 04fae9c NIFI-7069: Add groovy-dateutil module to scripting bundles add 850869c NIFI-7049 : SFTP processors shouldn't silently try to access known hosts file of the user No new revisions were added by this update. Summary of changes: .../processors/standard/util/SFTPTransfer.java | 20 +-- .../nifi/processors/standard/TestGetSFTP.java | 29 ++ 2 files changed, 41 insertions(+), 8 deletions(-)
