[GitHub] nifi pull request #3033: NIFI-5629 GetFile vast listing performance
Github user adyoun2 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3033#discussion_r225123174
--- Diff:
nifi-commons/nifi-properties/src/main/java/org/apache/nifi/util/NiFiProperties.java
---
@@ -1422,10 +1424,15 @@ public String
getDefaultBackPressureDataSizeThreshold() {
public static NiFiProperties createBasicNiFiProperties(final String
propertiesFilePath, final Map additionalProperties) {
final Map addProps = (additionalProperties ==
null) ? Collections.EMPTY_MAP : additionalProperties;
final Properties properties = new Properties();
-final String nfPropertiesFilePath = (propertiesFilePath == null)
+String nfPropertiesFilePath = (propertiesFilePath == null)
? System.getProperty(NiFiProperties.PROPERTIES_FILE_PATH)
: propertiesFilePath;
if (nfPropertiesFilePath != null) {
+try {
--- End diff --
it was relevant, to get it working in my local environment, where someone
had put spaces in the path, %20 in resource paths needed converting back to ' '
---
[jira] [Commented] (NIFI-5629) GetFile becomes slow listing vast directories
[
https://issues.apache.org/jira/browse/NIFI-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650069#comment-16650069
]
ASF GitHub Bot commented on NIFI-5629:
--
Github user adyoun2 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3033#discussion_r225122894
--- Diff:
nifi-mock/src/main/java/org/apache/nifi/util/MockProcessSession.java ---
@@ -458,6 +458,9 @@ public MockFlowFile importFrom(final Path path, final
boolean keepSourceFile, Fl
newFlowFile.setData(baos.toByteArray());
newFlowFile = putAttribute(newFlowFile,
CoreAttributes.FILENAME.key(), path.getFileName().toString());
+if (!keepSourceFile) {
--- End diff --
It is necessary, in that the mock was ignoring the attribute, so when
tested with multiple files, behaviour was non-deterministic, as it could quite
easily pick up the same file again.
> GetFile becomes slow listing vast directories
> -
>
> Key: NIFI-5629
> URL: https://issues.apache.org/jira/browse/NIFI-5629
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.6.0
>Reporter: Adam
>Priority: Minor
>
> GetFile repeatedly lists entire directories before applying batching, meaning
> for vast directories it spends a long time listing directories.
>
> Pull request to follow.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3033: NIFI-5629 GetFile vast listing performance
Github user adyoun2 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3033#discussion_r225122894
--- Diff:
nifi-mock/src/main/java/org/apache/nifi/util/MockProcessSession.java ---
@@ -458,6 +458,9 @@ public MockFlowFile importFrom(final Path path, final
boolean keepSourceFile, Fl
newFlowFile.setData(baos.toByteArray());
newFlowFile = putAttribute(newFlowFile,
CoreAttributes.FILENAME.key(), path.getFileName().toString());
+if (!keepSourceFile) {
--- End diff --
It is necessary, in that the mock was ignoring the attribute, so when
tested with multiple files, behaviour was non-deterministic, as it could quite
easily pick up the same file again.
---
[jira] [Commented] (NIFI-5629) GetFile becomes slow listing vast directories
[
https://issues.apache.org/jira/browse/NIFI-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650065#comment-16650065
]
ASF GitHub Bot commented on NIFI-5629:
--
Github user adyoun2 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3033#discussion_r225122035
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-documentation/src/test/java/org/apache/nifi/documentation/DocGeneratorTest.java
---
@@ -72,7 +74,12 @@ public void testProcessorLoadsNarResources() throws
IOException, ClassNotFoundEx
}
private NiFiProperties loadSpecifiedProperties(final String
propertiesFile, final String key, final String value) {
-String file =
DocGeneratorTest.class.getResource(propertiesFile).getFile();
+String file = propertiesFile;
--- End diff --
Sorry, it was relevant, to get it working in my local environment, where
someone had put spaces in the path, %20 in resource paths needed converting
back to ' '
> GetFile becomes slow listing vast directories
> -
>
> Key: NIFI-5629
> URL: https://issues.apache.org/jira/browse/NIFI-5629
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.6.0
>Reporter: Adam
>Priority: Minor
>
> GetFile repeatedly lists entire directories before applying batching, meaning
> for vast directories it spends a long time listing directories.
>
> Pull request to follow.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3033: NIFI-5629 GetFile vast listing performance
Github user adyoun2 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3033#discussion_r225122035
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-documentation/src/test/java/org/apache/nifi/documentation/DocGeneratorTest.java
---
@@ -72,7 +74,12 @@ public void testProcessorLoadsNarResources() throws
IOException, ClassNotFoundEx
}
private NiFiProperties loadSpecifiedProperties(final String
propertiesFile, final String key, final String value) {
-String file =
DocGeneratorTest.class.getResource(propertiesFile).getFile();
+String file = propertiesFile;
--- End diff --
Sorry, it was relevant, to get it working in my local environment, where
someone had put spaces in the path, %20 in resource paths needed converting
back to ' '
---
[jira] [Commented] (NIFI-3469) Add multipart request support to HandleHttpRequest Processor
[
https://issues.apache.org/jira/browse/NIFI-3469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650064#comment-16650064
]
ASF GitHub Bot commented on NIFI-3469:
--
Github user ekovacs commented on the issue:
https://github.com/apache/nifi/pull/2991
thanks @ijokarumawak you were right!
added the fix for it, and an UT as well
> Add multipart request support to HandleHttpRequest Processor
>
>
> Key: NIFI-3469
> URL: https://issues.apache.org/jira/browse/NIFI-3469
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Koji Kawamura
>Assignee: Endre Kovacs
>Priority: Major
>
> Currently, HandleHttpRequest outputs a single FlowFile containing all
> multipart values as following:
> {code}
> --ef07e8bf36c274d3
> Content-Disposition: form-data; name="p1"
> v1
> --ef07e8bf36c274d3
> Content-Disposition: form-data; name="p2"
> v2
> --ef07e8bf36c274d3--
> {code}
> Many users requested adding upload files support to NiFi.
> In order for HandleHttpRequest to support multipart data we need to add
> followings (this is based on a brief researching and can be more complex or
> simple):
> We need to use HttpServletRequest#getParts() as written in this stackoverflow
> thread:
> http://stackoverflow.com/questions/3337056/convenient-way-to-parse-incoming-multipart-form-data-parameters-in-a-servlet
> Also, we probably need a custom MultiPartInputStreamParser implementation.
> Because Jetty's default implementation writes input data to temporary
> directory on file system, instead, we'd like NiFi to write those into output
> FlowFiles content in streaming fashion.
> And we need request size validation checks, threshold for those validation
> should be passed via javax.servlet.MultipartConfigElement.
> Finally, we have to do something with HandleHttpResponse processor.
> Once HandleHttpRequest processor start splitting incoming request into
> multiple output FlowFiles, we need to wait for every fragment to be
> processed, then execute HandleHttpRequest.
> I think Wait/Notify processors (available from next version) will be helpful
> here.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #2991: NIFI-3469: multipart request support added to HandleHttpRe...
Github user ekovacs commented on the issue: https://github.com/apache/nifi/pull/2991 thanks @ijokarumawak you were right! added the fix for it, and an UT as well ---
[jira] [Commented] (NIFI-5629) GetFile becomes slow listing vast directories
[
https://issues.apache.org/jira/browse/NIFI-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650054#comment-16650054
]
ASF GitHub Bot commented on NIFI-5629:
--
Github user adyoun2 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3033#discussion_r225117222
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-documentation/src/test/java/org/apache/nifi/documentation/DocGeneratorTest.java
---
@@ -72,7 +74,12 @@ public void testProcessorLoadsNarResources() throws
IOException, ClassNotFoundEx
}
private NiFiProperties loadSpecifiedProperties(final String
propertiesFile, final String key, final String value) {
-String file =
DocGeneratorTest.class.getResource(propertiesFile).getFile();
+String file = propertiesFile;
--- End diff --
I certainly don't remember making this change. I suspect it changed on
master since I forked, but I'll check.
> GetFile becomes slow listing vast directories
> -
>
> Key: NIFI-5629
> URL: https://issues.apache.org/jira/browse/NIFI-5629
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.6.0
>Reporter: Adam
>Priority: Minor
>
> GetFile repeatedly lists entire directories before applying batching, meaning
> for vast directories it spends a long time listing directories.
>
> Pull request to follow.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3033: NIFI-5629 GetFile vast listing performance
Github user adyoun2 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3033#discussion_r225117222
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-documentation/src/test/java/org/apache/nifi/documentation/DocGeneratorTest.java
---
@@ -72,7 +74,12 @@ public void testProcessorLoadsNarResources() throws
IOException, ClassNotFoundEx
}
private NiFiProperties loadSpecifiedProperties(final String
propertiesFile, final String key, final String value) {
-String file =
DocGeneratorTest.class.getResource(propertiesFile).getFile();
+String file = propertiesFile;
--- End diff --
I certainly don't remember making this change. I suspect it changed on
master since I forked, but I'll check.
---
[jira] [Updated] (MINIFICPP-632) Testing CAPI
[ https://issues.apache.org/jira/browse/MINIFICPP-632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpad Boda updated MINIFICPP-632: - Description: Ideas for Testing: Testing Tailing Log File 1) Use TestController to create temp files and directories ( see PutFileTests as an example) 2) Test that you add appropriate attributes and that content exists 3) Test negative cases for removing log files 4) Negative test cases for invalid/null arguments was: Ideas for Testing: Testing Tailing Log File 1) Use TestController to create temp files and directories ( see PutFileTests as an example) 2) Test that you add appropriate attributes and that content exists 3) Test negative cases for removing log files 4) Negative test cases for invalid/null arguments > Testing CAPI > > > Key: MINIFICPP-632 > URL: https://issues.apache.org/jira/browse/MINIFICPP-632 > Project: NiFi MiNiFi C++ > Issue Type: Epic >Reporter: Mr TheSegfault >Priority: Major > > > Ideas for Testing: > > Testing Tailing Log File > > 1) Use TestController to create temp files and directories ( see > PutFileTests as an example) > 2) Test that you add appropriate attributes and that content exists > 3) Test negative cases for removing log files > 4) Negative test cases for invalid/null arguments -- This message was sent by Atlassian JIRA (v7.6.3#76005)
