[jira] [Commented] (NIFI-2851) Improve performance of SplitText
[ https://issues.apache.org/jira/browse/NIFI-2851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15662836#comment-15662836 ] ASF GitHub Bot commented on NIFI-2851: -- GitHub user ijokarumawak opened a pull request: https://github.com/apache/nifi/pull/1215 NIFI-2851: Fixed CheckStyle error. The latest master branch fails with CheckStyle by a test class introduced by NIFI-2851. https://travis-ci.org/apache/nifi/builds/175185419 Confirmed this PR fixes the CheckStyle error locally. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ijokarumawak/nifi nifi-2851 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1215.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1215 commit 15f314c0f07df898aafb08788859a6ba4f05f177 Author: Koji Kawamura Date: 2016-11-14T05:28:05Z NIFI-2851: Fixed CheckStyle error. > Improve performance of SplitText > > > Key: NIFI-2851 > URL: https://issues.apache.org/jira/browse/NIFI-2851 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Oleg Zhurakousky > Fix For: 1.1.0 > > > SplitText is fairly CPU-intensive and quite slow. A simple flow that splits a > 1.4 million line text file into 5k line chunks and then splits those 5k line > chunks into 1 line chunks is only capable of pushing through about 10k lines > per second. This equates to about 10 MB/sec. JVisualVM shows that the > majority of the time is spent in the locateSplitPoint() method. Isolating > this code and inspecting how it works, and using some micro-benchmarking, it > appears that if we refactor the calls to InputStream.read() to instead read > into a byte array, we can improve performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] nifi pull request #1215: NIFI-2851: Fixed CheckStyle error.
GitHub user ijokarumawak opened a pull request: https://github.com/apache/nifi/pull/1215 NIFI-2851: Fixed CheckStyle error. The latest master branch fails with CheckStyle by a test class introduced by NIFI-2851. https://travis-ci.org/apache/nifi/builds/175185419 Confirmed this PR fixes the CheckStyle error locally. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ijokarumawak/nifi nifi-2851 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1215.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1215 commit 15f314c0f07df898aafb08788859a6ba4f05f177 Author: Koji Kawamura Date: 2016-11-14T05:28:05Z NIFI-2851: Fixed CheckStyle error. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-3026) S2S initial connection behavior enhancement
[
https://issues.apache.org/jira/browse/NIFI-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15662528#comment-15662528
]
ASF GitHub Bot commented on NIFI-3026:
--
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1208#discussion_r87730175
--- Diff:
nifi-commons/nifi-site-to-site-client/src/main/java/org/apache/nifi/remote/client/SiteToSiteClient.java
---
@@ -560,7 +560,8 @@ public SiteToSiteClient build() {
}
/**
- * @return the configured URL for the remote NiFi instance
+ * @return the configured URL for the remote NiFi instance.
+ * The URL string possibly contains multiple URL entries in
comma-separated format.
*/
public String getUrl() {
--- End diff --
SiteToSiteReportingTask has its own UI and it doesn't allow URL in a
comma-separated format. We need to update the reporting task, too. However,
since there're other features such as HTTP transport and Proxy support to add
to the reporting task, I issued a separate JIRA
[SiteToSiteProvenanceReportingTask to support Site-to-Site recently added
features](https://issues.apache.org/jira/browse/NIFI-3030).
> S2S initial connection behavior enhancement
> ---
>
> Key: NIFI-3026
> URL: https://issues.apache.org/jira/browse/NIFI-3026
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework, Core UI
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>
> s2s client behavior and initial connection improvement is needed.
> Current experience is this: I, as a client (e.g. minifi), connect to a nifi
> cluster of e.g. 10 nodes. but i need to specify 1 node URL to establish this
> connection. this node may not be available 100% and go down, in which case my
> initial connection won't work.
> Once S2S makes the first connection, it then has a list of all nodes, and can
> check their status. But first connection failure would be a concern if the
> specified URL is somehow not working. Usually for these problems, the client
> should be able to specify multiple urls (according to multiple target cluster
> nodes), comma-separated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] nifi pull request #1208: NIFI-3026: Support multiple remote target URLs
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1208#discussion_r87730175
--- Diff:
nifi-commons/nifi-site-to-site-client/src/main/java/org/apache/nifi/remote/client/SiteToSiteClient.java
---
@@ -560,7 +560,8 @@ public SiteToSiteClient build() {
}
/**
- * @return the configured URL for the remote NiFi instance
+ * @return the configured URL for the remote NiFi instance.
+ * The URL string possibly contains multiple URL entries in
comma-separated format.
*/
public String getUrl() {
--- End diff --
SiteToSiteReportingTask has its own UI and it doesn't allow URL in a
comma-separated format. We need to update the reporting task, too. However,
since there're other features such as HTTP transport and Proxy support to add
to the reporting task, I issued a separate JIRA
[SiteToSiteProvenanceReportingTask to support Site-to-Site recently added
features](https://issues.apache.org/jira/browse/NIFI-3030).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Updated] (NIFI-3030) SiteToSiteProvenanceReportingTask to support Site-to-Site recently added features
[ https://issues.apache.org/jira/browse/NIFI-3030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Kawamura updated NIFI-3030: Summary: SiteToSiteProvenanceReportingTask to support Site-to-Site recently added features (was: SiteToSiteReportingTask to support Site-to-Site recently added features) > SiteToSiteProvenanceReportingTask to support Site-to-Site recently added > features > - > > Key: NIFI-3030 > URL: https://issues.apache.org/jira/browse/NIFI-3030 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.0.0 >Reporter: Koji Kawamura >Priority: Minor > > Site-to-Site is used by not only RemoteProcessGroup, but also > SiteToSiteProvenanceReportingTask, Storm and Spark integration. > Storm and Spark integration let users to specify SiteToSiteClientConfig from > user program source code, so newly added features will be exposed > automatically. > However, SiteToSiteProvenanceReportingTask has its own UI, and we should > update it to support these new features. > Features added recently are: > - HTTP transport protocol, HTTP Proxy configuration (NIFI-1857) > - Multiple URLs support (NIFI-3026) > I marked this as 'Minor' issue as there is a workaround to use these features > to send provenance via Site-to-Site, that is using > SiteToSiteProvenanceReportingTask to send provenance to the same NiFi > instance, then send it using a RemoteProcessGroup to the final remote > destination. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-3030) SiteToSiteReportingTask to support Site-to-Site recently added features
Koji Kawamura created NIFI-3030: --- Summary: SiteToSiteReportingTask to support Site-to-Site recently added features Key: NIFI-3030 URL: https://issues.apache.org/jira/browse/NIFI-3030 Project: Apache NiFi Issue Type: Improvement Components: Extensions Affects Versions: 1.0.0 Reporter: Koji Kawamura Priority: Minor Site-to-Site is used by not only RemoteProcessGroup, but also SiteToSiteProvenanceReportingTask, Storm and Spark integration. Storm and Spark integration let users to specify SiteToSiteClientConfig from user program source code, so newly added features will be exposed automatically. However, SiteToSiteProvenanceReportingTask has its own UI, and we should update it to support these new features. Features added recently are: - HTTP transport protocol, HTTP Proxy configuration (NIFI-1857) - Multiple URLs support (NIFI-3026) I marked this as 'Minor' issue as there is a workaround to use these features to send provenance via Site-to-Site, that is using SiteToSiteProvenanceReportingTask to send provenance to the same NiFi instance, then send it using a RemoteProcessGroup to the final remote destination. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2876) Refactor TextLineDemarcator and StreamDemarcator into a common abstract class
[ https://issues.apache.org/jira/browse/NIFI-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg Zhurakousky updated NIFI-2876: --- Status: Patch Available (was: In Progress) > Refactor TextLineDemarcator and StreamDemarcator into a common abstract class > - > > Key: NIFI-2876 > URL: https://issues.apache.org/jira/browse/NIFI-2876 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Oleg Zhurakousky >Assignee: Oleg Zhurakousky >Priority: Minor > Fix For: 1.1.0 > > > Based on the work that has been performed as part of the NIFI-2851 we now > have a new class with a significantly faster logic to perform demarcation of > the InputStream (TextLineDemarcator). This new class's initial starting point > was the existing LineDemarcator. They both now share ~60-70% of common code > which would be important to extract into a common abstract class as well as > incorporate the new (faster) demarcation logic int StreamDemarcator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2876) Refactor TextLineDemarcator and StreamDemarcator into a common abstract class
[ https://issues.apache.org/jira/browse/NIFI-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15661540#comment-15661540 ] ASF GitHub Bot commented on NIFI-2876: -- GitHub user olegz opened a pull request: https://github.com/apache/nifi/pull/1214 NIFI-2876 refactored demarcators into a common abstract class Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. NIFI-2876 polish You can merge this pull request into a Git repository by running: $ git pull https://github.com/olegz/nifi NIFI-2876 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1214.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1214 commit 8adf923f79a599a027a8066dca3a03fed160e2b8 Author: Oleg Zhurakousky Date: 2016-11-11T22:36:18Z NIFI-2876 refactored demarcators into a common abstract class NIFI-2876 polish > Refactor TextLineDemarcator and StreamDemarcator into a common abstract class > - > > Key: NIFI-2876 > URL: https://issues.apache.org/jira/browse/NIFI-2876 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Oleg Zhurakousky >Assignee: Oleg Zhurakousky >Priority: Minor > Fix For: 1.1.0 > > > Based on the work that has been performed as part of the NIFI-2851 we now > have a new class with a significantly faster logic to perform demarcation of > the InputStream (TextLineDemarcator). This new class's initial starting point > was the existing LineDemarcator. They both now share ~60-70% of common code > which would be important to extract into a common abstract class as well as > incorporate the new (faster) demarcation logic int StreamDemarcator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] nifi pull request #1214: NIFI-2876 refactored demarcators into a common abst...
GitHub user olegz opened a pull request: https://github.com/apache/nifi/pull/1214 NIFI-2876 refactored demarcators into a common abstract class Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. NIFI-2876 polish You can merge this pull request into a Git repository by running: $ git pull https://github.com/olegz/nifi NIFI-2876 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1214.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1214 commit 8adf923f79a599a027a8066dca3a03fed160e2b8 Author: Oleg Zhurakousky Date: 2016-11-11T22:36:18Z NIFI-2876 refactored demarcators into a common abstract class NIFI-2876 polish --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
