[GitHub] [nifi] archongum commented on issue #3599: NIFI-6480: PutORC/PutParquet can't overwrite file even if set 'Overwr…
archongum commented on issue #3599: NIFI-6480: PutORC/PutParquet can't overwrite file even if set 'Overwr… URL: https://github.com/apache/nifi/pull/3599#issuecomment-526784826 @bbende Any other things that need to be added? This is a minor change, cc if you have time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] rfellows opened a new pull request #3685: NIFI-6510 - Support enable disable of prediction in ui
rfellows opened a new pull request #3685: NIFI-6510 - Support enable disable of
prediction in ui
URL: https://github.com/apache/nifi/pull/3685
* Updated the Connection summary table to only show the prediction column if
predictions are available
* Updated back pressure tooltips to indicate a distinction between Queue
Prediction not being configured and not able to make a prediction just yet.
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
_Enables X functionality; fixes bug NIFI-._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [X] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [X] Has your PR been rebased against the latest commit within the target
branch (typically `master`)?
- [X] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on both JDK 8 and
JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] mattyb149 commented on issue #3684: NIFI-6295: Refactored NiFiRecordSerDe to handle nested complex types
mattyb149 commented on issue #3684: NIFI-6295: Refactored NiFiRecordSerDe to handle nested complex types URL: https://github.com/apache/nifi/pull/3684#issuecomment-526752138 @gideonkorir I started with your #3509 commit and did the refactor (a la JsonSerDe) as we discussed, plus tested with many types (both simple/complex and nested). Mind taking a look at this and/or trying it out? Would like to know if it works for you and if you're ok with the way it ended up. I welcome all comments and will incorporate any changes that should be made. I'll find an Apache NiFi Committer to review as well (as they'll need to be the ones to merge this in). Thanks very much for your contribution! Looking forward to getting this across the finish line together :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] mattyb149 opened a new pull request #3684: NIFI-6295: Refactored NiFiRecordSerDe to handle nested complex types
mattyb149 opened a new pull request #3684: NIFI-6295: Refactored
NiFiRecordSerDe to handle nested complex types
URL: https://github.com/apache/nifi/pull/3684
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
_Enables X functionality; fixes bug NIFI-._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically `master`)?
- [ ] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [x] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on both JDK 8 and
JDK 11?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] dbkegley closed pull request #3651: NIFI-6539 Nifi stateless init from flow xml
dbkegley closed pull request #3651: NIFI-6539 Nifi stateless init from flow xml URL: https://github.com/apache/nifi/pull/3651 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] dbkegley commented on issue #3651: NIFI-6539 Nifi stateless init from flow xml
dbkegley commented on issue #3651: NIFI-6539 Nifi stateless init from flow xml URL: https://github.com/apache/nifi/pull/3651#issuecomment-526732681 My goal for this was only to remove the requirement of using registry to run stateless flows so I'm in agreement about the points against using the flow.xml.gz. Using a configuration that's more easily version controlled makes perfect sense to me, though I would rather follow MiNiFi's approach of using YAML, but that's just personal preference. I'd still be happy to help out with this once there's a more acceptable approach. I'll keep an eye on the mailing list for those discussions. Closing this PR for now This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] rfellows opened a new pull request #3683: NIFI-6506 - Add ability to convert properties to parameters
rfellows opened a new pull request #3683: NIFI-6506 - Add ability to convert
properties to parameters
URL: https://github.com/apache/nifi/pull/3683
* open existing add param dialog from property table
* convert prop to param, update param context
* keep add parameter dialog open until update of the context completes when
converting property to param
* Updating parameter context while converting property to parameter.
* Disable the Apply button while updating
* When Cancel is pressed and there is an update in progress, issue the
DELETE request to cancel the update.
* Show some status in the Add Parameter dialog when parameter context is
updating following a property conversion.
* enforce character restrictions on property names.
* Add convert property to parameter to controller service config.
* Properly set sensitive setting when converting sensitive properties
* Allow converting of properties from controller services configuration into
parameters
* Refactor addNewParameter method to be able to share common parts with
convertPropertyToParameter
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
_Enables X functionality; fixes bug NIFI-._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [X] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [X] Has your PR been rebased against the latest commit within the target
branch (typically `master`)?
- [X] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on both JDK 8 and
JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] SamHjelmfelt edited a comment on issue #3651: NIFI-6539 Nifi stateless init from flow xml
SamHjelmfelt edited a comment on issue #3651: NIFI-6539 Nifi stateless init from flow xml URL: https://github.com/apache/nifi/pull/3651#issuecomment-526724080 (changed after talking to a couple others) I agree with Joe. The flow.xml.gz is not the best way to solve for scenarios where a registry is not available. The biggest risk in using the flow.xml.gz directly is that it makes Stateless NiFi look like a drop in replacement for a NiFi node and encourages a usage pattern that avoids version control. Not to mention that the flow.xml.gz was never intended as a public interface, so it could change unexpectedly between versions. MiNiFi chose to use templates (XML/YML files) as a way of loading flow definitions without the registry, but it sounds like templates may be on their way out. I will start a discussion thread on the dev list to talk about file-based flow representations and the best solution for when a registry is not available. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] SamHjelmfelt edited a comment on issue #3651: NIFI-6539 Nifi stateless init from flow xml
SamHjelmfelt edited a comment on issue #3651: NIFI-6539 Nifi stateless init from flow xml URL: https://github.com/apache/nifi/pull/3651#issuecomment-526724080 (changed after talking to others) I agree with Joe. The flow.xml.gz is not the best way to solve for scenarios where a registry is not available. The biggest risk in using the flow.xml.gz directly is that it makes Stateless NiFi look like a drop in replacement for a NiFi node and encourages a usage pattern that avoids version control. Not to mention that the flow.xml.gz was never intended as a public interface, so it could change unexpectedly between versions. MiNiFi chose to use templates (XML/YML files) as a way of loading flow definitions without the registry, but it sounds like templates may be on their way out. I will start a discussion thread on the dev list to talk about file-based flow representations and the best solution for when a registry is not available. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] SamHjelmfelt commented on issue #3651: NIFI-6539 Nifi stateless init from flow xml
SamHjelmfelt commented on issue #3651: NIFI-6539 Nifi stateless init from flow xml URL: https://github.com/apache/nifi/pull/3651#issuecomment-526724080 I agree with Joe. Providing the option of using Stateless NiFi without the NiFi Registry does make sense, but it would be better to use NiFi templates (XML or YML) like MiNiFi does today. Rather than using the internal configuration file of the node, users would export flows from NiFi and use them in Stateless NiFi. The biggest risk in using the flow.xml.gz directly is that it makes Stateless NiFi look like a drop in replacement for a NiFi node and encourages a usage pattern that avoids version control. The differences between Stateless NiFi and NiFi are far greater than NiFi and MiNiFi, which still does not use the flow.xml.gz. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (NIFI-6608) UI: Reporting Task Controller Services should not show that Parameters are supported
[ https://issues.apache.org/jira/browse/NIFI-6608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Lim updated NIFI-6608: - Attachment: reporting_task_cs_property.png > UI: Reporting Task Controller Services should not show that Parameters are > supported > > > Key: NIFI-6608 > URL: https://issues.apache.org/jira/browse/NIFI-6608 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Andrew Lim >Priority: Major > Attachments: reporting_task_cs_property.png > > > As shown in the attached screenshot, the UI shows that Parameters are > supported for Reporting Task Controller Services properties even though this > is not true. > The behavior is correct for Reporting Tasks. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (NIFI-6608) UI: Reporting Task Controller Services should not show that Parameters are supported
Andrew Lim created NIFI-6608: Summary: UI: Reporting Task Controller Services should not show that Parameters are supported Key: NIFI-6608 URL: https://issues.apache.org/jira/browse/NIFI-6608 Project: Apache NiFi Issue Type: Sub-task Components: Core UI Reporter: Andrew Lim As shown in the attached screenshot, the UI shows that Parameters are supported for Reporting Task Controller Services properties even though this is not true. The behavior is correct for Reporting Tasks. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (NIFI-6606) User should not be able to select to configure a PG with a parameter context for which they do not have read permissions
[ https://issues.apache.org/jira/browse/NIFI-6606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman reassigned NIFI-6606: - Assignee: Matt Gilman > User should not be able to select to configure a PG with a parameter context > for which they do not have read permissions > > > Key: NIFI-6606 > URL: https://issues.apache.org/jira/browse/NIFI-6606 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Scott Aslan >Assignee: Matt Gilman >Priority: Major > Attachments: Screen Shot 2019-08-30 at 1.17.08 PM.png > > > When configuring a PG with a parameter context for which the user do not have > read permissions for the parameter context UUID should not be selectable: > > !Screen Shot 2019-08-30 at 1.17.08 PM.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] mcgilman commented on issue #3637: NIFI-6400 Better options, consistent ids for ShellUserGroupProvider.
mcgilman commented on issue #3637: NIFI-6400 Better options, consistent ids for ShellUserGroupProvider. URL: https://github.com/apache/nifi/pull/3637#issuecomment-526702850 @natural Thanks for the update! We should be able to leverage the methods in `User.Builder` and `Group.Builder` to remove the private method for `getNameBasedUUID`. There is admittedly one scenario where the `Group` that is being created is only needed temporarily but I think this would use the APIs for as the were expected. I've typed up an example here [1]. Happy to discuss it and work through it more. [1] https://github.com/mcgilman/nifi/commit/c423a016cc00816436191962a4eda9445f7abe17 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (NIFI-6606) User should not be able to select to configure a PG with a parameter context for which they do not have read permissions
[ https://issues.apache.org/jira/browse/NIFI-6606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman updated NIFI-6606: -- Parent: NIFI-6276 Issue Type: Sub-task (was: Improvement) > User should not be able to select to configure a PG with a parameter context > for which they do not have read permissions > > > Key: NIFI-6606 > URL: https://issues.apache.org/jira/browse/NIFI-6606 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Scott Aslan >Priority: Major > Attachments: Screen Shot 2019-08-30 at 1.17.08 PM.png > > > When configuring a PG with a parameter context for which the user do not have > read permissions for the parameter context UUID should not be selectable: > > !Screen Shot 2019-08-30 at 1.17.08 PM.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (NIFI-6602) Parameters: "Apply" button should be greyed out if no changes have been
[ https://issues.apache.org/jira/browse/NIFI-6602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman updated NIFI-6602: -- Parent: NIFI-6276 Issue Type: Sub-task (was: Improvement) > Parameters: "Apply" button should be greyed out if no changes have been > --- > > Key: NIFI-6602 > URL: https://issues.apache.org/jira/browse/NIFI-6602 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Andrew Lim >Priority: Minor > > Open the Parameters Context window. Whether on the Settings tab or Parameters > tab, the "Apply" button is available for selection immediately even when no > edits have been made to apply. If selected, the window is closed which is > what the Cancel button is for. > > Noticed while testing the PR branch for NIFI-6282. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] YolandaMDavis commented on a change in pull request #3671: NIFI-6586 - Code comments and doc updates
YolandaMDavis commented on a change in pull request #3671: NIFI-6586 - Code comments and doc updates URL: https://github.com/apache/nifi/pull/3671#discussion_r319623326 ## File path: nifi-docs/src/main/asciidoc/user-guide.adoc ## @@ -1685,6 +1685,8 @@ The FlowFiles enqueued in a Connection can also be deleted when necessary. The r via `Empty queue` in the Connection's context menu. This action can also be performed if the source and destination are actively running. +Hovering over the queue will also reveal predicted statistics on when the queue may encounter back pressure, either due to the object count or content size meeting the current threshold +settings. Predictions will only be available when NiFi has enough data in it's internal repository and if it's model is accurate enough to broadcast a prediction. Review comment: @ottobackwards thanks for this question, it prompted us to ensure we add enabled/disabled property for the new features (commit has been added to the PR against master #3681. UI changes are pending (did ask @rfellows offline) but in short tooltip will distinguish when predictions are enabled yet not available vs when predictions are not enabled. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] YolandaMDavis commented on a change in pull request #3671: NIFI-6586 - Code comments and doc updates
YolandaMDavis commented on a change in pull request #3671: NIFI-6586 - Code comments and doc updates URL: https://github.com/apache/nifi/pull/3671#discussion_r319623326 ## File path: nifi-docs/src/main/asciidoc/user-guide.adoc ## @@ -1685,6 +1685,8 @@ The FlowFiles enqueued in a Connection can also be deleted when necessary. The r via `Empty queue` in the Connection's context menu. This action can also be performed if the source and destination are actively running. +Hovering over the queue will also reveal predicted statistics on when the queue may encounter back pressure, either due to the object count or content size meeting the current threshold +settings. Predictions will only be available when NiFi has enough data in it's internal repository and if it's model is accurate enough to broadcast a prediction. Review comment: @ottobackwards thanks for this question, it prompted us to ensure we add enabled/disabled property for the new features (commit has been added to the [PR against master](#3681 3681)). UI changes are pending (did ask @rfellows offline) but in short tooltip will distinguish when predictions are enabled yet not available vs when predictions are not enabled. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] YolandaMDavis commented on issue #3681: NIFI-6510 [WIP] - Analytics framework
YolandaMDavis commented on issue #3681: NIFI-6510 [WIP] - Analytics framework URL: https://github.com/apache/nifi/pull/3681#issuecomment-526696145 Relabeling this PR as WIP due to newer commit to support enable/disable of feature (with disable being the default). @rfellows this will require an update to the UI to ensure that user can determine if predictions are enabled vs available. Once that commit is available I'll remove WIP from heading. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (NIFI-6424) Null transit url in the Gremlin client service
[ https://issues.apache.org/jira/browse/NIFI-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-6424: --- Fix Version/s: 1.10.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Null transit url in the Gremlin client service > -- > > Key: NIFI-6424 > URL: https://issues.apache.org/jira/browse/NIFI-6424 > Project: Apache NiFi > Issue Type: Bug >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Major > Fix For: 1.10.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[
https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919803#comment-16919803
]
ASF subversion and git services commented on NIFI-6510:
---
Commit 7e6eddd0570ecceeb8e14e927cf257c731375fc9 in nifi's branch
refs/heads/analytics-framework from Yolanda M. Davis
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=7e6eddd ]
NIFI-6510 - documentation updates for enable/disable property
> Predictive Analytics for NiFi Metrics
> -
>
> Key: NIFI-6510
> URL: https://issues.apache.org/jira/browse/NIFI-6510
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Andrew Christianson
>Assignee: Yolanda M. Davis
>Priority: Major
> Fix For: 1.10.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> From Yolanda's email to the list:
>
> {noformat}
> Currently NiFi has lots of metrics available for areas including jvm and flow
> component usage (via component status) as well as provenance data which NiFi
> makes available either through the UI or reporting tasks (for consumption by
> other systems). Past discussions in the community cite users shipping this
> data to applications such as Prometheus, ELK stacks, or Ambari metrics for
> further analysis in order to capture/review performance issues, detect
> anomalies, and send alerts or notifications. These systems are efficient in
> capturing and helping to analyze these metrics however it requires
> customization work and knowledge of NiFi operations to provide meaningful
> analytics within a flow context.
> In speaking with Matt Burgess and Andy Christianson on this topic we feel
> that there is an opportunity to introduce an analytics framework that could
> provide users reasonable predictions on key performance indicators for flows,
> such as back pressure and flow rate, to help administrators improve
> operational management of NiFi clusters. This framework could offer several
> key features:
> - Provide a flexible internal analytics engine and model api which supports
> the addition of or enhancement to onboard models
> - Support integration of remote or cloud based ML models
> - Support both traditional and online (incremental) learning methods
> - Provide support for model caching (perhaps later inclusion into a model
> repository or registry)
> - UI enhancements to display prediction information either in existing
> summary data, new data visualizations, or directly within the flow/canvas
> (where applicable)
> For an initial target we thought that back pressure prediction would be a
> good starting point for this initiative, given that back pressure detection
> is a key indicator of flow performance and many of the metrics currently
> available would provide enough data points to create a reasonable performing
> model. We have some ideas on how this could be achieved however we wanted to
> discuss this more with the community to get thoughts about tackling this
> work, especially if there are specific use cases or other factors that should
> be considered.{noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (NIFI-6424) Null transit url in the Gremlin client service
[ https://issues.apache.org/jira/browse/NIFI-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919802#comment-16919802 ] ASF subversion and git services commented on NIFI-6424: --- Commit fc3477bd69be066ba7f75b90f9e58e18ee3b176c in nifi's branch refs/heads/master from Mike Thomsen [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=fc3477b ] NIFI-6424 Created a proper transit URL for Gremlin and OpenCypher services. Signed-off-by: Matthew Burgess This closes #3571 > Null transit url in the Gremlin client service > -- > > Key: NIFI-6424 > URL: https://issues.apache.org/jira/browse/NIFI-6424 > Project: Apache NiFi > Issue Type: Bug >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (NIFI-6607) UI: State of referencing processors in Parameter Context dialog do not update after "Apply"
Andrew Lim created NIFI-6607: Summary: UI: State of referencing processors in Parameter Context dialog do not update after "Apply" Key: NIFI-6607 URL: https://issues.apache.org/jira/browse/NIFI-6607 Project: Apache NiFi Issue Type: Sub-task Components: Core UI Reporter: Andrew Lim Change a parameter in a parameter context and select "Apply". The state of the referencing processors do not update. For example, if the referencing processor is in Invalid state and I corrected the problem with the parameter, the state will stay in invalid state but on the canvas it is no longer invalid. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] mattyb149 commented on issue #3571: NIFI-6424 Created a proper transit URL for Gremlin and OpenCypher ser…
mattyb149 commented on issue #3571: NIFI-6424 Created a proper transit URL for Gremlin and OpenCypher ser… URL: https://github.com/apache/nifi/pull/3571#issuecomment-526694582 +1 LGTM, thanks for the improvement! Merging to master This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] asfgit closed pull request #3571: NIFI-6424 Created a proper transit URL for Gremlin and OpenCypher ser…
asfgit closed pull request #3571: NIFI-6424 Created a proper transit URL for Gremlin and OpenCypher ser… URL: https://github.com/apache/nifi/pull/3571 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (NIFI-6424) Null transit url in the Gremlin client service
[ https://issues.apache.org/jira/browse/NIFI-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-6424: --- Status: Patch Available (was: Open) > Null transit url in the Gremlin client service > -- > > Key: NIFI-6424 > URL: https://issues.apache.org/jira/browse/NIFI-6424 > Project: Apache NiFi > Issue Type: Bug >Reporter: Mike Thomsen >Assignee: Mike Thomsen >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (NIFI-6418) UI - Need property editor that does not support parameters
[ https://issues.apache.org/jira/browse/NIFI-6418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Aslan updated NIFI-6418: -- Resolution: Fixed Status: Resolved (was: Patch Available) > UI - Need property editor that does not support parameters > -- > > Key: NIFI-6418 > URL: https://issues.apache.org/jira/browse/NIFI-6418 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Matt Gilman >Assignee: Matt Gilman >Priority: Major > > Reporting tasks and their Controller Services cannot reference parameters. We > need to update the property editors to also not allow parameter references. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (NIFI-6346) UI - Allow paramter reference within EL
[
https://issues.apache.org/jira/browse/NIFI-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Scott Aslan updated NIFI-6346:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
> UI - Allow paramter reference within EL
> ---
>
> Key: NIFI-6346
> URL: https://issues.apache.org/jira/browse/NIFI-6346
> Project: Apache NiFi
> Issue Type: Sub-task
> Components: Core UI
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Major
>
> Need to update `nfel` to support referencing parameters within expression
> language. A parameter reference should be allowed outside of an expression,
> as a subject to an expression, or as a parameter to a function in an
> expression.
> {code}
> ${#{abc}:toUpper()}
> #{abc}${attr:toUpper()}
> ${attr:contains(#{abc})}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (NIFI-6277) UI - Allow parameters to be referenced in component configuration
[ https://issues.apache.org/jira/browse/NIFI-6277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Aslan updated NIFI-6277: -- Resolution: Fixed Status: Resolved (was: Patch Available) > UI - Allow parameters to be referenced in component configuration > - > > Key: NIFI-6277 > URL: https://issues.apache.org/jira/browse/NIFI-6277 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Matt Gilman >Assignee: Matt Gilman >Priority: Major > > Update component configuration to allow for parameter references. This JIRA > specifically will be focused on property values that do not support EL. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (NIFI-6345) UI - Distinguish nfpr and nfel
[ https://issues.apache.org/jira/browse/NIFI-6345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Aslan updated NIFI-6345: -- Resolution: Fixed Status: Resolved (was: Patch Available) > UI - Distinguish nfpr and nfel > -- > > Key: NIFI-6345 > URL: https://issues.apache.org/jira/browse/NIFI-6345 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Matt Gilman >Assignee: Matt Gilman >Priority: Major > > Need to introduce some UX to better distinguish editor modes `nfpr` and > `nfel` so the user is aware of the available actions. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (NIFI-6290) UI - Support parameter reference when property contains allowable values
[ https://issues.apache.org/jira/browse/NIFI-6290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Aslan updated NIFI-6290: -- Resolution: Fixed Status: Resolved (was: Patch Available) > UI - Support parameter reference when property contains allowable values > > > Key: NIFI-6290 > URL: https://issues.apache.org/jira/browse/NIFI-6290 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Matt Gilman >Assignee: Matt Gilman >Priority: Major > > This Jira is to introduce support for referencing a parameter when a property > descriptor has allowable values. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] scottyaslan commented on issue #3636: Allow parameters to be used in property configuration
scottyaslan commented on issue #3636: Allow parameters to be used in property configuration URL: https://github.com/apache/nifi/pull/3636#issuecomment-526692328 @mcgilman Thanks! This has been merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (NIFI-6290) UI - Support parameter reference when property contains allowable values
[ https://issues.apache.org/jira/browse/NIFI-6290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919788#comment-16919788 ] ASF subversion and git services commented on NIFI-6290: --- Commit d18c98ce780447b2059872a45a0200f578587d4e in nifi's branch refs/heads/master from Matt Gilman [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=d18c98c ] NIFI-6290: - Updating comboEditor to allow for parameter reference. > UI - Support parameter reference when property contains allowable values > > > Key: NIFI-6290 > URL: https://issues.apache.org/jira/browse/NIFI-6290 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Matt Gilman >Assignee: Matt Gilman >Priority: Major > > This Jira is to introduce support for referencing a parameter when a property > descriptor has allowable values. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6290) UI - Support parameter reference when property contains allowable values
[ https://issues.apache.org/jira/browse/NIFI-6290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919791#comment-16919791 ] ASF subversion and git services commented on NIFI-6290: --- Commit f318a8c68524a268b819b1faa1b37f3f9e3190a4 in nifi's branch refs/heads/master from Matt Gilman [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f318a8c ] NIFI-6290: - Ensuring the first allowable value is selected. > UI - Support parameter reference when property contains allowable values > > > Key: NIFI-6290 > URL: https://issues.apache.org/jira/browse/NIFI-6290 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Matt Gilman >Assignee: Matt Gilman >Priority: Major > > This Jira is to introduce support for referencing a parameter when a property > descriptor has allowable values. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6346) UI - Allow paramter reference within EL
[
https://issues.apache.org/jira/browse/NIFI-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919787#comment-16919787
]
ASF subversion and git services commented on NIFI-6346:
---
Commit d93ae47afcdea8df6982f9d093830caa607a833c in nifi's branch
refs/heads/master from Matt Gilman
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=d93ae47 ]
NIFI-6346:
- Updating nfel to support referencing parameters.
> UI - Allow paramter reference within EL
> ---
>
> Key: NIFI-6346
> URL: https://issues.apache.org/jira/browse/NIFI-6346
> Project: Apache NiFi
> Issue Type: Sub-task
> Components: Core UI
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Major
>
> Need to update `nfel` to support referencing parameters within expression
> language. A parameter reference should be allowed outside of an expression,
> as a subject to an expression, or as a parameter to a function in an
> expression.
> {code}
> ${#{abc}:toUpper()}
> #{abc}${attr:toUpper()}
> ${attr:contains(#{abc})}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (NIFI-6277) UI - Allow parameters to be referenced in component configuration
[ https://issues.apache.org/jira/browse/NIFI-6277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919785#comment-16919785 ] ASF subversion and git services commented on NIFI-6277: --- Commit 75e452b45c5d3096667b8acbd1ae1ad1444e7e12 in nifi's branch refs/heads/master from Matt Gilman [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=75e452b ] NIFI-6277: - Allow parameters to be referenced from properties that do not support EL. > UI - Allow parameters to be referenced in component configuration > - > > Key: NIFI-6277 > URL: https://issues.apache.org/jira/browse/NIFI-6277 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Matt Gilman >Assignee: Matt Gilman >Priority: Major > > Update component configuration to allow for parameter references. This JIRA > specifically will be focused on property values that do not support EL. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6290) UI - Support parameter reference when property contains allowable values
[ https://issues.apache.org/jira/browse/NIFI-6290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919790#comment-16919790 ] ASF subversion and git services commented on NIFI-6290: --- Commit e5850fa1a3e37daabd6b6596c869ac903daf340a in nifi's branch refs/heads/master from Matt Gilman [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=e5850fa ] NIFI-6290: - Addressing timing issue when parameters listing request is not immediately resolved. > UI - Support parameter reference when property contains allowable values > > > Key: NIFI-6290 > URL: https://issues.apache.org/jira/browse/NIFI-6290 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Matt Gilman >Assignee: Matt Gilman >Priority: Major > > This Jira is to introduce support for referencing a parameter when a property > descriptor has allowable values. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6290) UI - Support parameter reference when property contains allowable values
[ https://issues.apache.org/jira/browse/NIFI-6290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919792#comment-16919792 ] ASF subversion and git services commented on NIFI-6290: --- Commit c6e500dfa44704568b58d5a867eb51eff91fd4da in nifi's branch refs/heads/master from Matt Gilman [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=c6e500d ] NIFI-6290: - Ensuring the first allowable value is selected. > UI - Support parameter reference when property contains allowable values > > > Key: NIFI-6290 > URL: https://issues.apache.org/jira/browse/NIFI-6290 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Matt Gilman >Assignee: Matt Gilman >Priority: Major > > This Jira is to introduce support for referencing a parameter when a property > descriptor has allowable values. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6290) UI - Support parameter reference when property contains allowable values
[ https://issues.apache.org/jira/browse/NIFI-6290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919793#comment-16919793 ] ASF subversion and git services commented on NIFI-6290: --- Commit bfbf3b753276015523fd9e56e397c8d3039fdfdb in nifi's branch refs/heads/master from Matt Gilman [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=bfbf3b7 ] NIFI-6290: - Addressing review feedback. This closes #3636 Signed-off-by: Scott Aslan > UI - Support parameter reference when property contains allowable values > > > Key: NIFI-6290 > URL: https://issues.apache.org/jira/browse/NIFI-6290 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Matt Gilman >Assignee: Matt Gilman >Priority: Major > > This Jira is to introduce support for referencing a parameter when a property > descriptor has allowable values. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6418) UI - Need property editor that does not support parameters
[ https://issues.apache.org/jira/browse/NIFI-6418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919789#comment-16919789 ] ASF subversion and git services commented on NIFI-6418: --- Commit 5b409d6facd92d288f87c0b379f9cb8b8cafb631 in nifi's branch refs/heads/master from Matt Gilman [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=5b409d6 ] NIFI-6418: - Updating property table editors to not support parameter referencing when necessary. > UI - Need property editor that does not support parameters > -- > > Key: NIFI-6418 > URL: https://issues.apache.org/jira/browse/NIFI-6418 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Matt Gilman >Assignee: Matt Gilman >Priority: Major > > Reporting tasks and their Controller Services cannot reference parameters. We > need to update the property editors to also not allow parameter references. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6345) UI - Distinguish nfpr and nfel
[ https://issues.apache.org/jira/browse/NIFI-6345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919786#comment-16919786 ] ASF subversion and git services commented on NIFI-6345: --- Commit 7228496801735054c44fb9d0dbc75ce51be05f56 in nifi's branch refs/heads/master from Matt Gilman [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=7228496 ] NIFI-6345: - Updating property table editor to distinguish between available modes (el vs parameter reference). > UI - Distinguish nfpr and nfel > -- > > Key: NIFI-6345 > URL: https://issues.apache.org/jira/browse/NIFI-6345 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Matt Gilman >Assignee: Matt Gilman >Priority: Major > > Need to introduce some UX to better distinguish editor modes `nfpr` and > `nfel` so the user is aware of the available actions. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] asfgit closed pull request #3636: Allow parameters to be used in property configuration
asfgit closed pull request #3636: Allow parameters to be used in property configuration URL: https://github.com/apache/nifi/pull/3636 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (NIFI-6606) User should not be able to select to configure a PG with a parameter context for which they do not have read permissions
Scott Aslan created NIFI-6606: - Summary: User should not be able to select to configure a PG with a parameter context for which they do not have read permissions Key: NIFI-6606 URL: https://issues.apache.org/jira/browse/NIFI-6606 Project: Apache NiFi Issue Type: Improvement Components: Core UI Reporter: Scott Aslan Attachments: Screen Shot 2019-08-30 at 1.17.08 PM.png When configuring a PG with a parameter context for which the user do not have read permissions for the parameter context UUID should not be selectable: !Screen Shot 2019-08-30 at 1.17.08 PM.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (NIFI-6605) EL/Param tooltip not visible for large property table editors
[ https://issues.apache.org/jira/browse/NIFI-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Aslan updated NIFI-6605: -- Description: When a user is editing a property table cell and resizes the editor to full screen or if the editor is too close to the top of the screen the tooltip for the EL/Param is no longer visible: was: When a user is editing a property table cell and resizes the editor the tooltip for the EL/Param is no longer visible: > EL/Param tooltip not visible for large property table editors > - > > Key: NIFI-6605 > URL: https://issues.apache.org/jira/browse/NIFI-6605 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Reporter: Scott Aslan >Priority: Major > Attachments: Screen Shot 2019-08-30 at 11.48.45 AM.png > > > When a user is editing a property table cell and resizes the editor to full > screen or if the editor is too close to the top of the screen the tooltip for > the EL/Param is no longer visible: > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6429) Provide option to include null values for SiteToSite Reporting Tasks
[ https://issues.apache.org/jira/browse/NIFI-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919762#comment-16919762 ] ASF subversion and git services commented on NIFI-6429: --- Commit 07baf723f72d0190cb8ba40b9ecf75ff3f23e3e0 in nifi's branch refs/heads/master from Yolanda Davis [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=07baf72 ] NIFI-6429 - Added optional field to allow variables with null values to be included in reporting records. Default functionality will exclude variables with null values for backwards compatibility. NIFI-6429 - checkstyle corrections NIFI-6429 - change property descriptor name > Provide option to include null values for SiteToSite Reporting Tasks > > > Key: NIFI-6429 > URL: https://issues.apache.org/jira/browse/NIFI-6429 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.9.2 >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > In some implementations of the SiteToSite Reporting Tasks there are variables > captured which could have null values. In these cases those values are > stripped out from the final output by default. Recommend adding a > configurable option for users to opt to include null values to help ensure a > consistent schema if needed (with default option being to remove values for > backwards compatibility). > > Affected Reporting Tasks: > SiteToSiteProvenanceReportingTask > SiteToSiteStatusReportingTask > SiteToSiteBulletinReportingTask > SiteToSiteMetricsReportingTask > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6429) Provide option to include null values for SiteToSite Reporting Tasks
[ https://issues.apache.org/jira/browse/NIFI-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919763#comment-16919763 ] ASF subversion and git services commented on NIFI-6429: --- Commit 07baf723f72d0190cb8ba40b9ecf75ff3f23e3e0 in nifi's branch refs/heads/master from Yolanda Davis [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=07baf72 ] NIFI-6429 - Added optional field to allow variables with null values to be included in reporting records. Default functionality will exclude variables with null values for backwards compatibility. NIFI-6429 - checkstyle corrections NIFI-6429 - change property descriptor name > Provide option to include null values for SiteToSite Reporting Tasks > > > Key: NIFI-6429 > URL: https://issues.apache.org/jira/browse/NIFI-6429 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.9.2 >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > In some implementations of the SiteToSite Reporting Tasks there are variables > captured which could have null values. In these cases those values are > stripped out from the final output by default. Recommend adding a > configurable option for users to opt to include null values to help ensure a > consistent schema if needed (with default option being to remove values for > backwards compatibility). > > Affected Reporting Tasks: > SiteToSiteProvenanceReportingTask > SiteToSiteStatusReportingTask > SiteToSiteBulletinReportingTask > SiteToSiteMetricsReportingTask > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6429) Provide option to include null values for SiteToSite Reporting Tasks
[ https://issues.apache.org/jira/browse/NIFI-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919764#comment-16919764 ] ASF subversion and git services commented on NIFI-6429: --- Commit 07baf723f72d0190cb8ba40b9ecf75ff3f23e3e0 in nifi's branch refs/heads/master from Yolanda Davis [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=07baf72 ] NIFI-6429 - Added optional field to allow variables with null values to be included in reporting records. Default functionality will exclude variables with null values for backwards compatibility. NIFI-6429 - checkstyle corrections NIFI-6429 - change property descriptor name > Provide option to include null values for SiteToSite Reporting Tasks > > > Key: NIFI-6429 > URL: https://issues.apache.org/jira/browse/NIFI-6429 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.9.2 >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > In some implementations of the SiteToSite Reporting Tasks there are variables > captured which could have null values. In these cases those values are > stripped out from the final output by default. Recommend adding a > configurable option for users to opt to include null values to help ensure a > consistent schema if needed (with default option being to remove values for > backwards compatibility). > > Affected Reporting Tasks: > SiteToSiteProvenanceReportingTask > SiteToSiteStatusReportingTask > SiteToSiteBulletinReportingTask > SiteToSiteMetricsReportingTask > > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] achristianson merged pull request #3586: NIFI-6429 - Added option to allow variables with null values to be included in S2S Reporting Tasks
achristianson merged pull request #3586: NIFI-6429 - Added option to allow variables with null values to be included in S2S Reporting Tasks URL: https://github.com/apache/nifi/pull/3586 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] achristianson commented on issue #3586: NIFI-6429 - Added option to allow variables with null values to be included in S2S Reporting Tasks
achristianson commented on issue #3586: NIFI-6429 - Added option to allow variables with null values to be included in S2S Reporting Tasks URL: https://github.com/apache/nifi/pull/3586#issuecomment-526686887 Looks good. Thank you for the contribution! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (NIFI-6510) Predictive Analytics for NiFi Metrics
[
https://issues.apache.org/jira/browse/NIFI-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919755#comment-16919755
]
ASF subversion and git services commented on NIFI-6510:
---
Commit 146af158182f764fa6d88517e81ad0d76c67616b in nifi's branch
refs/heads/analytics-framework from Yolanda M. Davis
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=146af15 ]
NIFI-6510 - add an enable/disable property for analytics
> Predictive Analytics for NiFi Metrics
> -
>
> Key: NIFI-6510
> URL: https://issues.apache.org/jira/browse/NIFI-6510
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Andrew Christianson
>Assignee: Yolanda M. Davis
>Priority: Major
> Fix For: 1.10.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> From Yolanda's email to the list:
>
> {noformat}
> Currently NiFi has lots of metrics available for areas including jvm and flow
> component usage (via component status) as well as provenance data which NiFi
> makes available either through the UI or reporting tasks (for consumption by
> other systems). Past discussions in the community cite users shipping this
> data to applications such as Prometheus, ELK stacks, or Ambari metrics for
> further analysis in order to capture/review performance issues, detect
> anomalies, and send alerts or notifications. These systems are efficient in
> capturing and helping to analyze these metrics however it requires
> customization work and knowledge of NiFi operations to provide meaningful
> analytics within a flow context.
> In speaking with Matt Burgess and Andy Christianson on this topic we feel
> that there is an opportunity to introduce an analytics framework that could
> provide users reasonable predictions on key performance indicators for flows,
> such as back pressure and flow rate, to help administrators improve
> operational management of NiFi clusters. This framework could offer several
> key features:
> - Provide a flexible internal analytics engine and model api which supports
> the addition of or enhancement to onboard models
> - Support integration of remote or cloud based ML models
> - Support both traditional and online (incremental) learning methods
> - Provide support for model caching (perhaps later inclusion into a model
> repository or registry)
> - UI enhancements to display prediction information either in existing
> summary data, new data visualizations, or directly within the flow/canvas
> (where applicable)
> For an initial target we thought that back pressure prediction would be a
> good starting point for this initiative, given that back pressure detection
> is a key indicator of flow performance and many of the metrics currently
> available would provide enough data points to create a reasonable performing
> model. We have some ideas on how this could be achieved however we wanted to
> discuss this more with the community to get thoughts about tackling this
> work, especially if there are specific use cases or other factors that should
> be considered.{noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (NIFI-6480) PutORC/PutParquet can't overwrite file even if set 'Overwrite Files' to true
[ https://issues.apache.org/jira/browse/NIFI-6480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919679#comment-16919679 ] HondaWei commented on NIFI-6480: I'll tackle this and send PR ASAP. > PutORC/PutParquet can't overwrite file even if set 'Overwrite Files' to true > > > Key: NIFI-6480 > URL: https://issues.apache.org/jira/browse/NIFI-6480 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: archon gum >Assignee: archon gum >Priority: Major > Attachments: Snipaste_2019-07-16_11-07-05.jpg > > Time Spent: 50m > Remaining Estimate: 0h > > !Snipaste_2019-07-16_11-07-05.jpg! > > Solution: > Reference to > [PutHDFS|https://github.com/apache/nifi/blob/72244d09ff193131119c10492b9327af35a64f02/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/PutHDFS.java], > delete dest file before rename temp file to dest file. > Tested on my local machine. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] scottyaslan commented on issue #3636: Allow parameters to be used in property configuration
scottyaslan commented on issue #3636: Allow parameters to be used in property configuration URL: https://github.com/apache/nifi/pull/3636#issuecomment-526654761 @mcgilman I noticed that when a user is editing a property table cell and resizes the editor that the tooltip for the EL/Param is off the screen:  I think we can address this in a follow on PR. I created https://issues.apache.org/jira/browse/NIFI-6605 to track this issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (NIFI-6605) EL/Param tooltip not visible for large property table editors
Scott Aslan created NIFI-6605: - Summary: EL/Param tooltip not visible for large property table editors Key: NIFI-6605 URL: https://issues.apache.org/jira/browse/NIFI-6605 Project: Apache NiFi Issue Type: Improvement Components: Core UI Reporter: Scott Aslan Attachments: Screen Shot 2019-08-30 at 11.48.45 AM.png When a user is editing a property table cell and resizes the editor the tooltip for the EL/Param is no longer visible: -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] rfellows commented on a change in pull request #3636: [WIP] Allow parameters to be used in property configuration
rfellows commented on a change in pull request #3636: [WIP] Allow parameters to be used in property configuration URL: https://github.com/apache/nifi/pull/3636#discussion_r319567192 ## File path: nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/jquery/nfeditor/jquery.nfeditor.js ## @@ -236,6 +234,28 @@ }); }, +/** + * Sets the size of the editor. Review comment: Looks like a `CTRL-C`, `CTRL-V` here. Should update this to reflect what the function actually does This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] joewitt commented on issue #3651: NIFI-6539 Nifi stateless init from flow xml
joewitt commented on issue #3651: NIFI-6539 Nifi stateless init from flow xml URL: https://github.com/apache/nifi/pull/3651#issuecomment-52661 @tpalfy this needs someone to actually vet/verify as it is very large update on a relatively new codebase @SamHjelmfelt this really looks like something you need to be reviewing and commenting on as it updates a lot of the initial stateless nifi contribution you've made. Does it make sense for a flow.xml.gz file to be executable directly? I'm not sure it is good to couple an internal detail of today's nifi runtime to the runtime of stateless nifi.. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi-cpp] arpadboda edited a comment on issue #615: [WIP]MINIFICPP-963 - Zero byte (no content) flow files cannot be received …
arpadboda edited a comment on issue #615: [WIP]MINIFICPP-963 - Zero byte (no content) flow files cannot be received … URL: https://github.com/apache/nifi-minifi-cpp/pull/615#issuecomment-526623380 For anyone reviewing this: I would suggest doing it commit by commit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi-cpp] arpadboda commented on issue #615: [WIP]MINIFICPP-963 - Zero byte (no content) flow files cannot be received …
arpadboda commented on issue #615: [WIP]MINIFICPP-963 - Zero byte (no content) flow files cannot be received … URL: https://github.com/apache/nifi-minifi-cpp/pull/615#issuecomment-526623380 For anyone review-ing this: I would suggest doing it commit by commit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi-cpp] arpadboda commented on issue #615: [WIP]MINIFICPP-963 - Zero byte (no content) flow files cannot be received …
arpadboda commented on issue #615: [WIP]MINIFICPP-963 - Zero byte (no content) flow files cannot be received … URL: https://github.com/apache/nifi-minifi-cpp/pull/615#issuecomment-526621458 @phrocker : updated this: -Connection filtering is verified by docker tests -0B transfer is now verified by docker tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (MINIFICPP-1022) Review passing shipped versions of libraries to other third parties (ExternalProjects)
Daniel Bakai created MINIFICPP-1022: --- Summary: Review passing shipped versions of libraries to other third parties (ExternalProjects) Key: MINIFICPP-1022 URL: https://issues.apache.org/jira/browse/MINIFICPP-1022 Project: Apache NiFi MiNiFi C++ Issue Type: Improvement Reporter: Daniel Bakai Assignee: Daniel Bakai Fix For: 0.7.0 Consider having a wrapper function to make this piece of CMake code reusable. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi-minifi-cpp] phrocker commented on issue #638: MINIFICPP-1018 - Fix linking issues
phrocker commented on issue #638: MINIFICPP-1018 - Fix linking issues URL: https://github.com/apache/nifi-minifi-cpp/pull/638#issuecomment-526620452 > @phrocker I would prefer the whole civetweb to be moved to an ExternalProject - I planned to do that with the review of passing our shipped libraries to our ExternalProjects. @bakaid sorry I deleted the comment after realizing I made a misinterpretation. I very much look forward to improving our external project management! I'm good with the PR. Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi-cpp] bakaid commented on a change in pull request #639: MINFICPP-1005 - Disable
bakaid commented on a change in pull request #639: MINFICPP-1005 - Disable
https://github.com/apache/nifi-minifi-cpp/pull/639#discussion_r319534998
##
File path: extensions/civetweb/processors/ListenHTTP.cpp
##
@@ -182,21 +182,12 @@ void ListenHTTP::onSchedule(core::ProcessContext
*context, core::ProcessSessionF
options.emplace_back("yes");
}
-if (sslMinVer == "SSL2") {
- options.emplace_back("ssl_protocol_version");
- options.emplace_back(std::to_string(0));
-} else if (sslMinVer == "SSL3") {
- options.emplace_back("ssl_protocol_version");
- options.emplace_back(std::to_string(1));
-} else if (sslMinVer == "TLS1.0") {
- options.emplace_back("ssl_protocol_version");
- options.emplace_back(std::to_string(2));
-} else if (sslMinVer == "TLS1.1") {
- options.emplace_back("ssl_protocol_version");
- options.emplace_back(std::to_string(3));
-} else {
+if (sslMinVer == "TLS1.2") {
options.emplace_back("ssl_protocol_version");
options.emplace_back(std::to_string(4));
+} else {
+ logger_->log_error("Invalid SSL Minimum Version specified!");
Review comment:
Fair point, I will change this to an exception.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi-minifi-cpp] phrocker commented on a change in pull request #639: MINFICPP-1005 - Disable
phrocker commented on a change in pull request #639: MINFICPP-1005 - Disable
https://github.com/apache/nifi-minifi-cpp/pull/639#discussion_r319534017
##
File path: extensions/civetweb/processors/ListenHTTP.cpp
##
@@ -182,21 +182,12 @@ void ListenHTTP::onSchedule(core::ProcessContext
*context, core::ProcessSessionF
options.emplace_back("yes");
}
-if (sslMinVer == "SSL2") {
- options.emplace_back("ssl_protocol_version");
- options.emplace_back(std::to_string(0));
-} else if (sslMinVer == "SSL3") {
- options.emplace_back("ssl_protocol_version");
- options.emplace_back(std::to_string(1));
-} else if (sslMinVer == "TLS1.0") {
- options.emplace_back("ssl_protocol_version");
- options.emplace_back(std::to_string(2));
-} else if (sslMinVer == "TLS1.1") {
- options.emplace_back("ssl_protocol_version");
- options.emplace_back(std::to_string(3));
-} else {
+if (sslMinVer == "TLS1.2") {
options.emplace_back("ssl_protocol_version");
options.emplace_back(std::to_string(4));
+} else {
+ logger_->log_error("Invalid SSL Minimum Version specified!");
Review comment:
I think if we throw an exception here we have a chance of catching this
earlier in the re-deployment process.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi-minifi-cpp] bakaid commented on issue #638: MINIFICPP-1018 - Fix linking issues
bakaid commented on issue #638: MINIFICPP-1018 - Fix linking issues URL: https://github.com/apache/nifi-minifi-cpp/pull/638#issuecomment-526619532 @phrocker I would prefer the whole civetweb to be moved to an ExternalProject - I planned to do that with the review of passing our shipped libraries to our ExternalProjects. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi-cpp] phrocker commented on a change in pull request #638: MINIFICPP-1018 - Fix linking issues
phrocker commented on a change in pull request #638: MINIFICPP-1018 - Fix
linking issues
URL: https://github.com/apache/nifi-minifi-cpp/pull/638#discussion_r319530810
##
File path: extensions/civetweb/CMakeLists.txt
##
@@ -33,6 +33,7 @@ set(BUILD_CIVET_TESTING OFF)
add_subdirectory(${CIVET_THIRDPARTY_ROOT}
${CIVET_BINARY_ROOT}
EXCLUDE_FROM_ALL)
+add_dependencies(c-library libressl-portable)
Review comment:
Is there no way to do this via the the main project? If not, can we create a
document in the root that adds this caveat so that we at least have a
documented path for continuing this? perhaps a new md tailed or, better yet,
to https://github.com/apache/nifi-minifi-cpp/blob/master/CONTRIB.md in a caveat
section ( or something of the like )?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi-minifi-cpp] bakaid opened a new pull request #639: MINFICPP-1005 - Disable
bakaid opened a new pull request #639: MINFICPP-1005 - Disable https://github.com/apache/nifi-minifi-cpp/pull/639 …ng secure connections Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFICPP- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (NIFI-6483) Apache Nifi (version 1.9.2) error when update data in a table (SQL server, MySQL, Postgresql) (it is running well with insert and update operations)
[ https://issues.apache.org/jira/browse/NIFI-6483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919545#comment-16919545 ] HondaWei edited comment on NIFI-6483 at 8/30/19 1:44 PM: - I traced the code, it failed if your column name in record schema is the lower case even though set "Translate Field Names" to false. I think it will work well if you set column name in upper case in record schema and set "Translate Field Names" to false. was (Author: hondawei): I traced the code, it still failed even though set "Translate Field Names" to false due to case-sensitive comparison of column name comparison. > Apache Nifi (version 1.9.2) error when update data in a table (SQL server, > MySQL, Postgresql) (it is running well with insert and update operations) > > > Key: NIFI-6483 > URL: https://issues.apache.org/jira/browse/NIFI-6483 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration >Affects Versions: 1.9.2 > Environment: error when update data in a table (SQL server, MySQL, > Postgresql) (it is running well with insert and update operations) >Reporter: thuy le >Assignee: HondaWei >Priority: Major > Attachments: image-2019-07-25-11-32-19-469.png > > > Apache Nifi (version 1.9.2) error when update data in a table (SQL server, > MySQL, Postgresql) (it is running well with insert and update operations) > > !https://community.hortonworks.com/storage/attachments/110025-1563909312644.png|width=736,height=242! > > _With Delete they do correct_ > !https://community.hortonworks.com/storage/attachments/110007-1563910599280.png|width=932,height=127! > !https://community.hortonworks.com/storage/attachments/110013-1563910653675.png|width=828,height=156! > > > *+with Update+*_: They automatic change Primarykey from_ *product_key* _to > PRODUCTKEY (_I tested with *_PutSQL_* and *_PutDatabaseRecord_* it have the > same issue) > !https://community.hortonworks.com/storage/attachments/110006-1563910561236.png|width=703,height=94!!https://community.hortonworks.com/storage/attachments/110005-1563910541777.png|width=605,height=83! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6483) Apache Nifi (version 1.9.2) error when update data in a table (SQL server, MySQL, Postgresql) (it is running well with insert and update operations)
[ https://issues.apache.org/jira/browse/NIFI-6483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919545#comment-16919545 ] HondaWei commented on NIFI-6483: I traced the code, it still failed even though set "Translate Field Names" to false due to case-sensitive comparison of column name comparison. > Apache Nifi (version 1.9.2) error when update data in a table (SQL server, > MySQL, Postgresql) (it is running well with insert and update operations) > > > Key: NIFI-6483 > URL: https://issues.apache.org/jira/browse/NIFI-6483 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration >Affects Versions: 1.9.2 > Environment: error when update data in a table (SQL server, MySQL, > Postgresql) (it is running well with insert and update operations) >Reporter: thuy le >Priority: Major > Attachments: image-2019-07-25-11-32-19-469.png > > > Apache Nifi (version 1.9.2) error when update data in a table (SQL server, > MySQL, Postgresql) (it is running well with insert and update operations) > > !https://community.hortonworks.com/storage/attachments/110025-1563909312644.png|width=736,height=242! > > _With Delete they do correct_ > !https://community.hortonworks.com/storage/attachments/110007-1563910599280.png|width=932,height=127! > !https://community.hortonworks.com/storage/attachments/110013-1563910653675.png|width=828,height=156! > > > *+with Update+*_: They automatic change Primarykey from_ *product_key* _to > PRODUCTKEY (_I tested with *_PutSQL_* and *_PutDatabaseRecord_* it have the > same issue) > !https://community.hortonworks.com/storage/attachments/110006-1563910561236.png|width=703,height=94!!https://community.hortonworks.com/storage/attachments/110005-1563910541777.png|width=605,height=83! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (NIFI-6483) Apache Nifi (version 1.9.2) error when update data in a table (SQL server, MySQL, Postgresql) (it is running well with insert and update operations)
[ https://issues.apache.org/jira/browse/NIFI-6483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] HondaWei reassigned NIFI-6483: -- Assignee: HondaWei > Apache Nifi (version 1.9.2) error when update data in a table (SQL server, > MySQL, Postgresql) (it is running well with insert and update operations) > > > Key: NIFI-6483 > URL: https://issues.apache.org/jira/browse/NIFI-6483 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration >Affects Versions: 1.9.2 > Environment: error when update data in a table (SQL server, MySQL, > Postgresql) (it is running well with insert and update operations) >Reporter: thuy le >Assignee: HondaWei >Priority: Major > Attachments: image-2019-07-25-11-32-19-469.png > > > Apache Nifi (version 1.9.2) error when update data in a table (SQL server, > MySQL, Postgresql) (it is running well with insert and update operations) > > !https://community.hortonworks.com/storage/attachments/110025-1563909312644.png|width=736,height=242! > > _With Delete they do correct_ > !https://community.hortonworks.com/storage/attachments/110007-1563910599280.png|width=932,height=127! > !https://community.hortonworks.com/storage/attachments/110013-1563910653675.png|width=828,height=156! > > > *+with Update+*_: They automatic change Primarykey from_ *product_key* _to > PRODUCTKEY (_I tested with *_PutSQL_* and *_PutDatabaseRecord_* it have the > same issue) > !https://community.hortonworks.com/storage/attachments/110006-1563910561236.png|width=703,height=94!!https://community.hortonworks.com/storage/attachments/110005-1563910541777.png|width=605,height=83! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6483) Apache Nifi (version 1.9.2) error when update data in a table (SQL server, MySQL, Postgresql) (it is running well with insert and update operations)
[ https://issues.apache.org/jira/browse/NIFI-6483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919528#comment-16919528 ] HondaWei commented on NIFI-6483: Hi [~thuylevn] Can you post your properties of PutDatabaseRecord? Do you set "Translate Field Names" to true? If this property is true, it removes underscore in the column name. > Apache Nifi (version 1.9.2) error when update data in a table (SQL server, > MySQL, Postgresql) (it is running well with insert and update operations) > > > Key: NIFI-6483 > URL: https://issues.apache.org/jira/browse/NIFI-6483 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration >Affects Versions: 1.9.2 > Environment: error when update data in a table (SQL server, MySQL, > Postgresql) (it is running well with insert and update operations) >Reporter: thuy le >Priority: Major > Attachments: image-2019-07-25-11-32-19-469.png > > > Apache Nifi (version 1.9.2) error when update data in a table (SQL server, > MySQL, Postgresql) (it is running well with insert and update operations) > > !https://community.hortonworks.com/storage/attachments/110025-1563909312644.png|width=736,height=242! > > _With Delete they do correct_ > !https://community.hortonworks.com/storage/attachments/110007-1563910599280.png|width=932,height=127! > !https://community.hortonworks.com/storage/attachments/110013-1563910653675.png|width=828,height=156! > > > *+with Update+*_: They automatic change Primarykey from_ *product_key* _to > PRODUCTKEY (_I tested with *_PutSQL_* and *_PutDatabaseRecord_* it have the > same issue) > !https://community.hortonworks.com/storage/attachments/110006-1563910561236.png|width=703,height=94!!https://community.hortonworks.com/storage/attachments/110005-1563910541777.png|width=605,height=83! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (NIFI-6604) Reduce the size of the generated assemblies
[ https://issues.apache.org/jira/browse/NIFI-6604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri updated NIFI-6604: -- Status: Patch Available (was: Open) > Reduce the size of the generated assemblies > --- > > Key: NIFI-6604 > URL: https://issues.apache.org/jira/browse/NIFI-6604 > Project: Apache NiFi > Issue Type: Task > Components: Tools and Build >Reporter: Aldrin Piri >Assignee: Aldrin Piri >Priority: Blocker > Fix For: 1.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > > As per the discussion on > https://lists.apache.org/thread.html/d45f33805dcb8096914a06853bc3b8eca1acf50c2ae7b68e75eabfe8@ > we must reduce the assembly size such that we can publish the convenience > binaries as part of the release. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi-minifi-cpp] apiri commented on issue #630: MINIFICPP-1006: minor updates for jni building using win build & correct artifact naming
apiri commented on issue #630: MINIFICPP-1006: minor updates for jni building using win build & correct artifact naming URL: https://github.com/apache/nifi-minifi-cpp/pull/630#issuecomment-526590300 License changes look good with the adjustments. +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (MINIFICPP-1021) Enable TLS 1.3 support
Daniel Bakai created MINIFICPP-1021: --- Summary: Enable TLS 1.3 support Key: MINIFICPP-1021 URL: https://issues.apache.org/jira/browse/MINIFICPP-1021 Project: Apache NiFi MiNiFi C++ Issue Type: New Feature Reporter: Daniel Bakai Since we now always ship libressl, curl and civetweb we can update them to the proper versions to support TLS 1.3 for both incoming and outgoing connections. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (NIFI-6488) NiFi web interface often non-responsive
[ https://issues.apache.org/jira/browse/NIFI-6488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Sargrad updated NIFI-6488: Attachment: image-2019-08-30-08-03-39-877.png > NiFi web interface often non-responsive > > > Key: NIFI-6488 > URL: https://issues.apache.org/jira/browse/NIFI-6488 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.9.2 > Environment: centos 7 >Reporter: David Sargrad >Priority: Major > Attachments: image-2019-08-28-08-15-51-702.png, > image-2019-08-30-08-03-39-877.png > > > I have been using NiFi for close to a year. Currently most of my experience > has been with building custom processors and prototyping a variety of flows. > > Often the NiFi web interface becomes non-responsive. Its typically during a > flow integration phase, where I leave flowfiles sitting in queues, sometimes > 3 or 4 queues may be maxed at 10,000 flow files. These are relatively small > flowfiles (perhaps 200 or 300 bytes). Yet, of course, the sum total of the > filled queues is in the tens of megabytes. > > I am guessing that the JVM is nearing, or perhaps has exceeded, its maximum > allocated heap size. I'd think that NiFi protects against non-responsive UI's > by shedding load, presenting a warning, or doing something to preserve its > operational state. > > What is the best way for me to isolate this issue? I'd like to understand > exactly what drives NiFi to be non-responsive. If there is anything that I > can do to isolate this, and to help the NiFi team make this more robust, I'm > glad to dig. > > Its been impossible for me to exactly characterize the conditions that lead > to a non-responsive NiFi UI. However I see this fairly frequently. > > I've updated the following picture. Since it did not seem to attach before. > I've also added a view of the memory and CPU consumption. In this particular > case I waited for about 30 seconds and eventually the NiFi user interface > came to life. > !image-2019-08-28-08-15-51-702.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6488) NiFi web interface often non-responsive
[ https://issues.apache.org/jira/browse/NIFI-6488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919480#comment-16919480 ] David Sargrad commented on NIFI-6488: - !image-2019-08-30-08-03-39-877.png! > NiFi web interface often non-responsive > > > Key: NIFI-6488 > URL: https://issues.apache.org/jira/browse/NIFI-6488 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.9.2 > Environment: centos 7 >Reporter: David Sargrad >Priority: Major > Attachments: image-2019-08-28-08-15-51-702.png, > image-2019-08-30-08-03-39-877.png > > > I have been using NiFi for close to a year. Currently most of my experience > has been with building custom processors and prototyping a variety of flows. > > Often the NiFi web interface becomes non-responsive. Its typically during a > flow integration phase, where I leave flowfiles sitting in queues, sometimes > 3 or 4 queues may be maxed at 10,000 flow files. These are relatively small > flowfiles (perhaps 200 or 300 bytes). Yet, of course, the sum total of the > filled queues is in the tens of megabytes. > > I am guessing that the JVM is nearing, or perhaps has exceeded, its maximum > allocated heap size. I'd think that NiFi protects against non-responsive UI's > by shedding load, presenting a warning, or doing something to preserve its > operational state. > > What is the best way for me to isolate this issue? I'd like to understand > exactly what drives NiFi to be non-responsive. If there is anything that I > can do to isolate this, and to help the NiFi team make this more robust, I'm > glad to dig. > > Its been impossible for me to exactly characterize the conditions that lead > to a non-responsive NiFi UI. However I see this fairly frequently. > > I've updated the following picture. Since it did not seem to attach before. > I've also added a view of the memory and CPU consumption. In this particular > case I waited for about 30 seconds and eventually the NiFi user interface > came to life. > !image-2019-08-28-08-15-51-702.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi-minifi-cpp] bakaid commented on issue #638: MINIFICPP-1018 - Fix linking issues
bakaid commented on issue #638: MINIFICPP-1018 - Fix linking issues URL: https://github.com/apache/nifi-minifi-cpp/pull/638#issuecomment-526570833 It fixed the parallel builds for me - I would prefer if @jdye64 could try it in his environment, to make sure. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi-cpp] arpadboda commented on issue #638: MINIFICPP-1018 - Fix linking issues
arpadboda commented on issue #638: MINIFICPP-1018 - Fix linking issues URL: https://github.com/apache/nifi-minifi-cpp/pull/638#issuecomment-526570377 @bakaid do you consider this complete now? Looks good (at least surely helps and doesn't break anything), so I'm happy to approve and merge. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] arpadboda commented on a change in pull request #3394: NIFI-6159 - Add BigQuery processor using the Streaming API
arpadboda commented on a change in pull request #3394: NIFI-6159 - Add BigQuery
processor using the Streaming API
URL: https://github.com/apache/nifi/pull/3394#discussion_r319470097
##
File path:
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/bigquery/AbstractBigQueryProcessor.java
##
@@ -82,23 +80,14 @@
.addValidator(StandardValidators.NON_EMPTY_EL_VALIDATOR)
.build();
-public static final PropertyDescriptor TABLE_SCHEMA = new
PropertyDescriptor.Builder()
-.name(BigQueryAttributes.TABLE_SCHEMA_ATTR)
-.displayName("Table Schema")
-.description(BigQueryAttributes.TABLE_SCHEMA_DESC)
-.required(false)
-
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
-.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
-.build();
-
-public static final PropertyDescriptor READ_TIMEOUT = new
PropertyDescriptor.Builder()
-.name(BigQueryAttributes.JOB_READ_TIMEOUT_ATTR)
-.displayName("Read Timeout")
-.description(BigQueryAttributes.JOB_READ_TIMEOUT_DESC)
+public static final PropertyDescriptor IGNORE_UNKNOWN = new
PropertyDescriptor.Builder()
+.name(BigQueryAttributes.IGNORE_UNKNOWN_ATTR)
+.displayName("Ignore Unknown Values")
+.description(BigQueryAttributes.IGNORE_UNKNOWN_DESC)
.required(true)
-.defaultValue("5 minutes")
-
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
-.addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
+.addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+.allowableValues("true", "false")
Review comment:
Do you actually need it?
Actually you further limit the values, boolean validator ignores case.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] arpadboda commented on a change in pull request #3394: NIFI-6159 - Add BigQuery processor using the Streaming API
arpadboda commented on a change in pull request #3394: NIFI-6159 - Add BigQuery
processor using the Streaming API
URL: https://github.com/apache/nifi/pull/3394#discussion_r319471431
##
File path:
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/bigquery/PutBigQueryStreaming.java
##
@@ -0,0 +1,206 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.gcp.bigquery;
+
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SystemResource;
+import org.apache.nifi.annotation.behavior.SystemResourceConsideration;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.LogLevel;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.serialization.RecordReader;
+import org.apache.nifi.serialization.RecordReaderFactory;
+import org.apache.nifi.serialization.record.MapRecord;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.util.StringUtils;
+
+import com.google.cloud.bigquery.BigQueryError;
+import com.google.cloud.bigquery.InsertAllRequest;
+import com.google.cloud.bigquery.InsertAllResponse;
+import com.google.cloud.bigquery.TableId;
+import com.google.common.collect.ImmutableList;
+
+/**
+ * A processor for streaming loading data into a Google BigQuery table. It
uses the BigQuery
+ * streaming insert API to insert data. This provides the lowest-latency
insert path into BigQuery,
+ * and therefore is the default method when the input is unbounded. BigQuery
will make a strong
+ * effort to ensure no duplicates when using this path, however there are some
scenarios in which
+ * BigQuery is unable to make this guarantee (see
+ * https://cloud.google.com/bigquery/streaming-data-into-bigquery). A query
can be run over the
+ * output table to periodically clean these rare duplicates. Alternatively,
using the Batch insert
+ * method does guarantee no duplicates, though the latency for the insert into
BigQuery will be much
+ * higher.
+ */
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@Tags({ "google", "google cloud", "bq", "gcp", "bigquery", "record" })
+@CapabilityDescription("Load data into Google BigQuery table using the
streaming API. This processor "
++ "is not intended to load large flow files as it will load the full
content into memory. If "
++ "you need to insert large flow files, consider using
PutBigQueryBatch instead.")
+@SeeAlso({ PutBigQueryBatch.class })
+@SystemResourceConsideration(resource = SystemResource.MEMORY)
+@WritesAttributes({
+@WritesAttribute(attribute = BigQueryAttributes.JOB_NB_RECORDS_ATTR,
description = BigQueryAttributes.JOB_NB_RECORDS_DESC)
+})
+public class PutBigQueryStreaming extends AbstractBigQueryProcessor {
+
+public static final PropertyDescriptor RECORD_READER = new
PropertyDescriptor.Builder()
+.name(BigQueryAttributes.RECORD_READER_ATTR)
+.displayName("Record Reader")
+.description(BigQueryAttributes.RECORD_READER_DESC)
+.identifiesControllerService(RecordReaderFactory.class)
+.required(true)
+.build();
+
+public static final PropertyDescriptor SKIP_INVALID_ROWS = new
PropertyDescriptor.Builder()
+.name(BigQueryAttributes.SKIP_INVALID_ROWS_ATTR)
+.displayName("Skip Invalid Rows")
+
[GitHub] [nifi-minifi-cpp] phrocker commented on issue #637: MINIFICPP-1020 - PutFile fails to create empty files
phrocker commented on issue #637: MINIFICPP-1020 - PutFile fails to create empty files URL: https://github.com/apache/nifi-minifi-cpp/pull/637#issuecomment-526534891 > @phrocker : the approach you suggest seems to be valid as well, although following that would require way more changes. As first examples the current implementation GenerateFlowFile and SiteToSiteClient::receiveFlowFiles doesn't follow this, both create the flowfile without clain in case the content size is zero. > > These could be changed as well, but my concern with this is some edgecases: you are allowed to route flowfiles from a failure relationship to PutFile. In these cases a lot of processors we have (opencv, the current implementation of FetchOPC, etc) don't create claim, but in my opinion this should still work. PutFile should only route to failure in case of IO issues (where the actual write or file creation cannot be performed). > > The NiFi approach is something I like, however following that would mean enforcing having a claim for all the flowfiles we have, and the current codebase is far from that. I suspect deser of FFM is creating a claim even when there is no data or claim to data -- I can see this causing confusion in how it would/should work. I'll dive further into that at some point today and get back to you; however, I was focused more on the GenerateFlowFile case. I'll also dive further into that PR to get a better understanding of whether that PR encapsulates the issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi-cpp] arpadboda edited a comment on issue #637: MINIFICPP-1020 - PutFile fails to create empty files
arpadboda edited a comment on issue #637: MINIFICPP-1020 - PutFile fails to create empty files URL: https://github.com/apache/nifi-minifi-cpp/pull/637#issuecomment-526525681 @phrocker : the approach you suggest seems to be valid as well, although following that would require way more changes. As first examples the current implementation GenerateFlowFile and SiteToSiteClient::receiveFlowFiles don't follow this, both create the flowfile without claim in case the content size is zero. These could be changed as well, but my concern with this is some edgecases: you are allowed to route flowfiles from a failure relationship to PutFile. In these cases a lot of processors we have (opencv, the current implementation of FetchOPC, etc) don't create claim, but in my opinion this should still work. PutFile should only route to failure in case of IO issues (where the actual write or file creation cannot be performed). The NiFi approach is something I like, however following that would mean enforcing having a claim for all the flowfiles we have, and the current codebase is far from that. In longer term we might consider following that as it would make things a bit cleaner, but that would break some contracts we currently use, so I wouldn't do that before 1.0. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi-cpp] arpadboda edited a comment on issue #637: MINIFICPP-1020 - PutFile fails to create empty files
arpadboda edited a comment on issue #637: MINIFICPP-1020 - PutFile fails to create empty files URL: https://github.com/apache/nifi-minifi-cpp/pull/637#issuecomment-526525681 @phrocker : the approach you suggest seems to be valid as well, although following that would require way more changes. As first examples the current implementation GenerateFlowFile and SiteToSiteClient::receiveFlowFiles don't follow this, both create the flowfile without claim in case the content size is zero. These could be changed as well, but my concern with this is some edgecases: you are allowed to route flowfiles from a failure relationship to PutFile. In these cases a lot of processors we have (opencv, the current implementation of FetchOPC, etc) don't create claim, but in my opinion this should still work. PutFile should only route to failure in case of IO issues (where the actual write or file creation cannot be performed). The NiFi approach is something I like, however following that would mean enforcing having a claim for all the flowfiles we have, and the current codebase is far from that. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi-cpp] arpadboda commented on issue #637: MINIFICPP-1020 - PutFile fails to create empty files
arpadboda commented on issue #637: MINIFICPP-1020 - PutFile fails to create empty files URL: https://github.com/apache/nifi-minifi-cpp/pull/637#issuecomment-526525681 @phrocker : the approach you suggest seems to be valid as well, although following that would require way more changes. As first examples the current implementation GenerateFlowFile and SiteToSiteClient::receiveFlowFiles doesn't follow this, both create the flowfile without clain in case the content size is zero. These could be changed as well, but my concern with this is some edgecases: you are allowed to route flowfiles from a failure relationship to PutFile. In these cases a lot of processors we have (opencv, the current implementation of FetchOPC, etc) don't create claim, but in my opinion this should still work. PutFile should only route to failure in case of IO issues (where the actual write or file creation cannot be performed). The NiFi approach is something I like, however following that would mean enforcing having a claim for all the flowfiles we have, and the current codebase is far from that. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi-cpp] phrocker edited a comment on issue #637: MINIFICPP-1020 - PutFile fails to create empty files
phrocker edited a comment on issue #637: MINIFICPP-1020 - PutFile fails to create empty files URL: https://github.com/apache/nifi-minifi-cpp/pull/637#issuecomment-526520615 @arpadboda I think the disconnect is that when, in NiFi, we deal with java output streams and write to the content repo. The result of that may be zero bytes but a content claim is created with no content. I believe what you did in the other PR is create no claim, correct? If this interpretation is correct, then the repo won't have any claim to the content and thus the path you attempt to address is correct. GenerateFlowFile should create an empty content and there should be a claim backing that, which would mean this PR isn't technically necessary since the repos will have that resource claim. I'll take a quick look at that other PR to get a better understanding of how these two link together if you disagree with this opinion and think it may help to have a more holistic understanding. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi-cpp] phrocker edited a comment on issue #637: MINIFICPP-1020 - PutFile fails to create empty files
phrocker edited a comment on issue #637: MINIFICPP-1020 - PutFile fails to create empty files URL: https://github.com/apache/nifi-minifi-cpp/pull/637#issuecomment-526520615 @arpadboda I think the disconnect is that when, in NiFi, we deal with java output streams and write to the content repo. The result of that may be zero bytes but a content claim is created with no content. I believe what you did in the other PR is create no claim, correct? If this interpretation is correct, then the repo won't have any claim to the content and thus the path you attempt to address is correct. GenerateFlowFile should create an empty content and there should be a claim backing that, which would mean this PR isn't technically necessary since the repos will have that resource claim. I'll take a quick look at that other PR to get a better understanding of how these two link together if you disagree with this opinion. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi-cpp] phrocker commented on issue #637: MINIFICPP-1020 - PutFile fails to create empty files
phrocker commented on issue #637: MINIFICPP-1020 - PutFile fails to create empty files URL: https://github.com/apache/nifi-minifi-cpp/pull/637#issuecomment-526520615 @arpadboda I think the disconnect is that when, in NiFi, we deal with java output streams and write to the content repo. The result of that may be zero bytes but a content claim is created with no content. I believe what you did in the other PR is create no claim, correct? If this interpretation is correct, then the repo won't have any claim to the content and thus the path you attempt to address is correct. GenerateFlowFile should create an empty content and there should be a claim backing that, which would mean this PR isn't technically necessary since the repos will have that resource claim. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi-cpp] bakaid commented on issue #605: MINIFICPP-550 - Implement RocksDB controller service and component st…
bakaid commented on issue #605: MINIFICPP-550 - Implement RocksDB controller service and component st… URL: https://github.com/apache/nifi-minifi-cpp/pull/605#issuecomment-526518410 @phrocker I think it's important as well - I raised the questions in my previous comment here, because I would like input on them, but I think someone would have to see the PR before they could be answered. So yes, by all means, I would appreciate if someone could review this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #637: MINIFICPP-1020 - PutFile fails to create empty files
arpadboda commented on a change in pull request #637: MINIFICPP-1020 - PutFile fails to create empty files URL: https://github.com/apache/nifi-minifi-cpp/pull/637#discussion_r319415633 ## File path: CPPLINT.cfg ## @@ -0,0 +1,2 @@ +set noparent +filter=-runtime/references Review comment: Sure, I can use nolint for the cases where I don't agree with this rule. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi-cpp] phrocker commented on a change in pull request #637: MINIFICPP-1020 - PutFile fails to create empty files
phrocker commented on a change in pull request #637: MINIFICPP-1020 - PutFile fails to create empty files URL: https://github.com/apache/nifi-minifi-cpp/pull/637#discussion_r319414243 ## File path: CPPLINT.cfg ## @@ -0,0 +1,2 @@ +set noparent +filter=-runtime/references Review comment: I'm afraid I don't agree with this. Can this be moved to a discussion outside of this PR to avoid stalling it? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi-cpp] phrocker commented on a change in pull request #637: MINIFICPP-1020 - PutFile fails to create empty files
phrocker commented on a change in pull request #637: MINIFICPP-1020 - PutFile fails to create empty files URL: https://github.com/apache/nifi-minifi-cpp/pull/637#discussion_r319414243 ## File path: CPPLINT.cfg ## @@ -0,0 +1,2 @@ +set noparent +filter=-runtime/references Review comment: I'm afraid I don't agree with this. Can this be moved to a discussion outside of this PR? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #637: MINIFICPP-1020 - PutFile fails to create empty files
arpadboda commented on a change in pull request #637: MINIFICPP-1020 - PutFile fails to create empty files URL: https://github.com/apache/nifi-minifi-cpp/pull/637#discussion_r319402435 ## File path: CPPLINT.cfg ## @@ -0,0 +1,2 @@ +set noparent +filter=-runtime/references Review comment: ``` Is this a non-const reference? If so, make const or use a pointer: std::vector& data [runtime/references] [2] ``` I think this hardly makes sense, passing a container by reference is much better than passing as a raw pointer. Moreover it's inconsistent as well, in case of lambdas it allows you to either capture by reference or have reference arguments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #637: MINIFICPP-1020 - PutFile fails to create empty files
arpadboda commented on a change in pull request #637: MINIFICPP-1020 - PutFile fails to create empty files URL: https://github.com/apache/nifi-minifi-cpp/pull/637#discussion_r319402435 ## File path: CPPLINT.cfg ## @@ -0,0 +1,2 @@ +set noparent +filter=-runtime/references Review comment: Is this a non-const reference? If so, make const or use a pointer: std::vector& data [runtime/references] [2] I think this hardly makes sense, passing a container by reference is much better than passing as a raw pointer. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi-cpp] arpadboda edited a comment on issue #637: MINIFICPP-1020 - PutFile fails to create empty files
arpadboda edited a comment on issue #637: MINIFICPP-1020 - PutFile fails to create empty files URL: https://github.com/apache/nifi-minifi-cpp/pull/637#issuecomment-526504548 > Can you provide an example flow? I created a flow of that pulled empty files via GetFile -> PutFile and it wrote those empty files. Is this the case where the resource claim has an invalid pointer to a file by virtue of GenerateFlowFile? That would be a separate issue in my opinion. I could be wrong but any help in reproducing this per the PR/ticket description would be helpful. > > Happy to provide logs where I did not see a failure or rollback. I connected GenerateFlowFile set to 0B with PutFile. If you check my other PR (the GenerateFlowFile one), and run the tests of that without having this change, you are going to see the issue. I agree that in the background having empty content and having no content might be different scenarios in code point of view, but they should be handled the same way. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi-cpp] arpadboda commented on issue #637: MINIFICPP-1020 - PutFile fails to create empty files
arpadboda commented on issue #637: MINIFICPP-1020 - PutFile fails to create empty files URL: https://github.com/apache/nifi-minifi-cpp/pull/637#issuecomment-526504548 > Can you provide an example flow? I created a flow of that pulled empty files via GetFile -> PutFile and it wrote those empty files. Is this the case where the resource claim has an invalid pointer to a file by virtue of GenerateFlowFile? That would be a separate issue in my opinion. I could be wrong but any help in reproducing this per the PR/ticket description would be helpful. > > Happy to provide logs where I did not see a failure or rollback. I connected GenerateFlowFile set to 0B with PutFile. If you check my other PR (the GenerateFlowFile one), and run that without having this change, you are going to see the issue. I agree that in the background having empty content and having no content might be different scenarios in code point of view, but they should be handled the same way. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi-cpp] arpadboda commented on a change in pull request #637: MINIFICPP-1020 - PutFile fails to create empty files
arpadboda commented on a change in pull request #637: MINIFICPP-1020 - PutFile
fails to create empty files
URL: https://github.com/apache/nifi-minifi-cpp/pull/637#discussion_r319400821
##
File path: extensions/standard-processors/processors/PutFile.cpp
##
@@ -218,11 +218,24 @@ bool PutFile::putFile(core::ProcessSession *session,
std::shared_ptrread(flowFile, );
+ bool success = false;
+
+ if (flowFile->getSize() > 0) {
Review comment:
In case there is no content of the flowfile, processsession::read is going
to rollback:
```
std::shared_ptr stream =
process_context_->getContentRepository()->read(claim);
if (nullptr == stream) {
rollback();
return;
}
```
Which is fine in a way as you are trying to read something that doesn't
exist.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
