[jira] [Assigned] (NIFI-5435) Prometheus /metrics http endpoint for monitoring integration
[ https://issues.apache.org/jira/browse/NIFI-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemantha kumara M S reassigned NIFI-5435: - Assignee: Hemantha kumara M S > Prometheus /metrics http endpoint for monitoring integration > > > Key: NIFI-5435 > URL: https://issues.apache.org/jira/browse/NIFI-5435 > Project: Apache NiFi > Issue Type: New Feature > Components: Configuration, Core Framework, Extensions, Tools and > Build >Affects Versions: 1.5.0 >Reporter: Hari Sekhon >Assignee: Hemantha kumara M S >Priority: Major > > Feature Request to add Prometheus /metrics http endpoint for monitoring > integration: > [https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E] > Prometheus metrics format for that endpoint: > [https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-5435) Prometheus /metrics http endpoint for monitoring integration
[ https://issues.apache.org/jira/browse/NIFI-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemantha kumara M S reassigned NIFI-5435: - Assignee: (was: Hemantha kumara M S) > Prometheus /metrics http endpoint for monitoring integration > > > Key: NIFI-5435 > URL: https://issues.apache.org/jira/browse/NIFI-5435 > Project: Apache NiFi > Issue Type: New Feature > Components: Configuration, Core Framework, Extensions, Tools and > Build >Affects Versions: 1.5.0 >Reporter: Hari Sekhon >Priority: Major > > Feature Request to add Prometheus /metrics http endpoint for monitoring > integration: > [https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E] > Prometheus metrics format for that endpoint: > [https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5886) Add RPM profile for NiFi Toolkit
[ https://issues.apache.org/jira/browse/NIFI-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yoshiaki Takahashi updated NIFI-5886: - Status: Patch Available (was: In Progress) > Add RPM profile for NiFi Toolkit > > > Key: NIFI-5886 > URL: https://issues.apache.org/jira/browse/NIFI-5886 > Project: Apache NiFi > Issue Type: Improvement > Components: Tools and Build >Reporter: Yoshiaki Takahashi >Assignee: Yoshiaki Takahashi >Priority: Major > > RPM for NiFi Toolkit is useful for easy installation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5887) unable to unmarshal json to an object
meh created NIFI-5887:
-
Summary: unable to unmarshal json to an object
Key: NIFI-5887
URL: https://issues.apache.org/jira/browse/NIFI-5887
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 1.7.0
Reporter: meh
i have a attribute (that produced by a REST service and catched by invokeHTTP
processor) in JSON format like this:
{code:java}
test => {"key":"value"}{code}
and then i want to put it in flows JSON content using JOLT processor, my
content is something like this:
{code:java}
{ "id": 123, "user": "foo" }{code}
and my JOLT specification is this:
{code:java}
[{ "operation": "default", "spec": { "interest": "${test}" } }]{code}
the problem is here that, in JOLT advanced window with test attribute nifi
cannot put json object and shown this error:
{quote}*"Error occurred during transformation"*
{quote}
and when run processor this detailed error is become alerted:
{quote}*"unable to unmarshal json to an object"*
{quote}
my desired result is this:
{code:java}
{ "id": 123, "user": "foo", "interest": {"key":"value"} }{code}
is this a bug or am i wrong?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5886) Add RPM profile for NiFi Toolkit
[ https://issues.apache.org/jira/browse/NIFI-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16716185#comment-16716185 ] ASF GitHub Bot commented on NIFI-5886: -- GitHub user yoshiata opened a pull request: https://github.com/apache/nifi/pull/3213 NIFI-5886: Add RPM profile for NiFi Toolkit Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/yoshiata/nifi NIFI-5886 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3213.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3213 commit bcc2898e67570bfb96a46695b987d3676f996031 Author: Yoshiaki Takahashi Date: 2018-12-11T01:24:10Z NIFI-5886: Add RPM profile for NiFi Toolkit > Add RPM profile for NiFi Toolkit > > > Key: NIFI-5886 > URL: https://issues.apache.org/jira/browse/NIFI-5886 > Project: Apache NiFi > Issue Type: Improvement > Components: Tools and Build >Reporter: Yoshiaki Takahashi >Assignee: Yoshiaki Takahashi >Priority: Major > > RPM for NiFi Toolkit is useful for easy installation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3213: NIFI-5886: Add RPM profile for NiFi Toolkit
GitHub user yoshiata opened a pull request: https://github.com/apache/nifi/pull/3213 NIFI-5886: Add RPM profile for NiFi Toolkit Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/yoshiata/nifi NIFI-5886 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3213.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3213 commit bcc2898e67570bfb96a46695b987d3676f996031 Author: Yoshiaki Takahashi Date: 2018-12-11T01:24:10Z NIFI-5886: Add RPM profile for NiFi Toolkit ---
[jira] [Commented] (NIFI-5882) Connector Prioritizers doesn't work together with Load Balance Strategy
[ https://issues.apache.org/jira/browse/NIFI-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715970#comment-16715970 ] Koji Kawamura commented on NIFI-5882: - [~jzahner] Thanks for confirming the doc is correct. Now, I'm a bit confused with the above comment. To me, the two processors queue and one processor queue examples are the same. Those FlowFiles are sorted in each queue only, not cluster wide. * qL: p1, p4 * qR1: p2, p5 * qR2: p3, p6 This doesn't guarantee that the p4 is processed after p3. The node processes FlowFiles from local queue and send FlowFiles to other node concurrently. And once a FlowFile is sent to another node, the processing order will be out-of-control from the sending node. If you would like to achieve exact synced order to process those data, then you can't distribute load or distribution doesn't add any benefit. If you need to process p1, p2, p3, p4, p5, p6 one by one in order, then I suggest not using load-balancing. > Connector Prioritizers doesn't work together with Load Balance Strategy > --- > > Key: NIFI-5882 > URL: https://issues.apache.org/jira/browse/NIFI-5882 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.8.0 > Environment: Centos 7.5, Secured 8 Node NiFi Cluster >Reporter: Josef Zahner >Priority: Major > Attachments: connector_config.png, queue_with_one_processor.png, > queue_with_two_processors.png, template_overview.png > > > For my template please check the picture "template_overview.png". On the left > hand side the working (two processor) example and on the right hand side the > not working one (one processor). > I have a ListSFTP Processor which reads files from 4 different folders. In > the filename of the files is a number (epochtime) which I'm parsing and set > it as "priority" attribute. We have a cluster, so I what I want to achieve > for the FetchSFTP is, that the files are fetched in order and are equally > distributed over our 8-node cluster. > However, it seems that if I'm combining to set the "priority" attribute on an > UpdateAttribute processor and on the directly attached connector use the > following features: > * Load Balance Strategy: Round Robin > * Select Prioritizers: PriorityAttributePrioritizer > the prioritizers doesn't seem to have any impact. > If i'm setting the priority attribute on an extra processor and use there > only the prioritizer - all files are in order but still on the primary node. > On the next processor then I'm setting the loadbalancing strategy for the > cluster (and add another attribute, but doesn't matter) and the prioritizer > together. That way it works. A picture of the queue for both examples is > attached (queue_with_one_processor & queue_with_two_processors.png). > *To sum up*, it seems if I'm setting the "priority" attribute on an > UpdateAttribute processor and directly try to use it on the attached > connector with a loadbalancing strategy and the prioritizer > (PriorityAttributePrioritizer) then the priority attribute doesn't work as > expected. If I'm setting the "priority" attribute on a separate processor and > then do on an additional processor the magic load balancing strategy stuff > together with the prioritizer then it works. > Cheers -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-5886) Add RPM profile for NiFi Toolkit
[ https://issues.apache.org/jira/browse/NIFI-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yoshiaki Takahashi reassigned NIFI-5886: Assignee: Yoshiaki Takahashi > Add RPM profile for NiFi Toolkit > > > Key: NIFI-5886 > URL: https://issues.apache.org/jira/browse/NIFI-5886 > Project: Apache NiFi > Issue Type: Improvement > Components: Tools and Build >Reporter: Yoshiaki Takahashi >Assignee: Yoshiaki Takahashi >Priority: Major > > RPM for NiFi Toolkit is useful for easy installation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-5875) PriorityAttributePrioritizer documentation is not clear.
[ https://issues.apache.org/jira/browse/NIFI-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Kawamura reassigned NIFI-5875: --- Assignee: Wil Selwood > PriorityAttributePrioritizer documentation is not clear. > > > Key: NIFI-5875 > URL: https://issues.apache.org/jira/browse/NIFI-5875 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation Website >Reporter: Wil Selwood >Assignee: Wil Selwood >Priority: Major > > The documentation for the PriorityAttributePrioritizer is not particularly > clear. The First part says highest goes first but then the example at the end > says that 1 would go before 9. This could be clearer. > I've just had to go look up the code to figure out what it was doing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (NIFI-5875) PriorityAttributePrioritizer documentation is not clear.
[ https://issues.apache.org/jira/browse/NIFI-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Koji Kawamura resolved NIFI-5875. - Resolution: Fixed Fix Version/s: 1.9.0 I've added 'contributor' role to your Jira account for NiFi project. You can assign yourself to any NiFi Jira. Thanks again [~wil.selwood] for improving docs! > PriorityAttributePrioritizer documentation is not clear. > > > Key: NIFI-5875 > URL: https://issues.apache.org/jira/browse/NIFI-5875 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation Website >Reporter: Wil Selwood >Assignee: Wil Selwood >Priority: Major > Fix For: 1.9.0 > > > The documentation for the PriorityAttributePrioritizer is not particularly > clear. The First part says highest goes first but then the example at the end > says that 1 would go before 9. This could be clearer. > I've just had to go look up the code to figure out what it was doing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5875) PriorityAttributePrioritizer documentation is not clear.
[ https://issues.apache.org/jira/browse/NIFI-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715948#comment-16715948 ] ASF subversion and git services commented on NIFI-5875: --- Commit 1a937b6573d77255984f9856bbc63f2c22f0 in nifi's branch refs/heads/master from [~wselw...@gmail.com] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=1a937b6 ] NIFI-5875 Improve docs around the PriorityAttributePrioritizer Clear up the documentation around the PriorityAttributePrioritizer to make it clearer that if the attributes are both number the largest wins but if they are not numbers they sort the other way around. I had to go and look at the code to work out what was going on after reading the existing documentation. Simplify PriorityAttributePrioritzer docs Simplify the main section of documentation for PriorityAttributePrioritzer and then add a notes section with more details about the ordering. Fix example. Put back comment setting the priority attribute Further improvements to the Priority ordering This closes #3205. Signed-off-by: Koji Kawamura > PriorityAttributePrioritizer documentation is not clear. > > > Key: NIFI-5875 > URL: https://issues.apache.org/jira/browse/NIFI-5875 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation Website >Reporter: Wil Selwood >Assignee: Wil Selwood >Priority: Major > > The documentation for the PriorityAttributePrioritizer is not particularly > clear. The First part says highest goes first but then the example at the end > says that 1 would go before 9. This could be clearer. > I've just had to go look up the code to figure out what it was doing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5875) PriorityAttributePrioritizer documentation is not clear.
[ https://issues.apache.org/jira/browse/NIFI-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715951#comment-16715951 ] ASF GitHub Bot commented on NIFI-5875: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3205 > PriorityAttributePrioritizer documentation is not clear. > > > Key: NIFI-5875 > URL: https://issues.apache.org/jira/browse/NIFI-5875 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation Website >Reporter: Wil Selwood >Assignee: Wil Selwood >Priority: Major > > The documentation for the PriorityAttributePrioritizer is not particularly > clear. The First part says highest goes first but then the example at the end > says that 1 would go before 9. This could be clearer. > I've just had to go look up the code to figure out what it was doing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3205: NIFI-5875 Improve docs around the PriorityAttribute...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3205 ---
[jira] [Commented] (NIFI-5875) PriorityAttributePrioritizer documentation is not clear.
[ https://issues.apache.org/jira/browse/NIFI-5875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715944#comment-16715944 ] ASF GitHub Bot commented on NIFI-5875: -- Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/3205 Looks perfect, +1 merging. Thanks @wselwood ! > PriorityAttributePrioritizer documentation is not clear. > > > Key: NIFI-5875 > URL: https://issues.apache.org/jira/browse/NIFI-5875 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation Website >Reporter: Wil Selwood >Priority: Major > > The documentation for the PriorityAttributePrioritizer is not particularly > clear. The First part says highest goes first but then the example at the end > says that 1 would go before 9. This could be clearer. > I've just had to go look up the code to figure out what it was doing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #3205: NIFI-5875 Improve docs around the PriorityAttributePriorit...
Github user ijokarumawak commented on the issue: https://github.com/apache/nifi/pull/3205 Looks perfect, +1 merging. Thanks @wselwood ! ---
[jira] [Commented] (NIFI-5885) EL expression of anyDelineatedValue() and or() fails with stacktrace
[
https://issues.apache.org/jira/browse/NIFI-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715755#comment-16715755
]
Yan Chen commented on NIFI-5885:
[~ijokarumawak] Thanks for the quick response and the work-around suggestion.
I'll wait for the fix from next official release (I already use other
work-around).
> EL expression of anyDelineatedValue() and or() fails with stacktrace
> -
>
> Key: NIFI-5885
> URL: https://issues.apache.org/jira/browse/NIFI-5885
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Yan Chen
>Assignee: Koji Kawamura
>Priority: Major
> Fix For: 1.9.0
>
> Attachments: anyDelineatedValue_or_test.xml,
> image-2018-12-10-11-37-55-545.png
>
>
> See attached Template test-case. Following EL expression failed:
> ${anyDelineatedValue('${names}',
> ','):equals('${my_name}'):or(${anyDelineatedValue('${types}',
> ','):equals('${my_type}')})}
> with stacktrace:
> {code:java}
> // code placeholder
> 2018-12-09 19:03:25,534 ERROR [Timer-Driven Process Thread-7]
> o.a.n.p.standard.RouteOnAttribute
> RouteOnAttribute[id=954d4a07-0167-1000--93233dd3]
> RouteOnAttribute[id=954d4a07-0167-1000--93233dd3] failed to process
> session due to java.lang.ArrayIndexOutOfBoundsException: 1; Processor
> Administratively Yielded for 1 sec: java.lang.ArrayIndexOutOfBoundsException:
> 1
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.nifi.attribute.expression.language.evaluation.selection.DelineatedAttributeEvaluator.evaluate(DelineatedAttributeEvaluator.java:65)
> at
> org.apache.nifi.attribute.expression.language.evaluation.functions.EqualsEvaluator.evaluate(EqualsEvaluator.java:40)
> at
> org.apache.nifi.attribute.expression.language.evaluation.selection.AnyAttributeEvaluator.evaluate(AnyAttributeEvaluator.java:38)
> at
> org.apache.nifi.attribute.expression.language.evaluation.functions.OrEvaluator.evaluate(OrEvaluator.java:47)
> at
> org.apache.nifi.attribute.expression.language.evaluation.selection.AnyAttributeEvaluator.evaluate(AnyAttributeEvaluator.java:49)
> at
> org.apache.nifi.attribute.expression.language.Query.evaluate(Query.java:363)
> at
> org.apache.nifi.attribute.expression.language.Query.evaluateExpression(Query.java:204)
> at
> org.apache.nifi.attribute.expression.language.CompiledExpression.evaluate(CompiledExpression.java:58)
> at
> org.apache.nifi.attribute.expression.language.StandardPreparedQuery.evaluateExpressions(StandardPreparedQuery.java:53)
> at
> org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:160)
> at
> org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:148)
> at
> org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:133)
> at
> org.apache.nifi.processors.standard.RouteOnAttribute.onTrigger(RouteOnAttribute.java:219)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5859) Update NAR maven plugin to include information about Extensions
[
https://issues.apache.org/jira/browse/NIFI-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715535#comment-16715535
]
ASF GitHub Bot commented on NIFI-5859:
--
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi-maven/pull/7#discussion_r240308506
--- Diff: src/main/java/org/apache/nifi/NarMojo.java ---
@@ -426,12 +479,260 @@
protected boolean cloneDuringInstanceClassLoading;
+/**
+ * The {@link RepositorySystemSession} used for obtaining the local
and remote artifact repositories.
+ */
+@Parameter(defaultValue = "${repositorySystemSession}", readonly =
true)
+private RepositorySystemSession repoSession;
+
+
+/**
+ * The {@link ProjectBuilder} used to generate the {@link
MavenProject} for the nar artifact the dependency tree is being generated for.
+ */
+@Component
+private ProjectBuilder projectBuilder;
+
+
+
@Override
-public void execute() throws MojoExecutionException,
MojoFailureException {
+public void execute() throws MojoExecutionException {
copyDependencies();
+
+try {
+generateDocumentation();
--- End diff --
Could we add an optional property like "enforceDocGeneration" (or some
better name) that when true we would let the exception be thrown here to fail
the build?
The use-case would be to know that when doing a release we have
successfully generated the docs if the build passed. Right now it can silently
fail with a warning and you'd have to manually inspect every NAR to really know.
> Update NAR maven plugin to include information about Extensions
> ---
>
> Key: NIFI-5859
> URL: https://issues.apache.org/jira/browse/NIFI-5859
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Tools and Build
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
>
> In order to have the NiFi Registry host any extensions, the registry will
> need a way to know what extensions exist in a given NAR. Currently, that
> information is not available directly.
> The NAR maven plugin should be updated to provide a list of extensions and
> for each one, provide at least the following minimal information:
> * Extension Type
> * Extension Name
> * Capability Description
> * Whether or not the component is Restricted, "sub-restrictions" it has, and
> explanations of both
> * Any Tags that the component has
> * If the component is a Controller Service, any Controller Service API's
> that it provides
> Additionally, it would be ideal to provide all documentation for the
> component within the NAR. It is best, though, not to write the documentation
> in HTML as is done now but rather in XML or some sort of form that provides
> the information in a structured way without any styling. This would allow the
> documentation to be rendered consistently, even if the styling changes from 1
> version to the next.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5859) Update NAR maven plugin to include information about Extensions
[
https://issues.apache.org/jira/browse/NIFI-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715538#comment-16715538
]
ASF GitHub Bot commented on NIFI-5859:
--
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi-maven/pull/7#discussion_r240314647
--- Diff: src/main/java/org/apache/nifi/XmlDefinitionWriter.java ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi;
+
+import org.apache.nifi.extension.definition.ExtensionDefinition;
+import org.apache.nifi.extension.definition.ExtensionType;
+import org.apache.nifi.extension.definition.Restriction;
+import org.apache.nifi.extension.definition.Restrictions;
+import org.apache.nifi.extension.definition.ServiceAPIDefinition;
+
+import javax.xml.stream.XMLOutputFactory;
+import javax.xml.stream.XMLStreamException;
+import javax.xml.stream.XMLStreamWriter;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+public class XmlDefinitionWriter {
--- End diff --
Can this be removed now that we are using the classes from nifi-api?
> Update NAR maven plugin to include information about Extensions
> ---
>
> Key: NIFI-5859
> URL: https://issues.apache.org/jira/browse/NIFI-5859
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Tools and Build
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
>
> In order to have the NiFi Registry host any extensions, the registry will
> need a way to know what extensions exist in a given NAR. Currently, that
> information is not available directly.
> The NAR maven plugin should be updated to provide a list of extensions and
> for each one, provide at least the following minimal information:
> * Extension Type
> * Extension Name
> * Capability Description
> * Whether or not the component is Restricted, "sub-restrictions" it has, and
> explanations of both
> * Any Tags that the component has
> * If the component is a Controller Service, any Controller Service API's
> that it provides
> Additionally, it would be ideal to provide all documentation for the
> component within the NAR. It is best, though, not to write the documentation
> in HTML as is done now but rather in XML or some sort of form that provides
> the information in a structured way without any styling. This would allow the
> documentation to be rendered consistently, even if the styling changes from 1
> version to the next.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5859) Update NAR maven plugin to include information about Extensions
[
https://issues.apache.org/jira/browse/NIFI-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715537#comment-16715537
]
ASF GitHub Bot commented on NIFI-5859:
--
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi-maven/pull/7#discussion_r240314701
--- Diff: src/main/java/org/apache/nifi/PropertiesDefinitionWriter.java ---
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi;
+
+import org.apache.nifi.extension.definition.ExtensionDefinition;
+import org.apache.nifi.extension.definition.Restriction;
+import org.apache.nifi.extension.definition.Restrictions;
+import org.apache.nifi.extension.definition.ServiceAPIDefinition;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.Objects;
+import java.util.Properties;
+
+public class PropertiesDefinitionWriter {
--- End diff --
Can this be removed now that we are using the classes from nifi-api?
> Update NAR maven plugin to include information about Extensions
> ---
>
> Key: NIFI-5859
> URL: https://issues.apache.org/jira/browse/NIFI-5859
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Tools and Build
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
>
> In order to have the NiFi Registry host any extensions, the registry will
> need a way to know what extensions exist in a given NAR. Currently, that
> information is not available directly.
> The NAR maven plugin should be updated to provide a list of extensions and
> for each one, provide at least the following minimal information:
> * Extension Type
> * Extension Name
> * Capability Description
> * Whether or not the component is Restricted, "sub-restrictions" it has, and
> explanations of both
> * Any Tags that the component has
> * If the component is a Controller Service, any Controller Service API's
> that it provides
> Additionally, it would be ideal to provide all documentation for the
> component within the NAR. It is best, though, not to write the documentation
> in HTML as is done now but rather in XML or some sort of form that provides
> the information in a structured way without any styling. This would allow the
> documentation to be rendered consistently, even if the styling changes from 1
> version to the next.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5859) Update NAR maven plugin to include information about Extensions
[
https://issues.apache.org/jira/browse/NIFI-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715540#comment-16715540
]
ASF GitHub Bot commented on NIFI-5859:
--
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi-maven/pull/7#discussion_r240310495
--- Diff: src/main/java/org/apache/nifi/NarMojo.java ---
@@ -426,12 +479,260 @@
protected boolean cloneDuringInstanceClassLoading;
+/**
+ * The {@link RepositorySystemSession} used for obtaining the local
and remote artifact repositories.
+ */
+@Parameter(defaultValue = "${repositorySystemSession}", readonly =
true)
+private RepositorySystemSession repoSession;
+
+
+/**
+ * The {@link ProjectBuilder} used to generate the {@link
MavenProject} for the nar artifact the dependency tree is being generated for.
+ */
+@Component
+private ProjectBuilder projectBuilder;
+
+
+
@Override
-public void execute() throws MojoExecutionException,
MojoFailureException {
+public void execute() throws MojoExecutionException {
copyDependencies();
+
+try {
+generateDocumentation();
+} catch (final Throwable t) { // Catch Throwable in case a linkage
error such as NoClassDefFoundError occurs
+getLog().warn("Could not generate extensions' documentation",
t);
+}
+
makeNar();
}
+private File getExtensionsDocumentationFile() {
+final File directory = new File(projectBuildDirectory,
"META-INF/docs");
+return new File(directory, "extension-docs.xml");
+}
+
+private void generateDocumentation() throws MojoExecutionException {
+getLog().info("Generating documentation for NiFi extensions in the
NAR...");
+
+// Create the ClassLoader for the NAR
+final ExtensionClassLoaderFactory classLoaderFactory =
createClassLoaderFactory();
+
+final ExtensionClassLoader extensionClassLoader;
+try {
+extensionClassLoader =
classLoaderFactory.createExtensionClassLoader();
+} catch (final Exception e) {
+if (getLog().isDebugEnabled()) {
+getLog().debug("Unable to create a ClassLoader for
documenting extensions. If this NAR contains any NiFi Extensions, those
extensions will not be documented.", e);
+} else {
+getLog().warn("Unable to create a ClassLoader for
documenting extensions. If this NAR contains any NiFi Extensions, those
extensions will not be documented. " +
+"Enable mvn DEBUG output for more information (mvn
-X).");
+}
+
+return;
+}
+
+
+final File docsFile = getExtensionsDocumentationFile();
+createDirectory(docsFile.getParentFile());
+
+final File additionalDetailsDir = new
File(docsFile.getParentFile(), "additional-details");
+createDirectory(additionalDetailsDir);
+
+try (final OutputStream out = new FileOutputStream(docsFile)) {
+
+final XMLStreamWriter xmlWriter =
XMLOutputFactory.newInstance().createXMLStreamWriter(out, "UTF-8");
+try {
+xmlWriter.writeStartElement("extensions");
+
+final String nifiApiVersion =
extensionClassLoader.getNiFiApiVersion();
+xmlWriter.writeStartElement("nifiApiVersion");
+xmlWriter.writeCharacters(nifiApiVersion);
+xmlWriter.writeEndElement();
+
+final Class docWriterClass;
+try {
+docWriterClass =
Class.forName(DOCUMENTATION_WRITER_CLASS_NAME, false, extensionClassLoader);
--- End diff --
Should we do this before entering the try block on line 542 so that if the
doc writer class can't be found we won't write anything to the docs file?
> Update NAR maven plugin to include information about Extensions
> ---
>
> Key: NIFI-5859
> URL: https://issues.apache.org/jira/browse/NIFI-5859
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Tools and Build
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
>
> In order to have the NiFi Registry host any extensions, the registry will
> need a way to know what extensions exist in a given NAR. Currently, that
> information is not available directly.
> The NAR maven plugin should be updated to provide a list of extensions and
> for each one, provide at least the
[jira] [Commented] (NIFI-5859) Update NAR maven plugin to include information about Extensions
[
https://issues.apache.org/jira/browse/NIFI-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715536#comment-16715536
]
ASF GitHub Bot commented on NIFI-5859:
--
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi-maven/pull/7#discussion_r240374218
--- Diff:
src/main/java/org/apache/nifi/extension/definition/extraction/ExtensionDefinitionFactory.java
---
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.extension.definition.extraction;
+
+import org.apache.maven.artifact.Artifact;
+import org.apache.maven.plugin.MojoExecutionException;
+import org.apache.nifi.extension.definition.ExtensionDefinition;
+import org.apache.nifi.extension.definition.ExtensionType;
+import org.apache.nifi.extension.definition.Restriction;
+import org.apache.nifi.extension.definition.Restrictions;
+import org.apache.nifi.extension.definition.ServiceAPIDefinition;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.Reader;
+import java.lang.annotation.Annotation;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.net.URL;
+import java.util.Collections;
+import java.util.Enumeration;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public class ExtensionDefinitionFactory {
+private static final String SERVICES_DIRECTORY = "META-INF/services/";
+
+private static final Map INTERFACE_NAMES = new
HashMap<>();
+static {
+INTERFACE_NAMES.put(ExtensionType.PROCESSOR,
"org.apache.nifi.processor.Processor");
+INTERFACE_NAMES.put(ExtensionType.CONTROLLER_SERVICE,
"org.apache.nifi.controller.ControllerService");
+INTERFACE_NAMES.put(ExtensionType.REPORTING_TASK,
"org.apache.nifi.reporting.ReportingTask");
+}
+
+private final ClassLoader extensionClassLoader;
+
+public ExtensionDefinitionFactory(final ClassLoader classLoader) {
+this.extensionClassLoader = classLoader;
+}
+
+public Set discoverExtensions(final ExtensionType
extensionType) throws IOException {
+final String interfaceName = INTERFACE_NAMES.get(extensionType);
+final Set classNames = discoverClassNames(interfaceName);
+
+if (classNames.isEmpty()) {
+return Collections.emptySet();
+}
+
+final Set definitions = new HashSet<>();
+for (final String className : classNames) {
+try {
+definitions.add(createExtensionDefinition(extensionType,
className));
+} catch (final Exception e) {
+throw new IOException("Failed to create Extension
Definition for " + extensionType + " " + className, e);
+}
+}
+
+return definitions;
+}
+
+private ExtensionDefinition createExtensionDefinition(final
ExtensionType extensionType, final String className) throws
ClassNotFoundException,
+IllegalAccessException, NoSuchMethodException,
InvocationTargetException {
+
+final Class extensionClass = Class.forName(className, false,
extensionClassLoader);
+
+final String capabilityDescription =
getCapabilityDescription(extensionClass);
+final Set tags = getTags(extensionClass);
+final Restrictions restrictions = getRestrictions(extensionClass);
+final Set serviceApis =
getProvidedServiceAPIs(extensionType, extensionClass);
+
+return new StandardExtensionDefinition(extensionType, className,
capabilityDescription, tags, restrictions, serviceApis);
+}
+
+private Set
[GitHub] nifi-maven pull request #7: NIFI-5859: Build NAR Extension Definitions/docs ...
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi-maven/pull/7#discussion_r240310495
--- Diff: src/main/java/org/apache/nifi/NarMojo.java ---
@@ -426,12 +479,260 @@
protected boolean cloneDuringInstanceClassLoading;
+/**
+ * The {@link RepositorySystemSession} used for obtaining the local
and remote artifact repositories.
+ */
+@Parameter(defaultValue = "${repositorySystemSession}", readonly =
true)
+private RepositorySystemSession repoSession;
+
+
+/**
+ * The {@link ProjectBuilder} used to generate the {@link
MavenProject} for the nar artifact the dependency tree is being generated for.
+ */
+@Component
+private ProjectBuilder projectBuilder;
+
+
+
@Override
-public void execute() throws MojoExecutionException,
MojoFailureException {
+public void execute() throws MojoExecutionException {
copyDependencies();
+
+try {
+generateDocumentation();
+} catch (final Throwable t) { // Catch Throwable in case a linkage
error such as NoClassDefFoundError occurs
+getLog().warn("Could not generate extensions' documentation",
t);
+}
+
makeNar();
}
+private File getExtensionsDocumentationFile() {
+final File directory = new File(projectBuildDirectory,
"META-INF/docs");
+return new File(directory, "extension-docs.xml");
+}
+
+private void generateDocumentation() throws MojoExecutionException {
+getLog().info("Generating documentation for NiFi extensions in the
NAR...");
+
+// Create the ClassLoader for the NAR
+final ExtensionClassLoaderFactory classLoaderFactory =
createClassLoaderFactory();
+
+final ExtensionClassLoader extensionClassLoader;
+try {
+extensionClassLoader =
classLoaderFactory.createExtensionClassLoader();
+} catch (final Exception e) {
+if (getLog().isDebugEnabled()) {
+getLog().debug("Unable to create a ClassLoader for
documenting extensions. If this NAR contains any NiFi Extensions, those
extensions will not be documented.", e);
+} else {
+getLog().warn("Unable to create a ClassLoader for
documenting extensions. If this NAR contains any NiFi Extensions, those
extensions will not be documented. " +
+"Enable mvn DEBUG output for more information (mvn
-X).");
+}
+
+return;
+}
+
+
+final File docsFile = getExtensionsDocumentationFile();
+createDirectory(docsFile.getParentFile());
+
+final File additionalDetailsDir = new
File(docsFile.getParentFile(), "additional-details");
+createDirectory(additionalDetailsDir);
+
+try (final OutputStream out = new FileOutputStream(docsFile)) {
+
+final XMLStreamWriter xmlWriter =
XMLOutputFactory.newInstance().createXMLStreamWriter(out, "UTF-8");
+try {
+xmlWriter.writeStartElement("extensions");
+
+final String nifiApiVersion =
extensionClassLoader.getNiFiApiVersion();
+xmlWriter.writeStartElement("nifiApiVersion");
+xmlWriter.writeCharacters(nifiApiVersion);
+xmlWriter.writeEndElement();
+
+final Class docWriterClass;
+try {
+docWriterClass =
Class.forName(DOCUMENTATION_WRITER_CLASS_NAME, false, extensionClassLoader);
--- End diff --
Should we do this before entering the try block on line 542 so that if the
doc writer class can't be found we won't write anything to the docs file?
---
[GitHub] nifi-maven pull request #7: NIFI-5859: Build NAR Extension Definitions/docs ...
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi-maven/pull/7#discussion_r240374218
--- Diff:
src/main/java/org/apache/nifi/extension/definition/extraction/ExtensionDefinitionFactory.java
---
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.extension.definition.extraction;
+
+import org.apache.maven.artifact.Artifact;
+import org.apache.maven.plugin.MojoExecutionException;
+import org.apache.nifi.extension.definition.ExtensionDefinition;
+import org.apache.nifi.extension.definition.ExtensionType;
+import org.apache.nifi.extension.definition.Restriction;
+import org.apache.nifi.extension.definition.Restrictions;
+import org.apache.nifi.extension.definition.ServiceAPIDefinition;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.Reader;
+import java.lang.annotation.Annotation;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.net.URL;
+import java.util.Collections;
+import java.util.Enumeration;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public class ExtensionDefinitionFactory {
+private static final String SERVICES_DIRECTORY = "META-INF/services/";
+
+private static final Map INTERFACE_NAMES = new
HashMap<>();
+static {
+INTERFACE_NAMES.put(ExtensionType.PROCESSOR,
"org.apache.nifi.processor.Processor");
+INTERFACE_NAMES.put(ExtensionType.CONTROLLER_SERVICE,
"org.apache.nifi.controller.ControllerService");
+INTERFACE_NAMES.put(ExtensionType.REPORTING_TASK,
"org.apache.nifi.reporting.ReportingTask");
+}
+
+private final ClassLoader extensionClassLoader;
+
+public ExtensionDefinitionFactory(final ClassLoader classLoader) {
+this.extensionClassLoader = classLoader;
+}
+
+public Set discoverExtensions(final ExtensionType
extensionType) throws IOException {
+final String interfaceName = INTERFACE_NAMES.get(extensionType);
+final Set classNames = discoverClassNames(interfaceName);
+
+if (classNames.isEmpty()) {
+return Collections.emptySet();
+}
+
+final Set definitions = new HashSet<>();
+for (final String className : classNames) {
+try {
+definitions.add(createExtensionDefinition(extensionType,
className));
+} catch (final Exception e) {
+throw new IOException("Failed to create Extension
Definition for " + extensionType + " " + className, e);
+}
+}
+
+return definitions;
+}
+
+private ExtensionDefinition createExtensionDefinition(final
ExtensionType extensionType, final String className) throws
ClassNotFoundException,
+IllegalAccessException, NoSuchMethodException,
InvocationTargetException {
+
+final Class extensionClass = Class.forName(className, false,
extensionClassLoader);
+
+final String capabilityDescription =
getCapabilityDescription(extensionClass);
+final Set tags = getTags(extensionClass);
+final Restrictions restrictions = getRestrictions(extensionClass);
+final Set serviceApis =
getProvidedServiceAPIs(extensionType, extensionClass);
+
+return new StandardExtensionDefinition(extensionType, className,
capabilityDescription, tags, restrictions, serviceApis);
+}
+
+private Set getProvidedServiceAPIs(final
ExtensionType extensionType, final Class extensionClass) throws
ClassNotFoundException {
+if (extensionType != ExtensionType.CONTROLLER_SERVICE) {
+return Collections.emptySet();
[GitHub] nifi-maven pull request #7: NIFI-5859: Build NAR Extension Definitions/docs ...
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi-maven/pull/7#discussion_r240314647
--- Diff: src/main/java/org/apache/nifi/XmlDefinitionWriter.java ---
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi;
+
+import org.apache.nifi.extension.definition.ExtensionDefinition;
+import org.apache.nifi.extension.definition.ExtensionType;
+import org.apache.nifi.extension.definition.Restriction;
+import org.apache.nifi.extension.definition.Restrictions;
+import org.apache.nifi.extension.definition.ServiceAPIDefinition;
+
+import javax.xml.stream.XMLOutputFactory;
+import javax.xml.stream.XMLStreamException;
+import javax.xml.stream.XMLStreamWriter;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+public class XmlDefinitionWriter {
--- End diff --
Can this be removed now that we are using the classes from nifi-api?
---
[GitHub] nifi-maven pull request #7: NIFI-5859: Build NAR Extension Definitions/docs ...
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi-maven/pull/7#discussion_r240314701
--- Diff: src/main/java/org/apache/nifi/PropertiesDefinitionWriter.java ---
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi;
+
+import org.apache.nifi.extension.definition.ExtensionDefinition;
+import org.apache.nifi.extension.definition.Restriction;
+import org.apache.nifi.extension.definition.Restrictions;
+import org.apache.nifi.extension.definition.ServiceAPIDefinition;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.Objects;
+import java.util.Properties;
+
+public class PropertiesDefinitionWriter {
--- End diff --
Can this be removed now that we are using the classes from nifi-api?
---
[GitHub] nifi-maven pull request #7: NIFI-5859: Build NAR Extension Definitions/docs ...
Github user bbende commented on a diff in the pull request:
https://github.com/apache/nifi-maven/pull/7#discussion_r240308506
--- Diff: src/main/java/org/apache/nifi/NarMojo.java ---
@@ -426,12 +479,260 @@
protected boolean cloneDuringInstanceClassLoading;
+/**
+ * The {@link RepositorySystemSession} used for obtaining the local
and remote artifact repositories.
+ */
+@Parameter(defaultValue = "${repositorySystemSession}", readonly =
true)
+private RepositorySystemSession repoSession;
+
+
+/**
+ * The {@link ProjectBuilder} used to generate the {@link
MavenProject} for the nar artifact the dependency tree is being generated for.
+ */
+@Component
+private ProjectBuilder projectBuilder;
+
+
+
@Override
-public void execute() throws MojoExecutionException,
MojoFailureException {
+public void execute() throws MojoExecutionException {
copyDependencies();
+
+try {
+generateDocumentation();
--- End diff --
Could we add an optional property like "enforceDocGeneration" (or some
better name) that when true we would let the exception be thrown here to fail
the build?
The use-case would be to know that when doing a release we have
successfully generated the docs if the build passed. Right now it can silently
fail with a warning and you'd have to manually inspect every NAR to really know.
---
[GitHub] nifi-minifi pull request #149: MINIFI-484: Fixing line breaks in system admi...
Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi/pull/149 ---
[jira] [Commented] (NIFI-5859) Update NAR maven plugin to include information about Extensions
[ https://issues.apache.org/jira/browse/NIFI-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715008#comment-16715008 ] ASF subversion and git services commented on NIFI-5859: --- Commit 1ac5b93144ebd92c907fb49664a524e196ae20c6 in nifi's branch refs/heads/master from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=1ac5b93 ] NIFI-5859: Added XML-based documentation writer that can be used to document a component. - Found several instances of nifi-framework-api's ProviderException being thrown from processors. Changed those to IllegalStateException, as ProviderException is not an appropriate Exception in those cases, and extensions should not depend on nifi-framework-api. - Performed some cleanup, moving Property Descriptors from Controller Service API's/specs into the implementations. Adding to the Service API results in bringing in nifi-utils to the nifi-standard-services-api-nar, which is a bad practice. The 'main' service api nar should not have a dependency on a util class. NIFI-5859: Added javadocs. Fixed pom.xml that was left pointing to snapshot version of nar maven plugin NIFI-5859: Addressing review feedback: adding component type, multiple additional details into separate file(s) This closes #3192. Signed-off-by: Bryan Bende > Update NAR maven plugin to include information about Extensions > --- > > Key: NIFI-5859 > URL: https://issues.apache.org/jira/browse/NIFI-5859 > Project: Apache NiFi > Issue Type: New Feature > Components: Tools and Build >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > > In order to have the NiFi Registry host any extensions, the registry will > need a way to know what extensions exist in a given NAR. Currently, that > information is not available directly. > The NAR maven plugin should be updated to provide a list of extensions and > for each one, provide at least the following minimal information: > * Extension Type > * Extension Name > * Capability Description > * Whether or not the component is Restricted, "sub-restrictions" it has, and > explanations of both > * Any Tags that the component has > * If the component is a Controller Service, any Controller Service API's > that it provides > Additionally, it would be ideal to provide all documentation for the > component within the NAR. It is best, though, not to write the documentation > in HTML as is done now but rather in XML or some sort of form that provides > the information in a structured way without any styling. This would allow the > documentation to be rendered consistently, even if the styling changes from 1 > version to the next. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5859) Update NAR maven plugin to include information about Extensions
[ https://issues.apache.org/jira/browse/NIFI-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715011#comment-16715011 ] ASF GitHub Bot commented on NIFI-5859: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3192 > Update NAR maven plugin to include information about Extensions > --- > > Key: NIFI-5859 > URL: https://issues.apache.org/jira/browse/NIFI-5859 > Project: Apache NiFi > Issue Type: New Feature > Components: Tools and Build >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > > In order to have the NiFi Registry host any extensions, the registry will > need a way to know what extensions exist in a given NAR. Currently, that > information is not available directly. > The NAR maven plugin should be updated to provide a list of extensions and > for each one, provide at least the following minimal information: > * Extension Type > * Extension Name > * Capability Description > * Whether or not the component is Restricted, "sub-restrictions" it has, and > explanations of both > * Any Tags that the component has > * If the component is a Controller Service, any Controller Service API's > that it provides > Additionally, it would be ideal to provide all documentation for the > component within the NAR. It is best, though, not to write the documentation > in HTML as is done now but rather in XML or some sort of form that provides > the information in a structured way without any styling. This would allow the > documentation to be rendered consistently, even if the styling changes from 1 > version to the next. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5859) Update NAR maven plugin to include information about Extensions
[ https://issues.apache.org/jira/browse/NIFI-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715007#comment-16715007 ] ASF subversion and git services commented on NIFI-5859: --- Commit 1ac5b93144ebd92c907fb49664a524e196ae20c6 in nifi's branch refs/heads/master from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=1ac5b93 ] NIFI-5859: Added XML-based documentation writer that can be used to document a component. - Found several instances of nifi-framework-api's ProviderException being thrown from processors. Changed those to IllegalStateException, as ProviderException is not an appropriate Exception in those cases, and extensions should not depend on nifi-framework-api. - Performed some cleanup, moving Property Descriptors from Controller Service API's/specs into the implementations. Adding to the Service API results in bringing in nifi-utils to the nifi-standard-services-api-nar, which is a bad practice. The 'main' service api nar should not have a dependency on a util class. NIFI-5859: Added javadocs. Fixed pom.xml that was left pointing to snapshot version of nar maven plugin NIFI-5859: Addressing review feedback: adding component type, multiple additional details into separate file(s) This closes #3192. Signed-off-by: Bryan Bende > Update NAR maven plugin to include information about Extensions > --- > > Key: NIFI-5859 > URL: https://issues.apache.org/jira/browse/NIFI-5859 > Project: Apache NiFi > Issue Type: New Feature > Components: Tools and Build >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > > In order to have the NiFi Registry host any extensions, the registry will > need a way to know what extensions exist in a given NAR. Currently, that > information is not available directly. > The NAR maven plugin should be updated to provide a list of extensions and > for each one, provide at least the following minimal information: > * Extension Type > * Extension Name > * Capability Description > * Whether or not the component is Restricted, "sub-restrictions" it has, and > explanations of both > * Any Tags that the component has > * If the component is a Controller Service, any Controller Service API's > that it provides > Additionally, it would be ideal to provide all documentation for the > component within the NAR. It is best, though, not to write the documentation > in HTML as is done now but rather in XML or some sort of form that provides > the information in a structured way without any styling. This would allow the > documentation to be rendered consistently, even if the styling changes from 1 > version to the next. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3192: NIFI-5859: Added XML-based documentation writer tha...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3192 ---
[jira] [Commented] (NIFI-5859) Update NAR maven plugin to include information about Extensions
[ https://issues.apache.org/jira/browse/NIFI-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715009#comment-16715009 ] ASF subversion and git services commented on NIFI-5859: --- Commit 1ac5b93144ebd92c907fb49664a524e196ae20c6 in nifi's branch refs/heads/master from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=1ac5b93 ] NIFI-5859: Added XML-based documentation writer that can be used to document a component. - Found several instances of nifi-framework-api's ProviderException being thrown from processors. Changed those to IllegalStateException, as ProviderException is not an appropriate Exception in those cases, and extensions should not depend on nifi-framework-api. - Performed some cleanup, moving Property Descriptors from Controller Service API's/specs into the implementations. Adding to the Service API results in bringing in nifi-utils to the nifi-standard-services-api-nar, which is a bad practice. The 'main' service api nar should not have a dependency on a util class. NIFI-5859: Added javadocs. Fixed pom.xml that was left pointing to snapshot version of nar maven plugin NIFI-5859: Addressing review feedback: adding component type, multiple additional details into separate file(s) This closes #3192. Signed-off-by: Bryan Bende > Update NAR maven plugin to include information about Extensions > --- > > Key: NIFI-5859 > URL: https://issues.apache.org/jira/browse/NIFI-5859 > Project: Apache NiFi > Issue Type: New Feature > Components: Tools and Build >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > > In order to have the NiFi Registry host any extensions, the registry will > need a way to know what extensions exist in a given NAR. Currently, that > information is not available directly. > The NAR maven plugin should be updated to provide a list of extensions and > for each one, provide at least the following minimal information: > * Extension Type > * Extension Name > * Capability Description > * Whether or not the component is Restricted, "sub-restrictions" it has, and > explanations of both > * Any Tags that the component has > * If the component is a Controller Service, any Controller Service API's > that it provides > Additionally, it would be ideal to provide all documentation for the > component within the NAR. It is best, though, not to write the documentation > in HTML as is done now but rather in XML or some sort of form that provides > the information in a structured way without any styling. This would allow the > documentation to be rendered consistently, even if the styling changes from 1 > version to the next. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5859) Update NAR maven plugin to include information about Extensions
[ https://issues.apache.org/jira/browse/NIFI-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16715006#comment-16715006 ] ASF GitHub Bot commented on NIFI-5859: -- Github user bbende commented on the issue: https://github.com/apache/nifi/pull/3192 LGTM, going to merge, thanks! > Update NAR maven plugin to include information about Extensions > --- > > Key: NIFI-5859 > URL: https://issues.apache.org/jira/browse/NIFI-5859 > Project: Apache NiFi > Issue Type: New Feature > Components: Tools and Build >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > > In order to have the NiFi Registry host any extensions, the registry will > need a way to know what extensions exist in a given NAR. Currently, that > information is not available directly. > The NAR maven plugin should be updated to provide a list of extensions and > for each one, provide at least the following minimal information: > * Extension Type > * Extension Name > * Capability Description > * Whether or not the component is Restricted, "sub-restrictions" it has, and > explanations of both > * Any Tags that the component has > * If the component is a Controller Service, any Controller Service API's > that it provides > Additionally, it would be ideal to provide all documentation for the > component within the NAR. It is best, though, not to write the documentation > in HTML as is done now but rather in XML or some sort of form that provides > the information in a structured way without any styling. This would allow the > documentation to be rendered consistently, even if the styling changes from 1 > version to the next. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #3192: NIFI-5859: Added XML-based documentation writer that can b...
Github user bbende commented on the issue: https://github.com/apache/nifi/pull/3192 LGTM, going to merge, thanks! ---
[jira] [Commented] (MINIFICPP-691) Improve method of gathering processor documentation and providing to C2
[ https://issues.apache.org/jira/browse/MINIFICPP-691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714974#comment-16714974 ] ASF GitHub Bot commented on MINIFICPP-691: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/457 > Improve method of gathering processor documentation and providing to C2 > --- > > Key: MINIFICPP-691 > URL: https://issues.apache.org/jira/browse/MINIFICPP-691 > Project: NiFi MiNiFi C++ > Issue Type: Task >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Labels: documentation > Fix For: 0.6.0 > > > Improve method of gathering processor documentation and providing to C2 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (MINIFICPP-691) Improve method of gathering processor documentation and providing to C2
[ https://issues.apache.org/jira/browse/MINIFICPP-691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri resolved MINIFICPP-691. --- Resolution: Fixed > Improve method of gathering processor documentation and providing to C2 > --- > > Key: MINIFICPP-691 > URL: https://issues.apache.org/jira/browse/MINIFICPP-691 > Project: NiFi MiNiFi C++ > Issue Type: Task >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Labels: documentation > Fix For: 0.6.0 > > > Improve method of gathering processor documentation and providing to C2 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #457: MINIFICPP-691: Add method for getting pro...
Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/457 ---
[jira] [Commented] (NIFI-5882) Connector Prioritizers doesn't work together with Load Balance Strategy
[ https://issues.apache.org/jira/browse/NIFI-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714665#comment-16714665 ] Josef Zahner commented on NIFI-5882: You are right, I see exactly the behaviour from the doc. The local queue for each node seems to be in order, so this is fine. I was confused because the order with two processors is different than the one with just one and we wanted to achieve a sorted order even over the cluster. *Two processors queue* (sorted overall and in the local queue) * qL: p1, p4 * qR1: p2, p5 * qR2: p3, p6 *One processor queue* (sorted just within the queue itself, but not over the whole cluster): * qL: p2,p6 * qR1: p1,p5 * qR2: p3,p6 With the two processor solution we would expect to get a fully synced queue right? > Connector Prioritizers doesn't work together with Load Balance Strategy > --- > > Key: NIFI-5882 > URL: https://issues.apache.org/jira/browse/NIFI-5882 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.8.0 > Environment: Centos 7.5, Secured 8 Node NiFi Cluster >Reporter: Josef Zahner >Priority: Major > Attachments: connector_config.png, queue_with_one_processor.png, > queue_with_two_processors.png, template_overview.png > > > For my template please check the picture "template_overview.png". On the left > hand side the working (two processor) example and on the right hand side the > not working one (one processor). > I have a ListSFTP Processor which reads files from 4 different folders. In > the filename of the files is a number (epochtime) which I'm parsing and set > it as "priority" attribute. We have a cluster, so I what I want to achieve > for the FetchSFTP is, that the files are fetched in order and are equally > distributed over our 8-node cluster. > However, it seems that if I'm combining to set the "priority" attribute on an > UpdateAttribute processor and on the directly attached connector use the > following features: > * Load Balance Strategy: Round Robin > * Select Prioritizers: PriorityAttributePrioritizer > the prioritizers doesn't seem to have any impact. > If i'm setting the priority attribute on an extra processor and use there > only the prioritizer - all files are in order but still on the primary node. > On the next processor then I'm setting the loadbalancing strategy for the > cluster (and add another attribute, but doesn't matter) and the prioritizer > together. That way it works. A picture of the queue for both examples is > attached (queue_with_one_processor & queue_with_two_processors.png). > *To sum up*, it seems if I'm setting the "priority" attribute on an > UpdateAttribute processor and directly try to use it on the attached > connector with a loadbalancing strategy and the prioritizer > (PriorityAttributePrioritizer) then the priority attribute doesn't work as > expected. If I'm setting the "priority" attribute on a separate processor and > then do on an additional processor the magic load balancing strategy stuff > together with the prioritizer then it works. > Cheers -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5629) GetFile becomes slow listing vast directories
[ https://issues.apache.org/jira/browse/NIFI-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714601#comment-16714601 ] ASF GitHub Bot commented on NIFI-5629: -- Github user adyoun2 commented on the issue: https://github.com/apache/nifi/pull/3033 Could someone take a look at this? Cheers > GetFile becomes slow listing vast directories > - > > Key: NIFI-5629 > URL: https://issues.apache.org/jira/browse/NIFI-5629 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.6.0 >Reporter: Adam >Priority: Minor > > GetFile repeatedly lists entire directories before applying batching, meaning > for vast directories it spends a long time listing directories. > > Pull request to follow. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #3033: NIFI-5629 GetFile vast listing performance
Github user adyoun2 commented on the issue: https://github.com/apache/nifi/pull/3033 Could someone take a look at this? Cheers ---
[jira] [Commented] (NIFI-5885) EL expression of anyDelineatedValue() and or() fails with stacktrace
[
https://issues.apache.org/jira/browse/NIFI-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714554#comment-16714554
]
ASF GitHub Bot commented on NIFI-5885:
--
Github user pvillard31 commented on the issue:
https://github.com/apache/nifi/pull/3212
+1, merging to master, thanks @ijokarumawak
> EL expression of anyDelineatedValue() and or() fails with stacktrace
> -
>
> Key: NIFI-5885
> URL: https://issues.apache.org/jira/browse/NIFI-5885
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Yan Chen
>Assignee: Koji Kawamura
>Priority: Major
> Attachments: anyDelineatedValue_or_test.xml,
> image-2018-12-10-11-37-55-545.png
>
>
> See attached Template test-case. Following EL expression failed:
> ${anyDelineatedValue('${names}',
> ','):equals('${my_name}'):or(${anyDelineatedValue('${types}',
> ','):equals('${my_type}')})}
> with stacktrace:
> {code:java}
> // code placeholder
> 2018-12-09 19:03:25,534 ERROR [Timer-Driven Process Thread-7]
> o.a.n.p.standard.RouteOnAttribute
> RouteOnAttribute[id=954d4a07-0167-1000--93233dd3]
> RouteOnAttribute[id=954d4a07-0167-1000--93233dd3] failed to process
> session due to java.lang.ArrayIndexOutOfBoundsException: 1; Processor
> Administratively Yielded for 1 sec: java.lang.ArrayIndexOutOfBoundsException:
> 1
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.nifi.attribute.expression.language.evaluation.selection.DelineatedAttributeEvaluator.evaluate(DelineatedAttributeEvaluator.java:65)
> at
> org.apache.nifi.attribute.expression.language.evaluation.functions.EqualsEvaluator.evaluate(EqualsEvaluator.java:40)
> at
> org.apache.nifi.attribute.expression.language.evaluation.selection.AnyAttributeEvaluator.evaluate(AnyAttributeEvaluator.java:38)
> at
> org.apache.nifi.attribute.expression.language.evaluation.functions.OrEvaluator.evaluate(OrEvaluator.java:47)
> at
> org.apache.nifi.attribute.expression.language.evaluation.selection.AnyAttributeEvaluator.evaluate(AnyAttributeEvaluator.java:49)
> at
> org.apache.nifi.attribute.expression.language.Query.evaluate(Query.java:363)
> at
> org.apache.nifi.attribute.expression.language.Query.evaluateExpression(Query.java:204)
> at
> org.apache.nifi.attribute.expression.language.CompiledExpression.evaluate(CompiledExpression.java:58)
> at
> org.apache.nifi.attribute.expression.language.StandardPreparedQuery.evaluateExpressions(StandardPreparedQuery.java:53)
> at
> org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:160)
> at
> org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:148)
> at
> org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:133)
> at
> org.apache.nifi.processors.standard.RouteOnAttribute.onTrigger(RouteOnAttribute.java:219)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5885) EL expression of anyDelineatedValue() and or() fails with stacktrace
[
https://issues.apache.org/jira/browse/NIFI-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Villard updated NIFI-5885:
-
Component/s: Extensions
> EL expression of anyDelineatedValue() and or() fails with stacktrace
> -
>
> Key: NIFI-5885
> URL: https://issues.apache.org/jira/browse/NIFI-5885
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Yan Chen
>Assignee: Koji Kawamura
>Priority: Major
> Fix For: 1.9.0
>
> Attachments: anyDelineatedValue_or_test.xml,
> image-2018-12-10-11-37-55-545.png
>
>
> See attached Template test-case. Following EL expression failed:
> ${anyDelineatedValue('${names}',
> ','):equals('${my_name}'):or(${anyDelineatedValue('${types}',
> ','):equals('${my_type}')})}
> with stacktrace:
> {code:java}
> // code placeholder
> 2018-12-09 19:03:25,534 ERROR [Timer-Driven Process Thread-7]
> o.a.n.p.standard.RouteOnAttribute
> RouteOnAttribute[id=954d4a07-0167-1000--93233dd3]
> RouteOnAttribute[id=954d4a07-0167-1000--93233dd3] failed to process
> session due to java.lang.ArrayIndexOutOfBoundsException: 1; Processor
> Administratively Yielded for 1 sec: java.lang.ArrayIndexOutOfBoundsException:
> 1
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.nifi.attribute.expression.language.evaluation.selection.DelineatedAttributeEvaluator.evaluate(DelineatedAttributeEvaluator.java:65)
> at
> org.apache.nifi.attribute.expression.language.evaluation.functions.EqualsEvaluator.evaluate(EqualsEvaluator.java:40)
> at
> org.apache.nifi.attribute.expression.language.evaluation.selection.AnyAttributeEvaluator.evaluate(AnyAttributeEvaluator.java:38)
> at
> org.apache.nifi.attribute.expression.language.evaluation.functions.OrEvaluator.evaluate(OrEvaluator.java:47)
> at
> org.apache.nifi.attribute.expression.language.evaluation.selection.AnyAttributeEvaluator.evaluate(AnyAttributeEvaluator.java:49)
> at
> org.apache.nifi.attribute.expression.language.Query.evaluate(Query.java:363)
> at
> org.apache.nifi.attribute.expression.language.Query.evaluateExpression(Query.java:204)
> at
> org.apache.nifi.attribute.expression.language.CompiledExpression.evaluate(CompiledExpression.java:58)
> at
> org.apache.nifi.attribute.expression.language.StandardPreparedQuery.evaluateExpressions(StandardPreparedQuery.java:53)
> at
> org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:160)
> at
> org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:148)
> at
> org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:133)
> at
> org.apache.nifi.processors.standard.RouteOnAttribute.onTrigger(RouteOnAttribute.java:219)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5885) EL expression of anyDelineatedValue() and or() fails with stacktrace
[
https://issues.apache.org/jira/browse/NIFI-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Villard updated NIFI-5885:
-
Resolution: Fixed
Fix Version/s: 1.9.0
Status: Resolved (was: Patch Available)
> EL expression of anyDelineatedValue() and or() fails with stacktrace
> -
>
> Key: NIFI-5885
> URL: https://issues.apache.org/jira/browse/NIFI-5885
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Yan Chen
>Assignee: Koji Kawamura
>Priority: Major
> Fix For: 1.9.0
>
> Attachments: anyDelineatedValue_or_test.xml,
> image-2018-12-10-11-37-55-545.png
>
>
> See attached Template test-case. Following EL expression failed:
> ${anyDelineatedValue('${names}',
> ','):equals('${my_name}'):or(${anyDelineatedValue('${types}',
> ','):equals('${my_type}')})}
> with stacktrace:
> {code:java}
> // code placeholder
> 2018-12-09 19:03:25,534 ERROR [Timer-Driven Process Thread-7]
> o.a.n.p.standard.RouteOnAttribute
> RouteOnAttribute[id=954d4a07-0167-1000--93233dd3]
> RouteOnAttribute[id=954d4a07-0167-1000--93233dd3] failed to process
> session due to java.lang.ArrayIndexOutOfBoundsException: 1; Processor
> Administratively Yielded for 1 sec: java.lang.ArrayIndexOutOfBoundsException:
> 1
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.nifi.attribute.expression.language.evaluation.selection.DelineatedAttributeEvaluator.evaluate(DelineatedAttributeEvaluator.java:65)
> at
> org.apache.nifi.attribute.expression.language.evaluation.functions.EqualsEvaluator.evaluate(EqualsEvaluator.java:40)
> at
> org.apache.nifi.attribute.expression.language.evaluation.selection.AnyAttributeEvaluator.evaluate(AnyAttributeEvaluator.java:38)
> at
> org.apache.nifi.attribute.expression.language.evaluation.functions.OrEvaluator.evaluate(OrEvaluator.java:47)
> at
> org.apache.nifi.attribute.expression.language.evaluation.selection.AnyAttributeEvaluator.evaluate(AnyAttributeEvaluator.java:49)
> at
> org.apache.nifi.attribute.expression.language.Query.evaluate(Query.java:363)
> at
> org.apache.nifi.attribute.expression.language.Query.evaluateExpression(Query.java:204)
> at
> org.apache.nifi.attribute.expression.language.CompiledExpression.evaluate(CompiledExpression.java:58)
> at
> org.apache.nifi.attribute.expression.language.StandardPreparedQuery.evaluateExpressions(StandardPreparedQuery.java:53)
> at
> org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:160)
> at
> org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:148)
> at
> org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:133)
> at
> org.apache.nifi.processors.standard.RouteOnAttribute.onTrigger(RouteOnAttribute.java:219)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5885) EL expression of anyDelineatedValue() and or() fails with stacktrace
[
https://issues.apache.org/jira/browse/NIFI-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714562#comment-16714562
]
ASF GitHub Bot commented on NIFI-5885:
--
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/3212
> EL expression of anyDelineatedValue() and or() fails with stacktrace
> -
>
> Key: NIFI-5885
> URL: https://issues.apache.org/jira/browse/NIFI-5885
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Yan Chen
>Assignee: Koji Kawamura
>Priority: Major
> Fix For: 1.9.0
>
> Attachments: anyDelineatedValue_or_test.xml,
> image-2018-12-10-11-37-55-545.png
>
>
> See attached Template test-case. Following EL expression failed:
> ${anyDelineatedValue('${names}',
> ','):equals('${my_name}'):or(${anyDelineatedValue('${types}',
> ','):equals('${my_type}')})}
> with stacktrace:
> {code:java}
> // code placeholder
> 2018-12-09 19:03:25,534 ERROR [Timer-Driven Process Thread-7]
> o.a.n.p.standard.RouteOnAttribute
> RouteOnAttribute[id=954d4a07-0167-1000--93233dd3]
> RouteOnAttribute[id=954d4a07-0167-1000--93233dd3] failed to process
> session due to java.lang.ArrayIndexOutOfBoundsException: 1; Processor
> Administratively Yielded for 1 sec: java.lang.ArrayIndexOutOfBoundsException:
> 1
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.nifi.attribute.expression.language.evaluation.selection.DelineatedAttributeEvaluator.evaluate(DelineatedAttributeEvaluator.java:65)
> at
> org.apache.nifi.attribute.expression.language.evaluation.functions.EqualsEvaluator.evaluate(EqualsEvaluator.java:40)
> at
> org.apache.nifi.attribute.expression.language.evaluation.selection.AnyAttributeEvaluator.evaluate(AnyAttributeEvaluator.java:38)
> at
> org.apache.nifi.attribute.expression.language.evaluation.functions.OrEvaluator.evaluate(OrEvaluator.java:47)
> at
> org.apache.nifi.attribute.expression.language.evaluation.selection.AnyAttributeEvaluator.evaluate(AnyAttributeEvaluator.java:49)
> at
> org.apache.nifi.attribute.expression.language.Query.evaluate(Query.java:363)
> at
> org.apache.nifi.attribute.expression.language.Query.evaluateExpression(Query.java:204)
> at
> org.apache.nifi.attribute.expression.language.CompiledExpression.evaluate(CompiledExpression.java:58)
> at
> org.apache.nifi.attribute.expression.language.StandardPreparedQuery.evaluateExpressions(StandardPreparedQuery.java:53)
> at
> org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:160)
> at
> org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:148)
> at
> org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:133)
> at
> org.apache.nifi.processors.standard.RouteOnAttribute.onTrigger(RouteOnAttribute.java:219)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3212: NIFI-5885 ArrayOutOfBoundsException at EL 'or' and ...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3212 ---
[jira] [Commented] (NIFI-5885) EL expression of anyDelineatedValue() and or() fails with stacktrace
[
https://issues.apache.org/jira/browse/NIFI-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714559#comment-16714559
]
ASF subversion and git services commented on NIFI-5885:
---
Commit 4e7a856f77fb8b7fbe1067ce667d8dff93dbd271 in nifi's branch
refs/heads/master from [~ijokarumawak]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=4e7a856 ]
NIFI-5885 ArrayOutOfBoundsException at EL 'or' and 'and' functions
EL 'or' and 'and' functions can be called multiple times within the same
context using the same evaluator instance.
That happens if their subject is derived from an IteratingEvaluator such as
'anyDelineatedValues'.
And if the right hand side expression for such 'or' and 'and' contains another
IteratingEvaluator,
then it can be evaluated more than the number of its candidates, ultimately an
ArrayOutOfBoundsException is thrown.
This commit makes Or/AndEvaluator caching its right hand side result to prevent
that happens.
For 'or' and 'and' functions, the right hand side expression is independant
from their subject boolean value.
It's enough evaluating right hand side once, because it returns the same result
even with different subjects.
Signed-off-by: Pierre Villard
This closes #3212.
> EL expression of anyDelineatedValue() and or() fails with stacktrace
> -
>
> Key: NIFI-5885
> URL: https://issues.apache.org/jira/browse/NIFI-5885
> Project: Apache NiFi
> Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Yan Chen
>Assignee: Koji Kawamura
>Priority: Major
> Fix For: 1.9.0
>
> Attachments: anyDelineatedValue_or_test.xml,
> image-2018-12-10-11-37-55-545.png
>
>
> See attached Template test-case. Following EL expression failed:
> ${anyDelineatedValue('${names}',
> ','):equals('${my_name}'):or(${anyDelineatedValue('${types}',
> ','):equals('${my_type}')})}
> with stacktrace:
> {code:java}
> // code placeholder
> 2018-12-09 19:03:25,534 ERROR [Timer-Driven Process Thread-7]
> o.a.n.p.standard.RouteOnAttribute
> RouteOnAttribute[id=954d4a07-0167-1000--93233dd3]
> RouteOnAttribute[id=954d4a07-0167-1000--93233dd3] failed to process
> session due to java.lang.ArrayIndexOutOfBoundsException: 1; Processor
> Administratively Yielded for 1 sec: java.lang.ArrayIndexOutOfBoundsException:
> 1
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.nifi.attribute.expression.language.evaluation.selection.DelineatedAttributeEvaluator.evaluate(DelineatedAttributeEvaluator.java:65)
> at
> org.apache.nifi.attribute.expression.language.evaluation.functions.EqualsEvaluator.evaluate(EqualsEvaluator.java:40)
> at
> org.apache.nifi.attribute.expression.language.evaluation.selection.AnyAttributeEvaluator.evaluate(AnyAttributeEvaluator.java:38)
> at
> org.apache.nifi.attribute.expression.language.evaluation.functions.OrEvaluator.evaluate(OrEvaluator.java:47)
> at
> org.apache.nifi.attribute.expression.language.evaluation.selection.AnyAttributeEvaluator.evaluate(AnyAttributeEvaluator.java:49)
> at
> org.apache.nifi.attribute.expression.language.Query.evaluate(Query.java:363)
> at
> org.apache.nifi.attribute.expression.language.Query.evaluateExpression(Query.java:204)
> at
> org.apache.nifi.attribute.expression.language.CompiledExpression.evaluate(CompiledExpression.java:58)
> at
> org.apache.nifi.attribute.expression.language.StandardPreparedQuery.evaluateExpressions(StandardPreparedQuery.java:53)
> at
> org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:160)
> at
> org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:148)
> at
> org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:133)
> at
> org.apache.nifi.processors.standard.RouteOnAttribute.onTrigger(RouteOnAttribute.java:219)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
>
[GitHub] nifi issue #3212: NIFI-5885 ArrayOutOfBoundsException at EL 'or' and 'and' f...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3212 +1, merging to master, thanks @ijokarumawak ---
[jira] [Commented] (NIFI-5816) SFTP cannot connect due to JSch limitations
[ https://issues.apache.org/jira/browse/NIFI-5816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714503#comment-16714503 ] Laurenceau Julien commented on NIFI-5816: - May be linked to NIFI-5763 > SFTP cannot connect due to JSch limitations > --- > > Key: NIFI-5816 > URL: https://issues.apache.org/jira/browse/NIFI-5816 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.8.0 >Reporter: Laurenceau Julien >Priority: Minor > > Hi, > The JSch library used for SFTP does not support HostKeyAlgorithms=ed25519 > whereas it is the current standard. This make SFTP / SSH unusable when > dealing with recent openssh config. > On dbeaver project they switched to sshj. > [https://github.com/dbeaver/dbeaver/issues/2202] > [https://community.hortonworks.com/answers/226377/view.html] > > https://stackoverflow.com/questions/2003419/com-jcraft-jsch-jschexception-unknownhostkey > One more argument against JSch is that it does not support rsa key length > other than default (2048). > ssh-keygen -o -t rsa -b 4096 -f id_rsa -> does not work with nifi > ssh-keygen -t rsa -f id_rsa -> works with nifi > Thanks and regards > JL > PS : sorry but I do not know nifi deep enough to fill all fields. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5886) Add RPM profile for NiFi Toolkit
Yoshiaki Takahashi created NIFI-5886: Summary: Add RPM profile for NiFi Toolkit Key: NIFI-5886 URL: https://issues.apache.org/jira/browse/NIFI-5886 Project: Apache NiFi Issue Type: Improvement Components: Tools and Build Reporter: Yoshiaki Takahashi RPM for NiFi Toolkit is useful for easy installation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
