[jira] [Commented] (NIFI-5847) AWS Region List Missing New Region

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16705609#comment-16705609
 ] 

ASF GitHub Bot commented on NIFI-5847:
--

Github user jmlogan commented on the issue:

https://github.com/apache/nifi/pull/3187
  
There are non public regions not in that list.

On Thu, Nov 29, 2018 at 11:34 PM Sivaprasanna 
wrote:

> @jmlogan  It is an enum on the AWS SDK side
> so whatever new regions that are added to AWS cloud services, the team
> managing the SDK project update the enum on their side.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or mute
> the thread
> 

> .
>



> AWS Region List Missing New Region
> --
>
> Key: NIFI-5847
> URL: https://issues.apache.org/jira/browse/NIFI-5847
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Jon logan
>Priority: Major
>
> A new region was added to AWS, the enum needs to be updated accordingly.
>  
> https://aws.amazon.com/blogs/aws/aws-govcloud-us-east-now-open/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #3187: NIFI-5847 Added GovCloud-East region to enum list.

[jmlogan]

Github user jmlogan commented on the issue:

https://github.com/apache/nifi/pull/3187
  
There are non public regions not in that list.

On Thu, Nov 29, 2018 at 11:34 PM Sivaprasanna 
wrote:

> @jmlogan  It is an enum on the AWS SDK side
> so whatever new regions that are added to AWS cloud services, the team
> managing the SDK project update the enum on their side.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or mute
> the thread
> 

> .
>



---

[jira] [Commented] (NIFI-5854) Enhance time unit features

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16705358#comment-16705358
 ] 

ASF GitHub Bot commented on NIFI-5854:
--

GitHub user alopresto opened a pull request:

https://github.com/apache/nifi/pull/3193

NIFI-5854 Added TimeUnit enhancements (microseconds, decimal parsing)

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alopresto/nifi NIFI-5854

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/3193.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3193


commit 1d4ecd3e4c078e67a6a04c1ffda48d635cf66fb7
Author: Andy LoPresto 
Date:   2018-11-30T04:17:58Z

NIFI-5854 Added skeleton logic to convert decimal time units.
Added helper methods.
Added unit tests.

commit 50b7b6436454b3b0d7e36fd1f2139dd1f10b9067
Author: Andy LoPresto 
Date:   2018-11-30T04:25:04Z

NIFI-5854 [WIP] Cleaned up logic.
Resolved failing unit tests due to error message change.

commit fc16cf37439e1bee5afe58adbfe61c31bc77df3f
Author: Andy LoPresto 
Date:   2018-11-30T04:48:21Z

NIFI-5854 [WIP] All helper method unit tests pass.

commit a0865667f03ad9e1a0ad46277045fa364720cd98
Author: Andy LoPresto 
Date:   2018-11-30T05:15:27Z

NIFI-5854 [WIP] FormatUtils#getPreciseTimeDuration() now handles all tested 
inputs correctly.
Added unit tests.

commit d2031d5a06713b69f1323215afe87efc8d93428f
Author: Andy LoPresto 
Date:   2018-11-30T19:39:34Z

NIFI-5854 [WIP] FormatUtils#getTimeDuration() still using long.
Added unit tests.
Renamed existing unit tests to reflect method under test.

commit ed51f48a3bee3750864b1415b5e85b42422ecaaf
Author: Andy LoPresto 
Date:   2018-11-30T20:13:57Z

NIFI-5854 FormatUtils#getTimeDuration() returns long but now accepts 
decimal inputs.
Added @Deprecation warnings (will update callers where possible).
All unit tests pass.




> Enhance time unit features
> --
>
> Key: NIFI-5854
> URL: https://issues.apache.org/jira/browse/NIFI-5854
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.8.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Major
>  Labels: parsing, time, units
>
> There is some ambiguity with time units (specifically around processor 
> properties). Two features which I think should be added:
> * Currently only whole numbers are parsed correctly. For example, {{10 
> milliseconds}} and {{0.010 seconds}} are functionally equivalent, but only 
> the former will be parsed. This is due to the regex used in 
> {{StandardValidators.TIME_PERIOD_VALIDATOR}} which relies on 
> {{FormatUtils.TIME_DURATION_REGEX}} (see below). Decimal amounts should be 
> parsed
> * The enumerated time units are *nanoseconds, milliseconds, seconds, minutes, 
> hours, days, weeks*. 

[GitHub] nifi pull request #3193: NIFI-5854 Added TimeUnit enhancements (microseconds...

[alopresto]

GitHub user alopresto opened a pull request:

https://github.com/apache/nifi/pull/3193

NIFI-5854 Added TimeUnit enhancements (microseconds, decimal parsing)

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alopresto/nifi NIFI-5854

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/3193.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3193


commit 1d4ecd3e4c078e67a6a04c1ffda48d635cf66fb7
Author: Andy LoPresto 
Date:   2018-11-30T04:17:58Z

NIFI-5854 Added skeleton logic to convert decimal time units.
Added helper methods.
Added unit tests.

commit 50b7b6436454b3b0d7e36fd1f2139dd1f10b9067
Author: Andy LoPresto 
Date:   2018-11-30T04:25:04Z

NIFI-5854 [WIP] Cleaned up logic.
Resolved failing unit tests due to error message change.

commit fc16cf37439e1bee5afe58adbfe61c31bc77df3f
Author: Andy LoPresto 
Date:   2018-11-30T04:48:21Z

NIFI-5854 [WIP] All helper method unit tests pass.

commit a0865667f03ad9e1a0ad46277045fa364720cd98
Author: Andy LoPresto 
Date:   2018-11-30T05:15:27Z

NIFI-5854 [WIP] FormatUtils#getPreciseTimeDuration() now handles all tested 
inputs correctly.
Added unit tests.

commit d2031d5a06713b69f1323215afe87efc8d93428f
Author: Andy LoPresto 
Date:   2018-11-30T19:39:34Z

NIFI-5854 [WIP] FormatUtils#getTimeDuration() still using long.
Added unit tests.
Renamed existing unit tests to reflect method under test.

commit ed51f48a3bee3750864b1415b5e85b42422ecaaf
Author: Andy LoPresto 
Date:   2018-11-30T20:13:57Z

NIFI-5854 FormatUtils#getTimeDuration() returns long but now accepts 
decimal inputs.
Added @Deprecation warnings (will update callers where possible).
All unit tests pass.




---

[jira] [Commented] (NIFI-5859) Update NAR maven plugin to include information about Extensions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16705214#comment-16705214
 ] 

ASF GitHub Bot commented on NIFI-5859:
--

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi/pull/3192#discussion_r237974350
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-standard-services-api-nar/pom.xml 
---
@@ -26,6 +26,12 @@
 true
 
 
+
+org.apache.nifi
+nifi-jetty-bundle
+1.9.0-SNAPSHOT
+nar
--- End diff --

I'm curious what caused this to be needed. At runtime we automatically make 
the Jetty bundle be the parent of any bundle that doesn't specify a dependency.


> Update NAR maven plugin to include information about Extensions
> ---
>
> Key: NIFI-5859
> URL: https://issues.apache.org/jira/browse/NIFI-5859
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Tools and Build
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
>
> In order to have the NiFi Registry host any extensions, the registry will 
> need a way to know what extensions exist in a given NAR. Currently, that 
> information is not available directly.
> The NAR maven plugin should be updated to provide a list of extensions and 
> for each one, provide at least the following minimal information:
>  * Extension Type
>  * Extension Name
>  * Capability Description
>  * Whether or not the component is Restricted, "sub-restrictions" it has, and 
> explanations of both
>  * Any Tags that the component has
>  * If the component is a Controller Service, any Controller Service API's 
> that it provides
> Additionally, it would be ideal to provide all documentation for the 
> component within the NAR. It is best, though, not to write the documentation 
> in HTML as is done now but rather in XML or some sort of form that provides 
> the information in a structured way without any styling. This would allow the 
> documentation to be rendered consistently, even if the styling changes from 1 
> version to the next.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5859) Update NAR maven plugin to include information about Extensions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16705212#comment-16705212
 ] 

ASF GitHub Bot commented on NIFI-5859:
--

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi/pull/3192#discussion_r237982934
  
--- Diff: 
nifi-api/src/main/java/org/apache/nifi/documentation/AbstractDocumentationWriter.java
 ---
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.documentation;
+
+import org.apache.nifi.annotation.behavior.DynamicProperties;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.DynamicRelationship;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.behavior.Restricted;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.SystemResourceConsideration;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.DeprecationNotice;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.ConfigurableComponent;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.ControllerService;
+import 
org.apache.nifi.documentation.init.DocumentationControllerServiceInitializationContext;
+import 
org.apache.nifi.documentation.init.DocumentationProcessorInitializationContext;
+import 
org.apache.nifi.documentation.init.DocumentationReportingInitializationContext;
+import org.apache.nifi.processor.Processor;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.reporting.ReportingTask;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.Reader;
+import java.io.StringWriter;
+import java.io.Writer;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * Base class for DocumentationWriter that simplifies iterating over all 
information for a component, creating a separate method
+ * for each, to ensure that implementations properly override all methods 
and therefore properly account for all information about
+ * a component.
+ *
+ * Please note that while this class lives within the nifi-api, it is 
provided primarily as a means for documentation components within
+ * the NiFi NAR Maven Plugin. Its home is the nifi-api, however, because 
the API is needed in order to extract the relevant information and
+ * the NAR Maven Plugin cannot have a direct dependency on nifi-api (doing 
so would cause a circular dependency). By having this homed within
+ * the nifi-api, the Maven plugin is able to discover the class 
dynamically and invoke the one or two methods necessary to create the 
documentation.
+ *
+ * This is a new capability in 1.9.0 in preparation for the Extension 
Registry and therefore, you should
+ * NOTE WELL: At this time, while this class is part of nifi-api, 
it is still evolving and may change in a non-backward-compatible manner or even 
be
+ * removed from one incremental release to the next. Use at your own risk!
+ */
+public abstract class AbstractDocumentationWriter implements 
DocumentationWriter {
+
+@Override
+public final void write(final ConfigurableComponent component) throws 

[jira] [Commented] (NIFI-5859) Update NAR maven plugin to include information about Extensions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16705213#comment-16705213
 ] 

ASF GitHub Bot commented on NIFI-5859:
--

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi/pull/3192#discussion_r237917798
  
--- Diff: 
nifi-api/src/main/java/org/apache/nifi/documentation/AbstractDocumentationWriter.java
 ---
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.documentation;
+
+import org.apache.nifi.annotation.behavior.DynamicProperties;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.DynamicRelationship;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.behavior.Restricted;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.SystemResourceConsideration;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.DeprecationNotice;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.ConfigurableComponent;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.ControllerService;
+import 
org.apache.nifi.documentation.init.DocumentationControllerServiceInitializationContext;
+import 
org.apache.nifi.documentation.init.DocumentationProcessorInitializationContext;
+import 
org.apache.nifi.documentation.init.DocumentationReportingInitializationContext;
+import org.apache.nifi.processor.Processor;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.reporting.ReportingTask;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.Reader;
+import java.io.StringWriter;
+import java.io.Writer;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * Base class for DocumentationWriter that simplifies iterating over all 
information for a component, creating a separate method
+ * for each, to ensure that implementations properly override all methods 
and therefore properly account for all information about
+ * a component.
+ *
+ * Please note that while this class lives within the nifi-api, it is 
provided primarily as a means for documentation components within
+ * the NiFi NAR Maven Plugin. Its home is the nifi-api, however, because 
the API is needed in order to extract the relevant information and
+ * the NAR Maven Plugin cannot have a direct dependency on nifi-api (doing 
so would cause a circular dependency). By having this homed within
+ * the nifi-api, the Maven plugin is able to discover the class 
dynamically and invoke the one or two methods necessary to create the 
documentation.
+ *
+ * This is a new capability in 1.9.0 in preparation for the Extension 
Registry and therefore, you should
+ * NOTE WELL: At this time, while this class is part of nifi-api, 
it is still evolving and may change in a non-backward-compatible manner or even 
be
+ * removed from one incremental release to the next. Use at your own risk!
+ */
+public abstract class AbstractDocumentationWriter implements 
DocumentationWriter {
+
+@Override
+public final void write(final ConfigurableComponent component) throws 

[GitHub] nifi pull request #3192: NIFI-5859: Added XML-based documentation writer tha...

[bbende]

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi/pull/3192#discussion_r237974350
  
--- Diff: 
nifi-nar-bundles/nifi-standard-services/nifi-standard-services-api-nar/pom.xml 
---
@@ -26,6 +26,12 @@
 true
 
 
+
+org.apache.nifi
+nifi-jetty-bundle
+1.9.0-SNAPSHOT
+nar
--- End diff --

I'm curious what caused this to be needed. At runtime we automatically make 
the Jetty bundle be the parent of any bundle that doesn't specify a dependency.


---

[GitHub] nifi pull request #3192: NIFI-5859: Added XML-based documentation writer tha...

[bbende]

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi/pull/3192#discussion_r237982934
  
--- Diff: 
nifi-api/src/main/java/org/apache/nifi/documentation/AbstractDocumentationWriter.java
 ---
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.documentation;
+
+import org.apache.nifi.annotation.behavior.DynamicProperties;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.DynamicRelationship;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.behavior.Restricted;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.SystemResourceConsideration;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.DeprecationNotice;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.ConfigurableComponent;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.ControllerService;
+import 
org.apache.nifi.documentation.init.DocumentationControllerServiceInitializationContext;
+import 
org.apache.nifi.documentation.init.DocumentationProcessorInitializationContext;
+import 
org.apache.nifi.documentation.init.DocumentationReportingInitializationContext;
+import org.apache.nifi.processor.Processor;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.reporting.ReportingTask;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.Reader;
+import java.io.StringWriter;
+import java.io.Writer;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * Base class for DocumentationWriter that simplifies iterating over all 
information for a component, creating a separate method
+ * for each, to ensure that implementations properly override all methods 
and therefore properly account for all information about
+ * a component.
+ *
+ * Please note that while this class lives within the nifi-api, it is 
provided primarily as a means for documentation components within
+ * the NiFi NAR Maven Plugin. Its home is the nifi-api, however, because 
the API is needed in order to extract the relevant information and
+ * the NAR Maven Plugin cannot have a direct dependency on nifi-api (doing 
so would cause a circular dependency). By having this homed within
+ * the nifi-api, the Maven plugin is able to discover the class 
dynamically and invoke the one or two methods necessary to create the 
documentation.
+ *
+ * This is a new capability in 1.9.0 in preparation for the Extension 
Registry and therefore, you should
+ * NOTE WELL: At this time, while this class is part of nifi-api, 
it is still evolving and may change in a non-backward-compatible manner or even 
be
+ * removed from one incremental release to the next. Use at your own risk!
+ */
+public abstract class AbstractDocumentationWriter implements 
DocumentationWriter {
+
+@Override
+public final void write(final ConfigurableComponent component) throws 
IOException {
+write(component, null);
+}
+
+@Override
+public final void write(final ConfigurableComponent component, final 
Collection providedServices) throws IOException {
+

[GitHub] nifi pull request #3192: NIFI-5859: Added XML-based documentation writer tha...

[bbende]

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi/pull/3192#discussion_r237917798
  
--- Diff: 
nifi-api/src/main/java/org/apache/nifi/documentation/AbstractDocumentationWriter.java
 ---
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.documentation;
+
+import org.apache.nifi.annotation.behavior.DynamicProperties;
+import org.apache.nifi.annotation.behavior.DynamicProperty;
+import org.apache.nifi.annotation.behavior.DynamicRelationship;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.ReadsAttribute;
+import org.apache.nifi.annotation.behavior.ReadsAttributes;
+import org.apache.nifi.annotation.behavior.Restricted;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.SystemResourceConsideration;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.DeprecationNotice;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.ConfigurableComponent;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.ControllerService;
+import 
org.apache.nifi.documentation.init.DocumentationControllerServiceInitializationContext;
+import 
org.apache.nifi.documentation.init.DocumentationProcessorInitializationContext;
+import 
org.apache.nifi.documentation.init.DocumentationReportingInitializationContext;
+import org.apache.nifi.processor.Processor;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.reporting.ReportingTask;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.Reader;
+import java.io.StringWriter;
+import java.io.Writer;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+
+/**
+ * Base class for DocumentationWriter that simplifies iterating over all 
information for a component, creating a separate method
+ * for each, to ensure that implementations properly override all methods 
and therefore properly account for all information about
+ * a component.
+ *
+ * Please note that while this class lives within the nifi-api, it is 
provided primarily as a means for documentation components within
+ * the NiFi NAR Maven Plugin. Its home is the nifi-api, however, because 
the API is needed in order to extract the relevant information and
+ * the NAR Maven Plugin cannot have a direct dependency on nifi-api (doing 
so would cause a circular dependency). By having this homed within
+ * the nifi-api, the Maven plugin is able to discover the class 
dynamically and invoke the one or two methods necessary to create the 
documentation.
+ *
+ * This is a new capability in 1.9.0 in preparation for the Extension 
Registry and therefore, you should
+ * NOTE WELL: At this time, while this class is part of nifi-api, 
it is still evolving and may change in a non-backward-compatible manner or even 
be
+ * removed from one incremental release to the next. Use at your own risk!
+ */
+public abstract class AbstractDocumentationWriter implements 
DocumentationWriter {
+
+@Override
+public final void write(final ConfigurableComponent component) throws 
IOException {
+write(component, null);
+}
+
+@Override
+public final void write(final ConfigurableComponent component, final 
Collection providedServices) throws IOException {
+

[jira] [Commented] (NIFI-5642) QueryCassandra processor : output FlowFiles as soon fetch_size is reached

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16705170#comment-16705170
 ] 

ASF GitHub Bot commented on NIFI-5642:
--

Github user aglotero commented on a diff in the pull request:

https://github.com/apache/nifi/pull/3051#discussion_r237965549
  
--- Diff: 
nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-processors/src/main/java/org/apache/nifi/processors/cassandra/QueryCassandra.java
 ---
@@ -191,76 +203,110 @@ public void onScheduled(final ProcessContext 
context) {
 
 @Override
 public void onTrigger(final ProcessContext context, final 
ProcessSession session) throws ProcessException {
+FlowFile inputFlowFile = null;
 FlowFile fileToProcess = null;
+
+Map attributes = null;
+
 if (context.hasIncomingConnection()) {
-fileToProcess = session.get();
+inputFlowFile = session.get();
 
 // If we have no FlowFile, and all incoming connections are 
self-loops then we can continue on.
 // However, if we have no FlowFile and we have connections 
coming from other Processors, then
 // we know that we should run only if we have a FlowFile.
-if (fileToProcess == null && context.hasNonLoopConnection()) {
+if (inputFlowFile == null && context.hasNonLoopConnection()) {
 return;
 }
+
+attributes = inputFlowFile.getAttributes();
 }
 
 final ComponentLog logger = getLogger();
-final String selectQuery = 
context.getProperty(CQL_SELECT_QUERY).evaluateAttributeExpressions(fileToProcess).getValue();
-final long queryTimeout = 
context.getProperty(QUERY_TIMEOUT).evaluateAttributeExpressions(fileToProcess).asTimePeriod(TimeUnit.MILLISECONDS);
+final String selectQuery = 
context.getProperty(CQL_SELECT_QUERY).evaluateAttributeExpressions(inputFlowFile).getValue();
+final long queryTimeout = 
context.getProperty(QUERY_TIMEOUT).evaluateAttributeExpressions(inputFlowFile).asTimePeriod(TimeUnit.MILLISECONDS);
 final String outputFormat = 
context.getProperty(OUTPUT_FORMAT).getValue();
-final Charset charset = 
Charset.forName(context.getProperty(CHARSET).evaluateAttributeExpressions(fileToProcess).getValue());
+final long maxRowsPerFlowFile = 
context.getProperty(MAX_ROWS_PER_FLOW_FILE).evaluateAttributeExpressions().asInteger();
+final Charset charset = 
Charset.forName(context.getProperty(CHARSET).evaluateAttributeExpressions(inputFlowFile).getValue());
 final StopWatch stopWatch = new StopWatch(true);
 
-if (fileToProcess == null) {
-fileToProcess = session.create();
+if(inputFlowFile != null){
+session.transfer(inputFlowFile, REL_ORIGINAL);
 }
 
 try {
 // The documentation for the driver recommends the session 
remain open the entire time the processor is running
 // and states that it is thread-safe. This is why 
connectionSession is not in a try-with-resources.
 final Session connectionSession = cassandraSession.get();
-final ResultSetFuture queryFuture = 
connectionSession.executeAsync(selectQuery);
+final ResultSet resultSet;
+
+if (queryTimeout > 0) {
+resultSet = connectionSession.execute(selectQuery, 
queryTimeout, TimeUnit.MILLISECONDS);
+}else{
+resultSet = connectionSession.execute(selectQuery);
+}
+
 final AtomicLong nrOfRows = new AtomicLong(0L);
 
-fileToProcess = session.write(fileToProcess, new 
OutputStreamCallback() {
-@Override
-public void process(final OutputStream out) throws 
IOException {
-try {
-logger.debug("Executing CQL query {}", new 
Object[]{selectQuery});
-final ResultSet resultSet;
-if (queryTimeout > 0) {
-resultSet = 
queryFuture.getUninterruptibly(queryTimeout, TimeUnit.MILLISECONDS);
-if (AVRO_FORMAT.equals(outputFormat)) {
-
nrOfRows.set(convertToAvroStream(resultSet, out, queryTimeout, 
TimeUnit.MILLISECONDS));
-} else if (JSON_FORMAT.equals(outputFormat)) {
-
nrOfRows.set(convertToJsonStream(resultSet, out, charset, queryTimeout, 
TimeUnit.MILLISECONDS));
-}
-} else {
-resultSet = 

[jira] [Commented] (NIFIREG-211) Add extension bundles as a type of versioned item

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16705171#comment-16705171
 ] 

ASF GitHub Bot commented on NIFIREG-211:


Github user asfgit closed the pull request at:

https://github.com/apache/nifi-registry/pull/148


> Add extension bundles as a type of versioned item
> -
>
> Key: NIFIREG-211
> URL: https://issues.apache.org/jira/browse/NIFIREG-211
> Project: NiFi Registry
>  Issue Type: Improvement
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Major
> Fix For: 0.4.0
>
>
> This ticket is to capture the work for adding extension bundles to NiFi 
> Registry.
> This work may require several follow on tickets, but at a high-level will 
> include some of the following:
> - Add a new type of item called an extension bundle, where each bundle
>  can contain one ore extensions or APIs
>  
>  - Support bundles for traditional NiFi (aka NARs) and also bundles for
>  MiNiFi CPP
>  
>  - Ability to upload the binary artifact for a bundle and extract the
>  metadata about the bundle, and metadata about the extensions contained
>  in the bundle (more on this later)
>  
>  - Provide a pluggable storage provider for saving the content of each
>  extension bundle so that we can have different implementations like
>  local fileysystem, S3, and other object stores
>  
>  - Provide a REST API for listing and retrieving available bundles,
>  integrate this into the registry Java client and NiFi CLI
> - Security considerations such as checksums and cryptographic signatures for 
> bundles



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi-registry pull request #148: NIFIREG-211 Initial work for adding extenio...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-registry/pull/148


---

[GitHub] nifi pull request #3051: NIFI-5642: QueryCassandra processor : output FlowFi...

[aglotero]

Github user aglotero commented on a diff in the pull request:

https://github.com/apache/nifi/pull/3051#discussion_r237965549
  
--- Diff: 
nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-processors/src/main/java/org/apache/nifi/processors/cassandra/QueryCassandra.java
 ---
@@ -191,76 +203,110 @@ public void onScheduled(final ProcessContext 
context) {
 
 @Override
 public void onTrigger(final ProcessContext context, final 
ProcessSession session) throws ProcessException {
+FlowFile inputFlowFile = null;
 FlowFile fileToProcess = null;
+
+Map attributes = null;
+
 if (context.hasIncomingConnection()) {
-fileToProcess = session.get();
+inputFlowFile = session.get();
 
 // If we have no FlowFile, and all incoming connections are 
self-loops then we can continue on.
 // However, if we have no FlowFile and we have connections 
coming from other Processors, then
 // we know that we should run only if we have a FlowFile.
-if (fileToProcess == null && context.hasNonLoopConnection()) {
+if (inputFlowFile == null && context.hasNonLoopConnection()) {
 return;
 }
+
+attributes = inputFlowFile.getAttributes();
 }
 
 final ComponentLog logger = getLogger();
-final String selectQuery = 
context.getProperty(CQL_SELECT_QUERY).evaluateAttributeExpressions(fileToProcess).getValue();
-final long queryTimeout = 
context.getProperty(QUERY_TIMEOUT).evaluateAttributeExpressions(fileToProcess).asTimePeriod(TimeUnit.MILLISECONDS);
+final String selectQuery = 
context.getProperty(CQL_SELECT_QUERY).evaluateAttributeExpressions(inputFlowFile).getValue();
+final long queryTimeout = 
context.getProperty(QUERY_TIMEOUT).evaluateAttributeExpressions(inputFlowFile).asTimePeriod(TimeUnit.MILLISECONDS);
 final String outputFormat = 
context.getProperty(OUTPUT_FORMAT).getValue();
-final Charset charset = 
Charset.forName(context.getProperty(CHARSET).evaluateAttributeExpressions(fileToProcess).getValue());
+final long maxRowsPerFlowFile = 
context.getProperty(MAX_ROWS_PER_FLOW_FILE).evaluateAttributeExpressions().asInteger();
+final Charset charset = 
Charset.forName(context.getProperty(CHARSET).evaluateAttributeExpressions(inputFlowFile).getValue());
 final StopWatch stopWatch = new StopWatch(true);
 
-if (fileToProcess == null) {
-fileToProcess = session.create();
+if(inputFlowFile != null){
+session.transfer(inputFlowFile, REL_ORIGINAL);
 }
 
 try {
 // The documentation for the driver recommends the session 
remain open the entire time the processor is running
 // and states that it is thread-safe. This is why 
connectionSession is not in a try-with-resources.
 final Session connectionSession = cassandraSession.get();
-final ResultSetFuture queryFuture = 
connectionSession.executeAsync(selectQuery);
+final ResultSet resultSet;
+
+if (queryTimeout > 0) {
+resultSet = connectionSession.execute(selectQuery, 
queryTimeout, TimeUnit.MILLISECONDS);
+}else{
+resultSet = connectionSession.execute(selectQuery);
+}
+
 final AtomicLong nrOfRows = new AtomicLong(0L);
 
-fileToProcess = session.write(fileToProcess, new 
OutputStreamCallback() {
-@Override
-public void process(final OutputStream out) throws 
IOException {
-try {
-logger.debug("Executing CQL query {}", new 
Object[]{selectQuery});
-final ResultSet resultSet;
-if (queryTimeout > 0) {
-resultSet = 
queryFuture.getUninterruptibly(queryTimeout, TimeUnit.MILLISECONDS);
-if (AVRO_FORMAT.equals(outputFormat)) {
-
nrOfRows.set(convertToAvroStream(resultSet, out, queryTimeout, 
TimeUnit.MILLISECONDS));
-} else if (JSON_FORMAT.equals(outputFormat)) {
-
nrOfRows.set(convertToJsonStream(resultSet, out, charset, queryTimeout, 
TimeUnit.MILLISECONDS));
-}
-} else {
-resultSet = queryFuture.getUninterruptibly();
-if (AVRO_FORMAT.equals(outputFormat)) {
-
nrOfRows.set(convertToAvroStream(resultSet, out, 0, null));
-

[jira] [Commented] (NIFI-5859) Update NAR maven plugin to include information about Extensions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704907#comment-16704907
 ] 

ASF GitHub Bot commented on NIFI-5859:
--

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/3192
  
Will review...


> Update NAR maven plugin to include information about Extensions
> ---
>
> Key: NIFI-5859
> URL: https://issues.apache.org/jira/browse/NIFI-5859
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Tools and Build
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
>
> In order to have the NiFi Registry host any extensions, the registry will 
> need a way to know what extensions exist in a given NAR. Currently, that 
> information is not available directly.
> The NAR maven plugin should be updated to provide a list of extensions and 
> for each one, provide at least the following minimal information:
>  * Extension Type
>  * Extension Name
>  * Capability Description
>  * Whether or not the component is Restricted, "sub-restrictions" it has, and 
> explanations of both
>  * Any Tags that the component has
>  * If the component is a Controller Service, any Controller Service API's 
> that it provides
> Additionally, it would be ideal to provide all documentation for the 
> component within the NAR. It is best, though, not to write the documentation 
> in HTML as is done now but rather in XML or some sort of form that provides 
> the information in a structured way without any styling. This would allow the 
> documentation to be rendered consistently, even if the styling changes from 1 
> version to the next.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #3192: NIFI-5859: Added XML-based documentation writer that can b...

[bbende]

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/3192
  
Will review...


---

[jira] [Commented] (NIFI-5859) Update NAR maven plugin to include information about Extensions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704855#comment-16704855
 ] 

ASF GitHub Bot commented on NIFI-5859:
--

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/3192

NIFI-5859: Added XML-based documentation writer that can be used to d…

…ocument a component.

- Found several instances of nifi-framework-api's ProviderException being 
thrown from processors. Changed those to IllegalStateException, as 
ProviderException is not an appropriate Exception in those cases, and 
extensions should not depend on nifi-framework-api.
- Performed some cleanup, moving Property Descriptors from Controller 
Service API's/specs into the implementations. Adding to the Service API results 
in bringing in nifi-utils to the nifi-standard-services-api-nar, which is a bad 
practice. The 'main' service api nar should not have a dependency on a util 
class.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-5859

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/3192.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3192


commit e07d9be3b07ade875c87d6463487807ca2bf5246
Author: Mark Payne 
Date:   2018-11-15T17:07:08Z

NIFI-5859: Added XML-based documentation writer that can be used to 
document a component.
- Found several instances of nifi-framework-api's ProviderException being 
thrown from processors. Changed those to IllegalStateException, as 
ProviderException is not an appropriate Exception in those cases, and 
extensions should not depend on nifi-framework-api.
- Performed some cleanup, moving Property Descriptors from Controller 
Service API's/specs into the implementations. Adding to the Service API results 
in bringing in nifi-utils to the nifi-standard-services-api-nar, which is a bad 
practice. The 'main' service api nar should not have a dependency on a util 
class.




> Update NAR maven plugin to include information about Extensions
> ---
>
> Key: NIFI-5859
> URL: https://issues.apache.org/jira/browse/NIFI-5859
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Tools and Build
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
>
> In order to have the NiFi Registry host any extensions, the registry will 
> need a way to know what extensions exist in a given NAR. Currently, that 
> information is not available directly.
> The NAR maven plugin should be updated to provide a list of extensions and 
> for each one, provide at least the following minimal information:
>  * Extension Type
>  * Extension Name
>  * Capability Description
>  * Whether or not the component is Restricted, "sub-restrictions" it has, and 
> explanations of both
>  * Any Tags that the 

[jira] [Commented] (NIFI-5859) Update NAR maven plugin to include information about Extensions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704843#comment-16704843
 ] 

ASF GitHub Bot commented on NIFI-5859:
--

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi-maven/pull/7

NIFI-5859: Build NAR Extension Definitions/docs at build time



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi-maven NIFI-5859

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-maven/pull/7.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #7


commit 7cecaa9a841af8c5729fd5b135db400318245bf3
Author: Mark Payne 
Date:   2018-11-14T20:09:25Z

NIFI-5859: Build NAR Extension Definitions/docs at build time




> Update NAR maven plugin to include information about Extensions
> ---
>
> Key: NIFI-5859
> URL: https://issues.apache.org/jira/browse/NIFI-5859
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Tools and Build
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
>
> In order to have the NiFi Registry host any extensions, the registry will 
> need a way to know what extensions exist in a given NAR. Currently, that 
> information is not available directly.
> The NAR maven plugin should be updated to provide a list of extensions and 
> for each one, provide at least the following minimal information:
>  * Extension Type
>  * Extension Name
>  * Capability Description
>  * Whether or not the component is Restricted, "sub-restrictions" it has, and 
> explanations of both
>  * Any Tags that the component has
>  * If the component is a Controller Service, any Controller Service API's 
> that it provides
> Additionally, it would be ideal to provide all documentation for the 
> component within the NAR. It is best, though, not to write the documentation 
> in HTML as is done now but rather in XML or some sort of form that provides 
> the information in a structured way without any styling. This would allow the 
> documentation to be rendered consistently, even if the styling changes from 1 
> version to the next.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #3192: NIFI-5859: Added XML-based documentation writer tha...

[markap14]

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/3192

NIFI-5859: Added XML-based documentation writer that can be used to d…

…ocument a component.

- Found several instances of nifi-framework-api's ProviderException being 
thrown from processors. Changed those to IllegalStateException, as 
ProviderException is not an appropriate Exception in those cases, and 
extensions should not depend on nifi-framework-api.
- Performed some cleanup, moving Property Descriptors from Controller 
Service API's/specs into the implementations. Adding to the Service API results 
in bringing in nifi-utils to the nifi-standard-services-api-nar, which is a bad 
practice. The 'main' service api nar should not have a dependency on a util 
class.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-5859

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/3192.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3192


commit e07d9be3b07ade875c87d6463487807ca2bf5246
Author: Mark Payne 
Date:   2018-11-15T17:07:08Z

NIFI-5859: Added XML-based documentation writer that can be used to 
document a component.
- Found several instances of nifi-framework-api's ProviderException being 
thrown from processors. Changed those to IllegalStateException, as 
ProviderException is not an appropriate Exception in those cases, and 
extensions should not depend on nifi-framework-api.
- Performed some cleanup, moving Property Descriptors from Controller 
Service API's/specs into the implementations. Adding to the Service API results 
in bringing in nifi-utils to the nifi-standard-services-api-nar, which is a bad 
practice. The 'main' service api nar should not have a dependency on a util 
class.




---

[GitHub] nifi-maven pull request #7: NIFI-5859: Build NAR Extension Definitions/docs ...

[markap14]

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi-maven/pull/7

NIFI-5859: Build NAR Extension Definitions/docs at build time



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi-maven NIFI-5859

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-maven/pull/7.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #7


commit 7cecaa9a841af8c5729fd5b135db400318245bf3
Author: Mark Payne 
Date:   2018-11-14T20:09:25Z

NIFI-5859: Build NAR Extension Definitions/docs at build time




---

[jira] [Created] (NIFI-5859) Update NAR maven plugin to include information about Extensions

[Mark Payne (JIRA)]

Mark Payne created NIFI-5859:


 Summary: Update NAR maven plugin to include information about 
Extensions
 Key: NIFI-5859
 URL: https://issues.apache.org/jira/browse/NIFI-5859
 Project: Apache NiFi
  Issue Type: New Feature
  Components: Tools and Build
Reporter: Mark Payne
Assignee: Mark Payne


In order to have the NiFi Registry host any extensions, the registry will need 
a way to know what extensions exist in a given NAR. Currently, that information 
is not available directly.

The NAR maven plugin should be updated to provide a list of extensions and for 
each one, provide at least the following minimal information:
 * Extension Type
 * Extension Name
 * Capability Description
 * Whether or not the component is Restricted, "sub-restrictions" it has, and 
explanations of both
 * Any Tags that the component has
 * If the component is a Controller Service, any Controller Service API's that 
it provides

Additionally, it would be ideal to provide all documentation for the component 
within the NAR. It is best, though, not to write the documentation in HTML as 
is done now but rather in XML or some sort of form that provides the 
information in a structured way without any styling. This would allow the 
documentation to be rendered consistently, even if the styling changes from 1 
version to the next.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5851) Add MFA to *S3Object Processors

[Sivaprasanna Sethuraman (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704777#comment-16704777
 ] 

Sivaprasanna Sethuraman commented on NIFI-5851:
---

MFA support is kinda tricky business in the NiFi world. The MFA token is not 
re-usable which means the flow should be updated with the MFA token for each 
run in order to not fail.

> Add MFA to *S3Object Processors
> ---
>
> Key: NIFI-5851
> URL: https://issues.apache.org/jira/browse/NIFI-5851
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Lukas Heusser
>Priority: Major
>  Labels: features, security
>
> Add the possibility to add MFA to the *S3Object processors. Due to security 
> this is a quiet important feature for customers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5456) PutKinesisStream - Fails to work with AWS Private Link endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704754#comment-16704754
 ] 

ASF GitHub Bot commented on NIFI-5456:
--

Github user amcdonaldccri commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2968#discussion_r237865566
  
--- Diff: 
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java
 ---
@@ -286,7 +286,7 @@ protected void 
initializeRegionAndEndpoint(ProcessContext context) {
 final String urlstr = 
StringUtils.trimToEmpty(context.getProperty(ENDPOINT_OVERRIDE).evaluateAttributeExpressions().getValue());
 if (!urlstr.isEmpty()) {
 getLogger().info("Overriding endpoint with {}", new 
Object[]{urlstr});
-this.client.setEndpoint(urlstr);
+this.client.setEndpoint(urlstr, 
this.client.getServiceName(), this.region.getName());
--- End diff --

Using this.region.getName() doesn't make sense to me because the aws docs 
for 
public void setEndpoint(String endpoint, String serviceName, String 
regionId)
say
 * @param regionId
 *The ID of the region in which this service resides AND the
 *overriding region for signing purposes.
so the regionId is the same as the overriding region but using this.region 
is using the enum.


> PutKinesisStream - Fails to work with AWS Private Link endpoint
> ---
>
> Key: NIFI-5456
> URL: https://issues.apache.org/jira/browse/NIFI-5456
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.6.0, 1.7.1
> Environment: RedHat 6
>Reporter: Ariel Godinez
>Assignee: Sivaprasanna Sethuraman
>Priority: Major
>  Labels: easyfix
> Fix For: 1.8.0
>
> Attachments: 
> 0001-NIFI-5456-AWS-clients-now-work-with-private-link-end.patch
>
>
> NiFi version: 1.6.0
> PutKinesisStream fails to put due to invalid signing information when using 
> an AWS Private Link as the endpoint override URL. The endpoint override URL 
> pattern for private links is like below along with the error that NiFi 
> outputs when we attempt to use this type of URL as the 'Endpoint Override 
> URL' property value.
> Endpoint Override URL: 
> [https://vpce-|https://vpce-/].kinesis.us-east-2.vpce.amazonaws.com
> ERROR [Timer-Driven Process Thread-11] "o.a.n.p.a.k.stream.PutKinesisStream" 
> PutKinesisStream[id=4c314e25-0164-1000--9bd79c77] Failed to publish 
> due to exception com.amazonaws.services.kinesis.model.AmazonKinesisException: 
> Credential should be scoped to a valid region, not 'vpce'.  (Service: 
> AmazonKinesis; Status Code: 400; Error Code: InvalidSignatureException; 
> Request ID: 6330b83c-a64e-4acf-b892-a505621cf78e) flowfiles 
> [StandardFlowFileRecord[uuid=ba299cec-7cbf-4750-a766-c348b5cd9c73,claim=StandardContentClaim
>  [resourceClaim=StandardResourceClaim[id=1532469012962-1, 
> container=content002, section=1], offset=2159750, 
> length=534625],offset=0,name=900966573101260,size=534625]]
>  
> It looks like 'vpce' is being extracted from the url as the region name when 
> it should be getting 'us-east-2'. We were able to get this processor to work 
> correctly by explicitly passing in the region and service using 
> 'setEndpoint(String endpoint, String serviceName, String regionId)' instead 
> of 'setEndpoint(String endpoint)' in 
> 'nifi/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java'
>  line 289



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (NIFI-5858) Existing unpacked custom processors fail to deploy after upgrade from 1.7.0 to 1.8.0

[Frederik Petersen (JIRA)]

Frederik Petersen created NIFI-5858:
---

 Summary: Existing unpacked custom processors fail to deploy after 
upgrade from 1.7.0 to 1.8.0
 Key: NIFI-5858
 URL: https://issues.apache.org/jira/browse/NIFI-5858
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core Framework
Affects Versions: 1.8.0
 Environment: HDF 3.2.1 -> HDF 3.3.0
Reporter: Frederik Petersen


Hi,

after upgrading our HDF cluster from 3.2.0.0 to 3.3.0.0 our custom processors 
didn't work anymore and were replaced by "ghost" processors. We had to roll 
back and it was kind of hard to find out what went wrong.

I think I figured it out now and I'm confused that there are no measures taken 
or at least documented to avoid this issue. Maybe I didn't find them though, so 
if that's the case please point them out to me, thanks.

So the root of the problem seems to be this change in nifi: 
[https://github.com/apache/nifi/commit/a27ccd8a56fb52a61d79f87711324142a5b4a141]

The corresponding Jira ticket is: https://jira.apache.org/jira/browse/NIFI-5479

What I think happened:
We have our custom processors in a custom library folder configured with 
"nifi.nar.library.directory.custom=/custom/folder". The last release of our 
processors happened during 1.7.0 so the NarUnpacker unpacked them into nifis 
work directory.

Before upgrading to nifi 1.8.0 we went through the changelog and checked 
whether there were any api incompatibilities, but it didn't seem like it so we 
thought we would be fine with our current nar.

During the upgrade we saw this log line:

2018-11-30 11:49:52,920 WARN [main] org.apache.nifi.nar.NarClassLoader 
/var/lib/nifi/work/nar/extensions/custom-nar-1.14.2.nar-unpacked does not 
contain NAR-INF/bundled-dependencies!

So I guess, the new NarUnpacker didn't re-unpack the .nar file (and thus there 
was no renaming from META-INF to NAR-INF. Thus the NarClassLoader cannot 
resolve everything properly.

So a new .nar release/deleting the work directory at this point should allow us 
to upgrade. But maybe it would be a good idea to make sure this is documented 
somewhere to save others from some of the frustration and investigation. Maybe 
it can even be handled (at least for HDF upgrades, but I know that this Jira is 
the wrong place for that) automatically.

Let me know if you have any questions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2968: NIFI-5456: AWS clients now work with private link e...

[amcdonaldccri]

Github user amcdonaldccri commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2968#discussion_r237865566
  
--- Diff: 
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java
 ---
@@ -286,7 +286,7 @@ protected void 
initializeRegionAndEndpoint(ProcessContext context) {
 final String urlstr = 
StringUtils.trimToEmpty(context.getProperty(ENDPOINT_OVERRIDE).evaluateAttributeExpressions().getValue());
 if (!urlstr.isEmpty()) {
 getLogger().info("Overriding endpoint with {}", new 
Object[]{urlstr});
-this.client.setEndpoint(urlstr);
+this.client.setEndpoint(urlstr, 
this.client.getServiceName(), this.region.getName());
--- End diff --

Using this.region.getName() doesn't make sense to me because the aws docs 
for 
public void setEndpoint(String endpoint, String serviceName, String 
regionId)
say
 * @param regionId
 *The ID of the region in which this service resides AND the
 *overriding region for signing purposes.
so the regionId is the same as the overriding region but using this.region 
is using the enum.


---

[jira] [Commented] (NIFI-5855) Optimize GenerateTableFetch to remove unnecessary ORDER BY clause

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704504#comment-16704504
 ] 

ASF subversion and git services commented on NIFI-5855:
---

Commit 8a12307d1a5269954cc64b6bab73590b15a67444 in nifi's branch 
refs/heads/master from [~ca9mbu]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=8a12307 ]

NIFI-5855: Remove unnecessary ORDER BY clause in GenerateTableFetch when 
Partition Size is zero

Signed-off-by: Pierre Villard 

This closes #3191.


> Optimize GenerateTableFetch to remove unnecessary ORDER BY clause
> -
>
> Key: NIFI-5855
> URL: https://issues.apache.org/jira/browse/NIFI-5855
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> Related to NIFI-5727, which for the query that GenerateTableFetch uses to get 
> column/table information removes an unnecessary COUNT(*) clause when 
> Partition Size = 0, the ORDER BY clause can also be removed. This is because 
> there will be a single generated query that will fetch all rows that match 
> the filters. The ORDER BY is necessary during pagination (when Partition Size 
> > 0) so it can fetch from Row X to Row Y based on the row number, which is 
> only constant if the column is ordered.
> This Jira proposes to remove the ORDER BY clause from the generated query if 
> Partition Size is set to 0. Note that this can cause the output of the 
> executed query to not guaranteed to be ordered; it is a possible change in 
> behavior, but we don't document any guarantees about row ordering anyway. If 
> the user needs the rows ordered, they can do it with QueryRecord, or they can 
> add the ORDER BY clause manually via ReplaceText.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5850) Replace custom AWS Region Enums with AWS SDK Region enums

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704508#comment-16704508
 ] 

ASF GitHub Bot commented on NIFI-5850:
--

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/3190
  
@zenfenan agree that bumping the version just before a release is a good 
idea (and we should try to do it for multiple bundles) but I still suggest to 
bump the version in this PR so that we can also close #3187 and allow the 
community to use NiFi or the AWS NAR (built from master) with the new region.


> Replace custom AWS Region Enums with AWS SDK Region enums
> -
>
> Key: NIFI-5850
> URL: https://issues.apache.org/jira/browse/NIFI-5850
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.6.0, 1.7.0, 1.8.0, 1.7.1
>Reporter: Sivaprasanna Sethuraman
>Assignee: Sivaprasanna Sethuraman
>Priority: Major
>
> With NIFI-5129, we introduced a custom Enum for displaying readable AWS 
> regions. This readable names have now been added to the official AWS Java SDK 
> so ideally this custom AWS Regions enum have to removed and the region names 
> should be read from the AWS SDK.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #3190: NIFI-5850: Replaced custom AWS regions enum with the one f...

[pvillard31]

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/3190
  
@zenfenan agree that bumping the version just before a release is a good 
idea (and we should try to do it for multiple bundles) but I still suggest to 
bump the version in this PR so that we can also close #3187 and allow the 
community to use NiFi or the AWS NAR (built from master) with the new region.


---

[jira] [Updated] (NIFI-5855) Optimize GenerateTableFetch to remove unnecessary ORDER BY clause

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-5855:
-
Description: 
Related to NIFI-5727, which for the query that GenerateTableFetch uses to get 
column/table information removes an unnecessary {{COUNT( * )}} clause when 
Partition Size = 0, the ORDER BY clause can also be removed. This is because 
there will be a single generated query that will fetch all rows that match the 
filters. The ORDER BY is necessary during pagination (when Partition Size > 0) 
so it can fetch from Row X to Row Y based on the row number, which is only 
constant if the column is ordered.

This Jira proposes to remove the ORDER BY clause from the generated query if 
Partition Size is set to 0. Note that this can cause the output of the executed 
query to not guaranteed to be ordered; it is a possible change in behavior, but 
we don't document any guarantees about row ordering anyway. If the user needs 
the rows ordered, they can do it with QueryRecord, or they can add the ORDER BY 
clause manually via ReplaceText.

  was:
Related to NIFI-5727, which for the query that GenerateTableFetch uses to get 
column/table information removes an unnecessary {{COUNT(*)}} clause when 
Partition Size = 0, the ORDER BY clause can also be removed. This is because 
there will be a single generated query that will fetch all rows that match the 
filters. The ORDER BY is necessary during pagination (when Partition Size > 0) 
so it can fetch from Row X to Row Y based on the row number, which is only 
constant if the column is ordered.

This Jira proposes to remove the ORDER BY clause from the generated query if 
Partition Size is set to 0. Note that this can cause the output of the executed 
query to not guaranteed to be ordered; it is a possible change in behavior, but 
we don't document any guarantees about row ordering anyway. If the user needs 
the rows ordered, they can do it with QueryRecord, or they can add the ORDER BY 
clause manually via ReplaceText.


> Optimize GenerateTableFetch to remove unnecessary ORDER BY clause
> -
>
> Key: NIFI-5855
> URL: https://issues.apache.org/jira/browse/NIFI-5855
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
> Fix For: 1.9.0
>
>
> Related to NIFI-5727, which for the query that GenerateTableFetch uses to get 
> column/table information removes an unnecessary {{COUNT( * )}} clause when 
> Partition Size = 0, the ORDER BY clause can also be removed. This is because 
> there will be a single generated query that will fetch all rows that match 
> the filters. The ORDER BY is necessary during pagination (when Partition Size 
> > 0) so it can fetch from Row X to Row Y based on the row number, which is 
> only constant if the column is ordered.
> This Jira proposes to remove the ORDER BY clause from the generated query if 
> Partition Size is set to 0. Note that this can cause the output of the 
> executed query to not guaranteed to be ordered; it is a possible change in 
> behavior, but we don't document any guarantees about row ordering anyway. If 
> the user needs the rows ordered, they can do it with QueryRecord, or they can 
> add the ORDER BY clause manually via ReplaceText.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5855) Optimize GenerateTableFetch to remove unnecessary ORDER BY clause

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704505#comment-16704505
 ] 

ASF GitHub Bot commented on NIFI-5855:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/3191


> Optimize GenerateTableFetch to remove unnecessary ORDER BY clause
> -
>
> Key: NIFI-5855
> URL: https://issues.apache.org/jira/browse/NIFI-5855
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
> Fix For: 1.9.0
>
>
> Related to NIFI-5727, which for the query that GenerateTableFetch uses to get 
> column/table information removes an unnecessary COUNT(*) clause when 
> Partition Size = 0, the ORDER BY clause can also be removed. This is because 
> there will be a single generated query that will fetch all rows that match 
> the filters. The ORDER BY is necessary during pagination (when Partition Size 
> > 0) so it can fetch from Row X to Row Y based on the row number, which is 
> only constant if the column is ordered.
> This Jira proposes to remove the ORDER BY clause from the generated query if 
> Partition Size is set to 0. Note that this can cause the output of the 
> executed query to not guaranteed to be ordered; it is a possible change in 
> behavior, but we don't document any guarantees about row ordering anyway. If 
> the user needs the rows ordered, they can do it with QueryRecord, or they can 
> add the ORDER BY clause manually via ReplaceText.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-5855) Optimize GenerateTableFetch to remove unnecessary ORDER BY clause

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-5855:
-
   Resolution: Fixed
Fix Version/s: 1.9.0
   Status: Resolved  (was: Patch Available)

> Optimize GenerateTableFetch to remove unnecessary ORDER BY clause
> -
>
> Key: NIFI-5855
> URL: https://issues.apache.org/jira/browse/NIFI-5855
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
> Fix For: 1.9.0
>
>
> Related to NIFI-5727, which for the query that GenerateTableFetch uses to get 
> column/table information removes an unnecessary COUNT(*) clause when 
> Partition Size = 0, the ORDER BY clause can also be removed. This is because 
> there will be a single generated query that will fetch all rows that match 
> the filters. The ORDER BY is necessary during pagination (when Partition Size 
> > 0) so it can fetch from Row X to Row Y based on the row number, which is 
> only constant if the column is ordered.
> This Jira proposes to remove the ORDER BY clause from the generated query if 
> Partition Size is set to 0. Note that this can cause the output of the 
> executed query to not guaranteed to be ordered; it is a possible change in 
> behavior, but we don't document any guarantees about row ordering anyway. If 
> the user needs the rows ordered, they can do it with QueryRecord, or they can 
> add the ORDER BY clause manually via ReplaceText.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-5855) Optimize GenerateTableFetch to remove unnecessary ORDER BY clause

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-5855:
-
Description: 
Related to NIFI-5727, which for the query that GenerateTableFetch uses to get 
column/table information removes an unnecessary {{COUNT(*)}} clause when 
Partition Size = 0, the ORDER BY clause can also be removed. This is because 
there will be a single generated query that will fetch all rows that match the 
filters. The ORDER BY is necessary during pagination (when Partition Size > 0) 
so it can fetch from Row X to Row Y based on the row number, which is only 
constant if the column is ordered.

This Jira proposes to remove the ORDER BY clause from the generated query if 
Partition Size is set to 0. Note that this can cause the output of the executed 
query to not guaranteed to be ordered; it is a possible change in behavior, but 
we don't document any guarantees about row ordering anyway. If the user needs 
the rows ordered, they can do it with QueryRecord, or they can add the ORDER BY 
clause manually via ReplaceText.

  was:
Related to NIFI-5727, which for the query that GenerateTableFetch uses to get 
column/table information removes an unnecessary COUNT(*) clause when Partition 
Size = 0, the ORDER BY clause can also be removed. This is because there will 
be a single generated query that will fetch all rows that match the filters. 
The ORDER BY is necessary during pagination (when Partition Size > 0) so it can 
fetch from Row X to Row Y based on the row number, which is only constant if 
the column is ordered.

This Jira proposes to remove the ORDER BY clause from the generated query if 
Partition Size is set to 0. Note that this can cause the output of the executed 
query to not guaranteed to be ordered; it is a possible change in behavior, but 
we don't document any guarantees about row ordering anyway. If the user needs 
the rows ordered, they can do it with QueryRecord, or they can add the ORDER BY 
clause manually via ReplaceText.


> Optimize GenerateTableFetch to remove unnecessary ORDER BY clause
> -
>
> Key: NIFI-5855
> URL: https://issues.apache.org/jira/browse/NIFI-5855
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
> Fix For: 1.9.0
>
>
> Related to NIFI-5727, which for the query that GenerateTableFetch uses to get 
> column/table information removes an unnecessary {{COUNT(*)}} clause when 
> Partition Size = 0, the ORDER BY clause can also be removed. This is because 
> there will be a single generated query that will fetch all rows that match 
> the filters. The ORDER BY is necessary during pagination (when Partition Size 
> > 0) so it can fetch from Row X to Row Y based on the row number, which is 
> only constant if the column is ordered.
> This Jira proposes to remove the ORDER BY clause from the generated query if 
> Partition Size is set to 0. Note that this can cause the output of the 
> executed query to not guaranteed to be ordered; it is a possible change in 
> behavior, but we don't document any guarantees about row ordering anyway. If 
> the user needs the rows ordered, they can do it with QueryRecord, or they can 
> add the ORDER BY clause manually via ReplaceText.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #3191: NIFI-5855: Remove unnecessary ORDER BY clause in Ge...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/3191


---

[jira] [Commented] (NIFI-5855) Optimize GenerateTableFetch to remove unnecessary ORDER BY clause

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704502#comment-16704502
 ] 

ASF GitHub Bot commented on NIFI-5855:
--

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/3191
  
+1, merging to master, thanks @mattyb149 


> Optimize GenerateTableFetch to remove unnecessary ORDER BY clause
> -
>
> Key: NIFI-5855
> URL: https://issues.apache.org/jira/browse/NIFI-5855
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> Related to NIFI-5727, which for the query that GenerateTableFetch uses to get 
> column/table information removes an unnecessary COUNT(*) clause when 
> Partition Size = 0, the ORDER BY clause can also be removed. This is because 
> there will be a single generated query that will fetch all rows that match 
> the filters. The ORDER BY is necessary during pagination (when Partition Size 
> > 0) so it can fetch from Row X to Row Y based on the row number, which is 
> only constant if the column is ordered.
> This Jira proposes to remove the ORDER BY clause from the generated query if 
> Partition Size is set to 0. Note that this can cause the output of the 
> executed query to not guaranteed to be ordered; it is a possible change in 
> behavior, but we don't document any guarantees about row ordering anyway. If 
> the user needs the rows ordered, they can do it with QueryRecord, or they can 
> add the ORDER BY clause manually via ReplaceText.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #3191: NIFI-5855: Remove unnecessary ORDER BY clause in GenerateT...

[pvillard31]

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/3191
  
+1, merging to master, thanks @mattyb149 


---