[jira] [Created] (NIFI-5474) ReplaceText RegexReplace evaluates payload as Expression language
Joseph Percivall created NIFI-5474:
--
Summary: ReplaceText RegexReplace evaluates payload as Expression
language
Key: NIFI-5474
URL: https://issues.apache.org/jira/browse/NIFI-5474
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 1.7.1, 1.7.0
Reporter: Joseph Percivall
To reproduce, add "${this will fail}" to the ReplaceTest unit test resource
"hello.txt" and run one of the tests (like testRegexWithExpressionLanguage).
You'll end up seeing an error message like this:
{quote}java.lang.AssertionError:
org.apache.nifi.attribute.expression.language.exception.AttributeExpressionLanguageException:
Invalid Expression: ${replaceValue}, World! ${this will fail} due to
Unexpected token 'will' at line 1, column 7. Query: ${this will fail}
at
org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:201)
at
org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:160)
at
org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:155)
at
org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:150)
at
org.apache.nifi.util.StandardProcessorTestRunner.run(StandardProcessorTestRunner.java:145)
at
org.apache.nifi.processors.standard.TestReplaceText.testRegexWithExpressionLanguage(TestReplaceText.java:382)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by:
org.apache.nifi.attribute.expression.language.exception.AttributeExpressionLanguageException:
Invalid Expression: ${replaceValue}, World! ${this will fail} due to
Unexpected token 'will' at line 1, column 7. Query: ${this will fail}
at
org.apache.nifi.attribute.expression.language.InvalidPreparedQuery.evaluateExpressions(InvalidPreparedQuery.java:49)
at
org.apache.nifi.attribute.expression.language.StandardPropertyValue.evaluateAttributeExpressions(StandardPropertyValue.java:160)
at
org.apache.nifi.util.MockPropertyValue.evaluateAttributeExpressions(MockPropertyValue.java:257)
at
org.apache.nifi.util.MockPropertyValue.evaluateAttributeExpressions(MockPropertyValue.java:244)
at
org.apache.nifi.processors.standard.ReplaceText$RegexReplace.replace(ReplaceText.java:564)
at
org.apache.nifi.processors.standard.ReplaceText.onTrigger(ReplaceText.java:299)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.util.StandardProcessorTestRunner$RunProcessor.call(StandardProcessorTestRunner.java:251)
at
org.apache.nifi.util.StandardProcessorTestRunner$RunProcessor.call(StandardProcessorTestRunner.java:245)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
[jira] [Commented] (NIFI-5473) Add documentation for using intermediate CA with TLS toolkit
[ https://issues.apache.org/jira/browse/NIFI-5473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563131#comment-16563131 ] ASF GitHub Bot commented on NIFI-5473: -- Github user andrewmlim commented on the issue: https://github.com/apache/nifi/pull/2927 Will review and provide feedback. > Add documentation for using intermediate CA with TLS toolkit > > > Key: NIFI-5473 > URL: https://issues.apache.org/jira/browse/NIFI-5473 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation Website, Security, Tools and Build >Affects Versions: 1.7.1 >Reporter: Andy LoPresto >Assignee: Andy LoPresto >Priority: Major > Labels: certificate, documentation, security, tls, tls-toolkit > > With some manual work, the TLS toolkit can be used with a pre-existing > certificate and private key that has been signed by an organization's > certificate authority (CA) to sign toolkit-generated certificates. The Admin > Guide should be improved to cover the necessary steps. > When the separate "Security Guide" / "Toolkit Guide" is created, this content > should be migrated there. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2927: NIFI-5473 Added section on using external signed CA for TL...
Github user andrewmlim commented on the issue: https://github.com/apache/nifi/pull/2927 Will review and provide feedback. ---
[jira] [Updated] (NIFI-5473) Add documentation for using intermediate CA with TLS toolkit
[ https://issues.apache.org/jira/browse/NIFI-5473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy LoPresto updated NIFI-5473: Status: Patch Available (was: In Progress) > Add documentation for using intermediate CA with TLS toolkit > > > Key: NIFI-5473 > URL: https://issues.apache.org/jira/browse/NIFI-5473 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation Website, Security, Tools and Build >Affects Versions: 1.7.1 >Reporter: Andy LoPresto >Assignee: Andy LoPresto >Priority: Major > Labels: certificate, documentation, security, tls, tls-toolkit > > With some manual work, the TLS toolkit can be used with a pre-existing > certificate and private key that has been signed by an organization's > certificate authority (CA) to sign toolkit-generated certificates. The Admin > Guide should be improved to cover the necessary steps. > When the separate "Security Guide" / "Toolkit Guide" is created, this content > should be migrated there. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5473) Add documentation for using intermediate CA with TLS toolkit
[ https://issues.apache.org/jira/browse/NIFI-5473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563123#comment-16563123 ] ASF GitHub Bot commented on NIFI-5473: -- Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2927 All reviews welcome, but would especially appreciate @andrewmlim @thesolson @pepov @YolandaMDavis on this if available. Thanks. > Add documentation for using intermediate CA with TLS toolkit > > > Key: NIFI-5473 > URL: https://issues.apache.org/jira/browse/NIFI-5473 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation Website, Security, Tools and Build >Affects Versions: 1.7.1 >Reporter: Andy LoPresto >Assignee: Andy LoPresto >Priority: Major > Labels: certificate, documentation, security, tls, tls-toolkit > > With some manual work, the TLS toolkit can be used with a pre-existing > certificate and private key that has been signed by an organization's > certificate authority (CA) to sign toolkit-generated certificates. The Admin > Guide should be improved to cover the necessary steps. > When the separate "Security Guide" / "Toolkit Guide" is created, this content > should be migrated there. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2927: NIFI-5473 Added section on using external signed CA for TL...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2927 All reviews welcome, but would especially appreciate @andrewmlim @thesolson @pepov @YolandaMDavis on this if available. Thanks. ---
[GitHub] nifi issue #2927: NIFI-5473 Added section on using external signed CA for TL...
Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2927 The decision to increase the Table of Contents level to *3* can be debated as well and can be split into a separate Jira/PR if necessary. ---
[jira] [Commented] (NIFI-5473) Add documentation for using intermediate CA with TLS toolkit
[ https://issues.apache.org/jira/browse/NIFI-5473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563122#comment-16563122 ] ASF GitHub Bot commented on NIFI-5473: -- Github user alopresto commented on the issue: https://github.com/apache/nifi/pull/2927 The decision to increase the Table of Contents level to *3* can be debated as well and can be split into a separate Jira/PR if necessary. > Add documentation for using intermediate CA with TLS toolkit > > > Key: NIFI-5473 > URL: https://issues.apache.org/jira/browse/NIFI-5473 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation Website, Security, Tools and Build >Affects Versions: 1.7.1 >Reporter: Andy LoPresto >Assignee: Andy LoPresto >Priority: Major > Labels: certificate, documentation, security, tls, tls-toolkit > > With some manual work, the TLS toolkit can be used with a pre-existing > certificate and private key that has been signed by an organization's > certificate authority (CA) to sign toolkit-generated certificates. The Admin > Guide should be improved to cover the necessary steps. > When the separate "Security Guide" / "Toolkit Guide" is created, this content > should be migrated there. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5473) Add documentation for using intermediate CA with TLS toolkit
[ https://issues.apache.org/jira/browse/NIFI-5473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563121#comment-16563121 ] ASF GitHub Bot commented on NIFI-5473: -- GitHub user alopresto opened a pull request: https://github.com/apache/nifi/pull/2927 NIFI-5473 Added section on using external signed CA for TLS Toolkit. Increased TOC level to 3 for ASCIIDOC generated documentation. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/alopresto/nifi NIFI-5473 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2927.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2927 commit 00b74f3e706cb29c043d9d3e3627bbdd030c15b0 Author: Andy LoPresto Date: 2018-07-31T03:47:14Z NIFI-5473 Added section on using external signed CA for TLS Toolkit. Increased TOC level to 3 for ASCIIDOC generated documentation. > Add documentation for using intermediate CA with TLS toolkit > > > Key: NIFI-5473 > URL: https://issues.apache.org/jira/browse/NIFI-5473 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation Website, Security, Tools and Build >Affects Versions: 1.7.1 >Reporter: Andy LoPresto >Assignee: Andy LoPresto >Priority: Major > Labels: certificate, documentation, security, tls, tls-toolkit > > With some manual work, the TLS toolkit can be used with a pre-existing > certificate and private key that has been signed by an organization's > certificate authority (CA) to sign toolkit-generated certificates. The Admin > Guide should be improved to cover the necessary steps. > When the separate "Security Guide" / "Toolkit Guide" is created, this content > should be migrated there. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2927: NIFI-5473 Added section on using external signed CA...
GitHub user alopresto opened a pull request: https://github.com/apache/nifi/pull/2927 NIFI-5473 Added section on using external signed CA for TLS Toolkit. Increased TOC level to 3 for ASCIIDOC generated documentation. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/alopresto/nifi NIFI-5473 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2927.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2927 commit 00b74f3e706cb29c043d9d3e3627bbdd030c15b0 Author: Andy LoPresto Date: 2018-07-31T03:47:14Z NIFI-5473 Added section on using external signed CA for TLS Toolkit. Increased TOC level to 3 for ASCIIDOC generated documentation. ---
[jira] [Updated] (NIFI-5471) QueryDatabaseTable does not correctly support arbitrary queries on Oracle
[
https://issues.apache.org/jira/browse/NIFI-5471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-5471:
---
Status: Patch Available (was: In Progress)
> QueryDatabaseTable does not correctly support arbitrary queries on Oracle
> -
>
> Key: NIFI-5471
> URL: https://issues.apache.org/jira/browse/NIFI-5471
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> NIFI-1706 added the capability to generate the "table" it uses for
> incremental fetch from a SQL statement, rather than generating it directly
> from "Columns to Return", "Max Value Columns", etc. When a SQL query is
> supplied, it is wrapped as a nested query in a SELECT, and the result is
> aliased to the provided "Table Name" property value.
> Currently this is done by simply generating a string "SELECT (" + query + ")
> AS " + tableName. However some databases (such as Oracle) do not allow the
> "AS" keywords for aliasing things like tables. The getWrappedQuery() method
> should delegate to the specified database adapter for generating a table
> alias clause. Many adapters will return "AS " + tableName, where the Oracle
> adapter(s) will simply return the table name.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5471) QueryDatabaseTable does not correctly support arbitrary queries on Oracle
[
https://issues.apache.org/jira/browse/NIFI-5471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563083#comment-16563083
]
ASF GitHub Bot commented on NIFI-5471:
--
GitHub user mattyb149 opened a pull request:
https://github.com/apache/nifi/pull/2926
NIFI-5471: Push table aliasing to DatabaseAdapter impls
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with NIFI- where is the JIRA number
you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [x] Is your initial contribution a single, squashed commit?
### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mattyb149/nifi NIFI-5471
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/2926.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2926
commit f007889d7c7c42023a3ab876b2e59372308fc0cf
Author: Matthew Burgess
Date: 2018-07-31T03:11:40Z
NIFI-5471: Push table aliasing to DatabaseAdapter impls
> QueryDatabaseTable does not correctly support arbitrary queries on Oracle
> -
>
> Key: NIFI-5471
> URL: https://issues.apache.org/jira/browse/NIFI-5471
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> NIFI-1706 added the capability to generate the "table" it uses for
> incremental fetch from a SQL statement, rather than generating it directly
> from "Columns to Return", "Max Value Columns", etc. When a SQL query is
> supplied, it is wrapped as a nested query in a SELECT, and the result is
> aliased to the provided "Table Name" property value.
> Currently this is done by simply generating a string "SELECT (" + query + ")
> AS " + tableName. However some databases (such as Oracle) do not allow the
> "AS" keywords for aliasing things like tables. The getWrappedQuery() method
> should delegate to the specified database adapter for generating a table
> alias clause. Many adapters will return "AS " + tableName, where the Oracle
> adapter(s) will simply return the table name.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2926: NIFI-5471: Push table aliasing to DatabaseAdapter i...
GitHub user mattyb149 opened a pull request: https://github.com/apache/nifi/pull/2926 NIFI-5471: Push table aliasing to DatabaseAdapter impls Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mattyb149/nifi NIFI-5471 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2926.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2926 commit f007889d7c7c42023a3ab876b2e59372308fc0cf Author: Matthew Burgess Date: 2018-07-31T03:11:40Z NIFI-5471: Push table aliasing to DatabaseAdapter impls ---
[jira] [Assigned] (NIFI-5471) QueryDatabaseTable does not correctly support arbitrary queries on Oracle
[
https://issues.apache.org/jira/browse/NIFI-5471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess reassigned NIFI-5471:
--
Assignee: Matt Burgess
> QueryDatabaseTable does not correctly support arbitrary queries on Oracle
> -
>
> Key: NIFI-5471
> URL: https://issues.apache.org/jira/browse/NIFI-5471
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> NIFI-1706 added the capability to generate the "table" it uses for
> incremental fetch from a SQL statement, rather than generating it directly
> from "Columns to Return", "Max Value Columns", etc. When a SQL query is
> supplied, it is wrapped as a nested query in a SELECT, and the result is
> aliased to the provided "Table Name" property value.
> Currently this is done by simply generating a string "SELECT (" + query + ")
> AS " + tableName. However some databases (such as Oracle) do not allow the
> "AS" keywords for aliasing things like tables. The getWrappedQuery() method
> should delegate to the specified database adapter for generating a table
> alias clause. Many adapters will return "AS " + tableName, where the Oracle
> adapter(s) will simply return the table name.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4371) Add support for query timeout in Hive processors
[
https://issues.apache.org/jira/browse/NIFI-4371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563055#comment-16563055
]
ASF GitHub Bot commented on NIFI-4371:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2138#discussion_r206380004
--- Diff:
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/AbstractHiveQLProcessor.java
---
@@ -75,6 +81,38 @@
.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
.build();
+public static final PropertyDescriptor QUERY_TIMEOUT = new
PropertyDescriptor.Builder()
+.name("hive-query-timeout")
+.displayName("Query timeout")
+.description("Sets the number of seconds the driver will wait
for a query to execute. "
++ "A value of 0 means no timeout. NOTE: Non-zero
values may not be supported by the driver.")
+.defaultValue("0")
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.expressionLanguageSupported(true)
+.build();
+
+@Override
+protected Collection
customValidate(ValidationContext validationContext) {
+final List problems = new ArrayList<>(1);
+
+if(validationContext.getProperty(QUERY_TIMEOUT).isSet()
+&&
!validationContext.getProperty(QUERY_TIMEOUT).isExpressionLanguagePresent()
+&&
validationContext.getProperty(QUERY_TIMEOUT).asInteger() != 0) {
+try(HiveStatement stmt = new HiveStatement(null, null, null)) {
+stmt.setQueryTimeout(0);
--- End diff --
Also I think yes, the idea is to catch the "not supported" thing during
customValidate() where we can, but we also have to add it to the runtime
checking because of EL
> Add support for query timeout in Hive processors
>
>
> Key: NIFI-4371
> URL: https://issues.apache.org/jira/browse/NIFI-4371
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
> Attachments: Screen Shot 2017-09-09 at 4.31.21 PM.png, Screen Shot
> 2017-09-09 at 6.38.51 PM.png, Screen Shot 2017-09-09 at 6.40.48 PM.png
>
>
> With HIVE-4924 it is possible to set a query timeout when executing a query
> against Hive (starting with Hive 2.1). Right now, NiFi is built using Hive
> 1.2.1 and this feature is not available by default (the method is not
> implemented in the driver). However, if building NiFi with specific profiles
> this feature can be used.
> The objective is to expose the query timeout parameter in the processor and
> enable expression language. If the version of the driver is not implementing
> the query timeout the processor will be in invalid state (unless expression
> language is used, and in this case, the flow file will be routed to the
> failure relationship).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2138: NIFI-4371 - add support for query timeout in Hive p...
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2138#discussion_r206380004
--- Diff:
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/AbstractHiveQLProcessor.java
---
@@ -75,6 +81,38 @@
.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
.build();
+public static final PropertyDescriptor QUERY_TIMEOUT = new
PropertyDescriptor.Builder()
+.name("hive-query-timeout")
+.displayName("Query timeout")
+.description("Sets the number of seconds the driver will wait
for a query to execute. "
++ "A value of 0 means no timeout. NOTE: Non-zero
values may not be supported by the driver.")
+.defaultValue("0")
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.expressionLanguageSupported(true)
+.build();
+
+@Override
+protected Collection
customValidate(ValidationContext validationContext) {
+final List problems = new ArrayList<>(1);
+
+if(validationContext.getProperty(QUERY_TIMEOUT).isSet()
+&&
!validationContext.getProperty(QUERY_TIMEOUT).isExpressionLanguagePresent()
+&&
validationContext.getProperty(QUERY_TIMEOUT).asInteger() != 0) {
+try(HiveStatement stmt = new HiveStatement(null, null, null)) {
+stmt.setQueryTimeout(0);
--- End diff --
Also I think yes, the idea is to catch the "not supported" thing during
customValidate() where we can, but we also have to add it to the runtime
checking because of EL
---
[jira] [Created] (NIFI-5473) Add documentation for using intermediate CA with TLS toolkit
Andy LoPresto created NIFI-5473: --- Summary: Add documentation for using intermediate CA with TLS toolkit Key: NIFI-5473 URL: https://issues.apache.org/jira/browse/NIFI-5473 Project: Apache NiFi Issue Type: Improvement Components: Documentation Website, Security, Tools and Build Affects Versions: 1.7.1 Reporter: Andy LoPresto Assignee: Andy LoPresto With some manual work, the TLS toolkit can be used with a pre-existing certificate and private key that has been signed by an organization's certificate authority (CA) to sign toolkit-generated certificates. The Admin Guide should be improved to cover the necessary steps. When the separate "Security Guide" / "Toolkit Guide" is created, this content should be migrated there. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5342) CaptureChangeMySQL didn't escape database name
[ https://issues.apache.org/jira/browse/NIFI-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562651#comment-16562651 ] ASF GitHub Bot commented on NIFI-5342: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2923 > CaptureChangeMySQL didn't escape database name > -- > > Key: NIFI-5342 > URL: https://issues.apache.org/jira/browse/NIFI-5342 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Tanapol Nearunchorn >Assignee: Uday Kale >Priority: Major > Fix For: 1.8.0 > > > CaptureChangeMySQL didn't escape database name as you can see: > [https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-cdc/nifi-cdc-mysql-bundle/nifi-cdc-mysql-processors/src/main/java/org/apache/nifi/cdc/mysql/processors/CaptureChangeMySQL.java#L1003] > > And also with the line 1004, which query from a table. > It can cause problems when there is database or table name that contains a > reserved keyword. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5342) CaptureChangeMySQL didn't escape database name
[ https://issues.apache.org/jira/browse/NIFI-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562650#comment-16562650 ] ASF subversion and git services commented on NIFI-5342: --- Commit e97ae921f759e51ee1709ef0884fc029bd40d26b in nifi's branch refs/heads/master from uday [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=e97ae92 ] NIFI-5342 CaptureChangeMySQL didn't escape database name Signed-off-by: Matthew Burgess This closes #2923 > CaptureChangeMySQL didn't escape database name > -- > > Key: NIFI-5342 > URL: https://issues.apache.org/jira/browse/NIFI-5342 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Tanapol Nearunchorn >Assignee: Uday Kale >Priority: Major > Fix For: 1.8.0 > > > CaptureChangeMySQL didn't escape database name as you can see: > [https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-cdc/nifi-cdc-mysql-bundle/nifi-cdc-mysql-processors/src/main/java/org/apache/nifi/cdc/mysql/processors/CaptureChangeMySQL.java#L1003] > > And also with the line 1004, which query from a table. > It can cause problems when there is database or table name that contains a > reserved keyword. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2923: NIFI-5342 CaptureChangeMySQL didn't escape database...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2923 ---
[jira] [Commented] (NIFI-5342) CaptureChangeMySQL didn't escape database name
[ https://issues.apache.org/jira/browse/NIFI-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562649#comment-16562649 ] ASF GitHub Bot commented on NIFI-5342: -- Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/2923 +1 LGTM, tried with keywords as database names and table names (and regression tested existing DBs and tables), everything looks good. Thanks for the improvement! Merging to master. > CaptureChangeMySQL didn't escape database name > -- > > Key: NIFI-5342 > URL: https://issues.apache.org/jira/browse/NIFI-5342 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Tanapol Nearunchorn >Assignee: Uday Kale >Priority: Major > Fix For: 1.8.0 > > > CaptureChangeMySQL didn't escape database name as you can see: > [https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-cdc/nifi-cdc-mysql-bundle/nifi-cdc-mysql-processors/src/main/java/org/apache/nifi/cdc/mysql/processors/CaptureChangeMySQL.java#L1003] > > And also with the line 1004, which query from a table. > It can cause problems when there is database or table name that contains a > reserved keyword. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2923: NIFI-5342 CaptureChangeMySQL didn't escape database name
Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/2923 +1 LGTM, tried with keywords as database names and table names (and regression tested existing DBs and tables), everything looks good. Thanks for the improvement! Merging to master. ---
[jira] [Commented] (NIFI-4371) Add support for query timeout in Hive processors
[
https://issues.apache.org/jira/browse/NIFI-4371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562577#comment-16562577
]
ASF GitHub Bot commented on NIFI-4371:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2138#discussion_r206336993
--- Diff:
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/AbstractHiveQLProcessor.java
---
@@ -75,6 +81,38 @@
.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
.build();
+public static final PropertyDescriptor QUERY_TIMEOUT = new
PropertyDescriptor.Builder()
+.name("hive-query-timeout")
+.displayName("Query timeout")
+.description("Sets the number of seconds the driver will wait
for a query to execute. "
++ "A value of 0 means no timeout. NOTE: Non-zero
values may not be supported by the driver.")
+.defaultValue("0")
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.expressionLanguageSupported(true)
+.build();
+
+@Override
+protected Collection
customValidate(ValidationContext validationContext) {
+final List problems = new ArrayList<>(1);
+
+if(validationContext.getProperty(QUERY_TIMEOUT).isSet()
+&&
!validationContext.getProperty(QUERY_TIMEOUT).isExpressionLanguagePresent()
+&&
validationContext.getProperty(QUERY_TIMEOUT).asInteger() != 0) {
+try(HiveStatement stmt = new HiveStatement(null, null, null)) {
+stmt.setQueryTimeout(0);
--- End diff --
Yeah anything but zero, just so it matches the logic
> Add support for query timeout in Hive processors
>
>
> Key: NIFI-4371
> URL: https://issues.apache.org/jira/browse/NIFI-4371
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
> Attachments: Screen Shot 2017-09-09 at 4.31.21 PM.png, Screen Shot
> 2017-09-09 at 6.38.51 PM.png, Screen Shot 2017-09-09 at 6.40.48 PM.png
>
>
> With HIVE-4924 it is possible to set a query timeout when executing a query
> against Hive (starting with Hive 2.1). Right now, NiFi is built using Hive
> 1.2.1 and this feature is not available by default (the method is not
> implemented in the driver). However, if building NiFi with specific profiles
> this feature can be used.
> The objective is to expose the query timeout parameter in the processor and
> enable expression language. If the version of the driver is not implementing
> the query timeout the processor will be in invalid state (unless expression
> language is used, and in this case, the flow file will be routed to the
> failure relationship).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2138: NIFI-4371 - add support for query timeout in Hive p...
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2138#discussion_r206336993
--- Diff:
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/AbstractHiveQLProcessor.java
---
@@ -75,6 +81,38 @@
.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
.build();
+public static final PropertyDescriptor QUERY_TIMEOUT = new
PropertyDescriptor.Builder()
+.name("hive-query-timeout")
+.displayName("Query timeout")
+.description("Sets the number of seconds the driver will wait
for a query to execute. "
++ "A value of 0 means no timeout. NOTE: Non-zero
values may not be supported by the driver.")
+.defaultValue("0")
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.expressionLanguageSupported(true)
+.build();
+
+@Override
+protected Collection
customValidate(ValidationContext validationContext) {
+final List problems = new ArrayList<>(1);
+
+if(validationContext.getProperty(QUERY_TIMEOUT).isSet()
+&&
!validationContext.getProperty(QUERY_TIMEOUT).isExpressionLanguagePresent()
+&&
validationContext.getProperty(QUERY_TIMEOUT).asInteger() != 0) {
+try(HiveStatement stmt = new HiveStatement(null, null, null)) {
+stmt.setQueryTimeout(0);
--- End diff --
Yeah anything but zero, just so it matches the logic
---
[jira] [Created] (NIFI-5472) Trackpad Support / AKA pan canvas with scroll
Daniel created NIFI-5472: Summary: Trackpad Support / AKA pan canvas with scroll Key: NIFI-5472 URL: https://issues.apache.org/jira/browse/NIFI-5472 Project: Apache NiFi Issue Type: Improvement Components: Core UI Reporter: Daniel Hi, many canvas based apps support so called "trackpad mode", which allow you to pan the canvas using the scroll up/down or left/right gestures. To zoom, you instead use the pinch gesture. This would be a wonder addition for trackpad users. Thanks for the consideration. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4371) Add support for query timeout in Hive processors
[
https://issues.apache.org/jira/browse/NIFI-4371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562568#comment-16562568
]
ASF GitHub Bot commented on NIFI-4371:
--
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2138#discussion_r206335572
--- Diff:
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/AbstractHiveQLProcessor.java
---
@@ -75,6 +81,38 @@
.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
.build();
+public static final PropertyDescriptor QUERY_TIMEOUT = new
PropertyDescriptor.Builder()
+.name("hive-query-timeout")
+.displayName("Query timeout")
+.description("Sets the number of seconds the driver will wait
for a query to execute. "
++ "A value of 0 means no timeout. NOTE: Non-zero
values may not be supported by the driver.")
+.defaultValue("0")
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.expressionLanguageSupported(true)
+.build();
+
+@Override
+protected Collection
customValidate(ValidationContext validationContext) {
+final List problems = new ArrayList<>(1);
+
+if(validationContext.getProperty(QUERY_TIMEOUT).isSet()
+&&
!validationContext.getProperty(QUERY_TIMEOUT).isExpressionLanguagePresent()
+&&
validationContext.getProperty(QUERY_TIMEOUT).asInteger() != 0) {
+try(HiveStatement stmt = new HiveStatement(null, null, null)) {
+stmt.setQueryTimeout(0);
--- End diff --
> True (at least for older Apache Hive drivers), but it is a tad confusing
to see the if statement check for non-zero then test with zero.
Do you suggest an arbitrary value like
java
stmt.setQueryTimeout(1); // just checking driver supports query timeout
?
> Add support for query timeout in Hive processors
>
>
> Key: NIFI-4371
> URL: https://issues.apache.org/jira/browse/NIFI-4371
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
> Attachments: Screen Shot 2017-09-09 at 4.31.21 PM.png, Screen Shot
> 2017-09-09 at 6.38.51 PM.png, Screen Shot 2017-09-09 at 6.40.48 PM.png
>
>
> With HIVE-4924 it is possible to set a query timeout when executing a query
> against Hive (starting with Hive 2.1). Right now, NiFi is built using Hive
> 1.2.1 and this feature is not available by default (the method is not
> implemented in the driver). However, if building NiFi with specific profiles
> this feature can be used.
> The objective is to expose the query timeout parameter in the processor and
> enable expression language. If the version of the driver is not implementing
> the query timeout the processor will be in invalid state (unless expression
> language is used, and in this case, the flow file will be routed to the
> failure relationship).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2138: NIFI-4371 - add support for query timeout in Hive p...
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2138#discussion_r206335572
--- Diff:
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/AbstractHiveQLProcessor.java
---
@@ -75,6 +81,38 @@
.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
.build();
+public static final PropertyDescriptor QUERY_TIMEOUT = new
PropertyDescriptor.Builder()
+.name("hive-query-timeout")
+.displayName("Query timeout")
+.description("Sets the number of seconds the driver will wait
for a query to execute. "
++ "A value of 0 means no timeout. NOTE: Non-zero
values may not be supported by the driver.")
+.defaultValue("0")
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.expressionLanguageSupported(true)
+.build();
+
+@Override
+protected Collection
customValidate(ValidationContext validationContext) {
+final List problems = new ArrayList<>(1);
+
+if(validationContext.getProperty(QUERY_TIMEOUT).isSet()
+&&
!validationContext.getProperty(QUERY_TIMEOUT).isExpressionLanguagePresent()
+&&
validationContext.getProperty(QUERY_TIMEOUT).asInteger() != 0) {
+try(HiveStatement stmt = new HiveStatement(null, null, null)) {
+stmt.setQueryTimeout(0);
--- End diff --
> True (at least for older Apache Hive drivers), but it is a tad confusing
to see the if statement check for non-zero then test with zero.
Do you suggest an arbitrary value like
java
stmt.setQueryTimeout(1); // just checking driver supports query timeout
?
---
[jira] [Commented] (NIFI-4371) Add support for query timeout in Hive processors
[
https://issues.apache.org/jira/browse/NIFI-4371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562566#comment-16562566
]
ASF GitHub Bot commented on NIFI-4371:
--
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2138#discussion_r206334896
--- Diff:
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/AbstractHiveQLProcessor.java
---
@@ -76,6 +86,38 @@
.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
.build();
+public static final PropertyDescriptor QUERY_TIMEOUT = new
PropertyDescriptor.Builder()
+.name("hive-query-timeout")
+.displayName("Query timeout")
+.description("Sets the number of seconds the driver will wait
for a query to execute. "
++ "A value of 0 means no timeout. NOTE: Non-zero
values may not be supported by the driver.")
+.defaultValue("0")
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
--- End diff --
True - will fix in AbstractHiveQLProcessor and AbstractHive3QLProcessor
> Add support for query timeout in Hive processors
>
>
> Key: NIFI-4371
> URL: https://issues.apache.org/jira/browse/NIFI-4371
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
> Attachments: Screen Shot 2017-09-09 at 4.31.21 PM.png, Screen Shot
> 2017-09-09 at 6.38.51 PM.png, Screen Shot 2017-09-09 at 6.40.48 PM.png
>
>
> With HIVE-4924 it is possible to set a query timeout when executing a query
> against Hive (starting with Hive 2.1). Right now, NiFi is built using Hive
> 1.2.1 and this feature is not available by default (the method is not
> implemented in the driver). However, if building NiFi with specific profiles
> this feature can be used.
> The objective is to expose the query timeout parameter in the processor and
> enable expression language. If the version of the driver is not implementing
> the query timeout the processor will be in invalid state (unless expression
> language is used, and in this case, the flow file will be routed to the
> failure relationship).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2138: NIFI-4371 - add support for query timeout in Hive p...
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2138#discussion_r206334896
--- Diff:
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/AbstractHiveQLProcessor.java
---
@@ -76,6 +86,38 @@
.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
.build();
+public static final PropertyDescriptor QUERY_TIMEOUT = new
PropertyDescriptor.Builder()
+.name("hive-query-timeout")
+.displayName("Query timeout")
+.description("Sets the number of seconds the driver will wait
for a query to execute. "
++ "A value of 0 means no timeout. NOTE: Non-zero
values may not be supported by the driver.")
+.defaultValue("0")
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
--- End diff --
True - will fix in AbstractHiveQLProcessor and AbstractHive3QLProcessor
---
[jira] [Commented] (NIFI-4371) Add support for query timeout in Hive processors
[
https://issues.apache.org/jira/browse/NIFI-4371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562564#comment-16562564
]
ASF GitHub Bot commented on NIFI-4371:
--
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2138#discussion_r206334616
--- Diff:
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/AbstractHiveQLProcessor.java
---
@@ -75,6 +81,38 @@
.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
.build();
+public static final PropertyDescriptor QUERY_TIMEOUT = new
PropertyDescriptor.Builder()
+.name("hive-query-timeout")
+.displayName("Query timeout")
+.description("Sets the number of seconds the driver will wait
for a query to execute. "
++ "A value of 0 means no timeout. NOTE: Non-zero
values may not be supported by the driver.")
+.defaultValue("0")
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.expressionLanguageSupported(true)
+.build();
+
+@Override
+protected Collection
customValidate(ValidationContext validationContext) {
+final List problems = new ArrayList<>(1);
+
+if(validationContext.getProperty(QUERY_TIMEOUT).isSet()
+&&
!validationContext.getProperty(QUERY_TIMEOUT).isExpressionLanguagePresent()
+&&
validationContext.getProperty(QUERY_TIMEOUT).asInteger() != 0) {
+try(HiveStatement stmt = new HiveStatement(null, null, null)) {
+stmt.setQueryTimeout(0);
--- End diff --
Did you mean this one:
https://github.com/apache/nifi/pull/2138#discussion_r159670913
> Add support for query timeout in Hive processors
>
>
> Key: NIFI-4371
> URL: https://issues.apache.org/jira/browse/NIFI-4371
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
> Attachments: Screen Shot 2017-09-09 at 4.31.21 PM.png, Screen Shot
> 2017-09-09 at 6.38.51 PM.png, Screen Shot 2017-09-09 at 6.40.48 PM.png
>
>
> With HIVE-4924 it is possible to set a query timeout when executing a query
> against Hive (starting with Hive 2.1). Right now, NiFi is built using Hive
> 1.2.1 and this feature is not available by default (the method is not
> implemented in the driver). However, if building NiFi with specific profiles
> this feature can be used.
> The objective is to expose the query timeout parameter in the processor and
> enable expression language. If the version of the driver is not implementing
> the query timeout the processor will be in invalid state (unless expression
> language is used, and in this case, the flow file will be routed to the
> failure relationship).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2138: NIFI-4371 - add support for query timeout in Hive p...
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2138#discussion_r206334616
--- Diff:
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/AbstractHiveQLProcessor.java
---
@@ -75,6 +81,38 @@
.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
.build();
+public static final PropertyDescriptor QUERY_TIMEOUT = new
PropertyDescriptor.Builder()
+.name("hive-query-timeout")
+.displayName("Query timeout")
+.description("Sets the number of seconds the driver will wait
for a query to execute. "
++ "A value of 0 means no timeout. NOTE: Non-zero
values may not be supported by the driver.")
+.defaultValue("0")
+.required(true)
+.addValidator(StandardValidators.INTEGER_VALIDATOR)
+.expressionLanguageSupported(true)
+.build();
+
+@Override
+protected Collection
customValidate(ValidationContext validationContext) {
+final List problems = new ArrayList<>(1);
+
+if(validationContext.getProperty(QUERY_TIMEOUT).isSet()
+&&
!validationContext.getProperty(QUERY_TIMEOUT).isExpressionLanguagePresent()
+&&
validationContext.getProperty(QUERY_TIMEOUT).asInteger() != 0) {
+try(HiveStatement stmt = new HiveStatement(null, null, null)) {
+stmt.setQueryTimeout(0);
--- End diff --
Did you mean this one:
https://github.com/apache/nifi/pull/2138#discussion_r159670913
---
[jira] [Commented] (NIFI-5450) Add endpoint override to GetSQS, PutSQS
[ https://issues.apache.org/jira/browse/NIFI-5450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562535#comment-16562535 ] ASF GitHub Bot commented on NIFI-5450: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2924 > Add endpoint override to GetSQS, PutSQS > > > Key: NIFI-5450 > URL: https://issues.apache.org/jira/browse/NIFI-5450 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Richard St. John >Assignee: Sivaprasanna Sethuraman >Priority: Major > Fix For: 1.8.0 > > Attachments: > 0001-NIFI-5450-Added-ENDPOINT_OVERRIDE-to-AWS-SQS-process.patch > > > The endpoint override property descriptor should be included for *SQS > processors to facilitate other regions not available in the Region property > descriptor. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5450) Add endpoint override to GetSQS, PutSQS
[ https://issues.apache.org/jira/browse/NIFI-5450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-5450: - Component/s: (was: Core Framework) Extensions > Add endpoint override to GetSQS, PutSQS > > > Key: NIFI-5450 > URL: https://issues.apache.org/jira/browse/NIFI-5450 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Richard St. John >Assignee: Sivaprasanna Sethuraman >Priority: Major > Fix For: 1.8.0 > > Attachments: > 0001-NIFI-5450-Added-ENDPOINT_OVERRIDE-to-AWS-SQS-process.patch > > > The endpoint override property descriptor should be included for *SQS > processors to facilitate other regions not available in the Region property > descriptor. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5450) Add endpoint override to GetSQS, PutSQS
[ https://issues.apache.org/jira/browse/NIFI-5450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-5450: - Resolution: Fixed Fix Version/s: 1.8.0 Status: Resolved (was: Patch Available) > Add endpoint override to GetSQS, PutSQS > > > Key: NIFI-5450 > URL: https://issues.apache.org/jira/browse/NIFI-5450 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Richard St. John >Assignee: Sivaprasanna Sethuraman >Priority: Major > Fix For: 1.8.0 > > Attachments: > 0001-NIFI-5450-Added-ENDPOINT_OVERRIDE-to-AWS-SQS-process.patch > > > The endpoint override property descriptor should be included for *SQS > processors to facilitate other regions not available in the Region property > descriptor. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2924: NIFI-5450: Added ENDPOINT_OVERRIDE to AWS SQS proce...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2924 ---
[jira] [Commented] (NIFI-5450) Add endpoint override to GetSQS, PutSQS
[ https://issues.apache.org/jira/browse/NIFI-5450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562533#comment-16562533 ] ASF subversion and git services commented on NIFI-5450: --- Commit 44face10bf9837308587dc1d46326e4a5764435f in nifi's branch refs/heads/master from zenfenan [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=44face1 ] NIFI-5450: Added ENDPOINT_OVERRIDE to AWS SQS processors Signed-off-by: Pierre Villard This closes #2924. > Add endpoint override to GetSQS, PutSQS > > > Key: NIFI-5450 > URL: https://issues.apache.org/jira/browse/NIFI-5450 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Richard St. John >Assignee: Sivaprasanna Sethuraman >Priority: Major > Attachments: > 0001-NIFI-5450-Added-ENDPOINT_OVERRIDE-to-AWS-SQS-process.patch > > > The endpoint override property descriptor should be included for *SQS > processors to facilitate other regions not available in the Region property > descriptor. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5450) Add endpoint override to GetSQS, PutSQS
[ https://issues.apache.org/jira/browse/NIFI-5450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562532#comment-16562532 ] ASF GitHub Bot commented on NIFI-5450: -- Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/2924 +1, merging to master, thanks @zenfenan > Add endpoint override to GetSQS, PutSQS > > > Key: NIFI-5450 > URL: https://issues.apache.org/jira/browse/NIFI-5450 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Richard St. John >Assignee: Sivaprasanna Sethuraman >Priority: Major > Attachments: > 0001-NIFI-5450-Added-ENDPOINT_OVERRIDE-to-AWS-SQS-process.patch > > > The endpoint override property descriptor should be included for *SQS > processors to facilitate other regions not available in the Region property > descriptor. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2924: NIFI-5450: Added ENDPOINT_OVERRIDE to AWS SQS processors
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/2924 +1, merging to master, thanks @zenfenan ---
[jira] [Created] (NIFI-5471) QueryDatabaseTable does not correctly support arbitrary queries on Oracle
Matt Burgess created NIFI-5471:
--
Summary: QueryDatabaseTable does not correctly support arbitrary
queries on Oracle
Key: NIFI-5471
URL: https://issues.apache.org/jira/browse/NIFI-5471
Project: Apache NiFi
Issue Type: Bug
Components: Extensions
Reporter: Matt Burgess
NIFI-1706 added the capability to generate the "table" it uses for incremental
fetch from a SQL statement, rather than generating it directly from "Columns to
Return", "Max Value Columns", etc. When a SQL query is supplied, it is wrapped
as a nested query in a SELECT, and the result is aliased to the provided "Table
Name" property value.
Currently this is done by simply generating a string "SELECT (" + query + ") AS
" + tableName. However some databases (such as Oracle) do not allow the "AS"
keywords for aliasing things like tables. The getWrappedQuery() method should
delegate to the specified database adapter for generating a table alias clause.
Many adapters will return "AS " + tableName, where the Oracle adapter(s) will
simply return the table name.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5469) Edits needed for LDAP and Kerberos login identity provider sections in Admin Guide
[ https://issues.apache.org/jira/browse/NIFI-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562444#comment-16562444 ] ASF GitHub Bot commented on NIFI-5469: -- GitHub user andrewmlim opened a pull request: https://github.com/apache/nifi/pull/2925 NIFI-5469 Additional italics and code formatting corrections for cons… …istency You can merge this pull request into a Git repository by running: $ git pull https://github.com/andrewmlim/nifi NIFI-5469-2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2925.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2925 commit ac62417eaede596d4f33c37aa12a048d197d08d1 Author: Andrew Lim Date: 2018-07-30T20:17:19Z NIFI-5469 Additional italics and code formatting corrections for consistency > Edits needed for LDAP and Kerberos login identity provider sections in Admin > Guide > -- > > Key: NIFI-5469 > URL: https://issues.apache.org/jira/browse/NIFI-5469 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation Website >Reporter: Andrew Lim >Assignee: Andrew Lim >Priority: Minor > > Going through the Authentication and Authorization sections of the Admin > Guide, I noticed the following improvements could be made: > * Removed “Kerberos Config File” property from kerberos-provider login > identity provider (this was done because the same property exists in > nifi.properties) > * Corrected the "LDAP-based Users/Groups Referencing User Attribute” login > identity provider example to refer to “member uid" > * Added titles to login identity provider examples for improved > readability/search > * Changed UserGroupProvider property examples from bulleted lists to tables > Also, text formatting for references to config files, directories, etc. need > to be made consistent. For example, config files like _nifi.properties_, > _authorizers.xml_ should be italicized. Directories, properties and default > values for properties should be monospaced. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2925: NIFI-5469 Additional italics and code formatting co...
GitHub user andrewmlim opened a pull request: https://github.com/apache/nifi/pull/2925 NIFI-5469 Additional italics and code formatting corrections for cons⦠â¦istency You can merge this pull request into a Git repository by running: $ git pull https://github.com/andrewmlim/nifi NIFI-5469-2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2925.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2925 commit ac62417eaede596d4f33c37aa12a048d197d08d1 Author: Andrew Lim Date: 2018-07-30T20:17:19Z NIFI-5469 Additional italics and code formatting corrections for consistency ---
[jira] [Created] (MINIFICPP-585) Add tests for updating full flow to empty flow and validate all internal services stop
Mr TheSegfault created MINIFICPP-585: Summary: Add tests for updating full flow to empty flow and validate all internal services stop Key: MINIFICPP-585 URL: https://issues.apache.org/jira/browse/MINIFICPP-585 Project: NiFi MiNiFi C++ Issue Type: Test Reporter: Mr TheSegfault Assignee: Mr TheSegfault -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5327) NetFlow Processors
[ https://issues.apache.org/jira/browse/NIFI-5327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562391#comment-16562391 ] ASF GitHub Bot commented on NIFI-5327: -- Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2820 As far as I can see, this PR is in great same. Good works @PrashanthVenkatesan. Tests and contrib tests are great. I manually tested with the sample generator and the output looks like it will be useful. Also, the factoring of the submittal will make the record reader follow up a breeze ;) +1 from me. @MikeThomsen, @bbende, @mattyb149 ? anyone have time to take a look at this? > NetFlow Processors > -- > > Key: NIFI-5327 > URL: https://issues.apache.org/jira/browse/NIFI-5327 > Project: Apache NiFi > Issue Type: New Feature > Components: Core Framework >Affects Versions: 1.6.0 >Reporter: Prashanth Venkatesan >Assignee: Prashanth Venkatesan >Priority: Major > > As network traffic data scopes for the big data use case, would like NiFi to > have processors to support parsing of those protocols. > Netflow is a protocol introduced by Cisco that provides the ability to > collect IP network traffic as it enters or exits an interface and is > described in detail in here: > [https://www.cisco.com/c/en/us/td/docs/net_mgmt/netflow_collection_engine/3-6/user/guide/format.html] > > Currently, I have created the following processor: > *ParseNetflowv5*: Parses the ingress netflowv5 bytes and ingest as either > NiFi flowfile attributes or as a JSON content. This also sends > one-time-template. > > Further ahead, we can add many processor specific to network protocols in > this nar bundle. > I will create a pull request. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2820 As far as I can see, this PR is in great same. Good works @PrashanthVenkatesan. Tests and contrib tests are great. I manually tested with the sample generator and the output looks like it will be useful. Also, the factoring of the submittal will make the record reader follow up a breeze ;) +1 from me. @MikeThomsen, @bbende, @mattyb149 ? anyone have time to take a look at this? ---
[jira] [Commented] (NIFI-5327) NetFlow Processors
[ https://issues.apache.org/jira/browse/NIFI-5327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562385#comment-16562385 ] ASF GitHub Bot commented on NIFI-5327: -- Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/nifi/pull/2820#discussion_r206290668 --- Diff: nifi-nar-bundles/nifi-network-bundle/nifi-network-utils/src/main/java/org/apache/nifi/processors/network/parser/Netflowv5Parser.java --- @@ -29,6 +32,7 @@ private static final int SHORT_TYPE = 0; private static final int INTEGER_TYPE = 1; private static final int LONG_TYPE = 2; --- End diff -- I did some research, it doesn't support it in this version. > NetFlow Processors > -- > > Key: NIFI-5327 > URL: https://issues.apache.org/jira/browse/NIFI-5327 > Project: Apache NiFi > Issue Type: New Feature > Components: Core Framework >Affects Versions: 1.6.0 >Reporter: Prashanth Venkatesan >Assignee: Prashanth Venkatesan >Priority: Major > > As network traffic data scopes for the big data use case, would like NiFi to > have processors to support parsing of those protocols. > Netflow is a protocol introduced by Cisco that provides the ability to > collect IP network traffic as it enters or exits an interface and is > described in detail in here: > [https://www.cisco.com/c/en/us/td/docs/net_mgmt/netflow_collection_engine/3-6/user/guide/format.html] > > Currently, I have created the following processor: > *ParseNetflowv5*: Parses the ingress netflowv5 bytes and ingest as either > NiFi flowfile attributes or as a JSON content. This also sends > one-time-template. > > Further ahead, we can add many processor specific to network protocols in > this nar bundle. > I will create a pull request. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/nifi/pull/2820#discussion_r206290668 --- Diff: nifi-nar-bundles/nifi-network-bundle/nifi-network-utils/src/main/java/org/apache/nifi/processors/network/parser/Netflowv5Parser.java --- @@ -29,6 +32,7 @@ private static final int SHORT_TYPE = 0; private static final int INTEGER_TYPE = 1; private static final int LONG_TYPE = 2; --- End diff -- I did some research, it doesn't support it in this version. ---
[jira] [Commented] (NIFI-3531) modify session.recover behaviour in nifi-jms-processors to cope with high-traffic JMS
[ https://issues.apache.org/jira/browse/NIFI-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562370#comment-16562370 ] Michael Moser commented on NIFI-3531: - I have another scenario where the session.recover() causes problems. If the consumer has prefetch enabled, then the call to session.recover() causes messages sent but not acknowledged to be marked as "redelivered". This can cause JMS providers to retransmit these messages to the client. Also, some providers limit the number of times a message can be redelivered, so a message could actually be lost. I'm having a really hard time justifying the call to session.recover() before each call to consumer.receive(). It seems to cause known problems rather than limit possible message loss. > modify session.recover behaviour in nifi-jms-processors to cope with > high-traffic JMS > - > > Key: NIFI-3531 > URL: https://issues.apache.org/jira/browse/NIFI-3531 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Dominik Benz >Priority: Major > > As described in this mailing list post > http://apache-nifi-developer-list.39713.n7.nabble.com/session-recover-behaviour-in-nifi-jms-processor-td14940.html > the current implementation of nifi-jms-processor, especially of JMSConsumer > https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/main/java/org/apache/nifi/jms/processors/JMSConsumer.java > causes frequent re-delivery of JMS messages due to the following behaviour: > 1) several threads perform the JMS session callback in parallel > 2) each callback performs a session.recover > 3) during high traffic, the situation arises that the ACKs from another > thread may not (yet) have arrived at the JMS server > 4) this implies that the pointer of session.recover will reconsume the > not-yet-acked message from another thread > I understood Nifi prefers message duplication over message loss - however, in > our case the number of re-delivered messages is "piling up" over time, > leading to a growing number of un-acked messages in JMS and finally to > storage problems on the JMS server side. > It would be great to modify the nifi-jms-processor package to reliably cope > with higher-traffic JMS sources. Ideas from my side would be / include: > 1) make synchronous / asynchronous delivery configurable > 2) add configuration option for custom ACKing modes (i.e. non-standard JMS > modes like individual ACKing or NO_ACK) > 3) add configuration option for JMS message selectors > From other projects, I found the JMS communication options in this > spark-jms-receiver package > https://github.com/tbfenet/spark-jms-receiver > very helpful (in which situations are synchronous / asynchronous approaches > reliable, how to buffer messages before acking them, how/when to ack, ...). > Here's also a java port of a subset: > https://github.com/bernhardschaefer/spark-jms-receiver-java > For any mentioned option I'm also happy to help! > Best & thanks for a great piece of software, > Dominik -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #383: Minificpp 582
GitHub user phrocker opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/383 Minificpp 582 Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file? - [ ] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/phrocker/nifi-minifi-cpp MINIFICPP-582 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/383.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #383 commit 5f53b4bf63f324167e06d0da9495537b70ef2916 Author: Marc Parisi Date: 2018-07-30T18:14:12Z MINFICIPP-582: Update readme to implement bustache and kafka constraints commit d16c500defd6be94f0a301f8984b50f059c0d533 Author: Marc Parisi Date: 2018-07-30T18:57:53Z MINIFICPP-582: Run bash beautify ---
[jira] [Commented] (NIFI-5442) Message Page uses raw X-ProxyContextPath
[
https://issues.apache.org/jira/browse/NIFI-5442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562363#comment-16562363
]
ASF GitHub Bot commented on NIFI-5442:
--
Github user mcg30005 commented on the issue:
https://github.com/apache/nifi/pull/2908
+1
Hi Andy, PR looks really good, i asked Shawna to take a look as well since
she has way more experience in this area. One question that we had was what
page render will do with empty init str, we're going to patch and try this.
> Message Page uses raw X-ProxyContextPath
>
>
> Key: NIFI-5442
> URL: https://issues.apache.org/jira/browse/NIFI-5442
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.6.0
>Reporter: Dan Fike
>Assignee: Andy LoPresto
>Priority: Major
>
> It looks like {{message-page.jsp}} uses {{X-ProxyContextPath}} verbatim
> without sanitizing it or anything. See
> [https://github.com/apache/nifi/blob/66783c18b24b1c6b1cfd662c58ca9df1e60b866e/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/WEB-INF/pages/message-page.jsp#L21]
>
> I verified this by hitting {{/nifi-api/access/oidc/callback}} on an unsecured
> NiFi host to get the *User authentication/authorization is only supported
> when running over HTTPS* message page.
>
> {code:java}
> $ curl http://hostname/nifi-api/access/oidc/callback
> ...
> type="text/css" />
> ...
> $ curl --header "X-ProxyContextPath: /nifi/assets/reset.css/reset.css\"
> type=\"text/css\" /> type=\"text/javascript\">alert(\"omg\"); href=\"" http://hostname/nifi-api/access/oidc/callback
> ...
> type="text/css" />alert("omg"); rel="stylesheet" href="/nifi/assets/reset.css/reset.css" type="text/css" />
> ...{code}
>
> Presumably we want to do something like this:
> [https://github.com/apache/nifi/commit/5d643edfaba4f5369c94ee1b4eaa5c59e3a9f37a#diff-91119fe15bb6f3b931662093e367b671R20]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #2908: NIFI-5442 Get X-ProxyContextPath value from request attrib...
Github user mcg30005 commented on the issue: https://github.com/apache/nifi/pull/2908 +1 Hi Andy, PR looks really good, i asked Shawna to take a look as well since she has way more experience in this area. One question that we had was what page render will do with empty init str, we're going to patch and try this. ---
[jira] [Assigned] (MINIFICPP-582) Readme constraints not defined in bootstrap
[ https://issues.apache.org/jira/browse/MINIFICPP-582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mr TheSegfault reassigned MINIFICPP-582: Assignee: Mr TheSegfault > Readme constraints not defined in bootstrap > --- > > Key: MINIFICPP-582 > URL: https://issues.apache.org/jira/browse/MINIFICPP-582 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Blocker > Fix For: 0.6.0 > > > MINIFICPP-427 and others have added some constraints that aren't captured in > the bootstrap. We can make these changes in this ticket -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-5456) PutKinesisStream - Fails to work with AWS Private Link endpoint
[ https://issues.apache.org/jira/browse/NIFI-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivaprasanna Sethuraman reassigned NIFI-5456: - Assignee: Sivaprasanna Sethuraman > PutKinesisStream - Fails to work with AWS Private Link endpoint > --- > > Key: NIFI-5456 > URL: https://issues.apache.org/jira/browse/NIFI-5456 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.6.0, 1.7.1 > Environment: RedHat 6 >Reporter: Ariel Godinez >Assignee: Sivaprasanna Sethuraman >Priority: Major > Labels: easyfix > > NiFi version: 1.6.0 > PutKinesisStream fails to put due to invalid signing information when using > an AWS Private Link as the endpoint override URL. The endpoint override URL > pattern for private links is like below along with the error that NiFi > outputs when we attempt to use this type of URL as the 'Endpoint Override > URL' property value. > Endpoint Override URL: > [https://vpce-|https://vpce-/].kinesis.us-east-2.vpce.amazonaws.com > ERROR [Timer-Driven Process Thread-11] "o.a.n.p.a.k.stream.PutKinesisStream" > PutKinesisStream[id=4c314e25-0164-1000--9bd79c77] Failed to publish > due to exception com.amazonaws.services.kinesis.model.AmazonKinesisException: > Credential should be scoped to a valid region, not 'vpce'. (Service: > AmazonKinesis; Status Code: 400; Error Code: InvalidSignatureException; > Request ID: 6330b83c-a64e-4acf-b892-a505621cf78e) flowfiles > [StandardFlowFileRecord[uuid=ba299cec-7cbf-4750-a766-c348b5cd9c73,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1532469012962-1, > container=content002, section=1], offset=2159750, > length=534625],offset=0,name=900966573101260,size=534625]] > > It looks like 'vpce' is being extracted from the url as the region name when > it should be getting 'us-east-2'. We were able to get this processor to work > correctly by explicitly passing in the region and service using > 'setEndpoint(String endpoint, String serviceName, String regionId)' instead > of 'setEndpoint(String endpoint)' in > 'nifi/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java' > line 289 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5450) Add endpoint override to GetSQS, PutSQS
[ https://issues.apache.org/jira/browse/NIFI-5450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivaprasanna Sethuraman updated NIFI-5450: -- Attachment: 0001-NIFI-5450-Added-ENDPOINT_OVERRIDE-to-AWS-SQS-process.patch Status: Patch Available (was: In Progress) > Add endpoint override to GetSQS, PutSQS > > > Key: NIFI-5450 > URL: https://issues.apache.org/jira/browse/NIFI-5450 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Richard St. John >Assignee: Sivaprasanna Sethuraman >Priority: Major > Attachments: > 0001-NIFI-5450-Added-ENDPOINT_OVERRIDE-to-AWS-SQS-process.patch > > > The endpoint override property descriptor should be included for *SQS > processors to facilitate other regions not available in the Region property > descriptor. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5450) Add endpoint override to GetSQS, PutSQS
[ https://issues.apache.org/jira/browse/NIFI-5450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562227#comment-16562227 ] ASF GitHub Bot commented on NIFI-5450: -- GitHub user zenfenan opened a pull request: https://github.com/apache/nifi/pull/2924 NIFI-5450: Added ENDPOINT_OVERRIDE to AWS SQS processors Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zenfenan/nifi NIFI-5450 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2924.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2924 commit 7ec2ac467f6a114037f4e2bfdd5df201cf2dd8d8 Author: zenfenan Date: 2018-07-30T17:48:37Z NIFI-5450: Added ENDPOINT_OVERRIDE to AWS SQS processors > Add endpoint override to GetSQS, PutSQS > > > Key: NIFI-5450 > URL: https://issues.apache.org/jira/browse/NIFI-5450 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Richard St. John >Assignee: Sivaprasanna Sethuraman >Priority: Major > > The endpoint override property descriptor should be included for *SQS > processors to facilitate other regions not available in the Region property > descriptor. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2924: NIFI-5450: Added ENDPOINT_OVERRIDE to AWS SQS proce...
GitHub user zenfenan opened a pull request: https://github.com/apache/nifi/pull/2924 NIFI-5450: Added ENDPOINT_OVERRIDE to AWS SQS processors Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zenfenan/nifi NIFI-5450 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2924.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2924 commit 7ec2ac467f6a114037f4e2bfdd5df201cf2dd8d8 Author: zenfenan Date: 2018-07-30T17:48:37Z NIFI-5450: Added ENDPOINT_OVERRIDE to AWS SQS processors ---
[GitHub] nifi pull request #2921: NIFI-5468 Add toolkit to NiFi docker image and use ...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2921#discussion_r206252629
--- Diff: nifi-docker/dockermaven/sh/secure.sh ---
@@ -45,13 +45,23 @@ prop_replace 'nifi.security.truststore'
"${TRUSTSTORE_PATH}"
prop_replace 'nifi.security.truststoreType' "${TRUSTSTORE_TYPE}"
prop_replace 'nifi.security.truststorePasswd' "${TRUSTSTORE_PASSWORD}"
+prop_replace 'keystore' "${KEYSTORE_PATH}"
${nifi_toolkit_props_file}
+prop_replace 'keystoreType' "${KEYSTORE_TYPE}"
${nifi_toolkit_props_file}
+prop_replace 'keystorePasswd' "${KEYSTORE_PASSWORD}"
${nifi_toolkit_props_file}
--- End diff --
Same comment about `keyPasswd`.
---
[jira] [Commented] (NIFI-5468) Add toolkit to NiFi docker image and use zip archive
[
https://issues.apache.org/jira/browse/NIFI-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562169#comment-16562169
]
ASF GitHub Bot commented on NIFI-5468:
--
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2921#discussion_r206252629
--- Diff: nifi-docker/dockermaven/sh/secure.sh ---
@@ -45,13 +45,23 @@ prop_replace 'nifi.security.truststore'
"${TRUSTSTORE_PATH}"
prop_replace 'nifi.security.truststoreType' "${TRUSTSTORE_TYPE}"
prop_replace 'nifi.security.truststorePasswd' "${TRUSTSTORE_PASSWORD}"
+prop_replace 'keystore' "${KEYSTORE_PATH}"
${nifi_toolkit_props_file}
+prop_replace 'keystoreType' "${KEYSTORE_TYPE}"
${nifi_toolkit_props_file}
+prop_replace 'keystorePasswd' "${KEYSTORE_PASSWORD}"
${nifi_toolkit_props_file}
--- End diff --
Same comment about `keyPasswd`.
> Add toolkit to NiFi docker image and use zip archive
>
>
> Key: NIFI-5468
> URL: https://issues.apache.org/jira/browse/NIFI-5468
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Docker
>Affects Versions: 1.8.0
>Reporter: Peter Wilcsinszky
>Assignee: Peter Wilcsinszky
>Priority: Major
> Fix For: 1.8.0
>
>
> NiFi toolkit is an essential tool for a NiFi installation and makes sense to
> include it by default in the NiFi Docker image.
> Using the zip archive is a step to make possible switching completely to the
> zip release and avoid generating the tar.gz anymore. For the discussion see
> http://apache-nifi-developer-list.39713.n7.nabble.com/DISCUSS-Tar-Gzip-vs-Zip-td19175.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5468) Add toolkit to NiFi docker image and use zip archive
[
https://issues.apache.org/jira/browse/NIFI-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562164#comment-16562164
]
ASF GitHub Bot commented on NIFI-5468:
--
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2921#discussion_r206251879
--- Diff: nifi-docker/dockerhub/sh/secure.sh ---
@@ -45,13 +45,23 @@ prop_replace 'nifi.security.truststore'
"${TRUSTSTORE_PATH}"
prop_replace 'nifi.security.truststoreType' "${TRUSTSTORE_TYPE}"
prop_replace 'nifi.security.truststorePasswd' "${TRUSTSTORE_PASSWORD}"
+prop_replace 'keystore' "${KEYSTORE_PATH}"
${nifi_toolkit_props_file}
+prop_replace 'keystoreType' "${KEYSTORE_TYPE}"
${nifi_toolkit_props_file}
+prop_replace 'keystorePasswd' "${KEYSTORE_PASSWORD}"
${nifi_toolkit_props_file}
--- End diff --
Sometimes the `keyPasswd` is different from the `keystorePasswd` so
probably needs to be included here as well.
> Add toolkit to NiFi docker image and use zip archive
>
>
> Key: NIFI-5468
> URL: https://issues.apache.org/jira/browse/NIFI-5468
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Docker
>Affects Versions: 1.8.0
>Reporter: Peter Wilcsinszky
>Assignee: Peter Wilcsinszky
>Priority: Major
> Fix For: 1.8.0
>
>
> NiFi toolkit is an essential tool for a NiFi installation and makes sense to
> include it by default in the NiFi Docker image.
> Using the zip archive is a step to make possible switching completely to the
> zip release and avoid generating the tar.gz anymore. For the discussion see
> http://apache-nifi-developer-list.39713.n7.nabble.com/DISCUSS-Tar-Gzip-vs-Zip-td19175.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2921: NIFI-5468 Add toolkit to NiFi docker image and use ...
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2921#discussion_r206251879
--- Diff: nifi-docker/dockerhub/sh/secure.sh ---
@@ -45,13 +45,23 @@ prop_replace 'nifi.security.truststore'
"${TRUSTSTORE_PATH}"
prop_replace 'nifi.security.truststoreType' "${TRUSTSTORE_TYPE}"
prop_replace 'nifi.security.truststorePasswd' "${TRUSTSTORE_PASSWORD}"
+prop_replace 'keystore' "${KEYSTORE_PATH}"
${nifi_toolkit_props_file}
+prop_replace 'keystoreType' "${KEYSTORE_TYPE}"
${nifi_toolkit_props_file}
+prop_replace 'keystorePasswd' "${KEYSTORE_PASSWORD}"
${nifi_toolkit_props_file}
--- End diff --
Sometimes the `keyPasswd` is different from the `keystorePasswd` so
probably needs to be included here as well.
---
[jira] [Commented] (NIFI-4731) BigQuery processors
[
https://issues.apache.org/jira/browse/NIFI-4731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16562017#comment-16562017
]
ASF GitHub Bot commented on NIFI-4731:
--
Github user danieljimenez commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2682#discussion_r206191230
--- Diff:
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/bigquery/PutBigQueryBatch.java
---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.gcp.bigquery;
+
+import com.google.cloud.bigquery.BigQuery;
+import com.google.cloud.bigquery.FormatOptions;
+import com.google.cloud.bigquery.Job;
+import com.google.cloud.bigquery.JobInfo;
+import com.google.cloud.bigquery.Schema;
+import com.google.cloud.bigquery.TableDataWriteChannel;
+import com.google.cloud.bigquery.TableId;
+import com.google.cloud.bigquery.WriteChannelConfiguration;
+import com.google.common.collect.ImmutableList;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.LogLevel;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.gcp.storage.DeleteGCSObject;
+import org.apache.nifi.processors.gcp.storage.PutGCSObject;
+import org.apache.nifi.util.StringUtils;
+
+import java.nio.ByteBuffer;
+import java.nio.channels.Channels;
+import java.nio.channels.ReadableByteChannel;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * A processor for batch loading data into a Google BigQuery table
+ */
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@Tags({"google", "google cloud", "bq", "bigquery"})
+@CapabilityDescription("Batch loads flow files to a Google BigQuery
table.")
+@SeeAlso({PutGCSObject.class, DeleteGCSObject.class,
PutBigQueryStream.class})
+
+@WritesAttributes({
+@WritesAttribute(attribute = BigQueryAttributes.DATASET_ATTR,
description = BigQueryAttributes.DATASET_DESC),
+@WritesAttribute(attribute = BigQueryAttributes.TABLE_NAME_ATTR,
description = BigQueryAttributes.TABLE_NAME_DESC),
+@WritesAttribute(attribute = BigQueryAttributes.TABLE_SCHEMA_ATTR,
description = BigQueryAttributes.TABLE_SCHEMA_DESC),
+@WritesAttribute(attribute = BigQueryAttributes.SOURCE_TYPE_ATTR,
description = BigQueryAttributes.SOURCE_TYPE_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.IGNORE_UNKNOWN_ATTR, description =
BigQueryAttributes.IGNORE_UNKNOWN_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.CREATE_DISPOSITION_ATTR, description =
BigQueryAttributes.CREATE_DISPOSITION_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.WRITE_DISPOSITION_ATTR, description =
BigQueryAttributes.WRITE_DISPOSITION_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.MAX_BADRECORDS_ATTR, description =
BigQueryAttributes.MAX_BADRECORDS_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.JOB_CREATE_TIME_ATTR, description =
BigQueryAttributes.JOB_CREATE_TIME_DESC),
+@WritesAttribute(attribute = BigQueryAttributes.JOB_END_TIME_ATTR,
description = BigQueryAttributes.JOB_END_TIME_DESC),
+
[GitHub] nifi pull request #2682: NIFI-4731: BQ Processors and GCP library update.
Github user danieljimenez commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2682#discussion_r206191230
--- Diff:
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/bigquery/PutBigQueryBatch.java
---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.gcp.bigquery;
+
+import com.google.cloud.bigquery.BigQuery;
+import com.google.cloud.bigquery.FormatOptions;
+import com.google.cloud.bigquery.Job;
+import com.google.cloud.bigquery.JobInfo;
+import com.google.cloud.bigquery.Schema;
+import com.google.cloud.bigquery.TableDataWriteChannel;
+import com.google.cloud.bigquery.TableId;
+import com.google.cloud.bigquery.WriteChannelConfiguration;
+import com.google.common.collect.ImmutableList;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.LogLevel;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.gcp.storage.DeleteGCSObject;
+import org.apache.nifi.processors.gcp.storage.PutGCSObject;
+import org.apache.nifi.util.StringUtils;
+
+import java.nio.ByteBuffer;
+import java.nio.channels.Channels;
+import java.nio.channels.ReadableByteChannel;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * A processor for batch loading data into a Google BigQuery table
+ */
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@Tags({"google", "google cloud", "bq", "bigquery"})
+@CapabilityDescription("Batch loads flow files to a Google BigQuery
table.")
+@SeeAlso({PutGCSObject.class, DeleteGCSObject.class,
PutBigQueryStream.class})
+
+@WritesAttributes({
+@WritesAttribute(attribute = BigQueryAttributes.DATASET_ATTR,
description = BigQueryAttributes.DATASET_DESC),
+@WritesAttribute(attribute = BigQueryAttributes.TABLE_NAME_ATTR,
description = BigQueryAttributes.TABLE_NAME_DESC),
+@WritesAttribute(attribute = BigQueryAttributes.TABLE_SCHEMA_ATTR,
description = BigQueryAttributes.TABLE_SCHEMA_DESC),
+@WritesAttribute(attribute = BigQueryAttributes.SOURCE_TYPE_ATTR,
description = BigQueryAttributes.SOURCE_TYPE_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.IGNORE_UNKNOWN_ATTR, description =
BigQueryAttributes.IGNORE_UNKNOWN_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.CREATE_DISPOSITION_ATTR, description =
BigQueryAttributes.CREATE_DISPOSITION_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.WRITE_DISPOSITION_ATTR, description =
BigQueryAttributes.WRITE_DISPOSITION_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.MAX_BADRECORDS_ATTR, description =
BigQueryAttributes.MAX_BADRECORDS_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.JOB_CREATE_TIME_ATTR, description =
BigQueryAttributes.JOB_CREATE_TIME_DESC),
+@WritesAttribute(attribute = BigQueryAttributes.JOB_END_TIME_ATTR,
description = BigQueryAttributes.JOB_END_TIME_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.JOB_START_TIME_ATTR, description =
BigQueryAttributes.JOB_START_TIME_DESC),
+@WritesAttribute(attribute = BigQueryAttributes.JOB_LINK_ATTR,
description =
[jira] [Commented] (NIFI-5469) Edits needed for LDAP and Kerberos login identity provider sections in Admin Guide
[ https://issues.apache.org/jira/browse/NIFI-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561987#comment-16561987 ] ASF GitHub Bot commented on NIFI-5469: -- Github user andrewmlim commented on a diff in the pull request: https://github.com/apache/nifi/pull/2922#discussion_r206166816 --- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc --- @@ -2434,82 +2538,82 @@ Before you begin, confirm that: + | |*Required Information*|*Description* -|Source ZooKeeper hostname (*sourceHostname*)|The hostname must be one of the hosts running in the ZooKeeper ensemble, which can be found in /conf/zookeeper.properties. Any of the hostnames declared in the *server.N* properties can be used. -|Destination ZooKeeper hostname (*destinationHostname*)|The hostname must be one of the hosts running in the ZooKeeper ensemble, which can be found in /conf/zookeeper.properties. Any of the hostnames declared in the *server.N* properties can be used. -|Source ZooKeeper port (*sourceClientPort*)|This can be found in *zookeeper.properties* of the /conf/zookeeper.properties. The port is specified in the *clientPort* property. -|Destination ZooKeeper port (*destinationClientPort*)|This can be found in *zookeeper.properties* of the /conf/zookeeper.properties. The port is specified in the *clientPort* property. +|Source ZooKeeper hostname (*sourceHostname*)|The hostname must be one of the hosts running in the ZooKeeper ensemble, which can be found in _/conf/zookeeper.properties_. Any of the hostnames declared in the *server.N* properties can be used. --- End diff -- Going to also remove the double reference to zookeeper.properties: This can be found in zookeeper.properties of the /conf/zookeeper.properties. > Edits needed for LDAP and Kerberos login identity provider sections in Admin > Guide > -- > > Key: NIFI-5469 > URL: https://issues.apache.org/jira/browse/NIFI-5469 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation Website >Reporter: Andrew Lim >Assignee: Andrew Lim >Priority: Minor > > Going through the Authentication and Authorization sections of the Admin > Guide, I noticed the following improvements could be made: > * Removed “Kerberos Config File” property from kerberos-provider login > identity provider (this was done because the same property exists in > nifi.properties) > * Corrected the "LDAP-based Users/Groups Referencing User Attribute” login > identity provider example to refer to “member uid" > * Added titles to login identity provider examples for improved > readability/search > * Changed UserGroupProvider property examples from bulleted lists to tables > Also, text formatting for references to config files, directories, etc. need > to be made consistent. For example, config files like _nifi.properties_, > _authorizers.xml_ should be italicized. Directories, properties and default > values for properties should be monospaced. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2922: NIFI-5469 Update Kerberos and LDAP login identity p...
Github user andrewmlim commented on a diff in the pull request: https://github.com/apache/nifi/pull/2922#discussion_r206166816 --- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc --- @@ -2434,82 +2538,82 @@ Before you begin, confirm that: + | |*Required Information*|*Description* -|Source ZooKeeper hostname (*sourceHostname*)|The hostname must be one of the hosts running in the ZooKeeper ensemble, which can be found in /conf/zookeeper.properties. Any of the hostnames declared in the *server.N* properties can be used. -|Destination ZooKeeper hostname (*destinationHostname*)|The hostname must be one of the hosts running in the ZooKeeper ensemble, which can be found in /conf/zookeeper.properties. Any of the hostnames declared in the *server.N* properties can be used. -|Source ZooKeeper port (*sourceClientPort*)|This can be found in *zookeeper.properties* of the /conf/zookeeper.properties. The port is specified in the *clientPort* property. -|Destination ZooKeeper port (*destinationClientPort*)|This can be found in *zookeeper.properties* of the /conf/zookeeper.properties. The port is specified in the *clientPort* property. +|Source ZooKeeper hostname (*sourceHostname*)|The hostname must be one of the hosts running in the ZooKeeper ensemble, which can be found in _/conf/zookeeper.properties_. Any of the hostnames declared in the *server.N* properties can be used. --- End diff -- Going to also remove the double reference to zookeeper.properties: This can be found in zookeeper.properties of the /conf/zookeeper.properties. ---
[jira] [Commented] (NIFI-5327) NetFlow Processors
[ https://issues.apache.org/jira/browse/NIFI-5327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561986#comment-16561986 ] ASF GitHub Bot commented on NIFI-5327: -- Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/nifi/pull/2820#discussion_r206164076 --- Diff: nifi-nar-bundles/nifi-network-bundle/nifi-network-utils/src/main/java/org/apache/nifi/processors/network/parser/Netflowv5Parser.java --- @@ -29,6 +32,7 @@ private static final int SHORT_TYPE = 0; private static final int INTEGER_TYPE = 1; private static final int LONG_TYPE = 2; --- End diff -- So netflow5 doesn't support IPV6? If not then cool > NetFlow Processors > -- > > Key: NIFI-5327 > URL: https://issues.apache.org/jira/browse/NIFI-5327 > Project: Apache NiFi > Issue Type: New Feature > Components: Core Framework >Affects Versions: 1.6.0 >Reporter: Prashanth Venkatesan >Assignee: Prashanth Venkatesan >Priority: Major > > As network traffic data scopes for the big data use case, would like NiFi to > have processors to support parsing of those protocols. > Netflow is a protocol introduced by Cisco that provides the ability to > collect IP network traffic as it enters or exits an interface and is > described in detail in here: > [https://www.cisco.com/c/en/us/td/docs/net_mgmt/netflow_collection_engine/3-6/user/guide/format.html] > > Currently, I have created the following processor: > *ParseNetflowv5*: Parses the ingress netflowv5 bytes and ingest as either > NiFi flowfile attributes or as a JSON content. This also sends > one-time-template. > > Further ahead, we can add many processor specific to network protocols in > this nar bundle. > I will create a pull request. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/nifi/pull/2820#discussion_r206164076 --- Diff: nifi-nar-bundles/nifi-network-bundle/nifi-network-utils/src/main/java/org/apache/nifi/processors/network/parser/Netflowv5Parser.java --- @@ -29,6 +32,7 @@ private static final int SHORT_TYPE = 0; private static final int INTEGER_TYPE = 1; private static final int LONG_TYPE = 2; --- End diff -- So netflow5 doesn't support IPV6? If not then cool ---
[jira] [Commented] (NIFI-4731) BigQuery processors
[
https://issues.apache.org/jira/browse/NIFI-4731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561984#comment-16561984
]
ASF GitHub Bot commented on NIFI-4731:
--
Github user danieljimenez commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2682#discussion_r206158466
--- Diff:
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/bigquery/PutBigQueryBatch.java
---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.gcp.bigquery;
+
+import com.google.cloud.bigquery.BigQuery;
+import com.google.cloud.bigquery.FormatOptions;
+import com.google.cloud.bigquery.Job;
+import com.google.cloud.bigquery.JobInfo;
+import com.google.cloud.bigquery.Schema;
+import com.google.cloud.bigquery.TableDataWriteChannel;
+import com.google.cloud.bigquery.TableId;
+import com.google.cloud.bigquery.WriteChannelConfiguration;
+import com.google.common.collect.ImmutableList;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.LogLevel;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.gcp.storage.DeleteGCSObject;
+import org.apache.nifi.processors.gcp.storage.PutGCSObject;
+import org.apache.nifi.util.StringUtils;
+
+import java.nio.ByteBuffer;
+import java.nio.channels.Channels;
+import java.nio.channels.ReadableByteChannel;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * A processor for batch loading data into a Google BigQuery table
+ */
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@Tags({"google", "google cloud", "bq", "bigquery"})
+@CapabilityDescription("Batch loads flow files to a Google BigQuery
table.")
+@SeeAlso({PutGCSObject.class, DeleteGCSObject.class,
PutBigQueryStream.class})
+
+@WritesAttributes({
+@WritesAttribute(attribute = BigQueryAttributes.DATASET_ATTR,
description = BigQueryAttributes.DATASET_DESC),
+@WritesAttribute(attribute = BigQueryAttributes.TABLE_NAME_ATTR,
description = BigQueryAttributes.TABLE_NAME_DESC),
+@WritesAttribute(attribute = BigQueryAttributes.TABLE_SCHEMA_ATTR,
description = BigQueryAttributes.TABLE_SCHEMA_DESC),
+@WritesAttribute(attribute = BigQueryAttributes.SOURCE_TYPE_ATTR,
description = BigQueryAttributes.SOURCE_TYPE_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.IGNORE_UNKNOWN_ATTR, description =
BigQueryAttributes.IGNORE_UNKNOWN_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.CREATE_DISPOSITION_ATTR, description =
BigQueryAttributes.CREATE_DISPOSITION_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.WRITE_DISPOSITION_ATTR, description =
BigQueryAttributes.WRITE_DISPOSITION_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.MAX_BADRECORDS_ATTR, description =
BigQueryAttributes.MAX_BADRECORDS_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.JOB_CREATE_TIME_ATTR, description =
BigQueryAttributes.JOB_CREATE_TIME_DESC),
+@WritesAttribute(attribute = BigQueryAttributes.JOB_END_TIME_ATTR,
description = BigQueryAttributes.JOB_END_TIME_DESC),
+
[GitHub] nifi pull request #2682: NIFI-4731: BQ Processors and GCP library update.
Github user danieljimenez commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2682#discussion_r206158466
--- Diff:
nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-processors/src/main/java/org/apache/nifi/processors/gcp/bigquery/PutBigQueryBatch.java
---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.gcp.bigquery;
+
+import com.google.cloud.bigquery.BigQuery;
+import com.google.cloud.bigquery.FormatOptions;
+import com.google.cloud.bigquery.Job;
+import com.google.cloud.bigquery.JobInfo;
+import com.google.cloud.bigquery.Schema;
+import com.google.cloud.bigquery.TableDataWriteChannel;
+import com.google.cloud.bigquery.TableId;
+import com.google.cloud.bigquery.WriteChannelConfiguration;
+import com.google.common.collect.ImmutableList;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.LogLevel;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.gcp.storage.DeleteGCSObject;
+import org.apache.nifi.processors.gcp.storage.PutGCSObject;
+import org.apache.nifi.util.StringUtils;
+
+import java.nio.ByteBuffer;
+import java.nio.channels.Channels;
+import java.nio.channels.ReadableByteChannel;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * A processor for batch loading data into a Google BigQuery table
+ */
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@Tags({"google", "google cloud", "bq", "bigquery"})
+@CapabilityDescription("Batch loads flow files to a Google BigQuery
table.")
+@SeeAlso({PutGCSObject.class, DeleteGCSObject.class,
PutBigQueryStream.class})
+
+@WritesAttributes({
+@WritesAttribute(attribute = BigQueryAttributes.DATASET_ATTR,
description = BigQueryAttributes.DATASET_DESC),
+@WritesAttribute(attribute = BigQueryAttributes.TABLE_NAME_ATTR,
description = BigQueryAttributes.TABLE_NAME_DESC),
+@WritesAttribute(attribute = BigQueryAttributes.TABLE_SCHEMA_ATTR,
description = BigQueryAttributes.TABLE_SCHEMA_DESC),
+@WritesAttribute(attribute = BigQueryAttributes.SOURCE_TYPE_ATTR,
description = BigQueryAttributes.SOURCE_TYPE_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.IGNORE_UNKNOWN_ATTR, description =
BigQueryAttributes.IGNORE_UNKNOWN_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.CREATE_DISPOSITION_ATTR, description =
BigQueryAttributes.CREATE_DISPOSITION_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.WRITE_DISPOSITION_ATTR, description =
BigQueryAttributes.WRITE_DISPOSITION_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.MAX_BADRECORDS_ATTR, description =
BigQueryAttributes.MAX_BADRECORDS_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.JOB_CREATE_TIME_ATTR, description =
BigQueryAttributes.JOB_CREATE_TIME_DESC),
+@WritesAttribute(attribute = BigQueryAttributes.JOB_END_TIME_ATTR,
description = BigQueryAttributes.JOB_END_TIME_DESC),
+@WritesAttribute(attribute =
BigQueryAttributes.JOB_START_TIME_ATTR, description =
BigQueryAttributes.JOB_START_TIME_DESC),
+@WritesAttribute(attribute = BigQueryAttributes.JOB_LINK_ATTR,
description =
[jira] [Commented] (NIFI-4535) Set the Page Title to the name of the Root Process Group
[ https://issues.apache.org/jira/browse/NIFI-4535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561970#comment-16561970 ] ASF GitHub Bot commented on NIFI-4535: -- Github user mcgilman commented on the issue: https://github.com/apache/nifi/pull/2899 @patricker Just had another look. I think there is one case that still hasn't been addressed. When a user has been granted permissions to the root group, the document title does update to the root group name. A check has been introduced that will only change the document title if the user has access to the root group. However, if the user does have access to the root group and those permissions are removed the document title does not change back to the previous value. It retains the name of the root group when they last loaded the page. We should be restoring the default value. The default value is loaded through the about endpoint [1]. It would probably make sense to store that value when the page loads so we can restore it in an else clause in your PR. Let me know if you any assistance here. I'm happy to help. [1] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/canvas/controllers/nf-ng-canvas-global-menu-controller.js#L318 > Set the Page Title to the name of the Root Process Group > > > Key: NIFI-4535 > URL: https://issues.apache.org/jira/browse/NIFI-4535 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Reporter: Peter Wicks >Assignee: Peter Wicks >Priority: Minor > Fix For: 1.8.0 > > > NiFi's UI has a hard coded page title of NiFi. I have many servers and it's > hard to keep track of the tabs in Chrome. > Please change the Title of the page so it matches the name of the Root > processor group. This way I can name the root group by server/instance, and > easily identify the right tab/window. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2899: NIFI-4535 Only update Page Title to root flow name when us...
Github user mcgilman commented on the issue: https://github.com/apache/nifi/pull/2899 @patricker Just had another look. I think there is one case that still hasn't been addressed. When a user has been granted permissions to the root group, the document title does update to the root group name. A check has been introduced that will only change the document title if the user has access to the root group. However, if the user does have access to the root group and those permissions are removed the document title does not change back to the previous value. It retains the name of the root group when they last loaded the page. We should be restoring the default value. The default value is loaded through the about endpoint [1]. It would probably make sense to store that value when the page loads so we can restore it in an else clause in your PR. Let me know if you any assistance here. I'm happy to help. [1] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/canvas/controllers/nf-ng-canvas-global-menu-controller.js#L318 ---
[jira] [Commented] (NIFI-5327) NetFlow Processors
[ https://issues.apache.org/jira/browse/NIFI-5327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561950#comment-16561950 ] ASF GitHub Bot commented on NIFI-5327: -- Github user PrashanthVenkatesan commented on a diff in the pull request: https://github.com/apache/nifi/pull/2820#discussion_r206144210 --- Diff: nifi-nar-bundles/nifi-network-bundle/nifi-network-utils/src/main/java/org/apache/nifi/processors/network/parser/Netflowv5Parser.java --- @@ -29,6 +32,7 @@ private static final int SHORT_TYPE = 0; private static final int INTEGER_TYPE = 1; private static final int LONG_TYPE = 2; --- End diff -- to my knowledge, there is no IPV6 format in the parsed output. > NetFlow Processors > -- > > Key: NIFI-5327 > URL: https://issues.apache.org/jira/browse/NIFI-5327 > Project: Apache NiFi > Issue Type: New Feature > Components: Core Framework >Affects Versions: 1.6.0 >Reporter: Prashanth Venkatesan >Assignee: Prashanth Venkatesan >Priority: Major > > As network traffic data scopes for the big data use case, would like NiFi to > have processors to support parsing of those protocols. > Netflow is a protocol introduced by Cisco that provides the ability to > collect IP network traffic as it enters or exits an interface and is > described in detail in here: > [https://www.cisco.com/c/en/us/td/docs/net_mgmt/netflow_collection_engine/3-6/user/guide/format.html] > > Currently, I have created the following processor: > *ParseNetflowv5*: Parses the ingress netflowv5 bytes and ingest as either > NiFi flowfile attributes or as a JSON content. This also sends > one-time-template. > > Further ahead, we can add many processor specific to network protocols in > this nar bundle. > I will create a pull request. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user PrashanthVenkatesan commented on a diff in the pull request: https://github.com/apache/nifi/pull/2820#discussion_r206144210 --- Diff: nifi-nar-bundles/nifi-network-bundle/nifi-network-utils/src/main/java/org/apache/nifi/processors/network/parser/Netflowv5Parser.java --- @@ -29,6 +32,7 @@ private static final int SHORT_TYPE = 0; private static final int INTEGER_TYPE = 1; private static final int LONG_TYPE = 2; --- End diff -- to my knowledge, there is no IPV6 format in the parsed output. ---
[jira] [Updated] (MINIFICPP-579) Break curl dependency within expression language.
[ https://issues.apache.org/jira/browse/MINIFICPP-579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mr TheSegfault updated MINIFICPP-579: - Issue Type: Bug (was: Question) > Break curl dependency within expression language. > -- > > Key: MINIFICPP-579 > URL: https://issues.apache.org/jira/browse/MINIFICPP-579 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Critical > Fix For: 0.6.0 > > > If a user requests that we disable curl we should not build or load any curl > dependencies; however, there is a link in Expression language. We can either > move that code into http-curl ( preferable ) or disable expression language > when curl is disabled entirely. Further logic is (enable all or (when exp is > enabled) or (when curl enabled)) – this should be changed to facilitate the > changes in this ticket. > > Created to facilitate the merge of MINIFICPP-559 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MINIFICPP-579) Break curl dependency within expression language.
[ https://issues.apache.org/jira/browse/MINIFICPP-579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mr TheSegfault updated MINIFICPP-579: - Description: If a user requests that we disable curl we should not build or load any curl dependencies; however, there is a link in Expression language. We can either move that code into http-curl ( preferable ) or disable expression language when curl is disabled entirely. Further logic is (enable all or (when exp is enabled) or (when curl enabled)) – this should be changed to facilitate the changes in this ticket. Created to facilitate the merge of MINIFICPP-559 was:If a user requests that we disable curl we should not build or load any curl dependencies; however, there is a link in Expression language. We can either move that code into http-curl ( preferable ) or disable expression language when curl is disabled entirely. Further logic is (enable all or (when exp is enabled) or (when Created to facilitate the merge of MINIFICPP-559 > Break curl dependency within expression language. > -- > > Key: MINIFICPP-579 > URL: https://issues.apache.org/jira/browse/MINIFICPP-579 > Project: NiFi MiNiFi C++ > Issue Type: Question >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Critical > Fix For: 0.6.0 > > > If a user requests that we disable curl we should not build or load any curl > dependencies; however, there is a link in Expression language. We can either > move that code into http-curl ( preferable ) or disable expression language > when curl is disabled entirely. Further logic is (enable all or (when exp is > enabled) or (when curl enabled)) – this should be changed to facilitate the > changes in this ticket. > > Created to facilitate the merge of MINIFICPP-559 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MINIFICPP-579) Break curl dependency within expression language.
[ https://issues.apache.org/jira/browse/MINIFICPP-579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mr TheSegfault updated MINIFICPP-579: - Description: If a user requests that we disable curl we should not build or load any curl dependencies; however, there is a link in Expression language. We can either move that code into http-curl ( preferable ) or disable expression language when curl is disabled entirely. Further logic is (enable all or (when exp is enabled) or (when Created to facilitate the merge of MINIFICPP-559 (was: If a user requests that we disable curl we should not build or load any curl dependencies; however, there is a link in Expression language. We can either move that code into http-curl ( preferable ) or disable expression language when curl is disabled entirely. Created to facilitate the merge of MINIFICPP-559) > Break curl dependency within expression language. > -- > > Key: MINIFICPP-579 > URL: https://issues.apache.org/jira/browse/MINIFICPP-579 > Project: NiFi MiNiFi C++ > Issue Type: Question >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Critical > Fix For: 0.6.0 > > > If a user requests that we disable curl we should not build or load any curl > dependencies; however, there is a link in Expression language. We can either > move that code into http-curl ( preferable ) or disable expression language > when curl is disabled entirely. Further logic is (enable all or (when exp is > enabled) or (when Created to facilitate the merge of MINIFICPP-559 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MINIFICPP-559) cURL built-in build fails in cross-compilation setup
[ https://issues.apache.org/jira/browse/MINIFICPP-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561939#comment-16561939 ] ASF GitHub Bot commented on MINIFICPP-559: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/374 > cURL built-in build fails in cross-compilation setup > > > Key: MINIFICPP-559 > URL: https://issues.apache.org/jira/browse/MINIFICPP-559 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Andrew Christianson >Assignee: Andrew Christianson >Priority: Major > > cURL internal build & package find fails when cross compiling. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #374: MINIFICPP-559 Move cURL external build up...
Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/374 ---
[jira] [Commented] (MINIFICPP-427) Add headers support in PublishKafka
[ https://issues.apache.org/jira/browse/MINIFICPP-427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561933#comment-16561933 ] ASF GitHub Bot commented on MINIFICPP-427: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/360 > Add headers support in PublishKafka > --- > > Key: MINIFICPP-427 > URL: https://issues.apache.org/jira/browse/MINIFICPP-427 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Pierre Villard >Assignee: Dustin Rodrigues >Priority: Major > > Would be nice to add headers support in PublishKafka (assuming Kafka 0.11+) > in order to preserve attributes of the flow files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #360: MINIFICPP-427 - add PublishKafka headers ...
Github user asfgit closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/360 ---
[jira] [Created] (MINIFICPP-584) PythonExecuteScriptTests appears to time out travis MAC builds
Mr TheSegfault created MINIFICPP-584: Summary: PythonExecuteScriptTests appears to time out travis MAC builds Key: MINIFICPP-584 URL: https://issues.apache.org/jira/browse/MINIFICPP-584 Project: NiFi MiNiFi C++ Issue Type: Bug Reporter: Mr TheSegfault Fix For: 0.6.0 Travis builds are timing out on PythonExecuteScriptTests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MINIFICPP-582) Readme constraints not defined in bootstrap
[ https://issues.apache.org/jira/browse/MINIFICPP-582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mr TheSegfault updated MINIFICPP-582: - Fix Version/s: 0.6.0 > Readme constraints not defined in bootstrap > --- > > Key: MINIFICPP-582 > URL: https://issues.apache.org/jira/browse/MINIFICPP-582 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Mr TheSegfault >Priority: Blocker > Fix For: 0.6.0 > > > MINIFICPP-427 and others have added some constraints that aren't captured in > the bootstrap. We can make these changes in this ticket -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (MINIFICPP-583) Cannot disable curl properly
Mr TheSegfault created MINIFICPP-583: Summary: Cannot disable curl properly Key: MINIFICPP-583 URL: https://issues.apache.org/jira/browse/MINIFICPP-583 Project: NiFi MiNiFi C++ Issue Type: Bug Reporter: Mr TheSegfault Fix For: 0.6.0 Build fails with curl disabled due to civet always being enabled. We should more intelligently allow this to be disabled. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MINIFICPP-427) Add headers support in PublishKafka
[ https://issues.apache.org/jira/browse/MINIFICPP-427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561925#comment-16561925 ] ASF GitHub Bot commented on MINIFICPP-427: -- Github user phrocker commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/360 @dtrodrigues thanks for pointing out the readme update for bustache. I'll add that via MINIFICPP-582 and merge this. > Add headers support in PublishKafka > --- > > Key: MINIFICPP-427 > URL: https://issues.apache.org/jira/browse/MINIFICPP-427 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Pierre Villard >Assignee: Dustin Rodrigues >Priority: Major > > Would be nice to add headers support in PublishKafka (assuming Kafka 0.11+) > in order to preserve attributes of the flow files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (MINIFICPP-582) Readme constraints not defined in bootstrap
Mr TheSegfault created MINIFICPP-582: Summary: Readme constraints not defined in bootstrap Key: MINIFICPP-582 URL: https://issues.apache.org/jira/browse/MINIFICPP-582 Project: NiFi MiNiFi C++ Issue Type: Bug Reporter: Mr TheSegfault MINIFICPP-427 and others have added some constraints that aren't captured in the bootstrap. We can make these changes in this ticket -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp issue #360: MINIFICPP-427 - add PublishKafka headers support
Github user phrocker commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/360 @dtrodrigues thanks for pointing out the readme update for bustache. I'll add that via MINIFICPP-582 and merge this. ---
[jira] [Updated] (NIFI-5342) CaptureChangeMySQL didn't escape database name
[ https://issues.apache.org/jira/browse/NIFI-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-5342: --- Affects Version/s: (was: 1.6.0) Status: Patch Available (was: Open) > CaptureChangeMySQL didn't escape database name > -- > > Key: NIFI-5342 > URL: https://issues.apache.org/jira/browse/NIFI-5342 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Tanapol Nearunchorn >Assignee: Uday Kale >Priority: Major > > CaptureChangeMySQL didn't escape database name as you can see: > [https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-cdc/nifi-cdc-mysql-bundle/nifi-cdc-mysql-processors/src/main/java/org/apache/nifi/cdc/mysql/processors/CaptureChangeMySQL.java#L1003] > > And also with the line 1004, which query from a table. > It can cause problems when there is database or table name that contains a > reserved keyword. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5327) NetFlow Processors
[ https://issues.apache.org/jira/browse/NIFI-5327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561899#comment-16561899 ] ASF GitHub Bot commented on NIFI-5327: -- Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/nifi/pull/2820#discussion_r206125096 --- Diff: nifi-nar-bundles/nifi-network-bundle/nifi-network-utils/src/main/java/org/apache/nifi/processors/network/parser/Netflowv5Parser.java --- @@ -29,6 +32,7 @@ private static final int SHORT_TYPE = 0; private static final int INTEGER_TYPE = 1; private static final int LONG_TYPE = 2; --- End diff -- I hate to ask, but is there no IPV 6 that could be in there? > NetFlow Processors > -- > > Key: NIFI-5327 > URL: https://issues.apache.org/jira/browse/NIFI-5327 > Project: Apache NiFi > Issue Type: New Feature > Components: Core Framework >Affects Versions: 1.6.0 >Reporter: Prashanth Venkatesan >Assignee: Prashanth Venkatesan >Priority: Major > > As network traffic data scopes for the big data use case, would like NiFi to > have processors to support parsing of those protocols. > Netflow is a protocol introduced by Cisco that provides the ability to > collect IP network traffic as it enters or exits an interface and is > described in detail in here: > [https://www.cisco.com/c/en/us/td/docs/net_mgmt/netflow_collection_engine/3-6/user/guide/format.html] > > Currently, I have created the following processor: > *ParseNetflowv5*: Parses the ingress netflowv5 bytes and ingest as either > NiFi flowfile attributes or as a JSON content. This also sends > one-time-template. > > Further ahead, we can add many processor specific to network protocols in > this nar bundle. > I will create a pull request. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2820: NIFI-5327 Adding Netflowv5 protocol parser
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/nifi/pull/2820#discussion_r206125096 --- Diff: nifi-nar-bundles/nifi-network-bundle/nifi-network-utils/src/main/java/org/apache/nifi/processors/network/parser/Netflowv5Parser.java --- @@ -29,6 +32,7 @@ private static final int SHORT_TYPE = 0; private static final int INTEGER_TYPE = 1; private static final int LONG_TYPE = 2; --- End diff -- I hate to ask, but is there no IPV 6 that could be in there? ---
[jira] [Commented] (NIFI-5468) Add toolkit to NiFi docker image and use zip archive
[
https://issues.apache.org/jira/browse/NIFI-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561834#comment-16561834
]
ASF GitHub Bot commented on NIFI-5468:
--
Github user pepov commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2921#discussion_r206107302
--- Diff: nifi-docker/dockerhub/Dockerfile ---
@@ -23,11 +23,15 @@ LABEL site="https://nifi.apache.org;
ARG UID=1000
ARG GID=1000
ARG NIFI_VERSION=1.8.0
-ARG MIRROR=https://archive.apache.org/dist
+ARG MIRROR_BASE_URL=https://archive.apache.org/dist
+ARG ARCHIVE_BASE_URL=https://archive.apache.org/dist
+
+ENV NIFI_BASE_DIR=/opt/nifi
+ENV NIFI_HOME=${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}
+
+ENV
NIFI_BINARY_PATH=${NIFI_BINARY_PATH:-/nifi/${NIFI_VERSION}/nifi-${NIFI_VERSION}-bin.zip}
--- End diff --
it's the path behind the domain in the url, not on the filesystem
> Add toolkit to NiFi docker image and use zip archive
>
>
> Key: NIFI-5468
> URL: https://issues.apache.org/jira/browse/NIFI-5468
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Docker
>Affects Versions: 1.8.0
>Reporter: Peter Wilcsinszky
>Assignee: Peter Wilcsinszky
>Priority: Major
> Fix For: 1.8.0
>
>
> NiFi toolkit is an essential tool for a NiFi installation and makes sense to
> include it by default in the NiFi Docker image.
> Using the zip archive is a step to make possible switching completely to the
> zip release and avoid generating the tar.gz anymore. For the discussion see
> http://apache-nifi-developer-list.39713.n7.nabble.com/DISCUSS-Tar-Gzip-vs-Zip-td19175.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2921: NIFI-5468 Add toolkit to NiFi docker image and use ...
Github user pepov commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2921#discussion_r206107302
--- Diff: nifi-docker/dockerhub/Dockerfile ---
@@ -23,11 +23,15 @@ LABEL site="https://nifi.apache.org;
ARG UID=1000
ARG GID=1000
ARG NIFI_VERSION=1.8.0
-ARG MIRROR=https://archive.apache.org/dist
+ARG MIRROR_BASE_URL=https://archive.apache.org/dist
+ARG ARCHIVE_BASE_URL=https://archive.apache.org/dist
+
+ENV NIFI_BASE_DIR=/opt/nifi
+ENV NIFI_HOME=${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}
+
+ENV
NIFI_BINARY_PATH=${NIFI_BINARY_PATH:-/nifi/${NIFI_VERSION}/nifi-${NIFI_VERSION}-bin.zip}
--- End diff --
it's the path behind the domain in the url, not on the filesystem
---
[jira] [Commented] (NIFI-5468) Add toolkit to NiFi docker image and use zip archive
[ https://issues.apache.org/jira/browse/NIFI-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561832#comment-16561832 ] ASF GitHub Bot commented on NIFI-5468: -- Github user pepov commented on a diff in the pull request: https://github.com/apache/nifi/pull/2921#discussion_r206107007 --- Diff: nifi-docker/dockerhub/pom.xml --- @@ -39,7 +39,7 @@ 1000 1000 -1.6.0 +1.7.0 --- End diff -- unfortunately it cannot, since there is no released binary for 1.8 yet > Add toolkit to NiFi docker image and use zip archive > > > Key: NIFI-5468 > URL: https://issues.apache.org/jira/browse/NIFI-5468 > Project: Apache NiFi > Issue Type: Improvement > Components: Docker >Affects Versions: 1.8.0 >Reporter: Peter Wilcsinszky >Assignee: Peter Wilcsinszky >Priority: Major > Fix For: 1.8.0 > > > NiFi toolkit is an essential tool for a NiFi installation and makes sense to > include it by default in the NiFi Docker image. > Using the zip archive is a step to make possible switching completely to the > zip release and avoid generating the tar.gz anymore. For the discussion see > http://apache-nifi-developer-list.39713.n7.nabble.com/DISCUSS-Tar-Gzip-vs-Zip-td19175.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5468) Add toolkit to NiFi docker image and use zip archive
[
https://issues.apache.org/jira/browse/NIFI-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561833#comment-16561833
]
ASF GitHub Bot commented on NIFI-5468:
--
Github user pepov commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2921#discussion_r206107026
--- Diff: nifi-docker/dockerhub/pom.xml ---
@@ -61,7 +61,7 @@
${project.version}-dockerhub
-1.6.0
+1.7.0
--- End diff --
unfortunately it cannot, since there is no released binary for 1.8 yet
> Add toolkit to NiFi docker image and use zip archive
>
>
> Key: NIFI-5468
> URL: https://issues.apache.org/jira/browse/NIFI-5468
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Docker
>Affects Versions: 1.8.0
>Reporter: Peter Wilcsinszky
>Assignee: Peter Wilcsinszky
>Priority: Major
> Fix For: 1.8.0
>
>
> NiFi toolkit is an essential tool for a NiFi installation and makes sense to
> include it by default in the NiFi Docker image.
> Using the zip archive is a step to make possible switching completely to the
> zip release and avoid generating the tar.gz anymore. For the discussion see
> http://apache-nifi-developer-list.39713.n7.nabble.com/DISCUSS-Tar-Gzip-vs-Zip-td19175.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2921: NIFI-5468 Add toolkit to NiFi docker image and use ...
Github user pepov commented on a diff in the pull request: https://github.com/apache/nifi/pull/2921#discussion_r206107007 --- Diff: nifi-docker/dockerhub/pom.xml --- @@ -39,7 +39,7 @@ 1000 1000 -1.6.0 +1.7.0 --- End diff -- unfortunately it cannot, since there is no released binary for 1.8 yet ---
[GitHub] nifi pull request #2921: NIFI-5468 Add toolkit to NiFi docker image and use ...
Github user pepov commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2921#discussion_r206107026
--- Diff: nifi-docker/dockerhub/pom.xml ---
@@ -61,7 +61,7 @@
${project.version}-dockerhub
-1.6.0
+1.7.0
--- End diff --
unfortunately it cannot, since there is no released binary for 1.8 yet
---
[GitHub] nifi pull request #2921: NIFI-5468 Add toolkit to NiFi docker image and use ...
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2921#discussion_r206105937
--- Diff: nifi-docker/dockerhub/Dockerfile ---
@@ -23,11 +23,15 @@ LABEL site="https://nifi.apache.org;
ARG UID=1000
ARG GID=1000
ARG NIFI_VERSION=1.8.0
-ARG MIRROR=https://archive.apache.org/dist
+ARG MIRROR_BASE_URL=https://archive.apache.org/dist
+ARG ARCHIVE_BASE_URL=https://archive.apache.org/dist
+
+ENV NIFI_BASE_DIR=/opt/nifi
+ENV NIFI_HOME=${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}
+
+ENV
NIFI_BINARY_PATH=${NIFI_BINARY_PATH:-/nifi/${NIFI_VERSION}/nifi-${NIFI_VERSION}-bin.zip}
--- End diff --
Did you forget to add `/opt` here to this? `PATH:-/nifi/${NIFI_VERSION}`?
From the diff, it doesn't look like `/nifi` would be a valid starting path.
---
[jira] [Commented] (NIFI-5468) Add toolkit to NiFi docker image and use zip archive
[
https://issues.apache.org/jira/browse/NIFI-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561831#comment-16561831
]
ASF GitHub Bot commented on NIFI-5468:
--
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2921#discussion_r206106274
--- Diff: nifi-docker/dockerhub/pom.xml ---
@@ -61,7 +61,7 @@
${project.version}-dockerhub
-1.6.0
+1.7.0
--- End diff --
Should be 1.8.0 now.
> Add toolkit to NiFi docker image and use zip archive
>
>
> Key: NIFI-5468
> URL: https://issues.apache.org/jira/browse/NIFI-5468
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Docker
>Affects Versions: 1.8.0
>Reporter: Peter Wilcsinszky
>Assignee: Peter Wilcsinszky
>Priority: Major
> Fix For: 1.8.0
>
>
> NiFi toolkit is an essential tool for a NiFi installation and makes sense to
> include it by default in the NiFi Docker image.
> Using the zip archive is a step to make possible switching completely to the
> zip release and avoid generating the tar.gz anymore. For the discussion see
> http://apache-nifi-developer-list.39713.n7.nabble.com/DISCUSS-Tar-Gzip-vs-Zip-td19175.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5468) Add toolkit to NiFi docker image and use zip archive
[ https://issues.apache.org/jira/browse/NIFI-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561829#comment-16561829 ] ASF GitHub Bot commented on NIFI-5468: -- Github user MikeThomsen commented on a diff in the pull request: https://github.com/apache/nifi/pull/2921#discussion_r206106251 --- Diff: nifi-docker/dockerhub/pom.xml --- @@ -39,7 +39,7 @@ 1000 1000 -1.6.0 +1.7.0 --- End diff -- Should be 1.8.0 now. > Add toolkit to NiFi docker image and use zip archive > > > Key: NIFI-5468 > URL: https://issues.apache.org/jira/browse/NIFI-5468 > Project: Apache NiFi > Issue Type: Improvement > Components: Docker >Affects Versions: 1.8.0 >Reporter: Peter Wilcsinszky >Assignee: Peter Wilcsinszky >Priority: Major > Fix For: 1.8.0 > > > NiFi toolkit is an essential tool for a NiFi installation and makes sense to > include it by default in the NiFi Docker image. > Using the zip archive is a step to make possible switching completely to the > zip release and avoid generating the tar.gz anymore. For the discussion see > http://apache-nifi-developer-list.39713.n7.nabble.com/DISCUSS-Tar-Gzip-vs-Zip-td19175.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2921: NIFI-5468 Add toolkit to NiFi docker image and use ...
Github user MikeThomsen commented on a diff in the pull request: https://github.com/apache/nifi/pull/2921#discussion_r206106251 --- Diff: nifi-docker/dockerhub/pom.xml --- @@ -39,7 +39,7 @@ 1000 1000 -1.6.0 +1.7.0 --- End diff -- Should be 1.8.0 now. ---
[GitHub] nifi pull request #2921: NIFI-5468 Add toolkit to NiFi docker image and use ...
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2921#discussion_r206106274
--- Diff: nifi-docker/dockerhub/pom.xml ---
@@ -61,7 +61,7 @@
${project.version}-dockerhub
-1.6.0
+1.7.0
--- End diff --
Should be 1.8.0 now.
---
[jira] [Commented] (NIFI-5468) Add toolkit to NiFi docker image and use zip archive
[
https://issues.apache.org/jira/browse/NIFI-5468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561830#comment-16561830
]
ASF GitHub Bot commented on NIFI-5468:
--
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2921#discussion_r206105937
--- Diff: nifi-docker/dockerhub/Dockerfile ---
@@ -23,11 +23,15 @@ LABEL site="https://nifi.apache.org;
ARG UID=1000
ARG GID=1000
ARG NIFI_VERSION=1.8.0
-ARG MIRROR=https://archive.apache.org/dist
+ARG MIRROR_BASE_URL=https://archive.apache.org/dist
+ARG ARCHIVE_BASE_URL=https://archive.apache.org/dist
+
+ENV NIFI_BASE_DIR=/opt/nifi
+ENV NIFI_HOME=${NIFI_BASE_DIR}/nifi-${NIFI_VERSION}
+
+ENV
NIFI_BINARY_PATH=${NIFI_BINARY_PATH:-/nifi/${NIFI_VERSION}/nifi-${NIFI_VERSION}-bin.zip}
--- End diff --
Did you forget to add `/opt` here to this? `PATH:-/nifi/${NIFI_VERSION}`?
From the diff, it doesn't look like `/nifi` would be a valid starting path.
> Add toolkit to NiFi docker image and use zip archive
>
>
> Key: NIFI-5468
> URL: https://issues.apache.org/jira/browse/NIFI-5468
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Docker
>Affects Versions: 1.8.0
>Reporter: Peter Wilcsinszky
>Assignee: Peter Wilcsinszky
>Priority: Major
> Fix For: 1.8.0
>
>
> NiFi toolkit is an essential tool for a NiFi installation and makes sense to
> include it by default in the NiFi Docker image.
> Using the zip archive is a step to make possible switching completely to the
> zip release and avoid generating the tar.gz anymore. For the discussion see
> http://apache-nifi-developer-list.39713.n7.nabble.com/DISCUSS-Tar-Gzip-vs-Zip-td19175.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5179) List based processor documentation should not say that DistributedMapCache service should be used.
[ https://issues.apache.org/jira/browse/NIFI-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561555#comment-16561555 ] Koji Kawamura commented on NIFI-5179: - NIFI-5406 updated DistributedMapCache description for the most List processors as it updates abstract class. The only remaining List processor needs to be updated is, ListHDFS. > List based processor documentation should not say that DistributedMapCache > service should be used. > -- > > Key: NIFI-5179 > URL: https://issues.apache.org/jira/browse/NIFI-5179 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation Website >Affects Versions: 1.1.0, 1.2.0, 1.3.0, 1.4.0, 1.5.0, 1.6.0 >Reporter: Matthew Clarke >Priority: Minor > > Some of the list based processors where developed pre Apache NiFi 1.0 release > and relied on DistributedMapCache Service to retain cluster state. The > documentation leads to confusion now that cluster state is stored in > zookeeper. Documentation should be updated to illustrate what purpose the > DistributedMapCache service has post NiFi 1.x. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
