[GitHub] nifi pull request #2530: NIFI-4800 Expose the flattenMode as property in Fla...
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2530#discussion_r173719906
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/groovy/org/apache/nifi/processors/standard/TestFlattenJson.groovy
---
@@ -134,4 +134,44 @@ class TestFlattenJson {
])
}
}
+
+@Test
+void testFlattenModeNormal() {
+def testRunner = TestRunners.newTestRunner(FlattenJson.class)
+def json = prettyPrint(toJson([
+first: [
+second: [
+third: [
+"one", "two", "three", "four",
"five"
+]
+]
+]
+]))
+
+testRunner.setValidateExpressionUsage(true);
+testRunner.setProperty(FlattenJson.FLATTEN_MODE,
FlattenJson.FLATTEN_MODE_NORMAL)
+baseTest(testRunner, json,5) { parsed ->
+Assert.assertEquals("Separator not applied.", "one",
parsed["first.second.third[0]"])
+}
+}
+
+@Test
+void testFlattenModeDotNotation() {
+def testRunner = TestRunners.newTestRunner(FlattenJson.class)
+def json = prettyPrint(toJson([
+first: [
+second: [
+third: [
+"one", "two", "three", "four",
"five"
+]
+]
+]
+]))
+
+testRunner.setValidateExpressionUsage(true);
--- End diff --
not sure this line is required
---
[GitHub] nifi pull request #2530: NIFI-4800 Expose the flattenMode as property in Fla...
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2530#discussion_r173719874
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/groovy/org/apache/nifi/processors/standard/TestFlattenJson.groovy
---
@@ -134,4 +134,44 @@ class TestFlattenJson {
])
}
}
+
+@Test
+void testFlattenModeNormal() {
+def testRunner = TestRunners.newTestRunner(FlattenJson.class)
+def json = prettyPrint(toJson([
+first: [
+second: [
+third: [
+"one", "two", "three", "four",
"five"
+]
+]
+]
+]))
+
+testRunner.setValidateExpressionUsage(true);
--- End diff --
not sure this line is required
---
[GitHub] nifi pull request #2530: NIFI-4800 Expose the flattenMode as property in Fla...
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2530#discussion_r173719297
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/FlattenJson.java
---
@@ -89,13 +92,32 @@
.expressionLanguageSupported(true)
.build();
+public static final AllowableValue FLATTEN_MODE_NORMAL = new
AllowableValue("normal", "normal",
+"Flattens every objects into a single level json");
+
+public static final AllowableValue FLATTEN_MODE_KEEP_ARRAYS = new
AllowableValue("keep arrays", "keep arrays",
+"Flattens every objects and keep arrays format");
+
+public static final AllowableValue FLATTEN_MODE_DOT_NOTATION = new
AllowableValue("dot notation", "dot notation",
+"Conforms to MongoDB dot notation to update also nested
documents");
+
+public static final PropertyDescriptor FLATTEN_MODE = new
PropertyDescriptor.Builder()
+.name("Flatten Mode")
--- End diff --
Could you use both displayName() and name()?
---
[jira] [Commented] (NIFI-4800) Expose the flattenMode as property in FlattenJSON processor
[
https://issues.apache.org/jira/browse/NIFI-4800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394943#comment-16394943
]
ASF GitHub Bot commented on NIFI-4800:
--
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2530#discussion_r173719297
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/FlattenJson.java
---
@@ -89,13 +92,32 @@
.expressionLanguageSupported(true)
.build();
+public static final AllowableValue FLATTEN_MODE_NORMAL = new
AllowableValue("normal", "normal",
+"Flattens every objects into a single level json");
+
+public static final AllowableValue FLATTEN_MODE_KEEP_ARRAYS = new
AllowableValue("keep arrays", "keep arrays",
+"Flattens every objects and keep arrays format");
+
+public static final AllowableValue FLATTEN_MODE_DOT_NOTATION = new
AllowableValue("dot notation", "dot notation",
+"Conforms to MongoDB dot notation to update also nested
documents");
+
+public static final PropertyDescriptor FLATTEN_MODE = new
PropertyDescriptor.Builder()
+.name("Flatten Mode")
--- End diff --
Could you use both displayName() and name()?
> Expose the flattenMode as property in FlattenJSON processor
> ---
>
> Key: NIFI-4800
> URL: https://issues.apache.org/jira/browse/NIFI-4800
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Alois Gruber
>Assignee: Deon Huang
>Priority: Trivial
>
> the flattening class supports 3 different modes, which cannot be selected in
> the processor. Especially the flattening of arrays would be helpful
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4800) Expose the flattenMode as property in FlattenJSON processor
[
https://issues.apache.org/jira/browse/NIFI-4800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394944#comment-16394944
]
ASF GitHub Bot commented on NIFI-4800:
--
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2530#discussion_r173719874
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/groovy/org/apache/nifi/processors/standard/TestFlattenJson.groovy
---
@@ -134,4 +134,44 @@ class TestFlattenJson {
])
}
}
+
+@Test
+void testFlattenModeNormal() {
+def testRunner = TestRunners.newTestRunner(FlattenJson.class)
+def json = prettyPrint(toJson([
+first: [
+second: [
+third: [
+"one", "two", "three", "four",
"five"
+]
+]
+]
+]))
+
+testRunner.setValidateExpressionUsage(true);
--- End diff --
not sure this line is required
> Expose the flattenMode as property in FlattenJSON processor
> ---
>
> Key: NIFI-4800
> URL: https://issues.apache.org/jira/browse/NIFI-4800
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Alois Gruber
>Assignee: Deon Huang
>Priority: Trivial
>
> the flattening class supports 3 different modes, which cannot be selected in
> the processor. Especially the flattening of arrays would be helpful
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4800) Expose the flattenMode as property in FlattenJSON processor
[
https://issues.apache.org/jira/browse/NIFI-4800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394942#comment-16394942
]
ASF GitHub Bot commented on NIFI-4800:
--
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2530#discussion_r173719906
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/groovy/org/apache/nifi/processors/standard/TestFlattenJson.groovy
---
@@ -134,4 +134,44 @@ class TestFlattenJson {
])
}
}
+
+@Test
+void testFlattenModeNormal() {
+def testRunner = TestRunners.newTestRunner(FlattenJson.class)
+def json = prettyPrint(toJson([
+first: [
+second: [
+third: [
+"one", "two", "three", "four",
"five"
+]
+]
+]
+]))
+
+testRunner.setValidateExpressionUsage(true);
+testRunner.setProperty(FlattenJson.FLATTEN_MODE,
FlattenJson.FLATTEN_MODE_NORMAL)
+baseTest(testRunner, json,5) { parsed ->
+Assert.assertEquals("Separator not applied.", "one",
parsed["first.second.third[0]"])
+}
+}
+
+@Test
+void testFlattenModeDotNotation() {
+def testRunner = TestRunners.newTestRunner(FlattenJson.class)
+def json = prettyPrint(toJson([
+first: [
+second: [
+third: [
+"one", "two", "three", "four",
"five"
+]
+]
+]
+]))
+
+testRunner.setValidateExpressionUsage(true);
--- End diff --
not sure this line is required
> Expose the flattenMode as property in FlattenJSON processor
> ---
>
> Key: NIFI-4800
> URL: https://issues.apache.org/jira/browse/NIFI-4800
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.5.0
>Reporter: Alois Gruber
>Assignee: Deon Huang
>Priority: Trivial
>
> the flattening class supports 3 different modes, which cannot be selected in
> the processor. Especially the flattening of arrays would be helpful
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-4960) Multiple Custom Processors don't work as espected.
Jorge Machado created NIFI-4960:
---
Summary: Multiple Custom Processors don't work as espected.
Key: NIFI-4960
URL: https://issues.apache.org/jira/browse/NIFI-4960
Project: Apache NiFi
Issue Type: Bug
Components: Configuration
Affects Versions: 1.5.0
Reporter: Jorge Machado
I we setting up the Nifi registry are we are getting this error when trying to
import a flow from Registry into Nifi:
|Multiple versions of …. exist. No exact match for default:...:unversioned.|
To note that we have multiple versions of the same processor in the lib folder
(which works for Templates)
Our Manifest inside the nar file looks like this:
{color:#33}Manifest-Version: 1.0{color}
{color:#33}Implementation-Title: componentName{color}
{color:#33}Implementation-Version: 1.5.9.rc.1{color}
{color:#33}Nar-Dependency-Group: org.apache.nifi{color}
{color:#33}Nar-Version: 1.5.9.rc.1{color}
{color:#33}Nar-Dependency-Version: 1.5.0{color}
{color:#33}Nar-Id: {color}componentName
{color:#33}Nar-Group: com.{color}componentName
{color:#33}Nar-Dependency-Id: nifi-standard-services-api-nar{color}
{color:#33} {color}
{color:#33}I see this on the Nifi Logs: {color}
{color:#33}2018-03-12 08:33:51,304 INFO [NiFi Web Server-94]
o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException:
Multiple versions of com.gfk.nifi.processor.customlogging.CustomLogger exist.
No exact match for default:ipmvp-nifi-customlogattribute:unversioned..
Returning Conflict response.{color}
{color:#33}After some debugging I found the bug on BundleUtils and I will
patch it. {color}
{code:java}
public static BundleDTO createBundleDto(final
org.apache.nifi.registry.flow.Bundle bundle) {
final BundleDTO dto = new BundleDTO();
dto.setArtifact(bundle.getArtifact());
dto.setGroup(dto.getGroup()); <-- Should be bundle right ?
dto.setVersion(dto.getVersion()); <-- should be bundle right ?
return dto;
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2531: NIFI-4960
GitHub user jomach opened a pull request: https://github.com/apache/nifi/pull/2531 NIFI-4960 * fix object setting. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? NIFI-4960 - [x ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jomach/nifi nifi-4960 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2531.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2531 commit 1c0afdce6fb9453b1ebac2821f03421e68425459 Author: Jorge Machado Date: 2018-03-12T09:55:05Z NIFI-4960 * fix object setting. ---
[jira] [Commented] (NIFI-4960) Multiple Custom Processors don't work as espected.
[
https://issues.apache.org/jira/browse/NIFI-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395130#comment-16395130
]
ASF GitHub Bot commented on NIFI-4960:
--
GitHub user jomach opened a pull request:
https://github.com/apache/nifi/pull/2531
NIFI-4960
* fix object setting.
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message? NIFI-4960
- [x ] Does your PR title start with NIFI- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ x] Have you ensured that the full suite of tests is executed via mvn
-Pcontrib-check clean install at the root nifi folder?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jomach/nifi nifi-4960
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/2531.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2531
commit 1c0afdce6fb9453b1ebac2821f03421e68425459
Author: Jorge Machado
Date: 2018-03-12T09:55:05Z
NIFI-4960
* fix object setting.
> Multiple Custom Processors don't work as espected.
> --
>
> Key: NIFI-4960
> URL: https://issues.apache.org/jira/browse/NIFI-4960
> Project: Apache NiFi
> Issue Type: Bug
> Components: Configuration
>Affects Versions: 1.5.0
>Reporter: Jorge Machado
>Priority: Major
>
> I we setting up the Nifi registry are we are getting this error when trying
> to import a flow from Registry into Nifi:
> |Multiple versions of …. exist. No exact match for default:...:unversioned.|
> To note that we have multiple versions of the same processor in the lib
> folder (which works for Templates)
> Our Manifest inside the nar file looks like this:
> {color:#33}Manifest-Version: 1.0{color}
> {color:#33}Implementation-Title: componentName{color}
> {color:#33}Implementation-Version: 1.5.9.rc.1{color}
> {color:#33}Nar-Dependency-Group: org.apache.nifi{color}
> {color:#33}Nar-Version: 1.5.9.rc.1{color}
> {color:#33}Nar-Dependency-Version: 1.5.0{color}
> {color:#33}Nar-Id: {color}componentName
> {color:#33}Nar-Group: com.{color}componentName
> {color:#33}Nar-Dependency-Id: nifi-standard-services-api-nar{color}
> {color:#33} {color}
> {color:#33}I see this on the Nifi Logs: {color}
> {color:#33}2018-03-12 08:33:51,304 INFO [NiFi Web Server-94]
> o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException:
> Multiple versions of com.gfk.nifi.processor.customlogging.CustomLogger exist.
> No exact match for default:ipmvp-nifi-customlogattribute:unversioned..
> Returning Conflict response.{color}
>
>
> {color:#33}After some debugging I found the bug on BundleUtils and I will
> patch it. {color}
> {code:java}
> public static BundleDTO createBundleDto(final
> org.apache.nifi.registry.flow.Bundle bundle) {
> final BundleDTO dto = new BundleDTO();
> dto.setArtifact(bundle.getArtifact());
> dto.setGroup(dto.getGroup()); <-- Should be bundle right ?
> dto.setVersion(dto.getVersion()); <-- should be bundle right ?
> return dto;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-4961) Allow data to be set on MockFlowFile
Glenn Widner created NIFI-4961: -- Summary: Allow data to be set on MockFlowFile Key: NIFI-4961 URL: https://issues.apache.org/jira/browse/NIFI-4961 Project: Apache NiFi Issue Type: Improvement Reporter: Glenn Widner While working on tests for a custom processor I noticed that the setData method is private which makes it hard to test that my processor is handling FlowFile content properly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (NIFI-4936) NiFi parent pom dependency management forcing versions to align defeating classloader isolation
[ https://issues.apache.org/jira/browse/NIFI-4936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman reassigned NIFI-4936: - Assignee: Matt Gilman (was: Joseph Witt) > NiFi parent pom dependency management forcing versions to align defeating > classloader isolation > --- > > Key: NIFI-4936 > URL: https://issues.apache.org/jira/browse/NIFI-4936 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Extensions, Tools and Build >Affects Versions: 1.1.0, 1.2.0, 1.0.1, 1.3.0, 1.4.0, 1.5.0 >Reporter: Joseph Witt >Assignee: Matt Gilman >Priority: Major > Fix For: 1.6.0 > > Attachments: NIFI-4936.patch, build-fixing, > old-vs-new-dependencies.txt > > > the top level pom in nifi has a massive dependency management section. this > was used initially to help enforce consistent usage of dependencies across > nifi but this also can defeat the purpose of the classloader isolation > offered by nars. We need to push down dependency version declarations to the > nar levels where appropriate. > there have been reported issues of defects happening due to us using much > newer versions (or sometimes older) of dependencies due to this dependency > management model. By pushing declarations down to the proper scope each > nar/etc.. can use the specific versions of components it needs and we'll stop > introducing issues by forcing a different version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (NIFI-4936) NiFi parent pom dependency management forcing versions to align defeating classloader isolation
[ https://issues.apache.org/jira/browse/NIFI-4936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman reopened NIFI-4936: --- Reopening to move the definition of the ranger version so it is accessible in both the ranger nar and the nifi assembly > NiFi parent pom dependency management forcing versions to align defeating > classloader isolation > --- > > Key: NIFI-4936 > URL: https://issues.apache.org/jira/browse/NIFI-4936 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Extensions, Tools and Build >Affects Versions: 1.1.0, 1.2.0, 1.0.1, 1.3.0, 1.4.0, 1.5.0 >Reporter: Joseph Witt >Assignee: Joseph Witt >Priority: Major > Fix For: 1.6.0 > > Attachments: NIFI-4936.patch, build-fixing, > old-vs-new-dependencies.txt > > > the top level pom in nifi has a massive dependency management section. this > was used initially to help enforce consistent usage of dependencies across > nifi but this also can defeat the purpose of the classloader isolation > offered by nars. We need to push down dependency version declarations to the > nar levels where appropriate. > there have been reported issues of defects happening due to us using much > newer versions (or sometimes older) of dependencies due to this dependency > management model. By pushing declarations down to the proper scope each > nar/etc.. can use the specific versions of components it needs and we'll stop > introducing issues by forcing a different version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4960) Multiple Custom Processors don't work as espected.
[
https://issues.apache.org/jira/browse/NIFI-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395242#comment-16395242
]
ASF subversion and git services commented on NIFI-4960:
---
Commit 06cb354c5e5654878eda582ddf620691360adb49 in nifi's branch
refs/heads/master from [~jomach]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=06cb354 ]
NIFI-4960: fix object setting. This closes #2531.
Signed-off-by: Mark Payne
> Multiple Custom Processors don't work as espected.
> --
>
> Key: NIFI-4960
> URL: https://issues.apache.org/jira/browse/NIFI-4960
> Project: Apache NiFi
> Issue Type: Bug
> Components: Configuration
>Affects Versions: 1.5.0
>Reporter: Jorge Machado
>Priority: Major
>
> I we setting up the Nifi registry are we are getting this error when trying
> to import a flow from Registry into Nifi:
> |Multiple versions of …. exist. No exact match for default:...:unversioned.|
> To note that we have multiple versions of the same processor in the lib
> folder (which works for Templates)
> Our Manifest inside the nar file looks like this:
> {color:#33}Manifest-Version: 1.0{color}
> {color:#33}Implementation-Title: componentName{color}
> {color:#33}Implementation-Version: 1.5.9.rc.1{color}
> {color:#33}Nar-Dependency-Group: org.apache.nifi{color}
> {color:#33}Nar-Version: 1.5.9.rc.1{color}
> {color:#33}Nar-Dependency-Version: 1.5.0{color}
> {color:#33}Nar-Id: {color}componentName
> {color:#33}Nar-Group: com.{color}componentName
> {color:#33}Nar-Dependency-Id: nifi-standard-services-api-nar{color}
> {color:#33} {color}
> {color:#33}I see this on the Nifi Logs: {color}
> {color:#33}2018-03-12 08:33:51,304 INFO [NiFi Web Server-94]
> o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException:
> Multiple versions of com.gfk.nifi.processor.customlogging.CustomLogger exist.
> No exact match for default:ipmvp-nifi-customlogattribute:unversioned..
> Returning Conflict response.{color}
>
>
> {color:#33}After some debugging I found the bug on BundleUtils and I will
> patch it. {color}
> {code:java}
> public static BundleDTO createBundleDto(final
> org.apache.nifi.registry.flow.Bundle bundle) {
> final BundleDTO dto = new BundleDTO();
> dto.setArtifact(bundle.getArtifact());
> dto.setGroup(dto.getGroup()); <-- Should be bundle right ?
> dto.setVersion(dto.getVersion()); <-- should be bundle right ?
> return dto;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #2531: NIFI-4960 - Multiple Custom Processors don't work as expec...
Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/2531 @jomach great catch and thanks for the PR! All looks good to me, +1 merged to master. ---
[GitHub] nifi pull request #2531: NIFI-4960 - Multiple Custom Processors don't work a...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2531 ---
[jira] [Commented] (NIFI-4960) Multiple Custom Processors don't work as espected.
[
https://issues.apache.org/jira/browse/NIFI-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395244#comment-16395244
]
ASF GitHub Bot commented on NIFI-4960:
--
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/2531
> Multiple Custom Processors don't work as espected.
> --
>
> Key: NIFI-4960
> URL: https://issues.apache.org/jira/browse/NIFI-4960
> Project: Apache NiFi
> Issue Type: Bug
> Components: Configuration
>Affects Versions: 1.5.0
>Reporter: Jorge Machado
>Priority: Major
> Fix For: 1.6.0
>
>
> I we setting up the Nifi registry are we are getting this error when trying
> to import a flow from Registry into Nifi:
> |Multiple versions of …. exist. No exact match for default:...:unversioned.|
> To note that we have multiple versions of the same processor in the lib
> folder (which works for Templates)
> Our Manifest inside the nar file looks like this:
> {color:#33}Manifest-Version: 1.0{color}
> {color:#33}Implementation-Title: componentName{color}
> {color:#33}Implementation-Version: 1.5.9.rc.1{color}
> {color:#33}Nar-Dependency-Group: org.apache.nifi{color}
> {color:#33}Nar-Version: 1.5.9.rc.1{color}
> {color:#33}Nar-Dependency-Version: 1.5.0{color}
> {color:#33}Nar-Id: {color}componentName
> {color:#33}Nar-Group: com.{color}componentName
> {color:#33}Nar-Dependency-Id: nifi-standard-services-api-nar{color}
> {color:#33} {color}
> {color:#33}I see this on the Nifi Logs: {color}
> {color:#33}2018-03-12 08:33:51,304 INFO [NiFi Web Server-94]
> o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException:
> Multiple versions of com.gfk.nifi.processor.customlogging.CustomLogger exist.
> No exact match for default:ipmvp-nifi-customlogattribute:unversioned..
> Returning Conflict response.{color}
>
>
> {color:#33}After some debugging I found the bug on BundleUtils and I will
> patch it. {color}
> {code:java}
> public static BundleDTO createBundleDto(final
> org.apache.nifi.registry.flow.Bundle bundle) {
> final BundleDTO dto = new BundleDTO();
> dto.setArtifact(bundle.getArtifact());
> dto.setGroup(dto.getGroup()); <-- Should be bundle right ?
> dto.setVersion(dto.getVersion()); <-- should be bundle right ?
> return dto;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (NIFI-4960) Multiple Custom Processors don't work as espected.
[
https://issues.apache.org/jira/browse/NIFI-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne resolved NIFI-4960.
--
Resolution: Fixed
Fix Version/s: 1.6.0
> Multiple Custom Processors don't work as espected.
> --
>
> Key: NIFI-4960
> URL: https://issues.apache.org/jira/browse/NIFI-4960
> Project: Apache NiFi
> Issue Type: Bug
> Components: Configuration
>Affects Versions: 1.5.0
>Reporter: Jorge Machado
>Priority: Major
> Fix For: 1.6.0
>
>
> I we setting up the Nifi registry are we are getting this error when trying
> to import a flow from Registry into Nifi:
> |Multiple versions of …. exist. No exact match for default:...:unversioned.|
> To note that we have multiple versions of the same processor in the lib
> folder (which works for Templates)
> Our Manifest inside the nar file looks like this:
> {color:#33}Manifest-Version: 1.0{color}
> {color:#33}Implementation-Title: componentName{color}
> {color:#33}Implementation-Version: 1.5.9.rc.1{color}
> {color:#33}Nar-Dependency-Group: org.apache.nifi{color}
> {color:#33}Nar-Version: 1.5.9.rc.1{color}
> {color:#33}Nar-Dependency-Version: 1.5.0{color}
> {color:#33}Nar-Id: {color}componentName
> {color:#33}Nar-Group: com.{color}componentName
> {color:#33}Nar-Dependency-Id: nifi-standard-services-api-nar{color}
> {color:#33} {color}
> {color:#33}I see this on the Nifi Logs: {color}
> {color:#33}2018-03-12 08:33:51,304 INFO [NiFi Web Server-94]
> o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException:
> Multiple versions of com.gfk.nifi.processor.customlogging.CustomLogger exist.
> No exact match for default:ipmvp-nifi-customlogattribute:unversioned..
> Returning Conflict response.{color}
>
>
> {color:#33}After some debugging I found the bug on BundleUtils and I will
> patch it. {color}
> {code:java}
> public static BundleDTO createBundleDto(final
> org.apache.nifi.registry.flow.Bundle bundle) {
> final BundleDTO dto = new BundleDTO();
> dto.setArtifact(bundle.getArtifact());
> dto.setGroup(dto.getGroup()); <-- Should be bundle right ?
> dto.setVersion(dto.getVersion()); <-- should be bundle right ?
> return dto;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4960) Multiple Custom Processors don't work as espected.
[
https://issues.apache.org/jira/browse/NIFI-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395243#comment-16395243
]
ASF GitHub Bot commented on NIFI-4960:
--
Github user markap14 commented on the issue:
https://github.com/apache/nifi/pull/2531
@jomach great catch and thanks for the PR! All looks good to me, +1 merged
to master.
> Multiple Custom Processors don't work as espected.
> --
>
> Key: NIFI-4960
> URL: https://issues.apache.org/jira/browse/NIFI-4960
> Project: Apache NiFi
> Issue Type: Bug
> Components: Configuration
>Affects Versions: 1.5.0
>Reporter: Jorge Machado
>Priority: Major
> Fix For: 1.6.0
>
>
> I we setting up the Nifi registry are we are getting this error when trying
> to import a flow from Registry into Nifi:
> |Multiple versions of …. exist. No exact match for default:...:unversioned.|
> To note that we have multiple versions of the same processor in the lib
> folder (which works for Templates)
> Our Manifest inside the nar file looks like this:
> {color:#33}Manifest-Version: 1.0{color}
> {color:#33}Implementation-Title: componentName{color}
> {color:#33}Implementation-Version: 1.5.9.rc.1{color}
> {color:#33}Nar-Dependency-Group: org.apache.nifi{color}
> {color:#33}Nar-Version: 1.5.9.rc.1{color}
> {color:#33}Nar-Dependency-Version: 1.5.0{color}
> {color:#33}Nar-Id: {color}componentName
> {color:#33}Nar-Group: com.{color}componentName
> {color:#33}Nar-Dependency-Id: nifi-standard-services-api-nar{color}
> {color:#33} {color}
> {color:#33}I see this on the Nifi Logs: {color}
> {color:#33}2018-03-12 08:33:51,304 INFO [NiFi Web Server-94]
> o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException:
> Multiple versions of com.gfk.nifi.processor.customlogging.CustomLogger exist.
> No exact match for default:ipmvp-nifi-customlogattribute:unversioned..
> Returning Conflict response.{color}
>
>
> {color:#33}After some debugging I found the bug on BundleUtils and I will
> patch it. {color}
> {code:java}
> public static BundleDTO createBundleDto(final
> org.apache.nifi.registry.flow.Bundle bundle) {
> final BundleDTO dto = new BundleDTO();
> dto.setArtifact(bundle.getArtifact());
> dto.setGroup(dto.getGroup()); <-- Should be bundle right ?
> dto.setVersion(dto.getVersion()); <-- should be bundle right ?
> return dto;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-4960) Having multiple versions of same component doesn't work as expected when using Flow Registry
[
https://issues.apache.org/jira/browse/NIFI-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-4960:
-
Summary: Having multiple versions of same component doesn't work as
expected when using Flow Registry (was: Multiple Custom Processors don't work
as espected.)
> Having multiple versions of same component doesn't work as expected when
> using Flow Registry
>
>
> Key: NIFI-4960
> URL: https://issues.apache.org/jira/browse/NIFI-4960
> Project: Apache NiFi
> Issue Type: Bug
> Components: Configuration
>Affects Versions: 1.5.0
>Reporter: Jorge Machado
>Priority: Major
> Fix For: 1.6.0
>
>
> I we setting up the Nifi registry are we are getting this error when trying
> to import a flow from Registry into Nifi:
> |Multiple versions of …. exist. No exact match for default:...:unversioned.|
> To note that we have multiple versions of the same processor in the lib
> folder (which works for Templates)
> Our Manifest inside the nar file looks like this:
> {color:#33}Manifest-Version: 1.0{color}
> {color:#33}Implementation-Title: componentName{color}
> {color:#33}Implementation-Version: 1.5.9.rc.1{color}
> {color:#33}Nar-Dependency-Group: org.apache.nifi{color}
> {color:#33}Nar-Version: 1.5.9.rc.1{color}
> {color:#33}Nar-Dependency-Version: 1.5.0{color}
> {color:#33}Nar-Id: {color}componentName
> {color:#33}Nar-Group: com.{color}componentName
> {color:#33}Nar-Dependency-Id: nifi-standard-services-api-nar{color}
> {color:#33} {color}
> {color:#33}I see this on the Nifi Logs: {color}
> {color:#33}2018-03-12 08:33:51,304 INFO [NiFi Web Server-94]
> o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException:
> Multiple versions of com.gfk.nifi.processor.customlogging.CustomLogger exist.
> No exact match for default:ipmvp-nifi-customlogattribute:unversioned..
> Returning Conflict response.{color}
>
>
> {color:#33}After some debugging I found the bug on BundleUtils and I will
> patch it. {color}
> {code:java}
> public static BundleDTO createBundleDto(final
> org.apache.nifi.registry.flow.Bundle bundle) {
> final BundleDTO dto = new BundleDTO();
> dto.setArtifact(bundle.getArtifact());
> dto.setGroup(dto.getGroup()); <-- Should be bundle right ?
> dto.setVersion(dto.getVersion()); <-- should be bundle right ?
> return dto;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4800) Expose the flattenMode as property in FlattenJSON processor
[ https://issues.apache.org/jira/browse/NIFI-4800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395247#comment-16395247 ] ASF GitHub Bot commented on NIFI-4800: -- Github user yjhyjhyjh0 commented on the issue: https://github.com/apache/nifi/pull/2530 Thanks for the review. I've added display name as most processor does and removed unnecessary line at unit tests. > Expose the flattenMode as property in FlattenJSON processor > --- > > Key: NIFI-4800 > URL: https://issues.apache.org/jira/browse/NIFI-4800 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.5.0 >Reporter: Alois Gruber >Assignee: Deon Huang >Priority: Trivial > > the flattening class supports 3 different modes, which cannot be selected in > the processor. Especially the flattening of arrays would be helpful -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2530: NIFI-4800 Expose the flattenMode as property in FlattenJSO...
Github user yjhyjhyjh0 commented on the issue: https://github.com/apache/nifi/pull/2530 Thanks for the review. I've added display name as most processor does and removed unnecessary line at unit tests. ---
[GitHub] nifi pull request #2532: NIFI-4936: Fix ranger version definition
GitHub user mcgilman opened a pull request: https://github.com/apache/nifi/pull/2532 NIFI-4936: Fix ranger version definition NIFI-4936: - Moving definition of ranger version into root pom. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mcgilman/nifi NIFI-4936 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2532.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2532 commit e786ff328fdbbe0632da5a09124430f4ab3b1a98 Author: Matt Gilman Date: 2018-03-12T13:53:07Z NIFI-4936: - Moving definition of ranger version into root pom. ---
[jira] [Commented] (NIFI-4936) NiFi parent pom dependency management forcing versions to align defeating classloader isolation
[ https://issues.apache.org/jira/browse/NIFI-4936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395255#comment-16395255 ] ASF GitHub Bot commented on NIFI-4936: -- GitHub user mcgilman opened a pull request: https://github.com/apache/nifi/pull/2532 NIFI-4936: Fix ranger version definition NIFI-4936: - Moving definition of ranger version into root pom. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mcgilman/nifi NIFI-4936 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2532.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2532 commit e786ff328fdbbe0632da5a09124430f4ab3b1a98 Author: Matt Gilman Date: 2018-03-12T13:53:07Z NIFI-4936: - Moving definition of ranger version into root pom. > NiFi parent pom dependency management forcing versions to align defeating > classloader isolation > --- > > Key: NIFI-4936 > URL: https://issues.apache.org/jira/browse/NIFI-4936 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Extensions, Tools and Build >Affects Versions: 1.1.0, 1.2.0, 1.0.1, 1.3.0, 1.4.0, 1.5.0 >Reporter: Joseph Witt >Assignee: Matt Gilman >Priority: Major > Fix For: 1.6.0 > > Attachments: NIFI-4936.patch, build-fixing, > old-vs-new-dependencies.txt > > > the top level pom in nifi has a massive dependency management section. this > was used initially to help enforce consistent usage of dependencies across > nifi but this also can defeat the purpose of the classloader isolation > offered by nars. We need to push down dependency version declarations to the > nar levels where appropriate. > there have been reported issues of defects happening due to us using much > newer versions (or sometimes older) of dependencies due to this dependency > management model. By pushing declarations down to the proper scope each > nar/etc.. can use the specific versions of components it needs and we'll stop > introducing issues by forcing a different version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2533: NIFI-4961 Allow data to be set on MockFlowFile
GitHub user kai5263499 opened a pull request: https://github.com/apache/nifi/pull/2533 NIFI-4961 Allow data to be set on MockFlowFile Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kai5263499/nifi NIFI-4961 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2533.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2533 commit bc3ec1d035a3834cb470918687deb2905fcfa244 Author: Wes Widner Date: 2018-03-12T13:10:25Z NIFI-4961 Allow data to be set on MockFlowFile ---
[jira] [Commented] (NIFI-4961) Allow data to be set on MockFlowFile
[ https://issues.apache.org/jira/browse/NIFI-4961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395258#comment-16395258 ] ASF GitHub Bot commented on NIFI-4961: -- GitHub user kai5263499 opened a pull request: https://github.com/apache/nifi/pull/2533 NIFI-4961 Allow data to be set on MockFlowFile Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kai5263499/nifi NIFI-4961 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2533.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2533 commit bc3ec1d035a3834cb470918687deb2905fcfa244 Author: Wes Widner Date: 2018-03-12T13:10:25Z NIFI-4961 Allow data to be set on MockFlowFile > Allow data to be set on MockFlowFile > > > Key: NIFI-4961 > URL: https://issues.apache.org/jira/browse/NIFI-4961 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Glenn Widner >Priority: Minor > > While working on tests for a custom processor I noticed that the setData > method is private which makes it hard to test that my processor is handling > FlowFile content properly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2532: NIFI-4936: Fix ranger version definition
Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2532 am a +1 assuming the build comes through clean with ranger profile activated. At first I was thinking this was counter to what we were doing but actually this is a cleaner way to go when versions are spelled out more than once for a given thing. This allows reuse without forcing dependency chains as we were. ---
[jira] [Commented] (NIFI-4936) NiFi parent pom dependency management forcing versions to align defeating classloader isolation
[ https://issues.apache.org/jira/browse/NIFI-4936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395274#comment-16395274 ] ASF GitHub Bot commented on NIFI-4936: -- Github user joewitt commented on the issue: https://github.com/apache/nifi/pull/2532 am a +1 assuming the build comes through clean with ranger profile activated. At first I was thinking this was counter to what we were doing but actually this is a cleaner way to go when versions are spelled out more than once for a given thing. This allows reuse without forcing dependency chains as we were. > NiFi parent pom dependency management forcing versions to align defeating > classloader isolation > --- > > Key: NIFI-4936 > URL: https://issues.apache.org/jira/browse/NIFI-4936 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Extensions, Tools and Build >Affects Versions: 1.1.0, 1.2.0, 1.0.1, 1.3.0, 1.4.0, 1.5.0 >Reporter: Joseph Witt >Assignee: Matt Gilman >Priority: Major > Fix For: 1.6.0 > > Attachments: NIFI-4936.patch, build-fixing, > old-vs-new-dependencies.txt > > > the top level pom in nifi has a massive dependency management section. this > was used initially to help enforce consistent usage of dependencies across > nifi but this also can defeat the purpose of the classloader isolation > offered by nars. We need to push down dependency version declarations to the > nar levels where appropriate. > there have been reported issues of defects happening due to us using much > newer versions (or sometimes older) of dependencies due to this dependency > management model. By pushing declarations down to the proper scope each > nar/etc.. can use the specific versions of components it needs and we'll stop > introducing issues by forcing a different version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4936) NiFi parent pom dependency management forcing versions to align defeating classloader isolation
[ https://issues.apache.org/jira/browse/NIFI-4936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395277#comment-16395277 ] Joseph Witt commented on NIFI-4936: --- [~JPercivall] I think it is worth mentioning in the migration guide. I'll add the flag/label to this JIRA. > NiFi parent pom dependency management forcing versions to align defeating > classloader isolation > --- > > Key: NIFI-4936 > URL: https://issues.apache.org/jira/browse/NIFI-4936 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Extensions, Tools and Build >Affects Versions: 1.1.0, 1.2.0, 1.0.1, 1.3.0, 1.4.0, 1.5.0 >Reporter: Joseph Witt >Assignee: Matt Gilman >Priority: Major > Fix For: 1.6.0 > > Attachments: NIFI-4936.patch, build-fixing, > old-vs-new-dependencies.txt > > > the top level pom in nifi has a massive dependency management section. this > was used initially to help enforce consistent usage of dependencies across > nifi but this also can defeat the purpose of the classloader isolation > offered by nars. We need to push down dependency version declarations to the > nar levels where appropriate. > there have been reported issues of defects happening due to us using much > newer versions (or sometimes older) of dependencies due to this dependency > management model. By pushing declarations down to the proper scope each > nar/etc.. can use the specific versions of components it needs and we'll stop > introducing issues by forcing a different version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-4936) NiFi parent pom dependency management forcing versions to align defeating classloader isolation
[ https://issues.apache.org/jira/browse/NIFI-4936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-4936: -- Labels: migration (was: ) > NiFi parent pom dependency management forcing versions to align defeating > classloader isolation > --- > > Key: NIFI-4936 > URL: https://issues.apache.org/jira/browse/NIFI-4936 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Extensions, Tools and Build >Affects Versions: 1.1.0, 1.2.0, 1.0.1, 1.3.0, 1.4.0, 1.5.0 >Reporter: Joseph Witt >Assignee: Matt Gilman >Priority: Major > Labels: migration > Fix For: 1.6.0 > > Attachments: NIFI-4936.patch, build-fixing, > old-vs-new-dependencies.txt > > > the top level pom in nifi has a massive dependency management section. this > was used initially to help enforce consistent usage of dependencies across > nifi but this also can defeat the purpose of the classloader isolation > offered by nars. We need to push down dependency version declarations to the > nar levels where appropriate. > there have been reported issues of defects happening due to us using much > newer versions (or sometimes older) of dependencies due to this dependency > management model. By pushing declarations down to the proper scope each > nar/etc.. can use the specific versions of components it needs and we'll stop > introducing issues by forcing a different version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2532: NIFI-4936: Fix ranger version definition
Github user bbende commented on the issue: https://github.com/apache/nifi/pull/2532 Looks good, will merge ---
[jira] [Commented] (NIFI-4936) NiFi parent pom dependency management forcing versions to align defeating classloader isolation
[ https://issues.apache.org/jira/browse/NIFI-4936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395322#comment-16395322 ] ASF GitHub Bot commented on NIFI-4936: -- Github user bbende commented on the issue: https://github.com/apache/nifi/pull/2532 Looks good, will merge > NiFi parent pom dependency management forcing versions to align defeating > classloader isolation > --- > > Key: NIFI-4936 > URL: https://issues.apache.org/jira/browse/NIFI-4936 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Extensions, Tools and Build >Affects Versions: 1.1.0, 1.2.0, 1.0.1, 1.3.0, 1.4.0, 1.5.0 >Reporter: Joseph Witt >Assignee: Matt Gilman >Priority: Major > Labels: migration > Fix For: 1.6.0 > > Attachments: NIFI-4936.patch, build-fixing, > old-vs-new-dependencies.txt > > > the top level pom in nifi has a massive dependency management section. this > was used initially to help enforce consistent usage of dependencies across > nifi but this also can defeat the purpose of the classloader isolation > offered by nars. We need to push down dependency version declarations to the > nar levels where appropriate. > there have been reported issues of defects happening due to us using much > newer versions (or sometimes older) of dependencies due to this dependency > management model. By pushing declarations down to the proper scope each > nar/etc.. can use the specific versions of components it needs and we'll stop > introducing issues by forcing a different version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2532: NIFI-4936: Fix ranger version definition
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2532 ---
[jira] [Commented] (NIFI-4936) NiFi parent pom dependency management forcing versions to align defeating classloader isolation
[ https://issues.apache.org/jira/browse/NIFI-4936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395325#comment-16395325 ] ASF subversion and git services commented on NIFI-4936: --- Commit 32a8db22530fa5b3726b6b08f36adf2ffd6cd5c5 in nifi's branch refs/heads/master from [~mcgilman] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=32a8db2 ] NIFI-4936: - Moving definition of ranger version into root pom. This closes #2532. Signed-off-by: Bryan Bende > NiFi parent pom dependency management forcing versions to align defeating > classloader isolation > --- > > Key: NIFI-4936 > URL: https://issues.apache.org/jira/browse/NIFI-4936 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Extensions, Tools and Build >Affects Versions: 1.1.0, 1.2.0, 1.0.1, 1.3.0, 1.4.0, 1.5.0 >Reporter: Joseph Witt >Assignee: Matt Gilman >Priority: Major > Labels: migration > Fix For: 1.6.0 > > Attachments: NIFI-4936.patch, build-fixing, > old-vs-new-dependencies.txt > > > the top level pom in nifi has a massive dependency management section. this > was used initially to help enforce consistent usage of dependencies across > nifi but this also can defeat the purpose of the classloader isolation > offered by nars. We need to push down dependency version declarations to the > nar levels where appropriate. > there have been reported issues of defects happening due to us using much > newer versions (or sometimes older) of dependencies due to this dependency > management model. By pushing declarations down to the proper scope each > nar/etc.. can use the specific versions of components it needs and we'll stop > introducing issues by forcing a different version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4936) NiFi parent pom dependency management forcing versions to align defeating classloader isolation
[ https://issues.apache.org/jira/browse/NIFI-4936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395328#comment-16395328 ] ASF GitHub Bot commented on NIFI-4936: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2532 > NiFi parent pom dependency management forcing versions to align defeating > classloader isolation > --- > > Key: NIFI-4936 > URL: https://issues.apache.org/jira/browse/NIFI-4936 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Extensions, Tools and Build >Affects Versions: 1.1.0, 1.2.0, 1.0.1, 1.3.0, 1.4.0, 1.5.0 >Reporter: Joseph Witt >Assignee: Matt Gilman >Priority: Major > Labels: migration > Fix For: 1.6.0 > > Attachments: NIFI-4936.patch, build-fixing, > old-vs-new-dependencies.txt > > > the top level pom in nifi has a massive dependency management section. this > was used initially to help enforce consistent usage of dependencies across > nifi but this also can defeat the purpose of the classloader isolation > offered by nars. We need to push down dependency version declarations to the > nar levels where appropriate. > there have been reported issues of defects happening due to us using much > newer versions (or sometimes older) of dependencies due to this dependency > management model. By pushing declarations down to the proper scope each > nar/etc.. can use the specific versions of components it needs and we'll stop > introducing issues by forcing a different version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (NIFI-4936) NiFi parent pom dependency management forcing versions to align defeating classloader isolation
[ https://issues.apache.org/jira/browse/NIFI-4936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Bende resolved NIFI-4936. --- Resolution: Fixed > NiFi parent pom dependency management forcing versions to align defeating > classloader isolation > --- > > Key: NIFI-4936 > URL: https://issues.apache.org/jira/browse/NIFI-4936 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Extensions, Tools and Build >Affects Versions: 1.1.0, 1.2.0, 1.0.1, 1.3.0, 1.4.0, 1.5.0 >Reporter: Joseph Witt >Assignee: Matt Gilman >Priority: Major > Labels: migration > Fix For: 1.6.0 > > Attachments: NIFI-4936.patch, build-fixing, > old-vs-new-dependencies.txt > > > the top level pom in nifi has a massive dependency management section. this > was used initially to help enforce consistent usage of dependencies across > nifi but this also can defeat the purpose of the classloader isolation > offered by nars. We need to push down dependency version declarations to the > nar levels where appropriate. > there have been reported issues of defects happening due to us using much > newer versions (or sometimes older) of dependencies due to this dependency > management model. By pushing declarations down to the proper scope each > nar/etc.. can use the specific versions of components it needs and we'll stop > introducing issues by forcing a different version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-4962) FlattenJson processor add unexpected backslash after flatten
Deon Huang created NIFI-4962: Summary: FlattenJson processor add unexpected backslash after flatten Key: NIFI-4962 URL: https://issues.apache.org/jira/browse/NIFI-4962 Project: Apache NiFi Issue Type: Bug Components: Extensions Affects Versions: 1.5.0 Reporter: Deon Huang Assignee: Deon Huang Fix For: 1.6.0 Attachments: 34BFD277-F8FA-4ACE-BE1F-536B0BD6D10A.png, 35E5F7FD-DD79-4556-8BF0-021EEF948520.png, CEECDC79-7CA2-468C-A057-7A3264EB46BB.png Flatten Json will unexpectedly prefix slash with a backslash. This cause data miss match problem after flatten when json value contains url or other slash character. Detail example are in the attachments. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2534: NIFI-4821
GitHub user jomach opened a pull request: https://github.com/apache/nifi/pull/2534 NIFI-4821 * Upgrade poi to 3.17 Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jomach/nifi nifi-4821 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2534.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2534 commit d78f0e8b854f1debbd86da63c61c0d419b7e56f1 Author: Jorge Machado Date: 2018-03-12T14:46:31Z NIFI-4821 * Upgrade poi to 3.17 ---
[jira] [Commented] (NIFI-4962) FlattenJson processor add unexpected backslash after flatten
[ https://issues.apache.org/jira/browse/NIFI-4962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395337#comment-16395337 ] Deon Huang commented on NIFI-4962: -- Fortunately, writer of json flatten library just release an update version that solved this problem. Detail discussion is at https://github.com/wnameless/json-flattener/issues/33 I could submit an pull request to update library version and solve the escape character problem in few days. > FlattenJson processor add unexpected backslash after flatten > > > Key: NIFI-4962 > URL: https://issues.apache.org/jira/browse/NIFI-4962 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.5.0 >Reporter: Deon Huang >Assignee: Deon Huang >Priority: Minor > Fix For: 1.6.0 > > Attachments: 34BFD277-F8FA-4ACE-BE1F-536B0BD6D10A.png, > 35E5F7FD-DD79-4556-8BF0-021EEF948520.png, > CEECDC79-7CA2-468C-A057-7A3264EB46BB.png > > > Flatten Json will unexpectedly prefix slash with a backslash. > This cause data miss match problem after flatten when json value contains url > or other slash character. > Detail example are in the attachments. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4821) Upgrade to Apache POI 3.16 or newer
[ https://issues.apache.org/jira/browse/NIFI-4821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395338#comment-16395338 ] ASF GitHub Bot commented on NIFI-4821: -- GitHub user jomach opened a pull request: https://github.com/apache/nifi/pull/2534 NIFI-4821 * Upgrade poi to 3.17 Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jomach/nifi nifi-4821 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2534.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2534 commit d78f0e8b854f1debbd86da63c61c0d419b7e56f1 Author: Jorge Machado Date: 2018-03-12T14:46:31Z NIFI-4821 * Upgrade poi to 3.17 > Upgrade to Apache POI 3.16 or newer > --- > > Key: NIFI-4821 > URL: https://issues.apache.org/jira/browse/NIFI-4821 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Joseph Witt >Assignee: Joseph Witt >Priority: Major > Fix For: 1.6.0 > > > CVE-2017-12626 was announced today with the text: > > Title: CVE-2017-12626 – Denial of Service Vulnerabilities in Apache POI < 3.17 > Severity: Important > Vendor: The Apache Software Foundation > Versions affected: versions prior to version 3.17 > Description: > Apache POI versions prior to release 3.17 are vulnerable to Denial of > Service Attacks: > * Infinite Loops while parsing specially crafted WMF, EMF, MSG and macros > (POI bugs 61338 [0] and 61294 [1]) > * Out of Memory Exceptions while parsing specially crafted DOC, PPT and > XLS > (POI bugs 52372 [2] and 61295 [3]) > Mitigation: Users with applications which accept content from external or > untrusted sources are advised to upgrade to Apache POI 3.17 or newer. > -Tim Allison > on behalf of the Apache POI PMC > > [0] [https://bz.apache.org/bugzilla/show_bug.cgi?id=61338] > [1] [https://bz.apache.org/bugzilla/show_bug.cgi?id=61294] > [2] [https://bz.apache.org/bugzilla/show_bug.cgi?id=52372] > [3] [https://bz.apache.org/bugzilla/show_bug.cgi?id=61295] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2527: FetchHBaseRow - log level and displayName
Github user bbende commented on the issue: https://github.com/apache/nifi/pull/2527 I think when I initially implemented this processor I was assuming someone would be fetching rows that were expected to be there, so if it wasn't there then it would be more an error scenario. However, looking back on it now... since we do have the explicit not found relationship, it is going to be obvious when something is not found, and we can use LogMessage as Pierre mentioned if we need to generate a bulletin there. So I'm ok with changing it to debug. ---
[GitHub] nifi-registry issue #97: NIFIREG-134 Enable SpringBoot Actuator endpoints
Github user bbende commented on the issue: https://github.com/apache/nifi-registry/pull/97 Tested this locally and verified the security of the actuator end-point, everything looks good to me, nice work! Will merge. ---
[jira] [Commented] (NIFIREG-134) Enable Spring Boot Actuator REST API endpoints
[ https://issues.apache.org/jira/browse/NIFIREG-134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395383#comment-16395383 ] ASF GitHub Bot commented on NIFIREG-134: Github user bbende commented on the issue: https://github.com/apache/nifi-registry/pull/97 Tested this locally and verified the security of the actuator end-point, everything looks good to me, nice work! Will merge. > Enable Spring Boot Actuator REST API endpoints > -- > > Key: NIFIREG-134 > URL: https://issues.apache.org/jira/browse/NIFIREG-134 > Project: NiFi Registry > Issue Type: New Feature >Reporter: Kevin Doran >Assignee: Kevin Doran >Priority: Minor > Fix For: 0.2.0 > > > Spring Boot comes with an optional module known as Actuator which enables > remote administration, management, and monitoring of the application through > the REST API: > https://docs.spring.io/spring-boot/docs/current-SNAPSHOT/reference/htmlsingle/#production-ready-endpoints > This task is to enable the Actuator module and expose it in such a way that > access is gated by the standard NiFi Registry Authorization framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-registry pull request #97: NIFIREG-134 Enable SpringBoot Actuator endpo...
Github user asfgit closed the pull request at: https://github.com/apache/nifi-registry/pull/97 ---
[jira] [Resolved] (NIFIREG-134) Enable Spring Boot Actuator REST API endpoints
[ https://issues.apache.org/jira/browse/NIFIREG-134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Bende resolved NIFIREG-134. - Resolution: Fixed > Enable Spring Boot Actuator REST API endpoints > -- > > Key: NIFIREG-134 > URL: https://issues.apache.org/jira/browse/NIFIREG-134 > Project: NiFi Registry > Issue Type: New Feature >Reporter: Kevin Doran >Assignee: Kevin Doran >Priority: Minor > Fix For: 0.2.0 > > > Spring Boot comes with an optional module known as Actuator which enables > remote administration, management, and monitoring of the application through > the REST API: > https://docs.spring.io/spring-boot/docs/current-SNAPSHOT/reference/htmlsingle/#production-ready-endpoints > This task is to enable the Actuator module and expose it in such a way that > access is gated by the standard NiFi Registry Authorization framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFIREG-134) Enable Spring Boot Actuator REST API endpoints
[ https://issues.apache.org/jira/browse/NIFIREG-134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395385#comment-16395385 ] ASF GitHub Bot commented on NIFIREG-134: Github user asfgit closed the pull request at: https://github.com/apache/nifi-registry/pull/97 > Enable Spring Boot Actuator REST API endpoints > -- > > Key: NIFIREG-134 > URL: https://issues.apache.org/jira/browse/NIFIREG-134 > Project: NiFi Registry > Issue Type: New Feature >Reporter: Kevin Doran >Assignee: Kevin Doran >Priority: Minor > Fix For: 0.2.0 > > > Spring Boot comes with an optional module known as Actuator which enables > remote administration, management, and monitoring of the application through > the REST API: > https://docs.spring.io/spring-boot/docs/current-SNAPSHOT/reference/htmlsingle/#production-ready-endpoints > This task is to enable the Actuator module and expose it in such a way that > access is gated by the standard NiFi Registry Authorization framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-registry issue #97: NIFIREG-134 Enable SpringBoot Actuator endpoints
Github user kevdoran commented on the issue: https://github.com/apache/nifi-registry/pull/97 Thanks for reviewing, @bbende! I'll open JIRAs for the follow on work that builds upon this capability to make it more useful for NiFi Registry users. ---
[jira] [Commented] (NIFIREG-134) Enable Spring Boot Actuator REST API endpoints
[ https://issues.apache.org/jira/browse/NIFIREG-134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395388#comment-16395388 ] ASF GitHub Bot commented on NIFIREG-134: Github user kevdoran commented on the issue: https://github.com/apache/nifi-registry/pull/97 Thanks for reviewing, @bbende! I'll open JIRAs for the follow on work that builds upon this capability to make it more useful for NiFi Registry users. > Enable Spring Boot Actuator REST API endpoints > -- > > Key: NIFIREG-134 > URL: https://issues.apache.org/jira/browse/NIFIREG-134 > Project: NiFi Registry > Issue Type: New Feature >Reporter: Kevin Doran >Assignee: Kevin Doran >Priority: Minor > Fix For: 0.2.0 > > > Spring Boot comes with an optional module known as Actuator which enables > remote administration, management, and monitoring of the application through > the REST API: > https://docs.spring.io/spring-boot/docs/current-SNAPSHOT/reference/htmlsingle/#production-ready-endpoints > This task is to enable the Actuator module and expose it in such a way that > access is gated by the standard NiFi Registry Authorization framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-4963) Add support for Hive 3.0 processors
Matt Burgess created NIFI-4963: -- Summary: Add support for Hive 3.0 processors Key: NIFI-4963 URL: https://issues.apache.org/jira/browse/NIFI-4963 Project: Apache NiFi Issue Type: New Feature Components: Extensions Reporter: Matt Burgess Apache Hive is working on Hive 3.0, this Jira is to add a bundle of components (much like the current Hive bundle) that supports Hive 3.0 (and Apache ORC if necessary). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-1295) Add UI option to interrupt a running processor
[ https://issues.apache.org/jira/browse/NIFI-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman updated NIFI-1295: -- Issue Type: Sub-task (was: Improvement) Parent: NIFI-78 > Add UI option to interrupt a running processor > -- > > Key: NIFI-1295 > URL: https://issues.apache.org/jira/browse/NIFI-1295 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Affects Versions: 0.4.0 >Reporter: Oleg Zhurakousky >Assignee: Matt Gilman >Priority: Major > > Basically we need an expose option to a user to kill Processors that can't be > shut down the usual way (see NIFI-78 for more details). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (NIFI-4896) Add option to UI for terminating a Processor when stopped but still has threads
[ https://issues.apache.org/jira/browse/NIFI-4896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman resolved NIFI-4896. --- Resolution: Duplicate > Add option to UI for terminating a Processor when stopped but still has > threads > --- > > Key: NIFI-4896 > URL: https://issues.apache.org/jira/browse/NIFI-4896 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Mark Payne >Assignee: Matt Gilman >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #275: Minificpp 414
GitHub user achristianson opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/275 Minificpp 414 Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] If applicable, have you updated the LICENSE file? - [x] If applicable, have you updated the NOTICE file? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/achristianson/nifi-minifi-cpp MINIFICPP-414 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/275.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #275 commit 1b851e4668aeb3b3e6a3588ba22b60485052cf25 Author: Andrew I. Christianson Date: 2018-03-09T17:56:37Z MINIFICPP-422 Refactored type system in EL to preserve type information between operations commit 11addb3114baae88b9b02c8c44d8b5cda6610554 Author: Andrew I. Christianson Date: 2018-03-12T15:29:41Z MINIFICPP-414 Added boolean Expression Language functions ---
[GitHub] nifi-minifi-cpp issue #275: MINIFICPP-414 Added boolean Expression Language ...
Github user achristianson commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/275 This one depends wholly on MINIFICPP-422 and so cannot be separated from that commit. ---
[jira] [Commented] (MINIFICPP-414) Implement Expression Language boolean logic operations
[ https://issues.apache.org/jira/browse/MINIFICPP-414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395448#comment-16395448 ] ASF GitHub Bot commented on MINIFICPP-414: -- Github user achristianson commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/275 This one depends wholly on MINIFICPP-422 and so cannot be separated from that commit. > Implement Expression Language boolean logic operations > -- > > Key: MINIFICPP-414 > URL: https://issues.apache.org/jira/browse/MINIFICPP-414 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Andrew Christianson >Assignee: Andrew Christianson >Priority: Major > > [Boolean > Logic|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#boolean] > * > [isNull|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#isnull] > * > [notNull|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#notnull] > * > [isEmpty|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#isempty] > * > [equals|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#equals] > * > [equalsIgnoreCase|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#equalsignorecase] > * > [gt|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#gt] > * > [ge|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#ge] > * > [lt|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#lt] > * > [le|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#le] > * > [and|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#and] > * > [or|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#or] > * > [not|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#not] > * > [ifElse|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#ifelse] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2515: NIFI-4885: Granular component restrictions
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2515#discussion_r173858341
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/CurrentUserEndpointMerger.java
---
@@ -53,6 +54,23 @@ protected void mergeResponses(final CurrentUserEntity
clientEntity, final Map
clientEntityComponentRestrictionsPermissions =
clientEntity.getComponentRestrictionPermissions();
+final Set
entityComponentRestrictionsPermissions =
entity.getComponentRestrictionPermissions();
+
+// only retain the component restriction permissions in
common
+
clientEntityComponentRestrictionsPermissions.retainAll(entityComponentRestrictionsPermissions);
+
+// merge the component restriction permissions
+
clientEntityComponentRestrictionsPermissions.forEach(clientEntityPermission -> {
+final ComponentRestrictionPermissionDTO
entityPermission =
entityComponentRestrictionsPermissions.stream().filter(entityComponentRestrictionsPermission
-> {
+return
entityComponentRestrictionsPermission.getRequiredPermission().getId().equals(clientEntityPermission.getRequiredPermission().getId());
+}).findFirst().orElse(null);
--- End diff --
Are we guaranteed at this point that there will be at least one entry? If
so, then we should probably just use findFirst().get() because it makes this
more clear. If not, then we could end up with a null value here, and the next
line would then throw a NPE.
---
[jira] [Commented] (NIFI-4885) More granular restricted component categories
[
https://issues.apache.org/jira/browse/NIFI-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395485#comment-16395485
]
ASF GitHub Bot commented on NIFI-4885:
--
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2515#discussion_r173858341
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/CurrentUserEndpointMerger.java
---
@@ -53,6 +54,23 @@ protected void mergeResponses(final CurrentUserEntity
clientEntity, final Map
clientEntityComponentRestrictionsPermissions =
clientEntity.getComponentRestrictionPermissions();
+final Set
entityComponentRestrictionsPermissions =
entity.getComponentRestrictionPermissions();
+
+// only retain the component restriction permissions in
common
+
clientEntityComponentRestrictionsPermissions.retainAll(entityComponentRestrictionsPermissions);
+
+// merge the component restriction permissions
+
clientEntityComponentRestrictionsPermissions.forEach(clientEntityPermission -> {
+final ComponentRestrictionPermissionDTO
entityPermission =
entityComponentRestrictionsPermissions.stream().filter(entityComponentRestrictionsPermission
-> {
+return
entityComponentRestrictionsPermission.getRequiredPermission().getId().equals(clientEntityPermission.getRequiredPermission().getId());
+}).findFirst().orElse(null);
--- End diff --
Are we guaranteed at this point that there will be at least one entry? If
so, then we should probably just use findFirst().get() because it makes this
more clear. If not, then we could end up with a null value here, and the next
line would then throw a NPE.
> More granular restricted component categories
> -
>
> Key: NIFI-4885
> URL: https://issues.apache.org/jira/browse/NIFI-4885
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework, Core UI
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Major
>
> Update the Restricted annotation to support more granular categories.
> Available categories will map to new access policies. Example categories and
> their corresponding access policies may be
> * read-filesystem (/restricted-components/read-filesystem)
> * write-filesystem (/restricted-components/write-filesystem)
> * code-execution (/restricted-components/code-execution)
> * keytab-access (/restricted-components/keytab-access)
> The hierarchical nature of the access policies will support backward
> compatibility with existing installations where the policy of
> /restricted-components was used to enforce all subcategories. Any users with
> /restricted-components permissions will be granted access to all
> subcategories. In order to leverage the new granular categories, an
> administrator will need to use NiFi to update their access policies (remove a
> user from /restricted-components and place them into the desired subcategory)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2515: NIFI-4885: Granular component restrictions
Github user mcgilman commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2515#discussion_r173869295
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/CurrentUserEndpointMerger.java
---
@@ -53,6 +54,23 @@ protected void mergeResponses(final CurrentUserEntity
clientEntity, final Map
clientEntityComponentRestrictionsPermissions =
clientEntity.getComponentRestrictionPermissions();
+final Set
entityComponentRestrictionsPermissions =
entity.getComponentRestrictionPermissions();
+
+// only retain the component restriction permissions in
common
+
clientEntityComponentRestrictionsPermissions.retainAll(entityComponentRestrictionsPermissions);
+
+// merge the component restriction permissions
+
clientEntityComponentRestrictionsPermissions.forEach(clientEntityPermission -> {
+final ComponentRestrictionPermissionDTO
entityPermission =
entityComponentRestrictionsPermissions.stream().filter(entityComponentRestrictionsPermission
-> {
+return
entityComponentRestrictionsPermission.getRequiredPermission().getId().equals(clientEntityPermission.getRequiredPermission().getId());
+}).findFirst().orElse(null);
--- End diff --
Because we're doing a retainAll right before this we know that both
collections will each have an entry for the current clientEntityPermission. I
will update to use get() instead.
---
[jira] [Commented] (NIFI-4885) More granular restricted component categories
[
https://issues.apache.org/jira/browse/NIFI-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395501#comment-16395501
]
ASF GitHub Bot commented on NIFI-4885:
--
Github user mcgilman commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2515#discussion_r173869295
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/endpoints/CurrentUserEndpointMerger.java
---
@@ -53,6 +54,23 @@ protected void mergeResponses(final CurrentUserEntity
clientEntity, final Map
clientEntityComponentRestrictionsPermissions =
clientEntity.getComponentRestrictionPermissions();
+final Set
entityComponentRestrictionsPermissions =
entity.getComponentRestrictionPermissions();
+
+// only retain the component restriction permissions in
common
+
clientEntityComponentRestrictionsPermissions.retainAll(entityComponentRestrictionsPermissions);
+
+// merge the component restriction permissions
+
clientEntityComponentRestrictionsPermissions.forEach(clientEntityPermission -> {
+final ComponentRestrictionPermissionDTO
entityPermission =
entityComponentRestrictionsPermissions.stream().filter(entityComponentRestrictionsPermission
-> {
+return
entityComponentRestrictionsPermission.getRequiredPermission().getId().equals(clientEntityPermission.getRequiredPermission().getId());
+}).findFirst().orElse(null);
--- End diff --
Because we're doing a retainAll right before this we know that both
collections will each have an entry for the current clientEntityPermission. I
will update to use get() instead.
> More granular restricted component categories
> -
>
> Key: NIFI-4885
> URL: https://issues.apache.org/jira/browse/NIFI-4885
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework, Core UI
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Major
>
> Update the Restricted annotation to support more granular categories.
> Available categories will map to new access policies. Example categories and
> their corresponding access policies may be
> * read-filesystem (/restricted-components/read-filesystem)
> * write-filesystem (/restricted-components/write-filesystem)
> * code-execution (/restricted-components/code-execution)
> * keytab-access (/restricted-components/keytab-access)
> The hierarchical nature of the access policies will support backward
> compatibility with existing installations where the policy of
> /restricted-components was used to enforce all subcategories. Any users with
> /restricted-components permissions will be granted access to all
> subcategories. In order to leverage the new granular categories, an
> administrator will need to use NiFi to update their access policies (remove a
> user from /restricted-components and place them into the desired subcategory)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4944) PutHiveStreaming multiple instances with Snappy fail intermittently
[
https://issues.apache.org/jira/browse/NIFI-4944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395547#comment-16395547
]
ASF subversion and git services commented on NIFI-4944:
---
Commit d4632bdd5dce85cc7adb8c70bafda44d6a333da9 in nifi's branch
refs/heads/master from [~ca9mbu]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=d4632bd ]
NIFI-4944: Guard against race condition in Snappy for PutHiveStreaming
NIFI-4944: Removed unnecessary synchronized block, added more comments
This closes #2519
> PutHiveStreaming multiple instances with Snappy fail intermittently
> ---
>
> Key: NIFI-4944
> URL: https://issues.apache.org/jira/browse/NIFI-4944
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> When data coming into PutHiveStreaming is compressed with Snappy, then
> multiple instances of PutHiveStreaming in a flow can cause a failure, the log
> often shows the following:
> {{org.apache.nifi.processors.hive.PutHiveStreaming$$Lambda$510/1467586448@68a5884d
> failed to process due to org.xerial.snappy.SnappyError:
> [FAILED_TO_LOAD_NATIVE_LIBRARY] null; rolling back session: {}}}
> This is due to a race condition in Snappy 1.0.5 (the version used by the Hive
> NAR) where two classloaders can try to define the native loader class, thus
> the second one would fail, giving the error above.
> The proposed solution is to guarantee that Snappy is loaded before this
> situation is encountered (i.e. before the InstanceClassLoaders are created).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #2519: NIFI-4944: Guard against race condition in Snappy for PutH...
Github user mcgilman commented on the issue: https://github.com/apache/nifi/pull/2519 Thanks @mattyb149! This has been merged to master. ---
[jira] [Commented] (NIFI-4944) PutHiveStreaming multiple instances with Snappy fail intermittently
[
https://issues.apache.org/jira/browse/NIFI-4944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395548#comment-16395548
]
ASF subversion and git services commented on NIFI-4944:
---
Commit d4632bdd5dce85cc7adb8c70bafda44d6a333da9 in nifi's branch
refs/heads/master from [~ca9mbu]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=d4632bd ]
NIFI-4944: Guard against race condition in Snappy for PutHiveStreaming
NIFI-4944: Removed unnecessary synchronized block, added more comments
This closes #2519
> PutHiveStreaming multiple instances with Snappy fail intermittently
> ---
>
> Key: NIFI-4944
> URL: https://issues.apache.org/jira/browse/NIFI-4944
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> When data coming into PutHiveStreaming is compressed with Snappy, then
> multiple instances of PutHiveStreaming in a flow can cause a failure, the log
> often shows the following:
> {{org.apache.nifi.processors.hive.PutHiveStreaming$$Lambda$510/1467586448@68a5884d
> failed to process due to org.xerial.snappy.SnappyError:
> [FAILED_TO_LOAD_NATIVE_LIBRARY] null; rolling back session: {}}}
> This is due to a race condition in Snappy 1.0.5 (the version used by the Hive
> NAR) where two classloaders can try to define the native loader class, thus
> the second one would fail, giving the error above.
> The proposed solution is to guarantee that Snappy is loaded before this
> situation is encountered (i.e. before the InstanceClassLoaders are created).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2519: NIFI-4944: Guard against race condition in Snappy f...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2519 ---
[jira] [Updated] (NIFI-4944) PutHiveStreaming multiple instances with Snappy fail intermittently
[
https://issues.apache.org/jira/browse/NIFI-4944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Gilman updated NIFI-4944:
--
Resolution: Fixed
Fix Version/s: 1.6.0
Status: Resolved (was: Patch Available)
> PutHiveStreaming multiple instances with Snappy fail intermittently
> ---
>
> Key: NIFI-4944
> URL: https://issues.apache.org/jira/browse/NIFI-4944
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
> Fix For: 1.6.0
>
>
> When data coming into PutHiveStreaming is compressed with Snappy, then
> multiple instances of PutHiveStreaming in a flow can cause a failure, the log
> often shows the following:
> {{org.apache.nifi.processors.hive.PutHiveStreaming$$Lambda$510/1467586448@68a5884d
> failed to process due to org.xerial.snappy.SnappyError:
> [FAILED_TO_LOAD_NATIVE_LIBRARY] null; rolling back session: {}}}
> This is due to a race condition in Snappy 1.0.5 (the version used by the Hive
> NAR) where two classloaders can try to define the native loader class, thus
> the second one would fail, giving the error above.
> The proposed solution is to guarantee that Snappy is loaded before this
> situation is encountered (i.e. before the InstanceClassLoaders are created).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4944) PutHiveStreaming multiple instances with Snappy fail intermittently
[
https://issues.apache.org/jira/browse/NIFI-4944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395549#comment-16395549
]
ASF GitHub Bot commented on NIFI-4944:
--
Github user mcgilman commented on the issue:
https://github.com/apache/nifi/pull/2519
Thanks @mattyb149! This has been merged to master.
> PutHiveStreaming multiple instances with Snappy fail intermittently
> ---
>
> Key: NIFI-4944
> URL: https://issues.apache.org/jira/browse/NIFI-4944
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
> Fix For: 1.6.0
>
>
> When data coming into PutHiveStreaming is compressed with Snappy, then
> multiple instances of PutHiveStreaming in a flow can cause a failure, the log
> often shows the following:
> {{org.apache.nifi.processors.hive.PutHiveStreaming$$Lambda$510/1467586448@68a5884d
> failed to process due to org.xerial.snappy.SnappyError:
> [FAILED_TO_LOAD_NATIVE_LIBRARY] null; rolling back session: {}}}
> This is due to a race condition in Snappy 1.0.5 (the version used by the Hive
> NAR) where two classloaders can try to define the native loader class, thus
> the second one would fail, giving the error above.
> The proposed solution is to guarantee that Snappy is loaded before this
> situation is encountered (i.e. before the InstanceClassLoaders are created).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4944) PutHiveStreaming multiple instances with Snappy fail intermittently
[
https://issues.apache.org/jira/browse/NIFI-4944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395550#comment-16395550
]
ASF GitHub Bot commented on NIFI-4944:
--
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/2519
> PutHiveStreaming multiple instances with Snappy fail intermittently
> ---
>
> Key: NIFI-4944
> URL: https://issues.apache.org/jira/browse/NIFI-4944
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
> Fix For: 1.6.0
>
>
> When data coming into PutHiveStreaming is compressed with Snappy, then
> multiple instances of PutHiveStreaming in a flow can cause a failure, the log
> often shows the following:
> {{org.apache.nifi.processors.hive.PutHiveStreaming$$Lambda$510/1467586448@68a5884d
> failed to process due to org.xerial.snappy.SnappyError:
> [FAILED_TO_LOAD_NATIVE_LIBRARY] null; rolling back session: {}}}
> This is due to a race condition in Snappy 1.0.5 (the version used by the Hive
> NAR) where two classloaders can try to define the native loader class, thus
> the second one would fail, giving the error above.
> The proposed solution is to guarantee that Snappy is loaded before this
> situation is encountered (i.e. before the InstanceClassLoaders are created).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (MINIFICPP-423) Implement encode/decode EL functions
Andrew Christianson created MINIFICPP-423: - Summary: Implement encode/decode EL functions Key: MINIFICPP-423 URL: https://issues.apache.org/jira/browse/MINIFICPP-423 Project: NiFi MiNiFi C++ Issue Type: Improvement Reporter: Andrew Christianson Assignee: Andrew Christianson [Encode/Decode Functions|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#encode] * [escapeJson|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#escapejson] * [escapeXml|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#escapexml] * [escapeCsv|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#escapecsv] * [escapeHtml3|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#escapehtml3] * [escapeHtml4|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#escapehtml4] * [unescapeJson|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#unescapejson] * [unescapeXml|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#unescapexml] * [unescapeCsv|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#unescapecsv] * [unescapeHtml3|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#unescapehtml3] * [unescapeHtml4|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#unescapehtml4] * [urlEncode|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#urlencode] * [urlDecode|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#urldecode] * [base64Encode|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#base64encode] * [base64Decode|https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#base64decode] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2519: NIFI-4944: Guard against race condition in Snappy for PutH...
Github user moonkev commented on the issue: https://github.com/apache/nifi/pull/2519 I was aware that PutHiveStreaming used snappy-java vs native snappy, but was unaware of the methods that snappy-java protect against loading in multiple class loaders. Many thanks for the detailed explanation @mcgilman! ---
[jira] [Commented] (NIFI-4944) PutHiveStreaming multiple instances with Snappy fail intermittently
[
https://issues.apache.org/jira/browse/NIFI-4944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395575#comment-16395575
]
ASF GitHub Bot commented on NIFI-4944:
--
Github user moonkev commented on the issue:
https://github.com/apache/nifi/pull/2519
I was aware that PutHiveStreaming used snappy-java vs native snappy, but
was unaware of the methods that snappy-java protect against loading in multiple
class loaders. Many thanks for the detailed explanation @mcgilman!
> PutHiveStreaming multiple instances with Snappy fail intermittently
> ---
>
> Key: NIFI-4944
> URL: https://issues.apache.org/jira/browse/NIFI-4944
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
> Fix For: 1.6.0
>
>
> When data coming into PutHiveStreaming is compressed with Snappy, then
> multiple instances of PutHiveStreaming in a flow can cause a failure, the log
> often shows the following:
> {{org.apache.nifi.processors.hive.PutHiveStreaming$$Lambda$510/1467586448@68a5884d
> failed to process due to org.xerial.snappy.SnappyError:
> [FAILED_TO_LOAD_NATIVE_LIBRARY] null; rolling back session: {}}}
> This is due to a race condition in Snappy 1.0.5 (the version used by the Hive
> NAR) where two classloaders can try to define the native loader class, thus
> the second one would fail, giving the error above.
> The proposed solution is to guarantee that Snappy is loaded before this
> situation is encountered (i.e. before the InstanceClassLoaders are created).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2113 Hey @mattyb149, what's your status for this review? From a cursory look, it appears like it just needs an updated pom from @MikeThomsen and the final approval. With talks of 1.6.0 happening it would be nice to get this in so those using ES 6 aren't limited to the HTTP processor. If help is needed to finalize things just let me know where I can help. ---
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395597#comment-16395597 ] ASF GitHub Bot commented on NIFI-4325: -- Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2113 Hey @mattyb149, what's your status for this review? From a cursory look, it appears like it just needs an updated pom from @MikeThomsen and the final approval. With talks of 1.6.0 happening it would be nice to get this in so those using ES 6 aren't limited to the HTTP processor. If help is needed to finalize things just let me know where I can help. > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2113: NIFI-4325 Added new processor that uses the JSON DSL.
Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/2113 I'm not sure I'll have time to close the loop before 1.6.0, so if you'd like to finish the review/merge after Mike's rebase that would be very cool, thanks! ---
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395609#comment-16395609 ] ASF GitHub Bot commented on NIFI-4325: -- Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/2113 I'm not sure I'll have time to close the loop before 1.6.0, so if you'd like to finish the review/merge after Mike's rebase that would be very cool, thanks! > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2497: NIFI-3599 Allowed back pressure object count and data size...
Github user mcgilman commented on the issue: https://github.com/apache/nifi/pull/2497 @mosermw @markap14 It's kind of a gray area. I suggested an endpoint like the /nifi-api/flow/about because that seemed like the best fit currently and was obviously trying to avoid requiring another request to load the UI. I'm a little hesitant to go the route of a /nifi-api/flow/properties endpoint because I'm not sure that's a concept we want to advertise/expose. I don't mind doing something more generic, but what if the concept was more related to default values or config. This is something that any client using the API may want to know ahead of time for this reason exactly. Would adding a DefaultsDTO or ConfigDTO which is set on the AboutDTO fit a little better? ---
[jira] [Commented] (NIFI-3599) Add nifi.properties value to globally set the default backpressure size threshold for each connection
[ https://issues.apache.org/jira/browse/NIFI-3599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395645#comment-16395645 ] ASF GitHub Bot commented on NIFI-3599: -- Github user mcgilman commented on the issue: https://github.com/apache/nifi/pull/2497 @mosermw @markap14 It's kind of a gray area. I suggested an endpoint like the /nifi-api/flow/about because that seemed like the best fit currently and was obviously trying to avoid requiring another request to load the UI. I'm a little hesitant to go the route of a /nifi-api/flow/properties endpoint because I'm not sure that's a concept we want to advertise/expose. I don't mind doing something more generic, but what if the concept was more related to default values or config. This is something that any client using the API may want to know ahead of time for this reason exactly. Would adding a DefaultsDTO or ConfigDTO which is set on the AboutDTO fit a little better? > Add nifi.properties value to globally set the default backpressure size > threshold for each connection > - > > Key: NIFI-3599 > URL: https://issues.apache.org/jira/browse/NIFI-3599 > Project: Apache NiFi > Issue Type: Improvement > Components: Configuration >Reporter: Jeremy Dyer >Assignee: Michael Moser >Priority: Major > > By default each new connection added to the workflow canvas will have a > default backpressure size threshold of 10,000 objects. While the threshold > can be changed on a connection level it would be convenient to have a global > mechanism for setting that value to something other than 10,000. This > enhancement would add a property to nifi.properties that would allow for this > threshold to be set globally unless otherwise overridden at the connection > level. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4885) More granular restricted component categories
[ https://issues.apache.org/jira/browse/NIFI-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395648#comment-16395648 ] ASF subversion and git services commented on NIFI-4885: --- Commit b1217f529bfc5ea9296d1d55c6b0fe92a881a485 in nifi's branch refs/heads/master from [~mcgilman] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=b1217f5 ] NIFI-4885: - Introducing more granular restricted component access policies. This closes #2515. Signed-off-by: Mark Payne > More granular restricted component categories > - > > Key: NIFI-4885 > URL: https://issues.apache.org/jira/browse/NIFI-4885 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Core UI >Reporter: Matt Gilman >Assignee: Matt Gilman >Priority: Major > > Update the Restricted annotation to support more granular categories. > Available categories will map to new access policies. Example categories and > their corresponding access policies may be > * read-filesystem (/restricted-components/read-filesystem) > * write-filesystem (/restricted-components/write-filesystem) > * code-execution (/restricted-components/code-execution) > * keytab-access (/restricted-components/keytab-access) > The hierarchical nature of the access policies will support backward > compatibility with existing installations where the policy of > /restricted-components was used to enforce all subcategories. Any users with > /restricted-components permissions will be granted access to all > subcategories. In order to leverage the new granular categories, an > administrator will need to use NiFi to update their access policies (remove a > user from /restricted-components and place them into the desired subcategory) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2515: NIFI-4885: Granular component restrictions
Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/2515 @mcgilman this all looks good to me as well! Given positive review feedback from @andrewmlim and a +1 from @scottyaslan I am happy with the changes and think this is a great improvement on our security model. +1 merged to master. Thanks! ---
[jira] [Commented] (NIFI-4885) More granular restricted component categories
[ https://issues.apache.org/jira/browse/NIFI-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395653#comment-16395653 ] ASF GitHub Bot commented on NIFI-4885: -- Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/2515 @mcgilman this all looks good to me as well! Given positive review feedback from @andrewmlim and a +1 from @scottyaslan I am happy with the changes and think this is a great improvement on our security model. +1 merged to master. Thanks! > More granular restricted component categories > - > > Key: NIFI-4885 > URL: https://issues.apache.org/jira/browse/NIFI-4885 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Core UI >Reporter: Matt Gilman >Assignee: Matt Gilman >Priority: Major > Fix For: 1.6.0 > > > Update the Restricted annotation to support more granular categories. > Available categories will map to new access policies. Example categories and > their corresponding access policies may be > * read-filesystem (/restricted-components/read-filesystem) > * write-filesystem (/restricted-components/write-filesystem) > * code-execution (/restricted-components/code-execution) > * keytab-access (/restricted-components/keytab-access) > The hierarchical nature of the access policies will support backward > compatibility with existing installations where the policy of > /restricted-components was used to enforce all subcategories. Any users with > /restricted-components permissions will be granted access to all > subcategories. In order to leverage the new granular categories, an > administrator will need to use NiFi to update their access policies (remove a > user from /restricted-components and place them into the desired subcategory) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (NIFI-4885) More granular restricted component categories
[ https://issues.apache.org/jira/browse/NIFI-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne resolved NIFI-4885. -- Resolution: Fixed Fix Version/s: 1.6.0 > More granular restricted component categories > - > > Key: NIFI-4885 > URL: https://issues.apache.org/jira/browse/NIFI-4885 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Core UI >Reporter: Matt Gilman >Assignee: Matt Gilman >Priority: Major > Fix For: 1.6.0 > > > Update the Restricted annotation to support more granular categories. > Available categories will map to new access policies. Example categories and > their corresponding access policies may be > * read-filesystem (/restricted-components/read-filesystem) > * write-filesystem (/restricted-components/write-filesystem) > * code-execution (/restricted-components/code-execution) > * keytab-access (/restricted-components/keytab-access) > The hierarchical nature of the access policies will support backward > compatibility with existing installations where the policy of > /restricted-components was used to enforce all subcategories. Any users with > /restricted-components permissions will be granted access to all > subcategories. In order to leverage the new granular categories, an > administrator will need to use NiFi to update their access policies (remove a > user from /restricted-components and place them into the desired subcategory) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2468: NIFI-4849: Implemented REST Endpoint and associated...
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2468#discussion_r173900631
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/dto/EntityFactory.java
---
@@ -77,6 +79,27 @@
public final class EntityFactory {
+public ProcessorDiagnosticsEntity
createProcessorDiagnosticsEntity(final ProcessorDiagnosticsDTO dto, final
RevisionDTO revision, final PermissionsDTO processorPermissions,
+final ProcessorStatusDTO status, final List
bulletins) {
+final ProcessorDiagnosticsEntity entity = new
ProcessorDiagnosticsEntity();
+entity.setRevision(revision);
+if (dto != null) {
+entity.setPermissions(processorPermissions);
+entity.setId(dto.getProcessor().getId());
+if (processorPermissions != null &&
processorPermissions.getCanRead()) {
+entity.setComponent(dto);
+entity.setBulletins(bulletins);
+}
+}
+
+entity.setBulletins(bulletins);
+return entity;
+}
+
+private void pairDownDiagnostics(final ProcessorDiagnosticsDTO dto,
final PermissionsDTO controllerPermissions) {
--- End diff --
Whoops, yes, good call.
---
[jira] [Commented] (NIFI-4849) Add REST Endpoint for gathering Processor Diagnostics information
[
https://issues.apache.org/jira/browse/NIFI-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395666#comment-16395666
]
ASF GitHub Bot commented on NIFI-4849:
--
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2468#discussion_r173900631
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/dto/EntityFactory.java
---
@@ -77,6 +79,27 @@
public final class EntityFactory {
+public ProcessorDiagnosticsEntity
createProcessorDiagnosticsEntity(final ProcessorDiagnosticsDTO dto, final
RevisionDTO revision, final PermissionsDTO processorPermissions,
+final ProcessorStatusDTO status, final List
bulletins) {
+final ProcessorDiagnosticsEntity entity = new
ProcessorDiagnosticsEntity();
+entity.setRevision(revision);
+if (dto != null) {
+entity.setPermissions(processorPermissions);
+entity.setId(dto.getProcessor().getId());
+if (processorPermissions != null &&
processorPermissions.getCanRead()) {
+entity.setComponent(dto);
+entity.setBulletins(bulletins);
+}
+}
+
+entity.setBulletins(bulletins);
+return entity;
+}
+
+private void pairDownDiagnostics(final ProcessorDiagnosticsDTO dto,
final PermissionsDTO controllerPermissions) {
--- End diff --
Whoops, yes, good call.
> Add REST Endpoint for gathering Processor Diagnostics information
> -
>
> Key: NIFI-4849
> URL: https://issues.apache.org/jira/browse/NIFI-4849
> Project: Apache NiFi
> Issue Type: Sub-task
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.6.0
>
>
> We need to add a REST endpoint that will use the appropriate resources to
> gather the Processor Diagnostics information. Information to return should
> include things like:
> * Processor config
> * Processor status
> * Garbage Collection info
> * Repo Sizes
> * Connection info for connections whose source or destination is the
> processor
> * Controller Services that the processor is referencing
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2468: NIFI-4849: Implemented REST Endpoint and associated...
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2468#discussion_r173902301
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java
---
@@ -4506,6 +4515,123 @@ public ComponentHistoryDTO
getComponentHistory(final String componentId) {
return history;
}
+private ControllerServiceEntity createControllerServiceEntity(final
String serviceId, final NiFiUser user) {
+final ControllerServiceNode serviceNode =
controllerServiceDAO.getControllerService(serviceId);
+return createControllerServiceEntity(serviceNode,
Collections.emptySet(), user);
+}
+
+@Override
+public ProcessorDiagnosticsEntity getProcessorDiagnostics(final String
id) {
+final ProcessorNode processor = processorDAO.getProcessor(id);
+final ProcessorStatus processorStatus =
controllerFacade.getProcessorStatus(id);
+
+// Generate Processor Diagnostics
+final NiFiUser user = NiFiUserUtils.getNiFiUser();
+final ProcessorDiagnosticsDTO dto =
controllerFacade.getProcessorDiagnostics(processor, processorStatus,
bulletinRepository, serviceId -> createControllerServiceEntity(serviceId,
user));
+
+// Filter anything out of diagnostics that the user is not
authorized to see.
+final List jvmDiagnosticsSnaphots = new
ArrayList<>();
+final JVMDiagnosticsDTO jvmDiagnostics = dto.getJvmDiagnostics();
+jvmDiagnosticsSnaphots.add(jvmDiagnostics.getAggregateSnapshot());
+
+// filter controller-related information
+final boolean canReadController =
authorizableLookup.getController().isAuthorized(authorizer, RequestAction.READ,
user);
+if (!canReadController) {
+for (final JVMDiagnosticsSnapshotDTO snapshot :
jvmDiagnosticsSnaphots) {
--- End diff --
I don't think that active event & timer driven threads need to be filtered
because they are available to anyone with access to /flow. I also feel like
uptime should be available. I could see an argument for the elected primary &
coordinator not being included, but I went back and forth on that a bit
personally, because currently that info is not exposed anywhere except if you
have that permissions. It seemed quite benign to me to include this, but If you
think they should be removed I am okay with it.
---
[GitHub] nifi pull request #2468: NIFI-4849: Implemented REST Endpoint and associated...
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2468#discussion_r173902565
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java
---
@@ -4506,6 +4515,123 @@ public ComponentHistoryDTO
getComponentHistory(final String componentId) {
return history;
}
+private ControllerServiceEntity createControllerServiceEntity(final
String serviceId, final NiFiUser user) {
+final ControllerServiceNode serviceNode =
controllerServiceDAO.getControllerService(serviceId);
+return createControllerServiceEntity(serviceNode,
Collections.emptySet(), user);
+}
+
+@Override
+public ProcessorDiagnosticsEntity getProcessorDiagnostics(final String
id) {
+final ProcessorNode processor = processorDAO.getProcessor(id);
+final ProcessorStatus processorStatus =
controllerFacade.getProcessorStatus(id);
+
+// Generate Processor Diagnostics
+final NiFiUser user = NiFiUserUtils.getNiFiUser();
+final ProcessorDiagnosticsDTO dto =
controllerFacade.getProcessorDiagnostics(processor, processorStatus,
bulletinRepository, serviceId -> createControllerServiceEntity(serviceId,
user));
+
+// Filter anything out of diagnostics that the user is not
authorized to see.
+final List jvmDiagnosticsSnaphots = new
ArrayList<>();
+final JVMDiagnosticsDTO jvmDiagnostics = dto.getJvmDiagnostics();
+jvmDiagnosticsSnaphots.add(jvmDiagnostics.getAggregateSnapshot());
+
+// filter controller-related information
+final boolean canReadController =
authorizableLookup.getController().isAuthorized(authorizer, RequestAction.READ,
user);
+if (!canReadController) {
+for (final JVMDiagnosticsSnapshotDTO snapshot :
jvmDiagnosticsSnaphots) {
+snapshot.setMaxEventDrivenThreads(null);
+snapshot.setMaxTimerDrivenThreads(null);
+snapshot.setBundlesLoaded(null);
+}
+}
+
+// filter system diagnostics information
+final boolean canReadSystem =
authorizableLookup.getSystem().isAuthorized(authorizer, RequestAction.READ,
user);
+if (!canReadSystem) {
+for (final JVMDiagnosticsSnapshotDTO snapshot :
jvmDiagnosticsSnaphots) {
--- End diff --
That's a good call. Will update that.
---
[jira] [Commented] (NIFI-4849) Add REST Endpoint for gathering Processor Diagnostics information
[
https://issues.apache.org/jira/browse/NIFI-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395672#comment-16395672
]
ASF GitHub Bot commented on NIFI-4849:
--
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2468#discussion_r173902565
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java
---
@@ -4506,6 +4515,123 @@ public ComponentHistoryDTO
getComponentHistory(final String componentId) {
return history;
}
+private ControllerServiceEntity createControllerServiceEntity(final
String serviceId, final NiFiUser user) {
+final ControllerServiceNode serviceNode =
controllerServiceDAO.getControllerService(serviceId);
+return createControllerServiceEntity(serviceNode,
Collections.emptySet(), user);
+}
+
+@Override
+public ProcessorDiagnosticsEntity getProcessorDiagnostics(final String
id) {
+final ProcessorNode processor = processorDAO.getProcessor(id);
+final ProcessorStatus processorStatus =
controllerFacade.getProcessorStatus(id);
+
+// Generate Processor Diagnostics
+final NiFiUser user = NiFiUserUtils.getNiFiUser();
+final ProcessorDiagnosticsDTO dto =
controllerFacade.getProcessorDiagnostics(processor, processorStatus,
bulletinRepository, serviceId -> createControllerServiceEntity(serviceId,
user));
+
+// Filter anything out of diagnostics that the user is not
authorized to see.
+final List jvmDiagnosticsSnaphots = new
ArrayList<>();
+final JVMDiagnosticsDTO jvmDiagnostics = dto.getJvmDiagnostics();
+jvmDiagnosticsSnaphots.add(jvmDiagnostics.getAggregateSnapshot());
+
+// filter controller-related information
+final boolean canReadController =
authorizableLookup.getController().isAuthorized(authorizer, RequestAction.READ,
user);
+if (!canReadController) {
+for (final JVMDiagnosticsSnapshotDTO snapshot :
jvmDiagnosticsSnaphots) {
+snapshot.setMaxEventDrivenThreads(null);
+snapshot.setMaxTimerDrivenThreads(null);
+snapshot.setBundlesLoaded(null);
+}
+}
+
+// filter system diagnostics information
+final boolean canReadSystem =
authorizableLookup.getSystem().isAuthorized(authorizer, RequestAction.READ,
user);
+if (!canReadSystem) {
+for (final JVMDiagnosticsSnapshotDTO snapshot :
jvmDiagnosticsSnaphots) {
--- End diff --
That's a good call. Will update that.
> Add REST Endpoint for gathering Processor Diagnostics information
> -
>
> Key: NIFI-4849
> URL: https://issues.apache.org/jira/browse/NIFI-4849
> Project: Apache NiFi
> Issue Type: Sub-task
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.6.0
>
>
> We need to add a REST endpoint that will use the appropriate resources to
> gather the Processor Diagnostics information. Information to return should
> include things like:
> * Processor config
> * Processor status
> * Garbage Collection info
> * Repo Sizes
> * Connection info for connections whose source or destination is the
> processor
> * Controller Services that the processor is referencing
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4849) Add REST Endpoint for gathering Processor Diagnostics information
[
https://issues.apache.org/jira/browse/NIFI-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395671#comment-16395671
]
ASF GitHub Bot commented on NIFI-4849:
--
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2468#discussion_r173902301
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java
---
@@ -4506,6 +4515,123 @@ public ComponentHistoryDTO
getComponentHistory(final String componentId) {
return history;
}
+private ControllerServiceEntity createControllerServiceEntity(final
String serviceId, final NiFiUser user) {
+final ControllerServiceNode serviceNode =
controllerServiceDAO.getControllerService(serviceId);
+return createControllerServiceEntity(serviceNode,
Collections.emptySet(), user);
+}
+
+@Override
+public ProcessorDiagnosticsEntity getProcessorDiagnostics(final String
id) {
+final ProcessorNode processor = processorDAO.getProcessor(id);
+final ProcessorStatus processorStatus =
controllerFacade.getProcessorStatus(id);
+
+// Generate Processor Diagnostics
+final NiFiUser user = NiFiUserUtils.getNiFiUser();
+final ProcessorDiagnosticsDTO dto =
controllerFacade.getProcessorDiagnostics(processor, processorStatus,
bulletinRepository, serviceId -> createControllerServiceEntity(serviceId,
user));
+
+// Filter anything out of diagnostics that the user is not
authorized to see.
+final List jvmDiagnosticsSnaphots = new
ArrayList<>();
+final JVMDiagnosticsDTO jvmDiagnostics = dto.getJvmDiagnostics();
+jvmDiagnosticsSnaphots.add(jvmDiagnostics.getAggregateSnapshot());
+
+// filter controller-related information
+final boolean canReadController =
authorizableLookup.getController().isAuthorized(authorizer, RequestAction.READ,
user);
+if (!canReadController) {
+for (final JVMDiagnosticsSnapshotDTO snapshot :
jvmDiagnosticsSnaphots) {
--- End diff --
I don't think that active event & timer driven threads need to be filtered
because they are available to anyone with access to /flow. I also feel like
uptime should be available. I could see an argument for the elected primary &
coordinator not being included, but I went back and forth on that a bit
personally, because currently that info is not exposed anywhere except if you
have that permissions. It seemed quite benign to me to include this, but If you
think they should be removed I am okay with it.
> Add REST Endpoint for gathering Processor Diagnostics information
> -
>
> Key: NIFI-4849
> URL: https://issues.apache.org/jira/browse/NIFI-4849
> Project: Apache NiFi
> Issue Type: Sub-task
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.6.0
>
>
> We need to add a REST endpoint that will use the appropriate resources to
> gather the Processor Diagnostics information. Information to return should
> include things like:
> * Processor config
> * Processor status
> * Garbage Collection info
> * Repo Sizes
> * Connection info for connections whose source or destination is the
> processor
> * Controller Services that the processor is referencing
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2468: NIFI-4849: Implemented REST Endpoint and associated...
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2468#discussion_r173906446
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java
---
@@ -4506,6 +4515,123 @@ public ComponentHistoryDTO
getComponentHistory(final String componentId) {
return history;
}
+private ControllerServiceEntity createControllerServiceEntity(final
String serviceId, final NiFiUser user) {
+final ControllerServiceNode serviceNode =
controllerServiceDAO.getControllerService(serviceId);
+return createControllerServiceEntity(serviceNode,
Collections.emptySet(), user);
+}
+
+@Override
+public ProcessorDiagnosticsEntity getProcessorDiagnostics(final String
id) {
+final ProcessorNode processor = processorDAO.getProcessor(id);
+final ProcessorStatus processorStatus =
controllerFacade.getProcessorStatus(id);
+
+// Generate Processor Diagnostics
+final NiFiUser user = NiFiUserUtils.getNiFiUser();
+final ProcessorDiagnosticsDTO dto =
controllerFacade.getProcessorDiagnostics(processor, processorStatus,
bulletinRepository, serviceId -> createControllerServiceEntity(serviceId,
user));
+
+// Filter anything out of diagnostics that the user is not
authorized to see.
+final List jvmDiagnosticsSnaphots = new
ArrayList<>();
+final JVMDiagnosticsDTO jvmDiagnostics = dto.getJvmDiagnostics();
+jvmDiagnosticsSnaphots.add(jvmDiagnostics.getAggregateSnapshot());
+
+// filter controller-related information
+final boolean canReadController =
authorizableLookup.getController().isAuthorized(authorizer, RequestAction.READ,
user);
+if (!canReadController) {
+for (final JVMDiagnosticsSnapshotDTO snapshot :
jvmDiagnosticsSnaphots) {
+snapshot.setMaxEventDrivenThreads(null);
+snapshot.setMaxTimerDrivenThreads(null);
+snapshot.setBundlesLoaded(null);
+}
+}
+
+// filter system diagnostics information
+final boolean canReadSystem =
authorizableLookup.getSystem().isAuthorized(authorizer, RequestAction.READ,
user);
+if (!canReadSystem) {
+for (final JVMDiagnosticsSnapshotDTO snapshot :
jvmDiagnosticsSnaphots) {
+snapshot.setContentRepositoryStorageUsage(null);
+snapshot.setCpuCores(null);
+snapshot.setCpuLoadAverage(null);
+snapshot.setFlowFileRepositoryStorageUsage(null);
+snapshot.setMaxHeap(null);
+snapshot.setMaxHeapBytes(null);
+snapshot.setProvenanceRepositoryStorageUsage(null);
+snapshot.setPhysicalMemory(null);
+snapshot.setPhysicalMemoryBytes(null);
+snapshot.setGarbageCollectionDiagnostics(null);
+}
+}
+
+// filter connections
+final Predicate connectionAuthorized =
connectionDiagnostics -> {
+final String connectionId =
connectionDiagnostics.getConnection().getId();
+return
authorizableLookup.getConnection(connectionId).getAuthorizable().isAuthorized(authorizer,
RequestAction.READ, user);
+};
+
+// Function that can be used to remove the Source or Destination
of a ConnectionDTO, if the user is not authorized.
+final Function
filterSourceDestination = connectionDiagnostics -> {
--- End diff --
Good call. That would mean that the second Function there is not really
needed. Will address.
---
[jira] [Commented] (NIFI-4849) Add REST Endpoint for gathering Processor Diagnostics information
[
https://issues.apache.org/jira/browse/NIFI-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395691#comment-16395691
]
ASF GitHub Bot commented on NIFI-4849:
--
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2468#discussion_r173906446
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/StandardNiFiServiceFacade.java
---
@@ -4506,6 +4515,123 @@ public ComponentHistoryDTO
getComponentHistory(final String componentId) {
return history;
}
+private ControllerServiceEntity createControllerServiceEntity(final
String serviceId, final NiFiUser user) {
+final ControllerServiceNode serviceNode =
controllerServiceDAO.getControllerService(serviceId);
+return createControllerServiceEntity(serviceNode,
Collections.emptySet(), user);
+}
+
+@Override
+public ProcessorDiagnosticsEntity getProcessorDiagnostics(final String
id) {
+final ProcessorNode processor = processorDAO.getProcessor(id);
+final ProcessorStatus processorStatus =
controllerFacade.getProcessorStatus(id);
+
+// Generate Processor Diagnostics
+final NiFiUser user = NiFiUserUtils.getNiFiUser();
+final ProcessorDiagnosticsDTO dto =
controllerFacade.getProcessorDiagnostics(processor, processorStatus,
bulletinRepository, serviceId -> createControllerServiceEntity(serviceId,
user));
+
+// Filter anything out of diagnostics that the user is not
authorized to see.
+final List jvmDiagnosticsSnaphots = new
ArrayList<>();
+final JVMDiagnosticsDTO jvmDiagnostics = dto.getJvmDiagnostics();
+jvmDiagnosticsSnaphots.add(jvmDiagnostics.getAggregateSnapshot());
+
+// filter controller-related information
+final boolean canReadController =
authorizableLookup.getController().isAuthorized(authorizer, RequestAction.READ,
user);
+if (!canReadController) {
+for (final JVMDiagnosticsSnapshotDTO snapshot :
jvmDiagnosticsSnaphots) {
+snapshot.setMaxEventDrivenThreads(null);
+snapshot.setMaxTimerDrivenThreads(null);
+snapshot.setBundlesLoaded(null);
+}
+}
+
+// filter system diagnostics information
+final boolean canReadSystem =
authorizableLookup.getSystem().isAuthorized(authorizer, RequestAction.READ,
user);
+if (!canReadSystem) {
+for (final JVMDiagnosticsSnapshotDTO snapshot :
jvmDiagnosticsSnaphots) {
+snapshot.setContentRepositoryStorageUsage(null);
+snapshot.setCpuCores(null);
+snapshot.setCpuLoadAverage(null);
+snapshot.setFlowFileRepositoryStorageUsage(null);
+snapshot.setMaxHeap(null);
+snapshot.setMaxHeapBytes(null);
+snapshot.setProvenanceRepositoryStorageUsage(null);
+snapshot.setPhysicalMemory(null);
+snapshot.setPhysicalMemoryBytes(null);
+snapshot.setGarbageCollectionDiagnostics(null);
+}
+}
+
+// filter connections
+final Predicate connectionAuthorized =
connectionDiagnostics -> {
+final String connectionId =
connectionDiagnostics.getConnection().getId();
+return
authorizableLookup.getConnection(connectionId).getAuthorizable().isAuthorized(authorizer,
RequestAction.READ, user);
+};
+
+// Function that can be used to remove the Source or Destination
of a ConnectionDTO, if the user is not authorized.
+final Function
filterSourceDestination = connectionDiagnostics -> {
--- End diff --
Good call. That would mean that the second Function there is not really
needed. Will address.
> Add REST Endpoint for gathering Processor Diagnostics information
> -
>
> Key: NIFI-4849
> URL: https://issues.apache.org/jira/browse/NIFI-4849
> Project: Apache NiFi
> Issue Type: Sub-task
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.6.0
>
>
> We need to add a REST endpoint that will use the appropriate resources to
> gather the Processor Diagnostics information. Information to return should
> include things like:
> * Processor config
> * Processor status
> * Garbage Collection info
> * Repo Sizes
> * Connection info for connections whose source or destination is the
> processor
>
[GitHub] nifi issue #2468: NIFI-4849: Implemented REST Endpoint and associated backen...
Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/2468 I think at this point, after looking at the data model for JVMDiagnosticsSnapshotDTO we can probably just break this into 3 distinct fields: JVMSystemDiagnosticsSnapshotDTO, JVMControllerDiagnosticsDTO, JVMFlowDiagnosticsDTO. Then, it makes the filtering and the permissions (and likely the merging) a lot cleaner. Will head down that path and see if that cleans things up. Thanks for the review so far @mcgilman! ---
[jira] [Commented] (NIFI-4849) Add REST Endpoint for gathering Processor Diagnostics information
[ https://issues.apache.org/jira/browse/NIFI-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395695#comment-16395695 ] ASF GitHub Bot commented on NIFI-4849: -- Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/2468 I think at this point, after looking at the data model for JVMDiagnosticsSnapshotDTO we can probably just break this into 3 distinct fields: JVMSystemDiagnosticsSnapshotDTO, JVMControllerDiagnosticsDTO, JVMFlowDiagnosticsDTO. Then, it makes the filtering and the permissions (and likely the merging) a lot cleaner. Will head down that path and see if that cleans things up. Thanks for the review so far @mcgilman! > Add REST Endpoint for gathering Processor Diagnostics information > - > > Key: NIFI-4849 > URL: https://issues.apache.org/jira/browse/NIFI-4849 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.6.0 > > > We need to add a REST endpoint that will use the appropriate resources to > gather the Processor Diagnostics information. Information to return should > include things like: > * Processor config > * Processor status > * Garbage Collection info > * Repo Sizes > * Connection info for connections whose source or destination is the > processor > * Controller Services that the processor is referencing -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #2515: NIFI-4885: Granular component restrictions
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2515 ---
[GitHub] nifi pull request #2527: FetchHBaseRow - log level and displayName
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2527 ---
[jira] [Commented] (NIFI-4953) FetchHBaseRow filling logs with unnecessary error messages
[ https://issues.apache.org/jira/browse/NIFI-4953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395756#comment-16395756 ] ASF subversion and git services commented on NIFI-4953: --- Commit 373cf090a46e03bca49335b9df7d5de0bd94a086 in nifi's branch refs/heads/master from [~Berezitsky] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=373cf09 ] NIFI-4953 - FetchHBaseRow - update log level for "not found" to DEBUG instead of ERROR Signed-off-by: Pierre Villard This closes #2527. > FetchHBaseRow filling logs with unnecessary error messages > -- > > Key: NIFI-4953 > URL: https://issues.apache.org/jira/browse/NIFI-4953 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.5.0 >Reporter: Ed Berezitsky >Assignee: Ed Berezitsky >Priority: Major > > FetchHbaseRow prints error messages into logs when rowkey is not found. Such > messages generate a lot of logs while unnecessary, and affect log-based > monitoring. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4885) More granular restricted component categories
[ https://issues.apache.org/jira/browse/NIFI-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395757#comment-16395757 ] ASF GitHub Bot commented on NIFI-4885: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2515 > More granular restricted component categories > - > > Key: NIFI-4885 > URL: https://issues.apache.org/jira/browse/NIFI-4885 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Core UI >Reporter: Matt Gilman >Assignee: Matt Gilman >Priority: Major > Fix For: 1.6.0 > > > Update the Restricted annotation to support more granular categories. > Available categories will map to new access policies. Example categories and > their corresponding access policies may be > * read-filesystem (/restricted-components/read-filesystem) > * write-filesystem (/restricted-components/write-filesystem) > * code-execution (/restricted-components/code-execution) > * keytab-access (/restricted-components/keytab-access) > The hierarchical nature of the access policies will support backward > compatibility with existing installations where the policy of > /restricted-components was used to enforce all subcategories. Any users with > /restricted-components permissions will be granted access to all > subcategories. In order to leverage the new granular categories, an > administrator will need to use NiFi to update their access policies (remove a > user from /restricted-components and place them into the desired subcategory) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (NIFI-4953) FetchHBaseRow filling logs with unnecessary error messages
[ https://issues.apache.org/jira/browse/NIFI-4953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard resolved NIFI-4953. -- Resolution: Fixed Fix Version/s: 1.6.0 > FetchHBaseRow filling logs with unnecessary error messages > -- > > Key: NIFI-4953 > URL: https://issues.apache.org/jira/browse/NIFI-4953 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.5.0 >Reporter: Ed Berezitsky >Assignee: Ed Berezitsky >Priority: Major > Fix For: 1.6.0 > > > FetchHbaseRow prints error messages into logs when rowkey is not found. Such > messages generate a lot of logs while unnecessary, and affect log-based > monitoring. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2533: NIFI-4961 Allow data to be set on MockFlowFile
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/2533 Hi @kai5263499, I'm curious, do you have an example of unit test you want to implement that requires access to this method? ---
[jira] [Commented] (NIFI-4961) Allow data to be set on MockFlowFile
[ https://issues.apache.org/jira/browse/NIFI-4961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395769#comment-16395769 ] ASF GitHub Bot commented on NIFI-4961: -- Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/2533 Hi @kai5263499, I'm curious, do you have an example of unit test you want to implement that requires access to this method? > Allow data to be set on MockFlowFile > > > Key: NIFI-4961 > URL: https://issues.apache.org/jira/browse/NIFI-4961 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Glenn Widner >Priority: Minor > > While working on tests for a custom processor I noticed that the setData > method is private which makes it hard to test that my processor is handling > FlowFile content properly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2533: NIFI-4961 Allow data to be set on MockFlowFile
Github user kai5263499 commented on the issue: https://github.com/apache/nifi/pull/2533 Yes, I have a grpc processor that allows binary data to be passed back and forth and I want to be able to write a unit test to verify the processor returns valid protobufs. I didn't see a way to do that with the current MockFlowFile class where setData is protected ---
[jira] [Commented] (NIFI-4961) Allow data to be set on MockFlowFile
[ https://issues.apache.org/jira/browse/NIFI-4961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395773#comment-16395773 ] ASF GitHub Bot commented on NIFI-4961: -- Github user kai5263499 commented on the issue: https://github.com/apache/nifi/pull/2533 Yes, I have a grpc processor that allows binary data to be passed back and forth and I want to be able to write a unit test to verify the processor returns valid protobufs. I didn't see a way to do that with the current MockFlowFile class where setData is protected > Allow data to be set on MockFlowFile > > > Key: NIFI-4961 > URL: https://issues.apache.org/jira/browse/NIFI-4961 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Glenn Widner >Priority: Minor > > While working on tests for a custom processor I noticed that the setData > method is private which makes it hard to test that my processor is handling > FlowFile content properly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2470: NIFI-4864 Fixing additional resources property pointing to...
Github user bbende commented on the issue: https://github.com/apache/nifi/pull/2470 @zenfenan this seems to be working well, I had a few minor changes I posted here: https://github.com/bbende/nifi/commits/NIFI-4864 If you are good with that last commit I made then I will go ahead and merge this. To summarize my changes... - Changed to using StringUtils.eqausl(oldFingerprintg, newFingerprint) because its possible old fingerprint is null or empty and we would still want to replace it with the new one if we have a new one - Made the reload method synchronized - Removed a the has/get/set fingerprint methods from the interface to try and keep all the fingerprint logic inside of AbstractConfiguredComponent ---
[jira] [Commented] (NIFI-4864) Additional Resources property pointing at a directory won't find new JARs
[ https://issues.apache.org/jira/browse/NIFI-4864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395794#comment-16395794 ] ASF GitHub Bot commented on NIFI-4864: -- Github user bbende commented on the issue: https://github.com/apache/nifi/pull/2470 @zenfenan this seems to be working well, I had a few minor changes I posted here: https://github.com/bbende/nifi/commits/NIFI-4864 If you are good with that last commit I made then I will go ahead and merge this. To summarize my changes... - Changed to using StringUtils.eqausl(oldFingerprintg, newFingerprint) because its possible old fingerprint is null or empty and we would still want to replace it with the new one if we have a new one - Made the reload method synchronized - Removed a the has/get/set fingerprint methods from the interface to try and keep all the fingerprint logic inside of AbstractConfiguredComponent > Additional Resources property pointing at a directory won't find new JARs > - > > Key: NIFI-4864 > URL: https://issues.apache.org/jira/browse/NIFI-4864 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.2.0, 1.3.0, 1.4.0, 1.5.0 >Reporter: Bryan Bende >Assignee: Sivaprasanna Sethuraman >Priority: Minor > > If you have a processor/Controller Service/Reporting Task that has a property > with dynamicallyModifiesClasspath(true) and you set the value to a directory, > the resources in that directory will only be calculated when that property > changes. This means if you added JARs to the directory later, and stopped and > started your processor, those new JARs still won't be available. You would > have to change the property to a new directory, or back and forth to some > other directory, to force a recalculation. > The setProperties method in AbstractConfiguredComponent is where it looks at > incoming property changes and determines if any were for classpath related > properties and then calls reload accordingly. > We would need to consider the case where setProperties is never even being > called, someone just stops and starts the processor and would want to pick up > any new JARs added. > A possible solution might be to computer some kind of hash/fingerprint of the > URLs each time reload is called, and then when starting the processor we > could recompute the fingerprint and compare it to the previous one. If they > are different then we call reload before starting the component. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-4964) Add bulk lookup feature in LookupRecord
Pierre Villard created NIFI-4964: Summary: Add bulk lookup feature in LookupRecord Key: NIFI-4964 URL: https://issues.apache.org/jira/browse/NIFI-4964 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: Pierre Villard When having a flow file with a large number of records it would be much more efficient to parse the whole flow file once to list all the coordinates to look for, then call a new method (lookupAll?) in the lookup service to get all the results, and then parse the file one more time to update the records. It should be added in the CS description/annotations that this approach could hold in memory a large number of objects but could result in better performances for lookup services accessing external systems (Mongo, HBase, etc). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2468: NIFI-4849: Implemented REST Endpoint and associated backen...
Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/2468 @mcgilman I have pushed a new commit that addresses above conversation. ---
