Re: [PR] NIFI-12333 MiNiFi property override through env and system variables [nifi]
briansolo1985 commented on code in PR #7997:
URL: https://github.com/apache/nifi/pull/7997#discussion_r1386758472
##
minifi/minifi-commons/minifi-utils/src/main/java/org/apache/nifi/minifi/commons/status/util/PropertyUtil.java:
##
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.commons.status.util;
+
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Stream;
+
+public class PropertyUtil {
Review Comment:
Minor but if this is a util class, we could add abstract, so nobody could
instantiate this class - even with reflection
##
minifi/minifi-commons/minifi-utils/src/test/java/org/apache/nifi/minifi/commons/status/util/PropertyUtilTest.java:
##
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.minifi.commons.status.util;
+
+import static
org.apache.nifi.minifi.commons.status.util.PropertyUtil.resolvePropertyValue;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+import java.util.Map;
+import org.junit.jupiter.api.Test;
+
+class PropertyUtilTest {
+
+private static final String PROPERTY_KEY_WITH_DOTS = "property.foo.bar";
+private static final String PROPERTY_KEY_WITH_HYPHENS = "property-foo-bar";
+private static final String PROPERTY_KEY_WITH_DOTS_AND_HYPHENS =
"property.foo-bar";
+private static final String PROPERTY_KEY_WITH_UNDERSCORE =
"property_foo_bar";
+private static final String VALUE = "value";
+
+@Test
+void
testResolveParameterValueReturnsNullWhenPropertiesDoesNotContainTheKey() {
+assertNull(resolvePropertyValue(PROPERTY_KEY_WITH_DOTS, Map.of()));
+}
+
+@Test
Review Comment:
These tests would be a good for for a parameterized test. Or is this
intentional because the of the descriptive test case name?
##
minifi/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-runtime/src/main/java/org/apache/nifi/minifi/util/MiNiFiProperties.java:
##
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.util;
+
+import static
org.apache.nifi.minifi.commons.status.util.PropertyUtil.resolvePropertyValue;
+
+import java.io.BufferedInputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.InputStream;
+import java.util.Optional;
+import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import org.apache.nifi.util.NiFiProperties;
+
+/**
+ * Extends NiFi properties functionality with System and
Re: [PR] NIFI-12313 - Update PutDatabaseRecord.java add property use database table column datatype [nifi]
mattyb149 commented on PR #7981: URL: https://github.com/apache/nifi/pull/7981#issuecomment-1802883745 > Thanks for the contribution @avasquezkudaw. As this presents an important behavioral change in the Processor, it would be helpful to add a unit test method that verifies the expected behavior when the property is disabled. Also note that property names should use [Title Case](https://apastyle.apa.org/style-grammar-guidelines/capitalization/title-case) for naming. Agreed, when the capability for using database types was added it was decided purposely to not offer a property, to simplify the user experience and be flexible in which types were used, to be as smart as possible at inferring the user's intent when putting data of one type in the record into a column in the database with a different type, such as String -> Timestamp. This PR should have at least a handful of tests (or multiple different types in one or more tests) for covering possible users' intents. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11995 Support multiple output formats for message headers in ConsumeAMQP [nifi]
codecov-commenter commented on PR #7652: URL: https://github.com/apache/nifi/pull/7652#issuecomment-1802843410 ## [Codecov](https://app.codecov.io/gh/apache/nifi/pull/7652?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report > :exclamation: No coverage uploaded for pull request base (`main@eb6a5a2`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#section-missing-base-commit). > The diff coverage is `n/a`. ```diff @@ Coverage Diff @@ ## main#7652 +/- ## === Coverage? 50.60% Complexity ?35103 === Files ? 5251 Lines ? 275605 Branches?30435 === Hits? 139476 Misses ? 126049 Partials?10080 ``` :mega: Codecov offers a browser extension for seamless coverage viewing on GitHub. Try it in [Chrome](https://chrome.google.com/webstore/detail/codecov/gedikamndpbemklijjkncpnolildpbgo) or [Firefox](https://addons.mozilla.org/en-US/firefox/addon/codecov/) today! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (NIFI-12336) Update mockito.version to 5.7.0
[ https://issues.apache.org/jira/browse/NIFI-12336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-12336. - Fix Version/s: 2.0.0 Resolution: Fixed > Update mockito.version to 5.7.0 > --- > > Key: NIFI-12336 > URL: https://issues.apache.org/jira/browse/NIFI-12336 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike R >Assignee: Mike R >Priority: Minor > Fix For: 2.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Update mockito.version to 5.7.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12336) Update mockito.version to 5.7.0
[ https://issues.apache.org/jira/browse/NIFI-12336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12336: Priority: Minor (was: Major) > Update mockito.version to 5.7.0 > --- > > Key: NIFI-12336 > URL: https://issues.apache.org/jira/browse/NIFI-12336 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike R >Assignee: Mike R >Priority: Minor > Time Spent: 0.5h > Remaining Estimate: 0h > > Update mockito.version to 5.7.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12336) Update mockito.version to 5.7.0
[ https://issues.apache.org/jira/browse/NIFI-12336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784222#comment-17784222 ] ASF subversion and git services commented on NIFI-12336: Commit 2c89e3a21d57dfc1b0939136435997e3ce02756c in nifi's branch refs/heads/main from mr1716 [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=2c89e3a21d ] NIFI-12336 Upgraded Mockito from 5.5.0 to 5.7.0 This closes #8000 Signed-off-by: David Handermann > Update mockito.version to 5.7.0 > --- > > Key: NIFI-12336 > URL: https://issues.apache.org/jira/browse/NIFI-12336 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.23.2 >Reporter: Mike R >Assignee: Mike R >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Update mockito.version to 5.7.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12336) Update mockito.version to 5.7.0
[ https://issues.apache.org/jira/browse/NIFI-12336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12336: Affects Version/s: (was: 1.23.2) > Update mockito.version to 5.7.0 > --- > > Key: NIFI-12336 > URL: https://issues.apache.org/jira/browse/NIFI-12336 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike R >Assignee: Mike R >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Update mockito.version to 5.7.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12336 Update mockito.version to 5.7.0 [nifi]
exceptionfactory closed pull request #8000: NIFI-12336 Update mockito.version to 5.7.0 URL: https://github.com/apache/nifi/pull/8000 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-11129 - adding PutMongoBulk processor to use the bulkWrite API [nifi]
sebastianrothbucher commented on PR #6918:
URL: https://github.com/apache/nifi/pull/6918#issuecomment-1802690448
I don't think it was sth I did, specifically; it's just that Mongo might not
yet have committed the deletion of the DB in question. I nonetheless did two
things: 1.) use different test docs in different tests (which should not be
necessary but gives additoinal security). and 2.) add more deletion and an
explicit close to the Mongo write tests - which hopefully got this from 0.1% to
0.001% or sth
I re-ran the full test suite - and I have some other spurious errors. One in
GridFS I totally cannot explain (except somehow the CI overwrites the mongo
docker image - and we use another MongoDB on Github CI; just didn't see it).
The other issue is in GetMongoIT. We seem to overwrite flowfile attrs with
"environment variables". Looks to me the following change makes it a little
more stable. Didn't include it in the commit; maybe still useful
```
diff --git
a/nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/test/java/org/apache/nifi/processors/mongodb/GetMongoIT.java
b/nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/test/java/org/apache/nifi/processors/mongodb/GetMongoIT.java
index 87a0ca3d56..58e9adea6f 100644
---
a/nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/test/java/org/apache/nifi/processors/mongodb/GetMongoIT.java
+++
b/nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/test/java/org/apache/nifi/processors/mongodb/GetMongoIT.java
@@ -502,10 +502,9 @@ public class GetMongoIT extends AbstractMongoIT {
db.getCollection(collections[x])
.insertOne(new Document().append("msg", "Hello, World"));
-Map attrs = new HashMap<>();
-attrs.put("db", dbs[x]);
-attrs.put("collection", collections[x]);
-runner.enqueue(query, attrs);
+runner.setEnvironmentVariableValue("db", dbs[x]);
+runner.setEnvironmentVariableValue("collection",
collections[x]);
+runner.enqueue(query);
runner.run();
db.drop();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NIFI-12338) Misspelling of "the" in PutKudu documentation for Operation RecordPath property
Andrew M. Lim created NIFI-12338: Summary: Misspelling of "the" in PutKudu documentation for Operation RecordPath property Key: NIFI-12338 URL: https://issues.apache.org/jira/browse/NIFI-12338 Project: Apache NiFi Issue Type: Bug Components: Documentation Website Reporter: Andrew M. Lim "The" is misspelled "hte" in the line: When evaluated, the RecordPath must evaluate to one of *hte* valid Kudu Operation Types -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12336 Update mockito.version to 5.7.0 [nifi]
mr1716 commented on PR #8000: URL: https://github.com/apache/nifi/pull/8000#issuecomment-1802590124 Workflow succeeded. But integration test didnt? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-12337) GitHub Action Workflows could better report test results and summaries
Chris Sampson created NIFI-12337:
Summary: GitHub Action Workflows could better report test results
and summaries
Key: NIFI-12337
URL: https://issues.apache.org/jira/browse/NIFI-12337
Project: Apache NiFi
Issue Type: Improvement
Reporter: Chris Sampson
Fix For: 2.latest
Several of the GitHub Action Workflows produce {{JUnit}} XML result output
files through {{surefire}} and {{failsafe}} plugins. These are currently
uploaded as raw logs/XML files for people to download from GitHub, but it
should be possible to upload these results as {{checks}} within GitHub instead,
which would be easier for people to consume and understand what tests are
failing in their PRs.
One possible action to use would be
[test-reporter|https://github.com/marketplace/actions/test-reporter], although
an attempt to use this in NIFI-12177 as part of the new {{integration-tests}}
Workflow failed because it seems we're unable to {{write}} to the GitHub repo
(even with {code:yaml}permissions.checks: write{code} set within the Workflow
definition). This might be an intentional setting, and could require discussion
with the ASF Infrastructure team to understand whether this is/could be
permitted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12192) GitHub Action Workflows should report Code Coverage for CI Unit Tests
[ https://issues.apache.org/jira/browse/NIFI-12192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sampson updated NIFI-12192: - Fix Version/s: 2.0.0 (was: 2.latest) Resolution: Fixed Status: Resolved (was: Patch Available) > GitHub Action Workflows should report Code Coverage for CI Unit Tests > - > > Key: NIFI-12192 > URL: https://issues.apache.org/jira/browse/NIFI-12192 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Chris Sampson >Assignee: David Handermann >Priority: Minor > Fix For: 2.0.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Consider adding test coverage reporting, at least for one of the > ci-integration jobs, see tools such as: > * https://github.com/marketplace/actions/jacoco-report > * https://github.com/codecov/codecov-action (see > https://github.com/exceptionfactory/jagged for example implementation) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12192) GitHub Action Workflows should report Code Coverage for CI Unit Tests
[
https://issues.apache.org/jira/browse/NIFI-12192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-12192:
-
Summary: GitHub Action Workflows should report Code Coverage for CI Unit
Tests (was: GitHub Action Workflows could better report Tests Results and
Summaries)
> GitHub Action Workflows should report Code Coverage for CI Unit Tests
> -
>
> Key: NIFI-12192
> URL: https://issues.apache.org/jira/browse/NIFI-12192
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Assignee: David Handermann
>Priority: Minor
> Fix For: 2.latest
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Several of the GitHub Action Workflows produce {{JUnit}} XML result output
> files through {{surefire}} and {{failsafe}} plugins. These are currently
> uploaded as raw logs/XML files for people to download from GitHub, but it
> should be possible to upload these results as {{checks}} within GitHub
> instead, which would be easier for people to consume and understand what
> tests are failing in their PRs.
> One possible action to use would be
> [test-reporter|https://github.com/marketplace/actions/test-reporter],
> although an attempt to use this in NIFI-12177 as part of the new
> {{integration-tests}} Workflow failed because it seems we're unable to
> {{write}} to the GitHub repo (even with {code:yaml}permissions.checks:
> write{code} set within the Workflow definition). This might be an intentional
> setting, and could require discussion with the ASF Infrastructure team to
> understand whether this is/could be permitted.
> Also consider adding test coverage reporting, at least for one of the
> ci-integration jobs, see tools such as:
> * https://github.com/marketplace/actions/jacoco-report
> * https://github.com/codecov/codecov-action (see
> https://github.com/exceptionfactory/jagged for example implementation)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12192) GitHub Action Workflows should report Code Coverage for CI Unit Tests
[
https://issues.apache.org/jira/browse/NIFI-12192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-12192:
-
Description:
Consider adding test coverage reporting, at least for one of the ci-integration
jobs, see tools such as:
* https://github.com/marketplace/actions/jacoco-report
* https://github.com/codecov/codecov-action (see
https://github.com/exceptionfactory/jagged for example implementation)
was:
Several of the GitHub Action Workflows produce {{JUnit}} XML result output
files through {{surefire}} and {{failsafe}} plugins. These are currently
uploaded as raw logs/XML files for people to download from GitHub, but it
should be possible to upload these results as {{checks}} within GitHub instead,
which would be easier for people to consume and understand what tests are
failing in their PRs.
One possible action to use would be
[test-reporter|https://github.com/marketplace/actions/test-reporter], although
an attempt to use this in NIFI-12177 as part of the new {{integration-tests}}
Workflow failed because it seems we're unable to {{write}} to the GitHub repo
(even with {code:yaml}permissions.checks: write{code} set within the Workflow
definition). This might be an intentional setting, and could require discussion
with the ASF Infrastructure team to understand whether this is/could be
permitted.
Also consider adding test coverage reporting, at least for one of the
ci-integration jobs, see tools such as:
* https://github.com/marketplace/actions/jacoco-report
* https://github.com/codecov/codecov-action (see
https://github.com/exceptionfactory/jagged for example implementation)
> GitHub Action Workflows should report Code Coverage for CI Unit Tests
> -
>
> Key: NIFI-12192
> URL: https://issues.apache.org/jira/browse/NIFI-12192
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Assignee: David Handermann
>Priority: Minor
> Fix For: 2.latest
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Consider adding test coverage reporting, at least for one of the
> ci-integration jobs, see tools such as:
> * https://github.com/marketplace/actions/jacoco-report
> * https://github.com/codecov/codecov-action (see
> https://github.com/exceptionfactory/jagged for example implementation)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12192) GitHub Action Workflows could better report Tests Results and Summaries
[
https://issues.apache.org/jira/browse/NIFI-12192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784189#comment-17784189
]
ASF subversion and git services commented on NIFI-12192:
Commit 5554bab51638b5860a0fd28a7522eba89e7bf893 in nifi's branch
refs/heads/main from David Handermann
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=5554bab516 ]
NIFI-12192 Added Codecov Reporting with JaCoCo
- Added report-code-coverage profile to nifi-assembly for aggregation
- Added Codecov reporting to Ubuntu Linux Job in ci-workflow
- Added Codecov badge to README.md
This closes #7995
Signed-off-by: Chris Sampson
> GitHub Action Workflows could better report Tests Results and Summaries
> ---
>
> Key: NIFI-12192
> URL: https://issues.apache.org/jira/browse/NIFI-12192
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Assignee: David Handermann
>Priority: Minor
> Fix For: 2.latest
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Several of the GitHub Action Workflows produce {{JUnit}} XML result output
> files through {{surefire}} and {{failsafe}} plugins. These are currently
> uploaded as raw logs/XML files for people to download from GitHub, but it
> should be possible to upload these results as {{checks}} within GitHub
> instead, which would be easier for people to consume and understand what
> tests are failing in their PRs.
> One possible action to use would be
> [test-reporter|https://github.com/marketplace/actions/test-reporter],
> although an attempt to use this in NIFI-12177 as part of the new
> {{integration-tests}} Workflow failed because it seems we're unable to
> {{write}} to the GitHub repo (even with {code:yaml}permissions.checks:
> write{code} set within the Workflow definition). This might be an intentional
> setting, and could require discussion with the ASF Infrastructure team to
> understand whether this is/could be permitted.
> Also consider adding test coverage reporting, at least for one of the
> ci-integration jobs, see tools such as:
> * https://github.com/marketplace/actions/jacoco-report
> * https://github.com/codecov/codecov-action (see
> https://github.com/exceptionfactory/jagged for example implementation)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] NIFI-12192 Add Codecov Reporting with JaCoCo [nifi]
asfgit closed pull request #7995: NIFI-12192 Add Codecov Reporting with JaCoCo URL: https://github.com/apache/nifi/pull/7995 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (NIFI-11308) Create an Expression Language function to return NiFi version
[ https://issues.apache.org/jira/browse/NIFI-11308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anna Nys reassigned NIFI-11308: --- Assignee: Anna Nys > Create an Expression Language function to return NiFi version > - > > Key: NIFI-11308 > URL: https://issues.apache.org/jira/browse/NIFI-11308 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.19.1 >Reporter: Mark Bean >Assignee: Anna Nys >Priority: Major > > Create a new Expression Language function which returns the version of NiFi > being run. This allows for programmatic access to version information within > the flow. Suggest the function be subjectless as it applies to the instance > making the EL call and be called "nifiVersion()" > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12335) MiNiFi startup failure
[
https://issues.apache.org/jira/browse/NIFI-12335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784152#comment-17784152
]
ASF subversion and git services commented on NIFI-12335:
Commit 4430effe00136321e686e866f8d08f93e577a2e0 in nifi's branch
refs/heads/main from Ferenc Erdei
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=4430effe00 ]
NIFI-12335 Fix MiNiFi startup issue
Signed-off-by: Csaba Bejan
This closes #7998.
> MiNiFi startup failure
> --
>
> Key: NIFI-12335
> URL: https://issues.apache.org/jira/browse/NIFI-12335
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Ferenc Erdei
>Assignee: Ferenc Erdei
>Priority: Major
> Labels: minifi-java
> Fix For: 2.0.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> If the current flow is empty and I don't set nifi.minifi.security properties,
> then I see the following error in the minifi-app.log and the MiNiFi can not
> start:
> {code:java}
> java.lang.Exception: Unable to load flow due to:
> org.apache.nifi.controller.serialization.FlowSynchronizationException:
> java.lang.NullPointerException: Cannot invoke "java.util.List.iterator()"
> because the return value of
> "org.apache.nifi.controller.flow.VersionedDataflow.getReportingTasks()" is
> null{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] NIFI-12335 Fix MiNiFi startup issue [nifi]
bejancsaba closed pull request #7998: NIFI-12335 Fix MiNiFi startup issue URL: https://github.com/apache/nifi/pull/7998 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12121 added the ability to modify the output filter in … [nifi]
mosermw commented on code in PR #7922:
URL: https://github.com/apache/nifi/pull/7922#discussion_r1386907257
##
nifi-nar-bundles/nifi-jslt-bundle/nifi-jslt-processors/src/main/java/org/apache/nifi/processors/jslt/JSLTTransformJSON.java:
##
@@ -107,6 +110,18 @@ public class JSLTTransformJSON extends AbstractProcessor {
.defaultValue(EACH_OBJECT.getValue())
.build();
+public static final PropertyDescriptor RESULT_FILTER = new
PropertyDescriptor.Builder()
+.name("jslt-transform-result-filter")
+.displayName("Transform Result Filter")
+.description("A filter of output results using another JSLT. This
property allows you to change the built-in filter,"
++ " which removes objects with null values, empty objects
and empty arrays from the output."
++ " Use a filter such as \"true\" to disable all
filtering.")
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.identifiesExternalResource(ResourceCardinality.SINGLE,
ResourceType.TEXT, ResourceType.FILE)
Review Comment:
I admit I made the new property support external resources to make the
customValidate logic simpler, and didn't think about it much after that. Since
the JSLT result filter must return true/false for each JSON object, I can't
imagine it can get complicated enough to require needing external resources. I
modified the property descriptor to make it a standard property.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12335 Fix MiNiFi startup issue [nifi]
bejancsaba commented on PR #7998: URL: https://github.com/apache/nifi/pull/7998#issuecomment-1802233628 Thanks for the details and the changes will merge as soon as the workflows conclude. +1 from my side -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12233 - Parameter context provider for 1Password [nifi]
exceptionfactory commented on code in PR #7884:
URL: https://github.com/apache/nifi/pull/7884#discussion_r1386864555
##
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-parameter-providers/src/main/java/org/apache/nifi/parameter/OnePasswordParameterProvider.java:
##
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.parameter;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.ConfigVerificationResult;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.web.client.api.HttpResponseEntity;
+import org.apache.nifi.web.client.provider.api.WebClientServiceProvider;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+
+@Tags({"1Password"})
+@CapabilityDescription("Fetches parameters from 1Password Connect Server")
+public class OnePasswordParameterProvider extends AbstractParameterProvider
implements VerifiableParameterProvider {
+
+
+public static final PropertyDescriptor WEB_CLIENT_SERVICE_PROVIDER = new
PropertyDescriptor.Builder()
+.name("Web Client Service Provider")
+.displayName("Web Client Service Provider")
+.description("Controller service for HTTP client operations.")
+.identifiesControllerService(WebClientServiceProvider.class)
+.required(true)
+.build();
+public static final PropertyDescriptor CONNECT_SERVER = new
PropertyDescriptor.Builder()
+.name("Connect Server")
+.displayName("Connect Server")
+.description("HTTP endpoint of the 1Password Connect Server to
connect to. Example: http://localhost:8080;)
+.required(true)
+.addValidator(StandardValidators.URL_VALIDATOR)
+.build();
+public static final PropertyDescriptor ACCESS_TOKEN = new
PropertyDescriptor.Builder()
+.name("access-token")
+.displayName("Access Token")
+.description("Access Token using for authentication against the
1Password APIs.")
+.sensitive(true)
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+private List properties;
Review Comment:
This list can be removed in favor of the `DESCRIPTORS` field.
```suggestion
```
##
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-parameter-providers/src/main/java/org/apache/nifi/parameter/OnePasswordParameterProvider.java:
##
@@ -0,0 +1,220 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.parameter;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.ConfigVerificationResult;
+import org.apache.nifi.components.PropertyDescriptor;
+import
[PR] NIFI-12336 Update mockito.version to 5.7.0 [nifi]
mr1716 opened a new pull request, #8000: URL: https://github.com/apache/nifi/pull/8000 # Summary [NIFI-12336](https://issues.apache.org/jira/browse/NIFI-12336) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI-12336) issue created ### Pull Request Tracking - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X] Pull Request based on current revision of the `main` branch - [X] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (NIFI-12336) Update mockito.version to 5.7.0
[ https://issues.apache.org/jira/browse/NIFI-12336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike R reassigned NIFI-12336: - Assignee: Mike R > Update mockito.version to 5.7.0 > --- > > Key: NIFI-12336 > URL: https://issues.apache.org/jira/browse/NIFI-12336 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.23.2 >Reporter: Mike R >Assignee: Mike R >Priority: Major > > Update mockito.version to 5.7.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-11129 - adding PutMongoBulk processor to use the bulkWrite API [nifi]
sebastianrothbucher commented on PR #6918: URL: https://github.com/apache/nifi/pull/6918#issuecomment-1802207437 I still have to look. I ran tests before each push - and they just worked. Also from the error messages - it seems like sth gets stuck due to load. Let me get back2u -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-12336) Update mockito.version to 5.7.0
Mike R created NIFI-12336: - Summary: Update mockito.version to 5.7.0 Key: NIFI-12336 URL: https://issues.apache.org/jira/browse/NIFI-12336 Project: Apache NiFi Issue Type: Improvement Affects Versions: 1.23.2 Reporter: Mike R Update mockito.version to 5.7.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12334: Implement GCS option for FileResourceService [nifi]
pvillard31 commented on PR #7999: URL: https://github.com/apache/nifi/pull/7999#issuecomment-1802147126 I think the end goal here is to provide very high performance when it's about moving data from an object store to another without any kind of transformation. Right now support for FileResourceService is only in Azure processors but I'd expect the same to be added in AWS/GCS supports and add FileResourceService implementations for Azure, AWS in addition to local file system and GCS in this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12334: Implement GCS option for FileResourceService [nifi]
mark-bathori commented on PR #7999: URL: https://github.com/apache/nifi/pull/7999#issuecomment-1802124689 Thanks for the question @exceptionfactory. Currently if someone would like to move their data from one storage to another they would need to use a List processor into a Fetch processor and then to a Put processor. The fetched data will be stored in the content repository before it can be used by the Put processor. The `FileResourceService` main purpose is to skip the content repository and reference the data directly in the Put processor so instead of eg.: `ListGCS` -> `FetchGCS` -> `PutADLS` the user can do `ListGCS` -> `PutADLS`. This should also provide better performance in the amount of processed data due avoiding the need of writing data in the content repository. The `FileResourceService` already have an implementation for local file systems(`StandardFileResourceService`) and my PR would provide and option for Google Cloud Storage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12335 Fix MiNiFi startup issue [nifi]
ferencerdei commented on code in PR #7998:
URL: https://github.com/apache/nifi/pull/7998#discussion_r1386754850
##
minifi/minifi-commons/minifi-commons-framework/src/main/java/org/apache/nifi/minifi/commons/service/FlowEnrichService.java:
##
@@ -103,10 +107,12 @@ public byte[] enrichFlow(byte[] flowCandidate) {
rootGroup.setInstanceIdentifier(randomUUID().toString());
}
+rootGroup.getControllerServices().forEach(cs ->
cs.setScheduledState(ENABLED));
+
Optional commonSslControllerService =
createCommonSslControllerService();
commonSslControllerService
.ifPresent(sslControllerService -> {
-List currentControllerServices =
ofNullable(versionedDataflow.getControllerServices()).orElseGet(ArrayList::new);
+List currentControllerServices =
new ArrayList<>(versionedDataflow.getControllerServices());
Review Comment:
I wanted to avoid passing the reference for the original list, but maybe you
are right. As this object doesn't exist outside of this class we can safely use
the value directly. I'm going to update this and the other as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (NIFI-11463) Add option for IdentifyMimeType custom mime definitions to be in addition to the default tika definitions
[ https://issues.apache.org/jira/browse/NIFI-11463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784084#comment-17784084 ] David Handermann commented on NIFI-11463: - You're welcome [~Nissim Shiman], thanks again for your work on this, and adapting the PR through the process! > Add option for IdentifyMimeType custom mime definitions to be in addition to > the default tika definitions > - > > Key: NIFI-11463 > URL: https://issues.apache.org/jira/browse/NIFI-11463 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > Fix For: 2.0.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > IdentifyMimeType has options for users to configure custom mime types > definitions. > These user defined definitions override the default tika mime type > definitions. > Add option to allow custom mime type definitions to be in addition to the > default tika mime type definitions. > Users will be able to decide if custom definitions should be used instead of > or in addition to default tika definitions. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11463) Add option for IdentifyMimeType custom mime definitions to be in addition to the default tika definitions
[ https://issues.apache.org/jira/browse/NIFI-11463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784082#comment-17784082 ] Nissim Shiman commented on NIFI-11463: -- Thank you [~exceptionfactory] for direction in the PR for this and [~markap14] for the really nice work for migrateProperties() (NIFI-12139 , NIFI-12301). Both were incorporated in this, and made the code more graceful. It is very much appreciated. > Add option for IdentifyMimeType custom mime definitions to be in addition to > the default tika definitions > - > > Key: NIFI-11463 > URL: https://issues.apache.org/jira/browse/NIFI-11463 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Nissim Shiman >Assignee: Nissim Shiman >Priority: Major > Fix For: 2.0.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > IdentifyMimeType has options for users to configure custom mime types > definitions. > These user defined definitions override the default tika mime type > definitions. > Add option to allow custom mime type definitions to be in addition to the > default tika mime type definitions. > Users will be able to decide if custom definitions should be used instead of > or in addition to default tika definitions. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12335 Fix MiNiFi startup issue [nifi]
ferencerdei commented on code in PR #7998:
URL: https://github.com/apache/nifi/pull/7998#discussion_r1386728669
##
minifi/minifi-commons/minifi-commons-framework/src/main/java/org/apache/nifi/minifi/commons/service/FlowEnrichService.java:
##
@@ -103,10 +107,12 @@ public byte[] enrichFlow(byte[] flowCandidate) {
rootGroup.setInstanceIdentifier(randomUUID().toString());
}
+rootGroup.getControllerServices().forEach(cs ->
cs.setScheduledState(ENABLED));
Review Comment:
This also caused issues because the controller services were not enabled
automatically so I got validation errors. I forgot to mention this in the
ticket. Because it makes no sense to disable separate components in the context
of MiNiFi, I think it's safe to always enable everything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] MINIFICPP-1415 Pass references to onTrigger and onSchedule instead of… [nifi-minifi-cpp]
adamdebreceni commented on code in PR #1693:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1693#discussion_r1386727993
##
extensions/http-curl/processors/InvokeHTTP.cpp:
##
@@ -202,15 +186,15 @@ bool InvokeHTTP::appendHeaders(const core::FlowFile&
flow_file, /*std::invocable
return true;
}
-void InvokeHTTP::onTrigger(const std::shared_ptr&
context, const std::shared_ptr& session) {
- gsl_Expects(session && context && client_queue_);
+void InvokeHTTP::onTrigger(core::ProcessContext& context,
core::ProcessSession& session) {
+ gsl_Expects(client_queue_);
- auto flow_file = session->get();
+ auto flow_file = session.get();
if (flow_file == nullptr) {
if (!shouldEmitFlowFile()) {
- logger_->log_debug("InvokeHTTP -- create flow file with {}",
magic_enum::enum_name(method_));
- flow_file = session->create();
+ logger_->log_debug("InvokeHTTP -- create flow file with {}",
std::string(magic_enum::enum_name(method_)));
Review Comment:
why do we need to construct and `std::string`? `enum_name` seems to be used
without that in other logs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12335 Fix MiNiFi startup issue [nifi]
ferencerdei commented on code in PR #7998:
URL: https://github.com/apache/nifi/pull/7998#discussion_r1386724790
##
minifi/minifi-commons/minifi-commons-framework/src/main/java/org/apache/nifi/minifi/commons/service/FlowEnrichService.java:
##
@@ -90,6 +91,9 @@ public byte[] enrichFlow(byte[] flowCandidate) {
}
VersionedDataflow versionedDataflow =
parseVersionedDataflow(flowCandidate);
+
versionedDataflow.setReportingTasks(ofNullable(versionedDataflow.getReportingTasks()).orElseGet(ArrayList::new));
+
versionedDataflow.setRegistries(ofNullable(versionedDataflow.getRegistries()).orElseGet(ArrayList::new));
+
versionedDataflow.setControllerServices(ofNullable(versionedDataflow.getControllerServices()).orElseGet(ArrayList::new));
Review Comment:
For ParameterProviders and FlowAnalysisRules it is handled in the
VersionedFlowSynchronizer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12335 Fix MiNiFi startup issue [nifi]
briansolo1985 commented on code in PR #7998:
URL: https://github.com/apache/nifi/pull/7998#discussion_r1386715480
##
minifi/minifi-commons/minifi-commons-framework/src/main/java/org/apache/nifi/minifi/commons/service/FlowEnrichService.java:
##
@@ -103,10 +107,12 @@ public byte[] enrichFlow(byte[] flowCandidate) {
rootGroup.setInstanceIdentifier(randomUUID().toString());
}
+rootGroup.getControllerServices().forEach(cs ->
cs.setScheduledState(ENABLED));
Review Comment:
Very minor, but in my opinion lamdbas are more expresseive if meaningiful
parameter names are used, in this case I would use controllerService, but leave
it you.
##
minifi/minifi-commons/minifi-commons-framework/src/main/java/org/apache/nifi/minifi/commons/service/FlowEnrichService.java:
##
@@ -118,7 +124,7 @@ public byte[] enrichFlow(byte[] flowCandidate) {
createProvenanceReportingTask(commonSslControllerService.map(VersionedComponent::getInstanceIdentifier).orElse(EMPTY))
.ifPresent(provenanceReportingTask -> {
-List currentReportingTasks =
ofNullable(versionedDataflow.getReportingTasks()).orElseGet(ArrayList::new);
+List currentReportingTasks = new
ArrayList<>(versionedDataflow.getReportingTasks());
Review Comment:
And this one also could be simplified
```
createProvenanceReportingTask(commonSslControllerService.map(VersionedComponent::getInstanceIdentifier).orElse(EMPTY))
.ifPresent(versionedDataflow.getReportingTasks()::add);
```
##
minifi/minifi-commons/minifi-commons-framework/src/main/java/org/apache/nifi/minifi/commons/service/FlowEnrichService.java:
##
@@ -103,10 +107,12 @@ public byte[] enrichFlow(byte[] flowCandidate) {
rootGroup.setInstanceIdentifier(randomUUID().toString());
}
+rootGroup.getControllerServices().forEach(cs ->
cs.setScheduledState(ENABLED));
+
Optional commonSslControllerService =
createCommonSslControllerService();
commonSslControllerService
.ifPresent(sslControllerService -> {
-List currentControllerServices =
ofNullable(versionedDataflow.getControllerServices()).orElseGet(ArrayList::new);
+List currentControllerServices =
new ArrayList<>(versionedDataflow.getControllerServices());
Review Comment:
Since the initialization was moved outside now it is guaranteed that we will
get a non-null list object here.
Is it necessary to recreate the list object for some reasons?
If not we could simplify the ifPresent block by reducing it to
```
commonSslControllerService.ifPresent(versionedDataflow.getControllerServices()::add);
```
##
minifi/minifi-commons/minifi-commons-framework/src/main/java/org/apache/nifi/minifi/commons/service/FlowEnrichService.java:
##
@@ -26,6 +26,7 @@
import static java.util.UUID.randomUUID;
import static java.util.stream.Collectors.toMap;
import static org.apache.commons.lang3.StringUtils.EMPTY;
+import static org.apache.nifi.flow.ScheduledState.ENABLED;
Review Comment:
Could we use the static import in line 159 as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12335 Fix MiNiFi startup issue [nifi]
bejancsaba commented on code in PR #7998:
URL: https://github.com/apache/nifi/pull/7998#discussion_r1386713417
##
minifi/minifi-commons/minifi-commons-framework/src/main/java/org/apache/nifi/minifi/commons/service/FlowEnrichService.java:
##
@@ -90,6 +91,9 @@ public byte[] enrichFlow(byte[] flowCandidate) {
}
VersionedDataflow versionedDataflow =
parseVersionedDataflow(flowCandidate);
+
versionedDataflow.setReportingTasks(ofNullable(versionedDataflow.getReportingTasks()).orElseGet(ArrayList::new));
+
versionedDataflow.setRegistries(ofNullable(versionedDataflow.getRegistries()).orElseGet(ArrayList::new));
+
versionedDataflow.setControllerServices(ofNullable(versionedDataflow.getControllerServices()).orElseGet(ArrayList::new));
Review Comment:
Do we need to do this for the others (e.g.: parameters) as well or only
these are needed to be initialised on case of an empty flow
##
minifi/minifi-commons/minifi-commons-framework/src/main/java/org/apache/nifi/minifi/commons/service/FlowEnrichService.java:
##
@@ -103,10 +107,12 @@ public byte[] enrichFlow(byte[] flowCandidate) {
rootGroup.setInstanceIdentifier(randomUUID().toString());
}
+rootGroup.getControllerServices().forEach(cs ->
cs.setScheduledState(ENABLED));
Review Comment:
This seems unrelated to the startup issue, is this specific to NiFi 2.0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[PR] NIFI-12334: Implement GCS option for FileResourceService [nifi]
mark-bathori opened a new pull request, #7999: URL: https://github.com/apache/nifi/pull/7999 # Summary [NIFI-12334](https://issues.apache.org/jira/browse/NIFI-12334) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [x] Pull Request based on current revision of the `main` branch - [x] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [x] Build completed using `mvn clean install -P contrib-check` - [x] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] NIFI-12335 Fix MiNiFi startup issue [nifi]
ferencerdei opened a new pull request, #7998: URL: https://github.com/apache/nifi/pull/7998 # Summary [NIFI-12335](https://issues.apache.org/jira/browse/NIFI-12335) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [ ] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-12335) MiNiFi startup failure
Ferenc Erdei created NIFI-12335:
---
Summary: MiNiFi startup failure
Key: NIFI-12335
URL: https://issues.apache.org/jira/browse/NIFI-12335
Project: Apache NiFi
Issue Type: Bug
Reporter: Ferenc Erdei
Assignee: Ferenc Erdei
Fix For: 2.0.0
If the current flow is empty and I don't set nifi.minifi.security properties,
then I see the following error in the minifi-app.log and the MiNiFi can not
start:
{code:java}
java.lang.Exception: Unable to load flow due to:
org.apache.nifi.controller.serialization.FlowSynchronizationException:
java.lang.NullPointerException: Cannot invoke "java.util.List.iterator()"
because the return value of
"org.apache.nifi.controller.flow.VersionedDataflow.getReportingTasks()" is
null{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12334) Implement GCS option for FileResourceService
[ https://issues.apache.org/jira/browse/NIFI-12334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Bathori updated NIFI-12334: Description: Add GCS based implementation for the FileResourceService interface to be able to fetch data directly from Google Cloud Storage. (was: Add GCS based implementation for the FileResourceService to be able to fetch data directly from Google Cloud Storage.) > Implement GCS option for FileResourceService > > > Key: NIFI-12334 > URL: https://issues.apache.org/jira/browse/NIFI-12334 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Mark Bathori >Assignee: Mark Bathori >Priority: Major > > Add GCS based implementation for the FileResourceService interface to be able > to fetch data directly from Google Cloud Storage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12334) Implement GCS option for FileResourceService
Mark Bathori created NIFI-12334: --- Summary: Implement GCS option for FileResourceService Key: NIFI-12334 URL: https://issues.apache.org/jira/browse/NIFI-12334 Project: Apache NiFi Issue Type: New Feature Reporter: Mark Bathori Assignee: Mark Bathori Add GCS based implementation for the FileResourceService to be able to fetch data directly from Google Cloud Storage. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12261) Refactor Schema Access Strategy and Remove Hortonworks Properties
[ https://issues.apache.org/jira/browse/NIFI-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Bende updated NIFI-12261: --- Fix Version/s: 2.0.0 (was: 2.latest) Resolution: Fixed Status: Resolved (was: Patch Available) > Refactor Schema Access Strategy and Remove Hortonworks Properties > - > > Key: NIFI-12261 > URL: https://issues.apache.org/jira/browse/NIFI-12261 > Project: Apache NiFi > Issue Type: Task > Components: Extensions >Reporter: Pierre Villard >Assignee: David Handermann >Priority: Major > Fix For: 2.0.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > The Hortonworks Schema Registry has been removed for the release of NiFi 2.0. > However we have remaining pieces in nifi-standard-record-utils where are > defined the schema access strategies. > * HortonworksAttributeSchemaReferenceStrategy > * HortonworksAttributeSchemaReferenceWriter > * HortonworksEncodedSchemaReferenceStrategy > * HortonworksEncodedSchemaReferenceWriter > * HortonworksProtocolVersions > Those classes are referenced in SchemaAccessUtils (nifi-avro-record-utils). > We probably want to rename all of this and leverage the migrateProperties > feature. We could also just rename the "display name" and not really care > about it. > Having those strategies in an extension point could be an option but is > probably not worth the effort. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12261) Refactor Schema Access Strategy and Remove Hortonworks Properties
[ https://issues.apache.org/jira/browse/NIFI-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784058#comment-17784058 ] ASF subversion and git services commented on NIFI-12261: Commit 3dcfc919bbf83f07d018f6ffe9c78b7029971bd7 in nifi's branch refs/heads/main from David Handermann [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=3dcfc919bb ] NIFI-12261 Added Schema Reference Reader and Writer Services (#7991) - Refactored Confluent Encoded handling to Schema Reference Reader and Schema Reference Writer implementations - Removed Schema Protocol Version from Schema Registry Service associated with Hortonworks Schema Registry > Refactor Schema Access Strategy and Remove Hortonworks Properties > - > > Key: NIFI-12261 > URL: https://issues.apache.org/jira/browse/NIFI-12261 > Project: Apache NiFi > Issue Type: Task > Components: Extensions >Reporter: Pierre Villard >Assignee: David Handermann >Priority: Major > Fix For: 2.latest > > Time Spent: 1h 20m > Remaining Estimate: 0h > > The Hortonworks Schema Registry has been removed for the release of NiFi 2.0. > However we have remaining pieces in nifi-standard-record-utils where are > defined the schema access strategies. > * HortonworksAttributeSchemaReferenceStrategy > * HortonworksAttributeSchemaReferenceWriter > * HortonworksEncodedSchemaReferenceStrategy > * HortonworksEncodedSchemaReferenceWriter > * HortonworksProtocolVersions > Those classes are referenced in SchemaAccessUtils (nifi-avro-record-utils). > We probably want to rename all of this and leverage the migrateProperties > feature. We could also just rename the "display name" and not really care > about it. > Having those strategies in an extension point could be an option but is > probably not worth the effort. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12261 Add Schema Reference Reader and Writer Services [nifi]
bbende merged PR #7991: URL: https://github.com/apache/nifi/pull/7991 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12192 Add Codecov Reporting with JaCoCo [nifi]
exceptionfactory commented on PR #7995: URL: https://github.com/apache/nifi/pull/7995#issuecomment-1801898174 > As discussed elsewhere, would it be sensible to update the `integration-tests-ci` profile to skip all `ITRedisDistributedMapCacheClientService` ITs as part of this PR - more of them failed the first time in the PR checks, but then succeeded on the retry? Thanks for the review and feedback @ChrisSamo632! I have also noticed that methods in that class have been unreliable in automated tests, so I changed the exclusion to apply to the entire `ITRedisDistributedMapCacheClientService` class. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12192 Add Codecov Reporting with JaCoCo [nifi]
exceptionfactory commented on code in PR #7995:
URL: https://github.com/apache/nifi/pull/7995#discussion_r1386624819
##
.github/workflows/ci-workflow.yml:
##
@@ -160,22 +159,19 @@ jobs:
${{ env.MAVEN_COMPILE_COMMAND }}
- name: Maven Verify
env:
- NIFI_CI_LOCALE: >-
--Duser.language=en
--Duser.country=AU
- SUREFIRE_OPTS: >-
Review Comment:
I considered similar options as described in the
[prepare-agent](https://www.jacoco.org/jacoco/trunk/doc/prepare-agent-mojo.html)
goal documentation, but part of the difficulty is that the build workflow was
setting a System property, which JaCoCo also needs to set. We could change the
System property in the build workflow to a different name, and then use that
name for late property evaluation in the maven-surefire-plugin configuration,
but that would have involved changes all of the build profiles. I kept the
scope of changes narrow in this case, but we could revisit in the future if we
need to introduce other arguments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] NIFI-12192 Add Codecov Reporting with JaCoCo [nifi]
exceptionfactory commented on code in PR #7995: URL: https://github.com/apache/nifi/pull/7995#discussion_r1386620132 ## pom.xml: ## @@ -29,6 +29,7 @@ nifi-framework-api nifi-server-api nifi-bootstrap +nifi-code-coverage Review Comment: Thanks for the feedback. I considered the option of individually running the JaCoCo [report](https://www.jacoco.org/jacoco/trunk/doc/report-mojo.html) goal, but I went with the [report-aggregate](https://www.jacoco.org/jacoco/trunk/doc/report-aggregate-mojo.html) goal and aggregating module to avoid changing individual Maven configurations. I agree that neither solution is perfect, as each approach requires an extra element to consider when adding new modules. However, after making recent changes to remove the need for JavaDoc and source plugin skipping in individual modules, I went with the centralized approach. Another positive of this option is that if we don't see coverage information for a particular module, there is only one place to check, versus going through multiple modules. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-12333) Env and System variable overrides for MiNiFi and MiNiFi bootstrap properties
[ https://issues.apache.org/jira/browse/NIFI-12333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784008#comment-17784008 ] Ferenc Erdei commented on NIFI-12333: - PR: https://github.com/apache/nifi/pull/7997 > Env and System variable overrides for MiNiFi and MiNiFi bootstrap properties > > > Key: NIFI-12333 > URL: https://issues.apache.org/jira/browse/NIFI-12333 > Project: Apache NiFi > Issue Type: Improvement > Components: MiNiFi >Reporter: Ferenc Erdei >Assignee: Ferenc Erdei >Priority: Minor > Labels: minifi-java > Fix For: 2.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > > MiNiFi and MiNiFi bootstrap properties can be configured currently only by > editing the relevant config files and there is no way to override those > easily. In k8s environment, this requires separate config maps for each > configuration or an init container to replace properties in the configuration > files. > This story is about extending the property resolution to check firstly the > System properties, Env properties, and finally the property files. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] NIFI-12333 MiNiFi property override through env and system variables [nifi]
ferencerdei opened a new pull request, #7997: URL: https://github.com/apache/nifi/pull/7997 # Summary [NIFI-12333](https://issues.apache.org/jira/browse/NIFI-12333) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [ ] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-1415 Pass references to onTrigger and onSchedule instead of… [nifi-minifi-cpp]
adamdebreceni commented on code in PR #1693:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1693#discussion_r1386514328
##
extensions/http-curl/processors/InvokeHTTP.cpp:
##
@@ -93,6 +64,9 @@ void
setupClientTransferEncoding(extensions::curl::HTTPClient& client, bool use_
} // namespace
void InvokeHTTP::setupMembersFromProperties(const core::ProcessContext&
context) {
+ if (!context.getProperty(URL, url_))
+throw Exception(PROCESS_SCHEDULE_EXCEPTION, "URL property missing or
invalid");
Review Comment:
oh I see it was moved, in that case it can stay as is to be easier to review
the PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] MINIFICPP-1415 Pass references to onTrigger and onSchedule instead of… [nifi-minifi-cpp]
adamdebreceni commented on code in PR #1693:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1693#discussion_r1386506918
##
extensions/http-curl/processors/InvokeHTTP.cpp:
##
@@ -93,6 +64,9 @@ void
setupClientTransferEncoding(extensions::curl::HTTPClient& client, bool use_
} // namespace
void InvokeHTTP::setupMembersFromProperties(const core::ProcessContext&
context) {
+ if (!context.getProperty(URL, url_))
+throw Exception(PROCESS_SCHEDULE_EXCEPTION, "URL property missing or
invalid");
Review Comment:
can we use `getRequiredPropertyOrThrow` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (NIFI-11439) Add Storage API URL to GCS Processors
[
https://issues.apache.org/jira/browse/NIFI-11439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17784000#comment-17784000
]
mayki commented on NIFI-11439:
--
Hello,
For the parameter Storage-api-url, it seems neither
"https://{*}storage{*}-xxx.p.googleapis.com/{*}storage{*}/v1/" nor
"https://{*}storage{*}-xxx.p.googleapis.com; working.
{code:java}
PutGCSObject[id=aea7e07c-018b-1000-] Failure completing upload
flowfile=test/minh.txt bucket=xxx key=test/minh.txt reason=404 Not Found
POST
https://storage-xxx.p.googleapis.com/upload/storage/v1/storage/v1/b/xxx/o?name=test/minh.txt=resumable
Not Found: com.google.cloud.storage.StorageException: 404 Not Found
PutGCSObject[id=aea7e07c-018b-1000-] Failure completing upload
flowfile=test/minh.txt bucket=xxx key=test/minh.txt reason=www.googleapis.com:
com.google.cloud.storage.StorageException: www.googleapis.com
- Caused by: java.net.UnknownHostException: www.googleapis.com {code}
Need we add something more in the processor ?
Thanks
> Add Storage API URL to GCS Processors
> -
>
> Key: NIFI-11439
> URL: https://issues.apache.org/jira/browse/NIFI-11439
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Paul Grey
>Assignee: Paul Grey
>Priority: Minor
> Fix For: 2.0.0, 1.22.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> AWS processors allow the override of the base endpoint for API calls to AWS
> services [1]. GCS libraries provide analogous capabilities, as documented
> here [2]. Expose a GCS processor property to enable GCS endpoint override
> behavior.
> [1]
> https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java#L146-L154
> [2]
> https://cloud.google.com/storage/docs/request-endpoints#storage-set-client-endpoint-java
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] MINIFICPP-2217 - Implement jolt processor shift operation [nifi-minifi-cpp]
adamdebreceni commented on code in PR #1692:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1692#discussion_r1386425162
##
extensions/standard-processors/processors/JoltTransformJSON.h:
##
@@ -0,0 +1,104 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include
+
+#include "core/Processor.h"
+#include "core/ProcessSession.h"
+#include "core/PropertyDefinition.h"
+#include "core/PropertyDefinitionBuilder.h"
+#include "utils/Enum.h"
+#include "utils/Searcher.h"
+#include "../utils/JoltUtils.h"
+
+namespace org::apache::nifi::minifi::processors::jolt_transform_json {
+enum class JoltTransform {
+ SHIFT
+};
+} // namespace org::apache::nifi::minifi::processors::jolt_transform_json
+
+namespace magic_enum::customize {
+using JoltTransform =
org::apache::nifi::minifi::processors::jolt_transform_json::JoltTransform;
+
+template <>
+constexpr customize_t enum_name(JoltTransform value) noexcept {
+ switch (value) {
+case JoltTransform::SHIFT:
+ return "Shift";
+ }
+ return invalid_tag;
+}
+} // namespace magic_enum::customize
Review Comment:
good idea, done
##
extensions/standard-processors/utils/JoltUtils.cpp:
##
@@ -0,0 +1,1120 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "JoltUtils.h"
+#include "rapidjson/error/en.h"
+#include "Exception.h"
+
+namespace org::apache::nifi::minifi::utils::jolt {
+
+bool Spec::Template::check(std::string_view str) {
+ enum class State {
+Plain,
+Escaped
+ } state = State::Plain;
+ for (char ch : str) {
+switch (state) {
+ case State::Plain: {
+if (ch == '&') {
+ return true;
+} else if (ch == '\\') {
+ state = State::Escaped;
+}
+break;
+ }
+ case State::Escaped: {
+state = State::Plain;
+break;
+ }
+}
+ }
+ return false;
+}
+
+nonstd::expected, std::string>
Spec::Template::parse(It begin, It end, std::string_view escapables) {
+ enum class State {
+Plain,
+Escaped,
+Template, // &
+SimpleIndex, // &1
+CanonicalTemplate, // &(
+ParentIndex, // &(1
+NextIndex, // &(1,
+MatchIndex // &(1,0
+ };
+
+ std::vector fragments;
+ std::vector> references;
+ fragments.push_back({});
+ State state = State::Plain;
+ std::string target;
+ // go beyond the last char on purpose
+ auto ch_it = begin;
+ while (ch_it <= end) {
+std::optional ch;
+if (ch_it < end) {
+ ch = *ch_it;
+}
+bool force_terminate = false;
+switch (state) {
+ case State::Plain: {
+if (ch == '\\') {
+ state = State::Escaped;
+} else if (ch == '&') {
+ references.push_back({});
+ fragments.push_back({});
+ state = State::Template;
+} else if (ch == ')' || ch == ']' || ch == '.' || ch == '[') {
+ force_terminate = true;
+} else if (ch) {
+ fragments.back() += ch.value();
+}
+break;
+ }
+ case State::Escaped: {
+if (!ch) {
+ return nonstd::make_unexpected("Unterminated escape sequence");
+}
+if (!(ch == '\\' || ch == '&' || std::find(escapables.begin(),
escapables.end(), ch.value()) != escapables.end())) {
+ return nonstd::make_unexpected(fmt::format("Unknown escape sequence
in template '\\{}'", ch.value()));
+}
+fragments.back() += ch.value();
+state =
Re: [PR] NIFI-12192 Add Codecov Reporting with JaCoCo [nifi]
ChrisSamo632 commented on PR #7995: URL: https://github.com/apache/nifi/pull/7995#issuecomment-1801443699 @exceptionfactory thanks for the update, the code coverage will help improve confidence in the code base going forward. I've suggeted a couple of options to think about for improving the use of JaCoCo - more for consideration than blockers to merging this PR. As discussed elsewhere, would it be sensible to update the `integration-tests-ci` profile to skip all `ITRedisDistributedMapCacheClientService` ITs as part of this PR - more of them failed the first time in the PR checks, but then succeeded on the retry? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12192 Add Codecov Reporting with JaCoCo [nifi]
ChrisSamo632 commented on code in PR #7995: URL: https://github.com/apache/nifi/pull/7995#discussion_r1386290226 ## pom.xml: ## @@ -29,6 +29,7 @@ nifi-framework-api nifi-server-api nifi-bootstrap +nifi-code-coverage Review Comment: I think the other possibie option, to avoid the new `nifi-code-coverage` sub-module, which may become out of step when new things are added to the repo, would be to add: ```xml true ``` to all sub-modules where we don't want to collect coverage, e.g. NAR and Bundle sub-modules. So this would remove the need for a new sub-module to maintain, but would mean updating lots of other existing sub-modules instead, then people understanding when to include the `skip` property for new sub-modules in future. I don't think either option is ideal, but thought I'd raise the alternative for consideration - I'd be happy with either considered option -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12192 Add Codecov Reporting with JaCoCo [nifi]
ChrisSamo632 commented on code in PR #7995:
URL: https://github.com/apache/nifi/pull/7995#discussion_r1386277971
##
.github/workflows/ci-workflow.yml:
##
@@ -160,22 +159,19 @@ jobs:
${{ env.MAVEN_COMPILE_COMMAND }}
- name: Maven Verify
env:
- NIFI_CI_LOCALE: >-
--Duser.language=en
--Duser.country=AU
- SUREFIRE_OPTS: >-
Review Comment:
It might be possible to retain these args (thus allowing use of JaCoCo in
other Actions in future if desired, without needing to change the `mvn` args,
by using something like:
```xml
org.apache.maven.plugins
maven-surefire-plugin
@{jaCoCoArgLine} -Xmx1024m
```
within ``, and (within ``):
```xml
org.jacoco
jacoco-maven-plugin
default-prepare-agent
prepare-agent
jaCoCoArgLine
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NIFI-12333) Env and System variable overrides for MiNiFi and MiNiFi bootstrap properties
Ferenc Erdei created NIFI-12333: --- Summary: Env and System variable overrides for MiNiFi and MiNiFi bootstrap properties Key: NIFI-12333 URL: https://issues.apache.org/jira/browse/NIFI-12333 Project: Apache NiFi Issue Type: Improvement Components: MiNiFi Reporter: Ferenc Erdei Assignee: Ferenc Erdei Fix For: 2.0.0 MiNiFi and MiNiFi bootstrap properties can be configured currently only by editing the relevant config files and there is no way to override those easily. In k8s environment, this requires separate config maps for each configuration or an init container to replace properties in the configuration files. This story is about extending the property resolution to check firstly the System properties, Env properties, and finally the property files. -- This message was sent by Atlassian Jira (v8.20.10#820010)
