[GitHub] [nifi] exceptionfactory commented on pull request #7446: NIFI-11536: implemented AbstractStoreScanner and KeyStoreScanner that…
exceptionfactory commented on PR #7446: URL: https://github.com/apache/nifi/pull/7446#issuecomment-1612209337 > I made the changes while still preserving `KeyStoreScanner` and `TrustStoreScanner` as the derived classes for two reasons. > > 1. So the logger is class-specific It is not necessary to define a class-specific logger, although maintaining two separate class implementations might be useful based on the second concern. > > 2. So the `server` would have beans with distinct types. Having all have the same type would cause some existing tests to fail since it uses `server.getBean`. That is a good point regarding `addBean` and `getBean`. With that in mind, it does seem reasonable to maintain two separate classes. > > > Some thoughts: I was wondering if we were to have just one class whether it's a good idea to have both a keystore scanner and truststore scanner in the same class. One advantage this would have is if both keystore and truststore changed, then the `SSLContext` would be reloaded only once, as opposed to having two scanners that reloaded it twice. Although having one class might be worth considering, in general the keystore and truststore should change independently, so keeping two separate classes sounds good. Thanks for considering the options. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on pull request #7446: NIFI-11536: implemented AbstractStoreScanner and KeyStoreScanner that…
emiliosetiadarma commented on PR #7446: URL: https://github.com/apache/nifi/pull/7446#issuecomment-1612201484 I made the changes while still preserving `KeyStoreScanner` and `TrustStoreScanner` as the derived classes for two reasons. 1. So the logger is class-specific 2. So the `server` would have beans with distinct types. Having all have the same type would cause some existing tests to fail since it uses `server.getBean`. Some thoughts: I was wondering if we were to have just one class whether it's a good idea to have both a keystore scanner and truststore scanner in the same class. One advantage this would have is if both keystore and truststore changed, then the `SSLContext` would be reloaded only once, as opposed to having two scanners that reloaded it twice. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #7446: NIFI-11536: implemented AbstractStoreScanner and KeyStoreScanner that…
emiliosetiadarma commented on code in PR #7446:
URL: https://github.com/apache/nifi/pull/7446#discussion_r1245806680
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-jetty/src/main/java/org/apache/nifi/web/server/util/AbstractStoreScanner.java:
##
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.web.server.util;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.security.util.TlsConfiguration;
+import org.apache.nifi.security.util.TlsException;
+import org.eclipse.jetty.util.Scanner;
+import org.eclipse.jetty.util.annotation.ManagedAttribute;
+import org.eclipse.jetty.util.annotation.ManagedOperation;
+import org.eclipse.jetty.util.component.ContainerLifeCycle;
+import org.eclipse.jetty.util.log.Logger;
+import org.eclipse.jetty.util.resource.Resource;
+import org.eclipse.jetty.util.ssl.SslContextFactory;
+
+import javax.net.ssl.SSLContext;
+import java.io.File;
+import java.io.IOException;
+import java.util.Collections;
+
+import static org.apache.nifi.security.util.SslContextFactory.createSslContext;
+
+public abstract class AbstractStoreScanner extends ContainerLifeCycle
implements Scanner.DiscreteListener {
+private final SslContextFactory sslContextFactory;
+private final TlsConfiguration tlsConfiguration;
+private final File file;
+private final Scanner _scanner;
+
+protected AbstractStoreScanner(final SslContextFactory sslContextFactory,
+ final TlsConfiguration tlsConfiguration) {
+this.sslContextFactory = sslContextFactory;
+this.tlsConfiguration = tlsConfiguration;
+try {
+final Resource resource = getResource();
+File monitoredFile = resource.getFile();
+if (monitoredFile == null || !monitoredFile.exists()) {
+throw new IllegalArgumentException(String.format("%s file does
not exist", getFileType()));
+}
+if (monitoredFile.isDirectory()) {
+throw new IllegalArgumentException(String.format("expected %s
file not directory", getFileType()));
+}
+
+if (resource.getAlias() != null) {
+// this resource has an alias, use the alias, as that's what's
returned in the Scanner
+monitoredFile = new File(resource.getAlias());
+}
+
+file = monitoredFile;
+if (getLogger().isDebugEnabled()) {
+getLogger().debug("Monitored {} file: {}", getFileType(),
monitoredFile);
+}
+} catch (final IOException e) {
+throw new IllegalArgumentException(String.format("could not obtain
%s file", getFileType()), e);
+}
+
+File parentFile = file.getParentFile();
+if (!parentFile.exists() || !parentFile.isDirectory()) {
+throw new IllegalArgumentException(String.format("error obtaining
%s dir", getFileType()));
+}
+
+_scanner = new Scanner();
+_scanner.setScanDirs(Collections.singletonList(parentFile));
+_scanner.setScanInterval(1);
+_scanner.setReportDirs(false);
+_scanner.setReportExistingFilesOnStartup(false);
+_scanner.setScanDepth(1);
+_scanner.addListener(this);
+addBean(_scanner);
+}
+
+@Override
+public void fileAdded(final String filename) {
+if (getLogger().isDebugEnabled()) {
+getLogger().debug("added {}", filename);
+}
+
+if (this.file.toPath().toString().equals(filename)) {
+this.reload();
+}
+
+}
+
+@Override
+public void fileChanged(final String filename) {
+if (getLogger().isDebugEnabled()) {
+getLogger().debug("changed {}", filename);
+}
+
+if (this.file.toPath().toString().equals(filename)) {
+this.reload();
+}
+
+}
+
+@Override
+public void fileRemoved(final String filename) {
+if (getLogger().isDebugEnabled()) {
+getLogger().debug("removed {}", filename);
+}
+
+if (this.file.toPath().toString().equals(filename)) {
+this.reload();
+}
+
+
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #7446: NIFI-11536: implemented AbstractStoreScanner and KeyStoreScanner that…
emiliosetiadarma commented on code in PR #7446:
URL: https://github.com/apache/nifi/pull/7446#discussion_r1245806477
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-jetty/src/main/java/org/apache/nifi/web/server/util/AbstractStoreScanner.java:
##
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.web.server.util;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.security.util.TlsConfiguration;
+import org.apache.nifi.security.util.TlsException;
+import org.eclipse.jetty.util.Scanner;
+import org.eclipse.jetty.util.annotation.ManagedAttribute;
+import org.eclipse.jetty.util.annotation.ManagedOperation;
+import org.eclipse.jetty.util.component.ContainerLifeCycle;
+import org.eclipse.jetty.util.log.Logger;
+import org.eclipse.jetty.util.resource.Resource;
+import org.eclipse.jetty.util.ssl.SslContextFactory;
+
+import javax.net.ssl.SSLContext;
+import java.io.File;
+import java.io.IOException;
+import java.util.Collections;
+
+import static org.apache.nifi.security.util.SslContextFactory.createSslContext;
+
+public abstract class AbstractStoreScanner extends ContainerLifeCycle
implements Scanner.DiscreteListener {
+private final SslContextFactory sslContextFactory;
+private final TlsConfiguration tlsConfiguration;
+private final File file;
+private final Scanner _scanner;
+
+protected AbstractStoreScanner(final SslContextFactory sslContextFactory,
+ final TlsConfiguration tlsConfiguration) {
+this.sslContextFactory = sslContextFactory;
+this.tlsConfiguration = tlsConfiguration;
+try {
+final Resource resource = getResource();
+File monitoredFile = resource.getFile();
+if (monitoredFile == null || !monitoredFile.exists()) {
+throw new IllegalArgumentException(String.format("%s file does
not exist", getFileType()));
+}
+if (monitoredFile.isDirectory()) {
+throw new IllegalArgumentException(String.format("expected %s
file not directory", getFileType()));
+}
+
+if (resource.getAlias() != null) {
+// this resource has an alias, use the alias, as that's what's
returned in the Scanner
+monitoredFile = new File(resource.getAlias());
+}
+
+file = monitoredFile;
+if (getLogger().isDebugEnabled()) {
+getLogger().debug("Monitored {} file: {}", getFileType(),
monitoredFile);
+}
+} catch (final IOException e) {
+throw new IllegalArgumentException(String.format("could not obtain
%s file", getFileType()), e);
+}
+
+File parentFile = file.getParentFile();
+if (!parentFile.exists() || !parentFile.isDirectory()) {
+throw new IllegalArgumentException(String.format("error obtaining
%s dir", getFileType()));
+}
+
+_scanner = new Scanner();
+_scanner.setScanDirs(Collections.singletonList(parentFile));
+_scanner.setScanInterval(1);
+_scanner.setReportDirs(false);
+_scanner.setReportExistingFilesOnStartup(false);
+_scanner.setScanDepth(1);
+_scanner.addListener(this);
+addBean(_scanner);
+}
+
+@Override
+public void fileAdded(final String filename) {
+if (getLogger().isDebugEnabled()) {
+getLogger().debug("added {}", filename);
+}
+
+if (this.file.toPath().toString().equals(filename)) {
+this.reload();
+}
+
+}
+
+@Override
+public void fileChanged(final String filename) {
+if (getLogger().isDebugEnabled()) {
+getLogger().debug("changed {}", filename);
+}
+
+if (this.file.toPath().toString().equals(filename)) {
+this.reload();
+}
+
+}
+
+@Override
+public void fileRemoved(final String filename) {
+if (getLogger().isDebugEnabled()) {
+getLogger().debug("removed {}", filename);
+}
+
+if (this.file.toPath().toString().equals(filename)) {
+this.reload();
+}
+
+
[GitHub] [nifi] emiliosetiadarma commented on a diff in pull request #7446: NIFI-11536: implemented AbstractStoreScanner and KeyStoreScanner that…
emiliosetiadarma commented on code in PR #7446:
URL: https://github.com/apache/nifi/pull/7446#discussion_r1245806247
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-jetty/src/main/java/org/apache/nifi/web/server/util/AbstractStoreScanner.java:
##
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.web.server.util;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.security.util.TlsConfiguration;
+import org.apache.nifi.security.util.TlsException;
+import org.eclipse.jetty.util.Scanner;
+import org.eclipse.jetty.util.annotation.ManagedAttribute;
+import org.eclipse.jetty.util.annotation.ManagedOperation;
+import org.eclipse.jetty.util.component.ContainerLifeCycle;
+import org.eclipse.jetty.util.log.Logger;
+import org.eclipse.jetty.util.resource.Resource;
+import org.eclipse.jetty.util.ssl.SslContextFactory;
+
+import javax.net.ssl.SSLContext;
+import java.io.File;
+import java.io.IOException;
+import java.util.Collections;
+
+import static org.apache.nifi.security.util.SslContextFactory.createSslContext;
+
+public abstract class AbstractStoreScanner extends ContainerLifeCycle
implements Scanner.DiscreteListener {
+private final SslContextFactory sslContextFactory;
+private final TlsConfiguration tlsConfiguration;
+private final File file;
+private final Scanner _scanner;
+
+protected AbstractStoreScanner(final SslContextFactory sslContextFactory,
+ final TlsConfiguration tlsConfiguration) {
+this.sslContextFactory = sslContextFactory;
+this.tlsConfiguration = tlsConfiguration;
+try {
+final Resource resource = getResource();
+File monitoredFile = resource.getFile();
+if (monitoredFile == null || !monitoredFile.exists()) {
+throw new IllegalArgumentException(String.format("%s file does
not exist", getFileType()));
+}
+if (monitoredFile.isDirectory()) {
+throw new IllegalArgumentException(String.format("expected %s
file not directory", getFileType()));
+}
+
+if (resource.getAlias() != null) {
+// this resource has an alias, use the alias, as that's what's
returned in the Scanner
+monitoredFile = new File(resource.getAlias());
+}
+
+file = monitoredFile;
+if (getLogger().isDebugEnabled()) {
+getLogger().debug("Monitored {} file: {}", getFileType(),
monitoredFile);
+}
+} catch (final IOException e) {
+throw new IllegalArgumentException(String.format("could not obtain
%s file", getFileType()), e);
+}
+
+File parentFile = file.getParentFile();
+if (!parentFile.exists() || !parentFile.isDirectory()) {
+throw new IllegalArgumentException(String.format("error obtaining
%s dir", getFileType()));
+}
+
+_scanner = new Scanner();
+_scanner.setScanDirs(Collections.singletonList(parentFile));
+_scanner.setScanInterval(1);
+_scanner.setReportDirs(false);
+_scanner.setReportExistingFilesOnStartup(false);
+_scanner.setScanDepth(1);
+_scanner.addListener(this);
+addBean(_scanner);
+}
+
+@Override
+public void fileAdded(final String filename) {
+if (getLogger().isDebugEnabled()) {
+getLogger().debug("added {}", filename);
+}
+
+if (this.file.toPath().toString().equals(filename)) {
+this.reload();
+}
+
+}
+
+@Override
+public void fileChanged(final String filename) {
+if (getLogger().isDebugEnabled()) {
+getLogger().debug("changed {}", filename);
+}
+
+if (this.file.toPath().toString().equals(filename)) {
+this.reload();
+}
+
+}
+
+@Override
+public void fileRemoved(final String filename) {
+if (getLogger().isDebugEnabled()) {
+getLogger().debug("removed {}", filename);
+}
+
+if (this.file.toPath().toString().equals(filename)) {
+this.reload();
+}
+
+
[GitHub] [nifi] emiliosetiadarma commented on pull request #7446: NIFI-11536: implemented AbstractStoreScanner and KeyStoreScanner that…
emiliosetiadarma commented on PR #7446: URL: https://github.com/apache/nifi/pull/7446#issuecomment-1612132072 Thanks @exceptionfactory for the quick review! Will address the comments -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #7446: NIFI-11536: implemented AbstractStoreScanner and KeyStoreScanner that…
exceptionfactory commented on code in PR #7446:
URL: https://github.com/apache/nifi/pull/7446#discussion_r1245778923
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-jetty/src/main/java/org/apache/nifi/web/server/util/AbstractStoreScanner.java:
##
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.web.server.util;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.nifi.security.util.TlsConfiguration;
+import org.apache.nifi.security.util.TlsException;
+import org.eclipse.jetty.util.Scanner;
+import org.eclipse.jetty.util.annotation.ManagedAttribute;
+import org.eclipse.jetty.util.annotation.ManagedOperation;
+import org.eclipse.jetty.util.component.ContainerLifeCycle;
+import org.eclipse.jetty.util.log.Logger;
+import org.eclipse.jetty.util.resource.Resource;
+import org.eclipse.jetty.util.ssl.SslContextFactory;
+
+import javax.net.ssl.SSLContext;
+import java.io.File;
+import java.io.IOException;
+import java.util.Collections;
+
+import static org.apache.nifi.security.util.SslContextFactory.createSslContext;
+
+public abstract class AbstractStoreScanner extends ContainerLifeCycle
implements Scanner.DiscreteListener {
+private final SslContextFactory sslContextFactory;
+private final TlsConfiguration tlsConfiguration;
+private final File file;
+private final Scanner _scanner;
+
+protected AbstractStoreScanner(final SslContextFactory sslContextFactory,
+ final TlsConfiguration tlsConfiguration) {
+this.sslContextFactory = sslContextFactory;
+this.tlsConfiguration = tlsConfiguration;
+try {
+final Resource resource = getResource();
+File monitoredFile = resource.getFile();
+if (monitoredFile == null || !monitoredFile.exists()) {
+throw new IllegalArgumentException(String.format("%s file does
not exist", getFileType()));
+}
+if (monitoredFile.isDirectory()) {
+throw new IllegalArgumentException(String.format("expected %s
file not directory", getFileType()));
+}
+
+if (resource.getAlias() != null) {
+// this resource has an alias, use the alias, as that's what's
returned in the Scanner
+monitoredFile = new File(resource.getAlias());
+}
+
+file = monitoredFile;
+if (getLogger().isDebugEnabled()) {
+getLogger().debug("Monitored {} file: {}", getFileType(),
monitoredFile);
+}
+} catch (final IOException e) {
+throw new IllegalArgumentException(String.format("could not obtain
%s file", getFileType()), e);
+}
+
+File parentFile = file.getParentFile();
+if (!parentFile.exists() || !parentFile.isDirectory()) {
+throw new IllegalArgumentException(String.format("error obtaining
%s dir", getFileType()));
+}
+
+_scanner = new Scanner();
+_scanner.setScanDirs(Collections.singletonList(parentFile));
+_scanner.setScanInterval(1);
+_scanner.setReportDirs(false);
+_scanner.setReportExistingFilesOnStartup(false);
+_scanner.setScanDepth(1);
+_scanner.addListener(this);
+addBean(_scanner);
+}
+
+@Override
+public void fileAdded(final String filename) {
+if (getLogger().isDebugEnabled()) {
+getLogger().debug("added {}", filename);
+}
+
+if (this.file.toPath().toString().equals(filename)) {
+this.reload();
+}
+
+}
+
+@Override
+public void fileChanged(final String filename) {
+if (getLogger().isDebugEnabled()) {
+getLogger().debug("changed {}", filename);
+}
+
+if (this.file.toPath().toString().equals(filename)) {
+this.reload();
+}
+
+}
+
+@Override
+public void fileRemoved(final String filename) {
+if (getLogger().isDebugEnabled()) {
+getLogger().debug("removed {}", filename);
+}
+
+if (this.file.toPath().toString().equals(filename)) {
+this.reload();
+}
+
+
[jira] [Updated] (NIFI-11480) PutElasticsearchRecord should have an option to output _bulk api response errors as flowfile attributes
[
https://issues.apache.org/jira/browse/NIFI-11480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-11480:
-
Fix Version/s: 1.latest
2.latest
> PutElasticsearchRecord should have an option to output _bulk api response
> errors as flowfile attributes
> ---
>
> Key: NIFI-11480
> URL: https://issues.apache.org/jira/browse/NIFI-11480
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Minor
> Fix For: 1.latest, 2.latest
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> https://github.com/apache/nifi/pull/6903 for NIFI-1 introduced a
> [conversation|https://github.com/apache/nifi/pull/6903#issuecomment-1513872398]
> about outputting the response error details for Records that are not
> processed by Elasticsearch.
> The same PR introduces a new {{elasticsearch.bulk.error}} attribute for the
> {{PutElasticsearchJson}} processor, but explains why [it's not so simple for
> PutElasticsearchRecord|https://github.com/apache/nifi/pull/6903#issuecomment-1514554132]
> due to input FlowFiles potentially containing many Records and there being
> no obvious way of expressing error details for all such Records in the single
> output flowfile.
> One [suggested
> approach|https://github.com/apache/nifi/pull/6903#issuecomment-1517903668]
> would be to "partition" the output {{errors}} Records into multiple
> flowfiles, grouped by the error {{type}} provided by Elasticsearch. This
> {{type}} could then be added to the flowfile(s) as the
> {{elasticsearch.bulk.error}} attribute. Flows could then {{RouteOnAttribute}}
> if they wanted to handle certain Elasticsearch errors in particular ways.
> Leaving all errors flowfiles in the same output queue avoids the problem of
> the [large (and changing) number of potential Elasticsearch error
> types|https://github.com/apache/nifi/pull/6903#issuecomment-1517863606]
> Such output partitioning (if implemented) should be optional, driven by a
> processor property that maintains the current "all in one" flowfile output by
> default.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-11480) PutElasticsearchRecord should have an option to output _bulk api response errors as flowfile attributes
[
https://issues.apache.org/jira/browse/NIFI-11480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-11480:
-
Status: Patch Available (was: In Progress)
> PutElasticsearchRecord should have an option to output _bulk api response
> errors as flowfile attributes
> ---
>
> Key: NIFI-11480
> URL: https://issues.apache.org/jira/browse/NIFI-11480
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> https://github.com/apache/nifi/pull/6903 for NIFI-1 introduced a
> [conversation|https://github.com/apache/nifi/pull/6903#issuecomment-1513872398]
> about outputting the response error details for Records that are not
> processed by Elasticsearch.
> The same PR introduces a new {{elasticsearch.bulk.error}} attribute for the
> {{PutElasticsearchJson}} processor, but explains why [it's not so simple for
> PutElasticsearchRecord|https://github.com/apache/nifi/pull/6903#issuecomment-1514554132]
> due to input FlowFiles potentially containing many Records and there being
> no obvious way of expressing error details for all such Records in the single
> output flowfile.
> One [suggested
> approach|https://github.com/apache/nifi/pull/6903#issuecomment-1517903668]
> would be to "partition" the output {{errors}} Records into multiple
> flowfiles, grouped by the error {{type}} provided by Elasticsearch. This
> {{type}} could then be added to the flowfile(s) as the
> {{elasticsearch.bulk.error}} attribute. Flows could then {{RouteOnAttribute}}
> if they wanted to handle certain Elasticsearch errors in particular ways.
> Leaving all errors flowfiles in the same output queue avoids the problem of
> the [large (and changing) number of potential Elasticsearch error
> types|https://github.com/apache/nifi/pull/6903#issuecomment-1517863606]
> Such output partitioning (if implemented) should be optional, driven by a
> processor property that maintains the current "all in one" flowfile output by
> default.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-11714) Add General Error Handling for Jetty Framework Server
[ https://issues.apache.org/jira/browse/NIFI-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-11714: Affects Version/s: 1.latest 2.latest Status: Patch Available (was: Open) > Add General Error Handling for Jetty Framework Server > - > > Key: NIFI-11714 > URL: https://issues.apache.org/jira/browse/NIFI-11714 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.latest, 2.latest >Reporter: David Handermann >Assignee: David Handermann >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > The Jetty server that supports framework operations should be updated to > include generalized error handling that avoids writing stack traces. > Application REST resources support error handling and simplified messages, > but Jetty handles some exceptions that can result from malformed HTTP > requests. Implementing fallback error handling for Jetty will avoid providing > stack trace information and other information to HTTP clients. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] exceptionfactory opened a new pull request, #7447: NIFI-11714 Add Error Handler to Jetty Server
exceptionfactory opened a new pull request, #7447: URL: https://github.com/apache/nifi/pull/7447 # Summary [NIFI-11714](https://issues.apache.org/jira/browse/NIFI-11714) Adds a configured Error Handler to the embedded Jetty Server for NiFi, NiFi Registry, and MiNiFi C2 Servers. The configured Error Handler disables writing stack traces to the response stream. This follows the existing implementation within REST resource classes, but applies the approach to errors that Jetty handles before interacting with application classes, such as when parsing HTTP query parameters. # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X] Pull Request based on current revision of the `main` branch - [X] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [X] Build completed using `mvn clean install -P contrib-check` - [X] JDK 17 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-11178) Improve memory efficiency of ListHDFS
[
https://issues.apache.org/jira/browse/NIFI-11178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17738247#comment-17738247
]
Lehel Boér commented on NIFI-11178:
---
Heap dump at the end of a run with the old HDFS while processing a 4-ary tree
with 1000 0KB sized files/nodes (~350 000 files).
The HdfsNamesFileStatus objects retain ~241 MBs.
!image-2023-06-28-20-48-57-012.png!
*Heap dump comparison between old and new version*
!image-2023-06-28-20-53-21-210.png!
*Old ListHDFS memory usage:*
!image-2023-06-28-20-54-00-887.png!
*New ListHDFS memory usage*
!image-2023-06-28-20-54-32-188.png!
> Improve memory efficiency of ListHDFS
> -
>
> Key: NIFI-11178
> URL: https://issues.apache.org/jira/browse/NIFI-11178
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Mark Payne
>Assignee: Lehel Boér
>Priority: Major
> Attachments: image-2023-06-28-20-48-40-182.png,
> image-2023-06-28-20-48-57-012.png, image-2023-06-28-20-53-21-210.png,
> image-2023-06-28-20-54-00-887.png, image-2023-06-28-20-54-32-188.png
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> ListHDFS is used extremely commonly. Typically, a listing consists of several
> hundred files or less. However, there are times (especially when performing
> the first listing) when the Processor is configured to recurse into
> subdirectories and creates a listing containing millions of files.
> Currently, performing a listing containing millions of files can occupy
> several GB of heap space. Analyzing a recent heap dump, it was found that a
> listing of 6.7 million files in HDFS occupied approximately 12 GB of heap
> space in NiFi. The heap usage can be tracked back to the fact that we hold in
> a local variable a HashSet and these FileStatus objects occupy
> approximately 1-2 KB of heap each.
> There are several improvements that can be made here, some small changes will
> have significant memory improvements. There are also large changes that could
> dramatically reduce heap utilization, to nearly nothing. But these would
> require complex rewrites of the Processor that would be much more difficult
> to maintain.
> As a simple analysis, I built a small test to see how many FileStatus objects
> could be kept in memory:
> {code:java}
> final Set statuses = new HashSet<>();
> for (int i=0; i < 10_000_000; i++) {
> if (i % 10_000 == 0) {
> System.out.println(i);
> }
> final FsPermission fsPermission = new FsPermission(777);
> final FileStatus status = new FileStatus(2, true, 1, 4_000_000,
> System.currentTimeMillis(), 0L, fsPermission,
> "owner-" + i, "group-" + i, null, new Path("/path/to/my/file-" + i +
> ".txt"), true, false, false);
> statuses.add(status);
> }
> {code}
> This gives us a way to see how many FileStatus objects can be added to our
> set before we encounter OOME. With a 512 MB heap I reached approximately 1.13
> million FileStatus objects. Note that the Paths here are very small, which
> occupies less memory than would normally be the case.
> Making one small change, to change the {{Set}} to an
> {{ArrayList}} yielded 1.21 million instead of 1.13 million
> objects. This is reasonable, since we don't expect any duplicates anyway.
> Another small change that was made was to introduce a new class that keeps
> only the fields we care about from the FileStatus:
> {code:java}
> private static class HdfsEntity {
> private final String path;
> private final long length;
> private final boolean isdir;
> private final long timestamp;
> private final boolean canRead;
> private final boolean canWrite;
> private final boolean canExecute;
> private final boolean encrypted;
> public HdfsEntity(final FileStatus status) {
> this.path = status.getPath().getName();
> this.length = status.getLen();
> this.isdir = status.isDirectory();
> this.timestamp = status.getModificationTime();
> this.canRead =
> status.getPermission().getGroupAction().implies(FsAction.READ);
> this.canWrite =
> status.getPermission().getGroupAction().implies(FsAction.WRITE);
> this.canExecute =
> status.getPermission().getGroupAction().implies(FsAction.EXECUTE);
> this.encrypted = status.isEncrypted();
> }
> }
> {code}
> This introduced significant savings, allowing a {{List}} to store
> 5.2 million objects instead of 1.13 million.
> It is worth noting here that the HdfsEntity doesn't store the 'group' and
> 'owner' that are part of the FileStatus. Interestingly, though, these values
> are strings that are extremely repetitive, but are unique strings on the
> heap. So a full solution here would mean tracking the owner and group, but
> doing so in a way
[jira] [Updated] (NIFI-11178) Improve memory efficiency of ListHDFS
[
https://issues.apache.org/jira/browse/NIFI-11178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lehel Boér updated NIFI-11178:
--
Attachment: image-2023-06-28-20-54-32-188.png
> Improve memory efficiency of ListHDFS
> -
>
> Key: NIFI-11178
> URL: https://issues.apache.org/jira/browse/NIFI-11178
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Mark Payne
>Assignee: Lehel Boér
>Priority: Major
> Attachments: image-2023-06-28-20-48-40-182.png,
> image-2023-06-28-20-48-57-012.png, image-2023-06-28-20-53-21-210.png,
> image-2023-06-28-20-54-00-887.png, image-2023-06-28-20-54-32-188.png
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> ListHDFS is used extremely commonly. Typically, a listing consists of several
> hundred files or less. However, there are times (especially when performing
> the first listing) when the Processor is configured to recurse into
> subdirectories and creates a listing containing millions of files.
> Currently, performing a listing containing millions of files can occupy
> several GB of heap space. Analyzing a recent heap dump, it was found that a
> listing of 6.7 million files in HDFS occupied approximately 12 GB of heap
> space in NiFi. The heap usage can be tracked back to the fact that we hold in
> a local variable a HashSet and these FileStatus objects occupy
> approximately 1-2 KB of heap each.
> There are several improvements that can be made here, some small changes will
> have significant memory improvements. There are also large changes that could
> dramatically reduce heap utilization, to nearly nothing. But these would
> require complex rewrites of the Processor that would be much more difficult
> to maintain.
> As a simple analysis, I built a small test to see how many FileStatus objects
> could be kept in memory:
> {code:java}
> final Set statuses = new HashSet<>();
> for (int i=0; i < 10_000_000; i++) {
> if (i % 10_000 == 0) {
> System.out.println(i);
> }
> final FsPermission fsPermission = new FsPermission(777);
> final FileStatus status = new FileStatus(2, true, 1, 4_000_000,
> System.currentTimeMillis(), 0L, fsPermission,
> "owner-" + i, "group-" + i, null, new Path("/path/to/my/file-" + i +
> ".txt"), true, false, false);
> statuses.add(status);
> }
> {code}
> This gives us a way to see how many FileStatus objects can be added to our
> set before we encounter OOME. With a 512 MB heap I reached approximately 1.13
> million FileStatus objects. Note that the Paths here are very small, which
> occupies less memory than would normally be the case.
> Making one small change, to change the {{Set}} to an
> {{ArrayList}} yielded 1.21 million instead of 1.13 million
> objects. This is reasonable, since we don't expect any duplicates anyway.
> Another small change that was made was to introduce a new class that keeps
> only the fields we care about from the FileStatus:
> {code:java}
> private static class HdfsEntity {
> private final String path;
> private final long length;
> private final boolean isdir;
> private final long timestamp;
> private final boolean canRead;
> private final boolean canWrite;
> private final boolean canExecute;
> private final boolean encrypted;
> public HdfsEntity(final FileStatus status) {
> this.path = status.getPath().getName();
> this.length = status.getLen();
> this.isdir = status.isDirectory();
> this.timestamp = status.getModificationTime();
> this.canRead =
> status.getPermission().getGroupAction().implies(FsAction.READ);
> this.canWrite =
> status.getPermission().getGroupAction().implies(FsAction.WRITE);
> this.canExecute =
> status.getPermission().getGroupAction().implies(FsAction.EXECUTE);
> this.encrypted = status.isEncrypted();
> }
> }
> {code}
> This introduced significant savings, allowing a {{List}} to store
> 5.2 million objects instead of 1.13 million.
> It is worth noting here that the HdfsEntity doesn't store the 'group' and
> 'owner' that are part of the FileStatus. Interestingly, though, these values
> are strings that are extremely repetitive, but are unique strings on the
> heap. So a full solution here would mean tracking the owner and group, but
> doing so in a way that we use a Map or something similar in
> order to reuse identical Strings on the heap (in much the same way that
> String.intern() does, but without interning the String as it's only reusable
> within a small context).
> These changes will yield significant memory improvements.
> However, more penetrating changes can make even large improvements:
> * Rather than keeping a Set or a List of entities at all, we could instead
> just
[jira] [Updated] (NIFI-11178) Improve memory efficiency of ListHDFS
[
https://issues.apache.org/jira/browse/NIFI-11178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lehel Boér updated NIFI-11178:
--
Attachment: image-2023-06-28-20-54-00-887.png
> Improve memory efficiency of ListHDFS
> -
>
> Key: NIFI-11178
> URL: https://issues.apache.org/jira/browse/NIFI-11178
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Mark Payne
>Assignee: Lehel Boér
>Priority: Major
> Attachments: image-2023-06-28-20-48-40-182.png,
> image-2023-06-28-20-48-57-012.png, image-2023-06-28-20-53-21-210.png,
> image-2023-06-28-20-54-00-887.png, image-2023-06-28-20-54-32-188.png
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> ListHDFS is used extremely commonly. Typically, a listing consists of several
> hundred files or less. However, there are times (especially when performing
> the first listing) when the Processor is configured to recurse into
> subdirectories and creates a listing containing millions of files.
> Currently, performing a listing containing millions of files can occupy
> several GB of heap space. Analyzing a recent heap dump, it was found that a
> listing of 6.7 million files in HDFS occupied approximately 12 GB of heap
> space in NiFi. The heap usage can be tracked back to the fact that we hold in
> a local variable a HashSet and these FileStatus objects occupy
> approximately 1-2 KB of heap each.
> There are several improvements that can be made here, some small changes will
> have significant memory improvements. There are also large changes that could
> dramatically reduce heap utilization, to nearly nothing. But these would
> require complex rewrites of the Processor that would be much more difficult
> to maintain.
> As a simple analysis, I built a small test to see how many FileStatus objects
> could be kept in memory:
> {code:java}
> final Set statuses = new HashSet<>();
> for (int i=0; i < 10_000_000; i++) {
> if (i % 10_000 == 0) {
> System.out.println(i);
> }
> final FsPermission fsPermission = new FsPermission(777);
> final FileStatus status = new FileStatus(2, true, 1, 4_000_000,
> System.currentTimeMillis(), 0L, fsPermission,
> "owner-" + i, "group-" + i, null, new Path("/path/to/my/file-" + i +
> ".txt"), true, false, false);
> statuses.add(status);
> }
> {code}
> This gives us a way to see how many FileStatus objects can be added to our
> set before we encounter OOME. With a 512 MB heap I reached approximately 1.13
> million FileStatus objects. Note that the Paths here are very small, which
> occupies less memory than would normally be the case.
> Making one small change, to change the {{Set}} to an
> {{ArrayList}} yielded 1.21 million instead of 1.13 million
> objects. This is reasonable, since we don't expect any duplicates anyway.
> Another small change that was made was to introduce a new class that keeps
> only the fields we care about from the FileStatus:
> {code:java}
> private static class HdfsEntity {
> private final String path;
> private final long length;
> private final boolean isdir;
> private final long timestamp;
> private final boolean canRead;
> private final boolean canWrite;
> private final boolean canExecute;
> private final boolean encrypted;
> public HdfsEntity(final FileStatus status) {
> this.path = status.getPath().getName();
> this.length = status.getLen();
> this.isdir = status.isDirectory();
> this.timestamp = status.getModificationTime();
> this.canRead =
> status.getPermission().getGroupAction().implies(FsAction.READ);
> this.canWrite =
> status.getPermission().getGroupAction().implies(FsAction.WRITE);
> this.canExecute =
> status.getPermission().getGroupAction().implies(FsAction.EXECUTE);
> this.encrypted = status.isEncrypted();
> }
> }
> {code}
> This introduced significant savings, allowing a {{List}} to store
> 5.2 million objects instead of 1.13 million.
> It is worth noting here that the HdfsEntity doesn't store the 'group' and
> 'owner' that are part of the FileStatus. Interestingly, though, these values
> are strings that are extremely repetitive, but are unique strings on the
> heap. So a full solution here would mean tracking the owner and group, but
> doing so in a way that we use a Map or something similar in
> order to reuse identical Strings on the heap (in much the same way that
> String.intern() does, but without interning the String as it's only reusable
> within a small context).
> These changes will yield significant memory improvements.
> However, more penetrating changes can make even large improvements:
> * Rather than keeping a Set or a List of entities at all, we could instead
> just
[jira] [Updated] (NIFI-11178) Improve memory efficiency of ListHDFS
[
https://issues.apache.org/jira/browse/NIFI-11178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lehel Boér updated NIFI-11178:
--
Attachment: image-2023-06-28-20-53-21-210.png
> Improve memory efficiency of ListHDFS
> -
>
> Key: NIFI-11178
> URL: https://issues.apache.org/jira/browse/NIFI-11178
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Mark Payne
>Assignee: Lehel Boér
>Priority: Major
> Attachments: image-2023-06-28-20-48-40-182.png,
> image-2023-06-28-20-48-57-012.png, image-2023-06-28-20-53-21-210.png
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> ListHDFS is used extremely commonly. Typically, a listing consists of several
> hundred files or less. However, there are times (especially when performing
> the first listing) when the Processor is configured to recurse into
> subdirectories and creates a listing containing millions of files.
> Currently, performing a listing containing millions of files can occupy
> several GB of heap space. Analyzing a recent heap dump, it was found that a
> listing of 6.7 million files in HDFS occupied approximately 12 GB of heap
> space in NiFi. The heap usage can be tracked back to the fact that we hold in
> a local variable a HashSet and these FileStatus objects occupy
> approximately 1-2 KB of heap each.
> There are several improvements that can be made here, some small changes will
> have significant memory improvements. There are also large changes that could
> dramatically reduce heap utilization, to nearly nothing. But these would
> require complex rewrites of the Processor that would be much more difficult
> to maintain.
> As a simple analysis, I built a small test to see how many FileStatus objects
> could be kept in memory:
> {code:java}
> final Set statuses = new HashSet<>();
> for (int i=0; i < 10_000_000; i++) {
> if (i % 10_000 == 0) {
> System.out.println(i);
> }
> final FsPermission fsPermission = new FsPermission(777);
> final FileStatus status = new FileStatus(2, true, 1, 4_000_000,
> System.currentTimeMillis(), 0L, fsPermission,
> "owner-" + i, "group-" + i, null, new Path("/path/to/my/file-" + i +
> ".txt"), true, false, false);
> statuses.add(status);
> }
> {code}
> This gives us a way to see how many FileStatus objects can be added to our
> set before we encounter OOME. With a 512 MB heap I reached approximately 1.13
> million FileStatus objects. Note that the Paths here are very small, which
> occupies less memory than would normally be the case.
> Making one small change, to change the {{Set}} to an
> {{ArrayList}} yielded 1.21 million instead of 1.13 million
> objects. This is reasonable, since we don't expect any duplicates anyway.
> Another small change that was made was to introduce a new class that keeps
> only the fields we care about from the FileStatus:
> {code:java}
> private static class HdfsEntity {
> private final String path;
> private final long length;
> private final boolean isdir;
> private final long timestamp;
> private final boolean canRead;
> private final boolean canWrite;
> private final boolean canExecute;
> private final boolean encrypted;
> public HdfsEntity(final FileStatus status) {
> this.path = status.getPath().getName();
> this.length = status.getLen();
> this.isdir = status.isDirectory();
> this.timestamp = status.getModificationTime();
> this.canRead =
> status.getPermission().getGroupAction().implies(FsAction.READ);
> this.canWrite =
> status.getPermission().getGroupAction().implies(FsAction.WRITE);
> this.canExecute =
> status.getPermission().getGroupAction().implies(FsAction.EXECUTE);
> this.encrypted = status.isEncrypted();
> }
> }
> {code}
> This introduced significant savings, allowing a {{List}} to store
> 5.2 million objects instead of 1.13 million.
> It is worth noting here that the HdfsEntity doesn't store the 'group' and
> 'owner' that are part of the FileStatus. Interestingly, though, these values
> are strings that are extremely repetitive, but are unique strings on the
> heap. So a full solution here would mean tracking the owner and group, but
> doing so in a way that we use a Map or something similar in
> order to reuse identical Strings on the heap (in much the same way that
> String.intern() does, but without interning the String as it's only reusable
> within a small context).
> These changes will yield significant memory improvements.
> However, more penetrating changes can make even large improvements:
> * Rather than keeping a Set or a List of entities at all, we could instead
> just iterate over each of the FileStatus objects returned from HDFS and
>
[jira] [Updated] (NIFI-11178) Improve memory efficiency of ListHDFS
[
https://issues.apache.org/jira/browse/NIFI-11178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lehel Boér updated NIFI-11178:
--
Attachment: image-2023-06-28-20-48-57-012.png
> Improve memory efficiency of ListHDFS
> -
>
> Key: NIFI-11178
> URL: https://issues.apache.org/jira/browse/NIFI-11178
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Mark Payne
>Assignee: Lehel Boér
>Priority: Major
> Attachments: image-2023-06-28-20-48-40-182.png,
> image-2023-06-28-20-48-57-012.png
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> ListHDFS is used extremely commonly. Typically, a listing consists of several
> hundred files or less. However, there are times (especially when performing
> the first listing) when the Processor is configured to recurse into
> subdirectories and creates a listing containing millions of files.
> Currently, performing a listing containing millions of files can occupy
> several GB of heap space. Analyzing a recent heap dump, it was found that a
> listing of 6.7 million files in HDFS occupied approximately 12 GB of heap
> space in NiFi. The heap usage can be tracked back to the fact that we hold in
> a local variable a HashSet and these FileStatus objects occupy
> approximately 1-2 KB of heap each.
> There are several improvements that can be made here, some small changes will
> have significant memory improvements. There are also large changes that could
> dramatically reduce heap utilization, to nearly nothing. But these would
> require complex rewrites of the Processor that would be much more difficult
> to maintain.
> As a simple analysis, I built a small test to see how many FileStatus objects
> could be kept in memory:
> {code:java}
> final Set statuses = new HashSet<>();
> for (int i=0; i < 10_000_000; i++) {
> if (i % 10_000 == 0) {
> System.out.println(i);
> }
> final FsPermission fsPermission = new FsPermission(777);
> final FileStatus status = new FileStatus(2, true, 1, 4_000_000,
> System.currentTimeMillis(), 0L, fsPermission,
> "owner-" + i, "group-" + i, null, new Path("/path/to/my/file-" + i +
> ".txt"), true, false, false);
> statuses.add(status);
> }
> {code}
> This gives us a way to see how many FileStatus objects can be added to our
> set before we encounter OOME. With a 512 MB heap I reached approximately 1.13
> million FileStatus objects. Note that the Paths here are very small, which
> occupies less memory than would normally be the case.
> Making one small change, to change the {{Set}} to an
> {{ArrayList}} yielded 1.21 million instead of 1.13 million
> objects. This is reasonable, since we don't expect any duplicates anyway.
> Another small change that was made was to introduce a new class that keeps
> only the fields we care about from the FileStatus:
> {code:java}
> private static class HdfsEntity {
> private final String path;
> private final long length;
> private final boolean isdir;
> private final long timestamp;
> private final boolean canRead;
> private final boolean canWrite;
> private final boolean canExecute;
> private final boolean encrypted;
> public HdfsEntity(final FileStatus status) {
> this.path = status.getPath().getName();
> this.length = status.getLen();
> this.isdir = status.isDirectory();
> this.timestamp = status.getModificationTime();
> this.canRead =
> status.getPermission().getGroupAction().implies(FsAction.READ);
> this.canWrite =
> status.getPermission().getGroupAction().implies(FsAction.WRITE);
> this.canExecute =
> status.getPermission().getGroupAction().implies(FsAction.EXECUTE);
> this.encrypted = status.isEncrypted();
> }
> }
> {code}
> This introduced significant savings, allowing a {{List}} to store
> 5.2 million objects instead of 1.13 million.
> It is worth noting here that the HdfsEntity doesn't store the 'group' and
> 'owner' that are part of the FileStatus. Interestingly, though, these values
> are strings that are extremely repetitive, but are unique strings on the
> heap. So a full solution here would mean tracking the owner and group, but
> doing so in a way that we use a Map or something similar in
> order to reuse identical Strings on the heap (in much the same way that
> String.intern() does, but without interning the String as it's only reusable
> within a small context).
> These changes will yield significant memory improvements.
> However, more penetrating changes can make even large improvements:
> * Rather than keeping a Set or a List of entities at all, we could instead
> just iterate over each of the FileStatus objects returned from HDFS and
> determine whether or not the file should be
[jira] [Updated] (NIFI-11178) Improve memory efficiency of ListHDFS
[
https://issues.apache.org/jira/browse/NIFI-11178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lehel Boér updated NIFI-11178:
--
Attachment: image-2023-06-28-20-48-40-182.png
> Improve memory efficiency of ListHDFS
> -
>
> Key: NIFI-11178
> URL: https://issues.apache.org/jira/browse/NIFI-11178
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Mark Payne
>Assignee: Lehel Boér
>Priority: Major
> Attachments: image-2023-06-28-20-48-40-182.png,
> image-2023-06-28-20-48-57-012.png
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> ListHDFS is used extremely commonly. Typically, a listing consists of several
> hundred files or less. However, there are times (especially when performing
> the first listing) when the Processor is configured to recurse into
> subdirectories and creates a listing containing millions of files.
> Currently, performing a listing containing millions of files can occupy
> several GB of heap space. Analyzing a recent heap dump, it was found that a
> listing of 6.7 million files in HDFS occupied approximately 12 GB of heap
> space in NiFi. The heap usage can be tracked back to the fact that we hold in
> a local variable a HashSet and these FileStatus objects occupy
> approximately 1-2 KB of heap each.
> There are several improvements that can be made here, some small changes will
> have significant memory improvements. There are also large changes that could
> dramatically reduce heap utilization, to nearly nothing. But these would
> require complex rewrites of the Processor that would be much more difficult
> to maintain.
> As a simple analysis, I built a small test to see how many FileStatus objects
> could be kept in memory:
> {code:java}
> final Set statuses = new HashSet<>();
> for (int i=0; i < 10_000_000; i++) {
> if (i % 10_000 == 0) {
> System.out.println(i);
> }
> final FsPermission fsPermission = new FsPermission(777);
> final FileStatus status = new FileStatus(2, true, 1, 4_000_000,
> System.currentTimeMillis(), 0L, fsPermission,
> "owner-" + i, "group-" + i, null, new Path("/path/to/my/file-" + i +
> ".txt"), true, false, false);
> statuses.add(status);
> }
> {code}
> This gives us a way to see how many FileStatus objects can be added to our
> set before we encounter OOME. With a 512 MB heap I reached approximately 1.13
> million FileStatus objects. Note that the Paths here are very small, which
> occupies less memory than would normally be the case.
> Making one small change, to change the {{Set}} to an
> {{ArrayList}} yielded 1.21 million instead of 1.13 million
> objects. This is reasonable, since we don't expect any duplicates anyway.
> Another small change that was made was to introduce a new class that keeps
> only the fields we care about from the FileStatus:
> {code:java}
> private static class HdfsEntity {
> private final String path;
> private final long length;
> private final boolean isdir;
> private final long timestamp;
> private final boolean canRead;
> private final boolean canWrite;
> private final boolean canExecute;
> private final boolean encrypted;
> public HdfsEntity(final FileStatus status) {
> this.path = status.getPath().getName();
> this.length = status.getLen();
> this.isdir = status.isDirectory();
> this.timestamp = status.getModificationTime();
> this.canRead =
> status.getPermission().getGroupAction().implies(FsAction.READ);
> this.canWrite =
> status.getPermission().getGroupAction().implies(FsAction.WRITE);
> this.canExecute =
> status.getPermission().getGroupAction().implies(FsAction.EXECUTE);
> this.encrypted = status.isEncrypted();
> }
> }
> {code}
> This introduced significant savings, allowing a {{List}} to store
> 5.2 million objects instead of 1.13 million.
> It is worth noting here that the HdfsEntity doesn't store the 'group' and
> 'owner' that are part of the FileStatus. Interestingly, though, these values
> are strings that are extremely repetitive, but are unique strings on the
> heap. So a full solution here would mean tracking the owner and group, but
> doing so in a way that we use a Map or something similar in
> order to reuse identical Strings on the heap (in much the same way that
> String.intern() does, but without interning the String as it's only reusable
> within a small context).
> These changes will yield significant memory improvements.
> However, more penetrating changes can make even large improvements:
> * Rather than keeping a Set or a List of entities at all, we could instead
> just iterate over each of the FileStatus objects returned from HDFS and
> determine whether or not the file should be
[GitHub] [nifi] emiliosetiadarma opened a new pull request, #7446: NIFI-11536: implemented AbstractStoreScanner and KeyStoreScanner that…
emiliosetiadarma opened a new pull request, #7446: URL: https://github.com/apache/nifi/pull/7446 … reloads SSL context, changed TrustStoreScanner to extend the AbstractStoreScanner, and implemented unit tests # Summary [NIFI-11536](https://issues.apache.org/jira/browse/NIFI-11536) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [x] Pull Request based on current revision of the `main` branch - [x] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [x] Build completed using `mvn clean install -P contrib-check` - [x] JDK 17 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #7401: NIFI-11706 Add capability to create dedicated Parameter Contexts for versioned flows when importing
exceptionfactory commented on code in PR #7401: URL: https://github.com/apache/nifi/pull/7401#discussion_r1245594809 ## nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/WEB-INF/partials/canvas/import-flow-version-dialog.jsp: ## @@ -17,6 +17,11 @@ <%@ page contentType="text/html" pageEncoding="UTF-8" session="false" %> + Review Comment: On the further review, placing this checkbox at the top of the dialog does not seem like the best approach. As this is an optional feature that may not be commonly used, recommend moving this checkbox to the end of the dialog window. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #7401: NIFI-11706 Add capability to create dedicated Parameter Contexts for versioned flows when importing
exceptionfactory commented on code in PR #7401:
URL: https://github.com/apache/nifi/pull/7401#discussion_r1245593903
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/ProcessGroupResource.java:
##
@@ -2024,7 +2029,13 @@ public Response createProcessGroup(
}
}
-// Step 4: Resolve Bundle info
+// Step 4: Replace parameter contexts if necessary
+if (!keepExistingParameterContext) {
+final ParameterContextReplacementUtil
parameterContextReplacementUtil =
ParameterContextReplacementUtil.getInstance(serviceFacade);
Review Comment:
This approach to creating an instance of ParameterContextReplacementUtil
seems contrary to other conventions. It seems like it would be better to make
the ParameterContext Replacement operations into an interface that could be
injected. As it is, coupling the entire Service Facade to the Parameter Context
Replacement instantiation is problematic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (NIFI-11762) Upgrade Hadoop Dependencies to 3.3.6
[ https://issues.apache.org/jira/browse/NIFI-11762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17738223#comment-17738223 ] ASF subversion and git services commented on NIFI-11762: Commit f14cf35ea5aed51785824b65d17909e3a6cca8f4 in nifi's branch refs/heads/support/nifi-1.x from David Handermann [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f14cf35ea5 ] NIFI-11762 Upgraded Hadoop dependencies from 3.3.5 to 3.3.6 Signed-off-by: Pierre Villard This closes #7445. > Upgrade Hadoop Dependencies to 3.3.6 > > > Key: NIFI-11762 > URL: https://issues.apache.org/jira/browse/NIFI-11762 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: David Handermann >Assignee: David Handermann >Priority: Minor > Labels: dependency-upgrade > Fix For: 1.latest, 2.latest > > Time Spent: 20m > Remaining Estimate: 0h > > Apache Hadoop [3.3.6|https://hadoop.apache.org/docs/r3.3.6/] includes several > minor improvements as well as upgrades to transitive dependencies. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11762) Upgrade Hadoop Dependencies to 3.3.6
[ https://issues.apache.org/jira/browse/NIFI-11762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-11762: -- Fix Version/s: 2.0.0 1.23.0 (was: 1.latest) (was: 2.latest) Resolution: Fixed Status: Resolved (was: Patch Available) > Upgrade Hadoop Dependencies to 3.3.6 > > > Key: NIFI-11762 > URL: https://issues.apache.org/jira/browse/NIFI-11762 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: David Handermann >Assignee: David Handermann >Priority: Minor > Labels: dependency-upgrade > Fix For: 2.0.0, 1.23.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Apache Hadoop [3.3.6|https://hadoop.apache.org/docs/r3.3.6/] includes several > minor improvements as well as upgrades to transitive dependencies. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11762) Upgrade Hadoop Dependencies to 3.3.6
[ https://issues.apache.org/jira/browse/NIFI-11762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17738222#comment-17738222 ] ASF subversion and git services commented on NIFI-11762: Commit 3aba769cc37491eddd623aecd49e54e4066e58dd in nifi's branch refs/heads/main from David Handermann [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=3aba769cc3 ] NIFI-11762 Upgraded Hadoop dependencies from 3.3.5 to 3.3.6 Signed-off-by: Pierre Villard This closes #7445. > Upgrade Hadoop Dependencies to 3.3.6 > > > Key: NIFI-11762 > URL: https://issues.apache.org/jira/browse/NIFI-11762 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: David Handermann >Assignee: David Handermann >Priority: Minor > Labels: dependency-upgrade > Fix For: 1.latest, 2.latest > > Time Spent: 10m > Remaining Estimate: 0h > > Apache Hadoop [3.3.6|https://hadoop.apache.org/docs/r3.3.6/] includes several > minor improvements as well as upgrades to transitive dependencies. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6918: NIFI-11129 - adding PutMongoBulk processor to use the bulkWrite API
exceptionfactory commented on code in PR #6918:
URL: https://github.com/apache/nifi/pull/6918#discussion_r1245397601
##
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/PutMongoBulk.java:
##
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.mongodb;
+
+import com.mongodb.WriteConcern;
+import com.mongodb.client.MongoClient;
+import com.mongodb.client.ClientSession;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.client.model.BulkWriteOptions;
+import com.mongodb.client.model.Collation;
+import com.mongodb.client.model.CollationAlternate;
+import com.mongodb.client.model.CollationCaseFirst;
+import com.mongodb.client.model.CollationMaxVariable;
+import com.mongodb.client.model.CollationStrength;
+import com.mongodb.client.model.DeleteManyModel;
+import com.mongodb.client.model.DeleteOneModel;
+import com.mongodb.client.model.DeleteOptions;
+import com.mongodb.client.model.InsertOneModel;
+import com.mongodb.client.model.ReplaceOneModel;
+import com.mongodb.client.model.ReplaceOptions;
+import com.mongodb.client.model.UpdateManyModel;
+import com.mongodb.client.model.UpdateOneModel;
+import com.mongodb.client.model.UpdateOptions;
+import com.mongodb.client.model.WriteModel;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SystemResource;
+import org.apache.nifi.annotation.behavior.SystemResourceConsideration;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.bson.BsonArray;
+import org.bson.BsonDocument;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@EventDriven
+@Tags({ "mongodb", "insert", "update", "write", "put", "bulk" })
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Writes the contents of a FlowFile to MongoDB as
bulk-update")
+@SystemResourceConsideration(resource = SystemResource.MEMORY)
+public class PutMongoBulk extends AbstractMongoProcessor {
+static final Relationship REL_SUCCESS = new
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to MongoDB are routed
to this relationship").build();
+static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to MongoDB are
routed to this relationship").build();
+
+static final PropertyDescriptor ORDERED = new PropertyDescriptor.Builder()
+.name("Ordered")
+.expressionLanguageSupported(ExpressionLanguageScope.NONE)
+.description("Ordered execution of bulk-writes & break on error -
otherwise arbitrary order & continue on error")
+.required(false)
+.addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+.allowableValues("true", "false")
+.defaultValue("false")
+.build();
+
+static final PropertyDescriptor USE_TRANSACTION = new
PropertyDescriptor.Builder()
+.name("Use transaction")
+.expressionLanguageSupported(ExpressionLanguageScope.NONE)
+.description("Run all actions in one MongoDB transaction - does
NOT work with a client service so far")
+.required(false)
+
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #6918: NIFI-11129 - adding PutMongoBulk processor to use the bulkWrite API
exceptionfactory commented on code in PR #6918:
URL: https://github.com/apache/nifi/pull/6918#discussion_r1245397297
##
nifi-nar-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/main/java/org/apache/nifi/processors/mongodb/PutMongoBulk.java:
##
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.mongodb;
+
+import com.mongodb.WriteConcern;
+import com.mongodb.client.MongoClient;
+import com.mongodb.client.ClientSession;
+import com.mongodb.client.MongoCollection;
+import com.mongodb.client.model.BulkWriteOptions;
+import com.mongodb.client.model.Collation;
+import com.mongodb.client.model.CollationAlternate;
+import com.mongodb.client.model.CollationCaseFirst;
+import com.mongodb.client.model.CollationMaxVariable;
+import com.mongodb.client.model.CollationStrength;
+import com.mongodb.client.model.DeleteManyModel;
+import com.mongodb.client.model.DeleteOneModel;
+import com.mongodb.client.model.DeleteOptions;
+import com.mongodb.client.model.InsertOneModel;
+import com.mongodb.client.model.ReplaceOneModel;
+import com.mongodb.client.model.ReplaceOptions;
+import com.mongodb.client.model.UpdateManyModel;
+import com.mongodb.client.model.UpdateOneModel;
+import com.mongodb.client.model.UpdateOptions;
+import com.mongodb.client.model.WriteModel;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SystemResource;
+import org.apache.nifi.annotation.behavior.SystemResourceConsideration;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.ExpressionLanguageScope;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.bson.BsonArray;
+import org.bson.BsonDocument;
+import org.bson.Document;
+import org.bson.conversions.Bson;
+
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+@EventDriven
+@Tags({ "mongodb", "insert", "update", "write", "put", "bulk" })
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Writes the contents of a FlowFile to MongoDB as
bulk-update")
+@SystemResourceConsideration(resource = SystemResource.MEMORY)
+public class PutMongoBulk extends AbstractMongoProcessor {
+static final Relationship REL_SUCCESS = new
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to MongoDB are routed
to this relationship").build();
+static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to MongoDB are
routed to this relationship").build();
+
+static final PropertyDescriptor ORDERED = new PropertyDescriptor.Builder()
+.name("Ordered")
+.expressionLanguageSupported(ExpressionLanguageScope.NONE)
+.description("Ordered execution of bulk-writes & break on error -
otherwise arbitrary order & continue on error")
+.required(false)
+.addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+.allowableValues("true", "false")
+.defaultValue("false")
+.build();
+
+static final PropertyDescriptor USE_TRANSACTION = new
PropertyDescriptor.Builder()
+.name("Use transaction")
+.expressionLanguageSupported(ExpressionLanguageScope.NONE)
+.description("Run all actions in one MongoDB transaction - does
NOT work with a client service so far")
+.required(false)
+
[jira] [Updated] (NIFI-11762) Upgrade Hadoop Dependencies to 3.3.6
[ https://issues.apache.org/jira/browse/NIFI-11762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-11762: Status: Patch Available (was: Open) > Upgrade Hadoop Dependencies to 3.3.6 > > > Key: NIFI-11762 > URL: https://issues.apache.org/jira/browse/NIFI-11762 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: David Handermann >Assignee: David Handermann >Priority: Minor > Labels: dependency-upgrade > Fix For: 1.latest, 2.latest > > Time Spent: 10m > Remaining Estimate: 0h > > Apache Hadoop [3.3.6|https://hadoop.apache.org/docs/r3.3.6/] includes several > minor improvements as well as upgrades to transitive dependencies. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] exceptionfactory opened a new pull request, #7445: NIFI-11762 Upgrade Hadoop dependencies from 3.3.5 to 3.3.6
exceptionfactory opened a new pull request, #7445: URL: https://github.com/apache/nifi/pull/7445 # Summary [NIFI-11762](https://issues.apache.org/jira/browse/NIFI-11762) Upgrades Apache Hadoop dependencies from 3.3.5 to [3.3.6](https://hadoop.apache.org/docs/r3.3.6/) incorporating several improvements and transitive dependency upgrades. # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X] Pull Request based on current revision of the `main` branch - [X] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 17 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] MikeThomsen commented on pull request #6918: NIFI-11129 - adding PutMongoBulk processor to use the bulkWrite API
MikeThomsen commented on PR #6918: URL: https://github.com/apache/nifi/pull/6918#issuecomment-1611624670 > Thanks for the reference @MikeThomsen, that is a helpful comparison. With that in mind, does it make sense to align this Processor to something more record-oriented, as opposed to requiring JSON? No, I think it's good the way it is, and it doesn't bill itself as a record-aware processor. Maybe adding "Operations" to the end of it to further clarify that it is calling the BulkWrite API, but as-is I think it should fit a useful Mongo niche. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-11762) Upgrade Hadoop Dependencies to 3.3.6
David Handermann created NIFI-11762: --- Summary: Upgrade Hadoop Dependencies to 3.3.6 Key: NIFI-11762 URL: https://issues.apache.org/jira/browse/NIFI-11762 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: David Handermann Assignee: David Handermann Fix For: 1.latest, 2.latest Apache Hadoop [3.3.6|https://hadoop.apache.org/docs/r3.3.6/] includes several minor improvements as well as upgrades to transitive dependencies. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11759) Remove Distributed Cache Map Service Client from ListHDFS
[ https://issues.apache.org/jira/browse/NIFI-11759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17738168#comment-17738168 ] ASF subversion and git services commented on NIFI-11759: Commit 7b5853363f2951d5c23583219370cb1df9e830a0 in nifi's branch refs/heads/main from Matt Burgess [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=7b5853363f ] NIFI-11759: Remove Distributed Map Cache Client property from ListHDFS This closes #7443 Signed-off-by: Mike Thomsen > Remove Distributed Cache Map Service Client from ListHDFS > - > > Key: NIFI-11759 > URL: https://issues.apache.org/jira/browse/NIFI-11759 > Project: Apache NiFi > Issue Type: Task > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 2.latest > > Time Spent: 10m > Remaining Estimate: 0h > > Continuing the work of NIFI-6023 which deprecated/ignored the use of the > Distributed Map Cache in ListHDFS which was replaced with State Management, > this Jira is to remove the property from the processor for the main (upcoming > 2.0) branch. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11759) Remove Distributed Cache Map Service Client from ListHDFS
[ https://issues.apache.org/jira/browse/NIFI-11759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Thomsen updated NIFI-11759: Resolution: Fixed Status: Resolved (was: Patch Available) > Remove Distributed Cache Map Service Client from ListHDFS > - > > Key: NIFI-11759 > URL: https://issues.apache.org/jira/browse/NIFI-11759 > Project: Apache NiFi > Issue Type: Task > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 2.latest > > Time Spent: 20m > Remaining Estimate: 0h > > Continuing the work of NIFI-6023 which deprecated/ignored the use of the > Distributed Map Cache in ListHDFS which was replaced with State Management, > this Jira is to remove the property from the processor for the main (upcoming > 2.0) branch. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] asfgit closed pull request #7443: NIFI-11759: Remove Distributed Map Cache Client property from ListHDFS
asfgit closed pull request #7443: NIFI-11759: Remove Distributed Map Cache Client property from ListHDFS URL: https://github.com/apache/nifi/pull/7443 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #7401: NIFI-11706 Add capability to create dedicated Parameter Contexts for versioned flows when importing
exceptionfactory commented on code in PR #7401:
URL: https://github.com/apache/nifi/pull/7401#discussion_r1245364933
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/WEB-INF/partials/canvas/import-flow-version-dialog.jsp:
##
@@ -17,6 +17,11 @@
<%@ page contentType="text/html" pageEncoding="UTF-8" session="false" %>
+
+
+Keep Existing Parameter
Contexts
+
Review Comment:
Recommend adjusting the wording:
```suggestion
```
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/util/ParameterContextNameCollusionResolver.java:
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.web.util;
+
+import org.apache.nifi.web.api.dto.ParameterContextDTO;
+import org.apache.nifi.web.api.entity.ParameterContextEntity;
+
+import java.util.Collection;
+import java.util.Set;
+import java.util.function.Supplier;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+class ParameterContextNameCollusionResolver {
+private static final String LINEAGE_FORMAT = "^(?.+?)(
\\((?[0-9]+)\\))?$";
+private static final Pattern LINEAGE_PATTERN =
Pattern.compile(LINEAGE_FORMAT);
+
+ParameterContextNameCollusionResolver(final
Supplier> parameterContextSource) {
+this.parameterContextSource = parameterContextSource;
+}
+
+private final Supplier>
parameterContextSource;
+
+public String resolveNameCollusion(final String
originalParameterContextName) {
+final Matcher lineageMatcher =
LINEAGE_PATTERN.matcher(originalParameterContextName);
+
+if (!lineageMatcher.matches()) {
+throw new IllegalArgumentException("Existing Parameter Context
name \"(" + originalParameterContextName + "\") cannot be processed");
+}
+
+final String lineName = lineageMatcher.group("name");
+final String originalIndex = lineageMatcher.group("index");
Review Comment:
Recommend defining static variables for `name` and `index` and reusing
through this method.
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/util/ParameterContextNameCollusionResolver.java:
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.web.util;
+
+import org.apache.nifi.web.api.dto.ParameterContextDTO;
+import org.apache.nifi.web.api.entity.ParameterContextEntity;
+
+import java.util.Collection;
+import java.util.Set;
+import java.util.function.Supplier;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+class ParameterContextNameCollusionResolver {
Review Comment:
`Collusion` needs to be replaced with `Collision` across all references.
```suggestion
class ParameterContextNameCollisionResolver {
```
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/canvas/nf-flow-version.js:
##
@@ -1045,7 +1048,8 @@
return $.ajax({
type: 'POST',
data: JSON.stringify(processGroupEntity),
-url: '../nifi-api/process-groups/' +
encodeURIComponent(nfCanvasUtils.getGroupId()) + '/process-groups',
+
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #7401: NIFI-11706 Add capability to create dedicated Parameter Contexts for versioned flows when importing
exceptionfactory commented on code in PR #7401:
URL: https://github.com/apache/nifi/pull/7401#discussion_r1245360645
##
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/web/util/ParameterContextNameCollusionResolverTest.java:
##
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.web.util;
+
+import org.apache.nifi.web.api.dto.ParameterContextDTO;
+import org.apache.nifi.web.api.entity.ParameterContextEntity;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.mockito.Mockito;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.function.Supplier;
+import java.util.stream.Stream;
+
+
+class ParameterContextNameCollusionResolverTest {
Review Comment:
Spelling correction:
```suggestion
class ParameterContextNameCollisionResolverTest {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Assigned] (NIFI-11303) Create direct go-to option from Provenance lineage
[ https://issues.apache.org/jira/browse/NIFI-11303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynaldo Rea reassigned NIFI-11303: --- Assignee: Reynaldo Rea > Create direct go-to option from Provenance lineage > -- > > Key: NIFI-11303 > URL: https://issues.apache.org/jira/browse/NIFI-11303 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.19.1 >Reporter: Mark Bean >Assignee: Reynaldo Rea >Priority: Major > > The process to navigate from a Provenance event in the lineage diagram to the > component which generated the event is a little cumbersome. It requires > right-clicking on the event, choose 'View details', locating the Component > Id, returning to the main graph, entering the Component Id in the Search bar > and finally selecting the component. > It would be useful and more direct to have a "go to" link option available on > the context menu when right-clicking on the event. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11761) Timed out minifi restart and kill doesn't work
Ferenc Kis created NIFI-11761: - Summary: Timed out minifi restart and kill doesn't work Key: NIFI-11761 URL: https://issues.apache.org/jira/browse/NIFI-11761 Project: Apache NiFi Issue Type: Bug Components: MiNiFi Reporter: Ferenc Kis Assignee: Ferenc Kis -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11761) Timed out minifi restart and kill doesn't work
[ https://issues.apache.org/jira/browse/NIFI-11761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Kis updated NIFI-11761: -- Description: It looks like when graceful shutdown period expires and minifi is killed during restart then the start is not happening. > Timed out minifi restart and kill doesn't work > -- > > Key: NIFI-11761 > URL: https://issues.apache.org/jira/browse/NIFI-11761 > Project: Apache NiFi > Issue Type: Bug > Components: MiNiFi >Reporter: Ferenc Kis >Assignee: Ferenc Kis >Priority: Major > > It looks like when graceful shutdown period expires and minifi is killed > during restart then the start is not happening. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11671) JoinEnrichment SQL strategy doesn't allow attributes in join statement
[
https://issues.apache.org/jira/browse/NIFI-11671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philipp Korniets updated NIFI-11671:
Description:
We use ForkEnrichement - JoinEnrichment pattern and want to include filtering
in join SQL. Filter value is coming from FlowFile attribute
{code:sql}
${test} = 'NewValue'
SELECT original.*, enrichment.*,'${test}'
FROM original
LEFT OUTER JOIN enrichment
ON original.Underlying = enrichment.Underlying
WHERE enrichment.MyField = '${test}'
{code}
However this doesnt work because JoinEnrichment doesnt use
evaluateAttributeExpressions
Additionally in version 1.18 - doesnt allow whole query to be passed as
attribute.
!screenshot-1.png|width=692,height=431!
{code:java}
2023-06-28 11:07:16,611 ERROR [Timer-Driven Process Thread-7]
o.a.n.processors.standard.JoinEnrichment
JoinEnrichment[id=dbe156ac-0187-1000-4477-0183899e0432] Failed to join
'original' FlowFile
StandardFlowFileRecord[uuid=2ab9f6ad-73a5-4763-b25e-fd26c44835e1,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1687948831976-629, container=default,
section=629], offset=8334082,
length=600557],offset=0,name=lmr_SY08C41-1_S_514682_20230627.csv,size=600557]
and 'enrichment' FlowFile
StandardFlowFileRecord[uuid=e4bb7769-fdce-4dfe-af18-443676103035,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1687949723375-631, container=default,
section=631], offset=5362822,
length=1999502],offset=0,name=lmr_SY08C41-1_S_514682_20230627.csv,size=1999502];
routing to failure
java.sql.SQLException: Error while preparing statement [${instrumentJoinSQL}]
at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
at
org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:224)
at
org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:203)
at
org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:99)
at
org.apache.calcite.avatica.AvaticaConnection.prepareStatement(AvaticaConnection.java:178)
at
org.apache.nifi.processors.standard.enrichment.SqlJoinCache.createCalciteParameters(SqlJoinCache.java:91)
at
org.apache.nifi.processors.standard.enrichment.SqlJoinCache.getCalciteParameters(SqlJoinCache.java:65)
at
org.apache.nifi.processors.standard.enrichment.SqlJoinStrategy.join(SqlJoinStrategy.java:49)
at
org.apache.nifi.processors.standard.JoinEnrichment.processBin(JoinEnrichment.java:387)
at
org.apache.nifi.processor.util.bin.BinFiles.processBins(BinFiles.java:233)
at
org.apache.nifi.processors.standard.JoinEnrichment.processBins(JoinEnrichment.java:503)
at org.apache.nifi.processor.util.bin.BinFiles.onTrigger(BinFiles.java:193)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1354)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:246)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.RuntimeException: parse failed: Encountered "$" at line 1,
column 1.
Was expecting one of:
"ABS" ...
{code}
As I understand issue is in following line of code
https://github.com/apache/nifi/blob/ce0122bd2530e37eb57ace919a42bb2eb3cd5c02/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/JoinEnrichment.java#L441
instead of:
{code:java}
final String sql = context.getProperty(SQL).getValue();
{code}
it should be:
{code:java}
final String sql =
context.getProperty(SQL).evaluateAttributeExpressions(attributes).getValue();
{code}
was:
We use ForkEnrichement - JoinEnrichment pattern and want to include filtering
in join SQL. Filter value is coming from FlowFile attribute
{code:sql}
${test} = 'NewValue'
SELECT original.*, enrichment.*,'${test}'
FROM original
LEFT OUTER JOIN enrichment
ON original.Underlying = enrichment.Underlying
WHERE enrichment.MyField = '${test}'
{code}
However this doesnt work because JoinEnrichment doesnt use
evaluateAttributeExpressions
Additionally
[GitHub] [nifi] exceptionfactory commented on pull request #7442: NIFI-11754 Refactored Groovy test in nifi-jetty to Java (and JUnit 5)
exceptionfactory commented on PR #7442: URL: https://github.com/apache/nifi/pull/7442#issuecomment-1611393422 @dan-s1 The GitHub Actions builds have the default log level set to warning, so they do not print every test class status. This may be something to revisit, but it does generate a lot of output. The `HostHeaderHandlerTest` shows up in a local build. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (MINIFICPP-2153) Change MergeContent default to Bin Packing
Adam Debreceni created MINIFICPP-2153: - Summary: Change MergeContent default to Bin Packing Key: MINIFICPP-2153 URL: https://issues.apache.org/jira/browse/MINIFICPP-2153 Project: Apache NiFi MiNiFi C++ Issue Type: Improvement Reporter: Adam Debreceni Assignee: Adam Debreceni -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11671) JoinEnrichment SQL strategy doesn't allow attributes in join statement
[
https://issues.apache.org/jira/browse/NIFI-11671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philipp Korniets updated NIFI-11671:
Description:
We use ForkEnrichement - JoinEnrichment pattern and want to include filtering
in join SQL. Filter value is coming from FlowFile attribute
{code:sql}
${test} = 'NewValue'
SELECT original.*, enrichment.*,'${test}'
FROM original
LEFT OUTER JOIN enrichment
ON original.Underlying = enrichment.Underlying
WHERE enrichment.MyField = '${test}'
{code}
However this doesnt work because JoinEnrichment doesnt use
evaluateAttributeExpressions
Additionally in version 1.18 - doesnt allow whole query to be passed as
attribute.
!screenshot-1.png|width=692,height=431!
{code:java}
2023-06-28 11:07:16,611 ERROR [Timer-Driven Process Thread-7]
o.a.n.processors.standard.JoinEnrichment
JoinEnrichment[id=dbe156ac-0187-1000-4477-0183899e0432] Failed to join
'original' FlowFile
StandardFlowFileRecord[uuid=2ab9f6ad-73a5-4763-b25e-fd26c44835e1,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1687948831976-629, container=default,
section=629], offset=8334082,
length=600557],offset=0,name=lmr_SY08C41-1_S_514682_20230627.csv,size=600557]
and 'enrichment' FlowFile
StandardFlowFileRecord[uuid=e4bb7769-fdce-4dfe-af18-443676103035,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1687949723375-631, container=default,
section=631], offset=5362822,
length=1999502],offset=0,name=lmr_SY08C41-1_S_514682_20230627.csv,size=1999502];
routing to failure
java.sql.SQLException: Error while preparing statement [${instrumentJoinSQL}]
at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
at
org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:224)
at
org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:203)
at
org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:99)
at
org.apache.calcite.avatica.AvaticaConnection.prepareStatement(AvaticaConnection.java:178)
at
org.apache.nifi.processors.standard.enrichment.SqlJoinCache.createCalciteParameters(SqlJoinCache.java:91)
at
org.apache.nifi.processors.standard.enrichment.SqlJoinCache.getCalciteParameters(SqlJoinCache.java:65)
at
org.apache.nifi.processors.standard.enrichment.SqlJoinStrategy.join(SqlJoinStrategy.java:49)
at
org.apache.nifi.processors.standard.JoinEnrichment.processBin(JoinEnrichment.java:387)
at
org.apache.nifi.processor.util.bin.BinFiles.processBins(BinFiles.java:233)
at
org.apache.nifi.processors.standard.JoinEnrichment.processBins(JoinEnrichment.java:503)
at org.apache.nifi.processor.util.bin.BinFiles.onTrigger(BinFiles.java:193)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1354)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:246)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.RuntimeException: parse failed: Encountered "$" at line 1,
column 1.
Was expecting one of:
"ABS" ...
{code}
was:
We use ForkEnrichement - JoinEnrichment pattern and want to include filtering
in join SQL. Filter value is coming from FlowFile attribute
{code:sql}
${test} = 'NewValue'
SELECT original.*, enrichment.*,'${test}'
FROM original
LEFT OUTER JOIN enrichment
ON original.Underlying = enrichment.Underlying
WHERE enrichment.MyField = '${test}'
{code}
However this doesnt work because JoinEnrichment doesnt use
evaluateAttributeExpressions
Additionally in version 1.18 - doesnt allow whole query to be passed as
attribute.
!screenshot-1.png|width=692,height=431!
> JoinEnrichment SQL strategy doesn't allow attributes in join statement
> --
>
> Key: NIFI-11671
> URL: https://issues.apache.org/jira/browse/NIFI-11671
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions:
[jira] [Updated] (NIFI-11671) JoinEnrichment SQL strategy doesn't allow attributes in join statement
[
https://issues.apache.org/jira/browse/NIFI-11671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philipp Korniets updated NIFI-11671:
Attachment: screenshot-1.png
> JoinEnrichment SQL strategy doesn't allow attributes in join statement
> --
>
> Key: NIFI-11671
> URL: https://issues.apache.org/jira/browse/NIFI-11671
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.18.0, 1.20.0
>Reporter: Philipp Korniets
>Priority: Major
> Attachments: screenshot-1.png
>
>
> We use ForkEnrichement - JoinEnrichment pattern and want to include filtering
> in join SQL. Filter value is coming from FlowFile attribute
> {code:sql}
> ${test} = 'NewValue'
> SELECT original.*, enrichment.*,'${test}'
> FROM original
> LEFT OUTER JOIN enrichment
> ON original.Underlying = enrichment.Underlying
> WHERE enrichment.MyField = '${test}'
> {code}
> However this doesnt work because JoinEnrichment doesnt use
> evaluateAttributeExpressions
> Additionally in version 1.18 - doesnt allow whole query to be passed as
> attribute.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-11671) JoinEnrichment SQL strategy doesn't allow attributes in join statement
[
https://issues.apache.org/jira/browse/NIFI-11671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philipp Korniets updated NIFI-11671:
Description:
We use ForkEnrichement - JoinEnrichment pattern and want to include filtering
in join SQL. Filter value is coming from FlowFile attribute
{code:sql}
${test} = 'NewValue'
SELECT original.*, enrichment.*,'${test}'
FROM original
LEFT OUTER JOIN enrichment
ON original.Underlying = enrichment.Underlying
WHERE enrichment.MyField = '${test}'
{code}
However this doesnt work because JoinEnrichment doesnt use
evaluateAttributeExpressions
Additionally in version 1.18 - doesnt allow whole query to be passed as
attribute.
!screenshot-1.png|width=692,height=431!
was:
We use ForkEnrichement - JoinEnrichment pattern and want to include filtering
in join SQL. Filter value is coming from FlowFile attribute
{code:sql}
${test} = 'NewValue'
SELECT original.*, enrichment.*,'${test}'
FROM original
LEFT OUTER JOIN enrichment
ON original.Underlying = enrichment.Underlying
WHERE enrichment.MyField = '${test}'
{code}
However this doesnt work because JoinEnrichment doesnt use
evaluateAttributeExpressions
Additionally in version 1.18 - doesnt allow whole query to be passed as
attribute.
!screenshot-1.png!
> JoinEnrichment SQL strategy doesn't allow attributes in join statement
> --
>
> Key: NIFI-11671
> URL: https://issues.apache.org/jira/browse/NIFI-11671
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.18.0, 1.20.0
>Reporter: Philipp Korniets
>Priority: Major
> Attachments: screenshot-1.png
>
>
> We use ForkEnrichement - JoinEnrichment pattern and want to include filtering
> in join SQL. Filter value is coming from FlowFile attribute
> {code:sql}
> ${test} = 'NewValue'
> SELECT original.*, enrichment.*,'${test}'
> FROM original
> LEFT OUTER JOIN enrichment
> ON original.Underlying = enrichment.Underlying
> WHERE enrichment.MyField = '${test}'
> {code}
> However this doesnt work because JoinEnrichment doesnt use
> evaluateAttributeExpressions
> Additionally in version 1.18 - doesnt allow whole query to be passed as
> attribute.
> !screenshot-1.png|width=692,height=431!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-11671) JoinEnrichment SQL strategy doesn't allow attributes in join statement
[
https://issues.apache.org/jira/browse/NIFI-11671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philipp Korniets updated NIFI-11671:
Description:
We use ForkEnrichement - JoinEnrichment pattern and want to include filtering
in join SQL. Filter value is coming from FlowFile attribute
{code:sql}
${test} = 'NewValue'
SELECT original.*, enrichment.*,'${test}'
FROM original
LEFT OUTER JOIN enrichment
ON original.Underlying = enrichment.Underlying
WHERE enrichment.MyField = '${test}'
{code}
However this doesnt work because JoinEnrichment doesnt use
evaluateAttributeExpressions
Additionally in version 1.18 - doesnt allow whole query to be passed as
attribute.
was:
We use ForkEnrichement - JoinEnrichment pattern and want to include filtering
in join SQL. Filter value is coming from FlowFile attribute
{code:sql}
${test} = 'NewValue'
SELECT original.*, enrichment.*,'${test}'
FROM original
LEFT OUTER JOIN enrichment
ON original.Underlying = enrichment.Underlying
WHERE enrichment.MyField = '${test}'
{code}
However this doesnt work because JoinEnrichment doesnt use
evaluateAttributeExpressions
Additionally in version 1.18 - doesnt allow whole query to be passed as
attribute.
!screenshot-1.png!
> JoinEnrichment SQL strategy doesn't allow attributes in join statement
> --
>
> Key: NIFI-11671
> URL: https://issues.apache.org/jira/browse/NIFI-11671
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.18.0, 1.20.0
>Reporter: Philipp Korniets
>Priority: Major
> Attachments: screenshot-1.png
>
>
> We use ForkEnrichement - JoinEnrichment pattern and want to include filtering
> in join SQL. Filter value is coming from FlowFile attribute
> {code:sql}
> ${test} = 'NewValue'
> SELECT original.*, enrichment.*,'${test}'
> FROM original
> LEFT OUTER JOIN enrichment
> ON original.Underlying = enrichment.Underlying
> WHERE enrichment.MyField = '${test}'
> {code}
> However this doesnt work because JoinEnrichment doesnt use
> evaluateAttributeExpressions
> Additionally in version 1.18 - doesnt allow whole query to be passed as
> attribute.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-11671) JoinEnrichment SQL strategy doesn't allow attributes in join statement
[
https://issues.apache.org/jira/browse/NIFI-11671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philipp Korniets updated NIFI-11671:
Description:
We use ForkEnrichement - JoinEnrichment pattern and want to include filtering
in join SQL. Filter value is coming from FlowFile attribute
{code:sql}
${test} = 'NewValue'
SELECT original.*, enrichment.*,'${test}'
FROM original
LEFT OUTER JOIN enrichment
ON original.Underlying = enrichment.Underlying
WHERE enrichment.MyField = '${test}'
{code}
However this doesnt work because JoinEnrichment doesnt use
evaluateAttributeExpressions
Additionally in version 1.18 - doesnt allow whole query to be passed as
attribute.
!screenshot-1.png!
was:
We use ForkEnrichement - JoinEnrichment pattern and want to include filtering
in join SQL. Filter value is coming from FlowFile attribute
{code:sql}
${test} = 'NewValue'
SELECT original.*, enrichment.*,'${test}'
FROM original
LEFT OUTER JOIN enrichment
ON original.Underlying = enrichment.Underlying
WHERE enrichment.MyField = '${test}'
{code}
However this doesnt work because JoinEnrichment doesnt use
evaluateAttributeExpressions
Additionally in version 1.18 - doesnt allow whole query to be passed as
attribute.
> JoinEnrichment SQL strategy doesn't allow attributes in join statement
> --
>
> Key: NIFI-11671
> URL: https://issues.apache.org/jira/browse/NIFI-11671
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.18.0, 1.20.0
>Reporter: Philipp Korniets
>Priority: Major
> Attachments: screenshot-1.png
>
>
> We use ForkEnrichement - JoinEnrichment pattern and want to include filtering
> in join SQL. Filter value is coming from FlowFile attribute
> {code:sql}
> ${test} = 'NewValue'
> SELECT original.*, enrichment.*,'${test}'
> FROM original
> LEFT OUTER JOIN enrichment
> ON original.Underlying = enrichment.Underlying
> WHERE enrichment.MyField = '${test}'
> {code}
> However this doesnt work because JoinEnrichment doesnt use
> evaluateAttributeExpressions
> Additionally in version 1.18 - doesnt allow whole query to be passed as
> attribute.
> !screenshot-1.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-11671) JoinEnrichment SQL strategy doesn't allow attributes in join statement
[
https://issues.apache.org/jira/browse/NIFI-11671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philipp Korniets updated NIFI-11671:
Attachment: (was: screenshot-1.png)
> JoinEnrichment SQL strategy doesn't allow attributes in join statement
> --
>
> Key: NIFI-11671
> URL: https://issues.apache.org/jira/browse/NIFI-11671
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.18.0, 1.20.0
>Reporter: Philipp Korniets
>Priority: Major
>
> We use ForkEnrichement - JoinEnrichment pattern and want to include filtering
> in join SQL. Filter value is coming from FlowFile attribute
> {code:sql}
> ${test} = 'NewValue'
> SELECT original.*, enrichment.*,'${test}'
> FROM original
> LEFT OUTER JOIN enrichment
> ON original.Underlying = enrichment.Underlying
> WHERE enrichment.MyField = '${test}'
> {code}
> However this doesnt work because JoinEnrichment doesnt use
> evaluateAttributeExpressions
> Additionally in version 1.18 - doesnt allow whole query to be passed as
> attribute.
> !screenshot-1.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-11671) JoinEnrichment SQL strategy doesn't allow attributes in join statement
[
https://issues.apache.org/jira/browse/NIFI-11671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philipp Korniets updated NIFI-11671:
Description:
We use ForkEnrichement - JoinEnrichment pattern and want to include filtering
in join SQL. Filter value is coming from FlowFile attribute
{code:sql}
${test} = 'NewValue'
SELECT original.*, enrichment.*,'${test}'
FROM original
LEFT OUTER JOIN enrichment
ON original.Underlying = enrichment.Underlying
WHERE enrichment.MyField = '${test}'
{code}
However this doesnt work because JoinEnrichment doesnt use
evaluateAttributeExpressions
Additionally in version 1.18 - doesnt allow whole query to be passed as
attribute.
!screenshot-1.png!
was:
We use ForkEnrichement - JoinEnrichment pattern and want to include filtering
in join SQL. Filter value is coming from FlowFile attribute
{code:sql}
${test} = 'NewValue'
SELECT original.*, enrichment.*,'${test}'
FROM original
LEFT OUTER JOIN enrichment
ON original.Underlying = enrichment.Underlying
WHERE enrichment.MyField = '${test}'
{code}
However this doesnt work because JoinEnrichment doesnt use
evaluateAttributeExpressions
Additionaly in version 1.18
!screenshot-1.png!
> JoinEnrichment SQL strategy doesn't allow attributes in join statement
> --
>
> Key: NIFI-11671
> URL: https://issues.apache.org/jira/browse/NIFI-11671
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.18.0, 1.20.0
>Reporter: Philipp Korniets
>Priority: Major
>
> We use ForkEnrichement - JoinEnrichment pattern and want to include filtering
> in join SQL. Filter value is coming from FlowFile attribute
> {code:sql}
> ${test} = 'NewValue'
> SELECT original.*, enrichment.*,'${test}'
> FROM original
> LEFT OUTER JOIN enrichment
> ON original.Underlying = enrichment.Underlying
> WHERE enrichment.MyField = '${test}'
> {code}
> However this doesnt work because JoinEnrichment doesnt use
> evaluateAttributeExpressions
> Additionally in version 1.18 - doesnt allow whole query to be passed as
> attribute.
> !screenshot-1.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-11671) JoinEnrichment SQL strategy doesn't allow attributes in join statement
[
https://issues.apache.org/jira/browse/NIFI-11671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philipp Korniets updated NIFI-11671:
Description:
We use ForkEnrichement - JoinEnrichment pattern and want to include filtering
in join SQL. Filter value is coming from FlowFile attribute
{code:sql}
${test} = 'NewValue'
SELECT original.*, enrichment.*,'${test}'
FROM original
LEFT OUTER JOIN enrichment
ON original.Underlying = enrichment.Underlying
WHERE enrichment.MyField = '${test}'
{code}
However this doesnt work because JoinEnrichment doesnt use
evaluateAttributeExpressions
Additionaly in version 1.18
!screenshot-1.png!
was:
We use ForkEnrichement - JoinEnrichment pattern and want to include filtering
in join SQL. Filter value is coming from FlowFile attribute
{code:sql}
${test} = 'NewValue'
SELECT original.*, enrichment.*,'${test}'
FROM original
LEFT OUTER JOIN enrichment
ON original.Underlying = enrichment.Underlying
WHERE enrichment.MyField = '${test}'
{code}
However this doesnt work because JoinEnrichment doesnt use
evaluateAttributeExpressions
> JoinEnrichment SQL strategy doesn't allow attributes in join statement
> --
>
> Key: NIFI-11671
> URL: https://issues.apache.org/jira/browse/NIFI-11671
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.18.0, 1.20.0
>Reporter: Philipp Korniets
>Priority: Major
> Attachments: screenshot-1.png
>
>
> We use ForkEnrichement - JoinEnrichment pattern and want to include filtering
> in join SQL. Filter value is coming from FlowFile attribute
> {code:sql}
> ${test} = 'NewValue'
> SELECT original.*, enrichment.*,'${test}'
> FROM original
> LEFT OUTER JOIN enrichment
> ON original.Underlying = enrichment.Underlying
> WHERE enrichment.MyField = '${test}'
> {code}
> However this doesnt work because JoinEnrichment doesnt use
> evaluateAttributeExpressions
> Additionaly in version 1.18
> !screenshot-1.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-11671) JoinEnrichment SQL strategy doesn't allow attributes in join statement
[
https://issues.apache.org/jira/browse/NIFI-11671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philipp Korniets updated NIFI-11671:
Attachment: screenshot-1.png
> JoinEnrichment SQL strategy doesn't allow attributes in join statement
> --
>
> Key: NIFI-11671
> URL: https://issues.apache.org/jira/browse/NIFI-11671
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.18.0, 1.20.0
>Reporter: Philipp Korniets
>Priority: Major
> Attachments: screenshot-1.png
>
>
> We use ForkEnrichement - JoinEnrichment pattern and want to include filtering
> in join SQL. Filter value is coming from FlowFile attribute
> {code:sql}
> ${test} = 'NewValue'
> SELECT original.*, enrichment.*,'${test}'
> FROM original
> LEFT OUTER JOIN enrichment
> ON original.Underlying = enrichment.Underlying
> WHERE enrichment.MyField = '${test}'
> {code}
> However this doesnt work because JoinEnrichment doesnt use
> evaluateAttributeExpressions
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [nifi] nandorsoma closed pull request #7045: NIFI-11288 Add missing dependencies required by AWS AssumeRoleWithWebIdentity method
nandorsoma closed pull request #7045: NIFI-11288 Add missing dependencies required by AWS AssumeRoleWithWebIdentity method URL: https://github.com/apache/nifi/pull/7045 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] nandorsoma commented on pull request #7045: NIFI-11288 Add missing dependencies required by AWS AssumeRoleWithWebIdentity method
nandorsoma commented on PR #7045: URL: https://github.com/apache/nifi/pull/7045#issuecomment-1611005391 > Sorry for the late reply @nandorsoma. > > > Regarding #5775. We can reference environment variables in properties with expression language, so the PR might work for you without the extra library. For that, we need to set the `ExpressionLanguageScope` of the property to anything other than `NONE`. Preferably to `FLOWFILE_ATTRIBUTES`. Could you test that the PR mentioned solves your issue? > > As our DevOps recycled the test environment, I currently do not have the conditions in which I can reproduce and test this solution. Perhaps I will test this solution in the future when I am in a position to do so. Maybe you can close this PR first for now. Thank you for the reply @zhangzhonglai. Then I'm closing this PR. If you need help in the future feel free to contact us on the mailing list or over slack. Thanks again! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-11760) Upgrade Riemann Client to 0.5.3
[ https://issues.apache.org/jira/browse/NIFI-11760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17738005#comment-17738005 ] ASF subversion and git services commented on NIFI-11760: Commit 506d311f1cc71ef21e51d6886aa7eabc192fb71c in nifi's branch refs/heads/support/nifi-1.x from David Handermann [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=506d311f1c ] NIFI-11760 Upgraded Riemann Client from 0.4.1 to 0.5.3 Signed-off-by: Pierre Villard This closes #7444. > Upgrade Riemann Client to 0.5.3 > --- > > Key: NIFI-11760 > URL: https://issues.apache.org/jira/browse/NIFI-11760 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: David Handermann >Assignee: David Handermann >Priority: Minor > Fix For: 1.latest, 2.latest > > Time Spent: 20m > Remaining Estimate: 0h > > The Riemann Java client library should be upgrade to > [0.5.3|https://github.com/riemann/riemann-java-client/releases/tag/0.5.3] to > incorporate transitive dependency upgrades including Netty and Protobuf. > Version 0.4.2 introduced a change in the Maven coordinates and package name, > but did not change the implementation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11760) Upgrade Riemann Client to 0.5.3
[ https://issues.apache.org/jira/browse/NIFI-11760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-11760: -- Fix Version/s: 2.0.0 1.23.0 (was: 1.latest) (was: 2.latest) Resolution: Fixed Status: Resolved (was: Patch Available) > Upgrade Riemann Client to 0.5.3 > --- > > Key: NIFI-11760 > URL: https://issues.apache.org/jira/browse/NIFI-11760 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: David Handermann >Assignee: David Handermann >Priority: Minor > Fix For: 2.0.0, 1.23.0 > > Time Spent: 20m > Remaining Estimate: 0h > > The Riemann Java client library should be upgrade to > [0.5.3|https://github.com/riemann/riemann-java-client/releases/tag/0.5.3] to > incorporate transitive dependency upgrades including Netty and Protobuf. > Version 0.4.2 introduced a change in the Maven coordinates and package name, > but did not change the implementation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] asfgit closed pull request #7444: NIFI-11760 Upgrade Riemann Client from 0.4.1 to 0.5.3
asfgit closed pull request #7444: NIFI-11760 Upgrade Riemann Client from 0.4.1 to 0.5.3 URL: https://github.com/apache/nifi/pull/7444 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-11760) Upgrade Riemann Client to 0.5.3
[ https://issues.apache.org/jira/browse/NIFI-11760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17738004#comment-17738004 ] ASF subversion and git services commented on NIFI-11760: Commit 12b7dd02431a829ab71566b38cfc22e1675518ce in nifi's branch refs/heads/main from David Handermann [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=12b7dd0243 ] NIFI-11760 Upgraded Riemann Client from 0.4.1 to 0.5.3 Signed-off-by: Pierre Villard This closes #7444. > Upgrade Riemann Client to 0.5.3 > --- > > Key: NIFI-11760 > URL: https://issues.apache.org/jira/browse/NIFI-11760 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: David Handermann >Assignee: David Handermann >Priority: Minor > Fix For: 1.latest, 2.latest > > Time Spent: 10m > Remaining Estimate: 0h > > The Riemann Java client library should be upgrade to > [0.5.3|https://github.com/riemann/riemann-java-client/releases/tag/0.5.3] to > incorporate transitive dependency upgrades including Netty and Protobuf. > Version 0.4.2 introduced a change in the Maven coordinates and package name, > but did not change the implementation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
