Re: [EXTERNAL] Re: ConsumeKafka_2_6 Processor issue

2024-08-19 Thread Joe Witt
by customers to query mongodb was not. Thank you, Joe, for helping me. > Appreciate it. > > > > *From:* Joe Witt > *Sent:* Thursday, August 15, 2024 11:55 AM > *To:* users@nifi.apache.org > *Subject:* Re: [EXTERNAL] Re: ConsumeKafka_2_6 Processor issue > > > > C

Re: Draining queues during NiFi cluster restart in Kubernetes

2024-08-17 Thread Joe Witt
want to deliver > data with as little latency as possible, so if FlowFiles stay queued during > the restart time window, by the time the node is up and running again the > data will be very late when time it finally leaves the NiFi cluster. > > Regards, > Alexis. > > On S

Re: Draining queues during NiFi cluster restart in Kubernetes

2024-08-17 Thread Joe Witt
Alexis If you are simply restarting nodes you dont need to ensure everything is processed first. It will recover. If you want to scale down by removing a node there is an api to invoke offloading so you could scale down to a single node even if needed. Thanks On Sat, Aug 17, 2024 at 2:02 AM Al

Re: [EXTERNAL] Re: ConsumeKafka_2_6 Processor issue

2024-08-15 Thread Joe Witt
c the flow is same in both the dataflows(Same > processors and same connections)? > > *From:* Joe Witt > *Sent:* Thursday, August 15, 2024 11:29 AM > *To:* users@nifi.apache.org > *Subject:* Re: [EXTERNAL] Re: ConsumeKafka_2_6 Processor issue > > > > CAUTION: The e-mail

Re: [EXTERNAL] Re: ConsumeKafka_2_6 Processor issue

2024-08-15 Thread Joe Witt
ly there is data loss between > kafka and nifi but in this case I can clearly see it by querying both the > collections with the same eventid/transactionid > > *From:* Joe Witt > *Sent:* Thursday, August 15, 2024 11:11 AM > *To:* users@nifi.apache.org > *Subject:* [EXTERNAL]

Re: ConsumeKafka_2_6 Processor issue

2024-08-15 Thread Joe Witt
Hello The most likely scenario at play here is that configuration of the flow results in certain messages/events/flowfiles being routed to a failure path or some path that does not end up in Mongo. It is highly unlikely there is loss between Kafka and NiFi and between NiFi and Mongo. The more li

Re: FlattenJSON fails on large json file

2024-06-14 Thread Joe Witt
James You may be able to use alternative JSON components such as those with record readers/writes. You could certainly write a nifi processor in either Java or Python that would do this and be super efficient. The processor you've chosen just isn't very flexible in regards to larger objects and

Re: Nifi on RockyLinux

2024-06-14 Thread Joe Witt
Deepak In Apache we don't have mechanisms to verify/validate that but indications are that it should work just fine. Thanks On Fri, Jun 14, 2024 at 10:47 AM Chirthani, Deepak Reddy < c-deepakreddy.chirth...@charter.com> wrote: > Hi, > > > > Writing this email to get a confirmation for if Apache

Re: Restarting Nifi Cluster Systems to add new user

2024-04-30 Thread Joe Witt
e UI (which will normally be overwritten). I'm > running version 1.23.2. > > Regards, > *Shamsudeen Jameer*Data Operations ManagerPriority > shamsudeen.jam...@prth.como: (516) 345-5015 M: (917) 374-6599‒‒ > > > > On Tue, Apr 30, 2024 at 6:01 PM Joe Witt wrote: >

Re: Restarting Nifi Cluster Systems to add new user

2024-04-30 Thread Joe Witt
Hello Can you share more about which authentication and authorization provider you're using with NiFi today? Also would be good to share which other authentication/authorization providers your organization could leverage as then it is a question of which ones are supported out of the box. Thanks

Re: java.lang.OutOfMemoryError: Java heap space : NIFI 1.23.2

2024-03-19 Thread Joe Witt
Hello The key output is java.lang.OutOfMemoryError: Java heap space Review batch property options to limit response sizes in the database calls. Thanks On Tue, Mar 19, 2024 at 6:15 AM wrote: > Hello > > I got the executeSQL processor does the sql command "select * from > public.table1" > It

Re: Can we access Queued Duration as an attribute?

2024-02-15 Thread Joe Witt
This [1] blog seems amazingly appropriate and wow do we need these/any such fields we intend to truly honor in a prominent place in the docs. Super useful... [1] https://jameswing.net/nifi/nifi-internal-fields.html Thanks On Thu, Feb 15, 2024 at 8:35 AM Mark Payne wrote: > Jim, > > You can ac

Re: ExecuteStreamCommand failing to unzip incoming flowfiles

2024-01-31 Thread Joe Witt
e started, I will do a concise and thorough description in the > ticket. > Sincerely, > Jim > > On Wed, Jan 31, 2024 at 12:12 PM Joe Witt wrote: > >> James, >> >> Makes sense to create a JIRA to improve UnpackContent to extract these >> attributes in the eve

Re: ExecuteStreamCommand failing to unzip incoming flowfiles

2024-01-31 Thread Joe Witt
James, Makes sense to create a JIRA to improve UnpackContent to extract these attributes in the event of a zip file that happens to present them. The concept of lastModifiedDate does appear easily accessed if available in the metadata. Owner/Creator/Creation information looks less standard in th

Re: Finding slow down in processing

2024-01-10 Thread Joe Witt
Aaron, The usual suspects are memory consumption leading to high GC leading to lower performance over time, or back pressure in the flow, etc.. But your description does not really fit either exactly. Does your flow see a mix of large objects and smaller objects? Thanks On Wed, Jan 10, 2024 at

Re: New Apache NiFi Website Design Launched

2024-01-08 Thread Joe Witt
Thanks so much to all involved and David for driving this to completion! It looks great On Mon, Jan 8, 2024 at 11:17 AM David Handermann < exceptionfact...@apache.org> wrote: > Team, > > Thanks to a collaborative effort from several designers and > developers, the Apache NiFi project website has

Re: AuditService NPE After Upgrading to 1.24.0

2023-12-19 Thread Joe Witt
Agreed - thanks for the details. Can you create a JIRA with it while you're at it? Either way it will get attention I'm sure. Thanks On Tue, Dec 19, 2023 at 10:18 AM Shawn Weeks wrote: > If I revert to 1.23.2 the issue goes away. Seems like there might be a bug > in the H2 to Xodus migration

Re: NiFi custom extension and deploy phase on Maven

2023-12-09 Thread Joe Witt
Etienne, Are you using our nifi poms as your parent pom of your extensions? I do wonder if perhaps we should offer one that doesn't tie to all our ASF-isms Thanks On Sat, Dec 9, 2023 at 10:54 AM Etienne Jouvin wrote: > Hello all; > > Since now, I never push my extensions to any Nexus or Artif

Re: Bug : PutGCSObject : Version NIFI 1.23.2

2023-11-08 Thread Joe Witt
oxy configuration service" because we are behind the corporate network. >> >> Since some month ago we have configured "private connection" with google. >> So I have setup "storage api url" with the private url and leave empty >> "proxy configu

Re: Bug : PutGCSObject : Version NIFI 1.23.2

2023-11-08 Thread Joe Witt
; with the private url and leave empty > "proxy configuration service" > > Regards > > > *Envoyé:* mercredi 8 novembre 2023 à 14:31 > *De:* "Joe Witt" > *À:* users@nifi.apache.org > *Objet:* Re: Bug : PutGCSObject : Version NIFI 1.23.2 > Minh > &g

Re: Bug : PutGCSObject : Version NIFI 1.23.2

2023-11-08 Thread Joe Witt
Minh Do you need to override that value? Did Google specify a different URL for you or you have a private connection? If you dont set that value what happens? Thanks On Wed, Nov 8, 2023 at 4:57 AM wrote: > Hello, > > I'm trying to user the overwriting storage url parameter in the nifi > vers

Re: unsubscribe me

2023-11-01 Thread Joe Witt
Hello Please follow the guidance on https://nifi.apache.org/mailing_lists.html. Primarily for this mailing list that means sending an email to users-unsubscr...@nifi.apache.org Thanks On Wed, Nov 1, 2023 at 1:49 PM McIntyre, Christina < c-christina.mcint...@charter.com> wrote: > I no longer wis

Re: Analyze : "Diagnostics on Stop" logs

2023-10-28 Thread Joe Witt
Hello We need to better understand what it means to say the node stopped. Did the nifi process die? The nifi-app.log should almost certainly expose something interesting. Is this on a Linux box, or in a container, etc. How is it run and what precisely stopped? Thanks On Sat, Oct 28, 2023 at 2

Re: How can I View my flowfile records?

2023-09-25 Thread Joe Witt
pected downstream content viewer >> behaviour. >> >> On Mon, 25 Sept 2023, 06:54 Matt Burgess, wrote: >> >>> I added MIME Type properties to ExecuteProcess and ExecuteStream command >>> so you can set it explicitly if you want [1]. They will be in the 1.24

Re: How can I View my flowfile records?

2023-09-24 Thread Joe Witt
> sourcing.sourceMD5 > b48840c161b645a0169e622dcb8f5083 > sourcing.sourceSHA256 > 4847ac157fd30d6f2e53cb3c4e879ae063d498709da2686c6f61ba6019456afa > triage.datatype > excel > uuid > d72ec2e9-cfbd-435e-9954-4f7fae55c550 > > Thanks for any help. Perhaps my data is there but I

Re: How can I View my flowfile records?

2023-09-24 Thread Joe Witt
Jim, If a content type attribute exists and is not a type NiFi understands it will not be able to render it. Can you show what flowfile attributes are present at the point you attempt to view it? Thanks On Sun, Sep 24, 2023 at 3:03 PM James McMahon wrote: > Hello. I have converted incoming Ex

Re: Large json string field errors

2023-09-18 Thread Joe Witt
Greg When you mention a flow file with a string field are you saying the 'content' of the flowfile OR are you saying there is a flowfile attribute? The reference to 'field' is throwing me off a bit I think. You might want to share a picture of the relevant portions of the flow and/or the actual

Re: Large json string field errors

2023-09-18 Thread Joe Witt
Greg Can you share details on how you're seeing this limit? Is there a resulting stack trace or other output for it in the nifi-app.log? What version of NiFi are you on? Thanks On Mon, Sep 18, 2023 at 11:19 AM Gregory M. Foreman < gfore...@spinnerconsulting.com> wrote: > Hello: > > It appears

Re: NiFi OpenTelemetry JavaAgent

2023-09-08 Thread Joe Witt
Eric Any sign anything in the nifi app log? Did it perhaps run out of mem and hang? In general such agents should be fine. Thanks On Fri, Sep 8, 2023 at 6:15 PM Eric Secules wrote: > Hello, > > I have tried adding opentelemetry to NiFi by running the OpenTelemetry > javaagent ( > https://ope

Re: [NIFI] ExecuteScript

2023-09-08 Thread Joe Witt
NiFi 2.0 should get you much further with its Python processor support. Are you able to build main or would you need help with that? We could put a dev build somewhere if needed. Thanks On Fri, Sep 8, 2023 at 4:28 PM Jeremy Dyer wrote: > Rafael is right. This isn’t reasonably possible today. Y

Re: NiFi hanging during large sql query

2023-09-02 Thread Joe Witt
Nice. Gald you found it. On Sat, Sep 2, 2023 at 5:07 AM Mike Thomsen wrote: > It was the PostgreSQL JDBC driver. If you don't paginate the query > aggressively, it will try to load a significant chunk of the table into > memory rather than just pulling chunks, even with fetchSize set low. > > O

Re: Re: Files in /tmp folder

2023-09-01 Thread Joe Witt
Van You might be able to run 'lsof -p ' periodically and figure out who/what is placing that file there. Thanks Joe On Fri, Sep 1, 2023 at 12:53 PM Williams, Van wrote: > Here are the uncommented parts of that file: > > > > > > local-provider > > > org.apache.nifi.controller.sta

Re: Recommended resource limits

2023-08-16 Thread Joe Witt
Eric Try using 2GB for the heap and seeing that helps. I also believe there are specific pod settings youll want to use to avoid it getting nuked by k8s. This blog may give you great things to consider https://home.robusta.dev/blog/kubernetes-memory-limit Thanks On Wed, Aug 16, 2023 at 3:29 PM

Re: Node/processor/queue status in grafana

2023-08-13 Thread Joe Witt
This list is perfectly fine. All of our metrics are available via push, pull, and prometheus so it can def be done. It would be great to see what you end up with. Thanks On Sun, Aug 13, 2023 at 3:56 PM Aaron Rich wrote: > Hi, > > Not sure if this belongs on the dev or user mailing list but f

Re: source build issue - Nifi 1.23.0

2023-08-07 Thread Joe Witt
Greg Yeah seems likely there is a build issue with the RPM profile. I think we just dont have enough people exercising that thing often enough anymore and so when it breaks it is silent. The only thing we meant to have an RPM for as far as I recall was the nifi assembly itself but perhaps we had

Re: Schema Registry

2023-07-19 Thread Joe Witt
And looks like with NiFi 2.0 though we will drop the Hortonworks Schema Registry as shown here [1]. If maintenance becomes more active for it we could revisit as needed. [1] https://issues.apache.org/jira/browse/NIFI-11095 On Wed, Jul 19, 2023 at 12:32 PM Joe Witt wrote: > Hello >

Re: Schema Registry

2023-07-19 Thread Joe Witt
Hello You will want to check with Cloudera for the status of the Schema Registry in question here. From a NiFi point of view we will continue to integrate with schema registries we see in common usage and it should be set up well to plug in new ones and phase older ones as we go. Support for the

Re: UI SocketTimeoutException - heavy IO

2023-07-12 Thread Joe Witt
e.repository.checkpoint.interval=300 secs > nifi.flowfile.repository.always.sync=false > nifi.flowfile.repository.retain.orphaned.flowfiles=true > > -Joe > On 7/12/2023 11:07 AM, Joe Witt wrote: > > Joe > > I dont recall the specific version in which we got it truly sorted but > there was an issue wi

Re: UI SocketTimeoutException - heavy IO

2023-07-12 Thread Joe Witt
Joe I dont recall the specific version in which we got it truly sorted but there was an issue with our default settings for an important content repo property and how we handled mixture of large/small flowfiles written within the same underlying slab/claim in the content repository. Please check

Re: NiFi not rolling logs

2023-07-07 Thread Joe Witt
H. Interesting. Can you capture these bits of fun in a jira? Thanks On Fri, Jul 7, 2023 at 7:17 PM Mike Thomsen wrote: > After doing some research, it appears that is a wonky > setting WRT how well it's honored by logback. I let a GenerateFlowFile > > LogAttribute flow run for a long tim

Fwd: [ANNOUNCE] Apache NiFi 1.22.0 release.

2023-06-11 Thread Joe Witt
-- Forwarded message - From: Joe Witt Date: Sun, Jun 11, 2023 at 9:27 PM Subject: [ANNOUNCE] Apache NiFi 1.22.0 release. To: Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.22.0. Apache NiFi is an easy to use, powerful, and reliable system to

Re: Processor ID/Processor Name as a default attribute

2023-05-24 Thread Joe Witt
Hello Can you describe how you would use this information? These kinds of details and more are present in provenance data now. Thanks On Wed, May 24, 2023 at 7:45 AM Chirthani, Deepak Reddy < c-deepakreddy.chirth...@charter.com> wrote: > Is there any chance where Processor_ID or Processor_Nam

Re: how SCP full directory to remote location

2023-05-11 Thread Joe Witt
Hello NiFi doesn't offer an SCP specific processor. Instead you would use List/FetchSFTP to pull from the target directory and remote source server (and recurse to subdirs as desired) and then use PutSFTP to write to the target directory (including subdirs as desired) of the remote destination se

Re: Need Help in migrating Giant CSV from S3 to SFTP

2023-05-09 Thread Joe Witt
Nilesh, These processors generally are not memory sensitive as they should only ever have small amounts in memory at a time so it is likely this should work well up to 100s of GB objects and so on. We of course dont really test that much but it is technically reasonable and designed as such. So

Re: How to debug why a node isn't rejoining a cluster

2023-05-02 Thread Joe Witt
Mike If nothing is in the logs on the impacted node or anything else in the cluster you will want to grab a thread dump on the affected node and see what is actually happening. There has to be logs though If you have a large back log then you could be waiting for the initial repository healt

Re: PutSFTP TransportException (Timeout expired: 30000 MILLISECONDS) for sending data to Synology NAS - Data Timeout?

2023-04-24 Thread Joe Witt
Hello If going from 8 nodes with many errors to 1 node with few errors then you likely hit max connection limits on the sftp server. You can change that value on fhe sftp server. How many concurrent tasks do you allow the processor. If Y tasks you will want Y times 8 connections allowed. Than

[ANNOUNCE] Apache NiFi 1.21.0 release.

2023-04-07 Thread Joe Witt
Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.21.0. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. Apache NiFi was made for dataflow. It supports highly configurable directed graphs of data routing, transformation, a

Re: Broken pipe: InvokeHTTP

2023-03-14 Thread Joe Witt
Hello I believe this is the remote service killing the socket. Do you have logs for that service to check? Thanks On Tue, Mar 14, 2023 at 7:54 PM Richard Beare wrote: > Hi Everyone, > I have an InvokeHTTP processor experiencing the error below on a small > proportion of flowfiles. The service

Re: Cluster modification

2023-03-14 Thread Joe Witt
Phil Yes it should happen then too. Thanks On Tue, Mar 14, 2023 at 7:05 AM Phillip Lord wrote: > I see... Thanks Pierre. This all makes sense... I'm going to assume the > reconnecting node uses the Cluster Coordinator to retrieve the latest > flow-version. In Mark's video he demonstrated rec

Re: Maximum number of processors?

2023-03-10 Thread Joe Witt
essors that have flowfiles in the input queue and processors that have no > input? > > Thanks, > Eric > > On Thu, Mar 9, 2023 at 10:32 AM Joe Witt wrote: >> >> Eric >> >> There is a practical limit in terms of memory, browser performance, >> etc... Bu

Re: Maximum number of processors?

2023-03-09 Thread Joe Witt
Eric There is a practical limit in terms of memory, browser performance, etc... But there isn't otherwise any real hard limit set. We've seen flows with many 10s of thousands of processors that are part of what can be many dozens or hundreds of process groups. But the challenge that comes up i

Re: Processor with cron scheduling in middle of flow

2023-02-22 Thread Joe Witt
Yep. Though I'd still say we should first look at simply making it targeted that certain processors do not allow cron scheduling such as MergeContent which is not designed to work well unless it is consistently getting a chance to crunch numbers/data. @CronSchedulingDisabled or something... I do

Re: Processor with cron scheduling in middle of flow

2023-02-21 Thread Joe Witt
John MergeContent cannot reliably work well with cron scheduling. That component is designed to get threads consistently so it can perform its bin packing function and time and size based kick out functions. If it ever worked with cron scheduling that was mostly on accident I'd say. Thanks Joe

Re: Web Based Nifi for Python

2023-02-21 Thread Joe Witt
great. Go have fun. Thanks On Tue, Feb 21, 2023 at 7:27 AM Patrick Timmins wrote: > > I guess Joe already responded on this front ... more than a year and a half > ago! > > > > Forwarded Message > From:Joe Witt > Date:Sat, 10 Jul 2021 09:4

[ANNOUNCE] Apache NiFi 1.20.0 release.

2023-02-09 Thread Joe Witt
Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.20.0. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. Apache NiFi was made for dataflow. It supports highly configurable directed graphs of data routing, transformation, a

[ANNOUNCE] Apache NiFi 1.20.0 release.

2023-02-09 Thread Joe Witt
Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.20.0. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. Apache NiFi was made for dataflow. It supports highly configurable directed graphs of data routing, transformation, a

Re: Which repositories would benefit from a higher performance disk?

2023-01-16 Thread Joe Witt
Eric So assuming each ff is 50KB you currently see 1.6MB/sec for 120,000 ff/hr and want to achieve at least 26MB/sec for 2,000,000 ff/hr. These should be quite achievable rates for single nodes with modest resources on a wide range of flow requirements. Can you share a bit more about the cpu, me

Re: CSV with header - but always, even for 0 record flowfiles

2022-12-19 Thread Joe Witt
Josef Please file a JIRA with this information above. Makes sense what you're looking for. Just not sure where this 'concern' would live whether it is in the processors themselves or the controller services for the writers. Thanks On Mon, Dec 19, 2022 at 9:21 AM wrote: > Hi guys > > > > I’ve

Re: Error on nifi start

2022-12-13 Thread Joe Witt
We support 8, 11, and 17 now. As far as NiFi 1.14 which James mentioned he was using we didn't support it then. Support started in NiFi 1.16. Thanks On Tue, Dec 13, 2022 at 8:43 AM James Srinivasan wrote: > I think nifi currently supports java 8 or 11, not 17: > > > https://nifi.apache.org/do

[ANNOUNCE] Apache NiFi 1.19.1 release.

2022-12-07 Thread Joe Witt
Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.19.1. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. Apache NiFi was made for dataflow. It supports highly configurable directed graphs of data routing, transformation, a

Re: Expected mergerecord performance

2022-12-02 Thread Joe Witt
Hello Run schedule should be 0. 50 should be the min number of records 5 seconds is the max bin age it sounds like you want. Start with these changes and let us know what youre seeing. Thanks On Fri, Dec 2, 2022 at 10:12 PM Richard Beare wrote: > Hi, > I'm having a great deal of trouble con

[ANNOUNCE] Apache NiFi 1.19.0 release

2022-11-28 Thread Joe Witt
Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.19.0. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. Apache NiFi was made for dataflow. It supports highly configurable directed graphs of data routing, transformation, a

Re: Exacly once from NiFi to Kafka

2022-11-19 Thread Joe Witt
; In any case, messages have json format (one json per flowfile) but since I > am sending and storing them in kafka in plain text I am using > *no-record-oriented* Kafka publisher. Is PublishKafkaRecord more > reliable? Would it be better to use it? > > Thanks, > > Aian > &g

Re: Nifi with Nginx(ssl) reverse proxy

2022-11-15 Thread Joe Witt
Thanks! On Tue, Nov 15, 2022 at 10:06 AM Ben .T.George wrote: > Hello, > > I have achieved this with nginx - reverse proxy method. > > Regards, > Ben > > On Sun, Nov 13, 2022 at 7:22 PM Ben .T.George > wrote: > >> Hello, >> >> i have a nifi instance running fine on port 8443, i would like to >>

Re: Exacly once from NiFi to Kafka

2022-11-15 Thread Joe Witt
Aian, How can you tell there are duplicates in Kafka and are you certain that no duplicates exist in the source topic? Given NiFi's data provenance capabilities you should be able to pin point a given duplicate and figure out whether it happened at the source, in nifi, or otherwise. Note much ha

Re: Deprecated Processors

2022-11-12 Thread Joe Witt
Please post to a single list. Can you show the log output you see? Thanks On Sat, Nov 12, 2022 at 5:28 PM Mohnish Vaid wrote: > Greetings! > > Not sure which email to use, but I wanted to reach out and ask if it's > possible to find a list of deprecated processors (or future deprecated > ones).

Re: CVE-2022-42889 and impact on Apache Nifi

2022-11-10 Thread Joe Witt
hat has been done! >> >> 1.18.1 could be a good idea I think as you say it would let people more >> comfortable as there is much noise around this CVE and the immediate >> conclusion that a patch is needed while it may be a product is not >> vulnerable to it althou

Re: Empty Queue - but UI shows messages

2022-11-09 Thread Joe Witt
Hello This likely means the processor consuming from this queue has the flowfiles held being processed. MergeContent is a common processor that would do this but others certainly can. What processor do you have there? If you stop the target processor then delete it should always work though you

Re: CVE-2022-42889 and impact on Apache Nifi

2022-10-21 Thread Joe Witt
Tom In the future if you're concerned or have questions about a vulnerability/potential vulnerability please follow the guidance here. https://nifi.apache.org/security.html Here you can see what we've done for this already on main https://issues.apache.org/jira/browse/NIFI-10648 with more info

[ANNOUNCE] Apache NiFi 1.18.0 release

2022-10-06 Thread Joe Witt
Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.18.0. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. Apache NiFi was made for dataflow. It supports highly configurable directed graphs of data routing, transformation, a

Re: Thank You community

2022-10-04 Thread Joe Witt
Carlos Thanks for taking the time to write a nice email like this and glad you found nifi valuable. Best wishes. Joe On Tue, Oct 4, 2022 at 2:39 AM Dobbernack, Harald (Key-Work) wrote: > > | Perhaps I will explain this better in a medium Post. > > oh please do! Am looking forward to reading ab

Re: Merge Content De-fragment Max Bins

2022-09-27 Thread Joe Witt
Cool. Thanks for both On Tue, Sep 27, 2022 at 6:23 PM Eric Secules wrote: > Hi Joe, > > It is done. https://issues.apache.org/jira/browse/NIFI-10553 I have a > unit test that I added on my fork and it is failing as expected. > > -Eric > > On Tue, Sep 27, 2022 at

Re: Merge Content De-fragment Max Bins

2022-09-27 Thread Joe Witt
Eric Yeah I think I agree. Can you file a JIRA? Thanks On Tue, Sep 27, 2022 at 4:55 PM Eric Secules wrote: > Hello, > > I am using MergeContent in defragment mode. I am having a problem when > bins complete too early because they were evicted due to reaching the max > bin count. However, thi

Re: Trigger content eviction manually?

2022-09-13 Thread Joe Witt
). And thanks > Joe (and Pat) for both you technical and emotional support :-) > > Enjoy your day, see you on the list and may the flow be with you. Always. > > Best, Lars > > On 22-09-13 20:17, Joe Witt wrote: > > Hahah Pat! > > > > Lars > > > > O

Re: Trigger content eviction manually?

2022-09-13 Thread Joe Witt
hours in Germany, but you have NiFi brothers and > >> sisters around the world that are counting on you ... please don't > >> let us down. > >> > >> :) <- international smiley/joking symbol > >> > >> > >> On 9/13/2022 10:15 AM, J

Re: Trigger content eviction manually?

2022-09-13 Thread Joe Witt
ntent-repository On Tue, Sep 13, 2022 at 10:15 AM Joe Witt wrote: > > read that again and hopefully it was obvious I was joking. But I am > looking forward to hearing what you learn. > > Thanks > > On Tue, Sep 13, 2022 at 10:10 AM Joe Witt wrote: > > > > Lars &g

Re: Trigger content eviction manually?

2022-09-13 Thread Joe Witt
read that again and hopefully it was obvious I was joking. But I am looking forward to hearing what you learn. Thanks On Tue, Sep 13, 2022 at 10:10 AM Joe Witt wrote: > > Lars > > I need you to drive back to work because now I am very vested in the outcome > :) > >

Re: Trigger content eviction manually?

2022-09-13 Thread Joe Witt
; On 13 September 2022 16:45:30 CEST, Joe Witt wrote: >> >> Lars >> >> You should not have to update to 1.17. While I'm always fond of >> peoople being on the latest the issue i mentioned is fixed in 1.16.3. >> >> HOWEVER, please do confirm your values.

Re: Trigger content eviction manually?

2022-09-13 Thread Joe Witt
ck if the issue persists. Your help is really > appreciated, thanks! > > On 13 September 2022 16:33:53 CEST, Joe Witt wrote: >> >> Lars >> >> The issue that came to mind is >> https://issues.apache.org/jira/browse/NIFI-10023 but that is fixed in >> 1.16.2

Re: Trigger content eviction manually?

2022-09-13 Thread Joe Witt
nifi.content.repository.always.sync=false Thanks On Tue, Sep 13, 2022 at 7:04 AM Lars Winderling wrote: > > I'm using 1.16.3 from upstream (no custom build) on java 11 temurin, debian > 10, virtualized, no docker setup. > > On 13 September 2022 13:37:15 CEST, Joe Witt wrote: >> >>

Re: Trigger content eviction manually?

2022-09-13 Thread Joe Witt
Lars What version are you using? Thanks On Tue, Sep 13, 2022 at 3:11 AM Lars Winderling wrote: > Dear community, > > sometimes our content repository grows out of bounds. Since it has been > separated on disk from the rest of NiFi, we can still use the NiFi UI and > empty the respective queues

Re: MiniFi codebase question

2022-09-07 Thread Joe Witt
Marc Please only post to one list. Moved security and dev to bcc as this is just a user question. Apache NiFi's MiNiFi has two implementations one is Java based and the other is CPP based. Given the version you reference I suspect you're asking about the MiNiFI Java implementation. We used to

Re: Crash on startup due to Output port issue

2022-08-01 Thread Joe Witt
ew?usp=sharing >> [2] https://issues.apache.org/jira/browse/NIFI-10306 >> >> On Mon, Aug 1, 2022 at 12:46 PM BeNJ wrote: >> > >> > Updates: >> > Mark: Deleting the json allows nifi to start up(!!) - note that restarting >> > (with the newly g

[ANNOUNCE] Apache NiFi 1.17.0 release

2022-08-01 Thread Joe Witt
Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.17.0. This is a feature, improvement, bug fix and security release with 310 JIRAs completed. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. Apache NiFi was made for dataf

Re: Crash on startup due to Output port issue

2022-08-01 Thread Joe Witt
zer.java:423) >> at >> org.apache.nifi.groups.StandardProcessGroupSynchronizer.synchronize(StandardProcessGroupSynchronizer.java:373) >> at >> org.apache.nifi.groups.StandardProcessGroupSynchronizer.addProcessGroup(StandardProcessGroupSynchronizer.java:1019) >&

Re: Crash on startup due to Output port issue

2022-07-31 Thread Joe Witt
ng on > Linux)? > > Dave > > > On Sun, Jul 31, 2022, 2:21 PM Benji Benning wrote: > >> Not really around the below error. >> Anything I can move to debug or trace in logback.xml to add some color >> around this? >> >> Thanks, >> Benji >> &g

Re: Crash on startup due to Output port issue

2022-07-31 Thread Joe Witt
my iPhone > > On Jul 31, 2022, at 12:25, Joe Witt wrote: > >  > > Can you share the flow or does it contain sensitve info? > > On Sun, Jul 31, 2022 at 10:01 AM BeNJ wrote: > >> Hi, >> I'm a little at a loss so I figured maybe someone here would have an id

Re: Crash on startup due to Output port issue

2022-07-31 Thread Joe Witt
Can you share the flow or does it contain sensitve info? On Sun, Jul 31, 2022 at 10:01 AM BeNJ wrote: > Hi, > I'm a little at a loss so I figured maybe someone here would have an idea > of how to get out of this situation. > My NiFI is crashing on startup with the following: > --

Re: Issue with SiteToSiteProvenanceReportingTask 1.15.3

2022-07-15 Thread Joe Witt
Hello Is that the totality of what the logs contain? Ideally we're going to need to see the full logs. The other case you link is rooted in having exhausted the open file limits. Your case appears different. Would be good to file a JIRA with the logs, details about the JVM version at play, det

Re: Changes not persisted to flow.xml.gz

2022-06-30 Thread Joe Witt
Jim, If I think back a very very long time I can think of when this might have occurred/been possible but it certainly isn't something I've heard of in a very long time. We'd need to understand the nifi version, the os/version, java version as a starter. But also please look in the nifi-app.log

Re: Encrypted smb v3

2022-06-17 Thread Joe Witt
Robert Would you be comfortable trying to make a pull request for it? Thanks On Fri, Jun 17, 2022 at 3:12 AM Richter, Robert wrote: > Hello, > > > > We’ve tried to put some files on a smb v3 with PutSmbFile processor. This > works fine if the share is not encrypted and throws access denied fai

Re: Issue with registry and Port error

2022-06-16 Thread Joe Witt
Dave The issue you flag might be related. In the 1.15/1.16 lines we've improved a good bit of funky lifecycle state handling. If you're able I recommend trying this on the 1.16.3 release for both NiFi and Registry. If that still occurs we would like to see the nifi-app.log entries of nifi. Tha

[ANNOUNCE] Apache NiFi 1.16.3 release

2022-06-14 Thread Joe Witt
Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.16.3. This is a pure bug fix and security focused release with 13 JIRAs completed. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. Apache NiFi was made for dataflow. It s

[ANNOUNCE] Apache NiFi 1.16.2 release

2022-05-25 Thread Joe Witt
Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.16.2. This is an improvement, bug fix and security focused release with more than 30 JIRAs completed. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. Apache NiFi was made

Re: Co-habitating log files and flow_repo - bad idea?

2022-05-17 Thread Joe Witt
Hello Performance wise not as concerning really unless this is a truly performance intense high volume of flowfile situation. But must ensure logs only take up limited space and plenty remains for the flowfile repo particularly when there are backlogs/etc. Joe On Tue, May 17, 2022 at 12:21 PM

Re: nifi-content-viewer with mime.type of image/webp

2022-05-12 Thread Joe Witt
Ryan. Nope - just means we dont have a viewer for that content type. Would need to be added. Thanks On Thu, May 12, 2022 at 9:19 AM Ryan Hendrickson < ryan.andrew.hendrick...@gmail.com> wrote: > Hi all, > I'm curious if anyone has used the nifi-content-viewer to view images with > a mime.type

Re: Rest API to query for nifi.properties settings

2022-05-11 Thread Joe Witt
James, There is no API to do so that I'm aware of - we'd intentionally not want that available. The nifi.properties file contains inherently node centric values and some sensitive values. Joe On Wed, May 11, 2022 at 10:03 AM James McMahon wrote: > Are there any nifi REST API calls that return

[ANNOUNCE] Apache NiFi 1.16.1 release

2022-04-29 Thread Joe Witt
Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.16.1. This is an improvement, bug fix and security focused release with more than 80 JIRAs completed. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. Apache NiFi was made

Re: NiFi web UI not responsive under load

2022-04-19 Thread Joe Witt
Eric When the UI isn't responsive it would be great to have a snapshot of: - CPU usage at that time - GC behavior/logging at/around that time. - IO Utilization around that time - NiFi Thread dump precisely during it and ideally also one after it responds again NiFi Restarting itself is very inter

Re: Possible to convert .msg to .eml?

2022-04-07 Thread Joe Witt
James, >From a tool/purpose point of view yes NiFi is an ideal tool for the job assuming you're talking about needing to do this continuously in an automated flow fashion. But, we of course need some transform written for this or you could certainly build your own custom one as a full on processo

  1   2   3   4   5   6   7   8   9   10   >