(Removed dev) Having done variations of this workflow many times, my recommendation would be to take a very hard look at how your DBAs and devs have set up the databases and the queries they gave you for extracting the data ad inserting the data. Most of the devils in ETL/ELT will be there, not in your orchestration software (be it NiFi, StreamSet, Airflow, etc.) SQL databases can also present a lot of subtle and very nasty gotchas that can cripple high performance ingestion such as index maintenance behavior and referential integrity checks.
Also, if you are not using it already the NiFi record api for this sort of work, it would be worth your time to read the docs and apply that flow. It generally makes a very noticeable performance difference between NiFi and its data sources as it allows NiFi to efficiently batch process streams of data rather than operating on one record at a time (which by itself can be a performance killer with this sort of flow). On Fri, Feb 28, 2025 at 7:04 AM Siddhanta Singh <siddhanta.si...@tothenew.com.invalid> wrote: > Hi NiFi Support Team, > > We are experiencing significant slowness in NiFi, even though it is > generating FlowFiles. Our data pipeline involves pulling data from > *MongoDB* > and dumping it into *SQL Server*, but the process is running much slower > than expected. > *Issue Details:* > > - NiFi is generating FlowFiles very slowly, but data transfer speed is > unusually slow. > - The slowness persists despite sufficient system resources and > increased instance sizes. > - We have already optimized the server capacity, but the issue remains. > - No major errors are observed in the logs, but the performance > degradation is impacting our workflow. > > *Request for Assistance:* > > - Could you help analyze potential causes for this slowness? > - Are there any specific configurations or optimizations we should > check? > - Any guidance on debugging and improving performance would be > appreciated. > > Please let us know if you require additional details or logs. > > -- > > Warm Regards, > > *Siddhanta Singh* > > *Software Engineer*| <https://www.linkedin.com/company/tothenew> > > <http://www.tothenew.com> >
