Re: NiFi 1.11 - High repository storage usage
Hi Wesley & Harald, Thanks for the quick replies! @Wesley C. Dias de Oliveira<mailto:wcdolive...@gmail.com> I have recently increased these settings due to other reasons, so that might be the primary cause. @Harald You are right, they are on the same drive. I assumed that usage is relative to the nifi usage/settings not the OS. I will now change so each repository gets its own partition. Thanks! Valentina From: Dobbernack, Harald (Key-Work) Sent: Thursday, 25 June 2020 15:08 To: users@nifi.apache.org Subject: AW: NiFi 1.11 - High repository storage usage * When I check the size of the respective folders on the disk they don't even add up to 12,24 GB shown on the screenshot. I’m guessing you have other stuff on the drive or partition as well and not only the repositories. Von: Dobbernack, Harald (Key-Work) Gesendet: Donnerstag, 25. Juni 2020 14:58 An: users@nifi.apache.org Betreff: AW: NiFi 1.11 - High repository storage usage Hi, presumably all three respositories are on the same partition or drive? I believe the screenshot view you posted shows what the OS reports as the space usage of the whole partition/drive on which the repositories are sitting. Best practice would be to place each repository on its own partition. Greetings Harald Von: Valentina Ivanova mailto:valentina.ivan...@ri.se>> Gesendet: Donnerstag, 25. Juni 2020 14:50 An: users@nifi.apache.org<mailto:users@nifi.apache.org> Betreff: NiFi 1.11 - High repository storage usage Hello! I see (screenshot attached) quite high (86%) storage usage for all three Flow File, Content & Provenance Repositories in the System Diagnostics. When I check the size of the respective folders on the disk they don't even add up to 12,24 GB shown on the screenshot. I also have the following settings for the content repository in nifi.properties: nifi.content.repository.archive.max.retention.period=12 hours nifi.content.repository.archive.max.usage.percentage=50% nifi.content.repository.archive.enabled=true nifi.content.repository.always.sync=false So I am wondering where these numbers are coming from and how to reduce the storage utilization (I assume such high utilization will impact the performance)? Why is the content repository usage 86% even though the max.usage.percentage is set at 50%. Many thanks in advance & all the best Valentina Harald Dobbernack Key-Work Consulting GmbH | Kriegsstr. 100 | 76133 | Karlsruhe | Germany | www.key-work.de<https://www.key-work.de> | Datenschutz<https://www.key-work.de/de/footer/datenschutz.html> Fon: +49-721-78203-264 | E-Mail: harald.dobbern...@key-work.de | Fax: +49-721-78203-10 Key-Work Consulting GmbH, Karlsruhe, HRB 108695, HRG Mannheim Geschäftsführer: Andreas Stappert, Tobin Wotring
AW: NiFi 1.11 - High repository storage usage
* When I check the size of the respective folders on the disk they don't even add up to 12,24 GB shown on the screenshot. I'm guessing you have other stuff on the drive or partition as well and not only the repositories. Von: Dobbernack, Harald (Key-Work) Gesendet: Donnerstag, 25. Juni 2020 14:58 An: users@nifi.apache.org Betreff: AW: NiFi 1.11 - High repository storage usage Hi, presumably all three respositories are on the same partition or drive? I believe the screenshot view you posted shows what the OS reports as the space usage of the whole partition/drive on which the repositories are sitting. Best practice would be to place each repository on its own partition. Greetings Harald Von: Valentina Ivanova mailto:valentina.ivan...@ri.se>> Gesendet: Donnerstag, 25. Juni 2020 14:50 An: users@nifi.apache.org<mailto:users@nifi.apache.org> Betreff: NiFi 1.11 - High repository storage usage Hello! I see (screenshot attached) quite high (86%) storage usage for all three Flow File, Content & Provenance Repositories in the System Diagnostics. When I check the size of the respective folders on the disk they don't even add up to 12,24 GB shown on the screenshot. I also have the following settings for the content repository in nifi.properties: nifi.content.repository.archive.max.retention.period=12 hours nifi.content.repository.archive.max.usage.percentage=50% nifi.content.repository.archive.enabled=true nifi.content.repository.always.sync=false So I am wondering where these numbers are coming from and how to reduce the storage utilization (I assume such high utilization will impact the performance)? Why is the content repository usage 86% even though the max.usage.percentage is set at 50%. Many thanks in advance & all the best Valentina Harald Dobbernack Key-Work Consulting GmbH | Kriegsstr. 100 | 76133 | Karlsruhe | Germany | www.key-work.de<https://www.key-work.de> | Datenschutz<https://www.key-work.de/de/footer/datenschutz.html> Fon: +49-721-78203-264 | E-Mail: harald.dobbern...@key-work.de | Fax: +49-721-78203-10 Key-Work Consulting GmbH, Karlsruhe, HRB 108695, HRG Mannheim Gesch?ftsf?hrer: Andreas Stappert, Tobin Wotring
AW: NiFi 1.11 - High repository storage usage
Hi, presumably all three respositories are on the same partition or drive? I believe the screenshot view you posted shows what the OS reports as the space usage of the whole partition/drive on which the repositories are sitting. Best practice would be to place each repository on its own partition. Greetings Harald Von: Valentina Ivanova Gesendet: Donnerstag, 25. Juni 2020 14:50 An: users@nifi.apache.org Betreff: NiFi 1.11 - High repository storage usage Hello! I see (screenshot attached) quite high (86%) storage usage for all three Flow File, Content & Provenance Repositories in the System Diagnostics. When I check the size of the respective folders on the disk they don't even add up to 12,24 GB shown on the screenshot. I also have the following settings for the content repository in nifi.properties: nifi.content.repository.archive.max.retention.period=12 hours nifi.content.repository.archive.max.usage.percentage=50% nifi.content.repository.archive.enabled=true nifi.content.repository.always.sync=false So I am wondering where these numbers are coming from and how to reduce the storage utilization (I assume such high utilization will impact the performance)? Why is the content repository usage 86% even though the max.usage.percentage is set at 50%. Many thanks in advance & all the best Valentina Harald Dobbernack Key-Work Consulting GmbH | Kriegsstr. 100 | 76133 | Karlsruhe | Germany | www.key-work.de<https://www.key-work.de> | Datenschutz<https://www.key-work.de/de/footer/datenschutz.html> Fon: +49-721-78203-264 | E-Mail: harald.dobbern...@key-work.de | Fax: +49-721-78203-10 Key-Work Consulting GmbH, Karlsruhe, HRB 108695, HRG Mannheim Gesch?ftsf?hrer: Andreas Stappert, Tobin Wotring
Re: NiFi 1.11 - High repository storage usage
Hi, Valentina. I've experienced the same issue on an old installation. In my case, it was related to memory usage. The system begins to dump things to the disk when there's no available memory. Have you checked the memory( params? # JVM memory settings java.arg.2=-Xms512m java.arg.3=-Xmx512m Em qui., 25 de jun. de 2020 às 09:50, Valentina Ivanova < valentina.ivan...@ri.se> escreveu: > Hello! > > I see (screenshot attached) quite high (86%) storage usage for all three > Flow File, Content & Provenance Repositories in the System Diagnostics. > When I check the size of the respective folders on the disk they don't even > add up to 12,24 GB shown on the screenshot. > I also have the following settings for the content repository in > nifi.properties: > nifi.content.repository.archive.max.retention.period=12 hours > nifi.content.repository.archive.max.usage.percentage=50% > nifi.content.repository.archive.enabled=true > nifi.content.repository.always.sync=false > > > So I am wondering where these numbers are coming from and how to reduce > the storage utilization (I assume such high utilization will impact the > performance)? Why is the content repository usage 86% even though the > max.usage.percentage is set at 50%. > > Many thanks in advance & all the best > > Valentina > > > > -- Grato, Wesley C. Dias de Oliveira. Linux User nº 576838.
NiFi 1.11 - High repository storage usage
Hello! I see (screenshot attached) quite high (86%) storage usage for all three Flow File, Content & Provenance Repositories in the System Diagnostics. When I check the size of the respective folders on the disk they don't even add up to 12,24 GB shown on the screenshot. I also have the following settings for the content repository in nifi.properties: nifi.content.repository.archive.max.retention.period=12 hours nifi.content.repository.archive.max.usage.percentage=50% nifi.content.repository.archive.enabled=true nifi.content.repository.always.sync=false So I am wondering where these numbers are coming from and how to reduce the storage utilization (I assume such high utilization will impact the performance)? Why is the content repository usage 86% even though the max.usage.percentage is set at 50%. Many thanks in advance & all the best Valentina
