Re: Limiting the conntrackd logs in virtual router
Hello Andrija Sorry for replying to your email this late. Just saw it. Basically there was an issue with logrotate service as it was running just once a day instead of hourly. I have created a pr for that in 4.15 On Tue, Jun 15, 2021 at 12:41 AM Andrija Panic wrote: > Rakesh, > > can you confirm if you are seeing this for ALL VPCs with S-2-S enabled > can you describe/quantify the MB/s log increase over time - does it have > anything to do with the traffic throughput through the tunnel, or is it > just happening even with no traffic (i.e. you can setup S-2-S tunnel > between 2 VPCs (one of them marked as "passive") - I would like to see/test > this in 4.15.1 (if you confirm that the amount of traffic does NOT > influence the size of the logs) > > Please respond ASAP - as we need to craft 4.15.1 RC2 - if there is a bug, > we need to fix it (and I will help testing). > > Best, > > > On Wed, 9 Jun 2021 at 13:31, Rakesh Venkatesh > wrote: > > > If I restart with cleanup then the modified config will be deleted and > the > > log starts populating very fast again. > > > > On Wed, Jun 9, 2021 at 1:19 PM Andrija Panic > > wrote: > > > > > Did you try restarting that VPC with cleanup - so brand new VR is > > created - > > > to see if the issue is still observed. > > > > > > Can you test the same now with 4.15.1 RC1 (voting is happening, please > > feel > > > free to test and participate!) > > > > > > Best, > > > > > > On Wed, 9 Jun 2021 at 10:57, Rakesh Venkatesh < > www.rakeshv@gmail.com > > > > > > wrote: > > > > > > > I have 4.14 version. The issue exists with only 1 customer. > > > > The VR's belonging to other customers are fine but for one customer > who > > > has > > > > enabled site 2 site vpn, its creating lot of logs. > > > > The config I mentioned above is a temporary fix as it will be deleted > > > once > > > > the network is restarted with cleanup. > > > > > > > > On Wed, Jun 9, 2021 at 10:52 AM Andrija Panic < > andrija.pa...@gmail.com > > > > > > > wrote: > > > > > > > > > Can you advise which version and have you tried isolating the > > problem - > > > > > this is first time I hear for such an issue (and I have customers > > with > > > 3y > > > > > VR uptime with no issues) > > > > > > > > > > Perhaps something is broken inside that VR, thus the excessive > > logging, > > > > > etc? > > > > > > > > > > Best, > > > > > > > > > > On Wed, 9 Jun 2021 at 10:07, Rakesh Venkatesh < > > > www.rakeshv@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > Hello > > > > > > > > > > > > I have an issue in the virtual router where > > > conntrackd(conntrack-tools) > > > > > is > > > > > > populating /var/log/daemon.log a lot and because of that VR is > > > running > > > > > out > > > > > > of disk space. Do you guys know how to prevent conntrackd from > > > logging > > > > > too > > > > > > much? I manually changed the logrotate.conf to rotate every hour > > and > > > > > retain > > > > > > only few copies but thats a quick fix. Is there any better fix to > > > avoid > > > > > > this issue? In our setup the VR has 2GB disk and even that got > > filled > > > > up > > > > > > pretty quickly. > > > > > > > > > > > > The logrotate config for daemon.log is present in > > > > > /etc/logrotate.d/rsyslog > > > > > > and I added > > > > > > > > > > > > /var/log/daemon.log > > > > > > { > > > > > > hourly > > > > > > rotate 1 > > > > > > missingok > > > > > > notifempty > > > > > > compress > > > > > > copytruncate > > > > > > } > > > > > > > > > > > > > > > > > > The config present in /etc/logrotate.d/conntrackd is pointing > > > > > > to /var/log/conntrackd-stats.log and so changing that config wont > > > help. > > > > > > > > > > > > -- > > > > > > Thanks and regards > > > > > > Rakesh > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Andrija Panić > > > > > > > > > > > > > > > > > -- > > > > Thanks and regards > > > > Rakesh venkatesh > > > > > > > > > > > > > -- > > > > > > Andrija Panić > > > > > > > > > -- > > Thanks and regards > > Rakesh venkatesh > > > > > -- > > Andrija Panić > -- Thanks and regards Rakesh venkatesh
Re: Limiting the conntrackd logs in virtual router
Rakesh, can you confirm if you are seeing this for ALL VPCs with S-2-S enabled can you describe/quantify the MB/s log increase over time - does it have anything to do with the traffic throughput through the tunnel, or is it just happening even with no traffic (i.e. you can setup S-2-S tunnel between 2 VPCs (one of them marked as "passive") - I would like to see/test this in 4.15.1 (if you confirm that the amount of traffic does NOT influence the size of the logs) Please respond ASAP - as we need to craft 4.15.1 RC2 - if there is a bug, we need to fix it (and I will help testing). Best, On Wed, 9 Jun 2021 at 13:31, Rakesh Venkatesh wrote: > If I restart with cleanup then the modified config will be deleted and the > log starts populating very fast again. > > On Wed, Jun 9, 2021 at 1:19 PM Andrija Panic > wrote: > > > Did you try restarting that VPC with cleanup - so brand new VR is > created - > > to see if the issue is still observed. > > > > Can you test the same now with 4.15.1 RC1 (voting is happening, please > feel > > free to test and participate!) > > > > Best, > > > > On Wed, 9 Jun 2021 at 10:57, Rakesh Venkatesh > > > wrote: > > > > > I have 4.14 version. The issue exists with only 1 customer. > > > The VR's belonging to other customers are fine but for one customer who > > has > > > enabled site 2 site vpn, its creating lot of logs. > > > The config I mentioned above is a temporary fix as it will be deleted > > once > > > the network is restarted with cleanup. > > > > > > On Wed, Jun 9, 2021 at 10:52 AM Andrija Panic > > > > wrote: > > > > > > > Can you advise which version and have you tried isolating the > problem - > > > > this is first time I hear for such an issue (and I have customers > with > > 3y > > > > VR uptime with no issues) > > > > > > > > Perhaps something is broken inside that VR, thus the excessive > logging, > > > > etc? > > > > > > > > Best, > > > > > > > > On Wed, 9 Jun 2021 at 10:07, Rakesh Venkatesh < > > www.rakeshv@gmail.com > > > > > > > > wrote: > > > > > > > > > Hello > > > > > > > > > > I have an issue in the virtual router where > > conntrackd(conntrack-tools) > > > > is > > > > > populating /var/log/daemon.log a lot and because of that VR is > > running > > > > out > > > > > of disk space. Do you guys know how to prevent conntrackd from > > logging > > > > too > > > > > much? I manually changed the logrotate.conf to rotate every hour > and > > > > retain > > > > > only few copies but thats a quick fix. Is there any better fix to > > avoid > > > > > this issue? In our setup the VR has 2GB disk and even that got > filled > > > up > > > > > pretty quickly. > > > > > > > > > > The logrotate config for daemon.log is present in > > > > /etc/logrotate.d/rsyslog > > > > > and I added > > > > > > > > > > /var/log/daemon.log > > > > > { > > > > > hourly > > > > > rotate 1 > > > > > missingok > > > > > notifempty > > > > > compress > > > > > copytruncate > > > > > } > > > > > > > > > > > > > > > The config present in /etc/logrotate.d/conntrackd is pointing > > > > > to /var/log/conntrackd-stats.log and so changing that config wont > > help. > > > > > > > > > > -- > > > > > Thanks and regards > > > > > Rakesh > > > > > > > > > > > > > > > > > -- > > > > > > > > Andrija Panić > > > > > > > > > > > > > -- > > > Thanks and regards > > > Rakesh venkatesh > > > > > > > > > -- > > > > Andrija Panić > > > > > -- > Thanks and regards > Rakesh venkatesh > -- Andrija Panić
Re: Limiting the conntrackd logs in virtual router
If I restart with cleanup then the modified config will be deleted and the log starts populating very fast again. On Wed, Jun 9, 2021 at 1:19 PM Andrija Panic wrote: > Did you try restarting that VPC with cleanup - so brand new VR is created - > to see if the issue is still observed. > > Can you test the same now with 4.15.1 RC1 (voting is happening, please feel > free to test and participate!) > > Best, > > On Wed, 9 Jun 2021 at 10:57, Rakesh Venkatesh > wrote: > > > I have 4.14 version. The issue exists with only 1 customer. > > The VR's belonging to other customers are fine but for one customer who > has > > enabled site 2 site vpn, its creating lot of logs. > > The config I mentioned above is a temporary fix as it will be deleted > once > > the network is restarted with cleanup. > > > > On Wed, Jun 9, 2021 at 10:52 AM Andrija Panic > > wrote: > > > > > Can you advise which version and have you tried isolating the problem - > > > this is first time I hear for such an issue (and I have customers with > 3y > > > VR uptime with no issues) > > > > > > Perhaps something is broken inside that VR, thus the excessive logging, > > > etc? > > > > > > Best, > > > > > > On Wed, 9 Jun 2021 at 10:07, Rakesh Venkatesh < > www.rakeshv@gmail.com > > > > > > wrote: > > > > > > > Hello > > > > > > > > I have an issue in the virtual router where > conntrackd(conntrack-tools) > > > is > > > > populating /var/log/daemon.log a lot and because of that VR is > running > > > out > > > > of disk space. Do you guys know how to prevent conntrackd from > logging > > > too > > > > much? I manually changed the logrotate.conf to rotate every hour and > > > retain > > > > only few copies but thats a quick fix. Is there any better fix to > avoid > > > > this issue? In our setup the VR has 2GB disk and even that got filled > > up > > > > pretty quickly. > > > > > > > > The logrotate config for daemon.log is present in > > > /etc/logrotate.d/rsyslog > > > > and I added > > > > > > > > /var/log/daemon.log > > > > { > > > > hourly > > > > rotate 1 > > > > missingok > > > > notifempty > > > > compress > > > > copytruncate > > > > } > > > > > > > > > > > > The config present in /etc/logrotate.d/conntrackd is pointing > > > > to /var/log/conntrackd-stats.log and so changing that config wont > help. > > > > > > > > -- > > > > Thanks and regards > > > > Rakesh > > > > > > > > > > > > > -- > > > > > > Andrija Panić > > > > > > > > > -- > > Thanks and regards > > Rakesh venkatesh > > > > > -- > > Andrija Panić > -- Thanks and regards Rakesh venkatesh
Re: Limiting the conntrackd logs in virtual router
Did you try restarting that VPC with cleanup - so brand new VR is created - to see if the issue is still observed. Can you test the same now with 4.15.1 RC1 (voting is happening, please feel free to test and participate!) Best, On Wed, 9 Jun 2021 at 10:57, Rakesh Venkatesh wrote: > I have 4.14 version. The issue exists with only 1 customer. > The VR's belonging to other customers are fine but for one customer who has > enabled site 2 site vpn, its creating lot of logs. > The config I mentioned above is a temporary fix as it will be deleted once > the network is restarted with cleanup. > > On Wed, Jun 9, 2021 at 10:52 AM Andrija Panic > wrote: > > > Can you advise which version and have you tried isolating the problem - > > this is first time I hear for such an issue (and I have customers with 3y > > VR uptime with no issues) > > > > Perhaps something is broken inside that VR, thus the excessive logging, > > etc? > > > > Best, > > > > On Wed, 9 Jun 2021 at 10:07, Rakesh Venkatesh > > > wrote: > > > > > Hello > > > > > > I have an issue in the virtual router where conntrackd(conntrack-tools) > > is > > > populating /var/log/daemon.log a lot and because of that VR is running > > out > > > of disk space. Do you guys know how to prevent conntrackd from logging > > too > > > much? I manually changed the logrotate.conf to rotate every hour and > > retain > > > only few copies but thats a quick fix. Is there any better fix to avoid > > > this issue? In our setup the VR has 2GB disk and even that got filled > up > > > pretty quickly. > > > > > > The logrotate config for daemon.log is present in > > /etc/logrotate.d/rsyslog > > > and I added > > > > > > /var/log/daemon.log > > > { > > > hourly > > > rotate 1 > > > missingok > > > notifempty > > > compress > > > copytruncate > > > } > > > > > > > > > The config present in /etc/logrotate.d/conntrackd is pointing > > > to /var/log/conntrackd-stats.log and so changing that config wont help. > > > > > > -- > > > Thanks and regards > > > Rakesh > > > > > > > > > -- > > > > Andrija Panić > > > > > -- > Thanks and regards > Rakesh venkatesh > -- Andrija Panić
Re: Limiting the conntrackd logs in virtual router
I have 4.14 version. The issue exists with only 1 customer. The VR's belonging to other customers are fine but for one customer who has enabled site 2 site vpn, its creating lot of logs. The config I mentioned above is a temporary fix as it will be deleted once the network is restarted with cleanup. On Wed, Jun 9, 2021 at 10:52 AM Andrija Panic wrote: > Can you advise which version and have you tried isolating the problem - > this is first time I hear for such an issue (and I have customers with 3y > VR uptime with no issues) > > Perhaps something is broken inside that VR, thus the excessive logging, > etc? > > Best, > > On Wed, 9 Jun 2021 at 10:07, Rakesh Venkatesh > wrote: > > > Hello > > > > I have an issue in the virtual router where conntrackd(conntrack-tools) > is > > populating /var/log/daemon.log a lot and because of that VR is running > out > > of disk space. Do you guys know how to prevent conntrackd from logging > too > > much? I manually changed the logrotate.conf to rotate every hour and > retain > > only few copies but thats a quick fix. Is there any better fix to avoid > > this issue? In our setup the VR has 2GB disk and even that got filled up > > pretty quickly. > > > > The logrotate config for daemon.log is present in > /etc/logrotate.d/rsyslog > > and I added > > > > /var/log/daemon.log > > { > > hourly > > rotate 1 > > missingok > > notifempty > > compress > > copytruncate > > } > > > > > > The config present in /etc/logrotate.d/conntrackd is pointing > > to /var/log/conntrackd-stats.log and so changing that config wont help. > > > > -- > > Thanks and regards > > Rakesh > > > > > -- > > Andrija Panić > -- Thanks and regards Rakesh venkatesh
Re: Limiting the conntrackd logs in virtual router
Can you advise which version and have you tried isolating the problem - this is first time I hear for such an issue (and I have customers with 3y VR uptime with no issues) Perhaps something is broken inside that VR, thus the excessive logging, etc? Best, On Wed, 9 Jun 2021 at 10:07, Rakesh Venkatesh wrote: > Hello > > I have an issue in the virtual router where conntrackd(conntrack-tools) is > populating /var/log/daemon.log a lot and because of that VR is running out > of disk space. Do you guys know how to prevent conntrackd from logging too > much? I manually changed the logrotate.conf to rotate every hour and retain > only few copies but thats a quick fix. Is there any better fix to avoid > this issue? In our setup the VR has 2GB disk and even that got filled up > pretty quickly. > > The logrotate config for daemon.log is present in /etc/logrotate.d/rsyslog > and I added > > /var/log/daemon.log > { > hourly > rotate 1 > missingok > notifempty > compress > copytruncate > } > > > The config present in /etc/logrotate.d/conntrackd is pointing > to /var/log/conntrackd-stats.log and so changing that config wont help. > > -- > Thanks and regards > Rakesh > -- Andrija Panić
Limiting the conntrackd logs in virtual router
Hello I have an issue in the virtual router where conntrackd(conntrack-tools) is populating /var/log/daemon.log a lot and because of that VR is running out of disk space. Do you guys know how to prevent conntrackd from logging too much? I manually changed the logrotate.conf to rotate every hour and retain only few copies but thats a quick fix. Is there any better fix to avoid this issue? In our setup the VR has 2GB disk and even that got filled up pretty quickly. The logrotate config for daemon.log is present in /etc/logrotate.d/rsyslog and I added /var/log/daemon.log { hourly rotate 1 missingok notifempty compress copytruncate } The config present in /etc/logrotate.d/conntrackd is pointing to /var/log/conntrackd-stats.log and so changing that config wont help. -- Thanks and regards Rakesh