vdombrovski opened a new issue, #7576:
URL: https://github.com/apache/cloudstack/issues/7576
<!--
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release and main branch are affected too.
Always add information AFTER of these HTML comments, but no need to delete
the comments.
-->
##### ISSUE TYPE
<!-- Pick one below and delete the rest -->
* Bug Report
##### COMPONENT NAME
<!--
Categorize the issue, e.g. API, VR, VPN, UI, etc.
-->
~~~
VR
~~~
##### CLOUDSTACK VERSION
<!--
New line separated list of affected versions, commit ID for issues on main
branch.
-->
~~~
4.17.2
~~~
##### CONFIGURATION
<!--
Information about the configuration if relevant, e.g. basic network,
advanced networking, etc. N/A otherwise
-->
Advanced networking, KVM based, VR using the official 4.17 System VM image
(Debian GNU/Linux 5.0 (64-bit))
##### OS / ENVIRONMENT
<!--
Information about the environment if relevant, N/A otherwise
-->
N/A
##### SUMMARY
<!-- Explain the problem/feature briefly -->
Haproxy log files in virtual router are not released by rsyslogd, causing a
full in VR
```
root@r-5033-VM:~# df -h
[...]
/dev/vda6 4.2G 4.2G 0 100% /
lsof | grep deleted
rsyslogd 342 root 16w REG 254,6
3142168576 128005 /var/log/haproxy.log.1 (deleted)
rsyslogd 342 346 in:imuxso root 16w REG 254,6
3142168576 128005 /var/log/haproxy.log.1 (deleted)
rsyslogd 342 347 in:imklog root 16w REG 254,6
3142168576 128005 /var/log/haproxy.log.1 (deleted)
rsyslogd 342 348 imudp(w0) root 16w REG 254,6
3142168576 128005 /var/log/haproxy.log.1 (deleted)
rsyslogd 342 349 rs:main root 16w REG 254,6
3142168576 128005 /var/log/haproxy.log.1 (deleted)
/etc/init.d/rsyslog restart
df -h
[...]
/dev/vda6 4.2G 1.3G 3.0G 30% /
```
##### STEPS TO REPRODUCE
<!--
For bugs, show exactly how to reproduce the problem, using a minimal
test-case. Use Screenshots if accurate.
For new features, show how the feature would be used.
-->
<!-- Paste example playbooks or commands between quotes below -->
~~~
- Deploy a VR
- Add a load-balancing rule to a public IP handled by this VR
- Wait :)
~~~
<!-- You can also paste gist.github.com links for larger files -->
##### EXPECTED RESULTS
<!-- What did you expect to happen when running the steps above? -->
~~~
Old log files get released
~~~
##### ACTUAL RESULTS
<!-- What actually happened? -->
<!-- Paste verbatim command output between quotes below -->
~~~
lsof | grep deleted
rsyslogd 342 root 16w REG 254,6
3142168576 128005 /var/log/haproxy.log.1 (deleted)
rsyslogd 342 346 in:imuxso root 16w REG 254,6
3142168576 128005 /var/log/haproxy.log.1 (deleted)
rsyslogd 342 347 in:imklog root 16w REG 254,6
3142168576 128005 /var/log/haproxy.log.1 (deleted)
rsyslogd 342 348 imudp(w0) root 16w REG 254,6
3142168576 128005 /var/log/haproxy.log.1 (deleted)
rsyslogd 342 349 rs:main root 16w REG 254,6
3142168576 128005 /var/log/haproxy.log.1 (deleted)
~~~
Deleted file is held, causing a full on the VR, causing all the deployments
inside the network to fail with
```
WARN [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-2:null)
(logid:9b4056c5) Failed to create file
/var/cache/cloud/monitor_service.json.10cd2d0e-fd3c-
4bc8-9868-7041cdaad464 in VR 169.254.154.26
java.io.IOException: Error during SCP transfer.
at com.trilead.ssh2.SCPClient.put(SCPClient.java:518)
at com.trilead.ssh2.SCPClient.put(SCPClient.java:471)
at com.cloud.utils.ssh.SshHelper.scpTo(SshHelper.java:191)
at com.cloud.utils.ssh.SshHelper.scpTo(SshHelper.java:64)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createFileInVR(LibvirtComputingResource.java:483)
at
com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.applyConfigToVR(VirtualRoutingResource.java:226)
at
com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.applyConfig(VirtualRoutingResource.java:244)
at
com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeRequest(VirtualRoutingResource.java:150)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:35)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:29)
at
com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at
com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1831)
at com.cloud.agent.Agent.processRequest(Agent.java:660)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1080)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.IOException: Remote scp terminated with error (scp:
/var/cache/cloud//monitor_service.json.10cd2d0e-fd3c-4bc8-9868-7041cdaad464: No
space left on device).
at com.trilead.ssh2.SCPClient.readResponse(SCPClient.java:51)
at com.trilead.ssh2.SCPClient.sendBytes(SCPClient.java:142)
at com.trilead.ssh2.SCPClient.put(SCPClient.java:514)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
