kaushaldeokar opened a new issue, #4136:
URL: https://github.com/apache/hertzbeat/issues/4136
### Question
1. Currently Hertzbeat uses apache arrow and hertzbeat has both in-memory
heap and external memory usage .
It uses apache arrow to write in external memory .
**Our scenario :** we have distributed actictecture where we spin
multi-collectors and warehouses . data get converted to arrow object in
collector after collection and is passed to manager via netty here in manager
data gets deserialized and stored into memory and suspect manager memory is
increased due to this and later the data send to warehouse for persistence via
kafka topic .
if we add more monitors the memory grows drastically -> need to solve this
here manager memory is increased and not decreased later .
tried to clear the objects from memory before sending to kafka topic but it
didnot work .
here is sample code from manager where it pushes the data to kafka .
attached the ss for Heap and Code as well
<img width="1920" height="1080" alt="Image"
src="https://github.com/user-attachments/assets/477ca1bb-f44c-4076-939c-9e4a47d79e29";
/>
<img width="1920" height="1080" alt="Image"
src="https://github.com/user-attachments/assets/229fca54-89cd-4c17-afda-6f5e30cef4b6";
/>
<img width="1708" height="595" alt="Image"
src="https://github.com/user-attachments/assets/7bcb2487-7417-451f-b959-f3fafe41057a";
/>
<img width="1920" height="1080" alt="Image"
src="https://github.com/user-attachments/assets/c0816e24-3ec9-4859-a266-1c7e189eab9c";
/>
<img width="1920" height="1080" alt="Image"
src="https://github.com/user-attachments/assets/73a25e66-87b2-4548-9cbb-f4c5e695d4de";
/>
<img width="1920" height="1080" alt="Image"
src="https://github.com/user-attachments/assets/9d1e9908-ec90-44c0-8cc9-238dcaafabf3";
/>
<img width="790" height="478" alt="Image"
src="https://github.com/user-attachments/assets/7b2e6c3f-dd35-4a16-bdb9-df2ed506d772";
/>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail:
[email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
