https://github.com/gluster/glusterfs/blob/master/extras/distributed-testing/distributed-test-runner.py#L334
This call can be expensive in terms of memory because it sends the entire zip of the files from client to server. This call be optimized to send in batches if there is memory issues. You can try the change or let me know if you want me to make the change. Otherwise this code should be very slim on memory. Thanks! From: Deepshikha Khandelwal <[email protected]> Date: Monday, March 19, 2018 at 9:24 PM To: Karthikeyan Radhakrishnan <[email protected]> Cc: Nigel Babu <[email protected]>, gluster-infra <[email protected]>, Jeff Darcy <[email protected]> Subject: Re: Distributed Testing and Memory issues On Sun, Mar 18, 2018 at 12:12 PM, Karthikeyan Radhakrishnan <[email protected]<mailto:[email protected]>> wrote: Hi Nigel, This is awesome! MemoryError is very weird. We @Facebook have never seen that. The test server/client is super thin to cause memory pressure, but the tests they run can cause such issues. How much memory does the machine you are running have? I'm running this on machines having 2GB memory. And I think this is enough to have this distributed test framework setup for us. Is the machine under pressure when you see the errors? The best way would be to add a rpc to query memory stat and observe. These are newly created machines running just XMLRPC server process. I checked with top and got to know that this process is utilizing about 77% of memory at initial stage itself when the tester part of code scans and skip kicking the host/server for availability. RPC is a new thing for me, so I'm not aware of RPC query calls. If you can brief me more about this, it would be helpful. Let me accelerate setting up some common space (like aws) where can re-pro such problems. It would be great. Thanks! -Karthik From: Nigel Babu <[email protected]<mailto:[email protected]>> Date: Saturday, March 17, 2018 at 7:03 AM To: Karthikeyan Radhakrishnan <[email protected]<mailto:[email protected]>> Cc: gluster-infra <[email protected]<mailto:[email protected]>>, Deepshikha Khandelwal <[email protected]<mailto:[email protected]>>, Jeff Darcy <[email protected]<mailto:[email protected]>> Subject: Distributed Testing and Memory issues Hey Karthik, Deepshikha has been working on testing the distributed test framework that you contributed (thank you!). Instead of writing our own code to chunk the tests, we've decided to just consume what you've written so we can work on making it run both at FB and upstream. We're running into MemoryError exception from the threads. Do you know what's the best way to debug or let us know how much memory your machines have? That'll help us figure out solving this sooner upstream. PS: This email is CC'd to gluster-infra and is archived publicly. -- nigelb
_______________________________________________ Gluster-infra mailing list [email protected] http://lists.gluster.org/mailman/listinfo/gluster-infra
