Hey Steven,
There are many factors that may be contributing to the latency spikes you are seeing. Most don't have anything to do with emane and are likely related to your host system.
Have you any other advice as to typical factors/os configuration options you've seen that cause high latencies for emane and/or how I might diagnose them?
We have spent a fair amount of time tuning our emulation servers (LXC host systems) to achieve that goal. We don't run X on our servers, we disable all unneeded services and we usually assign one or more dedicated cores to each container running emane. The number of cores per container can increase based on whatever else is executing in the container. Some of our large scale network emulations trade fidelity for node count and assign more than one container per dedicated CPU. This type of tuning in not needed by all. It depends are how satisfied you are with the resulting emulation fidelity and whether or not the radio models in use have tight timing constraints. As a first step, if you are generating high traffic loads, you may benefit from trying the emane develop branch. See Pull 34 [1] which added functionality to short circuit timer logic for enforcing transmit data rates. After that I would try assigning a dedicated core to each container. [1] https://github.com/adjacentlink/emane/pull/34
OK, I tried your patch but it didn't have much effect on either the latency spikes or the high avgTimedEventLatencyRatio statistics.
If I allow the n emane daemons to share n cores, with every other container service running on the remaining cores, I get a negligible reduction in avgTimedEventLatencyRatio (from 1-2 to about 0.75-1.75) and no effect on latency spikes.
If I run each emane daemon on its own dedicated core, with every other container service shared across the others cores, I see a massive increase in the avgTimedEventLatencyRatio (from 1-2) to about 6-7) but again no impact on the latency spikes.
Do you think there is any point in me assigning containers as a whole to specific cores, or is it sufficient to assign the emane daemons separately? The CPU utilization for the cores running my emane daemons seems low enough to me (see attached pdf), although the remaining cores are heavily loaded.
Is it possible that the avgTimedEventLatencyRatios can be high if the host machine/emulator cores aren't overloaded? For example, are there any memory usage/io stats you typically look at?
Any help much appreciated. Thanks, Dan
cpu_util_fixed_kmobsession_16-11-16-Thu191115.pdf
Description: application/
_______________________________________________ emane-users mailing list [email protected] http://pf.itd.nrl.navy.mil/mailman/listinfo/emane-users
