On Mon, 17 Mar 2014, Michael Schuster wrote:
Hi Bob, a few things come to mind to help narrow this down (if I misses something previous in this thread, please ignore ;-): - you mentioned using mmap; did you use other (standard?) malloc as well? Did you make sure libumem isn't adding some debugging information?
I have recently tested with mmap and umem's malloc. I will take time to test with the default malloc.
- how does the rest of the HW (I/O!) compare between the linux and illumos box? Have you done some iostat monitoring? - how do file system/volume management compare?
The Illumos system uses zfs with 8 SAS disks arranged as 4 mirrors. The Linux system uses a single SATA disk with the XFS filesystem. Regardless, the data is cached so there should be no I/O at all due to the reads.
My continuing theory is that Illumos is much slower to expand virtual memory (map fresh memory into the process) than Linux is. Linux does not actually give memory to a process until the address is accessed (modified) whereas Solaris assures that the memory is allocated in advance. Most of the memory allocated is scribbled on but perhaps the MMU (or software associated with it) has limits to its page-in rate and doing it gradually helps.
Previously I observed that using large pages helps speed up this sort of operation under Illumos.
In spite of my theory, using 1/2 the memory does not result in 1/2 the execution time under Illumos (whereas it does under Linux). My theory may be faulty.
Tools I have used thus far to attempt to find the root of the problem assume that programs run for a long time and are bottlenecked by CPU and are poor for evaluating this sort of issue.
Bob -- Bob Friesenhahn [email protected], http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ ------------------------------------------- illumos-discuss Archives: https://www.listbox.com/member/archive/182180/=now RSS Feed: https://www.listbox.com/member/archive/rss/182180/21175430-2e6923be Modify Your Subscription: https://www.listbox.com/member/?member_id=21175430&id_secret=21175430-6a77cda4 Powered by Listbox: http://www.listbox.com
