Hi folks, I'm playing around with huge pages on x86-64, and I'm running
into behavior I'm not understanding.
I'm on Linux 3.2 (Ubuntu Precise), with libhugetlbfs 2.11.
Here's a representative part of my /proc/cpuinfo, particularly note that
the CPU claims to support both 2MB and 1 GB pages:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Intel(R) Xeon(R) CPU X5680 @ 3.33GHz
stepping : 2
microcode : 0x14
cpu MHz : 1600.000
cache size : 12288 KB
physical id : 0
siblings : 12
core id : 0
cpu cores : 6
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx
pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl
xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx
smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm
ida arat epb dts tpr_shadow vnmi flexpriority ept vpid
bogomips : 6667.00
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
I've separately tested this machine with 2MB pages, and that works very
well. But 1GB pages dont seem to work at all.
I booted the kernel with "hugepagesz=1g hugepages=10
default_hugepagesz=1g", and got this in my dmesg:
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.2.0-26-generic
root=UUID=8e6ac55e-4c43-4f55-ae05-1264d442e8d1 ro quiet splash vt.handoff=7
hugepagesz=1g hugepages=10 default_hugepagesz=1g
…
[ 3.072854] HugeTLB registered 1 GB page size, pre-allocated 10 pages
My /proc/meminfo looks reasonable:
AnonHugePages: 0 kB
HugePages_Total: 10
HugePages_Free: 10
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 1048576 kB
So far it all looks right, right?
But when I try to actually *use* 1 gig pages, i get nearly as many TLB
misses as if I was using 4k pages.
I wrote a test program that malloc's a big array of uint32's and then reads
it in a random order, and I use 'perf stat -e dTLB-load-misses' to track
TLB misses. (I'd be happy to provide the source code for this program, but
it's really dead simple.) The test allocates an aligned 2 gig chunk of
memory and reads 10 billion random locations.
perf reports (within a few percent) the same number of dTLB load misses
with 1G pages as I get with 4k pages. What gives? Any ideas?
I noticed this errata from Intel:
http://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/xeon-5600-specification-update.pdf
Specifically notice BD64, which says:
BD64. A Page Fault May Not be Generated When the PS bit is set to "1" in a
PML4E or PDPTE
Problem: On processors supporting Intel® 64 architecture, the PS bit (Page
Size, bit 7) is reserved in PML4Es and PDPTEs. If the translation of the
linear address of a memory
access encounters a PML4E or a PDPTE with PS set to 1, a page fault should
occur. Due to this erratum, PS of such an entry is ignored and no page
fault will occur due to its
being set.
Implication: Software may not operate properly if it relies on the
processor to deliver page faults when reserved bits are set in
paging-structure entries.
Workaround: Software should not set bit 7 in any PML4E or PDPTE that has
Present Bit (Bit 0) set to "1".
Does anyone have experience with this particular CPU?
Are 1 gig pages working and I'm just doing it wrong?
--
Sebastian Kuzminsky
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Libhugetlbfs-devel mailing list
Libhugetlbfs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/libhugetlbfs-devel
