> Try repeating the test with MDB_NORDAHEAD set on the environment.
Thank you: with MDB_NORDAHEAD it works on both machines as expected. We have
a couple of questions and observations.
We have:
machine 1:
XFS filesystem, 148GB RAM, 3.13
# blockdev --getra /dev/fiob
256
Shared memory grows to 13GB with or without MDB_NORDAHEAD (as expected)
machine 2:
EXT4 filesystem, 105GB RAM, 4.15
# blockdev --getra /dev/vda2
256
Shared memory grows to 83GB without MDB_NORDAHEAD (unexpected) and to 13GB
with MDB_NORDAHEAD (as expected)
Questions and observations:
1. Since blockdev --getra shows the same 256 for both machines, why
MDB_NORDAHEAD was necessary only on machine2?
2. After dropping the cache with sysctl -w vm.drop_caches=3 on the
problematic machine2 we read 83GB into shared memory without MDB_NORDAHEAD.
Then we re-run with MDB_NORDAHEAD reading from the in-memory cached
file, but the shared memory still grows to 83GB.
This contrasts with the first read with MDB_NORDAHEAD where shared
memory is 13GB on all reads.
3. Both machines have the same raw disk read speeds (tested with hdparm
and dd). After sysctl -w vm.drop_caches=3 the initial read time on
machine2 is twice that of on machine1.
4. Initially even on the affected machine2 it read 13G into shared memory
without MDB_NORDAHEAD. We upgraded the Ubuntu kernel from 4.15.0-91-generic
to 4.15.0-101-generic and it started reading 83GB. Then we downgraded to
4.15.0-91-generic but it continued to read 83GB until we added
MDB_NORDAHEAD.
-----Original Message-----
From: Howard Chu [mailto:[email protected]]
Sent: Wednesday, June 03, 2020 12:56 PM
To: Alec Matusis <[email protected]>; [email protected]
Subject: Re: Unexpected LMDB RSS /performance difference on similar machines
Alec Matusis wrote:
> We have encountered an unexpected performance impact only by moving an
LMDB to a different linux machine with similar hardware characteristics.
>
>
>
> We have a 93GB LMDB environment with 3 databases. The database in
> question is 13GB. The test executable loops over key/value pairs in
> the 13GB database with a read-only cursor. For the same executable, we
observe two different behaviors on different machines (the same lmdb
environment was copied to the machines with scp). First machine has 148GB
RAM and the second has 105GB RAM and the same CPU.
>
>
>
> The expected and desired behavior on linux kernel 3.13 / eglibc 2.19
> shows that the process takes 13GB of shared memory (seen by top and
> confirmed with /proc/<pid>/smaps below)). In the alternate behavior on
kernel 4.15 / glibc 2.27 the process reads from disk 83GB into shared memory
(which takes 16 minutes instead of 8 minutes on the correct machine in the
corresponding initial runs).
glibc version is irrelevant, only the kernel version matters, since no
library calls of any kind are invoked in a read operation.
Try repeating the test with MDB_NORDAHEAD set on the environment.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
