https://bugs.openldap.org/show_bug.cgi?id=10346
Issue ID: 10346
Summary: mdb_env_copy2 on a database with a value larger than
(2GB-16) results in a corrupt copy
Product: LMDB
Version: 0.9.31
Hardware: x86_64
OS: Linux
Status: UNCONFIRMED
Keywords: needs_review
Severity: normal
Priority: ---
Component: liblmdb
Assignee: [email protected]
Reporter: [email protected]
Target Milestone: ---
Created attachment 1072
--> https://bugs.openldap.org/attachment.cgi?id=1072&action=edit
reproduction source code
Running mdb_env_copy2 with compaction on a database with a value larger than
(2GB-16)bytes appears to complete successfully in that there are no errors, but
the copied database cannot be opened and throws an MDB_CORRUPTED error. Looking
at the copied database size, it appears that the value is either being skipped
or significantly truncated. Running mdb_env_copy2 without compaction also
completes successfully, and the copied database can be opened.
I initially encountered this while using py-lmdb with v0.9.31 of LMDB, but was
able to write up a simple script that uses the library directly. The source for
the script is attached, and the results below are from running it with the
latest from master.
Without compaction:
$ ./lmdb_repro test.lmdb $((2 * 1024 * 1024 * 1024 - 16 + 1)) testbak.lmdb
LMDB Version: LMDB 0.9.70: (December 19, 2015)
Set LMDB map size to 21474836330 bytes
Successfully inserted key with 2147483633 bytes of zero-filled data
Retrieved 2147483633 bytes of data
First 16 bytes (hex): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ...
Copying database to testbak.lmdb...
Database copy completed successfully.
Opening copied database and reading value...
Retrieved 2147483633 bytes of data from copied database
First 16 bytes from copy (hex): 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 ...
Data size matches between original and copy
With compaction:
$ ./lmdb_repro -c test.lmdb $((2 * 1024 * 1024 * 1024 - 16 + 1))
testbak.lmdb
LMDB Version: LMDB 0.9.70: (December 19, 2015)
Set LMDB map size to 21474836330 bytes
Successfully inserted key with 2147483633 bytes of zero-filled data
Retrieved 2147483633 bytes of data
First 16 bytes (hex): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ...
Copying database to testbak.lmdb (with compaction)...
Database copy completed successfully.
Opening copied database and reading value...
mdb_get (copy) failed: MDB_CORRUPTED: Located page was wrong type
Size difference on corrupt DB:
$ du -sh ./*
312K ./lmdb_repro
24K ./testbak.lmdb
2.1G ./test.lmdb
With compaction at the perceived max size:
$ ./lmdb_repro -c test.lmdb $((2 * 1024 * 1024 * 1024 - 16)) testbak.lmdb
LMDB Version: LMDB 0.9.70: (December 19, 2015)
Set LMDB map size to 21474836320 bytes
Successfully inserted key with 2147483632 bytes of zero-filled data
Retrieved 2147483632 bytes of data
First 16 bytes (hex): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ...
Copying database to testbak.lmdb (with compaction)...
Database copy completed successfully.
Opening copied database and reading value...
Retrieved 2147483632 bytes of data from copied database
First 16 bytes from copy (hex): 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 ...
Data size matches between original and copy
--
You are receiving this mail because:
You are on the CC list for the issue.
