Hi Álvaro, Nathan, > This seems like a rather low limit that will definitely affect > folks in the field.
I am not a heavy user of pg_repack, just curious, why do we think 105 million is a very small number? > An "easy" fix for this might be to mark that allocation as > using palloc_huge() somehow. The request that fails is the comboCids array in GetComboCommandId(), not the hash table. 8 bytes times 209,715,200 entries is the 1677721600 in Radim's error. I do not see a limit in the hash table. I tried repalloc_array_extended() there. With 110 million replayed updates master fails with that error, and the patched build completes with correct data. It only changes runs that fail today. From reading the code, the next limits are the int array size at about 1.7 billion entries, and then 2^32 commands. It does not lower the memory use. I measured 48 bytes for each change, so the limit today also caps this at about 5 GB. Without it, 1 billion changes need about 50 GB, and a host with less gets the OOM killer, not an ERROR. Experimental patch attached. Thanks, Shihao
test-only-repalloc-huge.diff
Description: Binary data
