On 2018-Aug-04, Kohei KaiGai wrote: > I could load the same data (544GB csv, 789GB heap) using COPY FROM > successfully. > When I reported the problem, rss usage of postgresql process increased > about 10MB/s ratio, then OOM killer eliminated after a few hours.
OK, I think we can consider this particular bug closed, then. > Now, it consumed about 60MB rss at the beginning of COPY FROM, and it > grows up very slowly during the COPY FROM execution, then grew up to > 250MB before completion. > We may have another memory blocks which are not released during > execution, however, > I could not identify whether it is really memory leaking or not, and > who's jobs. Most likely, this is a different memory leak. I sugges that one way to track this down is first figure out *which* context is the one growing, which you can see by running MemoryContextStats a few times and noting for meaningful differences. Then we can try to narrow down what is allocating stuff in that context. > It may be an idea to put a debug code that raises a notice when > MemoryContext allocates more than the threshold. I don't think this is really practical, because whatever the threshold we set, there would be some corner-case scenario where the threshold is legitimately crossed. And some memory leak scenarios that don't cross any thresholds. -- Álvaro Herrera https://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services