ASF GitHub Bot commented on FLINK-3322:

Github user ggevay commented on a diff in the pull request:

    --- Diff: 
    @@ -68,4 +72,9 @@
         * method signals an interrupt to that procedure.
        void abort();
    +   /**
    +    * Allows to reset the iterator
    +    */
    +   void reset(MutableObjectIterator<V1> in1, MutableObjectIterator<V2> 
in2, TypeSerializer<V1> serializer1, TypeSerializer<V2> serializer2, 
TypeComparator<V1> comp1, TypeComparator<V2> comp2, 
TypePairComparatorFactory<V1, V2> pairComparatorFactory);
    --- End diff --
    Could you pass in only those objects here, which might change during a 
reset? I think these should only be the two inputs.

> MemoryManager creates too much GC pressure with iterative jobs
> --------------------------------------------------------------
>                 Key: FLINK-3322
>                 URL: https://issues.apache.org/jira/browse/FLINK-3322
>             Project: Flink
>          Issue Type: Bug
>          Components: Local Runtime
>    Affects Versions: 1.0.0
>            Reporter: Gabor Gevay
>            Assignee: ramkrishna.s.vasudevan
>            Priority: Critical
>             Fix For: 1.0.0
>         Attachments: FLINK-3322.docx, FLINK-3322_reusingmemoryfordrivers.docx
> When taskmanager.memory.preallocate is false (the default), released memory 
> segments are not added to a pool, but the GC is expected to take care of 
> them. This puts too much pressure on the GC with iterative jobs, where the 
> operators reallocate all memory at every superstep.
> See the following discussion on the mailing list:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Memory-manager-behavior-in-iterative-jobs-tt10066.html
> Reproducing the issue:
> https://github.com/ggevay/flink/tree/MemoryManager-crazy-gc
> The class to start is malom.Solver. If you increase the memory given to the 
> JVM from 1 to 50 GB, performance gradually degrades by more than 10 times. 
> (It will generate some lookuptables to /tmp on first run for a few minutes.) 
> (I think the slowdown might also depend somewhat on 
> taskmanager.memory.fraction, because more unused non-managed memory results 
> in rarer GCs.)

This message was sent by Atlassian JIRA

Reply via email to