Re: [I] [Bug]: Python SDFs (e.g. PeriodicImpulse) running in Flink and polling using tracker.defer_remainder have checkpoint size growing indefinitely [beam]

Sun, 15 Oct 2023 16:00:59 -0700 


nybbles commented on issue #27648:
URL: https://github.com/apache/beam/issues/27648#issuecomment-1763529241

   I took a heap dump from running my pipeline that uses a pure SDF to read 
from Redis (should have similar memory leak to the minimal example I posted 
that uses `PeriodicImpulse`.
   
   The retained memory is primarily in `registeredKVStates`, a 
`java.util.HashMap` that is in the `keyedStateStore` within the 
`runtimeContext` in `ExecutableStageDoFnOperator`.
   
   ![Screenshot from 2023-10-15 
15-53-56](https://github.com/apache/beam/assets/421369/afe29064-35c1-43db-ab8e-70a0f3661280)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to