nybbles commented on issue #27648: URL: https://github.com/apache/beam/issues/27648#issuecomment-1763243069
> Hi @nybbles, I see, for pure SDF this option (`use_deprecated_read`) then won't work. This option was for the deprecated UnboundedRead (only exist in Java SDK) > > Nevertheless given that UnboundedRead execution is not broken on portable flink streaming, one should be able to fix `translateExecutableStage` (#[#27648 (comment)](https://github.com/apache/beam/issues/27648#issuecomment-1657418771)) according to it to fix at least the basic functionality of unbounded SDF @Abacn sorry for the silence; I needed to meet a deadline so I ended up finding a temporary solution (calling `defer_remainder` as little as possible, to delay the issue I was seeing). However, this workaround is not really sufficient anymore, so I need to look into this issue again. I also turned off checkpointing and am still seeing this issue (now it is Java heap space growth, proportional to the number of calls to `defer_remainder` that I do). I am taking a look at `translateExecutableStage`, but I'm not really sure what to fix to allow for basic functionality of unbounded SDF (or at least to stop this memory leak from occurring). I would appreciate some more guidance here - where could this memory leak be coming from, how I could track it down, and/or what is meant by fixing `translateExecutableStage`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
