gaogaotiantian commented on PR #54211: URL: https://github.com/apache/spark/pull/54211#issuecomment-3869001984
I disagree with the root cause analysis. I don't think this will fix the issue. The problem was that my implementation was wrong to begin with. `_execute_and_fetch_as_iterator` is a generator, not a normal function. Decorator for it will only trigger once when it's being called the first time (when the generator is being created). After that, when we really iterate the generator, the decorator won't trigger so gc is not disabled. We should change the implementation of the decorator or how to apply it. I'll work on it soon. As for switching from gc based solution - sure we can do that. It will be an even bigger change though, unless we decide that we don't need to release the model on the server. Any communication in `__del__` is dangerous. In theory we should not do any gRPC call in `__del__`. Disabling gc is not the most elegant way to solve it, but it's a way. We should fix it for now and discuss the next step because that would require somen design decisions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
