peter-toth commented on pull request #31682: URL: https://github.com/apache/spark/pull/31682#issuecomment-788185220
> > During serialization GenericRowWithSchema(1.0, 1.0) will be memoized first > > Can we fix the problem if we use different `Pickler` instances for sending the input data to python and sending the UDF results to JVM? Not sure I get this. We don't use the same instance. How could we use the same instance? One lives in the JVM and the other in Python. We use `pyrolite` `Pickler` in the JVM. The `GenericRowWithSchema(1.0, 1.0)` and `GenericRowWithSchema(1, 1)` instances are the tuple values of `c1` and `c2` respectively. We send both of them from JVM to Python. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
