cloud-fan commented on pull request #31682: URL: https://github.com/apache/spark/pull/31682#issuecomment-788887152
correct me if I'm wrong: pickler recursively serializes the input and applies the cache. The input is a row of `(c1, c2)`, but pickler recursively serializes the row of `c1` and `c2`, and causes a problem because of the cache. Then I think it's not realistic to make one pickler instance to handle data with the same schema. Turning off `valueCompare` may be the only choice. To evaluate the severity of the problem, it seems only an issue when there are nested struct types? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
