HeartSaVioR edited a comment on pull request #31296: URL: https://github.com/apache/spark/pull/31296#issuecomment-765877705
So what is the domain object T for PipeRDD when your type in Dataset is complicated? Can you reason about the output format of T.toString which PipeRDD will send to the external process? Is the output format known to end users? If you'd like to propose a public API then you'll need to be a first user on the API at least. The easiest way to prove your theory is to create a project which can communicate with the API. In the project you'll need to show the forked process can deal with such complicated type, and also prove the approach of deserialization is "known one". ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
