rxin commented on pull request #31296: URL: https://github.com/apache/spark/pull/31296#issuecomment-765877782
> > Agree with @HeartSaVioR here. I think it is more general if we just provide a UDF for this... it also doesn’t pollute the Dataset API with something so rarely used. > > > > I'm not sure I understand your point here. So you mean when users have to use pipe, it is simply to ask them to write UDF instead of using pipe? No. My suggestion is to create a built in expression that can do that, rather than having a top level api called pipe. The expression would also allow users to choose the fields they use and how to serialize / deserialize. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
