gaogaotiantian commented on PR #57137: URL: https://github.com/apache/spark/pull/57137#issuecomment-4919401038
Honestly I don't think this is worth it. The code is really not maintainable and I'm not even sure this will pass on all arrow/numpy version matrix that we support. The code is cheap to generate now with LLM, but the maintenance effort is just too high. We don't know if there will corner cases in type coercion with `numpy`. This is something that should be done in low level libraries like `arrow`, not `pyspark`. The perf improvement is observable on certain dataset, but the issue is not "solved" - there is still a gap between arrow vs pickle, even with this much code. More importantly, we don't know what happens on other data types, or even how much this matters to our users. In general, I don't think taking over the work from arrow is a good idea. The benefit can't justify the cost. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
