Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/11347#issuecomment-190225735
Hi @davies
I did a simple benchmark for pickle serializer with 1 million (int, string)
rows. The execution time only includes serialize results at python side and
send bytes to JVM side, the result is: batched serializer is about 2.67 times
faster and serialized size is about 30% smaller.
I think this is a problem. We should find another way to overcome the issue
that binary data at JVM side has wrong row count, so that we can still use
batched serializer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
