Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/15556
@tgravescs I have tried it with a 150MB file and speedup is very noticeable
(refer the spark issue for results I see). I have noticed the JSON
deserialization is particularly slow, while simply running through the file not
so much. I would expect proportionate benefit on larger files wherever reading
the file/event stream is much less expensive than the Json deserialization +
replay.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
