Github user joehalliwell commented on the pull request:
https://github.com/apache/spark/pull/11221#issuecomment-185087175
Thanks for taking the time to review and comment.
@rxin Yes, I wondered about batching to (say) 64KB of source code before
splitting. That's the approach used in
https://github.com/apache/spark/pull/7076. Without measuring, I wasn't sure
what the performance impact would be -- for example I believe JIT can inline
small methods such as these -- but if you think it's worth it, I'll push in
that general direction in future work.
@davies Ahh... I'm only really familiar with few bits of codegen I needed
to write this patch. Are there tests for or examples of whole stage codgen you
could point me at?
In the meantime, I'll implement your suggestion of falling back to
interpretation. Shall I do that on this PR, or open a new one?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
