Github user saurfang commented on the pull request:
https://github.com/apache/spark/pull/7076#issuecomment-120438630
I agree with you this fix is kind of a hack. However I would argue if a
human were to write these code, he would also split the projection calls into
blocks. The difference is really a human would break them into logical blocks
(e.g. groups columns that are similar to each other) for readability purposes.
Machine doesn't care and would just split them as long as things work. This
seems to be the suggestion I have seen when I searched for this exception which
also happens in other code generation frameworks.
That being said, if we were to fall back to
``InterpretedMutableProjection``, would spark-unsafe not go into effect
anymore? I understand unsafe is currently predicated on codegen and unsafe is
kind of critical for our current use case.
I also think this kind of scenario where you have lots of columns that can
be code generated is where codegen would likely shine the most?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
