Github user srowen commented on the issue:
https://github.com/apache/spark/pull/15915
It seems to do the job. It seems a little funny that if you specify a huge
chunk size like 1GB, that's accepted, but if you specify ~2GB, it becomes 4MB.
Is the logic that this probably happens when something else is left to a
default value, so this is like a default value?
it seems more natural to clamp to Int.MaxValue if we believe that the
caller really means to specify a huge chunk size. But it sounds like the
argument is that this can never be sensible. Is there a lower hard limit to
enforce?
That is: is it too much to just cap this at 4MB?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
