Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/15178
So if the primary use of this is inside of SQL then that might be OK
(because we can just be very careful about it) - but since we are also exposing
it to the user it feels like that these behaviour will probably catch some
people by surprise (and at the very least we should document the behaviour).
Maybe it would make sense to update the cleaning logic somehow or store the
blocks differently so the currently cleaning logic behaves as expected - but it
would be really good to hear what @rxin or @JoshRosen think about this because
I'm a little uncertain.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
