Github user attilapiros commented on the issue:
https://github.com/apache/spark/pull/21601
I read your changes and in the test I was searching for an check/assert but
found none. I understand it is about checking no exception is thrown during the
directory content reading but I still missed some asserts and covering more
cases, at least:
- min split size per node < maxSplitSize && min split size per rack <
maxSplitSize
- min split size per node > maxSplitSize && min split size per rack <
maxSplitSize
- min split size per node < maxSplitSize && min split size per rack >
maxSplitSize
As I see it is hard to add checks/asserts but what about testing
WholeTextFileInputFormat directly?
In your test you could inherit from WholeTextFileInputFormat and override
the protected setters for maxSplitSize, minSplitSizeNode, minSplitSizeRack and
and store the values in your new test class so asserts and checks can be added.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
