Github user lxsmnv commented on the issue:
https://github.com/apache/spark/pull/16995
@HyukjinKwon the problem that I have found was in CSVFileFormat. So its
more of csv specific. However, it can be a problem of some other data source
types, but not all - it will depend on data source implementation. If the same
problem with other data source types exists, there can be some degree of a
common nature of that problem for those datasource types and the fix may
require a some significant amount of work and changes.
My fix is quite simple and doesn't introduce much changes. For now, I would
suggest to merge it. Otherwise aiming a more generic solution now may end up in
neither options being implemented.
If you tried to reproduce this issue with our datasource types, can you
create a new ticket and provide the details about the tests you have done and I
will have a look at it and think about more generic approach.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
