MaxGekk commented on issue #23257: [SPARK-26310][SQL] Verify applicability of JSON options URL: https://github.com/apache/spark/pull/23257#issuecomment-449707703 @HyukjinKwon Thank you for looking at this PR. I proposed the changes because I saw a few times already when our users tried to apply datasource specific options in wrong way. Maybe this is because of Spark's documentation is not clear enough, and we should point out more precisely when particular option can be applicable. ... but I believe it makes sense to solve the problem in code. > ... more generalised but on the other hand it looks an overkill I considered more generalized approach like creating a trait for `JSONOptions`, `CSVOptions`, `TextOptions`, `AvroOptions` and etc. like. The trait could contains applicable options for each direction (`in read`/`in write`). In this way, a caller could ask explicitly (or implicitly from the constructor) to verify passed options. > The configuration also looks an overkill ... It was introduced only because of behavior change. > ... we could warn instead but on the other hand I get why it's needed Logs are not always available to users or can be disabled. And who among users read logs ;-) . I prefer to throw an exception because it can safe user's time in debugging/trouble shooting. I cannot imagine a situation when in write options are passed as read options in purpose, and silent behavior could be expected.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
