michalsenkyr edited a comment on issue #20505: [SPARK-23251][SQL] Add checks for collection element Encoders URL: https://github.com/apache/spark/pull/20505#issuecomment-527632139 I have not worked on this for a year and a half, but I am willing to look into this again. I can confirm that the element types are still not checked on current master. Brief review of the last state of this PR: Essentially, there are three possible behaviors when creating Datasets with elements without Encoders: 1. Failure in runtime (with a confusing error message) - current behavior 2. Failure on compilation with a confusing error message - behavior with this PR in its (almost - see below) current state 3. Failure on compilation with a good error message - not sure if possible, would require further research I tried rebasing this on the current master and we're only missing a few Encoders (BigInt, Null and collection arrays) to get it up to date with current unit tests. Please note that these Encoders are missing for Datasets in general, not just in combination with collections, even though the types are used in collection tests, which does not seem to be a desired behavior. Also note that as it's not trivial to transfer the full serialization capabilities of ExpressionEncoder to compile-time constraints, there may be edge cases where backwards compatibility gets broken (as with the aforementioned unit tests). I can clean up and commit the rebased code, but I am unsure whether this feature is desirable enough to spend more time on. Can I get a second opinion on this?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
