I noticed that Spark handles CAST operation in fail-safe manner. i.e. if the casting operation will fail for some record(s), Spark doesn't fail the entire query instead it returns null data value for those failures.
For example following query: Looking at the code it seems that this behavior is implemented intentionally. As per my understanding, is this behavior is implemented to avoid failing the entire query due to some outlier records. Is this understanding correct? Is there any dev group mail thread where I can find details discussion on this decision? Also is there any configuration that can be used to make the query fail fast instead of returning null values? Any future development is planned on this similar line? -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
