Re: What are Analysis Errors With respect to Spark Sql DataFrames and DataSets?

Thanks a lot! On Wed, May 3, 2017 at 4:36 PM, Michael Armbrust wrote: > if I do dataset.select("nonExistentColumn") then the Analysis Error is >> thrown at compile time right? >> > > if you do df.as[MyClass].map(_.badFieldName) you will get a compile > error. However,

Re: What are Analysis Errors With respect to Spark Sql DataFrames and DataSets?

> > if I do dataset.select("nonExistentColumn") then the Analysis Error is > thrown at compile time right? > if you do df.as[MyClass].map(_.badFieldName) you will get a compile error. However, if df doesn't have the right columns for MyClass, that error will only be thrown at runtime (whether DF

Re: What are Analysis Errors With respect to Spark Sql DataFrames and DataSets?

got it! so if I do dataset.select("nonExistentColumn") then the Analysis Error is thrown at compile time right? but what if I have a dataset in memory and I go to a database server and execute a delete column query ? Will the dataset object be in sync with the underlying database table? Thanks!

Re: What are Analysis Errors With respect to Spark Sql DataFrames and DataSets?

An analysis exception occurs whenever the scala/java/python program is valid, but the dataframe operations being performed are not. For example, df.select("nonExistentColumn") would throw an analysis exception. On Wed, May 3, 2017 at 1:38 PM, kant kodali wrote: > Hi All, >