Re: What are Analysis Errors With respect to Spark Sql DataFrames and DataSets?

[kant kodali]

Thanks a lot!

On Wed, May 3, 2017 at 4:36 PM, Michael Armbrust 
wrote:

>  if I do dataset.select("nonExistentColumn") then the Analysis Error is
>> thrown at compile time right?
>>
>
> if you do df.as[MyClass].map(_.badFieldName) you will get a compile
> error.  However, if df doesn't have the right columns for MyClass, that
> error will only be thrown at runtime (whether DF is backed by something in
> memory or some remote database).
>

Re: What are Analysis Errors With respect to Spark Sql DataFrames and DataSets?

[Michael Armbrust]

>
>  if I do dataset.select("nonExistentColumn") then the Analysis Error is
> thrown at compile time right?
>

if you do df.as[MyClass].map(_.badFieldName) you will get a compile error.
However, if df doesn't have the right columns for MyClass, that error will
only be thrown at runtime (whether DF is backed by something in memory or
some remote database).

Re: What are Analysis Errors With respect to Spark Sql DataFrames and DataSets?

[kant kodali]

got it! so if I do dataset.select("nonExistentColumn") then the Analysis
Error is thrown at compile time right? but what if I have a dataset in
memory and I go to a database server and execute a delete column query ?
Will the dataset object be in sync with the underlying database table?

Thanks!

On Wed, May 3, 2017 at 2:25 PM, Michael Armbrust 
wrote:

> An analysis exception occurs whenever the scala/java/python program is
> valid, but the dataframe operations being performed are not.  For example,
> df.select("nonExistentColumn") would throw an analysis exception.
>
> On Wed, May 3, 2017 at 1:38 PM, kant kodali  wrote:
>
>> Hi All,
>>
>> I understand the compile time Errors this blog
>> 
>>  is
>> talking about but I don't understand what are Analysis Errors? Any Examples?
>>
>> Thanks!
>>
>
>

Re: What are Analysis Errors With respect to Spark Sql DataFrames and DataSets?

[Michael Armbrust]

An analysis exception occurs whenever the scala/java/python program is
valid, but the dataframe operations being performed are not.  For example,
df.select("nonExistentColumn") would throw an analysis exception.

On Wed, May 3, 2017 at 1:38 PM, kant kodali  wrote:

> Hi All,
>
> I understand the compile time Errors this blog
> 
>  is
> talking about but I don't understand what are Analysis Errors? Any Examples?
>
> Thanks!
>