Re: to_json not working with selectExpr

2017-07-19 Thread Matthew cao 
Another question, how can I find the version in which one function is added 
into SQL? 
> On 2017年7月20日, at 10:43, Hyukjin Kwon  wrote:
> 
> Yes, I guess it is.
> 
> 2017-07-20 11:31 GMT+09:00 Matthew cao  >:
> AH, I get it. So that’s why I get the not register error? Cuz it not added 
> into SQL in 2.1.0?
> 
>> On 2017年7月19日, at 22:35, Hyukjin Kwon > > wrote:
>> 
>> Yea, but it was added into SQL from Spark 2.2.0
>> 
>> 2017-07-19 23:02 GMT+09:00 Matthew cao > >:
>> I am using version 2.1.1 As I could remember, this function was added since 
>> 2.1.0.
>> 
>>> On 2017年7月17日, at 12:05, Burak Yavuz >> > wrote:
>>> 
>>> Hi Matthew,
>>> 
>>> Which Spark version are you using? The expression `to_json` was added in 
>>> 2.2 with this commit:
>>> https://github.com/apache/spark/commit/0cdcf9114527a2c359c25e46fd6556b3855bfb28
>>>  
>>> 
>>> 
>>> Best,
>>> Burak
>>> 
>>> On Sun, Jul 16, 2017 at 6:24 PM, Matthew cao >> > wrote:
>>> Hi all,
>>> I just read the databricks blog here: 
>>> https://docs.databricks.com/_static/notebooks/complex-nested-structured.html
>>>  
>>> 
>>> 
>>> When I try to follow the example about the to_json and selectExpr part, it 
>>> gave error: “org.apache.spark.sql.AnalysisException: Undefined function: 
>>> 'to_json'. This function is neither a registered temporary function nor a 
>>> permanent function registered in the database 'default'.; line 1 pos 0”.
>>> Also this error show in the original databricks notebook. I know that 
>>> to_json function works great with select. Do I miss something when using 
>>> selectExpr? THX.
>>> 
>>> Best,
>>> Matthew
>>> 
>>> 
>> 
>> 
> 
>

Re: to_json not working with selectExpr

2017-07-19 Thread Matthew cao 
Thank you so much! that confuse me quite a time!
> On 2017年7月20日, at 10:43, Hyukjin Kwon  wrote:
> 
> Yes, I guess it is.
> 
> 2017-07-20 11:31 GMT+09:00 Matthew cao  >:
> AH, I get it. So that’s why I get the not register error? Cuz it not added 
> into SQL in 2.1.0?
> 
>> On 2017年7月19日, at 22:35, Hyukjin Kwon > > wrote:
>> 
>> Yea, but it was added into SQL from Spark 2.2.0
>> 
>> 2017-07-19 23:02 GMT+09:00 Matthew cao > >:
>> I am using version 2.1.1 As I could remember, this function was added since 
>> 2.1.0.
>> 
>>> On 2017年7月17日, at 12:05, Burak Yavuz >> > wrote:
>>> 
>>> Hi Matthew,
>>> 
>>> Which Spark version are you using? The expression `to_json` was added in 
>>> 2.2 with this commit:
>>> https://github.com/apache/spark/commit/0cdcf9114527a2c359c25e46fd6556b3855bfb28
>>>  
>>> 
>>> 
>>> Best,
>>> Burak
>>> 
>>> On Sun, Jul 16, 2017 at 6:24 PM, Matthew cao >> > wrote:
>>> Hi all,
>>> I just read the databricks blog here: 
>>> https://docs.databricks.com/_static/notebooks/complex-nested-structured.html
>>>  
>>> 
>>> 
>>> When I try to follow the example about the to_json and selectExpr part, it 
>>> gave error: “org.apache.spark.sql.AnalysisException: Undefined function: 
>>> 'to_json'. This function is neither a registered temporary function nor a 
>>> permanent function registered in the database 'default'.; line 1 pos 0”.
>>> Also this error show in the original databricks notebook. I know that 
>>> to_json function works great with select. Do I miss something when using 
>>> selectExpr? THX.
>>> 
>>> Best,
>>> Matthew
>>> 
>>> 
>> 
>> 
> 
>

Re: to_json not working with selectExpr

2017-07-19 Thread Hyukjin Kwon 
Yes, I guess it is.

2017-07-20 11:31 GMT+09:00 Matthew cao :

> AH, I get it. So that’s why I get the not register error? Cuz it not added
> into SQL in 2.1.0?
>
> On 2017年7月19日, at 22:35, Hyukjin Kwon  wrote:
>
> Yea, but it was added into SQL from Spark 2.2.0
>
> 2017-07-19 23:02 GMT+09:00 Matthew cao :
>
>> I am using version 2.1.1 As I could remember, this function was added
>> since 2.1.0.
>>
>> On 2017年7月17日, at 12:05, Burak Yavuz  wrote:
>>
>> Hi Matthew,
>>
>> Which Spark version are you using? The expression `to_json` was added in
>> 2.2 with this commit:
>> https://github.com/apache/spark/commit/0cdcf9114527a2c359c25
>> e46fd6556b3855bfb28
>>
>> Best,
>> Burak
>>
>> On Sun, Jul 16, 2017 at 6:24 PM, Matthew cao  wrote:
>>
>>> Hi all,
>>> I just read the databricks blog here: https://docs.databricks.
>>> com/_static/notebooks/complex-nested-structured.html
>>>
>>> When I try to follow the example about the to_json and selectExpr part,
>>> it gave error: “org.apache.spark.sql.AnalysisException: Undefined
>>> function: 'to_json'. This function is neither a registered temporary
>>> function nor a permanent function registered in the database 'default'.;
>>> line 1 pos 0”.
>>> Also this error show in the original databricks notebook. I know that
>>> to_json function works great with select. Do I miss something when using
>>> selectExpr? THX.
>>>
>>> Best,
>>> Matthew
>>>
>>>
>>
>>
>
>

Re: to_json not working with selectExpr

2017-07-19 Thread Matthew cao 
AH, I get it. So that’s why I get the not register error? Cuz it not added into 
SQL in 2.1.0?
> On 2017年7月19日, at 22:35, Hyukjin Kwon  > wrote:
> 
> Yea, but it was added into SQL from Spark 2.2.0
> 
> 2017-07-19 23:02 GMT+09:00 Matthew cao  >:
> I am using version 2.1.1 As I could remember, this function was added since 
> 2.1.0.
> 
>> On 2017年7月17日, at 12:05, Burak Yavuz > > wrote:
>> 
>> Hi Matthew,
>> 
>> Which Spark version are you using? The expression `to_json` was added in 2.2 
>> with this commit:
>> https://github.com/apache/spark/commit/0cdcf9114527a2c359c25e46fd6556b3855bfb28
>>  
>> 
>> 
>> Best,
>> Burak
>> 
>> On Sun, Jul 16, 2017 at 6:24 PM, Matthew cao > > wrote:
>> Hi all,
>> I just read the databricks blog here: 
>> https://docs.databricks.com/_static/notebooks/complex-nested-structured.html 
>> 
>> 
>> When I try to follow the example about the to_json and selectExpr part, it 
>> gave error: “org.apache.spark.sql.AnalysisException: Undefined function: 
>> 'to_json'. This function is neither a registered temporary function nor a 
>> permanent function registered in the database 'default'.; line 1 pos 0”.
>> Also this error show in the original databricks notebook. I know that 
>> to_json function works great with select. Do I miss something when using 
>> selectExpr? THX.
>> 
>> Best,
>> Matthew
>> 
>> 
> 
>

Re: to_json not working with selectExpr

2017-07-19 Thread Hyukjin Kwon 
Yea, but it was added into SQL from Spark 2.2.0

2017-07-19 23:02 GMT+09:00 Matthew cao :

> I am using version 2.1.1 As I could remember, this function was added
> since 2.1.0.
>
> On 2017年7月17日, at 12:05, Burak Yavuz  wrote:
>
> Hi Matthew,
>
> Which Spark version are you using? The expression `to_json` was added in
> 2.2 with this commit:
> https://github.com/apache/spark/commit/0cdcf9114527a2c359c25e46fd6556
> b3855bfb28
>
> Best,
> Burak
>
> On Sun, Jul 16, 2017 at 6:24 PM, Matthew cao  wrote:
>
>> Hi all,
>> I just read the databricks blog here: https://docs.databricks.
>> com/_static/notebooks/complex-nested-structured.html
>>
>> When I try to follow the example about the to_json and selectExpr part,
>> it gave error: “org.apache.spark.sql.AnalysisException: Undefined
>> function: 'to_json'. This function is neither a registered temporary
>> function nor a permanent function registered in the database 'default'.;
>> line 1 pos 0”.
>> Also this error show in the original databricks notebook. I know that
>> to_json function works great with select. Do I miss something when using
>> selectExpr? THX.
>>
>> Best,
>> Matthew
>>
>>
>
>

Re: to_json not working with selectExpr

2017-07-19 Thread Matthew cao 
I am using version 2.1.1 As I could remember, this function was added since 
2.1.0.
> On 2017年7月17日, at 12:05, Burak Yavuz  > wrote:
> 
> Hi Matthew,
> 
> Which Spark version are you using? The expression `to_json` was added in 2.2 
> with this commit:
> https://github.com/apache/spark/commit/0cdcf9114527a2c359c25e46fd6556b3855bfb28
>  
> 
> 
> Best,
> Burak
> 
> On Sun, Jul 16, 2017 at 6:24 PM, Matthew cao  > wrote:
> Hi all,
> I just read the databricks blog here: 
> https://docs.databricks.com/_static/notebooks/complex-nested-structured.html 
> 
> 
> When I try to follow the example about the to_json and selectExpr part, it 
> gave error: “org.apache.spark.sql.AnalysisException: Undefined function: 
> 'to_json'. This function is neither a registered temporary function nor a 
> permanent function registered in the database 'default'.; line 1 pos 0”.
> Also this error show in the original databricks notebook. I know that to_json 
> function works great with select. Do I miss something when using selectExpr? 
> THX.
> 
> Best,
> Matthew
> 
>

Re: to_json not working with selectExpr

2017-07-16 Thread Burak Yavuz 
Hi Matthew,

Which Spark version are you using? The expression `to_json` was added in
2.2 with this commit:
https://github.com/apache/spark/commit/0cdcf9114527a2c359c25e46fd6556b3855bfb28

Best,
Burak

On Sun, Jul 16, 2017 at 6:24 PM, Matthew cao  wrote:

> Hi all,
> I just read the databricks blog here: https://docs.databricks.
> com/_static/notebooks/complex-nested-structured.html
>
> When I try to follow the example about the to_json and selectExpr part, it
> gave error: “org.apache.spark.sql.AnalysisException: Undefined function:
> 'to_json'. This function is neither a registered temporary function nor a
> permanent function registered in the database 'default'.; line 1 pos 0”.
> Also this error show in the original databricks notebook. I know that
> to_json function works great with select. Do I miss something when using
> selectExpr? THX.
>
> Best,
> Matthew
>
>