Re: IF statement doesn't work in Spark-SQL?
Hi, Yes, this is what I'm doing. I'm using hiveContext.hql() to run my query. But, the problem still happens. On Tue, Jan 20, 2015 at 7:24 PM, DEVAN M.S. msdeva...@gmail.com wrote: Add one more library libraryDependencies += org.apache.spark % spark-hive_2.10 % 1.2.0 val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc) repalce sqlContext with hiveContext. Its working while using HiveContext for me. Devan M.S. | Research Associate | Cyber Security | AMRITA VISHWA VIDYAPEETHAM | Amritapuri | Cell +919946535290 | On Tue, Jan 20, 2015 at 4:45 PM, DEVAN M.S. msdeva...@gmail.com wrote: Which context are you using HiveContext or SQLContext ? Can you try with HiveContext ?? Devan M.S. | Research Associate | Cyber Security | AMRITA VISHWA VIDYAPEETHAM | Amritapuri | Cell +919946535290 | On Tue, Jan 20, 2015 at 3:49 PM, Xuelin Cao xuelincao2...@gmail.com wrote: Hi, I'm using Spark 1.2 On Tue, Jan 20, 2015 at 5:59 PM, Wang, Daoyuan daoyuan.w...@intel.com wrote: Hi Xuelin, What version of Spark are you using? Thanks, Daoyuan *From:* Xuelin Cao [mailto:xuelincao2...@gmail.com] *Sent:* Tuesday, January 20, 2015 5:22 PM *To:* User *Subject:* IF statement doesn't work in Spark-SQL? Hi, I'm trying to migrate some hive scripts to Spark-SQL. However, I found some statement is incompatible in Spark-sql. Here is my SQL. And the same SQL works fine in HIVE environment. SELECT *if(ad_user_id1000, 1000, ad_user_id) as user_id* FROM ad_search_keywords What I found is, the parser reports error on the *if* statement: No function to evaluate expression. type: AttributeReference, tree: ad_user_id#4 Anyone have any idea about this?
Re: IF statement doesn't work in Spark-SQL?
Can you share your code ? Devan M.S. | Research Associate | Cyber Security | AMRITA VISHWA VIDYAPEETHAM | Amritapuri | Cell +919946535290 | On Tue, Jan 20, 2015 at 5:03 PM, Xuelin Cao xuelincao2...@gmail.com wrote: Hi, Yes, this is what I'm doing. I'm using hiveContext.hql() to run my query. But, the problem still happens. On Tue, Jan 20, 2015 at 7:24 PM, DEVAN M.S. msdeva...@gmail.com wrote: Add one more library libraryDependencies += org.apache.spark % spark-hive_2.10 % 1.2.0 val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc) repalce sqlContext with hiveContext. Its working while using HiveContext for me. Devan M.S. | Research Associate | Cyber Security | AMRITA VISHWA VIDYAPEETHAM | Amritapuri | Cell +919946535290 | On Tue, Jan 20, 2015 at 4:45 PM, DEVAN M.S. msdeva...@gmail.com wrote: Which context are you using HiveContext or SQLContext ? Can you try with HiveContext ?? Devan M.S. | Research Associate | Cyber Security | AMRITA VISHWA VIDYAPEETHAM | Amritapuri | Cell +919946535290 | On Tue, Jan 20, 2015 at 3:49 PM, Xuelin Cao xuelincao2...@gmail.com wrote: Hi, I'm using Spark 1.2 On Tue, Jan 20, 2015 at 5:59 PM, Wang, Daoyuan daoyuan.w...@intel.com wrote: Hi Xuelin, What version of Spark are you using? Thanks, Daoyuan *From:* Xuelin Cao [mailto:xuelincao2...@gmail.com] *Sent:* Tuesday, January 20, 2015 5:22 PM *To:* User *Subject:* IF statement doesn't work in Spark-SQL? Hi, I'm trying to migrate some hive scripts to Spark-SQL. However, I found some statement is incompatible in Spark-sql. Here is my SQL. And the same SQL works fine in HIVE environment. SELECT *if(ad_user_id1000, 1000, ad_user_id) as user_id* FROM ad_search_keywords What I found is, the parser reports error on the *if* statement: No function to evaluate expression. type: AttributeReference, tree: ad_user_id#4 Anyone have any idea about this?
Re: IF statement doesn't work in Spark-SQL?
Hi, I'm using Spark 1.2 On Tue, Jan 20, 2015 at 5:59 PM, Wang, Daoyuan daoyuan.w...@intel.com wrote: Hi Xuelin, What version of Spark are you using? Thanks, Daoyuan *From:* Xuelin Cao [mailto:xuelincao2...@gmail.com] *Sent:* Tuesday, January 20, 2015 5:22 PM *To:* User *Subject:* IF statement doesn't work in Spark-SQL? Hi, I'm trying to migrate some hive scripts to Spark-SQL. However, I found some statement is incompatible in Spark-sql. Here is my SQL. And the same SQL works fine in HIVE environment. SELECT *if(ad_user_id1000, 1000, ad_user_id) as user_id* FROM ad_search_keywords What I found is, the parser reports error on the *if* statement: No function to evaluate expression. type: AttributeReference, tree: ad_user_id#4 Anyone have any idea about this?
RE: IF statement doesn't work in Spark-SQL?
Hi Xuelin, What version of Spark are you using? Thanks, Daoyuan From: Xuelin Cao [mailto:xuelincao2...@gmail.com] Sent: Tuesday, January 20, 2015 5:22 PM To: User Subject: IF statement doesn't work in Spark-SQL? Hi, I'm trying to migrate some hive scripts to Spark-SQL. However, I found some statement is incompatible in Spark-sql. Here is my SQL. And the same SQL works fine in HIVE environment. SELECT if(ad_user_id1000, 1000, ad_user_id) as user_id FROM ad_search_keywords What I found is, the parser reports error on the if statement: No function to evaluate expression. type: AttributeReference, tree: ad_user_id#4 Anyone have any idea about this?
Re: IF statement doesn't work in Spark-SQL?
Which context are you using HiveContext or SQLContext ? Can you try with HiveContext ?? Devan M.S. | Research Associate | Cyber Security | AMRITA VISHWA VIDYAPEETHAM | Amritapuri | Cell +919946535290 | On Tue, Jan 20, 2015 at 3:49 PM, Xuelin Cao xuelincao2...@gmail.com wrote: Hi, I'm using Spark 1.2 On Tue, Jan 20, 2015 at 5:59 PM, Wang, Daoyuan daoyuan.w...@intel.com wrote: Hi Xuelin, What version of Spark are you using? Thanks, Daoyuan *From:* Xuelin Cao [mailto:xuelincao2...@gmail.com] *Sent:* Tuesday, January 20, 2015 5:22 PM *To:* User *Subject:* IF statement doesn't work in Spark-SQL? Hi, I'm trying to migrate some hive scripts to Spark-SQL. However, I found some statement is incompatible in Spark-sql. Here is my SQL. And the same SQL works fine in HIVE environment. SELECT *if(ad_user_id1000, 1000, ad_user_id) as user_id* FROM ad_search_keywords What I found is, the parser reports error on the *if* statement: No function to evaluate expression. type: AttributeReference, tree: ad_user_id#4 Anyone have any idea about this?
Re: IF statement doesn't work in Spark-SQL?
Add one more library libraryDependencies += org.apache.spark % spark-hive_2.10 % 1.2.0 val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc) repalce sqlContext with hiveContext. Its working while using HiveContext for me. Devan M.S. | Research Associate | Cyber Security | AMRITA VISHWA VIDYAPEETHAM | Amritapuri | Cell +919946535290 | On Tue, Jan 20, 2015 at 4:45 PM, DEVAN M.S. msdeva...@gmail.com wrote: Which context are you using HiveContext or SQLContext ? Can you try with HiveContext ?? Devan M.S. | Research Associate | Cyber Security | AMRITA VISHWA VIDYAPEETHAM | Amritapuri | Cell +919946535290 | On Tue, Jan 20, 2015 at 3:49 PM, Xuelin Cao xuelincao2...@gmail.com wrote: Hi, I'm using Spark 1.2 On Tue, Jan 20, 2015 at 5:59 PM, Wang, Daoyuan daoyuan.w...@intel.com wrote: Hi Xuelin, What version of Spark are you using? Thanks, Daoyuan *From:* Xuelin Cao [mailto:xuelincao2...@gmail.com] *Sent:* Tuesday, January 20, 2015 5:22 PM *To:* User *Subject:* IF statement doesn't work in Spark-SQL? Hi, I'm trying to migrate some hive scripts to Spark-SQL. However, I found some statement is incompatible in Spark-sql. Here is my SQL. And the same SQL works fine in HIVE environment. SELECT *if(ad_user_id1000, 1000, ad_user_id) as user_id* FROM ad_search_keywords What I found is, the parser reports error on the *if* statement: No function to evaluate expression. type: AttributeReference, tree: ad_user_id#4 Anyone have any idea about this?
Re: IF statement doesn't work in Spark-SQL?
|IF| is implemented as a generic UDF in Hive (|GenericUDFIf|). It seems that this function can’t be properly resolved. Could you provide a minimum code snippet that reproduces this issue? Cheng On 1/20/15 1:22 AM, Xuelin Cao wrote: Hi, I'm trying to migrate some hive scripts to Spark-SQL. However, I found some statement is incompatible in Spark-sql. Here is my SQL. And the same SQL works fine in HIVE environment. SELECT *if(ad_user_id1000, 1000, ad_user_id) as user_id* FROM ad_search_keywords What I found is, the parser reports error on the *if* statement: No function to evaluate expression. type: AttributeReference, tree: ad_user_id#4 Anyone have any idea about this?
