spark error when distinct on more than one cloume

2014-08-19 Thread wan...@testbird.com 







sql:SELECT app_id,COUNT(DISTINCT app_id, macaddr) cut from object group by 
app_id 

Error Log
14/08/19 17:58:26 INFO MemoryStore: Block broadcast_0 stored as values to 
memory (estimated size 158.6 KB, free 294.7 MB)

Exception in thread main java.lang.RuntimeException: [1.36] failure: ``)'' 
expected but `,' found



SELECT app_id,COUNT(DISTINCT app_id, macaddr) cut from object group by app_id

   ^

at scala.sys.package$.error(package.scala:27)

at org.apache.spark.sql.catalyst.SqlParser.apply(SqlParser.scala:47)

at org.apache.spark.sql.SQLContext.parseSql(SQLContext.scala:70)

at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:150)

at com.testbird.sparkapi.APILearn1$.main(APILearn1.scala:30)

at com.testbird.sparkapi.APILearn1.main(APILearn1.scala)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:292)

at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)

at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
any one can help me , thanks

Re: spark error when distinct on more than one cloume

2014-08-19 Thread Yin Huai 
Hi,

The SQLParser used by SQLContext is pretty limited. Instead, can you try
HiveContext?

Thanks,

Yin


On Tue, Aug 19, 2014 at 7:57 AM, wan...@testbird.com wan...@testbird.com
wrote:


 sql:SELECT app_id,COUNT(DISTINCT app_id, macaddr) cut from object group
 by app_id


 *Error Log*

 14/08/19 17:58:26 INFO MemoryStore: Block broadcast_0 stored as values to
 memory (estimated size 158.6 KB, free 294.7 MB)
 Exception in thread main java.lang.RuntimeException: [1.36] failure:
 ``)'' expected but `,' found

 SELECT app_id,COUNT(DISTINCT app_id, macaddr) cut from object group by
 app_id
 ^
 at scala.sys.package$.error(package.scala:27)
 at org.apache.spark.sql.catalyst.SqlParser.apply(SqlParser.scala:47)
 at org.apache.spark.sql.SQLContext.parseSql(SQLContext.scala:70)
 at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:150)
 at com.testbird.sparkapi.APILearn1$.main(APILearn1.scala:30)
 at com.testbird.sparkapi.APILearn1.main(APILearn1.scala)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

 at
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

 at java.lang.reflect.Method.invoke(Method.java:606)
 at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:292)
 at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
 at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

 any one can help me , thanks

 --