Re: Spark SQL Parser error

[Yin Huai]

I have not tried it. But, I guess you need to add your credential in the s3
path. Or, can you copy the jar to your driver node and try again?


On Sun, Aug 24, 2014 at 9:35 AM, S Malligarjunan 
wrote:

> Hello Yin,
>
> Additional note:
> In ./bin/spark-shell --jars "s3n:/mybucket/myudf.jar"  I got the following
> message in console.
> Waring skipped external jar..
>
> Thanks and Regards,
> Sankar S.
>
>
>
>   On , S Malligarjunan  wrote:
>
>
>  Hello Yin,
>
> I have tried use sc.addJar and hiveContext.sparkContext.addJar and
> ./bin/spark-shell --jars option,
>
> In all three option when I try to create temporary funtion i get the
> classNotFoundException. What would be the issue here?
>
> Thanks and Regards,
> Sankar S.
>
>
>
>   On Saturday, 23 August 2014, 0:53, Yin Huai 
> wrote:
>
>
> Hello Sankar,
>
> "Add JAR" in SQL is not supported at the moment. We are working on it (
> https://issues.apache.org/jira/browse/SPARK-2219). For now, can you try
> SparkContext.addJar or using "--jars " to launch spark shell?
>
> Thanks,
>
> Yin
>
>
> On Fri, Aug 22, 2014 at 2:01 PM, S Malligarjunan  > wrote:
>
> Hello Yin/All.
>
> @Yin - Thanks for helping. I solved the sql parser error. I am getting the
> following exception now
>
> scala> hiveContext.hql("ADD JAR s3n://hadoop.anonymous.com/lib/myudf.jar
> ");
> warning: there were 1 deprecation warning(s); re-run with -deprecation for
> details
> 14/08/22 17:58:55 INFO SessionState: converting to local s3n://
> hadoop.anonymous.com/lib/myudf.jar
> 14/08/22 17:58:56 ERROR SessionState: Unable to register
> /tmp/3d273a4c-0494-4bec-80fe-86aa56f11684_resources/myudf.jar
> Exception: org.apache.spark.repl.SparkIMain$TranslatingClassLoader cannot
> be cast to java.net.URLClassLoader
> java.lang.ClassCastException:
> org.apache.spark.repl.SparkIMain$TranslatingClassLoader cannot be cast to
> java.net.URLClassLoader
>  at
> org.apache.hadoop.hive.ql.exec.Utilities.addToClassPath(Utilities.java:1680)
>
>
> Thanks and Regards,
> Sankar S.
>
>
>
>   On Friday, 22 August 2014, 22:53, S Malligarjunan
>  wrote:
>
>
>  Hello Yin,
>
> Forgot to mention one thing, the same query works fine in Hive and Shark..
>
> Thanks and Regards,
> Sankar S.
>
>
>
>   On , S Malligarjunan  wrote:
>
>
>  Hello Yin,
>
> I have tried  the create external table command as well. I get the same
> error.
> Please help me to find the root cause.
>
> Thanks and Regards,
> Sankar S.
>
>
>
>   On Friday, 22 August 2014, 22:43, Yin Huai 
> wrote:
>
>
> Hi Sankar,
>
> You need to create an external table in order to specify the location of
> data (i.e. using CREATE EXTERNAL TABLE user1  LOCATION).  You can
> take a look at this page
> 
>  for
> reference.
>
> Thanks,
>
> Yin
>
>
> On Thu, Aug 21, 2014 at 11:12 PM, S Malligarjunan <
> smalligarju...@yahoo.com.invalid> wrote:
>
> Hello All,
>
> When i execute the following query
>
>
> val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>
> CREATE TABLE user1 (time string, id string, u_id string, c_ip string,
> user_agent string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES
> TERMINATED BY '\n' STORED AS TEXTFILE LOCATION 's3n://
> hadoop.anonymous.com/output/qa/cnv_px_ip_gnc/ds=2014-06-14/')
>
> I am getting the following error
> org.apache.spark.sql.hive.HiveQl$ParseException: Failed to parse: CREATE
> TABLE user1 (time string, id string, u_id string, c_ip string, user_agent
> string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ' ' LINES TERMINATED BY
> '
> ' STORED AS TEXTFILE LOCATION 's3n://
> hadoop.anonymous.com/output/qa/cnv_px_ip_gnc/ds=2014-06-14/')
>  at org.apache.spark.sql.hive.HiveQl$.parseSql(HiveQl.scala:215)
> at org.apache.spark.sql.hive.HiveContext.hiveql(HiveContext.scala:98)
>  at org.apache.spark.sql.hive.HiveContext.hql(HiveContext.scala:102)
> at $iwC$$iwC$$iwC$$iwC$$iwC.(:22)
>  at $iwC$$iwC$$iwC$$iwC.(:27)
> at $iwC$$iwC$$iwC.(:29)
>  at $iwC$$iwC.(:31)
> at $iwC.(:33)
>  at (:35)
>
> Kindly let me know what could be the issue here.
>
> I have cloned spark from github. Using Hadoop 1.0.3
>
> Thanks and Regards,
> Sankar S.
>
>
>
>
>
>
>
>
>
>
>
>
>
>

Re: Spark SQL Parser error

[S Malligarjunan]

Hello Yin,

Additional note:
In ./bin/spark-shell --jars "s3n:/mybucket/myudf.jar"  I got the following 
message in console.
Waring skipped external jar..
 
Thanks and Regards,
Sankar S.  



On , S Malligarjunan  wrote:
 


Hello Yin,

I have tried use sc.addJar and hiveContext.sparkContext.addJar and 
./bin/spark-shell --jars option,

In all three option when I try to create temporary funtion i get the 
classNotFoundException. What would be the issue here?
 
Thanks and Regards,
Sankar S.  



On Saturday, 23 August 2014, 0:53, Yin Huai  wrote:
 


Hello Sankar,

"Add JAR" in SQL is not supported at the moment. We are working on it 
(https://issues.apache.org/jira/browse/SPARK-2219). For now, can you try 
SparkContext.addJar or using "--jars " to launch spark shell?

Thanks,

Yin 



On Fri, Aug 22, 2014 at 2:01 PM, S Malligarjunan  
wrote:

Hello Yin/All.
>
>
>@Yin - Thanks for helping. I solved the sql parser error. I am getting the 
>following exception now
>
>
>scala> hiveContext.hql("ADD JAR s3n://hadoop.anonymous.com/lib/myudf.jar");
>warning: there were 1 deprecation warning(s); re-run with -deprecation for 
>details
>14/08/22 17:58:55 INFO SessionState: converting to local 
>s3n://hadoop.anonymous.com/lib/myudf.jar
>14/08/22 17:58:56 ERROR SessionState: Unable to register 
>/tmp/3d273a4c-0494-4bec-80fe-86aa56f11684_resources/myudf.jar
>Exception: org.apache.spark.repl.SparkIMain$TranslatingClassLoader cannot be 
>cast to java.net.URLClassLoader
>java.lang.ClassCastException: 
>org.apache.spark.repl.SparkIMain$TranslatingClassLoader cannot be cast to 
>java.net.URLClassLoader
>at org.apache.hadoop.hive.ql.exec.Utilities.addToClassPath(Utilities.java:1680)
>
>
> 
>Thanks and Regards,
>Sankar S.  
>
>
>
>
>
>On Friday, 22 August 2014, 22:53, S Malligarjunan 
> wrote:
> 
>
>
>Hello Yin,
>
>
>Forgot to mention one thing, the same query works fine in Hive and Shark..
> 
>Thanks and Regards,
>Sankar S.  
>
>
>
>
>
>On , S Malligarjunan  wrote:
> 
>
>
>Hello Yin,
>
>
>I have tried  the create external table command as well. I get the same error.
>Please help me to find the root cause.
> 
>Thanks and Regards,
>Sankar S.  
>
>
>
>
>
>On Friday, 22 August 2014, 22:43, Yin Huai  wrote:
> 
>
>
>Hi Sankar,
>
>
>You need to create an external table in order to specify the location of data 
>(i.e. using CREATE EXTERNAL TABLE user1  LOCATION).  You can take a look 
>at this page for reference. 
>
>
>Thanks,
>
>
>Yin
>
>
>
>On Thu, Aug 21, 2014 at 11:12 PM, S Malligarjunan 
> wrote:
>
>Hello All,
>>
>>
>>When i execute the following query 
>>
>>
>>
>>
>>val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>>
>>
>>CREATE TABLE user1 (time string, id string, u_id string, c_ip string, 
>>user_agent string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES 
>>TERMINATED BY '\n' STORED AS TEXTFILE LOCATION 
>>'s3n://hadoop.anonymous.com/output/qa/cnv_px_ip_gnc/ds=2014-06-14/')
>>
>>
>>I am getting the following error 
>>org.apache.spark.sql.hive.HiveQl$ParseException: Failed to parse: CREATE 
>>TABLE user1 (time string, id string, u_id string, c_ip string, user_agent 
>>string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '' LINES TERMINATED BY '
>>' STORED AS TEXTFILE LOCATION 
>>'s3n://hadoop.anonymous.com/output/qa/cnv_px_ip_gnc/ds=2014-06-14/')
>>at org.apache.spark.sql.hive.HiveQl$.parseSql(HiveQl.scala:215)
>>at org.apache.spark.sql.hive.HiveContext.hiveql(HiveContext.scala:98)
>>at org.apache.spark.sql.hive.HiveContext.hql(HiveContext.scala:102)
>>at $iwC$$iwC$$iwC$$iwC$$iwC.(:22)
>>at $iwC$$iwC$$iwC$$iwC.(:27)
>>at $iwC$$iwC$$iwC.(:29)
>>at $iwC$$iwC.(:31)
>>at $iwC.(:33)
>>at (:35)
>>
>>
>>Kindly let me know what could be the issue here.
>>
>>
>>I have cloned spark from github. Using Hadoop 1.0.3 
>> 
>>Thanks and Regards,
>>Sankar S.  
>>
>>
>
>
>
>
>
>
>

Re: Spark SQL Parser error

[S Malligarjunan]

Hello Yin,

I have tried use sc.addJar and hiveContext.sparkContext.addJar and 
./bin/spark-shell --jars option,

In all three option when I try to create temporary funtion i get the 
classNotFoundException. What would be the issue here?
 
Thanks and Regards,
Sankar S.  



On Saturday, 23 August 2014, 0:53, Yin Huai  wrote:
 


Hello Sankar,

"Add JAR" in SQL is not supported at the moment. We are working on it 
(https://issues.apache.org/jira/browse/SPARK-2219). For now, can you try 
SparkContext.addJar or using "--jars " to launch spark shell?

Thanks,

Yin 



On Fri, Aug 22, 2014 at 2:01 PM, S Malligarjunan  
wrote:

Hello Yin/All.
>
>
>@Yin - Thanks for helping. I solved the sql parser error. I am getting the 
>following exception now
>
>
>scala> hiveContext.hql("ADD JAR s3n://hadoop.anonymous.com/lib/myudf.jar");
>warning: there were 1 deprecation warning(s); re-run with -deprecation for 
>details
>14/08/22 17:58:55 INFO SessionState: converting to local 
>s3n://hadoop.anonymous.com/lib/myudf.jar
>14/08/22 17:58:56 ERROR SessionState: Unable to register 
>/tmp/3d273a4c-0494-4bec-80fe-86aa56f11684_resources/myudf.jar
>Exception: org.apache.spark.repl.SparkIMain$TranslatingClassLoader cannot be 
>cast to java.net.URLClassLoader
>java.lang.ClassCastException: 
>org.apache.spark.repl.SparkIMain$TranslatingClassLoader cannot be cast to 
>java.net.URLClassLoader
>at org.apache.hadoop.hive.ql.exec.Utilities.addToClassPath(Utilities.java:1680)
>
>
> 
>Thanks and Regards,
>Sankar S.  
>
>
>
>
>
>On Friday, 22 August 2014, 22:53, S Malligarjunan 
> wrote:
> 
>
>
>Hello Yin,
>
>
>Forgot to mention one thing, the same query works fine in Hive and Shark..
> 
>Thanks and Regards,
>Sankar S.  
>
>
>
>
>
>On , S Malligarjunan  wrote:
> 
>
>
>Hello Yin,
>
>
>I have tried  the create external table command as well. I get the same error.
>Please help me to find the root cause.
> 
>Thanks and Regards,
>Sankar S.  
>
>
>
>
>
>On Friday, 22 August 2014, 22:43, Yin Huai  wrote:
> 
>
>
>Hi Sankar,
>
>
>You need to create an external table in order to specify the location of data 
>(i.e. using CREATE EXTERNAL TABLE user1  LOCATION).  You can take a look 
>at this page for reference. 
>
>
>Thanks,
>
>
>Yin
>
>
>
>On Thu, Aug 21, 2014 at 11:12 PM, S Malligarjunan 
> wrote:
>
>Hello All,
>>
>>
>>When i execute the following query 
>>
>>
>>
>>
>>val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>>
>>
>>CREATE TABLE user1 (time string, id string, u_id string, c_ip string, 
>>user_agent string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES 
>>TERMINATED BY '\n' STORED AS TEXTFILE LOCATION 
>>'s3n://hadoop.anonymous.com/output/qa/cnv_px_ip_gnc/ds=2014-06-14/')
>>
>>
>>I am getting the following error 
>>org.apache.spark.sql.hive.HiveQl$ParseException: Failed to parse: CREATE 
>>TABLE user1 (time string, id string, u_id string, c_ip string, user_agent 
>>string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '' LINES TERMINATED BY '
>>' STORED AS TEXTFILE LOCATION 
>>'s3n://hadoop.anonymous.com/output/qa/cnv_px_ip_gnc/ds=2014-06-14/')
>>at org.apache.spark.sql.hive.HiveQl$.parseSql(HiveQl.scala:215)
>>at org.apache.spark.sql.hive.HiveContext.hiveql(HiveContext.scala:98)
>>at org.apache.spark.sql.hive.HiveContext.hql(HiveContext.scala:102)
>>at $iwC$$iwC$$iwC$$iwC$$iwC.(:22)
>>at $iwC$$iwC$$iwC$$iwC.(:27)
>>at $iwC$$iwC$$iwC.(:29)
>>at $iwC$$iwC.(:31)
>>at $iwC.(:33)
>>at (:35)
>>
>>
>>Kindly let me know what could be the issue here.
>>
>>
>>I have cloned spark from github. Using Hadoop 1.0.3 
>> 
>>Thanks and Regards,
>>Sankar S.  
>>
>>
>
>
>
>
>
>
>

Re: Spark SQL Parser error

[Yin Huai]

Hello Sankar,

"Add JAR" in SQL is not supported at the moment. We are working on it (
https://issues.apache.org/jira/browse/SPARK-2219). For now, can you try
SparkContext.addJar or using "--jars " to launch spark shell?

Thanks,

Yin


On Fri, Aug 22, 2014 at 2:01 PM, S Malligarjunan 
wrote:

> Hello Yin/All.
>
> @Yin - Thanks for helping. I solved the sql parser error. I am getting the
> following exception now
>
> scala> hiveContext.hql("ADD JAR s3n://hadoop.anonymous.com/lib/myudf.jar
> ");
> warning: there were 1 deprecation warning(s); re-run with -deprecation for
> details
> 14/08/22 17:58:55 INFO SessionState: converting to local s3n://
> hadoop.anonymous.com/lib/myudf.jar
> 14/08/22 17:58:56 ERROR SessionState: Unable to register
> /tmp/3d273a4c-0494-4bec-80fe-86aa56f11684_resources/myudf.jar
> Exception: org.apache.spark.repl.SparkIMain$TranslatingClassLoader cannot
> be cast to java.net.URLClassLoader
> java.lang.ClassCastException:
> org.apache.spark.repl.SparkIMain$TranslatingClassLoader cannot be cast to
> java.net.URLClassLoader
>  at
> org.apache.hadoop.hive.ql.exec.Utilities.addToClassPath(Utilities.java:1680)
>
>
> Thanks and Regards,
> Sankar S.
>
>
>
>   On Friday, 22 August 2014, 22:53, S Malligarjunan
>  wrote:
>
>
>  Hello Yin,
>
> Forgot to mention one thing, the same query works fine in Hive and Shark..
>
> Thanks and Regards,
> Sankar S.
>
>
>
>   On , S Malligarjunan  wrote:
>
>
>  Hello Yin,
>
> I have tried  the create external table command as well. I get the same
> error.
> Please help me to find the root cause.
>
> Thanks and Regards,
> Sankar S.
>
>
>
>   On Friday, 22 August 2014, 22:43, Yin Huai 
> wrote:
>
>
> Hi Sankar,
>
> You need to create an external table in order to specify the location of
> data (i.e. using CREATE EXTERNAL TABLE user1  LOCATION).  You can
> take a look at this page
> 
>  for
> reference.
>
> Thanks,
>
> Yin
>
>
> On Thu, Aug 21, 2014 at 11:12 PM, S Malligarjunan <
> smalligarju...@yahoo.com.invalid> wrote:
>
> Hello All,
>
> When i execute the following query
>
>
> val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>
> CREATE TABLE user1 (time string, id string, u_id string, c_ip string,
> user_agent string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES
> TERMINATED BY '\n' STORED AS TEXTFILE LOCATION 's3n://
> hadoop.anonymous.com/output/qa/cnv_px_ip_gnc/ds=2014-06-14/')
>
> I am getting the following error
> org.apache.spark.sql.hive.HiveQl$ParseException: Failed to parse: CREATE
> TABLE user1 (time string, id string, u_id string, c_ip string, user_agent
> string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ' ' LINES TERMINATED BY
> '
> ' STORED AS TEXTFILE LOCATION 's3n://
> hadoop.anonymous.com/output/qa/cnv_px_ip_gnc/ds=2014-06-14/')
>  at org.apache.spark.sql.hive.HiveQl$.parseSql(HiveQl.scala:215)
> at org.apache.spark.sql.hive.HiveContext.hiveql(HiveContext.scala:98)
>  at org.apache.spark.sql.hive.HiveContext.hql(HiveContext.scala:102)
> at $iwC$$iwC$$iwC$$iwC$$iwC.(:22)
>  at $iwC$$iwC$$iwC$$iwC.(:27)
> at $iwC$$iwC$$iwC.(:29)
>  at $iwC$$iwC.(:31)
> at $iwC.(:33)
>  at (:35)
>
> Kindly let me know what could be the issue here.
>
> I have cloned spark from github. Using Hadoop 1.0.3
>
> Thanks and Regards,
> Sankar S.
>
>
>
>
>
>
>
>
>

Re: Spark SQL Parser error

[S Malligarjunan]

Hello Yin/All.

@Yin - Thanks for helping. I solved the sql parser error. I am getting the 
following exception now

scala> hiveContext.hql("ADD JAR s3n://hadoop.anonymous.com/lib/myudf.jar");
warning: there were 1 deprecation warning(s); re-run with -deprecation for 
details
14/08/22 17:58:55 INFO SessionState: converting to local 
s3n://hadoop.anonymous.com/lib/myudf.jar
14/08/22 17:58:56 ERROR SessionState: Unable to register 
/tmp/3d273a4c-0494-4bec-80fe-86aa56f11684_resources/myudf.jar
Exception: org.apache.spark.repl.SparkIMain$TranslatingClassLoader cannot be 
cast to java.net.URLClassLoader
java.lang.ClassCastException: 
org.apache.spark.repl.SparkIMain$TranslatingClassLoader cannot be cast to 
java.net.URLClassLoader
at org.apache.hadoop.hive.ql.exec.Utilities.addToClassPath(Utilities.java:1680)

 
Thanks and Regards,
Sankar S.  



On Friday, 22 August 2014, 22:53, S Malligarjunan 
 wrote:
 


Hello Yin,

Forgot to mention one thing, the same query works fine in Hive and Shark..
 
Thanks and Regards,
Sankar S.  



On , S Malligarjunan  wrote:
 


Hello Yin,

I have tried  the create external table command as well. I get the same error.
Please help me to find the root cause.
 
Thanks and Regards,
Sankar S.  



On Friday, 22 August 2014, 22:43, Yin Huai  wrote:
 


Hi Sankar,

You need to create an external table in order to specify the location of data 
(i.e. using CREATE EXTERNAL TABLE user1  LOCATION).  You can take a look at 
this page for reference. 

Thanks,

Yin



On Thu, Aug 21, 2014 at 11:12 PM, S Malligarjunan 
 wrote:

Hello All,
>
>
>When i execute the following query 
>
>
>
>
>val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>
>
>CREATE TABLE user1 (time string, id string, u_id string, c_ip string, 
>user_agent string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES 
>TERMINATED BY '\n' STORED AS TEXTFILE LOCATION 
>'s3n://hadoop.anonymous.com/output/qa/cnv_px_ip_gnc/ds=2014-06-14/')
>
>
>I am getting the following error 
>org.apache.spark.sql.hive.HiveQl$ParseException: Failed to parse: CREATE TABLE 
>user1 (time string, id string, u_id string, c_ip string, user_agent string) 
>ROW FORMAT DELIMITED FIELDS TERMINATED BY '' LINES TERMINATED BY '
>' STORED AS TEXTFILE LOCATION 
>'s3n://hadoop.anonymous.com/output/qa/cnv_px_ip_gnc/ds=2014-06-14/')
>at org.apache.spark.sql.hive.HiveQl$.parseSql(HiveQl.scala:215)
>at org.apache.spark.sql.hive.HiveContext.hiveql(HiveContext.scala:98)
>at org.apache.spark.sql.hive.HiveContext.hql(HiveContext.scala:102)
>at $iwC$$iwC$$iwC$$iwC$$iwC.(:22)
>at $iwC$$iwC$$iwC$$iwC.(:27)
>at $iwC$$iwC$$iwC.(:29)
>at $iwC$$iwC.(:31)
>at $iwC.(:33)
>at (:35)
>
>
>Kindly let me know what could be the issue here.
>
>
>I have cloned spark from github. Using Hadoop 1.0.3 
> 
>Thanks and Regards,
>Sankar S.  
>
>

Re: Spark SQL Parser error

[S Malligarjunan]

Hello Yin,

Forgot to mention one thing, the same query works fine in Hive and Shark..
 
Thanks and Regards,
Sankar S.  



On , S Malligarjunan  wrote:
 


Hello Yin,

I have tried  the create external table command as well. I get the same error.
Please help me to find the root cause.
 
Thanks and Regards,
Sankar S.  



On Friday, 22 August 2014, 22:43, Yin Huai  wrote:
 


Hi Sankar,

You need to create an external table in order to specify the location of data 
(i.e. using CREATE EXTERNAL TABLE user1  LOCATION).  You can take a look at 
this page for reference. 

Thanks,

Yin



On Thu, Aug 21, 2014 at 11:12 PM, S Malligarjunan 
 wrote:

Hello All,
>
>
>When i execute the following query 
>
>
>
>
>val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>
>
>CREATE TABLE user1 (time string, id string, u_id string, c_ip string, 
>user_agent string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES 
>TERMINATED BY '\n' STORED AS TEXTFILE LOCATION 
>'s3n://hadoop.anonymous.com/output/qa/cnv_px_ip_gnc/ds=2014-06-14/')
>
>
>I am getting the following error 
>org.apache.spark.sql.hive.HiveQl$ParseException: Failed to parse: CREATE TABLE 
>user1 (time string, id string, u_id string, c_ip string, user_agent string) 
>ROW FORMAT DELIMITED FIELDS TERMINATED BY '' LINES TERMINATED BY '
>' STORED AS TEXTFILE LOCATION 
>'s3n://hadoop.anonymous.com/output/qa/cnv_px_ip_gnc/ds=2014-06-14/')
>at org.apache.spark.sql.hive.HiveQl$.parseSql(HiveQl.scala:215)
>at org.apache.spark.sql.hive.HiveContext.hiveql(HiveContext.scala:98)
>at org.apache.spark.sql.hive.HiveContext.hql(HiveContext.scala:102)
>at $iwC$$iwC$$iwC$$iwC$$iwC.(:22)
>at $iwC$$iwC$$iwC$$iwC.(:27)
>at $iwC$$iwC$$iwC.(:29)
>at $iwC$$iwC.(:31)
>at $iwC.(:33)
>at (:35)
>
>
>Kindly let me know what could be the issue here.
>
>
>I have cloned spark from github. Using Hadoop 1.0.3 
> 
>Thanks and Regards,
>Sankar S.  
>
>

Re: Spark SQL Parser error

[S Malligarjunan]

Hello Yin,

I have tried  the create external table command as well. I get the same error.
Please help me to find the root cause.
 
Thanks and Regards,
Sankar S.  



On Friday, 22 August 2014, 22:43, Yin Huai  wrote:
 


Hi Sankar,

You need to create an external table in order to specify the location of data 
(i.e. using CREATE EXTERNAL TABLE user1  LOCATION).  You can take a look at 
this page for reference. 

Thanks,

Yin



On Thu, Aug 21, 2014 at 11:12 PM, S Malligarjunan 
 wrote:

Hello All,
>
>
>When i execute the following query 
>
>
>
>
>val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>
>
>CREATE TABLE user1 (time string, id string, u_id string, c_ip string, 
>user_agent string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES 
>TERMINATED BY '\n' STORED AS TEXTFILE LOCATION 
>'s3n://hadoop.anonymous.com/output/qa/cnv_px_ip_gnc/ds=2014-06-14/')
>
>
>I am getting the following error 
>org.apache.spark.sql.hive.HiveQl$ParseException: Failed to parse: CREATE TABLE 
>user1 (time string, id string, u_id string, c_ip string, user_agent string) 
>ROW FORMAT DELIMITED FIELDS TERMINATED BY '' LINES TERMINATED BY '
>' STORED AS TEXTFILE LOCATION 
>'s3n://hadoop.anonymous.com/output/qa/cnv_px_ip_gnc/ds=2014-06-14/')
>at org.apache.spark.sql.hive.HiveQl$.parseSql(HiveQl.scala:215)
>at org.apache.spark.sql.hive.HiveContext.hiveql(HiveContext.scala:98)
>at org.apache.spark.sql.hive.HiveContext.hql(HiveContext.scala:102)
>at $iwC$$iwC$$iwC$$iwC$$iwC.(:22)
>at $iwC$$iwC$$iwC$$iwC.(:27)
>at $iwC$$iwC$$iwC.(:29)
>at $iwC$$iwC.(:31)
>at $iwC.(:33)
>at (:35)
>
>
>Kindly let me know what could be the issue here.
>
>
>I have cloned spark from github. Using Hadoop 1.0.3 
> 
>Thanks and Regards,
>Sankar S.  
>
>

Re: Spark SQL Parser error

[Yin Huai]

Hi Sankar,

You need to create an external table in order to specify the location of
data (i.e. using CREATE EXTERNAL TABLE user1  LOCATION).  You can take
a look at this page

for
reference.

Thanks,

Yin


On Thu, Aug 21, 2014 at 11:12 PM, S Malligarjunan <
smalligarju...@yahoo.com.invalid> wrote:

> Hello All,
>
> When i execute the following query
>
>
> val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>
> CREATE TABLE user1 (time string, id string, u_id string, c_ip string,
> user_agent string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES
> TERMINATED BY '\n' STORED AS TEXTFILE LOCATION 's3n://
> hadoop.anonymous.com/output/qa/cnv_px_ip_gnc/ds=2014-06-14/')
>
> I am getting the following error
> org.apache.spark.sql.hive.HiveQl$ParseException: Failed to parse: CREATE
> TABLE user1 (time string, id string, u_id string, c_ip string, user_agent
> string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ' ' LINES TERMINATED BY
> '
> ' STORED AS TEXTFILE LOCATION 's3n://
> hadoop.anonymous.com/output/qa/cnv_px_ip_gnc/ds=2014-06-14/')
>  at org.apache.spark.sql.hive.HiveQl$.parseSql(HiveQl.scala:215)
> at org.apache.spark.sql.hive.HiveContext.hiveql(HiveContext.scala:98)
>  at org.apache.spark.sql.hive.HiveContext.hql(HiveContext.scala:102)
> at $iwC$$iwC$$iwC$$iwC$$iwC.(:22)
>  at $iwC$$iwC$$iwC$$iwC.(:27)
> at $iwC$$iwC$$iwC.(:29)
>  at $iwC$$iwC.(:31)
> at $iwC.(:33)
>  at (:35)
>
> Kindly let me know what could be the issue here.
>
> I have cloned spark from github. Using Hadoop 1.0.3
>
> Thanks and Regards,
> Sankar S.
>
>

Spark SQL Parser error

[S Malligarjunan]

Hello All,

When i execute the following query 


val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)

CREATE TABLE user1 (time string, id string, u_id string, c_ip string, 
user_agent string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES 
TERMINATED BY '\n' STORED AS TEXTFILE LOCATION 
's3n://hadoop.anonymous.com/output/qa/cnv_px_ip_gnc/ds=2014-06-14/')

I am getting the following error 
org.apache.spark.sql.hive.HiveQl$ParseException: Failed to parse: CREATE TABLE 
user1 (time string, id string, u_id string, c_ip string, user_agent string) ROW 
FORMAT DELIMITED FIELDS TERMINATED BY '' LINES TERMINATED BY '
' STORED AS TEXTFILE LOCATION 
's3n://hadoop.anonymous.com/output/qa/cnv_px_ip_gnc/ds=2014-06-14/')
at org.apache.spark.sql.hive.HiveQl$.parseSql(HiveQl.scala:215)
at org.apache.spark.sql.hive.HiveContext.hiveql(HiveContext.scala:98)
at org.apache.spark.sql.hive.HiveContext.hql(HiveContext.scala:102)
at $iwC$$iwC$$iwC$$iwC$$iwC.(:22)
at $iwC$$iwC$$iwC$$iwC.(:27)
at $iwC$$iwC$$iwC.(:29)
at $iwC$$iwC.(:31)
at $iwC.(:33)
at (:35)

Kindly let me know what could be the issue here.

I have cloned spark from github. Using Hadoop 1.0.3 
 
Thanks and Regards,
Sankar S.