Hi Canan, If you mean the trunk branch as https://github.com/apache/pig/tree/trunk/ then no, Spark as execution engine is not available there yet, you'll have to check out the actual 'spark' branch: https://github.com/apache/pig/tree/spark/ Also, what log entries can you see before the exception?
On trunk the full log should be something like: *./pig -x spark_local* Listening for transport dt_socket at address: 5005 16/12/21 09:24:28 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL 16/12/21 09:24:28 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE 16/12/21 09:24:28 INFO pig.ExecTypeProvider: Trying ExecType : TEZ_LOCAL 16/12/21 09:24:28 INFO pig.ExecTypeProvider: Trying ExecType : TEZ 16/12/21 09:24:29 ERROR pig.Main: ERROR 2040: Unknown exec type: spark_local 16/12/21 09:24:29 WARN pig.Main: There is no log file to write to. 16/12/21 09:24:29 ERROR pig.Main: org.apache.pig.PigException: ERROR 2040: Unknown exec type: spark_local ... For the same command on spark branch you should see: *./pig -x spark_local* Listening for transport dt_socket at address: 5005 16/12/21 09:33:14 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL 16/12/21 09:33:14 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE 16/12/21 09:33:14 INFO pig.ExecTypeProvider: Trying ExecType : SPARK 16/12/21 09:33:14 INFO pig.ExecTypeProvider: Trying ExecType : SPARK_LOCAL 16/12/21 09:33:14 INFO pig.ExecTypeProvider: Picked SPARK_LOCAL as the ExecType 2016-12-21 09:33:14,073 [main] INFO org.apache.pig.Main - Apache Pig version 0.16.0-SNAPSHOT (rUnversioned directory) compiled Dec 21 2016, 09:32:12 ... Regards, Adam On 21 December 2016 at 08:20, canan chen <ccn...@gmail.com> wrote: > Thanks Divya, but this blog doesn't help me. I am using pig trunk which > already has spark. > > And I use this command to build pig > > ant clean -Dhadoopversion=23 jar > > > > > On Wed, Dec 21, 2016 at 2:54 PM Divya Gehlot <divya.htco...@gmail.com> > wrote: > > > > > http://blog.cloudera.com/blog/2014/09/pig-is-flying-apache- > pig-on-apache-spark/ > > > > Hope this helps > > > > > > Thanks, > > Divya > > > > On 21 December 2016 at 11:13, canan chen <ccn...@gmail.com> wrote: > > > > > I try to run pig on spark. But hit the following error. Could anyone > help > > > me on that ? And BTW where can I find document about how to use pig on > > > spark. > > > > > > > > > 16/12/21 11:11:44 ERROR pig.Main: org.apache.pig.PigException: ERROR > > 2040: > > > Unknown exec type: SPARK > > > at org.apache.pig.ExecTypeProvider.selectExecType( > > > ExecTypeProvider.java:50) > > > at org.apache.pig.impl.PigContext.<init>(PigContext.java:251) > > > at org.apache.pig.Main.run(Main.java:390) > > > at org.apache.pig.Main.main(Main.java:176) > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > at > > > > > sun.reflect.NativeMethodAccessorImpl.invoke( > NativeMethodAccessorImpl.java: > > > 62) > > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke( > > > DelegatingMethodAccessorImpl.java:43) > > > at java.lang.reflect.Method.invoke(Method.java:497) > > > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > > > at org.apache.hadoop.util.RunJar.main(RunJar.java:136) > > > > > >
