One core segment that frequently asks for systems like Pig & Hive are
analyst who want to deal with data. The key place I see pig fitting in is
getting non-developers deal with data at scale & free up developers to deal
with code, udf rather than manage day to day dataflow changes & updates.
A bypr
>> I've only had a quick look at Pig, but it seems that a declarative
>> layer on top of Spark couldn't be anything other than a big win, as it
>> allows developers to declare *what* they want, permitting the compiler
>> to determine how best poke at the RDD API to implement it.
The devil is in th
On Fri, Apr 25, 2014 at 6:30 AM, Mark Baker wrote:
> I've only had a quick look at Pig, but it seems that a declarative
> layer on top of Spark couldn't be anything other than a big win, as it
> allows developers to declare *what* they want, permitting the compiler
> to determine how best poke at
It depends, personally I have the opposite opinion.
IMO expressing pipelines in a functional language feels natural, you just
have to get used with the language (scala).
Testing spark jobs is easy where testing a Pig script is much harder and
not natural.
If you want a more high level language t
I've only had a quick look at Pig, but it seems that a declarative
layer on top of Spark couldn't be anything other than a big win, as it
allows developers to declare *what* they want, permitting the compiler
to determine how best poke at the RDD API to implement it.
In my brief time with Spark, I
> found the answer - Scala. With Scala's dynamic compilation, we can write
>>>>>> much power constructs than any DSL can provide.
>>>>>>
>>>>>> If I am a new organization and beginning to choose, I would go with
>>>>>> Scala
gt;>>> exec scala "$0" "$@"
>>>>> !#
>>>>> YOUR DSL GOES HERE BUT IN SCALA!
>>>>>
>>>>> You have DSL like scripting, functional and complete language power!
>>>>> If we can improve first 3 line
have DSL like scripting, functional and complete language power! If
>>>> we can improve first 3 lines, here you go, you have most powerful DSL to
>>>> solve data problems.
>>>>
>>>> -Bharath
>>>>
>>>>
>>>>
>>>>
>>
>
>>>
>>>
>>>
>>>
>>> On Mon, Mar 10, 2014 at 11:00 PM, Xiangrui Meng wrote:
>>>
>>>> Hi Sameer,
>>>>
>>>> Lin (cc'ed) could also give you some updates about Pig on Spark
>>>> development on her side.
>>>
r,
>>>
>>> Lin (cc'ed) could also give you some updates about Pig on Spark
>>> development on her side.
>>>
>>> Best,
>>> Xiangrui
>>>
>>> On Mon, Mar 10, 2014 at 12:52 PM, Sameer Tilak wrote:
>>> > Hi Mayur,
&g
>> Best,
>> Xiangrui
>>
>> On Mon, Mar 10, 2014 at 12:52 PM, Sameer Tilak wrote:
>> > Hi Mayur,
>> > We are planning to upgrade our distribution MR1> MR2 (YARN) and the
>> goal is
>> > to get SPROK set up next month. I will kee
eep you posted. Can you please
> keep
> > me informed about your progress as well.
> >
> > ________
> > From: mayur.rust...@gmail.com
> > Date: Mon, 10 Mar 2014 11:47:56 -0700
> >
> > Subject: Re: Pig on Spark
> > To: user@spark.
Hi,
We got spork working on spark 0.9.0
Repository available at:
https://github.com/sigmoidanalytics/pig/tree/spork-hadoopasm-fix
Please suggest your feedback.
-
Lalit Yadav
la...@sigmoidanalytics.com
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Pig
>>>>> Regards
>>>>>> Mayur
>>>>>>
>>>>>> Mayur Rustagi
>>>>>> Ph: +1 (760) 203 3257
>>>>>> http://www.sigmoidanalytics.com
>>>>>> @mayur_rustagi <https://twitter.com/mayur
on, Mar 10, 2014 at 11:00 PM, Xiangrui Meng wrote:
>>>>>
>>>>>> Hi Sameer,
>>>>>>
>>>>>> Lin (cc'ed) could also give you some updates about Pig on Spark
>>>>>> development on her side.
>>>>>>
&g
t; On Mon, Mar 10, 2014 at 11:00 PM, Xiangrui Meng wrote:
>>>>
>>>>> Hi Sameer,
>>>>>
>>>>> Lin (cc'ed) could also give you some updates about Pig on Spark
>>>>> development on her side.
>>>>>
>>>&
Hi,
I have been following Aniket's spork github repository.
https://github.com/aniket486/pig
I have done all the changes mentioned in recently modified pig-spark file.
I am using:
hadoop 2.0.5 alpha
spark-0.8.1-incubating
mesos 0.16.0
##PIG variables
export *HADOOP_CONF_DIR*=$HADOOP_INSTALL/etc/
ates about Pig on Spark
>>>> development on her side.
>>>>
>>>> Best,
>>>> Xiangrui
>>>>
>>>> On Mon, Mar 10, 2014 at 12:52 PM, Sameer Tilak
>>>> wrote:
>>>> > Hi Mayur,
>>>>
to upgrade our distribution MR1> MR2 (YARN) and the
>>> goal is
>>> > to get SPROK set up next month. I will keep you posted. Can you please
>>> keep
>>> > me informed about your progress as well.
>>> >
>>> > __
2:52 PM, Sameer Tilak wrote:
>> > Hi Mayur,
>> > We are planning to upgrade our distribution MR1> MR2 (YARN) and the
>> goal is
>> > to get SPROK set up next month. I will keep you posted. Can you please
>> keep
>> > me informed about your progress a
; From: mayur.rust...@gmail.com
> > Date: Mon, 10 Mar 2014 11:47:56 -0700
> >
> > Subject: Re: Pig on Spark
> > To: user@spark.apache.org
> >
> >
> > Hi Sameer,
> > Did you make any progress on this. My team is also trying it out would
> love
up next month. I will keep you posted. Can you please keep
> me informed about your progress as well.
>
>
> From: mayur.rust...@gmail.com
> Date: Mon, 10 Mar 2014 11:47:56 -0700
>
> Subject: Re: Pig on Spark
> To: user@spark.apache.org
>
&
Hi Mayur,We are planning to upgrade our distribution MR1> MR2 (YARN) and the
goal is to get SPROK set up next month. I will keep you posted. Can you please
keep me informed about your progress as well.
From: mayur.rust...@gmail.com
Date: Mon, 10 Mar 2014 11:47:56 -0700
Subject: Re: Pig on Sp
wrote:
> Hi Aniket,
> Many thanks! I will check this out.
>
> --
> Date: Thu, 6 Mar 2014 13:46:50 -0800
> Subject: Re: Pig on Spark
> From: aniket...@gmail.com
> To: user@spark.apache.org; tgraves...@yahoo.com
>
>
> There is some wo
Hi Aniket,Many thanks! I will check this out.
Date: Thu, 6 Mar 2014 13:46:50 -0800
Subject: Re: Pig on Spark
From: aniket...@gmail.com
To: user@spark.apache.org; tgraves...@yahoo.com
There is some work to make this work on yarn at
https://github.com/aniket486/pig. (So, compile pig with ant
There is some work to make this work on yarn at
https://github.com/aniket486/pig. (So, compile pig with ant
-Dhadoopversion=23)
You can look at https://github.com/aniket486/pig/blob/spork/pig-spark to
find out what sort of env variables you need (sorry, I haven't been able to
clean this up- in-pro
I had asked a similar question on the dev mailing list a while back (Jan 22nd).
See the archives:
http://mail-archives.apache.org/mod_mbox/spark-dev/201401.mbox/browser -> look
for spork.
Basically Matei said:
Yup, that was it, though I believe people at Twitter picked it up again
recently.
27 matches
Mail list logo
