Hi, FYI:
The project http://code.google.com/p/hadoop-gpl-packing/ is an effort to provide all the lzo and elephant bird code into one rpm or deb for x86_64, amd64-64 and i386-32, I've created this for use at my current work place and we've decided to open source and share this. Normally to start with lzo you need to compile and package different projects together, this project makes the task easier, just download and run rpm -i or dpkg. The libraries get installed into /opt/hadoopgpl from there you reference it from your pig scripts. The newest release contains pig-0.8.0 elephant bird compiled for branches (1) https://github.com/dvryaboy/elephant-bird (2)https://github.com/hirohanin/elephant-bird/ (3)https://github.com/gerritjvv/elephant-bird Some confusion might arise while these are being developed and pulled back into the original branch, but the current state as I understand is: Branch 1 contains all from (2). Branch 3 contains a merge from 1 and 2 with some extra classes I'm using at my work place. We're using elephant-bird lzo and GPB with pig-0.8.0 and its working without problems so far. Also on the home page of this project I've tried to provide easy to follow instructions of how to link in and configure pig, hadoop for use with lzo and GPB. Cheers, Gerrit -----Original Message----- From: Evert Lammerts [mailto:[email protected]] Sent: Thursday, January 13, 2011 8:19 PM To: [email protected]; [email protected] Subject: Re: LZO & Pig (Elephantbird?) Alright, thanks! I'm going to give that a try and will let you know how it goes. Cheers, Evert ----- Reply message ----- From: "Dmitriy Ryaboy" <[email protected]> Date: Thu, Jan 13, 2011 8:27 pm Subject: LZO & Pig (Elephantbird?) To: "[email protected]" <[email protected]> Depends on your definition of ready. I haven't put it into production and there's a bit of clean-up left, but as far as I know there are no critical bugs; just some interface issues which are shared between the current master and the version for Pig 0.8. Pig 0.8 should work with Hadoop 0.20.1; I've been working with it under CDH2 (which is a patched 0.20.1). -Dmitriy On Thu, Jan 13, 2011 at 7:43 AM, Evert Lammerts <[email protected]>wrote: > > Are you trying to load protobufs or thrift files, or do you just want > > Lzo > > support? > > Protobufs would be nice, but Elephant Bird is not ready yet for Pig 0.7 / > 0.8, right? > > > If you just want plain text lzo loading, the loaders in the > > pig-08 > > branch totally work. > > Does Pig 0.8 work with Hadoop 0.20.1 as well? > > Thanks for the support! > Evert > > > > > Let me know if you have any issues. > > > > D > > > > On Wed, Jan 12, 2011 at 12:23 PM, Tyler Coffin <[email protected]> wrote: > > > > > There's a fork of elephant-bird where pig-8 support is being worked > > on: > > > https://github.com/dvryaboy/elephant-bird/tree/pig-08 > > > > > > I haven't given it a shot yet. > > > > > > -----Original Message----- > > > From: Evert Lammerts [mailto:[email protected]] > > > Sent: January 12, 2011 15:10 > > > To: '[email protected]' > > > Subject: LZO & Pig (Elephantbird?) > > > > > > Hello list, > > > > > > I've installed the LZO codecs (https://github.com/kevinweil/hadoop- > > lzo) > > > and > > > now I'm looking into using LZO in Pig. Elephant Bird > > > (https://github.com/kevinweil/elephant-bird) seems to provide some > > nice > > > prefab loaders, but it's requirements do not fit out Hadoop > > installation > > > (we're on CDH3b2 with Pig 0.7, EB cannot be used with anything > > > 0.6). Also > > > the need for Thrift 0.2 is unclear to me - Thrift is now at 0.5. > > > > > > Now I did find this project, http://code.google.com/p/hadoop-gpl- > > packing/, > > > saying EB can handle even Pig 0.8. This confuses me - can I or can I > > not > > > use > > > Elephant Bird with Pig 0.7, or even upgrade to Pig 0.8? > > > > > > Since EB is probably not an option, does anybody have some pointers > > on how > > > to use LZO'ed files with Pig? > > > > > > Thanks! > > > > > > Evert Lammerts > > > > > > --------------------------------------------------------------------- > > > This transmission (including any attachments) may contain > > confidential > > > information, privileged material (including material protected by the > > > solicitor-client or other applicable privileges), or constitute non- > > public > > > information. Any use of this information by anyone other than the > > intended > > > recipient is prohibited. If you have received this transmission in > > error, > > > please immediately reply to the sender and delete this information > > from your > > > system. Use, dissemination, distribution, or reproduction of this > > > transmission by unintended recipients is not authorized and may be > > unlawful. > > > >
