Hubris aside (mine I mean), I'm probably not qualified to be the last word on
this with regard to advising you on your project's overall direction, so
hopefully others will chime in on this discussion...but that said, when I
started with hadoop a few years ago and our project exceed the capabilities of
direct-Java hadoop interfacing, we briefly entertained the notion of pipes and
learned (to the best of my recollection) that pipes was officially losing its
support within the hadoop community (it may have even been depricated although
I don't remember for sure) and that streaming was being officially advocated as
the replacement (unless I am grossly exaggerating the advice I received at the
time). In fact, the primary reason I abandoned pipes wasn't so much that the
technology itself wouldn't work, but that there was far less community support
for it than for streaming, so whenever I got stuck I couldn't really ask for
help (or I could ask but I would be inquiring a much smaller pool of potential
responders, thus decreasing the scope of any response).
At any rate, I have been quite satisfied with streaming. Gotchas have included
coding from a scratch an entire partially abstract C++ class family to offer a
streaming interface to communicate between hadoop and underlying native
processing, to say nothing of coding up binary/ascii conversion in a variety of
formats to shuffle data across the i/o streams (for reasons I can't recall now,
the binary streaming interface wasn't a good option for us at the time).
Still, all's well.
Best of luck.
On May 21, 2012, at 10:26 , Peter Cogan wrote:
>
> Hi Keith,
>
> Yeah - maybe that was my first mistake. Since I haven't had much luck getting
> pipes working I've started looking at streaming. I am still 'early in my
> hadoop efforts' so now is the time for big course corrections
>
> thanks
> Peter
>
> On 21 May 2012, at 18:11, Keith Wiley wrote:
>
>> Mind if I ask why you prefer pipes to streaming? I abandoned pipes early in
>> my hadoop efforts and went all engines full on streaming.
>>
>> On May 21, 2012, at 04:07 , Peter Cogan wrote:
>>
>>> Hi,
>>>
>>> I am trying to install pipes on os x following the instructions here
________________________________________________________________________________
Keith Wiley kwi...@keithwiley.com keithwiley.com music.keithwiley.com
"The easy confidence with which I know another man's religion is folly teaches
me to suspect that my own is also."
-- Mark Twain
________________________________________________________________________________
