Solved the issue with a Jython UDF.
REGISTER 'lazysimpleserde.py' USING jython AS myfuncs;
A = LOAD '000000_0' using PigStorage('\\u001') AS (params:chararray);
B = FOREACH pixels GENERATE myfuncs.extractMap(params);
@outputSchema("params:map[]")
def extractMap(lazy_map):
extracted = {}
entries = lazy_map.split('\x02')
for entry in entries:
split_entry = entry.split('\x03')
if len(split_entry) == 2:
extracted[split_entry[0]] = split_entry[1]
return extracted
On Tue, Mar 12, 2013 at 4:35 PM, Shawn Hermans <[email protected]>wrote:
> It uses ^A for record separator. That would be easy enough as I could
> just use PigStorage("\001") to pull in the records. The only issue is how
> to extract maps. It uses ^C to separate entires within the map and ^B to
> separate key/value pairs in the map. It wouldn't be too difficult to write
> a UDF to parse the map entries, I was just wondering if there was a
> built-in way of doing that.
>
> Thanks,
> Shawn
>
>
> On Tue, Mar 12, 2013 at 2:53 PM, Dmitriy Ryaboy <[email protected]>wrote:
>
>> How does LazySimpleSerde store data?
>>
>>
>> On Tue, Mar 12, 2013 at 11:17 AM, Shawn Hermans <[email protected]
>> >wrote:
>>
>> > All,
>> > Is there an easy way to read Hive LazySimpleSerde encoded files in Pig?
>> I
>> > did some research and found support for Hive's columnar format and for
>> > SequenceFiles, but did not see anything for LazySimpleSerde.
>> >
>> > Thanks,
>> > Shawn
>> >
>>
>
>
