It helps. Thanks.
-----Original Message-----
From: Daniel Dai [mailto:[email protected]]
Sent: Saturday, April 09, 2011 2:39 AM
To: [email protected]
Subject: Re: Dereferencing columns of nested bags
Bag dereference results a bag with less columns. It does not reduce the
nested levels.
$1 refer to visits: {(timestamp: bytearray,visit: {(Key:
chararray,Value: chararray)})}
$1.$1 slice the second column of the bag, all it does is drop timestamp
column from bag "visits". The bag is still there. The schema for $1.$1
is {(visit: {(Key: chararray,Value: chararray)})}. Note the mechanism is
different than tuple. If $1 is a tuple, $1.$1 does reduce one level and
get nested item out.
To reduce the level of a bag, you can only flatten the bag.
Daniel
On 04/08/2011 09:41 AM, Mridul Muralidharan wrote:
> On Friday 08 April 2011 08:16 PM, Badrinarayanan S wrote:
>> I am trying it within foreach as part of generate.
>>
>> I believe the innermost tuple of Key and Value is considered as a single
>> column. So I am able to refer only to $1.$1.($0) which gives the whole
>> tuple. However would it be possible to generate the Key and Value as
>> separate columns as part of foreach.
>>
>> The reference to $1.$1.($0, $1) results in an error like out of bound
>> access.
>
> You are right, it looks pretty broken.
> You can reference $1.$0 but not $1.$1 !
>
> You might want to file a JIRA I guess ...
>
>
>
> If you split it into multiple foreach/flatten invocations, you can get
> to the data you want (but it is not the same functionally since you
> loose record level aggregation that $1.$1.$0 (for ex) provides).
>
>
> Regards,
> Mridul
>
>> Thanks,
>> badri
>>
>> -----Original Message-----
>> From: Mridul Muralidharan [mailto:[email protected]]
>> Sent: Friday, April 08, 2011 4:32 PM
>> To: [email protected]
>> Cc: Badrinarayanan S
>> Subject: Re: Dereferencing columns of nested bags
>>
>>
>> How are you trying to reference it ? Within foreach ? Filter ? Or
>> elsewhere ?
>>
>> Doesn't something like $1.$1.($0, $1) not work to reference key, value
>> as a tuple ?
>>
>>
>> - Mridul
>>
>>
>> On Friday 08 April 2011 03:38 PM, Badrinarayanan S wrote:
>>> Is it possible to dereference a column part of a nested bag. In the
schema
>>> given below, I am trying to dereference the columns Key and Value which
is
>>> part of visit bag which is part of visits bag.
>>>
>>>
>>>
>>> (id, visits:bag{visittuple:tuple(timestamp,
>>> visit:bag{details:tuple(Key:chararray, Value:chararray)})})
>>>
>>>
>>>
>>> From the SVN trunk of Pig I could see a fix for this (PIG-1866, the
fix
>> for
>>> dereferencing a bag within tuple does not work). Does it also addresses
>>> nested bags? If so for the above example can it be dereferenced as
>>> visits.visit.Key. I tried it against the latest trunk, but it failed.
>>>
>>>
>>>
>>> Thanks,
>>>
>>> badri
>>>
>>>
>>>
>>> Disclaimer: This message (including any attachments) is being sent from
>> Fifth Generation Technologies India (P) Ltd. (5G) and may contain
>> information that is proprietary, confidential and privileged. If you are
not
>> the intended recipient, please inform the sender immediately by reply
e-mail
>> and delete this message and attachments from your system, without
retaining
>> a copy. Any unauthorized use or dissemination of this message in whole or
in
>> part is strictly prohibited. 5G shall not be liable for the improper or
>> incomplete transmission of the information contained in this
communication
>> nor for any delay in its receipt or damage to your system. 5G does not
>> guarantee that the integrity of this communication has been maintained
nor
>> that this communication is free of viruses, interceptions or
interference.
>>
>>
>>
>>
>>
>> Disclaimer: This message (including any attachments) is being sent from
Fifth Generation Technologies India (P) Ltd. (5G) and may contain
information that is proprietary, confidential and privileged. If you are not
the intended recipient, please inform the sender immediately by reply e-mail
and delete this message and attachments from your system, without retaining
a copy. Any unauthorized use or dissemination of this message in whole or in
part is strictly prohibited. 5G shall not be liable for the improper or
incomplete transmission of the information contained in this communication
nor for any delay in its receipt or damage to your system. 5G does not
guarantee that the integrity of this communication has been maintained nor
that this communication is free of viruses, interceptions or interference.
>>
Disclaimer: This message (including any attachments) is being sent from Fifth
Generation Technologies India (P) Ltd. (5G) and may contain information that is
proprietary, confidential and privileged. If you are not the intended
recipient, please inform the sender immediately by reply e-mail and delete this
message and attachments from your system, without retaining a copy. Any
unauthorized use or dissemination of this message in whole or in part is
strictly prohibited. 5G shall not be liable for the improper or incomplete
transmission of the information contained in this communication nor for any
delay in its receipt or damage to your system. 5G does not guarantee that the
integrity of this communication has been maintained nor that this communication
is free of viruses, interceptions or interference.
