[jira] [Commented] (PIG-3108) HBaseStorage returns empty maps when mixing wildcard- with other columns
[
https://issues.apache.org/jira/browse/PIG-3108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785792#comment-13785792
]
Harsh J commented on PIG-3108:
--
Moved from Release Notes to comments:
bq. Tested and committed. Thanks for the patch Christoph and sorry for the
delay!
HBaseStorage returns empty maps when mixing wildcard- with other columns
Key: PIG-3108
URL: https://issues.apache.org/jira/browse/PIG-3108
Project: Pig
Issue Type: Bug
Affects Versions: 0.9.0, 0.9.1, 0.9.2, 0.10.0, 0.11, 0.10.1, 0.12.0
Reporter: Christoph Bauer
Assignee: Christoph Bauer
Fix For: 0.12.0
Attachments: PIG-3108.patch, PIG-3108.patch
Consider the following:
A and B should be the same (with different order, of course).
{code}
/*
in hbase shell:
create 'pigtest', 'pig'
put 'pigtest' , '1', 'pig:name', 'A'
put 'pigtest' , '1', 'pig:has_legs', 'true'
put 'pigtest' , '1', 'pig:has_ribs', 'true'
*/
A = LOAD 'hbase://pigtest' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage('pig:name pig:has*') AS
(name:chararray,parts);
B = LOAD 'hbase://pigtest' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage('pig:has* pig:name') AS
(parts,name:chararray);
dump A;
dump B;
{code}
This is due to a bug in setLocation and initScan.
For _A_
# scan.addColumn(pig,name); // for 'pig:name'
# scan.addFamily(pig); // for the 'pig:has*'
So that's silently right.
But for _B_
# scan.addFamily(pig)
# scan.addColumn(pig,name)
will override the first call to addFamily, because you cannot mix them on the
same family.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (PIG-3108) HBaseStorage returns empty maps when mixing wildcard- with other columns
[
https://issues.apache.org/jira/browse/PIG-3108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13552090#comment-13552090
]
Bill Graham commented on PIG-3108:
--
Thanks Cristoph this patch looks good. I'm on vaca for a few days but after
that I will test this before committing.
Let's limit PIG-3067 to only refactoring the code without any new features,
improvements or bug fixes.
HBaseStorage returns empty maps when mixing wildcard- with other columns
Key: PIG-3108
URL: https://issues.apache.org/jira/browse/PIG-3108
Project: Pig
Issue Type: Bug
Affects Versions: 0.9.0, 0.9.1, 0.9.2, 0.10.0, 0.11, 0.10.1, 0.12
Reporter: Christoph Bauer
Assignee: Christoph Bauer
Fix For: 0.12
Attachments: PIG-3108.patch, PIG-3108.patch
Consider the following:
A and B should be the same (with different order, of course).
{code}
/*
in hbase shell:
create 'pigtest', 'pig'
put 'pigtest' , '1', 'pig:name', 'A'
put 'pigtest' , '1', 'pig:has_legs', 'true'
put 'pigtest' , '1', 'pig:has_ribs', 'true'
*/
A = LOAD 'hbase://pigtest' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage('pig:name pig:has*') AS
(name:chararray,parts);
B = LOAD 'hbase://pigtest' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage('pig:has* pig:name') AS
(parts,name:chararray);
dump A;
dump B;
{code}
This is due to a bug in setLocation and initScan.
For _A_
# scan.addColumn(pig,name); // for 'pig:name'
# scan.addFamily(pig); // for the 'pig:has*'
So that's silently right.
But for _B_
# scan.addFamily(pig)
# scan.addColumn(pig,name)
will override the first call to addFamily, because you cannot mix them on the
same family.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-3108) HBaseStorage returns empty maps when mixing wildcard- with other columns
[
https://issues.apache.org/jira/browse/PIG-3108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13545737#comment-13545737
]
Christoph Bauer commented on PIG-3108:
--
I will generate a new patch.
The reason addFamily/addColumn is in {{setLocation}} is that {{pushProjection}}
might have been called leading to a reduced number of columns actually needed
in the scan.
I looked around but haven't found a better place for the scanColumns code.
HBaseStorage returns empty maps when mixing wildcard- with other columns
Key: PIG-3108
URL: https://issues.apache.org/jira/browse/PIG-3108
Project: Pig
Issue Type: Bug
Affects Versions: 0.9.0, 0.9.1, 0.9.2, 0.10.0, 0.11, 0.10.1, 0.12
Reporter: Christoph Bauer
Assignee: Christoph Bauer
Fix For: 0.12
Attachments: PIG-3108.patch
Consider the following:
A and B should be the same (with different order, of course).
{code}
/*
in hbase shell:
create 'pigtest', 'pig'
put 'pigtest' , '1', 'pig:name', 'A'
put 'pigtest' , '1', 'pig:has_legs', 'true'
put 'pigtest' , '1', 'pig:has_ribs', 'true'
*/
A = LOAD 'hbase://pigtest' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage('pig:name pig:has*') AS
(name:chararray,parts);
B = LOAD 'hbase://pigtest' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage('pig:has* pig:name') AS
(parts,name:chararray);
dump A;
dump B;
{code}
This is due to a bug in setLocation and initScan.
For _A_
# scan.addColumn(pig,name); // for 'pig:name'
# scan.addFamily(pig); // for the 'pig:has*'
So that's silently right.
But for _B_
# scan.addFamily(pig)
# scan.addColumn(pig,name)
will override the first call to addFamily, because you cannot mix them on the
same family.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-3108) HBaseStorage returns empty maps when mixing wildcard- with other columns
[
https://issues.apache.org/jira/browse/PIG-3108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13545741#comment-13545741
]
Christoph Bauer commented on PIG-3108:
--
Come to think of it.
column filter initialization should also be postponed if possible.
I will/did adress that in https://issues.apache.org/jira/browse/PIG-3067 if you
agree.
HBaseStorage returns empty maps when mixing wildcard- with other columns
Key: PIG-3108
URL: https://issues.apache.org/jira/browse/PIG-3108
Project: Pig
Issue Type: Bug
Affects Versions: 0.9.0, 0.9.1, 0.9.2, 0.10.0, 0.11, 0.10.1, 0.12
Reporter: Christoph Bauer
Assignee: Christoph Bauer
Fix For: 0.12
Attachments: PIG-3108.patch
Consider the following:
A and B should be the same (with different order, of course).
{code}
/*
in hbase shell:
create 'pigtest', 'pig'
put 'pigtest' , '1', 'pig:name', 'A'
put 'pigtest' , '1', 'pig:has_legs', 'true'
put 'pigtest' , '1', 'pig:has_ribs', 'true'
*/
A = LOAD 'hbase://pigtest' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage('pig:name pig:has*') AS
(name:chararray,parts);
B = LOAD 'hbase://pigtest' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage('pig:has* pig:name') AS
(parts,name:chararray);
dump A;
dump B;
{code}
This is due to a bug in setLocation and initScan.
For _A_
# scan.addColumn(pig,name); // for 'pig:name'
# scan.addFamily(pig); // for the 'pig:has*'
So that's silently right.
But for _B_
# scan.addFamily(pig)
# scan.addColumn(pig,name)
will override the first call to addFamily, because you cannot mix them on the
same family.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-3108) HBaseStorage returns empty maps when mixing wildcard- with other columns
[
https://issues.apache.org/jira/browse/PIG-3108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13545642#comment-13545642
]
Bill Graham commented on PIG-3108:
--
Got it, thanks for the clarification. A few comments then:
1. Yes, we should rename {{addFiltersWithoutColumnPrefix}} to {{addScans}},
since the filter part is misleading and it seems we can always call that method
as implemented.
2. Instead of calling {{addFiltersWithoutColumnPrefix}} from {{setLocation}}
let's just remove the addFamily/addColumn block from {{setLocation}}. Then in
{{initScan()}} we can handle both scans and filters in one place with something
like this (to replace the existing conditional):
{noformat}
addScans(columnInfo_);
if (!columnPrefixExists) {
addFiltersWithoutColumnPrefix(columnInfo_);
}
{noformat}
3. Would you please update the javadocs in {{addFiltersWithoutColumnPrefix}}
(or {{addScan}}) to describe the new logic as best you can. This section of the
filter/scan code has been particularly nasty in the past so we should be as
clear as possible about what's happening here.
HBaseStorage returns empty maps when mixing wildcard- with other columns
Key: PIG-3108
URL: https://issues.apache.org/jira/browse/PIG-3108
Project: Pig
Issue Type: Bug
Affects Versions: 0.9.0, 0.9.1, 0.9.2, 0.10.0, 0.11, 0.10.1, 0.12
Reporter: Christoph Bauer
Fix For: 0.12
Attachments: PIG-3108.patch
Consider the following:
A and B should be the same (with different order, of course).
{code}
/*
in hbase shell:
create 'pigtest', 'pig'
put 'pigtest' , '1', 'pig:name', 'A'
put 'pigtest' , '1', 'pig:has_legs', 'true'
put 'pigtest' , '1', 'pig:has_ribs', 'true'
*/
A = LOAD 'hbase://pigtest' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage('pig:name pig:has*') AS
(name:chararray,parts);
B = LOAD 'hbase://pigtest' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage('pig:has* pig:name') AS
(parts,name:chararray);
dump A;
dump B;
{code}
This is due to a bug in setLocation and initScan.
For _A_
# scan.addColumn(pig,name); // for 'pig:name'
# scan.addFamily(pig); // for the 'pig:has*'
So that's silently right.
But for _B_
# scan.addFamily(pig)
# scan.addColumn(pig,name)
will override the first call to addFamily, because you cannot mix them on the
same family.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-3108) HBaseStorage returns empty maps when mixing wildcard- with other columns
[
https://issues.apache.org/jira/browse/PIG-3108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13543751#comment-13543751
]
Christoph Bauer commented on PIG-3108:
--
Sorry. I've been with this code too long. I will try to explain.
addFiltersWithColumnPrefix and addFiltersWithoutColumnPrefix actually do
different things:
- addFiltersWithColumnPrefix creates HBase scan filters
- addFiltersWithoutColumnPrefix tells the scan object the families and
columns to retrieve. This is much quicker than adding filters, thats why it
was changed.
The thing is: the scan object should always be limited to the
family/columns needed to speed things up. In fact we did this already - see
setLocation (it's basicly the same as in addFiltersWithoutColumnPrefix).
So what I did was to replace the code in setLocation with a call to
addFiltersWithoutColumnPrefix
To make things clear, we could remove addFiltersWithoutColumnPrefix from
the if/else in initScan() ()setLocation will called anyway) and rename it
to setScanColumns or something.
2013/1/3 Bill Graham (JIRA) j...@apache.org
HBaseStorage returns empty maps when mixing wildcard- with other columns
Key: PIG-3108
URL: https://issues.apache.org/jira/browse/PIG-3108
Project: Pig
Issue Type: Bug
Affects Versions: 0.9.0, 0.9.1, 0.9.2, 0.10.0, 0.11, 0.10.1, 0.12
Reporter: Christoph Bauer
Fix For: 0.12
Attachments: PIG-3108.patch
Consider the following:
A and B should be the same (with different order, of course).
{code}
/*
in hbase shell:
create 'pigtest', 'pig'
put 'pigtest' , '1', 'pig:name', 'A'
put 'pigtest' , '1', 'pig:has_legs', 'true'
put 'pigtest' , '1', 'pig:has_ribs', 'true'
*/
A = LOAD 'hbase://pigtest' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage('pig:name pig:has*') AS
(name:chararray,parts);
B = LOAD 'hbase://pigtest' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage('pig:has* pig:name') AS
(parts,name:chararray);
dump A;
dump B;
{code}
This is due to a bug in setLocation and initScan.
For _A_
# scan.addColumn(pig,name); // for 'pig:name'
# scan.addFamily(pig); // for the 'pig:has*'
So that's silently right.
But for _B_
# scan.addFamily(pig)
# scan.addColumn(pig,name)
will override the first call to addFamily, because you cannot mix them on the
same family.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-3108) HBaseStorage returns empty maps when mixing wildcard- with other columns
[
https://issues.apache.org/jira/browse/PIG-3108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13543237#comment-13543237
]
Bill Graham commented on PIG-3108:
--
Thanks for digging into this one. I'm a confused either by the example or the
implementation. The example seems to point out a bug when there are column
descriptors with prefixes (i.e. {{has*}} or {{name*}}), but the fix is in
{{addFiltersWithoutColumnPrefix}} which should only apply when there is no
column with a descriptor. This is an intentional fork in how filters and/or
scans are used which happens in this block of code:
{noformat}
if (!columnPrefixExists) {
addFiltersWithoutColumnPrefix(columnInfo_);
}
else {
addFiltersWithColumnPrefix(columnInfo_);
}
{noformat}
HBaseStorage returns empty maps when mixing wildcard- with other columns
Key: PIG-3108
URL: https://issues.apache.org/jira/browse/PIG-3108
Project: Pig
Issue Type: Bug
Affects Versions: 0.9.0, 0.9.1, 0.9.2, 0.10.0, 0.11, 0.10.1, 0.12
Reporter: Christoph Bauer
Fix For: 0.12
Attachments: PIG-3108.patch
Consider the following:
A and B should be the same (with different order, of course).
{code}
/*
in hbase shell:
create 'pigtest', 'pig'
put 'pigtest' , '1', 'pig:name', 'A'
put 'pigtest' , '1', 'pig:has_legs', 'true'
put 'pigtest' , '1', 'pig:has_ribs', 'true'
*/
A = LOAD 'hbase://pigtest' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage('pig:name pig:has*') AS
(name:chararray,parts);
B = LOAD 'hbase://pigtest' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage('pig:has* pig:name') AS
(parts,name:chararray);
dump A;
dump B;
{code}
This is due to a bug in setLocation and initScan.
For _A_
# scan.addColumn(pig,name); // for 'pig:name'
# scan.addFamily(pig); // for the 'pig:has*'
So that's silently right.
But for _B_
# scan.addFamily(pig)
# scan.addColumn(pig,name)
will override the first call to addFamily, because you cannot mix them on the
same family.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
