[jira] Updated: (PIG-922) Logical optimizer: push up project
[ https://issues.apache.org/jira/browse/PIG-922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated PIG-922: --- Attachment: PIG-922-p2_preview.patch Include initial patch for phase 2 (inclusive). Need more test, but upload patch first for comments. This patch include these codes: 1. Prune inputs for relational operator 2. Recursively prune inputs from root of a logical plan 3. PigStorage with input pruning Yet to do: 1. Map key pruning for a map input 2. Transformation rule to collect required fields and prune unused columns down Logical optimizer: push up project -- Key: PIG-922 URL: https://issues.apache.org/jira/browse/PIG-922 Project: Pig Issue Type: New Feature Components: impl Affects Versions: 0.3.0 Reporter: Daniel Dai Assignee: Daniel Dai Attachments: PIG-922-p1_0.patch, PIG-922-p1_1.patch, PIG-922-p1_2.patch, PIG-922-p1_3.patch, PIG-922-p1_4.patch, PIG-922-p2_preview.patch This is a continuation work of [PIG-697|https://issues.apache.org/jira/browse/PIG-697]. We need to add another rule to the logical optimizer: Push up project, ie, prune columns as early as possible. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (PIG-934) Merge join implementation currently does not seek to right point on the right side input based on the offset provided by the index
[
https://issues.apache.org/jira/browse/PIG-934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan reassigned PIG-934:
Assignee: Ashutosh Chauhan
Merge join implementation currently does not seek to right point on the right
side input based on the offset provided by the index
--
Key: PIG-934
URL: https://issues.apache.org/jira/browse/PIG-934
Project: Pig
Issue Type: Bug
Affects Versions: 0.3.1
Reporter: Pradeep Kamath
Assignee: Ashutosh Chauhan
Attachments: pig-934.patch
We use POLoad to seek into right file which has the following code:
{noformat}
public void setUp() throws IOException{
String filename = lFile.getFileName();
loader =
(LoadFunc)PigContext.instantiateFuncFromSpec(lFile.getFuncSpec());
is = FileLocalizer.open(filename, pc);
loader.bindTo(filename , new BufferedPositionedInputStream(is),
this.offset, Long.MAX_VALUE);
}
{noformat}
Between opening the stream and bindTo we do not seek to the right offset.
bindTo itself does not perform any seek.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-934) Merge join implementation currently does not seek to right point on the right side input based on the offset provided by the index
[
https://issues.apache.org/jira/browse/PIG-934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12749042#action_12749042
]
Pradeep Kamath commented on PIG-934:
The reason I thought a separate function with a singleFile in the name was
needed was because the current FileLocalizer.open() can handle directories and
hence returns a DataStorageInputStreamIterator which internally iterates over
the underlying multiple streams of the files in the directory. Keeping the same
name may give the impression that the same capability is present even for the
version which seeks to an offset. Seeking to an offset would only work for a
single file - hence maybe have a separate function where the name implies this
restriction might be cleaner.
Merge join implementation currently does not seek to right point on the right
side input based on the offset provided by the index
--
Key: PIG-934
URL: https://issues.apache.org/jira/browse/PIG-934
Project: Pig
Issue Type: Bug
Affects Versions: 0.3.1
Reporter: Pradeep Kamath
Assignee: Ashutosh Chauhan
Attachments: pig-934.patch
We use POLoad to seek into right file which has the following code:
{noformat}
public void setUp() throws IOException{
String filename = lFile.getFileName();
loader =
(LoadFunc)PigContext.instantiateFuncFromSpec(lFile.getFuncSpec());
is = FileLocalizer.open(filename, pc);
loader.bindTo(filename , new BufferedPositionedInputStream(is),
this.offset, Long.MAX_VALUE);
}
{noformat}
Between opening the stream and bindTo we do not seek to the right offset.
bindTo itself does not perform any seek.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (PIG-937) Task get stuck in BasicTable's BTScaner's atEnd() method
Task get stuck in BasicTable's BTScaner's atEnd() method
Key: PIG-937
URL: https://issues.apache.org/jira/browse/PIG-937
Project: Pig
Issue Type: Bug
Reporter: He Yongqiang
It seems is caused by the infinite loop in the code:
BasicTable, Line 698
{noformat}
while (true)
{
int index = random.nextInt(cgScanners.length - 1) + 1;
if (cgScanners[index] != null) {
if (cgScanners[index].atEnd() != ret) {
throw new IOException(
atEnd() failed: Column Groups are not evenly positioned.);
}
break;
}
}
{noformat}
I think it's fine to just use a for loop here, like:
{noformat}
for (int index = 0; index cgScanners.length; index++) {
if (cgScanners[index] != null) {
if (cgScanners[index].atEnd() != ret) {
throw new IOException(
atEnd() failed: Column Groups are not evenly positioned.);
}
break;
}
}
{noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
