[
https://issues.apache.org/jira/browse/PIG-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Advertising
Gunther Hagleitner updated PIG-627:
-----------------------------------
Attachment: file_cmds-0305.patch
This patch is for the multi query branch again. It mostly fixes the problem
with certain commands in the script that require immediate execution (in batch
mode).
So if you do stuff like:
...
store a into 'tmp_foo';
...
rm tmp_foo
...
The rm will trigger execution and the file will be there for you to delete,
copyToLocal, move, etc. You can also use the "exec" statement without params in
a script now, to force execution of what we've seen so far.
This patch also contains a minor fix with the computation of progress in MR
jobs (which I screwed up in the last patch).
> PERFORMANCE: multi-query optimization
> -------------------------------------
>
> Key: PIG-627
> URL: https://issues.apache.org/jira/browse/PIG-627
> Project: Pig
> Issue Type: Improvement
> Affects Versions: types_branch
> Reporter: Olga Natkovich
> Fix For: types_branch
>
> Attachments: file_cmds-0305.patch, multi-store-0303.patch,
> multi-store-0304.patch, multiquery_0223.patch, multiquery_0224.patch
>
>
> Currently, if your Pig script contains multiple stores and some shared
> computation, Pig will execute several independent queries. For instance:
> A = load 'data' as (a, b, c);
> B = filter A by a > 5;
> store B into 'output1';
> C = group B by b;
> store C into 'output2';
> This script will result in map-only job that generated output1 followed by a
> map-reduce job that generated output2. As the resuld data is read, parsed and
> filetered twice which is unnecessary and costly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
