[
https://issues.apache.org/jira/browse/PIG-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800164#comment-13800164
]
Aniket Mokashi commented on PIG-3047:
-------------------------------------
[~ihadanny], thanks for updating the patch!
Few more comments-
# comment on the test still mentions the old property name.
# We typically catch the exception in test and assert the error message. Can
you modify your test to do so?
I tried your patch on trunk and found some problems with it.
# If I modify the test to remove the catch exception annotation and increase
the configuration value to 18. I get following exception stack-
{code}
Caused by: org.apache.pig.impl.plan.VisitorException: ERROR 0: Replicated input
files size: 64063 exceeds pig.join.replicated.max.bytes: 18
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler$JoinDistributedCacheVisitor.visitFRJoin(JobControlCompiler.java:1578)
... 42 more
{code}
# With your old patch, I get following exception stack-
{code}
Caused by: org.apache.pig.impl.plan.VisitorException: ERROR 0: Replicated input
files size: 17 exceeds pig.fr_join.max.bytes.size: 14
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler$JoinDistributedCacheVisitor.visitFRJoin(JobControlCompiler.java:1579)
... 42 more
{code}
> Check the size of a relation before adding it to distributed cache in
> Replicated join
> -------------------------------------------------------------------------------------
>
> Key: PIG-3047
> URL: https://issues.apache.org/jira/browse/PIG-3047
> Project: Pig
> Issue Type: Improvement
> Reporter: Julien Le Dem
> Assignee: Ido Hadanny
> Fix For: 0.12.1
>
> Attachments: fix_pig_3047_1.diff, fix_pig_3047.diff
>
>
> Right now if someone makes a mistake and put the large relation last, Pig
> will copy a huge file into distributed cache and it will take a long time
> before the job eventually fails. It would be better to check before copying
> the relation that it is of reasonable size.
> <1 GB ?
--
This message was sent by Atlassian JIRA
(v6.1#6144)
