[
https://issues.apache.org/jira/browse/PIG-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated PIG-1466:
-------------------------------
Attachment: PIG-1466.patch
With the changes in the patch, the logs from SpillableMemoryManager will look
like this -
2010-08-18 18:22:44,365 INFO org.apache.pig.impl.util.SpillableMemoryManager:
first memory handler call- Usage threshold exceeded init = 5439488(5312K) used
= 657702912(642288K) committed = 715849728(699072K) max = 715849728(699072K)
2010-08-18 18:22:44,728 INFO org.apache.pig.impl.util.SpillableMemoryManager:
first memory handler call - Collection threshold exceeded init = 5439488(5312K)
used = 658437416(643005K) committed = 715849728(699072K) max =
715849728(699072K)
2010-08-18 18:27:51,174 INFO org.apache.pig.impl.util.SpillableMemoryManager:
Spilled an estimate of 73675890 bytes from 1 objects. init = 5439488(5312K)
used = 511215656(499234K) committed = 659881984(644416K) max =
715849728(699072K)
2010-08-18 18:29:16,739 INFO org.apache.pig.impl.util.SpillableMemoryManager:
Spilled an estimate of 418063724 bytes from 1 objects. init = 5439488(5312K)
used = 563165736(549966K) committed = 659881984(644416K) max =
715849728(699072K)
2010-08-18 18:29:22,274 INFO org.apache.pig.impl.util.SpillableMemoryManager:
Spilled an estimate of 443888012 bytes from 1 objects. init = 5439488(5312K)
used = 553174984(540209K) committed = 715849728(699072K) max =
715849728(699072K)
> Improve log messages for memory usage
> -------------------------------------
>
> Key: PIG-1466
> URL: https://issues.apache.org/jira/browse/PIG-1466
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.7.0
> Reporter: Ashutosh Chauhan
> Assignee: Thejas M Nair
> Priority: Minor
> Fix For: 0.8.0
>
> Attachments: PIG-1466.patch
>
>
> For anything more then a moderately sized dataset Pig usually spits following
> messages:
> {code}
> 2010-05-27 18:28:31,659 INFO org.apache.pig.impl.util.SpillableMemoryManager:
> low memory handler called (Usage
> threshold exceeded) init = 4194304(4096K) used = 672012960(656262K) committed
> = 954466304(932096K) max =
> 954466304(932096K)
> 2010-05-27 18:10:52,653 INFO org.apache.pig.impl.util.SpillableMemoryManager:
> low memory handler called (Collection
> threshold exceeded) init = 4194304(4096K) used = 954466304(932096K) committed
> = 954466304(932096K) max =
> 954466304(932096K)
> {code}
> This seems to confuse users a lot. Once these messages are printed, users
> tend to believe that Pig is having hard time with memory, is spilling to disk
> etc. but in fact Pig might be cruising along at ease. We should be little
> more careful what to print in logs. Currently these are printed when a
> notification is sent by JVM and some other conditions are met which may not
> necessarily indicate low memory condition. Furthermore, with
> {{InternalCachedBag}} embraced everywhere in favor of {{DefaultBag}}, these
> messages have lost their usefulness. At the every least, we should lower the
> log level at which these are printed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
