Which version of Pig are you using? -----Original Message----- From: Corbin Hoenes [mailto:[email protected]] Sent: Thursday, May 06, 2010 10:29 AM To: [email protected] Subject: SpillableMemoryManager - low memory handler called
Hi Piggers - Seeing an issue with a particular script where our job is taking 6hrs 42min to complete. syslogs are showing loads of these: INFO : org.apache.pig.impl.util.SpillableMemoryManager - low memory handler called (Usage threshold exceeded) init = 5439488(5312K) used = 283443200(276800K) committed = 357957632(349568K) max = 357957632(349568K) INFO : org.apache.pig.impl.util.SpillableMemoryManager - low memory handler called (Usage threshold exceeded) init = 5439488(5312K) used = 267128840(260868K) committed = 357957632(349568K) max = 357957632(349568K) One iteresting thing is it's the map phase that is slow and one of the mappers is getting 8GB of input while the other 2000 or so mappers are getting MBs and hundreds of MBs of data. Any where I can start looking?
