I still see the number of counters increasing in the job tracker :(. Can I also set it in my giraph-site.xml or directly in my MasterCompute class?
Cheers, Christian 2013/9/10 Claudio Martella <[email protected]> > one the command line, you can use the -D option after the GiraphRunner > class before the GiraphRunner specific parameters, e.g. -D giraph. > useSuperstepCounters=false > > > On Tue, Sep 10, 2013 at 1:15 PM, Christian Krause <[email protected]> wrote: > >> Thanks a lot. One last question: where do I set options like >> USE_SUPERSTEP_COUNTERS? >> >> Christian >> >> >> 2013/9/9 André Kelpe <[email protected]> >> >>> On older versions of hadoop, you cannot set the counters to a higher >>> value. That was only introduced later. I had this issue on CDH3 (~1.5 >>> years ago) and my solution was to disable all counters for the giraph >>> job, to make it work. If you use a more modern version of hadoop, it >>> should be possible to increase the limit though. >>> >>> - André >>> >>> 2013/9/9 Avery Ching <[email protected]>: >>> > If you are running out of counters, you can turn off the superstep >>> counters >>> > >>> > /** Use superstep counters? (boolean) */ >>> > BooleanConfOption USE_SUPERSTEP_COUNTERS = >>> > new BooleanConfOption("giraph.useSuperstepCounters", true, >>> > "Use superstep counters? (boolean)"); >>> > >>> > >>> > On 9/9/13 6:43 AM, Claudio Martella wrote: >>> > >>> > No, I used a different counters limit on that hadoop version. Setting >>> > mapreduce.job.counters.limit to a higher number and restarting JT and >>> TT >>> > worked for me. Maybe 64000 might be too high? Try setting it to 512. >>> Does >>> > not look like the case, but who knows. >>> > >>> > >>> > On Mon, Sep 9, 2013 at 2:57 PM, Christian Krause <[email protected]> >>> wrote: >>> >> >>> >> Sorry, it still doesn't work (I ran into a different problem before I >>> >> reached the limit). >>> >> >>> >> I am using Hadoop 0.20.203.0. Is the limit of 120 counters maybe >>> >> hardcoded? >>> >> >>> >> Cheers >>> >> Christian >>> >> >>> >> Am 09.09.2013 08:29 schrieb "Christian Krause" <[email protected]>: >>> >> >>> >>> I changed the property name to mapred.job.counters.limit and >>> restarted it >>> >>> again. Now it works. >>> >>> >>> >>> Thanks, >>> >>> Christian >>> >>> >>> >>> >>> >>> 2013/9/7 Claudio Martella <[email protected]> >>> >>>> >>> >>>> did you restart TT and JT? >>> >>>> >>> >>>> >>> >>>> On Sat, Sep 7, 2013 at 7:09 AM, Christian Krause <[email protected]> >>> wrote: >>> >>>>> >>> >>>>> Hi, >>> >>>>> I've increased the counter limit in mapred-site.xml, but I still >>> get >>> >>>>> the error: Exceeded counter limits - Counters=121 Limit=120. >>> Groups=6 >>> >>>>> Limit=50. >>> >>>>> >>> >>>>> This is my config: >>> >>>>> >>> >>>>> cat conf/mapred-site.xml >>> >>>>> <?xml version="1.0"?> >>> >>>>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> >>> >>>>> >>> >>>>> <!-- Put site-specific property overrides in this file. --> >>> >>>>> >>> >>>>> <configuration> >>> >>>>> ... >>> >>>>> <property> >>> >>>>> <name>mapreduce.job.counters.limit</name> >>> >>>>> <value>64000</value> >>> >>>>> </property> >>> >>>>> <property> >>> >>>>> <name>mapred.task.timeout</name> >>> >>>>> <value>2400000</value> >>> >>>>> </property> >>> >>>>> ... >>> >>>>> </configuration> >>> >>>>> >>> >>>>> Any ideas? >>> >>>>> >>> >>>>> Cheers, >>> >>>>> Christian >>> >>>>> >>> >>>> >>> >>>> >>> >>>> >>> >>>> -- >>> >>>> Claudio Martella >>> >>>> [email protected] >>> >>> >>> >>> >>> > >>> > >>> > >>> > -- >>> > Claudio Martella >>> > [email protected] >>> > >>> > >>> >> >> > > > -- > Claudio Martella > [email protected] >
