I'm setting this property and the number of compute threads in the giraph-site.xml. This works very well. Thanks again for your help.
Cheers, Christian 2013/9/10 Claudio Martella <[email protected]> > you can set it in your giraph-site.xml, but it should work on the command > line. > > > On Tue, Sep 10, 2013 at 1:44 PM, Christian Krause <[email protected]> wrote: > >> I still see the number of counters increasing in the job tracker :(. Can >> I also set it in my giraph-site.xml or directly in my MasterCompute class? >> >> Cheers, >> Christian >> >> >> 2013/9/10 Claudio Martella <[email protected]> >> >>> one the command line, you can use the -D option after the GiraphRunner >>> class before the GiraphRunner specific parameters, e.g. -D giraph. >>> useSuperstepCounters=false >>> >>> >>> On Tue, Sep 10, 2013 at 1:15 PM, Christian Krause <[email protected]>wrote: >>> >>>> Thanks a lot. One last question: where do I set options like >>>> USE_SUPERSTEP_COUNTERS? >>>> >>>> Christian >>>> >>>> >>>> 2013/9/9 André Kelpe <[email protected]> >>>> >>>>> On older versions of hadoop, you cannot set the counters to a higher >>>>> value. That was only introduced later. I had this issue on CDH3 (~1.5 >>>>> years ago) and my solution was to disable all counters for the giraph >>>>> job, to make it work. If you use a more modern version of hadoop, it >>>>> should be possible to increase the limit though. >>>>> >>>>> - André >>>>> >>>>> 2013/9/9 Avery Ching <[email protected]>: >>>>> > If you are running out of counters, you can turn off the superstep >>>>> counters >>>>> > >>>>> > /** Use superstep counters? (boolean) */ >>>>> > BooleanConfOption USE_SUPERSTEP_COUNTERS = >>>>> > new BooleanConfOption("giraph.useSuperstepCounters", true, >>>>> > "Use superstep counters? (boolean)"); >>>>> > >>>>> > >>>>> > On 9/9/13 6:43 AM, Claudio Martella wrote: >>>>> > >>>>> > No, I used a different counters limit on that hadoop version. Setting >>>>> > mapreduce.job.counters.limit to a higher number and restarting JT >>>>> and TT >>>>> > worked for me. Maybe 64000 might be too high? Try setting it to 512. >>>>> Does >>>>> > not look like the case, but who knows. >>>>> > >>>>> > >>>>> > On Mon, Sep 9, 2013 at 2:57 PM, Christian Krause <[email protected]> >>>>> wrote: >>>>> >> >>>>> >> Sorry, it still doesn't work (I ran into a different problem before >>>>> I >>>>> >> reached the limit). >>>>> >> >>>>> >> I am using Hadoop 0.20.203.0. Is the limit of 120 counters maybe >>>>> >> hardcoded? >>>>> >> >>>>> >> Cheers >>>>> >> Christian >>>>> >> >>>>> >> Am 09.09.2013 08:29 schrieb "Christian Krause" <[email protected]>: >>>>> >> >>>>> >>> I changed the property name to mapred.job.counters.limit and >>>>> restarted it >>>>> >>> again. Now it works. >>>>> >>> >>>>> >>> Thanks, >>>>> >>> Christian >>>>> >>> >>>>> >>> >>>>> >>> 2013/9/7 Claudio Martella <[email protected]> >>>>> >>>> >>>>> >>>> did you restart TT and JT? >>>>> >>>> >>>>> >>>> >>>>> >>>> On Sat, Sep 7, 2013 at 7:09 AM, Christian Krause <[email protected]> >>>>> wrote: >>>>> >>>>> >>>>> >>>>> Hi, >>>>> >>>>> I've increased the counter limit in mapred-site.xml, but I still >>>>> get >>>>> >>>>> the error: Exceeded counter limits - Counters=121 Limit=120. >>>>> Groups=6 >>>>> >>>>> Limit=50. >>>>> >>>>> >>>>> >>>>> This is my config: >>>>> >>>>> >>>>> >>>>> cat conf/mapred-site.xml >>>>> >>>>> <?xml version="1.0"?> >>>>> >>>>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> >>>>> >>>>> >>>>> >>>>> <!-- Put site-specific property overrides in this file. --> >>>>> >>>>> >>>>> >>>>> <configuration> >>>>> >>>>> ... >>>>> >>>>> <property> >>>>> >>>>> <name>mapreduce.job.counters.limit</name> >>>>> >>>>> <value>64000</value> >>>>> >>>>> </property> >>>>> >>>>> <property> >>>>> >>>>> <name>mapred.task.timeout</name> >>>>> >>>>> <value>2400000</value> >>>>> >>>>> </property> >>>>> >>>>> ... >>>>> >>>>> </configuration> >>>>> >>>>> >>>>> >>>>> Any ideas? >>>>> >>>>> >>>>> >>>>> Cheers, >>>>> >>>>> Christian >>>>> >>>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> -- >>>>> >>>> Claudio Martella >>>>> >>>> [email protected] >>>>> >>> >>>>> >>> >>>>> > >>>>> > >>>>> > >>>>> > -- >>>>> > Claudio Martella >>>>> > [email protected] >>>>> > >>>>> > >>>>> >>>> >>>> >>> >>> >>> -- >>> Claudio Martella >>> [email protected] >>> >> >> > > > -- > Claudio Martella > [email protected] >
