Re: Adding additional default JVM parameters to accumulo-env.sh boosts performance and prevent crashing of Tservers and masters.
Are you using native maps? Also, I'm in agreement with Josh that specific configuration flags may not apply to everybody. The conf/accumulo-env.sh script is a user-controllable file intended to be tweaked by system admins for their environment. That said, it's possible some of these configs may make sense for most users, and we'd want to make them default, but I'm not sure. On Fri, Mar 22, 2019 at 9:49 AM Jeffrey Manno wrote: > > To add some more detail, we are using Java 1.8 openJDK, and Hadoop 3.2.0. > These new JVM options were also added to HDFS_DATANODE_OPTS and > HDFS_NAMENODE_OPTs in hadoop-env,sh. This eliminated a replication warning > that kept occurring on our cluster. > > Here are the rest of the JAVA_OPTS in accumulo-env.sh for reference as what > is normally there by default: > '-XX:CMSInitiatingOccupancyFraction=75' > '-XX:+CMSClassUnloadingEnabled' > '-XX:OnOutOfMemoryError=kill -9 %p' > '-XX:-OmitStackTraceInFastThrow' > '-Djava.net.preferIPv4Stack=true' > "-Daccumulo.native.lib.path=${lib}/native") > > > > On Fri, Mar 22, 2019 at 9:29 AM Jeffrey Zeiberg wrote: > > > Jeffrey Manno (ASRC Federal) and Jeffrey Zeiberg (ASRC Federal) have > > discovered that adding a few new JVM options to the JAVA_OPTS set prevented > > crashing and increase system performance. > > > > They were added after line 68 in accumulo-env.sh. They are: > > 'server' > > '-XX:+UseParallelOldGC' > > '-XX:NewRatio=3' > > '-XX:AggressiveHeap' > > > > The machines we are using are 7 year old machines with 8G of main memory, > > between 500G - 1T HDD and Intel I5 or I7 processors. > > > > Maybe these parameters should be made in the Accumulo 2.0 distributions > > accumulo-env.sh file? > > > > Comments? > >
Re: Adding additional default JVM parameters to accumulo-env.sh boosts performance and prevent crashing of Tservers and masters.
To add some more detail, we are using Java 1.8 openJDK, and Hadoop 3.2.0. These new JVM options were also added to HDFS_DATANODE_OPTS and HDFS_NAMENODE_OPTs in hadoop-env,sh. This eliminated a replication warning that kept occurring on our cluster. Here are the rest of the JAVA_OPTS in accumulo-env.sh for reference as what is normally there by default: '-XX:CMSInitiatingOccupancyFraction=75' '-XX:+CMSClassUnloadingEnabled' '-XX:OnOutOfMemoryError=kill -9 %p' '-XX:-OmitStackTraceInFastThrow' '-Djava.net.preferIPv4Stack=true' "-Daccumulo.native.lib.path=${lib}/native") On Fri, Mar 22, 2019 at 9:29 AM Jeffrey Zeiberg wrote: > Jeffrey Manno (ASRC Federal) and Jeffrey Zeiberg (ASRC Federal) have > discovered that adding a few new JVM options to the JAVA_OPTS set prevented > crashing and increase system performance. > > They were added after line 68 in accumulo-env.sh. They are: > 'server' > '-XX:+UseParallelOldGC' > '-XX:NewRatio=3' > '-XX:AggressiveHeap' > > The machines we are using are 7 year old machines with 8G of main memory, > between 500G - 1T HDD and Intel I5 or I7 processors. > > Maybe these parameters should be made in the Accumulo 2.0 distributions > accumulo-env.sh file? > > Comments? >
Re: Adding additional default JVM parameters to accumulo-env.sh boosts performance and prevent crashing of Tservers and masters.
Very rarely do JVM GC properties universally apply to all users and workloads. I think it would be better to document why these options helped in your workload. Teach folks how to choose the correct JVM properties for their workloads is a better way forward, than encouraging folks to treat them as black-boxes. While this is extremely in-depth, I like the tone of this blog post: https://blogs.apache.org/hbase/entry/tuning_g1gc_for_your_hbase. The authors explain what they observed from a system, what they changed, what effect that change should have, and, finally, the change that they observe. On 3/22/19 9:28 AM, Jeffrey Zeiberg wrote: Jeffrey Manno (ASRC Federal) and Jeffrey Zeiberg (ASRC Federal) have discovered that adding a few new JVM options to the JAVA_OPTS set prevented crashing and increase system performance. They were added after line 68 in accumulo-env.sh. They are: 'server' '-XX:+UseParallelOldGC' '-XX:NewRatio=3' '-XX:AggressiveHeap' The machines we are using are 7 year old machines with 8G of main memory, between 500G - 1T HDD and Intel I5 or I7 processors. Maybe these parameters should be made in the Accumulo 2.0 distributions accumulo-env.sh file? Comments?
Adding additional default JVM parameters to accumulo-env.sh boosts performance and prevent crashing of Tservers and masters.
Jeffrey Manno (ASRC Federal) and Jeffrey Zeiberg (ASRC Federal) have discovered that adding a few new JVM options to the JAVA_OPTS set prevented crashing and increase system performance. They were added after line 68 in accumulo-env.sh. They are: 'server' '-XX:+UseParallelOldGC' '-XX:NewRatio=3' '-XX:AggressiveHeap' The machines we are using are 7 year old machines with 8G of main memory, between 500G - 1T HDD and Intel I5 or I7 processors. Maybe these parameters should be made in the Accumulo 2.0 distributions accumulo-env.sh file? Comments?