Ok, at least one tablet per tabletserver would be good. More than one
tablet per tabletserver is not an issue. 5-10 per tserver would be OK.
Try increasing tserver.mutation.queue.max to something like 50M or 100M.
4M is very small.
What is -Xmx in your accumulo-env.sh for ACCUMULO_TSERVER_OPTS?
You can also try setting table.durability=flush in accumulo-site.xml.
hujs wrote:
Sorry, the post was wrong, these days the network is not good, just to see,
figure in the "accumulo balance" this post can be seen. I'm tuning the
accumulo ingest rate now. I do not know what is the reason to limit the
accumulo ingest rate. I mainly used
Here are a few ways:
1, per-splite, I through the custom j, n points will be divided into three
tablets, each tserver one.
2, adjust table.file.max, tserver.compaction.minor.concurrent.max and
table.durability = flush
3, by simulating multiple clients, with multiple Bathwriter send data.
4, the opening of the native map
5, configure the tserver's accumuo-site.xml and accumulo-env.sh memory
settings to 2g.
However, ingest rate did not improve. My cluster is far from reaching the
load: cpu utilization of less than 1/8, memory usage less than 1/5, IO 8-14%
wa, with start-ingest.sh test ingest rate, but the average ingest rate less
than 18 Million entry / s. My cluster each machine configuration: cpu 8,32 g
Me. I would like to know what the impact ingest rate,
How do I tune my cluster?
Using the accumuluo 1.7.1 version.The accumulo-site.xml configuration is as
follows:
<configuration>
<property>
<name>instance.volumes</name>
<value>hdfs://master12:9000/accumulo</value>
<description>comma separated list of URIs for volumes. example:
hdfs://localhost:9000/accumulo</description>
</property>
<property>
<name>instance.zookeeper.host</name>
<value>master12:2181,slave13:2181,slave10:2181,slave11:2181</value>
<description>comma separated list of zookeeper servers</description>
</property>
<property>
<name>logger.dir.walog</name>
<value>walogs</value>
<description>The property only needs to be set if upgrading from 1.4
which used to store write-ahead logs on the local
filesystem. In 1.5 write-ahead logs are stored in DFS. When 1.5 is
started for the first time it will copy any 1.4
write ahead logs into DFS. It is possible to specify a
comma-separated list of directories.
</description>
</property>
<property>
<name>instance.secret</name>
<value>DEFAULT</value>
<description>A secret unique to a given instance that all servers must
know in order to communicate with one another.
Change it before initialization. To
change it later use ./bin/accumulo
org.apache.accumulo.server.util.ChangeSecret --old [oldpasswd] --new
[newpasswd],
and then update this file.
</description>
</property>
<property>
<name>tserver.memory.maps.max</name>
<value>2G</value>
</property>
<property>
<name>tserver.memory.maps.native.enabled</name>
<value>true</value>
</property>
<property>
<name>tserver.cache.data.size</name>
<value>128M</value>
</property>
<property>
<name>tserver.cache.index.size</name>
<value>128M</value>
</property>
<property>
<name>trace.token.property.password</name>
<value>123456</value>
</property>
<property>
<name>trace.user</name>
<value>root</value>
</property>
<property>
<name>tserver.sort.buffer.size</name>
<value>500M</value>
</property>
<property>
<name>tserver.walog.max.size</name>
<value>2G</value>
</property>
<property>
<name>tserver.wal.blocksize</name>
<value>2G</value>
</property>
<property>
<name>tserver.mutation.queue.max</name>
<value>4M</value>
</property>
<property>
<name>tserver.compaction.major.concurrent.max</name>
<value>8</value>
</property>
<property>
<name>tserver.compaction.minor.concurrent.max</name>
<value>8</value>
</property>
<property>
<name>general.classpaths</name>
<value>
$ACCUMULO_HOME/lib/accumulo-server.jar,
$ACCUMULO_HOME/lib/accumulo-core.jar,
$ACCUMULO_HOME/lib/accumulo-start.jar,
$ACCUMULO_HOME/lib/accumulo-fate.jar,
$ACCUMULO_HOME/lib/accumulo-proxy.jar,
$ACCUMULO_HOME/lib/[^.].*.jar,
$ZOOKEEPER_HOME/zookeeper[^.].*.jar,
$HADOOP_CONF_DIR,
$HADOOP_PREFIX/share/hadoop/common/[^.].*.jar,
$HADOOP_PREFIX/share/hadoop/common/lib/(?!slf4j)[^.].*.jar,
$HADOOP_PREFIX/share/hadoop/hdfs/[^.].*.jar,
$HADOOP_PREFIX/share/hadoop/mapreduce/[^.].*.jar,
$HADOOP_PREFIX/share/hadoop/yarn/[^.].*.jar,
$HADOOP_PREFIX/share/hadoop/yarn/lib/jersey.*.jar,
</value>
<description>Classpaths that accumulo checks for updates and class
files.</description>
</property>
</configuration>
Think you dear josh Elser;
--
View this message in context:
http://apache-accumulo.1065345.n5.nabble.com/Write-or-Ingest-bottleneck-tp19255p19319.html
Sent from the Developers mailing list archive at Nabble.com.
