[
https://issues.apache.org/jira/browse/HBASE-14490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14989860#comment-14989860
]
Zephyr Guo commented on HBASE-14490:
------------------------------------
{quote}
What does dumping information means?
{quote}
There are two stages(put and read).I mark "#CREATE#" and "#DSICARD#" in log.You
can look test-v12-patch that I attached.test-v12-patch is my way that test
BBBP.Your byteBufferPool use the same way to test.
"read create total bytes" means total of byte that was created in put stage.
15 seconds after put stage(first stage) done , I use following code to dump
information.
{code}
#!/bin/bash
REPORT=./report.my
if [ ! -e $REPORT ]; then
touch $REPORT
fi
create_t=$(grep -oE '#CREATE#' $LOG_FILE | wc -l)
discard_t=$(grep -oE '#DISCARD#' $LOG_FILE | wc -l)
sed -ibk '1,$ d' $REPORT
rm ${REPORT}bk
echo "put create times:"$create_t >> $REPORT
echo "put discard times:"$discard_t >> $REPORT
cal_sum() {
sum=0
for i in $1; do
sum=$(expr $sum + $i)
done
}
cal_sum "$(grep -oE '#CREATE#\d+' $LOG_FILE | grep -oE '\d+')"
echo "put create total bytes:"$sum >> $REPORT
cal_sum "$(grep -oE '#DISCARD#\d+' $LOG_FILE | grep -oE '\d+')"
echo "put discard total bytes:"$sum >> $REPORT
{code}
15 seconds after read stage(second stage) done, I use following code to dump
information.
{code}
#!/bin/bash
REPORT=./report.my
create_t=$(grep -oE 'put create times: *\d+' $REPORT | grep -oE '\d+')
discard_t=$(grep -oE 'put discard times: *\d+' $REPORT | grep -oE '\d+')
create_tot=$(grep -oE 'put create total bytes:\d+' $REPORT | grep -oE '\d+')
discard_tot=$(grep -oE 'put discard total bytes:\d+' $REPORT | grep -oE '\d+')
create_t=$(expr $(grep -oE '#CREATE#' $LOG_FILE | wc -l) - $create_t)
discard_t=$(expr $(grep -oE '#DISCARD#' $LOG_FILE | wc -l) - $discard_t)
cal_sum() {
sum=0
for i in $1; do
sum=$(expr $sum + $i)
done
}
echo "read create times:"$create_t >> $REPORT
echo "read discard times:"$discard_t >> $REPORT
cal_sum "$(grep -oE '#CREATE#\d+' $LOG_FILE | grep -oE '\d+')"
echo "read create total bytes:"$(expr $sum - $create_tot) >> $REPORT
cal_sum "$(grep -oE '#DISCARD#\d+' $LOG_FILE | grep -oE '\d+')"
echo "read discard total bytes:"$(expr $sum - $discard_tot) >> $REPORT
{code}
{quote}
Did you check the frequency of GC?
{quote}
I don't know how to test frequency of GC if the gap is small.
> [RpcServer] reuse request read buffer
> -------------------------------------
>
> Key: HBASE-14490
> URL: https://issues.apache.org/jira/browse/HBASE-14490
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC
> Affects Versions: 2.0.0, 1.0.2
> Reporter: Zephyr Guo
> Assignee: Zephyr Guo
> Labels: performance
> Fix For: 2.0.0, 1.0.2
>
> Attachments: ByteBufferPool.java, HBASE-14490-v1.patch,
> HBASE-14490-v10.patch, HBASE-14490-v11.patch, HBASE-14490-v12.patch,
> HBASE-14490-v2.patch, HBASE-14490-v3.patch, HBASE-14490-v4.patch,
> HBASE-14490-v5.patch, HBASE-14490-v6.patch, HBASE-14490-v7.patch,
> HBASE-14490-v8.patch, HBASE-14490-v9.patch, test-v12-patch
>
>
> Reuse buffer to read request.It's not necessary to every request free
> buffer.The idea of optimization is to reduce the times that allocate
> ByteBuffer.
> *Modification*
> 1. {{saslReadAndProcess}} ,{{processOneRpc}}, {{processUnwrappedData}},
> {{processConnectionHeader}} accept a ByteBuffer instead of byte[].They can
> move {{ByteBuffer.position}} correctly when we have read the data.
> 2. {{processUnwrappedData}} no longer use any extra memory.
> 3. Maintaining a reused ByteBuffer in each {{Connection}}.
> a) Size is dynamic to fit specific client.
> b) If size deviate far from average size, we replace a new one.
> c) Using a SoftReference to reference the buffer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
