[
https://issues.apache.org/jira/browse/HDDS-8469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17718368#comment-17718368
]
Ethan Rose commented on HDDS-8469:
----------------------------------
[~sumitagrawl] The test code linked above has a few gaps:
* It is using a stubbed out implementation for the buffer, and therefore not
testing production code. We need to A/B test the two cases:
1. {{SCMDBTransactionBufferImpl}} with flush disabled
2. {{SCMHADBTransactionBufferImpl}} with flush enabled.
* It does not document which configs are changed to get the numbers for
comparison. Ideally one run of the test would produce numbers for both cases 1
and 2 above. This makes results difficult to reproduce.
* What the summary logs call "flush time" is not timing WAL flush. It is
timing commit, disposal, and recreation of the RocksDB batch operation object.
** We do not need to time WAL flush as part of this test, only overall
insert/delete/commit time. None of the paths under test involve a WAL flush.
Could you please update the test code for a better comparison:
* Start with a fresh test suite. It is ok to use pieces from other suites but
we want a minimal repro here, so adding a test into an existing suite with its
own setup will muddy the waters.
* Have one run of the test do case 1, followed by case 2.
** Passing in RocksDB parameters from the test may require some new code. It
could partially be done by adding the options to \{RocksDBConfiguration} and
parsing it out in \{DBStoreBuilder#build}
* Make the final summary log contain the performance difference between the
two (subtract one run's average time from the other). This way someone only has
to click "run" to see the differences.
* Add logs in {{RDBStore}} constructor to log the values of
\{dbOptions#manualWalFlush, {writeOptions#sync}
and other parameters tuned to make sure the correct values are making it to
RocksDB.
There's a few other options that would be good to test as well.
* No buffer and the WAL completely disabled.
** This would be good to see, even though we can't use that with Recon's
current implementation. It could tell us if there's benefit on the SCM side to
making Recon a non-voting Ratis follower instead of using incremental WAL
updates.
* Small buffer flush threshold, maybe 10 or so transactions instead of 1000
These should be easy enough to test once the above framework is in place.
> SCM using WAL manual flush of rocksDB
> -------------------------------------
>
> Key: HDDS-8469
> URL: https://issues.apache.org/jira/browse/HDDS-8469
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Sumit Agrawal
> Assignee: Sumit Agrawal
> Priority: Major
> Labels: proton, pull-request-available
> Attachments: SCM WAL manual flush.docx
>
>
> SCM currently make use of transaction buffer with batchOperation. This
> impacts that db update is not available till snapshot is taken which flushes
> batch operation or other related event.
> As part of this, need remove batchOperation and for performance, using WAL
> with manual flush at similar interval,
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
