[
https://issues.apache.org/jira/browse/RATIS-541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16865710#comment-16865710
]
Josh Elser commented on RATIS-541:
----------------------------------
{code:java}
boolean batchWritesPresent = batchSize > 1 && numBatches > 0;
for (int i = 0; i < this.numRecords; i++) {
if(batchWritesPresent & (i % (numRecords/numBatches) == 0))
{
List<ByteBuffer> messages = new
ArrayList<ByteBuffer>(batchSize);
for(int b = 0; b < batchSize; b++) {
messages.add(createValue(MESSAGE_PREFIX + i));
i++;
}
writer.write(messages);
}
String message = MESSAGE_PREFIX + i;
if (i % logFreq == 0) {
LOG.info(logName + " Writing " + message);
}
writer.write(createValue(message));
}
{code}
This is a snippet from the final state of {{BulkWrite#run()}}. When we hit
{{batchWritesPresent=true}}, would will still fall down and write an extra
message. I think this will inflate the number of records written; I'd expect it
to cause the BulkReader to also fail (reading more records than it should).
{code:java}
+ @Parameter(names = {"-bs", "--batchSize"}, description = "Number of
records in a batch")
+ private int batchSize = 1;
+ @Parameter(names = {"-nb", "--numBatches"}, description = "Number of
batches to write per log")
+ private int numBatches = 0;
{code}
It looks like despite a {{numBatches}} option, I still have to supply {{-nr}}
to state the total number of rows to be written. Feels to me that {{-nb}}
should not be an option, but automatically figured out by the Tool. WDYT?
{code:java}
Operation(LogName logName, LogServiceClient client, int numRecords, int
logFreq, int valueSize) {
this.logName = logName;
@@ -150,8 +158,23 @@ public class VerificationTool {
this.numRecords = numRecords;
this.logFreq = logFreq;
this.valueSize = valueSize;
+ this.batchSize = 1;
+ this.numBatches = 0;
}
+
+ Operation(LogName logName, LogServiceClient client, int numRecords,
int logFreq, int valueSize, int batchSize,
+ int numBatches) {
+ this.logName = logName;
+ this.client = client;
+ this.numRecords = numRecords;
+ this.logFreq = logFreq;
+ this.valueSize = valueSize;
+ this.batchSize = batchSize;
+ this.numBatches = numBatches;
+ }
+
+{code}
Please consolidate these constructors by having the 5-arg constructor call the
7-arg constructor.
{code:java}
static class BulkWriter extends Operation {
- BulkWriter(LogName logName, LogServiceClient client, int numRecords,
int logFreq, int valueSize) {
- super(logName, client, numRecords, logFreq, valueSize);
+ BulkWriter(LogName logName, LogServiceClient client, int numRecords,
int logFreq, int valueSize, int batchSize,
+ int numBatches) {
+ super(logName, client, numRecords, logFreq, valueSize, batchSize,
numBatches);
{code}
Would it make things for simple if you pulled the logic for batched-writes out
into its own {{Operation}} implementation? That might help simplify the code
around my first comment.
> Add option to verify correctness of batch and mix of batch and single updates
> data from LogService in VerficiationTool
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: RATIS-541
> URL: https://issues.apache.org/jira/browse/RATIS-541
> Project: Ratis
> Issue Type: Improvement
> Components: LogService
> Reporter: Rajeshbabu Chintaguntla
> Assignee: Rajeshbabu Chintaguntla
> Priority: Major
> Attachments: RATIS-541.patch
>
>
> Currently, in the verification tool, we are verifying single updates
> correctness in verification but the same can do for batch and mix of batch
> and single updates.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
