I've downloaded, built and tested the src release. It looks fine.

The -bin releases contain jna.jar, which isn't mentioned in the NOTICE
file, so -1 for releasing the binary tarballs. You can still release the
source, just leave out the binaries. I think this is ok, since 4.3.0
isn't GA.

I ran a few benchmarks to compare against 4.2.2. The bookie benchmark
didn't run for 4.3.0. It just hangs on "Benchmarking latency". For a
single ledger on 3 bookies, performance is acceptable, but latency is
higher. For 1000 ledgers on 3 bookies, tpt and latency is much better,
but tpt is very variable. I did see a lot of timeouts in some runs
also.

single ledger, 3 bookies
| 4.2.2                    |   | 4.3.0                    |
|  tpt | lat p99 | lat p95 |   |  tpt | lat p99 | lat p95 |
| 9995 |   87.98 |   77.26 |   | 9988 |     132 |     117 |
| 9997 |   97.55 |   80.70 |   | 9991 |     111 |     107 |
| 9996 |  105.80 |   86.20 |   | 9989 |     103 |     100 |

1000 ledgers, 3 bookies
| 4.2.2                     |   | 4.3.0                     |
|   tpt | lat p99 | lat p95 |   |   tpt | lat p99 | lat p95 |
| 14588 |     668 |     662 |   | 23728 |     229 |     219 |
| 13571 |     701 |     649 |   | 39417 |     230 |     219 |
| 14062 |     697 |     688 |   | 40654 |     231 |     222 |
|       |         |         |   |       |         |         |

Overall, I'm not sure this release is fit to go without having -alpha or
-beta attached to its version, to make it clear that it shouldn't be
used as is in production. In particular I'm worried about the bookie
benchmark not working [1], and the variability of the 1000 ledger
benchmark. The latency increase for a single ledger can be attributed to
the added batching in the journal.

-Ivan

[1] bookkeeper-benchmark/bin/benchmark bookie -zookeeper <zk> -host <bookie>


Reply via email to