[
https://issues.apache.org/jira/browse/CASSANDRA-9619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14595924#comment-14595924
]
Jim Witschey commented on CASSANDRA-9619:
-----------------------------------------
Bisecting is going, but slowly -- I've only been able to reproduce the
regression with a lot of data written, so each run takes several hours. The
current bisect log is in the description of [this {{cstar_perf}}
job|http://cstar.datastax.com/tests/id/30987b60-18e3-11e5-9c22-42010af0688f].
> Read performance regression in tables with many columns on trunk and 2.2 vs.
> 2.1
> --------------------------------------------------------------------------------
>
> Key: CASSANDRA-9619
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9619
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jim Witschey
> Labels: perfomance
> Fix For: 2.2.0 rc2
>
>
> There seems to be a regression in read in 2.2 and trunk, as compared to 2.1
> and 2.0. I found it running cstar_perf jobs with 50-column tables. 2.2 may be
> worse than trunk, though my results on that aren't consistent. The relevant
> cstar_perf jobs are here:
> http://cstar.datastax.com/tests/id/273e2ea8-0fc8-11e5-816c-42010af0688f
> http://cstar.datastax.com/tests/id/3a8002d6-1480-11e5-97ff-42010af0688f
> http://cstar.datastax.com/tests/id/40ff2766-1248-11e5-bac8-42010af0688f
> The sequence of commands for these jobs is
> {code}
> stress write n=65000000 -rate threads=300 -col n=FIXED\(50\)
> stress read n=65000000 -rate threads=300
> stress read n=65000000 -rate threads=300
> {code}
> Have a look at the operations per second going from [the first read
> operation|http://cstar.datastax.com/graph?stats=273e2ea8-0fc8-11e5-816c-42010af0688f&metric=op_rate&operation=2_read&smoothing=1&show_aggregates=true&xmin=0&xmax=729.08&ymin=0&ymax=174379.7]
> to [the second read
> operation|http://cstar.datastax.com/graph?stats=273e2ea8-0fc8-11e5-816c-42010af0688f&metric=op_rate&operation=2_read&smoothing=1&show_aggregates=true&xmin=0&xmax=729.08&ymin=0&ymax=174379.7].
> They've fallen from ~135K to ~100K comparing trunk to 2.1 and 2.0. It's
> slightly worse for 2.2, and 2.2 operations per second fall continuously from
> the first to the second read operation.
> There's a corresponding increase in read latency -- it's noticable on trunk
> and pretty bad on 2.2. Again, the latency gets higher and higher on 2.2 as
> the read operations progress (see the graphs
> [here|http://cstar.datastax.com/graph?stats=273e2ea8-0fc8-11e5-816c-42010af0688f&metric=95th_latency&operation=2_read&smoothing=1&show_aggregates=true&xmin=0&xmax=729.08&ymin=0&ymax=17.27]
> and
> [here|http://cstar.datastax.com/graph?stats=273e2ea8-0fc8-11e5-816c-42010af0688f&metric=95th_latency&operation=3_read&smoothing=1&show_aggregates=true&xmin=0&xmax=928.62&ymin=0&ymax=14.52]).
> I see a similar regression in a [more recent
> test|http://cstar.datastax.com/graph?stats=40ff2766-1248-11e5-bac8-42010af0688f&metric=op_rate&operation=2_read&smoothing=1&show_aggregates=true&xmin=0&xmax=752.62&ymin=0&ymax=171799.1],
> though in this one trunk performed worse than 2.2. This run also didn't
> display the increasing latency in 2.2.
> This regression may show for smaller numbers of columns, but not as
> prominently, as shown [in the results to this test with the stress default of
> 5
> columns|http://cstar.datastax.com/graph?stats=227cb89e-0fc8-11e5-9f14-42010af0688f&metric=99.9th_latency&operation=3_read&smoothing=1&show_aggregates=true&xmin=0&xmax=498.19&ymin=0&ymax=334.29].
> There's an increase in latency variability on trunk and 2.2, but I don't see
> a regression in summary statistics.
> My measurements aren't confounded by [the recent regression in
> cassandra-stress|https://issues.apache.org/jira/browse/CASSANDRA-9558];
> cstar_perf uses the same stress program (from trunk) on all versions on the
> cluster.
> I'm currently working to
> - reproduce with a smaller workload so this is easier to bisect and debug.
> - get results with larger numbers of columns, since we've seen the regression
> on 50 columns but not the stress default of 5.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
