[jira] [Commented] (CASSANDRA-13533) ColumnIdentifier object size wrong when tables are not flushed
[ https://issues.apache.org/jira/browse/CASSANDRA-13533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026719#comment-16026719 ] Joel Knighton commented on CASSANDRA-13533: --- +1 - thanks for the patch. I ran tests for all relevant branches; no unit tests failed on 3.0/3.11/trunk, and dtests looked the same as the present state of those branches. Committed to 3.0 as {{8ffdd26cbee33c5dc1205c0f7292628e1a2c69e3}} and merged forward. > ColumnIdentifier object size wrong when tables are not flushed > -- > > Key: CASSANDRA-13533 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13533 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Eduard Tudenhoefner >Assignee: Eduard Tudenhoefner > Fix For: 3.0.14, 3.11.0, 4.0 > > Attachments: columnidentifier.png > > > It turns out that the object size of {{ColumnIdentifier}} is wrong when > *cassandra.test.flush_local_schema_changes: false*. This looks like stuff is > being wrongly reused when no flush is happening. > We only noticed this because we were using the prepared stmt cache and > noticed that prepared statements would account for *1-6mb* when > *cassandra.test.flush_local_schema_changes: false*. With > *cassandra.test.flush_local_schema_changes: true* (which is the default) > those would be around *5000 bytes*. > Attached is a test that reproduces the problem and also a fix. > Also after talking to [~jkni] / [~blerer] we shouldn't probably take > {{ColumnDefinition}} into account when measuring object sizes with > {{MemoryMeter}} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13533) ColumnIdentifier object size wrong when tables are not flushed
[ https://issues.apache.org/jira/browse/CASSANDRA-13533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023856#comment-16023856 ] Eduard Tudenhoefner commented on CASSANDRA-13533: - that works for me [~jkni]. The test was only there to show that stuff was broken and anything we can do to shorten the circumstances for reproduction is a +1 from my side. > ColumnIdentifier object size wrong when tables are not flushed > -- > > Key: CASSANDRA-13533 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13533 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Eduard Tudenhoefner >Assignee: Eduard Tudenhoefner > Fix For: 3.0.14, 3.11.0, 4.0 > > Attachments: columnidentifier.png > > > It turns out that the object size of {{ColumnIdentifier}} is wrong when > *cassandra.test.flush_local_schema_changes: false*. This looks like stuff is > being wrongly reused when no flush is happening. > We only noticed this because we were using the prepared stmt cache and > noticed that prepared statements would account for *1-6mb* when > *cassandra.test.flush_local_schema_changes: false*. With > *cassandra.test.flush_local_schema_changes: true* (which is the default) > those would be around *5000 bytes*. > Attached is a test that reproduces the problem and also a fix. > Also after talking to [~jkni] / [~blerer] we shouldn't probably take > {{ColumnDefinition}} into account when measuring object sizes with > {{MemoryMeter}} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13533) ColumnIdentifier object size wrong when tables are not flushed
[ https://issues.apache.org/jira/browse/CASSANDRA-13533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023753#comment-16023753 ] Joel Knighton commented on CASSANDRA-13533: --- As a concrete example, something like the following would suffice to show the interning issue on all active branches. {code} @Test public void testInterningUsesMinimalByteBuffer() { byte[] bytes = new byte[2]; bytes[0] = 0x63; ByteBuffer byteBuffer = ByteBuffer.wrap(bytes); byteBuffer.limit(1); ColumnIdentifier c1 = ColumnIdentifier.getInterned(byteBuffer, UTF8Type.instance); Assert.assertEquals(2, byteBuffer.capacity()); Assert.assertEquals(1, c1.bytes.capacity()); } {code} What do you think, [~eduard.tudenhoefner]? > ColumnIdentifier object size wrong when tables are not flushed > -- > > Key: CASSANDRA-13533 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13533 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Eduard Tudenhoefner >Assignee: Eduard Tudenhoefner > Fix For: 3.0.14, 3.11.0, 4.0 > > Attachments: columnidentifier.png > > > It turns out that the object size of {{ColumnIdentifier}} is wrong when > *cassandra.test.flush_local_schema_changes: false*. This looks like stuff is > being wrongly reused when no flush is happening. > We only noticed this because we were using the prepared stmt cache and > noticed that prepared statements would account for *1-6mb* when > *cassandra.test.flush_local_schema_changes: false*. With > *cassandra.test.flush_local_schema_changes: true* (which is the default) > those would be around *5000 bytes*. > Attached is a test that reproduces the problem and also a fix. > Also after talking to [~jkni] / [~blerer] we shouldn't probably take > {{ColumnDefinition}} into account when measuring object sizes with > {{MemoryMeter}} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13533) ColumnIdentifier object size wrong when tables are not flushed
[ https://issues.apache.org/jira/browse/CASSANDRA-13533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023663#comment-16023663 ] Joel Knighton commented on CASSANDRA-13533: --- The patch looks good. I merged this forward through 3.11 and trunk - testall and dtests look good for all branches relative to upstream. I'm not sure about including the test as written; it passes on 3.11 and trunk even before the fix because of the default of offheap memtables introduced in [CASSANDRA-9472]. It seems to me that we might as well reduce this test to just checking that interning a ColumnIdentifier uses a minimal bytebuffer and add it to {{ColumnIdentifierTest}}. > ColumnIdentifier object size wrong when tables are not flushed > -- > > Key: CASSANDRA-13533 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13533 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Eduard Tudenhoefner >Assignee: Eduard Tudenhoefner > Fix For: 3.0.14, 3.11.0, 4.0 > > Attachments: columnidentifier.png > > > It turns out that the object size of {{ColumnIdentifier}} is wrong when > *cassandra.test.flush_local_schema_changes: false*. This looks like stuff is > being wrongly reused when no flush is happening. > We only noticed this because we were using the prepared stmt cache and > noticed that prepared statements would account for *1-6mb* when > *cassandra.test.flush_local_schema_changes: false*. With > *cassandra.test.flush_local_schema_changes: true* (which is the default) > those would be around *5000 bytes*. > Attached is a test that reproduces the problem and also a fix. > Also after talking to [~jkni] / [~blerer] we shouldn't probably take > {{ColumnDefinition}} into account when measuring object sizes with > {{MemoryMeter}} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13533) ColumnIdentifier object size wrong when tables are not flushed
[ https://issues.apache.org/jira/browse/CASSANDRA-13533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16013284#comment-16013284 ] Eduard Tudenhoefner commented on CASSANDRA-13533: - [~jjirsa] we were using that flag in some of our integration tests where we found out that, _very rarely_, simple things (usually taking < 1 second) would take a long amount of time (> 30 seconds). We boiled it then down to the point where we saw that this time was 99% spent in the prepared stmt cache for adding/evicting elements. And in our case, eviction was happening because a single prepared stmt would account *1mb - 6mb* (prep stmt cache size was *8mb* for us). After looking closer why prep stmts would be so large, we eventually saw that the binary representation of {{ColumnIdentifier#text}} was be wrong and not what we would have expected (see attached screenshot). We also only detected all of this because on *cassandra-3.0* we don't use *.omitSharedBufferOverhead()* with {{MemoryMeter}} object measurements ( [QueryProcessor.java#L67|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/cql3/QueryProcessor.java#L67]). On *cassandra-3.11* things got refactored over time and the prep stmt cache was using {{MemoryMeter}} stuff from {{ObjectSizes}}, where *.omitSharedBufferOverhead()* is being applied and the issue wouldn't happen ([ObjectSizes.java#L35|https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/utils/ObjectSizes.java#L35]). > ColumnIdentifier object size wrong when tables are not flushed > -- > > Key: CASSANDRA-13533 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13533 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Eduard Tudenhoefner >Assignee: Eduard Tudenhoefner > Fix For: 3.0.14, 3.11.0, 4.0 > > Attachments: columnidentifier.png > > > It turns out that the object size of {{ColumnIdentifier}} is wrong when > *cassandra.test.flush_local_schema_changes: false*. This looks like stuff is > being wrongly reused when no flush is happening. > We only noticed this because we were using the prepared stmt cache and > noticed that prepared statements would account for *1-6mb* when > *cassandra.test.flush_local_schema_changes: false*. With > *cassandra.test.flush_local_schema_changes: true* (which is the default) > those would be around *5000 bytes*. > Attached is a test that reproduces the problem and also a fix. > Also after talking to [~jkni] / [~blerer] we shouldn't probably take > {{ColumnDefinition}} into account when measuring object sizes with > {{MemoryMeter}} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13533) ColumnIdentifier object size wrong when tables are not flushed
[ https://issues.apache.org/jira/browse/CASSANDRA-13533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16013227#comment-16013227 ] Jeff Jirsa commented on CASSANDRA-13533: {quote} This looks like stuff is being wrongly reused when no flush is happening {quote} Obviously that flag is only for unit tests, but can you describe the symptom that led you to finding/fixing this? > ColumnIdentifier object size wrong when tables are not flushed > -- > > Key: CASSANDRA-13533 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13533 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Eduard Tudenhoefner >Assignee: Eduard Tudenhoefner > Fix For: 3.0.14, 3.11.0, 4.0 > > > It turns out that the object size of {{ColumnIdentifier}} is wrong when > *cassandra.test.flush_local_schema_changes: false*. This looks like stuff is > being wrongly reused when no flush is happening. > We only noticed this because we were using the prepared stmt cache and > noticed that prepared statements would account for *1-6mb* when > *cassandra.test.flush_local_schema_changes: false*. With > *cassandra.test.flush_local_schema_changes: true* (which is the default) > those would be around *5000 bytes*. > Attached is a test that reproduces the problem and also a fix. > Also after talking to [~jkni] / [~blerer] we shouldn't probably take > {{ColumnDefinition}} into account when measuring object sizes with > {{MemoryMeter}} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13533) ColumnIdentifier object size wrong when tables are not flushed
[ https://issues.apache.org/jira/browse/CASSANDRA-13533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16012660#comment-16012660 ] Eduard Tudenhoefner commented on CASSANDRA-13533: - Branch: https://github.com/nastra/cassandra/tree/CASSANDRA-13533-30 > ColumnIdentifier object size wrong when tables are not flushed > -- > > Key: CASSANDRA-13533 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13533 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Eduard Tudenhoefner >Assignee: Eduard Tudenhoefner > > It turns out that the object size of {{ColumnIdentifier}} is wrong when > *cassandra.test.flush_local_schema_changes: false*. This looks like stuff is > being wrongly reused when no flush is happening. > We only noticed this because we were using the prepared stmt cache and > noticed that prepared statements would account for *1-6mb* when > *cassandra.test.flush_local_schema_changes: false*. With > *cassandra.test.flush_local_schema_changes: true* (which is the default) > those would be around *5000 bytes*. > Attached is a test that reproduces the problem and also a fix. > Also after talking to [~jkni] / [~blerer] we shouldn't probably take > {{ColumnDefinition}} into account when measuring object sizes with > {{MemoryMeter}} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org