On Fri, Aug 21, 2020 at 7:57 AM Attila Wind <attilaw@swf.technology> wrote:
> Hi Cassandra Gurus, > > Recently I captured a very interesting warning in the logs saying > > 2020-08-19 08:08:32.492 > [cassandra-client-keytiles_data_webhits-nio-worker-2] WARN > com.datastax.driver.core.RequestHandler - Query '[3 bound values] select * > from visit_sess > ion_by_start_time_v4 where container_id=? and first_action_time_frame_id > >= ? and first_action_time_frame_id <= ?;' generated server side > warning(s): > *Read 6628 live rows and 6628 tombstone cells* for query SELECT * FROM > keytiles_data_webhits.visit_session_by_start_time_v4 WHERE container_id = > 5YzsPfE2Gcu8sd-76626 AND first_action_time_frame_id > 4 > 43837 AND first_action_time_frame_id <= 443670 AND user_agent_type > > browser-mobile AND unique_webclient_id > > 045d1683-c702-48bd-9d2b-dcf1ca87ac7c AND first_action_ts > 15978 > 15766 LIMIT 6628 (see tombstone_warn_threshold) > > What makes this interesting to me is the fact we never issue not even row > level deletes but any kind of deletes against this table for now > So I'm wondering what can result in tombstone creation in Cassandra - > apart from explicit DELETE queries and TTL setup... > > My suspicion is (but I'm not sure) that as we are going with "select *" > read strategy, then calculate everything in-memory, eventually writing back > with kinda "update *" queries to Cassandra in this table (so not updating > just a few columns but everything) can lead to these... Can it? > I tried to search around this sympthom but was not successful - so decided > to ask you guys maybe someone can give us a pointer... > > Some more info: > > - the table does not have TTL set - this mechanism is turned off > - the LIMIT param in upper query comes from paging size > - we are using Cassandra4 alpha3 > - we also have a few similarly built tables where we follow the above > described "update *" policy on write path - however those tables are > counter tables... when we mass-read them into memory we also go with > "select *" logic reading up tons of rows. The point is we never saw such a > warning for these counter tables however we are handling them same > fashion... ok counter tables work differently but still interesting to me > why those never generated things like this > > Tombstones are also implicitly created any time you insert or update a row which has an (unfrozen) collection column: list<>, map<> or set<>. This has to be done in order to ensure the new write replaces any existing collection entries. >From an in-depth blog post from TLP: https://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html Note: When using collections, range tombstones will be generated by INSERT and UPDATE operations every time you are using an entire collection, and not updating parts of it. Inserting a collection over an existing collection, rather than appending it or updating only an item in it, leads to range tombstones insert followed by the insert of the new values for the collection. This DELETE operation is hidden leading to some weird and frustrating tombstones issues. I'd check if you have any of those in the table involved and see if you could use a frozen collection instead, for example. -- Alex
