[jira] [Commented] (CASSANDRA-13127) Materialized Views: View row expires too soon
[ https://issues.apache.org/jira/browse/CASSANDRA-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091325#comment-16091325 ] ZhaoYang commented on CASSANDRA-13127: -- [~doanduyhai] yes, it's considered. There are 3 cases: TTLed base column as view's PK or view's filter condition will wipe entire view row. TTLed base column as selected in view will work as it is TTLed base column not selected in view, if there is no other live cells or primary key of view, view row is gone. > Materialized Views: View row expires too soon > - > > Key: CASSANDRA-13127 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13127 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Materialized Views >Reporter: Duarte Nunes >Assignee: ZhaoYang > > Consider the following commands, ran against trunk: > {code} > echo "DROP MATERIALIZED VIEW ks.mv; DROP TABLE ks.base;" | bin/cqlsh > echo "CREATE TABLE ks.base (p int, c int, v int, PRIMARY KEY (p, c));" | > bin/cqlsh > echo "CREATE MATERIALIZED VIEW ks.mv AS SELECT p, c FROM base WHERE p IS NOT > NULL AND c IS NOT NULL PRIMARY KEY (c, p);" | bin/cqlsh > echo "INSERT INTO ks.base (p, c) VALUES (0, 0) USING TTL 10;" | bin/cqlsh > # wait for row liveness to get closer to expiration > sleep 6; > echo "UPDATE ks.base USING TTL 8 SET v = 0 WHERE p = 0 and c = 0;" | bin/cqlsh > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 7 > (1 rows) > c | p > ---+--- > 0 | 0 > (1 rows) > # wait for row liveness to expire > sleep 4; > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 3 > (1 rows) > c | p > ---+--- > (0 rows) > {code} > Notice how the view row is removed even though the base row is still live. I > would say this is because in ViewUpdateGenerator#computeLivenessInfoForEntry > the TTLs are compared instead of the expiration times, but I'm not sure I'm > getting that far ahead in the code when updating a column that's not in the > view. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13127) Materialized Views: View row expires too soon
[ https://issues.apache.org/jira/browse/CASSANDRA-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091304#comment-16091304 ] DOAN DuyHai commented on CASSANDRA-13127: - Hey guys A question about TTLed columns. I want to make sure we do cover this scenario. When an TTLed column expired in the base table, on the next compaction, this TTLed column is replaced by a tombstone. Since all this TTLed -> tombstone mechanism only occurs during compaction, do we have something in the code path that *remove* the corresponding entry in the view ? ping [~jasonstack] [~fsander] > Materialized Views: View row expires too soon > - > > Key: CASSANDRA-13127 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13127 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Materialized Views >Reporter: Duarte Nunes >Assignee: ZhaoYang > > Consider the following commands, ran against trunk: > {code} > echo "DROP MATERIALIZED VIEW ks.mv; DROP TABLE ks.base;" | bin/cqlsh > echo "CREATE TABLE ks.base (p int, c int, v int, PRIMARY KEY (p, c));" | > bin/cqlsh > echo "CREATE MATERIALIZED VIEW ks.mv AS SELECT p, c FROM base WHERE p IS NOT > NULL AND c IS NOT NULL PRIMARY KEY (c, p);" | bin/cqlsh > echo "INSERT INTO ks.base (p, c) VALUES (0, 0) USING TTL 10;" | bin/cqlsh > # wait for row liveness to get closer to expiration > sleep 6; > echo "UPDATE ks.base USING TTL 8 SET v = 0 WHERE p = 0 and c = 0;" | bin/cqlsh > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 7 > (1 rows) > c | p > ---+--- > 0 | 0 > (1 rows) > # wait for row liveness to expire > sleep 4; > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 3 > (1 rows) > c | p > ---+--- > (0 rows) > {code} > Notice how the view row is removed even though the base row is still live. I > would say this is because in ViewUpdateGenerator#computeLivenessInfoForEntry > the TTLs are compared instead of the expiration times, but I'm not sure I'm > getting that far ahead in the code when updating a column that's not in the > view. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13127) Materialized Views: View row expires too soon
[ https://issues.apache.org/jira/browse/CASSANDRA-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090043#comment-16090043 ] Fridtjof Sander commented on CASSANDRA-13127: - bq. More precisely only for base column/cell that has timestamp and not used in MV. (In cases of insertion, no column timestamp) You win. bq. For now, base table with view doesn't support drop columns. but there is a ticket open to support alter MV. I just checked it, and that restriction is actually only checked via {{view.includes(columnName)}} which is not the case here. Anyway, it's probably a little to early in the discussion for that kind of detail, we can just set a mental mark to remember this, I guess. > Materialized Views: View row expires too soon > - > > Key: CASSANDRA-13127 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13127 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Materialized Views >Reporter: Duarte Nunes >Assignee: ZhaoYang > > Consider the following commands, ran against trunk: > {code} > echo "DROP MATERIALIZED VIEW ks.mv; DROP TABLE ks.base;" | bin/cqlsh > echo "CREATE TABLE ks.base (p int, c int, v int, PRIMARY KEY (p, c));" | > bin/cqlsh > echo "CREATE MATERIALIZED VIEW ks.mv AS SELECT p, c FROM base WHERE p IS NOT > NULL AND c IS NOT NULL PRIMARY KEY (c, p);" | bin/cqlsh > echo "INSERT INTO ks.base (p, c) VALUES (0, 0) USING TTL 10;" | bin/cqlsh > # wait for row liveness to get closer to expiration > sleep 6; > echo "UPDATE ks.base USING TTL 8 SET v = 0 WHERE p = 0 and c = 0;" | bin/cqlsh > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 7 > (1 rows) > c | p > ---+--- > 0 | 0 > (1 rows) > # wait for row liveness to expire > sleep 4; > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 3 > (1 rows) > c | p > ---+--- > (0 rows) > {code} > Notice how the view row is removed even though the base row is still live. I > would say this is because in ViewUpdateGenerator#computeLivenessInfoForEntry > the TTLs are compared instead of the expiration times, but I'm not sure I'm > getting that far ahead in the code when updating a column that's not in the > view. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13127) Materialized Views: View row expires too soon
[ https://issues.apache.org/jira/browse/CASSANDRA-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090024#comment-16090024 ] ZhaoYang commented on CASSANDRA-13127: -- For now, base table with view doesn't support drop columns. but there is a ticket open to support alter MV. More precisely only for base column/cell that has timestamp and not used in MV. (In cases of insertion, no column timestamp) Good to carry column->timestamp mapping. > Materialized Views: View row expires too soon > - > > Key: CASSANDRA-13127 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13127 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Materialized Views >Reporter: Duarte Nunes >Assignee: ZhaoYang > > Consider the following commands, ran against trunk: > {code} > echo "DROP MATERIALIZED VIEW ks.mv; DROP TABLE ks.base;" | bin/cqlsh > echo "CREATE TABLE ks.base (p int, c int, v int, PRIMARY KEY (p, c));" | > bin/cqlsh > echo "CREATE MATERIALIZED VIEW ks.mv AS SELECT p, c FROM base WHERE p IS NOT > NULL AND c IS NOT NULL PRIMARY KEY (c, p);" | bin/cqlsh > echo "INSERT INTO ks.base (p, c) VALUES (0, 0) USING TTL 10;" | bin/cqlsh > # wait for row liveness to get closer to expiration > sleep 6; > echo "UPDATE ks.base USING TTL 8 SET v = 0 WHERE p = 0 and c = 0;" | bin/cqlsh > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 7 > (1 rows) > c | p > ---+--- > 0 | 0 > (1 rows) > # wait for row liveness to expire > sleep 4; > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 3 > (1 rows) > c | p > ---+--- > (0 rows) > {code} > Notice how the view row is removed even though the base row is still live. I > would say this is because in ViewUpdateGenerator#computeLivenessInfoForEntry > the TTLs are compared instead of the expiration times, but I'm not sure I'm > getting that far ahead in the code when updating a column that's not in the > view. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13127) Materialized Views: View row expires too soon
[ https://issues.apache.org/jira/browse/CASSANDRA-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1609#comment-1609 ] Fridtjof Sander commented on CASSANDRA-13127: - Exactly, but only from those columns not projected into the view. The timestamps of all projected columns are available from the cells as usual. Would we also need to carry information to identify from which column a timestamp comes from, in case a unprojected base-column is dropped and hence becomes irrelevant? > Materialized Views: View row expires too soon > - > > Key: CASSANDRA-13127 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13127 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Materialized Views >Reporter: Duarte Nunes >Assignee: ZhaoYang > > Consider the following commands, ran against trunk: > {code} > echo "DROP MATERIALIZED VIEW ks.mv; DROP TABLE ks.base;" | bin/cqlsh > echo "CREATE TABLE ks.base (p int, c int, v int, PRIMARY KEY (p, c));" | > bin/cqlsh > echo "CREATE MATERIALIZED VIEW ks.mv AS SELECT p, c FROM base WHERE p IS NOT > NULL AND c IS NOT NULL PRIMARY KEY (c, p);" | bin/cqlsh > echo "INSERT INTO ks.base (p, c) VALUES (0, 0) USING TTL 10;" | bin/cqlsh > # wait for row liveness to get closer to expiration > sleep 6; > echo "UPDATE ks.base USING TTL 8 SET v = 0 WHERE p = 0 and c = 0;" | bin/cqlsh > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 7 > (1 rows) > c | p > ---+--- > 0 | 0 > (1 rows) > # wait for row liveness to expire > sleep 4; > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 3 > (1 rows) > c | p > ---+--- > (0 rows) > {code} > Notice how the view row is removed even though the base row is still live. I > would say this is because in ViewUpdateGenerator#computeLivenessInfoForEntry > the TTLs are compared instead of the expiration times, but I'm not sure I'm > getting that far ahead in the code when updating a column that's not in the > view. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13127) Materialized Views: View row expires too soon
[ https://issues.apache.org/jira/browse/CASSANDRA-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089988#comment-16089988 ] ZhaoYang commented on CASSANDRA-13127: -- Then single base column timestamp in {{shadowable-liveness-info or shadowable-tombstone}} won't be sufficient. All base columns' timestamp are required in {{shadowable-liveness-info or shadowable tombstone}}. > Materialized Views: View row expires too soon > - > > Key: CASSANDRA-13127 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13127 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Materialized Views >Reporter: Duarte Nunes >Assignee: ZhaoYang > > Consider the following commands, ran against trunk: > {code} > echo "DROP MATERIALIZED VIEW ks.mv; DROP TABLE ks.base;" | bin/cqlsh > echo "CREATE TABLE ks.base (p int, c int, v int, PRIMARY KEY (p, c));" | > bin/cqlsh > echo "CREATE MATERIALIZED VIEW ks.mv AS SELECT p, c FROM base WHERE p IS NOT > NULL AND c IS NOT NULL PRIMARY KEY (c, p);" | bin/cqlsh > echo "INSERT INTO ks.base (p, c) VALUES (0, 0) USING TTL 10;" | bin/cqlsh > # wait for row liveness to get closer to expiration > sleep 6; > echo "UPDATE ks.base USING TTL 8 SET v = 0 WHERE p = 0 and c = 0;" | bin/cqlsh > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 7 > (1 rows) > c | p > ---+--- > 0 | 0 > (1 rows) > # wait for row liveness to expire > sleep 4; > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 3 > (1 rows) > c | p > ---+--- > (0 rows) > {code} > Notice how the view row is removed even though the base row is still live. I > would say this is because in ViewUpdateGenerator#computeLivenessInfoForEntry > the TTLs are compared instead of the expiration times, but I'm not sure I'm > getting that far ahead in the code when updating a column that's not in the > view. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13127) Materialized Views: View row expires too soon
[ https://issues.apache.org/jira/browse/CASSANDRA-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089940#comment-16089940 ] Fridtjof Sander commented on CASSANDRA-13127: - I found another bug, that I believe falls into the same ballpark as this issue: {code:java} @Test public void testFilterViewTombstones() throws Throwable { execute("USE " + keyspace()); executeNet(protocolVersion, "USE " + keyspace()); createTable("CREATE TABLE %s (k int, c int, a int, b int, PRIMARY KEY (k, c))"); createView("mv", "CREATE MATERIALIZED VIEW %s AS SELECT k,c FROM %%s WHERE k IS NOT NULL AND c IS NOT NULL PRIMARY KEY (k,c)"); updateView("UPDATE %s USING TIMESTAMP 10 SET b=1 WHERE k=1 AND c=1"); assertRows(execute("SELECT * from %s"), row(1, 1, null, 1)); assertRows(execute("SELECT * from mv"), row(1, 1)); updateView("DELETE b FROM %s USING TIMESTAMP 11 WHERE k=1 AND c=1"); assertEmpty(execute("SELECT * from %s")); assertEmpty(execute("SELECT * from mv")); updateView("UPDATE %s USING TIMESTAMP 1 SET a=1 WHERE k=1 AND c=1"); assertRows(execute("SELECT * from %s"), row(1, 1, 1, null)); assertRows(execute("SELECT * from mv"), row(1, 1)); } {code} A view entry is created based on {{b=1@ts=10}} and deleted with {{ts=11}}. Later another cell is added with {{a=1@ts=1}}. The view's liveness-info with {{ts=1}} does not supersede the previously created tombstone with {{ts=11}}. If you disagree on the similarity, I'll open a new ticket for this. Both issues are about handling the contribution of base-cells to the view-row's liveness that are projected out of the view. On trunk, this doesn't even create an mv-entry (because of the empty liveness-info), but also fails with [~jasonstack]'s patch. I'm not sure, if this can be fixed at all without keeping hidden timestamp/liveness-data of discarded columns. > Materialized Views: View row expires too soon > - > > Key: CASSANDRA-13127 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13127 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Materialized Views >Reporter: Duarte Nunes >Assignee: ZhaoYang > > Consider the following commands, ran against trunk: > {code} > echo "DROP MATERIALIZED VIEW ks.mv; DROP TABLE ks.base;" | bin/cqlsh > echo "CREATE TABLE ks.base (p int, c int, v int, PRIMARY KEY (p, c));" | > bin/cqlsh > echo "CREATE MATERIALIZED VIEW ks.mv AS SELECT p, c FROM base WHERE p IS NOT > NULL AND c IS NOT NULL PRIMARY KEY (c, p);" | bin/cqlsh > echo "INSERT INTO ks.base (p, c) VALUES (0, 0) USING TTL 10;" | bin/cqlsh > # wait for row liveness to get closer to expiration > sleep 6; > echo "UPDATE ks.base USING TTL 8 SET v = 0 WHERE p = 0 and c = 0;" | bin/cqlsh > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 7 > (1 rows) > c | p > ---+--- > 0 | 0 > (1 rows) > # wait for row liveness to expire > sleep 4; > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 3 > (1 rows) > c | p > ---+--- > (0 rows) > {code} > Notice how the view row is removed even though the base row is still live. I > would say this is because in ViewUpdateGenerator#computeLivenessInfoForEntry > the TTLs are compared instead of the expiration times, but I'm not sure I'm > getting that far ahead in the code when updating a column that's not in the > view. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13127) Materialized Views: View row expires too soon
[ https://issues.apache.org/jira/browse/CASSANDRA-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089502#comment-16089502 ] ZhaoYang commented on CASSANDRA-13127: -- Updated the patch. Now MV should handle partial update properly, UPDATE semantic is now same as INSERT which means as long as data exists in base, the data should exist in view. It introduces {{Shadowable-LivenessInfo}}. if {{livenessInfo}} is {{shadowable}} and {{not live}}, all columns in the row are considered deleted. And this {{Shadowable-LivenessInfo}} can be further extended for CASSANDRA-11500 to solve view timestamp tie. > Materialized Views: View row expires too soon > - > > Key: CASSANDRA-13127 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13127 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Materialized Views >Reporter: Duarte Nunes >Assignee: ZhaoYang > > Consider the following commands, ran against trunk: > {code} > echo "DROP MATERIALIZED VIEW ks.mv; DROP TABLE ks.base;" | bin/cqlsh > echo "CREATE TABLE ks.base (p int, c int, v int, PRIMARY KEY (p, c));" | > bin/cqlsh > echo "CREATE MATERIALIZED VIEW ks.mv AS SELECT p, c FROM base WHERE p IS NOT > NULL AND c IS NOT NULL PRIMARY KEY (c, p);" | bin/cqlsh > echo "INSERT INTO ks.base (p, c) VALUES (0, 0) USING TTL 10;" | bin/cqlsh > # wait for row liveness to get closer to expiration > sleep 6; > echo "UPDATE ks.base USING TTL 8 SET v = 0 WHERE p = 0 and c = 0;" | bin/cqlsh > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 7 > (1 rows) > c | p > ---+--- > 0 | 0 > (1 rows) > # wait for row liveness to expire > sleep 4; > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 3 > (1 rows) > c | p > ---+--- > (0 rows) > {code} > Notice how the view row is removed even though the base row is still live. I > would say this is because in ViewUpdateGenerator#computeLivenessInfoForEntry > the TTLs are compared instead of the expiration times, but I'm not sure I'm > getting that far ahead in the code when updating a column that's not in the > view. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13127) Materialized Views: View row expires too soon
[ https://issues.apache.org/jira/browse/CASSANDRA-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083764#comment-16083764 ] ZhaoYang commented on CASSANDRA-13127: -- {{when updating base's columns which are not used in view.}} is about whether we should consider `Update semantic the same as Insert`. (for now it's not the same. update-statement has no primary key liveness info. pk's liveness of update-statement is depending on liveness of normal columns) eg. {quote} base table: create table ks.base (p int, c int, v int, primary key (p, c)) view table: select p,c from ks.base ... primary key (c, p) an update query on base table normal column {{v}} will not generate any rows.. a delete query on base table normal column {{v}} will remove the base row. {quote} imo, to avoid user confusion, it's better to keep view matched with base regardless semantic of update..(the current patch is not general enough to cover all such cases yet) [~tjake][~slebresne] what do you think? > Materialized Views: View row expires too soon > - > > Key: CASSANDRA-13127 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13127 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Materialized Views >Reporter: Duarte Nunes >Assignee: ZhaoYang > > Consider the following commands, ran against trunk: > {code} > echo "DROP MATERIALIZED VIEW ks.mv; DROP TABLE ks.base;" | bin/cqlsh > echo "CREATE TABLE ks.base (p int, c int, v int, PRIMARY KEY (p, c));" | > bin/cqlsh > echo "CREATE MATERIALIZED VIEW ks.mv AS SELECT p, c FROM base WHERE p IS NOT > NULL AND c IS NOT NULL PRIMARY KEY (c, p);" | bin/cqlsh > echo "INSERT INTO ks.base (p, c) VALUES (0, 0) USING TTL 10;" | bin/cqlsh > # wait for row liveness to get closer to expiration > sleep 6; > echo "UPDATE ks.base USING TTL 8 SET v = 0 WHERE p = 0 and c = 0;" | bin/cqlsh > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 7 > (1 rows) > c | p > ---+--- > 0 | 0 > (1 rows) > # wait for row liveness to expire > sleep 4; > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 3 > (1 rows) > c | p > ---+--- > (0 rows) > {code} > Notice how the view row is removed even though the base row is still live. I > would say this is because in ViewUpdateGenerator#computeLivenessInfoForEntry > the TTLs are compared instead of the expiration times, but I'm not sure I'm > getting that far ahead in the code when updating a column that's not in the > view. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13127) Materialized Views: View row expires too soon
[ https://issues.apache.org/jira/browse/CASSANDRA-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079873#comment-16079873 ] ZhaoYang commented on CASSANDRA-13127: -- [~tjake] yes, it will have to read existing data more often. the extra cases are: when updating base's columns which are not used in view. If a user has multiple views on a base, read-before-write is very likely to happen, the original optimization won't help too much. another naive way is totally disable TTL on table with MV. because it may not make sense to denormalize a short-life data.. > Materialized Views: View row expires too soon > - > > Key: CASSANDRA-13127 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13127 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Materialized Views >Reporter: Duarte Nunes >Assignee: ZhaoYang > > Consider the following commands, ran against trunk: > {code} > echo "DROP MATERIALIZED VIEW ks.mv; DROP TABLE ks.base;" | bin/cqlsh > echo "CREATE TABLE ks.base (p int, c int, v int, PRIMARY KEY (p, c));" | > bin/cqlsh > echo "CREATE MATERIALIZED VIEW ks.mv AS SELECT p, c FROM base WHERE p IS NOT > NULL AND c IS NOT NULL PRIMARY KEY (c, p);" | bin/cqlsh > echo "INSERT INTO ks.base (p, c) VALUES (0, 0) USING TTL 10;" | bin/cqlsh > # wait for row liveness to get closer to expiration > sleep 6; > echo "UPDATE ks.base USING TTL 8 SET v = 0 WHERE p = 0 and c = 0;" | bin/cqlsh > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 7 > (1 rows) > c | p > ---+--- > 0 | 0 > (1 rows) > # wait for row liveness to expire > sleep 4; > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 3 > (1 rows) > c | p > ---+--- > (0 rows) > {code} > Notice how the view row is removed even though the base row is still live. I > would say this is because in ViewUpdateGenerator#computeLivenessInfoForEntry > the TTLs are compared instead of the expiration times, but I'm not sure I'm > getting that far ahead in the code when updating a column that's not in the > view. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13127) Materialized Views: View row expires too soon
[ https://issues.apache.org/jira/browse/CASSANDRA-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16078206#comment-16078206 ] T Jake Luciani commented on CASSANDRA-13127: [~jasonstack] What impact will the change to the update statement have? My guess it we would potentially read the existing data a lot more often. > Materialized Views: View row expires too soon > - > > Key: CASSANDRA-13127 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13127 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Materialized Views >Reporter: Duarte Nunes >Assignee: ZhaoYang > > Consider the following commands, ran against trunk: > {code} > echo "DROP MATERIALIZED VIEW ks.mv; DROP TABLE ks.base;" | bin/cqlsh > echo "CREATE TABLE ks.base (p int, c int, v int, PRIMARY KEY (p, c));" | > bin/cqlsh > echo "CREATE MATERIALIZED VIEW ks.mv AS SELECT p, c FROM base WHERE p IS NOT > NULL AND c IS NOT NULL PRIMARY KEY (c, p);" | bin/cqlsh > echo "INSERT INTO ks.base (p, c) VALUES (0, 0) USING TTL 10;" | bin/cqlsh > # wait for row liveness to get closer to expiration > sleep 6; > echo "UPDATE ks.base USING TTL 8 SET v = 0 WHERE p = 0 and c = 0;" | bin/cqlsh > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 7 > (1 rows) > c | p > ---+--- > 0 | 0 > (1 rows) > # wait for row liveness to expire > sleep 4; > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 3 > (1 rows) > c | p > ---+--- > (0 rows) > {code} > Notice how the view row is removed even though the base row is still live. I > would say this is because in ViewUpdateGenerator#computeLivenessInfoForEntry > the TTLs are compared instead of the expiration times, but I'm not sure I'm > getting that far ahead in the code when updating a column that's not in the > view. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13127) Materialized Views: View row expires too soon
[ https://issues.apache.org/jira/browse/CASSANDRA-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062967#comment-16062967 ] ZhaoYang commented on CASSANDRA-13127: -- || source || junit-result || dtest-result|| | [3.11|https://github.com/jasonstack/cassandra/commits/CASSANDRA-13127-trunk] | [utest|https://circleci.com/gh/jasonstack/cassandra/57] | | | [dtest| https://github.com/jasonstack/cassandra-dtest/commits/CASSANDRA-13127 ] | \ | \ | 1. View.mayBeAffectedBy will return true as long as view-key-values are not filtered 2. LivenessInfo.supersedes(another) will check localDeletionTime if timestamps are the same. greater localDeletionTimestamp supersedes. (for primary key) 3. Conflicts.resolveRegular will return LEFT_WIN if left is alive. (for cells) > Materialized Views: View row expires too soon > - > > Key: CASSANDRA-13127 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13127 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Materialized Views >Reporter: Duarte Nunes >Assignee: ZhaoYang > > Consider the following commands, ran against trunk: > {code} > echo "DROP MATERIALIZED VIEW ks.mv; DROP TABLE ks.base;" | bin/cqlsh > echo "CREATE TABLE ks.base (p int, c int, v int, PRIMARY KEY (p, c));" | > bin/cqlsh > echo "CREATE MATERIALIZED VIEW ks.mv AS SELECT p, c FROM base WHERE p IS NOT > NULL AND c IS NOT NULL PRIMARY KEY (c, p);" | bin/cqlsh > echo "INSERT INTO ks.base (p, c) VALUES (0, 0) USING TTL 10;" | bin/cqlsh > # wait for row liveness to get closer to expiration > sleep 6; > echo "UPDATE ks.base USING TTL 8 SET v = 0 WHERE p = 0 and c = 0;" | bin/cqlsh > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 7 > (1 rows) > c | p > ---+--- > 0 | 0 > (1 rows) > # wait for row liveness to expire > sleep 4; > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 3 > (1 rows) > c | p > ---+--- > (0 rows) > {code} > Notice how the view row is removed even though the base row is still live. I > would say this is because in ViewUpdateGenerator#computeLivenessInfoForEntry > the TTLs are compared instead of the expiration times, but I'm not sure I'm > getting that far ahead in the code when updating a column that's not in the > view. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13127) Materialized Views: View row expires too soon
[ https://issues.apache.org/jira/browse/CASSANDRA-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16062776#comment-16062776 ] ZhaoYang commented on CASSANDRA-13127: -- one more issue in Row.Merger: > if (row.primaryKeyLivenessInfo().supersedes(rowInfo)) // only timestamp is > used to compare >rowInfo = row.primaryKeyLivenessInfo(); 3. when timestamps are the same, whether TTL is considered or ignored, is depending which row is luckily compared first. > a. insert Row-A using timestamp 1; flush > b. insert Row-A using timestamp 1 and ttl 5; flush > c. sleep 6 second > d. check result, Row-A exists > reversed step a and b and try again. Row-A not longer exists. imo, when timestamps are same, livenessInfo with greater localDeletionTime should supersede another. > Materialized Views: View row expires too soon > - > > Key: CASSANDRA-13127 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13127 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Materialized Views >Reporter: Duarte Nunes > > Consider the following commands, ran against trunk: > {code} > echo "DROP MATERIALIZED VIEW ks.mv; DROP TABLE ks.base;" | bin/cqlsh > echo "CREATE TABLE ks.base (p int, c int, v int, PRIMARY KEY (p, c));" | > bin/cqlsh > echo "CREATE MATERIALIZED VIEW ks.mv AS SELECT p, c FROM base WHERE p IS NOT > NULL AND c IS NOT NULL PRIMARY KEY (c, p);" | bin/cqlsh > echo "INSERT INTO ks.base (p, c) VALUES (0, 0) USING TTL 10;" | bin/cqlsh > # wait for row liveness to get closer to expiration > sleep 6; > echo "UPDATE ks.base USING TTL 8 SET v = 0 WHERE p = 0 and c = 0;" | bin/cqlsh > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 7 > (1 rows) > c | p > ---+--- > 0 | 0 > (1 rows) > # wait for row liveness to expire > sleep 4; > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 3 > (1 rows) > c | p > ---+--- > (0 rows) > {code} > Notice how the view row is removed even though the base row is still live. I > would say this is because in ViewUpdateGenerator#computeLivenessInfoForEntry > the TTLs are compared instead of the expiration times, but I'm not sure I'm > getting that far ahead in the code when updating a column that's not in the > view. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13127) Materialized Views: View row expires too soon
[ https://issues.apache.org/jira/browse/CASSANDRA-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16002850#comment-16002850 ] Benjamin Roth commented on CASSANDRA-13127: --- [~zznate] I have never "stumbled upon" it but i was also never taking care of that. We also only use the default TTLs, so maybe this is a different thing. Sounds like it is worth investigating on it. I think there should be a consensus what the expected behaviour should be, then some tests should be written and then the desired behaviour should be implemented if it is not met, yet. Unfortunately I don't have the time at the moment to dig deep into that issue and go through all the details in the code to see what's going on here. Just from reading the description of the issue it totally looks like a bug - at least from a user's point of view. > Materialized Views: View row expires too soon > - > > Key: CASSANDRA-13127 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13127 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Materialized Views >Reporter: Duarte Nunes > > Consider the following commands, ran against trunk: > {code} > echo "DROP MATERIALIZED VIEW ks.mv; DROP TABLE ks.base;" | bin/cqlsh > echo "CREATE TABLE ks.base (p int, c int, v int, PRIMARY KEY (p, c));" | > bin/cqlsh > echo "CREATE MATERIALIZED VIEW ks.mv AS SELECT p, c FROM base WHERE p IS NOT > NULL AND c IS NOT NULL PRIMARY KEY (c, p);" | bin/cqlsh > echo "INSERT INTO ks.base (p, c) VALUES (0, 0) USING TTL 10;" | bin/cqlsh > # wait for row liveness to get closer to expiration > sleep 6; > echo "UPDATE ks.base USING TTL 8 SET v = 0 WHERE p = 0 and c = 0;" | bin/cqlsh > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 7 > (1 rows) > c | p > ---+--- > 0 | 0 > (1 rows) > # wait for row liveness to expire > sleep 4; > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 3 > (1 rows) > c | p > ---+--- > (0 rows) > {code} > Notice how the view row is removed even though the base row is still live. I > would say this is because in ViewUpdateGenerator#computeLivenessInfoForEntry > the TTLs are compared instead of the expiration times, but I'm not sure I'm > getting that far ahead in the code when updating a column that's not in the > view. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13127) Materialized Views: View row expires too soon
[ https://issues.apache.org/jira/browse/CASSANDRA-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16002718#comment-16002718 ] Nate McCall commented on CASSANDRA-13127: - [~brstgt] Thoughts/have you seen this on your cluster? > Materialized Views: View row expires too soon > - > > Key: CASSANDRA-13127 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13127 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Materialized Views >Reporter: Duarte Nunes > > Consider the following commands, ran against trunk: > {code} > echo "DROP MATERIALIZED VIEW ks.mv; DROP TABLE ks.base;" | bin/cqlsh > echo "CREATE TABLE ks.base (p int, c int, v int, PRIMARY KEY (p, c));" | > bin/cqlsh > echo "CREATE MATERIALIZED VIEW ks.mv AS SELECT p, c FROM base WHERE p IS NOT > NULL AND c IS NOT NULL PRIMARY KEY (c, p);" | bin/cqlsh > echo "INSERT INTO ks.base (p, c) VALUES (0, 0) USING TTL 10;" | bin/cqlsh > # wait for row liveness to get closer to expiration > sleep 6; > echo "UPDATE ks.base USING TTL 8 SET v = 0 WHERE p = 0 and c = 0;" | bin/cqlsh > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 7 > (1 rows) > c | p > ---+--- > 0 | 0 > (1 rows) > # wait for row liveness to expire > sleep 4; > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 3 > (1 rows) > c | p > ---+--- > (0 rows) > {code} > Notice how the view row is removed even though the base row is still live. I > would say this is because in ViewUpdateGenerator#computeLivenessInfoForEntry > the TTLs are compared instead of the expiration times, but I'm not sure I'm > getting that far ahead in the code when updating a column that's not in the > view. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13127) Materialized Views: View row expires too soon
[ https://issues.apache.org/jira/browse/CASSANDRA-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16002662#comment-16002662 ] ZhaoYang commented on CASSANDRA-13127: -- > I still think it's a bug that a live row exists in the base table, but not in > the view. As a user point of view, I agree. > Materialized Views: View row expires too soon > - > > Key: CASSANDRA-13127 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13127 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Materialized Views >Reporter: Duarte Nunes > > Consider the following commands, ran against trunk: > {code} > echo "DROP MATERIALIZED VIEW ks.mv; DROP TABLE ks.base;" | bin/cqlsh > echo "CREATE TABLE ks.base (p int, c int, v int, PRIMARY KEY (p, c));" | > bin/cqlsh > echo "CREATE MATERIALIZED VIEW ks.mv AS SELECT p, c FROM base WHERE p IS NOT > NULL AND c IS NOT NULL PRIMARY KEY (c, p);" | bin/cqlsh > echo "INSERT INTO ks.base (p, c) VALUES (0, 0) USING TTL 10;" | bin/cqlsh > # wait for row liveness to get closer to expiration > sleep 6; > echo "UPDATE ks.base USING TTL 8 SET v = 0 WHERE p = 0 and c = 0;" | bin/cqlsh > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 7 > (1 rows) > c | p > ---+--- > 0 | 0 > (1 rows) > # wait for row liveness to expire > sleep 4; > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 3 > (1 rows) > c | p > ---+--- > (0 rows) > {code} > Notice how the view row is removed even though the base row is still live. I > would say this is because in ViewUpdateGenerator#computeLivenessInfoForEntry > the TTLs are compared instead of the expiration times, but I'm not sure I'm > getting that far ahead in the code when updating a column that's not in the > view. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13127) Materialized Views: View row expires too soon
[ https://issues.apache.org/jira/browse/CASSANDRA-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16002660#comment-16002660 ] ZhaoYang commented on CASSANDRA-13127: -- UpdateStatement.java * addUpdateForKey(PartitionUpdate update, Clustering clustering, UpdateParameters params) {quote} // We update the row timestamp (ex-row marker) only on INSERT (#6782) // Further, COMPACT tables semantic differs from "CQL3" ones in that a row exists only if it has // a non-null column, so we don't want to set the row timestamp for them. if (type.isInsert() && cfm.isCQLTable()) params.addPrimaryKeyLivenessInfo(); {quote} > Materialized Views: View row expires too soon > - > > Key: CASSANDRA-13127 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13127 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Materialized Views >Reporter: Duarte Nunes > > Consider the following commands, ran against trunk: > {code} > echo "DROP MATERIALIZED VIEW ks.mv; DROP TABLE ks.base;" | bin/cqlsh > echo "CREATE TABLE ks.base (p int, c int, v int, PRIMARY KEY (p, c));" | > bin/cqlsh > echo "CREATE MATERIALIZED VIEW ks.mv AS SELECT p, c FROM base WHERE p IS NOT > NULL AND c IS NOT NULL PRIMARY KEY (c, p);" | bin/cqlsh > echo "INSERT INTO ks.base (p, c) VALUES (0, 0) USING TTL 10;" | bin/cqlsh > # wait for row liveness to get closer to expiration > sleep 6; > echo "UPDATE ks.base USING TTL 8 SET v = 0 WHERE p = 0 and c = 0;" | bin/cqlsh > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 7 > (1 rows) > c | p > ---+--- > 0 | 0 > (1 rows) > # wait for row liveness to expire > sleep 4; > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 3 > (1 rows) > c | p > ---+--- > (0 rows) > {code} > Notice how the view row is removed even though the base row is still live. I > would say this is because in ViewUpdateGenerator#computeLivenessInfoForEntry > the TTLs are compared instead of the expiration times, but I'm not sure I'm > getting that far ahead in the code when updating a column that's not in the > view. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13127) Materialized Views: View row expires too soon
[ https://issues.apache.org/jira/browse/CASSANDRA-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16002580#comment-16002580 ] Duarte Nunes commented on CASSANDRA-13127: -- What are the TTL issues you're mentioning? I still think it's a bug that a live row exists in the base table, but not in the view. > Materialized Views: View row expires too soon > - > > Key: CASSANDRA-13127 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13127 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Materialized Views >Reporter: Duarte Nunes > > Consider the following commands, ran against trunk: > {code} > echo "DROP MATERIALIZED VIEW ks.mv; DROP TABLE ks.base;" | bin/cqlsh > echo "CREATE TABLE ks.base (p int, c int, v int, PRIMARY KEY (p, c));" | > bin/cqlsh > echo "CREATE MATERIALIZED VIEW ks.mv AS SELECT p, c FROM base WHERE p IS NOT > NULL AND c IS NOT NULL PRIMARY KEY (c, p);" | bin/cqlsh > echo "INSERT INTO ks.base (p, c) VALUES (0, 0) USING TTL 10;" | bin/cqlsh > # wait for row liveness to get closer to expiration > sleep 6; > echo "UPDATE ks.base USING TTL 8 SET v = 0 WHERE p = 0 and c = 0;" | bin/cqlsh > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 7 > (1 rows) > c | p > ---+--- > 0 | 0 > (1 rows) > # wait for row liveness to expire > sleep 4; > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 3 > (1 rows) > c | p > ---+--- > (0 rows) > {code} > Notice how the view row is removed even though the base row is still live. I > would say this is because in ViewUpdateGenerator#computeLivenessInfoForEntry > the TTLs are compared instead of the expiration times, but I'm not sure I'm > getting that far ahead in the code when updating a column that's not in the > view. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13127) Materialized Views: View row expires too soon
[ https://issues.apache.org/jira/browse/CASSANDRA-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16002522#comment-16002522 ] ZhaoYang commented on CASSANDRA-13127: -- "UPDATE" semantic was designed to be different from "INSERT" to solve some TTL issue which will remove entire PK when only ttl of column is updated. In this ticket, in the update statement, there is no row-liveness info associated with update. ( so no base row lifetime...) > Materialized Views: View row expires too soon > - > > Key: CASSANDRA-13127 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13127 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths, Materialized Views >Reporter: Duarte Nunes > > Consider the following commands, ran against trunk: > {code} > echo "DROP MATERIALIZED VIEW ks.mv; DROP TABLE ks.base;" | bin/cqlsh > echo "CREATE TABLE ks.base (p int, c int, v int, PRIMARY KEY (p, c));" | > bin/cqlsh > echo "CREATE MATERIALIZED VIEW ks.mv AS SELECT p, c FROM base WHERE p IS NOT > NULL AND c IS NOT NULL PRIMARY KEY (c, p);" | bin/cqlsh > echo "INSERT INTO ks.base (p, c) VALUES (0, 0) USING TTL 10;" | bin/cqlsh > # wait for row liveness to get closer to expiration > sleep 6; > echo "UPDATE ks.base USING TTL 8 SET v = 0 WHERE p = 0 and c = 0;" | bin/cqlsh > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 7 > (1 rows) > c | p > ---+--- > 0 | 0 > (1 rows) > # wait for row liveness to expire > sleep 4; > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 3 > (1 rows) > c | p > ---+--- > (0 rows) > {code} > Notice how the view row is removed even though the base row is still live. I > would say this is because in ViewUpdateGenerator#computeLivenessInfoForEntry > the TTLs are compared instead of the expiration times, but I'm not sure I'm > getting that far ahead in the code when updating a column that's not in the > view. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13127) Materialized Views: View row expires too soon
[ https://issues.apache.org/jira/browse/CASSANDRA-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839597#comment-15839597 ] Duarte Nunes commented on CASSANDRA-13127: -- I think that the behaviour for an {{UPDATE}} should match that of an {{INSERT}}, since the latter is probably too late to change. So, I think a view row's lifetime should be tied to the base row's lifetime, and that's kind what the code tries to do (the view row liveness can be set to match that of a column not in the view). If an operation extends the lifetime of a base row, the corresponding views should be updated. Similarly, if an update resurrects a base row, the view row should also be resurrected even if it doesn't include the particular column that triggered it. > Materialized Views: View row expires too soon > - > > Key: CASSANDRA-13127 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13127 > Project: Cassandra > Issue Type: Bug >Reporter: Duarte Nunes > > Consider the following commands, ran against trunk: > {code} > echo "DROP MATERIALIZED VIEW ks.mv; DROP TABLE ks.base;" | bin/cqlsh > echo "CREATE TABLE ks.base (p int, c int, v int, PRIMARY KEY (p, c));" | > bin/cqlsh > echo "CREATE MATERIALIZED VIEW ks.mv AS SELECT p, c FROM base WHERE p IS NOT > NULL AND c IS NOT NULL PRIMARY KEY (c, p);" | bin/cqlsh > echo "INSERT INTO ks.base (p, c) VALUES (0, 0) USING TTL 10;" | bin/cqlsh > # wait for row liveness to get closer to expiration > sleep 6; > echo "UPDATE ks.base USING TTL 8 SET v = 0 WHERE p = 0 and c = 0;" | bin/cqlsh > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 7 > (1 rows) > c | p > ---+--- > 0 | 0 > (1 rows) > # wait for row liveness to expire > sleep 4; > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 3 > (1 rows) > c | p > ---+--- > (0 rows) > {code} > Notice how the view row is removed even though the base row is still live. I > would say this is because in ViewUpdateGenerator#computeLivenessInfoForEntry > the TTLs are compared instead of the expiration times, but I'm not sure I'm > getting that far ahead in the code when updating a column that's not in the > view. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13127) Materialized Views: View row expires too soon
[ https://issues.apache.org/jira/browse/CASSANDRA-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837348#comment-15837348 ] Vincent White commented on CASSANDRA-13127: --- I think I pretty much understand what's happening here. It basically all stems from the base upsert behaviour (creating a row via {{UPDATE}} so the primary key columns don't exist on their own vs {{INSERT}}). I'm still not sure it matches the MV docs though and the comments in the code say things like: {code}1) either the columns for the base and view PKs are exactly the same: in that case, the view entry should live as long as the base row lives. This means the view entry should only expire once *everything* in the base row has expired. Which means the row TTL should be the max of any other TTL.{code} I think the logic in {{computeLivenessInfoForEntry}} doesn't make sense for updates because it only ever expected inserts. It leads to some funky behaviour if you're mixing updates, inserts and TTL's. I didn't test with deletes but I guees they could cause similar results. Simply patching computeLivenessInfoForEntry like: {code:title=ViewUpdateGenerator.java#computeLivenessInfoForEntry} int expirationTime = baseLiveness.localExpirationTime(); for (Cell cell : baseRow.cells()) { -if (cell.ttl() > ttl) +if (cell.localDeletionTime() > expirationTime) { ttl = cell.ttl(); expirationTime = cell.localDeletionTime(); } } -return ttl == baseLiveness.ttl() +return expirationTime == baseLiveness.localExpirationTime() ? baseLiveness : LivenessInfo.withExpirationTime(baseLiveness.timestamp(), ttl, expirationTime); } {code} isn't enough because it leads to further unexpected behaviour where update statements will resurrect previously TTL'd MV entries in some cases. If an update statement sets a column that could cause the update of _any_ view in that keyspace it will resurrect entries in views that have PK's made up of only columns from the base PK, regardless of whether the statement updates non-PK columns in that view. If the update statement only sets values of columns that don't appear in the keyspace's MV's then no MV TTL'd entries for that PK will be resurrected. If there was never an entry in the MV for that MV PK then it won't create a a new one. This is because upserts don't create new MV entries unless they set the value of a non-PK column in that view (with or without this patch). I don't think I've seen it referenced anywhere but is that intended behaviour when using upserts and materialized views? That an {{UPDATE}} to a column not in a view will not create an entry in an MV if the veiw's PK is only made up of columns from the base table's PK, but the matching {{INSERT}} statement will? > Materialized Views: View row expires too soon > - > > Key: CASSANDRA-13127 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13127 > Project: Cassandra > Issue Type: Bug >Reporter: Duarte Nunes > > Consider the following commands, ran against trunk: > {code} > echo "DROP MATERIALIZED VIEW ks.mv; DROP TABLE ks.base;" | bin/cqlsh > echo "CREATE TABLE ks.base (p int, c int, v int, PRIMARY KEY (p, c));" | > bin/cqlsh > echo "CREATE MATERIALIZED VIEW ks.mv AS SELECT p, c FROM base WHERE p IS NOT > NULL AND c IS NOT NULL PRIMARY KEY (c, p);" | bin/cqlsh > echo "INSERT INTO ks.base (p, c) VALUES (0, 0) USING TTL 10;" | bin/cqlsh > # wait for row liveness to get closer to expiration > sleep 6; > echo "UPDATE ks.base USING TTL 8 SET v = 0 WHERE p = 0 and c = 0;" | bin/cqlsh > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 7 > (1 rows) > c | p > ---+--- > 0 | 0 > (1 rows) > # wait for row liveness to expire > sleep 4; > echo "SELECT p, c, ttl(v) FROM ks.base; SELECT * FROM ks.mv;" | bin/cqlsh > p | c | ttl(v) > ---+---+ > 0 | 0 | 3 > (1 rows) > c | p > ---+--- > (0 rows) > {code} > Notice how the view row is removed even though the base row is still live. I > would say this is because in ViewUpdateGenerator#computeLivenessInfoForEntry > the TTLs are compared instead of the expiration times, but I'm not sure I'm > getting that far ahead in the code when updating a column that's not in the > view. -- This message was sent by Atlassian JIRA (v6.3.4#6332)