subject:"\[jira\] \[Comment Edited\] \(CASSANDRA\-11500\) Obsolete MV entry may not be properly deleted"

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

2017-09-01 Thread ZhaoYang (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150303#comment-16150303
 ] 

ZhaoYang edited comment on CASSANDRA-11500 at 9/1/17 12:27 PM:
---

[~pauloricardomg] thanks for the feedback (y)

bq. I wasn't very comfortable with our previous approach of enforcing strict 
liveness during row merge, since it changes a lot of low-level 
structures/interfaces (like BTreeRow/MergeListener, etc) to enforce a 
table-level setting. Since we'll probably get rid of this when doing a proper 
implementation of virtual cells , I updated on this commit to perform the 
filtering during read instead which will give us the same result but with less 
change in unrelated code. Do you see any problem with this approach?

As we discussed offline, we need to make sure the raw data including tombstone, 
expired liveness are shipped to the coordinator side.  Enforcing strict 
liveness in {{ReadCommand.executeLocally()}} would remove the row before digest 
or data response. Instead, we add {{enforceStrictLiveness}} to {{Row.purge}} to 
get the same result but less interfaces changes for {{Row}}.

bq. One problem of replacing shadowable tombstones by expired liveness info is 
that it stores an additional unused ttl field for every shadowed view entry to 
solve the commutative view deletion problem. In order to avoid this I updated 
the patch to only use expired ttl when a shadowable tombstone would not work 
along with an explanation on why that is used since it's a hack

Shadowable tombstone will be deprecated and use expired livenessInfo if the 
deletion time is greater than merged-row deletion to avoid uncessary expired 
livenessInfo.

bq. in TableViews.java, the DeletionTracker should be applied even if existing 
has no data, eg. partition-deletion

It's tested by  "testRangeDeletionWithFlush()" in ViewTest. Without partition 
deletion info from deletion tracker, existing row is given as empty and it will 
resurrect deleted cells.

bq.  In order to prevent against this, I added a note to the Upgrading section 
of NEWS.txt explaining about this caveat and that running repair before the 
upgrade should be sufficient to avoid it.

(y)

| source | unit | [dtest| 
| [trunk|https://github.com/jasonstack/cassandra/commits/trunk-11500-squashed] 
|  https://circleci.com/gh/jasonstack/cassandra/551 | 
secondary_indexes_test.TestPreJoinCallback.resumt_test |
| 
[3.11|https://github.com/jasonstack/cassandra/commits/CASSANDRA-11500-strict-3.11]
 |  https://circleci.com/gh/jasonstack/cassandra/557 | 
counter_tests.TestCounters.test_13691 |
| 
[3.0|https://github.com/jasonstack/cassandra/commits/CASSANDRA-11500-strict-3.0]
 |  https://circleci.com/gh/jasonstack/cassandra/556|  
counter_tests.TestCounters.test_13691
authe_test.TestAuth.sysmtem_auth_ks_is_alterable_test |
| [dtest|https://github.com/riptano/cassandra-dtest/commits/11500-poc]|

Those failed dtests are not related.
{code}
Changes:
1. Using expired livenessInfo if computed deletion time is greater than merged 
row deletion. There are only 2 cases:
  a. non-pk base column used in view pk is removed by partial update or 
partial delete
  b. unselected base column is removed by partial update or partial delete
   
   Current shadowable tombstone is not used to avoid the issue of resurrecting 
deleted cells. We will expired-livenessInfo and merged base row deletion 
instead. 

2. It's strict-liveness iff there is non-key base column in view-pk. The 
existence of view row is solely base on this non-key base column.

3. If there is no non-pk base column in view-pk, the view's liveness/deletion 
is using max of base livenessIno + unselected column. unselected column's ttl 
is used only when it affects view row liveness. Selected columns won't 
contribute to livenessInfo or row deletion.
* this wouldn't support complex cases as explained above. eg. c/d 
unselected, update c@10, delete c@11, update d@5. view row should be alive but 
dead

4. in TableViews.java, the DeletionTracker should be applied even if existing 
has no data, eg. partition-deletion

5. When generating read command to read existing base data, need to query all 
base columns instead of view's queried column if base and view having same key 
columns to read unselected column. 
{code}


was (Author: jasonstack):
[~pauloricardomg] thanks for the feedback (y)

bq. I wasn't very comfortable with our previous approach of enforcing strict 
liveness during row merge, since it changes a lot of low-level 
structures/interfaces (like BTreeRow/MergeListener, etc) to enforce a 
table-level setting. Since we'll probably get rid of this when doing a proper 
implementation of virtual cells , I updated on this commit to perform the 
filtering during read instead which will give us the same result but with less 
change in unrelated code. Do you see any problem with this approach?

As

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

2017-08-16 Thread ZhaoYang (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124534#comment-16124534
]

ZhaoYang edited comment on CASSANDRA-11500 at 8/17/17 1:22 AM:
---

Thanks for reviewing and feedback.

Changing the semantic of MV and revising non-key column filtering
feature(CASSANDRA-10368) will indeed make it easier. It's a good idea to make a
simple non-disruptive change to stabilize basic features and wait for more
commiters involved.

Using an extended flag for {{Strict-Liveness}} will allow us to change to
future structure easily, either multiple livenessInfos or virtualcells.

About the {{Strict Liveness}} semantic:
* A strict row is only live iff it's row level liveness info is live,
regardless of the liveness of its columns.

My understanding is: view row is strict iff the view has non-key base row as
view pk. When it's {{Strict}}, the view's row liveness/deletion should use this
non-key base column's timestamp as well as ttl, unless there is a greater row
deletion.(It's like a simplified version of "VirtualCells" which only store
metadata for non-key base column in view pk)

For now, the semantic of MV:
* if it's strict(non-key base row as view pk), the existence of view row is
only with its row livenessInfo
* if it's not-strict, view row is alive if there is any live selected view
columns or live livenessInfo.

{code}
For 13127:
Unselected columns has no effect on liveness of view row, for now, till we
are ready for new design.
It cannot be properly supported without disruptive changes, like
VirtualCells or multiple livenessInfos.
The unsupported corner cases: if base row is removed after unselected column
removal(ttl or cell tombstone), we will have problem keeping the view row alive
again by setting smaller or equal timestamp on base livenessInfo or other
unselected column. (if updating smaller ts on selected column, view row will be
alive)
{code}

{code}
For 13547:
It's necessary to forbid dropping filtered columns from base columns.
The filtered column part needs to be reconsidered with 10368.
It cannot be properly supported without disruptive changes, like
VirtualCells or multiple livenessInfos
{code}

{code}
for 13409:
As paulo suggested, generating column tombstones when receiving a partial
update for a previously deleted row might be a non-disruptive solution if cell
tombstone can co-exist with row deletion which has greater timestamp.

It is resolved by avoid using shadowable tombstone, use expired
livenessInfo instead. We won't have the problem of resurrecting previous
deleted data.
{code}

{code}
Changes:
1. Removal is shadowable:
1.1 if the non-key base column in view-pk is updated or deleted by partial
update or partial delete. if this non-key column is removed by row deletion,
it's not shadowable.
1.2 if there is no non-key base column in view-pk.

Current shadowable tombstone is not used to avoid the issue of resurrecting
deleted cells. We will expired-livenessInfo instead.

2. It's strict-liveness iff there is non-key base column in view-pk. The
existence of view row is solely base on this non-key base column.

3. If there is no non-pk base column in view-pk, the view's liveness/deletion
is using max of base livenessIno + unselected column. unselected column's ttl
is used only when it affects view row liveness. Selected columns won't
contribute to livenessInfo or row deletion.
* this wouldn't support complex cases as explained above. eg. c/d
unselected, update c@10, delete c@11, update d@5. view row should be alive but
dead

4. in TableViews.java, the DeletionTracker should be applied even if existing
has no data, eg. partition-deletion

{code}

was (Author: jasonstack):
Thanks for reviewing and feedback.

Using an extended flag for {{Strict-Liveness}} will allow us to change to
future structure easily, either multiple livenessInfos or virtualcells.

About the {{Strict Liveness}} semantic:
* A strict row is only live iff it's row level liveness info is live,
regardless of the liveness of its columns.

For now, the

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

2017-08-14 Thread ZhaoYang (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124534#comment-16124534
 ] 

ZhaoYang edited comment on CASSANDRA-11500 at 8/14/17 12:30 PM:


Thanks for reviewing and feedback.

Changing the semantic of MV and revising non-key column filtering 
feature(CASSANDRA-10368) will indeed make it easier. It's a good idea to make a 
simple non-disruptive change to stabilize basic features and wait for more 
commiters involved.

Using an extended flag for {{Strict-Liveness}} will allow us to change to 
future structure easily, either multiple livenessInfos or virtualcells. 

About the {{Strict Liveness}} semantic:
* A strict row is only live iff it's row level liveness info is live, 
regardless of the liveness of its columns.

My understanding is: view row is strict iff the view has non-key base row as 
view pk. When it's {{Strict}}, the view's row liveness/deletion should use this 
non-key base column's timestamp as well as ttl, unless there is a greater row 
deletion.(It's like a simplified version of "VirtualCells" which only store 
metadata for non-key base column in view pk)

For now, the semantic of MV: 
* if it's strict(non-key base row as view pk), the existence of view row is 
only with its row livenessInfo
* if it's not-strict, view row is alive if there is any live selected view 
columns or live livenessInfo.

{code}
For 13127: 
   Unselected columns has no effect on liveness of view row, for now, till we 
are ready for new design.
   It cannot be properly supported without disruptive changes, like 
VirtualCells or multiple livenessInfos
{code}

{code}
For 13547:
It's necessary to forbid dropping filtered columns from base columns.
The filtered column part needs to be reconsidered with 10368.
It cannot be properly supported without disruptive changes, like 
VirtualCells or multiple livenessInfos
{code}

{code}
for 13409:
As paulo suggested, generating column tombstones when receiving a partial 
update for a previously deleted row might be a non-disruptive solution if cell 
tombstone can co-exist with row deletion which has greater timestamp.
I will reopen this ticket.
{code}

PATCH for 11500: 
| [trunk|https://github.com/jasonstack/cassandra/commits/11500-poc]|
| [dtest|https://github.com/riptano/cassandra-dtest/commits/11500-poc]| 

Changes:
1. deletion is shadowable if the non-key base column in view-pk is updated or 
deleted by partial update or partial delete. if this non-key column is removed 
by row deletion, it's not shadowable.
2. it's strict-liveness iff there is non-key base column in view-pk.
3. if it's not strict-liveness, the view's livenes/deletion is using max of all 
base columns. (this wouldn't support complex unselected columns. eg. c/d 
unselected, update c@10, delete c@11, update d@5. view row should be alive but 
dead)
4. in TableViews.java, the DeletionTracker should be applied even if one of the 
iterator has no data, eg. partition-deletion
5. sstabledump will include shadowable info



was (Author: jasonstack):
Thanks for reviewing and feedback.

Changing the semantic of MV and revising non-key column filtering 
feature(CASSANDRA-10368) will indeed make it easier. It's a good idea to make a 
simple non-disruptive change to stabilize basic features and wait for more 
commiters involved.

Using an extended flag for {{Strict-Liveness}} will allow us to change to 
future structure easily, either multiple livenessInfos or virtualcells. 

About the {{Strict Liveness}} semantic:
* A strict row is only live iff it's row level liveness info is live, 
regardless of the liveness of its columns.

My understanding is: view row is strict iff the view has non-key base row as 
view pk. When it's {{Strict}}, the view's row liveness/deletion should use this 
non-key base column's timestamp as well as ttl, unless there is a greater row 
deletion.(It's like a simplified version of "VirtualCells" which only store 
metadata for non-key base column in view pk)

For now, the semantic of MV: 
* if it's strict(non-key base row as view pk), the existence of view row is 
only with its row livenessInfo
* if it's not-strict, view row is alive if there is any live selected view 
columns or live livenessInfo.

{code}
For 13127: 
   Unselected columns has no effect on liveness of view row, for now, till we 
are ready for new design.
   It cannot be properly supported without disruptive changes, like 
VirtualCells or multiple livenessInfos
{code}

{code}
For 13547:
It's necessary to forbid dropping filtered columns from base columns.
The filtered column part needs to be reconsidered with 10368.
It cannot be properly supported without disruptive changes, like 
VirtualCells or multiple livenessInfos
{code}

{code}
for 13409:
As paulo suggested, generating column tombstones when receiving a partial 
update for a previously deleted

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

2017-08-09 Thread Paulo Motta (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119823#comment-16119823
 ] 

Paulo Motta edited comment on CASSANDRA-11500 at 8/9/17 12:49 PM:
--

Talking offline with Zhao, it seems like there is still an outstanding case 
derived from CASSANDRA-13547 not addressed by the strict liveness suggestion:

{code:none}
// liveness or deletion using max-timestamp of view-primary-key column in base
base:  (k), a, b, c
view:  (k, a), b, c=1

q1: insert (1,1,1,1) with timestamp 0

base: liveness=ts@0,  k=1, a=1@0, b=1@0, c=1@0
view: liveness=ts@0,  (k=1, a=1), b=1@0, c=1@0

q2: update c=1 with timestamp 10 where k = 1  

base: liveness=ts@0,  k=1, a=1@0, b=1@0, c=1@10
view: liveness=ts@0,  (k=1, a=1), b=1@0, c=1@10

q3: update c=2 with timestamp 11 where k = 1  

base: liveness=ts@0,  k=1, a=1@0, b=1@0, c=2@11
view:
  liveness=ts@0,  (k=1, a=1), b=1@0, c=1@10
  tombstone=ts@0,  (k=1, a=1)

  with strict-liveness flag, view row is dead

q4: update c=1 with timestamp 12 where k = 1  

base: liveness=ts@0,  k=1, a=1@0, b=1@0, c=1@12
view:
  liveness=ts@0,  (k=1, a=1), b=1@0, c=1@10
  tombstone=ts@0,  (k=1, a=1)
  liveness=ts@0,  (k=1, a=1), b=1@0, c=1@12
 
  view row should be live..but it's dead
{code}

It seems like this scenario where the row liveness depend on a non-view primary 
key was overlooked by CASSANDRA-10368 and seems to be analogous to the problem 
Tyler discovered on CASSANDRA-10226 (but with conditions rather than non-base 
view primary keys):

bq. It seems like when we include multiple non-PK columns in the view PK, we 
fundamentally have to accept that the view row's existence depends on multiple 
timestamps. I propose that we solve this by using a set of timestamps for the 
row's LivenessInfo.

The solution proposed on that ticket of keeping multiple deletion and liveness 
infos per primary key is similar to the virtual cells solution you 
independently came up (great job!). While I agree that a solution along those 
lines is the way to go moving forward, that's a pretty significant change in 
the storage engine which may introduce unforeseen problems, and would probably 
be nice to have [~slebresne] blessing given he seems to [feel 
strongly|https://issues.apache.org/jira/browse/CASSANDRA-10226?focusedCommentId=14740391=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14740391]
 about it and will likely want to chime in.

I personally think that before introducing disruptive changes to the storage 
engine and MV machinery to enable relatively new features (in this case, 
filtering on non-PK columns which didn't seem to have all of its repercussions 
considered on CASSANDRA-10368), we should take a conservative approach and 
spend our energy on stabilizing current MV features.

In practical terms, I'd suggest going with the simpler strict liveness approach 
I suggested above to fix the current problems (or any alternative which do not 
require disruptive changes on the storage engine) and disallow filtering on 
non-PK while the virtual cells are not implemented - MVs with it already 
enabled would not be affected but users would be susceptible to the problem 
above (we could maybe print a warning to inform this).

After we have current MV features stabilized we can then think of implementing 
the virtual cell idea to properly enable other features like filtering on 
non-view PK columns and support multiple non-PK cols in MV clustering key when 
partition key is shared (CASSANDRA-10226).

Please let me know what do you think.


was (Author: pauloricardomg):
Talking offline with Zhao, it seems like there is still an outstanding case 
derived from CASSANDRA-13547 not addressed by the strict liveness suggestion:

{code:none}
// liveness or deletion using max-timestamp of view-primary-key column in base
base:  (k), a, b, c
view:  (k, a), b, c=1

q1: insert (1,1,1,1) with timestamp 0

base: liveness=ts@0,  k=1, a=1@0, b=1@0, c=1@0
view: liveness=ts@0,  (k=1, a=1), b=1@0, c=1@0

q2: update c=1 with timestamp 10 where k = 1  

base: liveness=ts@0,  k=1, a=1@0, b=1@0, c=1@10
view: liveness=ts@0,  (k=1, a=1), b=1@0, c=1@10

q3: update c=2 with timestamp 11 where k = 1  

base: liveness=ts@0,  k=1, a=1@0, b=1@0, c=2@11
view:
  liveness=ts@0,  (k=1, a=1), b=1@0, c=1@10
  tombstone=ts@0,  (k=1, a=1)

  with strict-liveness flag, view row is dead

q4: update c=1 with timestamp 12 where k = 1  

base: liveness=ts@0,  k=1, a=1@0, b=1@0, c=1@12
view:
  liveness=ts@0,  (k=1, a=1), b=1@0, c=1@10
  tombstone=ts@0,  (k=1, a=1)
  liveness=ts@0,  (k=1, a=1), b=1@0, c=1@12
 
  view row should be live..but it's dead
{code}

It seems like this scenario where the row

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

2017-08-01 Thread ZhaoYang (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16101603#comment-16101603
 ] 

ZhaoYang edited comment on CASSANDRA-11500 at 8/1/17 7:17 AM:
--

WIP 
[branch|https://github.com/jasonstack/cassandra/commits/CASSANDRA-11500-cell] 
[dtest|https://github.com/riptano/cassandra-dtest/commits/CASSANDRA-11500]


Changed:
*  Extra "VirtualCell"(kind of special LivenessInfo for MV) in Row to maintain 
consistency of view row. It stores: 1. base column used in view PK and base 
column used in view filter conditions. if any of such column dead, entire view 
row dead, regardless LivenessInfo or DeletionTime status. or 2. unselected base 
columns. if any of such column alive, view's pk should be alive. 
* blocked dropping filter base column
* fix issue of creating view with token()  filter
* remove Row.Deletion, it was used to wrap DeletionTime with "Shadowable" flag
* fix missing partition deletion in view-update
* for now, ttl on non-frozen collections is not reflected in "VritualCells"

Todo: 
more dtest


was (Author: jasonstack):
WIP 
[branch|https://github.com/jasonstack/cassandra/commits/CASSANDRA-11500-cell] 
[dtest|https://github.com/riptano/cassandra-dtest/commits/CASSANDRA-11500]


Changed:
*  Extra "VirtualCell"(kind of special LivenessInfo for MV) in Row to maintain 
consistency of view row. It stores: 1. base column used in view PK and base 
column used in view filter conditions. if any of such column dead, entire view 
row dead, regardless LivenessInfo or DeletionTime status. or 2. unselected base 
columns. if any of such column alive, view's pk should be alive. 
* blocked dropping filter base column
* fix issue of creating view with token()  filter
* remove Row.Deletion, it was used to wrap DeletionTime with "Shadowable" flag
* fix missing partition deletion in view-update

Todo: 
more dtest

> Obsolete MV entry may not be properly deleted
> -
>
> Key: CASSANDRA-11500
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11500
> Project: Cassandra
>  Issue Type: Bug
>  Components: Materialized Views
>Reporter: Sylvain Lebresne
>Assignee: ZhaoYang
>
> When a Materialized View uses a non-PK base table column in its PK, if an 
> update changes that column value, we add the new view entry and remove the 
> old one. When doing that removal, the current code uses the same timestamp 
> than for the liveness info of the new entry, which is the max timestamp for 
> any columns participating to the view PK. This is not correct for the 
> deletion as the old view entry could have other columns with higher timestamp 
> which won't be deleted as can easily shown by the failing of the following 
> test:
> {noformat}
> CREATE TABLE t (k int PRIMARY KEY, a int, b int);
> CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS 
> NOT NULL PRIMARY KEY (k, a);
> INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
> UPDATE t USING TIMESTAMP 4 SET b = 2 WHERE k = 1;
> UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1;
> SELECT * FROM mv WHERE k = 1; // This currently return 2 entries, the old 
> (invalid) and the new one
> {noformat}
> So the correct timestamp to use for the deletion is the biggest timestamp in 
> the old view entry (which we know since we read the pre-existing base row), 
> and that is what CASSANDRA-11475 does (the test above thus doesn't fail on 
> that branch).
> Unfortunately, even then we can still have problems if further updates 
> requires us to overide the old entry. Consider the following case:
> {noformat}
> CREATE TABLE t (k int PRIMARY KEY, a int, b int);
> CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS 
> NOT NULL PRIMARY KEY (k, a);
> INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
> UPDATE t USING TIMESTAMP 10 SET b = 2 WHERE k = 1;
> UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1; // This will delete the 
> entry for a=1 with timestamp 10
> UPDATE t USING TIMESTAMP 3 SET a = 1 WHERE k = 1; // This needs to re-insert 
> an entry for a=1 but shouldn't be deleted by the prior deletion
> UPDATE t USING TIMESTAMP 4 SET a = 2 WHERE k = 1; // ... and we can play this 
> game more than once
> UPDATE t USING TIMESTAMP 5 SET a = 1 WHERE k = 1;
> ...
> {noformat}
> In a way, this is saying that the "shadowable" deletion mechanism is not 
> general enough: we need to be able to re-insert an entry when a prior one had 
> been deleted before, but we can't rely on timestamps being strictly bigger on 
> the re-insert. In that sense, this can be though as a similar problem than 
> CASSANDRA-10965, though the solution there of a single flag is not enough 
> since we can have to replace more than once.
> I think the proper solution would be to ship enough information to

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

2017-08-01 Thread ZhaoYang (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16101603#comment-16101603
 ] 

ZhaoYang edited comment on CASSANDRA-11500 at 8/1/17 6:27 AM:
--

WIP 
[branch|https://github.com/jasonstack/cassandra/commits/CASSANDRA-11500-cell] 
[dtest|https://github.com/riptano/cassandra-dtest/commits/CASSANDRA-11500]


Changed:
*  Extra "VirtualCell"(kind of special LivenessInfo for MV) in Row to maintain 
consistency of view row. It stores: 1. base column used in view PK and base 
column used in view filter conditions. if any of such column dead, entire view 
row dead, regardless LivenessInfo or DeletionTime status. or 2. unselected base 
columns. if any of such column alive, view's pk should be alive. 
* blocked dropping filter base column
* fix issue of creating view with token()  filter
* remove Row.Deletion, it was used to wrap DeletionTime with "Shadowable" flag
* fix missing partition deletion in view-update

Todo: 
more dtest


was (Author: jasonstack):
WIP 
[branch|https://github.com/jasonstack/cassandra/commits/CASSANDRA-11500-cell]

Changed:
*  Extra "VirtualCell"(kind of special LivenessInfo for MV) in Row to maintain 
liveness of view row. It stores: 1. base column used in view PK and base column 
used in view filter conditions. if any of such column dead, entire view row 
dead, regardless LivenessInfo or DeletionTime status. 2. unselected base 
columns. if any of such column alive, view's pk should be alive if it's not 
deleted by DeletionTime or those columns in <1>. 
* blocked dropping filter base column
* fix issue of creating view with token()  filter
* remove Row.Deletion, it was used to wrap DeletionTime with "Shadowable" flag

Todo: 
optimize in-memory/storage representation for "virtual-cells" to re-use 
AbstractRow/BTree
more dtest
optimize ViewUpdateGenerator process to reduce payload. eg, if view row has 
live pk or other live column, "virtualCells' unselected payload" is not 
necessary.

> Obsolete MV entry may not be properly deleted
> -
>
> Key: CASSANDRA-11500
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11500
> Project: Cassandra
>  Issue Type: Bug
>  Components: Materialized Views
>Reporter: Sylvain Lebresne
>Assignee: ZhaoYang
>
> When a Materialized View uses a non-PK base table column in its PK, if an 
> update changes that column value, we add the new view entry and remove the 
> old one. When doing that removal, the current code uses the same timestamp 
> than for the liveness info of the new entry, which is the max timestamp for 
> any columns participating to the view PK. This is not correct for the 
> deletion as the old view entry could have other columns with higher timestamp 
> which won't be deleted as can easily shown by the failing of the following 
> test:
> {noformat}
> CREATE TABLE t (k int PRIMARY KEY, a int, b int);
> CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS 
> NOT NULL PRIMARY KEY (k, a);
> INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
> UPDATE t USING TIMESTAMP 4 SET b = 2 WHERE k = 1;
> UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1;
> SELECT * FROM mv WHERE k = 1; // This currently return 2 entries, the old 
> (invalid) and the new one
> {noformat}
> So the correct timestamp to use for the deletion is the biggest timestamp in 
> the old view entry (which we know since we read the pre-existing base row), 
> and that is what CASSANDRA-11475 does (the test above thus doesn't fail on 
> that branch).
> Unfortunately, even then we can still have problems if further updates 
> requires us to overide the old entry. Consider the following case:
> {noformat}
> CREATE TABLE t (k int PRIMARY KEY, a int, b int);
> CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS 
> NOT NULL PRIMARY KEY (k, a);
> INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
> UPDATE t USING TIMESTAMP 10 SET b = 2 WHERE k = 1;
> UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1; // This will delete the 
> entry for a=1 with timestamp 10
> UPDATE t USING TIMESTAMP 3 SET a = 1 WHERE k = 1; // This needs to re-insert 
> an entry for a=1 but shouldn't be deleted by the prior deletion
> UPDATE t USING TIMESTAMP 4 SET a = 2 WHERE k = 1; // ... and we can play this 
> game more than once
> UPDATE t USING TIMESTAMP 5 SET a = 1 WHERE k = 1;
> ...
> {noformat}
> In a way, this is saying that the "shadowable" deletion mechanism is not 
> general enough: we need to be able to re-insert an entry when a prior one had 
> been deleted before, but we can't rely on timestamps being strictly bigger on 
> the re-insert. In that sense, this can be though as a similar problem than 
> CASSANDRA-10965, though the solution there of a single flag is not enough 
> since we

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

2017-07-28 Thread ZhaoYang (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16101603#comment-16101603
 ] 

ZhaoYang edited comment on CASSANDRA-11500 at 7/28/17 10:28 AM:


WIP 
[branch|https://github.com/jasonstack/cassandra/commits/CASSANDRA-11500-cell]

Changed:
*  Extra "VirtualCell"(kind of special LivenessInfo for MV) in Row to maintain 
liveness of view row. It stores: 1. base column used in view PK and base column 
used in view filter conditions. if any of such column dead, entire view row 
dead, regardless LivenessInfo or DeletionTime status. 2. unselected base 
columns. if any of such column alive, view's pk should be alive if it's not 
deleted by DeletionTime or those columns in <1>. 
* blocked dropping filter base column
* fix issue of creating view with token()  filter
* remove Row.Deletion, it was used to wrap DeletionTime with "Shadowable" flag

Todo: 
optimize in-memory/storage representation for "virtual-cells" to re-use 
AbstractRow/BTree
more dtest
optimize ViewUpdateGenerator process to reduce payload. eg, if view row has 
live pk or other live column, "virtualCells' unselected payload" is not 
necessary.


was (Author: jasonstack):
WIP 
[branch|https://github.com/jasonstack/cassandra/commits/CASSANDRA-11500-cell]

Changed:
*  Extra "VirtualCell"(kind of special LivenessInfo for MV) in Row to maintain 
liveness of view row. It stores: 1. base column used in view PK and base column 
used in view filter conditions. if any of such column dead, entire view row 
dead, regardless LivenessInfo or DeletionTime status. 2. unselected base 
columns. if any of such column alive, view's pk should be alive if it's not 
deleted by DeletionTime or those columns in <1>. 
* blocked dropping filter base column
* fix issue of creating view with token()  filter

Todo: 
optimize in-memory/storage representation for "virtual-cells" to re-use 
AbstractRow/BTree
more dtest
optimize ViewUpdateGenerator process to reduce payload. eg, if view row has 
live pk or other live column, "virtualCells' unselected payload" is not 
necessary.

> Obsolete MV entry may not be properly deleted
> -
>
> Key: CASSANDRA-11500
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11500
> Project: Cassandra
>  Issue Type: Bug
>  Components: Materialized Views
>Reporter: Sylvain Lebresne
>Assignee: ZhaoYang
>
> When a Materialized View uses a non-PK base table column in its PK, if an 
> update changes that column value, we add the new view entry and remove the 
> old one. When doing that removal, the current code uses the same timestamp 
> than for the liveness info of the new entry, which is the max timestamp for 
> any columns participating to the view PK. This is not correct for the 
> deletion as the old view entry could have other columns with higher timestamp 
> which won't be deleted as can easily shown by the failing of the following 
> test:
> {noformat}
> CREATE TABLE t (k int PRIMARY KEY, a int, b int);
> CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS 
> NOT NULL PRIMARY KEY (k, a);
> INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
> UPDATE t USING TIMESTAMP 4 SET b = 2 WHERE k = 1;
> UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1;
> SELECT * FROM mv WHERE k = 1; // This currently return 2 entries, the old 
> (invalid) and the new one
> {noformat}
> So the correct timestamp to use for the deletion is the biggest timestamp in 
> the old view entry (which we know since we read the pre-existing base row), 
> and that is what CASSANDRA-11475 does (the test above thus doesn't fail on 
> that branch).
> Unfortunately, even then we can still have problems if further updates 
> requires us to overide the old entry. Consider the following case:
> {noformat}
> CREATE TABLE t (k int PRIMARY KEY, a int, b int);
> CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS 
> NOT NULL PRIMARY KEY (k, a);
> INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
> UPDATE t USING TIMESTAMP 10 SET b = 2 WHERE k = 1;
> UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1; // This will delete the 
> entry for a=1 with timestamp 10
> UPDATE t USING TIMESTAMP 3 SET a = 1 WHERE k = 1; // This needs to re-insert 
> an entry for a=1 but shouldn't be deleted by the prior deletion
> UPDATE t USING TIMESTAMP 4 SET a = 2 WHERE k = 1; // ... and we can play this 
> game more than once
> UPDATE t USING TIMESTAMP 5 SET a = 1 WHERE k = 1;
> ...
> {noformat}
> In a way, this is saying that the "shadowable" deletion mechanism is not 
> general enough: we need to be able to re-insert an entry when a prior one had 
> been deleted before, but we can't rely on timestamps being strictly bigger on 
> the re-insert. In that sense, this can be though as a

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

2017-07-28 Thread ZhaoYang (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16101603#comment-16101603
 ] 

ZhaoYang edited comment on CASSANDRA-11500 at 7/28/17 10:27 AM:


WIP 
[branch|https://github.com/jasonstack/cassandra/commits/CASSANDRA-11500-cell]

Changed:
*  Extra "VirtualCell"(kind of special LivenessInfo for MV) in Row to maintain 
liveness of view row. It stores: 1. base column used in view PK and base column 
used in view filter conditions. if any of such column dead, entire view row 
dead, regardless LivenessInfo or DeletionTime status. 2. unselected base 
columns. if any of such column alive, view's pk should be alive if it's not 
deleted by DeletionTime or those columns in <1>. 
* blocked dropping filter base column
* fix issue of creating view with token()  filter

Todo: 
optimize in-memory/storage representation for "virtual-cells" to re-use 
AbstractRow/BTree
more dtest
optimize ViewUpdateGenerator process to reduce payload. eg, if view row has 
live pk or other live column, "virtualCells' unselected payload" is not 
necessary.


was (Author: jasonstack):
WIP 
[branch|https://github.com/jasonstack/cassandra/commits/CASSANDRA-11500-cell]

Changed:
Extra "VirtualCell"(kind of special LivenessInfo for MV) in Row. It stores: 1. 
base column used in view PK and base column used in view filter conditions. if 
any of such column dead, entire view row dead, regardless LivenessInfo or 
DeletionTime status. 2. unselected base columns. if any of such column alive, 
view's pk should be alive if it's not deleted by DeletionTime or those columns 
in <1>. 

Todo: 
optimize in-memory/storage representation for "virtual-cells" to re-use 
AbstractRow/BTree
more dtest
optimize ViewUpdateGenerator process to reduce payload. eg, if view row has 
live pk or other live column, "virtualCells' unselected payload" is not 
necessary.
support collection type in unselected column (for now, it won't generate 
"virtual-cells" for non-frozen collection type)

> Obsolete MV entry may not be properly deleted
> -
>
> Key: CASSANDRA-11500
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11500
> Project: Cassandra
>  Issue Type: Bug
>  Components: Materialized Views
>Reporter: Sylvain Lebresne
>Assignee: ZhaoYang
>
> When a Materialized View uses a non-PK base table column in its PK, if an 
> update changes that column value, we add the new view entry and remove the 
> old one. When doing that removal, the current code uses the same timestamp 
> than for the liveness info of the new entry, which is the max timestamp for 
> any columns participating to the view PK. This is not correct for the 
> deletion as the old view entry could have other columns with higher timestamp 
> which won't be deleted as can easily shown by the failing of the following 
> test:
> {noformat}
> CREATE TABLE t (k int PRIMARY KEY, a int, b int);
> CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS 
> NOT NULL PRIMARY KEY (k, a);
> INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
> UPDATE t USING TIMESTAMP 4 SET b = 2 WHERE k = 1;
> UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1;
> SELECT * FROM mv WHERE k = 1; // This currently return 2 entries, the old 
> (invalid) and the new one
> {noformat}
> So the correct timestamp to use for the deletion is the biggest timestamp in 
> the old view entry (which we know since we read the pre-existing base row), 
> and that is what CASSANDRA-11475 does (the test above thus doesn't fail on 
> that branch).
> Unfortunately, even then we can still have problems if further updates 
> requires us to overide the old entry. Consider the following case:
> {noformat}
> CREATE TABLE t (k int PRIMARY KEY, a int, b int);
> CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS 
> NOT NULL PRIMARY KEY (k, a);
> INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
> UPDATE t USING TIMESTAMP 10 SET b = 2 WHERE k = 1;
> UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1; // This will delete the 
> entry for a=1 with timestamp 10
> UPDATE t USING TIMESTAMP 3 SET a = 1 WHERE k = 1; // This needs to re-insert 
> an entry for a=1 but shouldn't be deleted by the prior deletion
> UPDATE t USING TIMESTAMP 4 SET a = 2 WHERE k = 1; // ... and we can play this 
> game more than once
> UPDATE t USING TIMESTAMP 5 SET a = 1 WHERE k = 1;
> ...
> {noformat}
> In a way, this is saying that the "shadowable" deletion mechanism is not 
> general enough: we need to be able to re-insert an entry when a prior one had 
> been deleted before, but we can't rely on timestamps being strictly bigger on 
> the re-insert. In that sense, this can be though as a similar problem than 
> CASSANDRA-10965, though the solution there of a single flag is

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

2017-07-26 Thread ZhaoYang (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082241#comment-16082241
]

ZhaoYang edited comment on CASSANDRA-11500 at 7/26/17 3:46 PM:
---

h3. Relation: base -> view

First of all, I think all of us should agree on what cases view row should
exists.

IMO, there are two main cases:

1. base pk and view pk are the same (order doesn't matter) and view has no
filter conditions or only conditions on base pk.
(filter condition mean: {{c = 1}} in view's where clause. filter condition is
not a concern here, since no previous view data to be cleared.)

view row exists if any of following is true:
* a. base row pk has live livenessInfo(timestamp) and base row pk satifies
view's filter conditions if any.
* b. or one of base row columns selected in view has live timestamp (via
update) and base row pk satifies view's filter conditions if any. this is
handled by existing mechanism of liveness and tombstone since all info are
included in view row
* c. or one of base row columns not selected in view has live timestamp (via
update) and base row pk satifies view's filter conditions if any. Those
unselected columns' timestamp/ttl/cell-deletion info are not currently stored
on view row.

2. base column used in view pk or view has filter conditions on base non-key
column which can also lead to entire view row being wiped.

view row exists if any of following is true:
* a. base row pk has live livenessInfo(timestamp) && base column used in view
pk is not null but no timestamp && conditions are satisfied. ( pk having live
livenesInfo means it is not deleted by tombstone)
* b. or base row column in view pk has timestamp (via update) && conditions are
satisfied. eg. if base column used in view pk is TTLed, entire view row should
be wiped.

Next thing is to model "view's tombstone and livenessInfo" to maintain view
data based on above cases.

h3. Previous known issues:
(I might miss some issues, feel free to ping me..)

ttl
* view row is not wiped when TTLed on base column used in view pk or TTLed on
base non-key column with filter condition
* cells with same timestamp, merging ttls are not deterministic.

partial update on base columns not selected in view
* it results in no view data. because of current update semantics, no view
updates are generated
* corresponding view row' liveness is not depending on liveness of base columns

filter conditions or base column used in view pk causes
* view row is shadowed after a few modification on base column used in view pk
if the base non-key column has TS greater than base pk's ts and view key
column's ts. (as mentioned by sylvain: we need to be able to re-insert an entry
when a prior one had been deleted need to be careful to hanlde timestamp tie)

tombstone merging is not commutative
* in current code, shadowable tombstone doesn't co-exist with regular tombstone

sstabledump not supporting current shadowable tombstone

h3. Model

I can think of two ways to ship all required base column info to view:
* make base columns that are not selected in view as "virtual cell" and
store their imestamp/ttl to view without their actual values. so we can reuse
current ts/tb/ttl mechanism with additional validation logic to check if a view
row is alive.
* or storing those info on view's livenessInfo/deletion with addition merge
logic to make sure view liveness/deletion are ordered properly even in the case
of timestamp tie. It's like a replacement stragy which uses inserted view row
to replace old view row. (In regular table, reconciliation is at cell level.
need to research more about the concurrent view update cases, fow now, it looks
fine). It also implies that every modification on base, view row will get
replaced entirely..

I will go ahead with -second way since there is an existing shadowable
tombstone mechanism.- VirtualCells to avoid changing low level timestamp
comparison..

{code}
ColumnInfo: // generated from base column as it is.
0. timestamp
1. ttl
2. localDeletionTime: could be used to represent tombstone or TTLed
depends on if there is ttl

supersedes(): if timestamps are different, greater timestamp
supersedes; if timestamps are same, greater localDeletionTime supersedes.

Row: // VirtualCells(keyOrConditions and unselected) are always merged with
another row's during row merging process

// base column that are used in view pk or has filter condition on
non-pk column.
// if any column is not live, entire view row is wiped.
// if a column in base is filtered and not selected, it's stored here.
// during base modification, if a view row is removed due to
base-column-in-view-pk or filter-contiions, then no deletion is issue,
// the virtual cell tombstone is added to Row's keyOrConditions.
2.

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

2017-07-26 Thread ZhaoYang (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16101603#comment-16101603
 ] 

ZhaoYang edited comment on CASSANDRA-11500 at 7/26/17 3:34 PM:
---

WIP 
[branch|https://github.com/jasonstack/cassandra/commits/CASSANDRA-11500-cell]

Changed:
Extra "VirtualCell"(kind of special LivenessInfo for MV) in Row. It stores: 1. 
base column used in view PK and base column used in view filter conditions. if 
any of such column dead, entire view row dead, regardless LivenessInfo or 
DeletionTime status. 2. unselected base columns. if any of such column alive, 
view's pk should be alive if it's not deleted by DeletionTime or those columns 
in <1>. 

Todo: 
optimize in-memory/storage representation for "virtual-cells" to re-use 
AbstractRow/BTree
more dtest
optimize ViewUpdateGenerator process to reduce payload. eg, if view row has 
live pk or other live column, "virtualCells' unselected payload" is not 
necessary.
support collection type in unselected column (for now, it won't generate 
"virtual-cells" for non-frozen collection type)


was (Author: jasonstack):
WIP 
[branch|https://github.com/jasonstack/cassandra/commits/CASSANDRA-11500-cell]

Changed:
Extra "VirtualCell"(kind of special LivenessInfo for MV) in Row. It stores: 1. 
base column used in view PK and base column used in view filter conditions. if 
any of such column dead, entire view row dead, regardless LivenessInfo or 
DeletionTime status. 2. unselected base columns. if any of such column alive, 
view's pk should be alive if it's not deleted by DeletionTime or those columns 
in <1>. 

Todo: 
optimize in-memory/storage representation for "virtual-cells" to re-use 
AbstractRow/BTree
discard dead view row shadowed by virtualCells in compaction after gc_grace
more dtest
support collection type in unselected column (for now, it won't generate 
"virtual-cells" for non-frozen collection type)

> Obsolete MV entry may not be properly deleted
> -
>
> Key: CASSANDRA-11500
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11500
> Project: Cassandra
>  Issue Type: Bug
>  Components: Materialized Views
>Reporter: Sylvain Lebresne
>Assignee: ZhaoYang
>
> When a Materialized View uses a non-PK base table column in its PK, if an 
> update changes that column value, we add the new view entry and remove the 
> old one. When doing that removal, the current code uses the same timestamp 
> than for the liveness info of the new entry, which is the max timestamp for 
> any columns participating to the view PK. This is not correct for the 
> deletion as the old view entry could have other columns with higher timestamp 
> which won't be deleted as can easily shown by the failing of the following 
> test:
> {noformat}
> CREATE TABLE t (k int PRIMARY KEY, a int, b int);
> CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS 
> NOT NULL PRIMARY KEY (k, a);
> INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
> UPDATE t USING TIMESTAMP 4 SET b = 2 WHERE k = 1;
> UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1;
> SELECT * FROM mv WHERE k = 1; // This currently return 2 entries, the old 
> (invalid) and the new one
> {noformat}
> So the correct timestamp to use for the deletion is the biggest timestamp in 
> the old view entry (which we know since we read the pre-existing base row), 
> and that is what CASSANDRA-11475 does (the test above thus doesn't fail on 
> that branch).
> Unfortunately, even then we can still have problems if further updates 
> requires us to overide the old entry. Consider the following case:
> {noformat}
> CREATE TABLE t (k int PRIMARY KEY, a int, b int);
> CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS 
> NOT NULL PRIMARY KEY (k, a);
> INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
> UPDATE t USING TIMESTAMP 10 SET b = 2 WHERE k = 1;
> UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1; // This will delete the 
> entry for a=1 with timestamp 10
> UPDATE t USING TIMESTAMP 3 SET a = 1 WHERE k = 1; // This needs to re-insert 
> an entry for a=1 but shouldn't be deleted by the prior deletion
> UPDATE t USING TIMESTAMP 4 SET a = 2 WHERE k = 1; // ... and we can play this 
> game more than once
> UPDATE t USING TIMESTAMP 5 SET a = 1 WHERE k = 1;
> ...
> {noformat}
> In a way, this is saying that the "shadowable" deletion mechanism is not 
> general enough: we need to be able to re-insert an entry when a prior one had 
> been deleted before, but we can't rely on timestamps being strictly bigger on 
> the re-insert. In that sense, this can be though as a similar problem than 
> CASSANDRA-10965, though the solution there of a single flag is not enough 
> since we can have to replace more than once.
> I think the proper solution

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

2017-07-25 Thread ZhaoYang (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082241#comment-16082241
]

ZhaoYang edited comment on CASSANDRA-11500 at 7/25/17 9:25 AM:
---

h3. Relation: base -> view

First of all, I think all of us should agree on what cases view row should
exists.

IMO, there are two main cases:

2. base column used in view pk or view has filter conditions on base non-key
column which can also lead to entire view row being wiped.

Next thing is to model "view's tombstone and livenessInfo" to maintain view
data based on above cases.

h3. Previous known issues:
(I might miss some issues, feel free to ping me..)

ttl
* view row is not wiped when TTLed on base column used in view pk or TTLed on
base non-key column with filter condition
* cells with same timestamp, merging ttls are not deterministic.

tombstone merging is not commutative
* in current code, shadowable tombstone doesn't co-exist with regular tombstone

sstabledump not supporting current shadowable tombstone

h3. Model

I will go ahead with ~second way since there is an existing shadowable
tombstone mechanism.~ VirtualCells..

{code}
ColumnInfo: // generated from base column as it is.
0. timestamp
1. ttl
2. localDeletionTime: could be used to represent tombstone or TTLed
depends on if there is ttl

supersedes(): if timestamps are different, greater timestamp
supersedes; if timestamps are same, greater localDeletionTime supersedes.

Row: // VirtualCells(keyOrConditions and unselected) are always merged with
another row's during row merging process

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

2017-07-25 Thread ZhaoYang (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082241#comment-16082241
]

ZhaoYang edited comment on CASSANDRA-11500 at 7/25/17 9:24 AM:
---

h3. Relation: base -> view

First of all, I think all of us should agree on what cases view row should
exists.

IMO, there are two main cases:

2. base column used in view pk or view has filter conditions on base non-key
column which can also lead to entire view row being wiped.

Next thing is to model "view's tombstone and livenessInfo" to maintain view
data based on above cases.

h3. Previous known issues:
(I might miss some issues, feel free to ping me..)

ttl
* view row is not wiped when TTLed on base column used in view pk or TTLed on
base non-key column with filter condition
* cells with same timestamp, merging ttls are not deterministic.

tombstone merging is not commutative
* in current code, shadowable tombstone doesn't co-exist with regular tombstone

sstabledump not supporting current shadowable tombstone

h3. Model

FIXME in one node, if multiple ViewLivenessInfo having same timestamp, we
do replacement.. how about cross node merging? how about partial update on
normal columns that are part of view normal column? always send entire row to
do replacement

I will go ahead with ~second way since there is an existing shadowable
tombstone mechanism.~

View PrimaryKey LivenessInfo, its timestamp, payloads, merging

{code}
ColumnInfo: // generated from base column as it is.
0. timestamp
1. ttl
2. localDeletionTime: could be used to represent tombstone or TTLed
depends on if there is ttl

supersedes(): if timestamps are different, greater timestamp
supersedes; if timestamps are same, greater localDeletionTime supersedes.

Row: // keyOrConditions and unselected are always merged with another row's
during row merging process

// base column that are used in view pk or has filter condition on
non-pk column.
// if any column is not live, entire view row is wiped.
// if a column in base is filtered and not selected, it's

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

2017-07-20 Thread ZhaoYang (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082241#comment-16082241
]

ZhaoYang edited comment on CASSANDRA-11500 at 7/20/17 2:43 PM:
---

h3. Relation: base -> view

First of all, I think all of us should agree on what cases view row should
exists.

IMO, there are two main cases:

2. base column used in view pk or view has filter conditions on base non-key
column which can also lead to entire view row being wiped.

Next thing is to model "view's tombstone and livenessInfo" to maintain view
data based on above cases.

h3. Previous known issues:
(I might miss some issues, feel free to ping me..)

ttl
* view row is not wiped when TTLed on base column used in view pk or TTLed on
base non-key column with filter condition
* cells with same timestamp, merging ttls are not deterministic.

tombstone merging is not commutative
* in current code, shadowable tombstone doesn't co-exist with regular tombstone

sstabledump not supporting current shadowable tombstone

h3. Model

I will go ahead with second way since there is an existing shadowable tombstone
mechanism.

View PrimaryKey LivenessInfo, its timestamp, payloads, merging

{code}
ColumnInfo: // generated from base column as it is.
0. timestamp
1. ttl
2. localDeletionTime: could be used to represent tombstone or TTLed
depends on if there is ttl

supersedes(): if timestamps are different, greater timestamp
supersedes; if timestamps are same, greater localDeletionTime supersedes.

// if a normal column in base row has no timestamp (aka. generated by
Insert statement), when it is sent to view, it remains no timestamp.
// it will implicitly inherit ViewLivenessInfo just like how it works with
standard LivenessInfo in regular table,
// unlike shadowable mechanism which will explicitly put base pk's
timestamp into not-updated base columns into view data to keep "select
writetime" correct in view.
// (because in shaowable mechanism, view's pk timestamp is promoted to a
bigger value which cannot be used for writetime in view)
ViewLivenessInfo
// corresponding to base pk livenessInfo
0. timestamp
1. ttl / localDeletionTime

// base column that are used in view pk or has filter condition.
// if any column is not live or doesn't exist, entire view row is wiped.
// if a column in base is filtered and not selected, it's stored here.
2. Map keyOrConditions;

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

2017-07-20 Thread ZhaoYang (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082241#comment-16082241
]

ZhaoYang edited comment on CASSANDRA-11500 at 7/20/17 2:07 PM:
---

h3. Relation: base -> view

First of all, I think all of us should agree on what cases view row should
exists.

IMO, there are two main cases:

1. base pk and view pk are the same (order doesn't matter) and view has no
filter conditions or only conditions on base pk.
(filter condition is not a concern here, since no previous view data to be
cleared)

2. base column used in view pk or view has filter conditions on base non-key
column which can also lead to entire view row being wiped.

Next thing is to model "shadowable tombstone or shadowable liveness" to
maintain view data based on above cases.

h3. Previous known issues:
(I might miss some issues, feel free to ping me..)

ttl
* view row is not wiped when TTLed on base column used in view pk or TTLed on
base non-key column with filter condition
* cells with same timestamp, merging ttls are not deterministic.

partial update on base columns not selected in view
* it results in no view data. because of current update semantics, no view
updates are generated
* corresponding view row liveness is not depending on liveness of base columns

tombstone merging is not commutative
* in current code, shadowable tombstone doesn't co-exist with regular tombstone

sstabledump doesn't not support current shadowable tombstone

h3. Model

I will go ahead with second way since there is an existing shadowable tombstone
mechanism.

View PrimaryKey LivenessInfo, its timestamp, payloads, merging

{code}
ColumnInfo: // generated from base column
0. timestamp
1. ttl
2. localDeletionTime: could be used to represent tombstone or TTLed
depends on if there is ttl

supersedes(): if timestamps are different, greater timestamp
supersedes; if timestamps are same, greater localDeletionTime supersedes.

ViewLivenessInfo
// corresponding to base pk livenessInfo
0. timestamp
1. ttl / localDeletionTime

// if any column is live
3. Map unselected;

// to determina if a row is live
isRowAlive(Deletion delete):
get timestamp or columnInfo that is greater than those in Deletion

if any colummn in {{keyOrConditions}} is TTLed or tombstone(dead)
or not existed, false
if {{timestamp or ttl}} are alive, true
if any column in {{unselected}} is alive, true
otherwise check any columns in view row are alive

// cannot use supersedes, because timestamp can tie, we cannot compare
keyOrConditions.

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

2017-07-20 Thread ZhaoYang (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082241#comment-16082241
]

ZhaoYang edited comment on CASSANDRA-11500 at 7/20/17 1:28 PM:
---

h3. Relation: base -> view

First of all, I think all of us should agree on what cases view row should
exists.

IMO, there are two main cases:

2. base column used in view pk or view has filter conditions on base non-key
column which can also lead to entire view row being wiped.

Next thing is to model "shadowable tombstone or shadowable liveness" to
maintain view data based on above cases.

h3. Previous known issues:
(I might miss some issues, feel free to ping me..)

ttl
* view row is not wiped when TTLed on base column used in view pk or TTLed on
base non-key column with filter condition
* cells with same timestamp, merging ttls are not deterministic.

tombstone merging is not commutative
* in current code, shadowable tombstone doesn't co-exist with regular tombstone

sstabledump doesn't not support current shadowable tombstone

h3. Model

I will go ahead with second way since there is an existing shadowable tombstone
mechanism.

View PrimaryKey LivenessInfo, its timestamp, payloads, merging

{code}
ColumnInfo: // generated from base column
0. timestamp
1. ttl
2. localDeletionTime: could be used to represent tombstone or TTLed
depends on if there is ttl

supersedes(): if timestamps are different, greater timestamp
supersedes; if timestamps are same, greater localDeletionTime supersedes.

ViewLivenessInfo
// corresponding to base pk livenessInfo
0. timestamp
1. ttl / localDeletionTime

// if any column is live
3. Map unselected;

// to determina if a row is live
isRowAlive(Deletion delete):
get timestamp or columnInfo that is greater than those in Deletion

// cannot use supersedes, because timestamp can tie, we cannot compare
keyOrConditions.

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

2017-07-20 Thread ZhaoYang (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082241#comment-16082241
]

ZhaoYang edited comment on CASSANDRA-11500 at 7/20/17 12:53 PM:

h3. Relation: base -> view

First of all, I think all of us should agree on what cases view row should
exists.

IMO, there are two main cases:

2. base column used in view pk or view has filter conditions on base non-key
column which can also lead to entire view row being wiped.

Next thing is to model "shadowable tombstone or shadowable liveness" to
maintain view data based on above cases.

h3. Previous known issues:
(I might miss some issues, feel free to ping me..)

ttl
* view row is not wiped when TTLed on base column used in view pk or TTLed on
base non-key column with filter condition
* cells with same timestamp, merging ttls are not deterministic.

tombstone merging is not commutative
* in current code, shadowable tombstone doesn't co-exist with regular tombstone

sstabledump doesn't not support current shadowable tombstone

h3. Model

I will go ahead with second way since there is an existing shadowable tombstone
mechanism.

View PrimaryKey LivenessInfo, its timestamp, payloads, merging

{code}
ColumnInfo: // generated from base column
0. timestamp
1. ttl
2. localDeletionTime: could be used to represent tombstone or TTLed
depends on if there is ttl

supersedes(): if timestamps are different, greater timestamp
supersedes; if timestamps are same, greater localDeletionTime supersedes.

ViewLivenessInfo
// corresponding to base pk livenessInfo
0. timestamp
1. ttl / localDeletionTime

// if any column is live
3. Map unselected;

// to determina if a row is live
isRowAlive(Deletion delete):
get timestamp or columnInfo that is greater than those in Deletion

// cannot use supersedes, because timestamp can tie, we cannot compare
keyOrConditions.

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

2017-07-19 Thread ZhaoYang (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089856#comment-16089856
 ] 

ZhaoYang edited comment on CASSANDRA-11500 at 7/19/17 2:26 PM:
---

I plan to solve: {{partial update}},{{ttl}}, {{co-existed shadowable 
tombstone}}, {{view timestamp tie}} all inside this ticket using extended 
shadowable approach(mentioned 
[here|https://issues.apache.org/jira/browse/CASSANDRA-11500?focusedCommentId=16082241=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16082241]).
 Because all these issues require some storage format changes(extendedFlag), 
it's better to fix them and refactor in one commit.

I will drafrt a patch using {{ViewTombstone}} and {{ViewLiveness}}.

Any suggestions would be appreciated.




was (Author: jasonstack):
I plan to solve: {{partial update}},{{ttl}}, {{co-existed shadowable 
tombstone}}, {{view timestamp tie}} all inside this ticket using extended 
shadowable approach(mentioned 
[here|https://issues.apache.org/jira/browse/CASSANDRA-11500?focusedCommentId=16082241=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16082241]).
 Because all these issues require some storage format changes(extendedFlag), 
it's better to fix them and refactor in one commit.

Drafted a 
[patch|https://github.com/jasonstack/cassandra/commits/CASSANDRA-11500-update-time]..(refactoring
 and adding more dtest)  

Any suggestions would be appreciated.



> Obsolete MV entry may not be properly deleted
> -
>
> Key: CASSANDRA-11500
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11500
> Project: Cassandra
>  Issue Type: Bug
>  Components: Materialized Views
>Reporter: Sylvain Lebresne
>Assignee: ZhaoYang
>
> When a Materialized View uses a non-PK base table column in its PK, if an 
> update changes that column value, we add the new view entry and remove the 
> old one. When doing that removal, the current code uses the same timestamp 
> than for the liveness info of the new entry, which is the max timestamp for 
> any columns participating to the view PK. This is not correct for the 
> deletion as the old view entry could have other columns with higher timestamp 
> which won't be deleted as can easily shown by the failing of the following 
> test:
> {noformat}
> CREATE TABLE t (k int PRIMARY KEY, a int, b int);
> CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS 
> NOT NULL PRIMARY KEY (k, a);
> INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
> UPDATE t USING TIMESTAMP 4 SET b = 2 WHERE k = 1;
> UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1;
> SELECT * FROM mv WHERE k = 1; // This currently return 2 entries, the old 
> (invalid) and the new one
> {noformat}
> So the correct timestamp to use for the deletion is the biggest timestamp in 
> the old view entry (which we know since we read the pre-existing base row), 
> and that is what CASSANDRA-11475 does (the test above thus doesn't fail on 
> that branch).
> Unfortunately, even then we can still have problems if further updates 
> requires us to overide the old entry. Consider the following case:
> {noformat}
> CREATE TABLE t (k int PRIMARY KEY, a int, b int);
> CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS 
> NOT NULL PRIMARY KEY (k, a);
> INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
> UPDATE t USING TIMESTAMP 10 SET b = 2 WHERE k = 1;
> UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1; // This will delete the 
> entry for a=1 with timestamp 10
> UPDATE t USING TIMESTAMP 3 SET a = 1 WHERE k = 1; // This needs to re-insert 
> an entry for a=1 but shouldn't be deleted by the prior deletion
> UPDATE t USING TIMESTAMP 4 SET a = 2 WHERE k = 1; // ... and we can play this 
> game more than once
> UPDATE t USING TIMESTAMP 5 SET a = 1 WHERE k = 1;
> ...
> {noformat}
> In a way, this is saying that the "shadowable" deletion mechanism is not 
> general enough: we need to be able to re-insert an entry when a prior one had 
> been deleted before, but we can't rely on timestamps being strictly bigger on 
> the re-insert. In that sense, this can be though as a similar problem than 
> CASSANDRA-10965, though the solution there of a single flag is not enough 
> since we can have to replace more than once.
> I think the proper solution would be to ship enough information to always be 
> able to decide when a view deletion is shadowed. Which means that both 
> liveness info (for updates) and shadowable deletion would need to ship the 
> timestamp of any base table column that is part the view PK (so {{a}} in the 
> example below).  It's doable (and not that hard really), but it does require 
> a change to the sstable and intra-node protocol, which makes this a bit 
> painful

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

2017-07-19 Thread ZhaoYang (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082241#comment-16082241
]

ZhaoYang edited comment on CASSANDRA-11500 at 7/19/17 2:23 PM:
---

h3. Relation: base -> view

First of all, I think all of us should agree on what cases view row should
exists.

IMO, there are two main cases:

2. base column used in view pk or view has filter conditions on base non-key
column which can also lead to entire view row being wiped.

Next thing is to model "shadowable tombstone or shadowable liveness" to
maintain view data based on above cases.

h3. Previous known issues:
(I might miss some issues, feel free to ping me..)

ttl
* view row is not wiped when TTLed on base column used in view pk or TTLed on
base non-key column with filter condition
* cells with same timestamp, merging ttls are not deterministic.

tombstone merging is not commutative
* in current code, shadowable tombstone doesn't co-exist with regular tombstone

sstabledump doesn't not support current shadowable tombstone

h3. Model

I will go ahead with second way since there is an existing shadowable tombstone
mechanism.

View PrimaryKey LivenessInfo, its timestamp, payloads, merging

{code}
ColumnInfo: // generated from base column
0. timestamp
1. ttl
2. localDeletionTime: could be used to represent tombstone or TTLed
depends on if there is ttl

supersedes(): if timestamps are different, greater timestamp
supersedes; if timestamps are same, greater localDeletionTime supersedes.

ViewLivenessInfo
// corresponding to base pk livenessInfo
0. timestamp
1. ttl / localDeletionTime

// if any column is live
3. Map unselected;

// to determina if a row is live
isRowAlive(Deletion delete):
get timestamp or columnInfo that is greater than those in Deletion

// cannot use supersedes, because timestamp can tie, we cannot compare
keyOrConditions.

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

2017-07-18 Thread ZhaoYang (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082241#comment-16082241
 ] 

ZhaoYang edited comment on CASSANDRA-11500 at 7/18/17 3:10 PM:
---

h3. Relation: base -> view

First of all, I think all of us should agree on what cases view row should 
exists.

IMO, there are two main cases:

1. base pk and view pk are the same (order doesn't matter) and view has no 
filter conditions or only conditions on base pk.
(filter condition is not a concern here, since no previous view data to be 
cleared)

view row exists if any of following is true:
* a. base row pk has live livenessInfo(timestamp) and base row pk satifies 
view's filter conditions if any.
* b. or one of base row columns selected in view has live timestamp (via 
update) and base row pk satifies view's filter conditions if any. this is 
handled by existing mechanism of liveness and tombstone since all info are 
included in view row
* c. or one of base row columns not selected in view has live timestamp (via 
update) and base row pk satifies view's filter conditions if any. Those 
unselected columns' timestamp/ttl/cell-deletion info currently are not stored 
on view row. 

2. base column used in view pk or view has filter conditions on base non-key 
column which can also lead to entire view row being wiped.

view row exists if any of following is true:
* a. base row pk has live livenessInfo(timestamp) && base column used in view 
pk is not null but no timestamp && conditions are satisfied. ( pk having live 
livenesInfo means it is not deleted by tombstone)
* b. or base row column in view pk has timestamp (via update) && conditions are 
satisfied. eg. if base column used in view pk is TTLed, entire view row should 
be wiped.

Next thing is to model "shadowable tombstone or shadowable liveness" to 
maintain view data based on above cases.
 
h3. Previous known issues: 
(I might miss some issues, feel free to ping me..)

ttl
* view row is not wiped when TTLed on base column used in view pk or TTLed on 
base non-key column with filter condition
* cells with same timestamp, merging ttls are not deterministic.

partial update on base columns not selected in view
* it results in no view data. because of current update semantics, no view 
updates are generated
* corresponding view row liveness is not depending on liveness of base columns

filter conditions or base column used in view pk causes
* view row is shadowed after a few modification on base column used in view pk 
if the base non-key column has TS greater than base pk's ts and view key 
column's ts. (as mentioned by sylvain: we need to be able to re-insert an entry 
when a prior one had been deleted need to be careful to hanlde timestamp tie)

tombstone merging is not commutative
* in current code, shadowable tombstone doesn't co-exist with regular tombstone

sstabledump doesn't not support current shadowable tombstone

h3. Model (TO BE UPDATED)

{{ShadowableTombstone}} : 
* deletion-time, isShadowable, and "viewKeyTs" aka. base column's ts which is 
part of view pk(used to reconcile when timestamp tie), if there is no timestamp 
associated with that column, use base pk timestamp instead.
* it's only generated when one base column is a pk in view and this base column 
value is changed in base row, to mark previous view row as deleted. (original 
definition of {{shadowable}} in CASSANDRA-10261).  in other cases, {{standard 
tombstone}} is generated for view rows.
* if {{ShadowableTombstone}} is superseded by {{LivenessInfo}}, columns 
shadowed by {{ShadowableTombstone}} will come back alive. (original definition 
of {{shadowable}} in CASSANDRA-10261)
* {{ShadowableTombstone}}  should co-exist with {{Standard Tombstone}} if 
{{shadowable}}'s deletion time supersedes {{standard tombstone}} to avoid 
bringing columns older than {{standard tombstone}} coming back alive( as in 
CASSANDRA-13409)

{{ShadowableLivenessInfo}}:  
* timestamp, and "viewKeyTs"
* if shadowable and not live, all columns in this row are considered deleted as 
in CASSANDRA-13657 and CASSANDRA-13127 to solve partial update issues

When reconcile {{ShadowableTombstone}} and {{ShadowableLivenessInfo}}: 
{quote}
if deletion-time greater than timestamp, tombstone wins
if deletion-time smaller than timestamp, livenessInfo wins
when deletion-time ties with timestamp, 
 - if {{ShadowableTombstone}}'s {{viewKeyTs}} >= {{ShadowableLivenessInfo}}'s, 
then tombstone wins
 - else livesnessInfo wins.
{quote}

When inserting to view, always use the greatest timestamp of all base columns 
in view similar to how view deletion timestamp is computed.

h3. *Example*

{quote}
CREATE TABLE t (k int PRIMARY KEY, a int, b int);
CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS NOT 
NULL PRIMARY KEY (k, a);

{{q1}} INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
{{q2}} UPDATE t

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

2017-07-17 Thread ZhaoYang (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082241#comment-16082241
 ] 

ZhaoYang edited comment on CASSANDRA-11500 at 7/17/17 2:13 PM:
---

h3. *Idea*

{{ShadowableTombstone}} : 
* deletion-time, isShadowable, and "viewKeyTs" aka. base column's ts which is 
part of view pk(used to reconcile when timestamp tie), if there is no timestamp 
associated with that column, use base pk timestamp instead.
* it's only generated when one base column is a pk in view and this base column 
value is changed in base row, to mark previous view row as deleted. (original 
definition of {{shadowable}} in CASSANDRA-10261).  in other cases, {{standard 
tombstone}} is generated for view rows.
* if {{ShadowableTombstone}} is superseded by {{LivenessInfo}}, columns 
shadowed by {{ShadowableTombstone}} will come back alive. (original definition 
of {{shadowable}} in CASSANDRA-10261)
* {{ShadowableTombstone}}  should co-exist with {{Standard Tombstone}} if 
{{shadowable}}'s deletion time supersedes {{standard tombstone}} to avoid 
bringing columns older than {{standard tombstone}} coming back alive( as in 
CASSANDRA-13409)

{{ShadowableLivenessInfo}}:  
* timestamp, and "viewKeyTs"
* if shadowable and not live, all columns in this row are considered deleted as 
in CASSANDRA-13657 and CASSANDRA-13127 to solve partial update issues

When reconcile {{ShadowableTombstone}} and {{ShadowableLivenessInfo}}: 
{quote}
if deletion-time greater than timestamp, tombstone wins
if deletion-time smaller than timestamp, livenessInfo wins
when deletion-time ties with timestamp, 
 - if {{ShadowableTombstone}}'s {{viewKeyTs}} >= {{ShadowableLivenessInfo}}'s, 
then tombstone wins
 - else livesnessInfo wins.
{quote}

When inserting to view, always use the greatest timestamp of all base columns 
in view similar to how view deletion timestamp is computed.

h3. *Example*

{quote}
CREATE TABLE t (k int PRIMARY KEY, a int, b int);
CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS NOT 
NULL PRIMARY KEY (k, a);

{{q1}} INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
{{q2}} UPDATE t USING TIMESTAMP 10 SET b = 2 WHERE k = 1;
{{q3}} UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1; 
{{q3}} UPDATE t USING TIMESTAMP 3 SET a = 1 WHERE k = 1; 
{quote}


* After {{q1}}:
** in base: {{k=1@0, a=1, b=1}}// 'k' is having value '1' with timestamp '0'
** in view: 
***  sstable1: {{(k=1&=1)@TS(0,0), b=1}}  // 'k:a' is having value '1:1' with 
timestamp '0' and viewKeyTs '0' from base's pk because column 'a' has no TS
* After {{q2}}
** in base(merged): {{k=1@0, a=1, b=2@10}} 
** in view:  
***  sstable1: {{(k=1&=1)@TS(0,0), b=1}}
***  sstable2: {{(k=1&=1)@TS(10,0), b=2@10}}
***  or merged: {{(k=1&=1)@TS(10,0), b=2@10}}
* After {{q3}}
** in base(merged): {{k=1@0, a=2@2, b=2@10}}  
** in view:  
***  sstable1: {{(k=1&=1)@TS(0,0), b=1}}
***  sstable2: {{(k=1&=1)@TS(10,0), b=2@10}}
***  sstable3: {{(k=1&=1)@Shadowable(10,0)}} & {{(k=1&=2)@TS(10,2), 
b=2@10}}  // '(k=1&=2)' is having biggest timestamp '10' and viewKeyTs '2' 
from column 'a'
***  or merged: {{(k=1&=2)@TS(10,2), b=2@10}}
* After {{q4}}
** in base(merged): {{k=1@0, a=1@3, b=2@10}}  
** in view:  
***  sstable1: {{(k=1&=1)@TS(0,0), b=1}}
***  sstable2: {{(k=1&=1)@TS(10,0), b=2@10}}
***  sstable3: {{(k=1&=1)@Shadowable(10,0)}} & {{(k=1&=2)@TS(10,2), 
b=2@10}} 
***  sstable4: {{(k=1&=2)@Shadowable(10,2)}} & {{(k=1&=1)@TS(10,3), 
b=2@10}}  // '(k=1&=1)' is having biggest timestamp '10' and viewKeyTs '3' 
from column 'a'
***  or merged: {{(k=1&=1)@TS(10,3), b=2@10}}

h3. *Changes*

* Extra flag in storage serialization format to facilitate {{viewKeyTs}} and 
{{co-existed standard tombstones under shadowable}}
* Message serialization to store {{viewKeyTs}}
* Row.Merger Process
* BTreeRow Filter Process


was (Author: jasonstack):
h3. *Idea*

{{ShadowableTombstone}} : 
* deletion-time, isShadowable, and "viewKeyTs" aka. base column's ts which is 
part of view pk(used to reconcile when timestamp tie), if there is no timestamp 
associated with that column, use base pk timestamp instead.
* it's only generated when one base column is a pk in view and this base column 
value is changed in base row, to mark previous view row as deleted. (original 
definition of {{shadowable}} in CASSANDRA-10261).  in other cases, {{standard 
tombstone}} is generated for view rows.
* if {{ShadowableTombstone}} is superseded by {{LivenessInfo}}, columns 
shadowed by {{ShadowableTombstone}} will come back alive. (original definition 
of {{shadowable}} in CASSANDRA-10261)
* {{ShadowableTombstone}}  should co-exist with {{Standard Tombstone}} if 
{{shadowable}}'s deletion time supersedes {{standard tombstone}} to avoid 
bringing columns older than {{standard tombstone}} coming back alive( as in 
CASSANDRA-13409)

{{ShadowableLivenessInfo}}:  
* timestamp, and

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

2017-07-17 Thread ZhaoYang (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089856#comment-16089856
 ] 

ZhaoYang edited comment on CASSANDRA-11500 at 7/17/17 1:56 PM:
---

I plan to solve: {{partial update}},{{ttl}}, {{co-existed shadowable 
tombstone}}, {{view timestamp tie}} all inside this ticket using extended 
shadowable approach(mentioned 
[here|https://issues.apache.org/jira/browse/CASSANDRA-11500?focusedCommentId=16082241=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16082241]).
 Because all these issues require some storage format changes(extendedFlag), 
it's better to fix them and refactor in one commit.

Drafted a 
[patch|https://github.com/jasonstack/cassandra/commits/CASSANDRA-11500-update-time]..(refactoring
 and adding more dtest)  

Any suggestions would be appreciated.




was (Author: jasonstack):
I plan to solve: partial update, ttl, co-existed shadowable tombstone, view 
timestamp tie all inside this ticket using extended shadowable 
approach(mentioned 
[here|https://issues.apache.org/jira/browse/CASSANDRA-11500?focusedCommentId=16082241=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16082241]).
 Because all these issues require some storage format changes(extendedFlag), 
it's better to fix them and refactor in one commit.

Drafted a 
[patch|https://github.com/jasonstack/cassandra/commits/CASSANDRA-11500-update-time]..(refactoring
 and adding more dtest)  

Any suggestions would be appreciated.



> Obsolete MV entry may not be properly deleted
> -
>
> Key: CASSANDRA-11500
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11500
> Project: Cassandra
>  Issue Type: Bug
>  Components: Materialized Views
>Reporter: Sylvain Lebresne
>Assignee: ZhaoYang
>
> When a Materialized View uses a non-PK base table column in its PK, if an 
> update changes that column value, we add the new view entry and remove the 
> old one. When doing that removal, the current code uses the same timestamp 
> than for the liveness info of the new entry, which is the max timestamp for 
> any columns participating to the view PK. This is not correct for the 
> deletion as the old view entry could have other columns with higher timestamp 
> which won't be deleted as can easily shown by the failing of the following 
> test:
> {noformat}
> CREATE TABLE t (k int PRIMARY KEY, a int, b int);
> CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS 
> NOT NULL PRIMARY KEY (k, a);
> INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
> UPDATE t USING TIMESTAMP 4 SET b = 2 WHERE k = 1;
> UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1;
> SELECT * FROM mv WHERE k = 1; // This currently return 2 entries, the old 
> (invalid) and the new one
> {noformat}
> So the correct timestamp to use for the deletion is the biggest timestamp in 
> the old view entry (which we know since we read the pre-existing base row), 
> and that is what CASSANDRA-11475 does (the test above thus doesn't fail on 
> that branch).
> Unfortunately, even then we can still have problems if further updates 
> requires us to overide the old entry. Consider the following case:
> {noformat}
> CREATE TABLE t (k int PRIMARY KEY, a int, b int);
> CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS 
> NOT NULL PRIMARY KEY (k, a);
> INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
> UPDATE t USING TIMESTAMP 10 SET b = 2 WHERE k = 1;
> UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1; // This will delete the 
> entry for a=1 with timestamp 10
> UPDATE t USING TIMESTAMP 3 SET a = 1 WHERE k = 1; // This needs to re-insert 
> an entry for a=1 but shouldn't be deleted by the prior deletion
> UPDATE t USING TIMESTAMP 4 SET a = 2 WHERE k = 1; // ... and we can play this 
> game more than once
> UPDATE t USING TIMESTAMP 5 SET a = 1 WHERE k = 1;
> ...
> {noformat}
> In a way, this is saying that the "shadowable" deletion mechanism is not 
> general enough: we need to be able to re-insert an entry when a prior one had 
> been deleted before, but we can't rely on timestamps being strictly bigger on 
> the re-insert. In that sense, this can be though as a similar problem than 
> CASSANDRA-10965, though the solution there of a single flag is not enough 
> since we can have to replace more than once.
> I think the proper solution would be to ship enough information to always be 
> able to decide when a view deletion is shadowed. Which means that both 
> liveness info (for updates) and shadowable deletion would need to ship the 
> timestamp of any base table column that is part the view PK (so {{a}} in the 
> example below).  It's doable (and not that hard really), but it does require 
> a change to the sstable and

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

2017-07-17 Thread ZhaoYang (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16066749#comment-16066749
 ] 

ZhaoYang edited comment on CASSANDRA-11500 at 7/17/17 1:46 PM:
---

{quote}
we need to be able to re-insert an entry when a prior one had been deleted 
before, but we can't rely on timestamps being strictly bigger on the re-insert. 
In that sense, this can be though as a similar problem than CASSANDRA-10965, 
though the solution there of a single flag is not enough since we can have to 
replace more than once.
{quote}

Agreed.

-How about shipping an extra "view-update-time" (`nowInSecond` that 
view-operation is triggered) per view row.  it will be used to check who's new 
when TS ties.-

-The `nowInSeconds` could still be the same in certain cases, but rare, similar 
to CASSANDRA-1096-



was (Author: jasonstack):
{quote}
we need to be able to re-insert an entry when a prior one had been deleted 
before, but we can't rely on timestamps being strictly bigger on the re-insert. 
In that sense, this can be though as a similar problem than CASSANDRA-10965, 
though the solution there of a single flag is not enough since we can have to 
replace more than once.
{quote}

Agreed.

How about shipping an extra "view-update-time" (`nowInSecond` that 
view-operation is triggered) per view row.  it will be used to check who's new 
when TS ties.

The `nowInSeconds` could still be the same in certain cases, but rare, similar 
to CASSANDRA-1096


> Obsolete MV entry may not be properly deleted
> -
>
> Key: CASSANDRA-11500
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11500
> Project: Cassandra
>  Issue Type: Bug
>  Components: Materialized Views
>Reporter: Sylvain Lebresne
>Assignee: ZhaoYang
>
> When a Materialized View uses a non-PK base table column in its PK, if an 
> update changes that column value, we add the new view entry and remove the 
> old one. When doing that removal, the current code uses the same timestamp 
> than for the liveness info of the new entry, which is the max timestamp for 
> any columns participating to the view PK. This is not correct for the 
> deletion as the old view entry could have other columns with higher timestamp 
> which won't be deleted as can easily shown by the failing of the following 
> test:
> {noformat}
> CREATE TABLE t (k int PRIMARY KEY, a int, b int);
> CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS 
> NOT NULL PRIMARY KEY (k, a);
> INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
> UPDATE t USING TIMESTAMP 4 SET b = 2 WHERE k = 1;
> UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1;
> SELECT * FROM mv WHERE k = 1; // This currently return 2 entries, the old 
> (invalid) and the new one
> {noformat}
> So the correct timestamp to use for the deletion is the biggest timestamp in 
> the old view entry (which we know since we read the pre-existing base row), 
> and that is what CASSANDRA-11475 does (the test above thus doesn't fail on 
> that branch).
> Unfortunately, even then we can still have problems if further updates 
> requires us to overide the old entry. Consider the following case:
> {noformat}
> CREATE TABLE t (k int PRIMARY KEY, a int, b int);
> CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS 
> NOT NULL PRIMARY KEY (k, a);
> INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
> UPDATE t USING TIMESTAMP 10 SET b = 2 WHERE k = 1;
> UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1; // This will delete the 
> entry for a=1 with timestamp 10
> UPDATE t USING TIMESTAMP 3 SET a = 1 WHERE k = 1; // This needs to re-insert 
> an entry for a=1 but shouldn't be deleted by the prior deletion
> UPDATE t USING TIMESTAMP 4 SET a = 2 WHERE k = 1; // ... and we can play this 
> game more than once
> UPDATE t USING TIMESTAMP 5 SET a = 1 WHERE k = 1;
> ...
> {noformat}
> In a way, this is saying that the "shadowable" deletion mechanism is not 
> general enough: we need to be able to re-insert an entry when a prior one had 
> been deleted before, but we can't rely on timestamps being strictly bigger on 
> the re-insert. In that sense, this can be though as a similar problem than 
> CASSANDRA-10965, though the solution there of a single flag is not enough 
> since we can have to replace more than once.
> I think the proper solution would be to ship enough information to always be 
> able to decide when a view deletion is shadowed. Which means that both 
> liveness info (for updates) and shadowable deletion would need to ship the 
> timestamp of any base table column that is part the view PK (so {{a}} in the 
> example below).  It's doable (and not that hard really), but it does require 
> a change to the sstable and intra-node protocol, which makes this a bit

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

2017-07-17 Thread ZhaoYang (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082241#comment-16082241
 ] 

ZhaoYang edited comment on CASSANDRA-11500 at 7/17/17 8:35 AM:
---

h3. *Idea*

{{ShadowableTombstone}} : 
* deletion-time, isShadowable, and "viewKeyTs" aka. base column's ts which is 
part of view pk(used to reconcile when timestamp tie), if there is no timestamp 
associated with that column, use base pk timestamp instead.
* it's only generated when one base column is a pk in view and this base column 
value is changed in base row, to mark previous view row as deleted. (original 
definition of {{shadowable}} in CASSANDRA-10261).  in other cases, {{standard 
tombstone}} is generated for view rows.
* if {{ShadowableTombstone}} is superseded by {{LivenessInfo}}, columns 
shadowed by {{ShadowableTombstone}} will come back alive. (original definition 
of {{shadowable}} in CASSANDRA-10261)
* {{ShadowableTombstone}}  should co-exist with {{Standard Tombstone}} if 
{{shadowable}}'s deletion time supersedes {{standard tombstone}} to avoid 
bringing columns older than {{standard tombstone}} coming back alive( as in 
CASSANDRA-13409)

{{ShadowableLivenessInfo}}:  
* timestamp, and "viewKeyTs"
* if shadowable and not live, all columns in this row are considered deleted as 
in CASSANDRA-13657 and CASSANDRA-13127 to solve partial update issues

When reconcile {{ShadowableTombstone}} and {{ShadowableLivenessInfo}}: 
{quote}
if deletion-time greater than timestamp, tombstone wins
if deletion-time smaller than timestamp, livenessInfo wins
when deletion-time ties with timestamp, 
 - if {{ShadowableTombstone}}'s {{viewKeyTs}} >= {{ShadowableLivenessInfo}}'s, 
then tombstone wins
 - else livesnessInfo wins.
{quote}

When inserting to view, always use the greatest timestamp of all base columns 
in view similar to how view deletion timestamp is computed.

h3. *Example*

{quote}
CREATE TABLE t (k int PRIMARY KEY, a int, b int);
CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS NOT 
NULL PRIMARY KEY (k, a);

{{q1}} INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
{{q2}} UPDATE t USING TIMESTAMP 10 SET b = 2 WHERE k = 1;
{{q3}} UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1; 
{{q3}} UPDATE t USING TIMESTAMP 3 SET a = 1 WHERE k = 1; 
{quote}


* After {{q1}}:
** in base: {{k=1@0, a=1, b=1}}// 'k' is having value '1' with timestamp '0'
** in view: 
***  sstable1: {{(k=1&=1)@TS(0,0), b=1}}  // 'k:a' is having value '1:1' with 
timestamp '0' and viewKeyTs '0' from base's pk because column 'a' has no TS
* After {{q2}}
** in base(merged): {{k=1@0, a=1, b=2@10}} 
** in view:  
***  sstable1: {{(k=1&=1)@TS(0,0), b=1}}
***  sstable2: {{(k=1&=1)@TS(10,0), b=2@10}}
***  or merged: {{(k=1&=1)@TS(10,0), b=2@10}}
* After {{q3}}
** in base(merged): {{k=1@0, a=2@2, b=2@10}}  
** in view:  
***  sstable1: {{(k=1&=1)@TS(0,0), b=1}}
***  sstable2: {{(k=1&=1)@TS(10,0), b=2@10}}
***  sstable3: {{(k=1&=1)@Shadowable(10,0)}} & {{(k=1&=2)@TS(10,2), 
b=2@10}}  // '(k=1&=2)' is having biggest timestamp '10' and viewKeyTs '2' 
from column 'a'
***  or merged: {{(k=1&=2)@TS(10,2), b=2@10}}
* After {{q4}}
** in base(merged): {{k=1@0, a=1@3, b=2@10}}  
** in view:  
***  sstable1: {{(k=1&=1)@TS(0,0), b=1}}
***  sstable2: {{(k=1&=1)@TS(10,0), b=2@10}}
***  sstable3: {{(k=1&=1)@Shadowable(10,0)}} & {{(k=1&=2)@TS(10,2), 
b=2@10}} 
***  sstable4: {{(k=1&=2)@Shadowable(10,2)}} & {{(k=1&=1)@TS(10,3), 
b=2@10}}  // '(k=1&=1)' is having biggest timestamp '10' and viewKeyTs '3' 
from column 'a'
***  or merged: {{(k=1&=1)@TS(10,3), b=2@10}}

h3. *Changes*

* Extra flag in storage serialization format to facilitate {{viewKeyTs}} and 
{{co-existed standard tombstones under shadowable}}
* Message serialization to store {{viewKeyTs}}
* Row.Merger Process


was (Author: jasonstack):
h3. *Idea*

{{ShadowableTombstone}} : 
* deletion-time, isShadowable, and "viewKeyTs" aka. base column's ts which is 
part of view pk(used to reconcile when timestamp tie), if there is no timestamp 
associated with that column, use base pk timestamp instead.
* it's only generated when one base column is a pk in view and this base column 
value is changed in base row, to mark previous view row as deleted. (original 
definition of {{shadowable}} in CASSANDRA-10261).  in other cases, {{standard 
tombstone}} is generated for view rows.
* if {{ShadowableTombstone}} is superseded by {{LivenessInfo}}, columns 
shadowed by {{ShadowableTombstone}} will come back alive. (original definition 
of {{shadowable}} in CASSANDRA-10261)
* {{ShadowableTombstone}}  should co-exist with {{Standard Tombstone}} if 
{{shadowable}}'s deletion time supersedes {{standard tombstone}} to avoid 
bringing columns older than {{standard tombstone}} coming back alive( as in 
CASSANDRA-13409)

{{ShadowableLivenessInfo}}:  
* timestamp, and "viewKeyTs"
* nothing

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

2017-07-11 Thread ZhaoYang (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082241#comment-16082241
 ] 

ZhaoYang edited comment on CASSANDRA-11500 at 7/12/17 5:28 AM:
---

h3. *Idea*

{{ShadowableTombstone}} : 
* deletion-time, isShadowable, and "viewKeyTs" aka. base column's ts which is 
part of view pk(used to reconcile when timestamp tie), if there is no timestamp 
associated with that column, use base pk timestamp instead.
* it's only generated when one base column is a pk in view and this base column 
value is changed in base row, to mark previous view row as deleted. (original 
definition of {{shadowable}} in CASSANDRA-10261).  in other cases, {{standard 
tombstone}} is generated for view rows.
* if {{ShadowableTombstone}} is superseded by {{LivenessInfo}}, columns 
shadowed by {{ShadowableTombstone}} will come back alive. (original definition 
of {{shadowable}} in CASSANDRA-10261)
* {{ShadowableTombstone}}  should co-exist with {{Standard Tombstone}} if 
{{shadowable}}'s deletion time supersedes {{standard tombstone}} to avoid 
bringing columns older than {{standard tombstone}} coming back alive( as in 
CASSANDRA-13409)

{{ShadowableLivenessInfo}}:  
* timestamp, and "viewKeyTs"
* nothing special, except for an extra "viewKeyTs"

When reconcile {{ShadowableTombstone}} and {{ShadowableLivenessInfo}}: 
{quote}
if deletion-time greater than timestamp, tombstone wins
if deletion-time smaller than timestamp, livenessInfo wins
when deletion-time ties with timestamp, 
 - if {{ShadowableTombstone}}'s {{viewKeyTs}} >= {{ShadowableLivenessInfo}}'s, 
then tombstone wins
 - else livesnessInfo wins.
{quote}

When inserting to view, always use the greatest timestamp of all base columns 
in view similar to how view deletion timestamp is computed.

h3. *Example*

{quote}
CREATE TABLE t (k int PRIMARY KEY, a int, b int);
CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS NOT 
NULL PRIMARY KEY (k, a);

{{q1}} INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
{{q2}} UPDATE t USING TIMESTAMP 10 SET b = 2 WHERE k = 1;
{{q3}} UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1; 
{{q3}} UPDATE t USING TIMESTAMP 3 SET a = 1 WHERE k = 1; 
{quote}


* After {{q1}}:
** in base: {{k=1@0, a=1, b=1}}// 'k' is having value '1' with timestamp '0'
** in view: 
***  sstable1: {{(k=1&=1)@TS(0,0), b=1}}  // 'k:a' is having value '1:1' with 
timestamp '0' and viewKeyTs '0' from base's pk because column 'a' has no TS
* After {{q2}}
** in base(merged): {{k=1@0, a=1, b=2@10}} 
** in view:  
***  sstable1: {{(k=1&=1)@TS(0,0), b=1}}
***  sstable2: {{(k=1&=1)@TS(10,0), b=2@10}}
***  or merged: {{(k=1&=1)@TS(10,0), b=2@10}}
* After {{q3}}
** in base(merged): {{k=1@0, a=2@2, b=2@10}}  
** in view:  
***  sstable1: {{(k=1&=1)@TS(0,0), b=1}}
***  sstable2: {{(k=1&=1)@TS(10,0), b=2@10}}
***  sstable3: {{(k=1&=1)@Shadowable(10,0)}} & {{(k=1&=2)@TS(10,2), 
b=2@10}}  // '(k=1&=2)' is having biggest timestamp '10' and viewKeyTs '2' 
from column 'a'
***  or merged: {{(k=1&=2)@TS(10,2), b=2@10}}
* After {{q4}}
** in base(merged): {{k=1@0, a=1@3, b=2@10}}  
** in view:  
***  sstable1: {{(k=1&=1)@TS(0,0), b=1}}
***  sstable2: {{(k=1&=1)@TS(10,0), b=2@10}}
***  sstable3: {{(k=1&=1)@Shadowable(10,0)}} & {{(k=1&=2)@TS(10,2), 
b=2@10}} 
***  sstable4: {{(k=1&=2)@Shadowable(10,2)}} & {{(k=1&=1)@TS(10,3), 
b=2@10}}  // '(k=1&=1)' is having biggest timestamp '10' and viewKeyTs '3' 
from column 'a'
***  or merged: {{(k=1&=1)@TS(10,3), b=2@10}}

h3. *Changes*

* Extra flag in storage serialization format to facilitate {{viewKeyTs}} and 
{{co-existed standard tombstones under shadowable}}
* Message serialization to store {{viewKeyTs}}
* Row.Merger Process


was (Author: jasonstack):
h3. *Idea*

{{ShadowableTombstone}} : deletion-time, isShadowable, and "viewKeyTs" aka. 
base column's ts which is part of view pk(used to reconcile when timestamp 
tie), if there is no timestamp associated with that column, use base pk 
timestamp instead.
{{ShadowableLivenessInfo}}:  timestamp, and "viewKeyTs"

When reconcile {{ShadowableTombstone}} and {{ShadowableLivenessInfo}}: 
{quote}
if deletion-time greater than timestamp, tombstone wins
if deletion-time smaller than timestamp, livenessInfo wins
when deletion-time ties with timestamp, 
 - if {{ShadowableTombstone}}'s {{viewKeyTs}} >= {{ShadowableLivenessInfo}}'s, 
then tombstone wins
 - else livesnessInfo wins.
{quote}

When inserting to view, always use the greatest timestamp of all base columns 
in view similar to how view deletion timestamp is computed.

h3. *Example*

{quote}
CREATE TABLE t (k int PRIMARY KEY, a int, b int);
CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS NOT 
NULL PRIMARY KEY (k, a);

{{q1}} INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
{{q2}} UPDATE t USING TIMESTAMP 10 SET b = 2 WHERE k = 1;
{{q3}} UPDATE t

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

2017-07-11 Thread ZhaoYang (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082241#comment-16082241
 ] 

ZhaoYang edited comment on CASSANDRA-11500 at 7/11/17 2:06 PM:
---

h3. *Idea*

{{ShadowableTombstone}} : deletion-time, isShadowable, and "viewKeyTs" aka. 
base column's ts which is part of view pk(used to reconcile when timestamp 
tie), if there is no timestamp associated with that column, use base pk 
timestamp instead.
{{ShadowableLivenessInfo}}:  timestamp, and "viewKeyTs"

When reconcile {{ShadowableTombstone}} and {{ShadowableLivenessInfo}}: 
{quote}
if deletion-time greater than timestamp, tombstone wins
if deletion-time smaller than timestamp, livenessInfo wins
when deletion-time ties with timestamp, 
 - if {{ShadowableTombstone}}'s {{viewKeyTs}} >= {{ShadowableLivenessInfo}}'s, 
then tombstone wins
 - else livesnessInfo wins.
{quote}

When inserting to view, always use the greatest timestamp of all base columns 
in view similar to how view deletion timestamp is computed.

h3. *Example*

{quote}
CREATE TABLE t (k int PRIMARY KEY, a int, b int);
CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS NOT 
NULL PRIMARY KEY (k, a);

{{q1}} INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
{{q2}} UPDATE t USING TIMESTAMP 10 SET b = 2 WHERE k = 1;
{{q3}} UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1; 
{{q3}} UPDATE t USING TIMESTAMP 3 SET a = 1 WHERE k = 1; 
{quote}


* After {{q1}}:
** in base: {{k=1@0, a=1, b=1}}// 'k' is having value '1' with timestamp '0'
** in view: 
***  sstable1: {{(k=1&=1)@TS(0,0), b=1}}  // 'k:a' is having value '1:1' with 
timestamp '0' and viewKeyTs '0' from base's pk because column 'a' has no TS
* After {{q2}}
** in base(merged): {{k=1@0, a=1, b=2@10}} 
** in view:  
***  sstable1: {{(k=1&=1)@TS(0,0), b=1}}
***  sstable2: {{(k=1&=1)@TS(10,0), b=2@10}}
***  or merged: {{(k=1&=1)@TS(10,0), b=2@10}}
* After {{q3}}
** in base(merged): {{k=1@0, a=2@2, b=2@10}}  
** in view:  
***  sstable1: {{(k=1&=1)@TS(0,0), b=1}}
***  sstable2: {{(k=1&=1)@TS(10,0), b=2@10}}
***  sstable3: {{(k=1&=1)@Shadowable(10,0)}} & {{(k=1&=2)@TS(10,2), 
b=2@10}}  // '(k=1&=2)' is having biggest timestamp '10' and viewKeyTs '2' 
from column 'a'
***  or merged: {{(k=1&=2)@TS(10,2), b=2@10}}
* After {{q4}}
** in base(merged): {{k=1@0, a=1@3, b=2@10}}  
** in view:  
***  sstable1: {{(k=1&=1)@TS(0,0), b=1}}
***  sstable2: {{(k=1&=1)@TS(10,0), b=2@10}}
***  sstable3: {{(k=1&=1)@Shadowable(10,0)}} & {{(k=1&=2)@TS(10,2), 
b=2@10}} 
***  sstable4: {{(k=1&=2)@Shadowable(10,2)}} & {{(k=1&=1)@TS(10,3), 
b=2@10}}  // '(k=1&=1)' is having biggest timestamp '10' and viewKeyTs '3' 
from column 'a'
***  or merged: {{(k=1&=1)@TS(10,3), b=2@10}}




was (Author: jasonstack):
h3. *Idea*

{{ShadowableTombstone}} : deletion-time, isShadowable, and "viewKeyTs" aka. 
base column's ts which is part of view pk(used to reconcile when timestamp 
tie), if there is no timestamp associated with that column, use base pk 
timestamp instead.
{{ShadowableLivenessInfo}}:  timestamp, and "viewKeyTs"

When reconcile {{ShadowableTombstone}} and {{ShadowableLivenessInfo}}: 
{quote}
if deletion-time greater than timestamp, tombstone wins
if deletion-time smaller than timestamp, livenessInfo wins
when deletion-time ties with timestamp, 
 - if {{ShadowableTombstone}}'s {{viewKeyTs}} >= {{ShadowableLivenessInfo}}', 
then tombstone wins
 - else livesnessInfo wins.
{quote}

When inserting to view, always use the greatest timestamp of all base columns 
in view similar to how view deletion timestamp is computed.

h3. *Example*

{quote}
CREATE TABLE t (k int PRIMARY KEY, a int, b int);
CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS NOT 
NULL PRIMARY KEY (k, a);

{{q1}} INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
{{q2}} UPDATE t USING TIMESTAMP 10 SET b = 2 WHERE k = 1;
{{q3}} UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1; 
{{q3}} UPDATE t USING TIMESTAMP 3 SET a = 1 WHERE k = 1; 
{quote}


* After {{q1}}:
** in base: {{k=1@0, a=1, b=1}}// 'k' is having value '1' with timestamp '0'
** in view: 
***  sstable1: {{(k=1&=1)@TS(0,0), b=1}}  // 'k:a' is having value '1:1' with 
timestamp '0' and viewKeyTs '0' from base's pk because column 'a' has no TS
* After {{q2}}
** in base(merged): {{k=1@0, a=1, b=2@10}} 
** in view:  
***  sstable1: {{(k=1&=1)@TS(0,0), b=1}}
***  sstable2: {{(k=1&=1)@TS(10,0), b=2@10}}
***  or merged: {{(k=1&=1)@TS(10,0), b=2@10}}
* After {{q3}}
** in base(merged): {{k=1@0, a=2@2, b=2@10}}  
** in view:  
***  sstable1: {{(k=1&=1)@TS(0,0), b=1}}
***  sstable2: {{(k=1&=1)@TS(10,0), b=2@10}}
***  sstable3: {{(k=1&=1)@Shadowable(10,0)}} & {{(k=1&=2)@TS(10,2), 
b=2@10}}  // '(k=1&=2)' is having biggest timestamp '10' and viewKeyTs '2' 
from column 'a'
***  or merged: {{(k=1&=2)@TS(10,2), b=2@10}}
* After {{q4}}
** in base(merged):

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

2017-07-11 Thread ZhaoYang (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082241#comment-16082241
 ] 

ZhaoYang edited comment on CASSANDRA-11500 at 7/11/17 2:03 PM:
---

h3. *Idea*

{{ShadowableTombstone}} : deletion-time, isShadowable, and "viewKeyTs" aka. 
base column's ts which is part of view pk(used to reconcile when timestamp 
tie), if there is no timestamp associated with that column, use base pk 
timestamp instead.
{{ShadowableLivenessInfo}}:  timestamp, and "viewKeyTs"

When reconcile {{ShadowableTombstone}} and {{ShadowableLivenessInfo}}: 
{quote}
if deletion-time greater than timestamp, tombstone wins
if deletion-time smaller than timestamp, livenessInfo wins
when deletion-time ties with timestamp, 
 - if {{ShadowableTombstone}}'s {{viewKeyTs}} >= {{ShadowableLivenessInfo}}', 
then tombstone wins
 - else livesnessInfo wins.
{quote}

When inserting to view, always use the greatest timestamp of all base columns 
in view similar to how view deletion timestamp is computed.

h3. *Example*

{quote}
CREATE TABLE t (k int PRIMARY KEY, a int, b int);
CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS NOT 
NULL PRIMARY KEY (k, a);

{{q1}} INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
{{q2}} UPDATE t USING TIMESTAMP 10 SET b = 2 WHERE k = 1;
{{q3}} UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1; 
{{q3}} UPDATE t USING TIMESTAMP 3 SET a = 1 WHERE k = 1; 
{quote}


* After {{q1}}:
** in base: {{k=1@0, a=1, b=1}}// 'k' is having value '1' with timestamp '0'
** in view: 
***  sstable1: {{(k=1&=1)@TS(0,0), b=1}}  // 'k:a' is having value '1:1' with 
timestamp '0' and viewKeyTs '0' from base's pk because column 'a' has no TS
* After {{q2}}
** in base(merged): {{k=1@0, a=1, b=2@10}} 
** in view:  
***  sstable1: {{(k=1&=1)@TS(0,0), b=1}}
***  sstable2: {{(k=1&=1)@TS(10,0), b=2@10}}
***  or merged: {{(k=1&=1)@TS(10,0), b=2@10}}
* After {{q3}}
** in base(merged): {{k=1@0, a=2@2, b=2@10}}  
** in view:  
***  sstable1: {{(k=1&=1)@TS(0,0), b=1}}
***  sstable2: {{(k=1&=1)@TS(10,0), b=2@10}}
***  sstable3: {{(k=1&=1)@Shadowable(10,0)}} & {{(k=1&=2)@TS(10,2), 
b=2@10}}  // '(k=1&=2)' is having biggest timestamp '10' and viewKeyTs '2' 
from column 'a'
***  or merged: {{(k=1&=2)@TS(10,2), b=2@10}}
* After {{q4}}
** in base(merged): {{k=1@0, a=1@3, b=2@10}}  
** in view:  
***  sstable1: {{(k=1&=1)@TS(0,0), b=1}}
***  sstable2: {{(k=1&=1)@TS(10,0), b=2@10}}
***  sstable3: {{(k=1&=1)@Shadowable(10,0)}} & {{(k=1&=2)@TS(10,2), 
b=2@10}} 
***  sstable4: {{(k=1&=2)@Shadowable(10,2)}} & {{(k=1&=1)@TS(10,3), 
b=2@10}}  // '(k=1&=1)' is having biggest timestamp '10' and viewKeyTs '3' 
from column 'a'
***  or merged: {{(k=1&=1)@TS(10,3), b=2@10}}




was (Author: jasonstack):
h3. idea

{{ShadowableTombstone}} : deletion-time, isShadowable, and "viewKeyTs" aka. 
base column's ts which is part of view pk(used to reconcile when timestamp 
tie), if there is no timestamp associated with that column, use base pk 
timestamp instead.
{{ShadowableLivenessInfo}}:  timestamp, and "viewKeyTs"

When reconcile {{ShadowableTombstone}} and {{ShadowableLivenessInfo}}: 
{quote}
if deletion-time greater than timestamp, tombstone wins
if deletion-time smaller than timestamp, livenessInfo wins
when deletion-time ties with timestamp, 
 - if {{ShadowableTombstone}}'s {{viewKeyTs}} >= {{ShadowableLivenessInfo}}', 
then tombstone wins
 - else livesnessInfo wins.
{quote}

When inserting to view, always use the greatest timestamp of all base columns 
in view similar to how view deletion timestamp is computed.

h3. example

{quote}
CREATE TABLE t (k int PRIMARY KEY, a int, b int);
CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS NOT 
NULL PRIMARY KEY (k, a);

{{q1}} INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
{{q2}} UPDATE t USING TIMESTAMP 10 SET b = 2 WHERE k = 1;
{{q3}} UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1; 
{{q3}} UPDATE t USING TIMESTAMP 3 SET a = 1 WHERE k = 1; 
{quote}


* After {{q1}}:
** in base: {{k=1@0, a=1, b=1}}// 'k' is having value '1' with timestamp '0'
** in view: 
***  sstable1: {{(k=1&=1)@TS(0,0), b=1}}  // 'k:a' is having value '1:1' with 
timestamp '0' and viewKeyTs '0' from base's pk because column 'a' has no TS
* After {{q2}}
** in base(merged): {{k=1@0, a=1, b=2@10}} 
** in view:  
***  sstable1: {{(k=1&=1)@TS(0,0), b=1}}
***  sstable2: {{(k=1&=1)@TS(10,0), b=2@10}}
***  or merged: {{(k=1&=1)@TS(10,0), b=2@10}}
* After {{q3}}
** in base(merged): {{k=1@0, a=2@2, b=2@10}}  
** in view:  
***  sstable1: {{(k=1&=1)@TS(0,0), b=1}}
***  sstable2: {{(k=1&=1)@TS(10,0), b=2@10}}
***  sstable3: {{(k=1&=1)@Shadowable(10,0)}} & {{(k=1&=2)@TS(10,2), 
b=2@10}}  // '(k=1&=2)' is having biggest timestamp '10' and viewKeyTs '2' 
from column 'a'
***  or merged: {{(k=1&=2)@TS(10,2), b=2@10}}
* After {{q4}}
** in base(merged): {{k=1@0,

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

[jira] [Comment Edited] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

26 matches

Site Navigation

Mail list logo

Footer information