[jira] [Commented] (CASSANDRA-7859) Extend freezing to collections
[ https://issues.apache.org/jira/browse/CASSANDRA-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201907#comment-14201907 ] Sylvain Lebresne commented on CASSANDRA-7859: - bq. Does that clarify things? Yes, you're right, I was a bit quick in doing my comments. That said, those functions are still a bit hard to read, and instead of having {{CollectionType.isMultiCellCompatibleWith}} and every concrete collection {{isCompatibleWith}} method starting by {{if isMultiCell() return isMultipleCellCompatibleWith}}, I'd prefer having {{CollectionType}} directly implement {{isCompatibleWith}} for multiCell, and delegate just the non-multiCell case to an abstract method implemented by all concrete collections. But I'm good with you doing that upon commit. bq. What do you think about my comment on 8264 and getRequestedColumns? I finally had a look at the changes and it does look reasonable so I'm fine including them here an leaving what remains to CASSANDRA-8264. Overall, +1. > Extend freezing to collections > -- > > Key: CASSANDRA-7859 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7859 > Project: Cassandra > Issue Type: Improvement >Reporter: Sylvain Lebresne >Assignee: Tyler Hobbs > Labels: cql > Fix For: 2.1.2 > > Attachments: 7859-v1.txt > > > This is the follow-up to CASSANDRA-7857, to extend {{frozen}} to collections. > This will allow things like {{map>>}} for > instance, as well as allowing {{frozen}} collections in PK columns. > Additionally (and that's alsmot a separate ticket but I figured we can start > discussing it here), we could decide that tuple is a frozen type by default. > This means that we would allow {{tuple}} without needing to add > {{frozen}}, but we would require {{frozen}} for complex type inside tuple, so > {{tuple>}} would be rejected, but not {{tuple frozen>>}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7859) Extend freezing to collections
[ https://issues.apache.org/jira/browse/CASSANDRA-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200482#comment-14200482 ] Tyler Hobbs commented on CASSANDRA-7859: bq. To reiter, collection values are sorted (even when used as cell values), and so it felt fishy to me to consider that changing to a type that change the internal sorting of those values is "compatible" Right, I agree with that, and the current patch doesn't change that. I think an example will clarify this. Take the type {{frozen>}}. If you use that type for a clustering column, then entire lists will be compared (is [1, 2, 3] > [2, 3, 4]? etc) to sort the cells. In this case, the sorting order of {{int}} must be maintained when you alter the type. However, if you use {{frozen>}} for a cell value, that sorting order no longer matters. It would be safe to alter it to {{frozen>}}. That does *not* apply to sets, because the items within a set should be sorted. It also does not apply to map *keys*, but map *values* can be treated like list values. Does that clarify things? I was a little tired when I tried to explain it last night. bq. Maybe we should just remove the comment in that case. Yup, already removed it in the last commit. What do you think about my comment on 8264 and getRequestedColumns? > Extend freezing to collections > -- > > Key: CASSANDRA-7859 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7859 > Project: Cassandra > Issue Type: Improvement >Reporter: Sylvain Lebresne >Assignee: Tyler Hobbs > Labels: cql > Fix For: 2.1.2 > > Attachments: 7859-v1.txt > > > This is the follow-up to CASSANDRA-7857, to extend {{frozen}} to collections. > This will allow things like {{map>>}} for > instance, as well as allowing {{frozen}} collections in PK columns. > Additionally (and that's alsmot a separate ticket but I figured we can start > discussing it here), we could decide that tuple is a frozen type by default. > This means that we would allow {{tuple}} without needing to add > {{frozen}}, but we would require {{frozen}} for complex type inside tuple, so > {{tuple>}} would be rejected, but not {{tuple frozen>>}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7859) Extend freezing to collections
[ https://issues.apache.org/jira/browse/CASSANDRA-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200421#comment-14200421 ] Sylvain Lebresne commented on CASSANDRA-7859: - bq. When frozen lists and maps are used as cell values, the collection value ordering doesn't matter. I'm confused because my earlier comment was exactly to say that I'm uncomfortable making that assumption and you seemed to agree initially. To reiter, collection values are sorted (even when used as cell values), and so it felt fishy to me to consider that changing to a type that change the internal sorting of those values is "compatible". Typically sending a value to clients that is not properly sorted feels wrong. And given we lost little in practice by requiring the sorting not to change, I'd prefer sticking to that for now. bq. I think it could potentially introduce assertion errors when the behavior is reasonable. Fair enough, it's just that the comments in there pretty much says that it should only be call for frozen collections. Maybe we should just remove the comment in that case. Not a big deal though. > Extend freezing to collections > -- > > Key: CASSANDRA-7859 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7859 > Project: Cassandra > Issue Type: Improvement >Reporter: Sylvain Lebresne >Assignee: Tyler Hobbs > Labels: cql > Fix For: 2.1.2 > > Attachments: 7859-v1.txt > > > This is the follow-up to CASSANDRA-7857, to extend {{frozen}} to collections. > This will allow things like {{map>>}} for > instance, as well as allowing {{frozen}} collections in PK columns. > Additionally (and that's alsmot a separate ticket but I figured we can start > discussing it here), we could decide that tuple is a frozen type by default. > This means that we would allow {{tuple}} without needing to add > {{frozen}}, but we would require {{frozen}} for complex type inside tuple, so > {{tuple>}} would be rejected, but not {{tuple frozen>>}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7859) Extend freezing to collections
[ https://issues.apache.org/jira/browse/CASSANDRA-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199487#comment-14199487 ] Tyler Hobbs commented on CASSANDRA-7859: bq. It needs a minor rebase Merged the latest 2.1 (changes are on the same branch) bq. SelectStatement.getRequestedColumns still has the unrelated code. Haven't really reviewed it so would be nice to pull it out for some other ticket. So, the problem had to do with multi-column {{IN}} queries on {{COMPACT STORAGE}} tables. I started CASSANDRA-8264 for this, but with those changes reversed, there are test failures in FrozenCollectionsTest. As described in 8264, it looks like there are some additional problems with {{COMPACT STORAGE}} and multi-column relations that are not resolved by this ticket. Would you prefer committing the current changes and doing the rest in 8264, or committing this with with the getRequestedColumns changes reversed and fixing it all together in 8264? bq. I kind of like the idea of shortening fullcollection to full, let's do that (unless someone else chime in with a better suggestion). Done bq. I don't think we need CollectionType.isValidCompatibleWithInternal at all since it's equivalent to isCompatibleWith. That's not actually true for frozen lists and maps. When frozen lists and maps are used as cell values, the collection value ordering doesn't matter. However, while thinking about this I realized the original implementation wasn't exactly correct, so there are a couple of new commits in the branch that fix this. Basically, the nameComparator and valueComparator shouldn't be used for frozen collections, since they are directly compared with {{elements}} and {{keys}}/{{values}} for maps. bq. We could now add a assert !isMultiCell in the collections compare() method. That's true, but I think it could potentially introduce assertion errors when the behavior is reasonable. For example, suppose a list of IN values are being sorted for a (non-frozen) collection column; it would be natural to use the non-frozen CollectionType instance's {{compare()}} for sorting. To avoid making that fail (probably unexpectedly), I would prefer not to add an assert. > Extend freezing to collections > -- > > Key: CASSANDRA-7859 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7859 > Project: Cassandra > Issue Type: Improvement >Reporter: Sylvain Lebresne >Assignee: Tyler Hobbs > Labels: cql > Fix For: 2.1.2 > > Attachments: 7859-v1.txt > > > This is the follow-up to CASSANDRA-7857, to extend {{frozen}} to collections. > This will allow things like {{map>>}} for > instance, as well as allowing {{frozen}} collections in PK columns. > Additionally (and that's alsmot a separate ticket but I figured we can start > discussing it here), we could decide that tuple is a frozen type by default. > This means that we would allow {{tuple}} without needing to add > {{frozen}}, but we would require {{frozen}} for complex type inside tuple, so > {{tuple>}} would be rejected, but not {{tuple frozen>>}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7859) Extend freezing to collections
[ https://issues.apache.org/jira/browse/CASSANDRA-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14196001#comment-14196001 ] Sylvain Lebresne commented on CASSANDRA-7859: - Mostly look good, last few minor remarks: * It needs a minor rebase * {{SelectStatement.getRequestedColumns}} still has the unrelated code. Haven't really reviewed it so would be nice to pull it out for some other ticket. * I kind of like the idea of shortening {{fullcollection}} to {{full}}, let's do that (unless someone else chime in with a better suggestion). * I don't think we need {{CollectionType.isValidCompatibleWithInternal}} at all since it's equivalent to {{isCompatibleWith}}. * We could now add a {{assert !isMultiCell}} in the collections {{compare()}} method. I'll note that part of me feels like this might be a bit too big for 2.1, but there is a fair amount of tests so I'm not strongly opposed to it either. > Extend freezing to collections > -- > > Key: CASSANDRA-7859 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7859 > Project: Cassandra > Issue Type: Improvement >Reporter: Sylvain Lebresne >Assignee: Tyler Hobbs > Labels: cql > Fix For: 2.1.2 > > Attachments: 7859-v1.txt > > > This is the follow-up to CASSANDRA-7857, to extend {{frozen}} to collections. > This will allow things like {{map>>}} for > instance, as well as allowing {{frozen}} collections in PK columns. > Additionally (and that's alsmot a separate ticket but I figured we can start > discussing it here), we could decide that tuple is a frozen type by default. > This means that we would allow {{tuple}} without needing to add > {{frozen}}, but we would require {{frozen}} for complex type inside tuple, so > {{tuple>}} would be rejected, but not {{tuple frozen>>}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7859) Extend freezing to collections
[ https://issues.apache.org/jira/browse/CASSANDRA-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14191005#comment-14191005 ] Tyler Hobbs commented on CASSANDRA-7859: My branch has been updated. bq. I'm not totally convinced about adding new Frozen*Type classes. It adds a bunch of code duplication and changes The code duplication was only around extremely simple methods (field accessors, getInstance(), etc), so I felt it was worth it for the clarity. But in any case, they have been removed as suggested, since I think that's reasonable as well. bq. need to be a tad smarter when dealing with serializing/deserializing types Good catch. I've modified toString() for serialization, and added a mostly-fake FrozenType class that's only used for type parsing to address this (it seems simpler than special-casing TypeParser). bq. The commit seems to include changes from CASSANDRA-6904, and some changes in StorageService/StreamCoordinator that are unrelated to this ticket. Hmm, not sure how that happened. I merged in the latest 2.1, so this should be resolved. bq. I wonder if we shouldn't allow frozen on any type syntax wise. This would simply have no impact on simple types but that's already what happens for tuples after this patch so this would be somewhat consistent. I suspect this might simplify validation at least. With the recent changes, that's what's effectively done internally now. However, I think allowing that in the syntax would be more confusing than helpful for users. I can imagine a lot of questions like "what's the difference between a frozen int and a non-frozen int?". The fact that tuple has this behavior is just an unfortunate oddity, imo. bq. In CQL3Type.Raw.RawCollection, in the prepare method, we could replace ... Done bq. I wonder if we shouldn't make it a bit more generic so that if we want to index full (frozen) UDT in the future we don't have to introduce a fullUDT and so on. What about generalizing to wholeValue (which is still a crappy name so better naming ideas welcome)? We already support indexing complete (frozen) UDTs. However, just shortening it to {{full()}} would be relatively clear, succinct, and flexible enough to cover other types if needed. bq. In Tuples and UserTypes bindInternal methods, not sure using isMultiCell over the existing isCollection is a good idea. Fixed, thanks. bq. In SelectStatement.getRequestedColumns, the changes don't seem related to this issue, are they? No, they aren't. It's a bug that I discovered while working on this, and I intended to break that out into a separate ticket but forgot to. I'll open a separate ticket as soon as I can remember exactly what the bug was :) bq. In SelectStatement.buildBound, the !r.isContains() test is a no-op since ... Fixed as suggested. bq. In SingleColumnRestriction.Contains.canEvaluateWithSlices, I don't understand the comment. It was just an idea for a potential future optimization. It wasn't really useful, though, so it's been removed. bq. Why do we need the newly added code in SecondaryIndexSearcher.validate? We don't, removed. bq. Note 100% sure about the isValueCompatible() for frozen collections... Fair point. It now uses {{isCompatible()}} for the name comparator. > Extend freezing to collections > -- > > Key: CASSANDRA-7859 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7859 > Project: Cassandra > Issue Type: Improvement >Reporter: Sylvain Lebresne >Assignee: Tyler Hobbs > Labels: cql > Fix For: 2.1.2 > > Attachments: 7859-v1.txt > > > This is the follow-up to CASSANDRA-7857, to extend {{frozen}} to collections. > This will allow things like {{map>>}} for > instance, as well as allowing {{frozen}} collections in PK columns. > Additionally (and that's alsmot a separate ticket but I figured we can start > discussing it here), we could decide that tuple is a frozen type by default. > This means that we would allow {{tuple}} without needing to add > {{frozen}}, but we would require {{frozen}} for complex type inside tuple, so > {{tuple>}} would be rejected, but not {{tuple frozen>>}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7859) Extend freezing to collections
[ https://issues.apache.org/jira/browse/CASSANDRA-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14185402#comment-14185402 ] Sylvain Lebresne commented on CASSANDRA-7859: - I'm not totally convinced about adding new Frozen*Type classes. It adds a bunch of code duplication and changes (and breaks a number of instanceof checks, see my last remark below) for unclear benefis imo. Instead, I'd prefer adding a {{isFrozen}} (or {{isMultiCell}}) in CollectionType. This would also remove also the {{IMap}}, {{ISet}}, {{IList}} and {{MultiCollectionCellType}} and would save a ton of diff from the patch. Also I think those new classes adds some confusion (and somewhat break backward compatibility) in the sense that if you have {noformat} CREATE TYPE myType (s set) CREATE TABLE foo (k frozen PRIMARY KEY) {noformat} then for table create before this patch, the validator for {{k}} will be {{UserType(foo, s:SetType(Int32Type))}}, while after this patch it will be {{UserType(foo, s:FrozenSetType(Int32Type))}}. So I think we'd also need to be a tad smarter when dealing with serializing/deserializing types: we'll continue to serialize the type below as {{UserType(foo, s:SetType(Int32Type))}} for compatibility sake, but when parsing that, we'd remember the set is inside a UDT and thus frozen/non-multi-cell. When we have frozen collections, say {{frozen>>}}, we could serialize it as {{FrozenType(SetType(ListType(Int32Type)))}}, which would be easier to handlefor drivers (when they read/decode schema tables). Other remarks: * The commit seems to include changes from CASSANDRA-6904, and some changes in StorageService/StreamCoordinator that are unrelated to this ticket. * I wonder if we shouldn't allow {{frozen}} on any type syntax wise. This would simply have no impact on simple types but that's already what happens for tuples after this patch so this would be somewhat consistent. I suspect this might simplify validation at least. * In {{CQL3Type.Raw.RawCollection}}, in the {{prepare}} method, we could replace {{if (!frozen && values.isCollection() && !values.frozen)}} by {{if (!frozen && values.supportsFreezing() && !values.frozen)}} so it works properly once we have non-frozen UDT. Also, I think the {{keys}} and {{values}} field can stay {{final}}. * A bit sad you didn't found a more brilliant name than my {{fullCollection}}. And I wonder if we shouldn't make it a bit more generic so that if we want to index full (frozen) UDT in the future we don't have to introduce a {{fullUDT}} and so on. What about generalizing to {{wholeValue}} (which is still a crappy name so better naming ideas welcome)? * In {{Tuples}} and {{UserTypes}} {{bindInternal}} methods, not sure using {{isMultiCell}} over the existing {{isCollection}} is a good idea. * In {{SelectStatement.getRequestedColumns}}, the changes don't seem related to this issue, are they? * In {{SelectStatement.buildBound}}, the {{!r.isContains()}} test is a no-op since the first {{isNullRestriction()}} branch will have been taken. Also not convinced having the {{CONTAINS} test in {{isNullRestriction}} is very intuitive. I'd rather replace the test by {{if (isNullRestriction(r, b) || !r.canEvaluateWithSlice())}} directly in {{buildBound}}. * In {{SingleColumnRestriction.Contains.canEvaluateWithSlices}}, I don't understand the comment. * Why do we need the newly added code in {{SecondaryIndexSearcher.validate}}? * In {{CompositesIndexOnCollectionKey.supportsOperator}}, let's add parenthesis rather than relying on the precedence of {{&&}} over {{||}}. * Note 100% sure about the {{isValueCompatible()}} for frozen collections. Frozen collection values are stored sorted, and so it feels allowing changes that modify the sorting could backfire in subtle ways. I haven't carefully checked if it would actually break or not, but I'd prefer keeping it more restrictive initially, and if many people complain about the restriction, we can think long and hard about whether that's safe or not. * As the patch stand, there is a few {{instanceof MapType}}, {{instanceof ListType}} and {{instanceof SetType}} that should have a {{I}} added in front (but as said above, I'd rather just get rid of those new classes entirely). > Extend freezing to collections > -- > > Key: CASSANDRA-7859 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7859 > Project: Cassandra > Issue Type: Improvement >Reporter: Sylvain Lebresne >Assignee: Tyler Hobbs > Labels: cql > Fix For: 2.1.2 > > Attachments: 7859-v1.txt > > > This is the follow-up to CASSANDRA-7857, to extend {{frozen}} to collections. > This will allow things like {{map>>}} for > instance, as well as allowing {{frozen}} collections in PK columns. > Additionally (and that's alsm
[jira] [Commented] (CASSANDRA-7859) Extend freezing to collections
[ https://issues.apache.org/jira/browse/CASSANDRA-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151547#comment-14151547 ] Sylvain Lebresne commented on CASSANDRA-7859: - bq. By default, indexes on frozen collections will index the entire collection. What bother me about that is the inconstency with non-frozen collections and I do feel that this inconsistency will be unintuitive. So I would prefer leaving the default be indexing by values, and add some new keyword for indexing entire collections. Something like {{CREATE INDEX ON foo(fullCollection(myMap))}}, excepted that I'm hoping someone can come with a less crappy name than {{fullCollection}}. More generally, on the indexing of frozen collections, indexing on the entire collection should be relatively simple but indexing on values and keys will require new indexes implementation (new subclasses of {{CompositesIndex}}) and so this should definitively be pushed to a follow-up ticket imo. Not even sure said ticket should be targeted at 2.1. > Extend freezing to collections > -- > > Key: CASSANDRA-7859 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7859 > Project: Cassandra > Issue Type: Improvement >Reporter: Sylvain Lebresne >Assignee: Tyler Hobbs > Fix For: 2.1.1 > > > This is the follow-up to CASSANDRA-7857, to extend {{frozen}} to collections. > This will allow things like {{map>>}} for > instance, as well as allowing {{frozen}} collections in PK columns. > Additionally (and that's alsmot a separate ticket but I figured we can start > discussing it here), we could decide that tuple is a frozen type by default. > This means that we would allow {{tuple}} without needing to add > {{frozen}}, but we would require {{frozen}} for complex type inside tuple, so > {{tuple>}} would be rejected, but not {{tuple frozen>>}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7859) Extend freezing to collections
[ https://issues.apache.org/jira/browse/CASSANDRA-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14149627#comment-14149627 ] Tyler Hobbs commented on CASSANDRA-7859: bq. Additionally (and that's alsmot a separate ticket but I figured we can start discussing it here), we could decide that tuple is a frozen type by default. This means that we would allow tuple without needing to add frozen, but we would require frozen for complex type inside tuple, so tuple> would be rejected, but not tuple>>. I think that's reasonable. There are very few cases where non-frozen tuples would be useful, let alone the best option, so I don't think we'll ever need them. It's actually pretty simple to make this change, so I can do that as part of this ticket. > Extend freezing to collections > -- > > Key: CASSANDRA-7859 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7859 > Project: Cassandra > Issue Type: Improvement >Reporter: Sylvain Lebresne >Assignee: Tyler Hobbs > Fix For: 2.1.1 > > > This is the follow-up to CASSANDRA-7857, to extend {{frozen}} to collections. > This will allow things like {{map>>}} for > instance, as well as allowing {{frozen}} collections in PK columns. > Additionally (and that's alsmot a separate ticket but I figured we can start > discussing it here), we could decide that tuple is a frozen type by default. > This means that we would allow {{tuple}} without needing to add > {{frozen}}, but we would require {{frozen}} for complex type inside tuple, so > {{tuple>}} would be rejected, but not {{tuple frozen>>}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7859) Extend freezing to collections
[ https://issues.apache.org/jira/browse/CASSANDRA-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14148341#comment-14148341 ] Tyler Hobbs commented on CASSANDRA-7859: For secondary indexes, do we want to support indexing keys and values of frozen collections? If so, do we want to introduce a {{values()}} modifier to indicate that values should be indexed and index the whole collection by default? If we assume that the values should be indexed when no modifier is used, it will be consistent with the behavior for non-frozen collections. On the other hand, we actually have the ability to index the entire frozen collection, and this may be more in line with how frozen collections will be used. My personal opinion is that we should: # Introduce a {{values()}} modifier that can be used on collection indexes. For non-frozen collections, this is implied. # Accept both {{keys()}} and {{values()}} indexes on frozen collections. # By default, indexes on frozen collections will index the entire collection. I can (and probably should) split secondary index work into a second ticket, but it would be good to decide some of this upfront in case that ticket can't make it into the same release as this one. > Extend freezing to collections > -- > > Key: CASSANDRA-7859 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7859 > Project: Cassandra > Issue Type: Improvement >Reporter: Sylvain Lebresne >Assignee: Tyler Hobbs > Fix For: 2.1.1 > > > This is the follow-up to CASSANDRA-7857, to extend {{frozen}} to collections. > This will allow things like {{map>>}} for > instance, as well as allowing {{frozen}} collections in PK columns. > Additionally (and that's alsmot a separate ticket but I figured we can start > discussing it here), we could decide that tuple is a frozen type by default. > This means that we would allow {{tuple}} without needing to add > {{frozen}}, but we would require {{frozen}} for complex type inside tuple, so > {{tuple>}} would be rejected, but not {{tuple frozen>>}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332)