[jira] [Commented] (CASSANDRA-7826) support non-frozen, nested collections
[ https://issues.apache.org/jira/browse/CASSANDRA-7826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318599#comment-15318599 ] Sylvain Lebresne commented on CASSANDRA-7826: - This is kind of "blocked" on CASSANDRA-7396, for which I have an almost ready updated patch for, but have been finishing CASSANDRA-10783. Took me longer than expected so the lateness on progress is my fault. > support non-frozen, nested collections > -- > > Key: CASSANDRA-7826 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7826 > Project: Cassandra > Issue Type: Improvement > Components: CQL >Reporter: Tupshin Harper >Assignee: Alex Petrov > Labels: ponies > Fix For: 3.x > > > The inability to nest collections is one of the bigger data modelling > limitations we have right now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7826) support non-frozen, nested collections
[ https://issues.apache.org/jira/browse/CASSANDRA-7826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15318514#comment-15318514 ] Jonathan Ellis commented on CASSANDRA-7826: --- How is progress looking here? > support non-frozen, nested collections > -- > > Key: CASSANDRA-7826 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7826 > Project: Cassandra > Issue Type: Improvement > Components: CQL >Reporter: Tupshin Harper >Assignee: Alex Petrov > Labels: ponies > Fix For: 3.x > > > The inability to nest collections is one of the bigger data modelling > limitations we have right now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7826) support non-frozen, nested collections
[
https://issues.apache.org/jira/browse/CASSANDRA-7826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278429#comment-15278429
]
Sylvain Lebresne commented on CASSANDRA-7826:
-
bq. there's also the question if we would like to support arbitrary updates or
deletes right away, too. That would require new syntax for selecting the path
for update or delete (maybe something like mymap\['a'\]\['b'\]\[1\]
You're right, we'll need some new syntax here. But as to what we support, I'd
say we can support the easily doable extensions of what we already support to
any level of nestedness using square brackets. So if {{m}} is a {{map>}}, we'd support:
{noformat}
* m = { 'foo' : { 'bar' : { 2, 3 } } } and m = m + { 'foo' : { 'bar' : { 2, 3 }
} };
* m['foo'] = { 'bar' : { 2, 3 } } and m['foo'] = m['foo'] + { 'bar' : { 2, 3 }
};
* m['foo']['bar'] = {2, 3}, m['foo']['bar'] = m['foo']['bar'] + {1} and
m['foo']['bar'] = m['foo']['bar'] - {1}.
{noformat}
But as we currently have no way to delete a range of complex cells, we can only
allow deleting elements in the most nested collection. Which is a bit annoying
but adding range tombstone of complex cells is definitvely a follow-up ticket
on its own imo.
bq. Also, BufferCell/tombstone can most likely handle deletes as they are,
without slices, as we'll always know at least the partial path, but if I
understand it correctly, it will require additional logic for discarding
columns whose partial path matches the path of the tombstone.
I'm not entirely sure to follow. Are you suggesting to use a partial
{{CellPath}} to implement deletion of a slice of complex cells?. If so, I
suppose that's an option, but as said above, it's definitively something I
would leave to a follow up ticket.
bq. most likely 7396 will land before this patch, so it we might have to add
support for selecting individual parts on arbitrary level (or just on the top
level at first and leave the rest for the next iterations). So the question is
what we'd like to have on the current step.
I would agree that it probably make more sense to finish/commit CASSANDRA-7396
first and rebasing this on top of it. Which does mean we'll have to think about
sub-selections. That said, I suspect the existing {{ColumnSubSelection}} should
be up to the task, aren't they? But if it's simpler to only support top level
only at first and leave the rest to a followup, I personally won't mind.
bq. Non-frozen Set and List can be nested arbitrarily deep within the map,
although nothing can be nested within Set or List.
Sgtm.
bq. At this particular moment I'm not entirely certain about the slices of keys
for maps.
That can definitively be left to a follow-up ticket, assuming we even want to
bother with it.
> support non-frozen, nested collections
> --
>
> Key: CASSANDRA-7826
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7826
> Project: Cassandra
> Issue Type: Improvement
> Components: CQL
>Reporter: Tupshin Harper
>Assignee: Alex Petrov
> Labels: ponies
> Fix For: 3.x
>
>
> The inability to nest collections is one of the bigger data modelling
> limitations we have right now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7826) support non-frozen, nested collections
[
https://issues.apache.org/jira/browse/CASSANDRA-7826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15276186#comment-15276186
]
Alex Petrov commented on CASSANDRA-7826:
After reviewing [https://issues.apache.org/jira/browse/CASSANDRA-7396] in more
details and finding common grounds with that ticket, we need to allow
{{SELECT}}ing, {{UPDATE}}ing and {{DELETE}}ing individual fields of the nested
collections (up to the deepest nesting level for maps, none for lists and
sets), with syntax similar to 7396. Otherwise there's no big difference between
frozen and non-frozen collections.
At this particular moment I'm not entirely certain about the slices of keys for
maps. It seems that there are many more cases for fetching a particular key
from the map by path than for slices.
> support non-frozen, nested collections
> --
>
> Key: CASSANDRA-7826
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7826
> Project: Cassandra
> Issue Type: Improvement
> Components: CQL
>Reporter: Tupshin Harper
>Assignee: Alex Petrov
> Labels: ponies
> Fix For: 3.x
>
>
> The inability to nest collections is one of the bigger data modelling
> limitations we have right now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7826) support non-frozen, nested collections
[
https://issues.apache.org/jira/browse/CASSANDRA-7826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271250#comment-15271250
]
Alex Petrov commented on CASSANDRA-7826:
Would the updates and deletes for particular cells be within the scope of the
ticket?
After talking to [~thobbs] there's also the question if we would like to
support arbitrary updates or deletes right away, too. That would require new
syntax for selecting the path for update or delete (maybe something like
{{mymap['a']['b'][1]}}). Also, {{BufferCell/tombstone}} can most likely handle
deletes as they are, without slices, as we'll always know at least the partial
path, but if I understand it correctly, it will require additional logic for
discarding columns whose partial path matches the path of the tombstone.
Also, most likely [7396|https://issues.apache.org/jira/browse/CASSANDRA-7396]
will land before this patch, so it we might have to add support for selecting
individual parts on arbitrary level (or just on the top level at first and
leave the rest for the next iterations). So the question is what we'd like to
have on the current step.
Right now, arbitrary nesting of maps is fully working (also, dtests and UDFs
with nested structures are working). Non-frozen Set and List can be nested
arbitrarily deep within the map, although nothing can be nested within Set or
List. As was mentioned above.
> support non-frozen, nested collections
> --
>
> Key: CASSANDRA-7826
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7826
> Project: Cassandra
> Issue Type: Improvement
> Components: CQL
>Reporter: Tupshin Harper
>Assignee: Alex Petrov
> Labels: ponies
> Fix For: 3.x
>
>
> The inability to nest collections is one of the bigger data modelling
> limitations we have right now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7826) support non-frozen, nested collections
[ https://issues.apache.org/jira/browse/CASSANDRA-7826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271221#comment-15271221 ] Alex Petrov commented on CASSANDRA-7826: S > support non-frozen, nested collections > -- > > Key: CASSANDRA-7826 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7826 > Project: Cassandra > Issue Type: Improvement > Components: CQL >Reporter: Tupshin Harper >Assignee: Alex Petrov > Labels: ponies > Fix For: 3.x > > > The inability to nest collections is one of the bigger data modelling > limitations we have right now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7826) support non-frozen, nested collections
[ https://issues.apache.org/jira/browse/CASSANDRA-7826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15258087#comment-15258087 ] Sylvain Lebresne commented on CASSANDRA-7826: - bq. wouldn't mind supporting frozen collections as map keys / set elements We already do and I suspect this will be the case with this patch with no real additional complexity. But anyway, agreed on all counts too. > support non-frozen, nested collections > -- > > Key: CASSANDRA-7826 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7826 > Project: Cassandra > Issue Type: Improvement > Components: CQL >Reporter: Tupshin Harper >Assignee: Alex Petrov > Labels: ponies > Fix For: 3.x > > > The inability to nest collections is one of the bigger data modelling > limitations we have right now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7826) support non-frozen, nested collections
[ https://issues.apache.org/jira/browse/CASSANDRA-7826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15258033#comment-15258033 ] Jonathan Ellis commented on CASSANDRA-7826: --- bq. complex data types (set and map) can (or should) be reasonably supported only as values. This would also mean that nesting map inside of set (since set elements are serialized into the cell path) would also not be supported Agreed. (I wouldn't mind supporting *frozen* collections as map keys / set elements, but it's not a priority for me either.) > support non-frozen, nested collections > -- > > Key: CASSANDRA-7826 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7826 > Project: Cassandra > Issue Type: Improvement > Components: CQL >Reporter: Tupshin Harper >Assignee: Alex Petrov > Labels: ponies > Fix For: 3.x > > > The inability to nest collections is one of the bigger data modelling > limitations we have right now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7826) support non-frozen, nested collections
[
https://issues.apache.org/jira/browse/CASSANDRA-7826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257833#comment-15257833
]
Alex Petrov commented on CASSANDRA-7826:
We can reasonably support nesting for {{map}} and {{set}} nested within
{{map}}. Although complex data types ({{set}} and {{map}}) can (or should) be
reasonably supported only as values. This would also mean that nesting {{map}}
inside of {{set}} (since set elements are serialized into the cell path) would
also not be supported. Allowing them as a key would make the writes much more
complicated, as, for example, in order to update map that has a nested map
element as a key:
{code}
{{a: 1} {b: 2 c: 3}}
{code}
Would result into cell paths with serialised map (schematically speaking): {{
{a: 1} b}} and {{ {a: 1} c }}.
Although having nested collections as value makes most of sense, too, since it
was exactly the point of simplifying nested lookups and allowing deep updates
without re-reading a whole structure.
So far, I've made the cell multi-level and adjusted {{MapType}} and {{SetType}}
to support nested traversals, allowed multi-level {{Value}}s for both Map and
Set. I'd need to do some more adjustments to make it more presentable, but
other than that it starts showing first promises.
> support non-frozen, nested collections
> --
>
> Key: CASSANDRA-7826
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7826
> Project: Cassandra
> Issue Type: Improvement
> Components: CQL
>Reporter: Tupshin Harper
>Assignee: Alex Petrov
> Labels: ponies
> Fix For: 3.x
>
>
> The inability to nest collections is one of the bigger data modelling
> limitations we have right now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7826) support non-frozen, nested collections
[ https://issues.apache.org/jira/browse/CASSANDRA-7826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15230770#comment-15230770 ] Robert Stupp commented on CASSANDRA-7826: - bq. We'll use as many component in the CellPath as there is level of nestedness +1 - also because it feels easier to implement > support non-frozen, nested collections > -- > > Key: CASSANDRA-7826 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7826 > Project: Cassandra > Issue Type: Improvement > Components: CQL >Reporter: Tupshin Harper >Assignee: Alex Petrov > Labels: ponies > Fix For: 3.x > > > The inability to nest collections is one of the bigger data modelling > limitations we have right now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7826) support non-frozen, nested collections
[
https://issues.apache.org/jira/browse/CASSANDRA-7826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15230309#comment-15230309
]
Sylvain Lebresne commented on CASSANDRA-7826:
-
bq. {{ComplexColumnData}} is a bag of Cells, although having nested collections
would require {{ComplexColumnData}} to also contain {{ComplexColumnData}}
There is certainly multiple ways this could be implemented, but the way I think
we should do this, at least for now, is by projecting, kind of like we
implemented collections pre-3.0. We'll use as many component in the
{{CellPath}} as there is level of nestedness and there _will_ be repetition.
So for instance, to store a {{map}} like \{ 'x' : \{ 1, 2\},
'y' : \{ 3, 4 \} \}, we'll store it as 4 cells, having respectively the cell
paths: {{('x', 1)}}, {{('x', 2)}}, {{('y', 3)}} and {{('y', 4)}}.
Of course this is not the most efficient way to do this, but this way can be
done without breaking any backward compatibility and without any change to the
file format. It is not the goal of this ticket to transform the storage engine
to be document based in particular, just to add a bit of flexibility, mainly in
term of being able to insert without needing to read-before-write (like you
have to do if you use frozen collections). We'll still be clear that nesting
too much or having collections too big will be a bad idea.
> support non-frozen, nested collections
> --
>
> Key: CASSANDRA-7826
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7826
> Project: Cassandra
> Issue Type: Improvement
> Components: CQL
>Reporter: Tupshin Harper
>Assignee: Alex Petrov
> Labels: ponies
> Fix For: 3.x
>
>
> The inability to nest collections is one of the bigger data modelling
> limitations we have right now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7826) support non-frozen, nested collections
[
https://issues.apache.org/jira/browse/CASSANDRA-7826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15230244#comment-15230244
]
Alex Petrov commented on CASSANDRA-7826:
I'm doing a small research on what it may take to get this implemented. It's
clear that it'll require new syntax for nested lookups and updates, but I
wanted to address some larger things.
* {{ComplexColumnData}} is a bag of {{Cells}}, although having nested
collections would require {{ComplexColumnData}} to also contain
{{ComplexColumnData}}.
* in most of places, such as {{Operation}}, there's a single-level mapping:
retrieve a {{ColumnDefinition}}, and then build {{CellPath}} directly within
this column
* it might be good to allow nested {{CellPath}} iteration (maybe? making
{{CellPath}} multi-level, and allowing nested access)
* {{Insert}} and {{Update}} operations that don't require nested access (for
example, {{UPDATE %s SET c[0] = [5,6] WHERE a = 1 AND b = 2}} given {{c is
list}} are a much simpler, since they don't require finding a nested
cell, although nested collections are now stored as {{blobs}}, which is a main
"issue" here
* {{Operation}}s such as {{Set|DeleteField|Addition|Subtraction}} would
probably require the biggest changes, aside the mentioned {{Cell}} changes
Please correct me if I'm wrong somewhere.
> support non-frozen, nested collections
> --
>
> Key: CASSANDRA-7826
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7826
> Project: Cassandra
> Issue Type: Improvement
> Components: CQL
>Reporter: Tupshin Harper
>Assignee: Alex Petrov
> Labels: ponies
> Fix For: 3.x
>
>
> The inability to nest collections is one of the bigger data modelling
> limitations we have right now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7826) support non-frozen, nested collections
[ https://issues.apache.org/jira/browse/CASSANDRA-7826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15229866#comment-15229866 ] Robert Stupp commented on CASSANDRA-7826: - Better wait until #7396 is patch-available. > support non-frozen, nested collections > -- > > Key: CASSANDRA-7826 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7826 > Project: Cassandra > Issue Type: Improvement > Components: CQL >Reporter: Tupshin Harper >Assignee: Alex Petrov > Labels: ponies > Fix For: 3.x > > > The inability to nest collections is one of the bigger data modelling > limitations we have right now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7826) support non-frozen, nested collections
[
https://issues.apache.org/jira/browse/CASSANDRA-7826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15229021#comment-15229021
]
Alex Petrov commented on CASSANDRA-7826:
[~snazy] that's great! I'm currently gathering enough information to at least
understand what it'd take to solve it. It certainly would be required to have
accessors in order to have nested update ops, such as (based on the syntax
mentioned in [#7396|https://issues.apache.org/jira/browse/CASSANDRA-7396]
{{UPDATE tbl SET a = a[0] + [ 'a', 'b' ] WHERE k = 0}} (in case with list
nested within the list) and so on.
Do you think it's a good idea to take a closer look at / maybe comment on the
branch? Or is it better to wait for a final version?
> support non-frozen, nested collections
> --
>
> Key: CASSANDRA-7826
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7826
> Project: Cassandra
> Issue Type: Improvement
> Components: CQL
>Reporter: Tupshin Harper
>Assignee: Alex Petrov
> Labels: ponies
> Fix For: 3.x
>
>
> The inability to nest collections is one of the bigger data modelling
> limitations we have right now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7826) support non-frozen, nested collections
[ https://issues.apache.org/jira/browse/CASSANDRA-7826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15228073#comment-15228073 ] Robert Stupp commented on CASSANDRA-7826: - FYI, I'm currently working on CASSANDRA-7396 and should have a patch ready in a couple of days. > support non-frozen, nested collections > -- > > Key: CASSANDRA-7826 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7826 > Project: Cassandra > Issue Type: Improvement > Components: CQL >Reporter: Tupshin Harper >Assignee: Alex Petrov > Labels: ponies > Fix For: 3.x > > > The inability to nest collections is one of the bigger data modelling > limitations we have right now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7826) support non-frozen, nested collections
[ https://issues.apache.org/jira/browse/CASSANDRA-7826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230526#comment-14230526 ] Philip Thompson commented on CASSANDRA-7826: [~thobbs] Will users be able to use secondary indexes on collections beyond the root collection? support non-frozen, nested collections -- Key: CASSANDRA-7826 URL: https://issues.apache.org/jira/browse/CASSANDRA-7826 Project: Cassandra Issue Type: Improvement Components: API, Core Reporter: Tupshin Harper Assignee: Tyler Hobbs Labels: ponies Fix For: 3.0 The inability to nest collections is one of the bigger data modelling limitations we have right now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7826) support non-frozen, nested collections
[
https://issues.apache.org/jira/browse/CASSANDRA-7826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230549#comment-14230549
]
Philip Thompson commented on CASSANDRA-7826:
Additionally, how will secondary indexes work on the root collections with the
nested collections?
If I have a list of lists of integers that looks like: {{[ [1,2,3], [4,5,6]
]}}, would only {{CONTAINS [1,2,3]}} or {{CONTAINS [4,5,6]}} return true?
support non-frozen, nested collections
--
Key: CASSANDRA-7826
URL: https://issues.apache.org/jira/browse/CASSANDRA-7826
Project: Cassandra
Issue Type: Improvement
Components: API, Core
Reporter: Tupshin Harper
Assignee: Tyler Hobbs
Labels: ponies
Fix For: 3.0
The inability to nest collections is one of the bigger data modelling
limitations we have right now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7826) support non-frozen, nested collections
[ https://issues.apache.org/jira/browse/CASSANDRA-7826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14206985#comment-14206985 ] Tyler Hobbs commented on CASSANDRA-7826: bq. Will this support arbitrary nesting? If not, to what depth will we be able to nest collections? Yes, up to the cell name size limit. bq. we will be able to update and retrieve individual elements from the root collection and any nested collections, correct? Yes, that's the goal. support non-frozen, nested collections -- Key: CASSANDRA-7826 URL: https://issues.apache.org/jira/browse/CASSANDRA-7826 Project: Cassandra Issue Type: Improvement Components: API, Core Reporter: Tupshin Harper Assignee: Tyler Hobbs Labels: ponies Fix For: 3.0 The inability to nest collections is one of the bigger data modelling limitations we have right now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7826) support non-frozen, nested collections
[ https://issues.apache.org/jira/browse/CASSANDRA-7826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205111#comment-14205111 ] Philip Thompson commented on CASSANDRA-7826: Will this support arbitrary nesting? If not, to what depth will we be able to nest collections? Given CASSANDRA-7396 and CASSANDRA-7423, and [~thobbs] earlier comments, we will be able to update and retrieve individual elements from the root collection and any nested collections, correct? support non-frozen, nested collections -- Key: CASSANDRA-7826 URL: https://issues.apache.org/jira/browse/CASSANDRA-7826 Project: Cassandra Issue Type: Improvement Components: API, Core Reporter: Tupshin Harper Assignee: Tyler Hobbs Labels: ponies Fix For: 3.0 The inability to nest collections is one of the bigger data modelling limitations we have right now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
