[jira] [Commented] (CASSANDRA-15064) Wrong ordering for timeuuid fields
[ https://issues.apache.org/jira/browse/CASSANDRA-15064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806602#comment-16806602 ] Andreas Andersen commented on CASSANDRA-15064: -- As for our use case we can simply use the gocql.TimeUUIDWith function ([https://github.com/gocql/gocql/blob/master/uuid.go#L132]) and supply our own clock sequence. We will never have more than 2^14 images created on a single second. When it comes to the cassandra part, simply changing the underlying field from a timeuuid to a uuid will solve all of our problems. It sorts in the way we expect it to. Having the behavior for timeuuid fields mentioned in the cassandra documentation would have been sufficient for solving our problems - we would simply have gone with the uuid datatype from the start. So simply adding more information to the docs seems like a good solution to this issue :) > Wrong ordering for timeuuid fields > -- > > Key: CASSANDRA-15064 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15064 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: Andreas Andersen >Assignee: Jon Meredith >Priority: Normal > Attachments: example.cql > > > Hi! > We're seeing some strange behavior for the ordering of timeuuid fields. They > seem to be sorted in the wrong order when the clock_seq_low field in a > timeuuid goes from 7f to 80. Consider the following example: > {noformat} > cqlsh:test> show version; > [cqlsh 5.0.1 | Cassandra 3.11.4 | CQL spec 3.4.4 | Native protocol v4] > cqlsh:test> CREATE TABLE t ( > ... partition int, > ... t timeuuid, > ... i int, > ... > ... PRIMARY KEY(partition, t) > ... ) > ... WITH CLUSTERING ORDER BY(t ASC); > cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1, > 84e2c963-4ef9-11e9-b57e-f0def1d0755e, 1); > cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1, > 84e2c963-4ef9-11e9-b57f-f0def1d0755e, 2); > cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1, > 84e2c963-4ef9-11e9-b580-f0def1d0755e, 3); > cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1, > 84e2c963-4ef9-11e9-b581-f0def1d0755e, 4); > cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1, > 84e2c963-4ef9-11e9-b582-f0def1d0755e, 5); > cqlsh:test> SELECT * FROM t WHERE partition = 1 ORDER BY t ASC; > > partition | t | i > ---+--+--- > 1 | 84e2c963-4ef9-11e9-b580-f0def1d0755e | 3 > 1 | 84e2c963-4ef9-11e9-b581-f0def1d0755e | 4 > 1 | 84e2c963-4ef9-11e9-b582-f0def1d0755e | 5 > 1 | 84e2c963-4ef9-11e9-b57e-f0def1d0755e | 1 > 1 | 84e2c963-4ef9-11e9-b57f-f0def1d0755e | 2 > > (5 rows) > cqlsh:test> > {noformat} > The expected behavior is that the rows are returned in the same order as they > were inserted (we inserted them with their clustering key in an ascending > order). Instead, the order "wraps" in the middle. > This issue only arises when the 9th octet (clock_seq_low) in the uuid goes > from 7f to 80. A guess would be that the comparison is implemented as a > signed integer instead of an unsigned integer, as 0x7f = 127 and 0x80 = -128. > According to the RFC, the field should be treated as an unsigned integer: > [https://tools.ietf.org/html/rfc4122#section-4.1.2] > Changing the field from a timeuuid to a uuid gives the expected correct > behavior: > {noformat} > cqlsh:test> CREATE TABLE t ( > ... partition int, > ... t uuid, > ... i int, > ... > ... PRIMARY KEY(partition, t) > ... ) > ... WITH CLUSTERING ORDER BY(t ASC); > cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1, > 84e2c963-4ef9-11e9-b57e-f0def1d0755e, 1); > cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1, > 84e2c963-4ef9-11e9-b57f-f0def1d0755e, 2); > cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1, > 84e2c963-4ef9-11e9-b580-f0def1d0755e, 3); > cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1, > 84e2c963-4ef9-11e9-b581-f0def1d0755e, 4); > cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1, > 84e2c963-4ef9-11e9-b582-f0def1d0755e, 5); > cqlsh:test> SELECT * FROM t WHERE partition = 1 ORDER BY t ASC; > > partition | t | i > ---+--+--- > 1 | 84e2c963-4ef9-11e9-b57e-f0def1d0755e | 1 > 1 | 84e2c963-4ef9-11e9-b57f-f0def1d0755e | 2 > 1 | 84e2c963-4ef9-11e9-b580-f0def1d0755e | 3 > 1 | 84e2c963-4ef9-11e9-b581-f0def1d0755e | 4 > 1 | 84e2c963-4ef9-11e9-b582-f0def1d0755e | 5 > > (5 rows) > cqlsh:test>{noformat} > > --
[jira] [Commented] (CASSANDRA-15064) Wrong ordering for timeuuid fields
[ https://issues.apache.org/jira/browse/CASSANDRA-15064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805425#comment-16805425 ] Jon Meredith commented on CASSANDRA-15064: -- Thinking a little more on this, if you went with using server-side now() you'd need to ensure the same node was used to coordinate all of the INSERTs and I'm not sure what the default behavior of the Go client is, so perhaps doing something client side would be best. > Wrong ordering for timeuuid fields > -- > > Key: CASSANDRA-15064 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15064 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: Andreas Andersen >Assignee: Jon Meredith >Priority: Normal > Attachments: example.cql > > > Hi! > We're seeing some strange behavior for the ordering of timeuuid fields. They > seem to be sorted in the wrong order when the clock_seq_low field in a > timeuuid goes from 7f to 80. Consider the following example: > {noformat} > cqlsh:test> show version; > [cqlsh 5.0.1 | Cassandra 3.11.4 | CQL spec 3.4.4 | Native protocol v4] > cqlsh:test> CREATE TABLE t ( > ... partition int, > ... t timeuuid, > ... i int, > ... > ... PRIMARY KEY(partition, t) > ... ) > ... WITH CLUSTERING ORDER BY(t ASC); > cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1, > 84e2c963-4ef9-11e9-b57e-f0def1d0755e, 1); > cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1, > 84e2c963-4ef9-11e9-b57f-f0def1d0755e, 2); > cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1, > 84e2c963-4ef9-11e9-b580-f0def1d0755e, 3); > cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1, > 84e2c963-4ef9-11e9-b581-f0def1d0755e, 4); > cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1, > 84e2c963-4ef9-11e9-b582-f0def1d0755e, 5); > cqlsh:test> SELECT * FROM t WHERE partition = 1 ORDER BY t ASC; > > partition | t | i > ---+--+--- > 1 | 84e2c963-4ef9-11e9-b580-f0def1d0755e | 3 > 1 | 84e2c963-4ef9-11e9-b581-f0def1d0755e | 4 > 1 | 84e2c963-4ef9-11e9-b582-f0def1d0755e | 5 > 1 | 84e2c963-4ef9-11e9-b57e-f0def1d0755e | 1 > 1 | 84e2c963-4ef9-11e9-b57f-f0def1d0755e | 2 > > (5 rows) > cqlsh:test> > {noformat} > The expected behavior is that the rows are returned in the same order as they > were inserted (we inserted them with their clustering key in an ascending > order). Instead, the order "wraps" in the middle. > This issue only arises when the 9th octet (clock_seq_low) in the uuid goes > from 7f to 80. A guess would be that the comparison is implemented as a > signed integer instead of an unsigned integer, as 0x7f = 127 and 0x80 = -128. > According to the RFC, the field should be treated as an unsigned integer: > [https://tools.ietf.org/html/rfc4122#section-4.1.2] > Changing the field from a timeuuid to a uuid gives the expected correct > behavior: > {noformat} > cqlsh:test> CREATE TABLE t ( > ... partition int, > ... t uuid, > ... i int, > ... > ... PRIMARY KEY(partition, t) > ... ) > ... WITH CLUSTERING ORDER BY(t ASC); > cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1, > 84e2c963-4ef9-11e9-b57e-f0def1d0755e, 1); > cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1, > 84e2c963-4ef9-11e9-b57f-f0def1d0755e, 2); > cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1, > 84e2c963-4ef9-11e9-b580-f0def1d0755e, 3); > cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1, > 84e2c963-4ef9-11e9-b581-f0def1d0755e, 4); > cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1, > 84e2c963-4ef9-11e9-b582-f0def1d0755e, 5); > cqlsh:test> SELECT * FROM t WHERE partition = 1 ORDER BY t ASC; > > partition | t | i > ---+--+--- > 1 | 84e2c963-4ef9-11e9-b57e-f0def1d0755e | 1 > 1 | 84e2c963-4ef9-11e9-b57f-f0def1d0755e | 2 > 1 | 84e2c963-4ef9-11e9-b580-f0def1d0755e | 3 > 1 | 84e2c963-4ef9-11e9-b581-f0def1d0755e | 4 > 1 | 84e2c963-4ef9-11e9-b582-f0def1d0755e | 5 > > (5 rows) > cqlsh:test>{noformat} > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15064) Wrong ordering for timeuuid fields
[ https://issues.apache.org/jira/browse/CASSANDRA-15064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805069#comment-16805069 ] Jon Meredith commented on CASSANDRA-15064: -- Thanks for the additional information. I'm not familiar with the Go client so thanks for the link. The Go client takes a different approach to UUID generation than the server-side now() function, which as I mentioned above, generates the clockseq once at startup time - which limits now() to generating a single UUID per 100ns (which is 10 million / second). I agree with you you'll get a wrapping problem with your UUID solution. It just takes the lower 14 bits from the 32-bit clockseq counter which obviously wraps. What do you think about these two possibilities # Insert using CQL now() - probably easiest # Write an alternate UUIDFromTime *Insert Using CQL now()* {noformat} INSERT INTO timeline(user_id, image_timestamp, image_id) VALUES(?, now(), ?) {noformat} That will generate timestamps that always sort as you expect, however you'll need an average insertion rate less than 10,000,000 / second / coordinator and move the cost of UUID generation from client to server which should be small. *Write an alternate UUIDFromTime to match server-side now()* It should be straightforward to provide an alternative UUIDFromTime with the same semantics as CQL now(). Initialize clockseq once at startup Keep an atomic 64-bit, lastNanos, to store the 60-bit UUID timestamp Atomically update to be monontonically increasing. Something like this (apologies for the pigeon Go, I don't normally write it and have not tested this fragment) {noformat} var currentNanos = atomic.LoadInt64() // compute nextNanos the same way as https://github.com/gocql/gocql/blame/master/uuid.go#L126 var nextNanos = int64(utcTime.Unix()-timeBase)*1000 + int64(utcTime.Nanosecond()/100) if (thisNanos <= currentNanos) { thisNanos = currentNanos + 1; // If the clock is drifting backwards, just pick the monontonic next timestamp } if(!atomic.CompareAndSwapUInt64(, currentNanos, nextNanos)) { // If unable to swap, another thread has generated a new UUID, so the time should be close // to when this thread would have, just take the next timestamp rather than loop forever // in case of high contention. thisNanos = atomic.AddInt64(, 1) } {noformat} What do you think? As for this JIRA, what I'd propose is that I'll submit documentation improvements so that users don't rely on TimeUUID clockseq sort order, and file an issue under the GoCql client driver in case they missed this JIRA. > Wrong ordering for timeuuid fields > -- > > Key: CASSANDRA-15064 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15064 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: Andreas Andersen >Assignee: Jon Meredith >Priority: Normal > Attachments: example.cql > > > Hi! > We're seeing some strange behavior for the ordering of timeuuid fields. They > seem to be sorted in the wrong order when the clock_seq_low field in a > timeuuid goes from 7f to 80. Consider the following example: > {noformat} > cqlsh:test> show version; > [cqlsh 5.0.1 | Cassandra 3.11.4 | CQL spec 3.4.4 | Native protocol v4] > cqlsh:test> CREATE TABLE t ( > ... partition int, > ... t timeuuid, > ... i int, > ... > ... PRIMARY KEY(partition, t) > ... ) > ... WITH CLUSTERING ORDER BY(t ASC); > cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1, > 84e2c963-4ef9-11e9-b57e-f0def1d0755e, 1); > cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1, > 84e2c963-4ef9-11e9-b57f-f0def1d0755e, 2); > cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1, > 84e2c963-4ef9-11e9-b580-f0def1d0755e, 3); > cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1, > 84e2c963-4ef9-11e9-b581-f0def1d0755e, 4); > cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1, > 84e2c963-4ef9-11e9-b582-f0def1d0755e, 5); > cqlsh:test> SELECT * FROM t WHERE partition = 1 ORDER BY t ASC; > > partition | t | i > ---+--+--- > 1 | 84e2c963-4ef9-11e9-b580-f0def1d0755e | 3 > 1 | 84e2c963-4ef9-11e9-b581-f0def1d0755e | 4 > 1 | 84e2c963-4ef9-11e9-b582-f0def1d0755e | 5 > 1 | 84e2c963-4ef9-11e9-b57e-f0def1d0755e | 1 > 1 | 84e2c963-4ef9-11e9-b57f-f0def1d0755e | 2 > > (5 rows) > cqlsh:test> > {noformat} > The expected behavior is that the rows are returned in the same order as they > were inserted (we inserted them with their clustering key in an ascending > order). Instead, the order "wraps" in the middle. >
[jira] [Commented] (CASSANDRA-15064) Wrong ordering for timeuuid fields
[ https://issues.apache.org/jira/browse/CASSANDRA-15064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804990#comment-16804990 ] Andreas Andersen commented on CASSANDRA-15064: -- The problem is a real world problem for us, and the timeuuid's in the example are not synthetic. We use the gocql client library for golang, which generates timeuuid's in this way. It's the function at https://github.com/gocql/gocql/blob/master/uuid.go#L121 we use to generate our timeuuid's from timestamps, and we rely on the sequence number to be incremented on each call. Our use case is sorting images on a timeline. Our table more or less look like: {noformat} CREATE TABLE timeline ( user_id uuid, image_timestamp timeuuid, image_id uuid, -- Lots of other metadata fields here... PRIMARY KEY(user_id, image_timestamp) ) WITH CLUSTERING ORDER BY(t ASC); {noformat} The images are sorted by the time that they were taken. However, images taken quickly in series by the same camera might all have equal timestamps. In this case, we want to order them by some other property to show them in the correct chronological order. As of today we use the filename for this. In order to keep the table structure simple, we just sort them in the correct order before inserting them, relying on each newly generated timeuuid being greater than the last one. Our client side code more or less look like: {noformat} type image struct { user_id uuid image_iduuid taken_attimestamp filenamestring } images := []image // Sort images in the order we want to store them in cassandra sort(images, [taken_at, filename]) // Now give every image a unique timeuuid and insert them into cassandra. We // rely on the timeuuid generator to generate id's in ascending order in case of // multiple images with the same timestamp. for i := range images { image_timestamp := uuidFromTime(i.taken_at) queryCassandra( "INSERT INTO timeline(user_id, image_timestamp, image_id) VALUES(?, ?, ?)", i.user_id, image_timestamp, i.image_id ) } {noformat} Our use case is kind of special, since we at the time of insertion have all images that will ever be created for a single user, which allows us to insert them in this way. Creating a table in the following way would give us the same result: {noformat} CREATE TABLE timeline ( user_id uuid, image_timestamp timestamp, sort_number int, image_id uuid, -- Lots of other metadata fields here... PRIMARY KEY(user_id, image_timestamp, sort_number) ) WITH CLUSTERING ORDER BY(t ASC); {noformat} In practicality this is not really a big issue for us. We can just change the column type from timeuuid to a uuid, and everything will behave the way that we expect. While reading the code for gocql.UUIDFromTime() I also realize the our implementation is broken, as the sequence number will wrap at times, since it's only 14 bits long :) > Wrong ordering for timeuuid fields > -- > > Key: CASSANDRA-15064 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15064 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: Andreas Andersen >Assignee: Jon Meredith >Priority: Normal > Attachments: example.cql > > > Hi! > We're seeing some strange behavior for the ordering of timeuuid fields. They > seem to be sorted in the wrong order when the clock_seq_low field in a > timeuuid goes from 7f to 80. Consider the following example: > {noformat} > cqlsh:test> show version; > [cqlsh 5.0.1 | Cassandra 3.11.4 | CQL spec 3.4.4 | Native protocol v4] > cqlsh:test> CREATE TABLE t ( > ... partition int, > ... t timeuuid, > ... i int, > ... > ... PRIMARY KEY(partition, t) > ... ) > ... WITH CLUSTERING ORDER BY(t ASC); > cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1, > 84e2c963-4ef9-11e9-b57e-f0def1d0755e, 1); > cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1, > 84e2c963-4ef9-11e9-b57f-f0def1d0755e, 2); > cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1, > 84e2c963-4ef9-11e9-b580-f0def1d0755e, 3); > cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1, > 84e2c963-4ef9-11e9-b581-f0def1d0755e, 4); > cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1, > 84e2c963-4ef9-11e9-b582-f0def1d0755e, 5); > cqlsh:test> SELECT * FROM t WHERE partition = 1 ORDER BY t ASC; > > partition | t | i > ---+--+--- > 1 | 84e2c963-4ef9-11e9-b580-f0def1d0755e | 3 > 1 | 84e2c963-4ef9-11e9-b581-f0def1d0755e | 4 > 1 |
[jira] [Commented] (CASSANDRA-15064) Wrong ordering for timeuuid fields
[ https://issues.apache.org/jira/browse/CASSANDRA-15064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16802209#comment-16802209 ] Jon Meredith commented on CASSANDRA-15064: -- I can confirm the behavior you're seeing, however I don't think it's something that should be changed. Cassandra uses the ClockSeq in the manner described in section 4.1.5 as a mechanism to avoid generating duplicate UUIDs if the server is restarted witht the clock in the past, or comes up with a different node id [1]. As an aside, if the time reported by the OS via System.currentTimeMillis goes backwards while the server is running, the *timestamp* (not clock seq) will be constrained to be 100ns later than the previous timestamp returned until time catches up (or Cassandra is restarted). The node id and clock sequence are initialized on server startup and should not change during execution, so I was wondering if you generated them synthetically? Here are the UUIDs you supplied broken into components by java.util.UUID {noformat} 1 UUID 84e2c963-4ef9-11e9-b57e-f0def1d0755e = variant[0b10] version[0b1] timestamp[0x01e94ef984e2c963] node[0xf0def1d0755e] clockseq[0x357e] 2 UUID 84e2c963-4ef9-11e9-b57f-f0def1d0755e = variant[0b10] version[0b1] timestamp[0x01e94ef984e2c963] node[0xf0def1d0755e] clockseq[0x357f] 3 UUID 84e2c963-4ef9-11e9-b580-f0def1d0755e = variant[0b10] version[0b1] timestamp[0x01e94ef984e2c963] node[0xf0def1d0755e] clockseq[0x3580] 4 UUID 84e2c963-4ef9-11e9-b581-f0def1d0755e = variant[0b10] version[0b1] timestamp[0x01e94ef984e2c963] node[0xf0def1d0755e] clockseq[0x3581] 5 UUID 84e2c963-4ef9-11e9-b582-f0def1d0755e = variant[0b10] version[0b1] timestamp[0x01e94ef984e2c963] node[0xf0def1d0755e] clockseq[0x3582] {noformat} The reason the UUID [3] and TimeUUID [4] types compare differently for the clock sequence (and node) is that they have slightly different comparison functions. The TimeUUID treats clk_seq_hi as a signed byte and then clk_seq_low/node as a series of unsigned bytes (not a single integer), however the UUIDType comparison just treats it as a single signed long. It looks like this behavior was preserved during the performance refactor in CASSANDRA-8730. As for the correct ordering, although the spec does say to treat the clock sequence as an unsigned integer, it doesn't really convey any meaning as it is assigned from a random number generator. (FWIW, The RFC also recommends lexical ordering by time_low, time_mid, then time_hi which seemed odd to me but is confirmed by reading the uuid_compare function in Appendix A). For both TimeUUID / UUID the primary ordering is by 100-nsec resolution timestamp first which is what we want when dealing with timestamps, then deterministically on the node/clockseq (although unfortunately in a different order depending on TimeUUID/UUID). Changing the comparison function affects the order data is stored on disk and would require users to perform some kind of migration (automated or manual), and although it would be nice to make them both the same, and perhaps change the clockseq to be treated as an unsigned int to reduce surprise I don't think the value is there to make any change. Just in case I've missed the point, would you mind answering a couple of questions # How did you notice this issue, was it with synthetic tests or through production usage? # Why does it matter to you? Are you trying to integrate with an external system that expects the TimeUUIDs to be ordered in a specific way? [1] node id for Cassandra is generated from a hash of local IP addresses as the comments claim the MAC address was not easily accessible from Java. I'm not sure if the order of the hashed addresses is stable so it's possible the same node with same networking could come up with a different node. https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/utils/UUIDGen.java#L349 The method has been changed for trunk, but assuming you're not running it. [2] Initializing clock seq https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/utils/UUIDGen.java#L295 [3] UUID comparison https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/db/marshal/UUIDType.java#L90 [4] TimeUUID comparison https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/db/marshal/TimeUUIDType.java#L44 > Wrong ordering for timeuuid fields > -- > > Key: CASSANDRA-15064 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15064 > Project: Cassandra > Issue Type: Bug >Reporter: Andreas Andersen >Assignee: Jon Meredith >Priority: Normal > Attachments: example.cql > > > Hi! > We're seeing some strange behavior for the ordering of