[jira] [Commented] (CASSANDRA-15413) Missing results on reading large frozen text map
[
https://issues.apache.org/jira/browse/CASSANDRA-15413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981478#comment-16981478
]
Alex Petrov commented on CASSANDRA-15413:
-
I wrote a simple in-jvm dtest that reproduced it in all versions (2.1, 2.2,
3.0, 3.11, and trunk) last week and have started working on the patch. I'll
post it here as soon as it's ready, unless [~spodi] would like to post a full
patch.
> Missing results on reading large frozen text map
>
>
> Key: CASSANDRA-15413
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15413
> Project: Cassandra
> Issue Type: Bug
> Components: Local/SSTable
>Reporter: Tyler Codispoti
>Assignee: Alex Petrov
>Priority: Normal
>
> Cassandra version: 2.2.15
> I have been running into a case where, when fetching the results from a table
> with a frozen>, if the number of results is greater than the
> fetch size (default 5000), we can end up with missing data.
> Side note: The table schema comes from using KairosDB, but we've isolated
> this issue to Cassandra itself. But it looks like this can cause problems for
> users of KairosDB as well.
> Repro case. Tested against fresh install of Cassandra 2.2.15.
> 1. Create table (csqlsh)
> {code:sql}
> CREATE KEYSPACE test
> WITH REPLICATION = {
>'class' : 'SimpleStrategy',
>'replication_factor' : 1
> };
> CREATE TABLE test.test (
> name text,
> tags frozen>,
> PRIMARY KEY (name, tags)
> ) WITH CLUSTERING ORDER BY (tags ASC);
> {code}
> 2. Insert data (python3)
> {code:python}
> import time
> from cassandra.cluster import Cluster
> cluster = Cluster(['127.0.0.1'])
> session = cluster.connect('test')
> for i in range(0, 2):
> session.execute(
> """
> INSERT INTO test (name, tags)
> VALUES (%s, %s)
> """,
> ("test_name", {'id':str(i)})
> )
> {code}
>
> 3. Flush
>
> {code:java}
> nodetools flush{code}
>
>
> 4. Fetch data (python3)
> {code:python}
> import time
> from cassandra.cluster import Cluster
> cluster = Cluster(['127.0.0.1'], control_connection_timeout=5000)
> session = cluster.connect('test')
> session.default_fetch_size = 5000
> session.default_timeout = 120
> count = 0
> rows = session.execute("select tags from test where name='test_name'")
> for row in rows:
> count += 1
> print(count)
> {code}
> Result: 10111 (expected 2)
>
> Changing the page size changes the result count. Some quick samples:
>
> ||default_fetch_size||count||
> |5000|10111|
> |1000|1830|
> |999|1840|
> |998|1850|
> |2|2|
> |10|2|
>
>
> In short, I cannot guarantee I'll get all the results back unless the page
> size > number of rows.
> This seems to get worse with multiple SSTables (eg nodetool flush between
> some of the insert batches). When using replication, the issue can get
> disgustingly bad - potentially giving a different result on each query.
> Interesting, if we pad the values on the tag map ("id" in this repro case) so
> that the insertion is in lexicographical order, there is no issue. I believe
> the issue also does not repro if I do not call "nodetools flush" before
> querying.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15413) Missing results on reading large frozen text map
[
https://issues.apache.org/jira/browse/CASSANDRA-15413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981373#comment-16981373
]
Sicheng Liu commented on CASSANDRA-15413:
-
Any update on this issue? This seems to be a very serious issue since the
functionality broken here is really fundamental and could affect a lot of
users.
> Missing results on reading large frozen text map
>
>
> Key: CASSANDRA-15413
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15413
> Project: Cassandra
> Issue Type: Bug
> Components: Local/SSTable
>Reporter: Tyler Codispoti
>Assignee: Alex Petrov
>Priority: Normal
>
> Cassandra version: 2.2.15
> I have been running into a case where, when fetching the results from a table
> with a frozen>, if the number of results is greater than the
> fetch size (default 5000), we can end up with missing data.
> Side note: The table schema comes from using KairosDB, but we've isolated
> this issue to Cassandra itself. But it looks like this can cause problems for
> users of KairosDB as well.
> Repro case. Tested against fresh install of Cassandra 2.2.15.
> 1. Create table (csqlsh)
> {code:sql}
> CREATE KEYSPACE test
> WITH REPLICATION = {
>'class' : 'SimpleStrategy',
>'replication_factor' : 1
> };
> CREATE TABLE test.test (
> name text,
> tags frozen>,
> PRIMARY KEY (name, tags)
> ) WITH CLUSTERING ORDER BY (tags ASC);
> {code}
> 2. Insert data (python3)
> {code:python}
> import time
> from cassandra.cluster import Cluster
> cluster = Cluster(['127.0.0.1'])
> session = cluster.connect('test')
> for i in range(0, 2):
> session.execute(
> """
> INSERT INTO test (name, tags)
> VALUES (%s, %s)
> """,
> ("test_name", {'id':str(i)})
> )
> {code}
>
> 3. Flush
>
> {code:java}
> nodetools flush{code}
>
>
> 4. Fetch data (python3)
> {code:python}
> import time
> from cassandra.cluster import Cluster
> cluster = Cluster(['127.0.0.1'], control_connection_timeout=5000)
> session = cluster.connect('test')
> session.default_fetch_size = 5000
> session.default_timeout = 120
> count = 0
> rows = session.execute("select tags from test where name='test_name'")
> for row in rows:
> count += 1
> print(count)
> {code}
> Result: 10111 (expected 2)
>
> Changing the page size changes the result count. Some quick samples:
>
> ||default_fetch_size||count||
> |5000|10111|
> |1000|1830|
> |999|1840|
> |998|1850|
> |2|2|
> |10|2|
>
>
> In short, I cannot guarantee I'll get all the results back unless the page
> size > number of rows.
> This seems to get worse with multiple SSTables (eg nodetool flush between
> some of the insert batches). When using replication, the issue can get
> disgustingly bad - potentially giving a different result on each query.
> Interesting, if we pad the values on the tag map ("id" in this repro case) so
> that the insertion is in lexicographical order, there is no issue. I believe
> the issue also does not repro if I do not call "nodetools flush" before
> querying.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15413) Missing results on reading large frozen text map
[
https://issues.apache.org/jira/browse/CASSANDRA-15413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976805#comment-16976805
]
Tyler Codispoti commented on CASSANDRA-15413:
-
As a temporary workaround, we made a change to compareNextTo() in
AbstractCompoundCellNameType to force using the BytesType comparator for this
column. We did more changes to ensure we don't mess with any other columns, but
essentially, the change boils down to:
{code:java}
ByteBuffer previous = null;
for (int i = 0; i < composite.size(); i++)
{
if (!hasComponent(i))
return nextEOC == Composite.EOC.END ? 1 : -1;
AbstractType comparator = type.subtype(i);
ByteBuffer value1 = nextComponents[i];
ByteBuffer value2 = composite.get(i);
// For frozen map, do not compare each key/value. Compare the
whole serilized binary
// as what it did when writing to sstables.
if (comparator instanceof MapType) {
comparator = BytesType.instance;
}
int cmp = comparator.compareCollectionMembers(value1, value2,
previous);
if (cmp != 0)
return cmp;
previous = value1;
}
{code}
This looks to resolve the issue. It seems what is happening is that, when
reading Map, it will compare when reading back using lexicographic
order, but it is stored in binary order. When reading data back for pages after
the 1st, it'll compare the last value of the previous page to the current page
to see if any records can be skipped. Since the sort comparator is of the wrong
type, you can easily end up in a state where it skips records in correctly.
> Missing results on reading large frozen text map
>
>
> Key: CASSANDRA-15413
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15413
> Project: Cassandra
> Issue Type: Bug
> Components: Local/SSTable
>Reporter: Tyler Codispoti
>Assignee: Alex Petrov
>Priority: Normal
>
> Cassandra version: 2.2.15
> I have been running into a case where, when fetching the results from a table
> with a frozen>, if the number of results is greater than the
> fetch size (default 5000), we can end up with missing data.
> Side note: The table schema comes from using KairosDB, but we've isolated
> this issue to Cassandra itself. But it looks like this can cause problems for
> users of KairosDB as well.
> Repro case. Tested against fresh install of Cassandra 2.2.15.
> 1. Create table (csqlsh)
> {code:sql}
> CREATE KEYSPACE test
> WITH REPLICATION = {
>'class' : 'SimpleStrategy',
>'replication_factor' : 1
> };
> CREATE TABLE test.test (
> name text,
> tags frozen>,
> PRIMARY KEY (name, tags)
> ) WITH CLUSTERING ORDER BY (tags ASC);
> {code}
> 2. Insert data (python3)
> {code:python}
> import time
> from cassandra.cluster import Cluster
> cluster = Cluster(['127.0.0.1'])
> session = cluster.connect('test')
> for i in range(0, 2):
> session.execute(
> """
> INSERT INTO test (name, tags)
> VALUES (%s, %s)
> """,
> ("test_name", {'id':str(i)})
> )
> {code}
>
> 3. Flush
>
> {code:java}
> nodetools flush{code}
>
>
> 4. Fetch data (python3)
> {code:python}
> import time
> from cassandra.cluster import Cluster
> cluster = Cluster(['127.0.0.1'], control_connection_timeout=5000)
> session = cluster.connect('test')
> session.default_fetch_size = 5000
> session.default_timeout = 120
> count = 0
> rows = session.execute("select tags from test where name='test_name'")
> for row in rows:
> count += 1
> print(count)
> {code}
> Result: 10111 (expected 2)
>
> Changing the page size changes the result count. Some quick samples:
>
> ||default_fetch_size||count||
> |5000|10111|
> |1000|1830|
> |999|1840|
> |998|1850|
> |2|2|
> |10|2|
>
>
> In short, I cannot guarantee I'll get all the results back unless the page
> size > number of rows.
> This seems to get worse with multiple SSTables (eg nodetool flush between
> some of the insert batches). When using replication, the issue can get
> disgustingly bad - potentially giving a different result on each query.
> Interesting, if we pad the values on the tag map ("id" in this repro case) so
> that the insertion is in lexicographical order, there is no issue. I believe
> the issue also does not repro if I do not call "nodetools flush" before
> querying.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail:
