[jira] [Commented] (CASSANDRA-12453) AutoSavingCache does not store required keys making RowCacheTests Flaky
[
https://issues.apache.org/jira/browse/CASSANDRA-12453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15793601#comment-15793601
]
Jay Zhuang commented on CASSANDRA-12453:
Thanks for the review.
Updated the patch.
> AutoSavingCache does not store required keys making RowCacheTests Flaky
> ---
>
> Key: CASSANDRA-12453
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12453
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: sankalp kohli
>Assignee: Jay Zhuang
>Priority: Minor
> Fix For: 2.2.x, 3.0.x
>
> Attachments: 12453-2.2-update.diff, 12453-2.2.txt
>
>
> RowCacheTests were flaky and while investigating, I found that it does not
> store all the keys to disk.
> The reason is that we use OHCache and call hotKeyIterator on it. This is not
> guaranteed to return the number of keys we want. Here is the documentation
> from OHCache
> /**
> * Builds an iterator over the N most recently used keys returning
> deserialized objects.
> * You must call {@code close()} on the returned iterator.
> *
> * Note: During a rehash, the implementation might return keys twice
> or not at all.
> *
> */
> CloseableIterator hotKeyIterator(int n);
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12453) AutoSavingCache does not store required keys making RowCacheTests Flaky
[
https://issues.apache.org/jira/browse/CASSANDRA-12453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15789506#comment-15789506
]
Robert Stupp commented on CASSANDRA-12453:
--
Yes, that's related to the number of segments.
I think the easiest way to make the test consistent is to force OHC to use 16
segments. Adding the line
{code}
System.setProperty("org.caffinitas.ohc.segmentCount", "16");
{code}
as the first statement in {{RowCacheTest.defineSchema()}} seems to solve the
issue. Replacing 16 with 64 makes the utest fail. I think that makes the utest
consistent. WDYT about this approach?
> AutoSavingCache does not store required keys making RowCacheTests Flaky
> ---
>
> Key: CASSANDRA-12453
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12453
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: sankalp kohli
>Assignee: Jay Zhuang
>Priority: Minor
> Fix For: 2.2.x, 3.0.x
>
> Attachments: 12453-2.2.txt
>
>
> RowCacheTests were flaky and while investigating, I found that it does not
> store all the keys to disk.
> The reason is that we use OHCache and call hotKeyIterator on it. This is not
> guaranteed to return the number of keys we want. Here is the documentation
> from OHCache
> /**
> * Builds an iterator over the N most recently used keys returning
> deserialized objects.
> * You must call {@code close()} on the returned iterator.
> *
> * Note: During a rehash, the implementation might return keys twice
> or not at all.
> *
> */
> CloseableIterator hotKeyIterator(int n);
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12453) AutoSavingCache does not store required keys making RowCacheTests Flaky
[
https://issues.apache.org/jira/browse/CASSANDRA-12453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771572#comment-15771572
]
Jay Zhuang commented on CASSANDRA-12453:
The issue can be consistently reproduced on machine with > 16 CPUs (which will
use [64 cache
segments|https://github.com/snazy/ohc/blob/develop/ohc-core/src/main/java/org/caffinitas/ohc/OHCacheBuilder.java#L159]),
or override the
[{{segmentCount}}|https://github.com/snazy/ohc/blob/develop/ohc-core/src/main/java/org/caffinitas/ohc/OHCacheBuilder.java#L162]
to [64 |
https://github.com/cooldoger/cassandra/commit/53facbb6e75a9adf69be8622ea0d951d3996cc8a]:
{code}
[junit] Testcase:
testRowCachePartialLoad(org.apache.cassandra.db.RowCacheTest):FAILED
[junit] expected:<50> but was:<48>
[junit] junit.framework.AssertionFailedError: expected:<50> but was:<48>
[junit] at
org.apache.cassandra.db.RowCacheTest.rowCacheLoad(RowCacheTest.java:413)
[junit] at
org.apache.cassandra.db.RowCacheTest.testRowCachePartialLoad(RowCacheTest.java:231)
{code}
When there're only 100 keys are [hashed to 64
segments|https://github.com/snazy/ohc/blob/develop/ohc-core/src/main/java/org/caffinitas/ohc/linked/OHCacheLinkedImpl.java#L275],
16 segments are actually empty. The hotKeyIterator try to get [one key per
segments|https://github.com/snazy/ohc/blob/develop/ohc-core/src/main/java/org/caffinitas/ohc/linked/OHCacheLinkedImpl.java#L1396],
which will only get 48.
The fix is to increase the data size to cover more segments, so the
hotKeyIterator is able to get 50 keys. I don't think it will impact Cassandra,
as in production, the cache segments should be filled with more data, even get
less cache keys should not be a problem.
The fix is attached, please review.
> AutoSavingCache does not store required keys making RowCacheTests Flaky
> ---
>
> Key: CASSANDRA-12453
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12453
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: sankalp kohli
>Assignee: Jay Zhuang
>Priority: Minor
> Fix For: 2.2.x, 3.0.x
>
> Attachments: 12453-2.2.txt
>
>
> RowCacheTests were flaky and while investigating, I found that it does not
> store all the keys to disk.
> The reason is that we use OHCache and call hotKeyIterator on it. This is not
> guaranteed to return the number of keys we want. Here is the documentation
> from OHCache
> /**
> * Builds an iterator over the N most recently used keys returning
> deserialized objects.
> * You must call {@code close()} on the returned iterator.
> *
> * Note: During a rehash, the implementation might return keys twice
> or not at all.
> *
> */
> CloseableIterator hotKeyIterator(int n);
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12453) AutoSavingCache does not store required keys making RowCacheTests Flaky
[
https://issues.apache.org/jira/browse/CASSANDRA-12453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439530#comment-15439530
]
Robert Stupp commented on CASSANDRA-12453:
--
mind providing the stack trace of the failure and the patch?
> AutoSavingCache does not store required keys making RowCacheTests Flaky
> ---
>
> Key: CASSANDRA-12453
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12453
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: sankalp kohli
>Priority: Minor
>
> RowCacheTests were flaky and while investigating, I found that it does not
> store all the keys to disk.
> The reason is that we use OHCache and call hotKeyIterator on it. This is not
> guaranteed to return the number of keys we want. Here is the documentation
> from OHCache
> /**
> * Builds an iterator over the N most recently used keys returning
> deserialized objects.
> * You must call {@code close()} on the returned iterator.
> *
> * Note: During a rehash, the implementation might return keys twice
> or not at all.
> *
> */
> CloseableIterator hotKeyIterator(int n);
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12453) AutoSavingCache does not store required keys making RowCacheTests Flaky
[
https://issues.apache.org/jira/browse/CASSANDRA-12453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439485#comment-15439485
]
sankalp kohli commented on CASSANDRA-12453:
---
The way I fixed it internally is to not call hotKeyIterator in the test path.
Can we do this?
> AutoSavingCache does not store required keys making RowCacheTests Flaky
> ---
>
> Key: CASSANDRA-12453
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12453
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: sankalp kohli
>Priority: Minor
>
> RowCacheTests were flaky and while investigating, I found that it does not
> store all the keys to disk.
> The reason is that we use OHCache and call hotKeyIterator on it. This is not
> guaranteed to return the number of keys we want. Here is the documentation
> from OHCache
> /**
> * Builds an iterator over the N most recently used keys returning
> deserialized objects.
> * You must call {@code close()} on the returned iterator.
> *
> * Note: During a rehash, the implementation might return keys twice
> or not at all.
> *
> */
> CloseableIterator hotKeyIterator(int n);
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12453) AutoSavingCache does not store required keys making RowCacheTests Flaky
[
https://issues.apache.org/jira/browse/CASSANDRA-12453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438796#comment-15438796
]
Robert Stupp commented on CASSANDRA-12453:
--
For the hotKeyIterator, OHC implementation requests {{n / #-of-segments}} keys
per hash segment. This is a trade-off to limit the number of locks and accesses
to each segment for each iterator. This means that the number of returned
entries is just a best-effort and the requested number of entries is more or
less a hint. In the extreme case, It may also return slightly more results.
It should be fine for production cases, when many keys are requested.
In order to make the test non-flaky, it would help to see the actual utest
errors (haven't seen any errors in the last CI tests).
> AutoSavingCache does not store required keys making RowCacheTests Flaky
> ---
>
> Key: CASSANDRA-12453
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12453
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: sankalp kohli
>Priority: Minor
>
> RowCacheTests were flaky and while investigating, I found that it does not
> store all the keys to disk.
> The reason is that we use OHCache and call hotKeyIterator on it. This is not
> guaranteed to return the number of keys we want. Here is the documentation
> from OHCache
> /**
> * Builds an iterator over the N most recently used keys returning
> deserialized objects.
> * You must call {@code close()} on the returned iterator.
> *
> * Note: During a rehash, the implementation might return keys twice
> or not at all.
> *
> */
> CloseableIterator hotKeyIterator(int n);
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12453) AutoSavingCache does not store required keys making RowCacheTests Flaky
[
https://issues.apache.org/jira/browse/CASSANDRA-12453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433217#comment-15433217
]
Joel Knighton commented on CASSANDRA-12453:
---
I haven't had much time to look at/think about this yet. Any thoughts here as
the OHC expert [~snazy]?
> AutoSavingCache does not store required keys making RowCacheTests Flaky
> ---
>
> Key: CASSANDRA-12453
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12453
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: sankalp kohli
>Priority: Minor
>
> RowCacheTests were flaky and while investigating, I found that it does not
> store all the keys to disk.
> The reason is that we use OHCache and call hotKeyIterator on it. This is not
> guaranteed to return the number of keys we want. Here is the documentation
> from OHCache
> /**
> * Builds an iterator over the N most recently used keys returning
> deserialized objects.
> * You must call {@code close()} on the returned iterator.
> *
> * Note: During a rehash, the implementation might return keys twice
> or not at all.
> *
> */
> CloseableIterator hotKeyIterator(int n);
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12453) AutoSavingCache does not store required keys making RowCacheTests Flaky
[
https://issues.apache.org/jira/browse/CASSANDRA-12453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15419269#comment-15419269
]
sankalp kohli commented on CASSANDRA-12453:
---
[~jkni] One of the ways I have fixed this internally is to pass a boolean from
test to not use hotKeyIterator. However this test shows a minor bug. Not sure
how we can fix the actual issue. What do you think?
> AutoSavingCache does not store required keys making RowCacheTests Flaky
> ---
>
> Key: CASSANDRA-12453
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12453
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: sankalp kohli
>Priority: Minor
>
> RowCacheTests were flaky and while investigating, I found that it does not
> store all the keys to disk.
> The reason is that we use OHCache and call hotKeyIterator on it. This is not
> guaranteed to return the number of keys we want. Here is the documentation
> from OHCache
> /**
> * Builds an iterator over the N most recently used keys returning
> deserialized objects.
> * You must call {@code close()} on the returned iterator.
> *
> * Note: During a rehash, the implementation might return keys twice
> or not at all.
> *
> */
> CloseableIterator hotKeyIterator(int n);
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
