[jira] [Commented] (CASSANDRA-17044) Refactor schema management to allow for schema source pluggability
[ https://issues.apache.org/jira/browse/CASSANDRA-17044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833303#comment-17833303 ] Jeff Jirsa commented on CASSANDRA-17044: [~jjordan] - you mentioned "A CEP around this is in process, should be ready to propose in the next week" - did you folks ever publish that? > Refactor schema management to allow for schema source pluggability > -- > > Key: CASSANDRA-17044 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17044 > Project: Cassandra > Issue Type: Improvement > Components: Cluster/Schema >Reporter: Jacek Lewandowski >Assignee: Jacek Lewandowski >Priority: Normal > Fix For: 4.1-alpha1, 4.1 > > Time Spent: 20m > Remaining Estimate: 0h > > The idea is decompose `Schema` into separate entities responsible for > different things. In particular extract what is related to schema storage and > synchronization into a separate class so that it is possible to create an > extension point there and store schema in a different way than > `system_schema` keyspace, for example in etcd. > This would also simplify the logic and reduce the number of special cases, > make all the things more testable and the logic of internal classes > encapsulated. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-18655) Unfinalise AbstractVirtualTable.select(..) for downstream patches
[ https://issues.apache.org/jira/browse/CASSANDRA-18655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17741976#comment-17741976 ] Jeff Jirsa edited comment on CASSANDRA-18655 at 7/11/23 12:42 PM: -- Aleksey's comment here: https://issues.apache.org/jira/browse/CASSANDRA-7622?focusedCommentId=16456572=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16456572 And my comment here: https://issues.apache.org/jira/browse/CASSANDRA-7622?focusedCommentId=16456749=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16456749 In particular, this matches my memory, and the reason I spent 60 minutes of NGCC talking about it: "We don't want it to be exposed in regular DDL. However, we would like this to be at least a little bit generic, so that one could plug-in extra virtual keyspaces on C* startup, similar to how some folks/forks add extra system keyspaces and system tables. There are some use cases for virtual tables that we want to experiment with (Jeff Jirsa gave a few examples in his NGCC talk) that are compelling enough to at least allow this possibility." The only written dissent I see is "I am also not convinced that we should do it. I would really love to avoid another Trigger like feature.". We're years in and adoption of vtables is high. I think it's clear this is not a trigger like feature. I would +1 a removal of `final` was (Author: jjirsa): Aleksey's comment here: https://issues.apache.org/jira/browse/CASSANDRA-7622?focusedCommentId=16456572=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16456572 And my comment here: https://issues.apache.org/jira/browse/CASSANDRA-7622?focusedCommentId=16456749=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16456749 In particular, this matches my memory, and the reason I spent 60 minutes of NGCC talking about it: "We don't want it to be exposed in regular DDL. However, we would like this to be at least a little bit generic, so that one could plug-in extra virtual keyspaces on C* startup, similar to how some folks/forks add extra system keyspaces and system tables. There are some use cases for virtual tables that we want to experiment with (Jeff Jirsa gave a few examples in his NGCC talk) that are compelling enough to at least allow this possibility." Sylvain agreed: "If by "plug-in", you mean "patch-in" (but easily so and without being too limited), then sure, that's just good coding and no issue on that from me." The only written dissent I see is "I am also not convinced that we should do it. I would really love to avoid another Trigger like feature.". We're years in and adoption of vtables is high. I think it's clear this is not a trigger like feature. I would +1 a removal of `final` > Unfinalise AbstractVirtualTable.select(..) for downstream patches > - > > Key: CASSANDRA-18655 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18655 > Project: Cassandra > Issue Type: Improvement > Components: Feature/Virtual Tables >Reporter: Michael Semb Wever >Assignee: Michael Semb Wever >Priority: Normal > > In AbstractVirtualTable the select methods are final. This prevents > downstream C* engineers from implementing their own virtual tables where the > select needs to be overridden. > This is not a C* API and is not intended for C* users and operators. > Extension of these methods should also clearly marked as experimental with no > maintenance or compatibility provided from any release to another (including > patch versions). > The original proposal for Virtual Tables (CASSANDRA-7622) was to have a table > "backed by an API, rather than data explicitly managed and stored as > sstables". A number of people on the ticket supported the eventual idea of > user-defined Virtual Tables. The consensus was that an incremental approach > should be taken, where this should not be part of the initial implementation, > and that use-cases and careful consideration around API security and > compatibility would be needed. > The next incremental approach can be to permit downstream patches to > experiment against an explicitly labelled experimental (non-stable) internal > code (so to protect the C* community from security and compatibility > concerns). Such experiments will help smoke out and promote more grounded > discussions for further work, if so found and desired. > The patch is two lines: to remove the final keyword from both select(..) > methods; and adding whatever comment/annotation we need to state their > experimental/non-stable state and limited audience. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CASSANDRA-18655) Unfinalise AbstractVirtualTable.select(..) for downstream patches
[ https://issues.apache.org/jira/browse/CASSANDRA-18655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17741976#comment-17741976 ] Jeff Jirsa commented on CASSANDRA-18655: Aleksey's comment here: https://issues.apache.org/jira/browse/CASSANDRA-7622?focusedCommentId=16456572=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16456572 And my comment here: https://issues.apache.org/jira/browse/CASSANDRA-7622?focusedCommentId=16456749=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16456749 In particular, this matches my memory, and the reason I spent 60 minutes of NGCC talking about it: "We don't want it to be exposed in regular DDL. However, we would like this to be at least a little bit generic, so that one could plug-in extra virtual keyspaces on C* startup, similar to how some folks/forks add extra system keyspaces and system tables. There are some use cases for virtual tables that we want to experiment with (Jeff Jirsa gave a few examples in his NGCC talk) that are compelling enough to at least allow this possibility." Sylvain agreed: "If by "plug-in", you mean "patch-in" (but easily so and without being too limited), then sure, that's just good coding and no issue on that from me." The only written dissent I see is "I am also not convinced that we should do it. I would really love to avoid another Trigger like feature.". We're years in and adoption of vtables is high. I think it's clear this is not a trigger like feature. I would +1 a removal of `final` > Unfinalise AbstractVirtualTable.select(..) for downstream patches > - > > Key: CASSANDRA-18655 > URL: https://issues.apache.org/jira/browse/CASSANDRA-18655 > Project: Cassandra > Issue Type: Improvement > Components: Feature/Virtual Tables >Reporter: Michael Semb Wever >Assignee: Michael Semb Wever >Priority: Normal > > In AbstractVirtualTable the select methods are final. This prevents > downstream C* engineers from implementing their own virtual tables where the > select needs to be overridden. > This is not a C* API and is not intended for C* users and operators. > Extension of these methods should also clearly marked as experimental with no > maintenance or compatibility provided from any release to another (including > patch versions). > The original proposal for Virtual Tables (CASSANDRA-7622) was to have a table > "backed by an API, rather than data explicitly managed and stored as > sstables". A number of people on the ticket supported the eventual idea of > user-defined Virtual Tables. The consensus was that an incremental approach > should be taken, where this should not be part of the initial implementation, > and that use-cases and careful consideration around API security and > compatibility would be needed. > The next incremental approach can be to permit downstream patches to > experiment against an explicitly labelled experimental (non-stable) internal > code (so to protect the C* community from security and compatibility > concerns). Such experiments will help smoke out and promote more grounded > discussions for further work, if so found and desired. > The patch is two lines: to remove the final keyword from both select(..) > methods; and adding whatever comment/annotation we need to state their > experimental/non-stable state and limited audience. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12183) compaction_history system table does not capture all historical compaction sessions
[
https://issues.apache.org/jira/browse/CASSANDRA-12183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17736523#comment-17736523
]
Jeff Jirsa commented on CASSANDRA-12183:
The fix here looks reasonable and fine, but for hygiene, may be nice to add a
regression test.
After a compaction, ensure we can read the same taskId out of the compaction
history table?
> compaction_history system table does not capture all historical compaction
> sessions
> ---
>
> Key: CASSANDRA-12183
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12183
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Observability, Local/Compaction
>Reporter: Wei Deng
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 5.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> It appears that some compaction sessions are not recorded in
> system.compaction_history table after the compaction session successfully
> finishes.
> The following is an example (test by simply running +cassandra-stress write
> n=1+):
> {noformat}
> automaton@wdengdse50google-98425b985-3:~$ nodetool compactionstats
> pending tasks: 46
> id compaction typekeyspace
> tablecompletedtotalunit progress
>fa8a4f30-4884-11e6-b916-1dbd340a212fCompaction keyspace1
> standard1 4233184044 4774209875 bytes 88.67%
>91e30d21-4887-11e6-b916-1dbd340a212fCompaction keyspace1
> standard1836983029889773060 bytes 94.07%
> Active compaction remaining time : 0h00m35s
> automaton@wdengdse50google-98425b985-3:~$ nodetool compactionstats
> pending tasks: 47
> id compaction typekeyspace
> tablecompletedtotalunit progress
>fa8a4f30-4884-11e6-b916-1dbd340a212fCompaction keyspace1
> standard1 4353251539 4774209875 bytes 91.18%
>28359094-4888-11e6-b916-1dbd340a212fCompaction keyspace1
> standard1 49732274 4071652280 bytes 1.22%
> Active compaction remaining time : 0h04m24s
> {noformat}
> At this point you know the previous compaction session
> 91e30d21-4887-11e6-b916-1dbd340a212f finished and confirmation can be found
> from debug.log
> {noformat}
> automaton@wdengdse50google-98425b985-3:~$ grep
> 91e30d21-4887-11e6-b916-1dbd340a212f /var/log/cassandra/debug.log
> DEBUG [CompactionExecutor:4] 2016-07-12 23:22:58,674 CompactionTask.java:146
> - Compacting (91e30d21-4887-11e6-b916-1dbd340a212f)
> [/var/lib/cassandra/data/keyspace1/standard1-9c02e9c1487c11e6b9161dbd340a212f/mb-290-big-Data.db:level=0,
>
> /var/lib/cassandra/data/keyspace1/standard1-9c02e9c1487c11e6b9161dbd340a212f/mb-279-big-Data.db:level=0,
>
> /var/lib/cassandra/data/keyspace1/standard1-9c02e9c1487c11e6b9161dbd340a212f/mb-281-big-Data.db:level=0,
>
> /var/lib/cassandra/data/keyspace1/standard1-9c02e9c1487c11e6b9161dbd340a212f/mb-280-big-Data.db:level=0,
>
> /var/lib/cassandra/data/keyspace1/standard1-9c02e9c1487c11e6b9161dbd340a212f/mb-284-big-Data.db:level=0,
>
> /var/lib/cassandra/data/keyspace1/standard1-9c02e9c1487c11e6b9161dbd340a212f/mb-283-big-Data.db:level=0,
>
> /var/lib/cassandra/data/keyspace1/standard1-9c02e9c1487c11e6b9161dbd340a212f/mb-287-big-Data.db:level=0,
>
> /var/lib/cassandra/data/keyspace1/standard1-9c02e9c1487c11e6b9161dbd340a212f/mb-292-big-Data.db:level=0,
>
> /var/lib/cassandra/data/keyspace1/standard1-9c02e9c1487c11e6b9161dbd340a212f/mb-286-big-Data.db:level=0,
>
> /var/lib/cassandra/data/keyspace1/standard1-9c02e9c1487c11e6b9161dbd340a212f/mb-289-big-Data.db:level=0,
> ]
> DEBUG [CompactionExecutor:4] 2016-07-12 23:26:56,054 CompactionTask.java:217
> - Compacted (91e30d21-4887-11e6-b916-1dbd340a212f) 10 sstables to
> [/var/lib/cassandra/data/keyspace1/standard1-9c02e9c1487c11e6b9161dbd340a212f/mb-293-big,]
> to level=0. 889,773,060 bytes to 890,473,350 (~100% of original) in
> 237,365ms = 3.577703MB/s. 0 total partitions merged to 3,871,921. Partition
> merge counts were {1:3871921, }
> {noformat}
> However, if you query system.compaction_history table or run "nodetool
> compactionhistory | grep 91e30d21-4887-11e6-b916-1dbd340a212f" you will get
> nothing:
> {noformat}
> automaton@wdengdse50google-98425b985-3:~$ cqlsh -u cassandra
> Password:
> Connected to dse50 at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 3.0.7.1158 | DSE 5.0.0 | CQL spec 3.4.0 | Native
> protocol v4]
> Use HELP for help.
> cassandra@cqlsh> select * from system.compaction_history where
> id=91e30d21-4887-11e6-b916-1dbd340a212f;
> id | bytes_in | bytes_out | columnfamily_name | compacted_at | keyspace_name
> | rows_merged
>
[jira] [Commented] (CASSANDRA-8460) Make it possible to move non-compacting sstables to slow/big storage in DTCS
[ https://issues.apache.org/jira/browse/CASSANDRA-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17722803#comment-17722803 ] Jeff Jirsa commented on CASSANDRA-8460: --- I think a lot of people would still find it useful, however, I think since 2014, the way most people think about storage has changed. Tiering to spinning disk or cheaper block devices is fine. It's a win. It's easy to reason about - probably just implement it via compaction and all the read and write path stay exactly the same. But I think the industry trends would suggest this is suboptimal - moving this to a fast object store (e.g. s3) would be even better. It's lower cost / higher durability, and it allows for other things "eventually", like sharing one sstable between replicas (or eventually erasure encoding pieces of data). That turns this ticket from ~easy to ~hard, because you also have to touch the read path (or, more likely, change / add a new sstablereader that can read from object storage, and then figure out how you want to upload to object storage). So "is there interest", probably, but in an s3 version of this feature, vs spinning disk. > Make it possible to move non-compacting sstables to slow/big storage in DTCS > > > Key: CASSANDRA-8460 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8460 > Project: Cassandra > Issue Type: Improvement > Components: Local/Compaction >Reporter: Marcus Eriksson >Assignee: Lerh Chuan Low >Priority: Normal > Labels: doc-impacting, dtcs > Fix For: 5.x > > > It would be nice if we could configure DTCS to have a set of extra data > directories where we move the sstables once they are older than > max_sstable_age_days. > This would enable users to have a quick, small SSD for hot, new data, and big > spinning disks for data that is rarely read and never compacted. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] (CASSANDRA-17883) Node will not get schema updates if it replaces a removed node with the same address
[ https://issues.apache.org/jira/browse/CASSANDRA-17883 ] Jeff Jirsa deleted comment on CASSANDRA-17883: was (Author: jjirsa): When you stream a host with an existing IP (and same token), you don't get ANY writes during bootstrap stream (you get the stream, but you wont get any other writes). IIRC, it's related to how the write logic would increment `blockFor`, but the coordinator can't tell which version of the IP acks a write (old one or new one), so it cant actually implement the extra write for the pending range. > Node will not get schema updates if it replaces a removed node with the same > address > > > Key: CASSANDRA-17883 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17883 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Distributed Metadata >Reporter: Jacek Lewandowski >Priority: Normal > Fix For: 3.0.x, 3.11.x, 4.0.x, 5.x > > > If a node is removed and then a node with the same address is added, it is > ignored by migration coordinator - that is, when that node fails to push the > schema change, the other nodes will not be able pull it later during periodic > update. > Such situation will last until the other nodes gets restarted. > This is caused by permanently adding the address of the node being removed to > the ignore list in migration coordinator. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17883) Node will not get schema updates if it replaces a removed node with the same address
[ https://issues.apache.org/jira/browse/CASSANDRA-17883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713382#comment-17713382 ] Jeff Jirsa commented on CASSANDRA-17883: When you stream a host with an existing IP (and same token), you don't get ANY writes during bootstrap stream (you get the stream, but you wont get any other writes). IIRC, it's related to how the write logic would increment `blockFor`, but the coordinator can't tell which version of the IP acks a write (old one or new one), so it cant actually implement the extra write for the pending range. > Node will not get schema updates if it replaces a removed node with the same > address > > > Key: CASSANDRA-17883 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17883 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Distributed Metadata >Reporter: Jacek Lewandowski >Priority: Normal > Fix For: 3.0.x, 3.11.x, 4.0.x, 5.x > > > If a node is removed and then a node with the same address is added, it is > ignored by migration coordinator - that is, when that node fails to push the > schema change, the other nodes will not be able pull it later during periodic > update. > Such situation will last until the other nodes gets restarted. > This is caused by permanently adding the address of the node being removed to > the ignore list in migration coordinator. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18134) Improve handling of min/max clustering in sstable
[
https://issues.apache.org/jira/browse/CASSANDRA-18134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17688583#comment-17688583
]
Jeff Jirsa commented on CASSANDRA-18134:
I think I've asked for downgradability for ~years, but Scott's proposal on the
mailing list is a proposal not a decision, and the wiki has it in pending, and
I haven't seen any other feature gated on downgradability.
Is there a vote thread that asserts that's a policy instead of a preference?
(I do agree this needs to land in at least a new minor, if not a new major,
rather than a patch release, like 4.2 or 5.0, but I'm not aware of the project
voting on downgradability as a general requirement for new work)
> Improve handling of min/max clustering in sstable
> -
>
> Key: CASSANDRA-18134
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18134
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/SSTable
>Reporter: Jacek Lewandowski
>Assignee: Jacek Lewandowski
>Priority: Normal
> Fix For: 5.x
>
>
> This patch improves the following things:
> # SSTable metadata will store a covered slice instead of min/max clusterings.
> The difference is that for slices there is available the type of a bound
> rather than just a clustering. In particular it will provide the information
> whether the lower and upper bound of an sstable is opened or closed.
> # SSTable metadata will store a flag whether the SSTable contains any
> partition level deletions or not
> # The above two changes required to introduce a new major format for SSTables
> - {{oa}}
> # Single partition read command makes use of the above changes. In particular
> an sstable can be skipped when it does not intersect with the column filter,
> does not have partition level deletions and does not have statics; In case
> there are partition level deletions, but the other conditions are satisfied,
> only the partition header needs to be accessed (tests attached)
> # Skipping sstables assuming those three conditions are satisfied has been
> implemented also for partition range queries (tests attached). Also added
> minor separate statistics to record the number of accessed sstables in
> partition reads because now not all of them need to be accessed. That
> statistics is also needed in tests to confirm skipping.
> # Artificial lower bound marker is now an object on its own and is not
> implemented as a special case of range tombstone bound. Instead it sorts
> right before the lowest available bound in the data
> # Extended the lower bound optimization usage due the 1 and 2
> # Do not initialize iterator just to get a cached partition and associated
> columns index. The purpose of using lower bound optimization was to avoid
> opening an iterator of an sstable if possible.
> See also CASSANDRA-14861
> The changes in this patch include work of [~blambov], [~slebresne],
> [~jakubzytka] and [~jlewandowski]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-18134) Improve handling of min/max clustering in sstable
[
https://issues.apache.org/jira/browse/CASSANDRA-18134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17679398#comment-17679398
]
Jeff Jirsa commented on CASSANDRA-18134:
Not a review, but really interesting patch.
This is not a suggestion to change, just curious: did you consider using
something other than a metadata flag for partition deletions? Like a second
bloom filter of partition deletions? I ask because if a use case regularly does
partition deletions, the metadata flag seems like it reverts back to the
current behavior (any delete = include the file)?
(Also reminded me CASSANDRA-9843 was a thing that was opened once, years ago).
> Improve handling of min/max clustering in sstable
> -
>
> Key: CASSANDRA-18134
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18134
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/SSTable
>Reporter: Jacek Lewandowski
>Assignee: Jacek Lewandowski
>Priority: Normal
> Fix For: 4.x
>
>
> This patch improves the following things:
> # SSTable metadata will store a covered slice instead of min/max clusterings.
> The difference is that for slices there is available the type of a bound
> rather than just a clustering. In particular it will provide the information
> whether the lower and upper bound of an sstable is opened or closed.
> # SSTable metadata will store a flag whether the SSTable contains any
> partition level deletions or not
> # The above two changes required to introduce a new major format for SSTables
> - {{oa}}
> # Single partition read command makes use of the above changes. In particular
> an sstable can be skipped when it does not intersect with the column filter,
> does not have partition level deletions and does not have statics; In case
> there are partition level deletions, but the other conditions are satisfied,
> only the partition header needs to be accessed (tests attached)
> # Skipping sstables assuming those three conditions are satisfied has been
> implemented also for partition range queries (tests attached). Also added
> minor separate statistics to record the number of accessed sstables in
> partition reads because now not all of them need to be accessed. That
> statistics is also needed in tests to confirm skipping.
> # Artificial lower bound marker is now an object on its own and is not
> implemented as a special case of range tombstone bound. Instead it sorts
> right before the lowest available bound in the data
> # Extended the lower bound optimization usage due the 1 and 2
> # Do not initialize iterator just to get a cached partition and associated
> columns index. The purpose of using lower bound optimization was to avoid
> opening an iterator of an sstable if possible.
> See also CASSANDRA-14861
> The changes in this patch include work of [~blambov], [~slebresne],
> [~jakubzytka] and [~jlewandowski]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17991) Possible data inconsistency during bootstrap/decommission
[
https://issues.apache.org/jira/browse/CASSANDRA-17991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625960#comment-17625960
]
Jeff Jirsa commented on CASSANDRA-17991:
> Let's consider a scenario in that _n4_ receives SSTable1 and SSTable3, but
> not yet SSTable2, and _n4_ compacts SSTable1 and SSTable3. In this case, _n4_
> would purge the key "1". So at this time, there are no traces of key "1" on
> {_}n4{_}. After some time, SSTable2 is streamed, and at this time it will
> stream the key "1" as well.
It should be atomic, it shouldn't let anything compact until the streaming is
done. In what situation are you seeing this be non-atomic?
> Possible data inconsistency during bootstrap/decommission
> -
>
> Key: CASSANDRA-17991
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17991
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Bootstrap and Decommission
>Reporter: Jaydeepkumar Chovatia
>Priority: Normal
>
> I am facing one corner case in which the deleted data resurrects.
> tl;dr: This could be because when we stream all the SSTables for a given
> token range to the new owner, then they are not sent atomically, so the new
> owner could do compaction on the partially received SSTables, which might
> remove the tombstones.
>
> Here are the reproducible steps:
> +*Prerequisite*+ # Three nodes Cassandra cluster n1, n2, and n3 (C* version
> 3.0.27)
> #
> {code:java}
> CREATE KEYSPACE KS1 WITH replication = {'class': 'NetworkTopologyStrategy',
> 'dc1': '3'};
> CREATE TABLE KS1.T1 (
> key int,
> c1 int,
> c2 int,
> c3 int
> PRIMARY KEY (key)
> ) WITH CLUSTERING ORDER BY (key ASC)
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
> AND gc_grace_seconds = 864000;
> {code}
>
> *Reproducible Steps*
> * Day1: Insert a new record followed by {_}nodetool flush on n1, n2, and
> n3{_}. A new SSTable ({_}SSTable1{_}) will be created.
> {code:java}
> INSERT INTO KS1.T1 (key, c1, c2, c3) values (1, 10, 20, 30){code}
> * Day2: Insert the same record again followed by _nodetool flush_ {_}on n1,
> n2, and n3{_}{_}.{_} A new SSTable ({_}SSTable2{_}) will be created
> {code:java}
> INSERT INTO KS1.T1 (key, c1, c2, c3) values (1, 10, 20, 30){code}
> * Day3: Here is the data layout on SSTables on n1, n2, and n3
> {code:java}
> SSTable1:
> {
> "partition" : {
> "key" : [ "1" ],
> "position" : 900
> },
> "rows" : [
> "type" : "row",
> "position" : 10,
> "liveness_info" : { "tstamp" : "2022-10-15T00:00:00.01Z"},
> "cells" : [
> { "name" : "c1", "value" : 10 },
> { "name" : "c2", "value" : 20 },
> { "name" : "c3", "value" : 30 },
> ]
> }
> .
> SSTable2:
> {
> "partition" : {
> "key" : [ "1" ],
> "position" : 900
> },
> "rows" : [
> "type" : "row",
> "position" : 10,
> "liveness_info" : { "tstamp" : "2022-10-16T00:00:00.01Z"},
> "cells" : [
> { "name" : "c1", "value" : 10 },
> { "name" : "c2", "value" : 20 },
> { "name" : "c3", "value" : 30 },
> ]
> }
> {code}
> * Day4: Delete the record followed by _nodetool flush_ on n1, n2, and n3
> {code:java}
> CONSISTENCY ALL; DELETE FROM KS1.T1 WHERE key = 1; {code}
> * Day5: Here is the data layout on SSTables on n1, n2, and n3
> {code:java}
> SSTable1:
> {
> "partition" : {
> "key" : [ "1" ],
> "position" : 900
> },
> "rows" : [
> "type" : "row",
> "position" : 10,
> "liveness_info" : { "tstamp" : "2022-10-15T00:00:00.01Z"},
> "cells" : [
> { "name" : "c1", "value" : 10 },
> { "name" : "c2", "value" : 20 },
> { "name" : "c3", "value" : 30 },
> ]
> }
> .
> SSTable2:
> {
> "partition" : {
> "key" : [ "1" ],
> "position" : 900
> },
> "rows" : [
> "type" : "row",
> "position" : 10,
> "liveness_info" : { "tstamp" : "2022-10-16T00:00:00.01Z"},
> "cells" : [
> { "name" : "c1", "value" : 10 },
> { "name" : "c2", "value" : 20 },
> { "name" : "c3", "value" : 30 },
> ]
> }
> .
> SSTable3 (Tombstone):
> {
> "partition" : {
> "key" : [ "1" ],
> "position" : 900
> },
> "rows" : [
> "type" : "row",
> "position" : 10,
> "deletion_info" : { "marked_deleted" : "2022-10-19T00:00:00.01Z",
> "local_delete_time" : "2022-10-19T00:00:00.01Z" },
> "cells" : [ ]
> }
> {code}
> * Day20: Nothing happens for more than 10
[jira] [Commented] (CASSANDRA-17679) Make resumable bootstrap feature optional
[
https://issues.apache.org/jira/browse/CASSANDRA-17679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17574949#comment-17574949
]
Jeff Jirsa commented on CASSANDRA-17679:
All {{-Dcassandra.reset_bootstrap_progress}} seems to do is truncate the system
keyspace of ranges, but none of the already-bootstrapped-data?
https://github.com/apache/cassandra/blob/afaeb9302008331d8520687594afcb9f9ee664dc/src/java/org/apache/cassandra/db/SystemKeyspace.java#L1388
> Make resumable bootstrap feature optional
> -
>
> Key: CASSANDRA-17679
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17679
> Project: Cassandra
> Issue Type: New Feature
> Components: Consistency/Streaming
>Reporter: Josh McKenzie
>Assignee: Josh McKenzie
>Priority: Normal
> Fix For: 4.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> From the patch I'm working on:
> {code}
> # In certain environments, operators may want to disable resumable bootstrap
> in order to avoid potential correctness
> # violations or data loss scenarios. Largely this centers around nodes going
> down during bootstrap, tombstones being
> # written, and potential races with repair. By default we leave this on as
> it's been enabled for quite some time,
> # however the option to disable it is more palatable now that we have zero
> copy streaming as that greatly accelerates
> # bootstraps. This defaults to true.
> # resumable_bootstrap_enabled: true
> {code}
> Not really a great fit for guardrails as it's less a "feature to be toggled
> on and off" and more a subset of a specific feature that in certain
> circumstances can lead to issues.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17132) Fix startup issue with internode_application_timeout_in_ms
[
https://issues.apache.org/jira/browse/CASSANDRA-17132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17491195#comment-17491195
]
Jeff Jirsa commented on CASSANDRA-17132:
This commit broke users upgrading from 4.0.1 to 4.0.2.
We should NOT be making breaking changes in minor versions.
We also missed the {{NEWS.txt}} entry that notifies customers of breaking
changes.
> Fix startup issue with internode_application_timeout_in_ms
> --
>
> Key: CASSANDRA-17132
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17132
> Project: Cassandra
> Issue Type: Bug
> Components: Build
>Reporter: Ekaterina Dimitrova
>Assignee: Ekaterina Dimitrova
>Priority: Normal
> Fix For: 4.0.2, 4.1
>
>
> While testing my patch for CASSANDRA-17131 I found that there is a problem
> with _internode_application_timeout_in_ms_ in 4.0 and trunk.
> Seems to me that we can just safely remove it for now?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17048) Replace sequential sstable generation identifier with ULID
[ https://issues.apache.org/jira/browse/CASSANDRA-17048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17430815#comment-17430815 ] Jeff Jirsa commented on CASSANDRA-17048: Not a review, but a reminder: We removed windows support in trunk, but we've had issues with long paths in the past. Worth checking CASSANDRA-6962 if we end up with a longer generation identifier. > Replace sequential sstable generation identifier with ULID > -- > > Key: CASSANDRA-17048 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17048 > Project: Cassandra > Issue Type: Improvement > Components: Local/SSTable >Reporter: Jacek Lewandowski >Assignee: Jacek Lewandowski >Priority: Normal > Fix For: 4.1 > > Time Spent: 10m > Remaining Estimate: 0h > > Replace the current sequential sstable generation identifier with ULID based. > ULID is better because we do not need to scan the existing files to pick the > starting number as well as we can generate globally unique identifiers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16899) Change select wording to be more inclusive
[ https://issues.apache.org/jira/browse/CASSANDRA-16899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408860#comment-17408860 ] Jeff Jirsa commented on CASSANDRA-16899: [~mck] I'll wait for the 16763 changes to land and re-do the patch :) Mind pinging me in slack when they get replaced? > Change select wording to be more inclusive > -- > > Key: CASSANDRA-16899 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16899 > Project: Cassandra > Issue Type: Improvement > Components: Documentation/Website >Reporter: Jeff Jirsa >Assignee: Jeff Jirsa >Priority: Normal > > The conscious language checker has flagged a few thousand places where we are > using words that may be insensitive or undesirable. > https://clc.diversity.apache.org/analysis.html?project=cassandra.git > This ticket will address the non-code comment text as a first pass. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16899) Change select wording to be more inclusive
[ https://issues.apache.org/jira/browse/CASSANDRA-16899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-16899: --- Source Control Link: https://github.com/jeffjirsa/cassandra/commit/3dfe9090929cd730c0545ea1f4316b8bb17d31dd > Change select wording to be more inclusive > -- > > Key: CASSANDRA-16899 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16899 > Project: Cassandra > Issue Type: Improvement > Components: Documentation/Website >Reporter: Jeff Jirsa >Assignee: Jeff Jirsa >Priority: Normal > > The conscious language checker has flagged a few thousand places where we are > using words that may be insensitive or undesirable. > https://clc.diversity.apache.org/analysis.html?project=cassandra.git > This ticket will address the non-code comment text as a first pass. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16899) Change select wording to be more inclusive
[ https://issues.apache.org/jira/browse/CASSANDRA-16899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-16899: --- Test and Documentation Plan: No code or functional changes expected. Status: Patch Available (was: Open) > Change select wording to be more inclusive > -- > > Key: CASSANDRA-16899 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16899 > Project: Cassandra > Issue Type: Improvement > Components: Documentation/Website >Reporter: Jeff Jirsa >Assignee: Jeff Jirsa >Priority: Normal > > The conscious language checker has flagged a few thousand places where we are > using words that may be insensitive or undesirable. > https://clc.diversity.apache.org/analysis.html?project=cassandra.git > This ticket will address the non-code comment text as a first pass. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16899) Change select wording to be more inclusive
[ https://issues.apache.org/jira/browse/CASSANDRA-16899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-16899: --- Change Category: Code Clarity Complexity: Low Hanging Fruit Component/s: Documentation/Website Status: Open (was: Triage Needed) > Change select wording to be more inclusive > -- > > Key: CASSANDRA-16899 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16899 > Project: Cassandra > Issue Type: Improvement > Components: Documentation/Website >Reporter: Jeff Jirsa >Assignee: Jeff Jirsa >Priority: Normal > > The conscious language checker has flagged a few thousand places where we are > using words that may be insensitive or undesirable. > https://clc.diversity.apache.org/analysis.html?project=cassandra.git > This ticket will address the non-code comment text as a first pass. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16899) Change select wording to be more inclusive
[ https://issues.apache.org/jira/browse/CASSANDRA-16899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17406840#comment-17406840 ] Jeff Jirsa commented on CASSANDRA-16899: Code here, trunk only: https://github.com/jeffjirsa/cassandra/commit/3dfe9090929cd730c0545ea1f4316b8bb17d31dd Comment and docs only, no intentional changes to any code Running CI in a bit > Change select wording to be more inclusive > -- > > Key: CASSANDRA-16899 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16899 > Project: Cassandra > Issue Type: Improvement >Reporter: Jeff Jirsa >Assignee: Jeff Jirsa >Priority: Normal > > The conscious language checker has flagged a few thousand places where we are > using words that may be insensitive or undesirable. > https://clc.diversity.apache.org/analysis.html?project=cassandra.git > This ticket will address the non-code comment text as a first pass. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-16899) Change select wording to be more inclusive
Jeff Jirsa created CASSANDRA-16899: -- Summary: Change select wording to be more inclusive Key: CASSANDRA-16899 URL: https://issues.apache.org/jira/browse/CASSANDRA-16899 Project: Cassandra Issue Type: Improvement Reporter: Jeff Jirsa Assignee: Jeff Jirsa The conscious language checker has flagged a few thousand places where we are using words that may be insensitive or undesirable. https://clc.diversity.apache.org/analysis.html?project=cassandra.git This ticket will address the non-code comment text as a first pass. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-11684) Cleanup key ranges during compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-11684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308939#comment-17308939 ] Jeff Jirsa commented on CASSANDRA-11684: Close as a dupe of 5051 In Slack I mentioned this morning: "I had implemented something like 5051 in an early version of the pluggable twcs jar before i realized how terrifying it is given cassandra's lack of actual cluster membership" The TLDR is I've learned a bit since 2015 and what I thought was a good idea then probably requires some technical support to do safely now. > Cleanup key ranges during compaction > > > Key: CASSANDRA-11684 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11684 > Project: Cassandra > Issue Type: Improvement > Components: Local/Compaction >Reporter: Stefan Podkowinski >Priority: Normal > > Currently cleanup is considered an optional, manual operation that users are > told to run to free disk space after a node was affected by topology changes. > However, unmanaged key ranges could also end up on a node through other ways, > e.g. manual added sstable files by an admin. > I'm also not sure unmanaged data is really that harmless and cleanup should > really be optional, if you don't need to reclaim the disk space. When it > comes to repairs, users are expected to purge a node after downtime in case > it was not fully covered by a repair within gc_grace afterwards, in order to > avoid re-introducing deleted data. But the same could happen with unmanaged > data, e.g. after topology changes activate unmanaged ranges again or after > restoring backups. > I'd therefor suggest to avoid rewriting key ranges no longer belonging to a > node and older than gc_grace during compactions. > Maybe we could also introduce another CLEANUP_COMPACTION operation to find > candidates based on SSTable.first/last in case we don't have pending regular > or tombstone compactions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-11684) Cleanup key ranges during compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-11684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284966#comment-17284966 ] Jeff Jirsa commented on CASSANDRA-11684: Until strong cluster membership is a thing, this is probably more unsafe than people realize during range movements (nodetool move, expansions, shrinks, etc). > Cleanup key ranges during compaction > > > Key: CASSANDRA-11684 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11684 > Project: Cassandra > Issue Type: Improvement > Components: Local/Compaction >Reporter: Stefan Podkowinski >Priority: Normal > Labels: gsoc2021, mentor > > Currently cleanup is considered an optional, manual operation that users are > told to run to free disk space after a node was affected by topology changes. > However, unmanaged key ranges could also end up on a node through other ways, > e.g. manual added sstable files by an admin. > I'm also not sure unmanaged data is really that harmless and cleanup should > really be optional, if you don't need to reclaim the disk space. When it > comes to repairs, users are expected to purge a node after downtime in case > it was not fully covered by a repair within gc_grace afterwards, in order to > avoid re-introducing deleted data. But the same could happen with unmanaged > data, e.g. after topology changes activate unmanaged ranges again or after > restoring backups. > I'd therefor suggest to avoid rewriting key ranges no longer belonging to a > node and older than gc_grace during compactions. > Maybe we could also introduce another CLEANUP_COMPACTION operation to find > candidates based on SSTable.first/last in case we don't have pending regular > or tombstone compactions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16261) Prevent unbounded number of flushing tasks
[ https://issues.apache.org/jira/browse/CASSANDRA-16261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1727#comment-1727 ] Jeff Jirsa commented on CASSANDRA-16261: There's so, so, so many unrelated whitespace and style changes in the 3.0 version of this patch :( > Prevent unbounded number of flushing tasks > -- > > Key: CASSANDRA-16261 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16261 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Local Write-Read Paths >Reporter: Ekaterina Dimitrova >Assignee: Ekaterina Dimitrova >Priority: Normal > Fix For: 3.0.x, 3.11.x, 4.0-beta > > > The cleaner thread is not prevented from queueing an unbounded number of > flushing tasks for memtables that are almost empty. > This patch adds a mechanism to track the number of pending flushing > tasks in the memtable cleaner. Above the maximum number (2x the flushing > threads by default), only memtables using at least MCT memory will be > flushed, where MCT stands for Memory Cleanup Threshold. > This patch also fixes a possible problem tracking the memory marked as > "reclaiming" in the memtable allocators and pool. Writes that complete > only after a memtable has been scheduled for flushing, did not report > their memory as reclaiming. Normally this should be a small value of no > consequence, but if the flushing tasks are blocked for a long period, > and there is a sufficient number of writes, or these writes use > a sufficiently large quantity of memory, this would cause the memtable > cleaning algorithm to schedule repeated flushing tasks because the used > memory is always > reclaiming memory + MCT. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16139) Safe Ring Membership Protocol
[ https://issues.apache.org/jira/browse/CASSANDRA-16139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203496#comment-17203496 ] Jeff Jirsa commented on CASSANDRA-16139: > Would you care to elaborate why? My high level goal here is to ensure we can > reliably add/remove/replace nodes to a cluster, and this seems to be > reasonably doable with consistent hashing as far as I understand. I'd love to > explore alternatives but I'd be interested in learning what requirements are > not fulfilled by the current architecture. Because it's a concept borrowed from the 2007 paper and never reconsidered and it has ALL SORTS of unpleasant failure realities, and we can do better in 2021. For example: why, when a single machine fails in a datacenter, and the rest of the hosts detect the failure, does the database do nothing to re-replicate that data, instead forcing a user to come along and run some magic commands that literally only a handful of people actually understand, when the database COULD do it all on its own without humans in the loop? Why would we rely on humans assigning tokens, anyway, or static token assignment, when the database can see imbalance, and could potentially deal with imbalance on its own? The whole existence of vnodes should have been a red flag that tokens as a distribution mechanism were flawed. Tokens are a simplistic concept that are easy to reason about but horrible to use. If we're rewriting it, please take the time to research how other distributed databases move data around when there's a hot shard or a lost shard, because it's a meaningful and critical missing part of Cassandra. > Safe Ring Membership Protocol > - > > Key: CASSANDRA-16139 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16139 > Project: Cassandra > Issue Type: Improvement > Components: Cluster/Gossip, Cluster/Membership >Reporter: Paulo Motta >Assignee: Paulo Motta >Priority: Normal > > This ticket presents a practical protocol for performing safe ring membership > updates in Cassandra. This protocol will enable reliable concurrent ring > membership updates. > The proposed protocol is composed of the following macro-steps: > *PROPOSE:* An initiator node wanting to make updates to the current ring > structure (such as joining, leaving the ring or changing token assignments) > must propose the change to the other members of the ring (cohort). > *ACCEPT:* Upon receiving a proposal the other ring members determine if the > change is compatible with their local version of the ring, and if so, they > promise to accept the change proposed by the initiator. The ring members do > not accept proposals if they had already promised to honor another proposal, > to avoid conflicting ring membership updates. > *COMMIT:* Once the initiator receives acceptances from all the nodes in the > cohort, it commits the proposal by broadcasting the proposed ring delta via > gossip. Upon receiving these changes, the other members of the cohort apply > the delta to their local version of the ring and broadcast their new computed > version via gossip. The initiator concludes the ring membership update > operation by checking that all nodes agree on the new proposed version. > *ABORT:* A proposal not accepted by all members of the cohort may be > automatically aborted by the initiator or manually via a command line tool. > For simplicity the protocol above requires that all nodes are up during the > proposal step, but it should be possible to optimize it to require only a > quorum of nodes up to perform ring changes. > A python pseudo-code of the protocol is available > [here|https://gist.github.com/pauloricardomg/1930c8cf645aa63387a57bb57f79a0f7#file-safe_ring_membership-py]. > With the abstraction above it becomes very simple to perform ring change > operations: > * > [bootstrap|https://gist.github.com/pauloricardomg/1930c8cf645aa63387a57bb57f79a0f7#file-bootstrap-py] > * > [replace|https://gist.github.com/pauloricardomg/1930c8cf645aa63387a57bb57f79a0f7#file-replace-py] > * > [move|https://gist.github.com/pauloricardomg/1930c8cf645aa63387a57bb57f79a0f7#file-move-py] > * [remove > node|https://gist.github.com/pauloricardomg/1930c8cf645aa63387a57bb57f79a0f7#file-remove_node-py] > * [remove > token|https://gist.github.com/pauloricardomg/1930c8cf645aa63387a57bb57f79a0f7#file-remove_token-py] > h4. Token Ring Data Structure > The token ring data structure can be seen as a [Delta State Replicated Data > Type|https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type#State-based_CRDTs] > (Delta CRDT) containing the state of all (virtual) nodes in the cluster > where updates to the ring are operations on this CRDT. > Each member publishes its latest local accepted state (delta state)
[jira] [Comment Edited] (CASSANDRA-16139) Safe Ring Membership Protocol
[
https://issues.apache.org/jira/browse/CASSANDRA-16139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17203496#comment-17203496
]
Jeff Jirsa edited comment on CASSANDRA-16139 at 9/28/20, 8:09 PM:
--
{quote}Would you care to elaborate why? My high level goal here is to ensure we
can reliably add/remove/replace nodes to a cluster, and this seems to be
reasonably doable with consistent hashing as far as I understand. I'd love to
explore alternatives but I'd be interested in learning what requirements are
not fulfilled by the current architecture.{quote}
Because it's a concept borrowed from the 2007 paper and never reconsidered and
it has ALL SORTS of unpleasant failure realities, and we can do better in 2021.
For example: why, when a single machine fails in a datacenter, and the rest of
the hosts detect the failure, does the database do nothing to re-replicate that
data, instead forcing a user to come along and run some magic commands that
literally only a handful of people actually understand, when the database COULD
do it all on its own without humans in the loop? Why would we rely on humans
assigning tokens, anyway, or static token assignment, when the database can see
imbalance, and could potentially deal with imbalance on its own? The whole
existence of vnodes should have been a red flag that tokens as a distribution
mechanism were flawed.
Tokens are a simplistic concept that are easy to reason about but horrible to
use. If we're rewriting it, please take the time to research how other
distributed databases move data around when there's a hot shard or a lost
shard, because it's a meaningful and critical missing part of Cassandra.
was (Author: jjirsa):
> Would you care to elaborate why? My high level goal here is to ensure we can
> reliably add/remove/replace nodes to a cluster, and this seems to be
> reasonably doable with consistent hashing as far as I understand. I'd love to
> explore alternatives but I'd be interested in learning what requirements are
> not fulfilled by the current architecture.
Because it's a concept borrowed from the 2007 paper and never reconsidered and
it has ALL SORTS of unpleasant failure realities, and we can do better in 2021.
For example: why, when a single machine fails in a datacenter, and the rest of
the hosts detect the failure, does the database do nothing to re-replicate that
data, instead forcing a user to come along and run some magic commands that
literally only a handful of people actually understand, when the database COULD
do it all on its own without humans in the loop? Why would we rely on humans
assigning tokens, anyway, or static token assignment, when the database can see
imbalance, and could potentially deal with imbalance on its own? The whole
existence of vnodes should have been a red flag that tokens as a distribution
mechanism were flawed.
Tokens are a simplistic concept that are easy to reason about but horrible to
use. If we're rewriting it, please take the time to research how other
distributed databases move data around when there's a hot shard or a lost
shard, because it's a meaningful and critical missing part of Cassandra.
> Safe Ring Membership Protocol
> -
>
> Key: CASSANDRA-16139
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16139
> Project: Cassandra
> Issue Type: Improvement
> Components: Cluster/Gossip, Cluster/Membership
>Reporter: Paulo Motta
>Assignee: Paulo Motta
>Priority: Normal
>
> This ticket presents a practical protocol for performing safe ring membership
> updates in Cassandra. This protocol will enable reliable concurrent ring
> membership updates.
> The proposed protocol is composed of the following macro-steps:
> *PROPOSE:* An initiator node wanting to make updates to the current ring
> structure (such as joining, leaving the ring or changing token assignments)
> must propose the change to the other members of the ring (cohort).
> *ACCEPT:* Upon receiving a proposal the other ring members determine if the
> change is compatible with their local version of the ring, and if so, they
> promise to accept the change proposed by the initiator. The ring members do
> not accept proposals if they had already promised to honor another proposal,
> to avoid conflicting ring membership updates.
> *COMMIT:* Once the initiator receives acceptances from all the nodes in the
> cohort, it commits the proposal by broadcasting the proposed ring delta via
> gossip. Upon receiving these changes, the other members of the cohort apply
> the delta to their local version of the ring and broadcast their new computed
> version via gossip. The initiator concludes the ring membership update
> operation by checking that all nodes agree on the new proposed
[jira] [Commented] (CASSANDRA-16139) Safe Ring Membership Protocol
[ https://issues.apache.org/jira/browse/CASSANDRA-16139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17202742#comment-17202742 ] Jeff Jirsa commented on CASSANDRA-16139: If we're rewriting all of the membership/ownership code, we should definitely be thinking about a world that isn't based on tokens and hash tables. > Safe Ring Membership Protocol > - > > Key: CASSANDRA-16139 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16139 > Project: Cassandra > Issue Type: Improvement > Components: Cluster/Gossip, Cluster/Membership >Reporter: Paulo Motta >Assignee: Paulo Motta >Priority: Normal > > This ticket presents a practical protocol for performing safe ring membership > updates in Cassandra. This protocol will enable reliable concurrent ring > membership updates. > The proposed protocol is composed of the following macro-steps: > *PROPOSE:* An initiator node wanting to make updates to the current ring > structure (such as joining, leaving the ring or changing token assignments) > must propose the change to the other members of the ring (cohort). > *ACCEPT:* Upon receiving a proposal the other ring members determine if the > change is compatible with their local version of the ring, and if so, they > promise to accept the change proposed by the initiator. The ring members do > not accept proposals if they had already promised to honor another proposal, > to avoid conflicting ring membership updates. > *COMMIT:* Once the initiator receives acceptances from all the nodes in the > cohort, it commits the proposal by broadcasting the proposed ring delta via > gossip. Upon receiving these changes, the other members of the cohort apply > the delta to their local version of the ring and broadcast their new computed > version via gossip. The initiator concludes the ring membership update > operation by checking that all nodes agree on the new proposed version. > *ABORT:* A proposal not accepted by all members of the cohort may be > automatically aborted by the initiator or manually via a command line tool. > For simplicity the protocol above requires that all nodes are up during the > proposal step, but it should be possible to optimize it to require only a > quorum of nodes up to perform ring changes. > A python pseudo-code of the protocol is available > [here|https://gist.github.com/pauloricardomg/1930c8cf645aa63387a57bb57f79a0f7#file-safe_ring_membership-py]. > With the abstraction above it becomes very simple to perform ring change > operations: > * > [bootstrap|https://gist.github.com/pauloricardomg/1930c8cf645aa63387a57bb57f79a0f7#file-bootstrap-py] > * > [replace|https://gist.github.com/pauloricardomg/1930c8cf645aa63387a57bb57f79a0f7#file-replace-py] > * > [move|https://gist.github.com/pauloricardomg/1930c8cf645aa63387a57bb57f79a0f7#file-move-py] > * [remove > node|https://gist.github.com/pauloricardomg/1930c8cf645aa63387a57bb57f79a0f7#file-remove_node-py] > * [remove > token|https://gist.github.com/pauloricardomg/1930c8cf645aa63387a57bb57f79a0f7#file-remove_token-py] > h4. Token Ring Data Structure > The token ring data structure can be seen as a [Delta State Replicated Data > Type|https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type#State-based_CRDTs] > (Delta CRDT) containing the state of all (virtual) nodes in the cluster > where updates to the ring are operations on this CRDT. > Each member publishes its latest local accepted state (delta state) via > gossip and the union of all delta states comprise the global ring state. The > delta state must be commutative and idempotent to ensure all nodes will > eventually reach the same global state no matter the order received. > The content-addressed fingerprint of the global ring state uniquely > identifies the ring version and provides a simple way to verify agreement > between nodes in the cluster. Any change to the ring membership must be > agreed using the described protocol, ensuring that both conditions are met: > * All nodes have the same current view of the cluster before the update > (verified via the ring version fingerprint). > * All nodes have agreed to make the exact same update and not accept any > other update before the current proposed update is committed or aborted. > h4. Ring Convergence Time > Assuming there are no network partitions, the ring membership convergence > time will be dominated by the commit step since that is performed via gossip > broadcast. > The gossip broadcast is performed by sending the ring delta to the seed > nodes, since other nodes will contact seed nodes with a #seeds / #nodes > probability. This will define an upper bound for the maximum time it takes to > propagate a ring update that was accepted by all members of the ring. > On a cluster with 10%
[jira] [Commented] (CASSANDRA-15823) Support for networking via identity instead of IP
[ https://issues.apache.org/jira/browse/CASSANDRA-15823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200879#comment-17200879 ] Jeff Jirsa commented on CASSANDRA-15823: Will note that CASSANDRA-10134 likely solves some of the things I describe, and my knowledge was out of date because CASSANDRA-10134 landed in a version I don't pay attention to. > Support for networking via identity instead of IP > - > > Key: CASSANDRA-15823 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15823 > Project: Cassandra > Issue Type: Improvement >Reporter: Christopher Bradford >Priority: Normal > Labels: kubernetes > Attachments: consul-mesh-gateways.png, > istio-multicluster-with-gateways.svg, linkerd-service-mirroring.svg > > > TL;DR: Instead of mapping host ids to IPs, use hostnames. This allows > resolution to different IP addresses per DC that may then be forwarded to > nodes on remote networks without requiring node to node IP connectivity for > cross-dc links. > > This approach should not affect existing deployments as those could continue > to use IPs as the hostname and skip resolution. > > With orchestration platforms like Kubernetes and the usage of ephemeral > containers in environments today we should consider some changes to how we > handle the tracking of nodes and their network location. Currently we > maintain a mapping between host ids and IP addresses. > > With traditional infrastructure, if a node goes down it, usually, comes back > up with the same IP. In some environments this contract may be explicit with > virtual IPs that may move between hosts. In newer deployments, like on > Kubernetes, this contract is not possible. Pods (analogous to nodes) are > assigned an IP address at start time. Should the pod be restarted or > scheduled on a different host there is no guarantee we would have the same > IP. Cassandra is protected here as we already have logic in place to update > peers when we come up with the same host id, but a different IP address. > > There are ways to get Kubernetes to assign a specific IP per Pod. Most > recommendations involve the use of a service per pod. Communication with the > fixed service IP would automatically forward to the associated pod, > regardless of address. We _could_ use this approach, but it seems like this > would needlessly create a number of extra resources in our k8s cluster to get > around the problem. Which, to be fair, doesn't seem like much of a problem > with the aforementioned mitigations built into C*. > > So what is the _actual_ problem? *Cross-region, cross-cloud, > hybrid-deployment connectivity between pods is a pain.* This can be solved > with significant investment by those who want to deploy these types of > topologies. You can definitely configure connectivity between clouds over > dedicated connections, or VPN tunnels. With a big chunk of time insuring that > pod to pod connectivity just works even if those pods are managed by separate > control planes, but that again requires time and talent. There are a number > of edge cases to support between the ever so slight, but very important, > differences in cloud vendor networks. > > Recently there have been a number of innovations that aid in the deployment > and operation of these types of applications on Kubernetes. Service meshes > support distributed microservices running across multiple k8s cluster control > planes in disparate networks. Instead of directly connecting to IP addresses > of remote services instead they use a hostname. With this approach, hostname > traffic may then be routed to a proxy that sends traffic over the WAN > (sometimes with mTLS) to another proxy pod in the remote cluster which then > forwards the data along to the correct pod in that network. (See attached > diagrams) > > Which brings us to the point of this ticket. Instead of mapping host ids to > IPs, use hostnames (and update the underlying address periodically instead of > caching indefinitely). This allows resolution to different IP addresses per > DC (k8s cluster) that may then be forwarded to nodes (pods) on remote > networks (k8s clusters) without requiring node to node (pod to pod) IP > connectivity between them. Traditional deployments can still function like > they do today (even if operators opt to keep using IPs as identifiers instead > of hostnames). This proxy approach is then enabled like those we see in > service meshes. > > _Notes_ > C* already has the concept of broadcast addresses vs those which are bound on > the node. This approach _could_ be leveraged to provide the behavior we're > looking for, but then the broadcast values would need to be pre-computed > _*and match*_ across all k8s control planes. By
[jira] [Commented] (CASSANDRA-15823) Support for networking via identity instead of IP
[
https://issues.apache.org/jira/browse/CASSANDRA-15823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200519#comment-17200519
]
Jeff Jirsa commented on CASSANDRA-15823:
{quote}
If a node comes up with a completely brand new IP that was never part of the
cluster before, then we do not have any issues that I know of. Do you know of
any problems that can happen for that case Jeff Jirsa ?
{quote}
Think this is true, yes
{quote}
While I agree it would be best to have all membership based on UUID's, I think
we need to allow people to have hostnames be the contact point, and have those
re-resolve on every "connect". While I agree "ips are best, dns is the devil",
I have seen bad DNS take down clusters, there are many systems being created
right now where hostnames are the invariant, not ips, and we need Cassandra to
be able to play in those environments.
{quote}
If someone decides to completely redesign all of the internals to make it work
by dns name instead of hostID, I think that would be a pretty horrible mistake.
Yes, working by DNS should work, but things like ownership need to get to the
point that we're referring to the (known unique) host IDs instead of just
slapping a bandaid on top of the IP problem.
> Support for networking via identity instead of IP
> -
>
> Key: CASSANDRA-15823
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15823
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Christopher Bradford
>Priority: Normal
> Labels: kubernetes
> Attachments: consul-mesh-gateways.png,
> istio-multicluster-with-gateways.svg, linkerd-service-mirroring.svg
>
>
> TL;DR: Instead of mapping host ids to IPs, use hostnames. This allows
> resolution to different IP addresses per DC that may then be forwarded to
> nodes on remote networks without requiring node to node IP connectivity for
> cross-dc links.
>
> This approach should not affect existing deployments as those could continue
> to use IPs as the hostname and skip resolution.
>
> With orchestration platforms like Kubernetes and the usage of ephemeral
> containers in environments today we should consider some changes to how we
> handle the tracking of nodes and their network location. Currently we
> maintain a mapping between host ids and IP addresses.
>
> With traditional infrastructure, if a node goes down it, usually, comes back
> up with the same IP. In some environments this contract may be explicit with
> virtual IPs that may move between hosts. In newer deployments, like on
> Kubernetes, this contract is not possible. Pods (analogous to nodes) are
> assigned an IP address at start time. Should the pod be restarted or
> scheduled on a different host there is no guarantee we would have the same
> IP. Cassandra is protected here as we already have logic in place to update
> peers when we come up with the same host id, but a different IP address.
>
> There are ways to get Kubernetes to assign a specific IP per Pod. Most
> recommendations involve the use of a service per pod. Communication with the
> fixed service IP would automatically forward to the associated pod,
> regardless of address. We _could_ use this approach, but it seems like this
> would needlessly create a number of extra resources in our k8s cluster to get
> around the problem. Which, to be fair, doesn't seem like much of a problem
> with the aforementioned mitigations built into C*.
>
> So what is the _actual_ problem? *Cross-region, cross-cloud,
> hybrid-deployment connectivity between pods is a pain.* This can be solved
> with significant investment by those who want to deploy these types of
> topologies. You can definitely configure connectivity between clouds over
> dedicated connections, or VPN tunnels. With a big chunk of time insuring that
> pod to pod connectivity just works even if those pods are managed by separate
> control planes, but that again requires time and talent. There are a number
> of edge cases to support between the ever so slight, but very important,
> differences in cloud vendor networks.
>
> Recently there have been a number of innovations that aid in the deployment
> and operation of these types of applications on Kubernetes. Service meshes
> support distributed microservices running across multiple k8s cluster control
> planes in disparate networks. Instead of directly connecting to IP addresses
> of remote services instead they use a hostname. With this approach, hostname
> traffic may then be routed to a proxy that sends traffic over the WAN
> (sometimes with mTLS) to another proxy pod in the remote cluster which then
> forwards the data along to the correct pod in that network. (See attached
> diagrams)
>
> Which brings us to the point of this ticket.
[jira] [Updated] (CASSANDRA-16126) Blog post on Cassandra Usage Report 2020
[ https://issues.apache.org/jira/browse/CASSANDRA-16126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-16126: --- Reviewers: Nate McCall, Sankalp Kohli (was: Jeff Jirsa, Nate McCall, Sankalp Kohli) > Blog post on Cassandra Usage Report 2020 > > > Key: CASSANDRA-16126 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16126 > Project: Cassandra > Issue Type: Task > Components: Documentation/Blog >Reporter: Melissa Logan >Assignee: Melissa Logan >Priority: Normal > Fix For: 4.0-beta > > > Blog post on the 2020 Cassandra Usage Report. > ML: > [https://lists.apache.org/thread.html/r8a91b1d63079739c8dfbdab1833c66d58326de91a659afa3fb2a41a7%40%3Cdev.cassandra.apache.org%3E] > > PR: [https://github.com/apache/cassandra-website/pull/21] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13994) Remove COMPACT STORAGE internals before 4.0 release
[
https://issues.apache.org/jira/browse/CASSANDRA-13994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17129473#comment-17129473
]
Jeff Jirsa commented on CASSANDRA-13994:
I promise you lots of people are still running native proto v3, please don't
remove it this far into the alphas.
> Remove COMPACT STORAGE internals before 4.0 release
> ---
>
> Key: CASSANDRA-13994
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13994
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Local Write-Read Paths
>Reporter: Alex Petrov
>Assignee: Ekaterina Dimitrova
>Priority: Low
> Fix For: 4.0, 4.0-rc
>
>
> 4.0 comes without thrift (after [CASSANDRA-5]) and COMPACT STORAGE (after
> [CASSANDRA-10857]), and since Compact Storage flags are now disabled, all of
> the related functionality is useless.
> There are still some things to consider:
> 1. One of the system tables (built indexes) was compact. For now, we just
> added {{value}} column to it to make sure it's backwards-compatible, but we
> might want to make sure it's just a "normal" table and doesn't have redundant
> columns.
> 2. Compact Tables were building indexes in {{KEYS}} mode. Removing it is
> trivial, but this would mean that all built indexes will be defunct. We could
> log a warning for now and ask users to migrate off those for now and
> completely remove it from future releases. It's just a couple of classes
> though.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15862) Use "allow list" or "safe list" instead of the term "whitelist"
[
https://issues.apache.org/jira/browse/CASSANDRA-15862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128645#comment-17128645
]
Jeff Jirsa commented on CASSANDRA-15862:
Worth noting that while the python api is not in apache cassandra, that doesn't
mean there's not room to fix things here:
{code}
jjirsa:cassandra jjirsa$ grep -ir whitelist src/ | wc -l
19
jjirsa:cassandra jjirsa$ grep -ir blacklist src/ | wc -l
65
jjirsa:cassandra jjirsa$
{code}
> Use "allow list" or "safe list" instead of the term "whitelist"
>
>
> Key: CASSANDRA-15862
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15862
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Ash Berlin-Taylor
>Priority: Normal
>
> Language matters. I'd like to remove all references in Apache Airflow to
> whitelist or black list, and the Cassandra Python API has some that we can't
> easily remove.
> The recent global events have made this even more relevant, but this has been
> on my radar for a while now. Here is a well written article for why I think
> it matters
> https://www.ncsc.gov.uk/blog-post/terminology-its-not-black-and-white
> {quote}It's fairly common to say whitelisting and blacklisting to describe
> desirable and undesirable things in cyber security.
> However, there's an issue with the terminology. It only makes sense if you
> equate white with 'good, permitted, safe' and black with 'bad, dangerous,
> forbidden'. There are some obvious problems with this. {quote}
> My exposure to is via the Python API where there is the
> cassandra.pollicies.WhiteListRoundRobinPolicy class. I propose that this be
> renamed to AllowListRoundRobinPolicy instead. I do not know if there are
> other references.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13618) CassandraRoleManager setup task improvement
[
https://issues.apache.org/jira/browse/CASSANDRA-13618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119111#comment-17119111
]
Jeff Jirsa commented on CASSANDRA-13618:
[~samt] i have no idea if this patch matters anymore. Not in a position to
drive it if it does. Just FYI.
> CassandraRoleManager setup task improvement
> ---
>
> Key: CASSANDRA-13618
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13618
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Authorization
>Reporter: Jeff Jirsa
>Assignee: Jeff Jirsa
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> {{CassandraRoleManager}} blocks some functionality during setup, using a
> delay added in CASSANDRA-9761 . Unfortunately, this setup is scheduled for
> 10s after startup, and may not be necessary, meaning immediately after
> startup some auth related queries may not behave as intended. We can skip
> this delay without any additional risk.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-13618) CassandraRoleManager setup task improvement
[
https://issues.apache.org/jira/browse/CASSANDRA-13618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa reassigned CASSANDRA-13618:
--
Assignee: (was: Jeff Jirsa)
> CassandraRoleManager setup task improvement
> ---
>
> Key: CASSANDRA-13618
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13618
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/Authorization
>Reporter: Jeff Jirsa
>Priority: Normal
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> {{CassandraRoleManager}} blocks some functionality during setup, using a
> delay added in CASSANDRA-9761 . Unfortunately, this setup is scheduled for
> 10s after startup, and may not be necessary, meaning immediately after
> startup some auth related queries may not behave as intended. We can skip
> this delay without any additional risk.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Issue Comment Deleted] (CASSANDRA-13313) Compaction leftovers not removed on upgrade 2.1/2.2 -> 3.0
[ https://issues.apache.org/jira/browse/CASSANDRA-13313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-13313: --- Comment: was deleted (was: Reopening, patch coming soon. ) > Compaction leftovers not removed on upgrade 2.1/2.2 -> 3.0 > -- > > Key: CASSANDRA-13313 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13313 > Project: Cassandra > Issue Type: Bug > Components: Local/Compaction >Reporter: Jeff Jirsa >Assignee: Jeff Jirsa >Priority: Normal > Labels: Correctness, remove-reopen > Fix For: 3.0.x, 3.11.x > > > Before 3.0 we used sstable ancestors to figure out if an sstable was left > over after a compaction. In 3.0 the ancestors are ignored and instead we use > LogTransaction files to figure it out. 3.0 should still clean up 2.1/2.2 > compaction leftovers using the on-disk sstable ancestors when available. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13313) Compaction leftovers not removed on upgrade 2.1/2.2 -> 3.0
[ https://issues.apache.org/jira/browse/CASSANDRA-13313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-13313: --- Authors: (was: Jeff Jirsa) > Compaction leftovers not removed on upgrade 2.1/2.2 -> 3.0 > -- > > Key: CASSANDRA-13313 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13313 > Project: Cassandra > Issue Type: Bug > Components: Local/Compaction >Reporter: Jeff Jirsa >Assignee: Jeff Jirsa >Priority: Normal > Labels: Correctness, remove-reopen > Fix For: 3.0.x, 3.11.x > > > Before 3.0 we used sstable ancestors to figure out if an sstable was left > over after a compaction. In 3.0 the ancestors are ignored and instead we use > LogTransaction files to figure it out. 3.0 should still clean up 2.1/2.2 > compaction leftovers using the on-disk sstable ancestors when available. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13313) Compaction leftovers not removed on upgrade 2.1/2.2 -> 3.0
[ https://issues.apache.org/jira/browse/CASSANDRA-13313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-13313: --- Resolution: Won't Fix Status: Resolved (was: Open) > Compaction leftovers not removed on upgrade 2.1/2.2 -> 3.0 > -- > > Key: CASSANDRA-13313 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13313 > Project: Cassandra > Issue Type: Bug > Components: Local/Compaction >Reporter: Jeff Jirsa >Priority: Normal > Labels: Correctness, remove-reopen > Fix For: 3.0.x, 3.11.x > > > Before 3.0 we used sstable ancestors to figure out if an sstable was left > over after a compaction. In 3.0 the ancestors are ignored and instead we use > LogTransaction files to figure it out. 3.0 should still clean up 2.1/2.2 > compaction leftovers using the on-disk sstable ancestors when available. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-13313) Compaction leftovers not removed on upgrade 2.1/2.2 -> 3.0
[ https://issues.apache.org/jira/browse/CASSANDRA-13313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa reassigned CASSANDRA-13313: -- Assignee: (was: Jeff Jirsa) > Compaction leftovers not removed on upgrade 2.1/2.2 -> 3.0 > -- > > Key: CASSANDRA-13313 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13313 > Project: Cassandra > Issue Type: Bug > Components: Local/Compaction >Reporter: Jeff Jirsa >Priority: Normal > Labels: Correctness, remove-reopen > Fix For: 3.0.x, 3.11.x > > > Before 3.0 we used sstable ancestors to figure out if an sstable was left > over after a compaction. In 3.0 the ancestors are ignored and instead we use > LogTransaction files to figure it out. 3.0 should still clean up 2.1/2.2 > compaction leftovers using the on-disk sstable ancestors when available. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Issue Comment Deleted] (CASSANDRA-6288) Make compaction a priority queue
[ https://issues.apache.org/jira/browse/CASSANDRA-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-6288: -- Comment: was deleted (was: Linking - If compaction uses a priorityqueue, we should consider both the issue in 11218 (prioritize certain types of compaction over others) and 6288 (prioritize typical compaction based on CF characteristics).) > Make compaction a priority queue > > > Key: CASSANDRA-6288 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6288 > Project: Cassandra > Issue Type: Improvement > Components: Local/Compaction >Reporter: Jonathan Ellis >Priority: Low > Labels: compaction > > We should prioritize compacting CFs by how many reads/s its preferred > candidate would save, divided by the number of bytes in the sstables. > (Note that STCS currently divides by number of keys; ISTM that bytes will > work better since that does not penalize narrow rows.) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-12200) Backlogged compactions can make repair on trivially small tables waiting for a long time to finish
[
https://issues.apache.org/jira/browse/CASSANDRA-12200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa reassigned CASSANDRA-12200:
--
Assignee: (was: Jeff Jirsa)
> Backlogged compactions can make repair on trivially small tables waiting for
> a long time to finish
> --
>
> Key: CASSANDRA-12200
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12200
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Core
>Reporter: Wei Deng
>Priority: Normal
>
> In C* 3.0 we started to use incremental repair by default. However, this
> seems to create a repair performance problem if you have a relatively
> write-heavy workload that can drive all available concurrent_compactors to be
> used by active compactions.
> I was able to demonstrate this issue by the following scenario:
> 1. On a three-node C* 3.0.7 cluster, use "cassandra-stress write n=1"
> to generate 100GB of data with keyspace1.standard1 table using LCS (ctrl+c
> the stress client once the data size on each node reaches 35+GB).
> 2. At this point, there will be hundreds of L0 SSTables waiting for LCS to
> digest on each node, and with concurrent_compactors set to default at 2, the
> two compaction threads are constantly busy processing the backlogged L0
> SSTables.
> 3. Now create a new keyspace called "trivial_ks" with RF=3 and create a small
> two-column CQL table in it, and insert 6 records.
> 4. Start a "nodetool repair trivial_ks" session on one of the nodes, and
> watch the following behavior:
> {noformat}
> automaton@wdengdse50google-98425b985-3:~$ nodetool repair trivial_ks
> [2016-07-13 01:57:28,364] Starting repair command #1, repairing keyspace
> trivial_ks with repair options (parallelism: parallel, primary range: false,
> incremental: true, job threads: 1, ColumnFamilies: [], dataCenters: [],
> hosts: [], # of ranges: 3)
> [2016-07-13 01:57:31,027] Repair session 27212dd0-489d-11e6-a6d6-cd06faa0aaa2
> for range [(3074457345618258602,-9223372036854775808],
> (-9223372036854775808,-3074457345618258603],
> (-3074457345618258603,3074457345618258602]] finished (progress: 66%)
> [2016-07-13 02:07:47,637] Repair completed successfully
> [2016-07-13 02:07:47,657] Repair command #1 finished in 10 minutes 19 seconds
> {noformat}
> Basically for such a small table it took 10+ minutes to finish the repair.
> Looking at debug.log for this particular repair session UUID, you will find
> that all nodes were able to pass through validation compaction within 15ms,
> but one of the nodes actually got stuck waiting for a compaction slot because
> it has to do an anti-compaction step before it can finally tell the
> initiating node that it's done with its part of the repair session, so it
> took 10+ minutes for one compaction slot to be freed up, like shown in the
> following debug.log entries:
> {noformat}
> DEBUG [AntiEntropyStage:1] 2016-07-13 01:57:30,956
> RepairMessageVerbHandler.java:149 - Got anticompaction request
> AnticompactionRequest{parentRepairSession=27103de0-489d-11e6-a6d6-cd06faa0aaa2}
> org.apache.cassandra.repair.messages.AnticompactionRequest@34449ff4
> <...>
>
> <...>
> DEBUG [CompactionExecutor:5] 2016-07-13 02:07:47,506 CompactionTask.java:217
> - Compacted (286609e0-489d-11e6-9e03-1fd69c5ec46c) 32 sstables to
> [/var/lib/cassandra/data/keyspace1/standard1-9c02e9c1487c11e6b9161dbd340a212f/mb-499-big,]
> to level=0. 2,892,058,050 bytes to 2,874,333,820 (~99% of original) in
> 616,880ms = 4.443617MB/s. 0 total partitions merged to 12,233,340.
> Partition merge counts were {1:12086760, 2:146580, }
> INFO [CompactionExecutor:5] 2016-07-13 02:07:47,512
> CompactionManager.java:511 - Starting anticompaction for trivial_ks.weitest
> on
> 1/[BigTableReader(path='/var/lib/cassandra/data/trivial_ks/weitest-538b07d1489b11e6a9ef61c6ff848952/mb-1-big-Data.db')]
> sstables
> INFO [CompactionExecutor:5] 2016-07-13 02:07:47,513
> CompactionManager.java:540 - SSTable
> BigTableReader(path='/var/lib/cassandra/data/trivial_ks/weitest-538b07d1489b11e6a9ef61c6ff848952/mb-1-big-Data.db')
> fully contained in range (-9223372036854775808,-9223372036854775808],
> mutating repairedAt instead of anticompacting
> INFO [CompactionExecutor:5] 2016-07-13 02:07:47,570
> CompactionManager.java:578 - Completed anticompaction successfully
> {noformat}
> Since validation compaction has its own threads outside of the regular
> compaction thread pool restricted by concurrent_compactors, we were able to
> pass through validation compaction without any issue. If we could treat
> anti-compaction the same way (i.e. to give it its own thread pool), we can
> avoid this kind of repair performance problem from happening.
--
This message
[jira] [Assigned] (CASSANDRA-6288) Make compaction a priority queue
[ https://issues.apache.org/jira/browse/CASSANDRA-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa reassigned CASSANDRA-6288: - Assignee: (was: Jeff Jirsa) > Make compaction a priority queue > > > Key: CASSANDRA-6288 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6288 > Project: Cassandra > Issue Type: Improvement > Components: Local/Compaction >Reporter: Jonathan Ellis >Priority: Low > Labels: compaction > > We should prioritize compacting CFs by how many reads/s its preferred > candidate would save, divided by the number of bytes in the sstables. > (Note that STCS currently divides by number of keys; ISTM that bytes will > work better since that does not penalize narrow rows.) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Issue Comment Deleted] (CASSANDRA-12200) Backlogged compactions can make repair on trivially small tables waiting for a long time to finish
[
https://issues.apache.org/jira/browse/CASSANDRA-12200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-12200:
---
Comment: was deleted
(was: Solution here could be a PriorityQueue for compaction, as discussed in
CASSANDRA-11218, and then prioritizing anticompaction in the same manner we
need to prioritize index builds, user defined compaction, etc.
)
> Backlogged compactions can make repair on trivially small tables waiting for
> a long time to finish
> --
>
> Key: CASSANDRA-12200
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12200
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Core
>Reporter: Wei Deng
>Assignee: Jeff Jirsa
>Priority: Normal
>
> In C* 3.0 we started to use incremental repair by default. However, this
> seems to create a repair performance problem if you have a relatively
> write-heavy workload that can drive all available concurrent_compactors to be
> used by active compactions.
> I was able to demonstrate this issue by the following scenario:
> 1. On a three-node C* 3.0.7 cluster, use "cassandra-stress write n=1"
> to generate 100GB of data with keyspace1.standard1 table using LCS (ctrl+c
> the stress client once the data size on each node reaches 35+GB).
> 2. At this point, there will be hundreds of L0 SSTables waiting for LCS to
> digest on each node, and with concurrent_compactors set to default at 2, the
> two compaction threads are constantly busy processing the backlogged L0
> SSTables.
> 3. Now create a new keyspace called "trivial_ks" with RF=3 and create a small
> two-column CQL table in it, and insert 6 records.
> 4. Start a "nodetool repair trivial_ks" session on one of the nodes, and
> watch the following behavior:
> {noformat}
> automaton@wdengdse50google-98425b985-3:~$ nodetool repair trivial_ks
> [2016-07-13 01:57:28,364] Starting repair command #1, repairing keyspace
> trivial_ks with repair options (parallelism: parallel, primary range: false,
> incremental: true, job threads: 1, ColumnFamilies: [], dataCenters: [],
> hosts: [], # of ranges: 3)
> [2016-07-13 01:57:31,027] Repair session 27212dd0-489d-11e6-a6d6-cd06faa0aaa2
> for range [(3074457345618258602,-9223372036854775808],
> (-9223372036854775808,-3074457345618258603],
> (-3074457345618258603,3074457345618258602]] finished (progress: 66%)
> [2016-07-13 02:07:47,637] Repair completed successfully
> [2016-07-13 02:07:47,657] Repair command #1 finished in 10 minutes 19 seconds
> {noformat}
> Basically for such a small table it took 10+ minutes to finish the repair.
> Looking at debug.log for this particular repair session UUID, you will find
> that all nodes were able to pass through validation compaction within 15ms,
> but one of the nodes actually got stuck waiting for a compaction slot because
> it has to do an anti-compaction step before it can finally tell the
> initiating node that it's done with its part of the repair session, so it
> took 10+ minutes for one compaction slot to be freed up, like shown in the
> following debug.log entries:
> {noformat}
> DEBUG [AntiEntropyStage:1] 2016-07-13 01:57:30,956
> RepairMessageVerbHandler.java:149 - Got anticompaction request
> AnticompactionRequest{parentRepairSession=27103de0-489d-11e6-a6d6-cd06faa0aaa2}
> org.apache.cassandra.repair.messages.AnticompactionRequest@34449ff4
> <...>
>
> <...>
> DEBUG [CompactionExecutor:5] 2016-07-13 02:07:47,506 CompactionTask.java:217
> - Compacted (286609e0-489d-11e6-9e03-1fd69c5ec46c) 32 sstables to
> [/var/lib/cassandra/data/keyspace1/standard1-9c02e9c1487c11e6b9161dbd340a212f/mb-499-big,]
> to level=0. 2,892,058,050 bytes to 2,874,333,820 (~99% of original) in
> 616,880ms = 4.443617MB/s. 0 total partitions merged to 12,233,340.
> Partition merge counts were {1:12086760, 2:146580, }
> INFO [CompactionExecutor:5] 2016-07-13 02:07:47,512
> CompactionManager.java:511 - Starting anticompaction for trivial_ks.weitest
> on
> 1/[BigTableReader(path='/var/lib/cassandra/data/trivial_ks/weitest-538b07d1489b11e6a9ef61c6ff848952/mb-1-big-Data.db')]
> sstables
> INFO [CompactionExecutor:5] 2016-07-13 02:07:47,513
> CompactionManager.java:540 - SSTable
> BigTableReader(path='/var/lib/cassandra/data/trivial_ks/weitest-538b07d1489b11e6a9ef61c6ff848952/mb-1-big-Data.db')
> fully contained in range (-9223372036854775808,-9223372036854775808],
> mutating repairedAt instead of anticompacting
> INFO [CompactionExecutor:5] 2016-07-13 02:07:47,570
> CompactionManager.java:578 - Completed anticompaction successfully
> {noformat}
> Since validation compaction has its own threads outside of the regular
> compaction thread pool restricted by concurrent_compactors, we were able
[jira] [Assigned] (CASSANDRA-11218) Prioritize Secondary Index rebuild
[ https://issues.apache.org/jira/browse/CASSANDRA-11218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa reassigned CASSANDRA-11218: -- Assignee: (was: Jeff Jirsa) > Prioritize Secondary Index rebuild > -- > > Key: CASSANDRA-11218 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11218 > Project: Cassandra > Issue Type: Improvement > Components: Feature/2i Index, Local/Compaction >Reporter: Sankalp Kohli >Priority: Low > > We have seen that secondary index rebuild get stuck behind other compaction > during a bootstrap and other operations. This causes things to not finish. We > should prioritize index rebuild via a separate thread pool or using a > priority queue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-6288) Make compaction a priority queue
[ https://issues.apache.org/jira/browse/CASSANDRA-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-6288: -- Authors: (was: Jeff Jirsa) > Make compaction a priority queue > > > Key: CASSANDRA-6288 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6288 > Project: Cassandra > Issue Type: Improvement > Components: Local/Compaction >Reporter: Jonathan Ellis >Assignee: Jeff Jirsa >Priority: Low > Labels: compaction > > We should prioritize compacting CFs by how many reads/s its preferred > candidate would save, divided by the number of bytes in the sstables. > (Note that STCS currently divides by number of keys; ISTM that bytes will > work better since that does not penalize narrow rows.) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15823) Support for networking via identity instead of IP
[ https://issues.apache.org/jira/browse/CASSANDRA-15823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111724#comment-17111724 ] Jeff Jirsa commented on CASSANDRA-15823: > Cassandra is protected here as we already have logic in place to update peers > when we come up with the same host id, but a different IP address. This definitely isn’t true / strictly safe. In fact it’s trivial to violate consistency / lose data by swapping the IP of two Pods/instances on the same host. We really need everything to be based on UUIDs, not ip or port or host name. And we really really really shouldn’t assume that dns is universally available or correct (because that’s just not always true, even in 2020). > Support for networking via identity instead of IP > - > > Key: CASSANDRA-15823 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15823 > Project: Cassandra > Issue Type: Improvement >Reporter: Christopher Bradford >Priority: Normal > Attachments: consul-mesh-gateways.png, > istio-multicluster-with-gateways.svg, linkerd-service-mirroring.svg > > > TL;DR: Instead of mapping host ids to IPs, use hostnames. This allows > resolution to different IP addresses per DC that may then be forwarded to > nodes on remote networks without requiring node to node IP connectivity for > cross-dc links. > > This approach should not affect existing deployments as those could continue > to use IPs as the hostname and skip resolution. > > With orchestration platforms like Kubernetes and the usage of ephemeral > containers in environments today we should consider some changes to how we > handle the tracking of nodes and their network location. Currently we > maintain a mapping between host ids and IP addresses. > > With traditional infrastructure, if a node goes down it, usually, comes back > up with the same IP. In some environments this contract may be explicit with > virtual IPs that may move between hosts. In newer deployments, like on > Kubernetes, this contract is not possible. Pods (analogous to nodes) are > assigned an IP address at start time. Should the pod be restarted or > scheduled on a different host there is no guarantee we would have the same > IP. Cassandra is protected here as we already have logic in place to update > peers when we come up with the same host id, but a different IP address. > > There are ways to get Kubernetes to assign a specific IP per Pod. Most > recommendations involve the use of a service per pod. Communication with the > fixed service IP would automatically forward to the associated pod, > regardless of address. We _could_ use this approach, but it seems like this > would needlessly create a number of extra resources in our k8s cluster to get > around the problem. Which, to be fair, doesn't seem like much of a problem > with the aforementioned mitigations built into C*. > > So what is the _actual_ problem? *Cross-region, cross-cloud, > hybrid-deployment connectivity between pods is a pain.* This can be solved > with significant investment by those who want to deploy these types of > topologies. You can definitely configure connectivity between clouds over > dedicated connections, or VPN tunnels. With a big chunk of time insuring that > pod to pod connectivity just works even if those pods are managed by separate > control planes, but that again requires time and talent. There are a number > of edge cases to support between the ever so slight, but very important, > differences in cloud vendor networks. > > Recently there have been a number of innovations that aid in the deployment > and operation of these types of applications on Kubernetes. Service meshes > support distributed microservices running across multiple k8s cluster control > planes in disparate networks. Instead of directly connecting to IP addresses > of remote services instead they use a hostname. With this approach, hostname > traffic may then be routed to a proxy that sends traffic over the WAN > (sometimes with mTLS) to another proxy pod in the remote cluster which then > forwards the data along to the correct pod in that network. (See attached > diagrams) > > Which brings us to the point of this ticket. Instead of mapping host ids to > IPs, use hostnames (and update the underlying address periodically instead of > caching indefinitely). This allows resolution to different IP addresses per > DC (k8s cluster) that may then be forwarded to nodes (pods) on remote > networks (k8s clusters) without requiring node to node (pod to pod) IP > connectivity between them. Traditional deployments can still function like > they do today (even if operators opt to keep using IPs as identifiers instead > of hostnames). This proxy approach is then enabled like those we see
[jira] [Updated] (CASSANDRA-15816) Transports are stopped in the wrong order
[ https://issues.apache.org/jira/browse/CASSANDRA-15816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-15816: --- Source Control Link: https://github.com/apache/cassandra/pull/593 > Transports are stopped in the wrong order > - > > Key: CASSANDRA-15816 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15816 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Internode >Reporter: Jeff Jirsa >Assignee: Jeff Jirsa >Priority: Normal > Time Spent: 10m > Remaining Estimate: 0h > > Stopping gossip while native is running is almost always wrong, change the > order of shutdown and log a warning when done manually -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15816) Transports are stopped in the wrong order
[ https://issues.apache.org/jira/browse/CASSANDRA-15816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-15816: --- Test and Documentation Plan: N/A. Status: Patch Available (was: In Progress) > Transports are stopped in the wrong order > - > > Key: CASSANDRA-15816 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15816 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Internode >Reporter: Jeff Jirsa >Assignee: Jeff Jirsa >Priority: Normal > Time Spent: 10m > Remaining Estimate: 0h > > Stopping gossip while native is running is almost always wrong, change the > order of shutdown and log a warning when done manually -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15816) Transports are stopped in the wrong order
[ https://issues.apache.org/jira/browse/CASSANDRA-15816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-15816: --- Bug Category: Parent values: Availability(12983) Complexity: Low Hanging Fruit Discovered By: User Report Severity: Low Assignee: Jeff Jirsa Status: Open (was: Triage Needed) > Transports are stopped in the wrong order > - > > Key: CASSANDRA-15816 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15816 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Internode >Reporter: Jeff Jirsa >Assignee: Jeff Jirsa >Priority: Normal > Time Spent: 10m > Remaining Estimate: 0h > > Stopping gossip while native is running is almost always wrong, change the > order of shutdown and log a warning when done manually -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15817) Prevent repair from overrunning compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-15817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-15817: --- Test and Documentation Plan: Tested by hand; disabled by default (via very large value) and not included in cassandra.yaml or NEWS.txt Status: Patch Available (was: In Progress) > Prevent repair from overrunning compaction > -- > > Key: CASSANDRA-15817 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15817 > Project: Cassandra > Issue Type: Bug > Components: Consistency/Repair >Reporter: Jeff Jirsa >Assignee: Jeff Jirsa >Priority: Low > Time Spent: 10m > Remaining Estimate: 0h > > Repair can run and stream faster than a host can compact. At some point, if a > host is sufficiently out of sync, or compaction is especially expensive, it > makes sense to intentionally block repair so that compaction can catch up -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15817) Prevent repair from overrunning compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-15817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa reassigned CASSANDRA-15817: -- Bug Category: Parent values: Degradation(12984)Level 1 values: Resource Management(12995) Complexity: Low Hanging Fruit Discovered By: User Report Severity: Low Source Control Link: https://github.com/apache/cassandra/pull/592 Assignee: Jeff Jirsa > Prevent repair from overrunning compaction > -- > > Key: CASSANDRA-15817 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15817 > Project: Cassandra > Issue Type: Bug > Components: Consistency/Repair >Reporter: Jeff Jirsa >Assignee: Jeff Jirsa >Priority: Low > Time Spent: 10m > Remaining Estimate: 0h > > Repair can run and stream faster than a host can compact. At some point, if a > host is sufficiently out of sync, or compaction is especially expensive, it > makes sense to intentionally block repair so that compaction can catch up -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15817) Prevent repair from overrunning compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-15817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-15817: --- Status: Open (was: Triage Needed) > Prevent repair from overrunning compaction > -- > > Key: CASSANDRA-15817 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15817 > Project: Cassandra > Issue Type: Bug > Components: Consistency/Repair >Reporter: Jeff Jirsa >Assignee: Jeff Jirsa >Priority: Low > Time Spent: 10m > Remaining Estimate: 0h > > Repair can run and stream faster than a host can compact. At some point, if a > host is sufficiently out of sync, or compaction is especially expensive, it > makes sense to intentionally block repair so that compaction can catch up -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15817) Prevent repair from overrunning compaction
Jeff Jirsa created CASSANDRA-15817: -- Summary: Prevent repair from overrunning compaction Key: CASSANDRA-15817 URL: https://issues.apache.org/jira/browse/CASSANDRA-15817 Project: Cassandra Issue Type: Bug Components: Consistency/Repair Reporter: Jeff Jirsa Repair can run and stream faster than a host can compact. At some point, if a host is sufficiently out of sync, or compaction is especially expensive, it makes sense to intentionally block repair so that compaction can catch up -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15816) Transports are stopped in the wrong order
Jeff Jirsa created CASSANDRA-15816: -- Summary: Transports are stopped in the wrong order Key: CASSANDRA-15816 URL: https://issues.apache.org/jira/browse/CASSANDRA-15816 Project: Cassandra Issue Type: Bug Components: Messaging/Internode Reporter: Jeff Jirsa Stopping gossip while native is running is almost always wrong, change the order of shutdown and log a warning when done manually -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-10726) Read repair inserts should not be blocking
[
https://issues.apache.org/jira/browse/CASSANDRA-10726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17049745#comment-17049745
]
Jeff Jirsa edited comment on CASSANDRA-10726 at 3/2/20 11:34 PM:
-
[~jjordan]:
{quote}In 3.11 you should set both read_repair_chance and
dc_local_read_repair_chance to 0 on all your tables.
{quote}
That's not what this is about - this is in foreground read repair where one of
the queried replicas doesn't match.
{quote}If such things are common on your cluster, then you will want to
investigate the performance issues further and try to resolve them. This change
means your application will no longer be trying to wait for the replicas to be
up to date before the read returns
{quote}
It helps when disks are failing, and it helps when GC is high. Both of those
things happen in the real world, and this patch turns a failing machine into a
non-issue.
That said - backporting this patch is almost certainly not going to happen.
It's a large patch in a critical piece of code and it's not suitable for a
stable branch like 3.11. It's a meaningful, important patch, but probably
safest to wait for 4.0
was (Author: jjirsa):
[~jjordan]:
bq. In 3.11 you should set both read_repair_chance and
dc_local_read_repair_chance to 0 on all your tables.
That's not what this is about - this is in foreground read repair where one of
the queried replicas doesn't match.
bq. If such things are common on your cluster, then you will want to
investigate the performance issues further and try to resolve them. This change
means your application will no longer be trying to wait for the replicas to be
up to date before the read returns
It helps when disks are failing, and it helps when GC is high. Both of those
things happen in the real world, and this patch turns a failing machine into a
non-issue.
That said - backporting this patch is almost certainly not going to happen.
It's a large patch in a critical piece of code and it's not suitable for a
stable branch like 3.11. It's a meaningful, important patch, but probably
safest to wait for 3.11
> Read repair inserts should not be blocking
> --
>
> Key: CASSANDRA-10726
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10726
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Coordination
>Reporter: Richard Low
>Assignee: Blake Eggleston
>Priority: Normal
> Fix For: 4.0
>
>
> Today, if there’s a digest mismatch in a foreground read repair, the insert
> to update out of date replicas is blocking. This means, if it fails, the read
> fails with a timeout. If a node is dropping writes (maybe it is overloaded or
> the mutation stage is backed up for some other reason), all reads to a
> replica set could fail. Further, replicas dropping writes get more out of
> sync so will require more read repair.
> The comment on the code for why the writes are blocking is:
> {code}
> // wait for the repair writes to be acknowledged, to minimize impact on any
> replica that's
> // behind on writes in case the out-of-sync row is read multiple times in
> quick succession
> {code}
> but the bad side effect is that reads timeout. Either the writes should not
> be blocking or we should return success for the read even if the write times
> out.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-10726) Read repair inserts should not be blocking
[
https://issues.apache.org/jira/browse/CASSANDRA-10726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17049745#comment-17049745
]
Jeff Jirsa commented on CASSANDRA-10726:
[~jjordan]:
bq. In 3.11 you should set both read_repair_chance and
dc_local_read_repair_chance to 0 on all your tables.
That's not what this is about - this is in foreground read repair where one of
the queried replicas doesn't match.
bq. If such things are common on your cluster, then you will want to
investigate the performance issues further and try to resolve them. This change
means your application will no longer be trying to wait for the replicas to be
up to date before the read returns
It helps when disks are failing, and it helps when GC is high. Both of those
things happen in the real world, and this patch turns a failing machine into a
non-issue.
That said - backporting this patch is almost certainly not going to happen.
It's a large patch in a critical piece of code and it's not suitable for a
stable branch like 3.11. It's a meaningful, important patch, but probably
safest to wait for 3.11
> Read repair inserts should not be blocking
> --
>
> Key: CASSANDRA-10726
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10726
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Coordination
>Reporter: Richard Low
>Assignee: Blake Eggleston
>Priority: Normal
> Fix For: 4.0
>
>
> Today, if there’s a digest mismatch in a foreground read repair, the insert
> to update out of date replicas is blocking. This means, if it fails, the read
> fails with a timeout. If a node is dropping writes (maybe it is overloaded or
> the mutation stage is backed up for some other reason), all reads to a
> replica set could fail. Further, replicas dropping writes get more out of
> sync so will require more read repair.
> The comment on the code for why the writes are blocking is:
> {code}
> // wait for the repair writes to be acknowledged, to minimize impact on any
> replica that's
> // behind on writes in case the out-of-sync row is read multiple times in
> quick succession
> {code}
> but the bad side effect is that reads timeout. Either the writes should not
> be blocking or we should return success for the read even if the write times
> out.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15352) Replica failure propagation to coordinator and client
[ https://issues.apache.org/jira/browse/CASSANDRA-15352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17042161#comment-17042161 ] Jeff Jirsa commented on CASSANDRA-15352: I don't understand this JIRA. Can someone summarize exactly what condition you're trying to improve? > Replica failure propagation to coordinator and client > - > > Key: CASSANDRA-15352 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15352 > Project: Cassandra > Issue Type: New Feature > Components: Messaging/Internode >Reporter: Alex Petrov >Priority: Normal > > We should add early reporting of replica-side errors, since currently we just > time-out requests. On normal read-write path this is not that important, but > this is a protocol change we will need to improve cheap quorums for transient > replication. This might have potential positive impact for regular read-write > path, since we’ll be aborting queries early instead of timing them out. Can > be useful for failing / going away nodes (which is also one of the changes > we’re planning to implement). > We do have means for propagating error both in client protocol through > and in internode through FAILURE_RSP, which is true and we do not > have to extend the protocol to implement this change, but this is still a > change in protocol behavior, since we’ll be sending a message where we would > usually silently timeout. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13019) Improve clearsnapshot to delete the snapshot files slowly
[
https://issues.apache.org/jira/browse/CASSANDRA-13019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-13019:
---
Reviewers: Aleksey Yeschenko, Chris Lohfink, maxwellguo, Jeff Jirsa (was:
Aleksey Yeschenko, Chris Lohfink, Jeff Jirsa, maxwellguo)
Aleksey Yeschenko, Chris Lohfink, maxwellguo, Jeff Jirsa
Status: Review In Progress (was: Patch Available)
> Improve clearsnapshot to delete the snapshot files slowly
> --
>
> Key: CASSANDRA-13019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13019
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Core
>Reporter: Dikang Gu
>Assignee: Jeff Jirsa
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.x
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> In our environment, we are creating snapshots for backup, after we finish the
> backup, we are running {{clearsnapshot}} to delete the snapshot files. At
> that time we may have thousands of files to delete, and it's causing sudden
> disk usage spike. As a result, we are experiencing a spike of drop messages
> from Cassandra.
> I think we should implement something like {{slowrm}} to delete the snapshot
> files slowly, avoid the sudden disk usage spike.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13019) Improve clearsnapshot to delete the snapshot files slowly
[
https://issues.apache.org/jira/browse/CASSANDRA-13019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-13019:
---
Status: Ready to Commit (was: Review In Progress)
> Improve clearsnapshot to delete the snapshot files slowly
> --
>
> Key: CASSANDRA-13019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13019
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Core
>Reporter: Dikang Gu
>Assignee: Jeff Jirsa
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.x
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> In our environment, we are creating snapshots for backup, after we finish the
> backup, we are running {{clearsnapshot}} to delete the snapshot files. At
> that time we may have thousands of files to delete, and it's causing sudden
> disk usage spike. As a result, we are experiencing a spike of drop messages
> from Cassandra.
> I think we should implement something like {{slowrm}} to delete the snapshot
> files slowly, avoid the sudden disk usage spike.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13019) Improve clearsnapshot to delete the snapshot files slowly
[
https://issues.apache.org/jira/browse/CASSANDRA-13019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976959#comment-16976959
]
Jeff Jirsa commented on CASSANDRA-13019:
Patch is approved by 3 people in GH PR (Aleksey, Chris, Maxwell)
> Improve clearsnapshot to delete the snapshot files slowly
> --
>
> Key: CASSANDRA-13019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13019
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Core
>Reporter: Dikang Gu
>Assignee: Jeff Jirsa
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.x
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> In our environment, we are creating snapshots for backup, after we finish the
> backup, we are running {{clearsnapshot}} to delete the snapshot files. At
> that time we may have thousands of files to delete, and it's causing sudden
> disk usage spike. As a result, we are experiencing a spike of drop messages
> from Cassandra.
> I think we should implement something like {{slowrm}} to delete the snapshot
> files slowly, avoid the sudden disk usage spike.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13019) Improve clearsnapshot to delete the snapshot files slowly
[
https://issues.apache.org/jira/browse/CASSANDRA-13019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971223#comment-16971223
]
Jeff Jirsa commented on CASSANDRA-13019:
All good suggestions. Here's how the nodetool command is setup:
{code}
(oss-ccm) jjirsa:cassandra jjirsa$ ccm node1 nodetool | grep -i snapshot
clearsnapshotRemove the snapshot with the given name from
the given keyspaces. If no snapshotName is specified we will remove all
snapshots
getsnapshotthrottle Print the hardlink-per-second throttle for
snapshot/clearsnapshot
listsnapshotsLists all the snapshots along with the size on
disk and true size.
setsnapshotthrottle Set the hardlinks-per-second cap for snapshot
and clearsnapshot throttling
snapshot Take a snapshot of specified keyspaces or a
snapshot of the specified table
(oss-ccm) jjirsa:cassandra jjirsa$ ccm node1 nodetool getsnapshotthrottle
Snapshot throttle is disabled
(oss-ccm) jjirsa:cassandra jjirsa$ time ccm node1 nodetool snapshot
Requested creating snapshot(s) for [all keyspaces] with snapshot name
[1573418615757] and options {skipFlush=false}
Snapshot directory: 1573418615757
real0m1.593s
user0m1.791s
sys 0m0.308s
(oss-ccm) jjirsa:cassandra jjirsa$ ccm node1 nodetool setsnapshotthrottle 2
(oss-ccm) jjirsa:cassandra jjirsa$ ccm node1 nodetool getsnapshotthrottle
Current snapshot throttle: 2 links/s
(oss-ccm) jjirsa:cassandra jjirsa$ time ccm node1 nodetool snapshot
Requested creating snapshot(s) for [all keyspaces] with snapshot name
[1573418645021] and options {skipFlush=false}
Snapshot directory: 1573418645021
real1m53.393s
user0m1.750s
sys 0m0.371s
{code}
> Improve clearsnapshot to delete the snapshot files slowly
> --
>
> Key: CASSANDRA-13019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13019
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Core
>Reporter: Dikang Gu
>Assignee: Jeff Jirsa
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.x
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> In our environment, we are creating snapshots for backup, after we finish the
> backup, we are running {{clearsnapshot}} to delete the snapshot files. At
> that time we may have thousands of files to delete, and it's causing sudden
> disk usage spike. As a result, we are experiencing a spike of drop messages
> from Cassandra.
> I think we should implement something like {{slowrm}} to delete the snapshot
> files slowly, avoid the sudden disk usage spike.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13019) Improve clearsnapshot to delete the snapshot files slowly
[
https://issues.apache.org/jira/browse/CASSANDRA-13019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-13019:
---
Impacts: Docs
Test and Documentation Plan:
Tested manually; unit tests probably needed, too.
Status: Patch Available (was: In Progress)
PR at https://github.com/apache/cassandra/pull/381
Tests at
https://circleci.com/gh/jeffjirsa/workflows/cassandra/tree/ratelimited-snapshot-trunk
> Improve clearsnapshot to delete the snapshot files slowly
> --
>
> Key: CASSANDRA-13019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13019
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Core
>Reporter: Dikang Gu
>Assignee: Jeff Jirsa
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In our environment, we are creating snapshots for backup, after we finish the
> backup, we are running {{clearsnapshot}} to delete the snapshot files. At
> that time we may have thousands of files to delete, and it's causing sudden
> disk usage spike. As a result, we are experiencing a spike of drop messages
> from Cassandra.
> I think we should implement something like {{slowrm}} to delete the snapshot
> files slowly, avoid the sudden disk usage spike.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13019) Improve clearsnapshot command to delete the snapshot files slowly.
[
https://issues.apache.org/jira/browse/CASSANDRA-13019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969313#comment-16969313
]
Jeff Jirsa commented on CASSANDRA-13019:
[~maxwellguo] I'm sorry I didnt see this issue and didnt realize you had
started working on it. I have a patch I wrote independently from
CASSANDRA-13019 - are you interested in reviewing that for me? Alternatively,
if you have this done, I'm happy to review your patch instead?
> Improve clearsnapshot command to delete the snapshot files slowly.
> --
>
> Key: CASSANDRA-13019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13019
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Core
>Reporter: Dikang Gu
>Assignee: maxwellguo
>Priority: Normal
> Fix For: 4.x
>
>
> In our environment, we are creating snapshots for backup, after we finish the
> backup, we are running {{clearsnapshot}} to delete the snapshot files. At
> that time we may have thousands of files to delete, and it's causing sudden
> disk usage spike. As a result, we are experiencing a spike of drop messages
> from Cassandra.
> I think we should implement something like {{slowrm}} to delete the snapshot
> files slowly, avoid the sudden disk usage spike.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15403) Throttle sstable hardlink creation/removal
[ https://issues.apache.org/jira/browse/CASSANDRA-15403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-15403: --- Resolution: Duplicate Status: Resolved (was: Open) > Throttle sstable hardlink creation/removal > -- > > Key: CASSANDRA-15403 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15403 > Project: Cassandra > Issue Type: Bug > Components: Local/SSTable >Reporter: Jeff Jirsa >Assignee: Jeff Jirsa >Priority: Normal > Fix For: 4.x > > > When there are tens of thousands of sstables, the snapshot and clear snapshot > operations can impact performance of the database itself. Adding a rate > limiter to the snapshot creation and deletion paths can improve response times -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15403) Throttle sstable hardlink creation/removal
[ https://issues.apache.org/jira/browse/CASSANDRA-15403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969311#comment-16969311 ] Jeff Jirsa commented on CASSANDRA-15403: Yep, I'll dupe this to that. The throttle needs to be on both create and remove, though. Also I have a patch I'll post shortly. > Throttle sstable hardlink creation/removal > -- > > Key: CASSANDRA-15403 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15403 > Project: Cassandra > Issue Type: Bug > Components: Local/SSTable >Reporter: Jeff Jirsa >Assignee: Jeff Jirsa >Priority: Normal > Fix For: 4.x > > > When there are tens of thousands of sstables, the snapshot and clear snapshot > operations can impact performance of the database itself. Adding a rate > limiter to the snapshot creation and deletion paths can improve response times -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15403) Throttle sstable hardlink creation/removal
[ https://issues.apache.org/jira/browse/CASSANDRA-15403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-15403: --- Bug Category: Parent values: Degradation(12984)Level 1 values: Resource Management(12995) (was: Parent values: Degradation(12984)Level 1 values: Performance Bug/Regression(12997)) > Throttle sstable hardlink creation/removal > -- > > Key: CASSANDRA-15403 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15403 > Project: Cassandra > Issue Type: Bug > Components: Local/SSTable >Reporter: Jeff Jirsa >Assignee: Jeff Jirsa >Priority: Normal > Fix For: 4.x > > > When there are tens of thousands of sstables, the snapshot and clear snapshot > operations can impact performance of the database itself. Adding a rate > limiter to the snapshot creation and deletion paths can improve response times -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15403) Throttle sstable hardlink creation/removal
[ https://issues.apache.org/jira/browse/CASSANDRA-15403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-15403: --- Bug Category: Parent values: Degradation(12984)Level 1 values: Performance Bug/Regression(12997) Complexity: Low Hanging Fruit Discovered By: User Report Fix Version/s: 4.x Severity: Low Status: Open (was: Triage Needed) > Throttle sstable hardlink creation/removal > -- > > Key: CASSANDRA-15403 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15403 > Project: Cassandra > Issue Type: Bug > Components: Local/SSTable >Reporter: Jeff Jirsa >Assignee: Jeff Jirsa >Priority: Normal > Fix For: 4.x > > > When there are tens of thousands of sstables, the snapshot and clear snapshot > operations can impact performance of the database itself. Adding a rate > limiter to the snapshot creation and deletion paths can improve response times -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15403) Throttle sstable hardlink creation/removal
Jeff Jirsa created CASSANDRA-15403: -- Summary: Throttle sstable hardlink creation/removal Key: CASSANDRA-15403 URL: https://issues.apache.org/jira/browse/CASSANDRA-15403 Project: Cassandra Issue Type: Bug Components: Local/SSTable Reporter: Jeff Jirsa Assignee: Jeff Jirsa When there are tens of thousands of sstables, the snapshot and clear snapshot operations can impact performance of the database itself. Adding a rate limiter to the snapshot creation and deletion paths can improve response times -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15092) Add a new Snitch for Alibaba Cloud Platform
[
https://issues.apache.org/jira/browse/CASSANDRA-15092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-15092:
---
Fix Version/s: 4.0-alpha
Source Control Link:
https://github.com/apache/cassandra/commit/a7a0e7fe40cd375c647d2f5e56d04fd98963cff2
Resolution: Fixed
Status: Resolved (was: Ready to Commit)
Committed. I fixed the log message, and some code style issues (tab instead of
spaces in a few places).
Noting for the record, that I asked in {{#cassandra-dev}} ASF slack on Sept 3
to see if anyone objected to landing this in the alpha since it's "new" and
received no objections (Benedict said "I don’t really consider a snitch for a
cloud provider to be a new feature, but more of a plugin", Aleksey said
"¯\_(ツ)_/¯ I don’t really mind having it in, and am fine with dropping it in
4.0 - it’s entirely self-contained after all", and Josh said "Maybe include but
mark experimental for a cycle?")
> Add a new Snitch for Alibaba Cloud Platform
> ---
>
> Key: CASSANDRA-15092
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15092
> Project: Cassandra
> Issue Type: New Feature
> Components: Legacy/Core, Local/Config
>Reporter: maxwellguo
>Assignee: maxwellguo
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-alpha
>
> Attachments: trunk-15092.txt
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Add snitch for Alibaba cloud platform, for we have saw cloud platform snitch
> for aws and google cloud ,for we can ge alibaba ecs (Elastic Compute Service)
> meta data from here :
> https://help.aliyun.com/document_detail/108460.html?spm=a2c4g.11186623.6.675.36684f8bLQrIMY
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15092) Add a new Snitch for Alibaba Cloud Platform
[ https://issues.apache.org/jira/browse/CASSANDRA-15092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-15092: --- Status: Ready to Commit (was: Review In Progress) > Add a new Snitch for Alibaba Cloud Platform > --- > > Key: CASSANDRA-15092 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15092 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Core, Local/Config >Reporter: maxwellguo >Assignee: maxwellguo >Priority: Normal > Labels: pull-request-available > Attachments: trunk-15092.txt > > Time Spent: 20m > Remaining Estimate: 0h > > Add snitch for Alibaba cloud platform, for we have saw cloud platform snitch > for aws and google cloud ,for we can ge alibaba ecs (Elastic Compute Service) > meta data from here : > https://help.aliyun.com/document_detail/108460.html?spm=a2c4g.11186623.6.675.36684f8bLQrIMY -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15092) Add a new Snitch for Alibaba Cloud Platform
[ https://issues.apache.org/jira/browse/CASSANDRA-15092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-15092: --- Reviewers: Brandon Williams, Jeff Jirsa Status: Review In Progress (was: Patch Available) > Add a new Snitch for Alibaba Cloud Platform > --- > > Key: CASSANDRA-15092 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15092 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Core, Local/Config >Reporter: maxwellguo >Assignee: maxwellguo >Priority: Normal > Labels: pull-request-available > Attachments: trunk-15092.txt > > Time Spent: 20m > Remaining Estimate: 0h > > Add snitch for Alibaba cloud platform, for we have saw cloud platform snitch > for aws and google cloud ,for we can ge alibaba ecs (Elastic Compute Service) > meta data from here : > https://help.aliyun.com/document_detail/108460.html?spm=a2c4g.11186623.6.675.36684f8bLQrIMY -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15092) Add a new Snitch for Alibaba Cloud Platform
[ https://issues.apache.org/jira/browse/CASSANDRA-15092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932049#comment-16932049 ] Jeff Jirsa commented on CASSANDRA-15092: [~brandon.williams] you've glanced at this before and you're familiar with snitches, interested in a final review/commit? > Add a new Snitch for Alibaba Cloud Platform > --- > > Key: CASSANDRA-15092 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15092 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Core, Local/Config >Reporter: maxwellguo >Assignee: maxwellguo >Priority: Normal > Labels: pull-request-available > Attachments: trunk-15092.txt > > Time Spent: 20m > Remaining Estimate: 0h > > Add snitch for Alibaba cloud platform, for we have saw cloud platform snitch > for aws and google cloud ,for we can ge alibaba ecs (Elastic Compute Service) > meta data from here : > https://help.aliyun.com/document_detail/108460.html?spm=a2c4g.11186623.6.675.36684f8bLQrIMY -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13938) Default repair is broken, crashes other nodes participating in repair (in trunk)
[
https://issues.apache.org/jira/browse/CASSANDRA-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931790#comment-16931790
]
Jeff Jirsa commented on CASSANDRA-13938:
Is this still broken?
> Default repair is broken, crashes other nodes participating in repair (in
> trunk)
>
>
> Key: CASSANDRA-13938
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13938
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Repair
>Reporter: Nate McCall
>Assignee: Joseph Lynch
>Priority: Urgent
> Fix For: 4.0-alpha
>
> Attachments: 13938.yaml, test.sh
>
>
> Running through a simple scenario to test some of the new repair features, I
> was not able to make a repair command work. Further, the exception seemed to
> trigger a nasty failure state that basically shuts down the netty connections
> for messaging *and* CQL on the nodes transferring back data to the node being
> repaired. The following steps reproduce this issue consistently.
> Cassandra stress profile (probably not necessary, but this one provides a
> really simple schema and consistent data shape):
> {noformat}
> keyspace: standard_long
> keyspace_definition: |
> CREATE KEYSPACE standard_long WITH replication = {'class':'SimpleStrategy',
> 'replication_factor':3};
> table: test_data
> table_definition: |
> CREATE TABLE test_data (
> key text,
> ts bigint,
> val text,
> PRIMARY KEY (key, ts)
> ) WITH COMPACT STORAGE AND
> CLUSTERING ORDER BY (ts DESC) AND
> bloom_filter_fp_chance=0.01 AND
> caching={'keys':'ALL', 'rows_per_partition':'NONE'} AND
> comment='' AND
> dclocal_read_repair_chance=0.00 AND
> gc_grace_seconds=864000 AND
> read_repair_chance=0.00 AND
> compaction={'class': 'SizeTieredCompactionStrategy'} AND
> compression={'sstable_compression': 'LZ4Compressor'};
> columnspec:
> - name: key
> population: uniform(1..5000) # 50 million records available
> - name: ts
> cluster: gaussian(1..50) # Up to 50 inserts per record
> - name: val
> population: gaussian(128..1024) # varrying size of value data
> insert:
> partitions: fixed(1) # only one insert per batch for individual partitions
> select: fixed(1)/1 # each insert comes in one at a time
> batchtype: UNLOGGED
> queries:
> single:
> cql: select * from test_data where key = ? and ts = ? limit 1;
> series:
> cql: select key,ts,val from test_data where key = ? limit 10;
> {noformat}
> The commands to build and run:
> {noformat}
> ccm create 4_0_test -v git:trunk -n 3 -s
> ccm stress user profile=./histo-test-schema.yml
> ops\(insert=20,single=1,series=1\) duration=15s -rate threads=4
> # flush the memtable just to get everything on disk
> ccm node1 nodetool flush
> ccm node2 nodetool flush
> ccm node3 nodetool flush
> # disable hints for nodes 2 and 3
> ccm node2 nodetool disablehandoff
> ccm node3 nodetool disablehandoff
> # stop node1
> ccm node1 stop
> ccm stress user profile=./histo-test-schema.yml
> ops\(insert=20,single=1,series=1\) duration=45s -rate threads=4
> # wait 10 seconds
> ccm node1 start
> # Note that we are local to ccm's nodetool install 'cause repair preview is
> not reported yet
> node1/bin/nodetool repair --preview
> node1/bin/nodetool repair standard_long test_data
> {noformat}
> The error outputs from the last repair command follow. First, this is stdout
> from node1:
> {noformat}
> $ node1/bin/nodetool repair standard_long test_data
> objc[47876]: Class JavaLaunchHelper is implemented in both
> /Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/bin/java
> (0x10274d4c0) and
> /Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/libinstrument.dylib
> (0x1047b64e0). One of the two will be used. Which one is undefined.
> [2017-10-05 14:31:52,425] Starting repair command #4
> (7e1a9150-a98e-11e7-ad86-cbd2801b8de2), repairing keyspace standard_long with
> repair options (parallelism: parallel, primary range: false, incremental:
> true, job threads: 1, ColumnFamilies: [test_data], dataCenters: [], hosts:
> [], previewKind: NONE, # of ranges: 3, pull repair: false, force repair:
> false)
> [2017-10-05 14:32:07,045] Repair session 7e2e8e80-a98e-11e7-ad86-cbd2801b8de2
> for range [(3074457345618258602,-9223372036854775808],
> (-9223372036854775808,-3074457345618258603],
> (-3074457345618258603,3074457345618258602]] failed with error Stream failed
> [2017-10-05 14:32:07,048] null
> [2017-10-05 14:32:07,050] Repair command #4 finished in 14 seconds
> error: Repair job has failed with the error message: [2017-10-05
> 14:32:07,048] null
> -- StackTrace --
> java.lang.RuntimeException: Repair job has failed with the error message:
> [2017-10-05 14:32:07,048] null
>
[jira] [Updated] (CASSANDRA-15295) Running into deadlock when do CommitLog initialization
[ https://issues.apache.org/jira/browse/CASSANDRA-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-15295: --- Test and Documentation Plan: Unspecified Status: Patch Available (was: Open) PR from reporter at https://github.com/apache/cassandra/pull/347 > Running into deadlock when do CommitLog initialization > -- > > Key: CASSANDRA-15295 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15295 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Normal > Attachments: jstack.log, pstack.log, screenshot-1.png, > screenshot-2.png, screenshot-3.png > > > Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a > long time. > I used jstack to saw what happened. The main thread stuck in > *AbstractCommitLogSegmentManager.awaitAvailableSegment* > !screenshot-1.png! > The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it > was not actually running. > !screenshot-2.png! > And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on > java class initialization. > !screenshot-3.png! > This is a deadlock obviously. CommitLog waits for a CommitLogSegment when > initializing. In this moment, the CommitLog class is not initialized and the > main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a > CommitLogSegment with exception and call *CommitLog.handleCommitError*(static > method). COMMIT-LOG-ALLOCATOR will block on this line because CommitLog > class is still initializing. > > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15295) Running into deadlock when do CommitLog initialization
[ https://issues.apache.org/jira/browse/CASSANDRA-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-15295: --- Bug Category: Parent values: Availability(12983)Level 1 values: Unavailable(12994) Complexity: Normal Discovered By: User Report Severity: Normal Status: Open (was: Triage Needed) > Running into deadlock when do CommitLog initialization > -- > > Key: CASSANDRA-15295 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15295 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Normal > Attachments: jstack.log, pstack.log, screenshot-1.png, > screenshot-2.png, screenshot-3.png > > > Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a > long time. > I used jstack to saw what happened. The main thread stuck in > *AbstractCommitLogSegmentManager.awaitAvailableSegment* > !screenshot-1.png! > The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it > was not actually running. > !screenshot-2.png! > And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on > java class initialization. > !screenshot-3.png! > This is a deadlock obviously. CommitLog waits for a CommitLogSegment when > initializing. In this moment, the CommitLog class is not initialized and the > main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a > CommitLogSegment with exception and call *CommitLog.handleCommitError*(static > method). COMMIT-LOG-ALLOCATOR will block on this line because CommitLog > class is still initializing. > > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15287) Inaccurate exception message with nodetool snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-15287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-15287:
---
Fix Version/s: 4.0
Since Version: 2.1.5
Source Control Link:
https://github.com/apache/cassandra/commit/a145a485206f6a32b07979dfb1b983addf49dd10
Resolution: Fixed
Status: Resolved (was: Ready to Commit)
Thanks for the patch. I've taken the liberty of rewording it SLIGHTLY to make
it more clear. Hope you don't object. Committed as
[a145a485206f6a32b07979dfb1b983addf49dd10|https://github.com/apache/cassandra/commit/a145a485206f6a32b07979dfb1b983addf49dd10]
> Inaccurate exception message with nodetool snapshot
>
>
> Key: CASSANDRA-15287
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15287
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/nodetool
>Reporter: DeepakVohra
>Assignee: Josh Turner
>Priority: Normal
> Fix For: 4.0
>
> Attachments: 15287.trunk, 15287.trunk.withchanges
>
>
> 1. Create a keyspace _cqlkeyspace_ (arbitrary name)
> 2. Create two tables (t1 and t2)
> 3. Run _nodetool snapshot_ to create snapshot for multiple tables in same
> command. Include the keyspace name as argument (which is not what the command
> syntax allows)
> {code:java}
> nodetool snapshot --kt-list cqlkeyspace.t1,cqlkeyspace.t2 --tag multi-table
> --cqlkeyspace
> {code}
> 4. Exception message is inaccurate in that it is referring to columnfamily
> when it should be referring to keyspace.
> {code:java}
> java.io.IOException: When specifying the Keyspace columfamily list for a
> snapshot, you should not specify columnfamily
> {code}
> Exception message should be:
> {code:java}
> java.io.IOException: When specifying the Keyspace columfamily list for a
> snapshot, you should not specify keyspace
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15287) Inaccurate exception message with nodetool snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-15287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-15287:
---
Reviewers: Jeff Jirsa, Jeff Jirsa (was: Jeff Jirsa)
Jeff Jirsa, Jeff Jirsa
Status: Review In Progress (was: Patch Available)
> Inaccurate exception message with nodetool snapshot
>
>
> Key: CASSANDRA-15287
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15287
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/nodetool
>Reporter: DeepakVohra
>Assignee: Josh Turner
>Priority: Normal
> Attachments: 15287.trunk, 15287.trunk.withchanges
>
>
> 1. Create a keyspace _cqlkeyspace_ (arbitrary name)
> 2. Create two tables (t1 and t2)
> 3. Run _nodetool snapshot_ to create snapshot for multiple tables in same
> command. Include the keyspace name as argument (which is not what the command
> syntax allows)
> {code:java}
> nodetool snapshot --kt-list cqlkeyspace.t1,cqlkeyspace.t2 --tag multi-table
> --cqlkeyspace
> {code}
> 4. Exception message is inaccurate in that it is referring to columnfamily
> when it should be referring to keyspace.
> {code:java}
> java.io.IOException: When specifying the Keyspace columfamily list for a
> snapshot, you should not specify columnfamily
> {code}
> Exception message should be:
> {code:java}
> java.io.IOException: When specifying the Keyspace columfamily list for a
> snapshot, you should not specify keyspace
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15287) Inaccurate exception message with nodetool snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-15287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-15287:
---
Status: Ready to Commit (was: Review In Progress)
> Inaccurate exception message with nodetool snapshot
>
>
> Key: CASSANDRA-15287
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15287
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/nodetool
>Reporter: DeepakVohra
>Assignee: Josh Turner
>Priority: Normal
> Attachments: 15287.trunk, 15287.trunk.withchanges
>
>
> 1. Create a keyspace _cqlkeyspace_ (arbitrary name)
> 2. Create two tables (t1 and t2)
> 3. Run _nodetool snapshot_ to create snapshot for multiple tables in same
> command. Include the keyspace name as argument (which is not what the command
> syntax allows)
> {code:java}
> nodetool snapshot --kt-list cqlkeyspace.t1,cqlkeyspace.t2 --tag multi-table
> --cqlkeyspace
> {code}
> 4. Exception message is inaccurate in that it is referring to columnfamily
> when it should be referring to keyspace.
> {code:java}
> java.io.IOException: When specifying the Keyspace columfamily list for a
> snapshot, you should not specify columnfamily
> {code}
> Exception message should be:
> {code:java}
> java.io.IOException: When specifying the Keyspace columfamily list for a
> snapshot, you should not specify keyspace
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15186) InternodeOutboundMetrics overloaded bytes/count mixup
[ https://issues.apache.org/jira/browse/CASSANDRA-15186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914804#comment-16914804 ] Jeff Jirsa edited comment on CASSANDRA-15186 at 8/24/19 4:16 AM: - Thanks! Committed as [0622288c9b2252e31bba0da73a8608ba2aa13ff3|https://github.com/apache/cassandra/commit/0622288c9b2252e31bba0da73a8608ba2aa13ff3] was (Author: jjirsa): Thanks! Committed as [0622288c9b2252e31bba0da73a8608ba2aa13ff3|[https://github.com/apache/cassandra/commit/0622288c9b2252e31bba0da73a8608ba2aa13ff3]] > InternodeOutboundMetrics overloaded bytes/count mixup > - > > Key: CASSANDRA-15186 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15186 > Project: Cassandra > Issue Type: Bug > Components: Observability/Metrics >Reporter: Marcus Olsson >Assignee: Josh Turner >Priority: Normal > Fix For: 4.0 > > Attachments: 15186-trunk.txt > > > In > [https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/metrics/InternodeOutboundMetrics.java] > there is a small mixup between overloaded count and bytes, in > [LargeMessageDroppedTasksDueToOverload|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/metrics/InternodeOutboundMetrics.java#L129] > and > [UrgentMessageDroppedTasksDueToOverload|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/metrics/InternodeOutboundMetrics.java#L151]. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15186) InternodeOutboundMetrics overloaded bytes/count mixup
[
https://issues.apache.org/jira/browse/CASSANDRA-15186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-15186:
---
Fix Version/s: 4.0
Since Version: 4.0
Source Control Link:
https://github.com/apache/cassandra/commit/0622288c9b2252e31bba0da73a8608ba2aa13ff3
Resolution: Fixed
Status: Resolved (was: Ready to Commit)
Thanks! Committed as
{{[0622288c9b2252e31bba0da73a8608ba2aa13ff3|[https://github.com/apache/cassandra/commit/0622288c9b2252e31bba0da73a8608ba2aa13ff3]]}}
> InternodeOutboundMetrics overloaded bytes/count mixup
> -
>
> Key: CASSANDRA-15186
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15186
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Marcus Olsson
>Assignee: Josh Turner
>Priority: Normal
> Fix For: 4.0
>
> Attachments: 15186-trunk.txt
>
>
> In
> [https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/metrics/InternodeOutboundMetrics.java]
> there is a small mixup between overloaded count and bytes, in
> [LargeMessageDroppedTasksDueToOverload|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/metrics/InternodeOutboundMetrics.java#L129]
> and
> [UrgentMessageDroppedTasksDueToOverload|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/metrics/InternodeOutboundMetrics.java#L151].
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15186) InternodeOutboundMetrics overloaded bytes/count mixup
[
https://issues.apache.org/jira/browse/CASSANDRA-15186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914804#comment-16914804
]
Jeff Jirsa edited comment on CASSANDRA-15186 at 8/24/19 4:15 AM:
-
Thanks! Committed as
[0622288c9b2252e31bba0da73a8608ba2aa13ff3|[https://github.com/apache/cassandra/commit/0622288c9b2252e31bba0da73a8608ba2aa13ff3]]
was (Author: jjirsa):
Thanks! Committed as
{{[0622288c9b2252e31bba0da73a8608ba2aa13ff3|[https://github.com/apache/cassandra/commit/0622288c9b2252e31bba0da73a8608ba2aa13ff3]]}}
> InternodeOutboundMetrics overloaded bytes/count mixup
> -
>
> Key: CASSANDRA-15186
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15186
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Marcus Olsson
>Assignee: Josh Turner
>Priority: Normal
> Fix For: 4.0
>
> Attachments: 15186-trunk.txt
>
>
> In
> [https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/metrics/InternodeOutboundMetrics.java]
> there is a small mixup between overloaded count and bytes, in
> [LargeMessageDroppedTasksDueToOverload|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/metrics/InternodeOutboundMetrics.java#L129]
> and
> [UrgentMessageDroppedTasksDueToOverload|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/metrics/InternodeOutboundMetrics.java#L151].
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15186) InternodeOutboundMetrics overloaded bytes/count mixup
[ https://issues.apache.org/jira/browse/CASSANDRA-15186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-15186: --- Status: Ready to Commit (was: Review In Progress) > InternodeOutboundMetrics overloaded bytes/count mixup > - > > Key: CASSANDRA-15186 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15186 > Project: Cassandra > Issue Type: Bug > Components: Observability/Metrics >Reporter: Marcus Olsson >Assignee: Josh Turner >Priority: Normal > Attachments: 15186-trunk.txt > > > In > [https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/metrics/InternodeOutboundMetrics.java] > there is a small mixup between overloaded count and bytes, in > [LargeMessageDroppedTasksDueToOverload|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/metrics/InternodeOutboundMetrics.java#L129] > and > [UrgentMessageDroppedTasksDueToOverload|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/metrics/InternodeOutboundMetrics.java#L151]. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15186) InternodeOutboundMetrics overloaded bytes/count mixup
[ https://issues.apache.org/jira/browse/CASSANDRA-15186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-15186: --- Reviewers: Jeff Jirsa, Jeff Jirsa (was: Jeff Jirsa) Jeff Jirsa, Jeff Jirsa Status: Review In Progress (was: Patch Available) > InternodeOutboundMetrics overloaded bytes/count mixup > - > > Key: CASSANDRA-15186 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15186 > Project: Cassandra > Issue Type: Bug > Components: Observability/Metrics >Reporter: Marcus Olsson >Assignee: Josh Turner >Priority: Normal > Attachments: 15186-trunk.txt > > > In > [https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/metrics/InternodeOutboundMetrics.java] > there is a small mixup between overloaded count and bytes, in > [LargeMessageDroppedTasksDueToOverload|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/metrics/InternodeOutboundMetrics.java#L129] > and > [UrgentMessageDroppedTasksDueToOverload|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/metrics/InternodeOutboundMetrics.java#L151]. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15186) InternodeOutboundMetrics overloaded bytes/count mixup
[ https://issues.apache.org/jira/browse/CASSANDRA-15186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-15186: --- Bug Category: Parent values: Code(13163)Level 1 values: Bug - Unclear Impact(13164) Discovered By: User Report Severity: Low Status: Open (was: Triage Needed) > InternodeOutboundMetrics overloaded bytes/count mixup > - > > Key: CASSANDRA-15186 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15186 > Project: Cassandra > Issue Type: Bug > Components: Observability/Metrics >Reporter: Marcus Olsson >Assignee: Josh Turner >Priority: Normal > Attachments: 15186-trunk.txt > > > In > [https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/metrics/InternodeOutboundMetrics.java] > there is a small mixup between overloaded count and bytes, in > [LargeMessageDroppedTasksDueToOverload|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/metrics/InternodeOutboundMetrics.java#L129] > and > [UrgentMessageDroppedTasksDueToOverload|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/metrics/InternodeOutboundMetrics.java#L151]. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15186) InternodeOutboundMetrics overloaded bytes/count mixup
[ https://issues.apache.org/jira/browse/CASSANDRA-15186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-15186: --- Test and Documentation Plan: N/A Status: Patch Available (was: Open) > InternodeOutboundMetrics overloaded bytes/count mixup > - > > Key: CASSANDRA-15186 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15186 > Project: Cassandra > Issue Type: Bug > Components: Observability/Metrics >Reporter: Marcus Olsson >Assignee: Josh Turner >Priority: Normal > Attachments: 15186-trunk.txt > > > In > [https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/metrics/InternodeOutboundMetrics.java] > there is a small mixup between overloaded count and bytes, in > [LargeMessageDroppedTasksDueToOverload|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/metrics/InternodeOutboundMetrics.java#L129] > and > [UrgentMessageDroppedTasksDueToOverload|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/metrics/InternodeOutboundMetrics.java#L151]. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15186) InternodeOutboundMetrics overloaded bytes/count mixup
[ https://issues.apache.org/jira/browse/CASSANDRA-15186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa reassigned CASSANDRA-15186: -- Assignee: Josh Turner > InternodeOutboundMetrics overloaded bytes/count mixup > - > > Key: CASSANDRA-15186 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15186 > Project: Cassandra > Issue Type: Bug > Components: Observability/Metrics >Reporter: Marcus Olsson >Assignee: Josh Turner >Priority: Normal > Attachments: 15186-trunk.txt > > > In > [https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/metrics/InternodeOutboundMetrics.java] > there is a small mixup between overloaded count and bytes, in > [LargeMessageDroppedTasksDueToOverload|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/metrics/InternodeOutboundMetrics.java#L129] > and > [UrgentMessageDroppedTasksDueToOverload|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/metrics/InternodeOutboundMetrics.java#L151]. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-8838) Resumable bootstrap streaming
[ https://issues.apache.org/jira/browse/CASSANDRA-8838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900234#comment-16900234 ] Jeff Jirsa commented on CASSANDRA-8838: --- For the record, this patch almost certainly causes us to violate consistency/correctness for the reasons discussed in the 4 comments above (missing writes while restarting / may not store hints / may not deliver hints before timeout), and that it's enabled by default and can't be disabled is really unfortunate for people who want to disable this feature. We need to do be more aware of new features and correctness in the future. > Resumable bootstrap streaming > - > > Key: CASSANDRA-8838 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8838 > Project: Cassandra > Issue Type: Sub-task > Components: Legacy/Streaming and Messaging >Reporter: Yuki Morishita >Assignee: Yuki Morishita >Priority: Low > Labels: dense-storage > Fix For: 2.2.0 beta 1 > > > This allows the bootstrapping node not to be streamed already received data. > The bootstrapping node records received keyspace/ranges as one stream session > completes. When some sessions with other nodes fail, bootstrapping fails > also, though next time it re-bootstraps, already received keyspace/ranges are > skipped to be streamed. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15080) Paxos tables should use a 4KB chunk length
[ https://issues.apache.org/jira/browse/CASSANDRA-15080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899160#comment-16899160 ] Jeff Jirsa commented on CASSANDRA-15080: Mentioned this in slack, but I think hard coding this is probably wrong. There are some real and valid use cases where payloads are much larger than 4k, so those would see a meaningful regression here. There may be a middle ground, or it may make sense to be somewhat configurable. > Paxos tables should use a 4KB chunk length > -- > > Key: CASSANDRA-15080 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15080 > Project: Cassandra > Issue Type: Improvement > Components: Feature/Lightweight Transactions >Reporter: Jon Haddad >Assignee: Jon Haddad >Priority: Normal > Labels: performance > Attachments: flamegraph-bad-perf.svg > > > In doing some research for a client on LWTs, I found that once we start > pushing a node hard enough, with enough LWTs, decompressing the data in the > paxos table takes up quite a bit of time. I've attached an SVG flame graph > showing about 10% of time is spent in LZ4_decompress_fast in queries hitting > the paxos table. We should be able to get a nice little performance bump > from changing this to 4KB chunk length or disabling it completely. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-12764) Compaction performance issues with many sstables, during transaction commit phase
[
https://issues.apache.org/jira/browse/CASSANDRA-12764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-12764:
---
Labels: lcs performance (was: lcs)
> Compaction performance issues with many sstables, during transaction commit
> phase
> -
>
> Key: CASSANDRA-12764
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12764
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction
>Reporter: Tom van der Woerdt
>Priority: Normal
> Labels: lcs, performance
>
> An issue with a script flooded my cluster with sstables. There is now a table
> with 100k sstables, all on the order of KBytes, and it's taking a long time
> (ETA 20 days) to compact, even though the table is only ~30GB.
> Stack trace :
> {noformat}
> "CompactionExecutor:308" #7541 daemon prio=1 os_prio=4 tid=0x7fa22af35400
> nid=0x41eb runnable [0x7fdbea48d000]
>java.lang.Thread.State: RUNNABLE
> at java.util.TimSort.countRunAndMakeAscending(TimSort.java:360)
> at java.util.TimSort.sort(TimSort.java:220)
> at java.util.Arrays.sort(Arrays.java:1438)
> at com.google.common.collect.Ordering.sortedCopy(Ordering.java:817)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.(IntervalTree.java:209)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.(IntervalTree.java:211)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.(IntervalTree.java:211)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.(IntervalTree.java:211)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.(IntervalTree.java:211)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.(IntervalTree.java:211)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.(IntervalTree.java:211)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.(IntervalTree.java:211)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.(IntervalTree.java:210)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.(IntervalTree.java:210)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.(IntervalTree.java:210)
> at
> org.apache.cassandra.utils.IntervalTree$IntervalNode.(IntervalTree.java:210)
> at org.apache.cassandra.utils.IntervalTree.(IntervalTree.java:50)
> at
> org.apache.cassandra.db.lifecycle.SSTableIntervalTree.(SSTableIntervalTree.java:40)
> at
> org.apache.cassandra.db.lifecycle.SSTableIntervalTree.build(SSTableIntervalTree.java:50)
> at org.apache.cassandra.db.lifecycle.View$4.apply(View.java:288)
> at org.apache.cassandra.db.lifecycle.View$4.apply(View.java:283)
> at
> com.google.common.base.Functions$FunctionComposition.apply(Functions.java:216)
> at org.apache.cassandra.db.lifecycle.Tracker.apply(Tracker.java:128)
> at org.apache.cassandra.db.lifecycle.Tracker.apply(Tracker.java:101)
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.checkpoint(LifecycleTransaction.java:307)
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.checkpoint(LifecycleTransaction.java:288)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:368)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:84)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:94)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:194)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:263)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at
[jira] [Updated] (CASSANDRA-15251) " cassandra.service failed " while running command "service cassandra start"
[
https://issues.apache.org/jira/browse/CASSANDRA-15251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-15251:
---
Status: Awaiting Feedback (was: Triage Needed)
> " cassandra.service failed " while running command "service cassandra start"
>
>
> Key: CASSANDRA-15251
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15251
> Project: Cassandra
> Issue Type: Bug
> Components: Build
>Reporter: Rajesh Vakcharla
>Priority: Normal
>
> Hi Team,
> I have installed Cassandra RPM Package in Redhat linux from AWS console.
> While trying to start cassandra getting error as shown below.
> [root@ip-172-31-41-5 ec2-user]# service cassandra start
> {color:#FF}Starting cassandra (via systemctl): Job for cassandra.service
> failed because the service did not take the steps required by its unit
> configuration.{color}
> {color:#FF}See "systemctl status cassandra.service" and "journalctl -xe"
> for details.{color}
> [FAILED]
> unable to findout the issue.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15251) " cassandra.service failed " while running command "service cassandra start"
[
https://issues.apache.org/jira/browse/CASSANDRA-15251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894575#comment-16894575
]
Jeff Jirsa commented on CASSANDRA-15251:
Which version of redhat linux? Which RPM? Which version of Cassandra?
> " cassandra.service failed " while running command "service cassandra start"
>
>
> Key: CASSANDRA-15251
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15251
> Project: Cassandra
> Issue Type: Bug
> Components: Build
>Reporter: Rajesh Vakcharla
>Priority: Normal
>
> Hi Team,
> I have installed Cassandra RPM Package in Redhat linux from AWS console.
> While trying to start cassandra getting error as shown below.
> [root@ip-172-31-41-5 ec2-user]# service cassandra start
> {color:#FF}Starting cassandra (via systemctl): Job for cassandra.service
> failed because the service did not take the steps required by its unit
> configuration.{color}
> {color:#FF}See "systemctl status cassandra.service" and "journalctl -xe"
> for details.{color}
> [FAILED]
> unable to findout the issue.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15250) Large cartesian produces in IN queries cause the server to run out of memory
[
https://issues.apache.org/jira/browse/CASSANDRA-15250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-15250:
---
Severity: Low
Complexity: Normal
Discovered By: User Report
Bug Category: Parent values: Degradation(12984)Level 1 values: Resource
Management(12995)
Status: Open (was: Triage Needed)
> Large cartesian produces in IN queries cause the server to run out of memory
>
>
> Key: CASSANDRA-15250
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15250
> Project: Cassandra
> Issue Type: Bug
> Components: CQL/Interpreter
>Reporter: Avi Kivity
>Priority: Normal
>
> The queries
>
> {{ create table tab (pk1 int, pk2 int, pk3 int, pk4 int, pk5 int, pk6 int,
> pk7 int, pk8 int, pk9 int, primary key((pk1, pk2, pk3, pk4, pk5, pk6, pk7,
> pk8, pk9)));}}
>
> {{ select * from tab where pk1 in (1, 2, 3, 4, 5, 6, 7, 8, 9, 10) and pk2
> in (1, 2, 3, 4, 5, 6, 7, 8, 9, 10) and pk3 in (1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
> and pk4 in (1, 2, 3, 4, 5, 6, 7, 8, 9, 10) and pk5 in (1, 2, 3, 4, 5, 6, 7,
> 8, 9, 10) and pk6 in (1, 2, 3, 4, 5, 6, 7, 8, 9, 10) and pk7 in (1, 2, 3, 4,
> 5, 6, 7, 8, 9, 10) and pk8 in (1, 2, 3, 4, 5, 6, 7, 8, 9, 10) and pk9 in (1,
> 2, 3, 4, 5, 6, 7, 8, 9, 10); }}
>
> Will cause the server to enter a garbage collection spiral from which it does
> not recover. The queries generate a large (1 billion row) cartesian product
> which the server presumably materializes in memory, and fails.
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15250) Large cartesian produces in IN queries cause the server to run out of memory
[
https://issues.apache.org/jira/browse/CASSANDRA-15250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894574#comment-16894574
]
Jeff Jirsa commented on CASSANDRA-15250:
Thanks Avi.
> Large cartesian produces in IN queries cause the server to run out of memory
>
>
> Key: CASSANDRA-15250
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15250
> Project: Cassandra
> Issue Type: Bug
> Components: CQL/Interpreter
>Reporter: Avi Kivity
>Priority: Normal
>
> The queries
>
> {{ create table tab (pk1 int, pk2 int, pk3 int, pk4 int, pk5 int, pk6 int,
> pk7 int, pk8 int, pk9 int, primary key((pk1, pk2, pk3, pk4, pk5, pk6, pk7,
> pk8, pk9)));}}
>
> {{ select * from tab where pk1 in (1, 2, 3, 4, 5, 6, 7, 8, 9, 10) and pk2
> in (1, 2, 3, 4, 5, 6, 7, 8, 9, 10) and pk3 in (1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
> and pk4 in (1, 2, 3, 4, 5, 6, 7, 8, 9, 10) and pk5 in (1, 2, 3, 4, 5, 6, 7,
> 8, 9, 10) and pk6 in (1, 2, 3, 4, 5, 6, 7, 8, 9, 10) and pk7 in (1, 2, 3, 4,
> 5, 6, 7, 8, 9, 10) and pk8 in (1, 2, 3, 4, 5, 6, 7, 8, 9, 10) and pk9 in (1,
> 2, 3, 4, 5, 6, 7, 8, 9, 10); }}
>
> Will cause the server to enter a garbage collection spiral from which it does
> not recover. The queries generate a large (1 billion row) cartesian product
> which the server presumably materializes in memory, and fails.
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15202) Deserialize merkle trees off-heap
[ https://issues.apache.org/jira/browse/CASSANDRA-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16879413#comment-16879413 ] Jeff Jirsa edited comment on CASSANDRA-15202 at 7/5/19 4:45 PM: Perf testing of this patch, using JMX toggling to enable/disable, resulted in the following GC graph: !offheap-mts-gc.png! This is repair of a 12 instance cluster with 100 tables running in a loop. Starting at 04/08@~1330, the old style repair was run. In the afternoon of 04/09, the prop was changed to use the offheap merkle trees, and the result is pretty clear: parnew collections drop from ~1-3s to ~100-300ms, and olg gen collections nearly completely disappear. was (Author: jjirsa): Perf testing of this patch, using JMX toggling to enable/disable, resulted in the following GC graph: !offheap-mts-gc.png! This is repair of a 12 instance cluster with 100 tables running in a loop. Starting at 04/08@~1330, the old style repair was run. In the afternoon of 04/09, the prop was changed to use the offheap merkle trees, and the result is pretty clear: parnew collections drop from ~3s to ~300ms, and olg gen collections nearly completely disappear. > Deserialize merkle trees off-heap > - > > Key: CASSANDRA-15202 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15202 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Repair >Reporter: Jeff Jirsa >Assignee: Aleksey Yeschenko >Priority: Normal > Fix For: 4.0 > > Attachments: offheap-mts-gc.png > > > CASSANDRA-14096 made the first step to address the heavy on-heap footprint of > merkle trees on repair coordinators - by reducing the time frame over which > they are referenced, and by more intelligently limiting depth of the trees > based on available heap size. > That alone improves GC profile and prevents OOMs, but doesn’t address the > issue entirely. The coordinator still must hold all the trees on heap at once > until it’s done diffing them with each other, which has a negative effect, > and, by reducing depth, we lose precision and thus cause more overstreaming > than before. > One way to improve the situation further is to build on CASSANDRA-14096 and > move the trees entirely off-heap. This is a trivial endeavor, given that we > are dealing with what should be full binary trees (though in practice aren’t > quite, yet). This JIRA makes the first step towards there - by moving just > deserialisation off-heap, leaving construction on the replicas on-heap still. > Additionally, the proposed patch fixes the issue of replica coordinators > sending merkle trees to itself over loopback, costing us a ser/deser loop per > tree. > Please note that there is more room for improvement here, and depending on > 4.0 timeline those improvements may or may not land in time. To name a few: > - with some minor modifications to init(), we can make sure that no matter > the range, the tree is *always* perfectly full; this would allow us to get > rid of child pointers in inner nodes, as child node addresses will be > trivially calculatable given fixed size of nodes > - the trees can be easily constructed off-heap so long as you run init() to > pre-size the tree to find out how large a buffer you need > - on-wire format doesn’t need to stream inner nodes, only leaves, and, > really, only the hashes of the leaves -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15202) Deserialize merkle trees off-heap
[ https://issues.apache.org/jira/browse/CASSANDRA-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16879413#comment-16879413 ] Jeff Jirsa commented on CASSANDRA-15202: Perf testing of this patch, using JMX toggling to enable/disable, resulted in the following GC graph: !offheap-mts-gc.png! This is repair of a 12 instance cluster with 100 tables running in a loop. Starting at 04/08@~1330, the old style repair was run. In the afternoon of 04/09, the prop was changed to use the offheap merkle trees, and the result is pretty clear: parnew collections drop from ~3s to ~300ms, and olg gen collections nearly completely disappear. > Deserialize merkle trees off-heap > - > > Key: CASSANDRA-15202 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15202 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Repair >Reporter: Jeff Jirsa >Assignee: Aleksey Yeschenko >Priority: Normal > Fix For: 4.0 > > Attachments: offheap-mts-gc.png > > > CASSANDRA-14096 made the first step to address the heavy on-heap footprint of > merkle trees on repair coordinators - by reducing the time frame over which > they are referenced, and by more intelligently limiting depth of the trees > based on available heap size. > That alone improves GC profile and prevents OOMs, but doesn’t address the > issue entirely. The coordinator still must hold all the trees on heap at once > until it’s done diffing them with each other, which has a negative effect, > and, by reducing depth, we lose precision and thus cause more overstreaming > than before. > One way to improve the situation further is to build on CASSANDRA-14096 and > move the trees entirely off-heap. This is a trivial endeavor, given that we > are dealing with what should be full binary trees (though in practice aren’t > quite, yet). This JIRA makes the first step towards there - by moving just > deserialisation off-heap, leaving construction on the replicas on-heap still. > Additionally, the proposed patch fixes the issue of replica coordinators > sending merkle trees to itself over loopback, costing us a ser/deser loop per > tree. > Please note that there is more room for improvement here, and depending on > 4.0 timeline those improvements may or may not land in time. To name a few: > - with some minor modifications to init(), we can make sure that no matter > the range, the tree is *always* perfectly full; this would allow us to get > rid of child pointers in inner nodes, as child node addresses will be > trivially calculatable given fixed size of nodes > - the trees can be easily constructed off-heap so long as you run init() to > pre-size the tree to find out how large a buffer you need > - on-wire format doesn’t need to stream inner nodes, only leaves, and, > really, only the hashes of the leaves -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15202) Deserialize merkle trees off-heap
[ https://issues.apache.org/jira/browse/CASSANDRA-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-15202: --- Attachment: offheap-mts-gc.png > Deserialize merkle trees off-heap > - > > Key: CASSANDRA-15202 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15202 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Repair >Reporter: Jeff Jirsa >Assignee: Aleksey Yeschenko >Priority: Normal > Fix For: 4.0 > > Attachments: offheap-mts-gc.png > > > CASSANDRA-14096 made the first step to address the heavy on-heap footprint of > merkle trees on repair coordinators - by reducing the time frame over which > they are referenced, and by more intelligently limiting depth of the trees > based on available heap size. > That alone improves GC profile and prevents OOMs, but doesn’t address the > issue entirely. The coordinator still must hold all the trees on heap at once > until it’s done diffing them with each other, which has a negative effect, > and, by reducing depth, we lose precision and thus cause more overstreaming > than before. > One way to improve the situation further is to build on CASSANDRA-14096 and > move the trees entirely off-heap. This is a trivial endeavor, given that we > are dealing with what should be full binary trees (though in practice aren’t > quite, yet). This JIRA makes the first step towards there - by moving just > deserialisation off-heap, leaving construction on the replicas on-heap still. > Additionally, the proposed patch fixes the issue of replica coordinators > sending merkle trees to itself over loopback, costing us a ser/deser loop per > tree. > Please note that there is more room for improvement here, and depending on > 4.0 timeline those improvements may or may not land in time. To name a few: > - with some minor modifications to init(), we can make sure that no matter > the range, the tree is *always* perfectly full; this would allow us to get > rid of child pointers in inner nodes, as child node addresses will be > trivially calculatable given fixed size of nodes > - the trees can be easily constructed off-heap so long as you run init() to > pre-size the tree to find out how large a buffer you need > - on-wire format doesn’t need to stream inner nodes, only leaves, and, > really, only the hashes of the leaves -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15200) Can a new adding node store other keyspaces' data and add permission of one more machine failure for a replicator 2's keyspace in the new even number nodes' DC
[ https://issues.apache.org/jira/browse/CASSANDRA-15200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-15200: --- Resolution: Not A Problem Status: Resolved (was: Triage Needed) JIRA is for bug reports and feature discussion, not questions and answers. > Can a new adding node store other keyspaces' data and add permission of > one more machine failure for a replicator 2's keyspace in the new even number > nodes' DC? > -- > > Key: CASSANDRA-15200 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15200 > Project: Cassandra > Issue Type: Improvement >Reporter: gloCalHelp.com >Priority: Normal > > Question on adding a new node, replicator and how many machine failes? > Can a new adding node store other keyspaces' data and add permission of > one more machine failure for a replicator 2's keyspace in the new even number > nodes' DC? > Cassandra's designing of adding a new node for only a keyspace is > quite good for a high load keyspace and token allocation. But I have > aquestions: > 1st, if the high load keyspace has a replicator of only 2, and > original cluster has only 3 datanodes, if adding a new node to form a 4 > datanodes for this keyspace, then there will permit how many machine failes > for this 2 replicator keyspace's 4 data nodes center? > 2nd, on cassandra's official web, there is a line of : To use this > approach, the new node must be started with the JVM option > -Dcassandra.allocate_tokens_for_keyspace=, where is the > keyspace from which the algorithm can find the load information to optimize > token assignment for. > Does this mean that a new adding node can only store a keyspace's data? > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15189) Cassandra3.11.3, an "insert same primary keys" doesn't report a error but work as update.
[
https://issues.apache.org/jira/browse/CASSANDRA-15189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-15189:
---
Resolution: Not A Problem
Status: Resolved (was: Triage Needed)
This is normal and expected
Insert and update are effectively the same thing in cassandra. Since two hosts
may be fully disconnected and not know what already exists in the database,
there’s not even at attempt to see if the row already exists unless you use
{{IF NOT EXISTS}} on the query, which makes it use the lightweight transaction
code path, using paxos internally to only conditionally accept the write.
> Cassandra3.11.3, an "insert same primary keys" doesn't report a error but
> work as update.
> --
>
> Key: CASSANDRA-15189
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15189
> Project: Cassandra
> Issue Type: Bug
>Reporter: gloCalHelp.com
>Priority: Normal
>
> Cassandra3.11.3, an "insert same primary keys" doesn't report a error but
> work as update.
> At the below, the 1st "-e", insert a row.
> the 2nd "-e" also insert a row with only the last column's value being
> changed to 375,
> The 2nd sql successfully executed as a update, checked by the 3rd "-e"
> bash-4.1$ bin/cqlsh hadoop6 -u dba -p *** --debug -e "INSERT INTO
> HYGL_JCSJ.hyjg_ods_yy_gps_novar3
> (clcph,dwsj,bc,blbs,cjbzh,ckryid,clid,clmc,ddfx,ddrq,fwj,gd,gdjd,gdwd,jsdlc,jszjl,jxzjl,sjid,sjsfzh,sjxm,sssd,xlmc)
> VALUES ('黑A00888D','2019-06-25
> 09:57:19',0,-20,'',-20,-20,'379-7038',1434,'2019-06-25',275,0,126723690,45726990
> ,796.0,2205,746,'null','null','null',0,'379');"
>
> Using CQL driver: '/home/cassandra/cas3.11.3/bin/../lib/cassandra-driver-internal-only-3.11.0-bb96859b.zip/cassandra-driver-3.11.0-bb96859b/cassandra/__init__.py'>
>
> Using connect timeout: 5 seconds
>
> Using 'utf-8' encoding
>
> Using ssl: False
>
> bash-4.1$ bin/cqlsh hadoop6 -u dba -p *** --debug -e "INSERT INTO
> HYGL_JCSJ.hyjg_ods_yy_gps_novar3
> (clcph,dwsj,bc,blbs,cjbzh,ckryid,clid,clmc,ddfx,ddrq,fwj,gd,gdjd,gdwd,jsdlc,jszjl,jxzjl,sjid,sjsfzh,sjxm,sssd,xlmc)
> VALUES ('黑A00888D','2019-06-25
> 09:57:19',0,-20,'',-20,-20,'379-7038',1434,'2019-06-25',275,0,126723690,45726990
> ,796.0,2205,746,'null','null','null',0,'375');"
>
> Using CQL driver: '/home/cassandra/cas3.11.3/bin/../lib/cassandra-driver-internal-only-3.11.0-bb96859b.zip/cassandra-driver-3.11.0-bb96859b/cassandra/__init__.py'>
>
> Using connect timeout: 5 seconds
>
> Using 'utf-8' encoding
>
> Using ssl: False
>
> bash-4.1$ bin/cqlsh hadoop4 -u dba -p *** --debug -e "select * from
> HYGL_JCSJ.hyjg_ods_yy_gps_novar3 where clcph='黑A00888D' and dwsj='2019-06-25
> 09:57:19';"Using CQL driver: '/home/cassandra/cas3.11.3/bin/../lib/cassandra-driver-internal-only-3.11.0-bb96859b.zip/cassandra-driver-3.11.0-bb96859b/cassandra/__init__.py'>
>
> Using connect timeout: 5 seconds
> Using 'utf-8' encoding
> Using ssl: False
> clcph | dwsj | bc | blbs | cjbzh | ckryid |
> clid | clmc | ddfx | ddrq | ddsj | dlzbs |
> dwrq | dwsk | fcsxh | fwj | gd | gdjd | gdwd | jd | jsdlc | jszjl
> | jxzjl | kxbs | sfaxlxs | sfcs | sjgxsj | sjid | sjlx | sjlyxt | sjsfzh |
> sjwtid | sjxm | sjzlfj | sssd | szzdid | szzdmc | szzdxh | wd | xlbm | xlid
> | xlmc | xslc | xxfssj | xxjssj | xxrksj | xzzdid | xzzdmc | xzzdxh | yxfx |
> yygpsxxjlid | yyzt
>
[jira] [Updated] (CASSANDRA-15188) WE are seeing high CPU and load average suddently on one of the node
[ https://issues.apache.org/jira/browse/CASSANDRA-15188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-15188: --- Resolution: Won't Do Status: Resolved (was: Triage Needed) This jira is intended for tracking development work and bug reports. High CPU could be caused by any number of things that are not bugs, including unusual query patterns and bad data models that are prone to high GC or inefficient reads. If you need help debugging the CPU issue, your best bet is the user mailing list or the slack channel. If you know the cause of the high CPU and think it should be fixed, please open a follow-up JIRA with the data model / query pattern you feel should be optimized. > WE are seeing high CPU and load average suddently on one of the node > - > > Key: CASSANDRA-15188 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15188 > Project: Cassandra > Issue Type: Bug > Components: Documentation/Blog >Reporter: chinta kiran >Priority: Normal > > We are seeing high CPU and load average suddently on one of the node . The > load average is almost 260. We are using apache Cassandra 3.11.2 version. > when we restart the node the load average is back to normal. We are having 4 > nodes cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15187) Cann't table and materialized view's timestamp column order by function well?
[
https://issues.apache.org/jira/browse/CASSANDRA-15187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874679#comment-16874679
]
Jeff Jirsa commented on CASSANDRA-15187:
This JIRA is meant for bug reports. Please use the mailing list or slack for
operational questions.
> Cann't table and materialized view's timestamp column order by function well?
> --
>
> Key: CASSANDRA-15187
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15187
> Project: Cassandra
> Issue Type: Bug
> Components: CQL/Syntax
>Reporter: gloCalHelp.com
>Priority: Normal
>
> Cann't table and materialized view's timestamp column order by function well?
> I am using cassandra3.11.3 on centos6.9 with python2.7.13, when I create a
> table as below:
> CREATE TABLE hygl_jcsj.hyjg_ods_yy_gps_novar4 (
> clcph text,
> dwsj timestamp,
> bc decimal,
> blbs decimal,
> cjbzh text,
> ckryid decimal,
> clid decimal,
> clmc text,
> ddfx decimal,
> ddrq timestamp,
> ddsj text,
> dlzbs decimal,
> dwrq timestamp,
> dwsk text,
> fcsxh decimal,
> fwj decimal,
> gd decimal,
> gdjd decimal,
> gdwd decimal,
> jd decimal,
> jsdlc decimal,
> jszjl decimal,
> jxzjl decimal,
> kxbs decimal,
> sfaxlxs decimal,
> sfcs decimal,
> sjgxsj timestamp,
> sjid text,
> sjlx decimal,
> sjlyxt decimal,
> sjsfzh text,
> sjwtid text,
> sjxm text,
> sjzlfj decimal,
> sssd decimal,
> szzdid decimal,
> szzdmc text,
> szzdxh decimal,
> wd decimal,
> xlbm text,
> xlid decimal,
> xlmc text,
> xslc decimal,
> xxfssj timestamp,
> xxjssj timestamp,
> xxrksj timestamp,
> xzzdid decimal,
> xzzdmc text,
> xzzdxh decimal,
> yxfx decimal,
> yygpsxxjlid decimal,
> yyzt decimal,
> PRIMARY KEY (clcph, dwsj)
> ) WITH CLUSTERING ORDER BY (dwsj ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = \{'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = 'GPS数据'
> AND compaction = \{'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = \{'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.0
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 360
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
> the column dwsj is order by nature order of ASC, but when I use cqlsh -e
> "select clcph,dwsj from hygl_jcsj.hyjg_ods_yy_gps_novar4 limit 18",
> the result is:
> | clcph \| dwsj| | | | |
> |-+-| | | | |
> | a85161782800835 \| 2019-06-27 11:39:42.00+| | | | |
> | a85161785390963 \| 2019-06-27 13:06:54.00+| | | | |
> | a8516178847003 \| 2019-06-25 10:51:18.00+| | | | |
> | a8516178847003 \| 2019-06-27 10:06:56.00+| | | | |
> |
[jira] [Updated] (CASSANDRA-15187) Cann't table and materialized view's timestamp column order by function well?
[
https://issues.apache.org/jira/browse/CASSANDRA-15187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-15187:
---
Resolution: Not A Bug
Status: Resolved (was: Triage Needed)
The clustering order by in the schema specifies the order of rows +within a
partition+.
In your example, you're querying without a partition key in the WHERE , so
+there's no ordering expected across partition keys+, and the results +ARE
sorted as expected+ in the rows where there are multiple rows per partition.
> Cann't table and materialized view's timestamp column order by function well?
> --
>
> Key: CASSANDRA-15187
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15187
> Project: Cassandra
> Issue Type: Bug
> Components: CQL/Syntax
>Reporter: gloCalHelp.com
>Priority: Normal
>
> Cann't table and materialized view's timestamp column order by function well?
> I am using cassandra3.11.3 on centos6.9 with python2.7.13, when I create a
> table as below:
> CREATE TABLE hygl_jcsj.hyjg_ods_yy_gps_novar4 (
> clcph text,
> dwsj timestamp,
> bc decimal,
> blbs decimal,
> cjbzh text,
> ckryid decimal,
> clid decimal,
> clmc text,
> ddfx decimal,
> ddrq timestamp,
> ddsj text,
> dlzbs decimal,
> dwrq timestamp,
> dwsk text,
> fcsxh decimal,
> fwj decimal,
> gd decimal,
> gdjd decimal,
> gdwd decimal,
> jd decimal,
> jsdlc decimal,
> jszjl decimal,
> jxzjl decimal,
> kxbs decimal,
> sfaxlxs decimal,
> sfcs decimal,
> sjgxsj timestamp,
> sjid text,
> sjlx decimal,
> sjlyxt decimal,
> sjsfzh text,
> sjwtid text,
> sjxm text,
> sjzlfj decimal,
> sssd decimal,
> szzdid decimal,
> szzdmc text,
> szzdxh decimal,
> wd decimal,
> xlbm text,
> xlid decimal,
> xlmc text,
> xslc decimal,
> xxfssj timestamp,
> xxjssj timestamp,
> xxrksj timestamp,
> xzzdid decimal,
> xzzdmc text,
> xzzdxh decimal,
> yxfx decimal,
> yygpsxxjlid decimal,
> yyzt decimal,
> PRIMARY KEY (clcph, dwsj)
> ) WITH CLUSTERING ORDER BY (dwsj ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = \{'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = 'GPS数据'
> AND compaction = \{'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = \{'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.0
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 360
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
> the column dwsj is order by nature order of ASC, but when I use cqlsh -e
> "select clcph,dwsj from hygl_jcsj.hyjg_ods_yy_gps_novar4 limit 18",
> the result is:
> | clcph \| dwsj| | | | |
> |-+-| | | | |
> | a85161782800835
[jira] [Updated] (CASSANDRA-15173) fix log for inconsistent-left-subrange in merkle-tree
[
https://issues.apache.org/jira/browse/CASSANDRA-15173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-15173:
---
Fix Version/s: 4.0
Source Control Link:
https://github.com/apache/cassandra/commit/802989dce75847f87a5a47e2fe5d590d02f086fa
Status: Resolved (was: Ready to Commit)
Resolution: Fixed
Committed as {{802989dce75847f87a5a47e2fe5d590d02f086fa}}
> fix log for inconsistent-left-subrange in merkle-tree
> -
>
> Key: CASSANDRA-15173
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15173
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Streaming and Messaging
>Reporter: Nathan Jackels
>Assignee: Nathan Jackels
>Priority: Low
> Labels: pull-request-available
> Fix For: 4.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> One of the logs added in CASSANDRA-13052 is called with incorrect arguments.
> In org.apache.cassandra.utils.MerkleTree::differenceHelper the right-side
> subrange logs with right:
> {code:java}
> else if (!rreso)
> {
> logger.debug("({}) Right sub-range fully inconsistent {}",
> active.depth, right);
> rdiff = FULLY_INCONSISTENT;
> }
> {code}
> but left side is also (incorrectly) logged with right:
> {code:java}
> else if (!lreso)
> {
> logger.debug("({}) Left sub-range fully inconsistent {}",
> active.depth, right);
> ldiff = FULLY_INCONSISTENT;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15173) fix log for inconsistent-left-subrange in merkle-tree
[
https://issues.apache.org/jira/browse/CASSANDRA-15173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-15173:
---
Test and Documentation Plan:
Patch in PR #327
Trivial patch. No tests/docs required.
Status: Patch Available (was: Open)
> fix log for inconsistent-left-subrange in merkle-tree
> -
>
> Key: CASSANDRA-15173
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15173
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Streaming and Messaging
>Reporter: Nathan Jackels
>Assignee: Nathan Jackels
>Priority: Low
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> One of the logs added in CASSANDRA-13052 is called with incorrect arguments.
> In org.apache.cassandra.utils.MerkleTree::differenceHelper the right-side
> subrange logs with right:
> {code:java}
> else if (!rreso)
> {
> logger.debug("({}) Right sub-range fully inconsistent {}",
> active.depth, right);
> rdiff = FULLY_INCONSISTENT;
> }
> {code}
> but left side is also (incorrectly) logged with right:
> {code:java}
> else if (!lreso)
> {
> logger.debug("({}) Left sub-range fully inconsistent {}",
> active.depth, right);
> ldiff = FULLY_INCONSISTENT;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
