[jira] [Commented] (CASSANDRA-14525) streaming failure during bootstrap makes new node into inconsistent state
[
https://issues.apache.org/jira/browse/CASSANDRA-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16727118#comment-16727118
]
Jay Zhuang commented on CASSANDRA-14525:
Rebased the code and started the tests:
| Branch | uTest | dTest |
| [14525-2.2|https://github.com/cooldoger/cassandra/tree/14525-2.2] |
[!https://circleci.com/gh/cooldoger/cassandra/tree/14525-2.2.svg?style=svg!|https://circleci.com/gh/cooldoger/cassandra/tree/14525-2.2]
|
[!https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/664/badge/icon!|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/664/]
|
| [14525-3.0|https://github.com/cooldoger/cassandra/tree/14525-3.0] |
[!https://circleci.com/gh/cooldoger/cassandra/tree/14525-3.0.svg?style=svg!|https://circleci.com/gh/cooldoger/cassandra/tree/14525-3.0]
|
[!https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/665/badge/icon!|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/665/]
|
| [14525-3.11|https://github.com/cooldoger/cassandra/tree/14525-3.11] |
[!https://circleci.com/gh/cooldoger/cassandra/tree/14525-3.11.svg?style=svg!|https://circleci.com/gh/cooldoger/cassandra/tree/14525-3.11]
|
[!https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/666/badge/icon!|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/666/]
|
| [14525-trunk|https://github.com/cooldoger/cassandra/tree/14525-trunk] |
[!https://circleci.com/gh/cooldoger/cassandra/tree/14525-trunk.svg?style=svg!|https://circleci.com/gh/cooldoger/cassandra/tree/14525-trunk]
|
[!https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/667/badge/icon!|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/667/]
|
> streaming failure during bootstrap makes new node into inconsistent state
> -
>
> Key: CASSANDRA-14525
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14525
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Major
> Fix For: 4.0, 2.2.x, 3.0.x
>
>
> If bootstrap fails for newly joining node (most common reason is due to
> streaming failure) then Cassandra state remains in {{joining}} state which is
> fine but Cassandra also enables Native transport which makes overall state
> inconsistent. This further creates NullPointer exception if auth is enabled
> on the new node, please find reproducible steps here:
> For example if bootstrap fails due to streaming errors like
> {quote}java.util.concurrent.ExecutionException:
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1256)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:894)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:660)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:573)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:330)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:567)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:695)
> [apache-cassandra-3.0.16.jar:3.0.16]
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> ~[guava-18.0.jar:na]
> at
>
[jira] [Commented] (CASSANDRA-14155) [TRUNK] Gossiper somewhat frequently hitting an NPE on node startup with dtests at org.apache.cassandra.gms.Gossiper.isSafeForStartup(Gossiper.java:769)
[
https://issues.apache.org/jira/browse/CASSANDRA-14155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16727108#comment-16727108
]
Ariel Weisberg commented on CASSANDRA-14155:
I haven't quite nailed it down, but basically nodes are sending gossip syn acks
to nodes that have just started that only contain information on the node
sending the ack. They are in response to previously received gossip syn as I
can see the message being created in the verb handler.
My guess is that since gossip doesn't do request/response correlation we are
getting an ack for some request, like maybe before the node was upgraded, and
the response from that looks like an ack to the shadow round.
I see stuff like
{noformat}
INFO [GossipStage:1] 2018-12-21 16:30:41,202
GossipDigestSynVerbHandler.java:104 - sending [] digests and
{127.0.0.2:7000=EndpointState: HeartBeatState = HeartBeat: generation =
1545427529, version = 2147483647, AppStateMap = {}} deltas
java.lang.Throwable: null
at
org.apache.cassandra.gms.GossipDigestSynVerbHandler.doVerb(GossipDigestSynVerbHandler.java:104)
at
org.apache.cassandra.net.MessageDeliveryTask.process(MessageDeliveryTask.java:92)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:54)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
DEBUG [GossipStage:1] 2018-12-21 16:30:41,202
OutboundMessagingConnection.java:257 - connection attempt 4 to 127.0.0.2:7000
(GOSSIP)
DEBUG [GossipStage:1] 2018-12-21 16:30:41,202 NettyFactory.java:326 - creating
outbound bootstrap to peer 127.0.0.2:7000, compression: false, encryption:
enabled (jdk), coalesce: DISABLED, protocolVersion: 11
{noformat}
and then later
{noformat}
DEBUG [GossipStage:1] 2018-12-21 16:30:59,320 Gossiper.java:1390 - Shadow
request received, adding all states, {127.0.0.3:7000=EndpointState:
HeartBeatState = HeartBeat: generation = 1545427545, version = 360, AppStateMap
= {STATUS=Value(NORMAL,3074457345618258602,18), LOAD=Value(1.1576835E7,349),
SCHEMA=Value(e8a984be-18d3-3322-b20c-036da60b86c9,46), DC=Value(datacenter1,6),
RACK=Value(rack1,8), RELEASE_VERSION=Value(3.11.3-SNAPSHOT,4),
RPC_ADDRESS=Value(127.0.0.3,3), NET_VERSION=Value(11,1),
HOST_ID=Value(e65cd54b-a568-42d9-ae25-70e566167fc1,2),
TOKENS=Value(^@^@^@^H*ªªª^@^@^@^@,17), RPC_READY=Value(true,32)},
127.0.0.2:7000=EndpointState: HeartBeatState = HeartBeat: generation =
1545427529, version = 2147483647, AppStateMap =
{STATUS=Value(shutdown,true,132), LOAD=Value(1.1576468E7,347),
SCHEMA=Value(e8a984be-18d3-3322-b20c-036da60b86c9,62), DC=Value(datacenter1,6),
RACK=Value(rack1,8), RELEASE_VERSION=Value(3.11.3-SNAPSHOT,4),
RPC_ADDRESS=Value(127.0.0.2,3), NET_VERSION=Value(11,1),
HOST_ID=Value(b14298c2-4322-49de-9747-407dc5f16bb6,2),
TOKENS=Value(^@^@^@^HÕUUU^@^@^@^@,17), RPC_READY=Value(false,133),
STATUS_WITH_PORT=Value(shutdown,true,131)}, 127.0.0.1:7000=EndpointState:
HeartBeatState = HeartBeat: generation = 1545427751, version = 152, AppStateMap
= {STATUS=Value(NORMAL,-9223372036854775808,31), LOAD=Value(4494006.0,104),
SCHEMA=Value(3b77100c-2a8c-318f-a9e0-c5bfc4a4bde4,42), DC=Value(datacenter1,7),
RACK=Value(rack1,9), RELEASE_VERSION=Value(4.0-SNAPSHOT,5),
RPC_ADDRESS=Value(127.0.0.1,4), NET_VERSION=Value(12,1),
HOST_ID=Value(01625b7d-2315-47e8-b031-da3cd9382161,2),
TOKENS=Value(^@^@^@^H<80>^@^@^@^@^@^@^@^@^@^@^@,29), RPC_READY=Value(true,54),
NATIVE_ADDRESS_AND_PORT=Value(127.0.0.1:9042,3),
STATUS_WITH_PORT=Value(NORMAL,-9223372036854775808,30)}}
INFO [GossipStage:1] 2018-12-21 16:30:59,322
GossipDigestSynVerbHandler.java:104 - sending [] digests and
{127.0.0.3:7000=EndpointState: HeartBeatState = HeartBeat: generation =
1545427545, version = 360, AppStateMap =
{STATUS=Value(NORMAL,3074457345618258602,18), LOAD=Value(1.1576835E7,349),
SCHEMA=Value(e8a984be-18d3-3322-b20c-036da60b86c9,46), DC=Value(datacenter1,6),
RACK=Value(rack1,8), RELEASE_VERSION=Value(3.11.3-SNAPSHOT,4),
RPC_ADDRESS=Value(127.0.0.3,3), NET_VERSION=Value(11,1),
HOST_ID=Value(e65cd54b-a568-42d9-ae25-70e566167fc1,2),
TOKENS=Value(^@^@^@^H*ªªª^@^@^@^@,17), RPC_READY=Value(true,32)},
127.0.0.2:7000=EndpointState: HeartBeatState = HeartBeat: generation =
1545427529, version = 2147483647, AppStateMap =
{STATUS=Value(shutdown,true,132), LOAD=Value(1.1576468E7,347),
[jira] [Commented] (CASSANDRA-12126) CAS Reads Inconsistencies
[ https://issues.apache.org/jira/browse/CASSANDRA-12126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726929#comment-16726929 ] Ariel Weisberg commented on CASSANDRA-12126: Having reviewed the code I think what Benedict says is correct. The criteria we use for identifying if there is an progress paxos round that needs resolution is incorrect because it assumes we have visibility to all accepted ballots when we only have visibility to a majority. I think this optimization can be done correctly, but it's a bit of surgery. Right now reads do a prepare and modify the promised ballot at each acceptor. If instead we only read the promised ballot from each acceptor then we could check the promised ballot matches the most recent committed ballot. If those are the same we know nothing is in progress because a higher ballot than the most recent accepted/committed ballot has not been promised by a majority which means there can be no lingering accepted ballot since a majority of promises must be collected first. If that isn't the case then we can go ahead and do a prepare + propose to make them match and subsequent reads won't have to do a propose. This may also impact our choice of how many replicas to contact in each phase since we want them to have consistent paxos state so reads can be one roundtrip. I am not sure if we contact them all (like with mutations) or just a majority. > CAS Reads Inconsistencies > -- > > Key: CASSANDRA-12126 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12126 > Project: Cassandra > Issue Type: Bug > Components: Coordination >Reporter: sankalp kohli >Priority: Major > Labels: LWT > > While looking at the CAS code in Cassandra, I found a potential issue with > CAS Reads. Here is how it can happen with RF=3 > 1) You issue a CAS Write and it fails in the propose phase. A machine replies > true to a propose and saves the commit in accepted filed. The other two > machines B and C does not get to the accept phase. > Current state is that machine A has this commit in paxos table as accepted > but not committed and B and C does not. > 2) Issue a CAS Read and it goes to only B and C. You wont be able to read the > value written in step 1. This step is as if nothing is inflight. > 3) Issue another CAS Read and it goes to A and B. Now we will discover that > there is something inflight from A and will propose and commit it with the > current ballot. Now we can read the value written in step 1 as part of this > CAS read. > If we skip step 3 and instead run step 4, we will never learn about value > written in step 1. > 4. Issue a CAS Write and it involves only B and C. This will succeed and > commit a different value than step 1. Step 1 value will never be seen again > and was never seen before. > If you read the Lamport “paxos made simple” paper and read section 2.3. It > talks about this issue which is how learners can find out if majority of the > acceptors have accepted the proposal. > In step 3, it is correct that we propose the value again since we dont know > if it was accepted by majority of acceptors. When we ask majority of > acceptors, and more than one acceptors but not majority has something in > flight, we have no way of knowing if it is accepted by majority of acceptors. > So this behavior is correct. > However we need to fix step 2, since it caused reads to not be linearizable > with respect to writes and other reads. In this case, we know that majority > of acceptors have no inflight commit which means we have majority that > nothing was accepted by majority. I think we should run a propose step here > with empty commit and that will cause write written in step 1 to not be > visible ever after. > With this fix, we will either see data written in step 1 on next serial read > or will never see it which is what we want. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14155) [TRUNK] Gossiper somewhat frequently hitting an NPE on node startup with dtests at org.apache.cassandra.gms.Gossiper.isSafeForStartup(Gossiper.java:769)
[
https://issues.apache.org/jira/browse/CASSANDRA-14155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726920#comment-16726920
]
Ariel Weisberg commented on CASSANDRA-14155:
Got the endpoint state map + some additional logging during the shadow round.
{noformat}
java.lang.AssertionError: previous (Gossip ApplicationState.HOST_ID for this
node) is null: Messages = Starting shadow gossip round to check for endpoint
collision at 127.0.0.2:7000
Sending shadow round GOSSIP DIGEST SYN to seeds [127.0.0.1:7000, 127.0.0.3:7000]
Received a regular ack from 127.0.0.1:7000, can now exit shadow round, epStates
= {127.0.0.2:7000=EndpointState: HeartBeatState = HeartBeat: generation =
154535, version = 2147483647, AppStateMap = {},
127.0.0.1:7000=EndpointState: HeartBeatState = HeartBeat: generation =
1545333481, version = 123, AppStateMap = {}}
at org.apache.cassandra.gms.Gossiper.isSafeForStartup(Gossiper.java:834)
at
org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:556)
at
org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:835)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:693)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:644)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:369)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:581)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:670)
{noformat}
{noformat}
java.lang.AssertionError: previous (Gossip ApplicationState.HOST_ID for this
node) is null: Messages = Starting shadow gossip round to check for endpoint
collision at 127.0.0.2:7000
Sending shadow round GOSSIP DIGEST SYN to seeds [127.0.0.1:7000, 127.0.0.3:7000]
Received a regular ack from 127.0.0.1:7000, can now exit shadow round,
epStateMap {127.0.0.3:7000=EndpointState: HeartBeatState = HeartBeat:
generation = 1545410617, version = 240, AppStateMap = {},
127.0.0.2:7000=EndpointState: HeartBeatState = HeartBeat: generation =
1545410603, version = 2147483647, AppStateMap = {},
127.0.0.1:7000=EndpointState: HeartBeatState = HeartBeat: generation =
1545410741, version = 123, AppStateMap = {}}, epStates =
{127.0.0.3:7000=EndpointState: HeartBeatState = HeartBeat: generation =
1545410617, version = 240, AppStateMap = {}, 127.0.0.2:7000=EndpointState:
HeartBeatState = HeartBeat: generation = 1545410603, version = 2147483647,
AppStateMap = {}, 127.0.0.1:7000=EndpointState: HeartBeatState = HeartBeat:
generation = 1545410741, version = 123, AppStateMap = {}}
at org.apache.cassandra.gms.Gossiper.isSafeForStartup(Gossiper.java:834)
at
org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:556)
at
org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:835)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:693)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:644)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:369)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:581)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:670)
{noformat}
> [TRUNK] Gossiper somewhat frequently hitting an NPE on node startup with
> dtests at
> org.apache.cassandra.gms.Gossiper.isSafeForStartup(Gossiper.java:769)
>
>
> Key: CASSANDRA-14155
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14155
> Project: Cassandra
> Issue Type: Bug
> Components: Lifecycle, Testing
>Reporter: Michael Kjellman
>Assignee: Jason Brown
>Priority: Major
> Labels: dtest
>
> Gossiper is somewhat frequently hitting an NPE on node startup with dtests at
> org.apache.cassandra.gms.Gossiper.isSafeForStartup(Gossiper.java:769)
> {code}
> test teardown failure
> Unexpected error found in node logs (see stdout for full details). Errors:
> [ERROR [main] 2018-01-08 21:41:01,832 CassandraDaemon.java:675 - Exception
> encountered during startup
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.gms.Gossiper.isSafeForStartup(Gossiper.java:769)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:511)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:761)
> ~[main/:na]
> at
>
[jira] [Commented] (CASSANDRA-14448) Improve the performance of CAS
[ https://issues.apache.org/jira/browse/CASSANDRA-14448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726921#comment-16726921 ] Ariel Weisberg commented on CASSANDRA-14448: Just to make sure this doesn't get lost. Benedict pointed out in conversation that we could do the read as part of the accept not the prepare. Basically we agree on what the next conditional operation is going to be. This means that dueling at the accept phase wouldn't have to do the work of the read. You would have to make it to accept to expend those resources. > Improve the performance of CAS > -- > > Key: CASSANDRA-14448 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14448 > Project: Cassandra > Issue Type: Improvement > Components: Coordination >Reporter: Dikang Gu >Assignee: Dikang Gu >Priority: Major > Fix For: 4.x > > > I'm working on some performance improvements of the lightweight transitions > (compare and set). > > As you know, current CAS requires 4 round trips to finish, which is not > efficient, especially in cross DC case. > 1) Prepare > 2) Quorum read current value > 3) Propose new value > 4) Commit > > I'm proposing the following improvements to reduce it to 2 round trips, which > is: > 1) Combine prepare and quorum read together, use only one round trip to > decide the ballot and also piggyback the current value in response. > 2) Propose new value, and then send out the commit request asynchronously, so > client will not wait for the ack of the commit. In case of commit failures, > we should still have chance to retry/repair it through hints or following > read/cas events. > > After the improvement, we should be able to finish the CAS operation using 2 > rounds trips. There can be following improvements as well, and this can be a > start point. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14905) if SizeEstimatesRecorder misses a 'onDropTable' notification, the size_estimates table will never be cleared for that table.
[ https://issues.apache.org/jira/browse/CASSANDRA-14905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ariel Weisberg updated CASSANDRA-14905: --- Reviewer: Ariel Weisberg > if SizeEstimatesRecorder misses a 'onDropTable' notification, the > size_estimates table will never be cleared for that table. > > > Key: CASSANDRA-14905 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14905 > Project: Cassandra > Issue Type: Bug > Components: Metrics >Reporter: Aleksandr Sorokoumov >Assignee: Aleksandr Sorokoumov >Priority: Minor > Fix For: 3.0.x, 3.11.x, 4.0.x > > Attachments: 14905-3.0-dtest.png, 14905-3.0-testall.png, > 14905-3.11-dtest.png, 14905-3.11-testall.png, 14905-4.0-dtest.png, > 14905-4.0-testall.png > > > if a node is down when a keyspace/table is dropped, it will receive the > schema notification before the size estimates listener is registered, so the > entries for the dropped keyspace/table will never be cleaned from the table. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-14155) [TRUNK] Gossiper somewhat frequently hitting an NPE on node startup with dtests at org.apache.cassandra.gms.Gossiper.isSafeForStartup(Gossiper.java:769)
[
https://issues.apache.org/jira/browse/CASSANDRA-14155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726920#comment-16726920
]
Ariel Weisberg edited comment on CASSANDRA-14155 at 12/21/18 5:22 PM:
--
Got the endpoint state map + some additional logging during the shadow round.
Container 5 , rolling upgrade ssl test
https://circleci.com/gh/aweisberg/cassandra/2329#artifacts/containers/99
https://2329-26126838-gh.circle-artifacts.com/5/dtest_upgrade_no_vnodes_logs/1545411075201_test_rolling_upgrade_with_internode_ssl/node1_debug.log
{noformat}
java.lang.AssertionError: previous (Gossip ApplicationState.HOST_ID for this
node) is null: Messages = Starting shadow gossip round to check for endpoint
collision at 127.0.0.2:7000
Sending shadow round GOSSIP DIGEST SYN to seeds [127.0.0.1:7000, 127.0.0.3:7000]
Received a regular ack from 127.0.0.1:7000, can now exit shadow round, epStates
= {127.0.0.2:7000=EndpointState: HeartBeatState = HeartBeat: generation =
154535, version = 2147483647, AppStateMap = {},
127.0.0.1:7000=EndpointState: HeartBeatState = HeartBeat: generation =
1545333481, version = 123, AppStateMap = {}}
at org.apache.cassandra.gms.Gossiper.isSafeForStartup(Gossiper.java:834)
at
org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:556)
at
org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:835)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:693)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:644)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:369)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:581)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:670)
{noformat}
https://2329-26126838-gh.circle-artifacts.com/5/dtest_upgrade_no_vnodes_logs/1545411075201_test_rolling_upgrade_with_internode_ssl/node2_debug.log
{noformat}
java.lang.AssertionError: previous (Gossip ApplicationState.HOST_ID for this
node) is null: Messages = Starting shadow gossip round to check for endpoint
collision at 127.0.0.2:7000
Sending shadow round GOSSIP DIGEST SYN to seeds [127.0.0.1:7000, 127.0.0.3:7000]
Received a regular ack from 127.0.0.1:7000, can now exit shadow round,
epStateMap {127.0.0.3:7000=EndpointState: HeartBeatState = HeartBeat:
generation = 1545410617, version = 240, AppStateMap = {},
127.0.0.2:7000=EndpointState: HeartBeatState = HeartBeat: generation =
1545410603, version = 2147483647, AppStateMap = {},
127.0.0.1:7000=EndpointState: HeartBeatState = HeartBeat: generation =
1545410741, version = 123, AppStateMap = {}}, epStates =
{127.0.0.3:7000=EndpointState: HeartBeatState = HeartBeat: generation =
1545410617, version = 240, AppStateMap = {}, 127.0.0.2:7000=EndpointState:
HeartBeatState = HeartBeat: generation = 1545410603, version = 2147483647,
AppStateMap = {}, 127.0.0.1:7000=EndpointState: HeartBeatState = HeartBeat:
generation = 1545410741, version = 123, AppStateMap = {}}
at org.apache.cassandra.gms.Gossiper.isSafeForStartup(Gossiper.java:834)
at
org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:556)
at
org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:835)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:693)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:644)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:369)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:581)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:670)
{noformat}
was (Author: aweisberg):
Got the endpoint state map + some additional logging during the shadow round.
{noformat}
java.lang.AssertionError: previous (Gossip ApplicationState.HOST_ID for this
node) is null: Messages = Starting shadow gossip round to check for endpoint
collision at 127.0.0.2:7000
Sending shadow round GOSSIP DIGEST SYN to seeds [127.0.0.1:7000, 127.0.0.3:7000]
Received a regular ack from 127.0.0.1:7000, can now exit shadow round, epStates
= {127.0.0.2:7000=EndpointState: HeartBeatState = HeartBeat: generation =
154535, version = 2147483647, AppStateMap = {},
127.0.0.1:7000=EndpointState: HeartBeatState = HeartBeat: generation =
1545333481, version = 123, AppStateMap = {}}
at org.apache.cassandra.gms.Gossiper.isSafeForStartup(Gossiper.java:834)
at
org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:556)
at
org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:835)
at
[jira] [Commented] (CASSANDRA-14905) if SizeEstimatesRecorder misses a 'onDropTable' notification, the size_estimates table will never be cleared for that table.
[ https://issues.apache.org/jira/browse/CASSANDRA-14905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726874#comment-16726874 ] Aleksandr Sorokoumov commented on CASSANDRA-14905: -- CI: ||3.0||3.11||4.0|| |[dtest|https://issues.apache.org/jira/secure/attachment/12952697/14905-3.0-dtest.png]|[dtest|https://issues.apache.org/jira/secure/attachment/12952699/14905-3.11-dtest.png]|[dtest|https://issues.apache.org/jira/secure/attachment/12952701/14905-4.0-dtest.png]| |[testall|https://issues.apache.org/jira/secure/attachment/12952698/14905-3.0-testall.png]|[testall|https://issues.apache.org/jira/secure/attachment/12952700/14905-3.11-testall.png]|[testall|https://issues.apache.org/jira/secure/attachment/12952702/14905-4.0-testall.png]| > if SizeEstimatesRecorder misses a 'onDropTable' notification, the > size_estimates table will never be cleared for that table. > > > Key: CASSANDRA-14905 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14905 > Project: Cassandra > Issue Type: Bug > Components: Metrics >Reporter: Aleksandr Sorokoumov >Assignee: Aleksandr Sorokoumov >Priority: Minor > Fix For: 3.0.x, 3.11.x, 4.0.x > > Attachments: 14905-3.0-dtest.png, 14905-3.0-testall.png, > 14905-3.11-dtest.png, 14905-3.11-testall.png, 14905-4.0-dtest.png, > 14905-4.0-testall.png > > > if a node is down when a keyspace/table is dropped, it will receive the > schema notification before the size estimates listener is registered, so the > entries for the dropped keyspace/table will never be cleaned from the table. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14905) if SizeEstimatesRecorder misses a 'onDropTable' notification, the size_estimates table will never be cleared for that table.
[ https://issues.apache.org/jira/browse/CASSANDRA-14905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksandr Sorokoumov updated CASSANDRA-14905: - Attachment: 14905-4.0-testall.png 14905-4.0-dtest.png 14905-3.11-testall.png 14905-3.11-dtest.png 14905-3.0-testall.png 14905-3.0-dtest.png > if SizeEstimatesRecorder misses a 'onDropTable' notification, the > size_estimates table will never be cleared for that table. > > > Key: CASSANDRA-14905 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14905 > Project: Cassandra > Issue Type: Bug > Components: Metrics >Reporter: Aleksandr Sorokoumov >Assignee: Aleksandr Sorokoumov >Priority: Minor > Fix For: 3.0.x, 3.11.x, 4.0.x > > Attachments: 14905-3.0-dtest.png, 14905-3.0-testall.png, > 14905-3.11-dtest.png, 14905-3.11-testall.png, 14905-4.0-dtest.png, > 14905-4.0-testall.png > > > if a node is down when a keyspace/table is dropped, it will receive the > schema notification before the size estimates listener is registered, so the > entries for the dropped keyspace/table will never be cleaned from the table. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14945) During garbagecollect use index check for tombstone removals
[ https://issues.apache.org/jira/browse/CASSANDRA-14945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vitalii Ishchenko updated CASSANDRA-14945: -- Priority: Minor (was: Major) > During garbagecollect use index check for tombstone removals > > > Key: CASSANDRA-14945 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14945 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Vitalii Ishchenko >Priority: Minor > > nodetool garbagecollect command may be further improved to actually remove > more tombstones. Right now bloom filter is used and may give false positives > preventing tombstone removal if min table timestamp is less than tombstone > timestamp. > Disabling bloom filter should do the trick, but currently there is a bug in > compaction with bloom filter turned off CASSANDRA-14944 > As improvement index can be always checked if garbagecollect compaction is > done -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14945) During garbagecollect use index check for tombstone removals
Vitalii Ishchenko created CASSANDRA-14945: - Summary: During garbagecollect use index check for tombstone removals Key: CASSANDRA-14945 URL: https://issues.apache.org/jira/browse/CASSANDRA-14945 Project: Cassandra Issue Type: Improvement Components: Compaction Reporter: Vitalii Ishchenko nodetool garbagecollect command may be further improved to actually remove more tombstones. Right now bloom filter is used and may give false positives preventing tombstone removal if min table timestamp is less than tombstone timestamp. Disabling bloom filter should do the trick, but currently there is a bug in compaction with bloom filter turned off CASSANDRA-14944 As improvement index can be always checked if garbagecollect compaction is done -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14944) Tombstones are not removed if bloom filter is turned off
Vitalii Ishchenko created CASSANDRA-14944:
-
Summary: Tombstones are not removed if bloom filter is turned off
Key: CASSANDRA-14944
URL: https://issues.apache.org/jira/browse/CASSANDRA-14944
Project: Cassandra
Issue Type: Bug
Components: Compaction
Reporter: Vitalii Ishchenko
Well actually they are deleted, but not always even though they should, because
check is done using Index of overlapped tables
When purge evaluator is constructed we are checking overlapping tables using
bloom filter, but when it is disabled we are checking against index, but if
condition is not properly constructed and we still check bloom filter which is
AlwaysPresentFilter and every overlapping sstable is used to get minTimestamp
and tombstones, that have their timestamp >= minTimestamp won't be deleted
{code:java}
if (sstable.getBloomFilter() instanceof AlwaysPresentFilter &&
sstable.getPosition(key, SSTableReader.Operator.EQ, false) != null
|| sstable.getBloomFilter().isPresent(key))
{
{code}
Should be something like this
{code:java}
boolean mightBePresentInTable = sstable.getBloomFilter() instanceof
AlwaysPresentFilter ? sstable.getPosition(key, SSTableReader.Operator.EQ,
false) != null : sstable.getBloomFilter().isPresent(key)
{code}
Code pointers in 3.11
[https://github.com/apache/cassandra/blob/08363afa5354c00a7ecd62fe273c392a678db28a/src/java/org/apache/cassandra/db/compaction/CompactionController.java#L274]
and 4.0
[https://github.com/apache/cassandra/blob/11384c3279a66e6c0fb7861e2b188b25e963580f/src/java/org/apache/cassandra/db/compaction/CompactionController.java#L281]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14943) Java Driver - insert UDT type
Pruteanu Dragos created CASSANDRA-14943:
---
Summary: Java Driver - insert UDT type
Key: CASSANDRA-14943
URL: https://issues.apache.org/jira/browse/CASSANDRA-14943
Project: Cassandra
Issue Type: New Feature
Reporter: Pruteanu Dragos
Inserting data in UDT fields is possible using string statements like:
INSERT INTO sampleTable( id, udt_filed ) VALUES ( 1, \{a:'aaa',b:'bbb} )
This seems not to work if using Prepared statement and passing udt_field as
setObject( map ) or setString( "JSON '\{a:'aaa',b:'bbb} '").
Could be one of this be fixed ? I need this for external tools which accept
less java objects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14942) cqlsh cannot describe keyspace when having table schema extensions
vincent royer created CASSANDRA-14942:
-
Summary: cqlsh cannot describe keyspace when having table schema
extensions
Key: CASSANDRA-14942
URL: https://issues.apache.org/jira/browse/CASSANDRA-14942
Project: Cassandra
Issue Type: Bug
Components: CQL
Environment: Cassandra 3.11.3
The issue comes from the cqlsh utility, in cassandra/metadata.py, function
_all_as_cql, viewkeys is not known.
Reporter: vincent royer
When adding a schema table extension to a table, cqlsh failed to describe
keyspace or table with the following error message: 'module' object has no
attribute 'viewkeys'
Step to reproduce the issue:
{{docker run --name some-cassandra -d docker.io/cassandra:3.11.3}}{{docker exec
-it cqlsh < cqlsh -e "DESC KEYSPACE ks"}}
{{'module' object has no attribute 'viewkeys'}}
docker exec -it c75a002959e2 cqlsh --debug -e "DESC KEYSPACE ks"
Using CQL driver:
Using connect timeout: 5 seconds
Using 'utf-8' encoding
Using ssl: False
Traceback (most recent call last):
File "/usr/bin/cqlsh.py", line 925, in onecmd
self.handle_statement(st, statementtext)
File "/usr/bin/cqlsh.py", line 962, in handle_statement
return custom_handler(parsed)
File "/usr/bin/cqlsh.py", line 1545, in do_describe
self.describe_keyspace(ksname)
File "/usr/bin/cqlsh.py", line 1281, in describe_keyspace
self.print_recreate_keyspace(self.get_keyspace_meta(ksname), sys.stdout)
File "/usr/bin/cqlsh.py", line 1231, in print_recreate_keyspace
out.write(ksdef.export_as_string())
File
"/usr/share/cassandra/lib/cassandra-driver-internal-only-3.11.0-bb96859b.zip/cassandra-driver-3.11.0-bb96859b/cassandra/metadata.py",
line 661, in export_as_string
+ [t.export_as_string() for t in self.tables.values()])
File
"/usr/share/cassandra/lib/cassandra-driver-internal-only-3.11.0-bb96859b.zip/cassandra-driver-3.11.0-bb96859b/cassandra/metadata.py",
line 1116, in export_as_string
ret = self._all_as_cql()
File
"/usr/share/cassandra/lib/cassandra-driver-internal-only-3.11.0-bb96859b.zip/cassandra-driver-3.11.0-bb96859b/cassandra/metadata.py",
line 1135, in _all_as_cql
for k in six.viewkeys(registry) & self.extensions: # no viewkeys on
OrderedMapSerializeKey
AttributeError: 'module' object has no attribute 'viewkeys'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13365) Nodes entering GC loop, does not recover
[
https://issues.apache.org/jira/browse/CASSANDRA-13365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726569#comment-16726569
]
Tuo Zhu commented on CASSANDRA-13365:
-
[~rushman] We don't have any materialized views. But I found the data is not
balanced in our 3 nodes cluster. I'm not sure if this related to the issue.
UN 172.16.213.213 1.46 TiB 256 ? c5853306-d369-46b6-94bb-cc38ff4ab0ff rack1
UN 172.16.213.215 1.05 TiB 256 ? 1ab8e142-0b1b-454f-97fa-ba4959e7e5e2 rack1
UN 172.16.213.217 1.09 TiB 256 ? 611b74e5-5d06-4a54-8bc0-39159d469ada rack1
I don't know if it's normal. We're using 256 tokens on each node.
> Nodes entering GC loop, does not recover
>
>
> Key: CASSANDRA-13365
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13365
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: 34-node cluster over 4 DCs
> Linux CentOS 7.2 x86
> Mix of 64GB/128GB RAM / node
> Mix of 32/40 hardware threads / node, Xeon ~2.4Ghz
> High read volume, low write volume, occasional sstable bulk loading
>Reporter: Mina Naguib
>Priority: Major
>
> Over the last week we've been observing two related problems affecting our
> Cassandra cluster
> Problem 1: 1-few nodes per DC entering GC loop, not recovering
> Checking the heap usage stats, there's a sudden jump of 1-3GB. Some nodes
> recover, but some don't and log this:
> {noformat}
> 2017-03-21T11:23:02.957-0400: 54099.519: [Full GC (Allocation Failure)
> 13G->11G(14G), 29.4127307 secs]
> 2017-03-21T11:23:45.270-0400: 54141.833: [Full GC (Allocation Failure)
> 13G->12G(14G), 28.1561881 secs]
> 2017-03-21T11:24:20.307-0400: 54176.869: [Full GC (Allocation Failure)
> 13G->13G(14G), 27.7019501 secs]
> 2017-03-21T11:24:50.528-0400: 54207.090: [Full GC (Allocation Failure)
> 13G->13G(14G), 27.1372267 secs]
> 2017-03-21T11:25:19.190-0400: 54235.752: [Full GC (Allocation Failure)
> 13G->13G(14G), 27.0703975 secs]
> 2017-03-21T11:25:46.711-0400: 54263.273: [Full GC (Allocation Failure)
> 13G->13G(14G), 27.3187768 secs]
> 2017-03-21T11:26:15.419-0400: 54291.981: [Full GC (Allocation Failure)
> 13G->13G(14G), 26.9493405 secs]
> 2017-03-21T11:26:43.399-0400: 54319.961: [Full GC (Allocation Failure)
> 13G->13G(14G), 27.5222085 secs]
> 2017-03-21T11:27:11.383-0400: 54347.945: [Full GC (Allocation Failure)
> 13G->13G(14G), 27.1769581 secs]
> 2017-03-21T11:27:40.174-0400: 54376.737: [Full GC (Allocation Failure)
> 13G->13G(14G), 27.4639031 secs]
> 2017-03-21T11:28:08.946-0400: 54405.508: [Full GC (Allocation Failure)
> 13G->13G(14G), 30.3480523 secs]
> 2017-03-21T11:28:40.117-0400: 54436.680: [Full GC (Allocation Failure)
> 13G->13G(14G), 27.8220513 secs]
> 2017-03-21T11:29:08.459-0400: 54465.022: [Full GC (Allocation Failure)
> 13G->13G(14G), 27.4691271 secs]
> 2017-03-21T11:29:37.114-0400: 54493.676: [Full GC (Allocation Failure)
> 13G->13G(14G), 27.0275733 secs]
> 2017-03-21T11:30:04.635-0400: 54521.198: [Full GC (Allocation Failure)
> 13G->13G(14G), 27.1902627 secs]
> 2017-03-21T11:30:32.114-0400: 54548.676: [Full GC (Allocation Failure)
> 13G->13G(14G), 27.8872850 secs]
> 2017-03-21T11:31:01.430-0400: 54577.993: [Full GC (Allocation Failure)
> 13G->13G(14G), 27.1609706 secs]
> 2017-03-21T11:31:29.024-0400: 54605.587: [Full GC (Allocation Failure)
> 13G->13G(14G), 27.3635138 secs]
> 2017-03-21T11:31:57.303-0400: 54633.865: [Full GC (Allocation Failure)
> 13G->13G(14G), 27.4143510 secs]
> 2017-03-21T11:32:25.110-0400: 54661.672: [Full GC (Allocation Failure)
> 13G->13G(14G), 27.8595986 secs]
> 2017-03-21T11:32:53.922-0400: 54690.485: [Full GC (Allocation Failure)
> 13G->13G(14G), 27.5242543 secs]
> 2017-03-21T11:33:21.867-0400: 54718.429: [Full GC (Allocation Failure)
> 13G->13G(14G), 30.8930130 secs]
> 2017-03-21T11:33:53.712-0400: 54750.275: [Full GC (Allocation Failure)
> 13G->13G(14G), 27.6523013 secs]
> 2017-03-21T11:34:21.760-0400: 54778.322: [Full GC (Allocation Failure)
> 13G->13G(14G), 27.3030198 secs]
> 2017-03-21T11:34:50.073-0400: 54806.635: [Full GC (Allocation Failure)
> 13G->13G(14G), 27.1594154 secs]
> 2017-03-21T11:35:17.743-0400: 54834.306: [Full GC (Allocation Failure)
> 13G->13G(14G), 27.3766949 secs]
> 2017-03-21T11:35:45.797-0400: 54862.360: [Full GC (Allocation Failure)
> 13G->13G(14G), 27.5756770 secs]
> 2017-03-21T11:36:13.816-0400: 54890.378: [Full GC (Allocation Failure)
> 13G->13G(14G), 27.5541813 secs]
> 2017-03-21T11:36:41.926-0400: 54918.488: [Full GC (Allocation Failure)
> 13G->13G(14G), 33.7510103 secs]
> 2017-03-21T11:37:16.132-0400: 54952.695: [Full GC (Allocation Failure)
> 13G->13G(14G), 27.4856611 secs]
> 2017-03-21T11:37:44.454-0400: 54981.017: [Full GC (Allocation Failure)
> 13G->13G(14G), 28.1269335 secs]
> 2017-03-21T11:38:12.774-0400: 55009.337: [Full
