[jira] [Commented] (CASSANDRA-11273) Exceptions during bootstrap cause bootstrap to hang (WORKAROUND)
[
https://issues.apache.org/jira/browse/CASSANDRA-11273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15172299#comment-15172299
]
Paulo Motta commented on CASSANDRA-11273:
-
Thanks for the clarification [~lonerzzz], you are correct, I misunderstood your
initial comment for some reason.
After investigation, I verified that while the consequence is different, this
has the same root cause of CASSANDRA-11050: on
[SchemaKeyspace.convertSchemaToMutations|https://github.com/apache/cassandra/blob/6237022e0234a71f5ca3d01aaefcecfd28bcdf71/src/java/org/apache/cassandra/schema/SchemaKeyspace.java#L341],
the {{dropped_columns}} table was not serialized during schema
synchronization, since it was not present in the tables listed in
{{SchemaKeyspace.ALL}}. The fix for CASSANDRA-11050, adding the
{{dropped_columns}} table to {{SchemaKeyspace.ALL}} will prevent this from
happening in the future. For this reason I'm closing this as a duplicate of
CASSANDRA-11050, while your workaround can be used if someone hit this before
3.4 is out.
Thanks for the report!
> Exceptions during bootstrap cause bootstrap to hang (WORKAROUND)
>
>
> Key: CASSANDRA-11273
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11273
> Project: Cassandra
> Issue Type: Bug
> Components: Lifecycle
> Environment: debian jesse patch current running Cassandra 3.0.3
>Reporter: Jason Kania
>
> When running bootstrap on a new node, the following problem can occur because
> Cassandra fails to recognize columns for some reason. The error prevents the
> bootstrap from finishing and hangs the bootstrap. If the bootstrap is
> resumed, it will get the same error and bootstrap cannot be completed. The
> workaround that I used is at the end.
> from 192.168.10.8
> ERROR [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857
> StreamSession.java:635 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
> Remote peer 192.168.10.10 failed stream session.
> INFO [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857
> StreamResultFuture.java:182 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
> Session with /192.168.10.10 is complete
> WARN [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,858
> StreamResultFuture.java:209 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
> Stream failed
> from 192.168.10.8 debug
> DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,414
> ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
> Received Received (79256340--11e5-9f70-7d76a8de8480, #0)
> DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,854
> ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
> Received Retry (f3a137e0-024b-11e5-bb31-0d2316086bf7, #0)
> DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,854
> ConnectionHandler.java:334 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
> Sending File (Header (cfId: f3a137e0-024b-11e5-bb31-0d2316086bf7, #0,
> version: ma, format: BIG, estimated keys: 128, transfer size: 4653,
> compressed?: true, repairedAt: 0, level: 0), file:
> /home/cassandra/data/sensordb/sensor/ma-76-big-Data.db)
> DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,854
> CompressedStreamWriter.java:63 - [Stream
> #c9868f90-ddbb-11e5-80c0-89f591237aca] Start streaming file
> /home/cassandra/data/sensordb/sensor/ma-76-big-Data.db to /192.168.10.10,
> repairedAt = 0, totalSize = 4653
> DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,854
> CompressedStreamWriter.java:94 - [Stream
> #c9868f90-ddbb-11e5-80c0-89f591237aca] Finished streaming file
> /home/cassandra/data/sensordb/sensor/ma-76-big-Data.db to /192.168.10.10,
> bytesTransferred = 4653, totalSize = 4653
> DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,855
> ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
> Received Retry (faa55490-024b-11e5-bb31-0d2316086bf7, #0)
> DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,855
> ConnectionHandler.java:334 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca]
> Sending File (Header (cfId: faa55490-024b-11e5-bb31-0d2316086bf7, #0,
> version: ma, format: BIG, estimated keys: 128, transfer size: 705,
> compressed?: true, repairedAt: 0, level: 0), file:
> /home/cassandra/data/sensordb/sensorUnit/ma-79-big-Data.db)
> DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,856
> CompressedStreamWriter.java:63 - [Stream
> #c9868f90-ddbb-11e5-80c0-89f591237aca] Start streaming file
> /home/cassandra/data/sensordb/sensorUnit/ma-79-big-Data.db to /192.168.10.10,
> repairedAt = 0, totalSize = 705
> DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,856
> CompressedStreamWriter.java:94 - [Stream
> #c9868f90-ddbb-11e5-80c0-89f591237aca] Finished streaming file
>
[jira] [Commented] (CASSANDRA-11273) Exceptions during bootstrap cause bootstrap to hang (WORKAROUND)
[ https://issues.apache.org/jira/browse/CASSANDRA-11273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15172145#comment-15172145 ] Jason Kania commented on CASSANDRA-11273: - The logs in the above Description are the errors that I saw during bootstrapping of new node 192.168.10.10. Node 192.168.10.8 is a working node in the cluster and not new. I ran nodetool repair on 192.168.10.8 without error prior to bootstrapping 192.168.10.10. If you look at the logs following the text "from 192.168.10.10" in the initial description text above, the errors there are what was seen during bootstrap. Previous to these logs, there were no other error logs. To work around, I did the steps highlighted following "Possible Workaround" in the above Description as I was unable to get Cassandra to bootstrap according automatically. > Exceptions during bootstrap cause bootstrap to hang (WORKAROUND) > > > Key: CASSANDRA-11273 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11273 > Project: Cassandra > Issue Type: Bug > Components: Lifecycle > Environment: debian jesse patch current running Cassandra 3.0.3 >Reporter: Jason Kania > > When running bootstrap on a new node, the following problem can occur because > Cassandra fails to recognize columns for some reason. The error prevents the > bootstrap from finishing and hangs the bootstrap. If the bootstrap is > resumed, it will get the same error and bootstrap cannot be completed. The > workaround that I used is at the end. > from 192.168.10.8 > ERROR [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857 > StreamSession.java:635 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] > Remote peer 192.168.10.10 failed stream session. > INFO [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857 > StreamResultFuture.java:182 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] > Session with /192.168.10.10 is complete > WARN [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,858 > StreamResultFuture.java:209 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] > Stream failed > from 192.168.10.8 debug > DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,414 > ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] > Received Received (79256340--11e5-9f70-7d76a8de8480, #0) > DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,854 > ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] > Received Retry (f3a137e0-024b-11e5-bb31-0d2316086bf7, #0) > DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,854 > ConnectionHandler.java:334 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] > Sending File (Header (cfId: f3a137e0-024b-11e5-bb31-0d2316086bf7, #0, > version: ma, format: BIG, estimated keys: 128, transfer size: 4653, > compressed?: true, repairedAt: 0, level: 0), file: > /home/cassandra/data/sensordb/sensor/ma-76-big-Data.db) > DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,854 > CompressedStreamWriter.java:63 - [Stream > #c9868f90-ddbb-11e5-80c0-89f591237aca] Start streaming file > /home/cassandra/data/sensordb/sensor/ma-76-big-Data.db to /192.168.10.10, > repairedAt = 0, totalSize = 4653 > DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,854 > CompressedStreamWriter.java:94 - [Stream > #c9868f90-ddbb-11e5-80c0-89f591237aca] Finished streaming file > /home/cassandra/data/sensordb/sensor/ma-76-big-Data.db to /192.168.10.10, > bytesTransferred = 4653, totalSize = 4653 > DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,855 > ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] > Received Retry (faa55490-024b-11e5-bb31-0d2316086bf7, #0) > DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,855 > ConnectionHandler.java:334 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] > Sending File (Header (cfId: faa55490-024b-11e5-bb31-0d2316086bf7, #0, > version: ma, format: BIG, estimated keys: 128, transfer size: 705, > compressed?: true, repairedAt: 0, level: 0), file: > /home/cassandra/data/sensordb/sensorUnit/ma-79-big-Data.db) > DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,856 > CompressedStreamWriter.java:63 - [Stream > #c9868f90-ddbb-11e5-80c0-89f591237aca] Start streaming file > /home/cassandra/data/sensordb/sensorUnit/ma-79-big-Data.db to /192.168.10.10, > repairedAt = 0, totalSize = 705 > DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,856 > CompressedStreamWriter.java:94 - [Stream > #c9868f90-ddbb-11e5-80c0-89f591237aca] Finished streaming file > /home/cassandra/data/sensordb/sensorUnit/ma-79-big-Data.db to /192.168.10.10, > bytesTransferred = 705, totalSize = 705 > DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857 > ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] > Received Session Failed > ERROR
[jira] [Commented] (CASSANDRA-11273) Exceptions during bootstrap cause bootstrap to hang (WORKAROUND)
[ https://issues.apache.org/jira/browse/CASSANDRA-11273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15172097#comment-15172097 ] Paulo Motta commented on CASSANDRA-11273: - This smells like some consequence of CASSANDRA-11050. As per your mailing list comments, you also had problems when boostrapping 192.168.10.10: bq. I was doing a bootstrapping on 192.168.10.10 and it had nothing on it to start with it. It was in the process of transferring the schema definitions that the bootstrap was failing. In the process of trying to get something working, I tried adding the dropped columns on the existing node and the new node but had no luck with that either. Do you have some logs of additional information of the boostrapping failure of 192.168.10.10? What did you do to fix? If the schema of 192.168.10.10 was corrupted (ie. missing dropped_columns data), then it could have propagated it to the new node (192.168.10.8), and thus causing the no columns found problem. So, the context of the failure of 192.168.10.10 to boostrap (and how you fixed it) is more important here to find out the root cause. > Exceptions during bootstrap cause bootstrap to hang (WORKAROUND) > > > Key: CASSANDRA-11273 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11273 > Project: Cassandra > Issue Type: Bug > Components: Lifecycle > Environment: debian jesse patch current running Cassandra 3.0.3 >Reporter: Jason Kania > > When running bootstrap on a new node, the following problem can occur because > Cassandra fails to recognize columns for some reason. The error prevents the > bootstrap from finishing and hangs the bootstrap. If the bootstrap is > resumed, it will get the same error and bootstrap cannot be completed. The > workaround that I used is at the end. > from 192.168.10.8 > ERROR [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857 > StreamSession.java:635 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] > Remote peer 192.168.10.10 failed stream session. > INFO [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857 > StreamResultFuture.java:182 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] > Session with /192.168.10.10 is complete > WARN [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,858 > StreamResultFuture.java:209 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] > Stream failed > from 192.168.10.8 debug > DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,414 > ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] > Received Received (79256340--11e5-9f70-7d76a8de8480, #0) > DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,854 > ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] > Received Retry (f3a137e0-024b-11e5-bb31-0d2316086bf7, #0) > DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,854 > ConnectionHandler.java:334 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] > Sending File (Header (cfId: f3a137e0-024b-11e5-bb31-0d2316086bf7, #0, > version: ma, format: BIG, estimated keys: 128, transfer size: 4653, > compressed?: true, repairedAt: 0, level: 0), file: > /home/cassandra/data/sensordb/sensor/ma-76-big-Data.db) > DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,854 > CompressedStreamWriter.java:63 - [Stream > #c9868f90-ddbb-11e5-80c0-89f591237aca] Start streaming file > /home/cassandra/data/sensordb/sensor/ma-76-big-Data.db to /192.168.10.10, > repairedAt = 0, totalSize = 4653 > DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,854 > CompressedStreamWriter.java:94 - [Stream > #c9868f90-ddbb-11e5-80c0-89f591237aca] Finished streaming file > /home/cassandra/data/sensordb/sensor/ma-76-big-Data.db to /192.168.10.10, > bytesTransferred = 4653, totalSize = 4653 > DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,855 > ConnectionHandler.java:262 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] > Received Retry (faa55490-024b-11e5-bb31-0d2316086bf7, #0) > DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,855 > ConnectionHandler.java:334 - [Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] > Sending File (Header (cfId: faa55490-024b-11e5-bb31-0d2316086bf7, #0, > version: ma, format: BIG, estimated keys: 128, transfer size: 705, > compressed?: true, repairedAt: 0, level: 0), file: > /home/cassandra/data/sensordb/sensorUnit/ma-79-big-Data.db) > DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,856 > CompressedStreamWriter.java:63 - [Stream > #c9868f90-ddbb-11e5-80c0-89f591237aca] Start streaming file > /home/cassandra/data/sensordb/sensorUnit/ma-79-big-Data.db to /192.168.10.10, > repairedAt = 0, totalSize = 705 > DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,856 > CompressedStreamWriter.java:94 - [Stream > #c9868f90-ddbb-11e5-80c0-89f591237aca] Finished streaming file >
