[
https://issues.apache.org/jira/browse/CASSANDRA-1992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12982288#action_12982288
]
Mateusz Korniak commented on CASSANDRA-1992:
--------------------------------------------
Another thing:
By luck, I discovered that restarting one or both nodes makes data to be served
again intact.
To avoid picking token 2nd node randomness I set it explicitly to
85070591730234615865843651857942052864
.
So scenario is now:
Clean pycassa installs with tokens set to 0 and
85070591730234615865843651857942052864.
Starting first node [1], uploading data, verifing ok, ring:
192.168.3.4 Up Normal 59.02 KB 100.00% 0
Bootstraping 2nd node, waiting to finish streaming, data verification bad:
column_family: 'CF0'
row_key: 'row=0 '
Traceback (most recent call last):
File "test.py", line 36, in <module>
loaded_cols_dict = current_cf.get(row_key)
File "/usr/share/python2.7/site-packages/pycassa/columnfamily.py", line 362,
in new_f
File "/usr/share/python2.7/site-packages/pycassa/columnfamily.py", line 429,
in get
pycassa.cassandra.ttypes.NotFoundException: NotFoundException()
Final ring (same on both nodes):
192.168.3.4 Up Normal 199.28 KB 50.00% 0
192.168.3.8 Up Normal 135.7 KB 50.00%
85070591730234615865843651857942052864
Restaring 192.168.3.4, data _same_ _error_ as above, ring changes to:
192.168.3.4 Up Normal 201.51 KB 50.00% 0
192.168.3.8 Up Normal 135.7 KB 50.00%
85070591730234615865843651857942052864
Restarting 192.168.3.8, data _verified_ _ok_ , ring changes on (same on both
nodes) to:
192.168.3.4 Up Normal 201.51 KB 50.00% 0
192.168.3.8 Up Normal 145.8 KB 50.00%
85070591730234615865843651857942052864
[1] Logs from 1st 192.168.3.4 node:
http://beauty.ant.gliwice.pl/bugs/cassandra-bootstrap/logs_with_restart/system-3.4.log
[2] Logs from 2nd 192.168.3.8 node:
http://beauty.ant.gliwice.pl/bugs/cassandra-bootstrap/logs_with_restart/system-3.8.log
HIH, regards
> Bootstrap breaks data stored (missing rows, extra rows, column values
> modified)
> -------------------------------------------------------------------------------
>
> Key: CASSANDRA-1992
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1992
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0
> Environment: Linux 2.6.36-1 #1 SMP Tue Nov 9 09:56:02 CET 2010 x86_64
> Intel(R)_Core(TM)2_Quad_CPU____Q8300__@_2.50GHz PLD Linux
> glibc-2.12-4.i686
> java-sun-1.6.0.22-1.i686
> Reporter: Mateusz Korniak
> Assignee: Brandon Williams
> Fix For: 0.7.1
>
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> Scenario:
> Two fresh (empty /data /commitog /saved_caches dirs) cassandra installs.
> Start first one.
> Run data inserting program [1], run again in verify mode - all data intact.
> Bootstrap 2nd node.
> Run verification again, now it fails.
> Issue is very strange to me as cassandra works perfectly for me when cluster
> nodes stay the same for days now but any bootstrap ( 1 -> 2 nodes, 2 -> 3
> nodes, 2->3 nodes RF=2) breaks data.
> I am running cassandra with 1GB heap size, 32bit userland on 64bit kernels,
> not sure what else could matter there.
> Any hints ?
> Thanks in advance, regards.
> [1] simple program generating data and later verifying data.
> http://beauty.ant.gliwice.pl/bugs/cassandra-bootstrap/test.py
> [2] Logs from 1st node:
> http://beauty.ant.gliwice.pl/bugs/cassandra-bootstrap/system-3.4.log
> [3] Logs from 2nd (bootstraping node)
> http://beauty.ant.gliwice.pl/bugs/cassandra-bootstrap/system-3.8.log
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
