Re: Trying to identify the problem with CQL ...
You do seem to be experiencing a problem indeed, but hard to say what it is from just this (could be anything from a configuration problem, a cassandra bug or a bug in the java driver). But since you seem to be able to reproduce easily, if you can provide a script that reproduce that issue, I'd gladly have a look (feel free to send it to me privately if you prefer). -- Sylvain On Tue, Mar 5, 2013 at 4:40 PM, Gabriel Ciuloaica wrote: > So, I have added more logging to the test app (comments inline). For some > reason I'm loosing updates. In a for loop I'm executing upload, read > writetime, download blob. Executed 10 times... See iteration number 2 and > 3 > > 1. initialize session > > 0[main] INFO com.datastax.driver.core.Cluster - New Cassandra host / > 10.11.1.109 added > 1[main] INFO com.datastax.driver.core.Cluster - New Cassandra host / > 10.11.1.200 added > > > > COORDINATOR: 10.11.1.108 > UPLOAD LENGTH:214 bytes --> uploading a blob (INSERT) > UPLOAD:1154 ms > Write time: 1362497519584000 ms --> reading writetime(avatar) > DOWNLOAD LENGTH: 214 bytes --> download the blob (SELECT) > DOWNLOAD: 134 ms > ---> md5 of the blob on > upload ---> md5 > of the blob on download > Upload Digest MD5 : b1944c41a25192520d33d15d00db2718 === Download Digest > MD5 : b1944c41a25192520d33d15d00db2718 > 0 > COORDINATOR: 10.11.1.109 > UPLOAD LENGTH:4031 bytes > UPLOAD:675 ms > Write time: 1362497521493000 ms > DOWNLOAD LENGTH: 4031 bytes > DOWNLOAD: 135 ms > Upload Digest MD5 : 20b71b77f90b3f8ae8995a7ce7f68295 === Download Digest > MD5 : 20b71b77f90b3f8ae8995a7ce7f68295 > 1 > COORDINATOR: 10.11.1.200 > UPLOAD LENGTH:3392 bytes > UPLOAD:668 ms > Write time: 1362497556815000 ms > DOWNLOAD LENGTH: 3392 bytes > DOWNLOAD: 136 ms > Upload Digest MD5 : 1158e1ea54d46a4d0bd45becc4523585 === Download Digest > MD5 : 1158e1ea54d46a4d0bd45becc4523585 > 2 > COORDINATOR: 10.11.1.108 > UPLOAD LENGTH:253 bytes > UPLOAD:668 ms > Write time: 1362497556815000 ms > DOWNLOAD LENGTH: 3392 bytes > DOWNLOAD: 136 ms > Upload Digest MD5 : fc9ce009530d6018a80c344d87b8ada4 === Download Digest > MD5 : 1158e1ea54d46a4d0bd45becc4523585 > 3 > COORDINATOR: 10.11.1.109 > UPLOAD LENGTH:266 bytes > UPLOAD:704 ms > Write time: 1362497556815000 ms > DOWNLOAD LENGTH: 3392 bytes > DOWNLOAD: 136 ms > Upload Digest MD5 : 5726af06e91a520deed093aba6afe112 === Download Digest > MD5 : 1158e1ea54d46a4d0bd45becc4523585 > 4 > COORDINATOR: 10.11.1.200 > UPLOAD LENGTH:3082 bytes > UPLOAD:901 ms > Write time: 1362497562076000 ms > DOWNLOAD LENGTH: 3082 bytes > DOWNLOAD: 135 ms > Upload Digest MD5 : fa2ea1972992cafea1c71b6c3e718058 === Download Digest > MD5 : fa2ea1972992cafea1c71b6c3e718058 > 5 > COORDINATOR: 10.11.1.108 > UPLOAD LENGTH:1481 bytes > UPLOAD:703 ms > Write time: 1362497562076000 ms > DOWNLOAD LENGTH: 3082 bytes > DOWNLOAD: 135 ms > Upload Digest MD5 : f208e4d3ea133fad5f9d175052ca70cf === Download Digest > MD5 : fa2ea1972992cafea1c71b6c3e718058 > 6 > COORDINATOR: 10.11.1.109 > UPLOAD LENGTH:5214 bytes > UPLOAD:801 ms > Write time: 1362497562076000 ms > DOWNLOAD LENGTH: 3082 bytes > DOWNLOAD: 134 ms > Upload Digest MD5 : c58d92d8273c7c9a7db76363b0b3e4c7 === Download Digest > MD5 : fa2ea1972992cafea1c71b6c3e718058 > 7 > COORDINATOR: 10.11.1.200 > UPLOAD LENGTH:2992 bytes > UPLOAD:665 ms > Write time: 1362497567779000 ms > DOWNLOAD LENGTH: 2992 bytes > DOWNLOAD: 134 ms > Upload Digest MD5 : 0848513c1b4214adf73c6ea5509ec294 === Download Digest > MD5 : 0848513c1b4214adf73c6ea5509ec294 > 8 > COORDINATOR: 10.11.1.108 > UPLOAD LENGTH:3670 bytes > UPLOAD:672 ms > Write time: 1362497567779000 ms > DOWNLOAD LENGTH: 2992 bytes > DOWNLOAD: 136 ms > Upload Digest MD5 : 27e235b9a90a22004d4098a0228ee07b === Download Digest > MD5 : 0848513c1b4214adf73c6ea5509ec294 > 9 > > > Thanks, > Gabi > > On 3/5/13 1:31 PM, Gabriel Ciuloaica wrote: > > Hi Sylvain, > > thanks for fast answer. I have updated keyspace definition and > cassandra-topologies.properties to all 3 nodes and restarted each node. > Both problems are still reproducible. I'm not able to read my writes and > also the selects shows same data as in my previous email. > > for write and read I'm using: > private static final String WRITE_STATEMENT = "INSERT INTO avatars (id, > image_type, avatar) VALUES (?,?,?);"; > private static final String READ_STATEMENT = "SELECT avatar, image_type > FROM avatars WHERE id=?"; > > I'm using java-driver (1.0.0-beta1) with prepared statement, sync calls. > > Write snippet: > > Session session; > try { > session = cassandraSession.getSess
Re: Trying to identify the problem with CQL ...
So, I have added more logging to the test app (comments inline). For some reason I'm loosing updates. In a for loop I'm executing upload, read writetime, download blob. Executed 10 times... See iteration number 2 and 3 1. initialize session 0[main] INFO com.datastax.driver.core.Cluster - New Cassandra host /10.11.1.109 added 1[main] INFO com.datastax.driver.core.Cluster - New Cassandra host /10.11.1.200 added COORDINATOR: 10.11.1.108 UPLOAD LENGTH:214 bytes --> uploading a blob (INSERT) UPLOAD:1154 ms Write time: 1362497519584000 ms --> reading writetime(avatar) DOWNLOAD LENGTH: 214 bytes --> download the blob (SELECT) DOWNLOAD: 134 ms ---> md5 of the blob on upload ---> md5 of the blob on download Upload Digest MD5 : b1944c41a25192520d33d15d00db2718 === Download Digest MD5 : b1944c41a25192520d33d15d00db2718 0 COORDINATOR: 10.11.1.109 UPLOAD LENGTH:4031 bytes UPLOAD:675 ms Write time: 1362497521493000 ms DOWNLOAD LENGTH: 4031 bytes DOWNLOAD: 135 ms Upload Digest MD5 : 20b71b77f90b3f8ae8995a7ce7f68295 === Download Digest MD5 : 20b71b77f90b3f8ae8995a7ce7f68295 1 COORDINATOR: 10.11.1.200 UPLOAD LENGTH:3392 bytes UPLOAD:668 ms Write time: 1362497556815000 ms DOWNLOAD LENGTH: 3392 bytes DOWNLOAD: 136 ms Upload Digest MD5 : 1158e1ea54d46a4d0bd45becc4523585 === Download Digest MD5 : 1158e1ea54d46a4d0bd45becc4523585 2 COORDINATOR: 10.11.1.108 UPLOAD LENGTH:253 bytes UPLOAD:668 ms Write time: 1362497556815000 ms DOWNLOAD LENGTH: 3392 bytes DOWNLOAD: 136 ms Upload Digest MD5 : fc9ce009530d6018a80c344d87b8ada4 === Download Digest MD5 : 1158e1ea54d46a4d0bd45becc4523585 3 COORDINATOR: 10.11.1.109 UPLOAD LENGTH:266 bytes UPLOAD:704 ms Write time: 1362497556815000 ms DOWNLOAD LENGTH: 3392 bytes DOWNLOAD: 136 ms Upload Digest MD5 : 5726af06e91a520deed093aba6afe112 === Download Digest MD5 : 1158e1ea54d46a4d0bd45becc4523585 4 COORDINATOR: 10.11.1.200 UPLOAD LENGTH:3082 bytes UPLOAD:901 ms Write time: 1362497562076000 ms DOWNLOAD LENGTH: 3082 bytes DOWNLOAD: 135 ms Upload Digest MD5 : fa2ea1972992cafea1c71b6c3e718058 === Download Digest MD5 : fa2ea1972992cafea1c71b6c3e718058 5 COORDINATOR: 10.11.1.108 UPLOAD LENGTH:1481 bytes UPLOAD:703 ms Write time: 1362497562076000 ms DOWNLOAD LENGTH: 3082 bytes DOWNLOAD: 135 ms Upload Digest MD5 : f208e4d3ea133fad5f9d175052ca70cf === Download Digest MD5 : fa2ea1972992cafea1c71b6c3e718058 6 COORDINATOR: 10.11.1.109 UPLOAD LENGTH:5214 bytes UPLOAD:801 ms Write time: 1362497562076000 ms DOWNLOAD LENGTH: 3082 bytes DOWNLOAD: 134 ms Upload Digest MD5 : c58d92d8273c7c9a7db76363b0b3e4c7 === Download Digest MD5 : fa2ea1972992cafea1c71b6c3e718058 7 COORDINATOR: 10.11.1.200 UPLOAD LENGTH:2992 bytes UPLOAD:665 ms Write time: 1362497567779000 ms DOWNLOAD LENGTH: 2992 bytes DOWNLOAD: 134 ms Upload Digest MD5 : 0848513c1b4214adf73c6ea5509ec294 === Download Digest MD5 : 0848513c1b4214adf73c6ea5509ec294 8 COORDINATOR: 10.11.1.108 UPLOAD LENGTH:3670 bytes UPLOAD:672 ms Write time: 1362497567779000 ms DOWNLOAD LENGTH: 2992 bytes DOWNLOAD: 136 ms Upload Digest MD5 : 27e235b9a90a22004d4098a0228ee07b === Download Digest MD5 : 0848513c1b4214adf73c6ea5509ec294 9 Thanks, Gabi On 3/5/13 1:31 PM, Gabriel Ciuloaica wrote: Hi Sylvain, thanks for fast answer. I have updated keyspace definition and cassandra-topologies.properties to all 3 nodes and restarted each node. Both problems are still reproducible. I'm not able to read my writes and also the selects shows same data as in my previous email. for write and read I'm using: private static final String WRITE_STATEMENT = "INSERT INTO avatars (id, image_type, avatar) VALUES (?,?,?);"; private static final String READ_STATEMENT = "SELECT avatar, image_type FROM avatars WHERE id=?"; I'm using java-driver (1.0.0-beta1) with prepared statement, sync calls. Write snippet: Session session; try { session = cassandraSession.getSession(); BoundStatement stmt = session.prepare(WRITE_STATEMENT) .setConsistencyLevel(ConsistencyLevel.LOCAL_QUORUM).bind(); stmt.enableTracing(); stmt.setLong("id", accountId); stmt.setString("image_type", image.getType()); stmt.setBytes("avatar", ByteBuffer.wrap(image.getBytes())); ResultSet result = session.execute(stmt); LOG.info("UPLOAD COORDINATOR: {}", result.getQueryTrace() .getCoordinator().getCanonicalHostName()); } catch (NoHostAvailableException e) { LOG.error("Could not prepare the statement.", e); throw new StorageUnavailableException(e); } finally {
Re: Trying to identify the problem with CQL ...
Hi Sylvain, thanks for fast answer. I have updated keyspace definition and cassandra-topologies.properties to all 3 nodes and restarted each node. Both problems are still reproducible. I'm not able to read my writes and also the selects shows same data as in my previous email. for write and read I'm using: private static final String WRITE_STATEMENT = "INSERT INTO avatars (id, image_type, avatar) VALUES (?,?,?);"; private static final String READ_STATEMENT = "SELECT avatar, image_type FROM avatars WHERE id=?"; I'm using java-driver (1.0.0-beta1) with prepared statement, sync calls. Write snippet: Session session; try { session = cassandraSession.getSession(); BoundStatement stmt = session.prepare(WRITE_STATEMENT) .setConsistencyLevel(ConsistencyLevel.LOCAL_QUORUM).bind(); stmt.enableTracing(); stmt.setLong("id", accountId); stmt.setString("image_type", image.getType()); stmt.setBytes("avatar", ByteBuffer.wrap(image.getBytes())); ResultSet result = session.execute(stmt); LOG.info("UPLOAD COORDINATOR: {}", result.getQueryTrace() .getCoordinator().getCanonicalHostName()); } catch (NoHostAvailableException e) { LOG.error("Could not prepare the statement.", e); throw new StorageUnavailableException(e); } finally { cassandraSession.releaseSession(); } Read snippet: Session session = null; byte[] imageBytes = null; String imageType = "png"; try { session = cassandraSession.getSession(); BoundStatement stmt = session.prepare(READ_STATEMENT) .setConsistencyLevel(ConsistencyLevel.LOCAL_QUORUM).bind(); stmt.setLong("id", accountId); ResultSet result = session.execute(stmt); Iterator it = result.iterator(); ByteBuffer avatar = null; while (it.hasNext()) { Row row = it.next(); avatar = row.getBytes("avatar"); imageType = row.getString("image_type"); } if (avatar == null) { throw new AvatarNotFoundException("Avatar hasn't been found"); } int length = avatar.remaining(); imageBytes = new byte[length]; avatar.get(imageBytes, 0, length); } catch (NoHostAvailableException e) { LOG.error("Could not prepare the statement.", e); throw new StorageUnavailableException(e); } finally { cassandraSession.releaseSession(); } Let me know what other information is need it. Thanks, Gabi On 3/5/13 12:52 PM, Sylvain Lebresne wrote: Without looking into details too closely, I'd say you're probably hitting https://issues.apache.org/jira/browse/CASSANDRA-5292 (since you use NTS+propertyFileSnitch+a DC name in caps). Long story short, the CREATE KEYSPACE interpret your DC-TORONTO as dc-toronto, which then probably don't match what you have in you property file. This will be fixed in 1.2.3. In the meantime, a workaround would be to use the cassandra-cli to create/update your keyspace definition. -- Sylvain On Tue, Mar 5, 2013 at 11:24 AM, Gabriel Ciuloaica mailto:gciuloa...@gmail.com>> wrote: Hello, I'm trying to find out what the problem is and where it is located. I have a 3 nodes Cassandra cluster (1.2.1), RF=3. I have a keyspace and a cf as defined (using PropertyFileSnitch): CREATE KEYSPACE backend WITH replication = { 'class': 'NetworkTopologyStrategy', 'DC-TORONTO': '3' }; USE backend; CREATE TABLE avatars ( id bigint PRIMARY KEY, avatar blob, image_type text ) WITH bloom_filter_fp_chance=0.01 AND caching='KEYS_ONLY' AND comment='' AND dclocal_read_repair_chance=0.00 AND gc_grace_seconds=864000 AND read_repair_chance=0.10 AND replicate_on_write='true' AND compaction={'class': 'SizeTieredCompactionStrategy'} AND compression={'sstable_compression': 'SnappyCompressor'}; Status of the cluster: Datacenter: DC-TORONTO == Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 10.11.1.109 44.98 MB 256 46.8% 726689df-edc3-49a0-b680-370953994a8c RAC2 UN 10.11.1.200 6.57 MB64 10.3% d6d700d4-28aa-4722-b215-a6a7d304b8e7 RAC3 UN 10.11.1.108 54.32 MB 256 42.8% 73cd86a9-4efb-4407-9fe8-9a1b3a277af7 RAC1 I'm trying to read my writes, by using CQL (datastax-java-driver), using LOCAL_QUORUM for reads and writes. For some reason, some of the writes are lost. Not sure if it is a driver issue or cassandra issue. Dinging further, using cqlsh client (1.2.1), I found a strange situat
Re: Trying to identify the problem with CQL ...
Without looking into details too closely, I'd say you're probably hitting https://issues.apache.org/jira/browse/CASSANDRA-5292 (since you use NTS+propertyFileSnitch+a DC name in caps). Long story short, the CREATE KEYSPACE interpret your DC-TORONTO as dc-toronto, which then probably don't match what you have in you property file. This will be fixed in 1.2.3. In the meantime, a workaround would be to use the cassandra-cli to create/update your keyspace definition. -- Sylvain On Tue, Mar 5, 2013 at 11:24 AM, Gabriel Ciuloaica wrote: > Hello, > > I'm trying to find out what the problem is and where it is located. > I have a 3 nodes Cassandra cluster (1.2.1), RF=3. > I have a keyspace and a cf as defined (using PropertyFileSnitch): > > CREATE KEYSPACE backend WITH replication = { > 'class': 'NetworkTopologyStrategy', > 'DC-TORONTO': '3' > }; > > USE backend; > > CREATE TABLE avatars ( > id bigint PRIMARY KEY, > avatar blob, > image_type text > ) WITH > bloom_filter_fp_chance=0.**01 AND > caching='KEYS_ONLY' AND > comment='' AND > dclocal_read_repair_chance=0.**00 AND > gc_grace_seconds=864000 AND > read_repair_chance=0.10 AND > replicate_on_write='true' AND > compaction={'class': 'SizeTieredCompactionStrategy'**} AND > compression={'sstable_**compression': 'SnappyCompressor'}; > > Status of the cluster: > Datacenter: DC-TORONTO > == > Status=Up/Down > |/ State=Normal/Leaving/Joining/**Moving > -- Address Load Tokens Owns Host ID > Rack > UN 10.11.1.109 44.98 MB 256 46.8% > 726689df-edc3-49a0-b680-**370953994a8c > RAC2 > UN 10.11.1.200 6.57 MB64 10.3% > d6d700d4-28aa-4722-b215-**a6a7d304b8e7 > RAC3 > UN 10.11.1.108 54.32 MB 256 42.8% > 73cd86a9-4efb-4407-9fe8-**9a1b3a277af7 > RAC1 > > I'm trying to read my writes, by using CQL (datastax-java-driver), using > LOCAL_QUORUM for reads and writes. For some reason, some of the writes are > lost. Not sure if it is a driver issue or cassandra issue. > Dinging further, using cqlsh client (1.2.1), I found a strange situation: > > select count(*) from avatars; > > count > --- >226 > > select id from avatars; > > id > - > 314 > 396 > 19 > .-> 77 rows in result > > select id, image_type from avatars; > > id | image_type > -+ > 332 |png > 314 |png > 396 | jpeg > 19 |png > 1250014 | jpeg > -> 226 rows in result. > > I do not understand why for second select I'm able to retrieve just a part > of the rows and not all rows. > > Not sure if this is related or not to the initial problem. > > Any help is really appreciated. > Thanks, > Gabi > > > > >
Trying to identify the problem with CQL ...
Hello, I'm trying to find out what the problem is and where it is located. I have a 3 nodes Cassandra cluster (1.2.1), RF=3. I have a keyspace and a cf as defined (using PropertyFileSnitch): CREATE KEYSPACE backend WITH replication = { 'class': 'NetworkTopologyStrategy', 'DC-TORONTO': '3' }; USE backend; CREATE TABLE avatars ( id bigint PRIMARY KEY, avatar blob, image_type text ) WITH bloom_filter_fp_chance=0.01 AND caching='KEYS_ONLY' AND comment='' AND dclocal_read_repair_chance=0.00 AND gc_grace_seconds=864000 AND read_repair_chance=0.10 AND replicate_on_write='true' AND compaction={'class': 'SizeTieredCompactionStrategy'} AND compression={'sstable_compression': 'SnappyCompressor'}; Status of the cluster: Datacenter: DC-TORONTO == Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 10.11.1.109 44.98 MB 256 46.8% 726689df-edc3-49a0-b680-370953994a8c RAC2 UN 10.11.1.200 6.57 MB64 10.3% d6d700d4-28aa-4722-b215-a6a7d304b8e7 RAC3 UN 10.11.1.108 54.32 MB 256 42.8% 73cd86a9-4efb-4407-9fe8-9a1b3a277af7 RAC1 I'm trying to read my writes, by using CQL (datastax-java-driver), using LOCAL_QUORUM for reads and writes. For some reason, some of the writes are lost. Not sure if it is a driver issue or cassandra issue. Dinging further, using cqlsh client (1.2.1), I found a strange situation: select count(*) from avatars; count --- 226 select id from avatars; id - 314 396 19 .-> 77 rows in result select id, image_type from avatars; id | image_type -+ 332 |png 314 |png 396 | jpeg 19 |png 1250014 | jpeg -> 226 rows in result. I do not understand why for second select I'm able to retrieve just a part of the rows and not all rows. Not sure if this is related or not to the initial problem. Any help is really appreciated. Thanks, Gabi