So, I have added more logging to the test app (comments inline). For
some reason I'm loosing updates. In a for loop I'm executing upload,
read writetime, download blob. Executed 10 times... See iteration number
2 and 3....
1. initialize session
0 [main] INFO com.datastax.driver.core.Cluster - New Cassandra host
/10.11.1.109 added
1 [main] INFO com.datastax.driver.core.Cluster - New Cassandra host
/10.11.1.200 added
COORDINATOR: 10.11.1.108
UPLOAD LENGTH:214 bytes --> uploading a blob (INSERT)
UPLOAD:1154 ms
Write time: 1362497519584000 ms --> reading writetime(avatar)
DOWNLOAD LENGTH: 214 bytes --> download the blob (SELECT)
DOWNLOAD: 134 ms
---> md5 of the blob on upload ---> md5 of the blob on download
Upload Digest MD5 : b1944c41a25192520d33d15d00db2718 === Download
Digest MD5 : b1944c41a25192520d33d15d00db2718
-------------------------------- 0
COORDINATOR: 10.11.1.109
UPLOAD LENGTH:4031 bytes
UPLOAD:675 ms
Write time: 1362497521493000 ms
DOWNLOAD LENGTH: 4031 bytes
DOWNLOAD: 135 ms
Upload Digest MD5 : 20b71b77f90b3f8ae8995a7ce7f68295 === Download
Digest MD5 : 20b71b77f90b3f8ae8995a7ce7f68295
-------------------------------- 1
COORDINATOR: 10.11.1.200
UPLOAD LENGTH:3392 bytes
UPLOAD:668 ms
Write time: 1362497556815000 ms
DOWNLOAD LENGTH: 3392 bytes
DOWNLOAD: 136 ms
Upload Digest MD5 : 1158e1ea54d46a4d0bd45becc4523585 === Download
Digest MD5 : 1158e1ea54d46a4d0bd45becc4523585
-------------------------------- 2
COORDINATOR: 10.11.1.108
UPLOAD LENGTH:253 bytes
UPLOAD:668 ms
Write time: 1362497556815000 ms
DOWNLOAD LENGTH: 3392 bytes
DOWNLOAD: 136 ms
Upload Digest MD5 : fc9ce009530d6018a80c344d87b8ada4 === Download
Digest MD5 : 1158e1ea54d46a4d0bd45becc4523585
-------------------------------- 3
COORDINATOR: 10.11.1.109
UPLOAD LENGTH:266 bytes
UPLOAD:704 ms
Write time: 1362497556815000 ms
DOWNLOAD LENGTH: 3392 bytes
DOWNLOAD: 136 ms
Upload Digest MD5 : 5726af06e91a520deed093aba6afe112 === Download
Digest MD5 : 1158e1ea54d46a4d0bd45becc4523585
-------------------------------- 4
COORDINATOR: 10.11.1.200
UPLOAD LENGTH:3082 bytes
UPLOAD:901 ms
Write time: 1362497562076000 ms
DOWNLOAD LENGTH: 3082 bytes
DOWNLOAD: 135 ms
Upload Digest MD5 : fa2ea1972992cafea1c71b6c3e718058 === Download
Digest MD5 : fa2ea1972992cafea1c71b6c3e718058
-------------------------------- 5
COORDINATOR: 10.11.1.108
UPLOAD LENGTH:1481 bytes
UPLOAD:703 ms
Write time: 1362497562076000 ms
DOWNLOAD LENGTH: 3082 bytes
DOWNLOAD: 135 ms
Upload Digest MD5 : f208e4d3ea133fad5f9d175052ca70cf === Download
Digest MD5 : fa2ea1972992cafea1c71b6c3e718058
-------------------------------- 6
COORDINATOR: 10.11.1.109
UPLOAD LENGTH:5214 bytes
UPLOAD:801 ms
Write time: 1362497562076000 ms
DOWNLOAD LENGTH: 3082 bytes
DOWNLOAD: 134 ms
Upload Digest MD5 : c58d92d8273c7c9a7db76363b0b3e4c7 === Download
Digest MD5 : fa2ea1972992cafea1c71b6c3e718058
-------------------------------- 7
COORDINATOR: 10.11.1.200
UPLOAD LENGTH:2992 bytes
UPLOAD:665 ms
Write time: 1362497567779000 ms
DOWNLOAD LENGTH: 2992 bytes
DOWNLOAD: 134 ms
Upload Digest MD5 : 0848513c1b4214adf73c6ea5509ec294 === Download
Digest MD5 : 0848513c1b4214adf73c6ea5509ec294
-------------------------------- 8
COORDINATOR: 10.11.1.108
UPLOAD LENGTH:3670 bytes
UPLOAD:672 ms
Write time: 1362497567779000 ms
DOWNLOAD LENGTH: 2992 bytes
DOWNLOAD: 136 ms
Upload Digest MD5 : 27e235b9a90a22004d4098a0228ee07b === Download
Digest MD5 : 0848513c1b4214adf73c6ea5509ec294
-------------------------------- 9
Thanks,
Gabi
On 3/5/13 1:31 PM, Gabriel Ciuloaica wrote:
Hi Sylvain,
thanks for fast answer. I have updated keyspace definition and
cassandra-topologies.properties to all 3 nodes and restarted each
node. Both problems are still reproducible. I'm not able to read my
writes and also the selects shows same data as in my previous email.
for write and read I'm using:
private static final String WRITE_STATEMENT = "INSERT INTO avatars
(id, image_type, avatar) VALUES (?,?,?);";
private static final String READ_STATEMENT = "SELECT avatar,
image_type FROM avatars WHERE id=?";
I'm using java-driver (1.0.0-beta1) with prepared statement, sync calls.
Write snippet:
Session session;
try {
session = cassandraSession.getSession();
BoundStatement stmt = session.prepare(WRITE_STATEMENT)
.setConsistencyLevel(ConsistencyLevel.LOCAL_QUORUM).bind();
stmt.enableTracing();
stmt.setLong("id", accountId);
stmt.setString("image_type", image.getType());
stmt.setBytes("avatar", ByteBuffer.wrap(image.getBytes()));
ResultSet result = session.execute(stmt);
LOG.info("UPLOAD COORDINATOR: {}", result.getQueryTrace()
.getCoordinator().getCanonicalHostName());
} catch (NoHostAvailableException e) {
LOG.error("Could not prepare the statement.", e);
throw new StorageUnavailableException(e);
} finally {
cassandraSession.releaseSession();
}
Read snippet:
Session session = null;
byte[] imageBytes = null;
String imageType = "png";
try {
session = cassandraSession.getSession();
BoundStatement stmt = session.prepare(READ_STATEMENT)
.setConsistencyLevel(ConsistencyLevel.LOCAL_QUORUM).bind();
stmt.setLong("id", accountId);
ResultSet result = session.execute(stmt);
Iterator<Row> it = result.iterator();
ByteBuffer avatar = null;
while (it.hasNext()) {
Row row = it.next();
avatar = row.getBytes("avatar");
imageType = row.getString("image_type");
}
if (avatar == null) {
throw new AvatarNotFoundException("Avatar hasn't been
found");
}
int length = avatar.remaining();
imageBytes = new byte[length];
avatar.get(imageBytes, 0, length);
} catch (NoHostAvailableException e) {
LOG.error("Could not prepare the statement.", e);
throw new StorageUnavailableException(e);
} finally {
cassandraSession.releaseSession();
}
Let me know what other information is need it.
Thanks,
Gabi
On 3/5/13 12:52 PM, Sylvain Lebresne wrote:
Without looking into details too closely, I'd say you're probably
hitting https://issues.apache.org/jira/browse/CASSANDRA-5292 (since
you use NTS+propertyFileSnitch+a DC name in caps).
Long story short, the CREATE KEYSPACE interpret your DC-TORONTO as
dc-toronto, which then probably don't match what you have in you
property file. This will be fixed in 1.2.3. In the meantime, a
workaround would be to use the cassandra-cli to create/update your
keyspace definition.
--
Sylvain
On Tue, Mar 5, 2013 at 11:24 AM, Gabriel Ciuloaica
<gciuloa...@gmail.com <mailto:gciuloa...@gmail.com>> wrote:
Hello,
I'm trying to find out what the problem is and where it is located.
I have a 3 nodes Cassandra cluster (1.2.1), RF=3.
I have a keyspace and a cf as defined (using PropertyFileSnitch):
CREATE KEYSPACE backend WITH replication = {
'class': 'NetworkTopologyStrategy',
'DC-TORONTO': '3'
};
USE backend;
CREATE TABLE avatars (
id bigint PRIMARY KEY,
avatar blob,
image_type text
) WITH
bloom_filter_fp_chance=0.010000 AND
caching='KEYS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.000000 AND
gc_grace_seconds=864000 AND
read_repair_chance=0.100000 AND
replicate_on_write='true' AND
compaction={'class': 'SizeTieredCompactionStrategy'} AND
compression={'sstable_compression': 'SnappyCompressor'};
Status of the cluster:
Datacenter: DC-TORONTO
======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN 10.11.1.109 44.98 MB 256 46.8%
726689df-edc3-49a0-b680-370953994a8c RAC2
UN 10.11.1.200 6.57 MB 64 10.3%
d6d700d4-28aa-4722-b215-a6a7d304b8e7 RAC3
UN 10.11.1.108 54.32 MB 256 42.8%
73cd86a9-4efb-4407-9fe8-9a1b3a277af7 RAC1
I'm trying to read my writes, by using CQL
(datastax-java-driver), using LOCAL_QUORUM for reads and writes.
For some reason, some of the writes are lost. Not sure if it is a
driver issue or cassandra issue.
Dinging further, using cqlsh client (1.2.1), I found a strange
situation:
select count(*) from avatars;
count
-------
226
select id from avatars;
id
---------
314
396
19
......... -> 77 rows in result
select id, image_type from avatars;
id | image_type
---------+------------
332 | png
314 | png
396 | jpeg
19 | png
1250014 | jpeg
........ -> 226 rows in result.
I do not understand why for second select I'm able to retrieve
just a part of the rows and not all rows.
Not sure if this is related or not to the initial problem.
Any help is really appreciated.
Thanks,
Gabi
