Re: Trying to identify the problem with CQL ...

2013-03-05 Thread Sylvain Lebresne
You do seem to be experiencing a problem indeed, but hard to say what it is
from just this (could be anything from a configuration problem, a cassandra
bug or a bug in the java driver).
But since you seem to be able to reproduce easily, if you can provide a
script that reproduce that issue, I'd gladly have a look (feel free to send
it to me privately if you prefer).

--
Sylvain


On Tue, Mar 5, 2013 at 4:40 PM, Gabriel Ciuloaica wrote:

>  So, I have added more logging to the test app (comments inline). For some
> reason I'm loosing updates.  In a for loop I'm executing upload, read
> writetime, download blob. Executed 10 times... See iteration number 2 and
> 3
>
> 1. initialize session
>
> 0[main] INFO  com.datastax.driver.core.Cluster  - New Cassandra host /
> 10.11.1.109 added
> 1[main] INFO  com.datastax.driver.core.Cluster  - New Cassandra host /
> 10.11.1.200 added
>
>
>
> COORDINATOR: 10.11.1.108
> UPLOAD LENGTH:214 bytes  --> uploading a blob (INSERT)
> UPLOAD:1154 ms
> Write time: 1362497519584000 ms --> reading writetime(avatar)
> DOWNLOAD LENGTH: 214 bytes   --> download the blob (SELECT)
> DOWNLOAD: 134 ms
> ---> md5 of the blob on
> upload ---> md5
> of the blob on download
> Upload Digest MD5 : b1944c41a25192520d33d15d00db2718 ===  Download Digest
> MD5 : b1944c41a25192520d33d15d00db2718
>   0
> COORDINATOR: 10.11.1.109
> UPLOAD LENGTH:4031 bytes
> UPLOAD:675 ms
> Write time: 1362497521493000 ms
> DOWNLOAD LENGTH: 4031 bytes
> DOWNLOAD: 135 ms
> Upload Digest MD5 : 20b71b77f90b3f8ae8995a7ce7f68295 ===  Download Digest
> MD5 : 20b71b77f90b3f8ae8995a7ce7f68295
>   1
> COORDINATOR: 10.11.1.200
> UPLOAD LENGTH:3392 bytes
> UPLOAD:668 ms
> Write time: 1362497556815000 ms
> DOWNLOAD LENGTH: 3392 bytes
> DOWNLOAD: 136 ms
> Upload Digest MD5 : 1158e1ea54d46a4d0bd45becc4523585 ===  Download Digest
> MD5 : 1158e1ea54d46a4d0bd45becc4523585
>   2
> COORDINATOR: 10.11.1.108
> UPLOAD LENGTH:253 bytes
> UPLOAD:668 ms
> Write time: 1362497556815000 ms
> DOWNLOAD LENGTH: 3392 bytes
> DOWNLOAD: 136 ms
> Upload Digest MD5 : fc9ce009530d6018a80c344d87b8ada4 ===  Download Digest
> MD5 : 1158e1ea54d46a4d0bd45becc4523585
>   3
> COORDINATOR: 10.11.1.109
> UPLOAD LENGTH:266 bytes
> UPLOAD:704 ms
> Write time: 1362497556815000 ms
> DOWNLOAD LENGTH: 3392 bytes
> DOWNLOAD: 136 ms
> Upload Digest MD5 : 5726af06e91a520deed093aba6afe112 ===  Download Digest
> MD5 : 1158e1ea54d46a4d0bd45becc4523585
>   4
> COORDINATOR: 10.11.1.200
> UPLOAD LENGTH:3082 bytes
> UPLOAD:901 ms
> Write time: 1362497562076000 ms
> DOWNLOAD LENGTH: 3082 bytes
> DOWNLOAD: 135 ms
> Upload Digest MD5 : fa2ea1972992cafea1c71b6c3e718058 ===  Download Digest
> MD5 : fa2ea1972992cafea1c71b6c3e718058
>   5
> COORDINATOR: 10.11.1.108
> UPLOAD LENGTH:1481 bytes
> UPLOAD:703 ms
> Write time: 1362497562076000 ms
> DOWNLOAD LENGTH: 3082 bytes
> DOWNLOAD: 135 ms
> Upload Digest MD5 : f208e4d3ea133fad5f9d175052ca70cf ===  Download Digest
> MD5 : fa2ea1972992cafea1c71b6c3e718058
>   6
> COORDINATOR: 10.11.1.109
> UPLOAD LENGTH:5214 bytes
> UPLOAD:801 ms
> Write time: 1362497562076000 ms
> DOWNLOAD LENGTH: 3082 bytes
> DOWNLOAD: 134 ms
> Upload Digest MD5 : c58d92d8273c7c9a7db76363b0b3e4c7 ===  Download Digest
> MD5 : fa2ea1972992cafea1c71b6c3e718058
>   7
> COORDINATOR: 10.11.1.200
> UPLOAD LENGTH:2992 bytes
> UPLOAD:665 ms
> Write time: 1362497567779000 ms
> DOWNLOAD LENGTH: 2992 bytes
> DOWNLOAD: 134 ms
> Upload Digest MD5 : 0848513c1b4214adf73c6ea5509ec294 ===  Download Digest
> MD5 : 0848513c1b4214adf73c6ea5509ec294
>   8
> COORDINATOR: 10.11.1.108
> UPLOAD LENGTH:3670 bytes
> UPLOAD:672 ms
> Write time: 1362497567779000 ms
> DOWNLOAD LENGTH: 2992 bytes
> DOWNLOAD: 136 ms
> Upload Digest MD5 : 27e235b9a90a22004d4098a0228ee07b ===  Download Digest
> MD5 : 0848513c1b4214adf73c6ea5509ec294
>   9
>
>
> Thanks,
> Gabi
>
> On 3/5/13 1:31 PM, Gabriel Ciuloaica wrote:
>
> Hi Sylvain,
>
> thanks for fast answer. I have updated keyspace definition and
> cassandra-topologies.properties to all 3 nodes and restarted each node.
> Both problems are still reproducible. I'm not able to read my writes and
> also the selects shows same data as in my previous email.
>
> for write and read I'm using:
> private static final String WRITE_STATEMENT = "INSERT INTO avatars (id,
> image_type, avatar) VALUES (?,?,?);";
> private static final String READ_STATEMENT = "SELECT avatar, image_type
> FROM avatars WHERE id=?";
>
> I'm using java-driver (1.0.0-beta1) with prepared statement, sync calls.
>
> Write snippet:
>
> Session session;
> try {
> session = cassandraSession.getSess

Re: Trying to identify the problem with CQL ...

2013-03-05 Thread Gabriel Ciuloaica
So, I have added more logging to the test app (comments inline). For 
some reason I'm loosing updates.  In a for loop I'm executing upload, 
read writetime, download blob. Executed 10 times... See iteration number 
2 and 3


1. initialize session

0[main] INFO  com.datastax.driver.core.Cluster  - New Cassandra host 
/10.11.1.109 added
1[main] INFO  com.datastax.driver.core.Cluster  - New Cassandra host 
/10.11.1.200 added




COORDINATOR: 10.11.1.108
UPLOAD LENGTH:214 bytes  --> uploading a blob (INSERT)
UPLOAD:1154 ms
Write time: 1362497519584000 ms --> reading writetime(avatar)
DOWNLOAD LENGTH: 214 bytes   --> download the blob (SELECT)
DOWNLOAD: 134 ms
---> md5 of the blob on upload ---> md5 of the blob on download
Upload Digest MD5 : b1944c41a25192520d33d15d00db2718 ===  Download 
Digest MD5 : b1944c41a25192520d33d15d00db2718

  0
COORDINATOR: 10.11.1.109
UPLOAD LENGTH:4031 bytes
UPLOAD:675 ms
Write time: 1362497521493000 ms
DOWNLOAD LENGTH: 4031 bytes
DOWNLOAD: 135 ms
Upload Digest MD5 : 20b71b77f90b3f8ae8995a7ce7f68295 ===  Download 
Digest MD5 : 20b71b77f90b3f8ae8995a7ce7f68295

  1
COORDINATOR: 10.11.1.200
UPLOAD LENGTH:3392 bytes
UPLOAD:668 ms
Write time: 1362497556815000 ms
DOWNLOAD LENGTH: 3392 bytes
DOWNLOAD: 136 ms
Upload Digest MD5 : 1158e1ea54d46a4d0bd45becc4523585 ===  Download 
Digest MD5 : 1158e1ea54d46a4d0bd45becc4523585

  2
COORDINATOR: 10.11.1.108
UPLOAD LENGTH:253 bytes
UPLOAD:668 ms
Write time: 1362497556815000 ms
DOWNLOAD LENGTH: 3392 bytes
DOWNLOAD: 136 ms
Upload Digest MD5 : fc9ce009530d6018a80c344d87b8ada4 ===  Download 
Digest MD5 : 1158e1ea54d46a4d0bd45becc4523585

  3
COORDINATOR: 10.11.1.109
UPLOAD LENGTH:266 bytes
UPLOAD:704 ms
Write time: 1362497556815000 ms
DOWNLOAD LENGTH: 3392 bytes
DOWNLOAD: 136 ms
Upload Digest MD5 : 5726af06e91a520deed093aba6afe112 ===  Download 
Digest MD5 : 1158e1ea54d46a4d0bd45becc4523585

  4
COORDINATOR: 10.11.1.200
UPLOAD LENGTH:3082 bytes
UPLOAD:901 ms
Write time: 1362497562076000 ms
DOWNLOAD LENGTH: 3082 bytes
DOWNLOAD: 135 ms
Upload Digest MD5 : fa2ea1972992cafea1c71b6c3e718058 ===  Download 
Digest MD5 : fa2ea1972992cafea1c71b6c3e718058

  5
COORDINATOR: 10.11.1.108
UPLOAD LENGTH:1481 bytes
UPLOAD:703 ms
Write time: 1362497562076000 ms
DOWNLOAD LENGTH: 3082 bytes
DOWNLOAD: 135 ms
Upload Digest MD5 : f208e4d3ea133fad5f9d175052ca70cf ===  Download 
Digest MD5 : fa2ea1972992cafea1c71b6c3e718058

  6
COORDINATOR: 10.11.1.109
UPLOAD LENGTH:5214 bytes
UPLOAD:801 ms
Write time: 1362497562076000 ms
DOWNLOAD LENGTH: 3082 bytes
DOWNLOAD: 134 ms
Upload Digest MD5 : c58d92d8273c7c9a7db76363b0b3e4c7 ===  Download 
Digest MD5 : fa2ea1972992cafea1c71b6c3e718058

  7
COORDINATOR: 10.11.1.200
UPLOAD LENGTH:2992 bytes
UPLOAD:665 ms
Write time: 1362497567779000 ms
DOWNLOAD LENGTH: 2992 bytes
DOWNLOAD: 134 ms
Upload Digest MD5 : 0848513c1b4214adf73c6ea5509ec294 ===  Download 
Digest MD5 : 0848513c1b4214adf73c6ea5509ec294

  8
COORDINATOR: 10.11.1.108
UPLOAD LENGTH:3670 bytes
UPLOAD:672 ms
Write time: 1362497567779000 ms
DOWNLOAD LENGTH: 2992 bytes
DOWNLOAD: 136 ms
Upload Digest MD5 : 27e235b9a90a22004d4098a0228ee07b ===  Download 
Digest MD5 : 0848513c1b4214adf73c6ea5509ec294

  9


Thanks,
Gabi
On 3/5/13 1:31 PM, Gabriel Ciuloaica wrote:

Hi Sylvain,

thanks for fast answer. I have updated keyspace definition and 
cassandra-topologies.properties to all 3 nodes and restarted each 
node. Both problems are still reproducible. I'm not able to read my 
writes and also the selects shows same data as in my previous email.


for write and read I'm using:
private static final String WRITE_STATEMENT = "INSERT INTO avatars 
(id, image_type, avatar) VALUES (?,?,?);";
private static final String READ_STATEMENT = "SELECT avatar, 
image_type FROM avatars WHERE id=?";


I'm using java-driver (1.0.0-beta1) with prepared statement, sync calls.

Write snippet:

Session session;
try {
session = cassandraSession.getSession();
BoundStatement stmt = session.prepare(WRITE_STATEMENT)
.setConsistencyLevel(ConsistencyLevel.LOCAL_QUORUM).bind();
stmt.enableTracing();
stmt.setLong("id", accountId);
stmt.setString("image_type", image.getType());
stmt.setBytes("avatar", ByteBuffer.wrap(image.getBytes()));
ResultSet result = session.execute(stmt);
LOG.info("UPLOAD COORDINATOR: {}", result.getQueryTrace()
.getCoordinator().getCanonicalHostName());

} catch (NoHostAvailableException e) {
LOG.error("Could not prepare the statement.", e);
throw new StorageUnavailableException(e);
} finally {

Re: Trying to identify the problem with CQL ...

2013-03-05 Thread Gabriel Ciuloaica

Hi Sylvain,

thanks for fast answer. I have updated keyspace definition and 
cassandra-topologies.properties to all 3 nodes and restarted each node. 
Both problems are still reproducible. I'm not able to read my writes and 
also the selects shows same data as in my previous email.


for write and read I'm using:
private static final String WRITE_STATEMENT = "INSERT INTO avatars (id, 
image_type, avatar) VALUES (?,?,?);";
private static final String READ_STATEMENT = "SELECT avatar, image_type 
FROM avatars WHERE id=?";


I'm using java-driver (1.0.0-beta1) with prepared statement, sync calls.

Write snippet:

Session session;
try {
session = cassandraSession.getSession();
BoundStatement stmt = session.prepare(WRITE_STATEMENT)
.setConsistencyLevel(ConsistencyLevel.LOCAL_QUORUM).bind();
stmt.enableTracing();
stmt.setLong("id", accountId);
stmt.setString("image_type", image.getType());
stmt.setBytes("avatar", ByteBuffer.wrap(image.getBytes()));
ResultSet result = session.execute(stmt);
LOG.info("UPLOAD COORDINATOR: {}", result.getQueryTrace()
.getCoordinator().getCanonicalHostName());

} catch (NoHostAvailableException e) {
LOG.error("Could not prepare the statement.", e);
throw new StorageUnavailableException(e);
} finally {
cassandraSession.releaseSession();
}

Read snippet:

Session session = null;
byte[] imageBytes = null;
String imageType = "png";
try {
session = cassandraSession.getSession();
BoundStatement stmt = session.prepare(READ_STATEMENT)
.setConsistencyLevel(ConsistencyLevel.LOCAL_QUORUM).bind();
stmt.setLong("id", accountId);
ResultSet result = session.execute(stmt);
Iterator it = result.iterator();
ByteBuffer avatar = null;

while (it.hasNext()) {
Row row = it.next();
avatar = row.getBytes("avatar");
imageType = row.getString("image_type");
}
if (avatar == null) {
throw new AvatarNotFoundException("Avatar hasn't been 
found");

}
int length = avatar.remaining();
imageBytes = new byte[length];
avatar.get(imageBytes, 0, length);

} catch (NoHostAvailableException e) {
LOG.error("Could not prepare the statement.", e);
throw new StorageUnavailableException(e);
} finally {
cassandraSession.releaseSession();
}

Let me know what other information is need it.

Thanks,
Gabi


On 3/5/13 12:52 PM, Sylvain Lebresne wrote:
Without looking into details too closely, I'd say you're probably 
hitting https://issues.apache.org/jira/browse/CASSANDRA-5292 (since 
you use NTS+propertyFileSnitch+a DC name in caps).


Long story short, the CREATE KEYSPACE interpret your DC-TORONTO as 
dc-toronto, which then probably don't match what you have in you 
property file. This will be fixed in 1.2.3. In the meantime, a 
workaround would be to use the cassandra-cli to create/update your 
keyspace definition.


--
Sylvain


On Tue, Mar 5, 2013 at 11:24 AM, Gabriel Ciuloaica 
mailto:gciuloa...@gmail.com>> wrote:


Hello,

I'm trying to find out what the problem is and where it is located.
I have a 3 nodes Cassandra cluster (1.2.1), RF=3.
I have a keyspace and a cf as defined (using PropertyFileSnitch):

CREATE KEYSPACE backend WITH replication = {
  'class': 'NetworkTopologyStrategy',
  'DC-TORONTO': '3'
};

USE backend;

CREATE TABLE avatars (
  id bigint PRIMARY KEY,
  avatar blob,
  image_type text
) WITH
  bloom_filter_fp_chance=0.01 AND
  caching='KEYS_ONLY' AND
  comment='' AND
  dclocal_read_repair_chance=0.00 AND
  gc_grace_seconds=864000 AND
  read_repair_chance=0.10 AND
  replicate_on_write='true' AND
  compaction={'class': 'SizeTieredCompactionStrategy'} AND
  compression={'sstable_compression': 'SnappyCompressor'};

Status of the cluster:
Datacenter: DC-TORONTO
==
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address   Load   Tokens  Owns   Host ID  
Rack

UN  10.11.1.109   44.98 MB   256 46.8%
726689df-edc3-49a0-b680-370953994a8c  RAC2
UN  10.11.1.200   6.57 MB64  10.3%
d6d700d4-28aa-4722-b215-a6a7d304b8e7  RAC3
UN  10.11.1.108   54.32 MB   256 42.8%
73cd86a9-4efb-4407-9fe8-9a1b3a277af7  RAC1

I'm trying to read my writes, by using CQL (datastax-java-driver),
using LOCAL_QUORUM for reads and writes. For some reason, some of
the writes are lost. Not sure if it is a driver issue or cassandra
issue.
Dinging further, using cqlsh client (1.2.1), I found a strange
situat

Re: Trying to identify the problem with CQL ...

2013-03-05 Thread Sylvain Lebresne
Without looking into details too closely, I'd say you're probably hitting
https://issues.apache.org/jira/browse/CASSANDRA-5292 (since you use
NTS+propertyFileSnitch+a DC name in caps).

Long story short, the CREATE KEYSPACE interpret your DC-TORONTO as
dc-toronto, which then probably don't match what you have in you property
file. This will be fixed in 1.2.3. In the meantime, a workaround would be
to use the cassandra-cli to create/update your keyspace definition.

--
Sylvain


On Tue, Mar 5, 2013 at 11:24 AM, Gabriel Ciuloaica wrote:

> Hello,
>
> I'm trying to find out what the problem is and where it is located.
> I have a 3 nodes Cassandra cluster (1.2.1), RF=3.
> I have a keyspace and a cf as defined (using PropertyFileSnitch):
>
> CREATE KEYSPACE backend WITH replication = {
>   'class': 'NetworkTopologyStrategy',
>   'DC-TORONTO': '3'
> };
>
> USE backend;
>
> CREATE TABLE avatars (
>   id bigint PRIMARY KEY,
>   avatar blob,
>   image_type text
> ) WITH
>   bloom_filter_fp_chance=0.**01 AND
>   caching='KEYS_ONLY' AND
>   comment='' AND
>   dclocal_read_repair_chance=0.**00 AND
>   gc_grace_seconds=864000 AND
>   read_repair_chance=0.10 AND
>   replicate_on_write='true' AND
>   compaction={'class': 'SizeTieredCompactionStrategy'**} AND
>   compression={'sstable_**compression': 'SnappyCompressor'};
>
> Status of the cluster:
> Datacenter: DC-TORONTO
> ==
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/**Moving
> --  Address   Load   Tokens  Owns   Host ID
> Rack
> UN  10.11.1.109   44.98 MB   256 46.8% 
> 726689df-edc3-49a0-b680-**370953994a8c
>  RAC2
> UN  10.11.1.200   6.57 MB64  10.3% 
> d6d700d4-28aa-4722-b215-**a6a7d304b8e7
>  RAC3
> UN  10.11.1.108   54.32 MB   256 42.8% 
> 73cd86a9-4efb-4407-9fe8-**9a1b3a277af7
>  RAC1
>
> I'm trying to read my writes, by using CQL (datastax-java-driver), using
> LOCAL_QUORUM for reads and writes. For some reason, some of the writes are
> lost. Not sure if it is a driver issue or cassandra issue.
> Dinging further, using cqlsh client (1.2.1), I found a strange situation:
>
> select count(*) from avatars;
>
>  count
> ---
>226
>
> select id from avatars;
>
>  id
> -
>  314
>  396
>   19
>  .->  77 rows in result
>
> select id, image_type from avatars;
>
>  id  | image_type
> -+
>  332 |png
>  314 |png
>  396 |   jpeg
>   19 |png
>  1250014 |   jpeg
>  -> 226 rows in result.
>
> I do not understand why for second select I'm able to retrieve just a part
> of the rows and not all rows.
>
> Not sure if this is related or not to the initial problem.
>
> Any help is really appreciated.
> Thanks,
> Gabi
>
>
>
>
>


Trying to identify the problem with CQL ...

2013-03-05 Thread Gabriel Ciuloaica

Hello,

I'm trying to find out what the problem is and where it is located.
I have a 3 nodes Cassandra cluster (1.2.1), RF=3.
I have a keyspace and a cf as defined (using PropertyFileSnitch):

CREATE KEYSPACE backend WITH replication = {
  'class': 'NetworkTopologyStrategy',
  'DC-TORONTO': '3'
};

USE backend;

CREATE TABLE avatars (
  id bigint PRIMARY KEY,
  avatar blob,
  image_type text
) WITH
  bloom_filter_fp_chance=0.01 AND
  caching='KEYS_ONLY' AND
  comment='' AND
  dclocal_read_repair_chance=0.00 AND
  gc_grace_seconds=864000 AND
  read_repair_chance=0.10 AND
  replicate_on_write='true' AND
  compaction={'class': 'SizeTieredCompactionStrategy'} AND
  compression={'sstable_compression': 'SnappyCompressor'};

Status of the cluster:
Datacenter: DC-TORONTO
==
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address   Load   Tokens  Owns   Host 
ID   Rack
UN  10.11.1.109   44.98 MB   256 46.8% 
726689df-edc3-49a0-b680-370953994a8c  RAC2
UN  10.11.1.200   6.57 MB64  10.3% 
d6d700d4-28aa-4722-b215-a6a7d304b8e7  RAC3
UN  10.11.1.108   54.32 MB   256 42.8% 
73cd86a9-4efb-4407-9fe8-9a1b3a277af7  RAC1


I'm trying to read my writes, by using CQL (datastax-java-driver), using 
LOCAL_QUORUM for reads and writes. For some reason, some of the writes 
are lost. Not sure if it is a driver issue or cassandra issue.

Dinging further, using cqlsh client (1.2.1), I found a strange situation:

select count(*) from avatars;

 count
---
   226

select id from avatars;

 id
-
 314
 396
  19
 .->  77 rows in result

select id, image_type from avatars;

 id  | image_type
-+
 332 |png
 314 |png
 396 |   jpeg
  19 |png
 1250014 |   jpeg
 -> 226 rows in result.

I do not understand why for second select I'm able to retrieve just a 
part of the rows and not all rows.


Not sure if this is related or not to the initial problem.

Any help is really appreciated.
Thanks,
Gabi