Re: is this correct, thrift unportable to CQL3Š.
On Tue, Sep 24, 2013 at 7:24 PM, Hiller, Dean dean.hil...@nrel.gov wrote:
Java has Integer as opposed to int which is what I represent golf scores
with so to speak in my example. In this case, Integer can be null but of
course maps to empty just fine. What about querying for all golf scores
that are empty then?
The DataStax java driver already maps a null Integer to a null CQL3
column, and so it *cannot* also map it to an empty value. And even though
you cannot have null CQL3 columns in the primary key, mapping a null
Integer to an empty value would be highly confusing/inconsistent (from the
point of view of the java driver) because:
1) having a null Integer map differently depending on whether the receiving
column is in the PK or not would be messy/surprising.
2) it's not really technically possible because internally the driver does
not always have the information of whether the receiving column is in the
PK or not
3) what about strings? Should we also map a null String to an empty value
when the column is in the PK? Because that would mean mapping a null java
String to an empty CQL3 string value and that just feel wrong. And if we
don't do it, then it means we treat Integer and String differently with
respect to null and that's too would be pretty confusing.
But anyway, all of that is pretty specific to the java driver, so maybe if
you guys want to debate that further we should move to the driver mailing
list:
https://groups.google.com/a/lists.datastax.com/forum/#!forum/java-driver-user
--
Sylvain
Ie. This sounds like this solution would be perfect if we can get rid of
the exception piece.
Thanks much!!!
Dean
From: Sylvain Lebresne sylv...@datastax.commailto:sylv...@datastax.com
Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
user@cassandra.apache.orgmailto:user@cassandra.apache.org
Date: Tuesday, September 24, 2013 10:58 AM
To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
user@cassandra.apache.orgmailto:user@cassandra.apache.org
Subject: Re: is this correct, thrift unportable to CQL3Š.
On Tue, Sep 24, 2013 at 6:17 PM, Vikas Goyal vi...@easility.commailto:
vi...@easility.com wrote:
Ok. Great. It works for String and Decimal/Float but not for integer data
type..
i.e,, if I am passing to the composite key column which is either text
or float, it works..
session.execute(boundStatement.bind(rowkey, , ByteBuffer.wrap(value)));
But not working with bigint, int or varint..and getting following
exception;
Exception:Invalid type for value 1 of CQL type varint, expecting class
java.math.BigInteger but class java.lang.String provided
..I am exploring more though..
We're getting into details of the java driver at this point. For int, what
you can do is:
boundStatement.setString(id, rowkey);
boundStatement.setBytesUnsafe(columnname, ByteBuffer.wrap(new
byte[0]));
boundStatement.setBytes(columnvalue, ByteBuffer.wrap(value));
In other words, the shorthand BoundStatement.bind() won't let you insert
an empty value for types
that don't naturally have an empty value (like int; keep in mind that an
empty value is not the same
than null). But you can circumvent that using setBytesUnsafe if you really
want.
Long story short, my advice would be to avoid using empty values for type
that don't naturally have
one (anything else than text and blob really). If you do, that's going to
be painful (but possible) to
work with, at least with the java driver (but for good reasons, java just
doesn't have a good to
represent an empty int value (again, you can have a null Integer but
that's different)).
--
Sylvain
Thanks a ton,
Vikas Goyal
On Tue, Sep 24, 2013 at 9:05 PM, Sylvain Lebresne sylv...@datastax.com
mailto:sylv...@datastax.com wrote:
However,we tried missing the value but it didn't work :(
Right, because not providing a value is akin to having a null value (in
the CQL3 sense of the term, which is different from what Dean asked about)
and null values are not allowed for primary key columns.
You could however insert an *empty* value if you wanted, which in you case
is just inserting an empty string since colname is a string. Thrift doesn't
allow more or less in that case.
--
Sylvain
So our code is like below where we are using 3 values if colname is not
null..else 2 values..
if (key != null) {
PreparedStatement statement = session.prepare(INSERT INTO
keys.StringIndice (id, colname, colvalue) VALUES (?, ?, ?));
BoundStatement boundStatement = new
BoundStatement(statement);
session.execute(boundStatement.bind(StandardConverters.convertFromBytes(String.class,
rowKey), key, ByteBuffer.wrap(value)));
} else {
PreparedStatement statement = session.prepare(INSERT INTO
+ keys + . + table + (id, colvalue) VALUES (?, ?));
BoundStatement boundStatement = new
BoundStatement(statement);
Mystery PIG issue with 1.2.10
Heya!
I am seeing something rather strange in the way Cass 1.2 + Pig seem to handle
integer values.
Setup: Cassandra 1.2.10, OSX 10.8, JDK 1.7u40, Pig 0.11.1. Single node for
testing this.
First a table:
CREATE TABLE testc (
key text PRIMARY KEY,
ivalue int,
svalue text,
value bigint
) WITH COMPACT STORAGE;
insert into testc (key,ivalue,svalue,value) values ('foo',10,'bar',65);
select * from testc;
key | ivalue | svalue | value
-+++---
foo | 10 |bar | 65
For my Pig setup, I then use libraries from different C* versions to actually
talk to my database (which stays on 1.2.10 all the time).
Cassandra 1.0.12 (using cassandra_storage.jar):
testc = LOAD 'cassandra://keyspace/testc' USING CassandraStorage();
dump testc
(foo,(svalue,bar),(ivalue,10),(value,65),{})
Cassandra 1.1.10:
testc = LOAD 'cassandra://keyspace/testc' USING CassandraStorage();
dump testc
(foo,(svalue,bar),(ivalue,10),(value,65),{})
Cassandra 1.2.10:
(testc = LOAD 'cassandra://keyspace/testc' USING CassandraStorage();
dump testc
foo,{(ivalue,
),(svalue,bar),(value,A)})
To me it appears that ints and bigints are interpreted as ascii values in cass
1.2.10. Did something change for CassandraStorage, is there a regression, or
am I doing something wrong? Quick perusal of the JIRA didn't reveal anything
that I could directly pin on this.
Note that using compact storage does not seem to affect the issue, though it
obviously changes the resulting pig format.
In addition, trying to use Pygmalion
tf = foreach testc generate key,
flatten(FromCassandraBag('ivalue,svalue,value',columns)) as
(ivalue:int,svalue:chararray,lvalue:long);
dump tf
(foo,
,bar,A)
So no help there. Explicitly casting the values to (long) or (int) just results
in a ClassCastException.
/Janne
Re: Migration LCS from 1.2.X to 2.0.x exception
this is the issue: https://issues.apache.org/jira/browse/CASSANDRA-5383 guess it fell between chairs, will poke around On Tue, Sep 24, 2013 at 4:26 PM, Nate McCall n...@thelastpickle.com wrote: What version of 1.2.x? Unfortunately, you must go through 1.2.9 first. See https://github.com/apache/cassandra/blob/cassandra-2.0.0/NEWS.txt#L19-L24 On Tue, Sep 24, 2013 at 8:57 AM, Desimpel, Ignace ignace.desim...@nuance.com wrote: Tested on WINDOWS : On startup of the 2.0.0 version from 1.2.x files I get an error as listed below. ** ** This is due to the code in LeveledManifest:: mutateLevel. The method already has a comment saying that it is scary … On windows, one cannot use the File::rename if the target file name already exists. Also, even on Linux, I’m not sure if a rename would actually ‘overwrite/implicit-delete’ the content of the target file. ** ** Anyway, adding code (below) before the FileUtils.renameWithConfirm should work in both cases (maybe even rename the fromFile to be able to recover…) File oTo = new File(filename); if ( oTo.exists() ) oTo.delete(); ** ** ** ** java.lang.RuntimeException: Failed to rename …..xxx\Function-ic-10-Statistics.db-tmp to …..xxx\Function-ic-10-Statistics.db at org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:136) ~[main/:na] at org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:125) ~[main/:na] at org.apache.cassandra.db.compaction.LeveledManifest.mutateLevel(LeveledManifest.java:601) ~[main/:na] at org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:103) ~[main/:na] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:247) ~[main/:na] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:443) ~[main/:na] ** ** Regards, ** ** Ignace Desimpel
Re: 1.2.10 - 2.0.1 migration issue
this is most likely a bug, filed https://issues.apache.org/jira/browse/CASSANDRA-6093 and will try to have a look today. On Wed, Sep 25, 2013 at 1:48 AM, Christopher Wirt chris.w...@struq.comwrote: Hi, ** ** Just had a go at upgrading a node to the latest stable c* 2 release and think I ran into some issues with manifest migration. ** ** On initial start up I hit this error as it starts to load the first of my CF. ** ** INFO [main] 2013-09-24 22:56:01,018 LegacyLeveledManifest.java (line 89) Migrating manifest for struqrealtime/impressionstorev2 INFO [main] 2013-09-24 22:56:01,019 LegacyLeveledManifest.java (line 119) Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration* *** ERROR [main] 2013-09-24 22:56:01,030 CassandraDaemon.java (line 459) Exception encountered during startup FSWriteError in /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:83)** ** at org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:138) at org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485) Caused by: java.nio.file.NoSuchFileException: /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json - /disk1/cassandra/data/struqrealtime/impressionstorev2/impressionstorev2.json at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:474) at java.nio.file.Files.createLink(Files.java:1037) at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:79)** ** ... 5 more ** ** I had already successful run a test migration on our dev server. Only real difference I can see if the number of data directories defined and the amount of data being held. ** ** I’ve run upgradesstables under 1.2.10. I have always been using vnodes and CQL3. I recently moved to using LZ4 instead of Snappy.. ** ** I tried to startup again and it gave me a slightly different error ** ** INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 89) Migrating manifest for struqrealtime/impressionstorev2 INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 119) Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration* *** ERROR [main] 2013-09-24 22:58:28,222 CassandraDaemon.java (line 459) Exception encountered during startup java.lang.RuntimeException: Tried to create duplicate hard link to /disk3/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/struqrealtime-impressionstorev2-ic-1030-TOC.txt at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)** ** at org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:129) at org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485) ** ** Will have a go recreating this tomorrow. ** ** Any insight or guesses at what the issue might be are always welcome. ** ** Thanks, Chris
Re: is this correct, thrift unportable to CQL3Š.
On Wed, Sep 25, 2013 at 12:22 AM, Sylvain Lebresne sylv...@datastax.comwrote:
On Tue, Sep 24, 2013 at 7:24 PM, Hiller, Dean dean.hil...@nrel.govwrote:
Java has Integer as opposed to int which is what I represent golf
scores with so to speak in my example. In this case, Integer can be null
but of course maps to empty just fine. What about querying for all golf
scores that are empty then?
The DataStax java driver already maps a null Integer to a null CQL3
column, and so it *cannot* also map it to an empty value. And even though
you cannot have null CQL3 columns in the primary key, mapping a null
Integer to an empty value would be highly confusing/inconsistent (from the
point of view of the java driver) because:
1) having a null Integer map differently depending on whether the
receiving column is in the PK or not would be messy/surprising.
2) it's not really technically possible because internally the driver does
not always have the information of whether the receiving column is in the
PK or not
3) what about strings? Should we also map a null String to an empty value
when the column is in the PK? Because that would mean mapping a null java
String to an empty CQL3 string value and that just feel wrong. And if we
don't do it, then it means we treat Integer and String differently with
respect to null and that's too would be pretty confusing.
But anyway, all of that is pretty specific to the java driver, so maybe if
you guys want to debate that further we should move to the driver mailing
list:
https://groups.google.com/a/lists.datastax.com/forum/#!forum/java-driver-user
Nulls should definitely stay null (that's an old DB debate), but if this
is a generic case, we might consider adding an emptyValue() helper method.
Would that be possible Sylvain?
--
Sylvain
Ie. This sounds like this solution would be perfect if we can get rid of
the exception piece.
Thanks much!!!
Dean
From: Sylvain Lebresne sylv...@datastax.commailto:sylv...@datastax.com
Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
user@cassandra.apache.orgmailto:user@cassandra.apache.org
Date: Tuesday, September 24, 2013 10:58 AM
To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
user@cassandra.apache.orgmailto:user@cassandra.apache.org
Subject: Re: is this correct, thrift unportable to CQL3Š.
On Tue, Sep 24, 2013 at 6:17 PM, Vikas Goyal vi...@easility.commailto:
vi...@easility.com wrote:
Ok. Great. It works for String and Decimal/Float but not for integer data
type..
i.e,, if I am passing to the composite key column which is either text
or float, it works..
session.execute(boundStatement.bind(rowkey, , ByteBuffer.wrap(value)));
But not working with bigint, int or varint..and getting following
exception;
Exception:Invalid type for value 1 of CQL type varint, expecting class
java.math.BigInteger but class java.lang.String provided
..I am exploring more though..
We're getting into details of the java driver at this point. For int,
what you can do is:
boundStatement.setString(id, rowkey);
boundStatement.setBytesUnsafe(columnname, ByteBuffer.wrap(new
byte[0]));
boundStatement.setBytes(columnvalue, ByteBuffer.wrap(value));
In other words, the shorthand BoundStatement.bind() won't let you insert
an empty value for types
that don't naturally have an empty value (like int; keep in mind that an
empty value is not the same
than null). But you can circumvent that using setBytesUnsafe if you
really want.
Long story short, my advice would be to avoid using empty values for type
that don't naturally have
one (anything else than text and blob really). If you do, that's going to
be painful (but possible) to
work with, at least with the java driver (but for good reasons, java just
doesn't have a good to
represent an empty int value (again, you can have a null Integer but
that's different)).
--
Sylvain
Thanks a ton,
Vikas Goyal
On Tue, Sep 24, 2013 at 9:05 PM, Sylvain Lebresne sylv...@datastax.com
mailto:sylv...@datastax.com wrote:
However,we tried missing the value but it didn't work :(
Right, because not providing a value is akin to having a null value (in
the CQL3 sense of the term, which is different from what Dean asked about)
and null values are not allowed for primary key columns.
You could however insert an *empty* value if you wanted, which in you
case is just inserting an empty string since colname is a string. Thrift
doesn't allow more or less in that case.
--
Sylvain
So our code is like below where we are using 3 values if colname is not
null..else 2 values..
if (key != null) {
PreparedStatement statement = session.prepare(INSERT
INTO keys.StringIndice (id, colname, colvalue) VALUES (?, ?, ?));
BoundStatement boundStatement = new
BoundStatement(statement);
session.execute(boundStatement.bind(StandardConverters.convertFromBytes(String.class,
Re: 1.2.10 - 2.0.1 migration issue
cant really reproduce, could you update the ticket with a bit more info about your setup? do you have multiple .json files in your data dirs? On Wed, Sep 25, 2013 at 10:07 AM, Marcus Eriksson krum...@gmail.com wrote: this is most likely a bug, filed https://issues.apache.org/jira/browse/CASSANDRA-6093 and will try to have a look today. On Wed, Sep 25, 2013 at 1:48 AM, Christopher Wirt chris.w...@struq.comwrote: Hi, ** ** Just had a go at upgrading a node to the latest stable c* 2 release and think I ran into some issues with manifest migration. ** ** On initial start up I hit this error as it starts to load the first of my CF. ** ** INFO [main] 2013-09-24 22:56:01,018 LegacyLeveledManifest.java (line 89) Migrating manifest for struqrealtime/impressionstorev2 INFO [main] 2013-09-24 22:56:01,019 LegacyLeveledManifest.java (line 119) Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration ERROR [main] 2013-09-24 22:56:01,030 CassandraDaemon.java (line 459) Exception encountered during startup FSWriteError in /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:83)* *** at org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:138) at org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485) Caused by: java.nio.file.NoSuchFileException: /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json - /disk1/cassandra/data/struqrealtime/impressionstorev2/impressionstorev2.json at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)*** * at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:474) at java.nio.file.Files.createLink(Files.java:1037) at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:79)* *** ... 5 more ** ** I had already successful run a test migration on our dev server. Only real difference I can see if the number of data directories defined and the amount of data being held. ** ** I’ve run upgradesstables under 1.2.10. I have always been using vnodes and CQL3. I recently moved to using LZ4 instead of Snappy.. ** ** I tried to startup again and it gave me a slightly different error ** ** INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 89) Migrating manifest for struqrealtime/impressionstorev2 INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 119) Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration ERROR [main] 2013-09-24 22:58:28,222 CassandraDaemon.java (line 459) Exception encountered during startup java.lang.RuntimeException: Tried to create duplicate hard link to /disk3/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/struqrealtime-impressionstorev2-ic-1030-TOC.txt at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)* *** at org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:129) at org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485) ** ** Will have a go recreating this tomorrow. ** ** Any insight or guesses at what the issue might be are always welcome. ** ** Thanks, Chris
Re: is this correct, thrift unportable to CQL3Š.
Nulls should definitely stay null (that's an old DB debate), but if this
is a generic case, we might consider adding an emptyValue() helper method.
Would that be possible Sylvain?
As said previously, debating this is outside the scope of this mailing
list, but to make it short, using empty values for types that don't
naturally support them like Integer is going to be painful (inserting is
one thing, handling them on the read side is another) and confusing no
matter what. So you should avoid it in the first place (and there is *tons*
of way to do that) and consequently adding an helper to do something you
should avoid is a bad idea.
--
Sylvain
--
Sylvain
Ie. This sounds like this solution would be perfect if we can get rid of
the exception piece.
Thanks much!!!
Dean
From: Sylvain Lebresne sylv...@datastax.commailto:sylv...@datastax.com
Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
user@cassandra.apache.orgmailto:user@cassandra.apache.org
Date: Tuesday, September 24, 2013 10:58 AM
To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
user@cassandra.apache.orgmailto:user@cassandra.apache.org
Subject: Re: is this correct, thrift unportable to CQL3Š.
On Tue, Sep 24, 2013 at 6:17 PM, Vikas Goyal vi...@easility.commailto:
vi...@easility.com wrote:
Ok. Great. It works for String and Decimal/Float but not for integer
data type..
i.e,, if I am passing to the composite key column which is either
text or float, it works..
session.execute(boundStatement.bind(rowkey, , ByteBuffer.wrap(value)));
But not working with bigint, int or varint..and getting following
exception;
Exception:Invalid type for value 1 of CQL type varint, expecting class
java.math.BigInteger but class java.lang.String provided
..I am exploring more though..
We're getting into details of the java driver at this point. For int,
what you can do is:
boundStatement.setString(id, rowkey);
boundStatement.setBytesUnsafe(columnname, ByteBuffer.wrap(new
byte[0]));
boundStatement.setBytes(columnvalue, ByteBuffer.wrap(value));
In other words, the shorthand BoundStatement.bind() won't let you insert
an empty value for types
that don't naturally have an empty value (like int; keep in mind that an
empty value is not the same
than null). But you can circumvent that using setBytesUnsafe if you
really want.
Long story short, my advice would be to avoid using empty values for
type that don't naturally have
one (anything else than text and blob really). If you do, that's going
to be painful (but possible) to
work with, at least with the java driver (but for good reasons, java
just doesn't have a good to
represent an empty int value (again, you can have a null Integer but
that's different)).
--
Sylvain
Thanks a ton,
Vikas Goyal
On Tue, Sep 24, 2013 at 9:05 PM, Sylvain Lebresne sylv...@datastax.com
mailto:sylv...@datastax.com wrote:
However,we tried missing the value but it didn't work :(
Right, because not providing a value is akin to having a null value (in
the CQL3 sense of the term, which is different from what Dean asked about)
and null values are not allowed for primary key columns.
You could however insert an *empty* value if you wanted, which in you
case is just inserting an empty string since colname is a string. Thrift
doesn't allow more or less in that case.
--
Sylvain
So our code is like below where we are using 3 values if colname is not
null..else 2 values..
if (key != null) {
PreparedStatement statement = session.prepare(INSERT
INTO keys.StringIndice (id, colname, colvalue) VALUES (?, ?, ?));
BoundStatement boundStatement = new
BoundStatement(statement);
session.execute(boundStatement.bind(StandardConverters.convertFromBytes(String.class,
rowKey), key, ByteBuffer.wrap(value)));
} else {
PreparedStatement statement = session.prepare(INSERT
INTO + keys + . + table + (id, colvalue) VALUES (?, ?));
BoundStatement boundStatement = new
BoundStatement(statement);
session.execute(boundStatement.bind(StandardConverters.convertFromBytes(String.class,
rowKey), ByteBuffer.wrap(value)));
}
And, I did that and getting this exception:
Exception:Missing PRIMARY KEY part colname since colvalue is set
And just FYI. Our Column Family definition is below:
CREATE TABLE keys.StringIndice (id text,
colname text,
colvalue blob,
PRIMARY KEY (id,colname, colvalue)) WITH COMPACT STORAGE)
Thanks again,
Vikas Goyal
On Tue, Sep 24, 2013 at 7:02 PM, Sylvain Lebresne sylv...@datastax.com
mailto:sylv...@datastax.com wrote:
Short answer: not, this is not correct.
Longer answer: what you call null is actually an empty value (which is
*not* the same thing, unless you consider an empty string is the same thing
than a null string). As it happens, C* always an empty value as a valid
value for any type and that's true of
Re: Bad Request: Invalid null value for clustering key part
I am generating the CSV file by other means than the COPY ... TO command
and I thought that the NULL option did not apply to the COPY ... FROM
command but, as it turns out, it does! This solved my problem.
The documentation here (
http://www.datastax.com/documentation/cql/3.1/webhelp/index.html#cql/cql_reference/copy_r.html)
is misleading. Should I report this by other means than this post?
Thanks,
Petter
2013/9/24 Sylvain Lebresne sylv...@datastax.com
Oh. That would be a COPY thing then. I'm not extremely familiar with cqlsh
code (which COPY is part of) but looking at the online help for it, it
seems to have a 'NULL' option that allows to define which character is used
to represent nulls. And by default, it does is an empty string. So you
could try something like:
COPY ... TO ... WITH NULL='null'
(assuming that if you do have nulls you use the string 'null' to encode it
in your csv and that your are sure nothing that's not supposed to be null
will be represented by the string 'null').
--
Sylvain
On Tue, Sep 24, 2013 at 9:41 AM, Petter von Dolwitz (Hem)
petter.von.dolw...@gmail.com wrote:
Hi Sylvain,
I was not describing the problem correctly. I'm sorry for this. This is
the situation:
1. I'm populating the DB with the java-driver (INSERT INTO...). Some
fields that are part of the primary key is *empty strings*. This works fine.
2. I'm trying to populate the DB from a CSV (COPY ... FROM) using cqlsh.
Some fields that are part of the primary key is *empty strings*. This
scenario gives me the Bad Request: Invalid null value for clustering key
part {field_name} message. Seems like empty strings are treated as NULL
when using the COPY .. FROM command?
This can obviously be me not knowing how to encode an empty string in a
CSV file. A simplified row from the CSV file can look like below:
field1_value,,,field4_value,field5_value
whereas field1 through field4 is part of the primary key.
Thanks for your time,
Petter
2013/9/23 Sylvain Lebresne sylv...@datastax.com
Is it not permitted to have null values in a field that is part a
primary key?
It's not.
This seems to be ok when using the java-driver
Are you sure? Because that would qualify as a bug (in the sense that
it's not supported by C* so there is not reason why this would work with
any driver). If you have some java driver code that show it possible, I'd
be happy to have a look.
--
Sylvain
Extracting throughput-heavy Cassandra operations to different process
Hello. I have been looking into the memory patterns of Cassandra (1.1.5), and I have a suggestion that I haven't seen discussed here. Cassandra is configured by default with a GC tailored for pauseless operation instead of throughput, and this makes sense since it needs to answer client queries with low latency. Under normal operation I see no Full GCs. But when repairs/compactions are run the memory pattern changes and I see frequent Full GCs, which take too long. Wouldn't it make sense to do repairs/compacts in a separate JVM process with a GC configured differently? In this case pauses wouldn't pose such a problem. I'm finding it difficult to configure the GC to behave correctly with and without repairs going on. Best regards, André signature.asc Description: Message signed with OpenPGP using GPGMail
Re: Bad Request: Invalid null value for clustering key part
In fact, the blog post on the subject (
http://www.datastax.com/dev/blog/simple-data-importing-and-exporting-with-cassandra)
also states that the NULL option only is used by the COPY ... TO command
but it also states that this might change in the future and the future
seems to be here now.
Thanks,
Petter
2013/9/25 Petter von Dolwitz (Hem) petter.von.dolw...@gmail.com
I am generating the CSV file by other means than the COPY ... TO command
and I thought that the NULL option did not apply to the COPY ... FROM
command but, as it turns out, it does! This solved my problem.
The documentation here (
http://www.datastax.com/documentation/cql/3.1/webhelp/index.html#cql/cql_reference/copy_r.html)
is misleading. Should I report this by other means than this post?
Thanks,
Petter
2013/9/24 Sylvain Lebresne sylv...@datastax.com
Oh. That would be a COPY thing then. I'm not extremely familiar with
cqlsh code (which COPY is part of) but looking at the online help for it,
it seems to have a 'NULL' option that allows to define which character is
used to represent nulls. And by default, it does is an empty string. So you
could try something like:
COPY ... TO ... WITH NULL='null'
(assuming that if you do have nulls you use the string 'null' to encode
it in your csv and that your are sure nothing that's not supposed to be
null will be represented by the string 'null').
--
Sylvain
On Tue, Sep 24, 2013 at 9:41 AM, Petter von Dolwitz (Hem)
petter.von.dolw...@gmail.com wrote:
Hi Sylvain,
I was not describing the problem correctly. I'm sorry for this. This is
the situation:
1. I'm populating the DB with the java-driver (INSERT INTO...). Some
fields that are part of the primary key is *empty strings*. This works fine.
2. I'm trying to populate the DB from a CSV (COPY ... FROM) using cqlsh.
Some fields that are part of the primary key is *empty strings*. This
scenario gives me the Bad Request: Invalid null value for clustering key
part {field_name} message. Seems like empty strings are treated as NULL
when using the COPY .. FROM command?
This can obviously be me not knowing how to encode an empty string in a
CSV file. A simplified row from the CSV file can look like below:
field1_value,,,field4_value,field5_value
whereas field1 through field4 is part of the primary key.
Thanks for your time,
Petter
2013/9/23 Sylvain Lebresne sylv...@datastax.com
Is it not permitted to have null values in a field that is part a
primary key?
It's not.
This seems to be ok when using the java-driver
Are you sure? Because that would qualify as a bug (in the sense that
it's not supported by C* so there is not reason why this would work with
any driver). If you have some java driver code that show it possible, I'd
be happy to have a look.
--
Sylvain
RE: 1.2.10 - 2.0.1 migration issue
Hi Marcus, Thanks for having a look at this. Just noticed this in the NEWS.txt For leveled compaction users, 2.0 must be atleast started before upgrading to 2.1 due to the fact that the old JSON leveled manifest is migrated into the sstable metadata files on startup in 2.0 and this code is gone from 2.1. Basically, my fault for skimming over this too quickly. We will move from 1.2.10 - 2.0 - 2.1 Thanks, Chris From: Marcus Eriksson [mailto:krum...@gmail.com] Sent: 25 September 2013 09:37 To: user@cassandra.apache.org Subject: Re: 1.2.10 - 2.0.1 migration issue cant really reproduce, could you update the ticket with a bit more info about your setup? do you have multiple .json files in your data dirs? On Wed, Sep 25, 2013 at 10:07 AM, Marcus Eriksson krum...@gmail.com wrote: this is most likely a bug, filed https://issues.apache.org/jira/browse/CASSANDRA-6093 and will try to have a look today. On Wed, Sep 25, 2013 at 1:48 AM, Christopher Wirt chris.w...@struq.com wrote: Hi, Just had a go at upgrading a node to the latest stable c* 2 release and think I ran into some issues with manifest migration. On initial start up I hit this error as it starts to load the first of my CF. INFO [main] 2013-09-24 22:56:01,018 LegacyLeveledManifest.java (line 89) Migrating manifest for struqrealtime/impressionstorev2 INFO [main] 2013-09-24 22:56:01,019 LegacyLeveledManifest.java (line 119) Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration ERROR [main] 2013-09-24 22:56:01,030 CassandraDaemon.java (line 459) Exception encountered during startup FSWriteError in /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem etamigration/impressionstorev2.json at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:83) at org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS( LegacyLeveledManifest.java:138) at org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(Le gacyLeveledManifest.java:91) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:4 42) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485) Caused by: java.nio.file.NoSuchFileException: /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem etamigration/impressionstorev2.json - /disk1/cassandra/data/struqrealtime/impressionstorev2/impressionstorev2.json at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:474 ) at java.nio.file.Files.createLink(Files.java:1037) at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:79) ... 5 more I had already successful run a test migration on our dev server. Only real difference I can see if the number of data directories defined and the amount of data being held. I've run upgradesstables under 1.2.10. I have always been using vnodes and CQL3. I recently moved to using LZ4 instead of Snappy.. I tried to startup again and it gave me a slightly different error INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 89) Migrating manifest for struqrealtime/impressionstorev2 INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 119) Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration ERROR [main] 2013-09-24 22:58:28,222 CassandraDaemon.java (line 459) Exception encountered during startup java.lang.RuntimeException: Tried to create duplicate hard link to /disk3/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem etamigration/struqrealtime-impressionstorev2-ic-1030-TOC.txt at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71) at org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS( LegacyLeveledManifest.java:129) at org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(Le gacyLeveledManifest.java:91) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:4 42) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485) Will have a go recreating this tomorrow. Any insight or guesses at what the issue might be are always welcome. Thanks, Chris
FW: Migration LCS from 1.2.X to 2.0.x exception
Same is true if started from 1.2.9 version to 2.0.x Seems like Cassandra-5383 is the related issue. The code in the function mutateLevel (from 1.2.5 to 1.2.9 and later ) is indirectly using the java file::renameTo function. And that will not work on Windows if the file already exists. Don't know about Linux, but even the java function description is already telling that the behavior of the function can be machine dependent. So even on linux based systems, and even if that functions returns with success, still does not confirm that actually the old file name/content was removed and the new file name/content is used as replacement. From: Marcus Eriksson [mailto:krum...@gmail.com] Sent: woensdag 25 september 2013 10:02 To: user@cassandra.apache.org Subject: Re: Migration LCS from 1.2.X to 2.0.x exception this is the issue: https://issues.apache.org/jira/browse/CASSANDRA-5383 guess it fell between chairs, will poke around On Tue, Sep 24, 2013 at 4:26 PM, Nate McCall n...@thelastpickle.commailto:n...@thelastpickle.com wrote: What version of 1.2.x? Unfortunately, you must go through 1.2.9 first. See https://github.com/apache/cassandra/blob/cassandra-2.0.0/NEWS.txt#L19-L24 On Tue, Sep 24, 2013 at 8:57 AM, Desimpel, Ignace ignace.desim...@nuance.commailto:ignace.desim...@nuance.com wrote: Tested on WINDOWS : On startup of the 2.0.0 version from 1.2.x files I get an error as listed below. This is due to the code in LeveledManifest:: mutateLevel. The method already has a comment saying that it is scary ... On windows, one cannot use the File::rename if the target file name already exists. Also, even on Linux, I'm not sure if a rename would actually 'overwrite/implicit-delete' the content of the target file. Anyway, adding code (below) before the FileUtils.renameWithConfirm should work in both cases (maybe even rename the fromFile to be able to recover...) File oTo = new File(filename); if ( oTo.exists() ) oTo.delete(); java.lang.RuntimeException: Failed to rename .xxx\Function-ic-10-Statistics.db-tmp to .xxx\Function-ic-10-Statistics.db at org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:136) ~[main/:na] at org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:125) ~[main/:na] at org.apache.cassandra.db.compaction.LeveledManifest.mutateLevel(LeveledManifest.java:601) ~[main/:na] at org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:103) ~[main/:na] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:247) ~[main/:na] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:443) ~[main/:na] Regards, Ignace Desimpel
Re: 1.2.10 - 2.0.1 migration issue
you are probably reading trunk NEWS.txt read the ticket for explanation of what the issue was (it is a proper bug) On Wed, Sep 25, 2013 at 12:59 PM, Christopher Wirt chris.w...@struq.comwrote: Hi Marcus, Thanks for having a look at this. ** ** Just noticed this in the NEWS.txt ** ** For *leveled* compaction users, 2.0 must be *atleast* started before upgrading to 2.1 due to the fact that the old JSON *leveled* manifest is migrated into the *sstable* *metadata* files on startup** ** in 2.0 and this code is gone from 2.1. ** ** Basically, my fault for skimming over this too quickly. ** ** We will move from 1.2.10 - 2.0 - 2.1 ** ** Thanks, Chris ** ** ** ** *From:* Marcus Eriksson [mailto:krum...@gmail.com] *Sent:* 25 September 2013 09:37 *To:* user@cassandra.apache.org *Subject:* Re: 1.2.10 - 2.0.1 migration issue ** ** cant really reproduce, could you update the ticket with a bit more info about your setup? ** ** do you have multiple .json files in your data dirs? ** ** On Wed, Sep 25, 2013 at 10:07 AM, Marcus Eriksson krum...@gmail.com wrote: this is most likely a bug, filed https://issues.apache.org/jira/browse/CASSANDRA-6093 and will try to have a look today. ** ** On Wed, Sep 25, 2013 at 1:48 AM, Christopher Wirt chris.w...@struq.com wrote: Hi, Just had a go at upgrading a node to the latest stable c* 2 release and think I ran into some issues with manifest migration. On initial start up I hit this error as it starts to load the first of my CF. INFO [main] 2013-09-24 22:56:01,018 LegacyLeveledManifest.java (line 89) Migrating manifest for struqrealtime/impressionstorev2 INFO [main] 2013-09-24 22:56:01,019 LegacyLeveledManifest.java (line 119) Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration* *** ERROR [main] 2013-09-24 22:56:01,030 CassandraDaemon.java (line 459) Exception encountered during startup FSWriteError in /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:83)** ** at org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:138) at org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485) Caused by: java.nio.file.NoSuchFileException: /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json - /disk1/cassandra/data/struqrealtime/impressionstorev2/impressionstorev2.json at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:474) at java.nio.file.Files.createLink(Files.java:1037) at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:79)** ** ... 5 more I had already successful run a test migration on our dev server. Only real difference I can see if the number of data directories defined and the amount of data being held. I’ve run upgradesstables under 1.2.10. I have always been using vnodes and CQL3. I recently moved to using LZ4 instead of Snappy.. I tried to startup again and it gave me a slightly different error INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 89) Migrating manifest for struqrealtime/impressionstorev2 INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 119) Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration* *** ERROR [main] 2013-09-24 22:58:28,222 CassandraDaemon.java (line 459) Exception encountered during startup java.lang.RuntimeException: Tried to create duplicate hard link to /disk3/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/struqrealtime-impressionstorev2-ic-1030-TOC.txt at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)** ** at org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:129) at org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91) at
Re: Query about class org.apache.cassandra.io.sstable.SSTableSimpleWriter
Can someone answer this doubt reg. SSTableSimpleWriter ? I'd asked about this earlier but it probably missed. Apologies for repeating the question (with minor additions) : Let's say I've initialized a *SSTableSimpleWriter* instance and a new column with TTL set : *org.apache.cassandra.io.sstable.SSTableSimpleWriter writer = new SSTableSimpleWriter( ... /* params here */);* *org.apache.cassandra.thrift.Column column; // initialize this with name, value, timestamp, TTL* What is the difference between calling *writer.addColumn()* on the column's name, value and timestamp, and *writer.addExpiringColumn()* on the column's name, value, TTL, timestamp and expiration timestamp ? Does the former result in the column expiring still , in cassandra 1.2.x (i.e. does setting the TTL on a Column object change the name or value in a way so as to ensure the column will expire as required) ? If not , what is the TTL attribute used for in the Column object ? Thanks, Jayadev On Tue, Sep 24, 2013 at 2:48 PM, Jayadev Jayaraman jdisal...@gmail.comwrote: Let's say I've initialized a *SSTableSimpleWriter* instance and a new column with TTL set : *SSTableSimpleWriter writer = new SSTableSimpleWriter( ... /* params here */);* *Column column;* What is the difference between calling *writer.addColumn()* on the column's name and value, and *writer.addExpiringColumn()* on the column and its TTL ? Does the former result in the column expiring still , in cassandra 1.2.x ? Or does it not ?
RE: 1.2.10 - 2.0.1 migration issue
Hi Marcus, I've seen your patch. This works with what I'm seeing. The first data directory only contained the JSON manifest at that time. As a workaround I've made sure that each of the snapshot directories now exist before starting up. I still end up with the second exception I posted regarding a duplicate hard link. Possibly two unrelated exceptions. After getting this error. Looking at the datadirs Data1 contains JSON manifests Loads of data files Snapshot directory Data2 contains Just the snapshot directory Data3 contains Just the snapshot directory INFO 12:56:22,766 Migrating manifest for struqrealtime/impressionstorev2 INFO 12:56:22,767 Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration ERROR 12:56:22,787 Exception encountered during startup java.lang.RuntimeException: Tried to create duplicate hard link to /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem etamigration/impressionstorev2.json at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71) at org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS( LegacyLeveledManifest.java:138) at org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(Le gacyLeveledManifest.java:91) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:247) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:4 43) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486) java.lang.RuntimeException: Tried to create duplicate hard link to /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem etamigration/impressionstorev2.json at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71) at org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS( LegacyLeveledManifest.java:138) at org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(Le gacyLeveledManifest.java:91) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:247) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:4 43) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486) Exception encountered during startup: Tried to create duplicate hard link to /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem etamigration/impressionstorev2.json Thanks, Chris From: Marcus Eriksson [mailto:krum...@gmail.com] Sent: 25 September 2013 13:11 To: user@cassandra.apache.org Subject: Re: 1.2.10 - 2.0.1 migration issue you are probably reading trunk NEWS.txt read the ticket for explanation of what the issue was (it is a proper bug) On Wed, Sep 25, 2013 at 12:59 PM, Christopher Wirt chris.w...@struq.com wrote: Hi Marcus, Thanks for having a look at this. Just noticed this in the NEWS.txt For leveled compaction users, 2.0 must be atleast started before upgrading to 2.1 due to the fact that the old JSON leveled manifest is migrated into the sstable metadata files on startup in 2.0 and this code is gone from 2.1. Basically, my fault for skimming over this too quickly. We will move from 1.2.10 - 2.0 - 2.1 Thanks, Chris From: Marcus Eriksson [mailto:krum...@gmail.com] Sent: 25 September 2013 09:37 To: user@cassandra.apache.org Subject: Re: 1.2.10 - 2.0.1 migration issue cant really reproduce, could you update the ticket with a bit more info about your setup? do you have multiple .json files in your data dirs? On Wed, Sep 25, 2013 at 10:07 AM, Marcus Eriksson krum...@gmail.com wrote: this is most likely a bug, filed https://issues.apache.org/jira/browse/CASSANDRA-6093 and will try to have a look today. On Wed, Sep 25, 2013 at 1:48 AM, Christopher Wirt chris.w...@struq.com wrote: Hi, Just had a go at upgrading a node to the latest stable c* 2 release and think I ran into some issues with manifest migration. On initial start up I hit this error as it starts to load the first of my CF. INFO [main] 2013-09-24 22:56:01,018 LegacyLeveledManifest.java (line 89) Migrating manifest for struqrealtime/impressionstorev2 INFO [main] 2013-09-24 22:56:01,019 LegacyLeveledManifest.java (line 119) Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration ERROR [main] 2013-09-24 22:56:01,030 CassandraDaemon.java (line 459) Exception encountered during startup FSWriteError in /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem etamigration/impressionstorev2.json at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:83) at org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS( LegacyLeveledManifest.java:138) at
RE: 1.2.10 - 2.0.1 migration issue
Should also say. I have managed to move one node from 1.2.10 to 2.0.0. I'm seeing this error on the machine I tried to migrate earlier to 2.0.1 Thanks From: Christopher Wirt [mailto:chris.w...@struq.com] Sent: 25 September 2013 14:04 To: 'user@cassandra.apache.org' Subject: RE: 1.2.10 - 2.0.1 migration issue Hi Marcus, I've seen your patch. This works with what I'm seeing. The first data directory only contained the JSON manifest at that time. As a workaround I've made sure that each of the snapshot directories now exist before starting up. I still end up with the second exception I posted regarding a duplicate hard link. Possibly two unrelated exceptions. After getting this error. Looking at the datadirs Data1 contains JSON manifests Loads of data files Snapshot directory Data2 contains Just the snapshot directory Data3 contains Just the snapshot directory INFO 12:56:22,766 Migrating manifest for struqrealtime/impressionstorev2 INFO 12:56:22,767 Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration ERROR 12:56:22,787 Exception encountered during startup java.lang.RuntimeException: Tried to create duplicate hard link to /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem etamigration/impressionstorev2.json at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71) at org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS( LegacyLeveledManifest.java:138) at org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(Le gacyLeveledManifest.java:91) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:247) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:4 43) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486) java.lang.RuntimeException: Tried to create duplicate hard link to /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem etamigration/impressionstorev2.json at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71) at org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS( LegacyLeveledManifest.java:138) at org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(Le gacyLeveledManifest.java:91) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:247) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:4 43) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486) Exception encountered during startup: Tried to create duplicate hard link to /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem etamigration/impressionstorev2.json Thanks, Chris From: Marcus Eriksson [mailto:krum...@gmail.com] Sent: 25 September 2013 13:11 To: user@cassandra.apache.org Subject: Re: 1.2.10 - 2.0.1 migration issue you are probably reading trunk NEWS.txt read the ticket for explanation of what the issue was (it is a proper bug) On Wed, Sep 25, 2013 at 12:59 PM, Christopher Wirt chris.w...@struq.com wrote: Hi Marcus, Thanks for having a look at this. Just noticed this in the NEWS.txt For leveled compaction users, 2.0 must be atleast started before upgrading to 2.1 due to the fact that the old JSON leveled manifest is migrated into the sstable metadata files on startup in 2.0 and this code is gone from 2.1. Basically, my fault for skimming over this too quickly. We will move from 1.2.10 - 2.0 - 2.1 Thanks, Chris From: Marcus Eriksson [mailto:krum...@gmail.com] Sent: 25 September 2013 09:37 To: user@cassandra.apache.org Subject: Re: 1.2.10 - 2.0.1 migration issue cant really reproduce, could you update the ticket with a bit more info about your setup? do you have multiple .json files in your data dirs? On Wed, Sep 25, 2013 at 10:07 AM, Marcus Eriksson krum...@gmail.com wrote: this is most likely a bug, filed https://issues.apache.org/jira/browse/CASSANDRA-6093 and will try to have a look today. On Wed, Sep 25, 2013 at 1:48 AM, Christopher Wirt chris.w...@struq.com wrote: Hi, Just had a go at upgrading a node to the latest stable c* 2 release and think I ran into some issues with manifest migration. On initial start up I hit this error as it starts to load the first of my CF. INFO [main] 2013-09-24 22:56:01,018 LegacyLeveledManifest.java (line 89) Migrating manifest for struqrealtime/impressionstorev2 INFO [main] 2013-09-24 22:56:01,019 LegacyLeveledManifest.java (line 119) Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration ERROR [main] 2013-09-24 22:56:01,030 CassandraDaemon.java (line 459) Exception encountered during startup FSWriteError in
Re: 1.2.10 - 2.0.1 migration issue
you probably have to remove the old snapshots before trying to restart On Wed, Sep 25, 2013 at 3:05 PM, Christopher Wirt chris.w...@struq.comwrote: Should also say. I have managed to move one node from 1.2.10 to 2.0.0. I’m seeing this error on the machine I tried to migrate earlier to 2.0.1 ** ** Thanks ** ** *From:* Christopher Wirt [mailto:chris.w...@struq.com] *Sent:* 25 September 2013 14:04 *To:* 'user@cassandra.apache.org' *Subject:* RE: 1.2.10 - 2.0.1 migration issue ** ** Hi Marcus, ** ** I’ve seen your patch. This works with what I’m seeing. The first data directory only contained the JSON manifest at that time. ** ** As a workaround I’ve made sure that each of the snapshot directories now exist before starting up. ** ** I still end up with the second exception I posted regarding a duplicate hard link. Possibly two unrelated exceptions. ** ** After getting this error. Looking at the datadirs Data1 contains JSON manifests Loads of data files Snapshot directory Data2 contains Just the snapshot directory Data3 contains Just the snapshot directory ** ** INFO 12:56:22,766 Migrating manifest for struqrealtime/impressionstorev2** ** INFO 12:56:22,767 Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration ERROR 12:56:22,787 Exception encountered during startup java.lang.RuntimeException: Tried to create duplicate hard link to /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)** ** at org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:138) at org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:247) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:443) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486) java.lang.RuntimeException: Tried to create duplicate hard link to /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)** ** at org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:138) at org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:247) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:443) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486) Exception encountered during startup: Tried to create duplicate hard link to /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json ** ** Thanks, ** ** Chris ** ** ** ** *From:* Marcus Eriksson [mailto:krum...@gmail.com] *Sent:* 25 September 2013 13:11 *To:* user@cassandra.apache.org *Subject:* Re: 1.2.10 - 2.0.1 migration issue ** ** you are probably reading trunk NEWS.txt ** ** read the ticket for explanation of what the issue was (it is a proper bug) ** ** On Wed, Sep 25, 2013 at 12:59 PM, Christopher Wirt chris.w...@struq.com wrote: Hi Marcus, Thanks for having a look at this. Just noticed this in the NEWS.txt For *leveled* compaction users, 2.0 must be *atleast* started before upgrading to 2.1 due to the fact that the old JSON *leveled* manifest is migrated into the *sstable* *metadata* files on startup** ** in 2.0 and this code is gone from 2.1. Basically, my fault for skimming over this too quickly. We will move from 1.2.10 - 2.0 - 2.1 Thanks, Chris *From:* Marcus Eriksson [mailto:krum...@gmail.com] *Sent:* 25 September 2013 09:37 *To:* user@cassandra.apache.org *Subject:* Re: 1.2.10 - 2.0.1 migration issue cant really reproduce, could you update the ticket with a bit more info about your setup? do you have multiple .json files in your data dirs? On Wed, Sep 25, 2013 at 10:07 AM, Marcus Eriksson krum...@gmail.com wrote: this is most likely a bug, filed https://issues.apache.org/jira/browse/CASSANDRA-6093 and will try to have a look today. On Wed, Sep 25, 2013 at 1:48 AM, Christopher Wirt chris.w...@struq.com
FW: Migration LCS from 1.2.X to 2.0.x exception
Did a test for File::rename on Ubuntu and windows. Seems that on Ubuntu the File::renameTo(FROM, TO) is actually replacing the existing (same as TO) file and content with the new (same as FROM) file and content. So that would make it a problem only on Windows. Adding a File::delete before the rename in case of Windows could do the (scary) trick. From: Desimpel, Ignace [mailto:ignace.desim...@nuance.com] Sent: woensdag 25 september 2013 13:13 To: user@cassandra.apache.org Subject: FW: Migration LCS from 1.2.X to 2.0.x exception Same is true if started from 1.2.9 version to 2.0.x Seems like Cassandra-5383 is the related issue. The code in the function mutateLevel (from 1.2.5 to 1.2.9 and later ) is indirectly using the java file::renameTo function. And that will not work on Windows if the file already exists. Don't know about Linux, but even the java function description is already telling that the behavior of the function can be machine dependent. So even on linux based systems, and even if that functions returns with success, still does not confirm that actually the old file name/content was removed and the new file name/content is used as replacement. From: Marcus Eriksson [mailto:krum...@gmail.com]mailto:[mailto:krum...@gmail.com] Sent: woensdag 25 september 2013 10:02 To: user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Re: Migration LCS from 1.2.X to 2.0.x exception this is the issue: https://issues.apache.org/jira/browse/CASSANDRA-5383 guess it fell between chairs, will poke around On Tue, Sep 24, 2013 at 4:26 PM, Nate McCall n...@thelastpickle.commailto:n...@thelastpickle.com wrote: What version of 1.2.x? Unfortunately, you must go through 1.2.9 first. See https://github.com/apache/cassandra/blob/cassandra-2.0.0/NEWS.txt#L19-L24 On Tue, Sep 24, 2013 at 8:57 AM, Desimpel, Ignace ignace.desim...@nuance.commailto:ignace.desim...@nuance.com wrote: Tested on WINDOWS : On startup of the 2.0.0 version from 1.2.x files I get an error as listed below. This is due to the code in LeveledManifest:: mutateLevel. The method already has a comment saying that it is scary ... On windows, one cannot use the File::rename if the target file name already exists. Also, even on Linux, I'm not sure if a rename would actually 'overwrite/implicit-delete' the content of the target file. Anyway, adding code (below) before the FileUtils.renameWithConfirm should work in both cases (maybe even rename the fromFile to be able to recover...) File oTo = new File(filename); if ( oTo.exists() ) oTo.delete(); java.lang.RuntimeException: Failed to rename .xxx\Function-ic-10-Statistics.db-tmp to .xxx\Function-ic-10-Statistics.db at org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:136) ~[main/:na] at org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:125) ~[main/:na] at org.apache.cassandra.db.compaction.LeveledManifest.mutateLevel(LeveledManifest.java:601) ~[main/:na] at org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:103) ~[main/:na] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:247) ~[main/:na] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:443) ~[main/:na] Regards, Ignace Desimpel
Re: Mystery PIG issue with 1.2.10
As an FYI, creating the table without the WITH COMPACT STORAGE and using
CqlStorage works just fine in 1.2.10.
I know that CqlStorage and AbstractCassandraStorage got changed for 1.2.10
- maybe there's a regression with the existing CassandraStorage?
Chad
On Wed, Sep 25, 2013 at 1:51 AM, Janne Jalkanen janne.jalka...@ecyrd.comwrote:
Heya!
I am seeing something rather strange in the way Cass 1.2 + Pig seem to
handle integer values.
Setup: Cassandra 1.2.10, OSX 10.8, JDK 1.7u40, Pig 0.11.1. Single node
for testing this.
First a table:
CREATE TABLE testc (
key text PRIMARY KEY,
ivalue int,
svalue text,
value bigint
) WITH COMPACT STORAGE;
insert into testc (key,ivalue,svalue,value) values ('foo',10,'bar',65);
select * from testc;
key | ivalue | svalue | value
-+++---
foo | 10 |bar | 65
For my Pig setup, I then use libraries from different C* versions to
actually talk to my database (which stays on 1.2.10 all the time).
Cassandra 1.0.12 (using cassandra_storage.jar):
testc = LOAD 'cassandra://keyspace/testc' USING CassandraStorage();
dump testc
(foo,(svalue,bar),(ivalue,10),(value,65),{})
Cassandra 1.1.10:
testc = LOAD 'cassandra://keyspace/testc' USING CassandraStorage();
dump testc
(foo,(svalue,bar),(ivalue,10),(value,65),{})
Cassandra 1.2.10:
(testc = LOAD 'cassandra://keyspace/testc' USING CassandraStorage();
dump testc
foo,{(ivalue,
),(svalue,bar),(value,A)})
To me it appears that ints and bigints are interpreted as ascii values in
cass 1.2.10. Did something change for CassandraStorage, is there a
regression, or am I doing something wrong? Quick perusal of the JIRA
didn't reveal anything that I could directly pin on this.
Note that using compact storage does not seem to affect the issue, though
it obviously changes the resulting pig format.
In addition, trying to use Pygmalion
tf = foreach testc generate key,
flatten(FromCassandraBag('ivalue,svalue,value',columns)) as
(ivalue:int,svalue:chararray,lvalue:long);
dump tf
(foo,
,bar,A)
So no help there. Explicitly casting the values to (long) or (int) just
results in a ClassCastException.
/Janne
Re: Bad Request: Invalid null value for clustering key part
On Wed, Sep 25, 2013 at 5:00 AM, Petter von Dolwitz (Hem) petter.von.dolw...@gmail.com wrote: I am generating the CSV file by other means than the COPY ... TO command and I thought that the NULL option did not apply to the COPY ... FROM command but, as it turns out, it does! This solved my problem. The documentation here ( http://www.datastax.com/documentation/cql/3.1/webhelp/index.html#cql/cql_reference/copy_r.html) is misleading. Should I report this by other means than this post? I've opened a ticket with our docs team to get this updated. Thanks! -- Tyler Hobbs DataStax http://datastax.com/
Re: 1.2.10 - 2.0.1 migration issue
On Wed, Sep 25, 2013 at 6:05 AM, Christopher Wirt chris.w...@struq.comwrote: Should also say. I have managed to move one node from 1.2.10 to 2.0.0. I’m seeing this error on the machine I tried to migrate earlier to 2.0.1 I'm confused... for the record : 1) you tried to upgrade from 1.2.10 to 2.0.1 2) the NEWS.txt snippet you posted refers to upgrading from versions below 2.0 to 2.1 3) 2.0.1 is 2.0, not 2.1 Therefore the problem is actually https://issues.apache.org/jira/browse/CASSANDRA-6093 ? =Rob
Nodes not added to existing cluster
Hi all, I have a three node cluster using the EC2 Multi-Region Snitch currently operating only in US-EAST. On having a node go down this morning, I started a new node with an identical configuration, except for the seed list, the listen address and the rpc address. The new node comes up and creates its own cluster rather than joining the pre-existing ring. I've tried creating a node both before ad after using `nodetool remove` for the bad node, each time with the same result. Does anyone have any suggestions for where to look that might put me on the right track? Thanks, -Skye
Re: Nodes not added to existing cluster
On Wed, Sep 25, 2013 at 12:41 PM, Skye Book skye.b...@gmail.com wrote: I have a three node cluster using the EC2 Multi-Region Snitch currently operating only in US-EAST. On having a node go down this morning, I started a new node with an identical configuration, except for the seed list, the listen address and the rpc address. The new node comes up and creates its own cluster rather than joining the pre-existing ring. I've tried creating a node both *before* ad *after* using `nodetool remove` for the bad node, each time with the same result. What version of Cassandra? This particular confusing behavior is fixed upstream, in a version you should not deploy to production yet. Take some solace, however, that you may be the last Cassandra administrator to die for a broken code path! https://issues.apache.org/jira/browse/CASSANDRA-5768 Does anyone have any suggestions for where to look that might put me on the right track? It must be that your seed list is wrong in some way, or your node state is wrong. If you're trying to bootstrap a node, note that you can't bootstrap a node when it is in its own seed list. If you have installed Cassandra via debian package, there is a possibility that your node has started before you explicitly started it. If so, it might have invalid node state. Have you tried wiping the data directory and trying again? What is your seed list? Are you sure the new node can reach the seeds on the network layer? =Rob
Re: Nodes not added to existing cluster
Check your security groups to be sure you have appropriate access. If in a VPC check both IN and OUT; if using ACLs check those. On Wed, Sep 25, 2013 at 3:41 PM, Skye Book skye.b...@gmail.com wrote: Hi all, I have a three node cluster using the EC2 Multi-Region Snitch currently operating only in US-EAST. On having a node go down this morning, I started a new node with an identical configuration, except for the seed list, the listen address and the rpc address. The new node comes up and creates its own cluster rather than joining the pre-existing ring. I've tried creating a node both *before* ad *after* using `nodetool remove` for the bad node, each time with the same result. Does anyone have any suggestions for where to look that might put me on the right track? Thanks, -Skye
Re: Nodes not added to existing cluster
Thank you, both Michael and Robert for your suggestions. I actually saw 5760, but we were running on 2.0.0, which it seems like this was fixed in. That said, I noticed that my Chef scripts were failing to set the broadcast_address correctly, which I'm guessing is the cause of the problem, fixing that and trying a redeploy. I am curious, though, how any of this worked in the first place spread across three AZ's without that being set? -Skye On Sep 25, 2013, at 3:56 PM, Robert Coli rc...@eventbrite.com wrote: On Wed, Sep 25, 2013 at 12:41 PM, Skye Book skye.b...@gmail.com wrote: I have a three node cluster using the EC2 Multi-Region Snitch currently operating only in US-EAST. On having a node go down this morning, I started a new node with an identical configuration, except for the seed list, the listen address and the rpc address. The new node comes up and creates its own cluster rather than joining the pre-existing ring. I've tried creating a node both before ad after using `nodetool remove` for the bad node, each time with the same result. What version of Cassandra? This particular confusing behavior is fixed upstream, in a version you should not deploy to production yet. Take some solace, however, that you may be the last Cassandra administrator to die for a broken code path! https://issues.apache.org/jira/browse/CASSANDRA-5768 Does anyone have any suggestions for where to look that might put me on the right track? It must be that your seed list is wrong in some way, or your node state is wrong. If you're trying to bootstrap a node, note that you can't bootstrap a node when it is in its own seed list. If you have installed Cassandra via debian package, there is a possibility that your node has started before you explicitly started it. If so, it might have invalid node state. Have you tried wiping the data directory and trying again? What is your seed list? Are you sure the new node can reach the seeds on the network layer? =Rob
[Cassandra] Initial Setup - VMs for Research
Request some initial setup guidance for Cassandra deployment I expect to mentor a project at the Oregon State University computer science department for a senior engineering student project. I am trying to pre-configure one or more VMware virtual machines to hold an initial Cassandra database for a NOSQL project. Any guidance on the steps for initial deployment would be appreciated. My VMware machines already have the necessary 3rd party tools such as Oracle Java 7 and are running on a Debian Linux 7.1.0 release. The Oregon State University computer science department will eventually host these virtual machines on their department servers if the student project is selected. Sincerely, Steven J. Hathaway (Senior IT Systems Architect)
RE: [Cassandra] Initial Setup - VMs for Research
What help are u looking for ? http://www.datastax.com/docs/datastax_enterprise3.1/install/install_deb_pkg -Original Message- From: shath...@e-z.net [mailto:shath...@e-z.net] Sent: 25 September 2013 15:27 To: user@cassandra.apache.org Subject: [Cassandra] Initial Setup - VMs for Research Request some initial setup guidance for Cassandra deployment I expect to mentor a project at the Oregon State University computer science department for a senior engineering student project. I am trying to pre-configure one or more VMware virtual machines to hold an initial Cassandra database for a NOSQL project. Any guidance on the steps for initial deployment would be appreciated. My VMware machines already have the necessary 3rd party tools such as Oracle Java 7 and are running on a Debian Linux 7.1.0 release. The Oregon State University computer science department will eventually host these virtual machines on their department servers if the student project is selected. Sincerely, Steven J. Hathaway (Senior IT Systems Architect)
Re: [Cassandra] Initial Setup - VMs for Research
There is also Debian for the standard Apache Cassandra build http://wiki.apache.org/cassandra/DebianPackaging Or you can use DS Community which is a totally free distro of Apache Cassandra http://www.datastax.com/documentation/cassandra/2.0/webhelp/index.html#cassandra/install/installDeb_t.html (Instructions for DS Community will work for both) Data Stax Enterprise is free for development purposes but can not be used in production without a licence. In addition is ships with extra installation companents (hadoop, mahoout, solr) that you probably want need. Hope that helps. - Aaron Morton New Zealand @aaronmorton Co-Founder Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 26/09/2013, at 2:42 PM, Kanwar Sangha kan...@mavenir.com wrote: What help are u looking for ? http://www.datastax.com/docs/datastax_enterprise3.1/install/install_deb_pkg -Original Message- From: shath...@e-z.net [mailto:shath...@e-z.net] Sent: 25 September 2013 15:27 To: user@cassandra.apache.org Subject: [Cassandra] Initial Setup - VMs for Research Request some initial setup guidance for Cassandra deployment I expect to mentor a project at the Oregon State University computer science department for a senior engineering student project. I am trying to pre-configure one or more VMware virtual machines to hold an initial Cassandra database for a NOSQL project. Any guidance on the steps for initial deployment would be appreciated. My VMware machines already have the necessary 3rd party tools such as Oracle Java 7 and are running on a Debian Linux 7.1.0 release. The Oregon State University computer science department will eventually host these virtual machines on their department servers if the student project is selected. Sincerely, Steven J. Hathaway (Senior IT Systems Architect)
Re: Nodes not added to existing cluster
I am curious, though, how any of this worked in the first place spread across three AZ's without that being set? boradcast_address is only needed when you are going cross region (IIRC it's the EC2MultiRegionSnitch) that sets it. As rob said, make sure the seed list includes on of the other nodes and that the cluster_name set. Cheers - Aaron Morton New Zealand @aaronmorton Co-Founder Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 26/09/2013, at 8:12 AM, Skye Book skye.b...@gmail.com wrote: Thank you, both Michael and Robert for your suggestions. I actually saw 5760, but we were running on 2.0.0, which it seems like this was fixed in. That said, I noticed that my Chef scripts were failing to set the broadcast_address correctly, which I'm guessing is the cause of the problem, fixing that and trying a redeploy. I am curious, though, how any of this worked in the first place spread across three AZ's without that being set? -Skye On Sep 25, 2013, at 3:56 PM, Robert Coli rc...@eventbrite.com wrote: On Wed, Sep 25, 2013 at 12:41 PM, Skye Book skye.b...@gmail.com wrote: I have a three node cluster using the EC2 Multi-Region Snitch currently operating only in US-EAST. On having a node go down this morning, I started a new node with an identical configuration, except for the seed list, the listen address and the rpc address. The new node comes up and creates its own cluster rather than joining the pre-existing ring. I've tried creating a node both before ad after using `nodetool remove` for the bad node, each time with the same result. What version of Cassandra? This particular confusing behavior is fixed upstream, in a version you should not deploy to production yet. Take some solace, however, that you may be the last Cassandra administrator to die for a broken code path! https://issues.apache.org/jira/browse/CASSANDRA-5768 Does anyone have any suggestions for where to look that might put me on the right track? It must be that your seed list is wrong in some way, or your node state is wrong. If you're trying to bootstrap a node, note that you can't bootstrap a node when it is in its own seed list. If you have installed Cassandra via debian package, there is a possibility that your node has started before you explicitly started it. If so, it might have invalid node state. Have you tried wiping the data directory and trying again? What is your seed list? Are you sure the new node can reach the seeds on the network layer? =Rob
Re: Mystery PIG issue with 1.2.10
(testc = LOAD 'cassandra://keyspace/testc' USING CassandraStorage();
dump testc
foo,{(ivalue,
),(svalue,bar),(value,A)})
If the CQL 3 data ye wish to read, CqlStorage be the driver of your success.
(btw there is a ticket out to update the example if you get excited
https://issues.apache.org/jira/browse/CASSANDRA-5709)
Cheers
-
Aaron Morton
New Zealand
@aaronmorton
Co-Founder Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com
On 26/09/2013, at 3:57 AM, Chad Johnston cjohns...@megatome.com wrote:
As an FYI, creating the table without the WITH COMPACT STORAGE and using
CqlStorage works just fine in 1.2.10.
I know that CqlStorage and AbstractCassandraStorage got changed for 1.2.10 -
maybe there's a regression with the existing CassandraStorage?
Chad
On Wed, Sep 25, 2013 at 1:51 AM, Janne Jalkanen janne.jalka...@ecyrd.com
wrote:
Heya!
I am seeing something rather strange in the way Cass 1.2 + Pig seem to handle
integer values.
Setup: Cassandra 1.2.10, OSX 10.8, JDK 1.7u40, Pig 0.11.1. Single node for
testing this.
First a table:
CREATE TABLE testc (
key text PRIMARY KEY,
ivalue int,
svalue text,
value bigint
) WITH COMPACT STORAGE;
insert into testc (key,ivalue,svalue,value) values ('foo',10,'bar',65);
select * from testc;
key | ivalue | svalue | value
-+++---
foo | 10 |bar | 65
For my Pig setup, I then use libraries from different C* versions to actually
talk to my database (which stays on 1.2.10 all the time).
Cassandra 1.0.12 (using cassandra_storage.jar):
testc = LOAD 'cassandra://keyspace/testc' USING CassandraStorage();
dump testc
(foo,(svalue,bar),(ivalue,10),(value,65),{})
Cassandra 1.1.10:
testc = LOAD 'cassandra://keyspace/testc' USING CassandraStorage();
dump testc
(foo,(svalue,bar),(ivalue,10),(value,65),{})
Cassandra 1.2.10:
(testc = LOAD 'cassandra://keyspace/testc' USING CassandraStorage();
dump testc
foo,{(ivalue,
),(svalue,bar),(value,A)})
To me it appears that ints and bigints are interpreted as ascii values in
cass 1.2.10. Did something change for CassandraStorage, is there a
regression, or am I doing something wrong? Quick perusal of the JIRA didn't
reveal anything that I could directly pin on this.
Note that using compact storage does not seem to affect the issue, though it
obviously changes the resulting pig format.
In addition, trying to use Pygmalion
tf = foreach testc generate key,
flatten(FromCassandraBag('ivalue,svalue,value',columns)) as
(ivalue:int,svalue:chararray,lvalue:long);
dump tf
(foo,
,bar,A)
So no help there. Explicitly casting the values to (long) or (int) just
results in a ClassCastException.
/Janne
Re: Nodes not added to existing cluster
Hi Aaron, thanks for the clarification. As might be expected, having the broadcast_address fixed hasn't fixed anything. What I did find after writing my last email is that output.log is littered with these: INFO 05:03:49,015 Cannot handshake version with /aa.bb.cc.dd INFO 05:03:49,017 Handshaking version with /aa.bb.cc.dd INFO 05:03:49,803 Cannot handshake version with /ww.xx.yy.zz INFO 05:03:49,805 Handshaking version with /ww.xx.yy.zz The two addresses that it is unable to handshake with are the other two addresses of nodes in the cluster I'm unable to join. I started thinking that maybe EC2 was having an-advertised problem communicating between AZ's but bringing up nodes in both of the other availability zones resulted in the same wrong behavior. I've gist'd my cassandra.yaml, its pretty standard and hasn't caused an issue in the past for me. https://gist.github.com/skyebook/ec9364cdcec02e803ffc Skye Book http://skyebook.net -- @sbook On Sep 26, 2013, at 12:34 AM, Aaron Morton aa...@thelastpickle.com wrote: I am curious, though, how any of this worked in the first place spread across three AZ's without that being set? boradcast_address is only needed when you are going cross region (IIRC it's the EC2MultiRegionSnitch) that sets it. As rob said, make sure the seed list includes on of the other nodes and that the cluster_name set. Cheers - Aaron Morton New Zealand @aaronmorton Co-Founder Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 26/09/2013, at 8:12 AM, Skye Book skye.b...@gmail.com wrote: Thank you, both Michael and Robert for your suggestions. I actually saw 5760, but we were running on 2.0.0, which it seems like this was fixed in. That said, I noticed that my Chef scripts were failing to set the broadcast_address correctly, which I'm guessing is the cause of the problem, fixing that and trying a redeploy. I am curious, though, how any of this worked in the first place spread across three AZ's without that being set? -Skye On Sep 25, 2013, at 3:56 PM, Robert Coli rc...@eventbrite.com wrote: On Wed, Sep 25, 2013 at 12:41 PM, Skye Book skye.b...@gmail.com wrote: I have a three node cluster using the EC2 Multi-Region Snitch currently operating only in US-EAST. On having a node go down this morning, I started a new node with an identical configuration, except for the seed list, the listen address and the rpc address. The new node comes up and creates its own cluster rather than joining the pre-existing ring. I've tried creating a node both before ad after using `nodetool remove` for the bad node, each time with the same result. What version of Cassandra? This particular confusing behavior is fixed upstream, in a version you should not deploy to production yet. Take some solace, however, that you may be the last Cassandra administrator to die for a broken code path! https://issues.apache.org/jira/browse/CASSANDRA-5768 Does anyone have any suggestions for where to look that might put me on the right track? It must be that your seed list is wrong in some way, or your node state is wrong. If you're trying to bootstrap a node, note that you can't bootstrap a node when it is in its own seed list. If you have installed Cassandra via debian package, there is a possibility that your node has started before you explicitly started it. If so, it might have invalid node state. Have you tried wiping the data directory and trying again? What is your seed list? Are you sure the new node can reach the seeds on the network layer? =Rob
