Hi, Daniel:
I didn't find any branch related to C* 2.1 in the
https://github.com/coursera/aegisthus, is there one?
It looks like there are big changes in the C* 2.1 of collections API. Just want
to know if there is any CQLMapper for C* 2.1 branch. Meantime, I will also try
to understand more about the new changes in C* 2.1
Thanks
Yong
From: danc...@coursera.org
Date: Wed, 10 Jun 2015 09:11:02 -0700
Subject: Re: Deserialize the collection type data from the SSTable file
To: java8...@hotmail.com
CC: user@cassandra.apache.org
Hi Yong,
Glad the code was helpful. I believe it serializes using List<Pair<ByteBuffer,
Column>> for maps so that it can store the Key of the map as well.
Thanks for pointing out the edge case!Thanks,Daniel
On Wed, Jun 10, 2015 at 6:39 AM, java8964 <java8...@hotmail.com> wrote:
Thanks, Daniel.
I didn't realize that Cassandra will serialize one more way using
List<Pair<ByteBuffer, Column>> for collection types. Reading your example code,
I make it work.
>From link you gave me, using my test data, I found out one issue though. In
>any one row, if the collection column is NULL, I think the logic of code will
>throw NullPointException on line 148:
Line 145 private void addValue(GenericRecord record, CFDefinition.Name name,
ColumnGroupMap group){
Line 146 if (name.type.isCollection()) {
Line 147 List<Pair<ByteBuffer, Column>> collection =
group.getCollection(name.name.key);
Line 148 ByteBuffer buffer =
((CollectionType)name.type).serialize(collection);
addCqlCollectionToRecord(record, name, buffer);
If the collection column in that row is NULL, then Line 147 will return NULL,
which will cause the following exception:
Exception in thread "main" java.lang.NullPointerException at
org.apache.cassandra.db.marshal.CollectionType.enforceLimit(CollectionType.java:113)
at org.apache.cassandra.db.marshal.ListType.serialize(ListType.java:120)
So I need to add a check to avoid that, as any regular columns in Cassandra
could just have NULL value.
Thanks
Yong
From: danc...@coursera.org
Date: Mon, 8 Jun 2015 15:13:02 -0700
Subject: Re: Deserialize the collection type data from the SSTable file
To: user@cassandra.apache.org
I'm not sure why sstable2json doesn't work for collections, but if you're into
reading raw sstables we use the following code with good success:
https://github.com/coursera/aegisthus/blob/77c73f6259f2a30d3d8ca64578be5c13ecc4e6f4/aegisthus-hadoop/src/main/java/org/coursera/mapreducer/CQLMapper.java#L85
Thanks,Daniel
On Mon, Jun 8, 2015 at 1:22 PM, java8964 <java8...@hotmail.com> wrote:
Hi, Cassandra users:
I have a question related to how to Deserialize the new collection types data
in the Cassandra 2.x. (The exactly version is C 2.0.10).
I create the following example tables in the CQLSH:
CREATE TABLE coupon ( account_id bigint, campaign_id uuid,
........................, discount_info map<text, text>,
........................, PRIMARY KEY (account_id, campaign_id))
The other columns can be ignored in this case. Then I inserted into the one
test data like this:
insert into coupon (account_id, campaign_id, discount_info) values (111,uuid(),
{'test_key':'test_value'});
After this, I got the SSTable files. I use the sstable2json file to check the
output:
$./resources/cassandra/bin/sstable2json /xxx/test-coupon-jb-1-Data.db[{"key":
"000000000000006f","columns":
[["0336e50d-21aa-4b3a-9f01-989a8c540e54:","",1433792922055000],
["0336e50d-21aa-4b3a-9f01-989a8c540e54:discount_info","0336e50d-21aa-4b3a-9f01-989a8c540e54:discount_info:!",1433792922054999,"t",1433792922],
["0336e50d-21aa-4b3a-9f01-989a8c540e54:discount_info:746573745f6b6579","746573745f76616c7565",1433792922055000]]}]
What I want to is to get the {"test_key" : "test_value"} as key/value pair that
I input into "discount_info" column. I followed the sstable2json code, and try
to deserialize the data by myself, but to my surprise, I cannot make it work,
even I tried several ways, but kept getting Exception.
>From what I researched, I know that Cassandra put the "campaign_id" +
>"discount_info" + "Another ByteBuffer" as composite column in this case. When
>I deserialize this columnName, I got the following dumped out as String:
"0336e50d-21aa-4b3a-9f01-989a8c540e54:discount_info:746573745f6b6579".
It includes 3 parts: the first part is the uuid for the campaign_id. The 2nd
part as "discount_info", which is the static name I defined in the table. The 3
part is a bytes array as length of 46, which I am not sure what it is.
The corresponding value part of this composite column is another byte array as
length of 10, hex as "746573745f76616c7565" if I dump it out.
Now, here is what I did and not sure why it doesn't work. First, I assume the
value part stores the real value I put in the Map, so I did the following:
ByteBuffer value = ByteBufferUtil.clone(column.value());MapType<String, String>
result = MapType.getInstance(UTF8Type.instance, UTF8Type.instance);
Map<String, String> output = result.compose(value);// it gave me the following
exception: org.apache.cassandra.serializers.MarshalException: Not enough bytes
to read a mapThen I am think that the real value must be stored as part of the
column names (the 3rd part of 46 bytes), so I did this:MapType<String, String>
result = MapType.getInstance(UTF8Type.instance, UTF8Type.instance);
Map<String, String> output = result.compose(third_part.value);// I got the
following exception:java.lang.IllegalArgumentException
at java.nio.Buffer.limit(Buffer.java:267)
at
org.apache.cassandra.utils.ByteBufferUtil.readBytes(ByteBufferUtil.java:587)
at
org.apache.cassandra.utils.ByteBufferUtil.readBytesWithShortLength(ByteBufferUtil.java:596)
at
org.apache.cassandra.serializers.MapSerializer.deserialize(MapSerializer.java:63)
at
org.apache.cassandra.serializers.MapSerializer.deserialize(MapSerializer.java:28)
at
org.apache.cassandra.db.marshal.AbstractType.compose(AbstractType.java:142)
I can get all other non-collection types data, but I cannot get the data from
the Map. My questions are:1) How does the Cassandra store the collection data
in the SSTable files? From the length of bytes, it is most likely as part of
the composite column. If so, why I got the exception as above? 2) The
sstable2json doesn't deserialize the real data out from the collection type. So
I don't have an example to follow. Do I use the wrong way trying to compose the
Map type data?
Thanks
Yong
