Re: [RELEASE] Apache Cassandra 1.1.11
Hi! I can't find the 1.1.11 package for the Debian at http://www.apache.org/dist/cassandra/debian/ The .deb package is there but Packages files still contain just version 1.1.10. Regards, Patrik
Re: [RELEASE] Apache Cassandra 1.1.11
It works now, thanks! P. On Wed, Apr 24, 2013 at 2:53 PM, Sylvain Lebresne sylv...@datastax.com wrote: Sorry, seems I screwed up somehow. That should be fixed now however.
Re: random partitioner and key scan
On Fri, Jul 20, 2012 at 11:17 AM, aaron morton aa...@thelastpickle.com wrote: Ordering the rows by row key locally would mean that every row on the node would have to be scanned to find the ones whose token was in the required token range. I don't know much about Cassandra internals, but from a user point of view, a scan for a range of tokens is not a common use-case. Does Cassandra need it so much that it would be impossible to implement the md5_distribution+natural_keys localy? Regards, P.
random partitioner and key scan
Hi, I know that RandomPartitioner does MD5 of a key and the MD5 is then used for key distribution AND key ordering. I was just wondering if it's possible to have RandomPartitioner just for key distribution and OrderedPartitioner just for per-node key ordering. That would solve the often requested key scan feature. Regards, Patrik
Re: random partitioner and key scan
Hi Prasenjit, I don't see the need to recalculate anything. One key has a one MD5 hash, it doesn't change. Just use the hash to select a node, than just the plain key. Can you elaborate on the redistribution please? Regards, P. On Thu, Jul 19, 2012 at 9:09 AM, prasenjit mukherjee prasen@gmail.com wrote: The probem could be while redistributing the tokens. In that case the hashes has to be recalculated on each fo the candidate node. -Thanks, Prasenjit On Thu, Jul 19, 2012 at 12:19 PM, Patrik Modesto patrik.mode...@gmail.com wrote: Hi, I know that RandomPartitioner does MD5 of a key and the MD5 is then used for key distribution AND key ordering. I was just wondering if it's possible to have RandomPartitioner just for key distribution and OrderedPartitioner just for per-node key ordering. That would solve the often requested key scan feature. Regards, Patrik
Re: Removing a counter columns using Thrift interface
On Mon, Jun 25, 2012 at 9:28 AM, Sylvain Lebresne sylv...@datastax.com wrote: On Mon, Jun 25, 2012 at 9:06 AM, Patrik Modesto patrik.mode...@gmail.com wrote: I'm used to use Mutation for everything, so the first thing I tried was Deletion on Counter column. Well, nothing happened. No error and the Counter column was still there. That shouldn't happen. The second try was the remove_counter() method. When I set just the column_family of ColumnPath, nothing happened. No error and the Counter column was still there. I supposed it would work like the remove() method which would remove whole row. It should. If if it doesn't that would be a bug. If you can reproduce such a bug, then please do open a ticket. I've tried again today and found one bug in my test program. Now Deletion works as expected. The remove_counter() works as well, I misinterpreted the results. Regards, P.
Removing a counter columns using Thrift interface
Hi, I was playing with Counters in Cassandra 1.0.10 and I see a behaviour that I didn't expect. It's about removing a Counter column. I'm used to use Mutation for everything, so the first thing I tried was Deletion on Counter column. Well, nothing happened. No error and the Counter column was still there. The second try was the remove_counter() method. When I set just the column_family of ColumnPath, nothing happened. No error and the Counter column was still there. I supposed it would work like the remove() method which would remove whole row. The third try was the remove_counter() method with fully specified Counter column in ColumnPath. Now the Counter column was gone. Maybe it behaves differenrtly in 1.1.x version but I'd expect working Deletion() on Counter column or error and remove_counter() to work like remove(). Why did I try to remove a Counter column? Because I need to reset the counter to zero before computing something and I don't know other way. Suppose I have thousands of countes I need to reset and I don't want to read every Counter column I need to reset and then to all of them write their negative value. Regards, Patrik
Re: How Cassandra determines the splits
Hi,
I had a simillar problem with Cassandra 0.8.x and the problem was when
configured Cassandra with rpc_address: 0.0.0.0 and starting Hadoop job
from outside the Cassandra cluster. But with version 1.0.x the problem
is gone.
You can debug the splits with thrift. This is a copypaste part of my
splits testing Python utility:
print describe_ring
res = client.describe_ring(argv[1])
for t in res:
print %s - %s [%s] [%s] % (t.start_token, t.end_token,
,.join(t.endpoints), ,.join(t.rpc_endpoints),)
for r in res:
res2 = client.describe_splits('PageData',
r.start_token, r.end_token,
24*1024)
It asks Cassandra for a list of nodes with their key ranges, then asks
each node for slits. You should adjust the 24*1024 split size.
Regards,
Patrik
On Tue, May 1, 2012 at 5:58 PM, Filippo Diotalevi fili...@ntoklo.com wrote:
Hi,
I'm having problems in my Cassandra/Hadoop (1.0.8 + cdh3u3) cluster related
to how cassandra splits the data to be processed by Hadoop.
I'm currently testing a map reduce job, starting from a CF of roughly 1500
rows, with
cassandra.input.split.size 10
cassandra.range.batch.size 1
but what I consistently see is that, while most of the task have 1-20 rows
assigned each, one of them is assigned 400+ rows, which gives me all sort of
problems in terms of timeouts and memory consumption (not to mention seeing
the mapper progress bar going to 4000% and more).
Do you have any suggestion to solve/troublehsoot this issue?
--
Filippo Diotalevi
Re: AssertionError: originally calculated column size ...
https://issues.apache.org/jira/browse/CASSANDRA-4206 Regards, Patrik On Tue, May 1, 2012 at 03:46, Jonathan Ellis jbel...@gmail.com wrote: On Mon, Apr 30, 2012 at 2:11 PM, Patrik Modesto patrik.mode...@gmail.com wrote: I think the problem is somehow connected to an IntegerType secondary index. Could be, but my money is on the supercolumns in the HH data model. Can you create a jira ticket? -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: AssertionError: originally calculated column size ...
Hi Aaron, it's still an issue. However it's difficult to reproduce because I'd need to insert data to Cassandra for day or more which is not possible. I think the problem is somehow connected to an IntegerType secondary index. I had a different problem with CF with two secondary indexes, the first UTF8Type, the second IntegerType. After a few hours of inserting data in the afternoon and midnight repair+compact, the next day I couldn't find any row using the IntegerType secondary index. The output was like this: [default@rfTest3] get IndexTest where col1 = '3230727:http://zaskolak.cz/download.php'; --- RowKey: 3230727:8383582:http://zaskolak.cz/download.php = (column=col1, value=3230727:http://zaskolak.cz/download.php, timestamp=1335348630332000) = (column=col2, value=8383582, timestamp=1335348630332000) --- RowKey: 3230727:8383583:http://zaskolak.cz/download.php = (column=col1, value=3230727:http://zaskolak.cz/download.php, timestamp=1335348449078000) = (column=col2, value=8383583, timestamp=1335348449078000) --- RowKey: 3230727:8383579:http://zaskolak.cz/download.php = (column=col1, value=3230727:http://zaskolak.cz/download.php, timestamp=1335348778577000) = (column=col2, value=8383579, timestamp=1335348778577000) 3 Rows Returned. Elapsed time: 292 msec(s). [default@rfTest3] get IndexTest where col2 = 8383583; 0 Row Returned. Elapsed time: 7 msec(s You can see there really is an 8383583 in col2 in on of the listed rows, but the search by secondary index returns nothing. The Assert Exception also happend only on CF with the secondary index of IntegerType. There were also secondary indexes of UTF8Type and LongType types. It's the first time I've tried secondary indexes of other type than UTF8Type. Regards, Patrik On Mon, Apr 30, 2012 at 06:05, aaron morton aa...@thelastpickle.com wrote: Looks a bit like https://issues.apache.org/jira/browse/CASSANDRA-3579 but that was fixed in 1.0.7 Is this still an issue ? Are you able to reproduce the fault ? Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 27/04/2012, at 6:56 PM, Patrik Modesto wrote: Hi, I've 4 node cluster of Cassandra 1.0.9. There is a rfTest3 keyspace with RF=3 and one CF with two secondary indexes. I'm importing data into this CF using Hadoop Mapreduce job, each row has less than 10 colkumns. From JMX: MaxRowSize: 1597 MeanRowSize: 369 And there are some tens of millions of rows. It's write-heavy usage and there is a big pressure on each node, there are quite some dropped mutations on each node. After ~12 hours of inserting I see these assertion exceptiona on 3 out of four nodes: ERROR 06:25:40,124 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:388) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:384) ... 7 more Caused by: java.lang.AssertionError: originally calculated column size of 629444349 but now it is 588008950 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:161) at org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:380) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more Few lines regarding Hints from the output.log: INFO 06:21:26,202 Compacting large row system/HintsColumnFamily:7000 (1712834057 bytes
Proper Mutation to IntegerType column w/ secondary index
Hi,
I see strange behaviour using CF with two secondary indexes, one
IntegerType one UTF8Type.
Using Cassanra 1.0.9 and CF:
create column family IndexTest
with column_type=Standard
and comparator=UTF8Type
and default_validation_class=UTF8Type
and key_validation_class=UTF8Type
and column_metadata=
[{
column_name: col1,
validation_class: UTF8Type,
index_type: KEYS
},
{
column_name: col2,
validation_class: IntegerType,
index_type: KEYS
}];
There are 13 928 437 rows. Col1 and col2 are created in MapReduce job
from other columns. The MapReruce job is used for recreating the index
columns. The col1 secondary index works the col2 behaves strangely.
Sometimes it returns correct result, sometimes nothing like in
following example.
[default@rfTest3] get IndexTest where col1 =
'3230727:http://zaskolak.cz/download.php';
---
RowKey: 3230727:8383582:http://zaskolak.cz/download.php
= (column=col1, value=3230727:http://zaskolak.cz/download.php,
timestamp=1335348630332000)
= (column=col2, value=8383582, timestamp=1335348630332000)
---
RowKey: 3230727:8383583:http://zaskolak.cz/download.php
= (column=col1, value=3230727:http://zaskolak.cz/download.php,
timestamp=1335348449078000)
= (column=col2, value=8383583, timestamp=1335348449078000)
---
RowKey: 3230727:8383579:http://zaskolak.cz/download.php
= (column=col1, value=3230727:http://zaskolak.cz/download.php,
timestamp=1335348778577000)
= (column=col2, value=8383579, timestamp=1335348778577000)
3 Rows Returned.
Elapsed time: 292 msec(s).
[default@rfTest3] get IndexTest where col2 = 8383583;
0 Row Returned.
Elapsed time: 7 msec(s
The col2 value is written from MapReduce like this 'ByteBufferUtil.bytes(Id)'.
What can be the problem here?
Regards,
P.
Re: Proper Mutation to IntegerType column w/ secondary index
I'll also try LongType index. I'll let you know how that works. Regards, P.
Re: Poor write performance with seconrady index
Hi Aaron, thanks for the reply. I suspected it might be the
read-and-write that causes the slower updates.
Regards,
P.
On Tue, Apr 17, 2012 at 11:52, aaron morton aa...@thelastpickle.com wrote:
Secondary indexes require a read and a write (potentially two) for every
update. Regular mutations are no look writes and are much faster.
Just like in a RDBMS, it's more efficient to insert data and then create the
index than to insert data with the index present.
An alternative is to create SSTables in the hadoop jobs and bulk load them
into the cluster.
Cheers
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 17/04/2012, at 2:51 AM, Patrik Modesto wrote:
Hi,
I've a 4 node test cluster running Cassandra 1.0.9, 32GB memory, 4x
1TB disks. I've two keyspaces, rfTest2 (RF=2) and rfTest3 (RF=3).
There are two CF, one with source data and one with secondary index:
create column family UrlGroup
with column_type=Standard
and comparator=UTF8Type
and default_validation_class=UTF8Type
and key_validation_class=UTF8Type
and column_metadata=
[{
column_name: groupId,
validation_class: UTF8Type,
index_type: KEYS
}];
I'm running Hadoop mapreduce job, reading the source CF and creating 3
mutations for each row-key in the UrlGroup CF.
The mapreduce runs for 30minutes. When I remove the secondary index,
the mapreduce runs just 10minutes. There are 26,273,544 mutations
total.
Also with the secondary index, the nodes show very high load 50+ and
iowait 70%+. Without secondary index the load is ~5 and iowait ~10%.
What may be the problem?
Regards,
Patrik
Re: newer Cassandra + Hadoop = TimedOutException()
I did change the rpc_endpoint to endpoints and now the splits are
computed correctly. So it's a bug in cassandra to hadoop interface. I
suspect that it has something to do with wide rows with tens of
thousands of columns we have because the unpatched getSubSplits()
works with small test data we have for development.
Regards,
P.
On Wed, Mar 7, 2012 at 11:02, Florent Lefillâtre flefi...@gmail.com wrote:
If you want try a test, in the CFIF.getSubSplits(String, String, TokenRange,
Configuration) method, replace the loop on 'range.rpc_endpoints' by the same
loop on 'range.endpoints'.
This method split token range of each node with describe_splits method, but
I think there is something wrong when you create Cassandra connection on
host '0.0.0.0'.
Le 7 mars 2012 09:07, Patrik Modesto patrik.mode...@gmail.com a écrit :
You're right, I wasn't looking in the right logs. Unfortunately I'd
need to restart hadoop takstracker with loglevel DEBUG and that is not
possilbe at the moment. Pitty it happens only in the production with
terrabytes of data, not in the test...
Regards,
P.
On Tue, Mar 6, 2012 at 14:31, Florent Lefillâtre flefi...@gmail.com
wrote:
CFRR.getProgress() is called by child mapper tasks on each TastTracker
node,
so the log must appear on
${hadoop_log_dir}/attempt_201202081707_0001_m_00_0/syslog (or
somethings
like this) on TaskTrackers, not on client job logs.
Are you sure to see the good log file, I say that because in your first
mail
you link the client job log.
And may be you can log the size of each split in CFIF.
Le 6 mars 2012 13:09, Patrik Modesto patrik.mode...@gmail.com a écrit
:
I've added a debug message in the CFRR.getProgress() and I can't find
it in the debug output. Seems like the getProgress() has not been
called at all;
Regards,
P.
On Tue, Mar 6, 2012 at 09:49, Jeremy Hanna jeremy.hanna1...@gmail.com
wrote:
you may be running into this -
https://issues.apache.org/jira/browse/CASSANDRA-3942 - I'm not sure
if it
really affects the execution of the job itself though.
On Mar 6, 2012, at 2:32 AM, Patrik Modesto wrote:
Hi,
I was recently trying Hadoop job + cassandra-all 0.8.10 again and
the
Timeouts I get are not because of the Cassandra can't handle the
requests. I've noticed there are several tasks that show proggess of
several thousands percents. Seems like they are looping their range
of
keys. I've run the job with debug enabled and the ranges look ok,
see
http://pastebin.com/stVsFzLM
Another difference between cassandra-all 0.8.7 and 0.8.10 is the
number of mappers the job creates:
0.8.7: 4680
0.8.10: 595
Task Complete
task_201202281457_2027_m_41 9076.81%
task_201202281457_2027_m_73 9639.04%
task_201202281457_2027_m_000105 10538.60%
task_201202281457_2027_m_000108 9364.17%
None of this happens with cassandra-all 0.8.7.
Regards,
P.
On Tue, Feb 28, 2012 at 12:29, Patrik Modesto
patrik.mode...@gmail.com wrote:
I'll alter these settings and will let you know.
Regards,
P.
On Tue, Feb 28, 2012 at 09:23, aaron morton
aa...@thelastpickle.com
wrote:
Have you tried lowering the batch size and increasing the time
out?
Even
just to get it to work.
If you get a TimedOutException it means CL number of servers did
not
respond
in time.
Cheers
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 28/02/2012, at 8:18 PM, Patrik Modesto wrote:
Hi aaron,
this is our current settings:
property
namecassandra.range.batch.size/name
value1024/value
/property
property
namecassandra.input.split.size/name
value16384/value
/property
rpc_timeout_in_ms: 3
Regards,
P.
On Mon, Feb 27, 2012 at 21:54, aaron morton
aa...@thelastpickle.com
wrote:
What settings do you have for cassandra.range.batch.size
and rpc_timeout_in_ms ? Have you tried reducing the first and/or
increasing
the second ?
Cheers
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 27/02/2012, at 8:02 PM, Patrik Modesto wrote:
On Sun, Feb 26, 2012 at 04:25, Edward Capriolo
edlinuxg...@gmail.com
wrote:
Did you see the notes here?
I'm not sure what do you mean by the notes?
I'm using the mapred.* settings suggested there:
property
namemapred.max.tracker.failures/name
value20/value
/property
property
namemapred.map.max.attempts/name
value20/value
/property
property
namemapred.reduce.max.attempts
Re: newer Cassandra + Hadoop = TimedOutException()
You're right, I wasn't looking in the right logs. Unfortunately I'd
need to restart hadoop takstracker with loglevel DEBUG and that is not
possilbe at the moment. Pitty it happens only in the production with
terrabytes of data, not in the test...
Regards,
P.
On Tue, Mar 6, 2012 at 14:31, Florent Lefillâtre flefi...@gmail.com wrote:
CFRR.getProgress() is called by child mapper tasks on each TastTracker node,
so the log must appear on
${hadoop_log_dir}/attempt_201202081707_0001_m_00_0/syslog (or somethings
like this) on TaskTrackers, not on client job logs.
Are you sure to see the good log file, I say that because in your first mail
you link the client job log.
And may be you can log the size of each split in CFIF.
Le 6 mars 2012 13:09, Patrik Modesto patrik.mode...@gmail.com a écrit :
I've added a debug message in the CFRR.getProgress() and I can't find
it in the debug output. Seems like the getProgress() has not been
called at all;
Regards,
P.
On Tue, Mar 6, 2012 at 09:49, Jeremy Hanna jeremy.hanna1...@gmail.com
wrote:
you may be running into this -
https://issues.apache.org/jira/browse/CASSANDRA-3942 - I'm not sure if it
really affects the execution of the job itself though.
On Mar 6, 2012, at 2:32 AM, Patrik Modesto wrote:
Hi,
I was recently trying Hadoop job + cassandra-all 0.8.10 again and the
Timeouts I get are not because of the Cassandra can't handle the
requests. I've noticed there are several tasks that show proggess of
several thousands percents. Seems like they are looping their range of
keys. I've run the job with debug enabled and the ranges look ok, see
http://pastebin.com/stVsFzLM
Another difference between cassandra-all 0.8.7 and 0.8.10 is the
number of mappers the job creates:
0.8.7: 4680
0.8.10: 595
Task Complete
task_201202281457_2027_m_41 9076.81%
task_201202281457_2027_m_73 9639.04%
task_201202281457_2027_m_000105 10538.60%
task_201202281457_2027_m_000108 9364.17%
None of this happens with cassandra-all 0.8.7.
Regards,
P.
On Tue, Feb 28, 2012 at 12:29, Patrik Modesto
patrik.mode...@gmail.com wrote:
I'll alter these settings and will let you know.
Regards,
P.
On Tue, Feb 28, 2012 at 09:23, aaron morton aa...@thelastpickle.com
wrote:
Have you tried lowering the batch size and increasing the time out?
Even
just to get it to work.
If you get a TimedOutException it means CL number of servers did not
respond
in time.
Cheers
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 28/02/2012, at 8:18 PM, Patrik Modesto wrote:
Hi aaron,
this is our current settings:
property
namecassandra.range.batch.size/name
value1024/value
/property
property
namecassandra.input.split.size/name
value16384/value
/property
rpc_timeout_in_ms: 3
Regards,
P.
On Mon, Feb 27, 2012 at 21:54, aaron morton aa...@thelastpickle.com
wrote:
What settings do you have for cassandra.range.batch.size
and rpc_timeout_in_ms ? Have you tried reducing the first and/or
increasing
the second ?
Cheers
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 27/02/2012, at 8:02 PM, Patrik Modesto wrote:
On Sun, Feb 26, 2012 at 04:25, Edward Capriolo
edlinuxg...@gmail.com
wrote:
Did you see the notes here?
I'm not sure what do you mean by the notes?
I'm using the mapred.* settings suggested there:
property
namemapred.max.tracker.failures/name
value20/value
/property
property
namemapred.map.max.attempts/name
value20/value
/property
property
namemapred.reduce.max.attempts/name
value20/value
/property
But I still see the timeouts that I haven't with cassandra-all 0.8.7.
P.
http://wiki.apache.org/cassandra/HadoopSupport#Troubleshooting
Re: newer Cassandra + Hadoop = TimedOutException()
Hi Florent, I don't change the server version, it is the Cassandra 0.8.10. I change just the version of cassandra-all in pom.xml of the mapreduce job. I have the 'rpc_address: 0.0.0.0' in cassandra.yaml, because I want cassandra to bind RPC to all interfaces. Regards, P. On Tue, Mar 6, 2012 at 09:44, Florent Lefillâtre flefi...@gmail.com wrote: Hi, I had the same problem on hadoop 0.20.2 and cassandra 1.0.5. In my case the split of token range failed. I have comment line 'rpc_address: 0.0.0.0' in cassandra.yaml. May be see if you have not configuration changes between 0.8.7 and 0.8.10 Le 6 mars 2012 09:32, Patrik Modesto patrik.mode...@gmail.com a écrit : Hi, I was recently trying Hadoop job + cassandra-all 0.8.10 again and the Timeouts I get are not because of the Cassandra can't handle the requests. I've noticed there are several tasks that show proggess of several thousands percents. Seems like they are looping their range of keys. I've run the job with debug enabled and the ranges look ok, see http://pastebin.com/stVsFzLM Another difference between cassandra-all 0.8.7 and 0.8.10 is the number of mappers the job creates: 0.8.7: 4680 0.8.10: 595 Task Complete task_201202281457_2027_m_41 9076.81% task_201202281457_2027_m_73 9639.04% task_201202281457_2027_m_000105 10538.60% task_201202281457_2027_m_000108 9364.17% None of this happens with cassandra-all 0.8.7. Regards, P. On Tue, Feb 28, 2012 at 12:29, Patrik Modesto patrik.mode...@gmail.com wrote: I'll alter these settings and will let you know. Regards, P. On Tue, Feb 28, 2012 at 09:23, aaron morton aa...@thelastpickle.com wrote: Have you tried lowering the batch size and increasing the time out? Even just to get it to work. If you get a TimedOutException it means CL number of servers did not respond in time. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 28/02/2012, at 8:18 PM, Patrik Modesto wrote: Hi aaron, this is our current settings: property namecassandra.range.batch.size/name value1024/value /property property namecassandra.input.split.size/name value16384/value /property rpc_timeout_in_ms: 3 Regards, P. On Mon, Feb 27, 2012 at 21:54, aaron morton aa...@thelastpickle.com wrote: What settings do you have for cassandra.range.batch.size and rpc_timeout_in_ms ? Have you tried reducing the first and/or increasing the second ? Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 27/02/2012, at 8:02 PM, Patrik Modesto wrote: On Sun, Feb 26, 2012 at 04:25, Edward Capriolo edlinuxg...@gmail.com wrote: Did you see the notes here? I'm not sure what do you mean by the notes? I'm using the mapred.* settings suggested there: property namemapred.max.tracker.failures/name value20/value /property property namemapred.map.max.attempts/name value20/value /property property namemapred.reduce.max.attempts/name value20/value /property But I still see the timeouts that I haven't with cassandra-all 0.8.7. P. http://wiki.apache.org/cassandra/HadoopSupport#Troubleshooting
Re: newer Cassandra + Hadoop = TimedOutException()
I've tryied cassandra-all 0.8.10 with fixed the rpc_endpoints == 0.0.0.0 bug, but the result is the same, there are still tasks over 1000%. The only change is that there are real host names instead of 0.0.0.0 in the debug output. Reconfiguring whole cluster is not possible, I can't test the rpc_address commented out. Regards, P. On Tue, Mar 6, 2012 at 12:26, Florent Lefillâtre flefi...@gmail.com wrote: I remember a bug on the ColumnFamilyInputFormat class 0.8.10. It was a test rpc_endpoints == 0.0.0.0 in place of rpc_endpoint.equals(0.0.0.0), may be it can help you Le 6 mars 2012 12:18, Florent Lefillâtre flefi...@gmail.com a écrit : Excuse me, I had not understood. So, for me, the problem comes from the change of ColumnFamilyInputFormat class between 0.8.7 and 0.8.10 where the splits are created (0.8.7 uses endpoints and 0.8.10 uses rpc_endpoints). With your config, splits fails, so Hadoop doesn't run a Map task on approximtively 16384 rows (your cassandra.input.split.size) but on all the rows of a node (certainly more over 16384). However Hadoop estimate the task progress on 16384 inputs, it's why you have something like 9076.81%. If you can't change rpc_adress configuration, I don't know how you can solve your problem :/, sorry. Le 6 mars 2012 11:53, Patrik Modesto patrik.mode...@gmail.com a écrit : Hi Florent, I don't change the server version, it is the Cassandra 0.8.10. I change just the version of cassandra-all in pom.xml of the mapreduce job. I have the 'rpc_address: 0.0.0.0' in cassandra.yaml, because I want cassandra to bind RPC to all interfaces. Regards, P. On Tue, Mar 6, 2012 at 09:44, Florent Lefillâtre flefi...@gmail.com wrote: Hi, I had the same problem on hadoop 0.20.2 and cassandra 1.0.5. In my case the split of token range failed. I have comment line 'rpc_address: 0.0.0.0' in cassandra.yaml. May be see if you have not configuration changes between 0.8.7 and 0.8.10 Le 6 mars 2012 09:32, Patrik Modesto patrik.mode...@gmail.com a écrit : Hi, I was recently trying Hadoop job + cassandra-all 0.8.10 again and the Timeouts I get are not because of the Cassandra can't handle the requests. I've noticed there are several tasks that show proggess of several thousands percents. Seems like they are looping their range of keys. I've run the job with debug enabled and the ranges look ok, see http://pastebin.com/stVsFzLM Another difference between cassandra-all 0.8.7 and 0.8.10 is the number of mappers the job creates: 0.8.7: 4680 0.8.10: 595 Task Complete task_201202281457_2027_m_41 9076.81% task_201202281457_2027_m_73 9639.04% task_201202281457_2027_m_000105 10538.60% task_201202281457_2027_m_000108 9364.17% None of this happens with cassandra-all 0.8.7. Regards, P. On Tue, Feb 28, 2012 at 12:29, Patrik Modesto patrik.mode...@gmail.com wrote: I'll alter these settings and will let you know. Regards, P. On Tue, Feb 28, 2012 at 09:23, aaron morton aa...@thelastpickle.com wrote: Have you tried lowering the batch size and increasing the time out? Even just to get it to work. If you get a TimedOutException it means CL number of servers did not respond in time. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 28/02/2012, at 8:18 PM, Patrik Modesto wrote: Hi aaron, this is our current settings: property namecassandra.range.batch.size/name value1024/value /property property namecassandra.input.split.size/name value16384/value /property rpc_timeout_in_ms: 3 Regards, P. On Mon, Feb 27, 2012 at 21:54, aaron morton aa...@thelastpickle.com wrote: What settings do you have for cassandra.range.batch.size and rpc_timeout_in_ms ? Have you tried reducing the first and/or increasing the second ? Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 27/02/2012, at 8:02 PM, Patrik Modesto wrote: On Sun, Feb 26, 2012 at 04:25, Edward Capriolo edlinuxg...@gmail.com wrote: Did you see the notes here? I'm not sure what do you mean by the notes? I'm using the mapred.* settings suggested there: property namemapred.max.tracker.failures/name value20/value /property property namemapred.map.max.attempts/name value20/value /property property namemapred.reduce.max.attempts/name value20/value /property But I still see the timeouts that I haven't with cassandra-all 0.8.7. P. http://wiki.apache.org
Re: newer Cassandra + Hadoop = TimedOutException()
I've added a debug message in the CFRR.getProgress() and I can't find it in the debug output. Seems like the getProgress() has not been called at all; Regards, P. On Tue, Mar 6, 2012 at 09:49, Jeremy Hanna jeremy.hanna1...@gmail.com wrote: you may be running into this - https://issues.apache.org/jira/browse/CASSANDRA-3942 - I'm not sure if it really affects the execution of the job itself though. On Mar 6, 2012, at 2:32 AM, Patrik Modesto wrote: Hi, I was recently trying Hadoop job + cassandra-all 0.8.10 again and the Timeouts I get are not because of the Cassandra can't handle the requests. I've noticed there are several tasks that show proggess of several thousands percents. Seems like they are looping their range of keys. I've run the job with debug enabled and the ranges look ok, see http://pastebin.com/stVsFzLM Another difference between cassandra-all 0.8.7 and 0.8.10 is the number of mappers the job creates: 0.8.7: 4680 0.8.10: 595 Task Complete task_201202281457_2027_m_41 9076.81% task_201202281457_2027_m_73 9639.04% task_201202281457_2027_m_000105 10538.60% task_201202281457_2027_m_000108 9364.17% None of this happens with cassandra-all 0.8.7. Regards, P. On Tue, Feb 28, 2012 at 12:29, Patrik Modesto patrik.mode...@gmail.com wrote: I'll alter these settings and will let you know. Regards, P. On Tue, Feb 28, 2012 at 09:23, aaron morton aa...@thelastpickle.com wrote: Have you tried lowering the batch size and increasing the time out? Even just to get it to work. If you get a TimedOutException it means CL number of servers did not respond in time. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 28/02/2012, at 8:18 PM, Patrik Modesto wrote: Hi aaron, this is our current settings: property namecassandra.range.batch.size/name value1024/value /property property namecassandra.input.split.size/name value16384/value /property rpc_timeout_in_ms: 3 Regards, P. On Mon, Feb 27, 2012 at 21:54, aaron morton aa...@thelastpickle.com wrote: What settings do you have for cassandra.range.batch.size and rpc_timeout_in_ms ? Have you tried reducing the first and/or increasing the second ? Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 27/02/2012, at 8:02 PM, Patrik Modesto wrote: On Sun, Feb 26, 2012 at 04:25, Edward Capriolo edlinuxg...@gmail.com wrote: Did you see the notes here? I'm not sure what do you mean by the notes? I'm using the mapred.* settings suggested there: property namemapred.max.tracker.failures/name value20/value /property property namemapred.map.max.attempts/name value20/value /property property namemapred.reduce.max.attempts/name value20/value /property But I still see the timeouts that I haven't with cassandra-all 0.8.7. P. http://wiki.apache.org/cassandra/HadoopSupport#Troubleshooting
Re: newer Cassandra + Hadoop = TimedOutException()
I'll alter these settings and will let you know. Regards, P. On Tue, Feb 28, 2012 at 09:23, aaron morton aa...@thelastpickle.com wrote: Have you tried lowering the batch size and increasing the time out? Even just to get it to work. If you get a TimedOutException it means CL number of servers did not respond in time. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 28/02/2012, at 8:18 PM, Patrik Modesto wrote: Hi aaron, this is our current settings: property namecassandra.range.batch.size/name value1024/value /property property namecassandra.input.split.size/name value16384/value /property rpc_timeout_in_ms: 3 Regards, P. On Mon, Feb 27, 2012 at 21:54, aaron morton aa...@thelastpickle.com wrote: What settings do you have for cassandra.range.batch.size and rpc_timeout_in_ms ? Have you tried reducing the first and/or increasing the second ? Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 27/02/2012, at 8:02 PM, Patrik Modesto wrote: On Sun, Feb 26, 2012 at 04:25, Edward Capriolo edlinuxg...@gmail.com wrote: Did you see the notes here? I'm not sure what do you mean by the notes? I'm using the mapred.* settings suggested there: property namemapred.max.tracker.failures/name value20/value /property property namemapred.map.max.attempts/name value20/value /property property namemapred.reduce.max.attempts/name value20/value /property But I still see the timeouts that I haven't with cassandra-all 0.8.7. P. http://wiki.apache.org/cassandra/HadoopSupport#Troubleshooting
Re: newer Cassandra + Hadoop = TimedOutException()
Hi aaron, this is our current settings: property namecassandra.range.batch.size/name value1024/value /property property namecassandra.input.split.size/name value16384/value /property rpc_timeout_in_ms: 3 Regards, P. On Mon, Feb 27, 2012 at 21:54, aaron morton aa...@thelastpickle.com wrote: What settings do you have for cassandra.range.batch.size and rpc_timeout_in_ms ? Have you tried reducing the first and/or increasing the second ? Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 27/02/2012, at 8:02 PM, Patrik Modesto wrote: On Sun, Feb 26, 2012 at 04:25, Edward Capriolo edlinuxg...@gmail.com wrote: Did you see the notes here? I'm not sure what do you mean by the notes? I'm using the mapred.* settings suggested there: property namemapred.max.tracker.failures/name value20/value /property property namemapred.map.max.attempts/name value20/value /property property namemapred.reduce.max.attempts/name value20/value /property But I still see the timeouts that I haven't with cassandra-all 0.8.7. P. http://wiki.apache.org/cassandra/HadoopSupport#Troubleshooting
Re: newer Cassandra + Hadoop = TimedOutException()
On Sun, Feb 26, 2012 at 04:25, Edward Capriolo edlinuxg...@gmail.com wrote: Did you see the notes here? I'm not sure what do you mean by the notes? I'm using the mapred.* settings suggested there: property namemapred.max.tracker.failures/name value20/value /property property namemapred.map.max.attempts/name value20/value /property property namemapred.reduce.max.attempts/name value20/value /property But I still see the timeouts that I haven't with cassandra-all 0.8.7. P. http://wiki.apache.org/cassandra/HadoopSupport#Troubleshooting
newer Cassandra + Hadoop = TimedOutException()
Hi, I can see some strange behaviour on my test cluster and in production. Both running cassandra 0.8.10. Strange is that when I compile my mapreduce job against cassandra-all 0.8.7 everything is ok, but if I use higher version I get quite a lots of TimedOutException. java.lang.RuntimeException: TimedOutException() at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.maybeInit(ColumnFamilyRecordReader.java:319) at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:333) at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(ColumnFamilyRecordReader.java:207) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:135) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:130) at org.apache.cassandra.hadoop.ColumnFamilyRecordReader.nextKeyValue(ColumnFamilyRecordReader.java:163) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:456) at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:32 There is nothing in the cassandra log, the cluster is idle, no one else is accessing the cluster. There are just few rows, nothing big: INFO mapred.JobClient: Reduce input groups=14 INFO mapred.JobClient: Combine output records=0 INFO mapred.JobClient: Map input records=544009 INFO mapred.JobClient: Reduce shuffle bytes=33876 INFO mapred.JobClient: Reduce output records=0 INFO mapred.JobClient: Spilled Records=38 INFO mapred.JobClient: Map output bytes=33656 INFO mapred.JobClient: Combine input records=0 INFO mapred.JobClient: Map output records=19 INFO mapred.JobClient: SPLIT_RAW_BYTES=3937 INFO mapred.JobClient: Reduce input records=19 What could be the problem? Regards, P.
Re: newer Cassandra + Hadoop = TimedOutException()
Some more info, running the cluster with debug enabled shows this error: DEBUG 14:23:05,749 Thrift transport error occurred during processing of message. org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:147) at org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:157) at org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.process(Cassandra.java:3221) at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889) at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145) ... 7 more Regards, P.
Re: newer Cassandra + Hadoop = TimedOutException()
There are two more similar exceptions: DEBUG 14:23:12,817 Thrift transport error occurred during processing of message. org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129) at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204) at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2877) at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) and DEBUG 14:23:13,938 Thrift transport error occurred during processing of message. org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:147) at org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:157) at org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.process(Cassandra.java:3221) at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889) at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.net.SocketException: Connection reset at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145) ... 7 more
Re: newer Cassandra + Hadoop = TimedOutException()
Hi Jeremy, I've seen the page and tried the values but to no help. Here goes tcpdump of one failed TCP connection: 15:06:20.231421 IP 10.0.18.87.9160 10.0.18.87.39396: Flags [P.], seq 137891735:137904068, ack 708, win 282, options [nop,nop,TS val 4119981636 ecr 4119981636], length 12333 15:06:20.231434 IP 10.0.18.87.39396 10.0.18.87.9160: Flags [.], ack 137891735, win 24576, options [nop,nop,TS val 4119981636 ecr 4119981636], length 0 15:06:20.231459 IP 10.0.18.87.39396 10.0.18.87.9160: Flags [.], ack 137904068, win 24576, options [nop,nop,TS val 4119981636 ecr 4119981636], length 0 15:06:20.271128 IP 10.0.18.87.39396 10.0.18.87.9160: Flags [P.], seq 708:712, ack 137904068, win 24576, options [nop,nop,TS val 4119981646 ecr 4119981636], length 4 15:06:20.271156 IP 10.0.18.87.39396 10.0.18.87.9160: Flags [P.], seq 712:901, ack 137904068, win 24576, options [nop,nop,TS val 4119981646 ecr 4119981636], length 189 15:06:20.271217 IP 10.0.18.87.9160 10.0.18.87.39396: Flags [.], ack 901, win 290, options [nop,nop,TS val 4119981646 ecr 4119981646], length 0 15:06:21.450164 IP 10.0.18.87.39396 10.0.18.87.9160: Flags [F.], seq 901, ack 137904068, win 24576, options [nop,nop,TS val 4119981941 ecr 4119981646], length 0 15:06:21.490323 IP 10.0.18.87.9160 10.0.18.87.39396: Flags [.], ack 902, win 290, options [nop,nop,TS val 4119981951 ecr 4119981941], length 0 15:06:38.487063 IP 10.0.18.87.9160 10.0.18.87.39396: Flags [P.], seq 137904068:137904072, ack 902, win 290, options [nop,nop,TS val 4119986200 ecr 4119981941], length 4 15:06:38.487079 IP 10.0.18.87.39396 10.0.18.87.9160: Flags [R], seq 1086151357, win 0, length 0 port 39396 is client, port 9160 is cassandra server. interresting are the last four lines. There if FIN from client to cassandra, then cassandra reply with ACK, then ACK+PSH and then gets RST and fails. It looks like cassandra writes to closed connection. Regards, P.
Re: newer Cassandra + Hadoop = TimedOutException()
No, we have our own servers, no cloud service. P. On Feb 24, 2012 5:57 PM, Jeremy Hanna jeremy.hanna1...@gmail.com wrote: By chance are you in EC2? On Feb 24, 2012, at 8:33 AM, Patrik Modesto wrote: Hi Jeremy, I've seen the page and tried the values but to no help. Here goes tcpdump of one failed TCP connection: 15:06:20.231421 IP 10.0.18.87.9160 10.0.18.87.39396: Flags [P.], seq 137891735:137904068, ack 708, win 282, options [nop,nop,TS val 4119981636 ecr 4119981636], length 12333 15:06:20.231434 IP 10.0.18.87.39396 10.0.18.87.9160: Flags [.], ack 137891735, win 24576, options [nop,nop,TS val 4119981636 ecr 4119981636], length 0 15:06:20.231459 IP 10.0.18.87.39396 10.0.18.87.9160: Flags [.], ack 137904068, win 24576, options [nop,nop,TS val 4119981636 ecr 4119981636], length 0 15:06:20.271128 IP 10.0.18.87.39396 10.0.18.87.9160: Flags [P.], seq 708:712, ack 137904068, win 24576, options [nop,nop,TS val 4119981646 ecr 4119981636], length 4 15:06:20.271156 IP 10.0.18.87.39396 10.0.18.87.9160: Flags [P.], seq 712:901, ack 137904068, win 24576, options [nop,nop,TS val 4119981646 ecr 4119981636], length 189 15:06:20.271217 IP 10.0.18.87.9160 10.0.18.87.39396: Flags [.], ack 901, win 290, options [nop,nop,TS val 4119981646 ecr 4119981646], length 0 15:06:21.450164 IP 10.0.18.87.39396 10.0.18.87.9160: Flags [F.], seq 901, ack 137904068, win 24576, options [nop,nop,TS val 4119981941 ecr 4119981646], length 0 15:06:21.490323 IP 10.0.18.87.9160 10.0.18.87.39396: Flags [.], ack 902, win 290, options [nop,nop,TS val 4119981951 ecr 4119981941], length 0 15:06:38.487063 IP 10.0.18.87.9160 10.0.18.87.39396: Flags [P.], seq 137904068:137904072, ack 902, win 290, options [nop,nop,TS val 4119986200 ecr 4119981941], length 4 15:06:38.487079 IP 10.0.18.87.39396 10.0.18.87.9160: Flags [R], seq 1086151357, win 0, length 0 port 39396 is client, port 9160 is cassandra server. interresting are the last four lines. There if FIN from client to cassandra, then cassandra reply with ACK, then ACK+PSH and then gets RST and fails. It looks like cassandra writes to closed connection. Regards, P.
truncate command fails
Hi, I've a strange problem with my test cluster. Calling truncate on a small ColumnFamily on idle cluster of 4 nodes returns UnavailableException after 10s. That 10s is set in rpc_timeout_in_ms. All nodes are up and running, here is nodetool ring: Address DC RackStatus State Load OwnsToken 148873535527910577765226390751398592512 10.0.18.129 GO RAC1Up Normal 7.93 GB 25.00% 21267647932558653966460912964485513216 10.0.18.99 GO RAC1Up Normal 7.44 GB 25.00% 63802943797675961899382738893456539648 10.0.18.98 GO RAC1Up Normal 7.11 GB 25.00% 106338239662793269832304564822427566080 10.0.18.87 GO RAC1Up Normal 7.78 GB 25.00% 148873535527910577765226390751398592512 #using Cassandra 0.8.9 on Debian Squeeze What may be the problem? Regards, Patrik
Re: truncate command fails
On Tue, Feb 7, 2012 at 18:35, Brandon Williams dri...@gmail.com wrote: Make sure you have JNA, without it the cost of forking ln to snapshot is expensive. I assume JNA is enabled. From the output.log: INFO 10:16:20,725 JNA mlockall successful Regards, P.
Re: rpc_address: 0.0.0.0
Thanks! I've created ticked https://issues.apache.org/jira/browse/CASSANDRA-3811 Regards, P. On Sun, Jan 29, 2012 at 20:00, Frisch, Michael michael.fri...@nuance.com wrote: If the code in the 0.8 branch is reflective of what is actually included in Cassandra 0.8.9 (here: http://svn.apache.org/repos/asf/cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/hadoop/ColumnFamilyInputFormat.java) then the problem is that line 202 is doing an == comparison on strings. The correct way to compare would be endpoint_address.equals(0.0.0.0) instead. - Mike
rpc_address: 0.0.0.0
Hi, #using cassandra 0.8.9 I used to have rpc_address set to 0.0.0.0 to bind cassandra to all interfaces. After upgrading out Hadoop jobs to cassandra 0.8.9 (from 0.8.7) there are lots of these messages, and the jobs fails. 12/01/26 11:15:21 DEBUG hadoop.ColumnFamilyInputFormat: failed connect to endpoint 0.0.0.0 java.io.IOException: unable to connect to server at org.apache.cassandra.hadoop.ConfigHelper.createConnection(ConfigHelper.java:389) at org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSubSplits(ColumnFamilyInputFormat.java:224) at org.apache.cassandra.hadoop.ColumnFamilyInputFormat.access$200(ColumnFamilyInputFormat.java:73) at org.apache.cassandra.hadoop.ColumnFamilyInputFormat$SplitCallable.call(ColumnFamilyInputFormat.java:193) at org.apache.cassandra.hadoop.ColumnFamilyInputFormat$SplitCallable.call(ColumnFamilyInputFormat.java:178) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused at org.apache.thrift.transport.TSocket.open(TSocket.java:183) at org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81) at org.apache.cassandra.hadoop.ConfigHelper.createConnection(ConfigHelper.java:385) ... 9 more Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:211) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at org.apache.thrift.transport.TSocket.open(TSocket.java:178) ... 11 more ... Caused by: java.util.concurrent.ExecutionException: java.io.IOException: failed connecting to all endpoints 10.0.18.129,10.0.18.99,10.0.18.98 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSplits(ColumnFamilyInputFormat.java:156) ... 19 more Caused by: java.io.IOException: failed connecting to all endpoints 10.0.18.129,10.0.18.99,10.0.18.98 at org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSubSplits(ColumnFamilyInputFormat.java:241) at org.apache.cassandra.hadoop.ColumnFamilyInputFormat.access$200(ColumnFamilyInputFormat.java:73) at org.apache.cassandra.hadoop.ColumnFamilyInputFormat$SplitCallable.call(ColumnFamilyInputFormat.java:193) at org.apache.cassandra.hadoop.ColumnFamilyInputFormat$SplitCallable.call(ColumnFamilyInputFormat.java:178) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) describe_ring returns: endpoints: 10.0.18.129,10.0.18.99,10.0.18.98 rpc_endpoints: 0.0.0.0,0.0.0.0,0.0.0.0 I've found CASSANDRA-3214 but there is ... then fall back to the gossip endpoint if we don't get what we want. But the job just fails. The job is run from outside of the cluster, which is valid. When I run the job from one of the nodes, there is of course cassandra on 0.0.0.0 and the job runs. Where can be the problem? Regards, P.
Re: Cassandra not suitable?
Thank you Jeremy, I've already changed the max.*.failures to 20, it help jobs to finish but doesn't solve the source of the timeouts. I'll try the other tips. Regards, Patrik On Wed, Dec 7, 2011 at 17:29, Jeremy Hanna jeremy.hanna1...@gmail.com wrote: If you're getting lots of timeout exceptions with mapreduce, you might take a look at http://wiki.apache.org/cassandra/HadoopSupport#Troubleshooting We saw that and tweaked a variety of things - all of which are listed there. Ultimately, we also boosted hadoop's tolerance for them as well and it was just fine - so that it could retry more. A coworker had the same experience running hadoop over elastic search - having to up that tolerance. An example configuration for modifying that is shown in the link above. Hopefully that will help for your mapreduce jobs at least. We've had good luck with MR/Pig over Cassandra, but it's after some lessons learned wrt configuration of both Cassandra and Hadoop. On Dec 6, 2011, at 3:50 AM, Patrik Modesto wrote: Hi, I'm quite desperate about Cassandra's performance in our production cluster. We have 8 real-HW nodes, 32core CPU, 32GB memory, 4 disks in raid10, cassandra 0.8.8, RF=3 and Hadoop. We four keyspaces, one is the large one, it has 2 CFs, one is kind of index, the other holds data. There are about 7milinon rows, mean row size is 7kB. We run several mapreduce tasks, most of them just reads from cassandra and writes to hdfs, but one fetch rows from cassnadra, compute something and write it back, for each row we compute three new json values, about 1kB each (they get overwritten next round). We got lots and lots of Timeout exceptions, LiveSSTablesCount over 100. Reapir doesn't finish even in 24hours, reading from the other keyspaces timeouts as well. We set compaction_throughput_mb_per_sec: 0 but it didn't help. Did we choose wrong DB for our usecase? Regards, Patrik This is from one node: INFO 10:28:40,035 Pool Name Active Pending Blocked INFO 10:28:40,036 ReadStage 96 695 0 INFO 10:28:40,037 RequestResponseStage 0 0 0 INFO 10:28:40,037 ReadRepairStage 0 0 0 INFO 10:28:40,037 MutationStage 1 1 0 INFO 10:28:40,038 ReplicateOnWriteStage 0 0 0 INFO 10:28:40,038 GossipStage 0 0 0 INFO 10:28:40,038 AntiEntropyStage 0 0 0 INFO 10:28:40,039 MigrationStage 0 0 0 INFO 10:28:40,039 StreamStage 0 0 0 INFO 10:28:40,040 MemtablePostFlusher 0 0 0 INFO 10:28:40,040 FlushWriter 0 0 0 INFO 10:28:40,040 MiscStage 0 0 0 INFO 10:28:40,041 FlushSorter 0 0 0 INFO 10:28:40,041 InternalResponseStage 0 0 0 INFO 10:28:40,041 HintedHandoff 1 5 0 INFO 10:28:40,042 CompactionManager n/a 27 INFO 10:28:40,042 MessagingService n/a 0,16559 And here is the nodetool ring output: 10.2.54.91 NG RAC1 Up Normal 118.04 GB 12.50% 0 10.2.54.92 NG RAC1 Up Normal 102.74 GB 12.50% 21267647932558653966460912964485513216 10.2.54.93 NG RAC1 Up Normal 76.95 GB 12.50% 42535295865117307932921825928971026432 10.2.54.94 NG RAC1 Up Normal 56.97 GB 12.50% 63802943797675961899382738893456539648 10.2.54.95 NG RAC1 Up Normal 75.55 GB 12.50% 85070591730234615865843651857942052864 10.2.54.96 NG RAC1 Up Normal 102.57 GB 12.50% 106338239662793269832304564822427566080 10.2.54.97 NG RAC1 Up Normal 68.03 GB 12.50% 127605887595351923798765477786913079296 10.2.54.98 NG RAC1 Up Normal 194.6 GB 12.50% 148873535527910577765226390751398592512
Re: Cassandra not suitable?
Hi Jake, I see the timeouts in mappers as well as at random-access backend daemons (for web services). There are now 10 mappers, 2 reducers on each node. There is one big 4-disk raid10 array on each node on which there is cassandra together with HDFS. We store just few GB of files on HDFS, otherwise we don't use it. Regards, P. On Wed, Dec 7, 2011 at 15:33, Jake Luciani jak...@gmail.com wrote: Where do you see the timeout exceptions? in the mappers? How many mappers reducers slots are you using? What does your disk setup look like? do you have HDFS on same disk as cassandra data dir? -Jake On Tue, Dec 6, 2011 at 4:50 AM, Patrik Modesto patrik.mode...@gmail.com wrote: Hi, I'm quite desperate about Cassandra's performance in our production cluster. We have 8 real-HW nodes, 32core CPU, 32GB memory, 4 disks in raid10, cassandra 0.8.8, RF=3 and Hadoop. We four keyspaces, one is the large one, it has 2 CFs, one is kind of index, the other holds data. There are about 7milinon rows, mean row size is 7kB. We run several mapreduce tasks, most of them just reads from cassandra and writes to hdfs, but one fetch rows from cassnadra, compute something and write it back, for each row we compute three new json values, about 1kB each (they get overwritten next round). We got lots and lots of Timeout exceptions, LiveSSTablesCount over 100. Reapir doesn't finish even in 24hours, reading from the other keyspaces timeouts as well. We set compaction_throughput_mb_per_sec: 0 but it didn't help. Did we choose wrong DB for our usecase? Regards, Patrik This is from one node: INFO 10:28:40,035 Pool Name Active Pending Blocked INFO 10:28:40,036 ReadStage 96 695 0 INFO 10:28:40,037 RequestResponseStage 0 0 0 INFO 10:28:40,037 ReadRepairStage 0 0 0 INFO 10:28:40,037 MutationStage 1 1 0 INFO 10:28:40,038 ReplicateOnWriteStage 0 0 0 INFO 10:28:40,038 GossipStage 0 0 0 INFO 10:28:40,038 AntiEntropyStage 0 0 0 INFO 10:28:40,039 MigrationStage 0 0 0 INFO 10:28:40,039 StreamStage 0 0 0 INFO 10:28:40,040 MemtablePostFlusher 0 0 0 INFO 10:28:40,040 FlushWriter 0 0 0 INFO 10:28:40,040 MiscStage 0 0 0 INFO 10:28:40,041 FlushSorter 0 0 0 INFO 10:28:40,041 InternalResponseStage 0 0 0 INFO 10:28:40,041 HintedHandoff 1 5 0 INFO 10:28:40,042 CompactionManager n/a 27 INFO 10:28:40,042 MessagingService n/a 0,16559 And here is the nodetool ring output: 10.2.54.91 NG RAC1 Up Normal 118.04 GB 12.50% 0 10.2.54.92 NG RAC1 Up Normal 102.74 GB 12.50% 21267647932558653966460912964485513216 10.2.54.93 NG RAC1 Up Normal 76.95 GB 12.50% 42535295865117307932921825928971026432 10.2.54.94 NG RAC1 Up Normal 56.97 GB 12.50% 63802943797675961899382738893456539648 10.2.54.95 NG RAC1 Up Normal 75.55 GB 12.50% 85070591730234615865843651857942052864 10.2.54.96 NG RAC1 Up Normal 102.57 GB 12.50% 106338239662793269832304564822427566080 10.2.54.97 NG RAC1 Up Normal 68.03 GB 12.50% 127605887595351923798765477786913079296 10.2.54.98 NG RAC1 Up Normal 194.6 GB 12.50% 148873535527910577765226390751398592512 -- http://twitter.com/tjake
Cassandra not suitable?
Hi, I'm quite desperate about Cassandra's performance in our production cluster. We have 8 real-HW nodes, 32core CPU, 32GB memory, 4 disks in raid10, cassandra 0.8.8, RF=3 and Hadoop. We four keyspaces, one is the large one, it has 2 CFs, one is kind of index, the other holds data. There are about 7milinon rows, mean row size is 7kB. We run several mapreduce tasks, most of them just reads from cassandra and writes to hdfs, but one fetch rows from cassnadra, compute something and write it back, for each row we compute three new json values, about 1kB each (they get overwritten next round). We got lots and lots of Timeout exceptions, LiveSSTablesCount over 100. Reapir doesn't finish even in 24hours, reading from the other keyspaces timeouts as well. We set compaction_throughput_mb_per_sec: 0 but it didn't help. Did we choose wrong DB for our usecase? Regards, Patrik This is from one node: INFO 10:28:40,035 Pool NameActive Pending Blocked INFO 10:28:40,036 ReadStage96 695 0 INFO 10:28:40,037 RequestResponseStage 0 0 0 INFO 10:28:40,037 ReadRepairStage 0 0 0 INFO 10:28:40,037 MutationStage 1 1 0 INFO 10:28:40,038 ReplicateOnWriteStage 0 0 0 INFO 10:28:40,038 GossipStage 0 0 0 INFO 10:28:40,038 AntiEntropyStage 0 0 0 INFO 10:28:40,039 MigrationStage0 0 0 INFO 10:28:40,039 StreamStage 0 0 0 INFO 10:28:40,040 MemtablePostFlusher 0 0 0 INFO 10:28:40,040 FlushWriter 0 0 0 INFO 10:28:40,040 MiscStage 0 0 0 INFO 10:28:40,041 FlushSorter 0 0 0 INFO 10:28:40,041 InternalResponseStage 0 0 0 INFO 10:28:40,041 HintedHandoff 1 5 0 INFO 10:28:40,042 CompactionManager n/a27 INFO 10:28:40,042 MessagingServicen/a 0,16559 And here is the nodetool ring output: 10.2.54.91 NG RAC1Up Normal 118.04 GB 12.50% 0 10.2.54.92 NG RAC1Up Normal 102.74 GB 12.50% 21267647932558653966460912964485513216 10.2.54.93 NG RAC1Up Normal 76.95 GB 12.50% 42535295865117307932921825928971026432 10.2.54.94 NG RAC1Up Normal 56.97 GB 12.50% 63802943797675961899382738893456539648 10.2.54.95 NG RAC1Up Normal 75.55 GB 12.50% 85070591730234615865843651857942052864 10.2.54.96 NG RAC1Up Normal 102.57 GB 12.50% 106338239662793269832304564822427566080 10.2.54.97 NG RAC1Up Normal 68.03 GB 12.50% 127605887595351923798765477786913079296 10.2.54.98 NG RAC1Up Normal 194.6 GB 12.50% 148873535527910577765226390751398592512
Cassandra 0.8.8
Hi, It's been almost 2 months since the release of the 0.8.7 version and there are quite some changes in 0.8.8, so I'd like to ask is there a release date? Regards, Patrik
Re: Cassandra 0.8.8
I've missed the vote email, thanks! Patrik On Fri, Dec 2, 2011 at 11:03, Sylvain Lebresne sylv...@datastax.com wrote: The release of 0.8.8 is under vote as of now. So it will be released as soon as it has been tested and voted on, which should hopefully happen soon (it could be today, it could be early next week). -- Sylvain On Fri, Dec 2, 2011 at 10:49 AM, Patrik Modesto patrik.mode...@gmail.com wrote: Hi, It's been almost 2 months since the release of the 0.8.7 version and there are quite some changes in 0.8.8, so I'd like to ask is there a release date? Regards, Patrik
Re: timeout while doing repair
Thanks for the reply. I know I can configure longer timeout but in our use case, reply longer than 1second is unacceptable. What I don't understand is why I get timeout while reading differrent keyspace than the repair is working on. I get timeouts even doing compaction. Besides usual access we do lots of reads and writes using Hadoop mapreduce jobs so we need to do compact/repair quite often. Regards Patrik Dne 24.11.2011 15:00 Jahangir Mohammed md.jahangi...@gmail.com napsal(a): Do you use any client which gives you this timeout ? If you don't specify any timeout from client, look at rpc_timeout_in_ms. Increase it and see if you still suffer this. Repair is a costly process. Thanks, Jahangir Mohammed. On Thu, Nov 24, 2011 at 2:45 AM, Patrik Modesto patrik.mode...@gmail.comwrote: Hi, I have a test cluster of 4 nodes running Debian and Cassandra 0.8.7, there are 3 keyspaces, all with RF=3, a node has load around 40GB. When I run nodetool repair after a while all thrift clients that read with CL.QUORUM get TimeoutException and even some that use just CL.ONE. I've tried to run repair on just one keyspace and read from other keyspace, but I still get the TimeoutException. I tried to tune compaction_throughput_mb_per_sec and concurrent_compactors but without success. The same problem is happening on our production cluster of 8 nodes (same setup). Where may be the problem? Regards, Patrik
Re: timeout while doing repair
We have our own servers, it is 16 core CPU, 32GB ram,8 1TB disks. I didn't check tpstats, just iotop where cassandra used all the io capacity when compacting/repairing. I had to completely clean the test cluster, but I'll check tpstats in the production. What should I look for? Regards, Patrik Dne 24.11.2011 19:13 Jahangir Mohammed md.jahangi...@gmail.com napsal(a): What I know is timeout is because of increased load on node due to repair. Hardware? EC2? Did you check tpstats? On Thu, Nov 24, 2011 at 11:42 AM, Patrik Modesto patrik.mode...@gmail.com wrote: Thanks for the reply. I know I can configure longer timeout but in our use case, reply longer than 1second is unacceptable. What I don't understand is why I get timeout while reading differrent keyspace than the repair is working on. I get timeouts even doing compaction. Besides usual access we do lots of reads and writes using Hadoop mapreduce jobs so we need to do compact/repair quite often. Regards Patrik Dne 24.11.2011 15:00 Jahangir Mohammed md.jahangi...@gmail.com napsal(a): Do you use any client which gives you this timeout ? If you don't specify any timeout from client, look at rpc_timeout_in_ms. Increase it and see if you still suffer this. Repair is a costly process. Thanks, Jahangir Mohammed. On Thu, Nov 24, 2011 at 2:45 AM, Patrik Modesto patrik.mode...@gmail.com wrote: Hi, I have a test cluster of 4 nodes running Debian and Cassandra 0.8.7, there are 3 keyspaces, all with RF=3, a node has load around 40GB. When I run nodetool repair after a while all thrift clients that read with CL.QUORUM get TimeoutException and even some that use just CL.ONE. I've tried to run repair on just one keyspace and read from other keyspace, but I still get the TimeoutException. I tried to tune compaction_throughput_mb_per_sec and concurrent_compactors but without success. The same problem is happening on our production cluster of 8 nodes (same setup). Where may be the problem? Regards, Patrik
timeout while doing repair
Hi, I have a test cluster of 4 nodes running Debian and Cassandra 0.8.7, there are 3 keyspaces, all with RF=3, a node has load around 40GB. When I run nodetool repair after a while all thrift clients that read with CL.QUORUM get TimeoutException and even some that use just CL.ONE. I've tried to run repair on just one keyspace and read from other keyspace, but I still get the TimeoutException. I tried to tune compaction_throughput_mb_per_sec and concurrent_compactors but without success. The same problem is happening on our production cluster of 8 nodes (same setup). Where may be the problem? Regards, Patrik
decommissioned node still in LoadMap in JMX Management Console
Hi, on our production cluster of 8 nodes which is running cassandra 0.8.7 we still see in the MBean org.apache.cassandra.db:type=StorageService.LoadMap in JMX Management console the 9th node we added for testing for a short time. After the testing we decommissioned the 9th node and has been reinstalled for another use. The node is not reported by `nodetool ring` not is it in org.apache.cassandra.db:type=StorageService.LiveNodes MBean. Is it a feature or a bug? 10.2.54.85 is the 9th node. # nodetool -h 0 ring Address DC RackStatus State Load OwnsToken 148873535527910577765226390751398592512 10.2.54.81 NG RAC1Up Normal 95.49 GB 12.50% 0 192.168.54.81 GO RAC1Up Normal 88.38 GB 12.50% 21267647932558653966460912964485513216 10.2.54.82 NG RAC1Up Normal 77.83 GB 12.50% 42535295865117307932921825928971026432 192.168.54.82 GO RAC1Up Normal 69.43 GB 12.50% 63802943797675961899382738893456539648 10.2.54.83 NG RAC1Up Normal 76.18 GB 12.50% 85070591730234615865843651857942052864 192.168.54.83 GO RAC1Up Normal 94.91 GB 12.50% 106338239662793269832304564822427566080 10.2.54.84 NG RAC1Up Normal 68.04 GB 12.50% 127605887595351923798765477786913079296 192.168.54.84 GO RAC1Up Normal 68.98 GB 12.50% 148873535527910577765226390751398592512 org.apache.cassandra.db:type=StorageService.LoadMap: 192.168.54.81 java.lang.String89.33 GBjava.lang.String 192.168.54.83 java.lang.String96.02 GBjava.lang.String 192.168.54.82 java.lang.String69.99 GBjava.lang.String 192.168.54.84 java.lang.String70.87 GBjava.lang.String 10.2.54.85 java.lang.String122.2 GBjava.lang.String 10.2.54.84 java.lang.String69.59 GBjava.lang.String 10.2.54.81 java.lang.String95.62 GBjava.lang.String 10.2.54.82 java.lang.String79.37 GBjava.lang.String 10.2.54.83 java.lang.String76.61 GBjava.lang.String Regards, Patrik
after upgrade to 0.8.7: InvalidRequestException(why:Column timestamp is required)
Hi, I've a 4 node cluster of cassandra 0.8.7 (upgraded just recently from 0.7.8). Upgrade went smothly, no problem with the data. Problem are my MapReduce tasks. The all report: java.io.IOException: InvalidRequestException(why:Column timestamp is required) at org.apache.cassandra.hadoop.ColumnFamilyRecordWriter$RangeClient.run(ColumnFamilyRecordWriter.java:310) Caused by: InvalidRequestException(why:Column timestamp is required) at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:19479) at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:1035) at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:1009) at org.apache.cassandra.hadoop.ColumnFamilyRecordWriter$RangeClient.run(ColumnFamilyRecordWriter.java:302) or java.io.IOException: InvalidRequestException(why:Deletion timestamp is not optional for non commutative column family PageData) at org.apache.cassandra.hadoop.ColumnFamilyRecordWriter$RangeClient.run(ColumnFamilyRecordWriter.java:310) Caused by: InvalidRequestException(why:Deletion timestamp is not optional for non commutative column family PageData) at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:19479) at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:1035) at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:1009) at org.apache.cassandra.hadoop.ColumnFamilyRecordWriter$RangeClient.run(ColumnFamilyRecordWriter.java:302) I only changed dependency in pom.xml on cassaandra-all 0.8.7 and replaced .avro. with .thift. where needed. No compilation error. The tasks run well with cassandra 0.7.x here on test and on the production cluster. I see there is a change in cassandra.thrift but it's I'm not using it in the java application. Can anybody help? Regards, Patrik
[SOLVED] Re: after upgrade to 0.8.7: InvalidRequestException(why:Column timestamp is required)
As usually happen, I've found the problem just after I sent the question. I have to use setters for setting values to thrift.* classes. So instead of: Deletion d = new Deletion(); d.timestamp = 1; use: Deletion d = new Deletion(); d.setTimestamp(1); etc. Regards, Patrik
Re: RF=1 w/ hadoop jobs
On Mon, Sep 5, 2011 at 09:39, Mick Semb Wever m...@apache.org wrote: I've entered a jira issue covering this request. https://issues.apache.org/jira/browse/CASSANDRA-3136 Would you mind attaching your patch to the issue. (No review of it will happen anywhere else.) I see Jonathan didn't change his mind, as the ticket was resolved won't fix. So that's it. I'm not going to attach the patch until another core Cassandra developer is interested in the use-cases mentioned there. I'm not sure about 0.8.x and 0.7.9 (to be released today with your patch) but 0.7.8 will fail even with RF1 when there is Hadoop TaskTracer without local Cassandra. So increasing RF is not a solution. Regards, Patrik
Re: RF=1 w/ hadoop jobs
Hi, On Thu, Sep 1, 2011 at 12:36, Mck m...@apache.org wrote: It's available here: http://pastebin.com/hhrr8m9P (for version 0.7.8) I'm interested in this patch and see it's usefulness but no one will act until you attach it to an issue. (I think a new issue is appropriate here). I'm glad someone is interestet in my patch usefull. As Jonathan already explained himself: ignoring unavailable ranges is a misfeature, imo I'm thinking opening a new ticket without support from more users is useless ATM. Please test the patch and if you like it, than there is time for ticket. Regards, P.
Re: RF=1 w/ hadoop jobs
On Fri, Sep 2, 2011 at 08:54, Mick Semb Wever m...@apache.org wrote: Patrik: is it possible to describe the use-case you have here? Sure. We use Cassandra as a storage for web-pages, we store the HTML, all URLs that has the same HTML data and some computed data. We run Hadoop MR jobs to compute lexical and thematical data for each page and for exporting the data to a binary files for later use. URL gets to a Cassandra on user request (a pageview) so if we delete an URL, it gets back quickly if the page is active. Because of that and because there is lots of data, we have the keyspace set to RF=1. We can drop the whole keyspace and it will regenerate quickly and would contain only fresh data, so we don't care about lossing a node. But Hadoop does care, well to be specific the Cassnadra ColumnInputFormat and ColumnRecortReader are the problem parts. If I stop one Cassandra node all MR jobs that read/write Cassandra fail. In our case, it doesn't matter, we can skip the range of URLs. The MR jobs run in a tight loop, so when the node is back with it's data, we use them. It's not only about some HW crash but it makes maintenance quite difficult. To stop a Cassandra node, you have to stop tasktracker there too which is unfortunate as there are another MR jobs that don't need Cassandra and can happily run. Regards, P.
Re: RF=1
Is there really no interest in the patch? P. On Thu, Aug 18, 2011 at 08:54, Patrik Modesto patrik.mode...@gmail.com wrote: On Wed, Aug 17, 2011 at 17:08, Jonathan Ellis jbel...@gmail.com wrote: See https://issues.apache.org/jira/browse/CASSANDRA-2388 Ok, thanks for the JIRA ticker. I've found that very same problem during my work on ignoring unavailabla ranges. But there is the another problem with Hadoop-Cassandra, if there is no node available for a range of keys, it fails on RuntimeError. For example having a keyspace with RF=1 and a node is down all MapReduce tasks fail. I've reworked my previous patch, that was addressing this issue and now there are ConfigHelper methods for enable/disable ignoring unavailable ranges. It's available here: http://pastebin.com/hhrr8m9P (for version 0.7.8) I hope it can be applied. Regards, Patrik
Re: RF=1
Hi, while I was investigating this issue, I've found that hadoop+cassandra don't work if you stop even just one node in the cluster. It doesn't depend on RF. ColumnFamilyRecordReader gets list of nodes (acording the RF) but chooses just the local host and if there is no cassandra running localy it throws RuntimeError exception. Which in turn marks the MapReduce task as failed. I've created a patch that makes ColumnFamilyRecordReader to try the local node and if it fails tries the other nodes in it's list. The patch is here http://pastebin.com/0RdQ0HMx I think attachements are not allowed on this ML. Please test it and apply. It's for 0.7.8 version. Regards, P. On Wed, Aug 3, 2011 at 13:59, aaron morton aa...@thelastpickle.com wrote: If you want to take a look o.a.c.hadoop.ColumnFamilyRecordReader.getSplits() is the function that gets the splits. Cheers - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 3 Aug 2011, at 16:18, Patrik Modesto wrote: On Tue, Aug 2, 2011 at 23:10, Jeremiah Jordan jeremiah.jor...@morningstar.com wrote: If you have RF=1, taking one node down is going to cause 25% of your data to be unavailable. If you want to tolerate a machines going down you need to have at least RF=2, if you want to use quorum and have a machine go down, you need at least RF=3. I know I can have RF 1 but I have limited resources and I don't care lossing 25% of the data. RF 1 basicaly means if a node goes down I have the data elsewhere, but what I need is if node goes down just ignore its range. I can handle it in my applications using thrift, but the hadoop-mapreduce can't handle it. It just fails with Exception in thread main java.io.IOException: Could not get input splits. Is there a way to say ignore this range to hadoop? Regards, P.
Re: RF=1
And one more patch: http://pastebin.com/zfNPjtQz This one handles a case where there are no nodes available for a slice. For example where the is a keyspace with RF=1 and a node is shut down. Its range of keys gets ignored. Regards, P. On Wed, Aug 17, 2011 at 13:28, Patrik Modesto patrik.mode...@gmail.com wrote: Hi, while I was investigating this issue, I've found that hadoop+cassandra don't work if you stop even just one node in the cluster. It doesn't depend on RF. ColumnFamilyRecordReader gets list of nodes (acording the RF) but chooses just the local host and if there is no cassandra running localy it throws RuntimeError exception. Which in turn marks the MapReduce task as failed. I've created a patch that makes ColumnFamilyRecordReader to try the local node and if it fails tries the other nodes in it's list. The patch is here http://pastebin.com/0RdQ0HMx I think attachements are not allowed on this ML. Please test it and apply. It's for 0.7.8 version. Regards, P. On Wed, Aug 3, 2011 at 13:59, aaron morton aa...@thelastpickle.com wrote: If you want to take a look o.a.c.hadoop.ColumnFamilyRecordReader.getSplits() is the function that gets the splits. Cheers - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 3 Aug 2011, at 16:18, Patrik Modesto wrote: On Tue, Aug 2, 2011 at 23:10, Jeremiah Jordan jeremiah.jor...@morningstar.com wrote: If you have RF=1, taking one node down is going to cause 25% of your data to be unavailable. If you want to tolerate a machines going down you need to have at least RF=2, if you want to use quorum and have a machine go down, you need at least RF=3. I know I can have RF 1 but I have limited resources and I don't care lossing 25% of the data. RF 1 basicaly means if a node goes down I have the data elsewhere, but what I need is if node goes down just ignore its range. I can handle it in my applications using thrift, but the hadoop-mapreduce can't handle it. It just fails with Exception in thread main java.io.IOException: Could not get input splits. Is there a way to say ignore this range to hadoop? Regards, P.
Internal error processing get_range_slices
Hi, on our dev cluster of 4 cassandra nodes 0.7.8 I'm suddenly getting: ERROR 13:40:50,848 Internal error processing get_range_slices java.lang.OutOfMemoryError: Java heap space at java.util.ArrayList.init(ArrayList.java:112) at org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:480) at org.apache.cassandra.thrift.CassandraServer.get_range_slices(CassandraServer.java:486) at org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.process(Cassandra.java:2868) at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2555) at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) I run get_range_slices() on all keys with 3 columns named in the Thrift request. columnParent.column_family = CATEGORIES_CATEGORY; keyRange.start_key = ; keyRange.end_key = ; keyRange.__isset.start_key = true; keyRange.__isset.end_key = true; keyRange.count = std::numeric_limitsint32_t::max(); slicePredicate.column_names.push_back(CATEGORIES_CATEGORY_ID); slicePredicate.column_names.push_back(CATEGORIES_CATEGORY_NAME); slicePredicate.column_names.push_back(CATEGORIES_CATEGORY_PARENT); slicePredicate.__isset.column_names = true; std::vectoroacassandra::KeySlice rangeSlices; cassandraWrapper-get_range_slices(rangeSlices, columnParent, slicePredicate, keyRange, oacassandra::ConsistencyLevel::QUORUM); There are just 102 rows each with 6 columns. Maximum rowsize is 3 379 391B, mean rowsize is 407 756B. Suddenly Cassandra needs 9GB of heap-space to fulfill this get_range_slices. There is no cache enabled. What could be the problem here? Regards, Patrik PS: while reading the email before I'd send it, I've noticed the keyRange.count =... is it possible that Cassandra is preallocating some internal data acording the KeyRange.count parameter?
Re: Internal error processing get_range_slices
On Mon, Aug 15, 2011 at 15:09, Jonathan Ellis jbel...@gmail.com wrote: On Mon, Aug 15, 2011 at 7:13 AM, Patrik Modesto patrik.mode...@gmail.com wrote: PS: while reading the email before I'd send it, I've noticed the keyRange.count =... is it possible that Cassandra is preallocating some internal data acording the KeyRange.count parameter? That's exactly what it does. Ok. But is this pre-alocating really needed? Can't cassandra deduce that it doesn't need that much space? Regards, P.
Re: Compacting large row
On Fri, Aug 5, 2011 at 15:02, Jonathan Ellis jbel...@gmail.com wrote: It's logging the actual key, not the md5. It's just converting the key bytes to hex first to make sure it's printable. Great! I'm using MD5 as a key so I didn't notice that. Thanks, P.
Compacting large row
Hi, I got several Compacting large row ... messages while runinng a compaction. Is there any way to get the row to see it? I know it's MD5 but it's MD5 of a key and I need the key to get to the row. The sstable is really huge so dump is not an option. Using Cassandra 0.7.8 Regards, P.
A way to break the cluster
Hi, I'm did break my test cluster again. It's really strange. I use cassandra 0.7.3. This is what I did: - install node1 - install node2, auto_bootstrap: true - install node3, auto_bootstrap: true - created a keyspace with RF 1, populate with data - create a keyspace with RF 3, populate with data - nodetool -h node1 loadbalance - after an hour of doing nothing I killed the nodetool, nodetool ring shows node1 as Up/Leaving - restarted node1 - nodetool ring shows node1 as Up/Normal From now on nodetool loadbalance or decommission failt with: nodetool -h skhd1.dev move 0 Exception in thread main java.lang.IllegalStateException: replication factor (3) exceeds number of endpoints (2) at org.apache.cassandra.locator.SimpleStrategy.calculateNaturalEndpoints(SimpleStrategy.java:60) at org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:929) at org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:895) at org.apache.cassandra.service.StorageService.startLeaving(StorageService.java:1595) at org.apache.cassandra.service.StorageService.move(StorageService.java:1733) at org.apache.cassandra.service.StorageService.move(StorageService.java:1708) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427) at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360) at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788) at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) at sun.rmi.transport.Transport$1.run(Transport.java:159) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:155) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) nodetool repair pass, but nothing changed. Regards, Patrik
Re: problem with bootstrap
On Fri, Mar 11, 2011 at 22:31, Aaron Morton aa...@thelastpickle.com wrote:
The assertion is interesting. Can you reproduce it with logging at debug and
post the results? Could you try to reproduce it with a clean cluster?
It was on a clean cluster last time. Anyway I started clean cluster
again, repeated the same steps with DEBUG enabled. This is the part of
log when the AssertionError happend:
DEBUG 08:32:50,513 reading RangeSliceCommand{keyspace='Context',
column_family='Url', super_column=null,
predicate=SlicePredicate(slice_range:SliceRange(start:80 01 00 01 00
00 00 10 67 65 74 5F 72 61 6E 67 65 5F 73 6C 69 63 65 73 00 00 00 07
0C 00 01 0B 00 03 00 00 00 03 55 72 6C 00 0C 00 02 0C 00 02 0B 00 01
00 00 00 00, finish:80 01 00 01 00 00 00 10 67 65 74 5F 72 61 6E 67 65
5F 73 6C 69 63 65 73 00 00 00 07 0C 00 01 0B 00 03 00 00 00 03 55 72
6C 00 0C 00 02 0C 00 02 0B 00 01 00 00 00 00 0B 00 02 00 00 00 00,
reversed:false, count:2147483647)),
range=(4839275280018186837186594882659755,53884050462475230985987955197486389848],
max_keys=100} from /10.0.18.99
DEBUG 08:32:50,513 reading RangeSliceCommand{keyspace='Context',
column_family='Url', super_column=null,
predicate=SlicePredicate(slice_range:SliceRange(start:80 01 00 01 00
00 00 10 67 65 74 5F 72 61 6E 67 65 5F 73 6C 69 63 65 73 00 00 00 07
0C 00 01 0B 00 03 00 00 00 03 55 72 6C 00 0C 00 02 0C 00 02 0B 00 01
00 00 00 00, finish:80 01 00 01 00 00 00 10 67 65 74 5F 72 61 6E 67 65
5F 73 6C 69 63 65 73 00 00 00 07 0C 00 01 0B 00 03 00 00 00 03 55 72
6C 00 0C 00 02 0C 00 02 0B 00 01 00 00 00 00 0B 00 02 00 00 00 00,
reversed:false, count:2147483647)),
range=(4839275280018186837186594882659755,53884050462475230985987955197486389848],
max_keys=100} from /10.0.18.129
DEBUG 08:32:50,515 Processing response on a callback from 159@/10.0.18.99
DEBUG 08:32:50,515 Processing response on a callback from 160@/10.0.18.129
ERROR 08:32:50,518 Fatal exception in thread
Thread[RequestResponseStage:3,5,main]
java.lang.AssertionError
at
org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127)
at
org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
It happens on every node I run the request on, the others report
nothing on that particular request.
HTH,
P.
Re: problem with bootstrap
Unfortunately I can't provide the info, I deleted it. It was in wery strange state. I started with new cluster today, 2 nodes, each with auto_bootstrap:true. I can create a keyspace with RF=3, but I can't insert any data in it. It didn't happen with the old cluster which made me think. How could I insert data in the old cluster in keyspace with RF=3 but with just 2 nodes? I found out that the cluster had 3 nodes for short time in the past. We had to remove/return one node but that was enough for the cluster to accept writes to keyspace with RF=3 even with just 2 nodes. So I tried to recreate the cluster state: I have 4 clean server, cassndra 0.7.3, auto_bootstrap:true 1) setup run node1 - success 2) create keyspace Context with rf=3 and create CF Url via cassandra-cli - success 3) list Url - Internal error processing get_range_slicesl node1: ERROR 09:46:28,725 Internal error processing get_range_slices java.lang.IllegalStateException: replication factor (3) exceeds number of endpoints (1) 4) setup run node2 - success 5) list Url on node1 - Internal error processing get_range_slicesl node1: ERROR 09:46:28,725 Internal error processing get_range_slices java.lang.IllegalStateException: replication factor (3) exceeds number of endpoints (1) 6) list Url on node2 - Internal error processing get_range_slicesl node2: ERROR 09:50:54,231 Internal error processing get_range_slices java.lang.IllegalStateException: replication factor (3) exceeds number of endpoints (2) 7) insert on node1 - Internal error processing insert node1: ERROR 09:53:11,669 Internal error processing insert java.lang.IllegalStateException: replication factor (3) exceeds number of endpoints (2) 8) insert on node2 - Internal error processing insert node2: ERROR 09:53:54,833 Internal error processing insert java.lang.IllegalStateException: replication factor (3) exceeds number of endpoints (2) 9) setup run node3 - success 10) list Url on node1 - success 11) insert in Url on node1 - success 12) stop cassandra on node3 - success 13) list insert on node12 - success 14) loadbalance on node1 - Exception in thread main java.lang.IllegalStateException: replication factor (3) exceeds number of endpoints (2) 15) setup run node4 - success 16) list Url on node4 - success BUT node4: ERROR 10:05:38,452 Fatal exception in thread Thread[RequestResponseStage:1,5,main] java.lang.AssertionError at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127) at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) ERROR 10:05:38,462 Fatal exception in thread Thread[RequestResponseStage:17,5,main] java.lang.AssertionError at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127) at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 17) loadbalance on node1 - success 18) list Url on node4 - success BUT node4: ERROR 10:09:58,251 Fatal exception in thread Thread[RequestResponseStage:18,5,main] java.lang.AssertionError at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127) at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) ERROR 10:09:58,257 Fatal exception in thread Thread[RequestResponseStage:5,5,main] java.lang.AssertionError at org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:127) at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 19) repair on node4 - after long long wait I killed it, non of the nodes report any error 20) list Url on node1 - success BUT node1: ERROR 10:18:53,715 Fatal
Re: problem with bootstrap
Hi, I'm stil fighting the Exception in thread main java.lang.IllegalStateException: replication factor (3) exceeds number of endpoints (2). When I have a 2-server cluster, create Keyspace with RF 3, I'm able to add (without auto_bootstrap) another node but cluster nodetool commands don't work and fail with the exception above. The new node serve data but I can't do loadbalance, decommission or move on the cluster. Patrik
problem with bootstrap
Hi, I've small test cluster, 2 servers, both running successfully cassandra 0.7.3. I've three keyspaces, two with RF1, one with RF3. Now when I try to bootstrap 3rd server (empty initial_token, auto_bootstrap: true), I get this exception on the new server. INFO 23:13:43,229 Joining: getting bootstrap token INFO 23:13:43,258 New token will be 127097301048222781806986236020167142093 to assume load from /10.0.18.99 INFO 23:13:43,259 switching in a fresh Memtable for LocationInfo at CommitLogContext(file='/mnt/disk8/cassandra/data/CommitLog-1299622332896.log', position=1578072) INFO 23:13:43,259 Enqueuing flush of Memtable-LocationInfo@1526249359(106 bytes, 3 operations) INFO 23:13:43,259 Writing Memtable-LocationInfo@1526249359(106 bytes, 3 operations) INFO 23:13:43,276 Completed flushing /mnt/disk3/cassandra/data/system/LocationInfo-f-2-Data.db (211 bytes) INFO 23:13:43,277 Joining: sleeping 3 ms for pending range setup INFO 23:14:13,277 Bootstrapping java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:160) Caused by: java.lang.IllegalStateException: replication factor (3) exceeds number of endpoints (2) at org.apache.cassandra.locator.SimpleStrategy.calculateNaturalEndpoints(SimpleStrategy.java:60) at org.apache.cassandra.locator.AbstractReplicationStrategy.getRangeAddresses(AbstractReplicationStrategy.java:212) at org.apache.cassandra.dht.BootStrapper.getRangesWithSources(BootStrapper.java:198) at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:83) at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:525) at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:453) at org.apache.cassandra.service.StorageService.initServer(StorageService.java:403) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:194) at org.apache.cassandra.service.AbstractCassandraDaemon.init(AbstractCassandraDaemon.java:217) ... 5 more Cannot load daemon Service exit with a return value of 3 On the other servers I get: ERROR 15:54:24,670 Error in ThreadPoolExecutor java.lang.IllegalStateException: replication factor (3) exceeds number of endpoints (2) at org.apache.cassandra.locator.SimpleStrategy.calculateNaturalEndpoints(SimpleStrategy.java:60) at org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:929) at org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:895) at org.apache.cassandra.service.StorageService.handleStateLeaving(StorageService.java:797) at org.apache.cassandra.service.StorageService.onChange(StorageService.java:651) at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:763) at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:753) at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:670) at org.apache.cassandra.gms.GossipDigestAckVerbHandler.doVerb(GossipDigestAckVerbHandler.java:68) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) ERROR 15:54:24,672 Fatal exception in thread Thread[GossipStage:1,5,main] java.lang.IllegalStateException: replication factor (3) exceeds number of endpoints (2) at org.apache.cassandra.locator.SimpleStrategy.calculateNaturalEndpoints(SimpleStrategy.java:60) at org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:929) at org.apache.cassandra.service.StorageService.calculatePendingRanges(StorageService.java:895) at org.apache.cassandra.service.StorageService.handleStateLeaving(StorageService.java:797) at org.apache.cassandra.service.StorageService.onChange(StorageService.java:651) at org.apache.cassandra.gms.Gossiper.doNotifications(Gossiper.java:763) at org.apache.cassandra.gms.Gossiper.applyApplicationStateLocally(Gossiper.java:753) at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:670) at org.apache.cassandra.gms.GossipDigestAckVerbHandler.doVerb(GossipDigestAckVerbHandler.java:68) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) at
Re: OOM during batch_mutate
On Tue, Feb 8, 2011 at 00:05, Jonathan Ellis jbel...@gmail.com wrote: Sounds like the keyspace was created on the 32GB machine, so it guessed memtable sizes that are too large when run on the 16GB one. Use update column family from the cli to cut the throughput and operations thresholds in half, or to 1/4 to be cautious. That was exactly the problem, thanks a lot Jonathan! Patrik
OOM during batch_mutate
Hi all! I'm running into OOM problem during batch_mutate. I've a test cluster of two servers, 32GB and 16GB RAM, real HW. I've one keyspace and one CF with 1,4mil rows, each 10 columns. A row is around 5k in size. I run Hadoop MR task that reads one column and generates Mutation that updates another column, it writes about 1k string there. Strange thing is I can run this MR three times without any error on a freshly created/filled CF but the 4th time Cassandra on the 16GB server hits OOM. After I run 'nodetool compact' on the 16GB server I can run the MR again without any OOM one or two times. I'm quite clueless on what to change in Cassandra/CF settings. Can anyone help me? Regards, Patrik output.log: [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor30] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor34] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor20] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor10] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor32] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor19] [Unloading class sun.reflect.GeneratedMethodAccessor9] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor33] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor16] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor15] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor23] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor43] WARN 09:28:46,172 Dropped 2 MUTATION messages in the last 5000ms WARN 09:30:10,555 Dropped 7 REQUEST_RESPONSE messages in the last 5000ms INFO 09:30:10,556 Pool NameActive Pending INFO 09:30:10,556 ReadStage 0 0 INFO 09:30:10,556 RequestResponseStage 050 INFO 09:30:10,557 ReadRepair0 0 INFO 09:30:10,557 MutationStage1626 INFO 09:30:10,557 GossipStage 1 8 INFO 09:30:10,558 AntiEntropyStage 0 0 INFO 09:30:10,558 MigrationStage0 0 INFO 09:30:10,558 StreamStage 0 0 ERROR 09:30:10,561 Internal error processing batch_mutate java.lang.NoClassDefFoundError: Could not initialize class org.apache.cassandra.thrift.TimedOutException at org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:417) at org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:385) at org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3036) at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2555) at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:167) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) ERROR 09:30:10,675 Fatal exception in thread Thread[MutationStage:14,5,main] java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2786) at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94) at java.io.DataOutputStream.write(DataOutputStream.java:90) at org.apache.cassandra.utils.FBUtilities.writeByteArray(FBUtilities.java:266) at org.apache.cassandra.db.ColumnSerializer.serialize(ColumnSerializer.java:56) at org.apache.cassandra.db.ColumnSerializer.serialize(ColumnSerializer.java:35) at org.apache.cassandra.db.ColumnFamilySerializer.serializeForSSTable(ColumnFamilySerializer.java:87) at org.apache.cassandra.db.ColumnFamilySerializer.serialize(ColumnFamilySerializer.java:73) at org.apache.cassandra.db.RowMutationSerializer.freezeTheMaps(RowMutation.java:364) at org.apache.cassandra.db.RowMutationSerializer.serialize(RowMutation.java:375) at org.apache.cassandra.db.RowMutationSerializer.serialize(RowMutation.java:353) at org.apache.cassandra.db.RowMutation.getSerializedBuffer(RowMutation.java:274) at org.apache.cassandra.db.RowMutation.apply(RowMutation.java:194) at org.apache.cassandra.service.StorageProxy$1.runMayThrow(StorageProxy.java:202) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) ERROR 09:30:10,675 Internal error processing batch_mutate java.lang.NoClassDefFoundError: Could not initialize class
[0.7.1] Error in ThreadPoolExecutor
Hi, on my two-node test setup I get repeatedly following error: The 10.0.18.129 server log: INFO 14:10:37,707 Node /10.0.18.99 has restarted, now UP again INFO 14:10:37,708 Checking remote schema before delivering hints INFO 14:10:37,708 Sleeping 45506ms to stagger hint delivery INFO 14:10:37,709 Node /10.0.18.99 state jump to normal INFO 14:11:23,215 Started hinted handoff for endpoint /10.0.18.99 ERROR 14:11:23,884 Error in ThreadPoolExecutor java.lang.RuntimeException: java.lang.IllegalArgumentException at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.IllegalArgumentException at java.nio.Buffer.position(Buffer.java:218) at org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:117) at org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:111) at org.apache.cassandra.db.HintedHandOffManager.getTableAndCFNames(HintedHandOffManager.java:237) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:306) at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:88) at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:385) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) ... 3 more ERROR 14:11:23,885 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.lang.IllegalArgumentException at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.IllegalArgumentException at java.nio.Buffer.position(Buffer.java:218) at org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:117) at org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:111) at org.apache.cassandra.db.HintedHandOffManager.getTableAndCFNames(HintedHandOffManager.java:237) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:306) at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:88) at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:385) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) ... 3 more The 10.0.18.99 server log: INFO 14:10:37,691 Binding thrift service to /0.0.0.0:9160 INFO 14:10:37,693 Using TFastFramedTransport with a max frame size of 15728640 bytes. INFO 14:10:37,695 Listening for thrift clients... INFO 14:10:38,337 GC for ParNew: 954 ms, 658827608 reclaimed leaving 966732432 used; max is 4265607168 INFO 14:11:27,142 Started hinted handoff for endpoint /10.0.18.129 ERROR 14:11:27,370 Error in ThreadPoolExecutor java.lang.RuntimeException: java.lang.IllegalArgumentException at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.IllegalArgumentException at java.nio.Buffer.position(Buffer.java:218) at org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:117) at org.apache.cassandra.utils.ByteBufferUtil.string(ByteBufferUtil.java:111) at org.apache.cassandra.db.HintedHandOffManager.getTableAndCFNames(HintedHandOffManager.java:237) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:306) at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:88) at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:385) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) ... 3 more ERROR 14:11:27,371 Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: java.lang.IllegalArgumentException at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by:
Re: OOM during batch_mutate
I forgot to mention I use 0.7.0 stable version. HTH, Patrik
Re: OOM during batch_mutate
Just tried current 0.7.1 from cassandra-0.7 branch and it does the same. OOM after three runs. -Xm* setting is computed by cassandra-env.sh like this: -Xms8022M -Xmx8022M -Xmn2005M What am I doing wrong? Thanks, Patrik On Mon, Feb 7, 2011 at 14:18, Patrik Modesto patrik.mode...@gmail.com wrote: I forgot to mention I use 0.7.0 stable version. HTH, Patrik
[0.7.1] more exceptions: Illegal mode
INFO 15:30:49,647 Compacted to /www/foo/cassandra/data/foo/Url-tmp-f-767-Data.db. 4,199,999,762 to 4,162,579,242 (~99% of original) bytes for 379,179 keys. Time: 137,149ms. ERROR 15:30:49,699 Fatal exception in thread Thread[CompactionExecutor:1,1,main] java.lang.RuntimeException: java.lang.IllegalArgumentException: Illegal mode w must be one of r, rw, rws, or rwd at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.IllegalArgumentException: Illegal mode w must be one of r, rw, rws, or rwd at java.io.RandomAccessFile.init(RandomAccessFile.java:197) at org.apache.cassandra.io.util.BufferedRandomAccessFile.init(BufferedRandomAccessFile.java:116) at org.apache.cassandra.io.sstable.CacheWriter.saveCache(CacheWriter.java:48) at org.apache.cassandra.db.CompactionManager$9.runMayThrow(CompactionManager.java:746) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) ... 6 more ERROR 15:30:49,699 Fatal exception in thread Thread[CompactionExecutor:1,1,main] java.lang.RuntimeException: java.lang.IllegalArgumentException: Illegal mode w must be one of r, rw, rws, or rwd at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.IllegalArgumentException: Illegal mode w must be one of r, rw, rws, or rwd at java.io.RandomAccessFile.init(RandomAccessFile.java:197) at org.apache.cassandra.io.util.BufferedRandomAccessFile.init(BufferedRandomAccessFile.java:116) at org.apache.cassandra.io.sstable.CacheWriter.saveCache(CacheWriter.java:48) at org.apache.cassandra.db.CompactionManager$9.runMayThrow(CompactionManager.java:746) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) ... 6 more
Re: OOM during batch_mutate
On Mon, Feb 7, 2011 at 15:44, sridhar basam s...@basam.org wrote: Looks like you don't have a big enough working set from your GC logs, there doesn't seem to be a lot being reclaimed in the GC process. The process is reclaiming a few hundred MB and is running every few seconds. How big are your caches? The probable reason that it works the first couple times when you create it due to nothing being in cache as it gets built up. Hi Sridhar, thanks for your answer. I didn't set the cache to any specific value, I use the defaults: [default@Foo] show keyspaces; Keyspace: Foo: Replication Strategy: org.apache.cassandra.locator.SimpleStrategy Replication Factor: 1 Column Families: ColumnFamily: Url Columns sorted by: org.apache.cassandra.db.marshal.UTF8Type Row cache size / save period: 0.0/0 Key cache size / save period: 20.0/3600 Memtable thresholds: 4.6640625/995/60 GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 1.0 Built indexes: [] My row-key is 16bytes long string, so I don't expect the cache would be a problem. Patrik
Re: [0.7.1] more exceptions: Illegal mode
On Mon, Feb 7, 2011 at 15:42, Thibaut Britz thibaut.br...@trendiction.com wrote: I think this is related to a faulty disk. I'm not sure thats the problem. Cassandra 0.7.0 didn't report any problem. It started with Cassandra 0.7.1. Patrik
get_range_slices and tombstones
Hi! I'm getting tombstones from get_range_slices(). I know that's normal. But is there a way to know that a key is tombstone? I know tombstone has no columns but I can create a row without any columns that would look like a tombstone in get_range_slices(). Regards, Patrik
Re: [mapreduce] ColumnFamilyRecordWriter hidden reuse
On Wed, Jan 26, 2011 at 08:58, Mck m...@apache.org wrote: You are correct that microseconds would be better but for the test it doesn't matter that much. Have you tried. I'm very new to cassandra as well, and always uncertain as to what to expect... IMHO it's matter of use-case. In my use-case there is no possibility for two (or more) processes to write/update the same key so miliseconds are fine for me. BTW how to get current time in microseconds in Java? As far as moving the clone(..) into ColumnFamilyRecordWriter.write(..) won't this hurt performance? Normally i would _always_ agree that a defensive copy of an array/collection argument be stored, but has this intentionally not been done (or should it) because of large reduce jobs (millions of records) and the performance impact here. The size of the queue is computed at runtime: ColumnFamilyOutputFormat.QUEUE_SIZE, 32 * Runtime.getRuntime().availableProcessors() So the queue is not too large so I'd say the performance shouldn't get hurt. -- Patrik
Re: [mapreduce] ColumnFamilyRecordWriter hidden reuse
Hi Mick,
attached is the very simple MR job, that deletes expired URL from my
test Cassandra DB. The keyspace looks like this:
Keyspace: Test:
Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
Replication Factor: 2
Column Families:
ColumnFamily: Url2
Columns sorted by: org.apache.cassandra.db.marshal.UTF8Type
Row cache size / save period: 0.0/0
Key cache size / save period: 20.0/3600
Memtable thresholds: 4.7015625/1003/60
GC grace seconds: 864000
Compaction min/max thresholds: 4/32
Read repair chance: 1.0
Built indexes: []
In the CF the key is URL and inside there are some data. My MR job
needs just expire_date which is int64 timestamp. For now I store it
as a string because I use Python and C++ to manipulate the data as
well.
For the MR Job to run you need a patch I did. You can find it here:
https://issues.apache.org/jira/browse/CASSANDRA-2014
The atttached file contains the working version with cloned key in
reduce() method. My other aproache was:
[code]
context.write(ByteBuffer.wrap(key.getBytes(), 0, key.getLength()),
Collections.singletonList(getMutation(key)));
[/code]
Which produce junk keys.
Best regards,
Patrik
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.nio.LongBuffer;
import java.util.*;
import org.apache.cassandra.avro.Mutation;
import org.apache.cassandra.avro.Deletion;
import org.apache.cassandra.avro.SliceRange;
import org.apache.cassandra.hadoop.ColumnFamilyOutputFormat;
import org.apache.cassandra.db.IColumn;
import org.apache.cassandra.hadoop.ColumnFamilyInputFormat;
import org.apache.cassandra.hadoop.ConfigHelper;
import org.apache.cassandra.thrift.SlicePredicate;
import org.apache.cassandra.utils.ByteBufferUtil;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
public class ContextExpirator extends Configured implements Tool
{
static final String KEYSPACE = Test;
static final String COLUMN_FAMILY = Url2;
static final String OUTPUT_COLUMN_FAMILY = Url2;
static final String COLUMN_VALUE = expire_date;
public static void main(String[] args) throws Exception
{
// Let ToolRunner handle generic command-line options
ToolRunner.run(new Configuration(), new ContextExpirator(), args);
System.exit(0);
}
public static class UrlFilterMapper
extends MapperByteBuffer, SortedMapByteBuffer, IColumn, Text, NullWritable
{
private final static NullWritable nic = NullWritable.get();
private ByteBuffer sourceColumn;
private static long now;
protected void setup(org.apache.hadoop.mapreduce.Mapper.Context context)
throws IOException, InterruptedException
{
sourceColumn = ByteBuffer.wrap(COLUMN_VALUE.getBytes());
now = System.currentTimeMillis() / 1000; // convert from ms
}
public void map(ByteBuffer key, SortedMapByteBuffer, IColumn columns, Context context)
throws IOException, InterruptedException
{
IColumn column = columns.get(sourceColumn);
if (column == null) {
return;
}
Text tKey = new Text(ByteBufferUtil.string(key));
Long value = Long.decode(ByteBufferUtil.string(column.value()));
if(now value) {
context.write(tKey, nic);
}
}
}
public static class RemoveUrlReducer
extends ReducerText, NullWritable, ByteBuffer, ListMutation
{
public void reduce(Text key, IterableNullWritable values, Context context)
throws IOException, InterruptedException
{
ByteBuffer bbKey = ByteBufferUtil.clone(ByteBuffer.wrap(key.getBytes(), 0, key.getLength()));
context.write(bbKey, Collections.singletonList(getMutation()));
}
private static Mutation getMutation()
{
Deletion d = new Deletion();
d.timestamp = System.currentTimeMillis();
Mutation m = new Mutation();
m.deletion = d;
return m;
}
}
public int run(String[] args) throws Exception
{
Job job = new Job(getConf(), context_expitator);
job.setJarByClass(ContextExpirator.class);
job.setInputFormatClass(ColumnFamilyInputFormat.class);
ConfigHelper.setInputColumnFamily(job.getConfiguration(), KEYSPACE, COLUMN_FAMILY);
job.setMapperClass(UrlFilterMapper.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(NullWritable.class);
Re: [mapreduce] ColumnFamilyRecordWriter hidden reuse
On Tue, Jan 25, 2011 at 19:09, Mick Semb Wever m...@apache.org wrote: In fact i have another problem (trying to write an empty byte[], or something, as a key, which put one whole row out of whack, ((one row in 25 million...))). But i'm debugging along the same code. I don't quite understand how the byte[] in ByteBuffer.wrap(key.getBytes(),...) gets clobbered. Code snippet would help here. Well your key is a mutable Text object, so i can see some possibility depending on how hadoop uses these objects. Is there something to ByteBuffer.allocate(..) i'm missing... I don't know, I'm quite new to Java (but with long C++ history). btw. is d.timestamp = System.currentTimeMillis(); ok? shouldn't this be microseconds so that each mutation has a different timestamp? http://wiki.apache.org/cassandra/DataModel You are correct that microseconds would be better but for the test it doesn't matter that much. Patrik
Re: WordCount example problem
Thanks, I'll do. P. On Wed, Nov 10, 2010 at 16:28, Aditya Muralidharan aditya.muralidha...@nisc.coop wrote: Also, your Mapper class needs to look like this: MyMapper extends MapperByteBuffer, SortedMapByteBuffer, IColumn,Text,SumWritable ... with all the necessary fixes to the map method. AD -Original Message- From: Jonathan Ellis [mailto:jbel...@gmail.com] Sent: Wednesday, November 10, 2010 8:40 AM To: user Subject: Re: WordCount example problem http://www.mail-archive.com/user@cassandra.apache.org/msg07093.html On Wed, Nov 10, 2010 at 5:47 AM, Patrik Modesto patrik.mode...@gmail.com wrote: Hi, I'm trying the WordCount example and getting this error: [12:33]$ ./bin/word_count 10/11/10 12:34:35 INFO WordCount: output reducer type: filesystem 10/11/10 12:34:36 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 10/11/10 12:34:36 INFO WordCount: XXX:text0 10/11/10 12:34:36 INFO mapred.JobClient: Running job: job_local_0001 10/11/10 12:34:36 INFO mapred.MapTask: io.sort.mb = 100 10/11/10 12:34:36 INFO mapred.MapTask: data buffer = 79691776/99614720 10/11/10 12:34:36 INFO mapred.MapTask: record buffer = 262144/327680 10/11/10 12:34:36 WARN mapred.LocalJobRunner: job_local_0001 java.lang.ClassCastException: java.nio.HeapByteBuffer cannot be cast to [B at WordCount$TokenizerMapper.map(WordCount.java:73) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177) 10/11/10 12:34:37 INFO mapred.JobClient: map 0% reduce 0% 10/11/10 12:34:37 INFO mapred.JobClient: Job complete: job_local_0001 10/11/10 12:34:37 INFO mapred.JobClient: Counters: 0 I'm using cassandra 0.7.0beta3 (from latest trunk) on just one machine. Is the example working for anybody? Thanks, P. -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com
Re: WordCount example problem
That's exactly what's happening to me. I wonder why Google did't find it. Thanks! P. On Wed, Nov 10, 2010 at 15:39, Jonathan Ellis jbel...@gmail.com wrote: http://www.mail-archive.com/user@cassandra.apache.org/msg07093.html On Wed, Nov 10, 2010 at 5:47 AM, Patrik Modesto patrik.mode...@gmail.com wrote: Hi, I'm trying the WordCount example and getting this error: [12:33]$ ./bin/word_count 10/11/10 12:34:35 INFO WordCount: output reducer type: filesystem 10/11/10 12:34:36 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 10/11/10 12:34:36 INFO WordCount: XXX:text0 10/11/10 12:34:36 INFO mapred.JobClient: Running job: job_local_0001 10/11/10 12:34:36 INFO mapred.MapTask: io.sort.mb = 100 10/11/10 12:34:36 INFO mapred.MapTask: data buffer = 79691776/99614720 10/11/10 12:34:36 INFO mapred.MapTask: record buffer = 262144/327680 10/11/10 12:34:36 WARN mapred.LocalJobRunner: job_local_0001 java.lang.ClassCastException: java.nio.HeapByteBuffer cannot be cast to [B at WordCount$TokenizerMapper.map(WordCount.java:73) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177) 10/11/10 12:34:37 INFO mapred.JobClient: map 0% reduce 0% 10/11/10 12:34:37 INFO mapred.JobClient: Job complete: job_local_0001 10/11/10 12:34:37 INFO mapred.JobClient: Counters: 0 I'm using cassandra 0.7.0beta3 (from latest trunk) on just one machine. Is the example working for anybody? Thanks, P. -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com
