Re: What does a write lock ?

Thu, Jul 7, 2011 at 3:21 PM, Yang wrote: > >> just to add onto what jonathan said >> >> the columns are immutable . if u overwrite/ reconcile a new obj is >> created and shoved into the memtable >> >> there is a shared lock for all writes though which guard agai

Re: What does a write lock ?

validation is on the new incoming column ,not the old row,right? On Jul 7, 2011 8:25 AM, "Jeffrey Kesselman" wrote: > Hi Jonnathan, > > This brings up an important question. I have been assuming that the > validation check is part of the atomic update operation. Is this NOT the > case? Which is t

Re: What does a write lock ?

just to add onto what jonathan said the columns are immutable . if u overwrite/ reconcile a new obj is created and shoved into the memtable there is a shared lock for all writes though which guard against an exclusive lock on memtable switching/flushing On Jul 7, 2011 7:51 AM, "A J" wrote:

Re: Strong Consistency with ONE read/writes

I'm no expert. So addressing the question to me probably give you real answers :) The single entry mode makes sure that all writes coming through the leader are received by replicas before ack to client. Probably wont be stale data On Jul 3, 2011 11:20 AM, "AJ" wrote: > Yang,

Re: Strong Consistency with ONE read/writes

can use different CL for each single write request. in that case you would need to keep both the current lightweight Phi-detector and add the ZK for leader election for single-entry mode write. Thanks Yang (I should correct my terminology it's not a "strong failure dete

Re: Strong Consistency with ONE read/writes

there is a JIRA completed in 0.7.x that "Prefers" a certain node in snitch, so this does roughly what you want MOST of the time but the problem is that it does not GUARANTEE that the same node will always be read. I recently read into the HBase vs Cassandra comparison thread that started after F

faster ByteBuffer comparison

mpare() looping over each byte doing a ByteBuffer.get() is there a faster (probably hardware-based) compare ? I tried doing 8 bytes at a time by doing getLong() and it actually seems slower thanks Yang

Re: custom reconciling columns?

https://issues.apache.org/jira/browse/CASSANDRA-2843 thanks Yang On Fri, Jul 1, 2011 at 12:09 AM, Sylvain Lebresne wrote: > I think it's an interesting solution. And we can probably avoid the two > getTopLevelColumns flavors with at bit a refactor. Let's open a ticket > howe

Re: custom reconciling columns?

? the only disadvantage is a slight change to getTopLevelColumns so we have 2 flavors of this method Thanks Yang On Wed, Jun 29, 2011 at 5:51 PM, Jonathan Ellis wrote: > On Tue, Jun 28, 2011 at 10:06 PM, Yang wrote: > > I'm trying to see whether there are some easy magic bu

Re: custom reconciling columns?

fact a sortedmap) in the internal process. we could very well just use a sorted list to do the read path, which would be much faster. (hacking out this idea today ...) yang On Thu, Jun 30, 2011 at 8:27 AM, Jeremiah Jordan < jeremiah.jor...@morningstar.com> wrote: > ** > The r

Re: custom reconciling columns? (improve performance of long rows )

return is going to be even slower... On Tue, Jun 28, 2011 at 10:07 PM, Yang wrote: > btw I use only one box now just because I'm running it on dev junit test, > not that it's going to be that way in production > > > On Tue, Jun 28, 2011 at 10:06 PM, Yang wr

Re: custom reconciling columns?

btw I use only one box now just because I'm running it on dev junit test, not that it's going to be that way in production On Tue, Jun 28, 2011 at 10:06 PM, Yang wrote: > ok, here is the profiling result. I think this is consistent (having been > trying to recover how to

Re: custom reconciling columns?

Cheers > > > - > Aaron Morton > Freelance Cassandra Developer > @aaronmorton > http://www.thelastpickle.com > > On 29 Jun 2011, at 04:20, Yang wrote: > > > for example, if I have an application that needs to read off a user > browsing history, a

custom reconciling columns?

ay to incorporate such custom reconcile under current code framework? (I see custom sorting, but no custom reconcile) thanks yang

Re: Clock skew

oftentimes people use time actually subconsciously to express causal relations ("before/after"), as long as you have some other means to establish causal relations, you don't really need to have an exactly clock. On Mon, Jun 27, 2011 at 4:54 PM, aaron morton wrote: > Without exception the timesta

Re: network/message simulation framework?

re static, then the tests can be carried out in a shell script organizing several java processes, slightly less convenient than a single junit process which orchestrates those multiple servers in a single JVM yang On Sat, Jun 25, 2011 at 3:40 AM, Watanabe Maki wrote: > iptables? > > From

Re: network/message simulation framework?

never mind, I see it does work for stopping messages. but still can't simulate multiple IPs on one box. On Fri, Jun 24, 2011 at 8:47 PM, Yang wrote: > Thanks Jonathan. > > this provides a way to essentially get a copy of the outgoing messages, > the messages onto the real con

Re: network/message simulation framework?

an even more difficult thing is that the ThriftSocket etc will need to open a java.net.Socket, which hooks onto the real IP address and port, in a simulation environment, we'd need to mock out the java.net.Socket too, that sounds more difficult. yang On Fri, Jun 24, 2011 at 8:23 PM, Jon

network/message simulation framework?

ious mock frameworks look like a good fit, but in reality is not enough to simulate complex network code, for example, it's basically impossible to setup a simulated N-node cluster on only one test box. Thanks Yang

Re: Concurrency: Does C* support a Happened-Before relation between processes' writes?

by "possible node N", I mean possible clients that will ever try to do the locking On Fri, Jun 24, 2011 at 12:28 PM, Yang wrote: > without a clear description of your pseudo-code, it's difficult to say > whether it will work. > > but I think it can work fine as an

Re: Concurrency: Does C* support a Happened-Before relation between processes' writes?

without a clear description of your pseudo-code, it's difficult to say whether it will work. but I think it can work fine as an election/agreement protocol, which you can use as a lock to some degree, but this requires all the potential lock contenders to all participate, you can't grab a lock bef

Re: simple get_slice() gives error?

another question: in current cassandra daemon, is it possible that a column (or more specifically its bytebuffer) is still being held by the read thread , while it was partially written by a write thread? this way the read thread could possibly get garbled result. Thanks Yang On Fri, Jun 10, 2011

Re: MR-hook for sstable compaction?

Thanks Jonathan. yes I did notice the RF issue, and thought , for example, to get a total salary, you'd need to divide it by RF, something like that. I'll take a look at 1608, Yang On Sun, Jun 19, 2011 at 12:12 AM, Jonathan Ellis wrote: > I'm skeptical that this is the rig

MR-hook for sstable compaction?

yang

Re: Propose new ConsistencyLevel.ALL_AVAIL for reads

consistency level definition should be a definition of requirement from the application perspective, it should not be tied to some ephemeral state in the system (: a node being deemed "available/up" or down is determined by the gossip and changes every second ) what you want can be simply achieve

Re: one way to make counter delete work better

patch in https://issues.apache.org/jira/browse/CASSANDRA-2774 <https://issues.apache.org/jira/browse/CASSANDRA-2774>some coding is messy and only intended for demonstration only, we could refine it after we agree this is a feasible way to go. Thanks Yang On Tue, Jun 14, 2011 at 11

Re: one way to make counter delete work better

both node eventually agree on the value 3 (i.e, only the second >> increment). >> I don't see how you could assign epoch numbers or anything to fix that. >> >> -- >> Sylvain >> >> On Mon, Jun 13, 2011 at 8:26 PM, Yang wrote: >> > ok, I think it

Re: one way to make counter delete work better

in "stronger reason", I mean the +3 is already merged up in memtable of node B, you can't find +1 and +2 any more On Tue, Jun 14, 2011 at 7:02 PM, Yang wrote: > I almost got the code done, should release in a bit. > > > > your scenario is not a problem conc

Re: one way to make counter delete work better

ompletely new counter, the counter name we are using is just kind of an alias. yang On Tue, Jun 14, 2011 at 11:21 AM, Sylvain Lebresne wrote: > Who assigns those epoch numbers ? > You need all nodes to agree on the epoch number somehow to have this work, > but then how do you maintai

Re: one way to make counter delete work better

ch number] then within each epoch, counter updates work as currently implemented On Mon, Jun 13, 2011 at 10:12 AM, Yang wrote: > I think this approach also works for your scenario: > > I thought that the issue is only concerned with merging within the same > leader; but you pointed o

Re: one way to make counter delete work better

ordering rule used in reconciliation; here we just need to find a similar ordering rule. the epoch thing could be a step on this direction. Thanks Yang On Mon, Jun 13, 2011 at 9:04 AM, Jonathan Ellis wrote: > I don't think that's bulletproof either. For instance, what if the >

one way to make counter delete work better

th, in the "no replicate-on-write" case, we need to add a read to sstable. but in the "replicate-on-write" case, we already read that, so it's no extra time cost. "no replicate-on-write" is not a very useful setup in reality anyway. does this sound a feasible way? if this works, expiring counter should also naturally work. Thanks Yang

Counter Column in Cassandra

Hi, All I am newbie to cassandra. I have a simple question but don't find any clear answer by searching google: What's the meaning of counter column in Cassandra? Best

Re: odd logs after repair

Hi, All I am newbie to cassandra. I have a simple question but don't find any clear answer by searching google: What's the meaning of count column in Cassandra? Thanks.

simple get_slice() gives error?

buffer, which also preserves the pos. is this some error on server code? (looks a flip() is needed somewhere) Thanks Yang

Re: auto-expiration of columns happens only during compaction?

thanks Nate On Fri, Jun 10, 2011 at 3:54 PM, Nate McCall wrote: > These are filtered out server side (see > o.a.c.db.filter.QueryFilter#isRelevant and o.a.c.db.ExpiringColumn for > specifics). > > On Fri, Jun 10, 2011 at 5:08 PM, Yang wrote: > > if I have a really shor

auto-expiration of columns happens only during compaction?

if I have a really short ttl, and the column expired before flush happens. then if I query on this column, would Cassandra recognize that it has lived past its ttl? or do I need to filter that out in application logic? Thanks Yang

Re: need some help with counters

something like this: https://issues.apache.org/jira/browse/CASSANDRA-2103 but this turns out not feasible On Thu, Jun 9, 2011 at 12:41 PM, Ian Holsman wrote: > Hi. > > I had a brief look at CASSANDRA-2103 (expiring counter columns), and I wa

caution for restarting CassandraDaemon in junit

one fails. it turns out that after "deactivate()" , the static caches for implementing singleton in Table is not cleared, so they are re-used. there could be more cases like this. so to get that working, you need to set the "forkMode" of surefire-plugin to let each test class be run in a separate JVM yang

Re: [RELEASE] 0.8.0

Is there anyone willing to upgrade the libcassandra for C++, to support new features in 0.8.0? Or has anyone started to work on it? Thanks On Jun 3, 2011, at 7:36 AM, Eric Evans wrote: > > I am very pleased to announce the official release of Cassandra 0.8.0. > > If you haven't been paying a

Re: "include" ability in schema file?

sorry wrong list ... please ignore On Tue, May 31, 2011 at 4:26 PM, Yang wrote: > I have a protocol schema hello.avpr > > { > types: { >{ name : input_msg , type: record : fields [ > { name: date , type int }, >{ na

"include" ability in schema file?

s this possible? also btw, for an avro-generate object class, would it be better to make the getSchema() method static? that way I can get its schema without creating an actual obj thanks Yang

Re: one idea on counters: simpler protocol due to FIFO

ath does not need a special-case: you only need to attach a repair flag to the mutation when sending out the repair command, it's up to the column to interpret this. anyway it's probably too inflexible to strictly mandate that all repair behaviors should be same as write, or just rep

Re: clarification of the consistency guarantees of Counters

never mind , I see that if leader/owner dies, the other replicas can simply use whoever has the highest count of the leader bucket, though not the authoritative number On Tue, May 31, 2011 at 1:21 AM, Yang wrote: > thanks Sylvain, I agree with what you said for the first few paragra

Re: clarification of the consistency guarantees of Counters

to reconcile, that's what I mean. but anyway it's not relevant now since CL can be > ONE but I'd really appreciate if you could give some review to my newer post on FIFO, I think that could be an interesting approach yang On Tue, May 31, 2011 at 12:59 AM, Sylvain Lebresn

one idea on counters: simpler protocol due to FIFO

me fast write/update as regular columns, without a read-write pattern, and essentially do "fire and forget", while the tally-up is left over to background work. we also avoid the extra "leader" network hop required by current implementation. Thanks Yang

Re: clarification of the consistency guarantees of Counters

thanks, got it I looked at the code more closely, the response handler between the coordinator and itself as leader, and between leader and replicas, are shared, so the coordinator can indeed wait for the count replication to finish for ALL yang On Mon, May 30, 2011 at 6:51 PM, Jeremy Hanna

clarification of the consistency guarantees of Counters

ned in the JIRA, and later do reconciliation on the history. because messages losses are less common than success, maybe this is not as bad a route as we thought?? Thanks Yang

Re: expiring + counter column?

plicate only the **sum** that I own, not individual delta that I just received. actually I'm not quite understanding why this approach was chosen, since it makes each write into read---write (when getReplicateOnWrite() ) , which can be slow. I'm still trying to understand that Thanks Yang

Re: expiring + counter column?

sorry in the notation, instead of "ttl" I mean "timestamp" On Sun, May 29, 2011 at 12:24 AM, Yang wrote: > sorry to beat on the dead horse. > > I looked at the link referred from #2103 : > https://issues.apache.org/jira/browse/CASSANDRA-2101 > I agree w

Re: expiring + counter column?

errata: "so c(123) means a counter column of ttl=1," > "so c(123) means a counter column of ttl=123," On Sun, May 29, 2011 at 12:24 AM, Yang wrote: > sorry to beat on the dead horse. > > I looked at the link referred from #2103 : > https://issues.apache

Re: expiring + counter column?

same logic can be carried over into expiring column, since they are essentially automatically inserted deletes. that way #2103 could be "solved" I'm attaching an example below, you can refer to them if needed. Thanks a lot Yang example: for simplicity we assume there is only on

expiring + counter column?

is this combination feature available , or on track ? thanks Yang

Re: issue/minor bug with counters ?

" is how > throughput is defined, so this is working as expected. > > On Tue, May 24, 2011 at 11:15 AM, Yang wrote: >> if you have only counter columns in your keyspace, and do a lot of >> updates on a few keys, >> the getLiveSize() of memtable actually returns the

issue/minor bug with counters ?

if you have only counter columns in your keyspace, and do a lot of updates on a few keys, the getLiveSize() of memtable actually returns the total amount of traffic that has gone into the Memtable, not the real size, so you end up producing very small SStables, with only a few KBytes. (I have alrea

Re: rainbird question (why is the 1minute buffer needed?)

(), which are still in-memory operations, and thus would not be so bad ? Yang On Mon, May 23, 2011 at 11:54 AM, Ryan King wrote: > On Sun, May 22, 2011 at 11:00 AM, Yang wrote: >> Thanks, >> >> I did read through that pdf doc, and went through the counters code in >> 0.

Re: batch dump of data from cassandra?

thanks Sri I am trying to make sure that Brisk underneath does a simple scraping of the rows, instead of doing foreach key ( keys ) { lookup (key) }.. after that, I can feel comfortable using Brisk for the import/export jobs yang On Mon, May 23, 2011 at 8:50 AM, SriSatish Ambati wrote

Re: batch dump of data from cassandra?

Thanks Jonathan. On Sun, May 22, 2011 at 9:56 PM, Jonathan Ellis wrote: > I'd modify SSTableExport.serializeRow (the sstable2json class) to > output to whatever system you are targeting. > > On Sun, May 22, 2011 at 11:19 PM, Yang wrote: >> let's say periodically (d

batch dump of data from cassandra?

tilize the file cache coherency as I read down the file. Thanks Yang

Re: rainbird question (why is the 1minute buffer needed?)

data nodes. the extra module I am proposing simply translates a single insert into multi-insert, and then cassandra takes over from there Thanks Yang On Sun, May 22, 2011 at 3:47 AM, aaron morton wrote: >  The implementation of distributed counters is  more complicated than your > example,

rainbird question (why is the 1minute buffer needed?)

. are there some points I'm missing? Thanks Yang

for counters: does read have to be ALL ?

ave a code example: rv = client.get_counter('key1', ColumnPath(column_family='Counter1', column='c1'), ConsistencyLevel.ONE) is one of them wrong? Thanks Yang

bootstrap question

hanks Yang // don't bootstrap if there are no tables defined. if (DatabaseDescriptor.getNonSystemTables().size() > 0) startBootstrap(token); else { isBootstrapMode = false; SystemTable.setBoot

Re: Cassandra + Zookeeper, what is the current state?

Thanks guys. but I feel it would probably be better to refactor out the hooks and make components like zookeeper pluggable , so users could use either zookeeper or the current config-file based seeds discovery Yang On Tue, Oct 19, 2010 at 9:02 AM, Jeremy Hanna wrote: > Further, when they o

Cassandra + Zookeeper, what is the current state?

e names is Zookeeper really used? docs/blog posts from online search kind of give conflicting answers Thanks Yang

Re: Cassandra security model? ( or, authentication docs ?)

Thanks a lot On Mon, Oct 18, 2010 at 11:44 AM, Eric Evans wrote: > On Sun, 2010-10-17 at 21:26 -0700, Yang wrote: >> I searched around, it seems that this is not clearly documented yet; >> the closest I found is: >> http://www.riptano.com/docs/0.6.5/install/auth-config >

Cassandra security model? ( or, authentication docs ?)

I see that the raw thrift API has a login() method, but when I setup a one-node cluster, I can simply connect localhost/9160 without any passwd what is the current picture of the security model? how can I enforce authentication? I searched around, it seems that this is not clearly documented y

Re: zookeeper, how do you feed the pets?

I guess you'd have better luck with the zookeeper user list but according to my very limited understanding of ZK, u can use the ZooKeeper.setData()/getData() API http://hadoop.apache.org/zookeeper/docs/current/api/index.html storing a simple count should be simple to ZK since each ZNode is desi

<    1   2   3