Not maybe. You are absolutely right. Bad idea. Hmpf. Am 05.03.2017 09:23 schrieb "benjamin roth" <brs...@gmail.com>:
> Sorry. Answer was to fast. Maybe you are right. > > Am 05.03.2017 09:21 schrieb "benjamin roth" <brs...@gmail.com>: > >> No. You just change the partitioner. That's all >> >> Am 05.03.2017 09:15 schrieb "DuyHai Doan" <doanduy...@gmail.com>: >> >>> "How can that be achieved? I haven't done "scientific researches" yet >>> but I >>> guess a "MV partitioner" could do the trick. Instead of applying the >>> regular partitioner, an MV partitioner would calculate the PK of the base >>> table (which is always possible) and then apply the regular partitioner." >>> >>> The main purpose of MV is to avoid the drawbacks of 2nd index >>> architecture, >>> e.g. to scan a lot of nodes to fetch the results. >>> >>> With MV, since you give the partition key, the guarantee is that you'll >>> hit >>> a single node. >>> >>> Now if you put MV data on the same node as base table data, you're doing >>> more-or-less the same thing as 2nd index. >>> >>> Let's take a dead simple example >>> >>> CREATE TABLE user (user_id uuid PRIMARY KEY, email text); >>> CREATE MV user_by_email AS SELECT * FROM user WHERE user_id IS NOT NULL >>> AND >>> email IS NOT NULL PRIMARY KEY((email),user_id); >>> >>> SELECT * FROM user_by_email WHERE email = xxx; >>> >>> With this query, how can you find the user_id that corresponds to email >>> 'xxx' so that your MV partitioner idea can work ? >>> >>> >>> >>> On Sun, Mar 5, 2017 at 9:05 AM, benjamin roth <brs...@gmail.com> wrote: >>> >>> > While I was reading the MV paragraph in your post, an idea popped up: >>> > >>> > The problem with MV inconsistencies and inconsistent range movement is >>> that >>> > the "MV contract" is broken. This only happens because base data and >>> > replica data reside on different hosts. If base data + replicas would >>> stay >>> > on the same host then a rebuild/remove would always stream both >>> matching >>> > parts of a base table + mv. >>> > >>> > So my idea: >>> > Why not make a replica ALWAYS stay local regardless where the token of >>> a MV >>> > would point at. That would solve these problems: >>> > 1. Rebuild / remove node would not break MV contract >>> > 2. A write always stays local: >>> > >>> > a) That means replication happens sync. That means a quorum write to >>> the >>> > base table guarantees instant data availability with quorum read on a >>> view >>> > >>> > b) It saves network roundtrips + request/response handling and helps to >>> > keep a cluster healthier in case of bulk operations (like repair >>> streams or >>> > rebuild stream). Write load stays local and is not spread across the >>> whole >>> > cluster. I think it makes the load in these situations more >>> predictable. >>> > >>> > How can that be achieved? I haven't done "scientific researches" yet >>> but I >>> > guess a "MV partitioner" could do the trick. Instead of applying the >>> > regular partitioner, an MV partitioner would calculate the PK of the >>> base >>> > table (which is always possible) and then apply the regular >>> partitioner. >>> > >>> > I'll create a proper Jira for it on monday. Currently it's sunday here >>> and >>> > my family wants me back so just a few thoughts on this right now. >>> > >>> > Any feedback is appreciated! >>> > >>> > 2017-03-05 6:34 GMT+01:00 Edward Capriolo <edlinuxg...@gmail.com>: >>> > >>> > > On Sat, Mar 4, 2017 at 10:26 AM, Jeff Jirsa <jji...@gmail.com> >>> wrote: >>> > > >>> > > > >>> > > > >>> > > > >>> > > > > On Mar 4, 2017, at 7:06 AM, Edward Capriolo < >>> edlinuxg...@gmail.com> >>> > > > wrote: >>> > > > > >>> > > > >> On Fri, Mar 3, 2017 at 12:04 PM, Jeff Jirsa <jji...@gmail.com> >>> > wrote: >>> > > > >> >>> > > > >> On Fri, Mar 3, 2017 at 5:40 AM, Edward Capriolo < >>> > > edlinuxg...@gmail.com> >>> > > > >> wrote: >>> > > > >> >>> > > > >>> >>> > > > >>> I used them. I built do it yourself secondary indexes with >>> them. >>> > They >>> > > > >> have >>> > > > >>> there gotchas, but so do all the secondary index >>> implementations. >>> > > Just >>> > > > >>> because datastax does not write about something. Lets see like >>> 5 >>> > > years >>> > > > >> ago >>> > > > >>> there was this: https://github.com/hmsonline/c >>> assandra-triggers >>> > > > >>> >>> > > > >>> >>> > > > >> Still in use? How'd it work? Production ready? Would you still >>> do it >>> > > > that >>> > > > >> way in 2017? >>> > > > >> >>> > > > >> >>> > > > >>> There is a fairly large divergence to what actual users do and >>> what >>> > > > other >>> > > > >>> groups 'say' actual users do in some cases. >>> > > > >>> >>> > > > >> >>> > > > >> A lot of people don't share what they're doing (for business >>> > reasons, >>> > > or >>> > > > >> because they don't think it's important, or because they don't >>> know >>> > > > >> how/where), and that's fine but it makes it hard for anyone to >>> know >>> > > what >>> > > > >> features are used, or how well they're really working in >>> production. >>> > > > >> >>> > > > >> I've seen a handful of "how do we use triggers" questions in >>> IRC, >>> > and >>> > > > they >>> > > > >> weren't unreasonable questions, but seemed like a lot of pain, >>> and >>> > > more >>> > > > >> than one of those people ultimately came back and said they used >>> > some >>> > > > other >>> > > > >> mechanism (and of course, some of them silently disappear, so we >>> > have >>> > > no >>> > > > >> idea if it worked or not). >>> > > > >> >>> > > > >> If anyone's actively using triggers, please don't keep it a >>> secret. >>> > > > Knowing >>> > > > >> that they're being used would be a great way to justify >>> continuing >>> > to >>> > > > >> maintain them. >>> > > > >> >>> > > > >> - Jeff >>> > > > >> >>> > > > > >>> > > > > "Still in use? How'd it work? Production ready? Would you still >>> do it >>> > > > that way in 2017?" >>> > > > > >>> > > > > I mean that is a loaded question. How long has cassandra had >>> > Secondary >>> > > > > Indexes? Did they work well? Would you use them? How many times >>> were >>> > > > they re-written? >>> > > > >>> > > > It wasn't really meant to be a loaded question; I was being sincere >>> > > > >>> > > > But I'll answer: secondary indexes suck for many use cases, but >>> they're >>> > > > invaluable for their actual intended purpose, and I have no idea >>> how >>> > many >>> > > > times they've been rewritten but they're production ready for their >>> > > narrow >>> > > > use case (defined by cardinality). >>> > > > >>> > > > Is there a real triggers use case still? Alternative to MVs? >>> > Alternative >>> > > > to CDC? I've never implemented triggers - since you have, what's >>> the >>> > > level >>> > > > of surprise for the developer? >>> > > >>> > > >>> > > :) You mention alternatives/: Lets break them down. >>> > > >>> > > MV: >>> > > They seem to have a lot pf promise. IE you can use them for things >>> other >>> > > then equality searches, and I do think the CQL example with the top N >>> > high >>> > > scores is pretty useful. Then again our buddy Mr Roth has a thread >>> named >>> > > "Rebuild / remove node with MV is inconsistent". I actually think a >>> lot >>> > of >>> > > the use case for mv falls into the category of "something you should >>> > > actually be doing with storm". I can vibe with the concept of not >>> > needing a >>> > > streaming platform, but i KNOW storm would do this correctly. I don't >>> > want >>> > > to land on something like 2x index v1 v2 where there was fundamental >>> > flaws >>> > > at scale.(not saying this is case but the rebuild thing seems a bit >>> > scary) >>> > > >>> > > CDC: >>> > > I slightly afraid of this. Rational: A extensible piece design >>> > specifically >>> > > for a close source implementation of hub and spoke replication. I >>> have >>> > some >>> > > experience trying to "play along" with extensible things >>> > > https://issues.apache.org/jira/browse/CASSANDRA-12627 >>> > > "Thus, I'm -1 on {[PropertyOrEnvironmentSeedProvider}}." >>> > > >>> > > Not a rub, but I can't even get something committed using an existing >>> > > extensible interface. Heaven forbid a use case I have would want to >>> > > *change* >>> > > the interface, I would probably get a -12. So I have no desire to >>> try and >>> > > maintain a CDC implementation. I see myself falling into the same old >>> > "why >>> > > you want to do this? -1" trap. >>> > > >>> > > Coordinator Triggers: >>> > > To bring things back really old-school coordinator triggers everyone >>> > always >>> > > wanted. In a nutshell, I DO believe they are easier to reason about >>> then >>> > > MV. It is pretty basic, it happens on the coordinator there is no >>> > batchlogs >>> > > or whatever, best effort possibly requiring more nodes then as the >>> keys >>> > > might be on different services. Actually I tend do like features >>> like. >>> > Once >>> > > something comes on the downswing of "software hype cycle" you know >>> it is >>> > > pretty stable as everyone's all excited about other things. >>> > > >>> > > As I said, I know I can use storm for top-n, so what is this feature? >>> > Well >>> > > I want to optimize my network transfer generally by building my batch >>> > > mutations on the server. Seems reasonable. Maybe I want to have my >>> own >>> > > little "read before write" thing like CQL lists. >>> > > >>> > > The warts, having tried it. First time i tried it found it did not >>> work >>> > > with non batches, patched in 3 hours. Took weeks before some CQL >>> user had >>> > > the same problem and it got fixed :) There was no dynamic stuff at >>> the >>> > time >>> > > so it was BYO class loader. Going against the grain and saying. >>> > > >>> > > The thing you have to realize with the best effort coordinator >>> triggers >>> > are >>> > > that "transaction" could be incomplete and well that sucks maybe for >>> some >>> > > cases. But I actually felt the 2x index implementations force all >>> > problems >>> > > into a type of "foreign key transnational integrity " that does not >>> make >>> > > sense for cassandra. >>> > > >>> > > Have you every used elastic search, there version of consistency is >>> write >>> > > something, keep reading and eventually you see it, wildly popular :) >>> It >>> > is >>> > > a crazy world. >>> > > >>> > >>> >>
