Hi all,
2013/2/15 Lewis John Mcgibbney <[email protected]>: > Hi, > OK so in an attempt to try and close this off I am proposing the following. > We follow up on Alfonso's road map... which is as follows > > *Proposals* > > 1. Create a configuration opt-in option for a DataStore to write > null-plus-onetype-unions without the index byte and delete the column when > null. Process properly when reading. This allows other systems to directly > read data easily. Ok, so I think I am understanding now. We are just trying to solve the null-plus-onetype-unions? Ok then, Alfonso's proposal is the way to go. Ignore the field when it is null, and persist it when it is not null. But my point was on supporting various-type unions. I know that HBase will be able to support any type of unions because it serializes data using Avro, but we have to decide on how other non-avro-based data stores will handle any type of unions. > 2. Create a *deprecated* configuration option that will parse schemas > ignoring unions and nulls, and will work like Gora 0.2.1. This will enable > anyone using pre 0.3 versions of Gora to upgrade. I think 0.2.1 with Alfonso's patch does handle this null-plus-onetype-unions correctly. So +1 on this one. > 3. Take on board Renato's suggestions regarding the CassandraStore. > @Renato, if possible it would be great for you to comment on GORA-174 with > your precise objectives. I think I have been confusing things here. GORA-174 is for supporting null-plus-onetype-unions, and I was addressing support for various-type-unions for non-avro-based data stores which is a different issue. > *Concerns* > > 1. AccumuloStore option as we do not seem to have a proposal to adapt > the AccumuloStore logic yet. The changes made into the compiler will alleviate further changes in other data stores. We should open another issue in JIRA to clarify that we need to address this in a near future. > 2. AFAIK we DO NOT need to worry about DynamoDB as we do not use Avro at > all but if you could confirm Renato that would be great. This is one thing I have been meaning to get around to. DynamoDB is not avro-based but Gora heavily relies in avro structures to maintain in memory data structures to then flush them into specific back-ends in a batch manner. As Amazon didn't provide any cost benefit for writing directly or in batch operations for DynamoDB, the Gora-DynamoDB does not support batch operations (operations performed when we do a "flush"). At the end of last year Amazon started providing support for batch operations for DynamoDB which now makes sense to build all the other features within the Gora-DynamoDB module ... *sight* ... that was a long explanation (: I will address this in separate issues. > 3. AvroStore... what is going on here? Do we need to make changes > locally to the store or are these already supported? My feeling is that > they are not... unless of course Alfonso's existing patches actually > address this. Can you please confrim Alfonso? IMHO we should probably make a roadmap for the AvroStore so we can take this store as a model for other data stores (as most of them rely on it), and get that Avro patch in!!! Maybe we should join strengths and tackle that issue after closing this one. > *Actions* > > 1. Address Proposals[1,2,3]: Produce concrete patches and get them > submitted to GORA-174 +1 > 2. Re-base on Concerns[1,2,3] and of course add to them with input from > you guys. +1 > 3. Test the functionality and comment on the issues opened by Renato > recently e.g. GORA-206 & 207. +1 I will do, and open different JIRA issues for supporting various-type-unions inside other data stores, and then we can discuss each case separately. > 4. Push the 0.3 RC. +1000!!! > *Notes* > > It should be noted that we have seen little or no non-backwards compatible > changes in the Gora API since the project was accepted into the Apache > Incubator. Although we should aim to always maintain backwards > compatibility, we should not shy away from engaging with proposals to move > the code base forward. +1 with this one as well. But again I think we should differ between backward compatibility and new features, as the null-one-type-union case. I know 0.2.1 release will only ignore them, or log a message saying that union data type is not FULLY supported in this version which is IMHO a new feature. > The PMC and Committer base as well as users determine in which direction > Gora goes. Without implementing some of the ideas Gora will go no where. > > On that note, have a great weekend everyone, I am looking forward to > ironing out GORA-174 as it has dominated the Gora agenda somewhat recently. ;) Renato M. > Thanks > > Lewis > > > > > -- > *Lewis*
