Re: Dynamic schema design: feedback requested

2013-03-11 Thread Yonik Seeley
On Wed, Mar 6, 2013 at 7:50 PM, Chris Hostetter hossman_luc...@fucit.org wrote: 2) If you wish to use the /schema REST API for read and write operations, then schema information will be persisted under the covers in a data store whose format is an implementation detail just like the index file

Re: Dynamic schema design: feedback requested

2013-03-11 Thread Chris Hostetter
: 2) If you wish to use the /schema REST API for read and write operations, : then schema information will be persisted under the covers in a data store : whose format is an implementation detail just like the index file format. : : This really needs to be driven by costs and benefits... :

Re: Dynamic schema design: feedback requested

2013-03-11 Thread Chris Hostetter
To revisit sarowes comment about how/when to decide if we are using the config file version of schema info (and hte API is read only) vs internal managed state data version of schema info (and the API is read/write)... On Wed, 6 Mar 2013, Steve Rowe wrote: : Two possible approaches: : : a.

Re: Dynamic schema design: feedback requested

2013-03-11 Thread Yonik Seeley
On Mon, Mar 11, 2013 at 2:50 PM, Chris Hostetter hossman_luc...@fucit.org wrote: : 2) If you wish to use the /schema REST API for read and write operations, : then schema information will be persisted under the covers in a data store : whose format is an implementation detail just like the

Re: Dynamic schema design: feedback requested

2013-03-11 Thread Chris Hostetter
: we needed to, we could just assert that the schema file is the : persistence mechanism, as opposed to the system of record, hence if : you hand edit it and then use the API to change it, your hand edit may : be lost. Or we may decide to do away with local FS mode altogether. presuming that

Re: Dynamic schema design: feedback requested

2013-03-11 Thread Yonik Seeley
On Mon, Mar 11, 2013 at 5:51 PM, Chris Hostetter hossman_luc...@fucit.org wrote: : I guess my main point is, we shouldn't decide a priori that using the : API means you can no longer hand edit. and my point is we should build a feature where solr has the ability to read/write some piece of

Re: Dynamic schema design: feedback requested

2013-03-08 Thread Steve Rowe
Hi Jan, On Mar 6, 2013, at 4:50 PM, Jan Høydahl jan@cominvent.com wrote: Will ZK get pushed the serialized monolithic schema.xml / schema.json from the node which changed it, and then trigger an update to the rest of the cluster? Yes. I was kind of hoping that once we have introduced

Re: Dynamic schema design: feedback requested

2013-03-08 Thread Steve Rowe
On Mar 6, 2013, at 7:50 PM, Chris Hostetter hossman_luc...@fucit.org wrote: I think it would make a lot of sense -- not just in terms of implementation but also for end user clarity -- to have some simple, straightforward to understand caveats about maintaining schema information... 1)

Re: Dynamic schema design: feedback requested

2013-03-08 Thread Steve Rowe
On Mar 8, 2013, at 2:57 PM, Steve Rowe sar...@gmail.com wrote: multiple collections may share the same config set and thus schema, so what happens if someone does not know this and hits PUT localhost:8983/solr/collection1/schema and it affects also the schema for collection2? Hmm, that's

Dynamic schema design: feedback requested

2013-03-06 Thread Steve Rowe
I'm working on SOLR-3251 https://issues.apache.org/jira/browse/SOLR-3251, to dynamically add fields to the Solr schema. I posted a rough outline of how I propose to do this:

Re: Dynamic schema design: feedback requested

2013-03-06 Thread Mark Miller
bq. Change Solr schema serialization from XML to JSON, and provide an XML-JSON conversion tool. What is the motivation for the change? I think if you are sitting down and looking to design a schema, working with the XML is fairly nice and fast. I picture that a lot of people would start by

Re: Dynamic schema design: feedback requested

2013-03-06 Thread Steve Rowe
In response to my thoughts about using DOM as an intermediate representation for schema elements, for use in lazy re-loading on schema change, Erik Hatcher argued against (solely) using XML for schema serialization

Re: Dynamic schema design: feedback requested

2013-03-06 Thread Mark Miller
Hmm…I think I'm missing some pieces. I agree with Erick that you should be able to load a schema from any object - a DB, a file in ZooKeeper, you name it. But it seems by default, having that object be schema.xml seems nicest to me. That doesn't mean you have to use DOM or XML internally -

Re: Dynamic schema design: feedback requested

2013-03-06 Thread Steve Rowe
I'm not sure what pieces you might be missing, sorry. I had thought about adding a web UI for schema composition, but that would be a major effort, and not in scope here. I agree, though, especially without a full schema modification REST API, that hand editing will have to be supported.

Re: Dynamic schema design: feedback requested

2013-03-06 Thread Mark Miller
On Mar 6, 2013, at 12:08 PM, Steve Rowe sar...@gmail.com wrote: I'm not sure what pieces you might be missing, sorry. My main confusion is around this: bq. When schema.xml is present, schema.json (if any) will be ignored. Basically, why have schema.json? Perhaps it's just me, but a json

Re: Dynamic schema design: feedback requested

2013-03-06 Thread Steve Rowe
On Mar 6, 2013, at 3:33 PM, Mark Miller markrmil...@gmail.com wrote: On Mar 6, 2013, at 12:08 PM, Steve Rowe sar...@gmail.com wrote: I'm not sure what pieces you might be missing, sorry. My main confusion is around this: bq. When schema.xml is present, schema.json (if any) will be ignored.

Re: Dynamic schema design: feedback requested

2013-03-06 Thread Jan Høydahl
How will this all work with ZooKeeper and cloud? Will ZK get pushed the serialized monolithic schema.xml / schema.json from the node which changed it, and then trigger an update to the rest of the cluster? I was kind of hoping that once we have introduced ZK into the mix as our centralized

Re: Dynamic schema design: feedback requested

2013-03-06 Thread Chris Hostetter
: As far as a user editing the file AND rest API access, I think that : seems fine. Yes, the user is in trouble if they break the file, but that Ignoring for a moment what format is used to persist schema information, I think it's important to have a conceptual distinction between data that

Re: Dynamic schema design: feedback requested

2013-03-06 Thread Mark Miller
On Mar 6, 2013, at 4:50 PM, Chris Hostetter hossman_luc...@fucit.org wrote: i don't think it's unreasable to say if you would like to manipulate the schema using an API, then you give up the ability to manipulate it as a config file on disk As long as you can initially work with an