Jack, Thanks for your response.
1. Flattening could be an option, although our scale and required functionality (runtime non DocValues backed facets) is beyond what solr3 can handle (billions of docs). We have flattened the meta data at the expense of "over"-generating solr documents. But to solve the problem I have described via flattening would make big impact on the scalability and price. 2. We have quite the opposite of what you have described about the dynamic fields: there will be very few per document. I agree, that caution should be taken here, as we have "suffered" (or should I say "experienced") having multivalued fields (the good thing is we never had to facet on them). Any other options? Maybe someone can share their experience with dynamic fields and discourage from pursuing this path? Dmitry On Mon, May 20, 2013 at 4:23 PM, Jack Krupansky <j...@basetechnology.com>wrote: > Before you dive off the deep end and "go crazy" with dynamic fields, try a > clean, simple, Solr-oriented static design. Yes, you CAN do an > over-complicated design with dynamic fields, but that doesn't mean you > should. > > In a single phrase, denormalize and flatten your design. Sure, that will > lead to a lot of rows, but Solr and Lucene are designed to do well in that > scenario. > > If you are still linking in terms of "C Struct", go for a long walk or do > SOMETHING else until you can get that idea out of your head. It is a > sub-optimal approach for exploiting the power of Lucene and Solr. > > Stay with a static schema design until you hit... just stay with a static > schema, period. > > Dynamic fields and multi-valued fields do have value, but only when used > in moderation - small numbers. If you start down a design path and find > that you are heavily dependent on dynamic fields and/or multi-valued fields > with large numbers of values per document, that is feedback that your > design needs to be denormalized and flattened further. > > -- Jack Krupansky > > -----Original Message----- From: Dmitry Kan > Sent: Monday, May 20, 2013 7:06 AM > To: solr-user@lucene.apache.org > Subject: [custom data structure] aligned dynamic fields > > > Hi all, > > Our current project requirement suggests that we should start storing > custom data structures in solr index. The custom data structure would be an > equivalent of C struct. > > The task is as follows. > > Suppose we have two types of fields, one is FieldName1 and the other > FieldName2. > > Suppose also that we can have multiple pairs of these two fields on a > document in Solr. > > That is, in notation of dynamic fields: > > doc1 > FieldName1_id1 > FieldName2_id1 > > FieldName1_id2 > FieldName2_id2 > > doc2 > FieldName1_id3 > FieldName2_id3 > > FieldName1_id4 > FieldName2_id4 > > FieldName1_id5 > FieldName2_id5 > > etc > > What we would like to have is a value for the Field1_(some_unique_id) and a > value for Field2_(some_unique_id) as input for search. That is we wouldn't > care about the some_unique_id in some search scenarios. And the search > would automatically iterate the pairs of dynamic fields and respect the > pairings. > > I know it used to be so, that with dynamic fields a client must provide the > dynamically generated field names coupled with their values up front when > searching. > > What data structure / solution could be used as an alternative approach to > help such a "structured search"? > > Thanks, > > Dmitry >
