We have key-value pairs that need to be searchable. We are looking for best approach, both in terms of indexing (fast as well as space efficient) as well as retrieval (fast search).
Right now, the two approaches that we have are: Nested docs or dynamic fields (myField_*_time:some date) The number of dynamic fields would definitely be > 1k. We wanted to get an idea which of these approaches work best or if there a third approach which is better than nested and dynamic fields. On Tue, Jun 27, 2017 at 5:39 AM, Susheel Kumar <[email protected]> wrote: > Can you describe your use case in terms of what business functionality you > are looking to achieve. > > Thanks, > Susheel > > On Mon, Jun 26, 2017 at 4:26 PM, Saurabh Sethi <[email protected] > > > wrote: > > > Number of dynamic fields will be in thousands (millions of users + > > thousands of events shared between subsets of users). > > > > We also thought about indexing in one field with value being > > fieldname_fieldvalue. Since we support range queries for dates and > numbers, > > it won't work out of box. > > > > On Mon, Jun 26, 2017 at 1:05 PM, Erick Erickson <[email protected] > > > > wrote: > > > > > How many distinct fields do you expect across _all_ documents? That > > > is, if doc1 has 10 dynamic fields and doc2 has 10 dynamic fields, will > > > there be exactly 10 fields total or more than 10 when you consider > > > both documents? > > > > > > 100s of fields total across all documents is a tractable problem. > > > thousands of dynamic fields total is going to be a problem. > > > > > > One technique that people do use is to index one field with a prefix > > > rather than N dynamic fields. So you have something like > > > dyn1_val1 > > > dyn1_val2 > > > dyn4_val67 > > > > > > Only really works with string fields of course. > > > > > > Best, > > > Erick > > > > > > On Mon, Jun 26, 2017 at 10:11 AM, Saurabh Sethi > > > <[email protected]> wrote: > > > > We have two requirements: > > > > > > > > 1. Indexing and storing event id and its timestamp. > > > > 2. Indexing and storing custom field name and value. The fields can > be > > of > > > > any type, but for now lets say they are of types string, date and > > number. > > > > > > > > The events and custom fields for any solr document can easily be in > > > > hundreds. > > > > > > > > We are looking at two different approaches to handle these scenarios: > > > > > > > > 1. *Dynamic fields* - Have the fields name start with a particular > > > pattern > > > > like for string, the pattern could be like str_* and for event could > be > > > > eventid_* > > > > 2. *Parent/child fields* - This seems to be an overkill for our use > > case > > > > since it's more for hierarchical data. Also, the parent and all its > > > > children need to be reindexed on update which defeats the purpose - > we > > > are > > > > now reindexing multiple docs instead of one with dynamic fields. But > it > > > > allows us to store custom field name along with its value unlike > > dynamic > > > > fields where we will have to map user supplied custom field to some > > other > > > > name based on type. > > > > > > > > Has anyone handled similar scenarios with Solr? If so, which approach > > > would > > > > you recommend based on your experience? > > > > > > > > We are using solr 6.6 > > > > > > > > Thanks, > > > > Saurabh > > > > > > > > > > > -- > > Saurabh Sethi > > Principal Engineer I | Engineering > > > -- Saurabh Sethi Principal Engineer I | Engineering
