Your biggest issue here is likely to be http connections. Making an HTTP connection to Solr is way more expensive than the ask of adding a single document to the index. If you are expecting to add 24 billion docs per day, I'd suggest that somehow merging those documents into batches before sending them to Solr will be necessary.
To my previous question - what do you gain by using Solr that you don't get from other solutions? I'd suggest that to make this system really work, you are going to need a deep understanding of how Lucene works - segments, segment merges, deletions, and many other things because when you start to work at that scale, the implementation details behind Lucene really start to matter and impact upon your ability to succeed. I'd suggest that what you are undertaking can certainly be done, but is a substantial project. Upayavira On Wed, Feb 10, 2016, at 09:48 PM, Mark Robinson wrote: > Thanks everyone for your suggestions. > Based on it I am planning to have one doc per event with sessionId > common. > > So in this case hopefully indexing each doc as and when it comes would be > okay? Or do we still need to batch and index to Solr? > > Also with 4M sessions a day with about 6000 docs (events) per session we > can expect about 24Billion docs per day! > > Will Solr still hold good. If so could some one please recommend a sizing > to cater to this levels of data. > The queries per second is around 320 qps. > > Thanks! > Mark > > > On Wed, Feb 10, 2016 at 3:38 AM, Emir Arnautovic < > emir.arnauto...@sematext.com> wrote: > > > Hi Mark, > > Appending session actions just to be able to return more than one session > > without retrieving large number of results is not good tradeoff. Like > > Upayavira suggested, you should consider storing one action per doc and > > aggregate on read time or push to Solr once session ends and aggregate on > > some other layer. > > If you are thinking handling infrastructure might be too much, you may > > consider using some of logging services to hold data. One such service is > > Sematext's Logsene (http://sematext.com/logsene). > > > > Thanks, > > Emir > > > > -- > > Monitoring * Alerting * Anomaly Detection * Centralized Log Management > > Solr & Elasticsearch Support * http://sematext.com/ > > > > > > > > On 10.02.2016 03:22, Mark Robinson wrote: > > > >> Thanks for your replies and suggestions! > >> > >> Why I store all events related to a session under one doc? > >> Each session can have about 500 total entries (events) corresponding to > >> it. > >> So when I try to retrieve a session's info it can back with around 500 > >> records. If it is this compounded one doc per session, I can retrieve more > >> sessions at a time with one doc per session. > >> eg under a sessionId an array of eventA activities, eventB activities > >> (using json). When an eventA activity again occurs, we will read all > >> that > >> data for that session, append this extra info to evenA data and push the > >> whole session related data back (indexing) to Solr. Like this for many > >> sessions parallely. > >> > >> > >> Why NRT? > >> Parallely many sessions are being written (4Million sessions hence > >> 4Million > >> docs per day). A person can do this querying any time. > >> > >> It is just a look up? > >> Yes. We just need to retrieve all info for a session and pass it on to > >> another system. We may even do some extra querying on some data like > >> timestamps, pageurl etc in that info added to a session. > >> > >> Thinking of having the data separate from the actual Solr Instance and > >> mention the loc of the dataDir in solrconfig. > >> > >> If Solr is not a good option could you please suggest something which will > >> satisfy this use case with min response time while querying. > >> > >> Thanks! > >> Mark > >> > >> On Tue, Feb 9, 2016 at 6:02 PM, Daniel Collins <danwcoll...@gmail.com> > >> wrote: > >> > >> So as I understand your use case, its effectively logging actions within a > >>> user session, why do you have to do the update in NRT? Why not just log > >>> all the user session events (with some unique key, and ensuring the > >>> session > >>> Id is in the document somewhere), then when you want to do the query, you > >>> join on the session id, and that gives you all the data records for that > >>> session. I don't really follow why it has to be 1 document (which you > >>> continually update). If you really need that aggregation, couldn't that > >>> happen offline? > >>> > >>> I guess your 1 saving grace is that you query using the unique ID (in > >>> your > >>> scenario) so you could use the real-time get handler, since you aren't > >>> doing a complex query (strictly its not a search, its a raw key lookup). > >>> > >>> But I would still question your use case, if you go the Solr route for > >>> that > >>> kind of scale with querying and indexing that much, you're going to have > >>> to > >>> throw a lot of hardware at it, as Jack says probably in the order of > >>> hundreds of machines... > >>> > >>> On 9 February 2016 at 19:00, Upayavira <u...@odoko.co.uk> wrote: > >>> > >>> Bear in mind that Lucene is optimised towards high read lower write. > >>>> That is, it puts in a lot of effort at write time to make reading > >>>> efficient. It sounds like you are going to be doing far more writing > >>>> than reading, and I wonder whether you are necessarily choosing the > >>>> right tool for the job. > >>>> > >>>> How would you later use this data, and what advantage is there to > >>>> storing it in Solr? > >>>> > >>>> Upayavira > >>>> > >>>> On Tue, Feb 9, 2016, at 03:40 PM, Mark Robinson wrote: > >>>> > >>>>> Hi, > >>>>> Thanks for all your suggestions. I took some time to get the details to > >>>>> be > >>>>> more accurate. Please find what I have gathered:- > >>>>> > >>>>> My data being indexed is something like this. > >>>>> I am basically capturing all data related to a user session. > >>>>> Inside a session I have categorized my actions like actionA, actionB > >>>>> etc.., > >>>>> per page. > >>>>> So each time an action pertaining to say actionA or actionB etc.. (in > >>>>> each > >>>>> page) happens, it is updated in Solr under that session (sessionId). > >>>>> > >>>>> So in short there is only one doc pertaining to a single session > >>>>> (identified by sessionid) in my Solr index and that is retrieved and > >>>>> updated > >>>>> whenever a new action under that session occurs. > >>>>> We expect upto 4Million session per day. > >>>>> > >>>>> On an average *one session's* *doc has a size* of *3MB to 20MB*. > >>>>> So if it is *4Million sessions per day*, each session writing around > >>>>> > >>>> *500 > >>> > >>>> times to Solr*, it is* 2Billion writes or (indexing) per day to Solr*. > >>>>> As it is one doc per session, it is *4Million docs per day*. > >>>>> This is around *80K docs indexed per second* during *peak* hours and > >>>>> around *15K > >>>>> docs indexed per second* into Solr during* non-peak* hours. > >>>>> Number of queries per second is around *320 queries per second*. > >>>>> > >>>>> > >>>>> 1. Average size of a doc > >>>>> 3MB to 20MB > >>>>> 2. Query types:- > >>>>> Until that session is in progress, whatever data is there for > >>>>> that > >>>>> session so far is queried and the new action's details captured and > >>>>> appended to existing data already captured related to that > >>>>> > >>>> session > >>> > >>>> and indexed back into Solr. So, longer the session the more data > >>>>> retrieved > >>>>> for each subsequent query to get current data captured for that > >>>>> > >>>> session. > >>> > >>>> Also querying can be done on timestamp etc... which is captured > >>>>> along > >>>>> with each action. > >>>>> 3. Are docs grouped somehow? > >>>>> All data related to a session are retrieved from Solr, updated > >>>>> and > >>>>> indexed back to Solr based on sessionId. No other grouping. > >>>>> 4. Are they time sensitive (NRT or offline process does this) > >>>>> As mentioned above this is in NRT. Each time a new user action in > >>>>> that > >>>>> session happens, we need to query existing session info already > >>>>> > >>>> captured > >>> > >>>> related to that session and append this new data to this > >>>>> > >>>> existing > >>> > >>>> info retrieved and index it back to Solr. > >>>>> 5. Will they update or it is rebuild every time, etc. > >>>>> Each time a new user action occurs, the full data pertaining to > >>>>> > >>>> that > >>> > >>>> session so far captured is retrieved from Solr, the extra latest data > >>>>> pertaining to this new action is appended and indexed back to > >>>>> > >>>> Solr. > >>> > >>>> 6. And the other thing you haven't told us is whether you plan on > >>>>> _adding_ > >>>>> 2B docs a day or whether that number is the total corpus size and you > >>>>> > >>>> are > >>> > >>>> re-indexing the 2B docs/day. IOW, if you are adding 2B docs/day, 30 > >>>>> > >>>> days > >>> > >>>> later do you have 2B docs or 60B docs in your > >>>>> corpus? > >>>>> We are expecting around 4 million sessions per day (per session 500 > >>>>> writes to Solr), which turns out to be 2B indexing done per day. So > >>>>> > >>>> after > >>> > >>>> 30 days it would be 4Milion*30 docs in the index. > >>>>> 7. Is there any aging of docs > >>>>> No we always query against the whole corpus present. > >>>>> 8. Is any doc deleted? > >>>>> No all data remains in the index > >>>>> > >>>>> Any suggestion is very welcome! > >>>>> > >>>>> Thanks! > >>>>> Mark. > >>>>> > >>>>> > >>>>> On Mon, Feb 8, 2016 at 3:30 PM, Jack Krupansky < > >>>>> > >>>> jack.krupan...@gmail.com > >>> > >>>> wrote: > >>>>> > >>>>> Oops... at 100 qps for a single node you would need 120 nodes to get > >>>>>> > >>>>> to 12K > >>>> > >>>>> qps and 800 nodes to get 80K qps, but that is just an extremely rough > >>>>>> ballpark estimate, not some precise and firm number. And that's if > >>>>>> > >>>>> all > >>> > >>>> the > >>>> > >>>>> queries can be evenly distributed throughout the cluster and don't > >>>>>> > >>>>> require > >>>> > >>>>> fanout to other shards, which effectively turns each incoming query > >>>>>> > >>>>> into n > >>>> > >>>>> queries where n is the number of shards. > >>>>>> > >>>>>> -- Jack Krupansky > >>>>>> > >>>>>> On Mon, Feb 8, 2016 at 12:07 PM, Jack Krupansky < > >>>>>> > >>>>> jack.krupan...@gmail.com> > >>>> > >>>>> wrote: > >>>>>> > >>>>>> So is there any aging or TTL (in database terminology) of older > >>>>>>> > >>>>>> docs? > >>> > >>>> And do all of your queries need to query all of the older documents > >>>>>>> > >>>>>> all > >>>> > >>>>> of > >>>>>> > >>>>>>> the time or is there a clear hierarchy of querying for aged > >>>>>>> > >>>>>> documents, > >>>> > >>>>> like > >>>>>> > >>>>>>> past 24-hours vs. past week vs. past year vs. older than a year? > >>>>>>> > >>>>>> Sure, > >>>> > >>>>> you > >>>>>> > >>>>>>> can always use a function query to boost by the inverse of document > >>>>>>> > >>>>>> age, > >>>> > >>>>> but Solr would be more efficient with filter queries or separate > >>>>>>> > >>>>>> indexes > >>>> > >>>>> for different time scales. > >>>>>>> > >>>>>>> Are documents ever updated or are they write-once? > >>>>>>> > >>>>>>> Are documents explicitly deleted? > >>>>>>> > >>>>>>> Technically you probably could meet those specs, but... how many > >>>>>>> organizations have the resources and the energy to do so? > >>>>>>> > >>>>>>> As a back of the envelope calculation, if Solr gave you 100 queries > >>>>>>> > >>>>>> per > >>>> > >>>>> second per node, that would mean you would need 1,200 nodes. It > >>>>>>> > >>>>>> would > >>> > >>>> also > >>>>>> > >>>>>>> depend on whether those queries are very narrow so that a single > >>>>>>> > >>>>>> node can > >>>> > >>>>> execute them or if they require fanout to other shards and then > >>>>>>> > >>>>>> aggregation > >>>>>> > >>>>>>> of results from those other shards. > >>>>>>> > >>>>>>> -- Jack Krupansky > >>>>>>> > >>>>>>> On Mon, Feb 8, 2016 at 11:24 AM, Erick Erickson < > >>>>>>> > >>>>>> erickerick...@gmail.com > >>>> > >>>>> wrote: > >>>>>>> > >>>>>>> Short form: You really have to prototype. Here's the long form: > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>> https://lucidworks.com/blog/2012/07/23/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/ > >>> > >>>> I've seen between 20M and 200M docs fit on a single piece of > >>>>>>>> > >>>>>>> hardware, > >>>> > >>>>> so you'll absolutely have to shard. > >>>>>>>> > >>>>>>>> And the other thing you haven't told us is whether you plan on > >>>>>>>> _adding_ 2B docs a day or whether that number is the total corpus > >>>>>>>> > >>>>>>> size > >>>> > >>>>> and you are re-indexing the 2B docs/day. IOW, if you are adding 2B > >>>>>>>> docs/day, 30 days later do you have 2B docs or 60B docs in your > >>>>>>>> corpus? > >>>>>>>> > >>>>>>>> Best, > >>>>>>>> Erick > >>>>>>>> > >>>>>>>> On Mon, Feb 8, 2016 at 8:09 AM, Susheel Kumar < > >>>>>>>> > >>>>>>> susheel2...@gmail.com> > >>>> > >>>>> wrote: > >>>>>>>> > >>>>>>>>> Also if you are expecting indexing of 2 billion docs as NRT or > >>>>>>>>> > >>>>>>>> if > >>> > >>>> it > >>>> > >>>>> will > >>>>>>>> > >>>>>>>>> be offline (during off hours etc). For more accurate sizing you > >>>>>>>>> > >>>>>>>> may > >>>> > >>>>> also > >>>>>>>> > >>>>>>>>> want to index say 10 million documents which may give you idea > >>>>>>>>> > >>>>>>>> how > >>> > >>>> much > >>>>>> > >>>>>>> is > >>>>>>>> > >>>>>>>>> your index size and then use that for extrapolation to come up > >>>>>>>>> > >>>>>>>> with > >>>> > >>>>> memory > >>>>>>>> > >>>>>>>>> requirements. > >>>>>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> Susheel > >>>>>>>>> > >>>>>>>>> On Mon, Feb 8, 2016 at 11:00 AM, Emir Arnautovic < > >>>>>>>>> emir.arnauto...@sematext.com> wrote: > >>>>>>>>> > >>>>>>>>> Hi Mark, > >>>>>>>>>> Can you give us bit more details: size of docs, query types, > >>>>>>>>>> > >>>>>>>>> are > >>> > >>>> docs > >>>> > >>>>> grouped somehow, are they time sensitive, will they update or > >>>>>>>>>> > >>>>>>>>> it > >>> > >>>> is > >>>> > >>>>> rebuild > >>>>>>>> > >>>>>>>>> every time, etc. > >>>>>>>>>> > >>>>>>>>>> Thanks, > >>>>>>>>>> Emir > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On 08.02.2016 16:56, Mark Robinson wrote: > >>>>>>>>>> > >>>>>>>>>> Hi, > >>>>>>>>>>> We have a requirement where we would need to index around 2 > >>>>>>>>>>> > >>>>>>>>>> Billion > >>>> > >>>>> docs > >>>>>>>> > >>>>>>>>> in > >>>>>>>>>>> a day. > >>>>>>>>>>> The queries against this indexed data set can be around 80K > >>>>>>>>>>> > >>>>>>>>>> queries > >>>> > >>>>> per > >>>>>>>> > >>>>>>>>> second during peak time and during non peak hours around 12K > >>>>>>>>>>> > >>>>>>>>>> queries > >>>> > >>>>> per > >>>>>>>> > >>>>>>>>> second. > >>>>>>>>>>> > >>>>>>>>>>> Can Solr realize this huge volumes. > >>>>>>>>>>> > >>>>>>>>>>> If so, assuming we have no constraints for budget what would > >>>>>>>>>>> > >>>>>>>>>> be > >>> > >>>> a > >>>> > >>>>> recommended Solr set up (number of shards, number of Solr > >>>>>>>>>>> > >>>>>>>>>> instances > >>>> > >>>>> etc...) > >>>>>>>>>>> > >>>>>>>>>>> Thanks! > >>>>>>>>>>> Mark > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>> Monitoring * Alerting * Anomaly Detection * Centralized Log > >>>>>>>>>> > >>>>>>>>> Management > >>>>>> > >>>>>>> Solr & Elasticsearch Support * http://sematext.com/ > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>
