Hi Peter I understand what you are saying but I think you are thinking more of report as graph and analysis and summary kind of data .. for my reports I do need to include all records that qualify certain criteria .. e.g. a listing of all orders placed in last 6 months .. now that could be 10000 orders and yes I will need probably a report that summarizes all that data but at the same time .. I need all those 10000 records to be exported in an excel file .. those are the reports that I am talking about ..
and 30000 probably is a stretch .. it might be 10-15000 at the most but I guess its still the same idea .. and yes I realize that its alot of data to be transferred over http .. but thats exactly why i am asking for suggestion on how to do .. I find it hard to believe that this is an unusual requirement .. I think most companies do reports that dump all records from databases in excel files .. so again to clarify I definitely need reports that present statistics and averages and yes I will be using facets and all kind of stuff there and I am not so concerned about those reports because like you pointed out, for those reports there will be very little data transfer but its the full data dump reports that I am trying to figure out the best way to handle. Thanks for your help Adeel On Thu, Sep 23, 2010 at 11:43 AM, Peter Sturge <peter.stu...@gmail.com>wrote: > Hi, > > Are you going to generate a report with 30000 records in it? That will > be a very large report - will anyone really want to read through that? > If you want/need 'summary' reports - i.e. stats on on the 30k records, > it is much more efficient to setup faceting and/or server-side > analysis to do this, rather than download > 30000 records to a client, then do statistical analysis on the result. > It will take a while to stream 30000 records over an http connection, > and, if you're building, say, a PDF table for 30k records, that will > take some time as well. > Server-side analysis then just send the results will work better, if > that fits your remit for reporting. > > Peter > > > > On Thu, Sep 23, 2010 at 4:14 PM, Adeel Qureshi <adeelmahm...@gmail.com> > wrote: > > Thank you for your suggestions .. makes sense and I didnt knew about the > > XsltResponseWriter .. that opens up door to all kind of possibilities > ..so > > its great to know about that > > > > but before I go that route .. what about performance .. In Solr Wiki it > > mentions that XSLT transformation isnt so bad in terms of memory usage > but I > > guess its all relative to the amount of data and obviously system > resources > > .. > > > > my data set will be around 15000 - 30'000 records at the most ..I do have > > about 30 some fields but all fields are either small strings (less than > 500 > > chars) or dates, int, booleans etc .. so should I be worried about > > performances problems while doing the XSLT translations .. secondly for > > reports Ill have to request solr to send all 15000 some records at the > same > > time to be entered in report output files .. is there a way to kind of > > stream that process .. well I think Solr native xml is already streamed > to > > you but sounds like for the translation it will have to load the whole > thing > > in RAM .. > > > > and again what about SolrJ .. isnt that supposed to provide better > > performance since its in java .. well I guess it shouldnt be much > different > > since it also uses the HTTP calls to communicate to Solr .. > > > > Thanks for your help > > Adeel > > > > On Thu, Sep 23, 2010 at 7:16 AM, kenf_nc <ken.fos...@realestate.com> > wrote: > > > >> > >> keep in mind that the <str name="id"> paradigm isn't completely useless, > >> the > >> str is a data type (string), it can be int, float, double, date, and > >> others. > >> So to not lose any information you may want to do something like: > >> > >> <id type="int">123</id> > >> <title type="str">xyz</title> > >> > >> Which I agree makes more sense to me. The name of the field is more > >> important than it's datatype, but I don't want to lose track of the data > >> type. > >> > >> Ken > >> -- > >> View this message in context: > >> > http://lucene.472066.n3.nabble.com/Solr-Reporting-tp1565271p1567604.html > >> Sent from the Solr - User mailing list archive at Nabble.com. > >> > > >
