Rickard Öberg wrote: > Richard Wallace wrote: > >> OTOH it would be fairly easy to fit the OfBiz Entity Engine in. But I >> also looked more at the IBatis store and that makes the most sense of >> all to me now if you're going to be storing to an RDBMS. Also, storing >> to something like HBase would be almost trivial or other BigTable, >> column-based, database. I kinda like that idea. >> > > I just looked at HBase the other day, and I agree, it would be a very > good fit! Cool to see how far Hadoop and friends have come. > > >> One question I still have about the separation of the search and >> persistence is doing data aggregation and reporting. In one system I >> worked on trying to generate a report from a Lucene index with 10's of >> thousands of entries took several minutes. When I did a spike to see >> how long it would take to generate the same report from the data in a >> relational database using an SQL query, it took only a few seconds. >> >> I'm wondering what would be the best way of generating this type of >> aggregated, report type data in Qi4j. >> > > As Niclas said, I think the best way to do it is to see reporting, which > is a very important function, as something you don't do with the main > persistence and domain model. Instead, create the tables you want for > your reporting needs, and export and map the domain model to it, and > then do reports from that, in offline mode. This way you can change the > tables at any time, make sure that they fit the reporting needs, and > don't impact the main application operation. > > Relational databases seem to be good at doing reporting. Let's use them > for that, and only that. Lucene and RDF systems are good at finding > stuff, so let's use them for that, and only that. JBDM, Amazon S3 and > friends seem to be good at storing data, so let's use them for that, and > only that. The sum total of all of this should be a reasonably good way > to do things, generally speaking. > >
Ok, this certainly makes sense. Use the right tool for the right job. Looking into these kinds of issues and reading more about OLAP and data mining and BI in general it definitely seems like off-loading reporting to one somewhere else is the right way to go. So, if we can have a side effect for pushing entities into a relational db then it should certainly be possible to embed something like the Mondrian OLAP server (http://mondrian.pentaho.org/) and do some pretty cool reporting stuff. Very neat. Rich > /Rickard > > > _______________________________________________ > qi4j-dev mailing list > [email protected] > http://lists.ops4j.org/mailman/listinfo/qi4j-dev > _______________________________________________ qi4j-dev mailing list [email protected] http://lists.ops4j.org/mailman/listinfo/qi4j-dev
