Hi Ashish, I support that idea fully. I don't yet have any great ideas on how to pull it off - that would probably require more than a few experiments and design strategies, but what you describe sounds good. Would love to see a simple example of it, maybe to begin with just to implement it for a single aspect such as WHERE items or so.
Best regards, Kasper 2015-05-20 19:12 GMT+02:00 Ashish Mukherjee <[email protected]>: > Hello, > > I was looking through some of the DataContext classes like ElasticSearch > etc. for my needs and was working on the Solr class recently. It seems > that it is not uncommon not to push-down many querying operations because > implementing executeQuery() can be tricky, as every kind of SQL is > supported by executeQuery(). Therefore, some of the connectors may not > scale very well due to lot of in-memory processing of large data-sets on > the application side. > > I was wondering if the following design would simplify implementation of > executeQuery() in each DataContext implementation - > > Let QueryPostProcessDataContext expose few more granular hooks (like > executeCountQuery() already exists), such as createFilters(), > createHaving() etc. > > These can have default implementations in QueryPostProcessDataContext class > which are called in a pipeline pattern for the entire query construction > and execution. > > Those implementing classes which can/want to implement joins etc. natively > can do so else the functionality can be satisfied by the base class method > call in the pipeline. > > By following such a design, it may be easier to support as many query > functions natively as possible while leaving the rest to the MM framework. > Then, it would not be an all or nothing implementation for a data connector > and keep memory footprint of the application manageable even for large > data-sets. > > These are just high level thoughts as of now, which I wanted to bounce off > with the group. > > Regards, > Ashish >
