I have a paging model whereby we stream data from CS by fetching 'pages' thereby reading (sequentially) entire datasets.
We're using the bucket approach where we write data for 5 minutes, then we can just fetch the bucket for that range. Our app now has TONS of data and we have a piece of middleware that filters it based on the client requests. So if they only want english they just get english and filter away about 60% of our data. but it doesn't support condition pushdown. So ALL this data has to be sent from our CS boxes to our middleware and filtered there (wasting a lot of network IO). Is there away (including refactoring the code) that I could push this this into CS? Maybe some way I could discovery the CS topology and put daemons on each of our CS boxes and fetch from CS directly (doing the filtering there). Thoughts? -- We’re hiring if you know of any awesome Java Devops or Linux Operations Engineers! Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile <https://plus.google.com/102718274791889610666/posts>
