Re: 1000's of CF's. virtual CFs possible Map/Reduce SOLUTION...

2012-10-02 Thread Hiller, Dean
Well, I think I know the direction we may follow so we can 1. Have Virtual CF's 2. Be able to map/reduce ONE Virtual CF Well, not map/reduce exactly but really really close. We use PlayOrm with it's partitioning so I am now thinking what we will do is have a compute grid where we can have each

Re: 1000's of CF's. virtual CFs possible Map/Reduce SOLUTION...

2012-10-02 Thread Brian O'Neill
Dean, We moved away from Hadoop and M/R, and instead we are using Storm as our compute grid. We queue keys in Kafka, then Storm distributes the work to the grid. Its working well so far, but we haven't taken it to prod yet. Data is read from Cassandra using a Cassandra-bolt. If you end up