Is the batch wise processing implemented or is all the data fetched still? On Fri, Aug 24, 2012 at 6:20 PM, Anjana Fernando <[email protected]> wrote:
> Hi, > > IMO, we just should not be doing the counting of the records. So > basically, you will not get the record count by default. But we can add it > as an option for the Cassandra explorer, to get the whole row count of a CF > when the user request it, i.e. pressing a button. So with this, pagination > will not be supported by default, and just give the user to advance a batch > of records with a "next" button. And also, it would be great, if we can > integrate CQL support for the explorer, so the user can filter the data in > a specific way and get the resultant records. > > Cheers, > Anjana. > > On Fri, Aug 24, 2012 at 6:01 PM, Shelan Perera <[email protected]> wrote: > >> Hi, >> >> Cassandra Explorer has been testing with 7 million row entries with BAM >> data and it gives timeout errors with such a load. The main reason for this >> is calculating the total no of rows >> to show how many row entries are available and to enable full numbered >> pagination. In Cassandra calculating total no of Rows is an anti pattern >> but that is the key information which is used >> heavily to verify inserted data by application. Almost all the available >> tools are using a limit such as 10000 rows as the limit and not going for >> total records. >> >> I have tried fetching records as batches (10000 each and 100,000 each on >> different occasions) but to complete a limit like 7 million it takes a >> considerable time. When i googled they have advised it is not a good >> idea to calculate the total row count as it can take really a long time >> to fetch all the records in a cluster and recommended to load off it to >> something such as a map reduce job. >> >> to fetch 100,000 records it took 2.73 Seconds. So all together it takes >> around 191 seconds to complete it. >> >> What would be the best way to overcome this ? >> >> Thanks >> >> -- >> *Shelan Perera* >> >> Software Engineer >> ** >> *WSO2, Inc. : wso2.com* >> lean.enterprise.middleware. >> >> *Home Page* : shelan.org >> *Blog* : blog.shelan.org >> *Linked-i*n :http://www.linkedin.com/pub/shelan-perera/a/194/465 >> *Twitter* :https://twitter.com/#!/shelan >> >> *Mobile* : +94 772 604 402 >> >> >> >> _______________________________________________ >> Dev mailing list >> [email protected] >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > > -- > *Anjana Fernando* > Associate Technical Lead > WSO2 Inc. | http://wso2.com > lean . enterprise . middleware > > _______________________________________________ > Dev mailing list > [email protected] > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- Regards, Tharindu blog: http://mackiemathew.com/ M: +94777759908
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
