Hi,
On Wed, Sep 4, 2013 at 11:22 AM, Shariq Muhammed <[email protected]> wrote: > On Wed, Sep 4, 2013 at 10:58 AM, Bhathiya Jayasekara <[email protected]>wrote: > >> Hi all, >> >> How about deleting archived data? As I remember, this requirement came up >> recently. IMO it's good to have a archive-delete feature which can >> be scheduled. >> > > Once the data is archived to an external Cassandra ring, we should have > the option to delete the archived data from the primary Cassandra cluster > (where raw stats are being published), > That is what happening now also, the data will be removed from the original column family and archived to another column family. This feature is basically purging and archiving. But yes, we can also have purging and archiving seperately, and giving the user to select whether he/she needs, purging/archiving/both. so that the new analytics job doesn't have to process the already analyzed > / old data ... Or else time taken to complete map-reduce jobs will grow > with the data growth ... > yes and currently we are working on incremental processing feature for BAM (refer thread 'Incremental Data Processing for BAM'), which will provide the capability to process only the unprocessed data. Thanks, Sinthuja. > >> Thanks, >> Bhathiya >> >> >> On Wed, Sep 4, 2013 at 10:18 AM, Malith Dhanushka <[email protected]>wrote: >> >>> Hi folks, >>> >>> We(BAM team, Sumedha) had a discussion about the $Subject and >>> following are the suggested improvements for the Cassandra data archival >>> feature in BAM. >>> >>> - Remove hive script based archiving and use hector API to directly >>> issue archive queries to Cassandra (Current implementation is >>> based on hive where it generates hive script and archiving process uses >>> map-reduce jobs to achieve the task and it has a limitation of discarding >>> custom key value pares in column family) >>> >>> - Use Task component for scheduling purposes >>> >>> - Archive data to external Cassandra ring >>> >>> - Major UI improvements >>> - List the current archiving tasks >>> - Edit, Remove and Schedule archiving tasks >>> - Add new archiving task >>> >>> If there is any additional requirements please raise. >>> >>> Thanks, >>> Malith >>> -- >>> Malith Dhanushka >>> >>> Engineer - Data Technologies >>> *WSO2, Inc. : wso2.com* >>> >>> *Mobile* : +94 716 506 693 >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> *Bhathiya Jayasekara* >> *Software Engineer,* >> *WSO2 inc., http://wso2.com* >> * >> * >> *Phone: +94715478185* >> *LinkedIn: http://www.linkedin.com/in/bhathiyaj* >> *Twitter: https://twitter.com/bhathiyax* >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > Thanks, > Shariq. > Phone: +94 777 202 225 > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- *Sinthuja Rajendran* Software Engineer <http://wso2.com/> WSO2, Inc.:http://wso2.com Blog: http://sinthu-rajan.blogspot.com/ Mobile: +94774273955
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
