I prefer incremental updates. "Wipe off everything and insert everything" is quite straightforward, but it is not scalable when the metadata is increasing.
My idea about "incremental updates": As eagle can get messages from "hdfs", we can also filter all the "delete/modify" message to update metadata in HBase. Regards, Daniel -----Original Message----- From: Edward Zhang [mailto:[email protected]] Sent: Tuesday, January 12, 2016 11:40 PM To: [email protected] Subject: Re: Is "Sensitive file info" in HBase always up-to-date? Hi Daniel, Eagle does not have maintain this consistency. As sensitivity metadata normally comes from external tools by scanning file system, this consistency can be enforced by external tools. But looks we need think about metadata update algorithm. Do we wipe off everything and insert everything or just do incremental updates? Thanks Edward On Tue, Jan 12, 2016 at 7:01 PM, Daniel Zhou <[email protected]> wrote: > Just a short question: > > Now in HBase eagle have stored the info of sensitive files, including > the file name and sensitive type defined by user. > If these sensitive files in hadoop got deleted, how would this "delete" > action affect the info stored in the HBase ? would their record in HBase > also get deleted? (let's say I didn't create policies for these files) > > Regards, > Daniel >
