Re: [Gluster-devel] HTIME API
Not exactly. The idea is to make use of information in the meta file (list of journals) so as to not actively maintain the backlog. The API would return a set of journals (or pathnames) starting from a given timestamp. This is similar to history, but used for live changes. Venky On 10/31/2014 11:10 AM, Kotresh Hiremath Ravishankar wrote: Hi Venky, If I understand correctly, the idea is to provide RPC based history API? If yes, how can that solve the different clients/consumers having to maintain state of how much it has processed? Thanks and Regards, Kotresh H R - Original Message - From: Venky Shankar vshan...@redhat.com To: Ajeet Jha a...@redhat.com Cc: Kotresh Hiremath Ravishankar khire...@redhat.com, gluster-devel@gluster.org Sent: Wednesday, October 29, 2014 6:23:40 PM Subject: HTIME API Hey Ajeet, Since changelog translator maintains a meta file (HTIME) containing a list of consumable changelogs, we could leverage this to provide an API invokable by libgfchangelog (once the interaction is RPC based) to get a list of changelogs for a given time period (similar to what history API does today). The benefit that this provides is that the consumers[1] need not maintain state (the API takes care of that) of the amount of changelogs [to be] processed and make use of this API to get freshly consumable changelogs. Thoughts? [1]: esp for lowlevel API (http://gluster.org/community/documentation/index.php/Features/Gnotify) Venky ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] HTIME API
Since HTIME file is available we need not maintain changelogs in two different places as we do now(backend and working dir), We can provide backend changelog names to consumer instead of working dir names. One problem with this is for parsing these changelogs they have to use one more RPC call. How about returning FOP details directly instead of changelog file names? If we use db(sqlite) internally in API itself then we can do multiple optimization. 1. If multiple data operations in requested time for a file then return only one 2. If a file/gfid is created and deleted in the requested time range then skip from the output. 3. If a file/gfid is deleted in the given time then do not return data fops. Other challenge is getting cluster view of these notifications.(and RENAME issues) -- regards Aravinda On 10/31/2014 11:10 AM, Kotresh Hiremath Ravishankar wrote: Hi Venky, If I understand correctly, the idea is to provide RPC based history API? If yes, how can that solve the different clients/consumers having to maintain state of how much it has processed? Thanks and Regards, Kotresh H R - Original Message - From: Venky Shankar vshan...@redhat.com To: Ajeet Jha a...@redhat.com Cc: Kotresh Hiremath Ravishankar khire...@redhat.com, gluster-devel@gluster.org Sent: Wednesday, October 29, 2014 6:23:40 PM Subject: HTIME API Hey Ajeet, Since changelog translator maintains a meta file (HTIME) containing a list of consumable changelogs, we could leverage this to provide an API invokable by libgfchangelog (once the interaction is RPC based) to get a list of changelogs for a given time period (similar to what history API does today). The benefit that this provides is that the consumers[1] need not maintain state (the API takes care of that) of the amount of changelogs [to be] processed and make use of this API to get freshly consumable changelogs. Thoughts? [1]: esp for lowlevel API (http://gluster.org/community/documentation/index.php/Features/Gnotify) Venky ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] HTIME API
On 10/31/2014 12:28 PM, Aravinda wrote: Since HTIME file is available we need not maintain changelogs in two different places as we do now(backend and working dir), We can provide backend changelog names to consumer instead of working dir names. One problem with this is for parsing these changelogs they have to use one more RPC call. How about returning FOP details directly instead of changelog file names? Nicely put Aravinda. That's exactly what's planned for Gnotify (lowlevel APIs), where the consumer does not have to _know_ the changelog format. The fops are notified (via callback) in a way similar to inotify(7) in an event buffer. Avoiding maintaining state given that we keep performance on par would be a huge plus point. Further, we could relax ordering and have callback invoked from multiple threads for cases when ordering does not really matter. If we use db(sqlite) internally in API itself then we can do multiple optimization. 1. If multiple data operations in requested time for a file then return only one 2. If a file/gfid is created and deleted in the requested time range then skip from the output. 3. If a file/gfid is deleted in the given time then do not return data fops. Other challenge is getting cluster view of these notifications.(and RENAME issues) Correct. Cluster view needs a good amount of thinking w.r.t functionality, performance, etc. -- regards Aravinda On 10/31/2014 11:10 AM, Kotresh Hiremath Ravishankar wrote: Hi Venky, If I understand correctly, the idea is to provide RPC based history API? If yes, how can that solve the different clients/consumers having to maintain state of how much it has processed? Thanks and Regards, Kotresh H R - Original Message - From: Venky Shankar vshan...@redhat.com To: Ajeet Jha a...@redhat.com Cc: Kotresh Hiremath Ravishankar khire...@redhat.com, gluster-devel@gluster.org Sent: Wednesday, October 29, 2014 6:23:40 PM Subject: HTIME API Hey Ajeet, Since changelog translator maintains a meta file (HTIME) containing a list of consumable changelogs, we could leverage this to provide an API invokable by libgfchangelog (once the interaction is RPC based) to get a list of changelogs for a given time period (similar to what history API does today). The benefit that this provides is that the consumers[1] need not maintain state (the API takes care of that) of the amount of changelogs [to be] processed and make use of this API to get freshly consumable changelogs. Thoughts? [1]: esp for lowlevel API (http://gluster.org/community/documentation/index.php/Features/Gnotify) Venky ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] HTIME API
Hi Venky, If I understand correctly, the idea is to provide RPC based history API? If yes, how can that solve the different clients/consumers having to maintain state of how much it has processed? Thanks and Regards, Kotresh H R - Original Message - From: Venky Shankar vshan...@redhat.com To: Ajeet Jha a...@redhat.com Cc: Kotresh Hiremath Ravishankar khire...@redhat.com, gluster-devel@gluster.org Sent: Wednesday, October 29, 2014 6:23:40 PM Subject: HTIME API Hey Ajeet, Since changelog translator maintains a meta file (HTIME) containing a list of consumable changelogs, we could leverage this to provide an API invokable by libgfchangelog (once the interaction is RPC based) to get a list of changelogs for a given time period (similar to what history API does today). The benefit that this provides is that the consumers[1] need not maintain state (the API takes care of that) of the amount of changelogs [to be] processed and make use of this API to get freshly consumable changelogs. Thoughts? [1]: esp for lowlevel API (http://gluster.org/community/documentation/index.php/Features/Gnotify) Venky ___ Gluster-devel mailing list Gluster-devel@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-devel
