I liked this topic. A better name might be 'stream storage primitives', as we treat DL as a stream storage. Comments inline.
On Wed, Nov 9, 2016 at 3:09 AM, Gerrit Sundaram <gerritsunda...@gmail.com> wrote: > As what Sijie suggested in the other email thread, I started this email > thread for discussing the stream operation primitives. > > The stream operations that I am aware of that DL supports are > > * Open a distributedlog stream > * Delete a distributedlog stream > * List all the distributedlog streams under a namespace > Are you also looking for listing streams under a 'sub-namespace' - (or streams have common prefix)? (Based on my understanding on your proposal, you might need this for a filesystem-like API?) > * Seal a distributedlog stream > * Truncate a distributedlog stream > Just to clarify this, the 'truncate' in DL is to trim the head of the stream not the tail. The 'truncate' in filesystem world is to a size of precisely *length* bytes, it is truncating the tail. Make sure we clarified it and are on same page. > > I am looking for a more filesystem-like API. for example, > > * Get the status/attributes of a stream (like stat in filesystem) > +1 for stream status/attributes. I think we might actually already have this in DL. since in kestrel, we use that for storing customized metadata. It might make sense to formalize it into 'stream status'. > * Rename a stream > we've talked about this for a while. +1. > * Symlink a stream Symlink a stream is probably easy to do. +1 we've thought about that for having the flexibility to move stream between different storage backend. Symlink would help this. But a more fundamental thought here is symlinks for log segments. So when a symlinked stream is deleted, the underneath log segments might not be deleted until its link count decreased to zero. > > Another operations that I can think of might be useful. > > * Split/Fork a stream (it can be useful for dynamic data partitioning) > Split and fork a stream sounds interesting. But it sounds like a more high-level feature rather than storage primitives. Actually, it might be a good separate discussion feature. > * Merge/Concat streams > I think there is already one outstanding jira for concatenating two DL streams. Jia and Arvind are working on that. https://issues.apache.org/jira/browse/DL-46 > > The above operations are based on my knowledge about DL. Feel free to add > more. > > - Gerrit >
