Since we just make use of the general FileSystem API to do listing, is it possible to make use of ' bucket index listing'?
Andrew Purtell <[email protected]> 于2021年5月22日周六 上午6:34写道： > > > > On May 20, 2021, at 4:00 AM, Wellington Chevreuil < > [email protected]> wrote: > > > > > >> > >> > >> IMO it should be a file per store. > >> Per region is not suitable here as compaction is per store. > >> Per file means we still need to list all the files. And usually, after > >> compaction, we need to do an atomic operation to remove several old > files > >> and add a new file, or even several files for stripe compaction. It > will be > >> easy if we just write one file to commit these changes. > >> > > > > Fine for me if it's simpler. Mentioned the per file approach because I > > thought it could be easier/faster to do that, rather than having to > update > > the store file list on every flush. AFAIK, append is out of the table, so > > updating this file would mean read it, write original content plus new > > hfile to a temp file, delete original file, rename it). > > > > That sounds right to be. > > A minor potential optimization is the filename could have a timestamp > component, so a bucket index listing at that path would pick up a list > including the latest, and the latest would be used as the manifest of valid > store files. The cloud object store is expected to provide an atomic > listing semantic where the file is written and closed and only then is it > visible, and it is visible at once to everyone. (I think this is available > on most.) Old manifest file versions could be lazily deleted. > > > >> Em qui., 20 de mai. de 2021 às 02:57, 张铎(Duo Zhang) < > [email protected]> > >> escreveu: > >> > >> IIRC S3 is the only object storage which does not guarantee > >> read-after-write consistency in the past... > >> > >> This is the quick result after googling > >> > >> AWS [1] > >> > >>> Amazon S3 delivers strong read-after-write consistency automatically > for > >>> all applications > >> > >> > >> Azure[2] > >> > >>> Azure Storage was designed to embrace a strong consistency model that > >>> guarantees that after the service performs an insert or update > operation, > >>> subsequent read operations return the latest update. > >> > >> > >> Aliyun[3] > >> > >>> A feature requires that object operations in OSS be atomic, which > >>> indicates that operations can only either succeed or fail without > >>> intermediate states. To ensure that users can access only complete > data, > >>> OSS does not return corrupted or partial data. > >>> > >>> Object operations in OSS are highly consistent. For example, when a > user > >>> receives an upload (PUT) success response, the uploaded object can be > >> read > >>> immediately, and copies of the object are written to multiple devices > for > >>> redundancy. Therefore, the situations where data is not obtained when > you > >>> perform the read-after-write operation do not exist. The same is true > for > >>> delete operations. After you delete an object, the object and its > copies > >> no > >>> longer exist. > >>> > >> > >> GCP[4] > >> > >>> Cloud Storage provides strong global consistency for the following > >>> operations, including both data and metadata: > >>> > >>> Read-after-write > >>> Read-after-metadata-update > >>> Read-after-delete > >>> Bucket listing > >>> Object listing > >>> > >> > >> I think these vendors could cover most end users in the world? > >> > >> 1. https://aws.amazon.com/cn/s3/consistency/ > >> 2. > >> > >> > https://docs.microsoft.com/en-us/azure/storage/blobs/concurrency-manage?tabs=dotnet > >> 3. https://www.alibabacloud.com/help/doc-detail/31827.htm > >> 4. https://cloud.google.com/storage/docs/consistency > >> > >> Nick Dimiduk <[email protected]> 于2021年5月19日周三 下午11:40写道： > >> > >>> On Wed, May 19, 2021 at 8:19 AM 张铎(Duo Zhang) <[email protected]> > >>> wrote: > >>> > >>>> What about just storing the hfile list in a file? Since now S3 has > >> strong > >>>> consistency, we could safely overwrite a file then I think? > >>>> > >>> > >>> My concern is about portability. S3 isn't the only blob store in town, > >> and > >>> consistent read-what-you-wrote semantics are not a standard feature, as > >> far > >>> as I know. If we want something that can work on 3 or 5 major public > >> cloud > >>> blobstore products as well as a smattering of on-prem technologies, we > >>> should be selective about what features we choose to rely on as > >>> foundational to our implementation. > >>> > >>> Or we are explicitly saying this will only work on S3 and we'll only > >>> support other services when they can achieve this level of > compatibility. > >>> > >>> Either way, we should be clear and up-front about what semantics we > >> demand. > >>> Implementing some kind of a test harness that can check compatibility > >> would > >>> help here, a similar effort to that of defining standard behaviors of > >> HDFS > >>> implementations. > >>> > >>> I love this discussion :) > >>> > >>> And since the hfile list file will be very small, renaming will not be > a > >>>> big problem. > >>>> > >>> > >>> Would this be a file per store? A file per region? Ah. Below you imply > >> it's > >>> per store. > >>> > >>> Wellington Chevreuil <[email protected]> 于2021年5月19日周三 > >>>> 下午10:43写道： > >>>> > >>>>> Thank you, Andrew and Duo, > >>>>> > >>>>> Talking internally with Josh Elser, initial idea was to rebase the > >>>> feature > >>>>> branch with master (in order to catch with latest commits), then > >> focus > >>> on > >>>>> work to have a minimal functioning hbase, in other words, together > >> with > >>>> the > >>>>> already committed work from HBASE-25391, make sure flush, > >> compactions, > >>>>> splits and merges all can take advantage of the persistent store file > >>>>> manager and complete with no need to rely on renames. These all map > >> to > >>>> the > >>>>> substasks HBASE-25391, HBASE-25392 and HBASE-25393. Once we could > >> test > >>>> and > >>>>> validate this works well for our goals, we can then focus on > >> snapshots, > >>>>> bulkloading and tooling. > >>>>> > >>>>> S3 now supports strong consistency, and I heard that they are also > >>>>>> implementing atomic renaming currently, so maybe that's one of the > >>>>> reasons > >>>>>> why the development is silent now.. > >>>>>> > >>>>> Interesting, I had no idea this was being implemented. I know, > >>> however, a > >>>>> version of this feature is already available on latest EMR releases > >> (at > >>>>> least from 6.2.0), and AWS team has published their own blog post > >> with > >>>>> their results: > >>>>> > >>>>> > >>>> > >>> > >> > https://aws.amazon.com/blogs/big-data/amazon-emr-6-2-0-adds-persistent-hfile-tracking-to-improve-performance-with-hbase-on-amazon-s3/ > >>>>> > >>>>> But I do not think store hfile list in meta is the only solution. It > >>> will > >>>>>> cause cyclic dependencies for hbase:meta, and then force us a have > >> a > >>>>>> fallback solution which makes the code a bit ugly. We should try to > >>> see > >>>>> if > >>>>>> this could be done with only the FileSystem. > >>>>>> > >>>>> This is indeed a relevant concern. One idea I had mentioned in the > >>>> original > >>>>> design doc was to track committed/non-committed files through xattr > >> (or > >>>>> tags), which may have its own performance issues as explained by > >>> Stephen > >>>>> Wu, but is something that could be attempted. > >>>>> > >>>>> Em qua., 19 de mai. de 2021 às 04:56, 张铎(Duo Zhang) < > >>>> [email protected] > >>>>>> > >>>>> escreveu: > >>>>> > >>>>>> S3 now supports strong consistency, and I heard that they are also > >>>>>> implementing atomic renaming currently, so maybe that's one of the > >>>>> reasons > >>>>>> why the development is silent now... > >>>>>> > >>>>>> For me, I also think deploying hbase on cloud storage is the > >> future, > >>>> so I > >>>>>> would also like to participate here. > >>>>>> > >>>>>> But I do not think store hfile list in meta is the only solution. > >> It > >>>> will > >>>>>> cause cyclic dependencies for hbase:meta, and then force us a have > >> a > >>>>>> fallback solution which makes the code a bit ugly. We should try to > >>> see > >>>>> if > >>>>>> this could be done with only the FileSystem. > >>>>>> > >>>>>> Thanks. > >>>>>> > >>>>>> Andrew Purtell <[email protected]> 于2021年5月19日周三 上午8:04写道： > >>>>>> > >>>>>>> Wellington (and et. al), > >>>>>>> > >>>>>>> S3 is also an important piece of our future production plans. > >>>>>>> Unfortunately, we were unable to assist much with last year's > >>> work, > >>>> on > >>>>>>> account of being sidetracked by more immediate concerns. > >>> Fortunately, > >>>>>> this > >>>>>>> renewed interest is timely in that we have an HBase 2 project > >>> where, > >>>> if > >>>>>>> this can land in a 2.5 or a 2.6, it could be an important cost to > >>>> serve > >>>>>>> optimization, and one we could and would make use of. Therefore I > >>>> would > >>>>>>> like to restate my employer's interest in this work too. It may > >>> just > >>>> be > >>>>>>> Viraj and myself in the early days. > >>>>>>> > >>>>>>> I'm not sure how best to collaborate. We could review changes > >> from > >>>> the > >>>>>>> original authors, new changes, and/or divide up the development > >>>> tasks. > >>>>> We > >>>>>>> can certainly offer our time for testing, and can afford the > >> costs > >>> of > >>>>>>> testing against the S3 service. > >>>>>>> > >>>>>>> > >>>>>>> On Tue, May 18, 2021 at 12:16 PM Wellington Chevreuil < > >>>>>>> [email protected]> wrote: > >>>>>>> > >>>>>>>> Greetings everyone, > >>>>>>>> > >>>>>>>> HBASE-24749 has been proposed almost a year ago, introducing a > >>> new > >>>>>>>> StoreFile tracker as a way to allow for any hbase hfile > >>>> modifications > >>>>>> to > >>>>>>> be > >>>>>>>> safely completed without needing a file system rename. This > >> seems > >>>>>> pretty > >>>>>>>> relevant for deployments over S3 file systems, where rename > >>>>> operations > >>>>>>> are > >>>>>>>> not atomic and can have a performance degradation when multiple > >>>>>> requests > >>>>>>>> get concurrently submitted to the same bucket. We had done > >>>>> superficial > >>>>>>>> tests and ycsb runs, where individual renames of files larger > >>> than > >>>>> 5GB > >>>>>>> can > >>>>>>>> take a few hundreds of seconds to complete. We also observed > >>>> impacts > >>>>> in > >>>>>>>> write loads throughput, the bottleneck potentially being the > >>>> renames. > >>>>>>>> > >>>>>>>> With S3 being an important piece of my employer cloud solution, > >>> we > >>>>>> would > >>>>>>>> like to help it move forward. We plan to contribute new patches > >>> per > >>>>> the > >>>>>>>> original design/Jira, but we’d also be happy to review changes > >>> from > >>>>> the > >>>>>>>> original authors, too. Please let us know if anyone has any > >>>> concerns, > >>>>>>>> otherwise we’ll start to self-assign issues on HBASE-24749 > >>>>>>>> > >>>>>>>> Wellington > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> Best regards, > >>>>>>> Andrew > >>>>>>> > >>>>>>> Words like orphans lost among the crosstalk, meaning torn from > >>>> truth's > >>>>>>> decrepit hands > >>>>>>> - A23, Crosstalk > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> >
