apologies, copying back in common-dev@ with my question about the code. On Tue, May 15, 2018 at 2:36 PM, Sean Busbey <bus...@cloudera.com> wrote:
> > Internal constraints prevented this feature from being developed in > Apache, so we want to ensure that all the code is discussed, maintainable, > and documented by the community before it merges. > > Has this code gone through ASF IP Clearance already? > > On Tue, May 15, 2018 at 10:34 AM, Steve Loughran <ste...@hortonworks.com> > wrote: > >> Hi >> >> Chris Douglas I and I've have a proposal for a short-lived feature branch >> for the Azure ABFS connector to go into the hadoop-azure package. This will >> connect to the new azure storage service, which will ultimately replace the >> one used by wasb. It's a big patch and, like all storage connectors, will >> inevitably take time to stabilize (i.e: nobody ever get seek() right, even >> when we think we have). >> >> Thomas & Esfandiar will do the coding: they've already done the >> paperwork. Chris, myself & anyone else interested can be involved in the >> review and testing. >> >> Comments? >> >> ------------- >> >> The initial HADOOP-15407 patch contains a new filesystem client for the >> forthcoming Azure ABFS, which is intended to replace Azure WASB as the >> Azure storage layer. The patch is large, as it contains the replacement >> client, tests, and generated code. >> >> We propose a feature branch, so the module can be broken into salient, >> reviewable chunks. Internal constraints prevented this feature from being >> developed in Apache, so we want to ensure that all the code is discussed, >> maintainable, and documented by the community before it merges. >> >> To effect this, we also propose adding two developers as branch >> committers: Thomas Marquardt tm...@microsoft.com<mailto:tma >> r...@microsoft.com> Esfandiar Manii esma...@microsoft.com<mailto:e >> sma...@microsoft.com> >> >> Beyond normal feature branch activity and merge criteria for FS modules, >> we want to add another merge criterion for ABFS. Some of the client APIs >> are not GA. It seems reasonable to require that this client works with >> public endpoints before it merges to trunk. >> >> To test the Blob FS driver, Blob FS team (including Esfandiar Manii and >> Thomas Marquardt) in Azure Storage will need the MSDN subscription ID(s) >> for all reviewers who want to run the tests. The ABFS team will then >> whitelist the subscription ID(s) for the Blob FS Preview. At that time, >> future storage accounts created will have the Blob FS endpoint, >> <accountName>.dfs.core.windows.net<http://dfs.core.windows.net>, which >> the Blob FS driver relies on. >> >> This is a temporary state during the (current) Private Preview and the >> early phases of Public Preview. In a few months, the whitelisting will not >> be required and anyone will be able to create a storage account with access >> to the Blob FS endpoint. >> >> Thomas and Esfandiar have been active in the Hadoop project working on >> the WASB connector (see https://issues.apache.org/jira >> /browse/HADOOP-14552). They understand the processes and requirements of >> the software. Working on the branch directly will let them bring this >> significant feature into the hadoop-azure module without disrupting >> existing users. >> > > > > -- > busbey > -- busbey
