I'm just curious why this is included in the 3.2.2 release? HDFS-15567 is tagged with 3.2.3 and the corresponding HDFS-14272 on server side is tagged with 3.3.0.
> If it goes into FS/FC, what does it do for a viewfs with >1 mounted HDFS? Should it take path, msync(path) so that viewFS knows where to forward it? The API shouldn't take any path - for viewFS I think it should call this on all the child file systems. It might also need to handle the case where some downstream clusters support this capability while others don't. > Options 1. I roll HDFS-15567 back "please be follow process" 2. Someone does a followup patch with specification and contract test, view FS. Add even more to the java 3. We do as per HADOOP-16898 into an MSyncable interface and then FileSystem & HDFS can implement. ViewFS and filterFS still need to pass through. I'm slightly in favor of the hasPathCapabilities approach and make this a mixin where FS impls can optionally support. Happy to hear what others think. Chao On Fri, Dec 11, 2020 at 9:00 AM Steve Loughran <ste...@cloudera.com.invalid> wrote: > Silence from the HDFS team > > > Hadoop 3.2.2 is in an RC; it has the new FS API call. I really don't want > to veto the release just because someone pulled up a method without doing > the due diligence. > > Is anyone in the HDFS going to do that due diligence or should we include > something in the release notes "msync()" must be considered unstable. > > Then we can do a proper msync(). > > If it goes into FS/FC, what does it do for a viewfs with >1 mounted HDFS? > Should it take path, msync(path) so that viewFS knows where to forward it? > > Alternatively: go with an MSync interface which those few FS which > implement it (hdfs) can do that, and the fact that it doesn't have doc or > tests won't be a blocker any more? > > -steve > > > > > On Thu, 10 Dec 2020 at 12:41, Steve Loughran <ste...@cloudera.com> wrote: > > > > > Gosh, has it really been only since february since I last asked the HDFS > > dev list to stop adding anything to FileSystem/FileContext APIs without > > > > * mentioning this on the hadoop-common list. > > * specifying what it does in filesystem.md > > * with a contract test > > * a new hasPathCapabilities probe. Throwing UnsupportedOperationException > > only lets people work out if it is unsupported through invocation. Being > > able to probe for it is better. > > * ViewFS support. > > * And, for any new API, one which works well for high-latency object > > stores: returning Future<Something> and Future<RemoteIterator<Something> > > when > 1 result is returned > > > > This needs to hold even for pulling something up from HDFS. Because if > > another FS wants to implement it, they need to know what it does, and > have > > tests to verify this. I say this as someone who has tried to document > HDFS > > rename() semantics and gave up. > > > > It's really frustrating that every time someone does an FS API change > like > > this in the past (most recently HDFS-13616) I am the one who has to keep > > sending the reminders out, and then having to try and clean up/. > > > > So what now? > > > > Options > > 1. I roll HDFS-15567 back "please be follow process" > > 2. Someone does a followup patch with specification and contract test, > > view FS. Add even more to the java > > 3. We do as per HADOOP-16898 into an MSyncable interface and then > > FileSystem & HDFS can implement. ViewFS and filterFS still need to pass > > through. > > > > *If nobody is going to volunteer for the specification/test changes, I'm > > happy for the rollback. It'll remind people about process, * > > > > Pre-emptive Warning: No matter what we do for this patch, I will roll > back > > the next change which adds a new API if it's not accompanied by > > specification and tests. > > > > Unhappily yours, > > > > Steve > > >
