I think I am OK with this. This will definitely fix the naming. I don't think IgfsHadoopFileSystem will ever be used directly, so the methods you are referring to will never be called.
D. On Fri, Feb 27, 2015 at 8:48 AM, Vladimir Ozerov <[email protected]> wrote: > How would you work with the secondary file system from Igfs in this case? > Lets say, you run Hadoop jobs which delegate to our IGFS and you want to > write-through the secondary file system. In this case we will have the > following flow: > 1) Hadoop -> FileSystem.get("igfs://") -> 2) IgfsHadoopFileSystem -> 3) > IgniteFs -> 4) IgfsHadoopFileSystemWrapper -> 5) actual FileSystem. > > 1) Hadoop get's instance of a file system through static get() method. > 2) We return our implementation of FileSystem called IgfsHadoopFileSystem. > 3) This implementation actually delegates to IgniteFs > 4) If secondary file system is configured, IgniteFs wirte/read-through the > secondary Igfs, which is IgfsHadoopFileSystemWrapper in our case. > 5) IgfsHadoopFileSystemWrapper, in turn, delegates to the real backing > Hadoop file system. > > So, the problem is that: > IgniteFs implements Igfs > IgfsHadoopFileSystemWrapper implements Igfs > > If we remove Igfs, then we will have to do the following: > "IgfsHadoopFileSystemWrapper extends IgniteFs" and all IgniteFs methods > which are not in Igfs will be implemented as "throw new > UnsupportedOperationException()". If we are ok with this, then let's remove > Igfs. > > Vladimir. > > On Fri, Feb 27, 2015 at 4:35 PM, Dmitriy Setrakyan <[email protected]> > wrote: > > > If possible, I would get rid of Igfs API and just keep IgniteFs. It is > very > > confusing now. > > > > D. > > > > On Fri, Feb 27, 2015 at 8:31 AM, Vladimir Ozerov <[email protected]> > > wrote: > > > > > 1) "Igfs" is an interface with several common file-system methods like > > > "create", "mkdirs", etc. > > > 2) "IgniteFs" is extension of "Igfs" which adds some advanced methods > > like > > > "execute()", "affinity()", etc. I.e. this is more feature-rich "Igfs". > > > > > > Earlier Igfs and IgniteFs were single interface called "GridGgfs". And > > > GridGgfs worked with secondary Hadoop FileSystem directly. This > approach > > > had one drawback - GridGgfs wihch is located in core module had > > references > > > to Hadoop libraries. > > > To get rid of these dependencies we had to extract some common > > file-system > > > methods from GridGgfs and name them "Igfs". And now secondary file > system > > > is not Hadoop FileSystem, but just another Igfs. And the rest > > > Ignite-specific rich API is now in "IgniteFs". > > > > > > On Fri, Feb 27, 2015 at 3:52 PM, Dmitriy Setrakyan < > > [email protected]> > > > wrote: > > > > > > > Now I am even more confused. Why do we have IgniteFs and IGFS APIs? > > What > > > is > > > > the point of having Igfs? What is the difference between the two? > > > > > > > > D. > > > > > > > > On Fri, Feb 27, 2015 at 4:11 AM, Vladimir Ozerov < > [email protected] > > > > > > > wrote: > > > > > > > > > My proposal can be found above: > > > > > > 1) *Igfs* is left as is - this is base interface; > > > > > > 2) IgniteFS is renamed to "*IgniteIgfs*", meaning that this is > > > Ignite's > > > > > implementation of IGFS; > > > > > > 3) IgfsHadoopFileSystemWrapper is renamed to "*HadoopIgfs*", > > meaning > > > > that > > > > > this another IGFS implementation which works over Hadoop > FileSystem; > > > > > > 4) IgfsHadoopFileSystem is renamed to *IgniteFileSystem*, meaning > > > that > > > > > this is Ignite's implementation of Hadoop FileSystem API. > > > > > > > > > > With such namings we will clearly indicate tow things: > > > > > - Everything, what ends with "Igfs" is IGFS implementation > > (IgniteIgfs > > > - > > > > > our in-memory impl, HadoopIgfs - another impl working over Hadoop > > > > > FileSystem); > > > > > - Everything, what has "FileSystem" in name is Hadoop FileSystem > > > > > implementation (IgniteFileSystem - *Ignite*'s implementation of > > > > > *FileSystem* > > > > > ). > > > > > > > > > > On Fri, Feb 27, 2015 at 8:44 AM, Dmitriy Setrakyan < > > > > [email protected]> > > > > > wrote: > > > > > > > > > > > Something definitely smells about the naming here. It is vague > and > > > > > > confusing. If it is only used as a secondary file system > > > configuration, > > > > > > maybe we should name it as such. > > > > > > > > > > > > Can you propose another name? > > > > > > > > > > > > D. > > > > > > > > > > > > On Fri, Feb 27, 2015 at 12:39 AM, Vladimir Ozerov < > > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > This wrapper is one of Igfs implementations. If user want to > > > > configure > > > > > > > secondary Igfs which is backed by Hadoop FileSystem, he should > > > create > > > > > > that > > > > > > > "wrapper" and set it as "secondaryFIleSystem" in > > > "IgfsConfiguration". > > > > > > > > > > > > > > On Fri, Feb 27, 2015 at 8:32 AM, Dmitriy Setrakyan < > > > > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > Sounds like something is wrong. I am still not clear why > should > > > our > > > > > > users > > > > > > > > even know about the wrapper. Can you please explain? > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > On Thu, Feb 26, 2015 at 8:56 AM, Vladimir Ozerov < > > > > > [email protected] > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > IgfsHadoopFileSystem is implementation of FileSystem > (Hadoop > > > > > > > interface). > > > > > > > > > IgfsHadoopFileSystemWrapper is implementation of Igfs which > > > > > delegates > > > > > > > > IGFS > > > > > > > > > API calls to underlying Hadoop FileSystem. > > > > > > > > > > > > > > > > > > On Thu, Feb 26, 2015 at 4:39 PM, Dmitriy Setrakyan < > > > > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Vova, > > > > > > > > > > > > > > > > > > > > I agree that it smells. Can you explain again the > > difference > > > > > > between > > > > > > > > > > IgfsHadoopFileSystem and IgfsHadoopFileSystemWrapper? > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > On Thu, Feb 26, 2015 at 5:21 AM, Vladimir Ozerov < > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > HI, > > > > > > > > > > > > > > > > > > > > > > Currently we have IgfsHadoopFileSystemWrapper class > whose > > > > sole > > > > > > > > purpose > > > > > > > > > is > > > > > > > > > > > to instantiate Hadoop FileSystem and delegate Igfs > calls > > to > > > > it. > > > > > > We > > > > > > > > use > > > > > > > > > > this > > > > > > > > > > > class to configure secondary Hadoop file system for > Igfs. > > > > > > > > > > > > > > > > > > > > > > It seems to me that "Wrapper" is wrong suffix here from > > > user > > > > > > > > > perspective, > > > > > > > > > > > as this is not a wrapper, but a functional unit which > > user > > > > uses > > > > > > in > > > > > > > > > > > configuration. It does not "wraps" anything from user > > point > > > > of > > > > > > > view. > > > > > > > > > > > > > > > > > > > > > > Furthermore, we do have public classes > > IgfsHadoopFileSystem > > > > > which > > > > > > > > have > > > > > > > > > no > > > > > > > > > > > realtion to IgfsHadoopFileSystemWrapper, what will also > > > > confuse > > > > > > > > users. > > > > > > > > > > > > > > > > > > > > > > We need to think about another class name here. May be > > > > > > > > > > > "IgfsHadoopFileSystemDelegator" or something like this? > > > > > > > > > > > > > > > > > > > > > > Any thoughts? > > > > > > > > > > > > > > > > > > > > > > Vladimir. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
