How would you work with the secondary file system from Igfs in this case?
Lets say, you run Hadoop jobs which delegate to our IGFS and you want to
write-through the secondary file system. In this case we will have the
following flow:
1) Hadoop -> FileSystem.get("igfs://") -> 2) IgfsHadoopFileSystem -> 3)
IgniteFs -> 4) IgfsHadoopFileSystemWrapper -> 5) actual FileSystem.1) Hadoop get's instance of a file system through static get() method. 2) We return our implementation of FileSystem called IgfsHadoopFileSystem. 3) This implementation actually delegates to IgniteFs 4) If secondary file system is configured, IgniteFs wirte/read-through the secondary Igfs, which is IgfsHadoopFileSystemWrapper in our case. 5) IgfsHadoopFileSystemWrapper, in turn, delegates to the real backing Hadoop file system. So, the problem is that: IgniteFs implements Igfs IgfsHadoopFileSystemWrapper implements Igfs If we remove Igfs, then we will have to do the following: "IgfsHadoopFileSystemWrapper extends IgniteFs" and all IgniteFs methods which are not in Igfs will be implemented as "throw new UnsupportedOperationException()". If we are ok with this, then let's remove Igfs. Vladimir. On Fri, Feb 27, 2015 at 4:35 PM, Dmitriy Setrakyan <[email protected]> wrote: > If possible, I would get rid of Igfs API and just keep IgniteFs. It is very > confusing now. > > D. > > On Fri, Feb 27, 2015 at 8:31 AM, Vladimir Ozerov <[email protected]> > wrote: > > > 1) "Igfs" is an interface with several common file-system methods like > > "create", "mkdirs", etc. > > 2) "IgniteFs" is extension of "Igfs" which adds some advanced methods > like > > "execute()", "affinity()", etc. I.e. this is more feature-rich "Igfs". > > > > Earlier Igfs and IgniteFs were single interface called "GridGgfs". And > > GridGgfs worked with secondary Hadoop FileSystem directly. This approach > > had one drawback - GridGgfs wihch is located in core module had > references > > to Hadoop libraries. > > To get rid of these dependencies we had to extract some common > file-system > > methods from GridGgfs and name them "Igfs". And now secondary file system > > is not Hadoop FileSystem, but just another Igfs. And the rest > > Ignite-specific rich API is now in "IgniteFs". > > > > On Fri, Feb 27, 2015 at 3:52 PM, Dmitriy Setrakyan < > [email protected]> > > wrote: > > > > > Now I am even more confused. Why do we have IgniteFs and IGFS APIs? > What > > is > > > the point of having Igfs? What is the difference between the two? > > > > > > D. > > > > > > On Fri, Feb 27, 2015 at 4:11 AM, Vladimir Ozerov <[email protected] > > > > > wrote: > > > > > > > My proposal can be found above: > > > > > 1) *Igfs* is left as is - this is base interface; > > > > > 2) IgniteFS is renamed to "*IgniteIgfs*", meaning that this is > > Ignite's > > > > implementation of IGFS; > > > > > 3) IgfsHadoopFileSystemWrapper is renamed to "*HadoopIgfs*", > meaning > > > that > > > > this another IGFS implementation which works over Hadoop FileSystem; > > > > > 4) IgfsHadoopFileSystem is renamed to *IgniteFileSystem*, meaning > > that > > > > this is Ignite's implementation of Hadoop FileSystem API. > > > > > > > > With such namings we will clearly indicate tow things: > > > > - Everything, what ends with "Igfs" is IGFS implementation > (IgniteIgfs > > - > > > > our in-memory impl, HadoopIgfs - another impl working over Hadoop > > > > FileSystem); > > > > - Everything, what has "FileSystem" in name is Hadoop FileSystem > > > > implementation (IgniteFileSystem - *Ignite*'s implementation of > > > > *FileSystem* > > > > ). > > > > > > > > On Fri, Feb 27, 2015 at 8:44 AM, Dmitriy Setrakyan < > > > [email protected]> > > > > wrote: > > > > > > > > > Something definitely smells about the naming here. It is vague and > > > > > confusing. If it is only used as a secondary file system > > configuration, > > > > > maybe we should name it as such. > > > > > > > > > > Can you propose another name? > > > > > > > > > > D. > > > > > > > > > > On Fri, Feb 27, 2015 at 12:39 AM, Vladimir Ozerov < > > > [email protected]> > > > > > wrote: > > > > > > > > > > > This wrapper is one of Igfs implementations. If user want to > > > configure > > > > > > secondary Igfs which is backed by Hadoop FileSystem, he should > > create > > > > > that > > > > > > "wrapper" and set it as "secondaryFIleSystem" in > > "IgfsConfiguration". > > > > > > > > > > > > On Fri, Feb 27, 2015 at 8:32 AM, Dmitriy Setrakyan < > > > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Sounds like something is wrong. I am still not clear why should > > our > > > > > users > > > > > > > even know about the wrapper. Can you please explain? > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > On Thu, Feb 26, 2015 at 8:56 AM, Vladimir Ozerov < > > > > [email protected] > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > IgfsHadoopFileSystem is implementation of FileSystem (Hadoop > > > > > > interface). > > > > > > > > IgfsHadoopFileSystemWrapper is implementation of Igfs which > > > > delegates > > > > > > > IGFS > > > > > > > > API calls to underlying Hadoop FileSystem. > > > > > > > > > > > > > > > > On Thu, Feb 26, 2015 at 4:39 PM, Dmitriy Setrakyan < > > > > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Vova, > > > > > > > > > > > > > > > > > > I agree that it smells. Can you explain again the > difference > > > > > between > > > > > > > > > IgfsHadoopFileSystem and IgfsHadoopFileSystemWrapper? > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > On Thu, Feb 26, 2015 at 5:21 AM, Vladimir Ozerov < > > > > > > [email protected] > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > HI, > > > > > > > > > > > > > > > > > > > > Currently we have IgfsHadoopFileSystemWrapper class whose > > > sole > > > > > > > purpose > > > > > > > > is > > > > > > > > > > to instantiate Hadoop FileSystem and delegate Igfs calls > to > > > it. > > > > > We > > > > > > > use > > > > > > > > > this > > > > > > > > > > class to configure secondary Hadoop file system for Igfs. > > > > > > > > > > > > > > > > > > > > It seems to me that "Wrapper" is wrong suffix here from > > user > > > > > > > > perspective, > > > > > > > > > > as this is not a wrapper, but a functional unit which > user > > > uses > > > > > in > > > > > > > > > > configuration. It does not "wraps" anything from user > point > > > of > > > > > > view. > > > > > > > > > > > > > > > > > > > > Furthermore, we do have public classes > IgfsHadoopFileSystem > > > > which > > > > > > > have > > > > > > > > no > > > > > > > > > > realtion to IgfsHadoopFileSystemWrapper, what will also > > > confuse > > > > > > > users. > > > > > > > > > > > > > > > > > > > > We need to think about another class name here. May be > > > > > > > > > > "IgfsHadoopFileSystemDelegator" or something like this? > > > > > > > > > > > > > > > > > > > > Any thoughts? > > > > > > > > > > > > > > > > > > > > Vladimir. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
