+1 on Vladimir’s suggestion. Let’s remember to document it whenever we make
the change.
I also think that we should have an out-of-the-box implementation of the
file-system-factory with a fix for Kerberos.
D.
On Thu, Dec 17, 2015 at 12:01 PM, Vladimir Ozerov <voze...@gridgain.com>
wrote:
> + 1 for idea in general, but I have several concerns:
>
> 1) SecondaryFileSystemProvider is internal class and cannot be exposed. We
> need some other interface here.
> 2) I would avoid setting properties through constructor because it is
> error-prone when Spring is used. Instead, I would create setters as in
> almost all other components.
> 3) Ideally we should not break existing interfaces.
>
> To summarize, it could looks as follows:
>
> class IgniteHadoopIgfsSecondaryFileSystem {
> /** These properties already exist. */
> void setUri(String fileSystemUri);
> void setConfig(String pathToConfig);
> void setUserName(String userName);
>
> /** This is a new property. */
> void setFileSystemFactory(HadoopFileSystemFactory factory);
> }
>
> interface HadoopFileSystemFactory {
> FileSystem create(String uri, String configPath, String userName);
> }
>
> Vladimir.
>
> On Thu, Dec 17, 2015 at 10:34 PM, Ivan V. <iveselovs...@gridgain.com>
> wrote:
>
> > To summarize suggested change in short: we add constructor
> >
> >
> org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem#IgniteHadoopIgfsSecondaryFileSystem(java.lang.String,
> >
> org.apache.ignite.internal.processors.hadoop.SecondaryFileSystemProvider) ,
> > allowing to plug an arbitrary SecondaryFileSystemProvider bean when
> > constructing IgniteHadoopIgfsSecondaryFileSystem , e.g:
> >
> > <bean
> > class="org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem">
> > <constructor-arg name="userName"
> value="ivan"/>
> > <constructor-arg name="provider">
> > <bean class="
> > *com.foo.MySuperSecondaryFileSystemProvider*">
> > <constructor-arg name="secUri"
> > value="hdfs://localhost:9000"/>
> > <constructor-arg
> > name="secConfPath"><null/></constructor-arg>
> > </bean>
> > </constructor-arg>
> > </bean>
> >
> >
> > On Thu, Dec 17, 2015 at 10:14 PM, Ivan V. <iveselovs...@gridgain.com>
> > wrote:
> >
> > > Hi, dev,
> > > one of Ignite users discovered that IGFS does not correctly work with
> > > secondary HDFS file system if that HDFS is "kerberised". The issue is
> > > described in https://issues.apache.org/jira/browse/IGNITE-2195 (This
> may
> > > also be close to this issue:
> > >
> >
> http://apache-ignite-users.70518.x6.nabble.com/Exception-in-Kerberos-Yarn-cluster-td1950.html
> > > ). The user suggests a fix that creates SecondaryFileSystem using some
> > > proxy UserGroupInformation, and that workaround works for him.
> > > However, we're not sure this fix is generic enough and will work well
> in
> > > all possible configurations.
> > > So, there is an idea to make SecondaryFileSystemProvider pluggable to
> > > allow users to write own code for specific configurations. An example
> can
> > > be seen in this pull request:
> > > https://github.com/apache/ignite/pull/350 .
> > > Pros, cons, other thoughts?
> > > Thanks in advance.
> > > --
> > > ivan
> > >
> >
>
