At some point, the state backends should probably have their own parent-
and subprojects, similar to the batch-connectors and streaming-connectors.
Then you have the flexibility to include and exclude them as needed.
Stephan
On Sat, Jan 23, 2016 at 6:42 PM, Gyula Fóra <gyula.f...@gmail.com> wrote:
> Alright, thanks.
>
> My dilemma is that currently all the out-of-core state stuff I put under
> streaming-contrib. So I either make the whole contrib package hadoop 2 only
> which is probably kind of awkward, or I move the main interfaces from
> contrib to somewhere else then I can put the HDFS backend stuff into a
> module that is hadoop 2 only.
>
> Gyula
>
> Robert Metzger <rmetz...@apache.org> ezt írta (időpont: 2016. jan. 22., P,
> 22:31):
>
> > Hi Gyula,
> >
> > will the out of core backend be in a separate maven module? If so, you
> can
> > include the module only in the "hadoop-2" profile.
> >
> > As you can see in the main pom.xml, "flink-yarn" and "flink-fs-tests"
> are
> > also "hadoop2" only modules:
> >
> > <profile>
> > <id>hadoop-2</id>
> > <activation>
> > <property>
> > <!-- Please do not remove the 'hadoop2' comment. See
> > ./tools/generate_specific_pom.sh -->
> > <!--hadoop2--><name>!hadoop.profile</name>
> > </property>
> > </activation>
> > <properties>
> > <hadoop.version>${hadoop-two.version}</hadoop.version>
> > <shading-artifact.name>flink-shaded-hadoop2</shading-artifact.name
> >
> > <shading-artifact-module.name>flink-shaded-hadoop2</
> > shading-artifact-module.name>
> > </properties>
> > <modules>
> > <module>flink-yarn</module>
> > <!-- Include the flink-fs-tests project only for HD2.
> > The HDFS minicluster interfaces changed between the two
> versions.
> > -->
> > <module>flink-fs-tests</module>
> > </modules>
> > </profile>
> >
> >
> > If the backend is not in a separate maven module, you can use reflection.
> > Check out the RollingSink#reflectHflushOrSync() method. It calls "hflush"
> > only if the method is available ;)
> >
> >
> >
> > On Fri, Jan 22, 2016 at 10:23 PM, Gyula Fóra <gyf...@apache.org> wrote:
> >
> > > Hi,
> > >
> > > While developing the out-of-core state backend that will store state
> > > directly to hdfs (either TFiles or BloomMapFiles), I realised that some
> > > file formats and features I use are hadoop 2.x only.
> > >
> > > What is the suggested way to handle features that use hadoop 2.x api?
> Can
> > > these be excluded from the travis build for the hadoop 1 profile
> somehow?
> > >
> > > Thanks,
> > > Gyula
> > >
> >
>
