Alright, thanks.
My dilemma is that currently all the out-of-core state stuff I put under
streaming-contrib. So I either make the whole contrib package hadoop 2 only
which is probably kind of awkward, or I move the main interfaces from
contrib to somewhere else then I can put the HDFS backend stuff into a
module that is hadoop 2 only.
Gyula
Robert Metzger <rmetz...@apache.org> ezt írta (időpont: 2016. jan. 22., P,
22:31):
> Hi Gyula,
>
> will the out of core backend be in a separate maven module? If so, you can
> include the module only in the "hadoop-2" profile.
>
> As you can see in the main pom.xml, "flink-yarn" and "flink-fs-tests" are
> also "hadoop2" only modules:
>
> <profile>
> <id>hadoop-2</id>
> <activation>
> <property>
> <!-- Please do not remove the 'hadoop2' comment. See
> ./tools/generate_specific_pom.sh -->
> <!--hadoop2--><name>!hadoop.profile</name>
> </property>
> </activation>
> <properties>
> <hadoop.version>${hadoop-two.version}</hadoop.version>
> <shading-artifact.name>flink-shaded-hadoop2</shading-artifact.name>
> <shading-artifact-module.name>flink-shaded-hadoop2</
> shading-artifact-module.name>
> </properties>
> <modules>
> <module>flink-yarn</module>
> <!-- Include the flink-fs-tests project only for HD2.
> The HDFS minicluster interfaces changed between the two versions.
> -->
> <module>flink-fs-tests</module>
> </modules>
> </profile>
>
>
> If the backend is not in a separate maven module, you can use reflection.
> Check out the RollingSink#reflectHflushOrSync() method. It calls "hflush"
> only if the method is available ;)
>
>
>
> On Fri, Jan 22, 2016 at 10:23 PM, Gyula Fóra <gyf...@apache.org> wrote:
>
> > Hi,
> >
> > While developing the out-of-core state backend that will store state
> > directly to hdfs (either TFiles or BloomMapFiles), I realised that some
> > file formats and features I use are hadoop 2.x only.
> >
> > What is the suggested way to handle features that use hadoop 2.x api? Can
> > these be excluded from the travis build for the hadoop 1 profile somehow?
> >
> > Thanks,
> > Gyula
> >
>
