Hi everyone!
We recently introduced type hints for the Java API. Since that is a pretty
useful feature, I wanted to quickly explain what it is.
Kudos to Timo Walther, who did a large part of this work.
*Background*
Flink tries to figure out as much information about what types enter and
leave user functions as possible.
- For the POJO API (where one refers to field names), we need that
information to make checks (for typos and type compatibility) before the
job is executed.
- For the upcoming logical programs (see roadmap draft) we need this to
know the "schema" of functions.
- The more we know, the better serialization and data layout schemes the
compiler/optimizer can develop. That is quite important for the memory
usage paradigm in Flink (work on serialized data inside/outside the heap
and make serialization very cheap)
- Finally, it also spares users having to worry about serialization
frameworks and having to register types at those frameworks.
*Problem*
Scala is an easy case, because it preserves generic type information
(ClassTags / Type Manifests), but Java erases generic type info in most
cases.
We do reflection analysis on the user function classes to get the generic
types. This logic also contains some simple type inference in case the
functions have type variables (such as a MapFunction<T, Tuple2<T, Long>>).
Not in all cases can we figure out the data types of functions reliably in
Java. Some issues remain with generic lambdas (we are trying to solve this
with the Java community, see below) and with generic type variables that we
cannot infer.
*Solution: Type Hints*
To make this cases work easily, a recent addition to the 0.9-SNAPSHOT
master introduced type hints. They allow you to tell the system types that
it cannot infer.
You can write code like
DataSet<SomeType> result =
dataSet.map(new MyGenericNonInferrableFunction<Long,
SomeType>()).returns(SomeType.class);
To make specification of generic types easier, it also comes with a parser
for simple string representations of generic types:
.returns("Tuple2<Integer, my.SomeType>")
We suggest to use this instead of the "ResultTypeQueryable" workaround that
has been used in some cases.
*Improving Type information in Java*
One Flink committer (Timo Walther) has actually become active in the
Eclipse JDT compiler community and in the OpenJDK community to try and
improve the way type information is available for lambdas.
Greetings,
Stephan
