Re: Function input type validation

[Chesnay Schepler]

On 09.11.2015 08:49, Aljoscha Krettek wrote:

In the case of the TupleTypeInfo subclass it only works because the equals 
method of TypleTypeInfo is used, IMHO.
I've overridden the equals method to check specifically for my 
implementation and not TupleTypeInfo, implemented a different serializer 
(and verified that it is used), and it still works.

Re: Function input type validation

[Stephan Ewen]

I think this originally comes from the fact that we need to match the input
TypeInfo against the generic signature, for example to figure out what "T"
means in a MapFunction, T>.
That is the reason why Flink can support generic functions even though
there is type erasure at runtime.

Much nice than nullable types is to use Option[Type] for a field that may
be null. In Scala, the OptionTypeInfo adds a null/not-null byte, which is
probably what your custom nullable type also does. By adding this for Java
(with Java 8 options, or a custom option type) would solve that issue as
well. It would also let us retain the benefit that non-nullable fields are
efficiently serialized.



On Mon, Nov 9, 2015 at 9:32 AM, Chesnay Schepler  wrote:

> On 09.11.2015 08:49, Aljoscha Krettek wrote:
>
>> In the case of the TupleTypeInfo subclass it only works because the
>> equals method of TypleTypeInfo is used, IMHO.
>>
> I've overridden the equals method to check specifically for my
> implementation and not TupleTypeInfo, implemented a different serializer
> (and verified that it is used), and it still works.
>

Function input type validation

[Gyula Fóra]

Hey All,

I am wondering what is the reason why Function input types are validated?

This might become an issue if the user wants to write his own TypeInfo for
a type that flink also handles natively.

Let's say I want to implement my own TupleTypeinfo that handles null
values, and I pass this typeinfo in the returns call of an operation. This
will most likely fail when the next operation validates the input although
I think it shouldn't.

But the funny thing about this is that this can actually be easily
circumvented as the validation function calls: actualType.equals(expected
type), so you can trick it by making your custom type's equal method return
true.

Thanks,
Gyula

Re: Function input type validation

[Timo Walther]

The reason for input validation is to check if the Function is fully 
compatible. Actually only the return types are necessary, but it 
prohibits stupid implementation mistakes and undesired behavior.


E.g. if you implement a "class MyMapper extends MapFunction{}" and use it for "env.fromElements(1,2,3).map(new MyMapper())"


Regards,
Timo

On 08.11.2015 23:27, Chesnay Schepler wrote:

On 08.11.2015 21:28, Gyula Fóra wrote:

Let's say I want to implement my own TupleTypeinfo that handles null
values, and I pass this typeinfo in the returns call of an operation. 
This
will most likely fail when the next operation validates the input 
although

I think it shouldn't.
So i just tried this. Removed the final modifier from TupleTypeInfo, 
created one that extended TupleTypeInfo,
threw in a returns() call into the ConnectedComponents example and it 
passed the input validation.


Unless i misunderstood something you're premise is invalid.