Hey everyone!
I would like to kick off a discussion on FLIP-609: Type Inference for Python
User Defined Functions[1].
While working with Python UDFs, I often noticed I was doing some duplicate
effort around type hinting - in one place
so the planner knows the Flink specific input/output types of the UDF, and also
on the function itself so I could have
an extra layer of checking my type assumptions/flow using type checking tools
like mypy. I also noticed that there was
a bit of friction in doing this, especially since the form of specifying input
types through the UDF constructor was
necessarily disconnected from the actual function arguments the types referred
to.
This FLIP proposes to add a type hints -> Flink types inference layer for UDFs,
so that users in ideal cases should
only have to annotate their function using native Python type hints, and we can
infer the input/output Flink types from
those. In more complex cases, where users want to specify a specific Flink type
rather than a Python type, I also propose
to add some shadow types that wrap the Flink types in a corresponding Python
type, so that both type checking works,
and the Flink specific type hints are bound to the actual arguments, rather
than just the argument positions via the udf
decorator. So that a user could do the following:
from dataclasses import dataclass
from typing import Optional
from pyflink.table import udf
from pyflink.table.typehints import TinyInt, SmallInt, Decimal
Money = Decimal(18, 2)
@dataclass
class PricingResult:
final_price: Money
discount_applied: bool
tier: TinyInt
@udf()
def apply_discount(
price: Money,
discount_pct: Optional[SmallInt],
tier: TinyInt,
) -> PricingResult:
pct = discount_pct or 0
discount = price * pct / 100
return PricingResult(
final_price=price - discount,
discount_applied=pct > 0,
tier=tier,
)
Would love any thoughts or feedback the community might have on this proposal!
Kind regards,
Mika Naylor
[1]
https://cwiki.apache.org/confluence/spaces/FLINK/pages/449286339/FLIP-609+Type+Inference+for+Python+User+Defined+Functions
