thanks for the proposal
and working in this direction
I haven't checked yet all the things
however I see at least one missing piece here.
In FlinkSQL we have views and materialized tables. Both are relying on
expanded ("rewritten" in Calcite terms) SQL.
And we have already faced a number of issues around that and fixed
them in Calcite (like just a few or them [1], [2], [3], [4], [5], [6],
[7], [8], [9], [10])
I'm pretty sure there will be something lambda related. For that
reason I would suggest also having test with views(or materialized
tables, does not matter) based on sql with lambda
and check that SELECT from them still able to produce output
[1] https://issues.apache.org/jira/browse/CALCITE-7660
[2] https://issues.apache.org/jira/browse/CALCITE-7575
[3] https://issues.apache.org/jira/browse/CALCITE-7465
[4] https://issues.apache.org/jira/browse/CALCITE-7480
[5] https://issues.apache.org/jira/browse/CALCITE-7471
[6] https://issues.apache.org/jira/browse/CALCITE-7470
[7] https://issues.apache.org/jira/browse/CALCITE-7466
[8] https://issues.apache.org/jira/browse/CALCITE-7312
[9] https://issues.apache.org/jira/browse/CALCITE-7217
[10] https://issues.apache.org/jira/browse/CALCITE-6944
On Fri, Jul 24, 2026 at 9:37 AM <[email protected]> wrote:
>
> Hi Gyula
>
> Thanks a lot for your reply and the +1!
>
> On a conceptual level, lambdas and scalar functions / UDFs are distinct. A
> lambda isn't a stored/registered value. It's represented by the planning-only
> FUNCTION logical type and only exists as an argument to higher-order
> functions like ARRAY_FILTER and TRANSFORM. Essentially, it's the wrapper that
> introduces the element variable x.
>
> Within the lambda you can freely use any scalar function or UDF:
> ARRAY_FILTER(arr, x -> is_positive(x))
> ARRAY_FILTER(arr, x -> is_positive(x) AND x < 100)
>
> What currently doesn't work is passing a bare UDF name in the lambda position
> (i.e. automatic expansion of is_positive into x -> is_positive(x)):
> ARRAY_FILTER(arr, is_positive) -- not supported
>
> We intentionally left this out of scope for now, but it's a straightforward
> follow-up if we see demand for it.
>
> Best,
> Dominik
>
> From: Gyula Fóra <[email protected]>
> Date: Thursday, 23 July 2026 at 16:07
> To: [email protected] <[email protected]>
> Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL
> (TRANSFORM and ARRAY_FILTER)
>
>
> Be aware: This is an external email.
>
>
>
> Hey!
>
> Without a good background knowledge about the syntax in other systems, the
> ARRAY_FILTER, TRANSFORM functions seem really useful, especially with
> lambda functions.
> Overall +1 for this idea from a very high level but it would be great with
> someone with more deep context on SQL functions / runtime to chime in.
>
> One question I had is how do lambda functions relate to existing scalar
> functions / UDFs? Would the user be able to use boolean valued scalar
> functions where lambdas are shown in the proposal?
>
> Cheers
> Gyula
>
> On Wed, Jul 15, 2026 at 8:40 AM <[email protected]> wrote:
>
> > Hi everyone,
> >
> > I'd like to start a discussion on a FLIP that introduces higher-order
> > functions
> > (functions that take a lambda expression as an argument) to Flink SQL,
> > together
> > with the first two built-ins that use them: TRANSFORM and ARRAY_FILTER.
> >
> > FLIP:
> > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C1ad1fb4b5c76419d815308dee8c3bbc2%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204124551661418%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=S3%2BjqrnvtFjxTyC%2BaRTwRV3ynymQB6Jru1fcO4gw5Ew%3D&reserved=0<https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing>
> >
> > Motivation
> >
> > Users migrating to Flink SQL from Spark, Databricks, Snowflake, DuckDB, and
> > Presto/Trino expect to manipulate collections (arrays and maps) inline
> > with a
> > lambda instead of UNNEST + re-aggregate rewrites or bespoke UDFs. The
> > absence of
> > higher-order collection functions forces verbose query rewrites during
> > migration
> > and raises time-to-first-query. TRANSFORM(array, x -> x + 1) and
> > ARRAY_FILTER(array, x -> x > 0) are the two most requested entry points
> > and,
> > importantly, they can be built on lambda infrastructure that already
> > exists in
> > Calcite (CALCITE-3679), so the surface area we add on the Flink side is
> > rather
> > small.
> >
> > Summary
> >
> > • Introduce a new logical type FUNCTION that describes the type of a
> > lambda
> > (its argument types and its result type). This is the type-system
> > foundation
> > every higher-order function needs; it is a planning/translation helper
> > type
> > and is not a persisted column type.
> > • Add ARRAY_FILTER(array, element -> predicate), which returns a new
> > array
> > containing only the elements for which the predicate holds.
> > • Add TRANSFORM(collection, lambda), which applies a lambda to every
> > element of
> > an array (TRANSFORM(array, x -> expr)) or every entry of a map
> > (TRANSFORM(map, (k, v) -> expr)), returning a new array/map.
> > • Reuse Calcite's lambda parsing, validation and
> > RexLambda/FunctionSqlType
> > infrastructure (CALCITE-3679) rather than inventing a Flink-specific
> > lambda
> > syntax. The lambda arrow syntax x -> expr and (k, v) -> expr is already
> > parseable by the Calcite version Flink bundles (1.41.0).
> >
> > The functions are net-new syntax and are additive: no existing query
> > changes
> > behavior. There is no new configuration option — the functions are always
> > available once the release ships.
> >
> > Examples:
> >
> > SELECT ARRAY_FILTER(ARRAY[1, 2, 3, 4], x -> x > 2); --
> > [3, 4]
> > SELECT TRANSFORM(ARRAY[1, 2, 3], x -> x * 10); --
> > [10, 20, 30]
> > SELECT TRANSFORM(ARRAY['a', 'bb', 'ccc'], s -> CHAR_LENGTH(s)); --
> > [1, 2, 3]
> > SELECT TRANSFORM(MAP['a', 1, 'b', 2], (k, v) -> v * 100); --
> > {a=100, b=200}
> >
> > Thanks,
> > Dominik Bünzli
> > Data, Analytics & AI Engineer
> >
--
Best regards,
Sergey
