Hi Sergey, Good catch. I didn’t have these cases in mind. Quickly added test cases for them to my branch.
The view test creates a view like "CREATE VIEW v AS SELECT id, ARRAY_FILTER(vals, x -> x > 2) FROM ...", then selects from it and checks that the arrays come back right. Same idea for TRANSFORM and the map lambda (k, v) -> v * 100. The materialized table test does the equivalent: a CONTINUOUS table using ARRAY_FILTER and TRANSFORM in its query, running on a mini-cluster, then a SELECT to confirm the output matches. So, both should (to my still limited knowledge), go through the expand-and-replan path and check real results. If Calcite mangles a lambda, these tests should fail. Looks good so far, tests are green locally. Best, Dominik From: Sergey Nuyanzin <[email protected]> Date: Friday, 24 July 2026 at 09:59 To: [email protected] <[email protected]> Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL (TRANSFORM and ARRAY_FILTER) Be aware: This is an external email. thanks for the proposal and working in this direction I haven't checked yet all the things however I see at least one missing piece here. In FlinkSQL we have views and materialized tables. Both are relying on expanded ("rewritten" in Calcite terms) SQL. And we have already faced a number of issues around that and fixed them in Calcite (like just a few or them [1], [2], [3], [4], [5], [6], [7], [8], [9], [10]) I'm pretty sure there will be something lambda related. For that reason I would suggest also having test with views(or materialized tables, does not matter) based on sql with lambda and check that SELECT from them still able to produce output [1] https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7660&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519482673%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=wy%2FmMDx7Bc%2Blkbjdyr1rWiEpGyOFweCNjHKHMhEGDYo%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7660> [2] https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7575&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519515485%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=jrSfxN8AhWe%2FY%2BeRtU7IWN9A5SIZrluz9a0jPCbrauE%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7575> [3] https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7465&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519540885%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2XqM%2FzWVAUM9M9v2BvhF2rG1Gxo2y12hPNJVV2OMbtU%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7465> [4] https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7480&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519565082%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=0MUvjzrXcdaphlgigoN0j6xW5m5JiGYJBCe0pXpehiE%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7480> [5] https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7471&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519590890%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=53w4cH9008310XzR343RfJKSpD%2BuiPJXL5sB4GivfMg%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7471> [6] https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7470&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519622955%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2FeYvSOwEryd6cE9JLIU%2Fyopj20WJzeu%2Fv8h2%2FSeGpUM%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7470> [7] https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7466&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519653261%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=PS6dyyL3YaK6Sefib5icMJZweVcQ3192NZI1owUwSIM%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7466> [8] https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7312&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519678467%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=56Zea5qfcDszsps%2B4VOdVCvn2UeVCVx2fOYaUWlX%2FDw%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7312> [9] https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7217&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519700401%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=KJka%2Fs%2F8YPH1b%2F16FCwTgFtsb8vFyK9oY3DGeT%2BUzxU%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7217> [10] https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-6944&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519722120%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2B1HgC5SAvuBelfegMB0CLTCYdwcb6NxCBcTzdjC0Qlw%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-6944> On Fri, Jul 24, 2026 at 9:37 AM <[email protected]> wrote: > > Hi Gyula > > Thanks a lot for your reply and the +1! > > On a conceptual level, lambdas and scalar functions / UDFs are distinct. A > lambda isn't a stored/registered value. It's represented by the planning-only > FUNCTION logical type and only exists as an argument to higher-order > functions like ARRAY_FILTER and TRANSFORM. Essentially, it's the wrapper that > introduces the element variable x. > > Within the lambda you can freely use any scalar function or UDF: > ARRAY_FILTER(arr, x -> is_positive(x)) > ARRAY_FILTER(arr, x -> is_positive(x) AND x < 100) > > What currently doesn't work is passing a bare UDF name in the lambda position > (i.e. automatic expansion of is_positive into x -> is_positive(x)): > ARRAY_FILTER(arr, is_positive) -- not supported > > We intentionally left this out of scope for now, but it's a straightforward > follow-up if we see demand for it. > > Best, > Dominik > > From: Gyula Fóra <[email protected]> > Date: Thursday, 23 July 2026 at 16:07 > To: [email protected] <[email protected]> > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL > (TRANSFORM and ARRAY_FILTER) > > > Be aware: This is an external email. > > > > Hey! > > Without a good background knowledge about the syntax in other systems, the > ARRAY_FILTER, TRANSFORM functions seem really useful, especially with > lambda functions. > Overall +1 for this idea from a very high level but it would be great with > someone with more deep context on SQL functions / runtime to chime in. > > One question I had is how do lambda functions relate to existing scalar > functions / UDFs? Would the user be able to use boolean valued scalar > functions where lambdas are shown in the proposal? > > Cheers > Gyula > > On Wed, Jul 15, 2026 at 8:40 AM <[email protected]> wrote: > > > Hi everyone, > > > > I'd like to start a discussion on a FLIP that introduces higher-order > > functions > > (functions that take a lambda expression as an argument) to Flink SQL, > > together > > with the first two built-ins that use them: TRANSFORM and ARRAY_FILTER. > > > > FLIP: > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519743111%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=MZQgSpOw4rNqYRG3Tm0mUigSQGXQi4dDreRhyV%2F4wZY%3D&reserved=0<https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519772568%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XNwMzQrplv6Pt7nGCO%2F0Jnk2YFD9v%2FJtP7b7iiyeGGQ%3D&reserved=0><https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing> > > > > Motivation > > > > Users migrating to Flink SQL from Spark, Databricks, Snowflake, DuckDB, and > > Presto/Trino expect to manipulate collections (arrays and maps) inline > > with a > > lambda instead of UNNEST + re-aggregate rewrites or bespoke UDFs. The > > absence of > > higher-order collection functions forces verbose query rewrites during > > migration > > and raises time-to-first-query. TRANSFORM(array, x -> x + 1) and > > ARRAY_FILTER(array, x -> x > 0) are the two most requested entry points > > and, > > importantly, they can be built on lambda infrastructure that already > > exists in > > Calcite (CALCITE-3679), so the surface area we add on the Flink side is > > rather > > small. > > > > Summary > > > > • Introduce a new logical type FUNCTION that describes the type of a > > lambda > > (its argument types and its result type). This is the type-system > > foundation > > every higher-order function needs; it is a planning/translation helper > > type > > and is not a persisted column type. > > • Add ARRAY_FILTER(array, element -> predicate), which returns a new > > array > > containing only the elements for which the predicate holds. > > • Add TRANSFORM(collection, lambda), which applies a lambda to every > > element of > > an array (TRANSFORM(array, x -> expr)) or every entry of a map > > (TRANSFORM(map, (k, v) -> expr)), returning a new array/map. > > • Reuse Calcite's lambda parsing, validation and > > RexLambda/FunctionSqlType > > infrastructure (CALCITE-3679) rather than inventing a Flink-specific > > lambda > > syntax. The lambda arrow syntax x -> expr and (k, v) -> expr is already > > parseable by the Calcite version Flink bundles (1.41.0). > > > > The functions are net-new syntax and are additive: no existing query > > changes > > behavior. There is no new configuration option — the functions are always > > available once the release ships. > > > > Examples: > > > > SELECT ARRAY_FILTER(ARRAY[1, 2, 3, 4], x -> x > 2); -- > > [3, 4] > > SELECT TRANSFORM(ARRAY[1, 2, 3], x -> x * 10); -- > > [10, 20, 30] > > SELECT TRANSFORM(ARRAY['a', 'bb', 'ccc'], s -> CHAR_LENGTH(s)); -- > > [1, 2, 3] > > SELECT TRANSFORM(MAP['a', 1, 'b', 2], (k, v) -> v * 100); -- > > {a=100, b=200} > > > > Thanks, > > Dominik Bünzli > > Data, Analytics & AI Engineer > > -- Best regards, Sergey
