On 2026/09/17 17:35:15 Martijn Visser wrote:
> Hi all,
>
> I was wondering if this proposal is still moving forward? I'd love to
> see some of these changes land in the upcoming Flink release.
>
> Thanks, Martijn
>
> Op vr 21 aug 2026 om 20:27 schreef David Anderson <[email protected]>:
> >
> > +1 for the overall direction; we've needed this. It will significantly
> > improve both usability and readability.
> >
> > Also +1 for Gustavo's recommendations. The overall scope is just too much
> > for one FLIP.
> >
> > On Fri, Aug 21, 2026 at 8:42 AM Gustavo de Morais <[email protected]>
> > wrote:
> >
> > > Hi Michal and Dominik,
> > >
> > > Thanks for working on this - the new document is much more mature than the
> > > first one! In general, I think we're in a good direction, +1.
> > >
> > > A few points:
> > >
> > > 1) Splitting the FLIP
> > >
> > > This is a lot of content for careful review, and each function really
> > > needs
> > > its own discussion. If we include so many in one FLIP, the review will
> > > probably take even longer. I'd suggest we focus on the foundation first
> > > and
> > > get that cleared, and have the function discussions in a second FLIP (and
> > > partially on the pull requests).
> > >
> > > Suggestion:
> > > - FLIP 1 (foundation): lambda syntax + FUNCTION type + capture lifting +
> > > codegen + Table API/Python etc. It can include a few example functions
> > > (e.g. ARRAY_TRANSFORM, ARRAY_FILTER, ARRAY_REDUCE) to exercise the design,
> > > but without committing to their shape.
> > > - FLIP 2+ (breadth): a second FLIP to discuss the first set of functions,
> > > their APIs and shapes.
> > >
> > > 2) Making it a FLIP document
> > >
> > > The document currently reads like a long decision log. To make it official
> > > as a FLIP, I think we should turn it into a concise, easy-to-read
> > > document:
> > > - Make it as short as possible with ofc still all the relevant information
> > > we need. Right now at 85 pages, it's really only consumable by AI.
> > > - Add an index at the top so it's easy to navigate. We often come back to
> > > a
> > > FLIP many times over the years, so this matters.
> > > - Add examples wherever we can - examples are in general super helpful to
> > > immediately grasp the content. There are some short ones in the
> > > "Composition" section - add more whetever you can.
> > > - As orientation, it helps to look at how recent FLIPs are structured. I'm
> > > happy to help make sure this lands on our Confluence when we're ready.
> > >
> > > 3) Content feedback (for later, not the first FLIP)
> > >
> > > Note: I don't think these belong in the first FLIP we should focus on, so
> > > feel free to ignore them for now - just sharing for the future and feel
> > > free to not reply directly to these here:
> > > - Predicates like exists/forall (Spark) and any_match/all_match/none_match
> > > (Trino). Very common, and hard to express well with FILTER + CARDINALITY.
> > > - An (element, index) variant of transform/filter, like Spark/Trino
> > > transform(a, (x, i) -> ...) with i the 0-based index. Pretty core and
> > > useful.
> > > - A custom comparator array_sort(array, comparator), as in Spark/Trino
> > > (2-arg comparator returning negative/0/positive).
> > >
> > > 4) EXPLAIN improvement
> > >
> > > For a capturing lambda, the plan shows the lifted form: the user writes x
> > > -> x + base, but the plan renders (x, cap$1) -> (x + cap$1), base. Could
> > > we
> > > de-lift at render time so EXPLAIN shows intent, not the optimization -
> > > i.e.
> > > Calc(select=[ARRAY_TRANSFORM(a, x -> (x + base)) AS EXPR$0])? The internal
> > > representation can stay as-is.
> > >
> > > Thanks again for driving this,
> > >
> > > Kind regards,
> > > Gustavo
> > >
> > > On Sun, 9 Aug 2026 at 10:05, Michal Stutzmann <[email protected]>
> > > wrote:
> > >
> > > > Hi Gustavo,
> > > >
> > > > Thanks for looking into this and the pointers. We have now included more
> > > > details in the FLIP draft (link shared by Dominik in the previous
> > > > e-mail:
> > > >
> > > >
> > > https://docs.google.com/document/d/1vd-l922wRJGkbhsGaT9TxRTG2UKjj7nAtxC8J9RaG1U/edit?usp=sharing
> > > > ).
> > > >
> > > > In short:
> > > > - A UDF called from a lambda body — supported.
> > > > - A UDF passed as the lambda, e.g. `ARRAY_TRANSFORM(a, my_udf)`, is not
> > > > supported — this would need eta-expansion. The same call written as
> > > > `ARRAY_TRANSFORM(a, x -> my_udf(x))` is supported.
> > > > - Nesting: supported; arbitrary depth, any combination of built-in and
> > > > user-defined, in both directions, with each level capturing the
> > > > parameters of all enclosing lambdas plus outer columns.
> > > >
> > > > All three are covered by a new section in the FLIP draft called Public
> > > > Interfaces / 6. Composition, which lists the supported compositions with
> > > an
> > > > example for each. The mechanism behind it is in Proposed Changes /
> > > > Lambda
> > > > captures.
> > > >
> > > > Every one of those is covered by an IT case that executes the query and
> > > > asserts the result. If you have other combinations in mind, we will add
> > > > them.
> > > >
> > > > Best,
> > > > Michal
> > > >
> > > >
> > > > On Fri, Aug 7, 2026 at 4:15 PM Gustavo de Morais <[email protected]
> > > >
> > > > wrote:
> > > >
> > > > > Hi Dominik,
> > > > >
> > > > > Thanks for the reply and for considering these. It's great that you're
> > > > both
> > > > > working on this. As mentioned, it's a very important feature, but
> > > > > there
> > > > are
> > > > > indeed some cases that might grow its scope - we should make sure we
> > > > design
> > > > > it in a complete way that fits the different parts of the project.
> > > > >
> > > > > I'll try to make some time next week to take a deeper look at what you
> > > > > shared. In the meantime, here are a few more pointers you can already
> > > > look
> > > > > into: you mentioned lambda arguments for user-defined functions - what
> > > > > about a UDF as an argument of a lambda function? Nesting is also worth
> > > > > considering: multiple nested lambda functions, lambda function -> UDF
> > > or
> > > > > regular function -> lambda function -> ... etc.
> > > > >
> > > > > Kind regards,
> > > > > Gustavo
> > > > >
> > > > > On Fri, 7 Aug 2026 at 11:31, <[email protected]> wrote:
> > > > >
> > > > > > Hi Gustavo,
> > > > > >
> > > > > > Sorry for the late response. I had "a few" discussions with my
> > > > colleague
> > > > > > Michal Stutzmann (you might remember him from the Flink Forward
> > > > > conference
> > > > > > in Barcelona) about the FLIP and the features it should cover.
> > > > > > Michal
> > > > > > extended the proposal<
> > > > > >
> > > > >
> > > >
> > > https://docs.google.com/document/d/1vd-l922wRJGkbhsGaT9TxRTG2UKjj7nAtxC8J9RaG1U/edit?usp=sharing
> > > > > >
> > > > > > and implementation to also support the Table & Python APIs, as well
> > > as
> > > > > > lambda arguments for user-defined functions. Additionally, we now
> > > > support
> > > > > > using columns within lambda expressions.
> > > > > >
> > > > > > We pushed the WIP changes to a branch in our fork, which you can
> > > > > > find
> > > > > here:
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > > https://github.com/swisscom-bigdata/flink/tree/flink-31207-higher-order-functions-and-lambdas
> > > > > >
> > > > > > The set of changes grew larger than expected, but in our opinion
> > > > > > they
> > > > > > significantly simplify the user experience by reducing the need to
> > > work
> > > > > > with UDFs or custom code to handle collections.
> > > > > >
> > > > > > Thanks a lot in advance for taking a look!
> > > > > >
> > > > > > Best,
> > > > > > Dominik
> > > > > >
> > > > > > From: Gustavo de Morais <[email protected]>
> > > > > > Date: Friday, 31 July 2026 at 12:05
> > > > > > To: [email protected] <[email protected]>
> > > > > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink
> > > > > > SQL
> > > > > > (TRANSFORM and ARRAY_FILTER)
> > > > > >
> > > > > >
> > > > > > Be aware: This is an external email.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Hey Dominik,
> > > > > >
> > > > > > Lambda functions are an important building block - not only for
> > > > > > these
> > > > > > built-in collection functions, but also for UDFs and PTFs whitin the
> > > > > whole
> > > > > > archictecture. So thanks for the proposal and for wanting to
> > > > contribute.
> > > > > >
> > > > > > I'll take a look at the proposal soon. You mentioned a WIP branch a
> > > > > couple
> > > > > > of times in the thread - could you share the link so we can take a
> > > look
> > > > > at
> > > > > > that too?
> > > > > >
> > > > > > Kind regards,
> > > > > > Gustavo
> > > > > >
> > > > > > On Thu, 30 Jul 2026 at 16:39, <[email protected]> wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > Following up on this FLIP to see if there are any comments or
> > > > concerns.
> > > > > > >
> > > > > > > Since the last mail I added tests for:
> > > > > > > - views using ARRAY_FILTER / TRANSFORM
> > > > > > > - materialized tables going through expand-and-replan
> > > > > > >
> > > > > > > Everything is green locally so far.
> > > > > > >
> > > > > > > Feedback on the planner integration, Calcite handling, missing
> > > > > > > edge
> > > > > > cases,
> > > > > > > or the overall direction would be appreciated.
> > > > > > >
> > > > > > > As I’m still new to the FLIP process, I’d also appreciate some
> > > > guidance
> > > > > > on
> > > > > > > the next steps from here. Many thanks in advance.
> > > > > > >
> > > > > > > Best,
> > > > > > > Dominik
> > > > > > >
> > > > > > > From: Bünzli Dominik, SCS-INI-DNA-INF <
> > > [email protected]>
> > > > > > > Date: Friday, 24 July 2026 at 11:17
> > > > > > > To: [email protected] <[email protected]>
> > > > > > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink
> > > SQL
> > > > > > > (TRANSFORM and ARRAY_FILTER)
> > > > > > >
> > > > > > > Hi Sergey,
> > > > > > >
> > > > > > > Good catch. I didn’t have these cases in mind. Quickly added test
> > > > cases
> > > > > > > for them to my branch.
> > > > > > >
> > > > > > > The view test creates a view like "CREATE VIEW v AS SELECT id,
> > > > > > > ARRAY_FILTER(vals, x -> x > 2) FROM ...", then selects from it and
> > > > > checks
> > > > > > > that the arrays come back right. Same idea for TRANSFORM and the
> > > map
> > > > > > lambda
> > > > > > > (k, v) -> v * 100.
> > > > > > >
> > > > > > > The materialized table test does the equivalent: a CONTINUOUS
> > > > > > > table
> > > > > using
> > > > > > > ARRAY_FILTER and TRANSFORM in its query, running on a
> > > > > > > mini-cluster,
> > > > > then
> > > > > > a
> > > > > > > SELECT to confirm the output matches.
> > > > > > >
> > > > > > > So, both should (to my still limited knowledge), go through the
> > > > > > > expand-and-replan path and check real results. If Calcite mangles
> > > > > > > a
> > > > > > lambda,
> > > > > > > these tests should fail. Looks good so far, tests are green
> > > locally.
> > > > > > >
> > > > > > > Best,
> > > > > > > Dominik
> > > > > > >
> > > > > > > From: Sergey Nuyanzin <[email protected]>
> > > > > > > Date: Friday, 24 July 2026 at 09:59
> > > > > > > To: [email protected] <[email protected]>
> > > > > > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink
> > > SQL
> > > > > > > (TRANSFORM and ARRAY_FILTER)
> > > > > > >
> > > > > > >
> > > > > > > Be aware: This is an external email.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > thanks for the proposal
> > > > > > >
> > > > > > > and working in this direction
> > > > > > >
> > > > > > > I haven't checked yet all the things
> > > > > > >
> > > > > > > however I see at least one missing piece here.
> > > > > > >
> > > > > > > In FlinkSQL we have views and materialized tables. Both are
> > > > > > > relying
> > > > on
> > > > > > > expanded ("rewritten" in Calcite terms) SQL.
> > > > > > > And we have already faced a number of issues around that and fixed
> > > > > > > them in Calcite (like just a few or them [1], [2], [3], [4], [5],
> > > > [6],
> > > > > > > [7], [8], [9], [10])
> > > > > > >
> > > > > > > I'm pretty sure there will be something lambda related. For that
> > > > > > > reason I would suggest also having test with views(or materialized
> > > > > > > tables, does not matter) based on sql with lambda
> > > > > > > and check that SELECT from them still able to produce output
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7660&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215787243%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=dhAIi9%2BVQbdypDURBS6hCR97oOt1Y%2BWDeYhcppgEH18%3D&reserved=0
> > > > > > <https://issues.apache.org/jira/browse/CALCITE-7660>
> > > > > > > <
> > > > > >
> > > > >
> > > >
> > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7660&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215837165%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=WJ6rRpo1oeuVua%2BU6r65we7Q%2FyAkiVC3drjq4y%2F37ZM%3D&reserved=0
> > > > > > <https://issues.apache.org/jira/browse/CALCITE-7660>>
> > > > > > > [2]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7575&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215876426%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2yAtLVRqA5lHAT4dKoiNgOEZmCDrNLlq6XnXvidm9dk%3D&reserved=0
> > > > > > <https://issues.apache.org/jira/browse/CALCITE-7575>
> > > > > > > <
> > > > > >
> > > > >
> > > >
> > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7575&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215913003%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=M%2Bv0m4nRXd%2B%2BbbvqPd8gB8PwbdSWqBG4eLVWMoUR4vo%3D&reserved=0
> > > > > > <https://issues.apache.org/jira/browse/CALCITE-7575>>
> > > > > > > [3]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7465&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215947484%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2F2pRc%2BwqNk80bEjsaexGT8uxTKn%2FGuw%2BBPSEYz3lGnk%3D&reserved=0
> > > > > > <https://issues.apache.org/jira/browse/CALCITE-7465>
> > > > > > > <
> > > > > >
> > > > >
> > > >
> > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7465&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215983673%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=n14%2FEnyO%2F2WIcPudrEWwFaQ8KHqRIppuqRfJxK3NqtA%3D&reserved=0
> > > > > > <https://issues.apache.org/jira/browse/CALCITE-7465>>
> > > > > > > [4]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7480&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216022139%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=6Gyk1uF2SfLH0J7lQqhBYPDIU53NVYIzCrg0GrhMN7w%3D&reserved=0
> > > > > > <https://issues.apache.org/jira/browse/CALCITE-7480>
> > > > > > > <
> > > > > >
> > > > >
> > > >
> > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7480&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216057890%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=f4gNZnMW6VBq12Yt8PkIzdFfUwdX8fYXTJOBOCxMOwg%3D&reserved=0
> > > > > > <https://issues.apache.org/jira/browse/CALCITE-7480>>
> > > > > > > [5]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7471&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216090334%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=d5YrH4adMbZBaEhqXEns9xR0IerRNTsjUZeC5DXQE50%3D&reserved=0
> > > > > > <https://issues.apache.org/jira/browse/CALCITE-7471>
> > > > > > > <
> > > > > >
> > > > >
> > > >
> > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7471&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216116472%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2ZyKirj6%2FrAmUM77mq35uKuogcNVn3vXfQqT5B9mTEU%3D&reserved=0
> > > > > > <https://issues.apache.org/jira/browse/CALCITE-7471>>
> > > > > > > [6]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7470&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216150666%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2Fuu4GC7suv7WSI4HtZqxGtvTkV4HarcODd5Q8W7zDpQ%3D&reserved=0
> > > > > > <https://issues.apache.org/jira/browse/CALCITE-7470>
> > > > > > > <
> > > > > >
> > > > >
> > > >
> > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7470&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216179872%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=JS14KTB4YncuYb7dkjjQFu%2FVRvqjaHnPfTEjCn2kWDw%3D&reserved=0
> > > > > > <https://issues.apache.org/jira/browse/CALCITE-7470>>
> > > > > > > [7]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7466&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216199448%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=mrskfIM%2FHIKFxL1foGM9XiGiphFwbPxO9q0n%2FubrWS8%3D&reserved=0
> > > > > > <https://issues.apache.org/jira/browse/CALCITE-7466>
> > > > > > > <
> > > > > >
> > > > >
> > > >
> > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7466&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216219518%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=r9y6sPdjB5UpVXSiVl8rq3XULGsl8SxlIWtY9LHlCkU%3D&reserved=0
> > > > > > <https://issues.apache.org/jira/browse/CALCITE-7466>>
> > > > > > > [8]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7312&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216239592%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Ra1qmGy%2Fnnxs98tuQZBMHMxceJ3NkYvyfKlwU1cHl84%3D&reserved=0
> > > > > > <https://issues.apache.org/jira/browse/CALCITE-7312>
> > > > > > > <
> > > > > >
> > > > >
> > > >
> > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7312&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216258677%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=GdHVoxqW6y2kiUi%2F3pckMK8szOLnGwqCtW4sconw4XQ%3D&reserved=0
> > > > > > <https://issues.apache.org/jira/browse/CALCITE-7312>>
> > > > > > > [9]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7217&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216284576%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=k%2FBsEShTSDCzRCfsgPZXz7cYJJnDsaeRnW45ldO5RiY%3D&reserved=0
> > > > > > <https://issues.apache.org/jira/browse/CALCITE-7217>
> > > > > > > <
> > > > > >
> > > > >
> > > >
> > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7217&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216319677%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XhpIIafvVURhq8tbckhdDytfZ9P%2FsJDoaEoIpYlOBW4%3D&reserved=0
> > > > > > <https://issues.apache.org/jira/browse/CALCITE-7217>>
> > > > > > > [10]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-6944&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216350464%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=ZiPFaESvPoORqz9KLJLh9I7z6IanGWhL2ZnCl2Ap9N4%3D&reserved=0
> > > > > > <https://issues.apache.org/jira/browse/CALCITE-6944>
> > > > > > > <
> > > > > >
> > > > >
> > > >
> > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-6944&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216379236%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Pqe2YCqIdEHIZbTSrLF6X6BAlGRej0vdDaOL%2F2J1%2BeM%3D&reserved=0
> > > > > > <https://issues.apache.org/jira/browse/CALCITE-6944>>
> > > > > > >
> > > > > > > On Fri, Jul 24, 2026 at 9:37 AM <[email protected]>
> > > > wrote:
> > > > > > > >
> > > > > > > > Hi Gyula
> > > > > > > >
> > > > > > > > Thanks a lot for your reply and the +1!
> > > > > > > >
> > > > > > > > On a conceptual level, lambdas and scalar functions / UDFs are
> > > > > > distinct.
> > > > > > > A lambda isn't a stored/registered value. It's represented by the
> > > > > > > planning-only FUNCTION logical type and only exists as an argument
> > > to
> > > > > > > higher-order functions like ARRAY_FILTER and TRANSFORM.
> > > Essentially,
> > > > > it's
> > > > > > > the wrapper that introduces the element variable x.
> > > > > > > >
> > > > > > > > Within the lambda you can freely use any scalar function or UDF:
> > > > > > > > ARRAY_FILTER(arr, x -> is_positive(x))
> > > > > > > > ARRAY_FILTER(arr, x -> is_positive(x) AND x < 100)
> > > > > > > >
> > > > > > > > What currently doesn't work is passing a bare UDF name in the
> > > > lambda
> > > > > > > position (i.e. automatic expansion of is_positive into x ->
> > > > > > is_positive(x)):
> > > > > > > > ARRAY_FILTER(arr, is_positive) -- not supported
> > > > > > > >
> > > > > > > > We intentionally left this out of scope for now, but it's a
> > > > > > > straightforward follow-up if we see demand for it.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Dominik
> > > > > > > >
> > > > > > > > From: Gyula Fóra <[email protected]>
> > > > > > > > Date: Thursday, 23 July 2026 at 16:07
> > > > > > > > To: [email protected] <[email protected]>
> > > > > > > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in
> > > > > > > > Flink
> > > > SQL
> > > > > > > (TRANSFORM and ARRAY_FILTER)
> > > > > > > >
> > > > > > > >
> > > > > > > > Be aware: This is an external email.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Hey!
> > > > > > > >
> > > > > > > > Without a good background knowledge about the syntax in other
> > > > > systems,
> > > > > > > the
> > > > > > > > ARRAY_FILTER, TRANSFORM functions seem really useful, especially
> > > > with
> > > > > > > > lambda functions.
> > > > > > > > Overall +1 for this idea from a very high level but it would be
> > > > great
> > > > > > > with
> > > > > > > > someone with more deep context on SQL functions / runtime to
> > > chime
> > > > > in.
> > > > > > > >
> > > > > > > > One question I had is how do lambda functions relate to existing
> > > > > scalar
> > > > > > > > functions / UDFs? Would the user be able to use boolean valued
> > > > scalar
> > > > > > > > functions where lambdas are shown in the proposal?
> > > > > > > >
> > > > > > > > Cheers
> > > > > > > > Gyula
> > > > > > > >
> > > > > > > > On Wed, Jul 15, 2026 at 8:40 AM <[email protected]>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi everyone,
> > > > > > > > >
> > > > > > > > > I'd like to start a discussion on a FLIP that introduces
> > > > > higher-order
> > > > > > > > > functions
> > > > > > > > > (functions that take a lambda expression as an argument) to
> > > Flink
> > > > > > SQL,
> > > > > > > > > together
> > > > > > > > > with the first two built-ins that use them: TRANSFORM and
> > > > > > ARRAY_FILTER.
> > > > > > > > >
> > > > > > > > > FLIP:
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216408066%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=UuKdaeEmSd6hP87gokZhU9leFfw0BrrD2buEDctKjBk%3D&reserved=0
> > > > > > <
> > > > > >
> > > > >
> > > >
> > > https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing
> > > > > > >
> > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216440590%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=woLr6CpsHkDCs7y6yZaJ4%2FoCNhxCxuU9hXqh8AaRSVI%3D&reserved=0
> > > > > > <
> > > > > >
> > > > >
> > > >
> > > https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing
> > > > > > >
> > > > > > > ><
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216476842%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=CuKWtmNDVEXK11ET%2BJDOqAp1b%2FHmHR5ionqmomI7cdE%3D&reserved=0
> > > > > > <
> > > > > >
> > > > >
> > > >
> > > https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing
> > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Motivation
> > > > > > > > >
> > > > > > > > > Users migrating to Flink SQL from Spark, Databricks,
> > > > > > > > > Snowflake,
> > > > > > > DuckDB, and
> > > > > > > > > Presto/Trino expect to manipulate collections (arrays and
> > > > > > > > > maps)
> > > > > > inline
> > > > > > > > > with a
> > > > > > > > > lambda instead of UNNEST + re-aggregate rewrites or bespoke
> > > UDFs.
> > > > > The
> > > > > > > > > absence of
> > > > > > > > > higher-order collection functions forces verbose query
> > > > > > > > > rewrites
> > > > > > during
> > > > > > > > > migration
> > > > > > > > > and raises time-to-first-query. TRANSFORM(array, x -> x + 1)
> > > and
> > > > > > > > > ARRAY_FILTER(array, x -> x > 0) are the two most requested
> > > entry
> > > > > > points
> > > > > > > > > and,
> > > > > > > > > importantly, they can be built on lambda infrastructure that
> > > > > already
> > > > > > > > > exists in
> > > > > > > > > Calcite (CALCITE-3679), so the surface area we add on the
> > > > > > > > > Flink
> > > > > side
> > > > > > is
> > > > > > > > > rather
> > > > > > > > > small.
> > > > > > > > >
> > > > > > > > > Summary
> > > > > > > > >
> > > > > > > > > • Introduce a new logical type FUNCTION that describes the
> > > type
> > > > > of
> > > > > > a
> > > > > > > > > lambda
> > > > > > > > > (its argument types and its result type). This is the
> > > > > type-system
> > > > > > > > > foundation
> > > > > > > > > every higher-order function needs; it is a
> > > > planning/translation
> > > > > > > helper
> > > > > > > > > type
> > > > > > > > > and is not a persisted column type.
> > > > > > > > > • Add ARRAY_FILTER(array, element -> predicate), which
> > > returns
> > > > a
> > > > > > new
> > > > > > > > > array
> > > > > > > > > containing only the elements for which the predicate
> > > > > > > > > holds.
> > > > > > > > > • Add TRANSFORM(collection, lambda), which applies a lambda
> > > to
> > > > > > every
> > > > > > > > > element of
> > > > > > > > > an array (TRANSFORM(array, x -> expr)) or every entry of a
> > > > map
> > > > > > > > > (TRANSFORM(map, (k, v) -> expr)), returning a new
> > > array/map.
> > > > > > > > > • Reuse Calcite's lambda parsing, validation and
> > > > > > > > > RexLambda/FunctionSqlType
> > > > > > > > > infrastructure (CALCITE-3679) rather than inventing a
> > > > > > > Flink-specific
> > > > > > > > > lambda
> > > > > > > > > syntax. The lambda arrow syntax x -> expr and (k, v) ->
> > > expr
> > > > is
> > > > > > > already
> > > > > > > > > parseable by the Calcite version Flink bundles (1.41.0).
> > > > > > > > >
> > > > > > > > > The functions are net-new syntax and are additive: no existing
> > > > > query
> > > > > > > > > changes
> > > > > > > > > behavior. There is no new configuration option — the functions
> > > > are
> > > > > > > always
> > > > > > > > > available once the release ships.
> > > > > > > > >
> > > > > > > > > Examples:
> > > > > > > > >
> > > > > > > > > SELECT ARRAY_FILTER(ARRAY[1, 2, 3, 4], x -> x > 2);
> > > > > > --
> > > > > > > > > [3, 4]
> > > > > > > > > SELECT TRANSFORM(ARRAY[1, 2, 3], x -> x * 10);
> > > > > > --
> > > > > > > > > [10, 20, 30]
> > > > > > > > > SELECT TRANSFORM(ARRAY['a', 'bb', 'ccc'], s ->
> > > > CHAR_LENGTH(s));
> > > > > > --
> > > > > > > > > [1, 2, 3]
> > > > > > > > > SELECT TRANSFORM(MAP['a', 1, 'b', 2], (k, v) -> v * 100);
> > > > > > --
> > > > > > > > > {a=100, b=200}
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Dominik Bünzli
> > > > > > > > > Data, Analytics & AI Engineer
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best regards,
> > > > > > > Sergey
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
Hi Martijn
We're actively working on the first foundational FLIP, as suggested by Gustavo,
and expect to return to the community with a proposal (FLIP) next week.
Best
Lorand
