I have pushed an implementation of the restricted lambdas we were
talking about. Guys, please review/test it.
I call these "local lambdas" in the source code (but I'm open for
suggestions), as the function they define can only be called in the
same variable scope where the lambda was (as we have no closures in
the template language, nor final variables). Also, they can only be
used as the parameters of the built-ins that explicitly support them.
As the subject shows, the main goal was just support filtering for
#list, without a nested #if, as that breaks #sep, #else, it?hasNext,
it?index, etc. (I will try to make more universal lambdas in FM3... I
guess it would be way too tricky in FM2.)
For now, I have only added two built-ins that support lambdas: ?filter
and ?map. Any ideas what other such built-ins would be often useful in
templates (with use case, if possible)?
Examples of using ?filter and ?map:
<#list products?filter(it -> it.price < 1000) as product>
${product.name}
</#list>
<#list products?map(it -> it.name) as name>
${name}
</#list>
Of course these built-ins aren't specific to #list, they can be used
anywhere. Naturally, they can be chained as well:
<#assign chepProdNames = products
?filter(it -> it.price < 1000)
?map(it -> it.name)
>
As a side note, ?filter and ?map also accepts FTL function-s and Java
methods as its parameter, not only lambdas.
A tricky aspect of this feature is lazy evaluation, in similar sense
as Java 8 Stream intermediate operations are lazy. On most places,
?filer and ?map are eager, that is, they return a completed sequence
of items. That's because our restricted lambdas only work correctly
"locally". However, there are very common situations where it's clear
that we can use lazy ("streaming") evaluation, as we know where the
resulting stream of elements will be consumed:
- One such case is when these kind of built-ins are chained. Like in
the last example, ?filter doesn't construct a List in memory,
instead the elements just flow through it (some is dropped, as it's
filtering), into ?map. Only the ?map at the end of the chain will
built a List eagerly. Some other built-ins also allow the left-hand
built-in to stream, like in ?filter(...)?map(...)?join(", ") no List
is built anywhere, instead the elements flow through both ?filter
and ?map, and ?join just appends them to the StringBuilder where it
creates its results.
- #list also enables lazy evaluation to its 1st parameter. So in the
#list examples above, yet again no List is built in memory.
- There are other such cases, which I didn't implement yet. For
example the sequence slice operator, like, xs?filter(f)[10..20],
should allow lazy evaluation.
Feedback is welcome!
Monday, December 17, 2018, 11:39:36 AM, Christoph Rüger wrote:
> Hey Daniel,
> I'm very sorry, but I didn't make any progress with this. I think I was a
> bit over-motivated, but unfortunately I cannot spend more time on this for
> various reasons.
> You can take this over. I'm glad to help out with testing and feedback.
>
> Thanks
> Christoph
>
>
>
>
> Am Mo., 17. Dez. 2018 um 11:04 Uhr schrieb Daniel Dekany <[email protected]
>>:
>
>> Any progress in this? I think I will give it a try in the coming days
>> otherwise.
>>
>>
>> Sunday, November 18, 2018, 10:31:29 PM, Daniel Dekany wrote:
>>
>> > See my answers inline...
>> >
>> > Sunday, November 18, 2018, 8:44:40 PM, Christoph Rüger wrote:
>> >
>> >> Thanks Daniel for your feedback. See my answers below
>> >>
>> >> Am So., 11. Nov. 2018 um 19:14 Uhr schrieb Daniel Dekany <
>> [email protected]
>> >>>:
>> >>
>> >>> Sunday, November 11, 2018, 11:40:50 AM, Christoph Rüger wrote:
>> >>>
>> >>> > Am So., 11. Nov. 2018 um 09:25 Uhr schrieb Daniel Dekany <
>> >>> [email protected]
>> >>> >>:
>> >>> >
>> >>> >> Saturday, November 10, 2018, 3:08:14 PM, Denis Bredelet wrote:
>> >>> >>
>> >>> >> > Hi,
>> >>> >> >
>> >>> >> > Le 9 novembre 2018 à 22:36, Christoph Rüger <[email protected]>
>> a
>> >>> >> écrit :
>> >>> >> >
>> >>> >> > Am Fr., 9. Nov. 2018 um 22:55 Uhr schrieb Daniel Dekany <
>> >>> >> [email protected]
>> >>> >> > :
>> >>> >> >
>> >>> >> > It's certainly tricky, but as far as I see possible (but then, who
>> >>> >> >
>> >>> >> > knows what will one find when actually working on it). It's also a
>> >>> >> > feature missing a lot. It's especially missing for #list (I know
>> that
>> >>> >> > you need it for something else), because if you filter the items
>> >>> >> > inside #list with #if-s, then #sep, ?hasNext, etc. will not be
>> usable.
>> >>> >> >
>> >>> >> > Let me say that I disagree here.
>> >>> >> >
>> >>> >> > I do not think that closures are required for FreeMarker, nor that
>> >>> they
>> >>> >> are a good idea.
>> >>> >> >
>> >>> >> > If we add new features to the FreeMarker *tempate engine* I would
>> >>> >> > rather we focus on multi-part macro body rather than an advanced
>> >>> >> language feature like closures.
>> >>> >> >
>> >>> >> > You can add ?filter and ?map if you want, a simple expression as
>> >>> >> parameter should be enough.
>> >>> >>
>> >>> >> Yes, as I said, we certainly start with only allowing lambdas in
>> >>> >> ?filter/?map, also certainly in ?contains.
>> >>> >>
>> >>> > Would be enough in my opinion and very useful.
>> >>> >
>> >>> > Is it possiblefor you to give some pointers to the code on how this
>> could
>> >>> > be implemented? I would maybe like to wrap my head around this a
>> little
>> >>> bit.
>> >>>
>> >>> Please feel yourself encouraged! (:
>> >>>
>> >>> > I started looking at seq_containsBI (
>> >>> >
>> >>>
>> https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L291
>> >>> )
>> >>> > and
>> >>> > and reverseBI (
>> >>> >
>> >>>
>> https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L264
>> >>> )
>> >>> > just to find something related (seq_containsBI checks something) and
>> >>> > reverseBI returns a new sequence.
>> >>> > What I haven't found is a function which takes an Expression as a
>> >>> > parameter.
>> >>> > Is there something similar already or would that be a new thing?
>> >>>
>> >>> It's a new thing in that it will be part of the expression syntax
>> >>> (even if for now we will only allow lambdas as the parameters of a few
>> >>> built-ins, so that we can get away without closures). So it's a new
>> >>> Expression subclass, and has to be part of the parser (ftl.jj) as
>> >>> well.
>> >>
>> >> Hmm, that parser stuff is new for me, it'll take me some time to get
>> into
>> >> it.
>> >
>> > So it's a JavaCC lexer+parser. With a few twists... sorry, but this
>> > code has history... :)
>> >
>> >>> As of lazy evaluation of parameters expressions, that's already
>> >>> done in the built-ins in BuiltInsWithParseTimeParameters, and you will
>> >>> see it's trivial to do, but the situation there is much simpler.
>> >>>
>> >>
>> >>
>> >>>
>> >>> In principle, a LambdaExpression should evaluate to a
>> >>> TemplateMethodModelEx, and then you pass that TemplateMethodModelEx to
>> >>> the called built-in or whatever it is. But with the approach of
>> >>> BuiltInsWithParseTimeParameters we can certainly even skip that, and
>> >>> just bind to the LambdaExpression directly, add a LocalContext that
>> >>> contains the lambda arguments, end evaluate the LambdaExpression right
>> >>> there, in the built-in implementation. Or at least at a very quick
>> >>> glance I think so.
>> >>>
>> >> Not sure I can follow completely but that hint with*
>> >> BuiltInsWithParseTimeParameters* got me started, but at the moment I'm
>> >> stuck as I need to get more familiar with the internals of Freemarker.
>> I am
>> >> also not sure I am on the same page regarding the syntax we are aiming
>> for
>> >> and why I would need to extend the parser when there is something like
>> >> BuiltInsWithParseTimeParameters....
>> >
>> > I assumed that the syntax will be similar to the Java lambda syntax
>> > (see below why), and that's of course needs lexer/parser changes.
>> >
>> >> Here is an example what I have in mind:
>> >> I started with the ?filter() builtin. I had a syntax like this in mind:
>> >>
>> >> *Example 1: ["a","b","c"]?filter(element, element == "c")*
>> >> *Example 2: ["a","b","c"]?filter(element, element == someOtherVariable,
>> >> someOtherVariable="c")*
>> >
>> > Looking at the above one believe that the value of `element` is passed
>> > in as first argument, and the the value of `element == "c"` as the
>> > second, but that's not the case. It's much better if it's visible that
>> > you got some kind of anonymous function definition there without
>> > knowing about the "filter" built-in.
>> >
>> > Java already has a syntax for expressing this kind of thing, so it
>> > would be better to use that familiar syntax. As far as I know it
>> > doesn't conflict with ours (it kind of it does, but not fatally).
>> >
>> > Also, even if for now we only allow this in said built-ins, we
>> > shouldn't exclude the possibility of making this kind of expression
>> > accessible elsewhere as well. Then, on many places the parsed won't
>> > know what kind of value is expected (consider passing a lambda to an
>> > user defined directive for example), so a syntax like above won't
>> > work.
>> >
>> >> Not sure if that's what you have in mind too, but to me it made sense
>> with
>> >> regards to BuiltInsWithParseTimeParameters and I could start without
>> >> touching parser stuff.
>> >
>> > BuiltInsWithParseTimeParameters doesn't affect the syntax (only a
>> > bit...), as a call to a such built-in looks like a call to any other
>> > built-in.
>> >
>> >> *1st argument 'element'* would just be the iterator variable similar to
>> >> <#list ["a","b","c"] as *element*>
>> >> 2nd argument is the filter lambda expression... aka our filter condition
>> >> 3rd+n argument are optional parameters in case used in the lambda
>> expression
>> >
>> > #list is special, similarly as `for ... in ...` is in most languages.
>> > It's not lambda-ish either; you can't write `as element/2` or such.
>> >
>> >> So at first I was looking how <#list> works and found IteratorBlock, and
>> >> though I could reuse it somehow.
>> >>
>> >> Here is some simple pseudo code I played around for the for Example 1:
>> >>
>> >> static class filter_BI extends BuiltInWithParseTimeParameters {
>> >>
>> >>
>> >>
>> >> TemplateModel _eval(Environment env) throws TemplateException {
>> >>
>> >> // sequence
>> >>
>> >> TemplateModel targetValue = target.evalToNonMissing(env);
>> >>
>> >>
>> >>
>> >> List parameters = this.parameters;
>> >>
>> >>
>> >>
>> >> Expression iteratorAlias = (Expression) parameters.get(0);
>> >>
>> >> Expression conditionExpression = (Expression)
>> parameters.get(1);
>> >>
>> >>
>> >>
>> >> TemplateSequenceModel seq = (TemplateSequenceModel)
>> target.eval
>> >> (env);
>> >>
>> >> for (int i = 0; i < seq.size(); i++) {
>> >>
>> >> TemplateModel cur = seq.get(i);
>> >>
>> >>
>> >> // this is where I am stuck at the moment
>> >>
>> >> // I basically want to evaluate conditionExpression
>> >> where iteratorAlias
>> >> is basically what I passed as 'element'
>> >>
>> >> // I am not sure if or how LocalContext could come into
>> play
>> >> here
>> >>
>> >> // basically for each iteration I would assign the
>> current
>> >> loop element to a context variable with the name 'element'
>> >>
>> >> // and then evaluate conditionExpression with that
>> context.
>> >>
>> >> // if conditionExpression is "true" then I would populate
>> >> add the current sequence element 'cur'
>> >>
>> >> // to a new result-List.... and return that.... something
>> >>
>> >> // I wanted to reuse IteratorBlock here somehow, but
>> didn't
>> >> get it to work yet.
>> >>
>> >> // maybe this is a stupid idea, or we just need something
>> >> similar
>> >>
>> >>
>> >>
>> >> }
>> >>
>> >> }
>> >>
>> >>
>> >>
>> >> Ok so far for my pseudo code.... Maybe you could give some more
>> pointers
>> >> based on that... in case this makes any sense ...
>> >
>> > For LocalContext, Environment has a pushLocalContext method. See the
>> > calls to it, like in visit(TemplateElement[], TemplateDirectiveModel,
>> > Map, List).
>> >
>> > To avoid running into more new things at once than necessary, perhaps
>> > you should start with seq?seq_contains(lambda). The end result should
>> > be like:
>> >
>> > users?contains(u -> u.paying)
>> >
>> >>> Another similarity to BuiltInsWithParseTimeParameters is that we won't
>> >>> allow separating the `?someBI` and `(arg)`. Like, with the example of
>> >>> `cond?then(1, 2)`, you aren't allowed to do this:
>> >>>
>> >>> <#assign t=cond?then>
>> >>> ${t(1, 2)}
>> >>>
>> >>> That maybe looks natural, but most other built-ins allow that. Of
>> >>> course we can't allow that for ?filter etc., because then we need
>> >>> closures again.
>> >>>
>> >>> >> Multi-part macro body is also planned. Means, I know it definitely
>> >>> >> should be added, but who knows when that's done... I mean, it's like
>> >>> >> that for what, a decade? (: It's not even decided what it exactly
>> >>> >> does, as there are many ways of approaching this. (I have my own
>> idea
>> >>> >> about what the right compromise would be, but others has other
>> >>> >> ideas...)
>> >>> >>
>> >>> >> Filtering lists bothers me because the template language should be
>> >>> >> (and somewhat indeed is) specialized on listing things on fancy ways
>> >>> >> that used to come up when generating document-like output. (If it
>> >>> >> doesn't do things like that, you might as well use a general purpose
>> >>> >> language.) Thus, that filter is unsolved (filtering with #if is
>> >>> >> verbose and spoils #sep etc.) bothers me a lot.
>> >>> >>
>> >>> >> BTW, ?filter and ?map is also especially handy in our case as
>> >>> >> FreeMarker doesn't support building new sequences (sequences are
>> >>> >> immutable). Although it has sequence concatenation with `+`, it's
>> not
>> >>> >> good for building a sequence one by one, unless the sequence will be
>> >>> >> quite short.
>> >>> >>
>> >>> > Good point.
>> >>> >
>> >>> >
>> >>> >>
>> >>> >> > Cheers,
>> >>> >> > -- Denis.
>> >>> >>
>> >>> >> --
>> >>> >> Thanks,
>> >>> >> Daniel Dekany
>> >>> >>
>> >>> >>
>> >>> >
>> >>>
>> >>> --
>> >>> Thanks,
>> >>> Daniel Dekany
>> >>>
>> >>>
>> >> Thanks
>> >> Christoph
>> >>
>> >
>>
>> --
>> Thanks,
>> Daniel Dekany
>>
>>
>
> --
> Christoph Rüger, Geschäftsführer
> Synesty <https://synesty.com/> - Anbinden und Automatisieren ohne
> Programmieren - Automatisierung, Schnittstellen, Datenfeeds
>
> Xing: https://www.xing.com/profile/Christoph_Rueger2
> LinkedIn: http://www.linkedin.com/pub/christoph-rueger/a/685/198
>
--
Thanks,
Daniel Dekany
