Sunday, February 24, 2019, 12:45:48 AM, Christoph Rüger wrote:
> Thanks you very much. This looks great.
> Our testsuite passes all freemarker related tests.
>
> Also some simple manual tests were successful e.g.
> <#list row.cols?map(col -> col.title)?filter(ct -> ct?contains("name")) as
coltitle >>${coltitle!}<#sep>,</#sep></#list>
> <#list description?split(" ")?filter(word -> word == "test") as
filteredWord>>${filteredWord}</#list>
>
> Regarding other use-cases for lazy-evaluation:
>
> Do you think a ?count (as terminal operation) without building up a list in
> memory?
> E.g.
> Number of products < 1000: ${products?filter(it -> it.price < 1000)?count}
>
> (I tested ${products?filter(it -> it.price < 1000)?size} but I think this
> builds up a list in memory which can be large)
Good point! We don't need to introduce ?count for this though, simply
?size has to be made smarter.
> Or what about other aggregations for numerical values like sum, avg, min,
> max e.g.:
> Cheapest price of products < 1000: ${products?filter(it -> it.price <
> 1000)?map(it -> it.price)?min}
We have ?min and ?max, they also need to be made smarter. As of ?sum
and ?acg... they are doable. But aggregates are mostly useful combined
with "group by". Technically, we can do all that, but I'm not 100%
sure if it's wise to do. The more such operations the template
language supports, the more people will tend to move such calculations
from the Java code that builds the data-model to the template. But the
good practice was always the opposite of that: only do presentation
logic in templates, not "business" calculations. But then again, there
are the situations when the template authors have little control over
the data model (typically, when you make a report/mail template for
some business application that was developed by another party), and
then they are happy that they can get the work done in the template.
So, it's not easy to decide which compromise is the better.
BTW, the kind of "group by" operation that doesn't aggregate anything,
just splits the collection into smaller collections for each distinct
"group by" value, like Collectors.groupBy in Java 8, would be often
useful for presentation logic. The use case is that you get a List of
rows, and let's say you have a Year column in there, and the List is
ordered by that. Then you should render it so that you don't show the
same year again and again. Like if it's a HTML table then you are
supposed to use td.@rowspan in the year column. It's quite awkward to
do such thing currently (try it and you will see...), despite that
it's a common requirement and is clearly a presentation decision, not
really "business logic". So, we could introduce a built-in for that:
collection?splitBy(lambdaOrFunctionOrMethod). I would avoid calling it
groupBy, since that has a different meaning in SQL (the aggregating
groupBy veriant). So now rowspan by year would be like:
<table>
<tr>
<th>Year</th>
<th>Department</th>
<th>Income</th>
</tr>
<#list rows?groupBy(row -> row.year) as sameYearRows>
<#list sameYearRows as row>
<tr>
<#if row?isFirst>
<td rowspan="${sameYearRows?size}">${row.year}</td>
</#if>
<td>${row.department}</td>
<td>${row.income}</td>
</tr>
</#list>
<#list>
</table>
Or if you have to print a new H2 and table for each year:
<#list rows?groupBy(row -> row.year) as sameYearRows>
<h2>${sameYearRows[0].year}<h2>
<table>
<tr>
<th>Department</th>
<th>Income</th>
</tr>
<#list sameYearRows as row>
<tr>
<td>${row.department}</td>
<td>${row.income}</td>
</tr>
</#list>
</table>
<#list>
Lambdas happen to be handy with this as well. Actually, some long
existing built-ins, like ?sortBy, ?contains, etc. could use them as
well (a bit tricky for them technically due to backward compatibility
requirements, but I belive I can work that around.)
> Didn't think this through, but just had the idea while testing.
>
> Anyway, good work!
>
>
>
>
>
>
> Am Sa., 23. Feb. 2019 um 18:19 Uhr schrieb Daniel Dekany <[email protected]
>>:
>
>> I have pushed an implementation of the restricted lambdas we were
>> talking about. Guys, please review/test it.
>>
>> I call these "local lambdas" in the source code (but I'm open for
>> suggestions), as the function they define can only be called in the
>> same variable scope where the lambda was (as we have no closures in
>> the template language, nor final variables). Also, they can only be
>> used as the parameters of the built-ins that explicitly support them.
>> As the subject shows, the main goal was just support filtering for
>> #list, without a nested #if, as that breaks #sep, #else, it?hasNext,
>> it?index, etc. (I will try to make more universal lambdas in FM3... I
>> guess it would be way too tricky in FM2.)
>>
>> For now, I have only added two built-ins that support lambdas: ?filter
>> and ?map. Any ideas what other such built-ins would be often useful in
>> templates (with use case, if possible)?
>>
>> Examples of using ?filter and ?map:
>>
>> <#list products?filter(it -> it.price < 1000) as product>
>> ${product.name}
>> </#list>
>>
>> <#list products?map(it -> it.name) as name>
>> ${name}
>> </#list>
>>
>> Of course these built-ins aren't specific to #list, they can be used
>> anywhere. Naturally, they can be chained as well:
>>
>> <#assign chepProdNames = products
>> ?filter(it -> it.price < 1000)
>> ?map(it -> it.name)
>> >
>>
>> As a side note, ?filter and ?map also accepts FTL function-s and Java
>> methods as its parameter, not only lambdas.
>>
>> A tricky aspect of this feature is lazy evaluation, in similar sense
>> as Java 8 Stream intermediate operations are lazy. On most places,
>> ?filer and ?map are eager, that is, they return a completed sequence
>> of items. That's because our restricted lambdas only work correctly
>> "locally". However, there are very common situations where it's clear
>> that we can use lazy ("streaming") evaluation, as we know where the
>> resulting stream of elements will be consumed:
>>
>> - One such case is when these kind of built-ins are chained. Like in
>> the last example, ?filter doesn't construct a List in memory,
>> instead the elements just flow through it (some is dropped, as it's
>> filtering), into ?map. Only the ?map at the end of the chain will
>> built a List eagerly. Some other built-ins also allow the left-hand
>> built-in to stream, like in ?filter(...)?map(...)?join(", ") no List
>> is built anywhere, instead the elements flow through both ?filter
>> and ?map, and ?join just appends them to the StringBuilder where it
>> creates its results.
>>
>> - #list also enables lazy evaluation to its 1st parameter. So in the
>> #list examples above, yet again no List is built in memory.
>>
>> - There are other such cases, which I didn't implement yet. For
>> example the sequence slice operator, like, xs?filter(f)[10..20],
>> should allow lazy evaluation.
>>
>> Feedback is welcome!
>>
>>
>> Monday, December 17, 2018, 11:39:36 AM, Christoph Rüger wrote:
>>
>> > Hey Daniel,
>> > I'm very sorry, but I didn't make any progress with this. I think I was a
>> > bit over-motivated, but unfortunately I cannot spend more time on this
>> for
>> > various reasons.
>> > You can take this over. I'm glad to help out with testing and feedback.
>> >
>> > Thanks
>> > Christoph
>> >
>> >
>> >
>> >
>> > Am Mo., 17. Dez. 2018 um 11:04 Uhr schrieb Daniel Dekany <
>> [email protected]
>> >>:
>> >
>> >> Any progress in this? I think I will give it a try in the coming days
>> >> otherwise.
>> >>
>> >>
>> >> Sunday, November 18, 2018, 10:31:29 PM, Daniel Dekany wrote:
>> >>
>> >> > See my answers inline...
>> >> >
>> >> > Sunday, November 18, 2018, 8:44:40 PM, Christoph Rüger wrote:
>> >> >
>> >> >> Thanks Daniel for your feedback. See my answers below
>> >> >>
>> >> >> Am So., 11. Nov. 2018 um 19:14 Uhr schrieb Daniel Dekany <
>> >> [email protected]
>> >> >>>:
>> >> >>
>> >> >>> Sunday, November 11, 2018, 11:40:50 AM, Christoph Rüger wrote:
>> >> >>>
>> >> >>> > Am So., 11. Nov. 2018 um 09:25 Uhr schrieb Daniel Dekany <
>> >> >>> [email protected]
>> >> >>> >>:
>> >> >>> >
>> >> >>> >> Saturday, November 10, 2018, 3:08:14 PM, Denis Bredelet wrote:
>> >> >>> >>
>> >> >>> >> > Hi,
>> >> >>> >> >
>> >> >>> >> > Le 9 novembre 2018 à 22:36, Christoph Rüger <
>> [email protected]>
>> >> a
>> >> >>> >> écrit :
>> >> >>> >> >
>> >> >>> >> > Am Fr., 9. Nov. 2018 um 22:55 Uhr schrieb Daniel Dekany <
>> >> >>> >> [email protected]
>> >> >>> >> > :
>> >> >>> >> >
>> >> >>> >> > It's certainly tricky, but as far as I see possible (but then,
>> who
>> >> >>> >> >
>> >> >>> >> > knows what will one find when actually working on it). It's
>> also a
>> >> >>> >> > feature missing a lot. It's especially missing for #list (I
>> know
>> >> that
>> >> >>> >> > you need it for something else), because if you filter the
>> items
>> >> >>> >> > inside #list with #if-s, then #sep, ?hasNext, etc. will not be
>> >> usable.
>> >> >>> >> >
>> >> >>> >> > Let me say that I disagree here.
>> >> >>> >> >
>> >> >>> >> > I do not think that closures are required for FreeMarker, nor
>> that
>> >> >>> they
>> >> >>> >> are a good idea.
>> >> >>> >> >
>> >> >>> >> > If we add new features to the FreeMarker *tempate engine* I
>> would
>> >> >>> >> > rather we focus on multi-part macro body rather than an
>> advanced
>> >> >>> >> language feature like closures.
>> >> >>> >> >
>> >> >>> >> > You can add ?filter and ?map if you want, a simple expression
>> as
>> >> >>> >> parameter should be enough.
>> >> >>> >>
>> >> >>> >> Yes, as I said, we certainly start with only allowing lambdas in
>> >> >>> >> ?filter/?map, also certainly in ?contains.
>> >> >>> >>
>> >> >>> > Would be enough in my opinion and very useful.
>> >> >>> >
>> >> >>> > Is it possiblefor you to give some pointers to the code on how
>> this
>> >> could
>> >> >>> > be implemented? I would maybe like to wrap my head around this a
>> >> little
>> >> >>> bit.
>> >> >>>
>> >> >>> Please feel yourself encouraged! (:
>> >> >>>
>> >> >>> > I started looking at seq_containsBI (
>> >> >>> >
>> >> >>>
>> >>
>> https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L291
>> >> >>> )
>> >> >>> > and
>> >> >>> > and reverseBI (
>> >> >>> >
>> >> >>>
>> >>
>> https://github.com/apache/freemarker/blob/a03a1473b65d9819674b285a0538fed824f37478/src/main/java/freemarker/core/BuiltInsForSequences.java#L264
>> >> >>> )
>> >> >>> > just to find something related (seq_containsBI checks something)
>> and
>> >> >>> > reverseBI returns a new sequence.
>> >> >>> > What I haven't found is a function which takes an Expression as a
>> >> >>> > parameter.
>> >> >>> > Is there something similar already or would that be a new thing?
>> >> >>>
>> >> >>> It's a new thing in that it will be part of the expression syntax
>> >> >>> (even if for now we will only allow lambdas as the parameters of a
>> few
>> >> >>> built-ins, so that we can get away without closures). So it's a new
>> >> >>> Expression subclass, and has to be part of the parser (ftl.jj) as
>> >> >>> well.
>> >> >>
>> >> >> Hmm, that parser stuff is new for me, it'll take me some time to get
>> >> into
>> >> >> it.
>> >> >
>> >> > So it's a JavaCC lexer+parser. With a few twists... sorry, but this
>> >> > code has history... :)
>> >> >
>> >> >>> As of lazy evaluation of parameters expressions, that's already
>> >> >>> done in the built-ins in BuiltInsWithParseTimeParameters, and you
>> will
>> >> >>> see it's trivial to do, but the situation there is much simpler.
>> >> >>>
>> >> >>
>> >> >>
>> >> >>>
>> >> >>> In principle, a LambdaExpression should evaluate to a
>> >> >>> TemplateMethodModelEx, and then you pass that TemplateMethodModelEx
>> to
>> >> >>> the called built-in or whatever it is. But with the approach of
>> >> >>> BuiltInsWithParseTimeParameters we can certainly even skip that, and
>> >> >>> just bind to the LambdaExpression directly, add a LocalContext that
>> >> >>> contains the lambda arguments, end evaluate the LambdaExpression
>> right
>> >> >>> there, in the built-in implementation. Or at least at a very quick
>> >> >>> glance I think so.
>> >> >>>
>> >> >> Not sure I can follow completely but that hint with*
>> >> >> BuiltInsWithParseTimeParameters* got me started, but at the moment
>> I'm
>> >> >> stuck as I need to get more familiar with the internals of
>> Freemarker.
>> >> I am
>> >> >> also not sure I am on the same page regarding the syntax we are
>> aiming
>> >> for
>> >> >> and why I would need to extend the parser when there is something
>> like
>> >> >> BuiltInsWithParseTimeParameters....
>> >> >
>> >> > I assumed that the syntax will be similar to the Java lambda syntax
>> >> > (see below why), and that's of course needs lexer/parser changes.
>> >> >
>> >> >> Here is an example what I have in mind:
>> >> >> I started with the ?filter() builtin. I had a syntax like this in
>> mind:
>> >> >>
>> >> >> *Example 1: ["a","b","c"]?filter(element, element == "c")*
>> >> >> *Example 2: ["a","b","c"]?filter(element, element ==
>> someOtherVariable,
>> >> >> someOtherVariable="c")*
>> >> >
>> >> > Looking at the above one believe that the value of `element` is passed
>> >> > in as first argument, and the the value of `element == "c"` as the
>> >> > second, but that's not the case. It's much better if it's visible that
>> >> > you got some kind of anonymous function definition there without
>> >> > knowing about the "filter" built-in.
>> >> >
>> >> > Java already has a syntax for expressing this kind of thing, so it
>> >> > would be better to use that familiar syntax. As far as I know it
>> >> > doesn't conflict with ours (it kind of it does, but not fatally).
>> >> >
>> >> > Also, even if for now we only allow this in said built-ins, we
>> >> > shouldn't exclude the possibility of making this kind of expression
>> >> > accessible elsewhere as well. Then, on many places the parsed won't
>> >> > know what kind of value is expected (consider passing a lambda to an
>> >> > user defined directive for example), so a syntax like above won't
>> >> > work.
>> >> >
>> >> >> Not sure if that's what you have in mind too, but to me it made sense
>> >> with
>> >> >> regards to BuiltInsWithParseTimeParameters and I could start without
>> >> >> touching parser stuff.
>> >> >
>> >> > BuiltInsWithParseTimeParameters doesn't affect the syntax (only a
>> >> > bit...), as a call to a such built-in looks like a call to any other
>> >> > built-in.
>> >> >
>> >> >> *1st argument 'element'* would just be the iterator variable similar
>> to
>> >> >> <#list ["a","b","c"] as *element*>
>> >> >> 2nd argument is the filter lambda expression... aka our filter
>> condition
>> >> >> 3rd+n argument are optional parameters in case used in the lambda
>> >> expression
>> >> >
>> >> > #list is special, similarly as `for ... in ...` is in most languages.
>> >> > It's not lambda-ish either; you can't write `as element/2` or such.
>> >> >
>> >> >> So at first I was looking how <#list> works and found IteratorBlock,
>> and
>> >> >> though I could reuse it somehow.
>> >> >>
>> >> >> Here is some simple pseudo code I played around for the for Example
>> 1:
>> >> >>
>> >> >> static class filter_BI extends BuiltInWithParseTimeParameters {
>> >> >>
>> >> >>
>> >> >>
>> >> >> TemplateModel _eval(Environment env) throws
>> TemplateException {
>> >> >>
>> >> >> // sequence
>> >> >>
>> >> >> TemplateModel targetValue = target.evalToNonMissing(env);
>> >> >>
>> >> >>
>> >> >>
>> >> >> List parameters = this.parameters;
>> >> >>
>> >> >>
>> >> >>
>> >> >> Expression iteratorAlias = (Expression)
>> parameters.get(0);
>> >> >>
>> >> >> Expression conditionExpression = (Expression)
>> >> parameters.get(1);
>> >> >>
>> >> >>
>> >> >>
>> >> >> TemplateSequenceModel seq = (TemplateSequenceModel)
>> >> target.eval
>> >> >> (env);
>> >> >>
>> >> >> for (int i = 0; i < seq.size(); i++) {
>> >> >>
>> >> >> TemplateModel cur = seq.get(i);
>> >> >>
>> >> >>
>> >> >> // this is where I am stuck at the moment
>> >> >>
>> >> >> // I basically want to evaluate conditionExpression
>> >> >> where iteratorAlias
>> >> >> is basically what I passed as 'element'
>> >> >>
>> >> >> // I am not sure if or how LocalContext could come
>> into
>> >> play
>> >> >> here
>> >> >>
>> >> >> // basically for each iteration I would assign the
>> >> current
>> >> >> loop element to a context variable with the name 'element'
>> >> >>
>> >> >> // and then evaluate conditionExpression with that
>> >> context.
>> >> >>
>> >> >> // if conditionExpression is "true" then I would
>> populate
>> >> >> add the current sequence element 'cur'
>> >> >>
>> >> >> // to a new result-List.... and return that....
>> something
>> >> >>
>> >> >> // I wanted to reuse IteratorBlock here somehow, but
>> >> didn't
>> >> >> get it to work yet.
>> >> >>
>> >> >> // maybe this is a stupid idea, or we just need
>> something
>> >> >> similar
>> >> >>
>> >> >>
>> >> >>
>> >> >> }
>> >> >>
>> >> >> }
>> >> >>
>> >> >>
>> >> >>
>> >> >> Ok so far for my pseudo code.... Maybe you could give some more
>> >> pointers
>> >> >> based on that... in case this makes any sense ...
>> >> >
>> >> > For LocalContext, Environment has a pushLocalContext method. See the
>> >> > calls to it, like in visit(TemplateElement[], TemplateDirectiveModel,
>> >> > Map, List).
>> >> >
>> >> > To avoid running into more new things at once than necessary, perhaps
>> >> > you should start with seq?seq_contains(lambda). The end result should
>> >> > be like:
>> >> >
>> >> > users?contains(u -> u.paying)
>> >> >
>> >> >>> Another similarity to BuiltInsWithParseTimeParameters is that we
>> won't
>> >> >>> allow separating the `?someBI` and `(arg)`. Like, with the example
>> of
>> >> >>> `cond?then(1, 2)`, you aren't allowed to do this:
>> >> >>>
>> >> >>> <#assign t=cond?then>
>> >> >>> ${t(1, 2)}
>> >> >>>
>> >> >>> That maybe looks natural, but most other built-ins allow that. Of
>> >> >>> course we can't allow that for ?filter etc., because then we need
>> >> >>> closures again.
>> >> >>>
>> >> >>> >> Multi-part macro body is also planned. Means, I know it
>> definitely
>> >> >>> >> should be added, but who knows when that's done... I mean, it's
>> like
>> >> >>> >> that for what, a decade? (: It's not even decided what it exactly
>> >> >>> >> does, as there are many ways of approaching this. (I have my own
>> >> idea
>> >> >>> >> about what the right compromise would be, but others has other
>> >> >>> >> ideas...)
>> >> >>> >>
>> >> >>> >> Filtering lists bothers me because the template language should
>> be
>> >> >>> >> (and somewhat indeed is) specialized on listing things on fancy
>> ways
>> >> >>> >> that used to come up when generating document-like output. (If it
>> >> >>> >> doesn't do things like that, you might as well use a general
>> purpose
>> >> >>> >> language.) Thus, that filter is unsolved (filtering with #if is
>> >> >>> >> verbose and spoils #sep etc.) bothers me a lot.
>> >> >>> >>
>> >> >>> >> BTW, ?filter and ?map is also especially handy in our case as
>> >> >>> >> FreeMarker doesn't support building new sequences (sequences are
>> >> >>> >> immutable). Although it has sequence concatenation with `+`, it's
>> >> not
>> >> >>> >> good for building a sequence one by one, unless the sequence
>> will be
>> >> >>> >> quite short.
>> >> >>> >>
>> >> >>> > Good point.
>> >> >>> >
>> >> >>> >
>> >> >>> >>
>> >> >>> >> > Cheers,
>> >> >>> >> > -- Denis.
>> >> >>> >>
>> >> >>> >> --
>> >> >>> >> Thanks,
>> >> >>> >> Daniel Dekany
>> >> >>> >>
>> >> >>> >>
>> >> >>> >
>> >> >>>
>> >> >>> --
>> >> >>> Thanks,
>> >> >>> Daniel Dekany
>> >> >>>
>> >> >>>
>> >> >> Thanks
>> >> >> Christoph
>> >> >>
>> >> >
>> >>
>> >> --
>> >> Thanks,
>> >> Daniel Dekany
>> >>
>> >>
>> >
>> > --
>> > Christoph Rüger, Geschäftsführer
>> > Synesty <https://synesty.com/> - Anbinden und Automatisieren ohne
>> > Programmieren - Automatisierung, Schnittstellen, Datenfeeds
>> >
>> > Xing: https://www.xing.com/profile/Christoph_Rueger2
>> > LinkedIn: http://www.linkedin.com/pub/christoph-rueger/a/685/198
>> >
>>
>> --
>> Thanks,
>> Daniel Dekany
>>
>>
>
--
Thanks,
Daniel Dekany
