A few quick thoughts on that: - We should replace the "document" term with something more speaking. It doesn't tell that it's some kind of input. Also, most of these inputs aren't something that people typically call documents. Like a csv file, or a database table, which is not even a file (OK we don't support such thing at the moment). I think, maybe "data source" is a safe enough term. (It also rhymes with data model.) - You have separate "template" and "document" "mode", that applies to a whole run. I think such specialization won't be helpful. We could just say, on the conceptual level at lest, that we need a set of "outputs generators". An output generator is an object (in the API) that specifies a template, a data-model (where the data-model is possibly populated with "documents"), and an output "sink" (a file path, or stdout), and can generate the output itself. A practical way of defining the output generators in a CLI application is via a bunch of files, each defining an output generator. Some of those files is maybe a template (that you can even detect from the file extension), or a data file that we currently call a "document". They could freely mix inside the same run. I have also met use case when you have a single table (single "document"), and each record in it yields an output file. That can also be described in some file format, or really in any other way, like directly in command line argument, via API, etc. - You have multiple documents without associated symbolical name in some examples. Templates can't identify those then in a well maintainable way. The actual file name is often not a good identifier, can change over time, and you might don't even have good control over it, like you already receive it as a parameter from somewhere else, or someone moves/renames that files that you need to read. Index is also not very good, but I have written about that earlier.
On Wed, Feb 26, 2020 at 9:33 AM Siegfried Goeschl < [email protected]> wrote: > Hi folks, > > still wrapping my side around but assembled some thoughts here - > https://gist.github.com/sgoeschl/b09b343a761b31a6c790d882167ff449 > > Thanks in advance, > > Siegfried Goeschl > > > > > On 23 Feb 2020, at 23:14, Daniel Dekany <[email protected]> wrote: > > > > What you are describing is more like the angle that FMPP took initially, > > where templates drive things, they generate the output for themselves > (even > > multiple output files if they wish). By default output files name (and > > relative path) is deduced from template name. There was also a global > > data-model, built in a configuration file (or equally, built via command > > line arguments, or both mixed), from which templates get whatever data > they > > are interested in. Take a look at the figures here: > > http://fmpp.sourceforge.net/qtour.html. Later, this concept was > generalized > > a bit more, because you could add XML files at the same place where you > > have the templates, and then you could associate transform templates to > the > > XML files (based on path pattern and/or the XML document element). Now > > that's like what freemarker-generator had initially (data files drive > > output, and the template is there to transform it). > > > > So I think the generic mental model would like this: > > > > 1. You got files that drive the process, let's call them *generator > > files* for now. Usually, each generator file yields an output file (but > > maybe even multiple output files, as you might saw in the last figure). > > These generator files can be of many types, like XML, JSON, XLSX (as > in the > > original freemarker-generator), and even templates (as is the norm in > > FMPP). If the file is not a template, then you got a set of transformer > > templates (-t CLI option) in a separate directory, which can be > associated > > with the generator files base on name patterns, and even based on > content > > (schema usually). If the generator file is a template (so that's a > > positional @Parameter CLI argument that happens to be an *.ftl, and is > not > > a template file specified after the "-t" option), then you just > > Template.process(...) it, and it prints what the output will be. > > 2. You also have a set of variables, the global data-model, that > > contains commonly useful stuff, like what you now call parameters (CLI > > -Pname=value), but also maybe data loaded from JSON, XML, etc.. Those > data > > files aren't "generator files". Templates just use them if they need > them. > > An important thing here is to reuse the same mechanism to read and > parse > > those data files, which was used in templates when transforming > generator > > files. So we need a common format for specifying how to load data > files. > > That's maybe just FTL that #assigns to the variables, or maybe more > > declarative format. > > > > What I have described in the original post here was a less generic form > of > > this, as I tried to be true with the original approach. I though the > > proposal will be drastic enough as it is... :) There, the "main" document > > is the "generator file" from point 1, the "-t" template is the transform > > template for the "main" document, and the other named documents ("users", > > "groups") is a poor man's shared data-model from point 2 (together with > > with -PName=value). > > > > There's further somewhat confusing thing to get right with the > > list-of-documents (`DocuentList`, `NamedDocumentLists`) thing though. In > > the model above, as per point 1, if you list multiple data files, each > will > > generate a separate output file. So, if you need take in a list of files > to > > transform it to a single output file (or at least with a single transform > > template execution), then you have to be explicit about that, as that's > not > > the default behavior anymore. But it's still absolutely possible. Imagine > > it as a "list of XLSX-es" is itself like a file format. You need some CLI > > (and Maven config, etc.) syntax to express that, but that shouldn't be a > > big deal. > > > > > > > > On Sun, Feb 23, 2020 at 9:43 PM Siegfried Goeschl < > > [email protected]> wrote: > > > >> Hi Daniel, > >> > >> Good timing - I was looking at a similar problem from different angle > >> yesterday (see below) > >> > >> Don't have enough time to answer your email in detail now - will do that > >> tomorrow evening > >> > >> Thanks in advance, > >> > >> Siegfried Goeschl > >> > >> > >> ===. START > >> # FreeMarker CLI Improvement > >> ## Support Of Multiple Template Files > >> Currently we support the following combinations > >> > >> * Single template and no data files > >> * Single template and one or more data files > >> > >> But we can not support the following use case which is quite typical in > >> the cloud > >> > >> __Convert multiple templates with a single data file, e.g copying a > >> directory of configuration files using a JSON configuration file__ > >> > >> ## Implementation notes > >> * When we copy a directory we can remove the `ftl`extension on the fly > >> * We might need an `exclude` filter for the copy operation > >> * Initially resolve to a list of template files and process one after > >> another > >> * Need to calculate the output file location and extension > >> * We need to rename the existing command line parameters (see below) > >> * Do we need multiple include and exclude filter? > >> * Do we need file versus directory filters? > >> > >> ### Command Line Options > >> ``` > >> --input-encoding : Encoding of the documents > >> --output-encoding : Encoding of the rendered template > >> --template-encoding : Encoding of the template > >> --output : Output file or directory > >> --include-document : Include pattern for documents > >> --exclude-document : Exclude pattern for documents > >> --include-template: Include pattern for templates > >> --exclude-template : Exclude pattern for templates > >> ``` > >> > >> ### Command Line Examples > >> ```text > >> # Copy all FTL templates found in "ext/config" to the "/config" > directory > >> using the data from "config.json" > >>> freemarker-cli -t ./ext/config --include-template *.ftl --o /config > >> config.json > >>> freemarker-cli --template ./ext/config --include-template *.ftl > --output > >> /config config.json > >> > >> # Bascically the same using a named document "configuration" > >> # It might make sense to expose "conf" directly in the FreeMarker data > >> model > >> # It might make sens to allow URIs for loading documents > >>> freemarker-cli -t ./ext/config/*.ftl -o /config -d > >> configuration=config.json > >>> freemarker-cli --template ./ext/config --include-template *.ftl > --output > >> /config --document configuration=config.json > >>> freemarker-cli --template ./ext/config --include-template *.ftl > --output > >> /config --document configuration=file:///config.json > >> > >> # Bascically the same using an environment variable as named document > >>> freemarker-cli -t ./ext/config --include-template *.ftl -o /config -d > >> configuration=env:///CONFIGURATION > >>> freemarker-cli --template ./ext/config --include-template *.ftl > --output > >> /config --document configuration=env:///CONFIGURATION > >> ``` > >> === END > >> > >> > >>> On 23.02.2020, at 16:37, Daniel Dekany <[email protected]> wrote: > >>> > >>> Input documents is a fundamental concept in freemarker-generator, so we > >>> should think about that more, and probably refine/rework how it's done. > >>> > >>> Currently it works like this, with CLI at least. > >>> > >>> freemarker-cli > >>> -t access-report.ftl > >>> somewhere/foo-access-log.csv > >>> > >>> Then in access-report.ftl you have to do something like this: > >>> > >>> <#assign doc = Documents.get(0)> > >>> ... process doc here > >>> > >>> (The more idiomatic Documents[0] won't work. Actually, that lead to a > >> funny > >>> chain of coincidences: It returned the string "D", then > >> CSVTool.parse(...) > >>> happily parsed that to a table with the single column "D", and 0 rows, > >> and > >>> as there were 0 rows, the template didn't run into an error because > >>> row.myExpectedColumn refers to a missing column either, so the process > >>> finished with success. (: Pretty unlucky for sure. The root was > >>> unintentionally breaking a FreeMarker idiom though; eventually we will > >> have > >>> to work on those too, but, different topic.) > >>> > >>> However, actually multiple input documents can be passed in: > >>> > >>> freemarker-cli > >>> -t access-report.ftl > >>> somewhere/foo-access-log.csv > >>> somewhere/bar-access-log.csv > >>> > >>> Above template will still work, though then you ignored all but the > first > >>> document. So if you expect any number of input documents, you probably > >> will > >>> have to do this: > >>> > >>> <#list Documents.list as doc> > >>> ... process doc here > >>> </#list> > >>> > >>> (The more idiomatic <#list Documents as doc> won't work; but again, > those > >>> we will work out in a different thread.) > >>> > >>> > >>> So, what would be better, in my opinion. I start out from what I think > >> are > >>> the common uses cases, in decreasing order of frequency. Goal is to > make > >>> those less error prone for the users, and simpler to express. > >>> > >>> USE CASE 1 > >>> > >>> You have exactly 1 input documents, which is therefore simply "the" > >>> document in the mind of the user. This is probably the typical use > case, > >>> but at least the use case users typically start out from when starting > >> the > >>> work. > >>> > >>> freemarker-cli > >>> -t access-report.ftl > >>> somewhere/foo-access-log.csv > >>> > >>> Then `Documents.get(0)` is not very fitting. Most importantly it's > error > >>> prone, because if the user passed in more than 1 documents (can even > >> happen > >>> totally accidentally, like if the user was lazy and used a wildcard > that > >>> the shell exploded), the template will silently ignore the rest of the > >>> documents, and the singe document processed will be practically picked > >>> randomly. The user might won't notice that and submits a bad report or > >> such. > >>> > >>> I think that in this use case the document should be simply referred as > >>> `Document` in the template. When you have multiple documents there, > >>> referring to `Document` should be an error, saying that the template > was > >>> made to process a single document only. > >>> > >>> > >>> USE CASE 2 > >>> > >>> You have multiple input documents, but each has different role > (different > >>> schema, maybe different file type). Like, you pass in users.csv and > >>> groups.csv. Each has difference schema, and so you want to access them > >>> differently, but in the same template. > >>> > >>> freemarker-cli > >>> [...] > >>> --named-document users somewhere/foo-users.csv > >>> --named-document groups somewhere/foo-groups.csv > >>> > >>> Then in the template you could refer to them as: > `NamedDocuments.users`, > >>> and `NamedDocuments.groups`. > >>> > >>> Use Case 1, and 2 can be unified into a coherent concept, where > >> `Document` > >>> is just a shorthand for `NamedDocuments.main`. It's called "main" > because > >>> that's "the" document the template is about, but then you have to added > >>> some helper documents, with symbolic names representing their role. > >>> > >>> freemarker-cli > >>> -t access-report.ftl > >>> --document-name=main somewhere/foo-access-log.csv > >>> --document-name=users somewhere/foo-users.csv > >>> --document-name=groups somewhere/foo-groups.csv > >>> > >>> Here, `Document` still works in the template, and it refers to > >>> `somewhere/foo-access-log.csv`. (While omitting --document-name=main > >> above > >>> would be cleaner, I couldn't figure out how to do that with Picocli. > >>> Anyway, for now the point is the concept, which is not specific to > CLI.) > >>> > >>> > >>> USE CASE 3 > >>> > >>> Here you have several of the same kind of documents. That has a more > >>> generic sub-use-case, when you have explicitly named documents (like > >>> "users" above), and for some you expect multiple input files. > >>> > >>> freemarker-cli > >>> -t access-report.ftl > >>> --document-name=main somewhere/foo-access-log.csv > >>> somewhere/bar-access-log.csv > >>> --document-name=users somewhere/foo-users.csv > >>> somewhere/bar-users.csv > >>> --document-name=groups somewhere/global-groups.csv > >>> > >>> The template must to be written with this use case in mind, as now it > has > >>> #list some of the documents. (I think in practice you hardly ever want > to > >>> get a document by hard coded index. Either you don't know how many > >>> documents you have, so you can't use hard coded indexes, or you do, and > >>> each index has a specific meaning, but then you should name the > documents > >>> instead, as using indexes is error prone, and hard to read.) > >>> Accessing that list of documents in the template, maybe could be done > >> like > >>> this: > >>> - For the "main" documents: `DocumentList` > >>> - For explicitly named documents, like "users": > >> `NamedDocumentLists.users` > >>> > >>> > >>> SUMMING UP > >>> > >>> To unify all 3 use cases into a coherent concept: > >>> - `NamedDocumentLists.<name>` is the most generic form, and while you > can > >>> achieve everything with it, using it requires your template to handle > the > >>> most generic case too. So, I think it would be rarely used. > >>> - `DocumentList` is just a shorthand for `NamedDocumentLists.main`. > It's > >>> used if you only have one kind of documents (single format and schema), > >> but > >>> potentially multiple of them. > >>> - `NamedDocuments.<name>` expresses that you expect exactly 1 document > of > >>> the given name. > >>> - `Document` is just a shorthand for `NamedDocuments.main`. This is for > >> the > >>> most natural/frequent use case. > >>> > >>> That's 4 possible ways of accessing your documents, which is a > trade-off > >>> for the sake of these: > >>> - Catching CLI (or Maven, etc.) input where the template output likely > >> will > >>> be wrong. That's only possible if the user can communicate its intent > in > >>> the template. > >>> - Users don't need to deal with concepts that are irrelevant in their > >>> concrete use case. Just start with the trivial, `Document`, and later > if > >>> the need arises, generalize to named documents, document lists, or > both. > >>> > >>> > >>> What do guys think? > >> > >> > >
