I have replaced the old TemplateLoader interface with the new one in
the freemarker-3 branch, and "migrated" all the included
TemplateLoader-s and tests.
More eyes see more, so check it out if you can, and tell if you see
anything to improve.
As we don't have a DatabaseTemplateLoader yet, one of the important
points of this new TemplateLoader wasn't demonstrated now. A good way
of spotting the rough edges would be if someone implements that for
example.
Thanks!
Monday, February 6, 2017, 12:40:15 AM, Daniel Dekany wrote:
> In FM2 we had some problems with the TemplateLoader interface:
>
> - The major problem with is that to load a template you need to do
> multiple round trips to the storage mechanism: check if the template
> "file" exists, then get its last modification date, then read its
> content. This is particularly problematic for databases, but often
> even HTTP could pack these into a single round trip.
>
> - When the <#ftl encoding=...> header disagrees with the actual
> charset used for reading the template, the whole template has to
> be re-read (I/O!). That's because we get a Reader, so we can't
> rewind the InputStream behind it and start reading it again with
> the new charset.
>
> - To detect changes one can only use the last modification date (not a
> revision number or hash). It can be especially problematic if
> templates can change so fast, that the clock may doesn't tick
> between them.
>
> - TemplateLoader can't return meta-info like the output format (MIME
> type basically) of the template. Some storages could do that.
>
> - Some storages (like databases) support some kind of atomicity and
> transaction isolation, but the TemplateLoader mechanism doesn't allow
> you to utilize these.
>
> - The "template source" is a tricky to understand for implementators, as
> it servers both as a handle that can be closed, and as a long lived cache
> key component.
>
> I propose a totally new TemplateLoader design to counter all these
> problems in FM3. I tell the idea mostly in code bellow. Please share
> your thoughts! (I was also thinking about backporting this to FM2, but
> it was not feasible.)
>
>
> /**
> * This is the one that replaces the TemplateLoader of FM2.
> */
> public interface TemplateLoader {
>
> /**
> * Creates a new session, or returns {@code null} if the
> template loader implementation doesn't support sessions.
> * See {@link TemplateLoaderSession} for more information about sessions.
> */
> TemplateLoaderSession createSession();
>
> /**
> * Loads the template content together with meta-data such as
> the version (usually the last modification time),
> * optionally conditionally. Note how all these operations
> (existence check, up-to-date check, opening for reading)
> * were put into one atomic unit. This allows you utilize the
> capabilities of the storage mechanism to spare round
> * trips, or even to add atomicity guarantees.
> *
> * @param name
> * The name (template root directory relative path) of the
> template; same as in FM2.
> * @param ifSourceDiffersFrom
> * If we only want to load the template if its
> source differs from this. {@code null} if you want the
> * template to be loaded unconditionally. If this is {@code
> null} then the
> * {@code ifVersionDiffersFrom} parameter must be {@code null}
> too. See
> * {@link TemplateLoadingResult#getSource()} for more about
> versions.
> * @param ifVersionDiffersFrom
> * If we only want to load the template if its
> version (which is usually the last modification time)
> * differs from this. {@code null} if {@code
> ifSourceDiffersFrom} is {@code null}, or if the backing
> * storage from which the {@code
> ifSourceDiffersFrom} template source comes from doesn't store a version.
> * See {@link TemplateLoadingResult#getVersion()} for more
> about versions.
> *
> * @return Not {@code null}.
> */
> TemplateLoadingResult load(String name, TemplateLoadingSource
> ifSourceDiffersFrom, Serializable ifVersionDiffersFrom,
> TemplateLoaderSession session) throws IOException;
>
> /**
> * Invoked by {@link Configuration#clearTemplateCache()} to
> instruct this template loader to throw away its current
> * state (some kind of cache usually) and start afresh. For
> most {@link TemplateLoader} implementations this does
> * nothing.
> */
> void resetState();
>
> }
>
> /**
> * Return value of {@link TemplateLoader#load(String,
> TemplateLoadingSource, Serializable, TemplateLoaderSession)}
> */
> public final class TemplateLoadingResult {
>
> public static final TemplateLoadingResult NOT_FOUND = new
> TemplateLoadingResult(
> TemplateLoadingResultStatus.NOT_FOUND);
> public static final TemplateLoadingResult NOT_MODIFIED = new
> TemplateLoadingResult(
> TemplateLoadingResultStatus.NOT_MODIFIED);
>
> /**
> * Creates an instance with status {@link
> TemplateLoadingResultStatus#OPENED}, for a storage mechanism that
> * naturally returns the template content as sequence of {@code
> char}-s as opposed to a sequence of {@code byte}-s.
> * This is the case for example when you store the template in
> a database in a varchar or CLOB. Do <em>not</em> use
> * this constructor for stores that naturally return binary
> data instead (like files, class loader resources,
> * BLOB-s, etc.), because using this constructor will disable
> FreeMarker's charset selection mechanism.
> */
> public TemplateLoadingResult(TemplateLoadingSource source,
> Serializable version, Reader reader,
> TemplateConfiguration templateConfiguration) { ... }
>
> /**
> * Creates an instance with status {@link
> TemplateLoadingResultStatus#OPENED}, for a storage mechanism that
> * naturally returns the template content as sequence of {@code
> byte}-s as opposed to a sequence of {@code char}-s.
> * This is the case for example when you store the template in
> a file, classpath resource, or BLOB. Do <em>not</em>
> * use this constructor for stores that naturally return text
> instead (like database varchar and CLOB columns).
> */
> public TemplateLoadingResult(TemplateLoadingSource source,
> Serializable version, InputStream inputStream,
> TemplateConfiguration templateConfiguration) { ... }
>
> /**
> * Returns non-{@code null} exactly if {@link #getStatus()} is
> {@link TemplateLoadingResultStatus#OPENED} and the
> * backing store mechanism returns content as {@code byte}-s,
> as opposed to as {@code chars}-s. The return value is
> * always the same instance, no mater when and how many times this method
> is called.
> */
> public InputStream getInputStream() {
> return inputStream;
> }
>
> /**
> * Tells what kind of result this is; see the documentation of
> {@link TemplateLoadingResultStatus}.
> */
> public TemplateLoadingResultStatus getStatus() {
> return status;
> }
>
> /**
> * Same as "template source" FM2, but it's simpler, as it only
> focuses the usage as part of the cache key,
> * and can't be closed. If you aren't familiar with FM2
> template sources, see {@link TemplateLoadingSource}
> * below.
> */
> public TemplateLoadingSource getSource() {
> return source;
> }
>
> /**
> * This replaced the lastModifed of FM2, and is more flexible
> as it can be a revision number, a cryptographic hash,
> * etc. Only set if the result status is {@link
> TemplateLoadingResultStatus#OPENED} and the backing storage stores
> * such information. Version objects are compared with each
> other with their {@link Object#equals(Object)} method.
> */
> public Serializable getVersion() {
> return version;
> }
>
> /**
> * Similar to {@link #getInputStream()}, but used when the
> backing storage mechanism returns content as
> * {@code char}-s, as opposed to as {@code byte}-s.
> */
> public Reader getReader() {
> return reader;
> }
>
> /**
> * If {@link #getStatus()} is {@link
> TemplateLoadingResultStatus#OPENED}, and the template loader stores such
> * information (which is rare) then it returns the {@link
> TemplateConfiguration} applicable to the template,
> * otherwise it returns {@code null}. If there are {@link
> TemplateConfiguration}-s coming from other
> * sources, such as from {@link
> Configuration#getTemplateConfigurations()}, this won't replace them, but will
> be
> * merged with them, with properties coming from the returned
> {@link TemplateConfiguration} having the highest
> * priority.
> */
> public TemplateConfiguration getTemplateConfiguration() {
> return templateConfiguration;
> }
>
> }
>
> /**
> * Used for the value of {@link TemplateLoadingResult#getStatus()}.
> */
> public enum TemplateLoadingResultStatus {
>
> /**
> * The template with the requested name doesn't exist (not to
> be confused with "wasn't accessible due to error").
> */
> NOT_FOUND,
>
> /**
> * If the template was found, but its source and version is the
> same as that which was provided to
> * {@link TemplateLoader#load(String, TemplateLoadingSource,
> Serializable, TemplateLoaderSession)} (from a cache
> * presumably), so its content wasn't opened for reading.
> */
> NOT_MODIFIED,
>
> /**
> * If the template was found and its content is ready for reading.
> */
> OPENED
>
> }
>
> /**
> * The point of {@link TemplateLoadingSource} is that with their
> {@link #equals(Object)} method we can tell if two cache
> * entries were generated from the same physical resource or not.
> Comparing the template names isn't enough, because a
> * {@link TemplateLoader} may uses some kind of fallback mechanism,
> such as delegating to other {@link TemplateLoader}-s
> * until the template is found. Like if we have two {@link
> FileTemplateLoader}-s with different physical root
> * directories, both can contain {@code "foo/bar.ftl"}, but
> obviously the two files aren't the same.
> */
> public interface TemplateLoadingSource extends Serializable {
> // Empty
> }
>
> /**
> * Stores shared state between {@link TemplateLoader} operations
> that are executed close to each other in the same
> * thread. For example, a {@link TemplateLoader} that reads from a
> database might wants to store the database
> * connection in it for reuse. The goal of sessions is mostly to
> increase performance. However, because a
> * {@link TemplateCache#getTemplate(String, java.util.Locale,
> Object, String, boolean)} call is executed inside a single
> * session, sessions can be also be utilized to ensure that the
> template lookup (see {@link TemplateLookupStrategy})
> * happens on a consistent view (a snapshot) of the backing
> storage, if the backing storage mechanism supports such
> * thing.
> *
> * <p>
> * The {@link TemplateLoaderSession} implementation is (usually)
> specific to the {@link TemplateLoader}
> * implementation. If your {@link TemplateLoader} implementation
> can't take advantage of sessions, you don't have to
> * implement this interface, just return {@code null} for {@link
> TemplateLoader#createSession()}.
> *
> * <p>
> * {@link TemplateLoaderSession}-s should be lazy, that is,
> creating an instance should be very fast and should not
> * cause I/O. Only when (and if ever) the shared resource stored in
> the session is needed for the first time should the
> * shared resource be initialized.
> *
> * <p>
> * {@link TemplateLoaderSession}-s need not be thread safe.
> */
> public interface TemplateLoaderSession {
>
> /**
> * Closes this session, freeing any resources it holds. Further
> operations involving this session should fail, with
> * the exception of {@link #close()} itself, which should be silently
> ignored.
> */
> public void close() throws IOException;
>
> public boolean isClosed();
>
> }
>
>
> So, the template loading sequence for template not yet in the cache is
> something like this:
>
> session = templateLoader.createSession();
>
> // TemplateLookupStrategy kicks in... of course its some loop in reality:
> res = templateLoader.load("foo_en_US.ftl", null, null, session);
> if (res.status == NOT_FOUND) res =
> templateLoader.load("foo_en.ftl", null, null, session);
> if (res.status == NOT_FOUND) res = templateLoader.load("foo.ftl", null,
> null, session);
> if (res.status == NOT_FOUND) throw new TemplateNotFoundException();
>
> Reader reader;
> if (res.getReader() != null) {
> reader = res.getReader(); // Charset is not relevant
> } else {
> reader = new InputStreamReader(res.getInputStream,
> figureOutCharsetFromCfgAndSuch());
> }
>
> template = new Template(reader, ...);
> reader.close();
>
> session.close();
>
> For cached templates after the update delay you will fill those two
> null parameters with the source and version from the cache entry, an
> then you skip most of the rest if res.status == NOT_MODIFIED.
>
> I would also note that I have simplified the case of the
> InputStreamReader above. In reality you will have markSupported()
> true, have to add a mark at position 0, then drop the mark during
> parsing when you know that no <#ftl encoding=...> can come anymore, and
> so you are safe. If <#ftl encoding=...> interferes, we will have to
> reset() the stream, recreate the InputStreamReader, and parse again.
> (We don't re-read from the TemplateLoader, we just re-parse.)
>
--
Thanks,
Daniel Dekany
