Re: [O] [ox-publish, patch] More flexible sitemaps
Hello, Rasmuswrites: > Sorry about the slow reply. Sometimes there's not enough time. So true. > I obviously agree with your criticism. It’s not obvious how to do this > properly. > > If we provide a backend we’d have to move the index preparation to > beginning of each export process as :publishing-function can be a list. > Also, I’m not sure how we’d know about the backend. Before the exporting > starts, we at most know the names of the functions right? If one of the > publishing functions is anonymous we don’t even have that. > > Perhaps the best way is to move keywords back to ox, though I’d rather opt > for something else. You're right, "ox-publish" is mostly back-end agnostic so the information is not readily available. However, we're creating an Org file here. We don't need to have parsed values for keywords. So, what about simply looking for, e.g., #+SUBTITLE value in the document and use it instead as a template replacement? IOW, what about eschewing so-called "environment" altogether? >>> +;; Call function to build sitemap based on files and the project-plist. >>> +(let* ((style (or (plist-get project-plist :sitemap-style) 'tree)) >>> + (fun (intern (format "org-publish-org-sitemap-as-%s" style >> >> Side note : I think this is a bit too smart. It prevents, e.g., from >> grepping for a function name. Maybe >> >> (if (eq style 'something) #'... #') >> >> would be better. > > The point is that it should be easy to supply your own functions. All > sorts of requirements for the index/sitemap could come up, and it is > important to not limit to the tree-style and the flat-style. But that's outside the scope of the patch currently considered. My point is that we should avoid forging function names as a UI. > I might want to provide an index ordered by keywords, say. > > Perhaps styles should be stores in an alist with elements like > > (STYLE STYLE-FUNCTION) ? Some functions may handle multiple styles, as `org-publish-org-sitemap' does. Besides, any user-provided sitemap function is likely to handle its own style without relying on :sitemap-style property. Therefore, providing :sitemap-function and :sitemap-style is enough, there's no need to merge them together. WDYT? Regards, -- Nicolas Goaziou
Re: [O] [ox-publish, patch] More flexible sitemaps
Hi, Sorry about the slow reply. Sometimes there's not enough time. Nicolas Goaziouwrites: >> + (unless (or (file-directory-p file) (directory-name-p file)) > > What is `directory-name-p'? Oh, it's new in Emacs-25. Thanks for pointing that out! The following is about the index function, which is the most "difficult" part. >> + (let* ((org-inhibit-startup t) >> + (visiting (find-buffer-visiting file)) >> + (buffer (or visiting (find-file-noselect file))) >> + (case-fold-search t)) >> +(with-current-buffer buffer >> + ;; protect local variables in open buffers >> + (org-export-with-buffer-copy >> + (let* ((backends (append (list nil) >> +(mapcar 'org-export-backend-name >> + >> org-export-registered-backends))) >> + (value (cl-loop for backend in backends >> + when (org-no-properties >> +(org-element-interpret-data >> + (plist-get (org-export-get-environment backend) >> +prop))) >> + return it))) > > Return value of `org-element-interpret-data' is never nil so the loop > always returns the first back-end. > > In any case, this is sub-optimal since common keywords are retrieved for > each back-end. Also, two back-ends could use the same keyword, with > different values (e.g., using `parsed' or not). > > One option could be to change the policy about keywords in ox.el and > move KEYWORDS and SUBTITLE there. Unfortunately, there is still a change > to miss some cases. > > Another option would be to provide BACKEND to sitemap-function. I think > it can be backward-compatible if we make it an optional argument. I obviously agree with your criticism. It’s not obvious how to do this properly. If we provide a backend we’d have to move the index preparation to beginning of each export process as :publishing-function can be a list. Also, I’m not sure how we’d know about the backend. Before the exporting starts, we at most know the names of the functions right? If one of the publishing functions is anonymous we don’t even have that. Perhaps the best way is to move keywords back to ox, though I’d rather opt for something else. >> + ;; Call function to build sitemap based on files and the project-plist. >> + (let* ((style (or (plist-get project-plist :sitemap-style) 'tree)) >> +(fun (intern (format "org-publish-org-sitemap-as-%s" style > > Side note : I think this is a bit too smart. It prevents, e.g., from > grepping for a function name. Maybe > > (if (eq style 'something) #'... #') > > would be better. The point is that it should be easy to supply your own functions. All sorts of requirements for the index/sitemap could come up, and it is important to not limit to the tree-style and the flat-style. I might want to provide an index ordered by keywords, say. Perhaps styles should be stores in an alist with elements like (STYLE STYLE-FUNCTION) ? >> +@item @code{:sitemap-preamble} >> +@item @code{:sitemap-postamble} >> +@tab Preamble and postamble for sitemap. Useful for inserting >> +@code{#+OPTIONS}, footers etc. See @code{org-publish-sitemap-preamble} >> +for details. > > I don't understand the "useful for inserting @code{#+OPTIONS}" part. > Maybe some examples could be useful. This part of the manual is rather > terse. It might be useful to suppress the printing of the title or the author or similar. Rasmus -- I feel emotional landscapes they puzzle me
Re: [O] [ox-publish, patch] More flexible sitemaps
Robert Kleinwrites: > are those Patches still being worked on? Yes, but they haven't been pushed yet. Rasmus -- May the Force be with you
Re: [O] [ox-publish, patch] More flexible sitemaps
Hi, are those Patches still being worked on? Thanks and best regards Robert On Wed, 01 Jun 2016 17:34:56 +0200 Nicolas Goaziouwrote: > Hello, > > Rasmus writes: > > > This was by far the hardest part... > > Thank you. Some comments follow. > > > +(defun org-publish-find-property (file property reset) > > + "Find the PROPERTY of FILE in project. > > +PROPERTY can be a string or a symbol-property." > > Could you also document RESET argument? > [rest deleted from reply]
Re: [O] [ox-publish, patch] More flexible sitemaps
Hello, Rasmuswrites: > This was by far the hardest part... Thank you. Some comments follow. > +(defun org-publish-find-property (file property reset) > + "Find the PROPERTY of FILE in project. > +PROPERTY can be a string or a symbol-property." Could you also document RESET argument? > + (unless (or (file-directory-p file) (directory-name-p file)) What is `directory-name-p'? > +(let ((prop (cond ((stringp property) > + (intern (downcase (format ":%s" property > + (t property The following might be more robust: (if (keywordp property) property (intern (downcase (format ":%s" property))) > + (and (not reset) (org-publish-cache-get-file-property file prop nil t)) (unless reset ...) Anyway, you don't seem to use the return value. If this is used for side-effect only, you may want to call `org-publish-initialize-cache' instead. > + (let* ((org-inhibit-startup t) > + (visiting (find-buffer-visiting file)) > + (buffer (or visiting (find-file-noselect file))) > + (case-fold-search t)) > +(with-current-buffer buffer > + ;; protect local variables in open buffers > + (org-export-with-buffer-copy > + (let* ((backends (append (list nil) > +(mapcar 'org-export-backend-name > +org-export-registered-backends))) > + (value (cl-loop for backend in backends > + when (org-no-properties > +(org-element-interpret-data > + (plist-get > (org-export-get-environment backend) > +prop))) > + return it))) Return value of `org-element-interpret-data' is never nil so the loop always returns the first back-end. In any case, this is sub-optimal since common keywords are retrieved for each back-end. Also, two back-ends could use the same keyword, with different values (e.g., using `parsed' or not). One option could be to change the policy about keywords in ox.el and move KEYWORDS and SUBTITLE there. Unfortunately, there is still a change to miss some cases. Another option would be to provide BACKEND to sitemap-function. I think it can be backward-compatible if we make it an optional argument. > +(defcustom org-publish-sitemap-file-entry-format "%- [[file:%link][%TITLE]]" >"Format string for site-map file entry. > -You could use brackets to delimit on what part the link will be. > > -%t is the title. > -%a is the author. > -%d is the date formatted using `org-publish-sitemap-date-format'." > +Most keywords can be accessed using a the name of the keyword > +prefixed with \"%\", e.g. \"%TITLE\" or \"%KEYWORDS\". In > +addition, the following keywords are defined. > + > + %fullpath > +The full path of the file. > + > + %relpath > +The relative path of the file. > + > + %filename > + %f > +The filename. > + > + %link > + %l > +The link. > + > + %* > +Leveled headline relative to the base directory. > + > + %* > +Leveled item relative to the base directory. > + > + %author > +The author of the document, the project, or `user-full-name'. > + > + %date > + Date formatted using `org-publish-sitemap-date-format'. > + > +See also `org-publish-sitemap-dir-entry-format'." Could it be possible to use single-char placeholders and `format-spec', which is more robust than replace-match (e.g., it is possibl to escape percent signs)... > +(defcustom org-publish-sitemap-dir-entry-format "%- %f" > + "Format string for site-map file entry. > + > +The following keywords are defined. > + > + %fullpath > +The full path of the file. > + > + %relpath > +The relative path of the file. > + > + %filename > + %f > +The filename. > + > + %link > + %l > +The link. > + > + %* > +Leveled headline relative to the base directory. > + > + %* > +Leveled item relative to the base directory. > + > + %author > +The author of the document, the project, or `user-full-name'. > + > + %date > + Date formatted using `org-publish-sitemap-date-format'. Ditto. > @@ -1292,11 +1394,11 @@ Returns value on success, else nil." > (defun org-publish-cache-ctime-of-src (file) >"Get the ctime of FILE as an integer." >(let ((attr (file-attributes > -(expand-file-name (or (file-symlink-p file) file) > - (file-name-directory file) > + (expand-file-name (or (file-symlink-p file) file) > + (file-name-directory file) Maybe (file-truename file) instead. > + `:sitemap-preamble' > + `:sitemap-postamble' > + > +Preamble and postamble for sitemap. Useful for inserting > +#+OPTIONS: keywords, footers etc. See > +
Re: [O] [ox-publish, patch] More flexible sitemaps
Hi, Thanks for the comments. Nicolas Goaziouwrites: >> (org-publish-find-subtitle): New function. >> (org-publish-org-sitemap-as-tree): New function. >> (org-publish--find-property): Find arbirary property. >> (org-publish-project-alist): Document changes. >> * doc/org.texi (Sitemap): Update documentation. > > All in all, I think this deserves to be split into 3 patches: one for > the preamble-postamble feature, another one for implementing > `org-publish--find-property' and associated refactoring, and the latter > for the sitemap itself. This was by far the hardest part... >> +(autoload 'message-flatten-list "message") >> +(autoload 'dired-tree-lessp "dired-aux") > > I hope we can avoid these. In particular, why are you using > `dired-tree-lessp' instead of `org-publish-compare-directory-files'? AFAIK, org-publish-compare-directory-files can’t be used on its own. It requires variables to be bound around it, e.g. org-publish-sitemap-ignore-case. See org-publish-get-base-files. (I don’t know why the project plist isn’t being passed around.) Anyway, I’m not using neither of these files anymore. Aside: the way ox-publish order files is not super, IMO, as it takes directory into account even if one use a flat list and chronological ordering. That’s a problem for another day, though. >> +(defun org-publish--tree-assoc (key tree) >> +(defun org-pubish--order-files-by-dir-tree (files) > > I don't understand why you need the 2 functions above. You are working > with plain lists, not nested ones. Besides, once the file name are > standardized, isn't tree order equivalent to lexicographic one? Removed. >> +(defun org-publish-find-subtitle (file reset) >> + "Find the title of FILE in project." >> + (org-publish--find-property file :subtitle reset)) > > I don't think this would work. :subtitle is not defined in default > export properties, it is back-end specific. `org-export-get-environment' > without any argument, doesn't catch these. You need to somehow provide > it the back-end. You are right. The "problem" is that the sitemap is done in a before the backend is known. So we can’t do (org-export-get-environment BACKEND). OTOH, only allowing ox.el keywords is too limiting IMO. E.g. a blog sitemap wouldn’t be able to include keywords, though a desirable format for each entry is TITLE DATE KEYWORDS #+INCLUDE: "file.org::lead" But KEYWORDS is specific to ox-html... A couple of workarounds: 1. Guess the backend(s) from the name of :publishing-function. This is not very robust... 2. Find the keyword by search in the file, thus sidestepping org-export-get-environment. It is is not without problems. E.g. if my file contains this, I’d probably only key1. #+Keywords: key1 #+Keywords: key2 3. Go through backend for org-export-get-environment until something is Found. I do this now. Rasmus -- This is the kind of tedious nonsense up with which I will not put >From 4e18d9141c509b03cc5579517547f6e7e73088f3 Mon Sep 17 00:00:00 2001 From: Rasmus Date: Fri, 27 May 2016 16:46:14 +0200 Subject: [PATCH 1/6] ox-publish: Tiny refactor * lisp/ox-publish.el: (org-publish-compare-directory-files): (org-publish-get-base-files-1): Change test. * lisp/ox-publish.el (org-publish-initialize-cache): Tiny fix. --- lisp/ox-publish.el | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lisp/ox-publish.el b/lisp/ox-publish.el index 8ccba99..d60562b 100644 --- a/lisp/ox-publish.el +++ b/lisp/ox-publish.el @@ -467,7 +467,7 @@ matching the regexp SKIP-DIR when recursing through BASE-DIR." (and skip-file (string-match skip-file fnd)) (not (file-exists-p (file-truename f))) (not (string-match match fnd))) - (cl-pushnew f org-publish-temp-files))) + (cl-pushnew f org-publish-temp-files :test #'file-equal-p))) (defun org-publish-get-base-files (project exclude-regexp) "Return a list of all files in PROJECT. @@ -511,7 +511,7 @@ matching filenames." (setq org-publish-temp-files nil) (when org-publish-sitemap-requested (cl-pushnew (expand-file-name (concat base-dir sitemap-filename)) - org-publish-temp-files)) + org-publish-temp-files :test #'file-equal-p)) (org-publish-get-base-files-1 base-dir recurse match ;; FIXME distinguish exclude regexp ;; for skip-file and skip-dir? @@ -1172,7 +1172,7 @@ If FREE-CACHE, empty the cache." (if cexists (load-file cache-file) (setq org-publish-cache - (make-hash-table :test 'equal :weakness nil :size 100)) + (make-hash-table :test #'equal :weakness nil :size 100)) (org-publish-cache-set ":project:" project-name) (org-publish-cache-set ":cache-file:" cache-file)) (unless cexists (org-publish-write-cache-file nil -- 2.8.3 >From 410dc552de2e846f7eed9f98275acffba45f5be0 Mon Sep 17 00:00:00 2001 From: Rasmus Date: Fri, 27 May 2016 15:46:31 +0200
Re: [O] [ox-publish, patch] More flexible sitemaps
Hello, Rasmuswrites: > I've long wanted to use ox to auto-generate something that looks like a > blog index. > > This patch makes ox sitemaps a bit more flexible. For instance, it would > allow me to use something like this for ‘:sitemap-file-entry-format’, > > :sitemap-file-entry-format "* [[file:%l][%t]] > #+include: \"%f::lead\" > > [[file:%l][Read more]]" > > Which would come out as; > > * [[file:link][Title]] > #+Include: "file.org::lead" > > [[File:link][Read more]] > > For the tests I did, it matches the "old" sitemap for list and tree. > > WDYT? It sounds interesting. Usual nitpicking follows. > * lisp/ox-publish.el (org-publish-sitemap-file-entry-format): Support > more formatters. > (org-publish-sitemap-dir-entry-format): New defcustom. > (org-publish-org-sitemap): Use new variables and functions. > (org-publish-org-sitemap-as-list): New function. > (org-publish--tree-assoc): New function. > (org-pubish--order-files-by-dir-tree): New function. > (org-publish-find-title): New function. This is not a new function. > (org-publish-find-subtitle): New function. > (org-publish-org-sitemap-as-tree): New function. > (org-publish--find-property): Find arbirary property. > (org-publish-project-alist): Document changes. > * doc/org.texi (Sitemap): Update documentation. All in all, I think this deserves to be split into 3 patches: one for the preamble-postamble feature, another one for implementing `org-publish--find-property' and associated refactoring, and the latter for the sitemap itself. > +(autoload 'message-flatten-list "message") > +(autoload 'dired-tree-lessp "dired-aux") I hope we can avoid these. In particular, why are you using `dired-tree-lessp' instead of `org-publish-compare-directory-files'? > @@ -399,6 +469,7 @@ This splices all the components into the list." > (defvar org-publish-sitemap-requested) > (defvar org-publish-sitemap-date-format) > (defvar org-publish-sitemap-file-entry-format) > +(defvar org-publish-sitemap-dir-entry-format) The above is not necessary. > + (files (nreverse > + ;; Sitemap shouldn't list itself. > + (cl-delete-if (lambda (f) > + (equal (file-truename f) > + (file-truename sitemap-filename))) See `file-equal-p'. > +(org-publish-get-base-files > + project > + (plist-get project-plist :exclude) >(sitemap-title (or (plist-get project-plist :sitemap-title) > - (concat "Sitemap for project " (car project > - (sitemap-style (or (plist-get project-plist :sitemap-style) > - 'tree)) > - (sitemap-sans-extension > - (plist-get project-plist :sitemap-sans-extension)) > + (concat "Sitemap for project " (car project >(visiting (find-buffer-visiting sitemap-filename)) > - file sitemap-buffer) > -(with-current-buffer > - (let ((org-inhibit-startup t)) > - (setq sitemap-buffer > - (or visiting (find-file sitemap-filename > + (sitemap-buffer (or visiting (find-file sitemap-filename))) > + (insert-pre-or-postamble (function (lambda (pre-or-postamble) No need to wrap `function' around `lambda'. Also, it doesn't "insert" anything, does it? IOW, isn't the name a bit confusing ? > + (when pre-or-postamble You can include the `when' in the cond: (cond ((not pre-or-postamble) nil) ((stringp pre-or-postamble) ...) ...) > + (cond ((stringp > pre-or-postamble) pre-or-postamble) > + ((listp pre-or-postamble) > +(mapconcat 'identity > preamble "\n")) > + ((functionp > pre-or-postamble) > +(funcall > pre-or-postamble project-plist)) > + (t (error (concat > "unknown `:sitemap-preamble' or " > + > "`:sitemap-postamble' format") > +(with-current-buffer (let ((org-inhibit-startup t)) sitemap-buffer) You can drop the `let' part, which is a no-op here. > + ;; Insert sitemap-preamble. > + (funcall insert-pre-or-postamble > +(plist-get project-plist :sitemap-preamble)) > + ;; Call function to build sitemap based on files and the project-plist. > + (insert (funcall (intern > + (concat "org-publish-org-sitemap-as-" > + (symbol-name (or (plist-get project-plist > :sitemap-style) 'tree > +files project-plist)) (intern (format "org-publish-org-sitemap-as-%s" (or
[O] [ox-publish, patch] More flexible sitemaps
Hi, I've long wanted to use ox to auto-generate something that looks like a blog index. This patch makes ox sitemaps a bit more flexible. For instance, it would allow me to use something like this for ‘:sitemap-file-entry-format’, :sitemap-file-entry-format "* [[file:%l][%t]] #+include: \"%f::lead\" [[file:%l][Read more]]" Which would come out as; * [[file:link][Title]] #+Include: "file.org::lead" [[File:link][Read more]] For the tests I did, it matches the "old" sitemap for list and tree. WDYT? I would particularly like feedback on simplification for the ordering of the tree’ed filenames. Rasmus -- This space is left intentionally blank >From e6b35524ba0959b6ca4057555325ec7d755248da Mon Sep 17 00:00:00 2001 From: RasmusDate: Sun, 27 Mar 2016 17:33:06 +0200 Subject: [PATCH 1/2] ox-publish: More flexible sitemaps * lisp/ox-publish.el (org-publish-sitemap-file-entry-format): Support more formatters. (org-publish-sitemap-dir-entry-format): New defcustom. (org-publish-org-sitemap): Use new variables and functions. (org-publish-org-sitemap-as-list): New function. (org-publish--tree-assoc): New function. (org-pubish--order-files-by-dir-tree): New function. (org-publish-find-title): New function. (org-publish-find-subtitle): New function. (org-publish-org-sitemap-as-tree): New function. (org-publish--find-property): Find arbirary property. (org-publish-project-alist): Document changes. * doc/org.texi (Sitemap): Update documentation. --- doc/org.texi | 20 ++-- lisp/ox-publish.el | 319 ++--- 2 files changed, 241 insertions(+), 98 deletions(-) diff --git a/doc/org.texi b/doc/org.texi index 025baaa..b3517c0 100644 --- a/doc/org.texi +++ b/doc/org.texi @@ -14570,8 +14570,9 @@ becomes @file{sitemap.html}). @item @code{:sitemap-function} @tab Plug-in function to use for generation of the sitemap. -Defaults to @code{org-publish-org-sitemap}, which generates a plain list -of links to all files in the project. +Defaults to @code{org-publish-org-sitemap}, which generates a plain list of +links to all files in the project. See further details in +@code{org-publish-project-alist}. @item @code{:sitemap-sort-folders} @tab Where folders should appear in the sitemap. Set this to @code{first} @@ -14590,12 +14591,9 @@ a file is retrieved with @code{org-publish-find-date}. @tab Should sorting be case-sensitive? Default @code{nil}. @item @code{:sitemap-file-entry-format} -@tab With this option one can tell how a sitemap's entry is formatted in the -sitemap. This is a format string with some escape sequences: @code{%t} stands -for the title of the file, @code{%a} stands for the author of the file and -@code{%d} stands for the date of the file. The date is retrieved with the -@code{org-publish-find-date} function and formatted with -@code{org-publish-sitemap-date-format}. Default @code{%t}. +@item @code{:sitemap-dir-entry-format} +@tab With this option one can tell how the entries of the sitemap is +formatted. See @code{org-publish-sitemap-file-entry-format} for details. @item @code{:sitemap-date-format} @tab Format string for the @code{format-time-string} function that tells how @@ -14607,6 +14605,12 @@ a sitemap entry's date is to be formatted. This property bypasses Useful to have cool URIs (see @uref{http://www.w3.org/Provider/Style/URI}). Defaults to @code{nil}. +@item @code{:sitemap-preamble} +@item @code{:sitemap-postamble} +@tab Preamble and postamble for sitemap. Useful for inserting +@code{#+OPTIONS}, footers etc. See @code{org-publish-sitemap-preamble} +for details. + @end multitable @node Generating an index diff --git a/lisp/ox-publish.el b/lisp/ox-publish.el index 8ccba99..b791e9a 100644 --- a/lisp/ox-publish.el +++ b/lisp/ox-publish.el @@ -41,6 +41,8 @@ (require 'cl-lib) (require 'format-spec) (require 'ox) +(autoload 'message-flatten-list "message") +(autoload 'dired-tree-lessp "dired-aux") @@ -217,10 +219,15 @@ a site-map of files or summary page for a given project. `:sitemap-style' -Can be `list' (site-map is just an itemized list of the -titles of the files involved) or `tree' (the directory -structure of the source files is reflected in the site-map). -Defaults to `tree'. +By default `list' (site-map is a list of files) or +`tree' (the directory structure of the source files is +reflected in the site-map). Defaults to `tree'. Files are +formatted according to `:sitemap-file-entry-format', +directories according to `:sitemap-dir-entry-format'. To add +new styles STYLE define a new function +`org-publish-org-sitemap-as-STYLE' that takes a list of files +and project-plist as arguments (assuming `:sitemap-function' +is `org-publish-org-sitemap'). `:sitemap-sans-extension' @@ -228,6 +235,20 @@ a site-map of files or summary page for a given project. cool URIs (see