Re: [Feature] add a new org-attach dispatcher command to offline save web page

[stardiviner]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256


Ihor Radchenko  writes:

>> As I said, PATCH welcome, I admired many times I don't have ability to build 
>> a
>> complex archive functionality on url.el or wget or curl.
>
> I have found the following solution [1] using wget:
>
> wget --mirror -p --convert-links -P ./LOCAL-DIR WEBSITE-URL

I know this "--mirror" option. I used this solution to archive web page before.
Make this command work in org-attach is a little bit of complicated. I have
considered it on current org-attach code. With my Elisp ability, can't write it
out. Still not very good at Elisp :(

>
> This will not bundle the page into a single file, but it is better than
> nothing. org-attach does not have to attach exactly one file.

You can check out function `org-attach-attach` source code, the "cond" structure
is the most commands.

I remember org-attach dispatcher used to have a command to move directory
directly. But it seems removed. Now, all org-attach commands only supports file.
I also search Elisp functions, can't find a function to move directory around.
So PATCH welcome. I remember you send patch on Org Mode mailing list, I believe
you at least is same kind of newbie like me who can write a little Elisp. :)

>
> [1] 
> https://askubuntu.com/questions/411540/how-to-get-wget-to-download-exact-same-web-page-html-as-browser
>
> Best,
> Ihor
>
> stardiviner  writes:
>
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA256
>>
>>
>> Matthew Lundin  writes:
>>
>>> Ihor Radchenko  writes:
>>>
 It does not mean that attaching URL directly is not worth including
 into org. This sounds pretty common use case, especially considering
 the number of packages providing similar feature. You mentioned
 org-board, but there is also org-download and org-web-tools.
>>>
>>> My uneasiness has more to do with the specificity of the dependence on
>>> monolith and the way that is hard-coded into the patch. When it comes to
>>> patches, I think priority should go to those that are configurable,
>>> accessible, and useful for everyone as opposed to those that have
>>> hard-coded work-flows or highly-specific user configurations.
>>
>> I don't think it's hardcoded, As I said in my first email, It's an 
>> `defcustom',
>> isn't it? User totally can change that. And I used `executable-find', the 
>> value
>> will be `nil' when monolith is not available. And I don't have ability to
>> archive same functionality with wget or curl. And as I said before, PATCH
>> welcome.
>>
>>>
 I agree that monolith is completely uncommon tool and I would not expect
 the majority of users to have it installed, but the same functionality
 utilising built-in url.el (as a default) should be acceptable.
>>>
>>> The question is: which functionality? A simple downloading tool or a
>>> full archival tool? Achieving similar functionality to org-board or
>>> monolith would a big task, since they aim to download an archival
>>> version of a webpage (including all resources). In addition, with
>>> archiving you also quickly run into the complexity of versioning based
>>> on time archived. There's also the challenge of mapping the downloaded
>>> files to metadata (specifically the original url). Org-board currently
>>> handles both of these very well.
>>
>> If you mean adding a timestamp and original URL, I don't think its our 
>> argument
>> point, because user totally can do it manually, or automatic this task in 
>> some
>> kind of way. WDYT?
>>
>>>
>>> I suppose there would be a few options depending on what the aims are:
>>>
>>> 1. At the simple end, include little more than than a quick and dirty
>>>way of downloading a single resource (html, pdf, jpeg) using url.el
>>>or wget (or optionally, monolith) and putting that in the attachment
>>>folder. Those who want full archiving of all resources could use
>>>other tools like org-board or org-web-tools.
>>
>> the [u] url downloading method already did the simple task.
>>
>>>
>>> 2. At the (much) more complex end, it would be to code out a robust
>>>archiving solution on top of url.el or wget.
>>
>> As I said, PATCH welcome, I admired many times I don't have ability to build 
>> a
>> complex archive functionality on url.el or wget or curl.
>>
>> Isn't Org Mode build up from small things? Can just become a big tool at 
>> once?
>>
>>>
>>> 3. Another, possibly simpler option... Add a command to the dispatcher
>>>that allows the user to invoke a custom function that is called with
>>>the attachment directory as the default-directory. This would enable
>>>more end-user flexibility, such as the ability to use
>>>wkhtmtoimage/wkhtmltopdf, monolith, phantom.js, archive.is, etc.
>>>
>>
>> This is leaved to `defcustom' option. I will improve the download function to
>> have enough elasticity.
>>
>> - -- 
>> [ stardiviner ]
>>I try to make every word tell the meaning that I want to express.
>>
>>Blog: https://stardiviner.g

Re: missing packages

[Colin Baxter]

Dear Kyle,

> Kyle Meyer  writes:

> Colin Baxter writes:
>> I notice a few packages seem to be absent from the org-mode dev
>> tree, e.g., org-bibtex.el and org-bbdb.el. Yet nothing I've seen
>> in ORG-NEWS suggests that they have been discontinued. I am
>> probably not understanding something rather elementary.

> In Org 9.3, link-related libraries were switched to an ol- prefix
> (ol-bibtex.el, ol-bbdb.el, ...).

Thank you very much for this information.

Best wishes,
Colin.


-- 
Colin Baxter
URL: http://www.Colin-Baxter.com

Re: Possible fix for :includes header argument in org-babel C source blocks

[Brandon Guttersohn]

Hey Kévin,

Apologies for the regression, and thank you for fixing it. I neglected to run 
the tests before suggesting that fix -- I'll try not to do that again..

I can at least confirm that the patch wasn't intended to change how C-header-files 
are specified in the org-babel-block-header. The goal was only to change how the 
headers are formatted in the generated C-language file during execution, and only for 
headers which were not wrapped in <>'s. Your fix looks right to me.

On 5/28/20 5:09 AM, Kévin Le Gouguec wrote:

Kévin Le Gouguec  writes:


That leads me to believe that the coercion was an unintended side-effect
of (format …).

Never mind, the ORG-NEWS entry for 9.1 shows an example of unquoted
header, so I guess it is intentional.

Here is a patch to fix the regression:


And here is a patch to add a test for the unquoted-single-header case,
since otherwise it's hard to tell whether this behaviour is intentional:


(Is the Org source for the C/C++/D Babel syntax[1] committed somewhere,
BTW?  I could not find it in the Org repo.)


I'd like to say that all tests pass now, but I'm getting failures on
master (even without my changes) for two other tests:


   FAILED  ob-tangle/jump-to-org
   FAILED  test-org-attach/dir


[1] https://orgmode.org/worg/org-contrib/babel/languages/ob-doc-C.html

Re: Possible fix for :includes header argument in org-babel C source blocks

[Kyle Meyer]

Kévin Le Gouguec writes:

> Here is a patch to fix the regression:

Thanks, applied (6506ea1).

> And here is a patch to add a test for the unquoted-single-header case,
> since otherwise it's hard to tell whether this behaviour is intentional:

Looks good to me.  Thanks for taking the time to add that.  Applied
(e9163591a).

> (Is the Org source for the C/C++/D Babel syntax[1] committed somewhere,
> BTW?  I could not find it in the Org repo.)
>  
> [1] https://orgmode.org/worg/org-contrib/babel/languages/ob-doc-C.html

The source for that page is in the worg repo:
https://code.orgmode.org/bzg/worg/src/master/org-contrib/babel/languages/ob-doc-C.org

> I'd like to say that all tests pass now, but I'm getting failures on
> master (even without my changes) for two other tests:
>
>>   FAILED  ob-tangle/jump-to-org
>>   FAILED  test-org-attach/dir

:(

After your first patch, all tests now pass on my end.  Would you mind
posting the details of those failures in a new thread?

Re: Possible fix for :includes header argument in org-babel C source blocks

[Kyle Meyer]

Kévin Le Gouguec writes:

> BTW, does the change from 44cb98fdb deserve an ORG-NEWS entry?

Hmm, my impression from the thread/patch was that it was a plain bug
fix, in which case I think the lack of an entry is fine.

Re: [Feature] add a new org-attach dispatcher command to offline save web page

[stardiviner]

I added monolith link to defcustom option docstring now. Update patch again.
  
From 6c667461b45e93059c6f801e485f7da4bfc3606c Mon Sep 17 00:00:00 2001
From: stardiviner 
Date: Fri, 29 May 2020 09:46:15 +0800
Subject: [PATCH] * org-attach.el: add a new command to archive web page

* lisp/org-attach.el (org-attach-url-archive): A new org-attach
dispatcher command to archive web page to a single file.

* lisp/org-agenda.el (org-attach-url-archive-command): A customize
option to specify external command for archiving web page.

* lisp/org-attach.el (org-attach-url-archive-page): A command
invoke external command to offline archive save web page.
---
 lisp/org-attach.el | 30 +-
 1 file changed, 29 insertions(+), 1 deletion(-)

diff --git a/lisp/org-attach.el b/lisp/org-attach.el
index 57d1360fc..80855d147 100644
--- a/lisp/org-attach.el
+++ b/lisp/org-attach.el
@@ -204,6 +204,8 @@ (defcustom org-attach-commands
  "Attach a file using symbolic-link method.")
 ((?u ?\C-u) org-attach-url
  "Attach a file from URL (downloading it).")
+((?U) org-attach-url-archive
+ "Attach an offline version of web page URL.")
 ((?b) org-attach-buffer
  "Select a buffer and attach its contents to the task.")
 ((?n ?\C-n) org-attach-new
@@ -467,6 +469,31 @@ (defun org-attach-url (url)
   (let ((org-attach-method 'url))
 (org-attach-attach url)))
 
+(defun org-attach-url-archive (url)
+  (interactive "MURL of the web page be archived to attach: \n")
+  (let ((org-attach-method 'archive))
+(org-attach-attach url)))
+
+(defcustom org-attach-url-archive-command
+  (file-name-nondirectory (executable-find "monolith"))
+  "The command used to offline archive web page.
+monolith can be found here: https://github.com/Y2Z/monolith.";
+  :type 'string
+  :safe #'stringp
+  :group 'org-attach)
+
+(defun org-attach-url-archive-page (url target-file)
+  "Save an offline archive of web page."
+  (pcase org-attach-url-archive-command
+("monolith"
+ (make-process
+  :name "org-attach-offline"
+  :command (list org-attach-url-archive-command url "-o" target-file)
+  :sentinel (lambda (proc event) (message "org-attach-offline finished!"))
+  :buffer "*org-attach-url-archive*"))
+(nil (warn "You must have a command availble for offline save web page!
+Set variable `org-attach-url-archive-command'."
+
 (defun org-attach-buffer (buffer-name)
   "Attach BUFFER-NAME's contents to current outline node.
 BUFFER-NAME is a string.  Signals a `file-already-exists' error
@@ -504,7 +531,8 @@ (defun org-attach-attach (file &optional visit-dir method)
((eq method 'cp) (copy-file file attach-file))
((eq method 'ln) (add-name-to-file file attach-file))
((eq method 'lns) (make-symbolic-link file attach-file))
-   ((eq method 'url) (url-copy-file file attach-file)))
+   ((eq method 'url) (url-copy-file file attach-file))
+   ((eq method 'archive) (org-attach-url-archive-page file attach-file)))
   (run-hook-with-args 'org-attach-after-change-hook attach-dir)
   (org-attach-tag)
   (cond ((eq org-attach-store-link-p 'attached)
-- 
2.26.2


stardiviner  writes:

> I attached the patch.
>
> I think this feature will be helpful for use who archive web page data usually
> like me. To be more portable, I also added an defcustom option for other
> external command. And use an if condition to detect whether external command
> available, else warning user.


-- 
[ stardiviner ]
   I try to make every word tell the meaning that I want to express.

   Blog: https://stardiviner.github.io/
   IRC(freenode): stardiviner, Matrix: stardiviner
   GPG: F09F650D7D674819892591401B5DF1C95AE89AC3


signature.asc
Description: PGP signature

[PATCH updated] Re: [Feature] add a new org-attach dispatcher command to offline save web page

[stardiviner]

I adopted some part of my patch, and make function name and docstring more clear
after this mail thread long discussion.
  
From 7e682ccd8d0d2a567de1bbbc0c8e02ee59e156bb Mon Sep 17 00:00:00 2001
From: stardiviner 
Date: Fri, 29 May 2020 09:46:15 +0800
Subject: [PATCH] * org-attach.el: add a new command to archive web page

* lisp/org-attach.el (org-attach-url-archive): A new org-attach
dispatcher command to archive web page to a single file.

* lisp/org-agenda.el (org-attach-url-archive-command): A customize
option to specify external command for archiving web page.

* lisp/org-attach.el (org-attach-url-archive-page): A command
invoke external command to offline archive save web page.
---
 lisp/org-attach.el | 29 -
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/lisp/org-attach.el b/lisp/org-attach.el
index 57d1360fc..14ac0f828 100644
--- a/lisp/org-attach.el
+++ b/lisp/org-attach.el
@@ -204,6 +204,8 @@ (defcustom org-attach-commands
  "Attach a file using symbolic-link method.")
 ((?u ?\C-u) org-attach-url
  "Attach a file from URL (downloading it).")
+((?U) org-attach-url-archive
+ "Attach an offline version of web page URL.")
 ((?b) org-attach-buffer
  "Select a buffer and attach its contents to the task.")
 ((?n ?\C-n) org-attach-new
@@ -467,6 +469,30 @@ (defun org-attach-url (url)
   (let ((org-attach-method 'url))
 (org-attach-attach url)))
 
+(defun org-attach-url-archive (url)
+  (interactive "MURL of the web page be archived to attach: \n")
+  (let ((org-attach-method 'archive))
+(org-attach-attach url)))
+
+(defcustom org-attach-url-archive-command
+  (file-name-nondirectory (executable-find "monolith"))
+  "The command used to offline archive web page."
+  :type 'string
+  :safe #'stringp
+  :group 'org-attach)
+
+(defun org-attach-url-archive-page (url target-file)
+  "Save an offline archive of web page."
+  (pcase org-attach-url-archive-command
+("monolith"
+ (make-process
+  :name "org-attach-offline"
+  :command (list org-attach-url-archive-command url "-o" target-file)
+  :sentinel (lambda (proc event) (message "org-attach-offline finished!"))
+  :buffer "*org-attach-url-archive*"))
+(nil (warn "You must have a command availble for offline save web page!
+Set variable `org-attach-url-archive-command'."
+
 (defun org-attach-buffer (buffer-name)
   "Attach BUFFER-NAME's contents to current outline node.
 BUFFER-NAME is a string.  Signals a `file-already-exists' error
@@ -504,7 +530,8 @@ (defun org-attach-attach (file &optional visit-dir method)
((eq method 'cp) (copy-file file attach-file))
((eq method 'ln) (add-name-to-file file attach-file))
((eq method 'lns) (make-symbolic-link file attach-file))
-   ((eq method 'url) (url-copy-file file attach-file)))
+   ((eq method 'url) (url-copy-file file attach-file))
+   ((eq method 'archive) (org-attach-url-archive-page file attach-file)))
   (run-hook-with-args 'org-attach-after-change-hook attach-dir)
   (org-attach-tag)
   (cond ((eq org-attach-store-link-p 'attached)
-- 
2.26.2


stardiviner  writes:

> I attached the patch.
>
> I think this feature will be helpful for use who archive web page data usually
> like me. To be more portable, I also added an defcustom option for other
> external command. And use an if condition to detect whether external command
> available, else warning user.


-- 
[ stardiviner ]
   I try to make every word tell the meaning that I want to express.

   Blog: https://stardiviner.github.io/
   IRC(freenode): stardiviner, Matrix: stardiviner
   GPG: F09F650D7D674819892591401B5DF1C95AE89AC3


signature.asc
Description: PGP signature

Re: [Feature] add a new org-attach dispatcher command to offline save web page

[Ihor Radchenko]

> As I said, PATCH welcome, I admired many times I don't have ability to build a
> complex archive functionality on url.el or wget or curl.

I have found the following solution [1] using wget:

wget --mirror -p --convert-links -P ./LOCAL-DIR WEBSITE-URL

This will not bundle the page into a single file, but it is better than
nothing. org-attach does not have to attach exactly one file.

[1] 
https://askubuntu.com/questions/411540/how-to-get-wget-to-download-exact-same-web-page-html-as-browser

Best,
Ihor

stardiviner  writes:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
>
> Matthew Lundin  writes:
>
>> Ihor Radchenko  writes:
>>
>>> It does not mean that attaching URL directly is not worth including
>>> into org. This sounds pretty common use case, especially considering
>>> the number of packages providing similar feature. You mentioned
>>> org-board, but there is also org-download and org-web-tools.
>>
>> My uneasiness has more to do with the specificity of the dependence on
>> monolith and the way that is hard-coded into the patch. When it comes to
>> patches, I think priority should go to those that are configurable,
>> accessible, and useful for everyone as opposed to those that have
>> hard-coded work-flows or highly-specific user configurations.
>
> I don't think it's hardcoded, As I said in my first email, It's an 
> `defcustom',
> isn't it? User totally can change that. And I used `executable-find', the 
> value
> will be `nil' when monolith is not available. And I don't have ability to
> archive same functionality with wget or curl. And as I said before, PATCH
> welcome.
>
>>
>>> I agree that monolith is completely uncommon tool and I would not expect
>>> the majority of users to have it installed, but the same functionality
>>> utilising built-in url.el (as a default) should be acceptable.
>>
>> The question is: which functionality? A simple downloading tool or a
>> full archival tool? Achieving similar functionality to org-board or
>> monolith would a big task, since they aim to download an archival
>> version of a webpage (including all resources). In addition, with
>> archiving you also quickly run into the complexity of versioning based
>> on time archived. There's also the challenge of mapping the downloaded
>> files to metadata (specifically the original url). Org-board currently
>> handles both of these very well.
>
> If you mean adding a timestamp and original URL, I don't think its our 
> argument
> point, because user totally can do it manually, or automatic this task in some
> kind of way. WDYT?
>
>>
>> I suppose there would be a few options depending on what the aims are:
>>
>> 1. At the simple end, include little more than than a quick and dirty
>>way of downloading a single resource (html, pdf, jpeg) using url.el
>>or wget (or optionally, monolith) and putting that in the attachment
>>folder. Those who want full archiving of all resources could use
>>other tools like org-board or org-web-tools.
>
> the [u] url downloading method already did the simple task.
>
>>
>> 2. At the (much) more complex end, it would be to code out a robust
>>archiving solution on top of url.el or wget.
>
> As I said, PATCH welcome, I admired many times I don't have ability to build a
> complex archive functionality on url.el or wget or curl.
>
> Isn't Org Mode build up from small things? Can just become a big tool at once?
>
>>
>> 3. Another, possibly simpler option... Add a command to the dispatcher
>>that allows the user to invoke a custom function that is called with
>>the attachment directory as the default-directory. This would enable
>>more end-user flexibility, such as the ability to use
>>wkhtmtoimage/wkhtmltopdf, monolith, phantom.js, archive.is, etc.
>>
>
> This is leaved to `defcustom' option. I will improve the download function to
> have enough elasticity.
>
> - -- 
> [ stardiviner ]
>I try to make every word tell the meaning that I want to express.
>
>Blog: https://stardiviner.github.io/
>IRC(freenode): stardiviner, Matrix: stardiviner
>GPG: F09F650D7D674819892591401B5DF1C95AE89AC3
>   
> -BEGIN PGP SIGNATURE-
>
> iQFIBAEBCAAyFiEE8J9lDX1nSBmJJZFAG13xyVromsMFAl7QbYMUHG51bWJjaGls
> ZEBnbWFpbC5jb20ACgkQG13xyVromsOFvAgAjTPtt5ZikNZnkp1gfuEGunVzblk6
> D6O+qgZAamT0EpR2nwK9U+x470KZAhXiLURkVb0l9/6uvA6MQZtj+umX/Qer/6bw
> vB4RICWenFKyyBMy9U1pZ9WzTNw2ZNfeaEiJ0XlgXx9ZPGjBgcLi/7FSM4NybV4O
> qbGwHbp4AzZ4mJOrMchNwlQfDYpg381xDewnHeuwtkVeBc34HpuFEGYKdmtOJ16S
> mnL/5LrbI7MahlIZuiqyAzeyU3DkSvFOC/ovvch9YlAk1SeK/BslUcaGrwl/Il70
> XG/28XaAvVW2OYrO9vOmwpdoguQqARhKeMuof6p85waf1OF8itd97P95Lw==
> =V/fu
> -END PGP SIGNATURE-

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong 
University, Xi'an, China
Email: yanta...@gmail.com, ihor_radche...@alumni.sutd.edu.sg

Re: [Feature] add a new org-attach dispatcher command to offline save web page

[stardiviner]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256


Matthew Lundin  writes:

> Ihor Radchenko  writes:
>
>> My view on this is bare-bones download, in a spirit of org-attach
>> itself. There is already 'url method in org-attach-attach, but it is
>> hard-coded to url-retrieve-synchronously. It would be handy if user
>> could configure alternative retrievers (like monolith, wget, curl, or
>> some user-defined function).
>
> Thanks. I'm learning about this for the first time after using
> org-attach for years! I have no idea why I've never noticed it.
>
>> Note that monolith does not crawl the website. It only collects
>> everything needed to show the page as you see it in browser into single
>> html file. This behaviour is what one expects to obtain when saving a
>> full web-page from browser.
>
> Agreed. This looks like the way to go. Keep it simple and abstract the
> call to url-copy-file to allow the user to configure the command for
> downloading the page/file. 

I tried to adopt my code to fit this design. But later, I realized this two
things are totally different functionality. url-retrieve-synchronously is used
to download URL file. Not for offline archiving.

>
> Best,
>
> Matt


- -- 
[ stardiviner ]
   I try to make every word tell the meaning that I want to express.

   Blog: https://stardiviner.github.io/
   IRC(freenode): stardiviner, Matrix: stardiviner
   GPG: F09F650D7D674819892591401B5DF1C95AE89AC3
  
-BEGIN PGP SIGNATURE-

iQFIBAEBCAAyFiEE8J9lDX1nSBmJJZFAG13xyVromsMFAl7QcEAUHG51bWJjaGls
ZEBnbWFpbC5jb20ACgkQG13xyVromsOJpgf/Yzib6/loAB+N3CMY7LBKxmH/xZ7d
u+isMgMlB2b9lVcC0BojwVG8Vo372Midjzu0+TJr2KuU/l9QV8xLqK90FOsLKzha
y2gbCghl2ehPvIfIKfI/LUPW3NOFIq01T4RYHgpEQ2GCWBdpNZ6Hki3fnvou2YzO
oIdtyablG5tZowvgRNQDNSg8XvaLTkkPqvXQUX1n7wba+buMADmns+rc3OTWBQqW
BHa8HC0X0n4TIcGy/pnet7+ooJtkHT05Ua9W0SrsaQthbaoqiDrfEyirL3Izm89D
rKYTo8dM7hKWLp7gQckPH6TYa8eiK0oQjp1riHa7myB2DjeWRqz2tKs2VA==
=JzUc
-END PGP SIGNATURE-

Re: org-babel :colnames yes

[ian martins]

Nick Dokos wrote:

> Would this work for you?
>
> #+BEGIN_SRC elisp :colnames yes
>   '((one two) hline (1 3) (1 6))
> #+END_SRC
>

Yes, that works. Thanks.

Re: [Feature] add a new org-attach dispatcher command to offline save web page

[stardiviner]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256


Thanks, Ihor, your explanation is helpful a lot!!!

Ihor Radchenko  writes:

>> My uneasiness has more to do with the specificity of the dependence on
>> monolith and the way that is hard-coded into the patch. When it comes to
>> patches, I think priority should go to those that are configurable,
>> accessible, and useful for everyone as opposed to those that have
>> hard-coded work-flows or highly-specific user configurations.
>
> Agree. Though I can see a use of having monolith as one of the options
> to help people discover what kind of tools they can use. I personally
> had a hard time finding command-line cli like monolith. Actually, it is
> the first time I heard about some offline tool handling js without a
> need to write python or ruby code. 
>
>> The question is: which functionality? A simple downloading tool or a
>> full archival tool? Achieving similar functionality to org-board or
>> monolith would a big task, since they aim to download an archival
>> version of a webpage (including all resources). 
>
> My view on this is bare-bones download, in a spirit of org-attach
> itself. There is already 'url method in org-attach-attach, but it is
> hard-coded to url-retrieve-synchronously. It would be handy if user
> could configure alternative retrievers (like monolith, wget, curl, or
> some user-defined function).
>
> Note that monolith does not crawl the website. It only collects
> everything needed to show the page as you see it in browser into single
> html file. This behaviour is what one expects to obtain when saving a
> full web-page from browser.
>
>> In addition, with
>> archiving you also quickly run into the complexity of versioning based
>> on time archived. 
>
> I guess that org-attach-git can be used for versioning, but I don't
> think that versioning is within scope of this patch. Monolith does not
> even support versioning.
>
>> There's also the challenge of mapping the downloaded
>> files to metadata (specifically the original url). Org-board currently
>> handles both of these very well.
>
> org-board is a great package, but it is not built-in. I do not think
> that all the org-board functionality needs to be included into
> org-attach. At least not within scope of this patch as I understand it.
>
> Best,
> Ihor
>
> Matthew Lundin  writes:
>
>> Ihor Radchenko  writes:
>>
>>> It does not mean that attaching URL directly is not worth including
>>> into org. This sounds pretty common use case, especially considering
>>> the number of packages providing similar feature. You mentioned
>>> org-board, but there is also org-download and org-web-tools.
>>
>> My uneasiness has more to do with the specificity of the dependence on
>> monolith and the way that is hard-coded into the patch. When it comes to
>> patches, I think priority should go to those that are configurable,
>> accessible, and useful for everyone as opposed to those that have
>> hard-coded work-flows or highly-specific user configurations.
>>
>>> I agree that monolith is completely uncommon tool and I would not expect
>>> the majority of users to have it installed, but the same functionality
>>> utilising built-in url.el (as a default) should be acceptable.
>>
>> The question is: which functionality? A simple downloading tool or a
>> full archival tool? Achieving similar functionality to org-board or
>> monolith would a big task, since they aim to download an archival
>> version of a webpage (including all resources). In addition, with
>> archiving you also quickly run into the complexity of versioning based
>> on time archived. There's also the challenge of mapping the downloaded
>> files to metadata (specifically the original url). Org-board currently
>> handles both of these very well.
>>
>> I suppose there would be a few options depending on what the aims are:
>>
>> 1. At the simple end, include little more than than a quick and dirty
>>way of downloading a single resource (html, pdf, jpeg) using url.el
>>or wget (or optionally, monolith) and putting that in the attachment
>>folder. Those who want full archiving of all resources could use
>>other tools like org-board or org-web-tools.
>>
>> 2. At the (much) more complex end, it would be to code out a robust
>>archiving solution on top of url.el or wget.
>>
>> 3. Another, possibly simpler option... Add a command to the dispatcher
>>that allows the user to invoke a custom function that is called with
>>the attachment directory as the default-directory. This would enable
>>more end-user flexibility, such as the ability to use
>>wkhtmtoimage/wkhtmltopdf, monolith, phantom.js, archive.is, etc.
>>
>> Best,
>>
>> Matt


- -- 
[ stardiviner ]
   I try to make every word tell the meaning that I want to express.

   Blog: https://stardiviner.github.io/
   IRC(freenode): stardiviner, Matrix: stardiviner
   GPG: F09F650D7D674819892591401B5DF1C95AE89AC3
  
-BEGIN PGP SIGNATURE

Re: [Feature] add a new org-attach dispatcher command to offline save web page

[stardiviner]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256


Matthew Lundin  writes:

> Ihor Radchenko  writes:
>
>> It does not mean that attaching URL directly is not worth including
>> into org. This sounds pretty common use case, especially considering
>> the number of packages providing similar feature. You mentioned
>> org-board, but there is also org-download and org-web-tools.
>
> My uneasiness has more to do with the specificity of the dependence on
> monolith and the way that is hard-coded into the patch. When it comes to
> patches, I think priority should go to those that are configurable,
> accessible, and useful for everyone as opposed to those that have
> hard-coded work-flows or highly-specific user configurations.

I don't think it's hardcoded, As I said in my first email, It's an `defcustom',
isn't it? User totally can change that. And I used `executable-find', the value
will be `nil' when monolith is not available. And I don't have ability to
archive same functionality with wget or curl. And as I said before, PATCH
welcome.

>
>> I agree that monolith is completely uncommon tool and I would not expect
>> the majority of users to have it installed, but the same functionality
>> utilising built-in url.el (as a default) should be acceptable.
>
> The question is: which functionality? A simple downloading tool or a
> full archival tool? Achieving similar functionality to org-board or
> monolith would a big task, since they aim to download an archival
> version of a webpage (including all resources). In addition, with
> archiving you also quickly run into the complexity of versioning based
> on time archived. There's also the challenge of mapping the downloaded
> files to metadata (specifically the original url). Org-board currently
> handles both of these very well.

If you mean adding a timestamp and original URL, I don't think its our argument
point, because user totally can do it manually, or automatic this task in some
kind of way. WDYT?

>
> I suppose there would be a few options depending on what the aims are:
>
> 1. At the simple end, include little more than than a quick and dirty
>way of downloading a single resource (html, pdf, jpeg) using url.el
>or wget (or optionally, monolith) and putting that in the attachment
>folder. Those who want full archiving of all resources could use
>other tools like org-board or org-web-tools.

the [u] url downloading method already did the simple task.

>
> 2. At the (much) more complex end, it would be to code out a robust
>archiving solution on top of url.el or wget.

As I said, PATCH welcome, I admired many times I don't have ability to build a
complex archive functionality on url.el or wget or curl.

Isn't Org Mode build up from small things? Can just become a big tool at once?

>
> 3. Another, possibly simpler option... Add a command to the dispatcher
>that allows the user to invoke a custom function that is called with
>the attachment directory as the default-directory. This would enable
>more end-user flexibility, such as the ability to use
>wkhtmtoimage/wkhtmltopdf, monolith, phantom.js, archive.is, etc.
>

This is leaved to `defcustom' option. I will improve the download function to
have enough elasticity.

- -- 
[ stardiviner ]
   I try to make every word tell the meaning that I want to express.

   Blog: https://stardiviner.github.io/
   IRC(freenode): stardiviner, Matrix: stardiviner
   GPG: F09F650D7D674819892591401B5DF1C95AE89AC3
  
-BEGIN PGP SIGNATURE-

iQFIBAEBCAAyFiEE8J9lDX1nSBmJJZFAG13xyVromsMFAl7QbYMUHG51bWJjaGls
ZEBnbWFpbC5jb20ACgkQG13xyVromsOFvAgAjTPtt5ZikNZnkp1gfuEGunVzblk6
D6O+qgZAamT0EpR2nwK9U+x470KZAhXiLURkVb0l9/6uvA6MQZtj+umX/Qer/6bw
vB4RICWenFKyyBMy9U1pZ9WzTNw2ZNfeaEiJ0XlgXx9ZPGjBgcLi/7FSM4NybV4O
qbGwHbp4AzZ4mJOrMchNwlQfDYpg381xDewnHeuwtkVeBc34HpuFEGYKdmtOJ16S
mnL/5LrbI7MahlIZuiqyAzeyU3DkSvFOC/ovvch9YlAk1SeK/BslUcaGrwl/Il70
XG/28XaAvVW2OYrO9vOmwpdoguQqARhKeMuof6p85waf1OF8itd97P95Lw==
=V/fu
-END PGP SIGNATURE-

Re: [Feature] add a new org-attach dispatcher command to offline save web page

[Samuel Wales]

i keep wondering if this is similar to the idea of making all external
links cached.  or specified ones.  using a specified cache dir.

for example, you could have various external links in your
html-exportable document, and run a command to cache them all.

Re: [Feature] add a new org-attach dispatcher command to offline save web page

[Matthew Lundin]

Ihor Radchenko  writes:

> My view on this is bare-bones download, in a spirit of org-attach
> itself. There is already 'url method in org-attach-attach, but it is
> hard-coded to url-retrieve-synchronously. It would be handy if user
> could configure alternative retrievers (like monolith, wget, curl, or
> some user-defined function).

Thanks. I'm learning about this for the first time after using
org-attach for years! I have no idea why I've never noticed it.

> Note that monolith does not crawl the website. It only collects
> everything needed to show the page as you see it in browser into single
> html file. This behaviour is what one expects to obtain when saving a
> full web-page from browser.

Agreed. This looks like the way to go. Keep it simple and abstract the
call to url-copy-file to allow the user to configure the command for
downloading the page/file. 

Best,

Matt

Re: missing packages

[Kyle Meyer]

Colin Baxter writes:

> I notice a few packages seem to be absent from the org-mode dev tree, 
> e.g., org-bibtex.el and org-bbdb.el. Yet nothing I've seen in ORG-NEWS
> suggests that they have been discontinued. I am probably not
> understanding something rather elementary.

In Org 9.3, link-related libraries were switched to an ol- prefix
(ol-bibtex.el, ol-bbdb.el, ...).

Re: org table: one column of random numbers (but natural ones)

[Stig Brautaset]

>> In row 67 you would have a random integer in the range [0..67)
>> f0 format removes any fractional part leaving only an integer number
>
> Aha thanks, a minor thing, which I thank, cannot be really done:
>
> Is it possible to avoid number repetition?
>
> So I want a random sequence of the column 67 but I don't want numbers to
> be repeated.

To avoid duplicates you could generate a sequence from [0..67), shuffle
it[1], then use the row number as an index into that list. (Or pop off the
front.) How to do that from an org table function I have no idea,
however.

[1]: Emacs lisp "Knuth shuffle" implementation: 
https://stackoverflow.com/a/49505968/5950

Regards,
Stig

26.3; org-indent-mode's line-prefix text property flickers near overlays

[Kévin Le Gouguec]

Hello,

The line-prefix text property set by org-indent-mode sometimes vanishes
when typing near a line where an overlay is applied.  To reproduce:

- emacs -Q
- C-x C-f repro.org
- M-x org-indent-mode
- M-: (insert "* heading\ntext")
- M-:
(let ((ov (make-overlay (point-at-bol) (point-at-bol)))
  (val (propertize " " 'display '((left-fringe right-triangle)
  (overlay-put ov 'before-string val))
- RET
- a

When "a" is inserted, org-indent-mode's line-prefix disappears on the
*previous* line ("text").  It remains gone as long as I type
self-inserting characters, until

- I type certain commands, e.g. RET or C-j, or

- I insert a closing delimiter that makes
  blink-paren-post-self-insert-function blink the corresponding opening
  delimiter, e.g. ')' or ']'.

Then the line-prefix shows up again.


This recipe is simplified; I originally found this bug in Org files
under version control with diff-hl-flydiff-mode enabled.  When typing in
new content, the preceding line is shoved to the left until I stop
typing; after diff-hl-flydiff-delay, diff-hl's idle timer kicks in and
updates its overlay, which as a side-effect makes org-indent-mode's
line-prefix come back up.

See [1] for my original bug report to diff-hl and some crude
"analysis".


I'm clearly out of my depth; at least I hope I used the correct terms to
describe the symptoms I've observed.  Let me know if there is anything I
can do to help debug this.

Thank you for your time.


[1] https://github.com/dgutov/diff-hl/issues/142


In GNU Emacs 28.0.50 (build 21, x86_64-pc-linux-gnu, GTK+ Version 3.24.20, 
cairo version 1.16.0)
 of 2020-05-18 built on my-little-tumbleweed
Repository revision: b1fe27d77db8f819641231ca46725f3eed0b4d9b
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12008000
System Description: openSUSE Tumbleweed

Configured using:
 'configure --with-xwidgets --with-cairo'

Configured features:
XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND GPM DBUS GSETTINGS GLIB NOTIFY
INOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF
ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS XWIDGETS JSON
PDUMPER LCMS2 GMP

Important settings:
  value of $LC_CTYPE: en_US.UTF-8
  value of $LC_TIME: en_GB.UTF-8
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=local
  locale-coding-system: utf-8-unix

Re: org table: one column of random numbers (but natural ones)

[Uwe Brauer]

   > Le 27/05/2020 à 22:40, Uwe Brauer a écrit :
   > What about:

   > #+TBLFM: $4=random($1);f0

   > In row 67 you would have a random integer in the range [0..67)
   > f0 format removes any fractional part leaving only an integer number

Aha thanks, a minor thing, which I thank, cannot be really done:

Is it possible to avoid number repetition?
So I want a random sequence of the column 67 but I don't want numbers to
be repeated.

Regards

Uwe 




smime.p7s
Description: S/MIME cryptographic signature

Re: [Feature] add a new org-attach dispatcher command to offline save web page

[Ihor Radchenko]

> I don't know how to implement similar functionality like monolith with
> url-retrieve-synchronously + save-buffer. PATCH welcome.

Sorry, I missed that the default 'url attach method exists already.

> Of course curl and wget is alternative options, just still same problem. Don't
> know how to download web page as a single file and save it as archive file. If
> use other ways, it's just another org-board. Then this patch will be 
> unnecessary.

Well. org-board is not built-in. And curl or wget are much more likely
to be installed. They can serve as a good fallback alternative to
monolith.

Best,
Ihor

stardiviner  writes:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
>
> Ihor Radchenko  writes:
>
>>> If have better solution, I totally agree with that.
>>
>> In the current state, your patch will be pretty much useless for users
>> without technical background.
>>
>> url-retrieve-synchronously + save-buffer should be a better default. It
>> will be available for all users.
>>
>
> I don't know how to implement similar functionality like monolith with
> url-retrieve-synchronously + save-buffer. PATCH welcome.
>
>> Ideally, there should be several customisation options. Something like:
>>  - url-retrieve-synchronously
>>  - curl
>>  - wget
>>  - monolith
>
> Of course curl and wget is alternative options, just still same problem. Don't
> know how to download web page as a single file and save it as archive file. If
> use other ways, it's just another org-board. Then this patch will be 
> unnecessary.
>
>>
>> Also, it would be great if you provide some description about each
>> customisation in the docstring. For example, a link to github page of
>> monolith and a brief description how it is different from other url
>> retrieval methods should be useful for general user.
>
> This is good solution, I will improve option description.
>
>>
>> Best,
>> Ihor
>>
>>
>> stardiviner  writes:
>>
>>> -BEGIN PGP SIGNED MESSAGE-
>>> Hash: SHA256
>>>
>>>
>>> Ihor Radchenko  writes:
>>>
> This looks helpful, but I think this should be an add-on rather than
> integrated into the org-attach. I'm glad to learn about monolith, but
> far fewer users are likely to have it installed than have wget or curl
> installed. You might also want to check out org-board, which has similar
> functionality:

 It does not mean that attaching URL directly is not worth including into
 org. This sounds pretty common use case, especially considering the number
 of packages providing similar feature. You mentioned org-board, but
 there is also org-download and org-web-tools.
>>>
>>> That's right, this phenomenon represents an obviously situation, Org Mode 
>>> should
>>> have this functionality.
>>>

 I agree that monolith is completely uncommon tool and I would not expect
 the majority of users to have it installed, but the same functionality
 utilising built-in url.el (as a default) should be acceptable.
>>>
>>> If have better solution, I totally agree with that.
>>>

 Best,
 Ihor


 Matthew Lundin  writes:

> stardiviner  writes:
>
>> I attached the patch.
>>
>> I think this feature will be helpful for use who archive web page data 
>> usually
>> like me. To be more portable, I also added an defcustom option for other
>> external command. And use an if condition to detect whether external 
>> command
>> available, else warning user.
>
> This looks helpful, but I think this should be an add-on rather than
> integrated into the org-attach. I'm glad to learn about monolith, but
> far fewer users are likely to have it installed than have wget or curl
> installed. You might also want to check out org-board, which has similar
> functionality:
>
> https://github.com/scallywag/org-board
>
> Best,
>
> Matt
>
>>>
>>>
>>> - -- 
>>> [ stardiviner ]
>>>I try to make every word tell the meaning that I want to express.
>>>
>>>Blog: https://stardiviner.github.io/
>>>IRC(freenode): stardiviner, Matrix: stardiviner
>>>GPG: F09F650D7D674819892591401B5DF1C95AE89AC3
>>>   
>>> -BEGIN PGP SIGNATURE-
>>>
>>> iQFIBAEBCAAyFiEE8J9lDX1nSBmJJZFAG13xyVromsMFAl7PXKoUHG51bWJjaGls
>>> ZEBnbWFpbC5jb20ACgkQG13xyVromsOiMQf9EY7ES42hWxdjgFqiN0UqPGssRd+l
>>> fcvhb+yiT5/71TYA8YbQOF0Pa2SpUBgGUw+hcHUd9aDXxAP+4ucmP4bZ/vx29UTN
>>> T7YpOvuc5ZkafiN5bWghij1dLYhfi2CMDGGduU9rvjSleaRMhoX12wo8OM2qBHDm
>>> 2bqWHrZsGVPsNlAK3ymN3PdmwOtxktRVUxWZOug3ZF6LYnEHWRp/rb6lvS2BF3Ej
>>> qNazerYVcpfDsOiCWTSrjos2gnITT2pSlnxmrQFO69Pvex+T2dEJL7raNyuXi3Qd
>>> jn5uayrnHn4ZeUwSfmAQiqvq2TsFZ5zvtH0nKM+4tp/6gSgEuTw9gYaoBQ==
>>> =qrFy
>>> -END PGP SIGNATURE-
>>>
>
>
> - -- 
> [ stardiviner ]
>I try to make every word tell the meaning that I want to express.
>
>Blog: https://stardiviner.github.io/
>IRC(freenode): stardiviner, Matrix: stardiviner
>G

Re: [Feature] add a new org-attach dispatcher command to offline save web page

[Ihor Radchenko]

> My uneasiness has more to do with the specificity of the dependence on
> monolith and the way that is hard-coded into the patch. When it comes to
> patches, I think priority should go to those that are configurable,
> accessible, and useful for everyone as opposed to those that have
> hard-coded work-flows or highly-specific user configurations.

Agree. Though I can see a use of having monolith as one of the options
to help people discover what kind of tools they can use. I personally
had a hard time finding command-line cli like monolith. Actually, it is
the first time I heard about some offline tool handling js without a
need to write python or ruby code. 

> The question is: which functionality? A simple downloading tool or a
> full archival tool? Achieving similar functionality to org-board or
> monolith would a big task, since they aim to download an archival
> version of a webpage (including all resources). 

My view on this is bare-bones download, in a spirit of org-attach
itself. There is already 'url method in org-attach-attach, but it is
hard-coded to url-retrieve-synchronously. It would be handy if user
could configure alternative retrievers (like monolith, wget, curl, or
some user-defined function).

Note that monolith does not crawl the website. It only collects
everything needed to show the page as you see it in browser into single
html file. This behaviour is what one expects to obtain when saving a
full web-page from browser.

> In addition, with
> archiving you also quickly run into the complexity of versioning based
> on time archived. 

I guess that org-attach-git can be used for versioning, but I don't
think that versioning is within scope of this patch. Monolith does not
even support versioning.

> There's also the challenge of mapping the downloaded
> files to metadata (specifically the original url). Org-board currently
> handles both of these very well.

org-board is a great package, but it is not built-in. I do not think
that all the org-board functionality needs to be included into
org-attach. At least not within scope of this patch as I understand it.

Best,
Ihor

Matthew Lundin  writes:

> Ihor Radchenko  writes:
>
>> It does not mean that attaching URL directly is not worth including
>> into org. This sounds pretty common use case, especially considering
>> the number of packages providing similar feature. You mentioned
>> org-board, but there is also org-download and org-web-tools.
>
> My uneasiness has more to do with the specificity of the dependence on
> monolith and the way that is hard-coded into the patch. When it comes to
> patches, I think priority should go to those that are configurable,
> accessible, and useful for everyone as opposed to those that have
> hard-coded work-flows or highly-specific user configurations.
>
>> I agree that monolith is completely uncommon tool and I would not expect
>> the majority of users to have it installed, but the same functionality
>> utilising built-in url.el (as a default) should be acceptable.
>
> The question is: which functionality? A simple downloading tool or a
> full archival tool? Achieving similar functionality to org-board or
> monolith would a big task, since they aim to download an archival
> version of a webpage (including all resources). In addition, with
> archiving you also quickly run into the complexity of versioning based
> on time archived. There's also the challenge of mapping the downloaded
> files to metadata (specifically the original url). Org-board currently
> handles both of these very well.
>
> I suppose there would be a few options depending on what the aims are:
>
> 1. At the simple end, include little more than than a quick and dirty
>way of downloading a single resource (html, pdf, jpeg) using url.el
>or wget (or optionally, monolith) and putting that in the attachment
>folder. Those who want full archiving of all resources could use
>other tools like org-board or org-web-tools.
>
> 2. At the (much) more complex end, it would be to code out a robust
>archiving solution on top of url.el or wget.
>
> 3. Another, possibly simpler option... Add a command to the dispatcher
>that allows the user to invoke a custom function that is called with
>the attachment directory as the default-directory. This would enable
>more end-user flexibility, such as the ability to use
>wkhtmtoimage/wkhtmltopdf, monolith, phantom.js, archive.is, etc.
>
> Best,
>
> Matt

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong 
University, Xi'an, China
Email: yanta...@gmail.com, ihor_radche...@alumni.sutd.edu.sg

Re: [Feature] add a new org-attach dispatcher command to offline save web page

[Matthew Lundin]

Ihor Radchenko  writes:

> It does not mean that attaching URL directly is not worth including
> into org. This sounds pretty common use case, especially considering
> the number of packages providing similar feature. You mentioned
> org-board, but there is also org-download and org-web-tools.

My uneasiness has more to do with the specificity of the dependence on
monolith and the way that is hard-coded into the patch. When it comes to
patches, I think priority should go to those that are configurable,
accessible, and useful for everyone as opposed to those that have
hard-coded work-flows or highly-specific user configurations.

> I agree that monolith is completely uncommon tool and I would not expect
> the majority of users to have it installed, but the same functionality
> utilising built-in url.el (as a default) should be acceptable.

The question is: which functionality? A simple downloading tool or a
full archival tool? Achieving similar functionality to org-board or
monolith would a big task, since they aim to download an archival
version of a webpage (including all resources). In addition, with
archiving you also quickly run into the complexity of versioning based
on time archived. There's also the challenge of mapping the downloaded
files to metadata (specifically the original url). Org-board currently
handles both of these very well.

I suppose there would be a few options depending on what the aims are:

1. At the simple end, include little more than than a quick and dirty
   way of downloading a single resource (html, pdf, jpeg) using url.el
   or wget (or optionally, monolith) and putting that in the attachment
   folder. Those who want full archiving of all resources could use
   other tools like org-board or org-web-tools.

2. At the (much) more complex end, it would be to code out a robust
   archiving solution on top of url.el or wget.

3. Another, possibly simpler option... Add a command to the dispatcher
   that allows the user to invoke a custom function that is called with
   the attachment directory as the default-directory. This would enable
   more end-user flexibility, such as the ability to use
   wkhtmtoimage/wkhtmltopdf, monolith, phantom.js, archive.is, etc.

Best,

Matt

Re: [Feature] add a new org-attach dispatcher command to offline save web page

[stardiviner]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256


Ihor Radchenko  writes:

>> If have better solution, I totally agree with that.
>
> In the current state, your patch will be pretty much useless for users
> without technical background.
>
> url-retrieve-synchronously + save-buffer should be a better default. It
> will be available for all users.
>

I don't know how to implement similar functionality like monolith with
url-retrieve-synchronously + save-buffer. PATCH welcome.

> Ideally, there should be several customisation options. Something like:
>  - url-retrieve-synchronously
>  - curl
>  - wget
>  - monolith

Of course curl and wget is alternative options, just still same problem. Don't
know how to download web page as a single file and save it as archive file. If
use other ways, it's just another org-board. Then this patch will be 
unnecessary.

>
> Also, it would be great if you provide some description about each
> customisation in the docstring. For example, a link to github page of
> monolith and a brief description how it is different from other url
> retrieval methods should be useful for general user.

This is good solution, I will improve option description.

>
> Best,
> Ihor
>
>
> stardiviner  writes:
>
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA256
>>
>>
>> Ihor Radchenko  writes:
>>
 This looks helpful, but I think this should be an add-on rather than
 integrated into the org-attach. I'm glad to learn about monolith, but
 far fewer users are likely to have it installed than have wget or curl
 installed. You might also want to check out org-board, which has similar
 functionality:
>>>
>>> It does not mean that attaching URL directly is not worth including into
>>> org. This sounds pretty common use case, especially considering the number
>>> of packages providing similar feature. You mentioned org-board, but
>>> there is also org-download and org-web-tools.
>>
>> That's right, this phenomenon represents an obviously situation, Org Mode 
>> should
>> have this functionality.
>>
>>>
>>> I agree that monolith is completely uncommon tool and I would not expect
>>> the majority of users to have it installed, but the same functionality
>>> utilising built-in url.el (as a default) should be acceptable.
>>
>> If have better solution, I totally agree with that.
>>
>>>
>>> Best,
>>> Ihor
>>>
>>>
>>> Matthew Lundin  writes:
>>>
 stardiviner  writes:

> I attached the patch.
>
> I think this feature will be helpful for use who archive web page data 
> usually
> like me. To be more portable, I also added an defcustom option for other
> external command. And use an if condition to detect whether external 
> command
> available, else warning user.

 This looks helpful, but I think this should be an add-on rather than
 integrated into the org-attach. I'm glad to learn about monolith, but
 far fewer users are likely to have it installed than have wget or curl
 installed. You might also want to check out org-board, which has similar
 functionality:

 https://github.com/scallywag/org-board

 Best,

 Matt

>>
>>
>> - -- 
>> [ stardiviner ]
>>I try to make every word tell the meaning that I want to express.
>>
>>Blog: https://stardiviner.github.io/
>>IRC(freenode): stardiviner, Matrix: stardiviner
>>GPG: F09F650D7D674819892591401B5DF1C95AE89AC3
>>   
>> -BEGIN PGP SIGNATURE-
>>
>> iQFIBAEBCAAyFiEE8J9lDX1nSBmJJZFAG13xyVromsMFAl7PXKoUHG51bWJjaGls
>> ZEBnbWFpbC5jb20ACgkQG13xyVromsOiMQf9EY7ES42hWxdjgFqiN0UqPGssRd+l
>> fcvhb+yiT5/71TYA8YbQOF0Pa2SpUBgGUw+hcHUd9aDXxAP+4ucmP4bZ/vx29UTN
>> T7YpOvuc5ZkafiN5bWghij1dLYhfi2CMDGGduU9rvjSleaRMhoX12wo8OM2qBHDm
>> 2bqWHrZsGVPsNlAK3ymN3PdmwOtxktRVUxWZOug3ZF6LYnEHWRp/rb6lvS2BF3Ej
>> qNazerYVcpfDsOiCWTSrjos2gnITT2pSlnxmrQFO69Pvex+T2dEJL7raNyuXi3Qd
>> jn5uayrnHn4ZeUwSfmAQiqvq2TsFZ5zvtH0nKM+4tp/6gSgEuTw9gYaoBQ==
>> =qrFy
>> -END PGP SIGNATURE-
>>


- -- 
[ stardiviner ]
   I try to make every word tell the meaning that I want to express.

   Blog: https://stardiviner.github.io/
   IRC(freenode): stardiviner, Matrix: stardiviner
   GPG: F09F650D7D674819892591401B5DF1C95AE89AC3
  
-BEGIN PGP SIGNATURE-

iQFIBAEBCAAyFiEE8J9lDX1nSBmJJZFAG13xyVromsMFAl7P4CwUHG51bWJjaGls
ZEBnbWFpbC5jb20ACgkQG13xyVromsMnIgf+MR3SvdlaxKofkSPZjYknvJ/MwUy4
iJOFYe0VRUXcNyjkQVGToNVVM50rWodR29LSXy8F1WbvgjDZb0arktsjOpjy+0Oq
R/7JiE7Y8lXYPlmn+Gcqoijcuoo0tNNaE8ia6U9uYbhp8I1Whd0Jhx5kWiXCxlWe
AMf2Yz4mw2CJc67W1MVUPM/2nq8+GhduLmY7UPw56A6ph+H2HD/AO+FzgL0dlivP
lV3EKCc7j5kA5uzXqbqfWJZdylH4cUbzdHHePsW+EpWodF/tcJcGJmlIm/PUNuEX
MsIkkKQt7QFLvr47vROVrJczEjT70ZdjWqjTZf5/esEsRUhb+m8uDbgc5g==
=c9Pj
-END PGP SIGNATURE-

[PATCH] ob-sql: Respect database param when using dbconnection

[Daniel Kraus]

Hi,

I use ob-sql with the :dbconnection param so I don't have my username and 
password in my org file.
But often I don't want to use the default database from the dbconnection alist 
but
rather specify it explicitly with :database.
Attached is a patch that fixes this.

Thanks,
  Daniel

>From a8dccff104d7426e2f353b1005e0bdcc51de6e99 Mon Sep 17 00:00:00 2001
From: Daniel Kraus 
Date: Tue, 26 May 2020 16:07:34 +0200
Subject: [PATCH] ob-sql: Respect database param when using dbconnection

---
 lisp/ob-sql.el | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lisp/ob-sql.el b/lisp/ob-sql.el
index 7c359b988..d7a8bf0a0 100644
--- a/lisp/ob-sql.el
+++ b/lisp/ob-sql.el
@@ -191,7 +191,8 @@ then look for the parameter into the corresponding connection
 defined in `sql-connection-alist`, otherwise look into PARAMS.
 Look `sql-connection-alist` (part of SQL mode) for how to define
 database connections."
-  (if (assq :dbconnection params)
+  (if (and (assq :dbconnection params)
+	   (not (and (assq :database params) (eq name :database
   (let* ((dbconnection (cdr (assq :dbconnection params)))
  (name-mapping '((:dbhost . sql-server)
  (:dbport . sql-port)
-- 
2.26.2

Re: Bug: ob-python mangles multiline :var values [9.3.6 (release_9.3.6-397-ga089600)]

[Jack Kamm]

An update on this -- I decided to revert my fix for multiline Python
variables. I left the unit test I added for this, but marked it as
expected to fail.

I'll try to submit a proper fix for this soon. However I've been swamped
at work so I can't promise when (hopefully a couple weeks). In the
meantime, I think it better to leave the original bug in place, rather
than break any existing ob-python use cases.

When I do get to this, I'll submit it as a patch to the list first, to
make sure I don't accidentally cause a new bug.

Best,
Jack

Jack Kamm  writes:

> Hi Matt --
>
>> A heads up... I believe this changes the scope of the :var variables,
>> since they previously were local to the main() function and now they are
>> declared globally. After this change, some of my existing python source
>> blocks (i.e., ones in which I attempt to assign a new value to a
>> variable defined by :var) now generate the following error:
>>
>> Traceback (most recent call last):
>>   File "", line 223, in 
>>   File "", line 214, in main
>> UnboundLocalError: local variable 'members' referenced before assignment
>
> Thanks for noticing this and pointing it out. This was an oversight on
> my end (I don't really use ":var" or non-session blocks).
>
> Unfortunately, the fix for the original bug will have to be a bit more
> complicated to avoid this error. I'll put it on my todo list, but if
> anyone wants to have a crack at it, that would be very welcome.
>
> We should also add a unit test for this regression so it doesn't happen
> again.
>
>> I hesitate to call this a bug, since it would be fine to think of
>> everything within the source block as local and the header :var
>> declarations as global.
>
> I think it's fair to call this a bug. I think it would be inconvenient
> to be unable to assign to these variables. Also, I did not intend to
> break any existing code with this.
>
>> And I suppose it would be worthwhile to ask: Is this change consistent
>> with other org-babel modules? Is there a canonical way that org-babel
>> handles scope?
>
> IMO everything ought to be in the same scope, and the user shouldn't
> have to think about the scope of the main() function that ob-python
> wraps the body in. Especially since non-session blocks are evaluated
> independently of other blocks, we really shouldn't have to think about
> their scope.
>
> I'm not sure if the Org manual provides any guidance here, but from a
> quick check I didn't see anything about it.
>
> Ideally I think everything should just be in global scope; however
> ob-python needs to use a wrapper function for the "return" keyword.

Re: [BUG] All drawers except property drawers are open at startup

[Matthew Lundin]

Nicolas Goaziou  writes:

> Matt Lundin  writes:
>
>> All drawers default to open at startup except for property drawers. This
>> includes :LOGBOOK: drawers, which can be quite long and block view of an
>> entry.
>
> I think all drawers are opened at startup, including property drawers.
>
>> I ran a git bisect and discovered that the bug was introduced with
>> commit 8b05c06d427e850d45684f69c5165cd7684e1071 on May 9. 
>
> Isn't it related to 88f5ed91c5cf9cb6962c1b441eac7dbab9a4734e instead?
>

No. I have the following in my emacs file:

(setq org-startup-folded t)

Here's a minimal recipe for reproducing:

Config file:

(add-to-list 'load-path "~/org-mode/lisp/")
(add-to-list 'load-path "~/org-mode/contrib/lisp/")

(setq org-startup-folded t)

Org file:

* Heading
   :PROPERTIES:
   :APROP: 1234
   :END:
   :LOGBOOK:
   - State "DONE"   from "NOW"[2020-05-27 Wed 23:03]
   :END:
   :RANDOM:
   Some text
   :END:

Navigate to file. Navigate to outline heading and hit TAB to cycle/open.

Before commit 8b05c06d427e850d45684f69c5165cd7684e1071 all drawers
remain folded (expected behavior):

* Heading
   :PROPERTIES:
   :LOGBOOK:

   :RANDOM:

Beginning with commit 8b05c06d427e850d45684f69c5165cd7684e1071 only the
PROPERTIES drawer remains folded, so the entry looks like this
(undesired behavior):

* Heading
   :PROPERTIES:
   :LOGBOOK:
   - State "DONE"   from "NOW"[2020-05-27 Wed 23:03]
   :END:

   :RANDOM:
   Some text
   :END:

Re: [Feature] add a new org-attach dispatcher command to offline save web page

[Ihor Radchenko]

> If have better solution, I totally agree with that.

In the current state, your patch will be pretty much useless for users
without technical background.

url-retrieve-synchronously + save-buffer should be a better default. It
will be available for all users.

Ideally, there should be several customisation options. Something like:
 - url-retrieve-synchronously
 - curl
 - wget
 - monolith

Also, it would be great if you provide some description about each
customisation in the docstring. For example, a link to github page of
monolith and a brief description how it is different from other url
retrieval methods should be useful for general user.

Best,
Ihor


stardiviner  writes:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
>
> Ihor Radchenko  writes:
>
>>> This looks helpful, but I think this should be an add-on rather than
>>> integrated into the org-attach. I'm glad to learn about monolith, but
>>> far fewer users are likely to have it installed than have wget or curl
>>> installed. You might also want to check out org-board, which has similar
>>> functionality:
>>
>> It does not mean that attaching URL directly is not worth including into
>> org. This sounds pretty common use case, especially considering the number
>> of packages providing similar feature. You mentioned org-board, but
>> there is also org-download and org-web-tools.
>
> That's right, this phenomenon represents an obviously situation, Org Mode 
> should
> have this functionality.
>
>>
>> I agree that monolith is completely uncommon tool and I would not expect
>> the majority of users to have it installed, but the same functionality
>> utilising built-in url.el (as a default) should be acceptable.
>
> If have better solution, I totally agree with that.
>
>>
>> Best,
>> Ihor
>>
>>
>> Matthew Lundin  writes:
>>
>>> stardiviner  writes:
>>>
 I attached the patch.

 I think this feature will be helpful for use who archive web page data 
 usually
 like me. To be more portable, I also added an defcustom option for other
 external command. And use an if condition to detect whether external 
 command
 available, else warning user.
>>>
>>> This looks helpful, but I think this should be an add-on rather than
>>> integrated into the org-attach. I'm glad to learn about monolith, but
>>> far fewer users are likely to have it installed than have wget or curl
>>> installed. You might also want to check out org-board, which has similar
>>> functionality:
>>>
>>> https://github.com/scallywag/org-board
>>>
>>> Best,
>>>
>>> Matt
>>>
>
>
> - -- 
> [ stardiviner ]
>I try to make every word tell the meaning that I want to express.
>
>Blog: https://stardiviner.github.io/
>IRC(freenode): stardiviner, Matrix: stardiviner
>GPG: F09F650D7D674819892591401B5DF1C95AE89AC3
>   
> -BEGIN PGP SIGNATURE-
>
> iQFIBAEBCAAyFiEE8J9lDX1nSBmJJZFAG13xyVromsMFAl7PXKoUHG51bWJjaGls
> ZEBnbWFpbC5jb20ACgkQG13xyVromsOiMQf9EY7ES42hWxdjgFqiN0UqPGssRd+l
> fcvhb+yiT5/71TYA8YbQOF0Pa2SpUBgGUw+hcHUd9aDXxAP+4ucmP4bZ/vx29UTN
> T7YpOvuc5ZkafiN5bWghij1dLYhfi2CMDGGduU9rvjSleaRMhoX12wo8OM2qBHDm
> 2bqWHrZsGVPsNlAK3ymN3PdmwOtxktRVUxWZOug3ZF6LYnEHWRp/rb6lvS2BF3Ej
> qNazerYVcpfDsOiCWTSrjos2gnITT2pSlnxmrQFO69Pvex+T2dEJL7raNyuXi3Qd
> jn5uayrnHn4ZeUwSfmAQiqvq2TsFZ5zvtH0nKM+4tp/6gSgEuTw9gYaoBQ==
> =qrFy
> -END PGP SIGNATURE-
>

-- 
Ihor Radchenko,
PhD,
Center for Advancing Materials Performance from the Nanoscale (CAMP-nano)
State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong 
University, Xi'an, China
Email: yanta...@gmail.com, ihor_radche...@alumni.sutd.edu.sg

Re: Help speeding up Org iCal export

[Ken Mankoff]

Hi Nicolas,

Thank you for providing suggestions about the appropriate tool. I've
profiled it, examined the variables you suggested, but am not sure what
next steps to take. I hope it is OK that I post questions about this on
this list...

On Thu, May 28, 2020 at 5:17 AM Nicolas Goaziou 
wrote:

> Ken Mankoff  writes:
>
> > When I eval `(org-icalendar-combine-agenda-files nil)`, it takes about
> > 15 seconds.
>
> OK. You may want to profile this, or use ELP.
>

Output from the profiler is here: https://pastebin.com/QvL9w69R but the key
lines seems to be:

  - org-icalendar-combine-agenda-files  15540  83%
   - apply  15540  83%
- org-icalendar--combine-files  15540  83%
 - mapconcat15536  83%
  - #   15536  83%
   - org-export-as  15162  81%
- org-export-data6493  34%
 - mapconcat 6493  34%
  - #6493  34%
   - org-export-data 6493  34%
- mapconcat  6164  33%
 - # 6160  33%
  - org-export-data  6156  33%
   - mapconcat   5674  30%

with org-export-data and mapconcat repeating, AND

- mapc   5672  30%
 - # 5672  30%
  - mapc 5672  30%
   - #   5668  30%
- #   4392  23%
 - org-id-find   4392  23%
  - org-id-update-id-locations
4376  23%
   - org-map-entries 3696  19%
- org-scan-tags  3668  19%
 + #990
  5%
 + #879
  4%
 + #830
  4%



> Some strange messages in the *Messages* buffer include:

> >
> > 22 files scanned, 7 files contains IDs and in total 23 IDs found.
>
> You may have set `org-icalendar-store-UID' somehow.
>

It is set to nil. I have only the *scratch* buffer open, and no other
buffers (although some where open earlier in this emacs session).


> > But my `org-agenda-files` variable only contains 12 files and no
> > #+INCLUDE statements.
>
> ID scanning is not limited to agenda files. Org also scans files known
> to contain ID. E.g., if, during the session, you insert an ID in
> "foo.org", later scans (in the same session) will include "foo.org".
>
> See `org-id-files' (internal variable), and `org-id-extra-files', which
> may contain even more files.
>

`org-id-files` contained a subset of my agenda files plus a few extra
(total: 7 files, and these 7 plus my agenda files do not sum to 22). When I
set it to nil, nothing significant changed, but it was reset after running
`org-icalendar-combine-agenda-files` again.

`org-id-extra-files` was set to org-agenda-text-search-extra-files. When I
set it to nil, nothing significant changed. It does not get reset when
`org-icalendar-combine-agenda-files` is run.

  -k.

missing packages

[Colin Baxter]

Hello,

I notice a few packages seem to be absent from the org-mode dev tree, 
e.g., org-bibtex.el and org-bbdb.el. Yet nothing I've seen in ORG-NEWS
suggests that they have been discontinued. I am probably not
understanding something rather elementary.

Best wishing

-- 
Colin Baxter
URL: http://www.Colin-Baxter.com

Re: org-babel :colnames yes

[ian martins]

I figured out that inserting `hline' works for some languages.  consistent
behavior with ":colnames yes" would be ideal but this solves my problem.

#+BEGIN_SRC elisp
  '((one two) hline (1 3) (1 6))
#+END_SRC

On Thu, May 28, 2020 at 7:48 AM ian martins  wrote:

> Hello, I'm trying to figure out how to tell org that the first row of a
> src block result is a table header.
>
> from readthedocs
>  it
> looks like ":colnames yes" should do this, but I haven't been able to get
> it to work, and the code
> 
> doesn't appear to want that to happen. maybe that is special handling for
> R .
>
> #+BEGIN_SRC elisp :colnames yes
>   '((one two) (1 3) (1 6))
> #+END_SRC
>
> I'm expecting an hline in the output but do not find one.
>
> I know ":colnames '(one two)" works, but I don't know the header value
> until I've generated the table.
>
> -Ian
>

[PATCH] [FEATURE] Re: `with` as a list.

[Mario Frasca]

I have added a couple of unit tests to the suite, describing the two 
functions I added.  I have no unexpectedly failing tests now.


I'm explicitly cc-ing Eric Schulte because he's in the header for 
org-plot.el, and —missing the unit tests for that source— I hope he can 
assist me not breaking what he wrote.  … I guess I can write also test 
cases for what already exists, but I will very likely need assistance.


best regards,
Mario Frasca

On 22/05/2020 11:07, Mario Frasca wrote:

good day to you all

now and then I use emacs to make graphs.  now recently I was plotting 
point data, and a running average "fit", so I wanted to have points, 
and lines, which I know it's possible in `gnuplot` but now how do I do 
that from org-plot …


I wrote a small patch for org-plot.el, I'm not a Lisp programmer so 
I'm sure the patch looks terrible, but it does allow me to do this:


#+PLOT: ind:1 deps:(3 6 4 7) with:(points lines points lines)

it's two additions:

1. it lets me specify the order in which the dependent columns should 
be considered.


2. it lets me specify a different `with` for each column, in the same 
order.


if you leave the `with` away, you get "lines" for all columns.

if you specify only one `with` value, that value is used for all columns.

if you specify more `deps` than `with`, the ones not specified will 
get "lines".


if you specify more `with` than `deps`, they are ignored.

I ran the tests, and I get two failing ones, quite unrelated according 
to me:


2 unexpected results:
   FAILED  ob-exp/evaluate-all-executables-in-order
   FAILED  ob-exp/export-call-line-information

I have not defined test cases for the new behaviour, I'm willing to do 
that (learning the way this test environment works), but I don't find 
the location of the other tests related to the area of the program, 
which I'm tweaking.


best regards all,

Mario Frasca

diff --git a/lisp/org-plot.el b/lisp/org-plot.el
index a23195d2a..524615d98 100644
--- a/lisp/org-plot.el
+++ b/lisp/org-plot.el
@@ -179,6 +179,28 @@ and dependent variables."
 	  (setf back-edge "") (setf front-edge ""
 row-vals))
 
+(defun org-plot/zip-deps-with (num-cols ind deps with)
+  "describe each column to be plotted as (col . with)"
+  ;; make 'deps explicit
+  (unless deps
+(setf deps (let (r)
+		 (dotimes (i num-cols r)
+		   (unless (eq num-cols (+ ind i))
+		 (setq r (cons (- num-cols i) r)))
+  ;; make sure 'with matches 'deps
+  (unless with
+(setf with "lines"))
+  (unless (listp with)
+(setf with (make-list (length deps) with)))
+  ;; invoke zipping function on converted data
+  (org-plot/zip deps with))
+
+(defun org-plot/zip (xs ys)
+  (unless
+  (null xs)
+(cons (cons (car xs) (or (car ys) "lines"))
+	  (org-plot/zip (cdr xs) (cdr ys)
+
 (defun org-plot/gnuplot-script (data-file num-cols params &optional preface)
   "Write a gnuplot script to DATA-FILE respecting the options set in PARAMS.
 NUM-COLS controls the number of columns plotted in a 2-d plot.
@@ -240,22 +262,24 @@ manner suitable for prepending to a user-specified script."
 			   "%Y-%m-%d-%H:%M:%S") "\"")))
 (unless preface
   (pcase type			; plot command
-	(`2d (dotimes (col num-cols)
-	   (unless (and (eq type '2d)
-			(or (and ind (equal (1+ col) ind))
-(and deps (not (member (1+ col) deps)
-		 (setf plot-lines
-		   (cons
-			(format plot-str data-file
-(or (and ind (> ind 0)
-	 (not text-ind)
-	 (format "%d:" ind)) "")
-(1+ col)
-(if text-ind (format ":xticlabel(%d)" ind) "")
-with
-(or (nth col col-labels)
-(format "%d" (1+ col
-			plot-lines)
+	(`2d (dolist
+		 (col-with
+		  (org-plot/zip-deps-with num-cols ind deps with))
+	   (setf plot-lines
+		 (cons
+		  (format plot-str data-file
+			  (or (and ind (> ind 0)
+   (not text-ind)
+   (format "%d:" ind)) "")
+			  (car col-with)
+			  (if text-ind (format ":xticlabel(%d)" ind) "")
+			  (cdr col-with)
+			  (apply (lambda (x)
+   (if (= 0 (length x))
+	   (format "%d" (car col-with))
+	 x))
+ (list (nth (1- (car col-with)) col-labels
+		  plot-lines
 	(`3d
 	 (setq plot-lines (list (format "'%s' matrix with %s title ''"
 	data-file with
@@ -310,7 +334,8 @@ line directly before or after the table."
 table data-file params)))
 		 (when y-labels (plist-put params :ylabels y-labels)
   ;; Check for timestamp ind column.
-  (let ((ind (1- (plist-get params :ind
+  (let ((ind (1- (plist-get params :ind)))
+	(with (plist-get params :with)))
 	(when (and (>= ind 0) (eq '2d (plist-get params :plot-type)))
 	  (if (= (length
 		  (delq 0 (mapcar
@@ -320,7 +345,7 @@ line directly before or after the table."
 		 0)
 	  (plist-put params :timeind t)
 	;; Check for text ind column.
-	(if (or (string= (plist-get params :with) "hist")
+	(if (or (and (stringp with) (string= with "h

Re: org-babel :colnames yes

[Nick Dokos]

Would this work for you?

--8<---cut here---start->8---
#+BEGIN_SRC elisp :colnames yes
 '((one two) hline (1 3) (1 6))
#+END_SRC

#+RESULTS:
| one | two |
|-+-|
|   1 |   3 |
|   1 |   6 |

--8<---cut here---end--->8---

-- 
Nick

"There are only two hard problems in computer science: cache
invalidation, naming things, and off-by-one errors." -Martin Fowler

Re: Bug: Subtle bug with calling "noweb" blocks with already computed :results output code and :cache yes [9.3.4 (9.3.4-5-ga0f3bb-elpa @ /home/lockywolf/.emacs.d/elpa/org-20200210/)]

[Nicolas Goaziou]

Hello,

Vladimir Nikishkin  writes:

> This bug is a bit hard to formulate. Have a look at the following
> org-file. Even though it's big, it's almost an MWE:

Emacs Lisp source code blocks would be way better.

Also, have you read cache limitations exposed at (info "(org)Limit code
block evaluation")? Note the part related to noweb expansion. Does that
explain what you observe?

Regards,

-- 
Nicolas Goaziou

Re: [BUG] All drawers except property drawers are open at startup

[Nicolas Goaziou]

Hello,

Matt Lundin  writes:

> All drawers default to open at startup except for property drawers. This
> includes :LOGBOOK: drawers, which can be quite long and block view of an
> entry.

I think all drawers are opened at startup, including property drawers.

> I ran a git bisect and discovered that the bug was introduced with
> commit 8b05c06d427e850d45684f69c5165cd7684e1071 on May 9. 

Isn't it related to 88f5ed91c5cf9cb6962c1b441eac7dbab9a4734e instead?

Regards,

-- 
Nicolas Goaziou

Re: Help speeding up Org iCal export

[Nicolas Goaziou]

Hello,

I see a few misconceptions here.

Ken Mankoff  writes:

> When I eval `(org-icalendar-combine-agenda-files nil)`, it takes about
> 15 seconds.

OK. You may want to profile this, or use ELP.

> Some strange messages in the *Messages* buffer include:
>
> 22 files scanned, 7 files contains IDs and in total 23 IDs found.

You may have set `org-icalendar-store-UID' somehow.

> But my `org-agenda-files` variable only contains 12 files and no
> #+INCLUDE statements.

ID scanning is not limited to agenda files. Org also scans files known
to contain ID. E.g., if, during the session, you insert an ID in
"foo.org", later scans (in the same session) will include "foo.org".

See `org-id-files' (internal variable), and `org-id-extra-files', which
may contain even more files.

> There are also several dozen lines of
>
> org-babel-exp process sh at position 74770...
> org-babel-exp process sh at position 74830...
> org-babel-exp process sh at position 75147...
> ...
> org-babel-exp process sqlite at position 27700...
> org-babel-exp process sqlite at position 27704...
> org-babel-exp process sh at position 27604...
> org-babel-exp process sh at position 27617...
> org-babel-exp process sh at position 27691...
> org-babel-exp process translate at position 27970...
>
> (and with bash, python, emacs-lisp, and other languages). When I view for 
> example the only #BEGIN_SRC translate block in any of my files it is just a 
> test in my notes file:
>
> #+BEGIN_SRC translate :src en :dest da
> This is a test
> #+END_SRC
>
>
> I cannot eval it with C-c C-c because I have set ":eval no"
> system-wide with:

This is a red herring, IMO. The message above means the block is being
processed for export, not necessarily evaluated. In order to know how to
handle the block, Org needs to process it.

>   (setq org-babel-default-header-args
>   (cons '(:eval . "no")
> (assq-delete-all :eval org-babel-default-header-args)))
>
>
> Sometimes the Messages contain confirmation that a code block is disabled:
>
> org-babel-exp process plantuml at position 27719...
> Evaluation of this plantuml code block is disabled.
> org-babel-exp process plantuml at position 27762...
> Evaluation of this plantuml code block is disabled.

Here, it tries to evaluate the blocks, obviously, but eventually
doesn't, per your settings.

> But those blocks don't have any special header:
>
> #+begin_src plantuml :file tryout.png
> ...misc plantuml code here.
> #+end_src

I see no issue here. Org checks if it should evaluate it, and cannot.

> Can anyone offer suggestions why it reports scanning 22 files, and how
> to speed it up since no code should be evaluated?

See above. Instead of wild guesses, you could use an appropriate tool.

Regards,

-- 
Nicolas Goaziou

Re: Bug: org export rewrites the org buffer. if Emacs crashes during the export, the file gets corrupted [9.3.4 (9.3.4-5-ga0f3bb-elpa @ /home/lockywolf/.emacs.d/elpa/org-20200210/)]

[Nicolas Goaziou]

Hello,

Vladimir Nikishkin  writes:

> When I was debugging export today, I found that while exporting, org
> destructively strips the header arguments from the blocks, such as
> :exports, :results, :file.

Export process takes place in a copy of the original buffer. You were
probably looking at that copy. If you do think this was the genuine
buffer, please provide an ECM.

Also, removing some source code headers is a natural part of the process.

Regards,

-- 
Nicolas Goaziou

org-babel :colnames yes

[ian martins]

Hello, I'm trying to figure out how to tell org that the first row of a src
block result is a table header.

from readthedocs
 it looks
like ":colnames yes" should do this, but I haven't been able to get it to
work, and the code

doesn't appear to want that to happen. maybe that is special handling for R
.

#+BEGIN_SRC elisp :colnames yes
  '((one two) (1 3) (1 6))
#+END_SRC

I'm expecting an hline in the output but do not find one.

I know ":colnames '(one two)" works, but I don't know the header value
until I've generated the table.

-Ian

Re: Possible fix for :includes header argument in org-babel C source blocks

[Kévin Le Gouguec]

Kévin Le Gouguec  writes:

> That leads me to believe that the coercion was an unintended side-effect
> of (format …).

Never mind, the ORG-NEWS entry for 9.1 shows an example of unquoted
header, so I guess it is intentional.

Here is a patch to fix the regression:

>From b68122821a26578470506938c3a358f52f5d7a46 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?K=C3=A9vin=20Le=20Gouguec?= 
Date: Thu, 28 May 2020 11:09:18 +0200
Subject: [PATCH 1/2] Coerce symbols found in :includes header arguments to
 strings

Fix regression from 2020-05-24T16:23:26Z!bran...@guttersohn.org
(commit 44cb98fdb), which broke test ob-C/string-var.

* lisp/ob-C.el (org-babel-C-expand-C): Make sure items in :includes
arguments are strings before performing string operations on them.
---
 lisp/ob-C.el | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lisp/ob-C.el b/lisp/ob-C.el
index c3e72c680..42c60c296 100644
--- a/lisp/ob-C.el
+++ b/lisp/ob-C.el
@@ -233,6 +233,9 @@ its header arguments."
 		;; includes
 		(mapconcat
 		 (lambda (inc)
+		   ;; :includes '( ) gives us a list of
+		   ;; symbols; convert those to strings.
+		   (when (symbolp inc) (setq inc (symbol-name inc)))
 		   (if (string-prefix-p "<" inc)
 		   (format "#include %s" inc)
 		 (format "#include \"%s\"" inc)))
-- 
2.17.1


And here is a patch to add a test for the unquoted-single-header case,
since otherwise it's hard to tell whether this behaviour is intentional:

>From cf1bb27215a46a521bb2f50d16b7dbc7441d81ec Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?K=C3=A9vin=20Le=20Gouguec?= 
Date: Thu, 28 May 2020 11:47:25 +0200
Subject: [PATCH 2/2] Add test case for symbol coercion in C Babel blocks

The ORG-NEWS entry for version 9.1 suggests that this coercion was
always intended, though AFAICT there was no test case for it.

* testing/lisp/test-ob-C.el (ob-C/symbol-include): Check explicitly
that :includes  (with no double-quotes around )
will be parsed correctly.
(ob-D/simple-program, ob-C/integer-var, ob-D/integer-var,
ob-C/two-integer-var, ob-D/two-integer-var, ob-C/string-var,
ob-D/string-var, ob-C/preprocessor): Adjust block indices.

* testing/examples/ob-C-test.org (Simple tests): Add input for the new
test.
---
 testing/examples/ob-C-test.org |  6 ++
 testing/lisp/test-ob-C.el  | 23 +++
 2 files changed, 21 insertions(+), 8 deletions(-)

diff --git a/testing/examples/ob-C-test.org b/testing/examples/ob-C-test.org
index 0faf630b9..347607cae 100644
--- a/testing/examples/ob-C-test.org
+++ b/testing/examples/ob-C-test.org
@@ -10,6 +10,12 @@
   return 0;
 #+end_src
 
+#+source: simple
+#+begin_src cpp :includes  :results silent
+  std::cout << 42;
+  return 0;
+#+end_src
+
 #+source: simple
 #+begin_src D :results silent
   writefln ("%s", 42);
diff --git a/testing/lisp/test-ob-C.el b/testing/lisp/test-ob-C.el
index 3e4a63f88..6b6b728a2 100644
--- a/testing/lisp/test-ob-C.el
+++ b/testing/lisp/test-ob-C.el
@@ -32,60 +32,67 @@
 		  (org-babel-next-src-block 1)
 		  (should (= 42 (org-babel-execute-src-block))
 
+(ert-deftest ob-C/symbol-include ()
+  "Hello world program with unquoted :includes."
+  (if (executable-find org-babel-C++-compiler)
+  (org-test-at-id "fa6db330-e960-4ea2-ac67-94bb845b8577"
+		  (org-babel-next-src-block 2)
+		  (should (= 42 (org-babel-execute-src-block))
+
 (ert-deftest ob-D/simple-program ()
   "Hello world program."
   (if (executable-find org-babel-D-compiler)
   (org-test-at-id "fa6db330-e960-4ea2-ac67-94bb845b8577"
-		  (org-babel-next-src-block 2)
+		  (org-babel-next-src-block 3)
 		  (should (= 42 (org-babel-execute-src-block))
 
 (ert-deftest ob-C/integer-var ()
   "Test of an integer variable."
   (if (executable-find org-babel-C++-compiler)
   (org-test-at-id "fa6db330-e960-4ea2-ac67-94bb845b8577"
-		  (org-babel-next-src-block 3)
+		  (org-babel-next-src-block 4)
 		  (should (= 12 (org-babel-execute-src-block))
 
 (ert-deftest ob-D/integer-var ()
   "Test of an integer variable."
   (if (executable-find org-babel-D-compiler)
   (org-test-at-id "fa6db330-e960-4ea2-ac67-94bb845b8577"
-		  (org-babel-next-src-block 4)
+		  (org-babel-next-src-block 5)
 		  (should (= 12 (org-babel-execute-src-block))
 
 (ert-deftest ob-C/two-integer-var ()
   "Test of two input variables"
   (if (executable-find org-babel-C++-compiler)
   (org-test-at-id "fa6db330-e960-4ea2-ac67-94bb845b8577"
-		  (org-babel-next-src-block 5)
+		  (org-babel-next-src-block 6)
 		  (should (= 22 (org-babel-execute-src-block))
 
 (ert-deftest ob-D/two-integer-var ()
   "Test of two input variables"
   (if (executable-find org-babel-D-compiler)
   (org-test-at-id "fa6db330-e960-4ea2-ac67-94bb845b8577"
-		  (org-babel-next-src-block 6)
+		  (org-babel-next-src-block 7)
 		  (should (= 22 (org-babel-execute-src-block))
 
 (ert-deftest ob-C/string-var ()
   "Test of a string input variable"
   (if (executable

Re: Emacs-orgmode Digest, Vol 171, Issue 30

[Daryl Manning]

On Thu, May 28, 2020 at 12:01 AM  wrote:

> Send Emacs-orgmode mailing list submissions to
> emacs-orgmode@gnu.org
>
> --
>
> Message: 24
> Date: Wed, 27 May 2020 07:11:08 -0400
> From: Matt Price 
> Cc: emacs-orgmode 
> Subject: Re: Improving Org Mode for VSCode - Thinking Aloud
> Message-ID:
>  tpbgzx70_hdu3_hkrjrnbuf29ct...@mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>

Yes, the VS Code org-mode extension is around, but it's still pretty
bare-bones in terms of overall functionality.
(also, for the jupyter comments, I think the MS Python plugin for VS Code
already includes being able to
use and run jupyter notebooks directly from VS Code already though unsure
if you can connect that
to something like GCP's Cloud Datalab or similar remote jupyter servers
rather than localhost)

For me, the grail would be getting org-agenda and deft working inside VS
Code. Those were the gateway
drugs that had me switch from Sublime Text to Emacs in the first place...
=]

ciao!
Daryl.

There are a number of interesting topics here. Like others, I would be very
> keen to share my org-based projects with non-Emacs users. I also only have
> very limited time & skills to contribute.  There is an existing extension
> for org-mode on vscode here:
> https://github.com/vscode-org-mode/vscode-org-mode.  That might be the
> best
> place to help out. One piece of low-hanging fruit would be simply to add an
> HTML renderer to that plugin (https://github.com/orgapp/orgajs is the
> obvious choice). A full-on literate programming engine seems like a very
> tough thing to engineer, but perhaps it's possible to imagine a new
> architecture in which there are, as you suggest, various code
> interpretation engines. It's an exciting idea; I wonder if it's possible to
> explore the possibilities in a more concrete and structured way?
>
> On Sun, May 24, 2020 at 4:20 PM rey-coyrehourcq <
> sebastien.rey-coyrehou...@univ-rouen.fr> wrote:
>
> > Hi,
> >
> > Really cool discussion here.
> >
> > My two cents, Jupyter & Kernel on various langage, which have a very
> > large community, could be an interesting backend for org-babel on
> > VSCode or anyeditoryouwantusehere.
> >
> > Lot of things start to appear to collaborate online around
> > online/scientific cnotebook/literate programing this last year, and that
> > continue, for example i discover recently :
> > Stencila : https://github.com/stencila=
> >
> > org-babel / org-mode outside of emacs has a great potential to lead an
> > alternative to other markdown (RStudio / Jupyter) solution.
> >
> > Best regards
> >
> > Le dimanche 24 mai 2020 à 06:06 -0700, Jack Kamm a écrit :
> > > It would be very good indeed for org-babel if it could be ported to
> > > other editors. One of the biggest drawbacks of org-babel notebooks is
> > > that I can't collaborate with my colleagues on them, since I can't
> > > expect them to use Emacs.
> > >
> > > Aside from VSCode, I think RStudio would be an excellent target for a
> > > few reasons:
> > >
> > > - Literate programming is already popular among R users (see also:
> > >   knitr, sweave, Rmarkdown)
> > > - There is a strong ob-R community here
> > > - There are some prominent Emacs users among the Rstudio developers
> > >   (e.g. Lionel Henry, who I think is both an Rstudio and ESS
> > > developer)
> > >
> > > However, this would be a massive undertaking, and ultimately would
> > > need
> > > a volunteer to step up to the plate. I don't have any bandwidth to do
> > > this in the foreseeable future but dream of working on it one day.
> > > The
> > > biggest downside -- it would require spending considerable time
> > > outside
> > > Emacs!
> > >
> > --
> >
> >
> > Sébastien Rey-Coyrehourcq
> > Research Engineer UMR IDEES
> > 02.35.14.69.30
> >
> > {Stronger security for your email, follow EFF tutorial :
> > https://ssd.eff.org/}
> >
> >
> >
>

Bug: org export rewrites the org buffer. if Emacs crashes during the export, the file gets corrupted [9.3.4 (9.3.4-5-ga0f3bb-elpa @ /home/lockywolf/.emacs.d/elpa/org-20200210/)]

[Vladimir Nikishkin]

Remember to cover the basics, that is, what you expected to happen and
what in fact did happen.  You don't know how to make a good report?  See

 https://orgmode.org/manual/Feedback.html#Feedback

Your bug report will be posted to the Org mailing list.


I have a very long org file, more than 7 lines.

Exporting it is a naturally long process.
Additionally, some blocks are malformed, so the export process fails.
to debug them, I use toggle-debug-on-error

When I was debugging export today, I found that while exporting, org
destructively strips the header arguments from the blocks, such as
:exports, :results, :file.

It should be normally putting them back, I guess, but if
toggle-debug-on-error is on, and there are places where (debug) is
invoked or if emacs crashes (or is killed) during the export process,
the headers never go back.

Test file: https://gitlab.com/Lockywolf/chibi-sicp/-/blob/master/index.org

Emacs  : GNU Emacs 26.3 (build 1, x86_64-slackware-linux-gnu, GTK+ Version 
3.24.12)
 of 2019-11-26
Package: Org mode version 9.3.4 (9.3.4-5-ga0f3bb-elpa @ 
/home/lockywolf/.emacs.d/elpa/org-20200210/)

current state:
==
(setq
 org-src-mode-hook '((lambda nil (modify-syntax-entry 60 ".") 
(modify-syntax-entry 62 ".")
  (setq geiser-autodoc--inhibit t))
 org-src-babel-configure-edit-buffer 
org-src-mode-configure-edit-buffer)
 org-latex-listings t
 org-link-shell-confirm-function 'yes-or-no-p
 org-babel-after-execute-hook '((lambda nil
 (if org-inline-image-overlays
  (progn (org-redisplay-inline-images)))
 )
)
 org-metadown-hook '(org-babel-pop-to-session-maybe)
 org-export-with-sub-superscripts '{}
 org-clock-out-hook '(org-clock-remove-empty-clock-drawer)
 org-html-format-inlinetask-function 
'org-html-format-inlinetask-default-function
 org-pretty-entities t
 org-odt-format-headline-function 'org-odt-format-headline-default-function
 org-link-descriptive nil
 org-latex-pdf-process '("latexmk  -lualatex -outdir=%o -f %f")
 org-agenda-files '("~/Personal_Planner/Planner.org" 
"~/Personal_Planner/lockywolf_net.org"
"~/Personal_Planner/cyberpunk_smarthome.org"
"~/Personal_Planner/notes.org"
"~/Personal_Planner/2020-04-28-hotkeys-to-remember.txt.org"
"~/DevLinux/chibi-sicp/index.org")
 org-ascii-format-inlinetask-function 'org-ascii-format-inlinetask-default
 org-modules '(org-habit ol-w3m ol-bbdb ol-bibtex ol-docview ol-gnus ol-info 
ol-irc ol-mhe
   ol-rmail ol-eww)
 org-plantuml-jar-path "/usr/local/bin/plantuml.jar"
 org-mode-hook '(#[0 "\301\211\207" [imenu-create-index-function 
org-imenu-get-tree] 2]
 turn-on-org-cdlatex org-tempo-setup
 (lambda nil
  (add-to-list (quote org-structure-template-alist)
   (quote ("p" . "src plantuml :noweb no :exports both :file 
")))
  (add-to-list (quote org-structure-template-alist)
   (quote ("SO" . "src scheme :exports both :results output")))
  (add-to-list (quote org-structure-template-alist)
   (quote ("SV" . "src scheme :exports both :results value")))
  (add-to-list (quote org-structure-template-alist)
   (quote ("SE" . "src scheme :exports code")))
  (add-to-list (quote org-structure-template-alist)
   (quote ("EL" . "src elisp :exports both :results output")))
  (add-to-list (quote org-structure-template-alist)
   (quote ("F" . "src f90 :exports both :results output")))
  )
 (lambda nil (modify-syntax-entry 60 ".") (modify-syntax-entry 
62 "."))
 (lambda nil (imenu-add-to-menubar "Imenu"))
 #[0 "\300\301\302\303\304$\207"
   [add-hook change-major-mode-hook org-show-all append local] 
5]
 #[0 "\300\301\302\303\304$\207"
   [add-hook change-major-mode-hook org-babel-show-result-all 
append local] 5]
 org-babel-result-hide-spec org-babel-hide-all-hashes 
org-eldoc-load)
 org-odt-format-drawer-function #[514 "\207" [] 3 "\n\n(fn NAME CONTENTS)"]
 org-archive-hook '(org-attach-archive-delete-maybe)
 org-confirm-elisp-link-function 'yes-or-no-p
 org-agenda-before-write-hook '(org-agenda-add-entry-text)
 org-preview-latex-default-process 'imagemagick
 org-metaup-hook '(org-babel-load-in-session-maybe)
 org-bibtex-headline-format-function #[257 "\300\236A\207" [:title] 3 "\n\n(fn 
ENTRY)"]
 org-latex-format-drawer-function #[514 "\207" [] 3 "\n\n(fn _ CONTENTS)"]
 org-babel-pre-tangle-hook '(save-buffer)
 org-latex-compiler "lualatex"
 org-tab-first-hook '(org-babel-h

Re: Possible fix for :includes header argument in org-babel C source blocks

[Kévin Le Gouguec]

Kyle Meyer  writes:

> I think this discussion was on emacs-devel only, so here are some links
> for others who might go looking for more context:
>
>   https://lists.gnu.org/archive/html/emacs-devel/2020-04/msg01880.html
>   https://lists.gnu.org/archive/html/emacs-devel/2020-05/msg03051.html

(Thanks, I should have thought to add some context before forwarding.)

>> I guess there might be some people out there who will expect things to
>> keep working without double-quotes?  I have never used Babel, so I have
>> no idea…
>
> I don't know either, but the test does make me think so.  Hopefully
> Brandon, Bastien, or someone else will chime in if that's not the case.

My opinion should only carry so much weight since I don't use Babel, but
from a quick reading of the sources, I couldn't find other examples of
this (symbol → string) coercion.  The only other instances of
lists-of-strings I could find in testing/examples were

> :var a='("abc" "def")

That leads me to believe that the coercion was an unintended side-effect
of (format …).

Of course, backward compatibility alone would mandate keeping the
coercion.

> Could you send the first patch with a commit message tacked on?

Will do ASAP.

BTW, does the change from 44cb98fdb deserve an ORG-NEWS entry?

Re: [O] Help speeding up Org iCal export

[Eric S Fraga]

On Wednesday, 27 May 2020 at 17:24, Ken Mankoff wrote:
> Can anyone offer suggestions why it reports scanning 22 files, and how
> to speed it up since no code should be evaluated?

Anecdotally, I have the impression that all org buffers in the current
Emacs instance are scanned as well as the files noted in the agenda
list.  But I have no proof of this nor do I know why it would do
this.  I have the same issue with org-caldav-sync.

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.3.6-640-g9bc0cc

Re: org table: one column of random numbers (but natural ones)

[Eric S Fraga]

On Wednesday, 27 May 2020 at 22:40, Uwe Brauer wrote:
> I have a org table and want to add a new colum, which natural numbers
> which are randomly ordered 

Something like 

#+TBLFM: $1=floor(100*random(0.1))

I don't know what argument you want to pass to random(), especially as
you had $1@4 which seems wrong?

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.3.6-640-g9bc0cc

Re: org table: one column of random numbers (but natural ones)

[Eric S Fraga]

Ignore previous reply.  I misread your email.  Too early in the morning
for me.

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.3.6-640-g9bc0cc