Re: [O] [ANN] Agenda speed up

[Kyle Meyer]

Nicolas Goaziou  writes:

> Kyle Meyer  writes:
>
>> Nicolas Goaziou  writes:
>>
>>> If there is no more feedback nor objection, I'll merge the branch in
>>> master before the end of the week.
>>
>> One issue I noticed: The sorting of categories defined in
>> org-agenda-sorting-strategy is not honored.  The default category
>> sorting for the agenda is to order the categories as they appear in the
>> agenda files.

[...]

> Fixed. Thank you.

Thank you.  Hit a second issue today.

(setq org-agenda-custom-commands
  '(("u" "Unschedule TODO entries" alltodo ""
 ((org-agenda-skip-function
   (lambda ()
 (org-agenda-skip-entry-if 'scheduled 'deadline)))
  (org-agenda-overriding-header "Unscheduled TODO entries: ")

On master and with an agenda file that contains

* TODO not scheduled

* TODO scheduled
SCHEDULED: <2017-10-07 Sat>

running "M-x org-agenda u" shows the "not scheduled" heading only.
wip-agenda-speedup show both entries.

-- 
Kyle

Re: [O] [ANN] Agenda speed up

[Samuel Wales]

bugs:

the following line will produce an entry in the agenda.

#   S CHEDULED: <%%(memq (calendar-day-of-week date) '(1 2 3 4 5))>

and tel: links are interpreted as timestamps.

and the closing time for clocks are not listed.

and it misses an entire, normal looking inactive timestamp that is the
entry after tel.

===

personal opinion: i was excited that the agenda was going to be sped
up, but i thought that that would apply to my regular 2d agenda, which
is the only view i commonly use.  but if the speedup will slow down 2d
in order to make 30d faster, i personally do not favor this branch.  i
almost never do n greater than 2.

i /expect/ 30d to be slow, but i want 1d and 2d to be extremely fast.
like instant.

ymmv.

===

unlikely i can contribute to further testing much if at all, for health reasons.

Re: [O] [ANN] Agenda speed up

[Samuel Wales]

On 10/3/17, Nicolas Goaziou  wrote:
>> org-element-at-point  1104
>> 17.213785437  0.0155921969
>> org-element--parse-to 1104
>> 17.098407592  0.0154876880
>> org-element--current-element  69978
>> 15.421312201  0.0002203737
>> org-at-clock-log-p12745
>> 12.127817233  0.0009515745
>
> Fixed. Could you test it again?

still much slower.  check out the filenames for the times.

--- "elp-results--(14.120262055 14 0.868587649)--release_9.1.1-114-gf5bc56" 
+++ "elp-results--(2.823281612 3
0.1366931619998)--release_9.1.1-99-g7a2b64"
@@ -1,85 +1,85 @@
-org-at-planning-p 19438
1.7517455880  9.011...e-05
-org-agenda-prepare-buffers1
1.280357753   1.280357753
-org-refresh-category-properties   8
0.5655261619  0.0706907702
-org-back-to-heading   3831
0.4543286779  0.0001185927
-org-parse-time-string 2739
0.3607386119  0.0001317044
-org-matcher-time  195
0.3364286490  0.0017252751
-org-2ft   192
0.3350386229  0.0017449928
-org-end-of-subtree197
0.2835889550  0.0014395378
-org-refresh-stats-properties  8
0.216994121   0.0271242651
-org-entry-get 1001
0.1716580229  0.0001714865
-org-entry-properties  994
0.1433018570  0.0001441668
-org-refresh-properties16
0.130269807   0.0081418629
-org-closest-date  1000
0.1084333750  0.0001084333
-org-at-property-p 121
0.1060563869  0.0008764990
-org-get-property-block128
0.092064553   0.0007192543
-org-refresh-effort-properties 8
0.071788969   0.0089736211
-org-get-priority  312
0.0678388049  0.0002174320
-org-set-regexps-and-options   8
0.0584633600  0.0073079200
-org--setup-collect-keywords   8
0.057983882   0.0072479852
-org-before-first-heading-p215
0.023083193   0.0001073636
-org-today 1985
0.0227823760  1.147...e-05
-org-in-commented-heading-p80
0.0167247850  0.0002090598
-org-outline-level 197
0.0147668000  7.495...e-05
-org-get-tags-at   321
0.0123168869  3.837...e-05
-org-heading-components80
0.0086148070  0.0001076850
-org-activate-links35
0.0079194529  0.0002262700
-org-get-agenda-file-buffer32
0.005901701   0.0001844281
-org-find-base-buffer-visiting 32
0.0057470019  0.0001795938
-org-get-limited-outline-regexp2009
0.0046851680  2.332...e-06
-org-date-to-gregorian 544
0.0042989290  7.902...e-06
-org-add-props 981
0.0042024999  4.283...e-06
-org-refresh-property  7
0.003612339   0.0005160484
-org--property-local-values7
0.0034815799  0.0004973685
-org-element-at-point  2
0.003142827   0.0015714135
-org-plist-delete  310
0.0022100709  7.129...e-06
-org-days-to-iso-week  2
0.0021368679  0.0010684339
-org-get-wdays 966
0.0021314989  2.206...e-06
-org-element-context   1
0.001709069   0.001709069
-org-element--parse-to 1
0.001696425   0.001696425
-org-get-todo-face 310
0.0016314150  5.262...e-06
-org-get-category  321
0.0013590150  4.233...e-06
-org-element-link-parser   34
0.001083772   3.187...e-05
-org-element-headline-parser   1
0.001021417   0.001021417
-org-eval  325
0.0009354789  2.878...e-06
-org-check-agenda-file 16
0.000722695   4.516...e-05
-org-defkey141
0.0006374779  4.521...e-06
-org-font-lock-add-tag-faces   1
0.000607730.00060773
-org-in-src-block-p47
0.000511424   1.088...e-05
-org-element--current-element  3
0.000489721   0.0001632403
-org-agenda-files  3
0.000378044   0.0001260146
-org-reduced-level 

Re: [O] [ANN] Agenda speed up

[Marco Wahl]

>> The overall outcome (from elp) was:
>>
>> | wip-agenda-speedup | 823.31343007 |
>> | master |  13.70077639 |
>
> Fixed.

Thanks!  Looks much better now.

>  Could you test it again?

Now the numbers (for the agenda today [2017-10-03 Tue 12:19]) are

| wip-agenda-speedup | 40.3 |
| master | 13.9 |

Details below.


Best regards,  Marco


*** wip-agenda-speedup(f5bc563a6)

- first agenda creation

#+begin_src emacs-lisp :results drawer
(elp-instrument-package "org-")
(org-agenda-list)
(elp-results)
(set-buffer "*ELP Profiling Results*")
(buffer-string)
#+end_src

#+RESULTS:
:RESULTS:
org-agenda-list   1   
40.319749877  40.319749877
org-agenda--all-filtered-data 1   
25.853502855  25.853502855
org-agenda--file-data 13  
24.455112311  1.8811624855
org-agenda--inactive-data 13  
20.348160329  1.5652431022
org-agenda-prepare1   
8.6519741709  8.6519741709
org-agenda-prepare-buffers1   
8.647676857   8.647676857
org-get-agenda-file-buffer39  
8.192441179   0.2100625943
org-mode  12  
7.821512181   0.6517926817
org-set-startup-visibility12  
7.3841486830  0.6153457235
org-cycle-hide-drawers12  
6.8701314540  0.5725109545
org-element-at-point  6009
4.7852410630  0.0007963456
org-at-planning-p 28631   
4.6709920739  0.0001631445
org-element--parse-to 6009
4.1859619890  0.0006966154
org-element--current-element  10699   
2.7955038890  0.0002612864
org-agenda--planning-data 13  
1.960562504   0.1508125003
org-agenda-day-entries13  
1.8965473350  0.1458882565
org-agenda--timestamp-data13  
1.5601358319  0.1200104486
org-agenda-skip   27955   
1.0537350700  3.769...e-05
org-flag-drawer   5938
0.8942473320  0.0001505973
org-inlinetask-in-task-p  2619
0.8825332480  0.0003369733
org-agenda-skip-eval  55910   
0.8576132569  1.533...e-05
org-entry-get 1021
0.8163370580  0.0007995465
org--property-local-values985 
0.7918385440  0.0008038969
org-is-habit-p929 
0.7781246909  0.0008375938
org-get-property-block1048
0.7403500450  0.0007064408
org-element-planning-parser   1963
0.6084338040  0.0003099509
org-element-timestamp-parser  2448
0.4564960249  0.0001864771
org-element-property-drawer-parser5684
0.4392262350  7.727...e-05
org-overview  12  
0.409869880.0341558233
org-element-drawer-parser 2454
0.3520127849  0.000143
org-get-limited-outline-regexp31331   
0.3462765439  1.105...e-05
org-at-heading-p  19729   
0.3450200739  1.748...e-05
org-back-to-heading   3113
0.3358626239  0.0001078903
org-agenda-finalize-entries   1   
0.277405615   0.277405615
org-inlinetask-outline-regexp 2619
0.2371466240  9.054...e-05
org-before-first-heading-p1108
0.2354297489  0.0002124817
org-refresh-properties38  
0.2335974369  0.0061473009
org-parse-time-string 3792
0.2318385470  6.113...e-05
org-element--collect-affiliated-keywords  3023
0.2182822999  7.220...e-05
org-refresh-effort-properties 25  
0.1549454639  0.0061978185
org-outline-level 7979
0.1289716940  1.616...e-05
org-refresh-category-properties   13  
0.1254479299  0.0096498407
org-get-tags-at   80  
0.1213341579  

Re: [O] [ANN] Agenda speed up

[Nicolas Goaziou]

Hello,

Samuel Wales  writes:

> fr: org-git-version to show branch name.
>
> On 10/2/17, Nicolas Goaziou  wrote:
>> Do you mean you get an error which was fixed earlier? What error?
>
> you fixed error.
>
>>> ... but 9maint and 9master produce agenda in about 3s ...
>>>
>>> ... while wip produces agenda in 26.75s ...
>>
>> This is obviously a bug. I would need a complete ELP report to fix it.
>
> org-element-at-point  1104
> 17.213785437  0.0155921969
> org-element--parse-to 1104
> 17.098407592  0.0154876880
> org-element--current-element  69978
> 15.421312201  0.0002203737
> org-at-clock-log-p12745
> 12.127817233  0.0009515745

Fixed. Could you test it again?

Thank you.

Regards,

-- 
Nicolas Goaziou

Re: [O] [ANN] Agenda speed up

[Nicolas Goaziou]

Hello,

Marco Wahl  writes:

> The overall outcome (from elp) was:
>
> | wip-agenda-speedup | 823.31343007 |
> | master |  13.70077639 |

Fixed. Could you test it again?

Thank you.

Regards,

-- 
Nicolas Goaziou

Re: [O] [ANN] Agenda speed up

[Samuel Wales]

fr: org-git-version to show branch name.

On 10/2/17, Nicolas Goaziou  wrote:
> Do you mean you get an error which was fixed earlier? What error?

you fixed error.

>> ... but 9maint and 9master produce agenda in about 3s ...
>>
>> ... while wip produces agenda in 26.75s ...
>
> This is obviously a bug. I would need a complete ELP report to fix it.

org-element-at-point  1104
17.213785437  0.0155921969
org-element--parse-to 1104
17.098407592  0.0154876880
org-element--current-element  69978
15.421312201  0.0002203737
org-at-clock-log-p12745
12.127817233  0.0009515745
org-element-clock-parser  38277
6.7017031099  0.0001750843
org-element-timestamp-parser  39144
3.5316997229  9.022...e-05
org-element--list-struct  28737
3.4365959609  0.0001195878
org-match-line30147
1.5677620349  5.200...e-05
org-at-planning-p 19352
1.5514231689  8.016...e-05
org-agenda-prepare-buffers1
1.280418243   1.280418243
org-element-plain-list-parser 28737
1.1539211430  4.015...e-05
org-parse-time-string 79904
1.0831344910  1.355...e-05
org-refresh-category-properties   8
0.5659694900  0.0707461862
org-element--collect-affiliated-keywords  29844
0.4167584269  1.396...e-05
org-back-to-heading   3587
0.3900127680  0.0001087295
org-element-drawer-parser 1095
0.3539328980  0.0003232263
org-end-of-subtree197
0.3254195049  0.0016518756
org-at-heading-p  100926
0.3098276209  3.069...e-06
org-get-limited-outline-regexp75242
0.2670187110  3.548...e-06
org-entry-get 850
0.2172587780  0.0002555985
org-refresh-stats-properties  8
0.215825033   0.0269781291
org-entry-properties  843
0.1923222459  0.0002281402
org-element-planning-parser   852
0.1566764160  0.0001838925
org-refresh-properties16
0.129169336   0.0080730835
org-element-property-drawer-parser1005
0.0981411630  9.765...e-05
org-element--cache-put69978
0.0852441850  1.218...e-06
org-in-src-block-p12813
0.0793080239  6.189...e-06
org-refresh-effort-properties 8
0.0712755180  0.0089094397
org-at-property-p 121
0.0647199699  0.0005348757
org-item-re   57486
0.0632800570  1.100...e-06
org-get-priority  277
0.0614306549  0.0002217713
org-set-regexps-and-options   8
0.058658  0.00733225
org--setup-collect-keywords   8
0.0581624289  0.0072703036
org-outline-level 197
0.0578927239  0.0002938716
org-get-property-block128
0.050565393   0.0003950421
org-closest-date  980
0.0252995829  2.581...e-05
org-before-first-heading-p216
0.0231139179  0.0001070088
org-today 1955
0.0225680109  1.154...e-05
org-in-commented-heading-p81
0.0170685310  0.0002107226
org-get-tags-at   277
0.0109338869  3.947...e-05
org-heading-components81
0.0088255099  0.0001089569
org-activate-links28
0.0064327669  0.0002297416
org-get-agenda-file-buffer32
0.0059095189  0.0001846724
org-find-base-buffer-visiting 32
0.005762442   0.0001800763
org-date-to-gregorian 520
0.0040966750  7.878...e-06
org-refresh-property  7
0.003666006   0.0005237151
org-add-props 847
0.0036215150  4.275...e-06
org--property-local-values7
0.003564942   0.0005092774
org-days-to-iso-week  2
0.002134759   0.0010673795
org-get-wdays 956
0.0020431799  2.137...e-06
org-plist-delete  269
0.001759547   6.541...e-06
org-get-todo-face 269
0.0013906500  5.169...e-06
org-get-category  277
0.0011122950  4.015...e-06
org-element-paragraph-parser  12
0.0009095570  7.579...e-05
org-element-link-parser 

Re: [O] [ANN] Agenda speed up

[Marco Wahl]

Hi,

Here is my contribution to the testing.

I used my personal setting.  I started a fresh Emacs and executed the
block

#+begin_src emacs-lisp :results drawer
(elp-instrument-package "org-")
(org-agenda-list)
(elp-results)
(set-buffer "*ELP Profiling Results*")
(buffer-string)
#+end_src

With branch
- wip-agenda-speedup
- master.

The overall outcome (from elp) was:

| wip-agenda-speedup | 823.31343007 |
| master |  13.70077639 |

Find details below.


HTH,  Marco


I have 13 org-agenda-files.

*** wip-agenda-speedup

- first agenda creation

#+begin_src emacs-lisp :results drawer
(elp-instrument-package "org-")
(org-agenda-list)
(elp-results)
(set-buffer "*ELP Profiling Results*")
(buffer-string)
#+end_src

#+RESULTS:
:RESULTS:
org-agenda-list   1   
823.31343007  823.31343007
org-agenda--all-filtered-data 1   
803.36990444  803.36990444
org-agenda--file-data 13  
802.25572081  61.711978524
org-agenda--inactive-data 12  
798.06812224  66.505676853
org-element-at-point  17192   
787.87165159  0.0458278066
org-element--parse-to 17192   
785.73708480  0.0457036461
org-at-clock-log-p25911   
784.47874411  0.0302758961
org-element--current-element  2547363 
712.31832548  0.0002796296
org-element-clock-parser  1369922 
290.01055289  0.0002116985
org-element-timestamp-parser  1381808 
233.29855592  0.0001688357
org-element--list-struct  1136002 
129.73331616  0.0001142016
org-parse-time-string 2752951 
107.34721023  3.899...e-05
org-at-heading-p  3714980 
34.288651923  9.229...e-06
org-element-plain-list-parser 1136003 
30.732969505  2.705...e-05
org-element--collect-affiliated-keywords  1150439 
24.256669741  2.108...e-05
org-get-limited-outline-regexp2601514 
23.139213300  8.894...e-06
org-item-re   2272266 
16.601373308  7.306...e-06
org-agenda-prepare1   
9.495683098   9.495683098
org-agenda-prepare-buffers1   
9.488834312   9.488834312
org-get-agenda-file-buffer38  
8.800481890.2315916286
org-mode  12  
8.0912638009  0.6742719834
org-element--cache-put2547363 
7.7647574699  3.048...e-06
org-set-startup-visibility12  
7.6124138650  0.6343678220
org-cycle-hide-drawers12  
6.696957008   0.5580797506
org-element-drawer-parser 13748   
4.0234334710  0.0002926559
org-at-planning-p 28671   
2.6218224390  9.144...e-05
org-element-planning-parser   10681   
2.5078197379  0.0002347925
org-match-line53012   
1.9656340269  3.707...e-05
org-agenda--planning-data 12  
1.880672274   0.1567226894
org-element-property-drawer-parser16252   
1.6516716070  0.0001016288
org-agenda-day-entries12  
1.299214825   0.1082679020
org-agenda--timestamp-data12  
1.2739375050  0.1061614587
org-flag-drawer   5937
1.1515499559  0.0001939615
org-entry-get 998 
0.8434770640  0.0008451673
org-is-habit-p918 
0.8054251420  0.0008773694
org-overview  12  
0.8051965750  0.0670997145
org--property-local-values974 
0.6132794840  0.0006296503
org-get-property-block1037
0.582717840.0005619265
org-inlinetask-in-task-p  2607
0.5429532830  0.0002082674
org-agenda-skip   16806   
0.3787984219  2.253...e-05
org-refresh-category-properties   13  
0.335916684   0.0258397449
org-back-to-heading   

Re: [O] [ANN] Agenda speed up

[Nicolas Goaziou]

Hello,

Samuel Wales  writes:

> ... and i am getting a fixed error from before ...

Do you mean you get an error which was fixed earlier? What error?

> ... but 9maint and 9master produce agenda in about 3s ...
>
> ... while wip produces agenda in 26.75s ...

This is obviously a bug. I would need a complete ELP report to fix it.

> for my 2d agenda.  elp as instructed shows no significant differences
>
> org-agenda-prepare-buffers1
> 1.221983848   1.221983848
> org-agenda-files  3
> 0.000375030.00012501

This is not a complete ELP report. Use 

  M-x elp-instrument-package RET org RET

use the 26.75s command then display the report with

  M-x elp-results

> also there are sorting differences in time grid for same minute and
> for scheduled for same day.  which i don't care about.

I think I fixed sorting order.

> also an inactive timestamp shows the headline above the headline that
> contains it.  this is a bug but cannot mce.

I couldn't reproduce it after some quick tests.

Regards,

-- 
Nicolas Goaziou

Re: [O] [ANN] Agenda speed up

[Nicolas Goaziou]

Hello,

Matt Lundin  writes:

> Nicolas Goaziou  writes:
>
>> Some feedback about the new agenda speed would be nice.
>
> One small bug I found with wip speedup branch. When trying to reschedule
> in the agenda with org-agenda-do-date-later or
> org-agenda-do-date-earlier, org mode gives a message:
>
> "Cannot find time stamp"

Fixed. Thank you.

Regards,
-- 
Nicolas Goaziou

Re: [O] [ANN] Agenda speed up

[Nicolas Goaziou]

Hello,

Kyle Meyer  writes:

> Nicolas Goaziou  writes:
>
>> If there is no more feedback nor objection, I'll merge the branch in
>> master before the end of the week.
>
> One issue I noticed: The sorting of categories defined in
> org-agenda-sorting-strategy is not honored.  The default category
> sorting for the agenda is to order the categories as they appear in the
> agenda files.
>
> Take the following text as the agenda file.
>
>
> * y
> :PROPERTIES:
> :CATEGORY: y
> :END:
>
> ** TODO a
> SCHEDULED: <2017-09-29 Fri>
>
> * x
> :PROPERTIES:
> :CATEGORY: x
> :END:
>
> ** TODO b
> SCHEDULED: <2017-09-29 Fri>
>
> Running emacs -Q and "M-x org-agenda a" on master displays
>
> Week-agenda (W39):
> Monday 25 September 2017 W39
> Tuesday26 September 2017
> Wednesday  27 September 2017
> Thursday   28 September 2017
> Friday 29 September 2017
>   y:  Scheduled:  TODO a
>   x:  Scheduled:  TODO b
> Saturday   30 September 2017
> Sunday  1 October 2017
>
> Running the same sequence on wip-agenda-speedup displays
>
> Week-agenda (W39):
> Monday 25 September 2017 W39
> Tuesday26 September 2017
> Wednesday  27 September 2017
> Thursday   28 September 2017
> Friday 29 September 2017
>   x:  Scheduled:  TODO b
>   y:  Scheduled:  TODO a
> Saturday   30 September 2017
> Sunday  1 October 2017

Fixed. Thank you.

Regards,

-- 
Nicolas Goaziou

Re: [O] [ANN] Agenda speed up

[Samuel Wales]

... and i am getting a fixed error from before ...

... but 9maint and 9master produce agenda in about 3s ...

... while wip produces agenda in 26.75s ...

for my 2d agenda.  elp as instructed shows no significant differences

org-agenda-prepare-buffers1
1.221983848   1.221983848
org-agenda-files  3
0.000375030.00012501

also there are sorting differences in time grid for same minute and
for scheduled for same day.  which i don't care about.

also a clocked entry gets its closing time which is an improvement i think.

also an inactive timestamp shows the headline above the headline that
contains it.  this is a bug but cannot mce.

9master is same as 9maint.  thus, all issues are in wip.

i am limited in ability to use computer.

Re: [O] [ANN] Agenda speed up

[Matt Lundin]

Nicolas Goaziou  writes:

> Some feedback about the new agenda speed would be nice.

One small bug I found with wip speedup branch. When trying to reschedule
in the agenda with org-agenda-do-date-later or
org-agenda-do-date-earlier, org mode gives a message:

"Cannot find time stamp"

Best,
Mat

Re: [O] [ANN] Agenda speed up

[Matt Lundin]

Nicolas Goaziou  writes:

> Matt Lundin  writes:
>
>> I am finding that the branch is still much slower than the current
>> master even when no agenda files have changed (i.e., when running
>> org-agenda-redo in an existing agenda buffer without changing
>> anything).
>
> This is interesting. Could you provide a report of the second call to
> the same view? Or better, both reports for the first and subsequent
> view. The second view should definitely be faster.

I've attached the profiling for master and the wip branch. Both were run
with identical agenda files and identical configurations

The first pair of files profiles calling org-agenda-list immediately
after emacs has started up (i.e., before opening agenda files). The
total times are:

master: 5.014304355
wip:6.68215677

The second pair of files profiles calling org-agenda-redo immediately
after generating the agenda with org-agenda-list (i.e., without changing
anything in the agenda files). Note: repeating the redo after that
results in virtually identical times.

master: 0.979825959
wip:1.17870

The third pair of files profiles calling org-agenda-redo after
changing (rescheduling) one item in the agenda.

master: 0.979580617
wip:1.189759099

> I can look into your issue with proper reports as there may be ways to
> improve it, but I'm more interesting in, e.g., halving the 30 s from
> a month view than reducing the 1 s it takes to display a day view.

I agree it is important to reduce the time it takes to display a month
view. Thanks for all your work on this! I'd like to clarify that I am
not asking for further reductions from what the master branch of
org-mode has already achieved. Rather, I am concerned with regressions
from those times. The gap in time above, of course, is not particularly
big. But I'm running into some slower times (e.g., a 1 second
difference) on some of my other custom agenda commands. Take, for
instance, the following command:

--8<---cut here---start->8---
(org-add-agenda-custom-command
 '("q" "Projects"
   ((stuck "")
(agenda ""
((org-agenda-include-deadlines t)
 (org-agenda-entry-types '(:deadline))
 ;; (org-agenda-skip-function
 ;;  '(org-agenda-skip-entry-if 'notregexp ":PROJ:"))
 (org-agenda-include-diary nil)
 (org-agenda-time-grid nil)))
(todo "PROJECT"
  ((org-agenda-todo-ignore-deadlines t)
   (org-agenda-prefix-format " %i %-12:c%l"
   ((org-deadline-warning-days 365
--8<---cut here---end--->8---

The startup times weren't that far apart.

master: 5.728477902
wip:6.68639764

But running org-agenda-redo results in a 1s gap (see the last two
attached files for profiling information):

master:   2.14623279
wip:  3.125571084

Best,
Matt

org-agenda-list   1   
5.014304355   5.014304355
org-agenda-prepare-buffers2   
3.7477554480  1.8738777240
org-agenda-prepare1   
3.303024766   3.303024766
org-get-agenda-file-buffer252 
2.8621154540  0.0113576010
org-mode  63  
2.021190203   0.0320823841
org-agenda-get-day-entries126 
1.043993129   0.0082856597
org-set-startup-visibility63  
0.9549281500  0.0151575896
org-element-at-point  1863
0.8281850030  0.0004445437
org-cycle-hide-drawers82  
0.8039815459  0.009804653
org-element--parse-to 1863
0.7838064640  0.0004207227
org-diary-sexp-entry  30  
0.6159707919  0.0205323597
org-agenda-finalize   1   
0.614693850.61469385
org-agenda-get-sexps  126 
0.604186440.0047951304
org-element--current-element  3412
0.5537156270  0.0001622847
org-agenda-to-appt1   
0.513418731   0.513418731
org-set-regexps-and-options   189 
0.2761626890  0.0014611782
org-refresh-category-properties   126 
0.271197041   0.0021523574
org-install-agenda-files-menu 63  
0.2459687320  0.0039042655
org-refresh-effort-properties 189 
0.2442077130  0.0012921043
org--setup-collect-keywords   189 
0.2426440819  0.0012838311
org-refresh-properties

Re: [O] [ANN] Agenda speed up

[Nicolas Goaziou]

Hello,

Samuel Wales  writes:

> i am getting errors.  in particular, it crashes when it is looking for
> log entries in a task that does not have log entries.  there is no
> rhyme or reason and i have no mce.  i remove teh task and another task
> produces teh error.

What is a "crash"? If Emacs crashes, this should be reported to
Emacs-Devel. If there is an error, you should send the backtrace here.

In any case, I fixed an issue with log entries. It could be worth
testing it again.

Regards,

-- 
Nicolas Goaziou

Re: [O] [ANN] Agenda speed up

[Samuel Wales]

maybe it is trying to parse a link that looks like a log entry?

On 10/1/17, Samuel Wales  wrote:
> my use is similar to matt's in that i use a 2d view.
>
> i am getting errors.  in particular, it crashes when it is looking for
> log entries in a task that does not have log entries.  there is no
> rhyme or reason and i have no mce.  i remove teh task and another task
> produces teh error.
>
> not ready fro prime time quite yet.
>
> --
> The Kafka Pandemic: 
>
> The disease DOES progress. MANY people have died from it. And ANYBODY
> can get it at any time.
>
> "You’ve really gotta quit this and get moving, because this is murder
> by neglect." ---
> .
>


-- 
The Kafka Pandemic: 

The disease DOES progress. MANY people have died from it. And ANYBODY
can get it at any time.

"You’ve really gotta quit this and get moving, because this is murder
by neglect." ---
.

Re: [O] [ANN] Agenda speed up

[Samuel Wales]

my use is similar to matt's in that i use a 2d view.

i am getting errors.  in particular, it crashes when it is looking for
log entries in a task that does not have log entries.  there is no
rhyme or reason and i have no mce.  i remove teh task and another task
produces teh error.

not ready fro prime time quite yet.

-- 
The Kafka Pandemic: 

The disease DOES progress. MANY people have died from it. And ANYBODY
can get it at any time.

"You’ve really gotta quit this and get moving, because this is murder
by neglect." ---
.

Re: [O] [ANN] Agenda speed up

[Nicolas Goaziou]

Matt Lundin  writes:

> I am finding that the branch is still much slower than the current
> master even when no agenda files have changed (i.e., when running
> org-agenda-redo in an existing agenda buffer without changing
> anything).

This is interesting. Could you provide a report of the second call to
the same view? Or better, both reports for the first and subsequent
view. The second view should definitely be faster.

> This is true even when I set org-element-use-cache to t.

This variable has nothing to do with the current patch set. You can
ignore it.

>> OTOH, displaying, e.g., a whole week, month, year should be a lot
>> faster.
>
> Is this an unavoidable trade-off? Since I am constantly refreshing
> single day agenda buffers and todo lists, I would much prefer a faster
> single day display to a faster week or month display.

Good question. 

Currently, Agenda is pretty much tailored for single day display. So,
this would be difficult to improve. Using a cache, we can avoid finding
again the same data, so we can speed up subsequent parsing. Yet, when
there is little data to find, the cache is not very helpful. Luckily, it
means the Agenda is quickly generated in those cases anyway.

However, current Agenda is terrible when displaying multiple days. Some
people wait more than 30 s for a month view. This is where the current
patch set really shines.

I can look into your issue with proper reports as there may be ways to
improve it, but I'm more interesting in, e.g., halving the 30 s from
a month view than reducing the 1 s it takes to display a day view.

Regards,

Re: [O] [ANN] Agenda speed up

[Nicolas Goaziou]

Hello,

Matt Lundin  writes:

> I think I have a fairly standard setup (some customizations, additional
> features such as habits). I'll do some testing with minimal examples to
> see if I can find out why the new branch is so much slower in my case.

Are 1.4 s "so much slower" than of 1 s. Granted, this is a terrible 40%
slowdown, but, all things being equal, 1.4 s is still acceptable,
considering, IIUC, you are using the worst case scenario, i.e., a single
day view from a cold cache.

> In the meantime, I'd like to that the branch *not* be merged until we
> are sure that it is actually faster for the majority of use cases.

Feedback is welcome.

Regards,

-- 
Nicolas Goaziou

Re: [O] [ANN] Agenda speed up

[Matt Lundin]

Matt Lundin  writes:
>
> Here is a quick comparison of the top elp-results using a couple of commands:

I'm including the full elp results for reference. These were run with my
org agenda files and with customizations that I don't have time to
isolate at the moment. I'll try to provide a report with a minimal
config soon. The attached files contain profiling for the two commands
reported in my previous email.

(org-todo-list "TODO")

(org-agenda-list)

Best,
Matt

org-todo-list 1   
0.954759710.95475971
org-agenda-prepare1   
0.425165363   0.425165363
org-agenda-prepare-buffers1   
0.394949431   0.394949431
org-agenda-get-day-entries63  
0.2680520310  0.0042547941
org-agenda-get-todos  63  
0.262784373   0.0041711805
org-agenda-finalize-entries   1   
0.202006798   0.202006798
org-get-tags-at   454 
0.200070026   0.0004406828
org-agenda-highlight-todo 227 
0.1841641769  0.0008112959
org-back-to-heading   1207
0.1734143220  0.0001436738
org-refresh-category-properties   63  
0.1109007029  0.0017603286
org-refresh-properties126 
0.0968679490  0.0007687932
org-set-regexps-and-options   63  
0.0660081399  0.0010477482
org--setup-collect-keywords   63  
0.061246143   0.000972161
org-refresh-effort-properties 63  
0.055947762   0.0008880597
org-refresh-stats-properties  63  
0.049975411   0.0007932604
org-agenda-finalize   1   
0.048449583   0.048449583
org-element-at-point  126 
0.0278754009  0.0002212333
org-element--parse-to 126 
0.0237195620  0.0001882504
org-get-priority  227 
0.0201254490  8.865...e-05
org-agenda-files  3   
0.0190556840  0.0063518946
org-element--current-element  189 
0.0178631470  9.451...e-05
org-up-heading-safe   398 
0.016063253   4.035...e-05
org-agenda-format-item227 
0.014556003   6.412...e-05
org-entries-lessp 859 
0.0112956840  1.314...e-05
org-agenda-align-tags 1   
0.010916710.01091671
org-agenda-mode   1   
0.009271944   0.009271944
org-get-property-block74  
0.0081775990  0.0001105080
org-outline-level 799 
0.0073207199  9.162...e-06
org-get-todo-state238 
0.005509511   2.314...e-05
org-entry-get 36  
0.0054365299  0.0001510147
org-element-keyword-parser189 
0.0054290560  2.872...e-05
org-at-property-p 38  
0.005193692   0.0001366761
org--property-local-values36  
0.0049136239  0.0001364895
org-agenda-skip   237 
0.0048559309  2.048...e-05
org-add-props 832 
0.0047682290  5.731...e-06
org-agenda-check-for-timestamp-as-reason-to-ignore-todo-item  237 
0.004598511.940...e-05
org-check-agenda-file 126 
0.0042163329  3.346...e-05
org-inlinetask-in-task-p  74  
0.003895827   5.264...e-05
org-get-agenda-file-buffer126 
0.0033892649  2.689...e-05
org-split-string  686 
0.0033632589  4.902...e-06
org-get-limited-outline-regexp641 
0.0033366020  5.205...e-06
org-activate-links9   
0.003008839   0.0003343154
org-remove-uninherited-tags   260 
0.0029396989  1.130...e-05
org-find-base-buffer-visiting 126 
0.0028278890  2.244...e-05
org-refresh-property 

Re: [O] [ANN] Agenda speed up

[Matt Lundin]

Nicolas Goaziou  writes:

> Hello,
>
> Samuel Wales  writes:
>
>> have not beena ble to respons for health reasons.  i have rsposne
>> partly done.  i think result is slightly slower but seems tob e
>> correct if you count recent maint as correct.
>
> It can be slightly slower if you start with a cold cache and never
> re-use it, e.g., when you display only a single day and the most
> important agenda files were modified since last agenda display.

I am finding that the branch is still much slower than the current
master even when no agenda files have changed (i.e., when running
org-agenda-redo in an existing agenda buffer without changing anything).
This is true even when I set org-element-use-cache to t.

> OTOH, displaying, e.g., a whole week, month, year should be a lot
> faster.

Is this an unavoidable trade-off? Since I am constantly refreshing
single day agenda buffers and todo lists, I would much prefer a faster
single day display to a faster week or month display.

Matt

Re: [O] [ANN] Agenda speed up

[Matt Lundin]

Nicolas Goaziou  writes:

> If there is no more feedback nor objection, I'll merge the branch in
> master before the end of the week.
>
> Until then, the changes are still available in wip-agenda-speedup branch
> for review.

Thanks for the heads up. I just had a chance to test the
wip-agenda-speedup branch and find that it significantly slows down the
creation of agenda buffers with my agenda files and custom commands.

I think I have a fairly standard setup (some customizations, additional
features such as habits). I'll do some testing with minimal examples to
see if I can find out why the new branch is so much slower in my case.
In the meantime, I'd like to that the branch *not* be merged until we
are sure that it is actually faster for the majority of use cases.

Here is a quick comparison of the top elp-results using a couple of commands:

(org-todo-list "TODO")

master:
--8<---cut here---start->8---
org-todo-list 1   
0.954759710.95475971
org-agenda-prepare1   
0.425165363   0.425165363
org-agenda-prepare-buffers1   
0.394949431   0.394949431
org-agenda-get-day-entries63  
0.2680520310  0.0042547941
org-agenda-get-todos  63  
0.262784373   0.0041711805
org-agenda-finalize-entries   1   
0.202006798   0.202006798
org-get-tags-at   454 
0.200070026   0.0004406828
org-agenda-highlight-todo 227 
0.1841641769  0.0008112959
org-back-to-heading   1207
0.1734143220  0.0001436738
--8<---cut here---end--->8---

wip-agenda-speedup:
--8<---cut here---start->8---
org-todo-list 1   
1.402434591   1.402434591
org-agenda-day-entries63  
0.4656588689  0.0073914106
org-agenda--entry-from-todo   2217
0.4304873449  0.0001941756
org-agenda-prepare1   
0.387713298   0.387713298
org-agenda-prepare-buffers1   
0.378589420.37858942
org-agenda--file-data 63  
0.2997486200  0.0047579146
org-entry-get 1402
0.2108398869  0.0001503850
org-entry-properties  1366
0.1953800049  0.0001430307
org-agenda-finalize-entries   1   
0.191974038   0.191974038
org-agenda-check-for-timestamp-as-reason-to-ignore-todo-item  237 
0.1819146310  0.0007675722
org-agenda-highlight-todo 227 
0.1735402220  0.0007644943
org-agenda--todo-data 7   
0.1687917040  0.0241131005
org-back-to-heading   2336
0.1648271410  7.055...e-05
--8<---cut here---end--->8---

(org-agenda-list)

master:
--8<---cut here---start->8---
org-agenda-list   1   
1.036426005   1.036426005
org-agenda-prepare1   
0.596309830.59630983
org-agenda-prepare-buffers1   
0.584742966   0.584742966
org-agenda-get-day-entries63  
0.388804281   0.0061714965
org-agenda-get-scheduled  63  
0.287089758   0.0045569802
org-refresh-category-properties   63  
0.280568592   0.0044534697
org-habit-parse-todo  30  
0.178230735   0.0059410245
org-time-string-to-time   219 
0.162822094   0.0007434798
--8<---cut here---end--->8---

wip-agenda-speedup:
--8<---cut here---start->8---
org-agenda-list   1   
1.377235021.37723502
org-agenda-prepare1   
0.594557456   0.594557456
org-agenda-prepare-buffers1   
0.582119253   0.582119253
org-agenda--all-filtered-data 1   
0.307176728   0.307176728
org-agenda--file-data 63  
0.279614084   0.0044383187

Re: [O] [ANN] Agenda speed up

[Nicolas Goaziou]

Hello,

Samuel Wales  writes:

> have not beena ble to respons for health reasons.  i have rsposne
> partly done.  i think result is slightly slower but seems tob e
> correct if you count recent maint as correct.

It can be slightly slower if you start with a cold cache and never
re-use it, e.g., when you display only a single day and the most
important agenda files were modified since last agenda display.

OTOH, displaying, e.g., a whole week, month, year should be a lot
faster.

In any case, an ELP report is necessary to detect any abnormality.

Regards,

-- 
Nicolas Goaziou

Re: [O] [ANN] Agenda speed up

[Kyle Meyer]

Nicolas Goaziou  writes:

> If there is no more feedback nor objection, I'll merge the branch in
> master before the end of the week.

One issue I noticed: The sorting of categories defined in
org-agenda-sorting-strategy is not honored.  The default category
sorting for the agenda is to order the categories as they appear in the
agenda files.

Take the following text as the agenda file.

--8<---cut here---start->8---

* y
:PROPERTIES:
:CATEGORY: y
:END:

** TODO a
SCHEDULED: <2017-09-29 Fri>

* x
:PROPERTIES:
:CATEGORY: x
:END:

** TODO b
SCHEDULED: <2017-09-29 Fri>
--8<---cut here---end--->8---

Running emacs -Q and "M-x org-agenda a" on master displays

Week-agenda (W39):
Monday 25 September 2017 W39
Tuesday26 September 2017
Wednesday  27 September 2017
Thursday   28 September 2017
Friday 29 September 2017
  y:  Scheduled:  TODO a
  x:  Scheduled:  TODO b
Saturday   30 September 2017
Sunday  1 October 2017

Running the same sequence on wip-agenda-speedup displays

Week-agenda (W39):
Monday 25 September 2017 W39
Tuesday26 September 2017
Wednesday  27 September 2017
Thursday   28 September 2017
Friday 29 September 2017
  x:  Scheduled:  TODO b
  y:  Scheduled:  TODO a
Saturday   30 September 2017
Sunday  1 October 2017

-- 
Kyle

Re: [O] [ANN] Agenda speed up

[Samuel Wales]

have not beena ble to respons for health reasons.  i have rsposne
partly done.  i think result is slightly slower but seems tob e
correct if you count recent maint as correct.

Re: [O] [ANN] Agenda speed up

[Nicolas Goaziou]

Nicolas Goaziou  writes:

> I rewrote the part responsible for agenda generation in "org-agenda.el".
> Basically, I refactored some hot loops and introduced a cache mechanism
> for data extracted out of agenda files.
>
> I expect to see some interesting improvements when viewing the agenda
> with a span larger than one day, or when generating an agenda view
> without having touched most of the agenda files since last view.
>
> Some feedback about the new agenda speed would be nice.

If there is no more feedback nor objection, I'll merge the branch in
master before the end of the week.

Until then, the changes are still available in wip-agenda-speedup branch
for review.

Regards,

Re: [O] [ANN] Agenda speed up

[Nicolas Goaziou]

Hello,

Samuel Wales  writes:

> On 8/27/17, Nicolas Goaziou  wrote:
>> I expect to see some interesting improvements when viewing the agenda
>> with a span larger than one day, or when generating an agenda view
>> without having touched most of the agenda files since last view.
>
> wow, great!  i have long wanted this.
>
>> The only thing that is expected to be defective is
>> `org-agenda-include-inactive-timestamps'. It has no effect at the
>> moment. I can activate it again, but I'm wondering if it's worth the
>> overhead. It is already possible to display clocks and closed timestamps
>> in the agenda. Other inactive timestamps could be ignored from the
>> agenda altogether. This is their sole purpose, after all. WDYT?
>
> totally disagree.  i totally rely on showing inactive timestamps.
>
> in fact, i created a face for inactive timestamps.
>
> i also do this:
>
> (setq org-agenda-inactive-leader "Inactive:  ")
>
> to better match Closed: and Clocked:.
>
> i keep logs like this:
>
> * CONVERSATION [#C] [2017-01-28 Sat 20:16] org changed its
> coloring for nokori, so past due shows tomorrow
>
> which show up nicely.  in fact, sometimes i do:
>
> * DONEKEEP [2017-08-27 Sun 12:52] sent email
>
> instead of doneifying.  showing closed tasks and donekeep at the same
> time makes sense to me.
>
> all of this keeps a record and can be sorted nicely in the outline
> with visual binary search, much better than date trees for me.
>
> i quite often will run agenda agenda on a restricted file, so that i
> can get all active timestamps, and closed timestamps, and clocking
> timestamps, and inactive timestamps for inside the restriction.  this
> gets sorted correctly, by timestamp.
>
> so i can see the active, closed, and clocking timestamps in context
> with my notes.  that is totally key for me.  missing inactive
> timestamps would violate the idea that i can bounce around the agenda
> agenda at various dates to see what i did on those dates.  for
> example, i go back 4 days to see what i did 4 days ago, and i see a
> record of everything, including random insertions of inactive
> timestamps, donekeep, conversation, closed, and clocked.  this feels
> like an essential org feature to me.
>
> i run agenda search also, to find relevant timestamps.  inactive are
> much more common than active.
>
> being able to see my logs with inactive timestamps in context with
> active timestamps and closed timestamps and clocking is useful.

The inactive time stamps are available again in the "wip-agenda-speedup"
branch.

Make sure to delete any local branch and start anew if you tried it
already. Also make sure to clear the cache, e.g., restarting the Emacs
session.

>From now on, there will be no more rebasing on that branch, only regular
commits.

Feedback still very welcome.

Regards,

-- 
Nicolas Goaziou

Re: [O] [ANN] Agenda speed up

[Eric S Fraga]

On Wednesday, 30 Aug 2017 at 17:00, Nicolas Goaziou wrote:
> I think I spotted the problem. I updated the branch. Could you try
> again?

Done.  Attached is the resulting file from elp; much much better!  I
haven't rerun the default agenda as my agenda files may have changed but
only by 1-3 new entries.

Question as an aside: how do I best get the updates with git?  If I
checkout the branch, it tells me that I am 1 commit ahead and the remote
is 7 commits ahead (for instance).  I find I have to delete the branch
and then reacquire.  I'm not a big git user (mercurial is my VCS of
choice).

> Also, are there missing entries (besides inactive timestamps)? If so,
> would they be related to inlinetasks?

I haven't seen any missing entries since that first time.  I do not tend
to use inlinetasks in my agenda files, only within project specific
files which are not searched by the default agenda views.

-- 
: Eric S Fraga via Emacs 26.0.50, Org release_9.0.10-715-g8b5b2c


signature.asc
Description: PGP signature

Re: [O] [ANN] Agenda speed up

[Nicolas Goaziou]

Eric S Fraga  writes:

> Default and speedup results attached with org from a minute or so
> ago.  Still quite a difference between the two and not in the desired
> direction :-(  Hope these results help!

Thank you.

I think I spotted the problem. I updated the branch. Could you try
again?

Also, are there missing entries (besides inactive timestamps)? If so,
would they be related to inlinetasks?

Regards,

Re: [O] [ANN] Agenda speed up

[Eric S Fraga]

Forgot to add: I re-enabled my 14 <%%()> lines for completeness.
-- 
: Eric S Fraga via Emacs 26.0.50, Org release_9.0.10-715-g8b5b2c


signature.asc
Description: PGP signature

Re: [O] [ANN] Agenda speed up

[Eric S Fraga]

On Wednesday, 30 Aug 2017 at 11:00, Nicolas Goaziou wrote:
> or by trying in a fresh Emacs session. I didn't change the cache format
> so far, though.

All tests are with fresh emacs sessions: emacs, instrument package,
agenda, view month, next month, elp results, exit emacs.

Default and speedup results attached with org from a minute or so
ago.  Still quite a difference between the two and not in the desired
direction :-(  Hope these results help!

(back to first computer in case you were keeping track)

Thanks,
eric

-- 
: Eric S Fraga via Emacs 26.0.50, Org release_9.0.10-715-g8b5b2c
org-agenda-list   3   
19.615536284  6.5385120946
org-agenda-redo   2   
18.338604917  9.1693024585
org-let   2   
16.744135962  8.372067981
org-agenda-get-day-entries768 
15.307202554  0.0199312533
org-agenda-get-scheduled  768 
9.8357005239  0.0128069017
org-agenda-later  1   
9.378686659   9.378686659
org-agenda-view-mode-dispatch 1   
9.167906739.16790673
org-agenda-month-view 1   
8.960431599   8.960431599
org-agenda-change-time-span   1   
8.960423986   8.960423986
org-agenda-prepare-buffers5   
5.656869958   1.1313739915
org-agenda-prepare3   
4.184791168   1.3949303893
org-at-planning-p 39168   
3.8068944489  9.719...e-05
org-agenda1   
3.056179293.05617929
org-back-to-heading   86854   
2.3679976289  2.726...e-05
org-agenda--timestamp-to-absolute 56598   
2.3283312469  4.113...e-05
org-indent-initialize-agent   9   
2.0770357720  0.2307817524
org-indent-initialize-buffer  9   
2.076821355   0.2307579283
org-agenda-skip   43642   
2.0651276480  4.731...e-05
org-time-string-to-absolute   56598   
2.0122050389  3.555...e-05
org-agenda-get-blocks 768 
1.9067488329  0.0024827458
org-agenda-get-deadlines  768 
1.8708426799  0.0024359930
org-get-agenda-file-buffer828 
1.8227216149  0.0022013546
org-end-of-subtree5014
1.7007795029  0.0003392061
org-mode  10  
1.650076729   0.1650076729
org-agenda-to-appt2   
1.592130069   0.7960650345
org-set-startup-visibility10  
1.4465168379  0.1446516837
org-get-todo-state38704   
1.4276157679  3.688...e-05
org-parse-time-string 66105   
1.3716788229  2.075...e-05
org-inlinetask-in-task-p  37306   
1.2408245400  3.326...e-05
org-agenda-get-timestamps 768 
1.206963151   0.0015715666
org-cycle-hide-drawers14  
1.111790466   0.0794136047
org-at-item-p 15272   
0.9609090140  6.291...e-05
org-in-commented-heading-p4670
0.9489873300  0.0002032092
org-element-at-point  2565
0.8369609749  0.0003263005
org-element--parse-to 2565
0.7870030489  0.0003068238
org-indent-set-line-properties16922   
0.6453445200  3.813...e-05
org-closest-date  19872   
0.6436876209  3.239...e-05
org-list-in-valid-context-p   2529
0.542946589   0.0002146882
org-in-block-p2529
0.5356518930  0.0002118038
org-indent-add-properties 9   
0.521490343   0.0579433714
org-heading-components4670
0.5198980630  0.0001113272
org-outline-level 12501   
0.4742613580  3.793...e-05
org-element--current-element  4245
0.4740368930  0.0001116694

Re: [O] [ANN] Agenda speed up

[Nicolas Goaziou]

Hello,

Eric S Fraga  writes:

>> I updated the "wip-agenda-speedup" branch (rebasing needed). It should
>> now call `org-agenda-skip' less often. Could you try again using that?
>
> I am not sure what "rebasing needed" means

It means that "git pull" may not be sufficient, since I overwrite
history on this branch.

> and whether I need to do anything special.

At the very least, you need to make sure the cache is clean, with

  (clrhash org-agenda--data-cache)

or by trying in a fresh Emacs session. I didn't change the cache format
so far, though.

> Default version:
>
> | org-agenda-list   | 3 | 25.169072093 | 8.3896906976 |
> | org-agenda-redo   | 2 | 23.729199963 | 11.864599981 |
> | org-let   | 2 | 21.949337096 | 10.974668548 |
> | org-agenda-get-day-entries|   768 | 20.485554483 | 0.0266738990 |
> | org-agenda-get-scheduled  |   768 | 13.140805835 | 0.0171104242 |
> | org-agenda-later  | 1 | 12.218471156 | 12.218471156 |
> | org-agenda-view-mode-dispatch | 1 | 11.774425814 | 11.774425814 |
> | org-agenda-month-view | 1 | 11.511459292 | 11.511459292 |
> | org-agenda-change-time-span   | 1 | 11.511454414 | 11.511454414 |
> | org-agenda-prepare-buffers| 5 |  6.235086298 | 1.2470172596 |
> | org-at-planning-p | 38286 | 4.9494982779 | 0.0001292769 |
> | org-agenda-prepare| 3 | 4.6090207959 | 1.5363402653 |
> | org-agenda| 1 |  3.444506372 |  3.444506372 |
> | org-agenda-skip   | 42757 | 3.0357010199 | 7.099...e-05 |
> | org-agenda-get-deadlines  |   768 | 2.9264708689 | 0.0038105089 |
> | org-back-to-heading   | 85980 | 2.7460824440 | 3.193...e-05 |
> | org-agenda--timestamp-to-absolute | 57206 | 2.7053754779 | 4.729...e-05 |
> | org-get-todo-state| 37819 | 2.5136005120 | 6.646...e-05 |
> | org-time-string-to-absolute   | 57206 | 2.2163205409 | 3.874...e-05 |
> | org-agenda-get-blocks |   768 | 2.1708883639 | 0.0028266775 |
> | org-inlinetask-in-task-p  | 37309 | 2.0235439769 | 5.423...e-05 |
> | org-get-agenda-file-buffer|   828 | 1.9225380609 | 0.0023219058 |
> | org-agenda-to-appt| 2 |  1.776863076 |  0.888431538 |
>
> New wip-agenda-speedup version:
>
> | org-agenda-list   |  3 | 31.765126901 | 10.588375633 |
> | org-agenda-redo   |  2 | 30.228140779 | 15.114070389 |
> | org-let   |  2 | 27.439146923 | 13.719573461 |
> | org-agenda-day-entries|706 | 25.781168166 | 0.0365172353 |
> | org-agenda-later  |  1 | 15.415702875 | 15.415702875 |
> | org-agenda-view-mode-dispatch |  1 | 15.065425835 | 15.065425835 |
> | org-agenda-month-view |  1 | 14.817610403 | 14.817610403 |
> | org-agenda-change-time-span   |  1 | 14.817599293 | 14.817599293 |
> | org-get-todo-state| 150556 | 12.739302215 | 8.461...e-05 |
> | org-back-to-heading   | 162734 | 11.384205458 | 6.995...e-05 |
> | org-agenda-prepare-buffers|  5 |  6.309316915 | 1.2618633830 |
> | org-agenda|  1 | 4.5347066179 | 4.5347066179 |
> | org-agenda-prepare|  3 |  4.526386149 |  1.508795383 |
> | org-agenda--timestamp-to-absolute |  65195 | 3.6421088350 | 5.586...e-05 |
> | org-agenda-to-appt|  2 |   2.78557571 |  1.392787855 |
> | org-time-string-to-absolute   |  65195 | 2.7304234570 | 4.188...e-05 |
> | org-indent-initialize-agent   | 11 | 2.5848328029 | 0.2349848002 |
> | org-indent-initialize-buffer  | 11 |  2.584644208 | 0.2349676552 |
> | org-get-repeat| 113814 | 2.2632576760 | 1.988...e-05 |
> | org-agenda--all-filtered-data |  3 |  1.997229961 | 0.6657433203 |
> | org-get-agenda-file-buffer|862 | 1.9403902009 | 0.0022510327 |
> | org-end-of-subtree|   4920 | 1.7806221550 | 0.0003619150 |
> | org-mode  | 10 |  1.736419648 | 0.1736419648 |
> | org-agenda--file-data | 60 | 1.5899973689 | 0.0264999561 |

Still no luck. At least, it is obvious where the hanging fruits are.

Unfortunately, I'm not sure where those numbers of `org-get-todo-state'
and `org-back-to-heading' come from. For example, there are as many
`org-get-todo-state' calls from "org-agenda.el" in both "master" and
"wip-agenda-speedup" branches.

Information is missing in your report. For example, I don't know how
many times `org-agenda-skip' was called in the "wip-agenda-speedup"
version.

Could you try again with a fresh "wip-agenda-speedup" branch (I fixed
"<%%...>" timestamps) and post a full ELP report?

Thank you!


Regards,

-- 
Nicolas Goaziou

Re: [O] [ANN] Agenda speed up

[Robert Horn]

I'll note that I use %%( entries for sunset/sunrise, e.g.,
%%(diary-sunrise), and for some schedules that need logic that does "shift one
day if Monday is a holiday".

That means I will need the %%( functionality to keep working.  I suspect
I'm not alone.  These are some fairly old and stable functions that I'd
actually forgotten about until I tried org-super-agenda and noticed that
it broke them.  (Now fixed).

R Horn

Re: [O] [ANN] Agenda speed up

[Eric S Fraga]

On Monday, 28 Aug 2017 at 16:24, Nicolas Goaziou wrote:
> I updated the "wip-agenda-speedup" branch (rebasing needed). It should
> now call `org-agenda-skip' less often. Could you try again using that?

I am not sure what "rebasing needed" means and whether I need to do
anything special.  I did "git pull", "git checkout wip-agenda-speedup"
and "make".

I get the following results now:

Default version:

| org-agenda-list   | 3 | 25.169072093 | 8.3896906976 |
| org-agenda-redo   | 2 | 23.729199963 | 11.864599981 |
| org-let   | 2 | 21.949337096 | 10.974668548 |
| org-agenda-get-day-entries|   768 | 20.485554483 | 0.0266738990 |
| org-agenda-get-scheduled  |   768 | 13.140805835 | 0.0171104242 |
| org-agenda-later  | 1 | 12.218471156 | 12.218471156 |
| org-agenda-view-mode-dispatch | 1 | 11.774425814 | 11.774425814 |
| org-agenda-month-view | 1 | 11.511459292 | 11.511459292 |
| org-agenda-change-time-span   | 1 | 11.511454414 | 11.511454414 |
| org-agenda-prepare-buffers| 5 |  6.235086298 | 1.2470172596 |
| org-at-planning-p | 38286 | 4.9494982779 | 0.0001292769 |
| org-agenda-prepare| 3 | 4.6090207959 | 1.5363402653 |
| org-agenda| 1 |  3.444506372 |  3.444506372 |
| org-agenda-skip   | 42757 | 3.0357010199 | 7.099...e-05 |
| org-agenda-get-deadlines  |   768 | 2.9264708689 | 0.0038105089 |
| org-back-to-heading   | 85980 | 2.7460824440 | 3.193...e-05 |
| org-agenda--timestamp-to-absolute | 57206 | 2.7053754779 | 4.729...e-05 |
| org-get-todo-state| 37819 | 2.5136005120 | 6.646...e-05 |
| org-time-string-to-absolute   | 57206 | 2.2163205409 | 3.874...e-05 |
| org-agenda-get-blocks |   768 | 2.1708883639 | 0.0028266775 |
| org-inlinetask-in-task-p  | 37309 | 2.0235439769 | 5.423...e-05 |
| org-get-agenda-file-buffer|   828 | 1.9225380609 | 0.0023219058 |
| org-agenda-to-appt| 2 |  1.776863076 |  0.888431538 |

New wip-agenda-speedup version:

| org-agenda-list   |  3 | 31.765126901 | 10.588375633 |
| org-agenda-redo   |  2 | 30.228140779 | 15.114070389 |
| org-let   |  2 | 27.439146923 | 13.719573461 |
| org-agenda-day-entries|706 | 25.781168166 | 0.0365172353 |
| org-agenda-later  |  1 | 15.415702875 | 15.415702875 |
| org-agenda-view-mode-dispatch |  1 | 15.065425835 | 15.065425835 |
| org-agenda-month-view |  1 | 14.817610403 | 14.817610403 |
| org-agenda-change-time-span   |  1 | 14.817599293 | 14.817599293 |
| org-get-todo-state| 150556 | 12.739302215 | 8.461...e-05 |
| org-back-to-heading   | 162734 | 11.384205458 | 6.995...e-05 |
| org-agenda-prepare-buffers|  5 |  6.309316915 | 1.2618633830 |
| org-agenda|  1 | 4.5347066179 | 4.5347066179 |
| org-agenda-prepare|  3 |  4.526386149 |  1.508795383 |
| org-agenda--timestamp-to-absolute |  65195 | 3.6421088350 | 5.586...e-05 |
| org-agenda-to-appt|  2 |   2.78557571 |  1.392787855 |
| org-time-string-to-absolute   |  65195 | 2.7304234570 | 4.188...e-05 |
| org-indent-initialize-agent   | 11 | 2.5848328029 | 0.2349848002 |
| org-indent-initialize-buffer  | 11 |  2.584644208 | 0.2349676552 |
| org-get-repeat| 113814 | 2.2632576760 | 1.988...e-05 |
| org-agenda--all-filtered-data |  3 |  1.997229961 | 0.6657433203 |
| org-get-agenda-file-buffer|862 | 1.9403902009 | 0.0022510327 |
| org-end-of-subtree|   4920 | 1.7806221550 | 0.0003619150 |
| org-mode  | 10 |  1.736419648 | 0.1736419648 |
| org-agenda--file-data | 60 | 1.5899973689 | 0.0264999561 |

Please do not compare the times from these runs with those of
yesterday.  This is a different computer, a much older one but with Xeon
processors so difficult to compare...

(also ignore org version in this email signature as it comes from before
updating the repository)

I hope this helps!

Thanks,
eric

-- 
: Eric S Fraga via Emacs 26.0.50, Org release_9.0.9-573-g09e612


signature.asc
Description: PGP signature

Re: [O] [ANN] Agenda speed up

[Eric S Fraga]

On Monday, 28 Aug 2017 at 16:24, Nicolas Goaziou wrote:
> How great! I managed to achieve a negative speed up.

:-)

Happens to me more often than I wish to admit to...

>> I had to remove a bunch of <%% (...)> items in one of my agenda files as
>> these gave me error messages about the sexp.
>
> Could you show one of the culprits?

One example:

  <%%(and (= 5 (calendar-day-of-week date)) (diary-block 2009 11 16 2009 12 
18))>

I had a number of ancient lines related to my teaching schedule.  I now
use cloning of subtrees so I don't have any relevant lines like these
any more.

>> Finally, most importantly, the actual items shown in the day view with
>> the new code are only a subset of those shown with the old view.  I
>> cannot see any pattern in those omitted.
>
> There may be more than one pattern involved. Maybe the types involved
> would help.

>From memory, the skipped entries were SCHEDULED TODO items.  But so were
the ones shown as the majority of my agenda view is such items.  Not
sure if they had DEADLINEs as well or not.  Will check later today.

> I updated the "wip-agenda-speedup" branch (rebasing needed). It should
> now call `org-agenda-skip' less often. Could you try again using that?

Sure.  Hopefully later today.

thanks,
eric

-- 
: Eric S Fraga via Emacs 26.0.50, Org release_9.0.9-573-g09e612


signature.asc
Description: PGP signature

Re: [O] [ANN] Agenda speed up

[Nicolas Goaziou]

Hello,

Eric S Fraga  writes:

> I've compared timings old with new (from git a few minutes ago) by
> starting up emacs, instrumenting package org, viewing agenda (defaults
> to 1 day), switching to month view and then moving to the following
> month:
>
> Old agenda:
>
> | org-agenda-list   | 3 | 19.77036 | 6.5901203703 |

[...]

> New agenda:
>
> | org-agenda-list   |  3 | 29.473988928 |  9.824662976 |

How great! I managed to achieve a negative speed up.

> | org-agenda-skip   |  97251 | 8.0815792529 | 8.310...e-05 |

I overlooked `org-agenda-skip', which does nothing fancy on my side. It
is indeed called more often...

Skipping is actually harder with the cache, because you want to cache
everything anyway (or the cache cannot be trusted).

> I had to remove a bunch of <%% (...)> items in one of my agenda files as
> these gave me error messages about the sexp.

Could you show one of the culprits?

> Finally, most importantly, the actual items shown in the day view with
> the new code are only a subset of those shown with the old view.  I
> cannot see any pattern in those omitted.

There may be more than one pattern involved. Maybe the types involved
would help.

I updated the "wip-agenda-speedup" branch (rebasing needed). It should
now call `org-agenda-skip' less often. Could you try again using that?

Thank you for the feedback.

Regards,

-- 
Nicolas Goaziou

Re: [O] [ANN] Agenda speed up

[Colin Baxter]

> "Russell" == Russell Adams  writes:

Russell> On Sun, Aug 27, 2017 at 06:16:50PM +0200, Nicolas Goaziou
Russell> wrote:
>> Hello,
>> 
>> I rewrote the part responsible for agenda generation in
>> "org-agenda.el".  Basically, I refactored some hot loops and
>> introduced a cache mechanism for data extracted out of agenda
>> files.

Russell> Sounds awesome.

>> The only thing that is expected to be defective is
>> `org-agenda-include-inactive-timestamps'. It has no effect at the
>> moment. I can activate it again, but I'm wondering if it's worth
>> the overhead. It is already possible to display clocks and closed
>> timestamps in the agenda. Other inactive timestamps could be
>> ignored from the agenda altogether. This is their sole purpose,
>> after all. WDYT?

--- snip ---

Russell> Please consider preserving the existing inactive timestamps
Russell> functionality.

I too find inactive timestamps to be extremely useful, and would like
the ability to use them in their present form to be preserved. For me,
they are one of the (many :-) great things about org-mode.

Best wishes,

-- 
--
Colin Baxter
m43...@yandex.com
GnuPG fingerprint: 68A8 799C 0230 16E7 BF68  2A27 BBFA 2492 91F5 41C8

Re: [O] [ANN] Agenda speed up

[Eric S Fraga]

On Sunday, 27 Aug 2017 at 18:16, Nicolas Goaziou wrote:
> Hello,
>
> I rewrote the part responsible for agenda generation in "org-agenda.el".
> Basically, I refactored some hot loops and introduced a cache mechanism
> for data extracted out of agenda files.
>
> I expect to see some interesting improvements when viewing the agenda
> with a span larger than one day, or when generating an agenda view
> without having touched most of the agenda files since last view.
>
> Some feedback about the new agenda speed would be nice.

Dear Nicolas,

thank you for this.  Speeding up the agenda is something that appeals
given the complexity of my agenda and the often need to view weeks or
months.

I've compared timings old with new (from git a few minutes ago) by
starting up emacs, instrumenting package org, viewing agenda (defaults
to 1 day), switching to month view and then moving to the following
month:

Old agenda:

| org-agenda-list   | 3 | 19.77036 | 6.5901203703 |
| org-agenda-redo   | 2 | 18.346457274 | 9.1732286370 |
| org-let   | 2 | 16.772160558 |  8.386080279 |
| org-agenda-get-day-entries|   768 | 15.158740033 | 0.0197379427 |
| org-agenda-get-scheduled  |   768 | 9.5280991699 | 0.0124063791 |
| org-agenda-later  | 1 |  9.455895455 |  9.455895455 |
| org-agenda-view-mode-dispatch | 1 |  9.181260158 |  9.181260158 |
| org-agenda-month-view | 1 |  8.891024524 |  8.891024524 |
| org-agenda-change-time-span   | 1 |  8.891016702 |  8.891016702 |
| org-agenda-prepare-buffers| 5 | 5.8218686260 | 1.1643737252 |
| org-agenda-prepare| 3 | 4.4909116949 | 1.4969705649 |
| org-at-planning-p | 38979 | 3.8525446340 | 9.883...e-05 |
| org-agenda| 1 |  3.198932055 |  3.198932055 |

New agenda:

| org-agenda-list   |  3 | 29.473988928 |  9.824662976 |
| org-agenda-redo   |  2 | 28.332199373 | 14.166099686 |
| org-let   |  2 | 25.852282215 | 12.926141107 |
| org-agenda-day-entries|768 | 25.210189072 | 0.0328257670 |
| org-agenda-view-mode-dispatch |  1 | 14.441177311 | 14.441177311 |
| org-agenda-month-view |  1 | 14.281358385 | 14.281358385 |
| org-agenda-change-time-span   |  1 | 14.281353565 | 14.281353565 |
| org-agenda-later  |  1 | 14.051216182 | 14.051216182 |
| org-back-to-heading   | 110331 | 9.5532106970 | 8.658...e-05 |
| org-get-todo-state|  96677 | 9.5383469060 | 9.866...e-05 |
| org-agenda-skip   |  97251 | 8.0815792529 | 8.310...e-05 |
| org-agenda-prepare-buffers|  5 | 5.7887248999 |   1.15774498 |
| org-agenda-prepare|  3 |  4.180059331 | 1.3933531103 |
| org-agenda|  1 |  3.805856309 |  3.805856309 |
| org-agenda-to-appt|  2 |  2.477811826 |  1.238905913 |
| org-get-agenda-file-buffer|888 | 1.8626567679 | 0.0020975864 |

I had to remove a bunch of <%% (...)> items in one of my agenda files as
these gave me error messages about the sexp.

Finally, most importantly, the actual items shown in the day view with
the new code are only a subset of those shown with the old view.  I
cannot see any pattern in those omitted.

Thanks again,
eric

-- 
: Eric S Fraga via Emacs 26.0.50, Org release_9.0.9-796-gbae41a


signature.asc
Description: PGP signature

Re: [O] [ANN] Agenda speed up

[Russell Adams]

On Sun, Aug 27, 2017 at 06:16:50PM +0200, Nicolas Goaziou wrote:
> Hello,
>
> I rewrote the part responsible for agenda generation in "org-agenda.el".
> Basically, I refactored some hot loops and introduced a cache mechanism
> for data extracted out of agenda files.

Sounds awesome.

> The only thing that is expected to be defective is
> `org-agenda-include-inactive-timestamps'. It has no effect at the
> moment. I can activate it again, but I'm wondering if it's worth the
> overhead. It is already possible to display clocks and closed timestamps
> in the agenda. Other inactive timestamps could be ignored from the
> agenda altogether. This is their sole purpose, after all. WDYT?

Count me in as another major user of inactive timestamps. I use active 
timestamps only for appointments, and when I'm
working on a task I constantly add inactive timestamps via single keypress 
macro. When I review what has happened in the
agenda, I only toggle on the inactives when I need to see them.

Professionally they have been lifesavers. When did I execute that command on 
that host? When did the user call? What
hours do I need to bill? Etc.

Please consider preserving the existing inactive timestamps functionality.




--
Russell Adamsrlad...@adamsinfoserv.com

PGP Key ID: 0x1160DCB3   http://www.adamsinfoserv.com/

Fingerprint:1723 D8CA 4280 1EC9 557F  66E8 1154 E018 1160 DCB3

Re: [O] [ANN] Agenda speed up

[Samuel Wales]

On 8/27/17, Nicolas Goaziou  wrote:
> I expect to see some interesting improvements when viewing the agenda
> with a span larger than one day, or when generating an agenda view
> without having touched most of the agenda files since last view.

wow, great!  i have long wanted this.

> The only thing that is expected to be defective is
> `org-agenda-include-inactive-timestamps'. It has no effect at the
> moment. I can activate it again, but I'm wondering if it's worth the
> overhead. It is already possible to display clocks and closed timestamps
> in the agenda. Other inactive timestamps could be ignored from the
> agenda altogether. This is their sole purpose, after all. WDYT?

totally disagree.  i totally rely on showing inactive timestamps.

in fact, i created a face for inactive timestamps.

i also do this:

(setq org-agenda-inactive-leader "Inactive:  ")

to better match Closed: and Clocked:.

i keep logs like this:

* CONVERSATION [#C] [2017-01-28 Sat 20:16] org changed its
coloring for nokori, so past due shows tomorrow

which show up nicely.  in fact, sometimes i do:

* DONEKEEP [2017-08-27 Sun 12:52] sent email

instead of doneifying.  showing closed tasks and donekeep at the same
time makes sense to me.

all of this keeps a record and can be sorted nicely in the outline
with visual binary search, much better than date trees for me.

i quite often will run agenda agenda on a restricted file, so that i
can get all active timestamps, and closed timestamps, and clocking
timestamps, and inactive timestamps for inside the restriction.  this
gets sorted correctly, by timestamp.

so i can see the active, closed, and clocking timestamps in context
with my notes.  that is totally key for me.  missing inactive
timestamps would violate the idea that i can bounce around the agenda
agenda at various dates to see what i did on those dates.  for
example, i go back 4 days to see what i did 4 days ago, and i see a
record of everything, including random insertions of inactive
timestamps, donekeep, conversation, closed, and clocked.  this feels
like an essential org feature to me.

i run agenda search also, to find relevant timestamps.  inactive are
much more common than active.

being able to see my logs with inactive timestamps in context with
active timestamps and closed timestamps and clocking is useful.

-- 
The Kafka Pandemic: 

The disease DOES progress. MANY people have died from it. And ANYBODY
can get it at any time.

"You’ve really gotta quit this and get moving, because this is murder
by neglect." ---
.

Re: [O] [ANN] Agenda speed up

[Nicolas Goaziou]

Completing myself:

Caveat: there's an incompatible change in `org-agenda-log-mode-items'.
You need to apply a trivial change to the values:

  closed -> :closed
  clock  -> :clock
  state  -> :state