Bug: org-agenda search is slow on Windows 10 [9.4.4 (release_9.4.4 @ c:/Program Files/Emacs/x86_64/share/emacs/27.2/lisp/org/)]

[Robert Mastragostino]

Hello,

Org-agenda search is being unreasonably slow, taking several seconds to search 
through ~100 small files (a simple python program is practically 
instantaneous). I'm not familiar with proper emacs profiling but pausing in the 
middle of the search essentially always gives the stack trace below, where some 
recursion attempting to figure out file names is waiting on w32-long-file-name.

I've also attached my init.el, though as far as I can tell this isn't 
interference from another package.

Robert

Emacs  : GNU Emacs 27.2 (build 1, x86_64-w64-mingw32)
 of 2021-03-26
Package: Org mode version 9.4.4 (release_9.4.4 @ c:/Program 
Files/Emacs/x86_64/share/emacs/27.2/lisp/org/)

--
stack trace:

Debugger entered--Lisp error: (quit)
  w32-long-file-name("c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam")
  file-truename("c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam" (99) (nil))
  file-truename("c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2...")
  file-equal-p("c:/Users/rmast/AppData/Roaming/.emacs.d/orgfiles/t..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2...")
  #f(compiled-function (a b) #)("c:/Users/rmast/AppData/Roaming/.emacs.d/orgfiles/t..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2...")
  cl--position("c:/Users/rmast/AppData/Roaming/.emacs.d/orgfiles/t..." 
("c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." ...) 0 125)
  
cl--delete-duplicates(("c:/Users/rmast/AppData/Roaming/.emacs.d/orgfiles/B..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/orgfiles/n..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/orgfiles/t..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 
"c:/Users/rmast/AppData/Roaming/.emacs.d/org-roam/2..." 

Re: org-refile, org-capture byte-compiling failure with recent emacs-git

[Kyle Meyer]

Kyle Meyer writes:

> Kyle Meyer writes:
>
>> However, it seems unintended that with-demoted-errors would actually
>> error here now rather than give a warning; again, will need to take a
>> closer look.
>
> The patch at  demotes the error to the
> intended warning.

And Emacs's d52c929e3 has been ported to Org's bugfix branch to avoid
these failures regardless of whether the patch there is applied.

Re: org-refile, org-capture byte-compiling failure with recent emacs-git

[Kyle Meyer]

Kyle Meyer writes:

> However, it seems unintended that with-demoted-errors would actually
> error here now rather than give a warning; again, will need to take a
> closer look.

The patch at  demotes the error to the
intended warning.

Re: org-refile, org-capture byte-compiling failure with recent emacs-git

[Kyle Meyer]

Vladimir Lomov writes:

> Hello,
>
> org-refile and org-capture failed to byte-compile by emacs from git (commit:
> 8c8093aa6f20a7a4f3799cbdb9270cac013ef71d) with following messages:
>
> ...
>
> Compiling 
> /home/vladimir/pkgs/emacs-org-mode/src/emacs-org-mode/lisp/org-capture.el...
>
> In toplevel form:
> org-capture.el:1455:12: Error: Wrong number of arguments: (1 . 2), 4
>
> ...
>
> Compiling 
> /home/vladimir/pkgs/emacs-org-mode/src/emacs-org-mode/lisp/org-refile.el...
>
> In toplevel form:
> org-refile.el:566:17: Error: Wrong number of arguments: (1 . 2), 4

Thanks for reporting.

I'll need to look more closely, but this is very likely triggered by
d52c929e31f ((with-demoted-errors): Warn on missing `format` arg,
2022-02-04) on Emacs's side.  That commit is slated to be ported to the
Org repo but has been held for the time being with the rest of the
Org-touching commits that are in Emacs's master branch but not the
emacs-28 branch.

In d52c929e31f, the with-demoted-errors calls in org-refile and
org-capture were changed to use the format argument, which has been
available since Emacs 24.4.  So, the likely fix will just be porting
that commit.  However, it seems unintended that with-demoted-errors
would actually error here now rather than give a warning; again, will
need to take a closer look.

org-refile, org-capture byte-compiling failure with recent emacs-git

[Vladimir Lomov]

Hello,

org-refile and org-capture failed to byte-compile by emacs from git (commit:
8c8093aa6f20a7a4f3799cbdb9270cac013ef71d) with following messages:

...

Compiling 
/home/vladimir/pkgs/emacs-org-mode/src/emacs-org-mode/lisp/org-capture.el...

In toplevel form:
org-capture.el:1455:12: Error: Wrong number of arguments: (1 . 2), 4

...

Compiling 
/home/vladimir/pkgs/emacs-org-mode/src/emacs-org-mode/lisp/org-refile.el...

In toplevel form:
org-refile.el:566:17: Error: Wrong number of arguments: (1 . 2), 4

...

I also see this warning:

...

Compiling 
/home/vladimir/pkgs/emacs-org-mode/src/emacs-org-mode/lisp/org-agenda.el...
org-refile.el: Warning: Error: %S
org-refile.el: Warning: Error: %S

...

but not sure if it is related to above errors.

P.S. I tried the latest git commit (ef0cf495b2bc6a4eb5b3c21e26d9df8c7577).

---
WBR, Vladimir Lomov

-- 
I come from a small town whose population never changed.  Each time a woman
got pregnant, someone left town.
-- Michael Prichard


signature.asc
Description: PGP signature

Re: How do you manage complex project with Org-mode

[Eric Abrahamsen]

Sébastien Gendre  writes:

> Hello Eric,
>
> Thanks for your reply and advice.
>
> I will look at custom agenda view and clocking. But I have bad memory of
> clocking tools because of the way previous jobs used it (It was not
> Emacs).

It's a much more positive experience when you're using it to improve
your own quality of life, rather than someone else's business metrics!

Re: How do you manage complex project with Org-mode

[Tim Cross]

Sébastien Gendre  writes:

> Hello Tim,
>
> Thanks for your response and advice.
>
> I want to keep Org-mode as simple as possible. As you suggest.
>
> In the past, I ended up several times with a too complex Org-mode
> workflow and stop using it because of that. That because, today, I want
> to keep it simple. Usually, I apply a GTD workflow (or what I think it
> is, I'm not an expert).
>
> As you say, I need to learn skills for project management. But the
> project management methods we learned at school where to rigid. And, at
> work, the method is more "do the job, stop thinking, be professional".
> But it's, or was, the kind of job where you are asked to "not write test
> to save time". I generally have bad experiences at work.

Sadly, the software industry is full of some very poor middle managers
who don't really understand the complexities of the software
life-cycle. Too often, they focus on immediate deadlines and overlook
long-term maintenance. How you deal with such situations is down to
experience, confidence and where your own personal values lie. There has
been more than one job I've left because the way management was running
the project was poor and almost certainly going to lead to eventual
failure. There has been more than one job interview where I've stated
that if they are looking for someone to write code by the pound, I'm not
there man. I will ask leading questions in the interview to evaluate
what 'style' of development/management they use - for example, if they
measure productivity by the number of lines of code you right in a day,
I will thank them for their time and quietly walk away.

>
> To manage school big work, I think of managing them as projects.
>
> I want to apply a simple "Project" workflow:
>
> * Each project is a headline with the status "PROJECT"
> * Each project have the deadline defined by the school work deadline
> * Each project have a complete description with every info needed to work
> * Each project have one or many tasks (as sub headlines with a status)
> * Each task have a importance, time and effort estimation
> * Each task have its own deadline, distributed along the remaining time
> * When I set a task deadline, I look at its estimations and also other 
> projects tasks
> * To create a new project, I use Org-capture with a template
>
> Every time I create a new project, it start with one task: "Planning the
> project". With a deadline at 2 days max. The description of this task is
> a checkbox list of thing to do when planning the project.
>
> And finally, 2 times per week, I got a repetitive task: "Review the
> projects progress". With this, I should be able to adjust spending time
> and effort.
>

All seems like a reasonable starting point. The key is to regularly
review and refine your workflow. I would avoid the tendency to
think you have to put everything into your workflow. For example, I
would not have a task which says to review my tasks twice a week. Do you
really need a task to remind you to do this twice a week? Do you really
need to track that you have done this? I would classify such tasks as
'noise' tasks. They really don't perform any real purpose.  It is
similar to brain dead project policies which state things like "every
function must have a unit test". Many functions don't need a unit test
and forcing people to write pointless unit tests does more damage than
not having them. A unit test needs to be a positive addition and
deciding when it is or is not is part of what being a professional
developer is all about. Likewise with project management. You don't want
tasks for every simple thing you do. You want to record and track the
important tasks. No tool is a replacement for using your brain - these
are just tools and it is down to us to use them in an intelligent and
efficient manner.

Use org mode to make your life easier. If it isn't making it easier,
then you are doing something wrong and need to review and
re-prioritise.

Re: How do you manage complex project with Org-mode

[Sébastien Gendre]

Hello Eric,

Thanks for your reply and advice.

I will look at custom agenda view and clocking. But I have bad memory of
clocking tools because of the way previous jobs used it (It was not
Emacs).


Eric Abrahamsen  writes:

> Sébastien Gendre  writes:
>
>> Hello,
>>
>> I don't know if it's the correct place to ask it. If not, sorry to ask in
>> the wrong place.
>>
>> How do you manage complex project with Org-mode ?
>>
>> I used Org-mode for several periods of time in recent years. It worked
>> very well for short and day to day tasks. When only a few of theme have
>> deadlines and when you have plenty of time to do them.
>>
>> But, as a student, I regularly have big and important projects to do for
>> the school. The kind of project who need several days to be done, with
>> deadlines too soon, and if you fail one them the consequences can be
>> disastrous. And generally, I have to many of these project in the same
>> time and not enough time to do all the work. So, I also need to follow
>> the progress of each project to choose which is sufficiently advanced to
>> be stop for the benefit of another less advanced project.
>>
>> And I don't know how to manage this kind of projects with Org-mode. How
>> to do it, without failing a 6 days project because I spent to much time
>> on something else and I have only 3 days left with 3 half-day important
>> appointment I cannot cancel. I can't risk failing a single one of these
>> project by trying. So, when I am in a period with a lot of these
>> projects, I stop using Org-mode and concentrate on doing these project
>> as fast as I can. And because I often have this kind of project, I spend
>> most of the year without being able to use Org-mode.
>>
>> So, if you have any suggestion on how to manage, in Org-mode, projects
>> with:
>> * Lot of work to do (many days)
>> * Short deadline (not enough time)
>> * High importance (disastrous consequences in my future in case of fail)
>> * Many of them in the same time
>> * Progression need to be followed to chose where to sacrifice time to
>>   limit the damages
>
> Interesting questions! I have the same general problem of, when things
> heat up too much, I stop using the Org agenda. It's quite the opposite
> of how it's supposed to work, but I guess it's something about human
> psychology. I'll be interested to see what people say in this thread.
>
> Some suggestions that come to mind:
>
> - Create custom agenda views for each project, providing an overview of
>   that project only, but use a single unified agenda view for each day's
>   schedule. Look at the per-project agenda to decide if/how to complete
>   it, but draw the action items into the unified schedule when deciding
>   how to spend your day. It should become evident pretty quickly what
>   you actually have time for. Projects are many, but there is only one
>   of you.
> - Maybe consider using `org-trigger-hook' and `org-blocker-hook' to cut
>   down on TODO overwhelm.
> - Use time estimates and then TODO clocking to more swiftly disabuse
>   yourself of unrealistic expectations. This plus a schedule agenda can
>   also help you make sure you stop work at a reasonable time and go do
>   something else.
> - Say no to more work :) Looking at your solid-packed agenda for the
>   next day works wonders for saying no.
>
> Good luck!

Re: How do you manage complex project with Org-mode

[Sébastien Gendre]

Hello Quiliro,

Thank for you reply and advice. :)

Generally, I use the GTD workflow. Or what I think it is (I'm not an
expert). I started with 3 files:

* Tasks.org
* Notes.org
* Inbox.org

Now, the file "Notes.org" is a folder. And inside I got a file per
subject.

I capture notes and tasks, with org-capture, that end in the file "Inbox.org".

On my phone, I use Orgzly as a pocket inbox, for when I'm not in front of
my computer. The files of Orgzly are sync with my PC using Syncthing.

Everyday, at the end of the day, I review the inbox(s):
* I do little tasks (< 3 min)
* I complete the description of the other tasks and refile them (with
  org-refile)

Everyday, at the beginning of the day, I look at the agenda (Org-agenda)
to see what I need to do. If I have free time and still energy, I search
for tasks to do in advance.

This workflow work very well and ask few effort for managing it. But I
never take the risk to use it for important project. But maybe it's an
irrational fear. Maybe it would work.

Based on suggestions in this mailing list, I take time to think about
what define a work given by the school and what I need to do to
accomplish them best. And I think of a simple "Project" workflow that
can integrate with my actual workflow.

I think this simple "Project" workflow will be:

* Each project is a headline with the status "PROJECT"
* Each project have the deadline defined by the school work deadline
* Each project have a complete description with every info needed to work
* Each project have one or many tasks (as sub headlines with a TODO/NEXT/DONE 
status)
* Each task have an importance, time and effort estimation
* Each task have its own deadline, distributed along the remaining time
* When I set a task deadline, I look at its estimations and also other projects 
tasks
* To create a new project, I use Org-capture with a template

Every time I create a new project, it start with one task: "Planning the
project". With a deadline at 2 days max. The description of this task is
a checkbox list of thing to do when planning the project.

And finally, 2 times per week, I got a repetitive task: "Review the
projects progress". With this, I should be able to adjust spending time
and effort.

I think it would be simple and need only a few Org-mode configurations.
And by doing like that, every time I look at the agenda, I will see what
work I need to do every days. Task of each project or review of all
projects.




Quiliro Ordóñez  writes:

> Hello Seb.
>
> It is great that you want to use org for your planning.  It is the best
> tool I know of.  Tim's advice sounds the best introductory one and
> others even gave you advanced advice.  Nevertheless, you never told us
> how you use org and why it takes so much time for you.  Perhaps you
> could take less time to plan and more time to do, without completely
> trashing the planning.  Then you could increment the planning until it
> proves to be more time consuming.  At that point you can reduce a little
> planning again.  It is a tuning process.
>
> Happy Hacking!
>
> Quiliro

Re: How do you manage complex project with Org-mode

[Sébastien Gendre]

Hello Antonio,

Thanks for your reply and advice. :)

I looked at TaskJuggler and I see how it could help. But, by looking at the
screenshot, I see too much information in the same time. And, as someone
with difficulty to concentrating, I need more minimalist tools.

But I will try it one time.


Antonio Carlos Padoan Junior  writes:

> Hello,
>
> I'm not sure if I will help you with my advice, but I really appreciate
> to use taskjuggler mixed with org-mode. There is somewhere in the web an
> org-exporter to taskjuggler. I think recently a topic on this matter
> appeared in this list.
>
> Taskjuggler can help you optimizing complex plannings and can be used in a
> professional context.
>
> For sure you will need to invest some time on it but it is a good
> middle-term investment.
>
> Good luck,

Re: How do you manage complex project with Org-mode

[Sébastien Gendre]

Hello Milan,

Thank you for your reply and advice. :)

Indeed, using deadline with longer in-advance warnings is a good idea. I
completely forget this feature. Using priority and split big task
is also a good advice. Today I pay attention to keep Org-mode as simple as
possible. It was not the case in the past and few times I stopped using
Org-mode because of that.



Milan Zamazal  writes:

>> "SG" == Sébastien Gendre  writes:
>
> SG> But, as a student, I regularly have big and important projects
> SG> to do for the school. The kind of project who need several days
> SG> to be done, with deadlines too soon, and if you fail one them
> SG> the consequences can be disastrous. And generally, I have to
> SG> many of these project in the same time and not enough time to do
> SG> all the work. So, I also need to follow the progress of each
> SG> project to choose which is sufficiently advanced to be stop for
> SG> the benefit of another less advanced project.
>
> SG> And I don't know how to manage this kind of projects with
> SG> Org-mode. How to do it, without failing a 6 days project because
> SG> I spent to much time on something else and I have only 3 days
> SG> left with 3 half-day important appointment I cannot cancel. I
> SG> can't risk failing a single one of these project by trying. So,
> SG> when I am in a period with a lot of these projects, I stop using
> SG> Org-mode and concentrate on doing these project as fast as I
> SG> can. And because I often have this kind of project, I spend most
> SG> of the year without being able to use Org-mode.
>
> Hi, I’d join the suggestion to keep things simple in the beginning.  My
> task flow is different from yours but in order not to miss really
> important things, I use the following:
>
> - Deadlines, with longer in-advance warnings when needed (e.g. “-3w” in
>   DEADLINE).
>
> - I use priority A for and only for stuff that is on risk of really bad
>   consequences if not handled ASAP.  And I schedule such stuff to a
>   future date if it doesn’t make sense to work on it now for any reason.
>
> As for progress, I’d say that if you don’t know how far are you with
> your short-term tasks and which of them require attention currently then
> you might have a problem with your workflow.  Maybe you are too
> overloaded or you don’t split your time among the tasks appropriately.
> Org mode is a good tool to implement support for different workflows but
> cannot help if a used workflow doesn’t work very well for you.  Again,
> starting simple with Org mode and paying attention first to how you work
> and how it could be improved generally might be a good idea (and a
> life-long process for many of us).
>
> Regards,
> Milan

Re: How do you manage complex project with Org-mode

[Sébastien Gendre]

Hello Tim,

Thanks for your response and advice.

I want to keep Org-mode as simple as possible. As you suggest.

In the past, I ended up several times with a too complex Org-mode
workflow and stop using it because of that. That because, today, I want
to keep it simple. Usually, I apply a GTD workflow (or what I think it
is, I'm not an expert).

As you say, I need to learn skills for project management. But the
project management methods we learned at school where to rigid. And, at
work, the method is more "do the job, stop thinking, be professional".
But it's, or was, the kind of job where you are asked to "not write test
to save time". I generally have bad experiences at work.

To manage school big work, I think of managing them as projects.

I want to apply a simple "Project" workflow:

* Each project is a headline with the status "PROJECT"
* Each project have the deadline defined by the school work deadline
* Each project have a complete description with every info needed to work
* Each project have one or many tasks (as sub headlines with a status)
* Each task have a importance, time and effort estimation
* Each task have its own deadline, distributed along the remaining time
* When I set a task deadline, I look at its estimations and also other projects 
tasks
* To create a new project, I use Org-capture with a template

Every time I create a new project, it start with one task: "Planning the
project". With a deadline at 2 days max. The description of this task is
a checkbox list of thing to do when planning the project.

And finally, 2 times per week, I got a repetitive task: "Review the
projects progress". With this, I should be able to adjust spending time
and effort.

I think it would be simple and need only a few Org-mode configurations.
And by doing like that, every time I look at the agenda, I will see what
work I need to do every days. Task of each project or review of all
projects.






Tim Cross  writes:

> Sébastien Gendre  writes:
>
>> Hello,
>>
>> I don't know if it's the correct place to ask it. If not, sorry to ask in
>> the wrong place.
>>
>
> I think it is the correct place. This is a list for general org mode
> discussions. Such discussions can be technical or about how to use org mode.
>
>> How do you manage complex project with Org-mode ?
>>
>> I used Org-mode for several periods of time in recent years. It worked
>> very well for short and day to day tasks. When only a few of theme have
>> deadlines and when you have plenty of time to do them.
>>
>> But, as a student, I regularly have big and important projects to do for
>> the school. The kind of project who need several days to be done, with
>> deadlines too soon, and if you fail one them the consequences can be
>> disastrous. And generally, I have to many of these project in the same
>> time and not enough time to do all the work. So, I also need to follow
>> the progress of each project to choose which is sufficiently advanced to
>> be stop for the benefit of another less advanced project.
>>
>> And I don't know how to manage this kind of projects with Org-mode. How
>> to do it, without failing a 6 days project because I spent to much time
>> on something else and I have only 3 days left with 3 half-day important
>> appointment I cannot cancel. I can't risk failing a single one of these
>> project by trying. So, when I am in a period with a lot of these
>> projects, I stop using Org-mode and concentrate on doing these project
>> as fast as I can. And because I often have this kind of project, I spend
>> most of the year without being able to use Org-mode.
>>
>
> The first thing I would say is that org mode is NOT going to solve your
> problems of too many tasks with competing priorities in too short a time
> period or mitigate the impact from missed deadlines etc. In fact, no
> project management software can do this.
>
> Org mode is merely a tool which can help you manage tasks, deadlines,
> schedules, priorities and project information. It can help you track
> your tasks so fewer are overlooked/forgotten, it can help you manage
> your deadlines and scheduling of time and it can help you manage
> priorities and provide you with an overview of things that can alert you
> to issues earlier, allowing for a wider range of mitigation
> strategies. However, it is just a tool and how well you use that tool
> will come down to experience and self discipline.
>
>
>> So, if you have any suggestion on how to manage, in Org-mode, projects
>> with:
>> * Lot of work to do (many days)
>> * Short deadline (not enough time)
>> * High importance (disastrous consequences in my future in case of fail)
>> * Many of them in the same time
>> * Progression need to be followed to chose where to sacrifice time to
>>   limit the damages
>>
>
> That list is almost an exact match for the job criteria on the last job
> I applied for, which highlights the point these are individual skills
> which need to be learned, primarily through experience and not 

Re: How do you manage complex project with Org-mode

[Sébastien Gendre]

Hello Arne,

Thank you for your reply and advice. :)

"Dr. Arne Babenhauserheide"  writes:
> Do I understand it right that what you need is to track the time
> required to reach milestones, not following the *progress*?

Well, what I want by "follow the progress" is to see:

* How many work I have done
* How many work remains to be done
* How many time left
* How many time other project need

I want to avoid spending to much time on a project sufficiently finished
to the detriment of another project. Or, if no project can be finished,
do enough of work to have a not so bad grade.

As you suggest, having milestone with their own deadline seems to be a
great idea. A better idea that having one big task for representing an
entire project (like I wanted to do). I will try using effort
estimation. But I will avoid customizing too much Org-mode. In the past,
I ended up several times with a too much complex workflow.

I think I will make a "Project" workflow. Something simple:

* Each project is a headline with the status "PROJECT"
* Each project have the deadline defined by the school work deadline
* Each project have a complete description with every info needed to work
* Each project have one or many tasks (as sub headlines with a status)
* Each task have a importance, time and effort estimation
* Each task have its own deadline, distributed along the remaining time
* When I set a task deadline, I look at its estimations and also other projects 
tasks
* To create a new project, I use Org-capture with a template

Every time I create a new project, it start with one task: "Planning the
project". With a deadline at 2 days max. The description of this task is
a checkbox list of thing to do when planning the project.

And finally, 2 times per week, I got a repetitive task: "Review the
projects progress". With this, I should be able to adjust spending time
and effort.

I think it would be simple and need only a few Org-mode configurations.
And by doing like that, every time I look at the agenda, I will see what
work I need to do every days. Task of each project or review of all
projects.



"Dr. Arne Babenhauserheide"  writes:

> [[PGP Signed Part:Undecided]]
> Hello Seb,
>
> It sounds like org-mode can be a great fit.
>
> Sébastien Gendre  writes:
>> But, as a student, I regularly have big and important projects to do for
>> the school. The kind of project who need several days to be done, with
>> deadlines too soon, and if you fail one them the consequences can be
>> disastrous. And generally, I have to many of these project in the same
>> time and not enough time to do all the work. So, I also need to follow
>> the progress of each project to choose which is sufficiently advanced to
>> be stop for the benefit of another less advanced project.
>
> Do I understand it right that what you need is to track the time
> required to reach milestones, not following the *progress*? So following
> progress seems like it would take more mental bandwidth than needed.
>
> You could have one org-mode task (headline) per milestone, with a
> DEADLINE (org-deadline), and then SCHEDULE (org-schedule) appointments.
>
> To track how much time you will still need, you can use org-set-effort.
> I did that at work for a while to train to get better at estimating. By
> having the effort in a clocktable I could see progress *when needed*.
> Hitting R in the org-agenda shows the clockreport-mode and you can see
> the Effort in the agenda by setting
> (org-agenda-clockreport-parameter-plist (quote (:link t :maxlevel 2
> :properties ("Effort". For a while I had the clocktable active by
> default.
>
> You can also add that to the column-mode (org-columns) to get a quick
> overview for a file (leave with org-columns-quit). Customize:
> (org-columns-default-format
>"%25ITEM %TODO %3PRIORITY %TAGS %17Effort(Estimated Effort){:} %CLOCKSUM")
>
>> And I don't know how to manage this kind of projects with Org-mode. How
>> to do it, without failing a 6 days project because I spent to much time
>> on something else and I have only 3 days left with 3 half-day important
>> appointment I cannot cancel. I can't risk failing a single one of these
>> project by trying. So, when I am in a period with a lot of these
>> projects, I stop using Org-mode and concentrate on doing these project
>> as fast as I can. And because I often have this kind of project, I spend
>> most of the year without being able to use Org-mode.
>
> I found that org-mode is the only organization tool for which using the
> tool actually reduces the time I need for organization. That works by
> taking notes in org-mode, too, and keeping it simple.
>
> What I do:
>
> ** Custom starting point: agenda-with-kanban
>   A function to show the agenda it besides the Kanban table. I start
>   each day and after each larger break by hitting F12. It shows me the
>   agenda and entry points into my work. This enables me to stay focussed.
>
> ** One planning file
>   I have a single file for 

Re: How do you manage complex project with Org-mode

[Sébastien Gendre]

Hello Matt,

Thanks for your reply and advice. :)

I already use Org-mode since a few years and it work great with little
tasks. Tasks that take a few hours maximum. My workflow is GTD, or
something very close (I'm not an expert of this subject).

My concern was more for school works that need many days. I don't want
to see, too late, on my agenda, a work to do. Like seeing 3 days before
the deadline that I need to do a 5 days work. It never append because,
when I got a lot to do for school, I stop using Org-mode to don't take
the risk. So, in reality, I don't know if it will append. Maybe it's an
irrational fear. But, because I often got a lot of work to do for school,
I spend a lot of time not using Org-mode.


Matt  writes:

>   On Mon, 28 Feb 2022 20:43:47 -0500 Sébastien Gendre  wrote 
> 
>
>  > And I don't know how to manage this kind of projects with Org-mode. How
>  > to do it, without failing a 6 days project because I spent to much time
>  > on something else and I have only 3 days left with 3 half-day important
>  > appointment I cannot cancel. I can't risk failing a single one of these
>  > project by trying. So, when I am in a period with a lot of these
>  > projects, I stop using Org-mode and concentrate on doing these project
>  > as fast as I can. And because I often have this kind of project, I spend
>  > most of the year without being able to use Org-mode.
>
> It sounds like you have a lot going on! If none of what you need to do
> explicitly requires Org, you may have to scale back what you learn
> about Org to fit the time you've got. Give yourself permission to
> accept that the time you have right now for Org isn't what you want.
> (I hope that's because you're learning lots of other cool things in
> school.) In situations like these, I like to do just a little each
> day. Maybe that means reading one paragraph a night before bed. It
> sounds like you're really excited about Org. (If you are, you've come
> to the right place. The people here love Org :) Reading about Org
> would be something fun to look forward to each night. You might be
> surprised at how motivating that one little paragraph can be! Working
> through the manual in this way will give you a good overview of how
> Org can be used and what you personally might use Org it for.
>
>  > So, if you have any suggestion on how to manage, in Org-mode, projects
>  > with:
>  > * Lot of work to do (many days)
>  > * Short deadline (not enough time)
>  > * High importance (disastrous consequences in my future in case of fail)
>  > * Many of them in the same time
>  > * Progression need to be followed to chose where to sacrifice time to
>  >   limit the damages
>  > 
>  > I will be happy to read them. :)
>
> My suggestion is to not try something new on anything that has a tight
> deadline. Org is new to you and learning things takes time. This is
> all normal. Life will throw a lot at you. Some times all I get is 20
> minutes at the end of the day. Some days, I get nothing at all. This
> is because I've filled my life with other cool things, like a partner,
> a house, friends, etc. If I get to spend some time doing something I
> think is worthwhile (like trying to help a fellow Org enthusiast),
> that's time well spent in my book.
>
> Isn't there some saying that goes like, "every avalance starts as a 
> snowflake?" Read a little, experiment a little, and over time, you'll be 
> surprised at how much you've learned. You'll get there!  

Re: [Script] Firefox Bookmarks To Org Buffer Importer

[Samuel Banya]

Sweet idea, you should make this into a MELPA project.

On Wed, Mar 2, 2022, at 5:50 AM, Sai Karthik wrote:
> Hello group! I have started exploring org mode since a while &  I just 
> want to share this small tool i made in org mode for my use case. It 
> parses the firefox bookmarks json file to urls in the org buffer with 
> their respective sections.
> 
> May be this could be useful for anyone of you.
> 
> Source: https://gitlab.com/kskarthik/fox2org/-/blob/main/fox2org.org
> 
> Regards,
> 
> Karthik
> https://kskarthik.gitlab.io/
> 
> 

Re: org-modern

[autofrettage]

Call me old-fashioned if you like, but when I see efforts like org-modern, I 
cannot help but thinking about Neo in Matrix. It was his ability to see the 
source code, which gave him his god-like powers.

I therefore draw the line at (conventional) syntax highlighting.

(I am painfully aware my standpoint could have its roots in the following 
phenomenon:
You tend to take new inventions for granted, if they appeared when you were 15 
years or younger. Inventions appearing between the age of 15 and 35, are not 
only accepted, but often hailed as "break-throughs", "game changing", and the 
like. Once you are older than 35, most new things tend to be dangerous, or even 
evil.)

Cheers
Rasmus

Re: How do you manage complex project with Org-mode

[Quiliro Ordóñez]

Hello Seb.

It is great that you want to use org for your planning.  It is the best
tool I know of.  Tim's advice sounds the best introductory one and
others even gave you advanced advice.  Nevertheless, you never told us
how you use org and why it takes so much time for you.  Perhaps you
could take less time to plan and more time to do, without completely
trashing the planning.  Then you could increment the planning until it
proves to be more time consuming.  At that point you can reduce a little
planning again.  It is a tuning process.

Happy Hacking!

Quiliro

Re: profiling latency in large org-mode buffers (under both main & org-fold feature)

[Ihor Radchenko]

Max Nikulin  writes:

> On 27/02/2022 13:43, Ihor Radchenko wrote:
>> 
>> Now, I did an extended profiling of what is happening using perf:
>> 
>>   6.20%   [.] buf_bytepos_to_charpos
>
> Maybe I am interpreting such results wrongly, but it does not look like 
> a bottleneck. Anyway thank you very much for such efforts, however it is 
> unlikely that I will join to profiling in near future.

The perf data I provided is a bit tricky. I recorded statistics over the
whole Emacs session + used fairly small number of iterations in your
benchmark code.

Now, I repeated the testing plugging perf to Emacs only during the
benchmark execution:

With refile cache and markers:
22.82%  emacs-29.0.50.1  emacs-29.0.50.1   [.] 
buf_bytepos_to_charpos
16.68%  emacs-29.0.50.1  emacs-29.0.50.1   [.] 
rpl_re_search_2
 8.02%  emacs-29.0.50.1  emacs-29.0.50.1   [.] 
re_match_2_internal
 6.93%  emacs-29.0.50.1  emacs-29.0.50.1   [.] Fmemq
 4.05%  emacs-29.0.50.1  emacs-29.0.50.1   [.] 
allocate_vectorlike
 1.88%  emacs-29.0.50.1  emacs-29.0.50.1   [.] 
mark_object

Without refile cache:
17.25%  emacs-29.0.50.1  emacs-29.0.50.1 [.] 
rpl_re_search_2
15.84%  emacs-29.0.50.1  emacs-29.0.50.1 [.] 
buf_bytepos_to_charpos
 8.89%  emacs-29.0.50.1  emacs-29.0.50.1 [.] 
re_match_2_internal
 8.00%  emacs-29.0.50.1  emacs-29.0.50.1 [.] Fmemq
 4.35%  emacs-29.0.50.1  emacs-29.0.50.1 [.] 
allocate_vectorlike
 2.01%  emacs-29.0.50.1  emacs-29.0.50.1 [.] 
mark_object

Percents should be adjusted for larger execution time in the first
dataset, but otherwise it is clear that buf_bytepos_to_charpos dominates
the time delta.

>> I am not sure if I understand the code correctly, but that loop is
>> clearly scaling performance with the number of markers
>
> I may be terribly wrong, but it looks like an optimization attempt that 
> may actually ruin performance. My guess is the following. Due to 
> multibyte characters position in buffer counted in characters may 
> significantly differ from index in byte sequence. Since markers have 
> both values bytepos and charpos, they are used (when available) to 
> narrow down initial estimation interval [0, buffer size) to nearest 
> existing markers. The code below even creates temporary markers to make 
> next call of the function faster.

I tend to agree after reading the code again.
I tried to play around with that marker loop. It seems that the loop
should not be mindlessly disabled, but it can be sufficient to check
only a small number of markers in front of the marker list. The cached
temporary markers are always added in front of the list.

Limiting the number of checked markers to 10, I got the following
result:

With threshold and refile cache:
| 9.5.2  ||   ||
| nm-tst |   28.060029337 | 4 | 1.842760862996 |
| org-refile-get-targets | 3.244561543997 | 0 |0.0 |
| nm-tst | 33.64825913704 | 4 | 1.230431054003 |
| org-refile-cache-clear |0.034879062 | 0 |0.0 |
| nm-tst |   23.974124596 | 5 | 1.429148814996 |

Markers add +~5.6sec.

Original Emacs code and refile cache:
| 9.5.2  |  |   ||
| nm-tst | 29.494383528 | 4 | 3.036850853002 |
| org-refile-get-targets |  3.635947646 | 1 | 0.454247973002 |
| nm-tst | 36.537926593 | 4 | 1.129757634998 |
| org-refile-cache-clear | 0.0096653649 | 0 |0.0 |
| nm-tst | 23.283457105 | 4 | 1.053649649997 |

Markers add +7sec.

The improvement is there, though markers still somehow come into play. I
speculate that limiting the number of checked markers might also force
adding extra temporary markers to the list, but I haven't looked into
that possibility for now. It might be better to discuss with emacs-devel
before trying too hard.

>> Finally, FYI. I plan to work on an alternative mechanism to access Org
>> headings - generic Org query library. It will not use markers and
>> implement ideas from org-ql. org-refile will eventually use that generic
>> library instead of current mechanism.
>
> I suppose that markers might be implemented in an efficient way, and 
> much better performance may be achieved when low-level data structures 
> are accessible. I am in doubts concerning attempts to create something 
> that resembles markers but based purely on high-level API.

I am currently using a custom version of org-ql utilising the new
element cache. It is substantially faster compared to current
org-refile-get-targets. The 

[Script] Firefox Bookmarks To Org Buffer Importer

[Sai Karthik]

Hello group! I have started exploring org mode since a while &  I just 
want to share this small tool i made in org mode for my use case. It 
parses the firefox bookmarks json file to urls in the org buffer with 
their respective sections.


May be this could be useful for anyone of you.

Source: https://gitlab.com/kskarthik/fox2org/-/blob/main/fox2org.org

Regards,

Karthik
https://kskarthik.gitlab.io/

Re: profiling latency in large org-mode buffers (under both main & org-fold feature)

[Max Nikulin]

On 27/02/2022 13:43, Ihor Radchenko wrote:


Now, I did an extended profiling of what is happening using perf:

  6.20%   [.] buf_bytepos_to_charpos


Maybe I am interpreting such results wrongly, but it does not look like 
a bottleneck. Anyway thank you very much for such efforts, however it is 
unlikely that I will join to profiling in near future.



buf_bytepos_to_charpos contains the following loop:

   for (tail = BUF_MARKERS (b); tail; tail = tail->next)
 {
   CONSIDER (tail->bytepos, tail->charpos);

   /* If we are down to a range of 50 chars,
 don't bother checking any other markers;
 scan the intervening chars directly now.  */
   if (best_above - bytepos < distance
   || bytepos - best_below < distance)
break;
   else
 distance += BYTECHAR_DISTANCE_INCREMENT;
 }

I am not sure if I understand the code correctly, but that loop is
clearly scaling performance with the number of markers


I may be terribly wrong, but it looks like an optimization attempt that 
may actually ruin performance. My guess is the following. Due to 
multibyte characters position in buffer counted in characters may 
significantly differ from index in byte sequence. Since markers have 
both values bytepos and charpos, they are used (when available) to 
narrow down initial estimation interval [0, buffer size) to nearest 
existing markers. The code below even creates temporary markers to make 
next call of the function faster.


It seems, buffers do not have any additional structures that track size 
in bytes and in characters of spans (I would not expect that 
representation of whole buffer in memory is single contiguous byte 
array). When there are no markers at all, the function has to iterate 
over each character and count its length.


The problem is that when the buffer has a lot of markers far aside from 
the position passed as argument, then iteration over markers just 
consumes CPU with no significant improvement of original estimation of 
boundaries.


If markers were organized in a tree than search would be much faster (at 
least for buffers with a lot of markers.


In some cases such function may take a hint: previous known 
bytepos+charpos pair.


I hope I missed something, but what I can expect from the code of 
buf_bytepos_to_charpos is that it is necessary to iterate over all 
markers to update positions after each typed character.



Finally, FYI. I plan to work on an alternative mechanism to access Org
headings - generic Org query library. It will not use markers and
implement ideas from org-ql. org-refile will eventually use that generic
library instead of current mechanism.


I suppose that markers might be implemented in an efficient way, and 
much better performance may be achieved when low-level data structures 
are accessible. I am in doubts concerning attempts to create something 
that resembles markers but based purely on high-level API.