Re: [BUG] Error in data input and output format for org-columns--summary-estimate

[Ihor Radchenko]

and...@fedeli.eu writes:

>IR > May you share your changes?
>Sure!
>Here they are: In these slices I take the upper part of the fork (where in 
> case, assuming a small-big usage convention ;)) as that is the value that 
> surely testify the effort estimation overrun. Being so, at the time of this 
> writing I just realized the use of pcase is likely replaceable by a simpler 
> (car (last (mapcar...))) call :).
>> (let* ((effort-in-minutes
>
>> (pcase (mapcar #'org-duration-to-minutes (split-string 
> org-clock-effort "-"))
>
>>   (`(,_ ,value) value)
>
>>   (`(,value) value)))

I see.
As I said, Org does not currently support EFFORT to be a range.
Ranges only appear in column est+ summaries.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [BUG] Error in data input and output format for org-columns--summary-estimate

[andrea]

   IR > May you share your changes?
   Sure!
   Here they are: In these slices I take the upper part of the fork (where in 
case, assuming a small-big usage convention ;)) as that is the value that 
surely testify the effort estimation overrun. Being so, at the time of this 
writing I just realized the use of pcase is likely replaceable by a simpler 
(car (last (mapcar...))) call :).
   Cheers,
 Andrea

   Job: diff -bwi org-clock.el{.old,}

   720c720,723

   < (let* ((effort-in-minutes (org-duration-to-minutes org-clock-effort))

   ---

   > (let* ((effort-in-minutes

   > (pcase (mapcar #'org-duration-to-minutes (split-string 
org-clock-effort "-"))

   >   (`(,_ ,value) value)

   >   (`(,value) value)))

   828c831,833

   < (let ((effort-in-minutes (org-duration-to-minutes org-clock-effort))

   ---

   > (let ((effort-in-minutes (pcase (mapcar #'org-duration-to-minutes 
(split-string org-clock-effort "-"))

   >(`(,_ ,value) value)

   >(`(,value) value)))

   Da emacs-orgmode-bounces+andrea=fedeli...@gnu.org
   A and...@fedeli.eu
   Cc emacs-orgmode@gnu.org
   Data Wed, 16 Aug 2023 10:15:13 +
   Oggetto Re: [BUG] Error in data input and output format for 
org-columns--summary-estimate
   and...@fedeli.eu writes:

   > Howdy!
   > I'm back to a previous element partially discussed as I found other org 
places where the duration had to be adapted to be able to deal with ranges: 
org-clock-get-clock-string and
   > org-clock-notify-once-if-expired, both in og-clock.el; both get into 
action if you have a task you estimated and for which you're now tracking 
development time (quite handy, I have to say, as you're immediately warned 
you've running beyond estimations. For those two functions, I have introduced a 
similar change to the one I did originally to go from the basic string-to 
number on split-string to org-duration to minutes. Thanks, Sant Ignucius, for 
the debug-on-entry feature :))

   May you share your changes? I am not sure if I fully understand what and
   why you did without seeing the diff.

   > Two considerations here:
   > 1. I understand the fact that est+ doesn't have to necessarily be 
associated with effort, but it is quite clear from the docs which is the intent 
with which it was introduced: the only provided example is on times, and there 
we have to consider that time is expressed in durations.
   > What I mean is that it does NOT make much sense to me to tell users the 
effort is to be written as 3d if given as a single value, and it has to be 
rewritten as 3-5 if we want to say "in a fork of 3 to 5 days", especially if 
somewhere else some other duration unit is used..

   It should not. The reason `org-columns--summary-estimate' uses
   split-string is because it may have to work with recursively calculated
   estimates from subtrees. EFFORT property itself does not officially
   support ranges.

   --
   Ihor Radchenko // yantar92,
   Org mode contributor,
   Learn more about Org mode at .
   Support Org development at ,
   or support my work at 

Re: [BUG] Error in data input and output format for org-columns--summary-estimate

[Ihor Radchenko]

and...@fedeli.eu writes:

>Howdy!
>I'm back to a previous element partially discussed as I found other org 
> places where the duration had to be adapted to be able to deal with ranges: 
> org-clock-get-clock-string and
>org-clock-notify-once-if-expired, both in og-clock.el; both get into 
> action if you have a task you estimated and for which you're now tracking 
> development time (quite handy, I have to say, as you're immediately warned 
> you've running beyond estimations. For those two functions, I have introduced 
> a similar change to the one I did originally to go from the basic string-to 
> number on split-string to org-duration to minutes. Thanks, Sant Ignucius, for 
> the debug-on-entry feature :))

May you share your changes? I am not sure if I fully understand what and
why you did without seeing the diff.

>Two considerations here:
>1. I understand the fact that est+ doesn't have to necessarily be 
> associated with effort, but it is quite clear from the docs which is the 
> intent with which it was introduced: the only provided example is on times, 
> and there we have to consider that time is expressed in durations.
>What I mean is that it does NOT make much sense to me to tell users the 
> effort is to be written as 3d if given as a single value, and it has to be 
> rewritten as 3-5 if we want to say "in a fork of 3 to 5 days", especially if 
> somewhere else some other duration unit is used..

It should not. The reason `org-columns--summary-estimate' uses
split-string is because it may have to work with recursively calculated
estimates from subtrees. EFFORT property itself does not officially
support ranges.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [BUG] Error in data input and output format for org-columns--summary-estimate

[andrea]

   Howdy!
   I'm back to a previous element partially discussed as I found other org 
places where the duration had to be adapted to be able to deal with ranges: 
org-clock-get-clock-string and
   org-clock-notify-once-if-expired, both in og-clock.el; both get into action 
if you have a task you estimated and for which you're now tracking development 
time (quite handy, I have to say, as you're immediately warned you've running 
beyond estimations. For those two functions, I have introduced a similar change 
to the one I did originally to go from the basic string-to number on 
split-string to org-duration to minutes. Thanks, Sant Ignucius, for the 
debug-on-entry feature :))
   Two considerations here:
   1. I understand the fact that est+ doesn't have to necessarily be associated 
with effort, but it is quite clear from the docs which is the intent with which 
it was introduced: the only provided example is on times, and there we have to 
consider that time is expressed in durations.
   What I mean is that it does NOT make much sense to me to tell users the 
effort is to be written as 3d if given as a single value, and it has to be 
rewritten as 3-5 if we want to say "in a fork of 3 to 5 days", especially if 
somewhere else some other duration unit is used..
   2. Said differently, whether it was practically used also somewhere else, 
and not for time estimation, I can say nothing about; it is already quite hard 
to find it in doc, to date, and there  the only example given is about time.
   As without my changes the effort fork would not work properly in org-clock 
functions, if we really think estimation ranges shall be used also somewhere 
else I think we need to consider a possible generalization of what a duration 
is. In facts if we want to utilize it for other kind of measuring units 
(weight, money, etc.), it might make sense to pair it with a proper translation 
table like the one of duration that allows to convert days, hours, weeks, etc. 
into minutes, back and forth; this way we might allow both a proper type check 
according to the utilized measure unit, and would be able to avoid having to 
misleadlingly allow to sum tons to kg, for instance. (Recall: the ending letter 
today is just discarded hence 1kg-1ton is today taken as 1-1).
   Cheers,
 Andrea.

   Da "Ihor Radchenko" yanta...@posteo.net
   A and...@fedeli.eu
   Cc emacs-orgmode@gnu.org
   Data Tue, 18 Jul 2023 09:10:33 +
   Oggetto Re: [BUG] Error in data input and output format for 
org-columns--summary-estimate
   and...@fedeli.eu writes:

   >> About possibile abuses, org documentation, to date, clearly tells
   >> org estimate utilizes times.
   >
   >> May you please elaborate?

   > AF: Sure! Clause 8.5 of current
   > (2023.Jul.14) org documentation,
   > https://orgmode.org/manual/Effort-Estimates.html, refers to effort
   > estimates giving examples that are all appearing to fall in time
   > domain.

   I see what you mean.
   However, est+ column summary does not have to be applied to EFFORT
   columns.

   One can, for example, use %SCORE{est+} column specification to summarize
   values stored in SCORE property. Org has no right to demand SCORE to be
   in time units and only time units.

   > > Similarly, I'd not spend too much code to check the format; i
   > > understand the intent to preserve backward compatibility, bit if
   > > that format adaptation mistake slipped forward to these days I
   > > doubt that nice feature was used much so far...
   >
   > Yes, but it is easy to have a fallback, so why not.

   > Because this way you persist in allowing a mistaken usage of that
   > function. A number is a number but a duration is NOT just a number,
   > and having something like (just for example) 10kg-0.5ton be taken
   > as 10-0.5 is in my opinion NOT helpful to any user.

   We may provide a linter (for M-x org-lint) that will ensure EFFORT
   estimates to be in understandable format.

   --
   Ihor Radchenko // yantar92,
   Org mode contributor,
   Learn more about Org mode at .
   Support Org development at ,
   or support my work at 

Re: [BUG] Error in data input and output format for org-columns--summary-estimate

[Ihor Radchenko]

and...@fedeli.eu writes:

>> About possibile abuses, org documentation, to date, clearly tells
>> org estimate utilizes times.
>
>> May you please elaborate?

> AF: Sure! Clause 8.5 of current
>(2023.Jul.14) org documentation,
>https://orgmode.org/manual/Effort-Estimates.html, refers to effort
>estimates giving examples that are all appearing to fall in time
>domain.

I see what you mean.
However, est+ column summary does not have to be applied to EFFORT
columns.

One can, for example, use %SCORE{est+} column specification to summarize
values stored in SCORE property. Org has no right to demand SCORE to be
in time units and only time units.

>> Similarly, I'd not spend too much code to check the format; i
>> understand the intent to preserve backward compatibility, bit if
>> that format adaptation mistake slipped forward to these days I
>> doubt that nice feature was used much so far...
>
>Yes, but it is easy to have a fallback, so why not.

>Because this way you persist in allowing a mistaken usage of that
>function. A number is a number but a duration is NOT just a number,
>and having something like (just for example) 10kg-0.5ton be taken
>as 10-0.5 is in my opinion NOT helpful to any user.

We may provide a linter (for M-x org-lint) that will ensure EFFORT
estimates to be in understandable format.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [BUG] Error in data input and output format for org-columns--summary-estimate

[andrea]

   Howdy!

   Da "Ihor Radchenko" yanta...@posteo.net
   A and...@fedeli.eu
   Cc emacs-orgmode@gnu.org
   Data Fri, 14 Jul 2023 09:02:04 +
   Oggetto Re: [BUG] Error in data input and output format for 
org-columns--summary-estimate
   [ Adding Org ML back to CC. Please use "reply all" to reply on the
   mailing list ]

   "and...@fedeli.eu"  writes:

   > I do apologize, I noticed only now the patch content: the output
   > format of duration-from-minutes Is controlled by the
   > org-duration-format variable, so if the user wants results in
   > months (s)he can easily get It that way.

   `org-duration-format' is controlling many other aspects of formatting.
   Customizing, for example, agenda should probably not affect column views.
   AF: I think we need to univoquely clarify which unit is to be used in 
estimation. As shown below org documentation suggests time it to be used for 
that, from there it cames the idea to turn times into minutes for computation 
reason; maybe we may introduce a new format variable for estimation results 
reporting. I thought I might use org-duration-from-minutes for that, as it came 
very handy.
   > About possibile abuses, org documentation, to date, clearly tells
   > org estimate utilizes times.

   May you please elaborate?
   AF: Sure! Clause 8.5 of current (2023.Jul.14) org documentation, 
https://orgmode.org/manual/Effort-Estimates.html, refers to effort estimates 
giving examples that are all appearing to fall in time domain. In particular it 
refers to org-duration.el for format information. That, IMO, establish that 
efforts have to refer to org-duration units. Now, the *default* values of 
org-duration are *clearly* time measures. From there my assumption about the 
fact that I may use the org-duration-to-minutes and org-duration-from-minutes 
adapters.
   Clearly, you /may/ decide to completely redefine the org-duration basis, but 
that would mean to coherently propagate the information change everywhere 
somebody used org-duration-units which, even though LisP is a very flexible 
language, is NOT what I'd expect org-duration utilizer always foresaw to have 
to be flexible in terms of.
   > Similarly, I'd not spend too much code to check the format; i
   > understand the intent to preserve backward compatibility, bit if
   > that format adaptation mistake slipped forward to these days I
   > doubt that nice feature was used much so far...

   Yes, but it is easy to have a fallback, so why not.
   Because this way you persist in allowing a mistaken usage of that function. 
A number is a number but a duration is NOT just a number, and having something 
like (just for example) 10kg-0.5ton be taken as 10-0.5 is in my opinion NOT 
helpful to any user.
   The pcase matches either value pairs or single values, and 
org-duration-to-minutes can be charged to decide on what to do if values did 
not match the proper format (with the default assumption, I'd say, that simple 
integers are minutes).
   In facts org-duration-to-minutes already has a cond classical closing (t 
...) branch to deal with that case. I'd leave the rest as I suggested; maybe, 
if you --as maintainer hence exposed to the global amount of supports request 
and use cases-- see that as convenient, with a different output format adapter 
than the one I picked (org-duration-from-minutes without an extra format 
specification actual argument); already allowing a custom format adapter would 
introduce an extra flexibility knob BUT consider that org-duration-to-minutes 
does NOT do the same (it implicitly utilizes the org-duration-format content, 
and has hardcoded assumption on quantities being representing times, so there 
too you'd need to change something, if it was not just a matter of definining a 
different alternative to display result times).
   Cheers!
 Andrea.

   --
   Ihor Radchenko // yantar92,
   Org mode contributor,
   Learn more about Org mode at .
   Support Org development at ,
   or support my work at 

Re: [BUG] Error in data input and output format for org-columns--summary-estimate

[Ihor Radchenko]

[ Adding Org ML back to CC. Please use "reply all" to reply on the
mailing list ]

"and...@fedeli.eu"  writes:

> I do apologize, I noticed only now the patch content: the output
> format of duration-from-minutes Is controlled by the
> org-duration-format variable, so if the user wants results in
> months (s)he can easily get It that way.

`org-duration-format' is controlling many other aspects of formatting.
Customizing, for example, agenda should probably not affect column views.
 
> About possibile abuses, org documentation, to date, clearly tells
> org estimate utilizes times.

May you please elaborate?
 
> Similarly, I'd not spend too much code to check the format; i
> understand the intent to preserve backward compatibility, bit if
> that format adaptation mistake slipped forward to these days I
> doubt that nice feature was used much so far...

Yes, but it is easy to have a fallback, so why not.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [BUG] Error in data input and output format for org-columns--summary-estimate

[Ihor Radchenko]

and...@fedeli.eu writes:

>#+PROPERTY: Effort_ALL 0 0:10 0:30 1:00 2:00 3:00 4:00 5:00 6:00 7:00
>
>#+COLUMNS: %30ITEM(Task) %30Effort(Estimated Effort){est+} %CLOCKSUM
>...
>Originally produces
>image.png
>That shows that days and months are wrongly counted the same

Thanks!
Confirmed.

>> The matter is related to the fact that org-columns--summary-estimate, 
> from org-colview.el, uses function string-to-number to take value on ranges; 
> acting this way

>   Both issues can be very simply addressed by adoption of 
> org-duration-to-minutes as
>
>   input adapted and org-duration-from-minutes as output adapter.
>   A similar concern affects also the simpler case of a single value instead 
> of a range (second pcase branch)

This will not be backwards-compatible.
Some people may abuse this summary type to count something that is not
time, like cost.

We rather need something like the attached diff.
However, this will use `org-duration-format', which may not always be
desirable - the default value will force days even when all the estimates
are months:

3m
3m-4m
---
180d 0:00 - 210d 0:00

diff --git a/lisp/org-colview.el b/lisp/org-colview.el
index 7aa5ef645..da9a11104 100644
--- a/lisp/org-colview.el
+++ b/lisp/org-colview.el
@@ -1345,18 +1345,32 @@ (defun org-columns--summary-estimate (estimates _)
 The mean and variance of the result will be the sum of the means
 and variances (respectively) of the individual estimates."
   (let ((mean 0)
-(var 0))
+(var 0)
+(durationp
+ (catch :no-match
+   (dolist (e estimates)
+ (dolist (val (split-string e "-"))
+   (unless (org-duration-p val) (throw :no-match nil
+   'all-values-are-durations)))
 (dolist (e estimates)
-  (pcase (mapcar #'string-to-number (split-string e "-"))
+  (pcase (mapcar
+  (if durationp
+  #'org-duration-to-minutes
+#'string-to-number)
+  (split-string e "-"))
 	(`(,low ,high)
 	 (let ((m (/ (+ low high) 2.0)))
 	   (cl-incf mean m)
 	   (cl-incf var (- (/ (+ (* low low) (* high high)) 2.0) (* m m)
 	(`(,value) (cl-incf mean value
 (let ((sd (sqrt var)))
-  (format "%s-%s"
-	  (format "%.0f" (- mean sd))
-	  (format "%.0f" (+ mean sd))
+  (if durationp
+  (format "%s - %s"
+  (org-duration-from-minutes (- mean sd))
+  (org-duration-from-minutes (+ mean sd)))
+(format "%s-%s"
+	(format "%.0f" (- mean sd))
+	(format "%.0f" (+ mean sd)))
 
 
 

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [BUG] Error in data input and output format for org-columns--summary-estimate

[andrea]

   #+PROPERTY: Effort_ALL 0 0:10 0:30 1:00 2:00 3:00 4:00 5:00 6:00 7:00

   #+COLUMNS: %30ITEM(Task) %30Effort(Estimated Effort){est+} %CLOCKSUM

   * efforts

   ** task 1

   :PROPERTIES:

   :Effort:   3d

   :END:

   ** task 2

   :PROPERTIES:

   :Effort:   3m-4m

   :END:

   Originally produces
   image.png
   That shows that days and months are wrongly counted the same
   With the suggested patch produces
   image.png
   Showing that day and months are correctly handled, and reported.
   Cheers,
 Andrea.

   Da emacs-orgmode-bounces+andrea=fedeli...@gnu.org
   A and...@fedeli.eu
   Cc emacs-orgmode@gnu.org
   Data Sun, 09 Jul 2023 14:17:59 +
   Oggetto Re: [BUG] Error in data input and output format for 
org-columns--summary-estimate
   and...@fedeli.eu writes:

   > The matter is related to the fact that org-columns--summary-estimate, from 
org-colview.el, uses function string-to-number to take value on ranges; acting 
this way
   >
   > the unit is removed making impossible to distinguish between 1y and 1min . 
Similarly, on output, the two margins are produced a (format "%.0f" ), 
which --again-- is generally wrong as it does not tell anything about the 
utilized time unit.

   May you please provide a concrete example demonstrating the problem?

   --
   Ihor Radchenko // yantar92,
   Org mode contributor,
   Learn more about Org mode at .
   Support Org development at ,
   or support my work at 

Re: [BUG] Error in data input and output format for org-columns--summary-estimate

[Ihor Radchenko]

and...@fedeli.eu writes:

>The matter is related to the fact that org-columns--summary-estimate, from 
> org-colview.el, uses function string-to-number to take value on ranges; 
> acting this way
>
>the unit is removed making impossible to distinguish between 1y and 1min . 
> Similarly, on output, the two margins are produced a (format "%.0f" ), 
> which --again-- is generally wrong as it does not tell anything about the 
> utilized time unit.

May you please provide a concrete example demonstrating the problem?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

[BUG] Error in data input and output format for org-columns--summary-estimate

[andrea]

   Howdy!

   This has been tested to happen with org 9.5.5, a few of the

   following, and on the HEAD version on git (9.6.7+); it's a source

   matter, afflicting org independently of the running platform (GNU/Linux,

   Windows, etc.)

   The matter is related to the fact that org-columns--summary-estimate, from 
org-colview.el, uses function string-to-number to take value on ranges; acting 
this way

   the unit is removed making impossible to distinguish between 1y and 1min . 
Similarly, on output, the two margins are produced a (format "%.0f" ), 
which --again-- is generally wrong as it does not tell anything about the 
utilized time unit.

   Both issues can be very simply addressed by adoption of 
org-duration-to-minutes as

   input adapted and org-duration-from-minutes as output adapter.
   A similar concern affects also the simpler case of a single value instead of 
a range (second pcase branch)

   Here is the patched code with the wrong code lines commented out.

   (defun org-columns--summary-estimate (estimates _)

 "Combine a list of estimates, using mean and variance.

   The mean and variance of the result will be the sum of the means

   and variances (respectively) of the individual estimates."

 (let ((mean 0)

   (var 0))

   (dolist (e estimates)

   ;;  (pcase (mapcar #'string-to-number (split-string e "-"))

 (pcase (mapcar #'org-duration-to-minutes (split-string e "-"))

   (`(,low ,high)

(let ((m (/ (+ low high) 2.0)))

  (cl-incf mean m)

  (cl-incf var (- (/ (+ (* low low) (* high high)) 2.0) (* m m)

   ;;(`(,value) (cl-incf mean value

   (`(,value) (cl-incf mean (org-duration-to-minutes value)

   (let ((sd (sqrt var)))

 (format "%s-%s"

   ;;  (format "%.0f" (- mean sd))

   ;;  (format "%.0f" (+ mean sd))

 (org-duration-from-minutes (- mean sd))

 (org-duration-from-minutes (+ mean sd))

   Cheers!
 Andrea Fedeli.