Re: Embedded platforms (without FPU's), etc.

[Daniel Stenberg]

On Fri, 13 Apr 2018, Philip Prindeville wrote:


We're still doing C89, remember?


Is it worth throwing a switch, cutting the shackle, etc. at some point?


I think that's indeed a discussion worth having. An early question will of 
course thhen be how we should proceed to take that decision. Whenever we do, 
there will be N% of upset users somewhere. How many N is small enough? What 
upset level is acceptable? And what's the gain. What's the grand upside we get 
by cutting off the far end of the tail?


We can't ask users this question, because the users most in need for support 
ancient machines and systems aren't around regularly. It's complicated 
basically!


Okay, let me ask the question differently….  We could alias a 
seconds/microseconds counter as a curl_off_t if the sizeof(curl_off_t) == 8… 
and not expose the functionality otherwise.


Is that acceptable?


Why would it *require* 64 bits? Even within 32 bits, or say 31 bits when 
signed, we can have 2000 seconds with microsecond accuracy and most times that 
curl measures are surely within that time period.


Secondly; even if you would limit the functionality to only work on 64 bit 
machines, I don't think you should #ifdef the header file like you've done in


  https://github.com/curl/curl/pull/2495

Unrelated question, why is there CURL_SIZEOF_CURL_OFF_T and 
SIZEOF_CURL_OFF_T?


Global symbols *must* be prefixed with curl_ or CURL_. So anything define in 
curl/system.h will be named like that.


Symbols that are private to the curl or libcurl build can be named whatever.

We *used* to provide CURL_SIZEOF_CURL_OFF_T in curl/system.h but we don't 
anymore, so there's a #define CURL_SIZEOF_CURL_OFF_T SIZEOF_CURL_OFF_T in 
lib/curl_setup.h to make the older code remain functional. So internally 
they're identical but CURL_SIZEOF_CURL_OFF_T is deprecated.


And I’m building on Fedora 27 running on x86_64 hardware, but curl_off_t is 
being defined as “long”… What did I not do right?


long is 64 bit on your hardware so what's the problem?

What about a struct with an uint32_t upper and lower half as a workaround 
elsewhere?


Why?

--

 / daniel.haxx.se---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Re: Embedded platforms (without FPU's), etc.

[Philip Prindeville]

> On Apr 13, 2018, at 3:18 AM, Daniel Stenberg  wrote:
> 
> On Thu, 12 Apr 2018, Philip Prindeville wrote:
> 
>> I was looking at  and it’s a little hairy.
> 
> Yes it is, but it works pretty good.
> 
>> Can we just do something simple/stupid, like include 
> 
> That's a standard header since *C99* so we can't reliably include that 
> without knowing that it exists. We're still doing C89, remember?


Is it worth throwing a switch, cutting the shackle, etc. at some point?


> 
>> and if uint64_t is included then we support these timestamps, and if not, 
>> then we don’t?
> 
> uint64_t is not a standard type in C89, so that takes us back to #ifdefs …


Okay, let me ask the question differently….  We could alias a 
seconds/microseconds counter as a curl_off_t if the sizeof(curl_off_t) == 8… 
and not expose the functionality otherwise.

Is that acceptable?


> 
 So we might want to express times as uint64_t
>>> For what option(s) would this be used?
>> 
>> /* Fill in new entries below here! */
>> CURLINFO_TOTAL_TIME_T = CURLINFO_USECS + 50,
>> CURLINFO_NAMELOOKUP_TIME_T = CURLINFO_USECS + 51,
>> CURLINFO_CONNECT_TIME_T   = CURLINFO_USECS + 52,
>> CURLINFO_PRETRANSFER_TIME_T = CURLINFO_USECS + 53,
>> CURLINFO_STARTTRANSFER_TIME_T = CURLINFO_USECS + 54,
>> CURLINFO_REDIRECT_TIME_T  = CURLINFO_USECS + 55,
>> CURLINFO_APPCONNECT_TIME_T = CURLINFO_USECS_T + 56,
> 
> These *_T time options already return the times as curl_off_t, which is a 
> 64bit type on most systems. Isn't that good enough?


Okay, you lost me.  I’m proposing adding the above CURLINFO_*’s.

Currently there’s CURLINFO_TOTAL_TIME which takes a &double.

What’s the homologous interface which returns a curl_off_t?  I’m not seeing it.

Looking in lib/getinfo.c I’m not seeing any time-based interrogatories in 
getinfo_offt() other than CURLINFO_FILETIME_T, using curl_off_t *… as an alias 
for time_t.  But even time_t is limited to seconds resolution.

A uint64_t when representing microseconds wouldn’t wrap until 18446744073709 
seconds had passed, or 584574 years, or the year 586,544AD (assuming a 1970 
base).

Unrelated question, why is there CURL_SIZEOF_CURL_OFF_T and SIZEOF_CURL_OFF_T?  
What’s the difference?  My config.log is showing:

#define SIZEOF_SIZE_T 8
#define SIZEOF_LONG 8
#define SIZEOF_INT 4
#define SIZEOF_SHORT 2
#define SIZEOF_TIME_T 8
#define SIZEOF_OFF_T 8
#define SIZEOF_CURL_OFF_T 8
#define HAVE_LONGLONG 1
#define CURL_SIZEOF_CURL_SOCKLEN_T 4

But include/curl/curl.h doesn’t seem to have access to SIZEOF_CURL_OFF_T …  Is 
there a way to make something be conditional in  on the value of 
SIZEOF_CURL_OFF_T.

And I’m building on Fedora 27 running on x86_64 hardware, but curl_off_t is 
being defined as “long”… What did I not do right?  Not seeing anything in 
./configure that I missed… 

Trying to get https://github.com/curl/curl/pull/2495 submission worthy.



> 
>>> I think we offer most of those values as 'curl_off_t'
>> 
>> I couldn’t figure out if that’s guaranteed to be 64-bits in all cases.
> 
> No, as It can't be. There's no general *guaranteed* 64 bit availability in 
> curl code (except some parts that #ifdefs on that fact). But in general 
> that's not a problem. You get it as a 64bit type for all the system on which 
> we can use 64 bits (which in reality is most systems you use in modern life).


What about a struct with an uint32_t upper and lower half as a workaround 
elsewhere?

Ugly, but…


> 
 Is there some easy way to use a callback that gets called immediately 
 after the connect() completes?
>>> 
>>> No, we have no such dedicated callback.
>> 
>> Could the debug function be overloaded?
> 
> Everything is possible as we're talking about code here, but I would first go 
> back and ask: why - what's the benefit here and the use case that you can't 
> just work-around or fulfill with existing functionality. Then, as a secondary 
> follow-up, we could discuss that if we deem this to be a really good idea 
> then how would we implement it.
> 
> To me, this idea smells a lot like the old idea to expose the internal state 
> of the multi handle which also has been mentioned from time to time but I've 
> never been persuaded the price is worth the minor benefit to some apps.
> 
> -- 


No, like I mentioned previously, I’m involved in LMAP and we tend to decompose 
tests into constituent phases, e.g. if I do a 50MB download, it’s not enough to 
know that it took 8 seconds…  It might be the case that the download proper 
took 2 seconds but name resolution took 6…

-Philip


---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Re: Embedded platforms (without FPU's), etc.

[Daniel Stenberg]

On Thu, 12 Apr 2018, Philip Prindeville wrote:


I was looking at  and it’s a little hairy.


Yes it is, but it works pretty good.


Can we just do something simple/stupid, like include 


That's a standard header since *C99* so we can't reliably include that without 
knowing that it exists. We're still doing C89, remember?


and if uint64_t is included then we support these timestamps, and if not, 
then we don’t?


uint64_t is not a standard type in C89, so that takes us back to #ifdefs ...


So we might want to express times as uint64_t

For what option(s) would this be used?


 /* Fill in new entries below here! */
 CURLINFO_TOTAL_TIME_T = CURLINFO_USECS + 50,
 CURLINFO_NAMELOOKUP_TIME_T = CURLINFO_USECS + 51,
 CURLINFO_CONNECT_TIME_T   = CURLINFO_USECS + 52,
 CURLINFO_PRETRANSFER_TIME_T = CURLINFO_USECS + 53,
 CURLINFO_STARTTRANSFER_TIME_T = CURLINFO_USECS + 54,
 CURLINFO_REDIRECT_TIME_T  = CURLINFO_USECS + 55,
 CURLINFO_APPCONNECT_TIME_T = CURLINFO_USECS_T + 56,


These *_T time options already return the times as curl_off_t, which is a 
64bit type on most systems. Isn't that good enough?



I think we offer most of those values as 'curl_off_t'


I couldn’t figure out if that’s guaranteed to be 64-bits in all cases.


No, as It can't be. There's no general *guaranteed* 64 bit availability in 
curl code (except some parts that #ifdefs on that fact). But in general that's 
not a problem. You get it as a 64bit type for all the system on which we can 
use 64 bits (which in reality is most systems you use in modern life).


Is there some easy way to use a callback that gets called immediately 
after the connect() completes?


No, we have no such dedicated callback.


Could the debug function be overloaded?


Everything is possible as we're talking about code here, but I would first go 
back and ask: why - what's the benefit here and the use case that you can't 
just work-around or fulfill with existing functionality. Then, as a secondary 
follow-up, we could discuss that if we deem this to be a really good idea then 
how would we implement it.


To me, this idea smells a lot like the old idea to expose the internal state 
of the multi handle which also has been mentioned from time to time but I've 
never been persuaded the price is worth the minor benefit to some apps.


--

 / daniel.haxx.se---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Re: Embedded platforms (without FPU's), etc.

[Philip Prindeville]

> On Apr 2, 2018, at 3:27 AM, Daniel Stenberg  wrote:
> 
> On Sat, 31 Mar 2018, Philip Prindeville wrote:
> 
>> Couple of questions.  One is a broader question about tweaking the API so 
>> that it uses fewer “doubles” for get curl_easy_getinfo() call, in particular 
>> using fixed point since you might be running on a platform such as an 
>> embedded router with a MIPS-32 or ARMv6 CPU which doesn’t have 
>> floating-point… but is reasonably capable of 64-bit operations (shifting, 
>> adding, subtracting, etc).
> 
> We've slowly been moving in that direction over time. As Harold mentioned, 
> that easily conflicts with other limitation: not having 64 bit support.


I was looking at  and it’s a little hairy.   Can we just 
do something simple/stupid, like include  and if uint64_t is included 
then we support these timestamps, and if not, then we don’t?



> 
> IMO, the most important thing would be to not have floating point operations 
> in "the critical path" since most systems still have soft float support.
> 
>> So we might want to express times as uint64_t’s as (tv.tv_sec * 100) + 
>> tv.usec.
> 
> For what option(s) would this be used?

  /* Fill in new entries below here! */
  CURLINFO_TOTAL_TIME_T = CURLINFO_USECS + 50,
  CURLINFO_NAMELOOKUP_TIME_T = CURLINFO_USECS + 51,
  CURLINFO_CONNECT_TIME_T   = CURLINFO_USECS + 52,
  CURLINFO_PRETRANSFER_TIME_T = CURLINFO_USECS + 53,
  CURLINFO_STARTTRANSFER_TIME_T = CURLINFO_USECS + 54,
  CURLINFO_REDIRECT_TIME_T  = CURLINFO_USECS + 55,
  CURLINFO_APPCONNECT_TIME_T = CURLINFO_USECS_T + 56,


> 
>> Not sure why some values like bytes downloaded need to be “doubles”. Again, 
>> uint64_t would be ideal here.
> 
> I think we offer most of those values as 'curl_off_t' numbers if you use the 
> modern versions.


I couldn’t figure out if that’s guaranteed to be 64-bits in all cases.  Like 
here:

#else
/* generic "safe guess" on old 32 bit style */
# define CURL_TYPEOF_CURL_OFF_T long
# define CURL_FORMAT_CURL_OFF_T "ld"
# define CURL_FORMAT_CURL_OFF_TU"lu"
# define CURL_SUFFIX_CURL_OFF_T L
# define CURL_SUFFIX_CURL_OFF_TUUL
# define CURL_TYPEOF_CURL_SOCKLEN_T int
#endif

I haven’t touched a System 370 since college, for instance…  or a Watcom or 
Borland compiler, for that matter.


> 
>> I can come up with a set of patches that support uint64_t’s in more places.
> 
> Sure. If those changes actually make a difference in a real world application 
> then I think those can be good improvements.
> 
>> Lastly—as you might guess, I’m doing development for platforms which don’t 
>> have FPU’s—and I need to retrieve the delay from calling curl_easy_perform() 
>> and the socket completing a connect().
> 
> And you can explain some of that time on floating point operations? Well then 
> we should certainly make sure those don't happen.
> 
>> Is there some easy way to use a callback that gets called immediately after 
>> the connect() completes?
> 
> No, we have no such dedicated callback.


Could the debug function be overloaded?

-Philip



> 
>> For instance, the first time XFERFUNCTION gets called and then immediately 
>> unset the hook (inside the handler)?
> 
> XFERFUNCTION may be called numerous times before the connect() completes if 
> you have slow name lookup or otherwise slow network.
> 
> libcurl is purposedly focused on transfers and doesn't really try to expose 
> such specific events or states as "connect completed" within each transfer. 
> libcurl also supports transfers without connects.
> 


---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Re: Embedded platforms (without FPU's), etc.

[Philip Prindeville]

> On Apr 2, 2018, at 3:49 PM, Philip Prindeville 
>  wrote:
> 
>> 
>> 
>>> Not sure why some values like bytes downloaded need to be “doubles”. Again, 
>>> uint64_t would be ideal here.
>> 
>> I think we offer most of those values as 'curl_off_t' numbers if you use the 
>> modern versions.
> 
> 
> Some of the platforms I have to use have regrettably old toolchains and 
> run-times.
> 
> The link https://curl.haxx.se/libcurl/c/CURLINFO_SIZE_DOWNLOAD.html says:
> 
> CURLcode curl_easy_getinfo(CURL *handle, CURLINFO_SIZE_DOWNLOAD, double *dlp);
> 
> but maybe that’s not current.


Ah, you’re referring to CURLINFO_SIZE_DOWNLOAD_T… ding.

Sorry for being thick-headed.

-Philip


---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Re: Embedded platforms (without FPU's), etc.

[Philip Prindeville]

> On Apr 2, 2018, at 3:27 AM, Daniel Stenberg  wrote:
> 
> On Sat, 31 Mar 2018, Philip Prindeville wrote:
> 
>> Couple of questions.  One is a broader question about tweaking the API so 
>> that it uses fewer “doubles” for get curl_easy_getinfo() call, in particular 
>> using fixed point since you might be running on a platform such as an 
>> embedded router with a MIPS-32 or ARMv6 CPU which doesn’t have 
>> floating-point… but is reasonably capable of 64-bit operations (shifting, 
>> adding, subtracting, etc).
> 
> We've slowly been moving in that direction over time. As Harold mentioned, 
> that easily conflicts with other limitation: not having 64 bit support.
> 
> IMO, the most important thing would be to not have floating point operations 
> in "the critical path" since most systems still have soft float support.
> 
>> So we might want to express times as uint64_t’s as (tv.tv_sec * 100) + 
>> tv.usec.
> 
> For what option(s) would this be used?


Anything timestamp related, so CURLINFO_TOTAL_TIME, CURLINFO_CONNECTTIME, 
CURLINFO_APPCONNECT_TIME, CURLINFO_PRETRANSFER_TIME, 
CURLINFO_STARTTRANSFER_TIME, et al.


> 
>> Not sure why some values like bytes downloaded need to be “doubles”. Again, 
>> uint64_t would be ideal here.
> 
> I think we offer most of those values as 'curl_off_t' numbers if you use the 
> modern versions.


Some of the platforms I have to use have regrettably old toolchains and 
run-times.

The link https://curl.haxx.se/libcurl/c/CURLINFO_SIZE_DOWNLOAD.html says:

CURLcode curl_easy_getinfo(CURL *handle, CURLINFO_SIZE_DOWNLOAD, double *dlp);

but maybe that’s not current.


> 
>> I can come up with a set of patches that support uint64_t’s in more places.
> 
> Sure. If those changes actually make a difference in a real world application 
> then I think those can be good improvements.


Well, my universe is perhaps different from most: it’s about trying to breathe 
new life and functionality into $40 routers that are in some cases 10 years old 
and haven’t been updated in almost as long…

This is a set of constraints that most people are (happily) not burdened with.


> 
>> Lastly—as you might guess, I’m doing development for platforms which don’t 
>> have FPU’s—and I need to retrieve the delay from calling curl_easy_perform() 
>> and the socket completing a connect().
> 
> And you can explain some of that time on floating point operations? Well then 
> we should certainly make sure those don't happen.


I don’t know the overhead from using soft FP libraries on all of the platforms, 
but the point is that there are means that are both faster and don’t sacrifice 
an precision.  It also means not having to link the sometimes largish libraries 
into the apps (for platforms which don’t support DSO’s or support them poorly, 
due to broken library versioning, inconsistent ABI enforcement, etc).



> 
>> Is there some easy way to use a callback that gets called immediately after 
>> the connect() completes?
> 
> No, we have no such dedicated callback.


Ah.  Because then could store whatever representation one wanted… well, no 
matter.


> 
>> For instance, the first time XFERFUNCTION gets called and then immediately 
>> unset the hook (inside the handler)?
> 
> XFERFUNCTION may be called numerous times before the connect() completes if 
> you have slow name lookup or otherwise slow network.
> 
> libcurl is purposedly focused on transfers and doesn't really try to expose 
> such specific events or states as "connect completed" within each transfer. 
> libcurl also supports transfers without connects.
> 



For… TFTP and such?

-Philip


---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Re: Embedded platforms (without FPU's), etc.

[Dennis Clarke]

On 02/04/18 05:27 AM, Daniel Stenberg wrote:

On Sat, 31 Mar 2018, Philip Prindeville wrote:

Couple of questions.  One is a broader question about tweaking the API 
so that it uses fewer “doubles” for get curl_easy_getinfo() call, in 
particular using fixed point since you might be running on a platform 
such as an embedded router with a MIPS-32 or ARMv6 CPU which doesn’t 
have floating-point… but is reasonably capable of 64-bit operations 
(shifting, adding, subtracting, etc).


We've slowly been moving in that direction over time. As Harold 
mentioned, that easily conflicts with other limitation: not having 64 
bit support.


IMO, the most important thing would be to not have floating point 
operations in "the critical path" since most systems still have soft 
float support.




This same problem shows up in the openssl codebase also. Recently some
usage of things like "fabs()" fell into the codebase.  This was caught
in the openssl-1.1.1 beta code but released out to the world in the
openssl-1.1.0h sources.  Seems to happen when people go looking for a
diff in time even though the datatypes in the time structs are just
integers. Not sure where the floating point is used in the curl code
but I suspect it may be related to time calcs ?

For the sake of reference the openssl test ct_test.c was fixed in the
beta release of openssl 1.1.1 :


***
*** 538,556 
   * Tests that the CT_POLICY_EVAL_CTX default time is approximately now.
   * Allow +-10 minutes, as it may compensate for clock skew.
   */
! static int test_default_ct_policy_eval_ctx_time_is_now()
  {
  int success = 0;
  CT_POLICY_EVAL_CTX *ct_policy_ctx = CT_POLICY_EVAL_CTX_new();
  const time_t default_time = 
CT_POLICY_EVAL_CTX_get_time(ct_policy_ctx) /

! 1000;
  const time_t time_tolerance = 600;  /* 10 minutes */

! if (fabs(difftime(time(NULL), default_time)) > time_tolerance) {
! fprintf(stderr,
! "Default CT_POLICY_EVAL_CTX time is not approximately 
now.\n");

  goto end;
- }

  success = 1;
  end:
--- 496,512 
   * Tests that the CT_POLICY_EVAL_CTX default time is approximately now.
   * Allow +-10 minutes, as it may compensate for clock skew.
   */
! static int test_default_ct_policy_eval_ctx_time_is_now(void)
  {
  int success = 0;
  CT_POLICY_EVAL_CTX *ct_policy_ctx = CT_POLICY_EVAL_CTX_new();
  const time_t default_time = 
CT_POLICY_EVAL_CTX_get_time(ct_policy_ctx) /

! 1000;
  const time_t time_tolerance = 600;  /* 10 minutes */

! if (!TEST_time_t_le(abs((int)difftime(time(NULL), default_time)),
! time_tolerance))
  goto end;

  success = 1;
  end:
***

Lovely .. people are still using difftime for some reason and then
 casting it. Anyways .. it happens.

Dennis

ps: why did umich.edu send that post to curl maillist a dozen times?
---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Re: Embedded platforms (without FPU's), etc.

[Daniel Stenberg]

On Sat, 31 Mar 2018, Philip Prindeville wrote:

Couple of questions.  One is a broader question about tweaking the API so 
that it uses fewer “doubles” for get curl_easy_getinfo() call, in particular 
using fixed point since you might be running on a platform such as an 
embedded router with a MIPS-32 or ARMv6 CPU which doesn’t have 
floating-point… but is reasonably capable of 64-bit operations (shifting, 
adding, subtracting, etc).


We've slowly been moving in that direction over time. As Harold mentioned, 
that easily conflicts with other limitation: not having 64 bit support.


IMO, the most important thing would be to not have floating point operations 
in "the critical path" since most systems still have soft float support.


So we might want to express times as uint64_t’s as (tv.tv_sec * 100) + 
tv.usec.


For what option(s) would this be used?

Not sure why some values like bytes downloaded need to be “doubles”. 
Again, uint64_t would be ideal here.


I think we offer most of those values as 'curl_off_t' numbers if you use the 
modern versions.



I can come up with a set of patches that support uint64_t’s in more places.


Sure. If those changes actually make a difference in a real world application 
then I think those can be good improvements.


Lastly—as you might guess, I’m doing development for platforms which don’t 
have FPU’s—and I need to retrieve the delay from calling curl_easy_perform() 
and the socket completing a connect().


And you can explain some of that time on floating point operations? Well then 
we should certainly make sure those don't happen.


Is there some easy way to use a callback that gets called immediately after 
the connect() completes?


No, we have no such dedicated callback.

For instance, the first time XFERFUNCTION gets called and then immediately 
unset the hook (inside the handler)?


XFERFUNCTION may be called numerous times before the connect() completes if 
you have slow name lookup or otherwise slow network.


libcurl is purposedly focused on transfers and doesn't really try to expose 
such specific events or states as "connect completed" within each transfer. 
libcurl also supports transfers without connects.


--

 / daniel.haxx.se---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html

Re: Embedded platforms (without FPU's), etc.

[Harold Tessmann III]

On 3/31/18, Philip Prindeville  wrote:

> Couple of questions.  One is a broader question about tweaking the API so
> that it uses fewer “doubles” for get curl_easy_getinfo() call, in particular
> using fixed point since you might be running on a platform such as an
> embedded router with a MIPS-32 or ARMv6 CPU which doesn’t have
> floating-point… but is reasonably capable of 64-bit operations (shifting,
> adding, subtracting, etc).

Does your toolchain not support software floating-point? I'm curious
because I work with an armv5 that does have hardware floating point
(if only we could get our hardware vendor to give us a toolchain with
hard-float…).

> So we might want to express times as uint64_t’s as (tv.tv_sec * 100) +
> tv.usec.

If we're allowing for platforms without any kind of floating-point
support, should we also allow for platforms that don't support 64-bit
ints?

Perhaps we could copy the local system's struct timespec
?

Harold

---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette:   https://curl.haxx.se/mail/etiquette.html