On Thu, 11 Dec 2025, Bastian Jesuiter via curl-users wrote:
Would it be architectural a challenge to move URL/Domain parsing to the
beginning of the command parsing?
Well, there are several challenges involved here.
Currently curl parses the entire command line and sets up all the options
first then it runs all the transfers. URL-globs is expanded when found in the
transfer loop.
To illustrate how this is challenging: there is for example only one
--oauth2-bearer option which is set for any number of URLs, until you use
--next and then you can set another.
Imagine a command line like this:
curl -K special https://{www.,}{example,google}.{com,org}/[1-100] -O
It will download 800 files (if the config file doesn't add any additional
URLs). What if we want a config file that sets a unique bearer token for...
each domain.
I believe these are the steps we need to make curl do:
- When initially parsing the config file, store the parts that are "URL
specific" in memory for later evalution.
- Set everything up as per usual.
- When doing the transfers, for each URL, evaluate the URL specific config
section stored in memory and apply the options set for this URL.
- Make sure that options that were URL-specifically set specifically for the
*previous* URL are not applied anymore. Maybe user-agent was set for
domain1, but the URL specific option for domain2 does not set it. Then it
should remain at default.
Or can the domain part also be dynamic with globbing?
The globbing is text based and applied before it is parsed as a URL so the
globbing can be done on any part or section of the URL we like. We can even
glob parts of the scheme, the port number etc. Whatever crazy we fancy.
This is a valid glob:
'htt{p,ps}:/{/localhost:,/example.com:8}{80,81}/
Remember that the command line can also consist of 10,000 different URLs
listed one after the other.
At what point are the variables for write-out calculated? Maybe those could
partially be calculated earlier, if it doesn't fragment the code too much.
They are calculated after each transfer, as lots of them are times, speeds and
data from the transfer.
--
/ daniel.haxx.se || https://rock-solid.curl.dev
--
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-users
Etiquette: https://curl.se/mail/etiquette.html
