Re: [Tails-dev] Proposal: Add support for curl (does ALL_PROXY fix it?)

[sajolida]

boyska:

@boyska: Any concern about that since curl is already installed?

no specific concern, but of course, this moves the needle towards making 
it work out of the box.


All right, see you on #19448, then!

https://gitlab.tails.boum.org/tails/tails/-/issues/19488

--
sajolida
Tails — https://tails.boum.org/
UX · Fundraising · Technical Writing
___
Tails-dev mailing list
Tails-dev@boum.org
https://www.autistici.org/mailman/listinfo/tails-dev
To unsubscribe from this list, send an empty email to 
tails-dev-unsubscr...@boum.org.

Re: [Tails-dev] Proposal: Add support for curl (does ALL_PROXY fix it?)

[boyska]

On 08/03/2023 18:12, sajolida wrote:
I wanted to try adding curl by default to see the impact on the image 
size but ... it seems to be installed already:


https://gitlab.tails.boum.org/tails/tails/-/commit/8aa013f3b549d90609fb2a4dd7e2bd14e3439151 



ah! thanks for making me notice. I didn't remember, and indeed we don't 
really use the curl binary itself.




I created #19488 to fix this.

I'm even more convinced that we should fix that :)

@david: Do you want to give it a try?

@boyska: Any concern about that since curl is already installed?

no specific concern, but of course, this moves the needle towards making 
it work out of the box.


--
boyska



OpenPGP_signature
Description: OpenPGP digital signature
___
Tails-dev mailing list
Tails-dev@boum.org
https://www.autistici.org/mailman/listinfo/tails-dev
To unsubscribe from this list, send an empty email to 
tails-dev-unsubscr...@boum.org.

Re: [Tails-dev] Proposal: Add support for curl (does ALL_PROXY fix it?)

[sajolida]

David A. Wheeler:

I've created a proposal to briefly describe how to use
additional software within Tails. It includes some information on how to use 
curl:
https://gitlab.tails.boum.org/tails/tails/-/merge_requests/1044


Thanks! I'll have a look.


I had in mind investigative journalists & similar researchers with a *modest* 
technical background.
More generally, people who gather data (and sometimes send it) from the 
Internet,
especially those who can write simple scripts to do that.


Right, I have the same people in mind and already met several of them.

I wanted to try adding curl by default to see the impact on the image 
size but ... it seems to be installed already:


https://gitlab.tails.boum.org/tails/tails/-/commit/8aa013f3b549d90609fb2a4dd7e2bd14e3439151

Maybe I missed something!

That said, it doesn't work out of the box:

amnesia@amnesia:~$ curl https://tails.boum.org/
curl: (7) Failed to connect to 127.0.0.1 port 9050: Connection refused

I created #19488 to fix this.

I'm even more convinced that we should fix that :)

@david: Do you want to give it a try?

@boyska: Any concern about that since curl is already installed?

--
sajolida
Tails — https://tails.boum.org/
UX · Fundraising · Technical Writing
___
Tails-dev mailing list
Tails-dev@boum.org
https://www.autistici.org/mailman/listinfo/tails-dev
To unsubscribe from this list, send an empty email to 
tails-dev-unsubscr...@boum.org.

Re: [Tails-dev] Proposal: Add support for curl (does ALL_PROXY fix it?)

[David A. Wheeler]

> oh yeah, we all like curl. I wonder what kind of user needs curl, though. 
> ...


> On Feb 28, 2023, at 2:20 PM, sajolida  wrote:
> I wanted to complete a bit the picture on "who the heck would use curl"?
> Daniel Moßbrucker used wget for his investigative journalism:
> https://tails.boum.org/contribute/how/user_experience/interviews/daniel
> In his own words:
> « For the research on child abuse, we used wget for most of our crawling
>  and it's super useful to have it already installed. »
> ...
> There I also met other journos who have a bit of a technical profile and 
> investigate online data in ways that could totally benefit from wget and curl.
> In that regard (and many others), the personas on 
> https://tails.boum.org/contribute/personas/ are quite outdated, I think. 
> Thinking about these more techie-journos, curl could totally makes sense in 
> Tails.

I've created a proposal to briefly describe how to use
additional software within Tails. It includes some information on how to use 
curl:
https://gitlab.tails.boum.org/tails/tails/-/merge_requests/1044

If curl is later added the curl specifics could be removed, but at least that 
information makes it
easier for people to use curl if they wish to.

> David: Actually, I'm curious what made you suggest adding curl to Tails.
> Did you had any particular use case or users in mind?

I had in mind investigative journalists & similar researchers with a *modest* 
technical background.
More generally, people who gather data (and sometimes send it) from the 
Internet,
especially those who can write simple scripts to do that.

There are many websites that provide data access via an API (e.g., a RESTful API
with JSON data formats). It's possible to use wget, but curl is more typically 
used for this
purpose. Curl is *designed* to be used this way. E.g.:
* curl includes the libcurl library; it's not just a command line system. 
Various
  programming language's libraries use libcurl directly (e.g., pycurl for 
Python). This
  ease of access via libraries makes curl the easy/obvious choice for using 
curl in
  simple scripting in languages like Python.
* curl supports HTTP multipart/form-data sending. That is, it better supports 
HTTP(s) upload.
* For other pluses, see: https://daniel.haxx.se/docs/curl-vs-wget.html

Wget is far better than curl for recursively downloading websites, but for other
kinds of interactions, curl is often more capable than wget. Both have their 
uses.

--- David A. Wheeler
___
Tails-dev mailing list
Tails-dev@boum.org
https://www.autistici.org/mailman/listinfo/tails-dev
To unsubscribe from this list, send an empty email to 
tails-dev-unsubscr...@boum.org.

Re: [Tails-dev] Proposal: Add support for curl (does ALL_PROXY fix it?)

[sajolida]

David A. Wheeler:

oh yeah, we all like curl. I wonder what kind of user needs curl, though.
You mentioned that some packages in Debian will "just work" if curl is working. 
Can you name any of them? I think that would be very valuable info!

Sorry, what I meant was, "I expect that some packages in Debian will "just work" if 
curl is working".
I don't have a specific list.


However, we try to add feature to Tails when they are improving something for 
our personas (see https://tails.boum.org/contribute/personas/ ). In this 
specific case, I think we don't have an example of how can curl, directly or 
indirectly, help our personas.

It's really easy to add the configuration you propose to Tails. But it will add 
to our QA process: if we tell our users that they can expect curl to work fine, 
we'd better test it regularly!

All in all, the cost is low but not negligible, while the benefit for our 
personas seems to be zero. So my personal opinion is that it's not worth it 
*until* there is a clear use case that is helped by supporting curl.


Here again, I'm sorry for the huge delay in answering.

I wanted to complete a bit the picture on "who the heck would use curl"?

Daniel Moßbrucker used wget for his investigative journalism:

https://tails.boum.org/contribute/how/user_experience/interviews/daniel

In his own words:

« For the research on child abuse, we used wget for most of our crawling
  and it's super useful to have it already installed. »

« I wouldn't call myself a technology expert. I'm able to code a little
  bit, but for sophisticated projects like these, I still need someone
  with a real information security background to ask them how do certain
  things, which tools to use, and to check and improve my 30 lines of
  script. »

When we did the Tails workshop at Netzwerk Recherche in October, it was 
really sweet to see Daniel advocate for wget as one of the really cool 
tools included in Tails :)


There I also met other journos who have a bit of a technical profile and 
investigate online data in ways that could totally benefit from wget and 
curl.


In that regard (and many others), the personas on 
https://tails.boum.org/contribute/personas/ are quite outdated, I think. 
Thinking about these more techie-journos, curl could totally makes sense 
in Tails.


That said, I haven't really read the other emails in this thread about 
what it would take from a technical point of view; and it's true that 
everything that we had to Tails, even as small as curl, can turn into 
more maintenance work down the line.


David: Actually, I'm curious what made you suggest adding curl to Tails.

Did you had any particular use case or users in mind?


That said, clearly the Tails project already expects that some users will 
install additional software. So I suggest modifying the documentation to 
provide *some* tips to users on how to deal with differences when running on 
Tails (in particular the network and dot files). The current documentation 
never mentions torsocks or any other related issues. Then you don't have to do 
more QA, and if later on someone raises a persona, you're a step ahead.


We've had https://gitlab.tails.boum.org/tails/tails/-/issues/18013 for a 
while but I never gave it enough prioritize to tackle it myself. I added 
your suggestions there for now.


--
sajolida
Tails — https://tails.boum.org/
UX · Fundraising · Technical Writing
___
Tails-dev mailing list
Tails-dev@boum.org
https://www.autistici.org/mailman/listinfo/tails-dev
To unsubscribe from this list, send an empty email to 
tails-dev-unsubscr...@boum.org.

Re: [Tails-dev] Proposal: Add support for curl (does ALL_PROXY fix it?)

[David A. Wheeler]

> On Feb 7, 2023, at 4:04 AM, boyska  wrote:
> 
> On 2/6/23 19:00, David A. Wheeler wrote:
>>> oh yeah, we all like curl. I wonder what kind of user needs curl, though.
>>> You mentioned that some packages in Debian will "just work" if curl is 
>>> working. Can you name any of them? I think that would be very valuable info!
>> Sorry, what I meant was, "I expect that some packages in Debian will "just 
>> work" if curl is working".
>> I don't have a specific list.
> 
> However, we try to add feature to Tails when they are improving something for 
> our personas (see https://tails.boum.org/contribute/personas/ ). In this 
> specific case, I think we don't have an example of how can curl, directly or 
> indirectly, help our personas.
> 
> It's really easy to add the configuration you propose to Tails. But it will 
> add to our QA process: if we tell our users that they can expect curl to work 
> fine, we'd better test it regularly!
> 
> All in all, the cost is low but not negligible, while the benefit for our 
> personas seems to be zero. So my personal opinion is that it's not worth it 
> *until* there is a clear use case that is helped by supporting curl.

Fair enough. Quick *technical* arguments for curl:
* curl, when configured using ALL_PROXY, can handle IPv6. torsocks can't handle 
IPv6, and thus wget can't either. The Tor Browser handles IPv6 just fine.
* curl has a lot of capabilities not in wget. E.g., it can work as a pipe, 
support more protocols, enables uploading, and is easily called as a library. 
See: https://daniel.haxx.se/docs/curl-vs-wget.html
* I believe many packages depend on curl, though I didn't find a way to quickly 
do reverse dependency searches on Debian packages to verify this.

Those aren't personas, of course. Also, I *completely* understand the concern 
about adding to the QA process. Also, segfault made a good point that with 
torsocks, people can add "--isolate". I can see why that'd be important to some 
users. The same effect is possible with curl and SOCKS, but that can't be done 
with the trivial curl configuration file approach I presented (sadly). I've 
learned that torsocks creates this isolation by creating a special SOCKS "user" 
for each invocation that uses the process ID and seconds-since-epoch as part of 
the name. That's doable with curl, by setting the environment variable 
ALL_PROXY with this dynamic information at runtime. I believe this is how to do 
this (which even duplicates the torsocks username format even though that's not 
necessary; if I'm missing something let me know):
ALL_PROXY="socks5://torsocks-$$%3a$(date +%s):0@127.0.0.1:9050" 
curl-or-some-command-using-curl ...

That said, clearly the Tails project already expects that some users will 
install additional software. So I suggest modifying the documentation to 
provide *some* tips to users on how to deal with differences when running on 
Tails (in particular the network and dot files). The current documentation 
never mentions torsocks or any other related issues. Then you don't have to do 
more QA, and if later on someone raises a persona, you're a step ahead.

So I suggest modifying this chapter:
https://tails.boum.org/doc/persistent_storage/additional_software/index.en.html
by adding a new section just after "Installing additional software" called
"Using additional software". Move the text of "Additional configuration" (Some 
packages...)
into that section, followed by something like the proposed text below.

--- David A. Wheeler


Proposed new text for "Using additional software" (after the "Additional 
configuration" text):


Tails handles network access differently than a typical system.
Application programs that access the network must send their "normal" network 
communication ("TCP packets")
through something called a SOCKS service. This usually requires extra steps 
before the application
will successfully communicate over a network.

In many cases, you can get networking applications to work using the "torsocks" 
command line application. Torsocks enables many programs to work, but note that 
it does *not* support IPv6. To use torsocks, on the command line, run this 
followed by the COMMAND you wish to use with network access:

torsocks --isolate COMMAND

The program "curl" includes support for SOCKS. It's been reported to us that 
you can also run curl, and many programs that use curl, on the command line 
using this format (the "username" before the "@" symbol implements Tor stream 
isolation):

ALL_PROXY="socks5://torsocks-$$%3a$(date +%s):0@127.0.0.1:9050" COMMAND

More generally, for other applications that access the network, configure them 
to use the SOCKS proxy at IP address 127.0.0.1 (aka "home") with port 9050. We 
would suggest creating a SOCKS username which is different each time the 
application is used in a new session; this counters correlation between 
different sessions. The password for the SOCKS username is "0"; this isn't a 
security problem because 

Re: [Tails-dev] Proposal: Add support for curl (does ALL_PROXY fix it?)

[boyska]

On 2/6/23 19:00, David A. Wheeler wrote:

oh yeah, we all like curl. I wonder what kind of user needs curl, though.
You mentioned that some packages in Debian will "just work" if curl is working. 
Can you name any of them? I think that would be very valuable info!

Sorry, what I meant was, "I expect that some packages in Debian will "just work" if 
curl is working".
I don't have a specific list.


However, we try to add feature to Tails when they are improving 
something for our personas (see 
https://tails.boum.org/contribute/personas/ ). In this specific case, I 
think we don't have an example of how can curl, directly or indirectly, 
help our personas.


It's really easy to add the configuration you propose to Tails. But it 
will add to our QA process: if we tell our users that they can expect 
curl to work fine, we'd better test it regularly!


All in all, the cost is low but not negligible, while the benefit for 
our personas seems to be zero. So my personal opinion is that it's not 
worth it *until* there is a clear use case that is helped by supporting 
curl.


Bye,

--
boyska



OpenPGP_signature
Description: OpenPGP digital signature
___
Tails-dev mailing list
Tails-dev@boum.org
https://www.autistici.org/mailman/listinfo/tails-dev
To unsubscribe from this list, send an empty email to 
tails-dev-unsubscr...@boum.org.

Re: [Tails-dev] Proposal: Add support for curl (does ALL_PROXY fix it?)

[David A. Wheeler]

> On Feb 6, 2023, at 2:44 AM, segfault  wrote:
> I agree that it might be useful to document in the Additional Software docs 
> that some applications need to be run via torsocks. I'll let our UX and 
> documentation person decide on that.
> 
>> However, in the case of *curl*, using torsocks has drawbacks.
>> The torsocks program uses the LD_PRELOAD trick that is
>> sometimes unreliable
> 
> I'm not aware of torsocks being unreliable. It's used in Tails for many 
> applications and I use it myself for others and in my experience when it 
> works once it works every time.

LD_PRELOAD has its uses, but it works by intercepting lower-level function 
calls.
This means that future "minor" changes to a program or library can cause
things to unexpectedly break. E.g., If a function call is internally added or 
changed
that should be intercepted (but isn't) the interception won't work.
Calls that get statically linked don't get intercepted.
Also, glib will just plow ahead if loading LD_PRELOAD libraries fails
.
I've written several programs that use LD_PRELOAD, and found it's
remarkably fiddly & that upgrades can easily make things go wrong.
I don't think Debian designs or tests its programs to work with LD_PRELOAD.

So while LD_PRELOAD certainly has its uses, I think it's better to use
more reliable alternatives where they're sensible. E.g.,
configuring a tool to directly use SOCKS (if it can do so) and/or
implementing a transparent proxy (again, so it reliably works).
Implementing a transparent proxy takes some effort,
so I was trying to find a simple but reliable solution.


> On Feb 6, 2023, at 4:08 AM, boyska  wrote:
> oh yeah, we all like curl. I wonder what kind of user needs curl, though.
> You mentioned that some packages in Debian will "just work" if curl is 
> working. Can you name any of them? I think that would be very valuable info!

Sorry, what I meant was, "I expect that some packages in Debian will "just 
work" if curl is working".
I don't have a specific list. That said, as you stated, *many* people like 
curl, so
I expect a lot of tools use curl internally. I think a reverse dependency lookup
(of the Debian metadata, not of a running instance) would show a long list.

--- David A. Wheeler
___
Tails-dev mailing list
Tails-dev@boum.org
https://www.autistici.org/mailman/listinfo/tails-dev
To unsubscribe from this list, send an empty email to 
tails-dev-unsubscr...@boum.org.

Re: [Tails-dev] Proposal: Add support for curl (does ALL_PROXY fix it?)

[boyska]

On 04/02/2023 20:18, David A. Wheeler wrote:

Currently Tails includes and supports wget. I propose *also* adding support for 
curl.

Curl is widely-used and is the "go-to" tool for many people at the command line.
Thus, including it would make Tails more convenient for some. In addition,
enabling curl would make a number of added packages in Debian "just work",

oh yeah, we all like curl. I wonder what kind of user needs curl, though.
You mentioned that some packages in Debian will "just work" if curl is 
working. Can you name any of them? I think that would be very valuable info!


--
boyska
___
Tails-dev mailing list
Tails-dev@boum.org
https://www.autistici.org/mailman/listinfo/tails-dev
To unsubscribe from this list, send an empty email to 
tails-dev-unsubscr...@boum.org.

Re: [Tails-dev] Proposal: Add support for curl (does ALL_PROXY fix it?)

[segfault]

David A. Wheeler:

You might want to try `torsocks --isolate curl` (I didn't test it but that's 
often enough to make any program use Tor).


Currently torsocks is never mentioned in the Tails user documentation,
so a typical Tails user has a good chance of not knowing about it. (yes, 
torsocks *is*
mentioned in the design docs, but that doesn't count as *user* documentation 
:-) ).
So: No matter what, documenting that "additional programs"
might need to use torsocks (and how to use it) would be a good idea.


I agree that it might be useful to document in the Additional Software 
docs that some applications need to be run via torsocks. I'll let our UX 
and documentation person decide on that.



However, in the case of *curl*, using torsocks has drawbacks.
The torsocks program uses the LD_PRELOAD trick that is
sometimes unreliable


I'm not aware of torsocks being unreliable. It's used in Tails for many 
applications and I use it myself for others and in my experience when it 
works once it works every time.

___
Tails-dev mailing list
Tails-dev@boum.org
https://www.autistici.org/mailman/listinfo/tails-dev
To unsubscribe from this list, send an empty email to 
tails-dev-unsubscr...@boum.org.

Re: [Tails-dev] Proposal: Add support for curl (does ALL_PROXY fix it?)

[David A. Wheeler]

> On Feb 5, 2023, at 4:05 PM, segfault  wrote:
> 
> Hi,
> 
> David A. Wheeler:
>>> On Feb 4, 2023, at 2:18 PM, David A. Wheeler  wrote:
>>> 
>>> Currently Tails includes and supports wget. I propose *also* adding support 
>>> for curl.
>> Replying to myself, I think there's a slightly better way to automatically 
>> support curl.
>> I previously proposed setting the "ALL_PROXY" environment variable, but the 
>> best
>> setting for curl using ALL_PROXY uses prefixes that might confuse other tools
>> that might *also* read from ALL_PROXY.
>> So instead I propose this, to make Tails automatically support curl, while 
>> not
>> interfering with any other program.


> 
> You can also make Tails install curl automatically via the Additional 
> Software feature.

That *installs* curl automatically, but it doesn't *work* without special 
effort.
Instead, curl just fails. That's true if you call it directly, *or* if it's 
called
indirectly (by the many additional programs that use curl to download 
something).

Currently you have to take additional steps instead of simply using it as 
usual, and
that's what I would like to fix. If you have to type in anything extra, that's 
a problem.
After all, you don't have to type in anything special for wget.


> You might want to try `torsocks --isolate curl` (I didn't test it but that's 
> often enough to make any program use Tor).

Currently torsocks is never mentioned in the Tails user documentation,
so a typical Tails user has a good chance of not knowing about it. (yes, 
torsocks *is*
mentioned in the design docs, but that doesn't count as *user* documentation 
:-) ).
So: No matter what, documenting that "additional programs"
might need to use torsocks (and how to use it) would be a good idea.

However, in the case of *curl*, using torsocks has drawbacks.
The torsocks program uses the LD_PRELOAD trick that is
sometimes unreliable and is *completely*
unnecessary for curl. Curl already knows how to use SOCKS5
natively, has many SOCKS options, and curl listens to the resolver 
configuration.

The only advantage I can see to using torsocks is its --isolate option.
If the torsocks --isolate option really is valuable, then I think that simply
invoking "curl" should automatically invoke its equivalent,
without the user needing to know about torsocks.
How exactly does torsocks implement --isolate to create isolation? It looks like
it just creates a new SOCKS username/pw each time. Maybe we can
do the same thing without needing to use LD_PRELOAD tricks.

The only way I can see of make curl "just work" with torsocks --isolate
 is to use a wrapping technique similar to the one used for wget. But maybe
there's another way, since I don't fully grok the --isolate mechanism.

--- David A. Wheeler
___
Tails-dev mailing list
Tails-dev@boum.org
https://www.autistici.org/mailman/listinfo/tails-dev
To unsubscribe from this list, send an empty email to 
tails-dev-unsubscr...@boum.org.

Re: [Tails-dev] Proposal: Add support for curl (does ALL_PROXY fix it?)

[segfault]

Hi,

David A. Wheeler:



On Feb 4, 2023, at 2:18 PM, David A. Wheeler  wrote:

Currently Tails includes and supports wget. I propose *also* adding support for 
curl.


Replying to myself, I think there's a slightly better way to automatically 
support curl.
I previously proposed setting the "ALL_PROXY" environment variable, but the best
setting for curl using ALL_PROXY uses prefixes that might confuse other tools
that might *also* read from ALL_PROXY.

So instead I propose this, to make Tails automatically support curl, while not
interfering with any other program.


You might want to try `torsocks --isolate curl` (I didn't test it but 
that's often enough to make any program use Tor).


You can also make Tails install curl automatically via the Additional 
Software feature.


Cheers
segfault
___
Tails-dev mailing list
Tails-dev@boum.org
https://www.autistici.org/mailman/listinfo/tails-dev
To unsubscribe from this list, send an empty email to 
tails-dev-unsubscr...@boum.org.

Re: [Tails-dev] Proposal: Add support for curl (does ALL_PROXY fix it?)

[David A. Wheeler]

> On Feb 4, 2023, at 2:18 PM, David A. Wheeler  wrote:
> 
> Currently Tails includes and supports wget. I propose *also* adding support 
> for curl.

Replying to myself, I think there's a slightly better way to automatically 
support curl.
I previously proposed setting the "ALL_PROXY" environment variable, but the best
setting for curl using ALL_PROXY uses prefixes that might confuse other tools
that might *also* read from ALL_PROXY.

So instead I propose this, to make Tails automatically support curl, while not
interfering with any other program. In short, use CURL_HOME instead, as follows:

A new file 

CURL_HOME=/etc/curl


A new file :

# By default use the Tor proxy for communication.
# Tails uses this curl configuration file by default;
# unset the CURL_HOME environment variable to avoid using this file.
proxy=socks5://127.0.0.1:9050


Just be clear, don't include the lines with "", those just surround the 
file contents.

By adding these two files, curl "just works" on Tails
(as far as I can tell).

Note: Earlier I proposed used "socks5h://" as the prefix. This *works*, but
it means that every invocation of curl will make a DNS request directly to the
SOCKS proxy. Here I'm using "socks5://" instead; this will tell curl to use its
normal DNS resolution process, which on Tails will invoke the local resolver.
The local resolver can cache DNS request results; if it does, that should 
reduce the
number of requests if the same DNS name is used multiple times.
If the local DNS resolver isn't caching, and the SOCKS proxy is, then never 
mind :-).

Note that *no* flexibility is lost. If a user wants a special configuration, the
user can just unset CURL_HOME and then do whatever they want.

While I think it'd be convenient to also have curl itself installed, at the
*least* Tails should add these few lines listed above (or something like them).
so curl will "automatically work".

I don't fully understand how Tor deals with other proxies
(before and/or after), nor Tor bridges, so there may be cases
where this doesn't work. In that case, I'm hoping someone can see that
and figure out the tweaks to make those cases work too.
Again, I think curl should work "out of the box" on Tails without someone
having to figure out proxy configurations.

--- David A. Wheeler
___
Tails-dev mailing list
Tails-dev@boum.org
https://www.autistici.org/mailman/listinfo/tails-dev
To unsubscribe from this list, send an empty email to 
tails-dev-unsubscr...@boum.org.