Re: Search by metadata in GnuNet

[madmurphy]

Hi there!

Back in september I wrote an email
<https://lists.gnu.org/archive/html/gnunet-developers/2022-09/msg00056.html>
to this list more or less about this topic (you can read the whole
conversation here
<https://lists.gnu.org/archive/html/gnunet-developers/2022-09/threads.html#00056>
).

Personally I believe metadata is everything. In my opinion people must be
free to catalogue things as they like (even by writing fake metadata),
however they should face the trust of the community and accept that their
metadata will not spread too much if they have a bad reputation.

What function are you looking for, specifically?

--madmurphy

On Fri, Nov 25, 2022 at 10:36 PM Jimmy Bionic  wrote:

> Hi there,
>
> I am keen to find out about one particular aspect of search
> implementation in GnuNet. I belong to a small community of like minds who
> would like to exchange in a safer possible way some parts of our
> collections. Within our community we don't know each other personally and
> often reside in different countries. While the nature of our collections is
> somewhat peculiar because we collect meta-tagged images (primarily multiple
> *.JPG and *.PNG files tagged by IPTC/MWG with the use of ExifTool), and the
> content of images could be anything like pages from a new book that's just
> come out, a figure from an electronic publication that can be only
> purchased,  or just photos from posh magazines. This means that the content
> can be potentially subject to copyright claims, so trust and privacy
> (anonymity) in the process of exchange is crucial. The thing about ExifTool
> is that there are multiple tools which allow one to run search queries by
> metadata, but all of them do it only locally. While the nature of metadata
> is somewhat shaky because there is currently no way to ensure the validity
> of the tags and their claimed match with the content. To put it simply, an
> image may be picturing a pet, but the tags are saying that this is a lion
> or planet Jupiter. That's why we rely so much on trust and welcome only
> sensible people to avoid fakes. The reason we've paid some attention to
> existing P2P solutions is because our collections are vast and we would
> love to search in each other's collections and so to exchange. Not many P2P
> actually have in-built search engines (eg. eDonkey has, BitTorrent
> doesn't), while search by metadata is desirable but even a more distant
> perspective. GnuNet seems to have the F2F topology which separates it from
> many other existing solutions. In order for it to fit in with our
> requirements, I suppose, some additional development may be required. So I
> wonder how flexible you are with regards to the implementation of the
> search by metadata in GnuNet?
>
> I would appreciate any comments from your side.
>
> With best regards.
>
>
>
>
>

Re: FS as user or not

[madmurphy]

Still related to this thread
<https://lists.gnu.org/archive/html/gnunet-developers/2022-10/msg4.html>,
If I launch gnunet-fs -i *as normal user*, first I get printed part of the
list, then I get printed

2022-10-24T20:05:56.483363+0100 gnunet-fs-284292 WARNING Failed to
receive response from `fs' service (error code is 1).
Segmentation fault (core dumped)

The moment in which the error is thrown varies. Sometimes gnunet-fs -i
prints two files that I am sharing and then it throws an error. Other times
it prints ten files and then it throws an error. Etc.

No errors instead if I launch gnunet-fs -i *as gnunet user*.

Error code 1 means GNUNET_MQ_ERROR_READ. There is only one point in the
entire GNUnet project that throws GNUNET_MQ_ERROR_READ, and that is
src/util/client.c at line 454
<https://git.gnunet.org/gnunet.git/tree/src/util/client.c?id=d435321cfca9a667ec6fbc3bcf675142d58ab463#n454>
.

Anyone able to reproduce this bug? Attached is my configuration.

--madmurphy

On Sun, Oct 9, 2022 at 11:37 AM madmurphy  wrote:

> Hi Christian,
>
> Yes, my username has always been in the gnunet group. But in the past I
> always used the system services of GNUnet, except for the cases in which I
> needed the user services (like for creating egos).
>
> The only (unlikely) explanation I can think of is that when I search the
> network using the gnunet system user it shows a lot of results that were
> cached earlier, while when I search from my personal user it shows only new
> results, and for some reason there are no more files in the network
> published under “commons”. This is why I asked if people see the same
> behavior I see.
>
> If that is not the reason, then I have no explanations.
>
> --madmurphy
>
> On Sat, Oct 8, 2022 at 11:08 AM Christian Grothoff 
> wrote:
>
>> Hmm. Is your $USER in the 'gnunet' group?
>>
>> On 10/6/22 08:54, madmurphy wrote:
>> > After one of the last commits, if I launch
>> >
>> > gnunet-search commons
>> >
>> > I get only one file that I am sharing, while if I launch
>> >
>> > sudo -u gnunet gnunet-search commons
>> >
>> > I get more than 40 results.
>> >
>> > For a second I thought that this commit from Jacki
>> > <
>> https://git.gnunet.org/gnunet.git/commit/?id=1672c85ad702146521dc830298dcb3f802533539
>> >
>> > was the reason, but it happens also if I restore the configuration
>> > before Jacki's commit. So I have no idea what happened.
>> >
>> > Anyone getting the same behavior?
>> >
>> > --madmurphy
>> >
>>
>>
<>

Re: contributing

[madmurphy]

On Tue, Oct 25, 2022 at 6:28 PM  wrote:

I would like to send some code amendments to the gnunet-ext repo. What are
the necessary steps, please?

The easieast way:

git clone https://git.gnunet.org/gnunet-ext.git
cd gnunet-ext

Edit whatever you want to edit in the gnunet-ext directory, then launch the
following commands (feel free to change the message as it best suits you):

git add --all
git commit -m 'Write your message here'
git format-patch -1

Finally, send the 0001-Write-your-message-here.patch file generated to this
mailing list (the actual name will change according to your message).

--madmurphy

On Tue, Oct 25, 2022 at 6:28 PM  wrote:

> Hello,
>
> there's still TODO wtr link to copyright waiver for contributing in the
> docs:
>
>
> file:///home/lash/src/build/gnunet/0.17.6-198-g1436e4266/doc/handbook/html/developers/contributing.html
>
> I would like to send some code amendments to the gnunet-ext repo. What
> are the necessary steps, please?
>
> Thanks,
> l
>

Re: FS as user or not

[madmurphy]

Hi Christian,

Yes, my username has always been in the gnunet group. But in the past I
always used the system services of GNUnet, except for the cases in which I
needed the user services (like for creating egos).

The only (unlikely) explanation I can think of is that when I search the
network using the gnunet system user it shows a lot of results that were
cached earlier, while when I search from my personal user it shows only new
results, and for some reason there are no more files in the network
published under “commons”. This is why I asked if people see the same
behavior I see.

If that is not the reason, then I have no explanations.

--madmurphy

On Sat, Oct 8, 2022 at 11:08 AM Christian Grothoff 
wrote:

> Hmm. Is your $USER in the 'gnunet' group?
>
> On 10/6/22 08:54, madmurphy wrote:
> > After one of the last commits, if I launch
> >
> > gnunet-search commons
> >
> > I get only one file that I am sharing, while if I launch
> >
> > sudo -u gnunet gnunet-search commons
> >
> > I get more than 40 results.
> >
> > For a second I thought that this commit from Jacki
> > <
> https://git.gnunet.org/gnunet.git/commit/?id=1672c85ad702146521dc830298dcb3f802533539
> >
> > was the reason, but it happens also if I restore the configuration
> > before Jacki's commit. So I have no idea what happened.
> >
> > Anyone getting the same behavior?
> >
> > --madmurphy
> >
>
>

FS as user or not

[madmurphy]

After one of the last commits, if I launch

gnunet-search commons

I get only one file that I am sharing, while if I launch

sudo -u gnunet gnunet-search commons

I get more than 40 results.

For a second I thought that this commit from Jacki
<https://git.gnunet.org/gnunet.git/commit/?id=1672c85ad702146521dc830298dcb3f802533539>
was the reason, but it happens also if I restore the configuration before
Jacki's commit. So I have no idea what happened.

Anyone getting the same behavior?

--madmurphy

Re: The future of the file-sharing service

[madmurphy]

Jacki,
On Wed, Sep 21, 2022 at 12:23 AM TheJackiMonster 
wrote:

I generally just like the idea that we could use GNUnet for contribution
instead of the current central servers.

I did think about a decentralized git on top of GNUnet more than once, and
it would be awesome.

David,
On Wed, Sep 21, 2022 at 1:37 AM David Barksdale via Mailinglist for GNUnet
developers  wrote:


   1. Possibility of sharing a file while it is still being downloaded
   (parts of it, of course)

This is already possible, the GAP routing layer can decide to cache any
block of data it receives in the datastore.

Yes, you also have (and automatically share) chunks of files you never
downloaded, but (as far as I know) at the moment you cannot *explictly*
publish foobar.mkv while you are still downloading it (you will have to
wait for the whole file, and if you don't launch gnunet-publish after you
have downloaded it these chunks will eventually get lost).

What I am imagining is an option (let's say a -s argument for
gnunet-download) that allows you explicitly to publish a file, using the
metadata that you decide in the moment, *already when you are still
downloading that file* – basically a full-fledged gnunet-publish nested
inside gnunet-download.


   1. Metadata must be editable and sharable

You can always publish new metadata under a keyword. What does sharable
mean here?

You can add new keywords to a file, but you cannot see the current keywords
whereby a file has been indexed, nor edit them, nor remove them (as far as
I know). If I am not wrong you also cannot edit a metadata field that you
had previously assigned to a file. Sharable means that the metadata can be
exported and imported using an exchange format if the user wants (JSON?
INI?), but also *queryable* (see next point).


   1. Search keywords must be visible, editable, sharable (part of the
   metadata?)

Metadata (along with the URI of the file) is published under keywords, a
keyword search returns this information. Are you suggesting including all
the known keywords as part of the metadata?

If I publish a file under the keywords “foo” and “bar”, these keywords
should be visible, not just guessable indirectly (at the moment if I search
for “foo” and I find a file I can guess that that keyword has been used for
that file, but I cannot access the other keywords, if there are any).

Basically, I imagine the entire metadata associated with an URI as an open
book (or better, a collection of open books). That means that I must be
able to access the metadata that *I* assigned to that URI, but also the
metadata that *other users* (anonymously, of course, i.e. “the network”)
assigned to that same URI. For example, if I publish hamlet.pdf and my
metadata contains “comment=best drama ever” I must be able to access that
metadata field and know that that is what I am using. If you publish the
same pdf and your metadata contains “comment=worst drama ever”, I must also
be able to know that *someone* is publishing the same file as “worst drama
ever” (without having any info on whom). The old eMule/aMule did something
similar and you could “search” the network for the metadata associated with
a specific file (but without any privacy and without too much
sophistication).


   1. Introduction of a rating mechanism for files (against spam)
   2. Allow reverse search (i.e. chk-URI lookup)

This could be done by using the URI as a keyword when publishing.

I don't know if using the URI as a keyword when publishing would be the
best implementation (sounds a bit redundant, doesn't it?), but what I am
suggesting is making it a default GNUnet feature for public files.


   1. Automatically and fully auto-unindex a file when it is missing
   2. Autoshare the dynamic content of a directory and update its index in
   real time (e.g. if I “autoshare” the content of
   /srv/filesharing/gnunet/madmurphy/, when I add foobar.txt to that
   directory it must be automatically indexed – the opposite if I remove it)

This is what gnunet-auto-share does.

Uh I missed that. Is there a “libgnunetautoshare” too?


   1. Implement file statistics (download counter? last seen? etc.) – this
   should allow the network to get rid easily of “lost” content

The gtk gui tries to download parts of search results to display how
available a file is.

This info would need to be stored also for files that you don't download
(i.e. it must be part of the DHT, not just of a GUI). It should be like a
sort of collective consciousness of the network behind the scenes, with the
aim of preventing the propagation of orphan chunks or regulating the
propagation of files that have not been downloaded often.

--madmurphy

Re: The future of the file-sharing service

[madmurphy]

On Tue, Sep 20, 2022 at 3:45 PM  wrote:

My 2 cents - general top priorities for file sharing applications:

• Make it as anonymous/secure and distributed as possible - 5-eye-safe
• Make it streaming-capable

Hi Bastian,

Personally I like both. List updated below.

   1. Possibility of sharing a file while it is still being downloaded
   (parts of it, of course)
   2. Metadata must be editable and sharable
   3. Search keywords must be visible, editable, sharable (part of the
   metadata?)
   4. Introduction of a rating mechanism for files (against spam)
   5. Allow reverse search (i.e. chk-URI lookup)
   6. Automatically and fully auto-unindex a file when it is missing
   7. Autoshare the dynamic content of a directory and update its index in
   real time (e.g. if I “autoshare” the content of
   /srv/filesharing/gnunet/madmurphy/, when I add foobar.txt to that
   directory it must be automatically indexed – the opposite if I remove it)
   8. Implement file statistics (download counter? last seen? etc.) – this
   should allow the network to get rid easily of “lost” content
   9. Implement a NOT operator for search keywords (a tilde, “~”?)
   10. Implement an OR operator (a vertical bar, “|”? currently not writing
   any operator equals OR, but we need an explicit OR operator if we want to
   implement the next point)
   11. Allow parenthesis parsing for AND/OR/NOT operators; this will
   require that operators can be followed by spaces (e.g. gnunet-search
   +required '+' '(' optional1 '|' optional2 ')')
   12. The output filename must become optional in gnunet-download. If not
   specified, the file/directory must be downloaded using the original
   filename it was published with – if currently this information is not
   always saved during publishing, then we must make sure that it is always
   saved (this is part of the metadata too and must be editable)
   13. “Private file sharing so that URIs don't reveal the actual hash and
   the actual file size. So either the data in the URI or file needs to be
   encrypted with a symmetric key” (Jacki)
   14. Local operations must not require the GNUnet services to be running
   (e.g.: querying the list of indexed files)
   15. “Make it as anonymous/secure and distributed as possible -
   5-eye-safe” (Bastian)
   16. “Make it streaming-capable” (Bastian)


On Tue, Sep 20, 2022 at 3:45 PM  wrote:

> My 2 cents - general top priorities for file sharing applications:
>
> • Make it as anonymous/secure and distributed as possible - 5-eye-safe
> • Make it streaming-capable
>
>
> Greetings,
> Bastian Schmidt
>
> --- Ursprüngliche Nachricht ---
> Von: madmurphy 
> Datum: 19.09.2022 20:42:45
> An: gnunet-developers 
> Betreff: The future of the file-sharing service
>
> > Hi all,
> > On Thu, Sep 15, 2022 at 2:55 PM Maxime Devos 
> > wrote:
> >
> > Myself, I'm very interested in the file-sharing service and DHT service
> --
> >
> > I would like to implement (Guix) substitutes over GNUnet (I already sent
> > a
> > POC implementation previously, but it wasn't a very nice implementation,
> >
> > e.g. there was no fallback to regular http / https, that's why I started
> >
> > scheme-gnunet, to make GNUnet easier to use from Guile.
> >
> > I know that probably the FS service is not the priority at the moment,
> but
> >
> > I was thinking that nothing forbids to imagine now how an ideal FS
> service
> >
> > could look like. Time ago I had written down some of the features that I
> >
> > believe a future FS module will need to implement. I paste my list here,
> >
> > with the hope of leaving food for thought for a brain storming.
> >
> > What FS needs to have:
> >
> >1. Possibility of sharing a file while it is still being downloaded
> >(parts of it, of course)
> >2. Metadata must be editable and sharable
> >3. Search keywords must be visible, editable, sharable (part of the
> >metadata?)
> >4. Introduction of a rating mechanism for files (against spam)
> >5. Allow reverse search (i.e. chk-URI lookup)
> >6. Automatically and fully auto-unindex a file when it is missing
> >7. Autoshare the dynamic content of a directory and update its index
> in
> >
> >real time (e.g. if I “autoshare” the content of
> >/srv/filesharing/gnunet/madmurphy/, when I add foobar.txt to that
> >directory it must be automatically indexed – the opposite if I remove
> > it)
> >8. Implement file statistics (download counter? last seen? etc.) –
> this
> >
> >should allow the network to get rid easily of “lost” content
> >9. Implement a NOT operator for search keywords (a tilde, “~”?)
> >10. Implement an OR operator (a 

Re: The future of the file-sharing service

[madmurphy]

Forgot about this:
On Mon, Sep 19, 2022 at 9:06 PM TheJackiMonster 
wrote:

Also I would like to see git over GNUnet FS or CADET (depends on what makes
more sense). Because when I think about the autoshare functionality, I
already start thinking about syncing files between devices which can cause
conflicts.

That seems like a complex task, which maybe could be accomplished by a
further service on top of FS (something like “FSGIT”)?

On Mon, Sep 19, 2022 at 11:39 PM madmurphy  wrote:

> Here is the list updated with Jacki's suggestion and a further point from
> me I had forgotten to mention…
>
>1. Possibility of sharing a file while it is still being downloaded
>(parts of it, of course)
>2. Metadata must be editable and sharable
>3. Search keywords must be visible, editable, sharable (part of the
>metadata?)
>4. Introduction of a rating mechanism for files (against spam)
>5. Allow reverse search (i.e. chk-URI lookup)
>6. Automatically and fully auto-unindex a file when it is missing
>7. Autoshare the dynamic content of a directory and update its index
>in real time (e.g. if I “autoshare” the content of
>/srv/filesharing/gnunet/madmurphy/, when I add foobar.txt to that
>directory it must be automatically indexed – the opposite if I remove it)
>8. Implement file statistics (download counter? last seen? etc.) –
>this should allow the network to get rid easily of “lost” content
>9. Implement a NOT operator for search keywords (a tilde, “~”?)
>10. Implement an OR operator (a vertical bar, “|”? currently not
>writing any operator equals OR, but we need an explicit OR operator if we
>want to implement the next point)
>11. Allow parenthesis parsing for AND/OR/NOT operators; this will
>require that operators can be followed by spaces (e.g. gnunet-search
>+required '+' '(' optional1 '|' optional2 ')')
>12. The output filename must become optional in gnunet-download. If
>not specified, the file/directory must be downloaded using the original
>filename it was published with – if currently this information is not
>always saved during publishing, then we must make sure that it is always
>saved (this is part of the metadata too and must be editable)
>13. “Private file sharing so that URIs don't reveal the actual hash
>and the actual file size. So either the data in the URI or file needs to be
>encrypted with a symmetric key” (Jacki)
>14. Local operations must not require the GNUnet services to be
>running (e.g.: querying the list of indexed files)
>
> On Mon, Sep 19, 2022 at 9:50 PM Luis Soeiro  wrote:
>
>
>1. Metadata must be editable and sharable
>2. Search keywords must be visible, editable, sharable (part of the
>metadata?)
>
> Hummm. Ok. But would it lead to spammers?
>
> Hi Luis, the rating mechanism will prevent that.
> On Mon, Sep 19, 2022 at 9:50 PM Luis Soeiro  wrote:
>
>
>1. Allow reverse search (i.e. chk-URI lookup)
>
> This might be troublesome. Imagine if some entity is looking for people
> who might donwload some special file (a planted file or an unauthorized
> file). That entity might try to locate the URI and then the users by
> looking at the pieces...
>
> That should be prevented by the network even if chk-URI lookup is allowed…
>
> --madmurphy
>
> On Mon, Sep 19, 2022 at 9:50 PM Luis Soeiro  wrote:
>
>> Hello
>>
>> Em 2022-09-19 20:42, madmurphy escreveu:
>>
>> > What FS needs to have:
>> >
>> > * Possibility of sharing a file while it is still being downloaded
>> > (parts of it, of course)
>>
>> That would be interesting. It would be cool to have it swarm the pieces
>> (like torrent)...
>>
>> > * Metadata must be editable and sharable
>> > * Search keywords must be visible, editable, sharable (part of the
>> > metadata?)
>>
>> Hummm. Ok. But would it lead to spammers?
>>
>> > * Introduction of a rating mechanism for files (against spam)
>>
>> +1
>>
>> > * Allow reverse search (i.e. chk-URI lookup)
>>
>> This might be troublesome. Imagine if some entity is looking for people
>> who might donwload some special file (a planted file or an unauthorized
>> file). That entity might try to locate the URI and then the users by
>> looking at the pieces...
>>
>> > * Automatically and fully auto-unindex a file when it is missing
>> > * Autoshare the dynamic content of a directory and update its index in
>> > real time (e.g. if I "autoshare" the content of
>> > /srv/filesharing/gnunet/madmurphy/, when I add foobar.txt to that
>> > d

Re: The future of the file-sharing service

[madmurphy]

Here is the list updated with Jacki's suggestion and a further point from
me I had forgotten to mention…

   1. Possibility of sharing a file while it is still being downloaded
   (parts of it, of course)
   2. Metadata must be editable and sharable
   3. Search keywords must be visible, editable, sharable (part of the
   metadata?)
   4. Introduction of a rating mechanism for files (against spam)
   5. Allow reverse search (i.e. chk-URI lookup)
   6. Automatically and fully auto-unindex a file when it is missing
   7. Autoshare the dynamic content of a directory and update its index in
   real time (e.g. if I “autoshare” the content of
   /srv/filesharing/gnunet/madmurphy/, when I add foobar.txt to that
   directory it must be automatically indexed – the opposite if I remove it)
   8. Implement file statistics (download counter? last seen? etc.) – this
   should allow the network to get rid easily of “lost” content
   9. Implement a NOT operator for search keywords (a tilde, “~”?)
   10. Implement an OR operator (a vertical bar, “|”? currently not writing
   any operator equals OR, but we need an explicit OR operator if we want to
   implement the next point)
   11. Allow parenthesis parsing for AND/OR/NOT operators; this will
   require that operators can be followed by spaces (e.g. gnunet-search
   +required '+' '(' optional1 '|' optional2 ')')
   12. The output filename must become optional in gnunet-download. If not
   specified, the file/directory must be downloaded using the original
   filename it was published with – if currently this information is not
   always saved during publishing, then we must make sure that it is always
   saved (this is part of the metadata too and must be editable)
   13. “Private file sharing so that URIs don't reveal the actual hash and
   the actual file size. So either the data in the URI or file needs to be
   encrypted with a symmetric key” (Jacki)
   14. Local operations must not require the GNUnet services to be running
   (e.g.: querying the list of indexed files)

On Mon, Sep 19, 2022 at 9:50 PM Luis Soeiro  wrote:


   1. Metadata must be editable and sharable
   2. Search keywords must be visible, editable, sharable (part of the
   metadata?)

Hummm. Ok. But would it lead to spammers?

Hi Luis, the rating mechanism will prevent that.
On Mon, Sep 19, 2022 at 9:50 PM Luis Soeiro  wrote:


   1. Allow reverse search (i.e. chk-URI lookup)

This might be troublesome. Imagine if some entity is looking for people who
might donwload some special file (a planted file or an unauthorized file).
That entity might try to locate the URI and then the users by looking at
the pieces...

That should be prevented by the network even if chk-URI lookup is allowed…

--madmurphy

On Mon, Sep 19, 2022 at 9:50 PM Luis Soeiro  wrote:

> Hello
>
> Em 2022-09-19 20:42, madmurphy escreveu:
>
> > What FS needs to have:
> >
> > * Possibility of sharing a file while it is still being downloaded
> > (parts of it, of course)
>
> That would be interesting. It would be cool to have it swarm the pieces
> (like torrent)...
>
> > * Metadata must be editable and sharable
> > * Search keywords must be visible, editable, sharable (part of the
> > metadata?)
>
> Hummm. Ok. But would it lead to spammers?
>
> > * Introduction of a rating mechanism for files (against spam)
>
> +1
>
> > * Allow reverse search (i.e. chk-URI lookup)
>
> This might be troublesome. Imagine if some entity is looking for people
> who might donwload some special file (a planted file or an unauthorized
> file). That entity might try to locate the URI and then the users by
> looking at the pieces...
>
> > * Automatically and fully auto-unindex a file when it is missing
> > * Autoshare the dynamic content of a directory and update its index in
> > real time (e.g. if I "autoshare" the content of
> > /srv/filesharing/gnunet/madmurphy/, when I add foobar.txt to that
> > directory it must be automatically indexed - the opposite if I remove
> > it)
> > * Implement file statistics (download counter? last seen? etc.) - this
> > should allow the network to get rid easily of "lost" content
> > * Implement a NOT operator for search keywords (a tilde, "~"?)
> > * Implement an OR operator (a vertical bar, "|"? currently not writing
> > any operator equals OR, but we need an explicit OR operator if we want
> > to implement the next point)
> > * Allow parenthesis parsing for AND/OR/NOT operators; this will require
> > that operators can be followed by spaces (e.g. gnunet-search +required
> > '+' '(' optional1 '|' optional2 ')')
> > * The output filename must become optional in gnunet-download. If not
> > specified, the file/directory must be downloaded using the original
> > filename it was published with - if currently this information is not
> > always saved during publishing, then we must make sure that it is
> > always saved (this is part of the metadata too and must be editable)
>
> +1
>
> []s,
> Luis
>
>

The future of the file-sharing service

[madmurphy]

Hi all,
On Thu, Sep 15, 2022 at 2:55 PM Maxime Devos  wrote:

Myself, I'm very interested in the file-sharing service and DHT service --
I would like to implement (Guix) substitutes over GNUnet (I already sent a
POC implementation previously, but it wasn't a very nice implementation,
e.g. there was no fallback to regular http / https, that's why I started
scheme-gnunet, to make GNUnet easier to use from Guile.

I know that probably the FS service is not the priority at the moment, but
I was thinking that nothing forbids to imagine now how an ideal FS service
could look like. Time ago I had written down some of the features that I
believe a future FS module will need to implement. I paste my list here,
with the hope of leaving food for thought for a brain storming.

What FS needs to have:

   1. Possibility of sharing a file while it is still being downloaded
   (parts of it, of course)
   2. Metadata must be editable and sharable
   3. Search keywords must be visible, editable, sharable (part of the
   metadata?)
   4. Introduction of a rating mechanism for files (against spam)
   5. Allow reverse search (i.e. chk-URI lookup)
   6. Automatically and fully auto-unindex a file when it is missing
   7. Autoshare the dynamic content of a directory and update its index in
   real time (e.g. if I “autoshare” the content of
   /srv/filesharing/gnunet/madmurphy/, when I add foobar.txt to that
   directory it must be automatically indexed – the opposite if I remove it)
   8. Implement file statistics (download counter? last seen? etc.) – this
   should allow the network to get rid easily of “lost” content
   9. Implement a NOT operator for search keywords (a tilde, “~”?)
   10. Implement an OR operator (a vertical bar, “|”? currently not writing
   any operator equals OR, but we need an explicit OR operator if we want to
   implement the next point)
   11. Allow parenthesis parsing for AND/OR/NOT operators; this will
   require that operators can be followed by spaces (e.g. gnunet-search
   +required '+' '(' optional1 '|' optional2 ')')
   12. The output filename must become optional in gnunet-download. If not
   specified, the file/directory must be downloaded using the original
   filename it was published with – if currently this information is not
   always saved during publishing, then we must make sure that it is always
   saved (this is part of the metadata too and must be editable)

My two cents. Please do not hesitate to post your comments on these
proposals or add new ideas.

--madmurphy

Re: About GNUrl and cURL

[madmurphy]

No, libcurl.so is a symlink to libcurl.so.4.8.0. However my glue package
<https://aur.archlinux.org/packages/libcurl-gnutls.so> (a super-minimal
<https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=libcurl-gnutls.so>
one) now provides a libcurl-gnutls.so, which is a symlink to
libcurl-gnutls.so.4.8.0.

By the way, after the last commits GNUnet's configure script does not
detect libcurl-gnutls also with my glue package (config.log attached).
However I just discovered these two packages in that fantastic place that
is AUR: libcurl-gnutls-minimal-git
<https://aur.archlinux.org/packages/libcurl-gnutls-minimal-git>,
libcurl3-gnutls <https://aur.archlinux.org/packages/libcurl3-gnutls>. I
think I should definitely give them a try.

--madmurphy

On Thu, Sep 8, 2022 at 7:55 AM Martin Schanzenbach 
wrote:

> I checked the log and the test runs correctly.
> The condition
>
> CURLSSLSET_OK != curl_global_sslset(CURLSSLBACKEND_GNUTLS, NULL, ))
>
> is evaluated against "-lcurl" and it is false.
> Hence the library linked against (-lcurl) does not support GnuTLS.
> The detection works.
> Are you sure that your libcurl.so is linked against the
> libcurl-gnutls.so* files?
>
> BR
>
> Excerpts from madmurphy's message of 2022-09-08 06:49:00 +0100:
> > I tried again, just to be sure. Still get
> >
> > ...
> > HTTP Client:curl (OpenSSL)
> > ...
> >
> > My config.log attached.
> >
> > --madmurphy
> >
> > On Wed, Sep 7, 2022 at 8:17 PM Martin Schanzenbach <
> mschanzenb...@posteo.de>
> > wrote:
> >
> > > I am quite sure it works now as expected so you would need to provide
> me
> > > with the config.log to debug.
> > > Maybe your link now points to the "Normal" curl because of the testing?
> > >
> > > Excerpts from madmurphy's message of 2022-09-07 18:55:19 +0100:
> > > > I now commited a programmatic check for GnuTLS. Try it out. It
> should not
> > > > require your fix.
> > > >
> > > > Mmm without my trick the configure script still prints
> > > >
> > > > ...
> > > > HTTP Client:curl (OpenSSL)
> > > > ...
> > > >
> > > > --madmurphy
> > > >
> > > > On Wed, Sep 7, 2022 at 3:44 PM Schanzenbach, Martin <
> > > mschanzenb...@posteo.de>
> > > > wrote:
> > > >
> > > > >
> > > > >
> > > > > > On 7. Sep 2022, at 14:59, madmurphy 
> wrote:
> > > > > >
> > > > > > Hi Martin,
> > > > > >
> > > > > > That means, if you can find out how the packages linked against
> > > > > libcurl-compat or libcurl-gnutls are built from source, you can do
> the
> > > same
> > > > > with the gnunet package.
> > > > > > The packages in the official repositories that explicitly require
> > > > > libcurl-gnutls (and not simply libcurl) are spotify-launcher
> (PKGBUILD
> > > /
> > > > > package source code) and steam-native-runtime (PKGBUILD / no source
> > > code
> > > > > here, it's just glue). But it is a mystery to me how they would
> find
> > > out.
> > > > >
> > > > > They probably do not distinguish between the two versions. The
> package
> > > > > build simply makes sure that during linking, the correct link is
> set.
> > > > >
> > > > > >
> > > > > > Ah look here:
> > > > >
> > >
> https://github.com/archlinux/svntogit-packages/blob/packages/curl/trunk/PKGBUILD#L94
> > > > > > The curl-compat package does link libcurl.so against the
> versioned
> > > files.
> > > > > > And curl-gnutls does the same:
> > > > >
> > >
> https://github.com/archlinux/svntogit-packages/blob/packages/curl/trunk/PKGBUILD#L94
> > > > > > That libcurl-compat package compiles curl with different build
> > > settings,
> > > > > then renames libcurl.so.4.8.0 to libcurl-compat.so.4.8.0, and
> finally
> > > links
> > > > > libcurl.so.3, libcurl.so.4.0.0, libcurl.so.4.1.0, libcurl.so.4.2.0,
> > > > > libcurl.so.4.3.0, libcurl.so.4.4.0, libcurl.so.4.5.0,
> libcurl.so.4.6.0,
> > > > > libcurl.so.4.7.0 to libcurl-compat.so.4.8.0 (these version are all
> old
> > > > > versions, and the files libcurl.so, libcurl.so.4 and
> libcurl.so.4.8.0
> > > > 

Re: About GNUrl and cURL

[madmurphy]

I tried again, just to be sure. Still get

...
HTTP Client:curl (OpenSSL)
...

My config.log attached.

--madmurphy

On Wed, Sep 7, 2022 at 8:17 PM Martin Schanzenbach 
wrote:

> I am quite sure it works now as expected so you would need to provide me
> with the config.log to debug.
> Maybe your link now points to the "Normal" curl because of the testing?
>
> Excerpts from madmurphy's message of 2022-09-07 18:55:19 +0100:
> > I now commited a programmatic check for GnuTLS. Try it out. It should not
> > require your fix.
> >
> > Mmm without my trick the configure script still prints
> >
> > ...
> > HTTP Client:curl (OpenSSL)
> > ...
> >
> > --madmurphy
> >
> > On Wed, Sep 7, 2022 at 3:44 PM Schanzenbach, Martin <
> mschanzenb...@posteo.de>
> > wrote:
> >
> > >
> > >
> > > > On 7. Sep 2022, at 14:59, madmurphy  wrote:
> > > >
> > > > Hi Martin,
> > > >
> > > > That means, if you can find out how the packages linked against
> > > libcurl-compat or libcurl-gnutls are built from source, you can do the
> same
> > > with the gnunet package.
> > > > The packages in the official repositories that explicitly require
> > > libcurl-gnutls (and not simply libcurl) are spotify-launcher (PKGBUILD
> /
> > > package source code) and steam-native-runtime (PKGBUILD / no source
> code
> > > here, it's just glue). But it is a mystery to me how they would find
> out.
> > >
> > > They probably do not distinguish between the two versions. The package
> > > build simply makes sure that during linking, the correct link is set.
> > >
> > > >
> > > > Ah look here:
> > >
> https://github.com/archlinux/svntogit-packages/blob/packages/curl/trunk/PKGBUILD#L94
> > > > The curl-compat package does link libcurl.so against the versioned
> files.
> > > > And curl-gnutls does the same:
> > >
> https://github.com/archlinux/svntogit-packages/blob/packages/curl/trunk/PKGBUILD#L94
> > > > That libcurl-compat package compiles curl with different build
> settings,
> > > then renames libcurl.so.4.8.0 to libcurl-compat.so.4.8.0, and finally
> links
> > > libcurl.so.3, libcurl.so.4.0.0, libcurl.so.4.1.0, libcurl.so.4.2.0,
> > > libcurl.so.4.3.0, libcurl.so.4.4.0, libcurl.so.4.5.0, libcurl.so.4.6.0,
> > > libcurl.so.4.7.0 to libcurl-compat.so.4.8.0 (these version are all old
> > > versions, and the files libcurl.so, libcurl.so.4 and libcurl.so.4.8.0
> > > remain linked to the binary shipped by the original curl package).
> > > >
> > > > That libcurl-gnutls package does exactly the same, but the basename
> of
> > > the symlinks becomes libcurl-gnutls.so.* instead of simply
> libcurl.so.*.
> > > >
> > > > FYI I updated the detection logic again. You may check if that works
> for
> > > you know.
> > > > If I try to build the last GNUnet commit with libcurl-gnutls from the
> > > official Arch repository I still get
> > > >
> > > > ...
> > > > HTTP Client:curl (OpenSSL)
> > > > ...
> > > >
> > > > while if I use my hacked libcurl-gnutls I get
> > > >
> > > > ...
> > > > HTTP Client:curl-gnutls
> > > > ...
> > > >
> > > > I think I found a solution. I will publish a glue package on AUR
> named
> > > libcurl-gnutls.so, which will depend on the official libcurl-gnutls,
> and on
> > > which GNUnet will depend. All this glue package will do is simply
> creating
> > > an unversioned symlink.
> > >
> > > Yeah.. curl-config just seems to be a bash script where the supported
> > > backends are hardcoded when it is "compiled".
> > > So even if you install curl-gnutls it will still say "openssl"...
> great.
> > >
> > > I now commited a programmatic check for GnuTLS. Try it out. It should
> not
> > > require your fix.
> > > No idea if anybody actually ships curl with multiple TLS backends, but
> we
> > > check on runtime anyway so its fine.
> > > We may want to set the backend explicity maybe with
> curl_global_sslset...
> > >
> > > BR
> > > Martin
> > >
> > > >
> > > > --madmurphy
> > > >
> > > >
> > > > On Wed, Sep 7, 2022 at 7:33 AM Martin Schanz

Re: About GNUrl and cURL

[madmurphy]

I now commited a programmatic check for GnuTLS. Try it out. It should not
require your fix.

Mmm without my trick the configure script still prints

...
HTTP Client:curl (OpenSSL)
...

--madmurphy

On Wed, Sep 7, 2022 at 3:44 PM Schanzenbach, Martin 
wrote:

>
>
> > On 7. Sep 2022, at 14:59, madmurphy  wrote:
> >
> > Hi Martin,
> >
> > That means, if you can find out how the packages linked against
> libcurl-compat or libcurl-gnutls are built from source, you can do the same
> with the gnunet package.
> > The packages in the official repositories that explicitly require
> libcurl-gnutls (and not simply libcurl) are spotify-launcher (PKGBUILD /
> package source code) and steam-native-runtime (PKGBUILD / no source code
> here, it's just glue). But it is a mystery to me how they would find out.
>
> They probably do not distinguish between the two versions. The package
> build simply makes sure that during linking, the correct link is set.
>
> >
> > Ah look here:
> https://github.com/archlinux/svntogit-packages/blob/packages/curl/trunk/PKGBUILD#L94
> > The curl-compat package does link libcurl.so against the versioned files.
> > And curl-gnutls does the same:
> https://github.com/archlinux/svntogit-packages/blob/packages/curl/trunk/PKGBUILD#L94
> > That libcurl-compat package compiles curl with different build settings,
> then renames libcurl.so.4.8.0 to libcurl-compat.so.4.8.0, and finally links
> libcurl.so.3, libcurl.so.4.0.0, libcurl.so.4.1.0, libcurl.so.4.2.0,
> libcurl.so.4.3.0, libcurl.so.4.4.0, libcurl.so.4.5.0, libcurl.so.4.6.0,
> libcurl.so.4.7.0 to libcurl-compat.so.4.8.0 (these version are all old
> versions, and the files libcurl.so, libcurl.so.4 and libcurl.so.4.8.0
> remain linked to the binary shipped by the original curl package).
> >
> > That libcurl-gnutls package does exactly the same, but the basename of
> the symlinks becomes libcurl-gnutls.so.* instead of simply libcurl.so.*.
> >
> > FYI I updated the detection logic again. You may check if that works for
> you know.
> > If I try to build the last GNUnet commit with libcurl-gnutls from the
> official Arch repository I still get
> >
> > ...
> > HTTP Client:curl (OpenSSL)
> > ...
> >
> > while if I use my hacked libcurl-gnutls I get
> >
> > ...
> > HTTP Client:curl-gnutls
> > ...
> >
> > I think I found a solution. I will publish a glue package on AUR named
> libcurl-gnutls.so, which will depend on the official libcurl-gnutls, and on
> which GNUnet will depend. All this glue package will do is simply creating
> an unversioned symlink.
>
> Yeah.. curl-config just seems to be a bash script where the supported
> backends are hardcoded when it is "compiled".
> So even if you install curl-gnutls it will still say "openssl"... great.
>
> I now commited a programmatic check for GnuTLS. Try it out. It should not
> require your fix.
> No idea if anybody actually ships curl with multiple TLS backends, but we
> check on runtime anyway so its fine.
> We may want to set the backend explicity maybe with curl_global_sslset...
>
> BR
> Martin
>
> >
> > --madmurphy
> >
> >
> > On Wed, Sep 7, 2022 at 7:33 AM Martin Schanzenbach <
> mschanzenb...@posteo.de> wrote:
> > FYI I updated the detection logic again. You may check if that works for
> > you know.
> > Know that even if it detected "curl-openssl" for you the last time, it
> > probably was correctly linked against the "drop-in" libcurl-gnutls.
> > We just were not able to detect that.
> >
> > BR
> >
> > Excerpts from Martin Schanzenbach's message of 2022-09-07 06:06:52 +:
> > > Excerpts from madmurphy's message of 2022-09-06 22:17:53 +0100:
> > > > Okay, about the libcurl-gnutls package, Martin was right. If I add
> this
> > > > line to the PKGBUILD of that package,
> > > >
> > > > ln -s libcurl-gnutls.so.4.8.0 "${pkgdir}"/usr/lib/libcurl-gnutls.so
> > > >
> > > > Everything goes well in GNUnet and the configure script prints
> > > >
> > > > ...
> > > > HTTP Client:curl-gnutls
> > > > ...
> > > >
> > > > Now the question is what to do. In theory I could publish my own
> version of
> > > > libcurl-gnutls on AUR with only that line added, and make GNUnet
> depend on
> > > > it. But I wonder why Arch developers did that. My guess is that for
> > > > creating the libcurl-gnutls package they copied and hacked 

Re: A more general question about curl

[madmurphy]

I never used the curl API, so I don't know what the multi interface is, but
if I remember correctly wget2 introduced non-blocking sockets. That's all I
know. I did not find a lot of info on Google, except maybe for this email
on gnutls mailing list:
https://lists.gnutls.org/pipermail/gnutls-devel/2019-June/014051.html

--madmurphy

On Wed, Sep 7, 2022 at 2:54 PM Schanzenbach, Martin 
wrote:

> We need a non-blocking API such as curl_multi.
> Last time I checked, libwget2 does not have that.
>
> BR
>
> > On 7. Sep 2022, at 15:46, madmurphy  wrote:
> >
> > I don't know all the reasons behind using curl and all GNUnet's
> requirements, but have you guys thought about switching to wget2? It is a
> GNU package and has a nice library (libwget). It supports GNU TLS natively,
> it is supposed to download faster than curl, and if a minor feature is
> missing it might be an opportunity to make libwget grow.
> >
> > A comparison table (by curl):
> >
> > https://curl.se/docs/comparison-table.html
> >
> > --madmurphy
>
>

A more general question about curl

[madmurphy]

I don't know all the reasons behind using curl and all GNUnet's
requirements, but have you guys thought about switching to wget2? It is a
GNU package and has a nice library (libwget). It supports GNU TLS natively,
it is supposed to download faster than curl, and if a minor feature is
missing it might be an opportunity to make libwget grow.

A comparison table (by curl):

https://curl.se/docs/comparison-table.html

--madmurphy

Re: About GNUrl and cURL

[madmurphy]

Hi Martin,

That means, if you can find out how the packages linked against
libcurl-compat or libcurl-gnutls are built from source, you can do the same
with the gnunet package.

The packages in the official repositories that explicitly require
libcurl-gnutls (and not simply libcurl) are spotify-launcher
<https://archlinux.org/packages/community/x86_64/spotify-launcher/> (
PKGBUILD
<https://github.com/archlinux/svntogit-community/blob/packages/spotify-launcher/trunk/PKGBUILD>
/ package source code
<https://github.com/kpcyrd/spotify-launcher/archive/refs/tags/v0.3.0.tar.gz>)
and steam-native-runtime
<https://archlinux.org/packages/multilib/x86_64/steam-native-runtime/> (
PKGBUILD
<https://github.com/archlinux/svntogit-community/blob/packages/steam-native-runtime/trunk/PKGBUILD>
/ no source code here, it's just glue). But it is a mystery to me how they
would find out.

Ah look here:
https://github.com/archlinux/svntogit-packages/blob/packages/curl/trunk/PKGBUILD#L94
The curl-compat package does link libcurl.so against the versioned files.
And curl-gnutls does the same:
https://github.com/archlinux/svntogit-packages/blob/packages/curl/trunk/PKGBUILD#L94

That libcurl-compat package compiles curl with different build settings,
then renames libcurl.so.4.8.0 to libcurl-compat.so.4.8.0, and finally links
libcurl.so.3, libcurl.so.4.0.0, libcurl.so.4.1.0, libcurl.so.4.2.0,
libcurl.so.4.3.0, libcurl.so.4.4.0, libcurl.so.4.5.0, libcurl.so.4.6.0,
libcurl.so.4.7.0 to libcurl-compat.so.4.8.0 (these version are all old
versions, and the files libcurl.so, libcurl.so.4 and libcurl.so.4.8.0
remain linked to the binary shipped by the original curl package).

That libcurl-gnutls package does exactly the same, but the basename of the
symlinks becomes libcurl-gnutls.so.* instead of simply libcurl.so.*.

FYI I updated the detection logic again. You may check if that works for
you know.

If I try to build the last GNUnet commit
<https://git.gnunet.org/gnunet.git/tree/?id=6745bed70af20a11aee5e0c8f6e7bd8479031e9e>
with libcurl-gnutls from the official Arch repository I still get

...
HTTP Client:curl (OpenSSL)
...

while if I use my hacked libcurl-gnutls I get

...
HTTP Client:curl-gnutls
...

I think I found a solution. I will publish a glue package on AUR named
libcurl-gnutls.so, which will depend on the official libcurl-gnutls, and on
which GNUnet will depend. All this glue package will do is simply creating
an unversioned symlink.

--madmurphy

On Wed, Sep 7, 2022 at 7:33 AM Martin Schanzenbach 
wrote:

> FYI I updated the detection logic again. You may check if that works for
> you know.
> Know that even if it detected "curl-openssl" for you the last time, it
> probably was correctly linked against the "drop-in" libcurl-gnutls.
> We just were not able to detect that.
>
> BR
>
> Excerpts from Martin Schanzenbach's message of 2022-09-07 06:06:52 +:
> > Excerpts from madmurphy's message of 2022-09-06 22:17:53 +0100:
> > > Okay, about the libcurl-gnutls package, Martin was right. If I add this
> > > line to the PKGBUILD of that package,
> > >
> > > ln -s libcurl-gnutls.so.4.8.0 "${pkgdir}"/usr/lib/libcurl-gnutls.so
> > >
> > > Everything goes well in GNUnet and the configure script prints
> > >
> > > ...
> > > HTTP Client:curl-gnutls
> > > ...
> > >
> > > Now the question is what to do. In theory I could publish my own
> version of
> > > libcurl-gnutls on AUR with only that line added, and make GNUnet
> depend on
> > > it. But I wonder why Arch developers did that. My guess is that for
> > > creating the libcurl-gnutls package they copied and hacked the section
> of
> > > the PKGBUILD that builds libcurl-compat
> > > <https://archlinux.org/packages/core/x86_64/libcurl-compat/>, which
> is a
> > > glue package that also does not ship the unversioned .so file
> > > <https://archlinux.org/packages/core/x86_64/libcurl-compat/files/>.
> Who
> > > knows…
> > >
> >
> > Ah look here:
> https://github.com/archlinux/svntogit-packages/blob/packages/curl/trunk/PKGBUILD#L94
> > The curl-compat package does link libcurl.so against the versioned
> > files.
> > And curl-gnutls does the same:
> https://github.com/archlinux/svntogit-packages/blob/packages/curl/trunk/PKGBUILD#L94
> >
> > So, this would actually confirm my initial thoughts that those are
> > drop-in replacements and that we should not check for libcurl-gnutls at
> > all.
> > I have no idea how to "detect" the version of curl in this case.
> > But, I also do not think it really matters.
> > So maybe we should just remove the

Re: About GNUrl and cURL

[madmurphy]

Okay, about the libcurl-gnutls package, Martin was right. If I add this
line to the PKGBUILD of that package,

ln -s libcurl-gnutls.so.4.8.0 "${pkgdir}"/usr/lib/libcurl-gnutls.so

Everything goes well in GNUnet and the configure script prints

...
HTTP Client:curl-gnutls
...

Now the question is what to do. In theory I could publish my own version of
libcurl-gnutls on AUR with only that line added, and make GNUnet depend on
it. But I wonder why Arch developers did that. My guess is that for
creating the libcurl-gnutls package they copied and hacked the section of
the PKGBUILD that builds libcurl-compat
<https://archlinux.org/packages/core/x86_64/libcurl-compat/>, which is a
glue package that also does not ship the unversioned .so file
<https://archlinux.org/packages/core/x86_64/libcurl-compat/files/>. Who
knows…

Jacki, what do you suggest? The PKGBUILD of libcurl-gnutls attached to this
email works well for GNUnet. But for publishing on AUR we would need to
rename it in some way.

--madmurphy

On Tue, Sep 6, 2022 at 9:06 PM Martin Schanzenbach 
wrote:

> Excerpts from Christian Grothoff's message of 2022-09-06 21:34:45 +0200:
> > On 9/6/22 14:43, madmurphy wrote:
> > > Just out of curiosity, why do I get
> > >
> > > gstreamer:  no
> >
> > You need also certain related gstreamer libraries
> > (gstreamer-plugins-base or something like that) before gstreamer is
> > considered "complete enough" to work for GNUnet.
> >
>
> I have to disagree that this is what configure.ac checks. I invite you
> to investigate as well. I am at a loss as what exactly the logic
> currently is. It sais "gstreamer no" for me, too, but conversation is
> built with the "gst" "backend". Whatever that means. Anyway.
>
> > -Christian
>


curl-7.85.0-1.src.tar.gz
Description: application/gzip

Re: About GNUrl and cURL

[madmurphy]

Okay, the first thing I notice in the list of the files installed by
libcurl-gnutls
<https://archlinux.org/packages/core/x86_64/libcurl-gnutls/files/> is that
there is no libcurl-gnutls.pc file. Do you think there is a way to check
without passing through pkgconf?

On Tue, Sep 6, 2022 at 1:47 PM madmurphy  wrote:

> I will try to check the configure.ac file. I don't get an error btw, only
> that "curl-openssl" http client...
>
> On Tue, Sep 6, 2022 at 1:45 PM Schanzenbach, Martin <
> mschanzenb...@posteo.de> wrote:
>
>> Yes. I *think* I have found a better way to check for this and it should
>> be compatible with debian as well.
>>
>> BR
>>
>> > On 6. Sep 2022, at 12:32, madmurphy  wrote:
>> >
>> > Hi Martin,
>> >
>> > If "normal" curl is found, the check for curl-gnutls is skipped.
>> > I guess we should prefer curl-gnutls.
>> > Yes, libcurl-gnutls should be checked first. Also because apparently
>> Arch is not the only distro that has a split package of cURL linked against
>> GNU TLS…
>> >
>> > --madmurphy
>> >
>> >
>> > On Tue, Sep 6, 2022 at 9:59 AM Schanzenbach, Martin <
>> mschanzenb...@posteo.de> wrote:
>> > Ah sorry I just checked:
>> > If "normal" curl is found, the check for curl-gnutls is skipped.
>> > I guess we should prefer curl-gnutls.
>> >
>> > Br
>> >
>> > > On 6. Sep 2022, at 10:51, Schanzenbach, Martin <
>> mschanzenb...@posteo.de> wrote:
>> > >
>> > > Hi,
>> > >
>> > > check your config.log and check for the test against libcurl-gnutls.
>> > > There should be test somewhere in there that fails for reasons.
>> > >
>> > > BR
>> > > Martin
>> > >
>> > >> On 5. Sep 2022, at 20:21, madmurphy  wrote:
>> > >>
>> > >> Hi Christian,
>> > >>
>> > >> I tried to run ./configure twice. The first time I had installed on
>> my machine curl (which is linked against openssl) and gnurl, but I did not
>> have libcurl-gnutls installed. Under these conditions at the end of the
>> configure script I got printed:
>> > >>
>> > >> ...
>> > >> HTTP Client:gnurl
>> > >> ...
>> > >>
>> > >> The second time I had curl and libcurl-gnutls installed, but gnurl
>> was not installed. Under these conditions at the end of the configure
>> script I got printed:
>> > >>
>> > >> ...
>> > >> HTTP Client:curl-openssl
>> > >> ...
>> > >>
>> > >> For some reasons in the second scenario the configure script sees
>> the curl package but does not see libcurl-gnutls and so the latter is not
>> used (I guess). The files shipped by the latter are:
>> > >>
>> > >> /usr/
>> > >> /usr/lib/
>> > >> /usr/lib/libcurl-gnutls.so.3
>> > >> /usr/lib/libcurl-gnutls.so.4
>> > >> /usr/lib/libcurl-gnutls.so.4.0.0
>> > >> /usr/lib/libcurl-gnutls.so.4.1.0
>> > >> /usr/lib/libcurl-gnutls.so.4.2.0
>> > >> /usr/lib/libcurl-gnutls.so.4.3.0
>> > >> /usr/lib/libcurl-gnutls.so.4.4.0
>> > >> /usr/lib/libcurl-gnutls.so.4.5.0
>> > >> /usr/lib/libcurl-gnutls.so.4.6.0
>> > >> /usr/lib/libcurl-gnutls.so.4.7.0
>> > >> /usr/lib/libcurl-gnutls.so.4.8.0
>> > >> /usr/share/
>> > >> /usr/share/licenses/
>> > >> /usr/share/licenses/libcurl-gnutls
>> > >>
>> > >> How can we ensure that the configure script correctly sees the
>> libcurl-gnutls package?
>> > >>
>> > >> I paste below both complete outputs.
>> > >>
>> > >> Installed: curl, gnurl
>> > >> Not installed: libcurl-gnutls:
>> > >>
>> > >> $ ./configure
>> > >> checking build system type... x86_64-pc-linux-gnu
>> > >> checking host system type... x86_64-pc-linux-gnu
>> > >> checking target system type... x86_64-pc-linux-gnu
>> > >> checking for a BSD-compatible install... /usr/bin/install -c
>> > >> checking whether build environment is sane... yes
>> > >> checking for a race-free mkdir -p... /usr/bin/mkdir -p
>> > >> checking for gawk... gawk
>> > >> checking whether make sets

Re: About GNUrl and cURL

[madmurphy]

No, sorry, I meant it doesn't work :-( Too much expectations, and I read
tls instead of ssl...

On Tue, Sep 6, 2022 at 1:43 PM madmurphy  wrote:

> Alright, it works now :-)
>
> HTTP Client:curl-openssl
>
> Just out of curiosity, why do I get
>
> gstreamer:  no
>
> ?
>
> On Tue, Sep 6, 2022 at 11:35 AM Martin Schanzenbach <
> mschanzenb...@posteo.de> wrote:
>
>> I cleaned up the curl detection in
>> afea0eea1ecfa41150fdf9ee052acac75eee6534 (next release 0.17.6).
>> Feel free to try.
>> It should be a bit more intelligent with respect to curl-gnutls
>> detection.
>>
>> BR
>>
>> Excerpts from Schanzenbach, Martin's message of 2022-09-06 08:59:23 +:
>> > Ah sorry I just checked:
>> > If "normal" curl is found, the check for curl-gnutls is skipped.
>> > I guess we should prefer curl-gnutls.
>> >
>> > Br
>> >
>> > > On 6. Sep 2022, at 10:51, Schanzenbach, Martin <
>> mschanzenb...@posteo.de> wrote:
>> > >
>> > > Hi,
>> > >
>> > > check your config.log and check for the test against libcurl-gnutls.
>> > > There should be test somewhere in there that fails for reasons.
>> > >
>> > > BR
>> > > Martin
>> > >
>> > >> On 5. Sep 2022, at 20:21, madmurphy  wrote:
>> > >>
>> > >> Hi Christian,
>> > >>
>> > >> I tried to run ./configure twice. The first time I had installed on
>> my machine curl (which is linked against openssl) and gnurl, but I did not
>> have libcurl-gnutls installed. Under these conditions at the end of the
>> configure script I got printed:
>> > >>
>> > >> ...
>> > >> HTTP Client:gnurl
>> > >> ...
>> > >>
>> > >> The second time I had curl and libcurl-gnutls installed, but gnurl
>> was not installed. Under these conditions at the end of the configure
>> script I got printed:
>> > >>
>> > >> ...
>> > >> HTTP Client:curl-openssl
>> > >> ...
>> > >>
>> > >> For some reasons in the second scenario the configure script sees
>> the curl package but does not see libcurl-gnutls and so the latter is not
>> used (I guess). The files shipped by the latter are:
>> > >>
>> > >> /usr/
>> > >> /usr/lib/
>> > >> /usr/lib/libcurl-gnutls.so.3
>> > >> /usr/lib/libcurl-gnutls.so.4
>> > >> /usr/lib/libcurl-gnutls.so.4.0.0
>> > >> /usr/lib/libcurl-gnutls.so.4.1.0
>> > >> /usr/lib/libcurl-gnutls.so.4.2.0
>> > >> /usr/lib/libcurl-gnutls.so.4.3.0
>> > >> /usr/lib/libcurl-gnutls.so.4.4.0
>> > >> /usr/lib/libcurl-gnutls.so.4.5.0
>> > >> /usr/lib/libcurl-gnutls.so.4.6.0
>> > >> /usr/lib/libcurl-gnutls.so.4.7.0
>> > >> /usr/lib/libcurl-gnutls.so.4.8.0
>> > >> /usr/share/
>> > >> /usr/share/licenses/
>> > >> /usr/share/licenses/libcurl-gnutls
>> > >>
>> > >> How can we ensure that the configure script correctly sees the
>> libcurl-gnutls package?
>> > >>
>> > >> I paste below both complete outputs.
>> > >>
>> > >> Installed: curl, gnurl
>> > >> Not installed: libcurl-gnutls:
>> > >>
>> > >> $ ./configure
>> > >> checking build system type... x86_64-pc-linux-gnu
>> > >> checking host system type... x86_64-pc-linux-gnu
>> > >> checking target system type... x86_64-pc-linux-gnu
>> > >> checking for a BSD-compatible install... /usr/bin/install -c
>> > >> checking whether build environment is sane... yes
>> > >> checking for a race-free mkdir -p... /usr/bin/mkdir -p
>> > >> checking for gawk... gawk
>> > >> checking whether make sets $(MAKE)... yes
>> > >> checking whether make supports nested variables... yes
>> > >> checking whether UID '1000' is supported by ustar format... yes
>> > >> checking whether GID '1000' is supported by ustar format... yes
>> > >> checking how to create a ustar tar archive... gnutar
>> > >> checking whether make supports nested variables... (cached) yes
>> > >> checking for gawk... (cached) gawk
>> > >> checking for gcc... gcc
>

Re: About GNUrl and cURL

[madmurphy]

Alright, it works now :-)

HTTP Client:curl-openssl

Just out of curiosity, why do I get

gstreamer:  no

?

On Tue, Sep 6, 2022 at 11:35 AM Martin Schanzenbach 
wrote:

> I cleaned up the curl detection in
> afea0eea1ecfa41150fdf9ee052acac75eee6534 (next release 0.17.6).
> Feel free to try.
> It should be a bit more intelligent with respect to curl-gnutls
> detection.
>
> BR
>
> Excerpts from Schanzenbach, Martin's message of 2022-09-06 08:59:23 +:
> > Ah sorry I just checked:
> > If "normal" curl is found, the check for curl-gnutls is skipped.
> > I guess we should prefer curl-gnutls.
> >
> > Br
> >
> > > On 6. Sep 2022, at 10:51, Schanzenbach, Martin <
> mschanzenb...@posteo.de> wrote:
> > >
> > > Hi,
> > >
> > > check your config.log and check for the test against libcurl-gnutls.
> > > There should be test somewhere in there that fails for reasons.
> > >
> > > BR
> > > Martin
> > >
> > >> On 5. Sep 2022, at 20:21, madmurphy  wrote:
> > >>
> > >> Hi Christian,
> > >>
> > >> I tried to run ./configure twice. The first time I had installed on
> my machine curl (which is linked against openssl) and gnurl, but I did not
> have libcurl-gnutls installed. Under these conditions at the end of the
> configure script I got printed:
> > >>
> > >> ...
> > >> HTTP Client:gnurl
> > >> ...
> > >>
> > >> The second time I had curl and libcurl-gnutls installed, but gnurl
> was not installed. Under these conditions at the end of the configure
> script I got printed:
> > >>
> > >> ...
> > >> HTTP Client:curl-openssl
> > >> ...
> > >>
> > >> For some reasons in the second scenario the configure script sees the
> curl package but does not see libcurl-gnutls and so the latter is not used
> (I guess). The files shipped by the latter are:
> > >>
> > >> /usr/
> > >> /usr/lib/
> > >> /usr/lib/libcurl-gnutls.so.3
> > >> /usr/lib/libcurl-gnutls.so.4
> > >> /usr/lib/libcurl-gnutls.so.4.0.0
> > >> /usr/lib/libcurl-gnutls.so.4.1.0
> > >> /usr/lib/libcurl-gnutls.so.4.2.0
> > >> /usr/lib/libcurl-gnutls.so.4.3.0
> > >> /usr/lib/libcurl-gnutls.so.4.4.0
> > >> /usr/lib/libcurl-gnutls.so.4.5.0
> > >> /usr/lib/libcurl-gnutls.so.4.6.0
> > >> /usr/lib/libcurl-gnutls.so.4.7.0
> > >> /usr/lib/libcurl-gnutls.so.4.8.0
> > >> /usr/share/
> > >> /usr/share/licenses/
> > >> /usr/share/licenses/libcurl-gnutls
> > >>
> > >> How can we ensure that the configure script correctly sees the
> libcurl-gnutls package?
> > >>
> > >> I paste below both complete outputs.
> > >>
> > >> Installed: curl, gnurl
> > >> Not installed: libcurl-gnutls:
> > >>
> > >> $ ./configure
> > >> checking build system type... x86_64-pc-linux-gnu
> > >> checking host system type... x86_64-pc-linux-gnu
> > >> checking target system type... x86_64-pc-linux-gnu
> > >> checking for a BSD-compatible install... /usr/bin/install -c
> > >> checking whether build environment is sane... yes
> > >> checking for a race-free mkdir -p... /usr/bin/mkdir -p
> > >> checking for gawk... gawk
> > >> checking whether make sets $(MAKE)... yes
> > >> checking whether make supports nested variables... yes
> > >> checking whether UID '1000' is supported by ustar format... yes
> > >> checking whether GID '1000' is supported by ustar format... yes
> > >> checking how to create a ustar tar archive... gnutar
> > >> checking whether make supports nested variables... (cached) yes
> > >> checking for gawk... (cached) gawk
> > >> checking for gcc... gcc
> > >> checking whether the C compiler works... yes
> > >> checking for C compiler default output file name... a.out
> > >> checking for suffix of executables...
> > >> checking whether we are cross compiling... no
> > >> checking for suffix of object files... o
> > >> checking whether the compiler supports GNU C... yes
> > >> checking whether gcc accepts -g... yes
> > >> checking for gcc option to enable C11 features... none needed
> > >> checking whether gcc understands -c and -o together

Re: About GNUrl and cURL

[madmurphy]

In the meanwhile I have asked <https://bugs.archlinux.org/task/75824> Arch
developers to make sure that a libcurl-gnutls.pc file gets installed…

--madmurphy

On Tue, Sep 6, 2022 at 2:03 PM madmurphy  wrote:

> Okay, the first thing I notice in the list of the files installed by
> libcurl-gnutls
> <https://archlinux.org/packages/core/x86_64/libcurl-gnutls/files/> is
> that there is no libcurl-gnutls.pc file. Do you think there is a way to
> check without passing through pkgconf?
>
> On Tue, Sep 6, 2022 at 1:47 PM madmurphy  wrote:
>
>> I will try to check the configure.ac file. I don't get an error btw,
>> only that "curl-openssl" http client...
>>
>> On Tue, Sep 6, 2022 at 1:45 PM Schanzenbach, Martin <
>> mschanzenb...@posteo.de> wrote:
>>
>>> Yes. I *think* I have found a better way to check for this and it should
>>> be compatible with debian as well.
>>>
>>> BR
>>>
>>> > On 6. Sep 2022, at 12:32, madmurphy  wrote:
>>> >
>>> > Hi Martin,
>>> >
>>> > If "normal" curl is found, the check for curl-gnutls is skipped.
>>> > I guess we should prefer curl-gnutls.
>>> > Yes, libcurl-gnutls should be checked first. Also because apparently
>>> Arch is not the only distro that has a split package of cURL linked against
>>> GNU TLS…
>>> >
>>> > --madmurphy
>>> >
>>> >
>>> > On Tue, Sep 6, 2022 at 9:59 AM Schanzenbach, Martin <
>>> mschanzenb...@posteo.de> wrote:
>>> > Ah sorry I just checked:
>>> > If "normal" curl is found, the check for curl-gnutls is skipped.
>>> > I guess we should prefer curl-gnutls.
>>> >
>>> > Br
>>> >
>>> > > On 6. Sep 2022, at 10:51, Schanzenbach, Martin <
>>> mschanzenb...@posteo.de> wrote:
>>> > >
>>> > > Hi,
>>> > >
>>> > > check your config.log and check for the test against libcurl-gnutls.
>>> > > There should be test somewhere in there that fails for reasons.
>>> > >
>>> > > BR
>>> > > Martin
>>> > >
>>> > >> On 5. Sep 2022, at 20:21, madmurphy  wrote:
>>> > >>
>>> > >> Hi Christian,
>>> > >>
>>> > >> I tried to run ./configure twice. The first time I had installed on
>>> my machine curl (which is linked against openssl) and gnurl, but I did not
>>> have libcurl-gnutls installed. Under these conditions at the end of the
>>> configure script I got printed:
>>> > >>
>>> > >> ...
>>> > >> HTTP Client:gnurl
>>> > >> ...
>>> > >>
>>> > >> The second time I had curl and libcurl-gnutls installed, but gnurl
>>> was not installed. Under these conditions at the end of the configure
>>> script I got printed:
>>> > >>
>>> > >> ...
>>> > >> HTTP Client:curl-openssl
>>> > >> ...
>>> > >>
>>> > >> For some reasons in the second scenario the configure script sees
>>> the curl package but does not see libcurl-gnutls and so the latter is not
>>> used (I guess). The files shipped by the latter are:
>>> > >>
>>> > >> /usr/
>>> > >> /usr/lib/
>>> > >> /usr/lib/libcurl-gnutls.so.3
>>> > >> /usr/lib/libcurl-gnutls.so.4
>>> > >> /usr/lib/libcurl-gnutls.so.4.0.0
>>> > >> /usr/lib/libcurl-gnutls.so.4.1.0
>>> > >> /usr/lib/libcurl-gnutls.so.4.2.0
>>> > >> /usr/lib/libcurl-gnutls.so.4.3.0
>>> > >> /usr/lib/libcurl-gnutls.so.4.4.0
>>> > >> /usr/lib/libcurl-gnutls.so.4.5.0
>>> > >> /usr/lib/libcurl-gnutls.so.4.6.0
>>> > >> /usr/lib/libcurl-gnutls.so.4.7.0
>>> > >> /usr/lib/libcurl-gnutls.so.4.8.0
>>> > >> /usr/share/
>>> > >> /usr/share/licenses/
>>> > >> /usr/share/licenses/libcurl-gnutls
>>> > >>
>>> > >> How can we ensure that the configure script correctly sees the
>>> libcurl-gnutls package?
>>> > >>
>>> > >> I paste below both complete outputs.
>>> > >>
>>> > >> Installed: curl, gnurl
>>> > >> Not installed: l

Re: About GNUrl and cURL

[madmurphy]

Hi Martin,

If "normal" curl is found, the check for curl-gnutls is skipped.
I guess we should prefer curl-gnutls.

Yes, libcurl-gnutls should be checked first. Also because apparently Arch
is not the only distro
<https://packages.debian.org/search?keywords=libcurl3-gnutls> that has a
split package of cURL linked against GNU TLS…

--madmurphy

On Tue, Sep 6, 2022 at 9:59 AM Schanzenbach, Martin 
wrote:

> Ah sorry I just checked:
> If "normal" curl is found, the check for curl-gnutls is skipped.
> I guess we should prefer curl-gnutls.
>
> Br
>
> > On 6. Sep 2022, at 10:51, Schanzenbach, Martin 
> wrote:
> >
> > Hi,
> >
> > check your config.log and check for the test against libcurl-gnutls.
> > There should be test somewhere in there that fails for reasons.
> >
> > BR
> > Martin
> >
> >> On 5. Sep 2022, at 20:21, madmurphy  wrote:
> >>
> >> Hi Christian,
> >>
> >> I tried to run ./configure twice. The first time I had installed on my
> machine curl (which is linked against openssl) and gnurl, but I did not
> have libcurl-gnutls installed. Under these conditions at the end of the
> configure script I got printed:
> >>
> >> ...
> >> HTTP Client:gnurl
> >> ...
> >>
> >> The second time I had curl and libcurl-gnutls installed, but gnurl was
> not installed. Under these conditions at the end of the configure script I
> got printed:
> >>
> >> ...
> >> HTTP Client:curl-openssl
> >> ...
> >>
> >> For some reasons in the second scenario the configure script sees the
> curl package but does not see libcurl-gnutls and so the latter is not used
> (I guess). The files shipped by the latter are:
> >>
> >> /usr/
> >> /usr/lib/
> >> /usr/lib/libcurl-gnutls.so.3
> >> /usr/lib/libcurl-gnutls.so.4
> >> /usr/lib/libcurl-gnutls.so.4.0.0
> >> /usr/lib/libcurl-gnutls.so.4.1.0
> >> /usr/lib/libcurl-gnutls.so.4.2.0
> >> /usr/lib/libcurl-gnutls.so.4.3.0
> >> /usr/lib/libcurl-gnutls.so.4.4.0
> >> /usr/lib/libcurl-gnutls.so.4.5.0
> >> /usr/lib/libcurl-gnutls.so.4.6.0
> >> /usr/lib/libcurl-gnutls.so.4.7.0
> >> /usr/lib/libcurl-gnutls.so.4.8.0
> >> /usr/share/
> >> /usr/share/licenses/
> >> /usr/share/licenses/libcurl-gnutls
> >>
> >> How can we ensure that the configure script correctly sees the
> libcurl-gnutls package?
> >>
> >> I paste below both complete outputs.
> >>
> >> Installed: curl, gnurl
> >> Not installed: libcurl-gnutls:
> >>
> >> $ ./configure
> >> checking build system type... x86_64-pc-linux-gnu
> >> checking host system type... x86_64-pc-linux-gnu
> >> checking target system type... x86_64-pc-linux-gnu
> >> checking for a BSD-compatible install... /usr/bin/install -c
> >> checking whether build environment is sane... yes
> >> checking for a race-free mkdir -p... /usr/bin/mkdir -p
> >> checking for gawk... gawk
> >> checking whether make sets $(MAKE)... yes
> >> checking whether make supports nested variables... yes
> >> checking whether UID '1000' is supported by ustar format... yes
> >> checking whether GID '1000' is supported by ustar format... yes
> >> checking how to create a ustar tar archive... gnutar
> >> checking whether make supports nested variables... (cached) yes
> >> checking for gawk... (cached) gawk
> >> checking for gcc... gcc
> >> checking whether the C compiler works... yes
> >> checking for C compiler default output file name... a.out
> >> checking for suffix of executables...
> >> checking whether we are cross compiling... no
> >> checking for suffix of object files... o
> >> checking whether the compiler supports GNU C... yes
> >> checking whether gcc accepts -g... yes
> >> checking for gcc option to enable C11 features... none needed
> >> checking whether gcc understands -c and -o together... yes
> >> checking whether make supports the include directive... yes (GNU style)
> >> checking dependency style of gcc... gcc3
> >> checking whether gcc and cc understand -c and -o together... yes
> >> checking whether ln -s works... yes
> >> checking whether make sets $(MAKE)... (cached) yes
> >> checking for pkg-config... /usr/bin/pkg-config
> >> checking pkg-config is at least version 0.29.2... yes
> >> checking how to print strings... printf
> >> checking for 

Re: About GNUrl and cURL

[madmurphy]

etcore.pc
config.status: creating pkgconfig/gnunetdatacache.pc
config.status: creating pkgconfig/gnunetdatastore.pc
config.status: creating pkgconfig/gnunetdht.pc
config.status: creating pkgconfig/gnunetdns.pc
config.status: creating pkgconfig/gnunetenv.pc
config.status: creating pkgconfig/gnunetfragmentation.pc
config.status: creating pkgconfig/gnunetfs.pc
config.status: creating pkgconfig/gnunetgns.pc
config.status: creating pkgconfig/gnunethello.pc
config.status: creating pkgconfig/gnunetidentity.pc
config.status: creating pkgconfig/gnunetmicrophone.pc
config.status: creating pkgconfig/gnunetmysql.pc
config.status: creating pkgconfig/gnunetnamestore.pc
config.status: creating pkgconfig/gnunetnat.pc
config.status: creating pkgconfig/gnunetnse.pc
config.status: creating pkgconfig/gnunetpeerinfo.pc
config.status: creating pkgconfig/gnunetpq.pc
config.status: creating pkgconfig/gnunetregex.pc
config.status: creating pkgconfig/gnunetrevocation.pc
config.status: creating pkgconfig/gnunetrps.pc
config.status: creating pkgconfig/gnunetscalarproduct.pc
config.status: creating pkgconfig/gnunetset.pc
config.status: creating pkgconfig/gnunetspeaker.pc
config.status: creating pkgconfig/gnunetstatistics.pc
config.status: creating pkgconfig/gnunettestbed.pc
config.status: creating pkgconfig/gnunettesting.pc
config.status: creating pkgconfig/gnunettransport.pc
config.status: creating pkgconfig/gnunetutil.pc
config.status: creating pkgconfig/gnunetvpn.pc
config.status: creating gnunet_config.h
config.status: executing depfiles commands
config.status: executing libtool commands
config.status: executing po-directories commands
config.status: creating po/POTFILES
config.status: creating po/Makefile
configure: WARNING: GnuTLS lacks DANE support; validation using it
will not be possible
configure: WARNING: Your version of Python is not supported, you might
see issues
configure:
Detected system
===

GNUnet version: 0.17.5

Host Setup: x86_64-pc-linux-gnu
Install Prefix: /usr
Compiler:   gcc
CFLAGS: -march=x86-64 -mtune=generic -O2 -pipe
-fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=2 -Wformat
-Werror=format-security -fstack-clash-protection
-fcf-protection -fno-strict-aliasing -Wno-address-of-packed-member
CPPFLAGS:
LDFLAGS:
-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now
-Wl,--unresolved-symbols=report-all
LIBS:   -lm
Build Target:   linux

Default Interface:  eth0

sqlite3:yes
MySQL:  no
PostgreSQL: yes

HTTP Client:curl-openssl
Bluetooth:  yes
iptables:   yes
ifconfig:   yes
UPnPc:  yes
GnuTLS: yes (without DANE support)

LaTeX:  yes

libextractor:   yes
libzbar:yes
libpng: yes
libidn: libidn2
libopus:yes
libpulse:   yes
gstreamer:  no

Java:   no

sphinx: yes
mandoc: no

GNUnet configuration


Transports: tcp udp unix http wlan
Conversation:   yes (pulse)
Database Backends   postgres sqlite
Experimental Code:  yes

Transpiled mdocml Manual:

configure: For detailed setup instructions, type 'info gnunet' after
the installation or visit https://docs.gnunet.org/


On Mon, Sep 5, 2022 at 7:14 PM Christian Grothoff 
wrote:

> Did you happen to have both libcurl-gnutls and libcurl-openssl
> installed, and maybe configure found the wrong one?
>
> On 9/5/22 18:56, madmurphy wrote:
> > Mmm I just checked better, and the |configure| script now says:
> >
> > ...
> > HTTP Client:    curl-openssl
> > ...
> >
> > That is not a good sign, right?
> >
> >
> > On Mon, Sep 5, 2022 at 5:47 PM madmurphy  > <mailto:madmurphy...@gmail.com>> wrote:
> >
> > Indeed, there is a dedicated package on Arch, |libcurl-gnutls|
> > <https://archlinux.org/packages/core/x86_64/libcurl-gnutls/>. I just
> > checked, and it seems that GNUnet works fine with it. Package
> > updated
> > <
> https://aur.archlinux.org/cgit/aur.git/commit/?h=gnunet=3693252e6dcdf53bfa428b4c078729c08f68ecde
> >.
> >
> >     --madmurphy
> >
> >
> > On Mon, Sep 5, 2022 at 4:25 PM Christian Grothoff
> > mailto:groth...@gnunet.org>> wrote:
> >
> > If Arch has a curl linked against GnuTLS, then yes. -Christian
> >
> > On 9/5/22 17:11, madmurphy wrote:
> > > On Arch GNUnet still depends on GNUrl, but as far as I
> > understood now
> > > cURL is preferred. Would I do the right thing if updated the
> Arch
> > > package accordingly and dropped the GNUrl dependency for good?
> > >
> > > --madmurphy
> >
>
>

Re: About GNUrl and cURL

[madmurphy]

Mmm I just checked better, and the configure script now says:

...
HTTP Client:curl-openssl
...

That is not a good sign, right?

On Mon, Sep 5, 2022 at 5:47 PM madmurphy  wrote:

> Indeed, there is a dedicated package on Arch, libcurl-gnutls
> <https://archlinux.org/packages/core/x86_64/libcurl-gnutls/>. I just
> checked, and it seems that GNUnet works fine with it. Package updated
> <https://aur.archlinux.org/cgit/aur.git/commit/?h=gnunet=3693252e6dcdf53bfa428b4c078729c08f68ecde>
> .
>
> --madmurphy
>
> On Mon, Sep 5, 2022 at 4:25 PM Christian Grothoff 
> wrote:
>
>> If Arch has a curl linked against GnuTLS, then yes. -Christian
>>
>> On 9/5/22 17:11, madmurphy wrote:
>> > On Arch GNUnet still depends on GNUrl, but as far as I understood now
>> > cURL is preferred. Would I do the right thing if updated the Arch
>> > package accordingly and dropped the GNUrl dependency for good?
>> >
>> > --madmurphy
>>
>>

Re: About GNUrl and cURL

[madmurphy]

Indeed, there is a dedicated package on Arch, libcurl-gnutls
<https://archlinux.org/packages/core/x86_64/libcurl-gnutls/>. I just
checked, and it seems that GNUnet works fine with it. Package updated
<https://aur.archlinux.org/cgit/aur.git/commit/?h=gnunet=3693252e6dcdf53bfa428b4c078729c08f68ecde>
.

--madmurphy

On Mon, Sep 5, 2022 at 4:25 PM Christian Grothoff 
wrote:

> If Arch has a curl linked against GnuTLS, then yes. -Christian
>
> On 9/5/22 17:11, madmurphy wrote:
> > On Arch GNUnet still depends on GNUrl, but as far as I understood now
> > cURL is preferred. Would I do the right thing if updated the Arch
> > package accordingly and dropped the GNUrl dependency for good?
> >
> > --madmurphy
>
>

About GNUrl and cURL

[madmurphy]

On Arch GNUnet still depends on GNUrl, but as far as I understood now cURL
is preferred. Would I do the right thing if updated the Arch package
accordingly and dropped the GNUrl dependency for good?

--madmurphy

Re: Small question about gnunet-search

[madmurphy]

That exists, it's just --timeout=forever

Alright. So it is only needed to implement the reactivity to the
configuration then. And also rewrite the entire FS module afterwards… :)

--madmurphy

On Mon, Jun 27, 2022 at 7:59 AM Christian Grothoff 
wrote:

> On 6/27/22 08:13, madmurphy wrote:
> > That said, I strongly suspect the *timeout* option the man page
> > refers to may have been removed a long time ago. Or maybe there is
> > one hiding in libgnunetfs that I'm not thinking of. ;-)
> >
> > Yes, I meant reacting specifically to that “timeout” option (I don't
> > even know how what key in the configuration file would map it,
> > eventually)… But the idea is not bad! Although in that case it would
> > also be needed to allow a |--timeout=infinity| option, for when a finite
> > timeout is set in the configuration file…
>
> That exists, it's just --timeout=forever
>
>
>

Re: Small question about gnunet-search

[madmurphy]

That said, I strongly suspect the *timeout* option the man page refers to
may have been removed a long time ago. Or maybe there is one hiding in
libgnunetfs that I'm not thinking of. ;-)

Yes, I meant reacting specifically to that “timeout” option (I don't even
know how what key in the configuration file would map it, eventually)… But
the idea is not bad! Although in that case it would also be needed to allow
a --timeout=infinity option, for when a finite timeout is set in the
configuration file…

By the way, back then I had also written down some of the features that I
believe a future FS module will need to implement:

   1. Possibility of sharing a file while it is still being downloaded
   2. Metadata must be editable and sharable
   3. Search keywords must be visible, editable, sharable (part of the
   metadata?)
   4. Introduction of a rating mechanism
   5. Allow reverse search (i.e. chk-URI lookup)
   6. Completely auto-unindex a file when it is missing
   7. Autoshare the dynamic content of a directory and update its index in
   real time
   8. Implement file statistics (download counter? last seen? etc.)
   9. Implement a “not” operator for search keywords (a tilde, “~”?)

My two cents

--madmurphy

On Sun, Jun 26, 2022 at 8:59 PM Christian Grothoff 
wrote:

> On 6/26/22 17:28, madmurphy wrote:
> > Hi everyone,
> >
> > I have a question about something I had noticed some months ago while
> > reviewing the man page for |gnunet-search|. After typing |man
> > gnunet-search| and scrolling down, the following text appears:
> >
> > FILES
> > |~/.config/gnunet.conf| GNUnet configuration file; specifies the
> > default value for the timeout
> >
> > However there is nothing in the source code of |src/fs/gnunet-search.c|
> > <
> https://git.gnunet.org/gnunet.git/tree/src/fs/gnunet-search.c?id=b6cbb6f800ef9aeebcfb76a7ba721d4b95a2e2ca>
>
> > that reacts to the configuration.
> >
>
> GNUNET_PROGRAM_run() reacts to the configuration, as does the rest of
> libgnunetutil and thus also libgnunetfs when it interacts with GNUnet
> services.
>
> That said, I strongly suspect the *timeout* option the man page refers
> to may have been removed a long time ago. Or maybe there is one hiding
> in libgnunetfs that I'm not thinking of. ;-)
>
> -Christian
>
>

Small question about gnunet-search

[madmurphy]

Hi everyone,

I have a question about something I had noticed some months ago while
reviewing the man page for gnunet-search. After typing man gnunet-search
and scrolling down, the following text appears:

FILES~/.config/gnunet.conf GNUnet configuration file; specifies the default
value for the timeout

However there is nothing in the source code of src/fs/gnunet-search.c
<https://git.gnunet.org/gnunet.git/tree/src/fs/gnunet-search.c?id=b6cbb6f800ef9aeebcfb76a7ba721d4b95a2e2ca>
that reacts to the configuration.

Was that a planned feature?

--madmurphy

Re: Packaging problems

[madmurphy]

I have added the Arch packages
<https://git.gnunet.org/gnunet.git/tree/contrib/packages/arch?id=22082c234c2013c65fc49a59d01c467a78c12f3e>
to the repo, as other distros might find them useful.

--madmurphy

On Mon, Jun 6, 2022 at 7:55 PM Maxime Devos  wrote:

> Martin Schanzenbach schreef op ma 06-06-2022 om 16:52 [+]:
> > > As Maxime says, GNUnet takes a long time to compile (when it
> > actually
> > > does - I'm having problems with that right now), and presumably
> > quite
> > > a
> > > while to test too. The obvious way to reduce those times is to
> > simply
> > > *reduce the amount of code being compiled and tested*. Breaking up
> > > the
> > > big repo would achieve that quite nicely.
> >
> > It really does not (on modern hardware).
> > See:
> > https://copr.fedorainfracloud.org/coprs/schanzen/gnunet/build/4501586/
> >
> > It takes around 7mins to install & compile from scratch (this
> > includes
> > installing all dependencies!).
> >
> > IMO right now "make check" is kind of annoying because it takes too
> > long and fails because of bad test design. It needs some love.
> > Maybe a high-level, quick "make check" and an optional "make check-
> > thorough" idk.
>
> FWIW, I included "make check" time in compilation time (from Guix'
> perspective, running tests is just yet another compilation).  And the
> computer I use for compilation isn't exactly modern -- according to the
> timestamp on /etc/host.conf, it's ~16 years old ... wait no, cannot be
> right ...  I'm not sure about the exact year, but at least ~5 years I'd
> say?  And that's the current computer (*) which has an SSD, whereas the
> computer(^) I used back then for GNUnet was older and had an old
> spinning disk.  I guess that isn't representative.
>
> (*) currently held together by duck tape, missing a few screws, keys,
> part of the frame and has a crack in the screen -- I cannot recommend
> Lenovo Yoga computers.
>
> (^) screen is broken unless a sea star is inserted between the keyboard
> and the screen in the right place at the right depth and angle.
>
> Greetings,
> Maxime.
>

Re: Reducing the number of executables to one

[madmurphy]

Hi Martin,

I am relatively unpassionate about this but: Isn't this kind of completion
shell-specific?
As in, wouldn't you have to provide different shell scripts for different
shells?

Yes, we would need a script for every shell. But we would have to do it
once (the list of the available executables would be generated
automatically), and online there are many autocompletion templates for most
shells ready to use.

On Thu, Mar 3, 2022 at 7:00 AM Schanzenbach, Martin 
wrote:

>
>
> > On 3. Mar 2022, at 01:51, madmurphy  wrote:
> >
> > Hi Christian,
> >
> > As I said, it is more of a long term planning. The main argument in
> favor is that (hopefully!) the number of “gnunet-XXX” utilities is only
> destined to grow, so eventually at some point it will be needed anyway. I
> can try to be the devil's advocate and answer point by point…
> >
> > It just makes it less obvious how to run the binaries under
> valgrind/gdb, adds just another process as overhead
> > It will always be possible to run the binaries directly. They will still
> exist, they will only be outside /usr/bin. Already now GNUnet does
> something similar with /usr/lib/gnunet/libexec.
> >
> > may require re-writing documentation.
> > That yes, will be required. But can be done slowly. If people see in the
> documentation gnunet-peerinfo instead of gnunet peerinfo they are going to
> understand anyway. It could even become an opportunity to do a very much
> needed documentation review.
> >
> > It also removes the ability to get a list of possible invocations via
> 'tab' easily (right now, you can type "gnunet-" and get a list of
> all available gnunet-commands).
> > That no, it is not a problem. It is possible to add command completion
> to the gnunet script/program (depending on whether we decide that it is a
> script or a C program), so that a user can type gnunet  and get the
> list of commands available – the same way as you can currently type make
>  and get the list of make targets available…
>
> I am relatively unpassionate about this but: Isn't this kind of completion
> shell-specific?
> As in, wouldn't you have to provide different shell scripts for different
> shells?
>
> BR
>
> >
> > --madmurphy
> >
> >
> > On Wed, Mar 2, 2022 at 11:01 PM Christian Grothoff 
> wrote:
> > Personally, I'm not a fan of this style. It just makes it less obvious
> > how to run the binaries under valgrind/gdb, adds just another process as
> > overhead, and may require re-writing documentation. It also removes the
> > ability to get a list of possible invocations via 'tab' easily (right
> > now, you can type "gnunet-" and get a list of all available
> > gnunet-commands).
> >
> > So overall, the "benefit" of being able to remove the hyphen seems,
> > well, questionable. But I'm aware that it is the current fashion. But I
> > personally don't get that fashion.
> >
> > On 2/27/22 11:20 AM, madmurphy wrote:
> > > This is more like a long term plan and nothing really important…
> > >
> > > I saw that the amount of command line utilities that GNUnet ships is
> > > quite sizeable and is probably only destined to grow (I have counted 70
> > > executables in |/usr/bin|); so I was thinking that GNUnet could follow
> > > git's approach, that of having one single executable in |/usr/bin|, and
> > > do something like |gnunet COMMAND OPTIONS ARGUMENTS|.
> > >
> > > As all the executables are named |gnunet-SOMETHING|, this would
> > > basically only remove the hyphen. For example, |gnunet-search
> 'commons'|
> > > would become |gnunet search 'commons'|.
> > >
> > > It can be done with a shell script as simple as:
> > >
> > > #!/bin/sh
> > > #
> > > # /usr/bin/gnunet
> > > #
> > >
> > > _GNUNET_UTIL_DIR_='/foo/bar'
> > >
> > > if [[ -f "${_GNUNET_UTIL_DIR_}/gnunet-${1}" ]]; then
> > >   "${_GNUNET_UTIL_DIR_}/gnunet-${1}" "${@:2}"
> > > else
> > >   echo "Unknown command \"${1}\"."
> > > fi
> > >
> > > (where |/foo/bar| is the directory where the executables are actually
> > > installed.)
> > >
> > > What do you think?
> > >
> > > --madmurphy
> > >
> >
>
>

Re: Reducing the number of executables to one

[madmurphy]

Hi Christian,

As I said, it is more of a long term planning. The main argument in favor
is that (hopefully!) the number of “gnunet-XXX” utilities is only destined
to grow, so eventually at some point it will be needed anyway. I can try to
be the devil's advocate and answer point by point…

It just makes it less obvious how to run the binaries under valgrind/gdb,
adds just another process as overhead

It will always be possible to run the binaries directly. They will still
exist, they will only be outside /usr/bin. Already now GNUnet does
something similar with /usr/lib/gnunet/libexec.

may require re-writing documentation.

That yes, will be required. But can be done slowly. If people see in the
documentation gnunet-peerinfo instead of gnunet peerinfo they are going to
understand anyway. It could even become an opportunity to do a very much
needed documentation review.

It also removes the ability to get a list of possible invocations via 'tab'
easily (right now, you can type "gnunet-" and get a list of all
available gnunet-commands).

That no, it is not a problem. It is possible to add command completion to
the gnunet script/program (depending on whether we decide that it is a
script or a C program), so that a user can type gnunet  and get the
list of commands available – the same way as you can currently type make
 and get the list of make targets available…

--madmurphy

On Wed, Mar 2, 2022 at 11:01 PM Christian Grothoff 
wrote:

> Personally, I'm not a fan of this style. It just makes it less obvious
> how to run the binaries under valgrind/gdb, adds just another process as
> overhead, and may require re-writing documentation. It also removes the
> ability to get a list of possible invocations via 'tab' easily (right
> now, you can type "gnunet-" and get a list of all available
> gnunet-commands).
>
> So overall, the "benefit" of being able to remove the hyphen seems,
> well, questionable. But I'm aware that it is the current fashion. But I
> personally don't get that fashion.
>
> On 2/27/22 11:20 AM, madmurphy wrote:
> > This is more like a long term plan and nothing really important…
> >
> > I saw that the amount of command line utilities that GNUnet ships is
> > quite sizeable and is probably only destined to grow (I have counted 70
> > executables in |/usr/bin|); so I was thinking that GNUnet could follow
> > git's approach, that of having one single executable in |/usr/bin|, and
> > do something like |gnunet COMMAND OPTIONS ARGUMENTS|.
> >
> > As all the executables are named |gnunet-SOMETHING|, this would
> > basically only remove the hyphen. For example, |gnunet-search 'commons'|
> > would become |gnunet search 'commons'|.
> >
> > It can be done with a shell script as simple as:
> >
> > #!/bin/sh
> > #
> > # /usr/bin/gnunet
> > #
> >
> > _GNUNET_UTIL_DIR_='/foo/bar'
> >
> > if [[ -f "${_GNUNET_UTIL_DIR_}/gnunet-${1}" ]]; then
> >   "${_GNUNET_UTIL_DIR_}/gnunet-${1}" "${@:2}"
> > else
> >   echo "Unknown command \"${1}\"."
> > fi
> >
> > (where |/foo/bar| is the directory where the executables are actually
> > installed.)
> >
> > What do you think?
> >
> > --madmurphy
> >
>
>

Re: Attacking the documentation monster

[madmurphy]

I don't know if this will be a popular proposal, but I really believe that
setting up a self-hosted Wiki could be a very good choice. No complicate
git clone, no complaints, just read/edit what you need, and distributed
responsibilities about its design and direction.

My two cents

On Tue, Mar 1, 2022 at 8:02 PM William Liquorice 
wrote:

> Hello,
>
> A year or two ago, I tried to wrap my head around GNUnet so that I could
> try to make parallel implementations of small bits in Rust, but found
> its documentation to be utterly impenetrable. Not even from a technical
> standpoint, the massive reference manual / "handbook" is quite
> overwhelming and more akin to the Lord of the Rings.
>
> Just now, I decided to look up the documentation for GNUnet's IPC
> functionality. This is a pretty important component of GNUnet's
> modularity. One rather important section about sending an example
> AddressLookupMessage between has the immediate subheading "FIXME: This
> is very outdated, see the tutorial for the current API!"
>
> To my annoyance, there is no link provided to this tutorial. Where is
> it? I will put the link in there myself if it exists.
>
> I am unfamiliar with how exactly documentation is put together for
> GNUnet, but just separating out the contributor's/developer's handbook
> (most of the page) would make the actual user manual significantly less
> intimidating. It doesn't need to all be on one page.
>
> I also made some SVG diagrams of specific GNUnet subsystems (the core
> transport -> CADET stack, GNS, VPN), but I don't recall ever hearing
> back from anyone about them. Are they any good?
>
> Thanks,
> Will
>
>

Reducing the number of executables to one

[madmurphy]

This is more like a long term plan and nothing really important…

I saw that the amount of command line utilities that GNUnet ships is quite
sizeable and is probably only destined to grow (I have counted 70
executables in /usr/bin); so I was thinking that GNUnet could follow git's
approach, that of having one single executable in /usr/bin, and do
something like gnunet COMMAND OPTIONS ARGUMENTS.

As all the executables are named gnunet-SOMETHING, this would basically
only remove the hyphen. For example, gnunet-search 'commons' would
become gnunet
search 'commons'.

It can be done with a shell script as simple as:

#!/bin/sh
#
# /usr/bin/gnunet
#

_GNUNET_UTIL_DIR_='/foo/bar'

if [[ -f "${_GNUNET_UTIL_DIR_}/gnunet-${1}" ]]; then
"${_GNUNET_UTIL_DIR_}/gnunet-${1}" "${@:2}"
else
echo "Unknown command \"${1}\"."
fi

(where /foo/bar is the directory where the executables are actually
installed.)

What do you think?

--madmurphy

Re: printf-like output for gnunet-search

[madmurphy]

t search, print only the URI that points to
   this search
  -c, --config=FILENAME  use configuration file FILENAME
  -F, --dir-printf=FORMATwrite search results for directories according to
   FORMAT; accepted placeholders are: %a, %f, %j,
   %l, %m, %n, %s; defaults to the value of
   --printf when omitted; if --printf is omitted
   too defaults to `#%n:\ngnunet-download -o "%f"
   -R %u\n\n`
  -f, --printf=FORMATwrite search results according to FORMAT;
   accepted placeholders are: %a, %f, %j, %l, %m,
   %n, %s; defaults to `#%n:\ngnunet-download -o
   "%f" %u\n\n` when omitted
  -h, --help print this help
  -i, --iter-printf=FORMAT   when the %a or %j format specifiers appear in
   --printf or --dir-printf, list each metadata
   property according to FORMAT; accepted
   placeholders are: %i, %l, %n, %p, %t, %w;
   defaults to `  %t: %p\n` when omitted
  -L, --log=LOGLEVEL configure logging to use LOGLEVEL
  -l, --logfile=FILENAME configure logging to write logs to FILENAME
  -N, --results=VALUEautomatically terminate search after VALUE
   results are found
  -n, --no-network   only search the local peer (no P2P network
   search)
  -o, --output=FILENAME  create a GNUnet directory with search results at
   FILENAME (e.g. `gnunet-search
   --output=commons.gnd commons`)
  -s, --silent   silent mode (requires the --output argument)
  -t, --timeout=DELAYautomatically terminate search after DELAY; the
   value given must be a number followed by a
   space and a time unit, for example "500 ms";
   without a unit it defaults to microseconds -
   100 = 1 second; if 0 or omitted it means to
   wait for CTRL-C
  -V, --verbose  be verbose (append "%a\n" to the default --printf
   and --dir-printf arguments - ignored when these
   are provided by the user)
  -v, --version  print the version number

Report bugs to gnunet-developers@gnu.org.
Home page: http://www.gnu.org/s/gnunet/
General help using GNU software: http://www.gnu.org/gethelp/

*Option 3: Any other new proposal?*

...

--madmurphy

On Sat, Feb 12, 2022 at 1:58 AM madmurphy  wrote:

> Hi Alessio,
>
> the line saying "The main reason is that it's so long to be basically
> unreadable." wasn't related only to the program's description, but to the
> whole output, sorry for the confusion.
>
> [...]
>
> Writing something like:
>
> "Format output according to FORMAT.
> Accepted placeholders: %j %l %f %u %n
> Defaults to the value of --printf when omitted"
>
> is perfectly fine.
>
> We can do that (see file attached), but the output would not become
> terribly smaller, and the reason is that the program accepts many
> arguments. The modified help page would look like this:
>
> $ gnunet-search --help
>
> gnunet-search [OPTIONS] KEYWORD1 KEYWORD2 ...
> Search for files that have been published on GNUnet
>
> Arguments mandatory for long options are also mandatory for short options.
>   -a, --anonymity=LEVEL  set the desired LEVEL of receiver-anonymity
>(default: 1)
>   -b, --bookmark-onlydo not search, print only the URI that points to
>this search
>   -c, --config=FILENAME  use configuration file FILENAME
>   -F, --dir-printf=FORMATwrite search results for directories according to
>FORMAT; accepted placeholders are: %a, %f, %j,
>%l, %m, %n, %s; defaults to the value of
>--printf when omitted; if --printf is omitted
>too defaults to `#%n:\ngnunet-download -o "%f"
>-R %u\n\n`
>   -f, --printf=FORMATwrite search results according to FORMAT;
>accepted placeholders are: %a, %f, %j, %l, %m,
>%n, %s; defaults to `#%n:\ngnunet-download -o
>"%f" %u\n\n` when omitted
>   -h, --help print this help
>   -i, --iter-printf=FORMAT   when the %a or %j fo

Re: printf-like output for gnunet-search

[madmurphy]

Hi Alessio,

the line saying "The main reason is that it's so long to be basically
unreadable." wasn't related only to the program's description, but to the
whole output, sorry for the confusion.

[...]

Writing something like:

"Format output according to FORMAT.
Accepted placeholders: %j %l %f %u %n
Defaults to the value of --printf when omitted"

is perfectly fine.

We can do that (see file attached), but the output would not become
terribly smaller, and the reason is that the program accepts many
arguments. The modified help page would look like this:

$ gnunet-search --help

gnunet-search [OPTIONS] KEYWORD1 KEYWORD2 ...
Search for files that have been published on GNUnet

Arguments mandatory for long options are also mandatory for short options.
  -a, --anonymity=LEVEL  set the desired LEVEL of receiver-anonymity
   (default: 1)
  -b, --bookmark-onlydo not search, print only the URI that points to
   this search
  -c, --config=FILENAME  use configuration file FILENAME
  -F, --dir-printf=FORMATwrite search results for directories according to
   FORMAT; accepted placeholders are: %a, %f, %j,
   %l, %m, %n, %s; defaults to the value of
   --printf when omitted; if --printf is omitted
   too defaults to `#%n:\ngnunet-download -o "%f"
   -R %u\n\n`
  -f, --printf=FORMATwrite search results according to FORMAT;
   accepted placeholders are: %a, %f, %j, %l, %m,
   %n, %s; defaults to `#%n:\ngnunet-download -o
   "%f" %u\n\n` when omitted
  -h, --help print this help
  -i, --iter-printf=FORMAT   when the %a or %j format specifiers appear in
   --printf or --dir-printf, list each metadata
   property according to FORMAT; accepted
   placeholders are: %i, %l, %n, %p, %t, %w;
   defaults to `  %t: %p\n` when omitted
  -L, --log=LOGLEVEL configure logging to use LOGLEVEL
  -l, --logfile=FILENAME configure logging to write logs to FILENAME
  -N, --results=VALUEautomatically terminate search after VALUE
   results are found
  -n, --no-network   only search the local peer (no P2P network
   search)
  -o, --output=FILENAME  create a GNUnet directory with search results at
   FILENAME (e.g. `gnunet-search
   --output=commons.gnd commons`)
  -s, --silent   silent mode (requires the --output argument)
  -t, --timeout=DELAYautomatically terminate search after DELAY; the
   value given must be a number followed by a
   space and a time unit, for example "500 ms";
   without a unit it defaults to microseconds -
   100 = 1 second; if 0 or omitted it means to
   wait for CTRL-C
  -V, --verbose  be verbose (append "%a\n" to the default --printf
   and --dir-printf arguments - ignored when these
   are provided by the user)
  -v, --version  print the version number

Report bugs to gnunet-developers@gnu.org.
Home page: http://www.gnu.org/s/gnunet/
General help using GNU software: http://www.gnu.org/gethelp/

As for the man page, I can think of a few example to add. But wouldn't
putting all the format specifiers together create confusion, since --printf
and --iter-printf use different format specifiers? I would use a unified
section for the examples, but I would leave the specifier lists where they
are.

--madmurphy

On Fri, Feb 11, 2022 at 11:56 PM Alessio Vanni  wrote:

> Before replying on your points, a correction: the line saying "The main
> reason is that it's so long to be basically unreadable." wasn't related
> only to the program's description, but to the whole output, sorry for
> the confusion.
>
> As for the rest:
>
> > Man pages are not always installed by people, and a help page should
> > be able to explain the bare minimum.
>
> In my opinion, if people don't install man pages and then need them,
> it's their own fault if then they don't know what to do.  Of course, if
> the OS doesn't come shipped with them it's not the user's fault, but it
> still doesn't mean the description of the flag has to be a full paragraph.
> I believe simply listing the accepted formats is enough, e.g.
>
> "Format output according to FORMAT.
> Accepted placeholders: %j %l %f %u %n"
>

Re: printf-like output for gnunet-search

[madmurphy]

verbose  be verbose (append "%a\n" to the default --printf
   and --dir-printf arguments - ignored when these
   are provided by the user)
  -v, --version  print the version number

Report bugs to gnunet-developers@gnu.org.
Home page: http://www.gnu.org/s/gnunet/
General help using GNU software: http://www.gnu.org/gethelp/

Point by point…

Even the tool's description should be reverted to the simpler one-liner
that was used previously.

The main reason is that it's so long to be basically unreadable.

Okay, I can do that (I already striked out the addition in the tool's
description).

As an example, the help string for `--printf', which other than being about
four sentences long also contains a list of items, should probably be
reworded like so:

"Write search results according to FORMAT. See the documentation for the
available placeholders."

Then you can add a dedicated section in the man page titled "Formatting the
output" or something like that.

Here I am not sure if a description of the format specifiers can be
completely absent in the help page. Man pages are not always installed by
people, and a help page should be able to explain the bare minimum.

What do other people think about removing all mentions of the % format
specifiers from the help page?

`--dir-printf' should use the same string, instead of referencing some
other option. It is generally a good idea to treat each help message as an
independent entity — at least personally speaking, oftentimes I find myself
scanning the left column for a specific optiong and only after I find it I
read the help text, so having to navigate back and forth because the option
I need just says "alias for foo" ends up being rather annoying in the long
run.

If we do mention the format specifiers in the help page, then the text of
--dir-printf does become too long, and the best is that it refers to
--printf like it does now. But if we remove the format specifiers from the
help page, then yes, it can be its own text. However, since when not
specified --dir-printf defaults to --printf, a mention of --printf will
always be there in a way or another.

Maybe we can amend the output of gnunet-search --help that I pasted above
directly on this mail thread?

I haven't checked the actual man pages yet, so I can't comment on that.

Below is the current output of man gnunet-search (we should also update the
date at the bottom, currently “February 25, 2012”)

--madmurphy

GNUNET-SEARCH(1) BSD General Commands Manual GNUNET-SEARCH(1)

*NAME*

*gnunet-search* — a command line interface to search for content on GNUnet

*SYNOPSIS*

*gnunet-search* [*−a **LEVEL *| *−-anonymity=**LEVEL*] [*−b *|
*−-bookmark-only*] [*−c **FILENAME *| *−-config=**FILENAME*] [*−F **FORMAT *
| *−-dir-printf=**FORMAT*] [*−f **FORMAT *| *−-printf=**FORMAT*] [*−h *|
*−-help*] [*−i **FORMAT *| *−-iter-printf=**FORMAT*] [*−L **LOGLEVEL *|
*−-loglevel=**LOGLEVEL*] [*−l **FILENAME *| *−-logfile=**FILENAME*] [*−o *
*FILENAME *| *−-output=**FILENAME*] [*−n *| *−-no-network*] [*−N **VALUE *|
*−-results=**VALUE*] [*−s *| *−-silent*] [*−t **DELAY *| *−-timeout=**DELAY*]
[*−v *| *−-version*] [*−V *| *−-verbose*] ⟨ KEYWORD ⟩ ⟨ +KEYWORD ⟩ | ⟨ *URI* ⟩
⟨ *+URI* ⟩

*DESCRIPTION*

Search for content on GNUnet. The keywords are case-sensitive.
*gnunet-search* can be used both for a search in the global namespace as
well as for searching a private subspace. The options are as follows:

*−a* *LEVEL* | *−-anonymity=**LEVEL*

This option can be used to specify additional anonymity constraints. The
default is 1. If set to 0, GNUnet will publish the file non-anonymously and
in fact sign the advertisement for the file using your peer’s private key.
This will allow other users to download the file as fast as possible,
including using non-anonymous methods (discovery via DHT and CADET
transfer). If you set it to 1 (default), you use the standard anonymous
routing algorithm (which does not explicitly leak your identity). However,
a powerful adversary may still be able to perform traffic analysis
(statistics) to over time discovery your identity. You can gain better
privacy by specifying a higher level of anonymity (using values above 1).
This tells FS that it must hide your own requests in equivalent-looking
cover traffic. This should confound an adversaries traffic analysis,
increasing the time and effort it would take to discover your identity.
However, it also can significantly reduce performance, as your requests
will be delayed until sufficient cover traffic is available. The specific
numeric value (for anonymity levels above 1) is simple: Given an anonymity
level L (above 1), each request FS makes on your behalf must be hidden in
L-1 equivalent requests of cover traffic (traffic your peer routes for
others) in the same time-period. The time-period is twice the average delay
by which GNUnet artificially delays traffic. Note that regardless

Re: printf-like output for gnunet-search

[madmurphy]

I have pushed the new gnunet-search with printf-like capabilities to the
git repository. I have also updated the man page. Feel free to play with it
and write your feedbacks :)

--madmurphy

On Tue, Feb 8, 2022 at 1:39 PM madmurphy  wrote:

> Last proposal about gnunet-search, then I will stop because the program
> is becoming hardcore: add a %[NUM]#[SPEC] syntax for the metadata
> specifiers (now they are two, %a and %j) in order to extract only
> specific metatypes. For example, -f '%f%49#j\n' -i ' (%p)' will print the
> file's keywords in brackets, if found. Something like (emphasis mine)
>
> $ gnunet-search commons -f '%f%49#j\n' -i ' (%p)'
>
> 2019 Helfrich; Bollier_ Frei, fair und lebendig - Die Macht der Commons.pdf
> Rose, Carol (1986)_ The Comedy of the Commons_ Commerce, Custom, and 
> inherently Public Property.pdf
> De Angelis (2017)_ Omnia Sunt Communia. On the Commons and the Transformation 
> to Postcapitalism.pdf
> Ostrom, Elinor (1990)_ Governing the Commons. The evolution of institutions 
> for collective action.pdf
> Gemeingueter_Report_Commons.pdf*File with keywords.pdf (some keyword, some 
> other keyword, etc.)*
> Rose, Carol (1986)_ The Comedy of the Commons_ Commerce, Custom, and 
> inherently Public Property.pdf
> Klick, Jonna (2021)_ Massimo de Angelis - Omnia Sunt Communia - Eine 
> Grundlegung der Commons - Rezension.pdf
> Helfrich, Silke_ Logik der Commons und des Marktes.png
> Hardin, Garett (1968)_ Tragedy of the Commons.pdf*Another file with 
> keywords.pdf (some keyword, some other keyword, etc.)*
> Liotard (2017)_ Fablab - a new space for commons based peer production.pdf
> Neustart Schweiz (2016)_ Nach Hause kommen. Nachbarschaften als Commons.pdf
>
> The number 49 is the EXTRACTOR_METATYPE_KEYWORDS identifier from
> extractor.h (not the search keywords). Once again, the new help page
> should explain it:
>
> $ gnunet-search --help
>
> gnunet-search [OPTIONS] KEYWORD1 KEYWORD2 ...
> Search for files that have been published on GNUnet
>
> Keywords should start with a plus sign to indicate that they are required -
> e.g. `gnunet-search commons gpl` searches for files that match either
> "commons" or "gpl", whereas `gnunet-search +commons +gpl` searches for files
> that match both "commons" and "gpl".
>
> Arguments mandatory for long options are also mandatory for short options.
>   -a, --anonymity=LEVEL  set the desired LEVEL of receiver-anonymity
>   -c, --config=FILENAME  use configuration file FILENAME
>   -F, --dir-printf=FORMATwrite search results for directories according to
>FORMAT; the format specifiers supported here
>are identical to the --printf argument (please
>see there for more information); if missing,
>--dir-printf defaults to --printf; if the
>latter is missing too --dir-printf defaults to
>`#%n:\ngnunet-download -o '%f' -R %u\n\n`
>   -f, --printf=FORMATwrite search results according to FORMAT, where
>%a is the complete list of all the printable
>metadata available (each member will be
>displayed according to the --prop-printf
>argument) - use %j for printing one field only
>- %f is the file's name, %l is the file name's
>length, %m is the file's mime type, %n is the
>search result number, %s is the file's size in
>bytes and %u is the file's URI; the %a and %j
>specifiers optionally support metatype
>filtering via hash sign (e.g. `%5#j` prints a
>book title, if present - see libextractor's
>metatypes for the complete list of numerical
>identifiers); if missing, --printf defaults to
>`#%n:\ngnunet-download -o '%f' %u\n\n`
>   -h, --help print this help
>   -i, --prop-printf=FORMAT   when the %a format specifier appears in --printf
>or --dir-printf, list each metadata property
>according to FORMAT, where %p is the property's
>content, %l is the content's length in bytes,
>%t is the property type, %i is the property
>type's unique identifier, %n is the property
> 

Re: libextractor - key-value pairs and mime types

[madmurphy]

Thank you, Christian

On Tue, Feb 8, 2022 at 10:17 PM Christian Grothoff 
wrote:

> I've added your patch and also clarified the use-case for RESERVED
> (which is still different from your proposed 'NONE'). Anyway, for
> further discussions on this, please use the libextractor list...
>
> -Christian
>
> On 2/8/22 7:21 PM, madmurphy wrote:
> > Actually, the first thing that I thought when I read
> > |EXTRACTOR_METATYPE_RESERVED| was “not used today, but tomorrow will be
> > used for the book's cover” :)
> >
> > It definitely needs a renaming!
> >
> > --madmurphy
> >
> >
> > On Tue, Feb 8, 2022 at 5:52 PM madmurphy  > <mailto:madmurphy...@gmail.com>> wrote:
> >
> > Of course, the value of |EXTRACTOR_METATYPE_RESERVED| would be even
> > better (zero would be the natural value for something like this).
> >
> > But then it's a lexical problem. If I see something marked as
> > “reserved” I read “do not ever try to use this label”.
> >
> > Since already libextractor uses |EXTRACTOR_METATYPE_RESERVED| with
> > the meaning of |EXTRACTOR_METATYPE_NONE|, would it not make sense to
> > rename |EXTRACTOR_METATYPE_RESERVED| to |EXTRACTOR_METATYPE_NONE|
> > and tell the user that there is nothing “reserved” about it?
> >
> > By instinct if I see a label named |EXTRACTOR_METATYPE_RESERVED| I
> > might think that there are cases in which libextractor marks a
> > metatype with |EXTRACTOR_METATYPE_RESERVED|, expecting me to treat
> > is as an opaque label. Instead, |EXTRACTOR_METATYPE_NONE| to be
> > usable requires libextractor never to mark anything publicly with it
> > (or throw it as a return value). Since apparently this is the case,
> > a comment similar to the one I had left in the patch would also be
> > useful (“used by libextractor only internally; available to the user
> > for marking an |enum EXTRACTOR_MetaType| as not carrying any
> > meaningful value”).
> >
> > --madmurphy
> >
>
>

Re: libextractor - key-value pairs and mime types

[madmurphy]

Actually, the first thing that I thought when I read
EXTRACTOR_METATYPE_RESERVED was “not used today, but tomorrow will be used
for the book's cover” :)

It definitely needs a renaming!

--madmurphy

On Tue, Feb 8, 2022 at 5:52 PM madmurphy  wrote:

> Of course, the value of EXTRACTOR_METATYPE_RESERVED would be even better
> (zero would be the natural value for something like this).
>
> But then it's a lexical problem. If I see something marked as “reserved” I
> read “do not ever try to use this label”.
>
> Since already libextractor uses EXTRACTOR_METATYPE_RESERVED with the
> meaning of EXTRACTOR_METATYPE_NONE, would it not make sense to rename
> EXTRACTOR_METATYPE_RESERVED to EXTRACTOR_METATYPE_NONE and tell the user
> that there is nothing “reserved” about it?
>
> By instinct if I see a label named EXTRACTOR_METATYPE_RESERVED I might
> think that there are cases in which libextractor marks a metatype with
> EXTRACTOR_METATYPE_RESERVED, expecting me to treat is as an opaque label.
> Instead, EXTRACTOR_METATYPE_NONE to be usable requires libextractor never
> to mark anything publicly with it (or throw it as a return value). Since
> apparently this is the case, a comment similar to the one I had left in the
> patch would also be useful (“used by libextractor only internally;
> available to the user for marking an enum EXTRACTOR_MetaType as not
> carrying any meaningful value”).
>
> --madmurphy
>

Re: libextractor - key-value pairs and mime types

[madmurphy]

Of course, the value of EXTRACTOR_METATYPE_RESERVED would be even better
(zero would be the natural value for something like this).

But then it's a lexical problem. If I see something marked as “reserved” I
read “do not ever try to use this label”.

Since already libextractor uses EXTRACTOR_METATYPE_RESERVED with the
meaning of EXTRACTOR_METATYPE_NONE, would it not make sense to rename
EXTRACTOR_METATYPE_RESERVED to EXTRACTOR_METATYPE_NONE and tell the user
that there is nothing “reserved” about it?

By instinct if I see a label named EXTRACTOR_METATYPE_RESERVED I might
think that there are cases in which libextractor marks a metatype with
EXTRACTOR_METATYPE_RESERVED, expecting me to treat is as an opaque label.
Instead, EXTRACTOR_METATYPE_NONE to be usable requires libextractor never
to mark anything publicly with it (or throw it as a return value). Since
apparently this is the case, a comment similar to the one I had left in the
patch would also be useful (“used by libextractor only internally;
available to the user for marking an enum EXTRACTOR_MetaType as not
carrying any meaningful value”).

--madmurphy

Re: libextractor - key-value pairs and mime types

[madmurphy]

I have prepared a mini-patch for explaining better what I mean with the
third point (I have used EXTRACTOR_METATYPE_NONE in the end, I think it is
more clear).

Please find it attached

--madmurphy

On Tue, Feb 8, 2022 at 1:38 PM madmurphy  wrote:

> Got it! I agree about your solution for the duplicate mime types.
>
> but until that is done, a key-value pair type would at least be better
> than 'unknown'.
>
> “Unknown” can continue to exist as an identifier for other cases, just not
> the key-value ones :)
>
> Also I forgot to mention a third point:
>
> 3. Add an EXTRACTOR_METATYPE_NO_METATYPE = -1 to enum EXTRACTOR_MetaType
> (more or less like NULL if that was a pointer). Without a
> EXTRACTOR_METATYPE_NO_METATYPE a programmer is forced to save the
> have_metatype information in another variable. The fact that it is a
> negative number is not a problem, because as the name suggests, *it is
> not a metatype*.
>
> P.S. Sorry for picking the wrong mailing list!
>
> On Tue, Feb 8, 2022 at 9:57 AM Christian Grothoff 
> wrote:
>
>> Hi madmurphy,
>>
>> The 'correct' place for GNU libextractor discussions would be
>>
>>   https://lists.gnu.org/mailman/listinfo/libextractor
>>
>> Alas, with my libextractor maintainer hat on, I would say this:
>>
>> On 2/7/22 10:01 PM, madmurphy wrote:
>> > Hi again, GNUnet people.
>> >
>> > Is this the place where to discuss about libextractor? I have two
>> points.
>> >
>> > #1 I often see something interesting. Key-value pairs are categorized as
>> > |EXTRACTOR_METATYPE_UNKNOWN|:
>> >
>> > unknown: chroma-format=4:2:0
>> > unknown: bit-depth-chroma=8
>> > unknown: colorimetry=bt709
>> > unknown: stream-format=avc
>> > unknown: stream-format=raw
>> > unknown: bit-depth-luma=8
>> > unknown: base-profile=lc
>> > unknown: mpegversion=4
>> > unknown: profile=high
>> > unknown: alignment=au
>> > unknown: parsed=true
>> > unknown: framed=true
>> > unknown: variant=iso
>> > unknown: profile=lc
>> > unknown: level=4.1
>> >
>> > But one point is that they are often numerous, and another point is that
>> > that of a key-value type is a really interesting metatype to have (and
>> > is not really “unknown”, since the key is self-explanatory). Would it
>> > not make sense to add an |EXTRACTOR_METATYPE_KEY_VALUE_PAIR| to the list
>> > of MetaTypes?
>>
>> We could do that. Sometimes I think it would be better to add new
>> specific LE types for some of the above, but until that is done, a
>> key-value pair type would at least be better than 'unknown'.
>>
>> > ...
>> >
>> >   /* generic attributes */
>> >   EXTRACTOR_METATYPE_UNKNOWN = 45,
>> >   EXTRACTOR_METATYPE_DESCRIPTION = 46,
>> >   EXTRACTOR_METATYPE_COPYRIGHT = 47,
>> >   EXTRACTOR_METATYPE_RIGHTS = 48,
>> >   EXTRACTOR_METATYPE_KEYWORDS = 49,
>> >   EXTRACTOR_METATYPE_ABSTRACT = 50,
>> >   EXTRACTOR_METATYPE_SUMMARY = 51,
>> >   EXTRACTOR_METATYPE_SUBJECT = 52,
>> >   EXTRACTOR_METATYPE_CREATOR = 53,
>> >   EXTRACTOR_METATYPE_FORMAT = 54,
>> >   EXTRACTOR_METATYPE_FORMAT_VERSION = 55,
>> >   *EXTRACTOR_METATYPE_KEY_VALUE_PAIR* = XXX,
>> >
>> > ...
>> >
>> > #2 I often see that files get tagged with multiple mime types according
>> > to libextractor:
>> >
>> > mimetype: video/quicktime
>> > mimetype: video/x-h264
>> > mimetype: audio/mpeg
>> > mimetype: video/mp4
>>
>> That is because different plugins (using different methods/libraries)
>> disagree on the 'correct' mime-type. Ideally, we'd identify which plugin
>> gets it wrong (and why), and unify the mime-types.
>>
>> > But that never reflects the reality, since files should have only one
>> > mime type (or at most, multiple mime types that mean the same thing).
>> > But then I see what happens with file names: there is only one
>> > |EXTRACTOR_METATYPE_GNUNET_ORIGINAL_FILENAME|, but there can be many
>> > |EXTRACTOR_METATYPE_FILENAME|s (in the case of archives, for example):
>> >
>> > EXTRACTOR_METATYPE_FILENAME = 2,
>> > ...
>> > EXTRACTOR_METATYPE_GNUNET_ORIGINAL_FILENAME = 180,
>> >
>> > Would it not make sense to do something similar for mime types? Only one
>> > “original mime type”, and an infinity of secondary mime types…?
>> >
>> > EXTRACTOR_METATYPE_MIMETYPE = 1,
>> > ...

Re: libextractor - key-value pairs and mime types

[madmurphy]

Got it! I agree about your solution for the duplicate mime types.

but until that is done, a key-value pair type would at least be better than
'unknown'.

“Unknown” can continue to exist as an identifier for other cases, just not
the key-value ones :)

Also I forgot to mention a third point:

3. Add an EXTRACTOR_METATYPE_NO_METATYPE = -1 to enum EXTRACTOR_MetaType
(more or less like NULL if that was a pointer). Without a
EXTRACTOR_METATYPE_NO_METATYPE a programmer is forced to save the
have_metatype information in another variable. The fact that it is a
negative number is not a problem, because as the name suggests, *it is not
a metatype*.

P.S. Sorry for picking the wrong mailing list!

On Tue, Feb 8, 2022 at 9:57 AM Christian Grothoff 
wrote:

> Hi madmurphy,
>
> The 'correct' place for GNU libextractor discussions would be
>
>   https://lists.gnu.org/mailman/listinfo/libextractor
>
> Alas, with my libextractor maintainer hat on, I would say this:
>
> On 2/7/22 10:01 PM, madmurphy wrote:
> > Hi again, GNUnet people.
> >
> > Is this the place where to discuss about libextractor? I have two points.
> >
> > #1 I often see something interesting. Key-value pairs are categorized as
> > |EXTRACTOR_METATYPE_UNKNOWN|:
> >
> > unknown: chroma-format=4:2:0
> > unknown: bit-depth-chroma=8
> > unknown: colorimetry=bt709
> > unknown: stream-format=avc
> > unknown: stream-format=raw
> > unknown: bit-depth-luma=8
> > unknown: base-profile=lc
> > unknown: mpegversion=4
> > unknown: profile=high
> > unknown: alignment=au
> > unknown: parsed=true
> > unknown: framed=true
> > unknown: variant=iso
> > unknown: profile=lc
> > unknown: level=4.1
> >
> > But one point is that they are often numerous, and another point is that
> > that of a key-value type is a really interesting metatype to have (and
> > is not really “unknown”, since the key is self-explanatory). Would it
> > not make sense to add an |EXTRACTOR_METATYPE_KEY_VALUE_PAIR| to the list
> > of MetaTypes?
>
> We could do that. Sometimes I think it would be better to add new
> specific LE types for some of the above, but until that is done, a
> key-value pair type would at least be better than 'unknown'.
>
> > ...
> >
> >   /* generic attributes */
> >   EXTRACTOR_METATYPE_UNKNOWN = 45,
> >   EXTRACTOR_METATYPE_DESCRIPTION = 46,
> >   EXTRACTOR_METATYPE_COPYRIGHT = 47,
> >   EXTRACTOR_METATYPE_RIGHTS = 48,
> >   EXTRACTOR_METATYPE_KEYWORDS = 49,
> >   EXTRACTOR_METATYPE_ABSTRACT = 50,
> >   EXTRACTOR_METATYPE_SUMMARY = 51,
> >   EXTRACTOR_METATYPE_SUBJECT = 52,
> >   EXTRACTOR_METATYPE_CREATOR = 53,
> >   EXTRACTOR_METATYPE_FORMAT = 54,
> >   EXTRACTOR_METATYPE_FORMAT_VERSION = 55,
> >   *EXTRACTOR_METATYPE_KEY_VALUE_PAIR* = XXX,
> >
> > ...
> >
> > #2 I often see that files get tagged with multiple mime types according
> > to libextractor:
> >
> > mimetype: video/quicktime
> > mimetype: video/x-h264
> > mimetype: audio/mpeg
> > mimetype: video/mp4
>
> That is because different plugins (using different methods/libraries)
> disagree on the 'correct' mime-type. Ideally, we'd identify which plugin
> gets it wrong (and why), and unify the mime-types.
>
> > But that never reflects the reality, since files should have only one
> > mime type (or at most, multiple mime types that mean the same thing).
> > But then I see what happens with file names: there is only one
> > |EXTRACTOR_METATYPE_GNUNET_ORIGINAL_FILENAME|, but there can be many
> > |EXTRACTOR_METATYPE_FILENAME|s (in the case of archives, for example):
> >
> > EXTRACTOR_METATYPE_FILENAME = 2,
> > ...
> > EXTRACTOR_METATYPE_GNUNET_ORIGINAL_FILENAME = 180,
> >
> > Would it not make sense to do something similar for mime types? Only one
> > “original mime type”, and an infinity of secondary mime types…?
> >
> > EXTRACTOR_METATYPE_MIMETYPE = 1,
> > ...
> > *EXTRACTOR_METATYPE_GNUNET_ORIGINAL_MIMETYPE* = XXX,
>
> I guess it depends. If this is for archives where files _inside_ the
> archive are given mime-types, then a different metatype makes sense
> (ditto for FILENAME: here we probably could have two types, one for the
> 'archive' and one for the 'contents'). But if the different mime-types
> are all about the 'original' file, then we should rather figure out
> which plugin gets it wrong. As for the "_GNUNET_" in the
> "_GNUNET_ORIGINAL_FILENAME" there, IIRC this again different because
> that is NOT a metatype used by GNU libextractor, but one that GNUnet
> itself generates and puts with the '

Re: printf-like output for gnunet-search

[madmurphy]

e a GNUnet directory with search results at
   FILENAME (e.g. `gnunet-search
   --output=commons.gnd commons`)
  -s, --silent   silent mode (requires the --output argument)
  -t, --timeout=DELAYautomatically terminate search after DELAY (in
   number of microseconds); if 0 or omitted it
   means to wait for CTRL-C
  -V, --verbose  be verbose
  -v, --version  print the version number
Report bugs to gnunet-developers@gnu.org.
Home page: http://www.gnu.org/s/gnunet/
General help using GNU software: http://www.gnu.org/gethelp/

What do you think? If it sounds already too hardcore I will go back to the
previous version. But then we will not have any means to print only
specific metatypes…

Please find the new proposal attached.

--madmurphy

On Mon, Feb 7, 2022 at 12:47 PM madmurphy  wrote:

> Thank you a lot, Carlo.
>
> Yes, --format (--dir-format) was the original name. Then I saw that the
> tendency favoured --printf (--dir-printf) more and I changed it – but I
> have intentionally left -f (-F) for the short name(s). The new name is
> actually inspired to the -printf argument of find (%s and %f have even
> identical meaning in both find and gnunet-search), but if you guys think
> that --format (--dir-format) is better, then I will change it back.
>
> In the meanwhile, silly me, I had forgotten to actually implement the %s
> specifier for the file size. I did that now (patch attached). I also added
> a --silent argument for when people only want to create a GNUnet
> directory and don't care about the output printed on screen. And finally, I
> tried to expand the help page a bit. Here is the new text:
>
> $ gnunet-search --help
>
> gnunet-search [OPTIONS] KEYWORD
> Search for files that have been published on GNUnet
> Arguments mandatory for long options are also mandatory for short options.
>   -a, --anonymity=LEVEL  set the desired LEVEL of receiver-anonymity
>   -c, --config=FILENAME  use configuration file FILENAME
>   -F, --dir-printf=FORMATwrite search results for directories according to
>FORMAT, where %a is the complete list of all
>the printable metadata available (each member
>of which will be displayed according to the
>--prop-printf argument), %f is the directory's
>name, %l is the directory name's length, %m is
>the directory's mime type (always equal to
>`application/gnunet-directory`), %n is the
>search result number, %s is the directory's
>size in bytes and %u is the directory's URI; if
>missing, --dir-printf defaults to the --printf
>argument; if the latter is missing too
>--dir-printf defaults to `#%n:\ngnunet-download
>-o "%f" -R %u\n\n`
>   -f, --printf=FORMATwrite search results according to FORMAT, where
>%a is the complete list of all the printable
>metadata available (each member of which will
>be displayed according to the --prop-printf
>argument), %f is the file's name, %l is the
>file name's length, %m is the file's mime type,
>%n is the search result number, %s is the
>file's size in bytes and %u is the file's URI;
>if missing, --printf defaults to
>`#%n:\ngnunet-download -o "%f" %u\n\n`
>   -h, --help print this help
>   -i, --prop-printf=FORMAT   when the %a format specifier appears in --printf
>or --dir-printf, list each metadata property
>according to FORMAT, where %p is the property's
>content, %l is the content's length in bytes,
>%t is the property type, %i is the property
>type's unique identifier, %n is the property
>number and %w is the name of the plugin that
>provided the information; if missing,
>--prop-printf defaults to `\t%t: %p\n`
>   -L, --log=LOGLEVEL configure logging to use LOGLEVEL
>   -l, --logfile=FILENAME configur

libextractor - key-value pairs and mime types

[madmurphy]

Hi again, GNUnet people.

Is this the place where to discuss about libextractor? I have two points.

#1 I often see something interesting. Key-value pairs are categorized as
EXTRACTOR_METATYPE_UNKNOWN:

unknown: chroma-format=4:2:0
unknown: bit-depth-chroma=8
unknown: colorimetry=bt709
unknown: stream-format=avc
unknown: stream-format=raw
unknown: bit-depth-luma=8
unknown: base-profile=lc
unknown: mpegversion=4
unknown: profile=high
unknown: alignment=au
unknown: parsed=true
unknown: framed=true
unknown: variant=iso
unknown: profile=lc
unknown: level=4.1

But one point is that they are often numerous, and another point is that
that of a key-value type is a really interesting metatype to have (and is
not really “unknown”, since the key is self-explanatory). Would it not make
sense to add an EXTRACTOR_METATYPE_KEY_VALUE_PAIR to the list of MetaTypes?

...

  /* generic attributes */
  EXTRACTOR_METATYPE_UNKNOWN = 45,
  EXTRACTOR_METATYPE_DESCRIPTION = 46,
  EXTRACTOR_METATYPE_COPYRIGHT = 47,
  EXTRACTOR_METATYPE_RIGHTS = 48,
  EXTRACTOR_METATYPE_KEYWORDS = 49,
  EXTRACTOR_METATYPE_ABSTRACT = 50,
  EXTRACTOR_METATYPE_SUMMARY = 51,
  EXTRACTOR_METATYPE_SUBJECT = 52,
  EXTRACTOR_METATYPE_CREATOR = 53,
  EXTRACTOR_METATYPE_FORMAT = 54,
  EXTRACTOR_METATYPE_FORMAT_VERSION = 55,
  *EXTRACTOR_METATYPE_KEY_VALUE_PAIR* = XXX,

...

#2 I often see that files get tagged with multiple mime types according to
libextractor:

mimetype: video/quicktime
mimetype: video/x-h264
mimetype: audio/mpeg
mimetype: video/mp4

But that never reflects the reality, since files should have only one mime
type (or at most, multiple mime types that mean the same thing). But then I
see what happens with file names: there is only one
EXTRACTOR_METATYPE_GNUNET_ORIGINAL_FILENAME, but there can be many
EXTRACTOR_METATYPE_FILENAMEs (in the case of archives, for example):

EXTRACTOR_METATYPE_FILENAME = 2,
...
EXTRACTOR_METATYPE_GNUNET_ORIGINAL_FILENAME = 180,

Would it not make sense to do something similar for mime types? Only one
“original mime type”, and an infinity of secondary mime types…?

EXTRACTOR_METATYPE_MIMETYPE = 1,
...*EXTRACTOR_METATYPE_GNUNET_ORIGINAL_MIMETYPE* = XXX,

So, two simple proposals:

   1. Create EXTRACTOR_METATYPE_KEY_VALUE_PAIR
   2. Create EXTRACTOR_METATYPE_GNUNET_ORIGINAL_MIMETYPE

What do you think? Does it make sense?

--madmurphy

Re: printf-like output for gnunet-search

[madmurphy]

Hi Martin!

Absolutely nothing is wrong with “the look” of the current output (I have
preserved it!). But what the current program cannot do is manipulating it.

The point of having a format field is that it allows to specify a lot of
different input very easily and produce a corresponding amount of output.
However, what actually triggered it was the fact that some information is
not printed at all by the current program (e.g.: the plugin name) and the
possibility to do advanced output manipulation for shell scripting in order
to do advanced search (and I think that the official “flagship” GNUnet
program for searching the network should be able to do that and beyond).

In the future it will be possible to add other ideas easily, but I would
keep it the current proposal for a while first (if it is accepted), because
further expansions require a bit of good planning – by the way, I have
reviewed the code further (only minor things) – you can find it attached to
this email…

Also if this code becomes stable enough, it will become part of GNUnet's
code base it can be reused in the future by other GNUnet command line
utilities that require printf-like capabilities…

--madmurphy

On Sun, Feb 6, 2022 at 3:23 PM Schanzenbach, Martin 
wrote:

> Hi!
>
> If there is a use case for this kind of functionality this lgtm.
> Altough, I do wonder what specifically triggered this. Is there anything
> wrong with the default output?
>
> BR
> Martin
>
> > On 5. Feb 2022, at 09:09, madmurphy  wrote:
> >
> > Okay, after thinking about it I did not like that the --verbose argument
> was ignored when a format was specified. But since, as it turns out, the
> --verbose argument was just a way to print all the metadata, I have added
> an argument for formatting the metadata too. So now we are even. In the
> meanwhile I have also renamed the new arguments and the format specifiers.
> >
> > Again, the help page will explain the new situation:
> >
> > $ gnunet-search --help
> >
> > gnunet-search [OPTIONS] KEYWORD
> > Search GNUnet for files that were published on GNUnet
> > Arguments mandatory for long options are also mandatory for short
> options.
> >   -a, --anonymity=LEVEL  set the desired LEVEL of receiver-anonymity
> >   -c, --config=FILENAME  use configuration file FILENAME
> >   -F, --dir-printf=FORMATwrite the search results for directories
> >according to FORMAT, where %f is the
> >directory's name, %u is the directory's
> URI, %m
> >is the directory's mime type (always
> equal to
> >`application/gnunet-directory`), %n is the
> >search result number and %a is the
> complete
> >list of all the printable metadata
> available,
> >in which each field is displayed
> according to
> >the --prop-printf argument; if missing
> defaults
> >to the --printf argument; if the latter is
> >missing too defaults to
> `#%n:\ngnunet-download
> >-o "%f" -R %u\n\n`
> >   -f, --printf=FORMATwrite the search results according to
> FORMAT,
> >where %f is the file's name, %u is the
> file's
> >URI, %m is the file's mime type, %n is the
> >search result number and %a is the
> complete
> >list of all the printable metadata
> available,
> >in which each field is displayed
> according to
> >the --prop-printf argument; if missing
> defaults
> >to `#%n:\ngnunet-download -o "%f" %u\n\n`
> >   -h, --help print this help
> >   -L, --log=LOGLEVEL configure logging to use LOGLEVEL
> >   -l, --logfile=FILENAME configure logging to write logs to FILENAME
> >   -N, --results=VALUEautomatically terminate search after VALUE
> >results are found
> >   -n, --no-network   only search the local peer (no P2P network
> >search)
> >   -o, --output=PREFIXwrite search results to file starting with
> PREFIX
> >   -p, --prop-printf=FORMAT   when the %a format specifier appears in
> --printf
> >or --dir-printf, list each property
> according
> >to

Re: printf-like output for gnunet-search

[madmurphy]

e it is a standalone program, if you want to play with this
patched version without having to recompile the entire GNUnet, you can just
launch:

unzip gnunet-search.rev2.zip
ln -s /usr/include/gnunet/gnunet_fs_service.h gnunet_fs_service.h
ln -s /usr/include/gnunet/platform.h platform.h
gcc -lgnunetutil -lgnunetfs -I/usr/include/gnunet -o
'gnunet-search-test' gnunet-search.c
./gnunet-search-test -f '%n. %f\n' 'commons'

(if you follow these steps the program will be compiled without libextractor
and the %t specifier in --prop-printf will not be available.)

--madmurphy

On Fri, Feb 4, 2022 at 6:21 PM madmurphy  wrote:

> Hi GNUnet folks!
>
> I have edited the gnunet-search utility to accept a --format and a
> --dir-format parameters and produce a printf-like output. It was pretty
> easy to do, I only had to edit gnunet/src/fs/gnunet-search.c
> <https://git.gnunet.org/gnunet.git/tree/src/fs/gnunet-search.c?id=65f9e37ce036acdfab29b25b9b6f69de1b126962>.
> The new text printed by gnunet-search --help can explain the two new
> arguments in detail:
>
> $ gnunet-search --help
>
> gnunet-search [OPTIONS] KEYWORD
> Search GNUnet for files that were published on GNUnet
> Arguments mandatory for long options are also mandatory for short options.
>   -a, --anonymity=LEVEL  set the desired LEVEL of receiver-anonymity
>   -c, --config=FILENAME  use configuration file FILENAME
>   -F, --dir-format=DIRFORMAT write search results for directories according to
>DIRFORMAT, where %n is the result number, %f is
>the file's name and %u is the file's URI; if
>missing defaults to FORMAT; if the latter is
>missing too defaults to '#%n:\ngnunet-download
>-o "%f" -R %u\n\n'
>   -f, --format=FORMATwrite search results according to FORMAT, where
>%n is the result number, %f is the file's name
>and %u is the file's URI; if missing defaults
>to '#%n:\ngnunet-download -o "%f" %u\n\n')
>   -h, --help print this help
>   -L, --log=LOGLEVEL configure logging to use LOGLEVEL
>   -l, --logfile=FILENAME configure logging to write logs to FILENAME
>   -N, --results=VALUEautomatically terminate search after VALUE
>results are found
>   -n, --no-network   only search the local peer (no P2P network
>search)
>   -o, --output=PREFIXwrite search results to file starting with PREFIX
>   -t, --timeout=DELAYautomatically terminate search after DELAY
>   -V, --verbose  be verbose
>   -v, --version  print the version number
> Report bugs to gnunet-developers@gnu.org.
> Home page: http://www.gnu.org/s/gnunet/
> General help using GNU software: http://www.gnu.org/gethelp/
>
> Basically with this new patch launching
>
> gnunet-search commons
>
> and launching
>
> gnunet-search --format='#%n:\ngnunet-download -o "%f" %u\n\n' \
>   --dir-format='#%n:\ngnunet-download -o "%f" -R %u\n\n' \
>   commons
>
> are equivalent.
>
> What do you think? I hope you like the idea.
>
> I have done my best to stick to the coding style of the rest of the
> program. But a good code review is more than welcomed.
>
> Please find attached a patch, or alternatively the single file I have
> edited.
>
> --madmurphy
>
<>
<>

printf-like output for gnunet-search

[madmurphy]

Hi GNUnet folks!

I have edited the gnunet-search utility to accept a --format and a
--dir-format parameters and produce a printf-like output. It was pretty
easy to do, I only had to edit gnunet/src/fs/gnunet-search.c
<https://git.gnunet.org/gnunet.git/tree/src/fs/gnunet-search.c?id=65f9e37ce036acdfab29b25b9b6f69de1b126962>.
The new text printed by gnunet-search --help can explain the two new
arguments in detail:

$ gnunet-search --help

gnunet-search [OPTIONS] KEYWORD
Search GNUnet for files that were published on GNUnet
Arguments mandatory for long options are also mandatory for short options.
  -a, --anonymity=LEVEL  set the desired LEVEL of receiver-anonymity
  -c, --config=FILENAME  use configuration file FILENAME
  -F, --dir-format=DIRFORMAT write search results for directories according to
   DIRFORMAT, where %n is the result number, %f is
   the file's name and %u is the file's URI; if
   missing defaults to FORMAT; if the latter is
   missing too defaults to '#%n:\ngnunet-download
   -o "%f" -R %u\n\n'
  -f, --format=FORMATwrite search results according to FORMAT, where
   %n is the result number, %f is the file's name
   and %u is the file's URI; if missing defaults
   to '#%n:\ngnunet-download -o "%f" %u\n\n')
  -h, --help print this help
  -L, --log=LOGLEVEL configure logging to use LOGLEVEL
  -l, --logfile=FILENAME configure logging to write logs to FILENAME
  -N, --results=VALUEautomatically terminate search after VALUE
   results are found
  -n, --no-network   only search the local peer (no P2P network
   search)
  -o, --output=PREFIXwrite search results to file starting with PREFIX
  -t, --timeout=DELAYautomatically terminate search after DELAY
  -V, --verbose  be verbose
  -v, --version  print the version number
Report bugs to gnunet-developers@gnu.org.
Home page: http://www.gnu.org/s/gnunet/
General help using GNU software: http://www.gnu.org/gethelp/

Basically with this new patch launching

gnunet-search commons

and launching

gnunet-search --format='#%n:\ngnunet-download -o "%f" %u\n\n' \
--dir-format='#%n:\ngnunet-download -o "%f" -R %u\n\n' \
commons

are equivalent.

What do you think? I hope you like the idea.

I have done my best to stick to the coding style of the rest of the
program. But a good code review is more than welcomed.

Please find attached a patch, or alternatively the single file I have
edited.

--madmurphy
<>
<>

New library: GNUnet Worker

[madmurphy]

Hi everyone!

It happened that I needed a way to launch the GNUnet scheduler in a
dedicated thread and schedule tasks from other threads, so I came out with
a general purpose library for multithreading with GNUnet: the *GNUnet
Worker* library.

The code is still in development and will probably change quite a bit. In
the meanwhile I would be happy if anyone would like to give it a try and
experiment with it. The package is now available at
https://github.com/madmurphy/libgnunet-worker.

I have been already in contact with Christian, Martin and TheJackiMonster,
who gave me precious suggestions and who I would like to thank.

For further information please find below the README file.

--madmurphy

GNUnet Worker

Multithreading with GNUnet
Overview

As it is often the case with network applications, GNUnet is built
following a single-threaded event-driven model. This is an optimal model
when dealing with high concurrency scenarios, but is poorly suited for
other contexts (e.g. graphical user interfaces).

To accomplish its event-driven flow, GNUnet uses a scheduler. Once such
scheduler is started, it is not designed to be invoked by other threads,
but can schedule only routines requested by its own thread. What to do then
if an application needs to deal with multiple threads and let the latter
interface with the GNUnet scheduler?

This framework offers a simple solution by creating a “bearing” between the
threads and the scheduler. The latter is run in its own dedicated thread
and is unaware of the existence of other threads. Such a bearing consists
in a “wish list” of routines to schedule, which can be populated
asynchronously by any thread and gets emptied synchronously only by the
scheduler according to the latter's natural flow.

When using this framework, threads must never invoke the scheduler directly
(preferably they must not even include gnunet/gnunet_scheduler_lib.h in
their scope), and can only use GNUNET_WORKER_push_load() or
GNUNET_WORKER_push_load_with_priority() to schedule new functions. They
will also have access to all the functions declared in
gnunet/gnunet_worker_lib.h.

Functions scheduled in this way, on the other hand, will have full access
to the scheduler's framework and will follow its single-threaded
event-driven flow (indeed they will run in the scheduler's thread).

Under examples/articulated-example you can find an example of such division
of labour, where all-other-threads.c launches multiple threads and
gnunet-thread.c contains only functions that will run in the scheduler's
thread.
Simple example

#include 
#include 

static void task_for_the_scheduler (void * const data) {

printf("Hello world\n");

}

int main (const int argc, const char * const * const argv) {

/*  Create a separate thread where GNUnet's scheduler is run  */
GNUNET_WORKER_Handle * my_current_worker = GNUNET_WORKER_create(
NULL,
NULL,
NULL
);

/*  Run a function in the scheduler's thread  */
GNUNET_WORKER_push_load(my_current_worker, _for_the_scheduler, NULL);

/*  Make sure that threads have had enough time to start...  */
sleep(1);

/*  Shutdown the scheduler and wait until it returns  */
GNUNET_WORKER_synch_destroy(my_current_worker);

}

See the examples subdirectory for further examples.
A minimal tutorial

There are three ways to create a GNUnet worker:

   1. GNUNET_WORKER_create(): the GNUnet scheduler will be launched in a
   new thread, equipped with a “load listener” for scheduling routines pushed
   by other threads
   2. GNUNET_WORKER_start_serving(): the current thread will launch the
   GNUnet scheduler and equip it with a “load listener” for scheduling
   routines pushed by other threads
   3. GNUNET_WORKER_adopt_running_scheduler(): this function assumes that
   the GNUnet scheduler is already running in the current thread (i.e. the
   user has previously launched either GNUNET_SCHEDULER_run() or
   GNUNET_PROGRAM_run()) and equipping the scheduler with a “load listener”
   for scheduling routines pushed by other threads has become necessary

As soon as a handle for a new worker is made available, it is immediately
possible to push load into it using GNUNET_WORKER_push_load() or
GNUNET_WORKER_push_load_with_priority(). The routines added in this way
will be launched asynchronously in the worker's thread.

There are three ways for terminating a worker and shutting down its
associated GNUnet scheduler:

   1. GNUNET_WORKER_asynch_destroy(): the worker will be terminated and its
   memory freed, without waiting for the scheduler to complete the shutdown
   (asynchronous)
   2. GNUNET_WORKER_synch_destroy(): the worker will be terminated and its
   memory freed, waiting for the scheduler to complete the shutdown
   (synchronous, “join”)
   3. GNUNET_WORKER_timedsynch_destroy(): the worker will be terminated and
   its memory freed, waiting for the scheduler to complete the shutdown only
   if this happens within a ce

Re: Git server migration

[madmurphy]

 Any plans to install GitLab FOSS 
on the GNUnet git server (i.e., a fork of GitLab with all proprietary code
removed)?

On Wed, Sep 1, 2021 at 6:21 PM Schanzenbach, Martin 
wrote:

> Hi,
>
> along with some other services, we are moving the gitolite service to
> another server.
> Unfortunately, we cannot move the gnunet.org domain just yet, so you may
> have to change your
> repository urls from g...@gnunet.org/repo to g...@git.gnunet.org/repo
> ( you need to add a "git." in front of gnunet.org)
>
> In case this should not already work for you, please contact me.
> The cgit web UI is still under construction.
>
> Sorry for the inconvenience.
>
> Martin
>