Re: [fossil-users] "fossil http" doubts

[Ross Berteig]

On 6/9/2017 1:21 PM, Warren Young wrote:

On Jun 9, 2017, at 12:17 PM, Ross Berteig  wrote:

I do think that the JSON support is close to solid enough to be on by default.

For functionality alone, that is surely true, but in the face of malice?  
Parsers are notoriously difficult to make bomb-proof.

Even if the JSON API is 100% solid, it acts as an API to the rest of Fossil.  
Some fuzzing the JSON API might find a way to break Fossil itself, a good thing 
if we do it before the black hats do.


I agree 100%, which is why I haven't pushed to flip the configure script 
option to enable JSON by default.


I hope to be buying some round tuits soon.

I also encourage others to build with JSON enabled and try to break it. 
Ideally then reducing any breaks down to a minimal sample so we can add 
them to the test suite for regression testing.


--
Ross Berteig   r...@cheshireeng.com
Cheshire Engineering Corp.   http://www.CheshireEng.com/
+1 626 303 1602

___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] "fossil http" doubts

[Warren Young]

On Jun 9, 2017, at 12:17 PM, Ross Berteig  wrote:
> 
> I do think that the JSON support is close to solid enough to be on by default.

For functionality alone, that is surely true, but in the face of malice?  
Parsers are notoriously difficult to make bomb-proof.

Even if the JSON API is 100% solid, it acts as an API to the rest of Fossil.  
Some fuzzing the JSON API might find a way to break Fossil itself, a good thing 
if we do it before the black hats do.

___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] "fossil http" doubts

[Ross Berteig]

On 6/8/2017 9:17 PM, Stephan Beal wrote:
On Thu, Jun 8, 2017 at 10:43 PM, Ross Berteig > wrote:


For building tools to generally interact with a repo, take a look
at the JSON support. It's (still) not compiled in by default, but
builds easily if requested by ./configure --json.


Trivia: the reason it's not compiled in by default is the lack of test 
coverage. Richard told me, way back when (2012, maybe?), that we could 
enable it if it had complete test coverage, including fuzz tests 
(making sure that random/garbage inputs/attacks don't break the repo, 
allow DoS, butter-overrun attacks, and similar). i  never got 
around to doing that  and am now physically incapable of doing 
so. Thus... if it's going to happen, someone else will have to do it :/.


I created the existing JSON tests as a first step down that path. 
Certainly not complete coverage yet, and no fuzz testing yet. The 
existing tests call every documented JSON API at least once, and go to 
some effort to exercise more features of some than others. I put some 
effort into trying to trigger every documented error response code, but 
IIRC there were a couple that I never found a reliable way to provoke.


I think fuzz testing is a great idea against all of fossil, but haven't 
taken the time (yet) to begin working on it. Covering all of the 
/webpages is probably wise.


I did run the test suite over a build of fossil instrumented to measure 
test coverage once. There are plenty of areas where more tests are welcome.


I haven't rocked the boat lately since I'm comfortable with using my own 
builds internally, but I do think that the JSON support is close to 
solid enough to be on by default.


--
Ross Berteig   r...@cheshireeng.com
Cheshire Engineering Corp.   http://www.CheshireEng.com/
+1 626 303 1602

___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] "fossil http" doubts

[Johan Kuuse]

> A test case that validates all of the HTML output would be great. That
> should be content agnostic, of course, so that it can be maintained without
> requiring a lot of work for new versions.


IMHO, the W3C HTML Validator
https://validator.w3.org/
has always done a good job for validating HTML, both off- and online,
so I have no intention to reinvent the wheel with a new tool.
I just tested their new version, Nu Html Checker
https://validator.w3.org/nu/about.html
running a small script to validate the Fossil built-in web pages.

Maybe this discussion should be continued on the fossil-dev list instead.
Anyway, here goes the script (run it anywhere inside a Fossil repository):

---
#!/bin/sh

# Validate Fossil built-in web pages with the W3C Nu HTML Checker
# https://validator.w3.org/nu/about.html
# The validator requires Java 8 to be installed.
# Install prequisites (tested on Ubuntu):
#   sudo apt-get install default-jre
#   wget 
https://github.com/validator/validator/releases/download/17.3.0/vnu.jar_17.3.0.zip
#   unzip vnu.jar_17.3.0.zip

mkdir -p fhtml
for w in `fossil help -w | xargs printf "%s\n"`
do
printf "GET $w HTTP/1.0\n\n" | fossil test-http > fhtml/${w}.html
grep -q 'Content-Type: text/html' fhtml/${w}.html
if [ $? -ne 0 ];then
# Delete non-HTML files
rm fhtml/${w}.html
else
# Strip HTTP header from HTML files
sed -i '1,/^\r\{0,1\}$/d' fhtml/${w}.html
fi
done
java -jar dist/vnu.jar fhtml/* > validate.log 2>&1
---

Short resume of validation result:

There are lots of closing  tags in the Fossil HTML, which is
invalid in HTML5. Just remove them.
For other errors, check validate.log

BR,
Johan
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] "fossil http" doubts

[Johan Kuuse]

> A test case that validates all of the HTML output would be great. That
> should be content agnostic, of course, so that it can be maintained without
> requiring a lot of work for new versions.


IMHO, the W3C HTML Validator
https://validator.w3.org/
has always done a good job for validating HTML, both off- and online,
so I have no intention to reinvent the wheel with a new tool.
I just tested their new version, Nu Html Checker
https://validator.w3.org/nu/about.html
running a small script to validate the Fossil built-in web pages.

Maybe this discussion should be continued on the fossil-dev list instead.
Anyway, here goes the script (run it anywhere inside a Fossil repository):

---
#!/bin/sh

# Validate Fossil built-in web pages with the W3C Nu HTML Checker
# https://validator.w3.org/nu/about.html
# The validator requires Java 8 to be installed.
# Install prequisites (tested on Ubuntu):
#   sudo apt-get install default-jre
#   wget 
https://github.com/validator/validator/releases/download/17.3.0/vnu.jar_17.3.0.zip
#   unzip vnu.jar_17.3.0.zip

mkdir -p fhtml
for w in `fossil help -w | xargs printf "%s\n"`
do
printf "GET $w HTTP/1.0\n\n" | fossil test-http > fhtml/${w}.html
grep -q 'Content-Type: text/html' fhtml/${w}.html
if [ $? -ne 0 ];then
# Delete non-HTML files
rm fhtml/${w}.html
else
# Strip HTTP header from HTML files
sed -i '1,/^\r\{0,1\}$/d' fhtml/${w}.html
fi
done
java -jar dist/vnu.jar fhtml/* > validate.log 2>&1
---

Short resume of validation result:

There are lots of closing  tags in the Fossil HTML, which is
invalid in HTML5. Just remove them.
For other errors, check validate.log

BR,
Johan
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] "fossil http" doubts

[Stephan Beal]

On Thu, Jun 8, 2017 at 10:43 PM, Ross Berteig  wrote:

> For building tools to generally interact with a repo, take a look at the
> JSON support. It's (still) not compiled in by default, but builds easily if
> requested by ./configure --json.


Trivia: the reason it's not compiled in by default is the lack of test
coverage. Richard told me, way back when (2012, maybe?), that we could
enable it if it had complete test coverage, including fuzz tests (making
sure that random/garbage inputs/attacks don't break the repo, allow DoS,
butter-overrun attacks, and similar). i  never got around to doing
that  and am now physically incapable of doing so. Thus... if it's
going to happen, someone else will have to do it :/.

-- 
- stephan beal
http://wanderinghorse.net/home/stephan/
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] "fossil http" doubts

[Ross Berteig]

On 6/7/2017 6:42 AM, Johan Kuuse wrote:

Thanks for the input.
"fossil test-http" did the trick:

webpage=/setup
printf "GET $webpage HTTP/1.0\n\n" | fossil test-http

My purposes for parsing the built-in web pages are basically two:

1. I want to hack the Fossil code:
 a. I parse and save the HTML from the builtin pages (excluding
pages where the output makes no sense, for example /zip)
 b. I modify the source code (related to one or more web pages)
 c. I parse the same pages again to check that I have changed only
the web page output I intended to change. A kind of self test before
committing.

This WoW is obviously only helpful for a developer changing the Fossil
source code of one single (official) commit.
It does not make sense to save the output for later use and compare it
between commits, as the content/structure may change.


Take a look at the test harness in the test folder of the repository. 
It's in Tcl, and has pretty good coverage of the internals despite a 
lack of dedicated test developers. Most of the tests drive the CLI. Even 
the tests for fossil json largely drive the CLI instead of /json URLs 
via HTTP, but there are examples in there.


Naturally, I'd love to see the test suite expand to further coverage of 
the web face. So if you have thoughts about that, chime in here, or on 
the fossil-dev list where chatter about things like test harnesses won't 
distract the end users.


For building tools to generally interact with a repo, take a look at the 
JSON support. It's (still) not compiled in by default, but builds easily 
if requested by ./configure --json.



2. I want to validate the web pages: Validate the HTML, check for
broken links, etc, using for example the W3C validation tools.

This kind of validation could be used both to find existing, and to
avoid introducing new invalid HTML.
Could possibly be useful both for programmers and for skin-makers.


A test case that validates all of the HTML output would be great. That 
should be content agnostic, of course, so that it can be maintained 
without requiring a lot of work for new versions.



--
Ross Berteig   r...@cheshireeng.com
Cheshire Engineering Corp.   http://www.CheshireEng.com/
+1 626 303 1602

___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] "fossil http" doubts

[Johan Kuuse]

On Wed, Jun 7, 2017 at 3:50 PM, Warren Young  wrote:
> On Jun 7, 2017, at 7:42 AM, Johan Kuuse  wrote:
>>
>> 2. I want to validate the web pages: Validate the HTML, check for
>> broken links, etc, using for example the W3C validation tools.
>
> If you’re using something like curl or wget to pull the web pages, there’s 
> typically a way to set up a “cookie jar” so that you can log in with one HTTP 
> request, then make the remaining HTTP requests as that user, with the HTTP 
> client automatically sending the necessary session cookie.


Thanks for the suggestion, but I wanted to avoid both a running web
server and the cookie jar setup.
"fossil test-http" made my day.
Get the HTML (including the HTTP Response header) from all Fossil
built-in web pages, using the output from 'fossil help -w' as a list:

mkdir -p fhtml && for w in `fossil help -w | xargs printf "%s\n"`; do
printf "GET $w HTTP/1.0\n\n" | fossil test-http > fhtml/${w}.html;done

BR,
Johan
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] "fossil http" doubts

[Warren Young]

On Jun 7, 2017, at 7:42 AM, Johan Kuuse  wrote:
> 
> 2. I want to validate the web pages: Validate the HTML, check for
> broken links, etc, using for example the W3C validation tools.

If you’re using something like curl or wget to pull the web pages, there’s 
typically a way to set up a “cookie jar” so that you can log in with one HTTP 
request, then make the remaining HTTP requests as that user, with the HTTP 
client automatically sending the necessary session cookie.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] "fossil http" doubts

[Johan Kuuse]

Thanks for the input.
"fossil test-http" did the trick:

webpage=/setup
printf "GET $webpage HTTP/1.0\n\n" | fossil test-http

My purposes for parsing the built-in web pages are basically two:

1. I want to hack the Fossil code:
a. I parse and save the HTML from the builtin pages (excluding
pages where the output makes no sense, for example /zip)
b. I modify the source code (related to one or more web pages)
c. I parse the same pages again to check that I have changed only
the web page output I intended to change. A kind of self test before
committing.

This WoW is obviously only helpful for a developer changing the Fossil
source code of one single (official) commit.
It does not make sense to save the output for later use and compare it
between commits, as the content/structure may change.


2. I want to validate the web pages: Validate the HTML, check for
broken links, etc, using for example the W3C validation tools.

This kind of validation could be used both to find existing, and to
avoid introducing new invalid HTML.
Could possibly be useful both for programmers and for skin-makers.


BR,
Johan



On Wed, Jun 7, 2017 at 11:53 AM, Stephan Beal  wrote:
> On Wed, Jun 7, 2017 at 8:48 AM, Johan Kuuse  wrote:
>>
>> My idea is to make a script which parses the output from all builtin
>> pages.
>
>
> Be aware that fossil makes NO GUARANTEES about the stability of
> content/structure of any pages (or CLI command output, for that matter).
> Thus any work you do on parsing any pages might be undone by any given
> commit to the main fossil tree.
>
> --
> - stephan beal
> http://wanderinghorse.net/home/stephan/
> "Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
> those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
>
> ___
> fossil-users mailing list
> fossil-users@lists.fossil-scm.org
> http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
>
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] "fossil http" doubts

[Stephan Beal]

On Wed, Jun 7, 2017 at 8:48 AM, Johan Kuuse  wrote:

> My idea is to make a script which parses the output from all builtin pages.
>

Be aware that fossil makes NO GUARANTEES about the stability of
content/structure of any pages (or CLI command output, for that matter).
Thus any work you do on parsing any pages might be undone by any given
commit to the main fossil tree.

-- 
- stephan beal
http://wanderinghorse.net/home/stephan/
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] "fossil http" doubts

[Martin Gagnon]

> Le 7 juin 2017 à 02:48, Johan Kuuse  a écrit :
> 
>> On Tue, Jun 6, 2017 at 6:15 PM, Richard Hipp  wrote:
>>> On 6/6/17, Johan Kuuse  wrote:
>>> Hi,
>>> 
>>> Is there any way to access the Fossil built-in webpages using the "fossil
>>> http"?
>>> In the example below, I try to access the /setup page from localhost,
>>> but it seems I don't get authorized. Instead I get redirected to the
>>> /login page.
>> 
>> You have to log in as a user with "s" (superuser) permission in order
>> to access the /setup page.  If you try to access /setup without the
>> right permissions, you get redirected.
>> 
>> Log in first, then try again.
> 
> Maybe I did not make myself very clear.
> I am running "fossil http" from the command line.
> Is there a way to first "login" from the command line, and then access
> built-in pages such as "/setup", also from the command line?
> 
> My idea is to make a script which parses the output from all builtin pages.
> 

I'm not sure to understand what you want to do but you probably want to use: 
"fossil test-http" instead. 
  https://www.fossil-scm.org/index.html/help?cmd=test-http

-- 
Martin G.___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] "fossil http" doubts

[Johan Kuuse]

On Tue, Jun 6, 2017 at 6:15 PM, Richard Hipp  wrote:
> On 6/6/17, Johan Kuuse  wrote:
>> Hi,
>>
>> Is there any way to access the Fossil built-in webpages using the "fossil
>> http"?
>> In the example below, I try to access the /setup page from localhost,
>> but it seems I don't get authorized. Instead I get redirected to the
>> /login page.
>
> You have to log in as a user with "s" (superuser) permission in order
> to access the /setup page.  If you try to access /setup without the
> right permissions, you get redirected.
>
> Log in first, then try again.

Maybe I did not make myself very clear.
I am running "fossil http" from the command line.
Is there a way to first "login" from the command line, and then access
built-in pages such as "/setup", also from the command line?

My idea is to make a script which parses the output from all builtin pages.

BR,
Johan


>
> --
> D. Richard Hipp
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] "fossil http" doubts

[Richard Hipp]

On 6/6/17, Johan Kuuse  wrote:
> Hi,
>
> Is there any way to access the Fossil built-in webpages using the "fossil
> http"?
> In the example below, I try to access the /setup page from localhost,
> but it seems I don't get authorized. Instead I get redirected to the
> /login page.

You have to log in as a user with "s" (superuser) permission in order
to access the /setup page.  If you try to access /setup without the
right permissions, you get redirected.

Log in first, then try again.

-- 
D. Richard Hipp
d...@sqlite.org
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users