Re: [fossil-users] REST API and client for same

[Paul Hammant]

OK, so I don't think there's any interest in this beyond me :-(

On Wed, Apr 5, 2017 at 1:58 PM, Paul Hammant  wrote:

> I don't really need Fossil to become an application server.  I just need
> it to handle CRUD over HTTPS on specific resources, and have configurable
> permissions for that.  Though TH1 scripts exist, I'd not use them, nor
> anything that purports to be JSP/ASP application scripting model. I'd not
> need them, if I've shifted my application code to JavaScript.
>
> Just like for CouchDB in a "backend as a service" configuration, *in-house non
> financially critical* *applications* are the type of apps that would fit
> well. CouchDB implements CORS
> 
> which all the modern browsers are savvy with too, and that is a partial
> answer to XSS/CSRF concerns.
>
> Anyway - I'd want everything that CouchDB does in the BaaS
>  model, but lose the
> query capability if I had to. I'd want to gain security features that CouchDB
> doesn't do by default  
> ("AdminParty"
> and HTTP *off* by default, HTTPS *on* by default).  Why VCS - who
> wouldn't want to be able to checkout documents as a set, perform functions
> on them, and commit them back as a set, atomically?
>
> - Paul
>
> - Paul
>
> On Tue, Apr 4, 2017 at 2:53 PM, Warren Young  wrote:
>
>> On Apr 4, 2017, at 11:24 AM, Paul Hammant  wrote:
>> >
>> > > I have little need for such a thing myself, so I’m just throwing this
>> idea out
>> > > there for anyone who thinks it looks like a good itch to scratch.
>> >
>> > I do have a need for this class of use. My thread "Fossil as an app
>> server" (nearly a week ago on this list) is in the same direction.
>>
>> Only in the vaguest sort of way.
>>
>> My idea is just that you should be able to replace the fossil binary as a
>> client with a series of HTTP calls, which lets you replace
>> fossil-the-client without duplicating all of its internal functionality.
>>
>> This idea of turning Fossil into a generic application server is off on a
>> completely wild tangent from that point.
>>
>> While thinking about this sort of thing, consider the XSS problem just
>> brought up on the mailing list.  To me, “generic application server”
>> implies multiple independent — possibly mutually-untrusting — applications
>> running on a single platform.  So, you’d better solve the XSS problem first.
>> ___
>> fossil-users mailing list
>> fossil-users@lists.fossil-scm.org
>> http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
>>
>
>
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] REST API and client for same

[Paul Hammant]

I don't really need Fossil to become an application server.  I just need it
to handle CRUD over HTTPS on specific resources, and have configurable
permissions for that.  Though TH1 scripts exist, I'd not use them, nor
anything that purports to be JSP/ASP application scripting model. I'd not
need them, if I've shifted my application code to JavaScript.

Just like for CouchDB in a "backend as a service" configuration, *in-house non
financially critical* *applications* are the type of apps that would fit
well. CouchDB implements CORS

which all the modern browsers are savvy with too, and that is a partial
answer to XSS/CSRF concerns.

Anyway - I'd want everything that CouchDB does in the BaaS
 model, but lose the
query capability if I had to. I'd want to gain security features that CouchDB
doesn't do by default
 ("AdminParty"
and HTTP *off* by default, HTTPS *on* by default).  Why VCS - who wouldn't
want to be able to checkout documents as a set, perform functions on them,
and commit them back as a set, atomically?

- Paul

- Paul

On Tue, Apr 4, 2017 at 2:53 PM, Warren Young  wrote:

> On Apr 4, 2017, at 11:24 AM, Paul Hammant  wrote:
> >
> > > I have little need for such a thing myself, so I’m just throwing this
> idea out
> > > there for anyone who thinks it looks like a good itch to scratch.
> >
> > I do have a need for this class of use. My thread "Fossil as an app
> server" (nearly a week ago on this list) is in the same direction.
>
> Only in the vaguest sort of way.
>
> My idea is just that you should be able to replace the fossil binary as a
> client with a series of HTTP calls, which lets you replace
> fossil-the-client without duplicating all of its internal functionality.
>
> This idea of turning Fossil into a generic application server is off on a
> completely wild tangent from that point.
>
> While thinking about this sort of thing, consider the XSS problem just
> brought up on the mailing list.  To me, “generic application server”
> implies multiple independent — possibly mutually-untrusting — applications
> running on a single platform.  So, you’d better solve the XSS problem first.
> ___
> fossil-users mailing list
> fossil-users@lists.fossil-scm.org
> http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
>
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] REST API and client for same

[Warren Young]

On Apr 4, 2017, at 11:24 AM, Paul Hammant  wrote:
> 
> > I have little need for such a thing myself, so I’m just throwing this idea 
> > out
> > there for anyone who thinks it looks like a good itch to scratch.
> 
> I do have a need for this class of use. My thread "Fossil as an app server" 
> (nearly a week ago on this list) is in the same direction. 

Only in the vaguest sort of way.

My idea is just that you should be able to replace the fossil binary as a 
client with a series of HTTP calls, which lets you replace fossil-the-client 
without duplicating all of its internal functionality.

This idea of turning Fossil into a generic application server is off on a 
completely wild tangent from that point.

While thinking about this sort of thing, consider the XSS problem just brought 
up on the mailing list.  To me, “generic application server” implies multiple 
independent — possibly mutually-untrusting — applications running on a single 
platform.  So, you’d better solve the XSS problem first.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] REST API and client for same

[Paul Hammant]

>
>
> I have little need for such a thing myself, so I’m just throwing this idea
> out there for anyone who thinks it looks like a good itch to scratch.
>


I do have a need for this class of use. My thread "Fossil as an app server"
(nearly a week ago on this list) is in the same direction.

Here's something I made 8/9 years ago -  a WYSInWYG wiki on top of
Subversion using it's auto-increment feature - https://www.youtube.com/watc
h?v=WfjK0Pb6IIM. and GET/PUT (and an application in the middle). It bumped
head revision and did nothing for clash detection. I'm not going to breath
life into that again.

Here's where I'm going more recently -

0. Here's the "seatmap" app I made using CouchDB again -
http://paulhammant.com/2015/12/21/angular-and-svg-and-couchdb/. It works
with CORS enabled directly from JavaScript to CouchDB - REST-centric
PUT/GET of a resource.  CouchDB isn't a SCM, but it could be with some
enhancements. Prospective users of CouchDB should know it is insecure in
ways that security experts roll their eyes - https://blog.couchdb.
org/2017/01/24/couchdb-ransom-notes/.

1. http://paulhammant.com/2016/06/26/using-rhodecode-and-
angular1-as-an-editor-for-a-config-as-code-system/ - the last incarnation
of a thing I'm still chasing. An Angular 'form' - directly saving JSON to
HEAD revision of a Subversion or Git or Mercurial repo. Heavy lifting
courtesy of Rhodecode, which is probably maintaining working copy for me,
somehow. While the class of application (SCM backed apps) is what I'm
after, there's no REST involved here.

2. Here's me stretching my skills to make a similar editor for Github as a
backing store - http://paulhammant.com/2015/06/07/custom-json-editors-for-
github/. Again the right class of application, but no REST involved.

3. Going back in time - I coached a new starter (Logan McGrath) at my old
company into making a real config editing thing with Perforce as a backing
store - http://loganmcgrath.com/blog/2012/11/16/scm-backed-applica
tion-configuration-with-perforce/.  We wrote this thing, so we sure as shit
knew we were maintaining working copy for the client on the server. Right
class of application, no REST involved.

4. A pre-cursor for #3 - http://paulhammant.com/2012/
08/14/app-config-using-git-and-angular/ - using Git a the backing store

My nirvana is Fossil upgraded to allow discrete CRUD operations over HTTPS.
I'd prefer REST, but will live with anything that I can perform from
JavaScript in a page that was a] served up (with mime types) from the same
HTTPS interface, or b] from another site/server and interoperating with
Fossil resources via CORS.  Towards that #0 is the reference that's
relevant for the 'ask' now, and #1-#4 are supporting on the class of
application, not the precise mechanism (REST or something close enough).

I'm pretty sure it's just a question of time until doc-store and VCS
convergence happens.

As I blog around this topic area,
http://paulhammant.com/2012/11/20/very-small-data/ might be interesting too,

- Paul
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] REST API and client for same

[Stephan Beal]

On Mon, Apr 3, 2017 at 3:25 PM, Warren Young  wrote:

> On Apr 3, 2017, at 1:29 AM, Stephan Beal  wrote:
> >
> > Commits can't be done without a checkout
>
> Given a way to ask Fossil over HTTP for the set of artifacts that makes up
> $reference, where the latter is anything Fossil currently accepts in
> “fossil up $reference” you’ll then have a way to check out that version:
>

Unfortunately my aching hands won't let me respond nearly as fully as i'd
like, so i'm just gonna leave it at:

emacs

;)

(Today the doc finally passed me on to a neurosurgeon, meaning the way for
an operation is paved. With any luck, i'll be able to type full-time again
within a few months.)

-- 
- stephan beal
http://wanderinghorse.net/home/stephan/
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] REST API and client for same

[Warren Young]

On Apr 3, 2017, at 1:29 AM, Stephan Beal  wrote:
> 
> Commits can't be done without a checkout

Given a way to ask Fossil over HTTP for the set of artifacts that makes up 
$reference, where the latter is anything Fossil currently accepts in “fossil up 
$reference” you’ll then have a way to check out that version:

$ frapi checkout https://m...@fossil-scm.org
Password, remember, etc.
…fossil-scm/ created and populated with tip-of-trunk
$ cd fossil-scm
…hack, hack, hack…
$ frapi ci
…diff checkout against pristine copies, send diffs up to server

This hypothetical lightweight client (which I’m calling frapi) would then 
record the version it checked out, so that on checkin, it can run the diffs 
against the pristine copies it squirreled away (like Subversion does) and 
submit the diffs to the backend to be checked in or rejected.

On success, it gets back a new checkin ID, which it records in the same place 
it recorded the initial checkin ID, ready to go again.

I think a local JSON file would make sense for that place, again a lightweight 
alternative to .fslckout and ~/.fossil.  This client already has to have a JSON 
parser built in, but does *not* have a reason for an embedded SQLite DB.

The use case that prompted all of this — and it is admittedly silly — is 
thinking about what it would take to port Fossil to a PDP-11.  Whether anyone 
actually does that is not the point.  The point is that such constraints should 
guide the design of this lightweight client.  

Such a client would necessarily be different from the current Fossil 
implementation.  We must get something new and different out of it, else why 
bother?

Incidentally, I don’t mean with all of this to disrespect Fossil itself.  I’m 
not proposing replacing it because it’s bad.  I’m just proposing a…shuttle 
craft, for use when we can’t fly the Enterprise itself on the mission.  The 
shuttlecraft always comes back home, and we can’t always use them, as when 
Ubering around Lwaxana Troi.  Gotta use the whole Enterprise for such things, 
you know. :)

> Fetching binary blobs directly with JSON

While that might be useful, I’m proposing that there be a way to get artifact 
data raw, directly over HTTP, just like downloading a file.

This is what I meant by JSON only being a default, when no better 
representation exists.  For “frapi up $version” and such, that better 
representation is “raw.”  Content-Length, binary data, the whole bit.

I suppose it could optionally be delta-compressed against the version the 
client claims to have, to save bandwidth.

> (my preferred solution) the JSON sends back links which, when visited, return 
> the raw blobs.

While that sounds like a useful API, a two round-trip solution introduces the 
possibility of race conditions.

For “frapi up”, you’d want a way to say “I have version $uuid, give me the tip 
of that artifact’s branch, optionally delta-compressed.”

This is also what I meant by this REST API being its own thing, designed with 
stateless HTTP in mind: you give everything in the request that tells the 
server what it needs to know to process your entire request in one go.  
Ideally, every operation takes only one round-trip.  The server carries the 
bulk of the processing workload.

That not only avoids race conditions, it means you pay for only one HTTP 
round-trip delay.

> i was going by past experience with at-work projects, where my colleagues are 
> sticklers for differentiating between PUT/POST (insert vs. update).

Yes, all experienced web devs know exactly how it should be done…and no one 
agrees. :)

It’s spaces vs tabs and emacs vs vi all over again.

The point here, though, is that doing everything with GET and POST is entirely 
doable and legal within the REST concept.

> CGI accounts for 100% of my fossil access :). (That was the killer feature 
> for me when i very first looked at fossil.)

I’m not suggesting that we change anything that already exists.  I’m just 
saying that if, for some reason, the person implementing this decided that they 
absolutely positively had to use the PUT verb for one new API that only frapi 
used, that would be fine by me.

> The majority (i suspect) of us have $5/month hosters which support PHP and 
> CGI, but have no way to run servers jobs full-time.

$5 gets you a root access VPS these days, plenty big enough to run Fossil.  
Shop around.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] REST API and client for same

[Stephan Beal]

On Sun, Apr 2, 2017 at 11:38 PM, Warren Young  wrote:

> On Apr 2, 2017, at 2:48 PM, Stephan Beal  wrote:
> >
> > a) that's essentially what the JSON API is
>
> …minus the lightweight Subversion-like client, of course.
>
> But, it’s good to know that most of the work is already done.
>

A good deal of it is. There are certainly a few holes in the API.


> > with the notable exception of missing blob support (since JSON has no
> definition for blobs).
>
> When I said that the default return representation should be JSON, I did
> not mean that all data must be in JSON format.  I just mean that, when no
> better representation exists, the API should use JSON.
>

i meant more for pushing new data, e.g. commits. Commits can't be done
without a checkout, though (not without significant changes, anyway).
Fetching binary blobs directly with JSON can't be done without an
intermediary format (base64 or whatever) or (my preferred solution) the
JSON sends back links which, when visited, return the raw blobs. That would
be one solution for such cases as downloads, as you mention here:


> Contrast the Fossil HTTP API here, where sometimes pulling a specific file
> shows it in the browser as-is, sometimes it displays it inline with other
> material, and sometimes it downloads it.





> With this REST API, I’d expect to get a file download every time, because
> the server cannot make any assumptions about the nature of the client,
> particularly about the display semantics.  Web applications like Fossil UI
> often make such assumptions.  The REST API would be more policy-free, in
> this sense.
>

Agreed. The current JSON API makes no assumptions about how the data will
be consumed/rendered.

> A strictly-REST-compliant API
>
> Who specifies “strict” REST?


> All I see are a bunch of cliques and camps, no actual standards.
>

Good point. AFAIK there is no official definition (or even RFC). i was
going by past experience with at-work projects, where my colleagues are
sticklers for differentiating between PUT/POST (insert vs. update).


That said, if some of this didn’t work with CGI, I don’t see a problem with
> that, since it is merely one alternative way to access Fossil, not the main
> one.


CGI accounts for 100% of my fossil access :). (That was the killer feature
for me when i very first looked at fossil.)


>   If the requirement is that you use “fossil server” with this, that seems
> fine to me.
>

But that immediately excludes everyone who doesn't have their own root
server (or otherwise has access to run server processes full-time). The
majority (i suspect) of us have $5/month hosters which support PHP and CGI,
but have no way to run servers jobs full-time.

(Apologies for my brevity - my left hand is offline again.)

-- 
- stephan beal
http://wanderinghorse.net/home/stephan/
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] REST API and client for same

[Warren Young]

On Apr 2, 2017, at 2:48 PM, Stephan Beal  wrote:
> 
> a) that's essentially what the JSON API is

…minus the lightweight Subversion-like client, of course.

But, it’s good to know that most of the work is already done.

> with the notable exception of missing blob support (since JSON has no 
> definition for blobs).

When I said that the default return representation should be JSON, I did not 
mean that all data must be in JSON format.  I just mean that, when no better 
representation exists, the API should use JSON.

Contrast the Fossil HTTP API here, where sometimes pulling a specific file 
shows it in the browser as-is, sometimes it displays it inline with other 
material, and sometimes it downloads it.  The Fossil HTTP API knows it’s 
talking to a web browser, and that a human is likely driving that browser, so 
it is trying to present the information in the manner the human most likely 
wants it.

With this REST API, I’d expect to get a file download every time, because the 
server cannot make any assumptions about the nature of the client, particularly 
about the display semantics.  Web applications like Fossil UI often make such 
assumptions.  The REST API would be more policy-free, in this sense.

> b) CGI does not specify certain methods which REST conventionally relies on, 
> most notably PUT.

PUT is nice to have for REST, but not essential.

The typical need for PUT vs POST is to distinguish “insert” from “update” — to 
put it in SQL terms — but that’s just one flavor of REST.  Another flavor 
distinguishes the cases based on whether you give a record ID or not; if not, 
it’s “insert.”

The PUT vs POST idempotency distinction also works in our favor here: it’s 
better that the one verb we get be the one that doesn’t try to guarantee 
idempotency.

> A strictly-REST-compliant API

Who specifies “strict” REST?

All I see are a bunch of cliques and camps, no actual standards.

That said, if some of this didn’t work with CGI, I don’t see a problem with 
that, since it is merely one alternative way to access Fossil, not the main 
one.  If the requirement is that you use “fossil server” with this, that seems 
fine to me.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] REST API and client for same

[Stephan Beal]

On Sun, Apr 2, 2017 at 8:58 PM, Warren Young  wrote:

> In a conversation off-list, I had an idea that might solve several
> existing problems.  What if the current HTTP URL interface of Fossil were
> expanded to be able to do everything that Fossil internally can do, such
> that it eventually implements REST API interface that is functionally
> equivalent to fossil CLI + Fossil UI?
>

Very briefly before bed:

a) that's essentially what the JSON API is, with the notable exception of
missing blob support (since JSON has no definition for blobs).

b) CGI does not specify certain methods which REST conventionally relies
on, most notably PUT. A strictly-REST-compliant API is not, IMO, possible
with fossil unless it is limited to server mode (as opposed to CGI mode).

-- 
- stephan beal
http://wanderinghorse.net/home/stephan/
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users