First some general notes. One of the real headaches in web app programming is state handling. With a thick client, what you do is you open a connection, perform your operations, commit your changes, etc. and then close the connection when you log off. With a web application, a single atomic unit of work may span several network connections. As a result a lot of things may have to be tracked in this way, and in some of our workloads, a large majority of database processing time is actually spent managing the application state. A lot of this time could be cut out if we didn't have to make sure that this information persisted when the database connection was closed and a new one opened.
I will give you an example. One of my customers pays a large number of invoices per week. I would say probably as many as 5000 invoices may be paid in a single batch payment workflow. I have good information on the profiling of the web app as to where time is spent. The basic selection takes only a couple of seconds (2-3), but once we add in the necessity to track that someone has selected these, we end up with about 45-50 sec of database time in the actual selection for payment. The XHTML document generation in some cases takes another couple of minutes. We can't use cursors to page through results because the cursors can't survive the connection teardown process. Consequently there is very limited performance tuning possible at present. Now, if this wasn't over HTTP, we could make the application far more responsive, and lead to better productivity, but as it is, we are fairly limited. This is one reason why I see the database-level API as such a big deal. It allows one to tune performance by removing the limitations of HTTP in this regard. If we can do it, I would much prefer to confine these problems to the web application than I would to export them to every other application that might interface to LedgerSMB. I would therefore suggest that running LedgerSMB over HTTP is a bit kludgy, and that long-term I would like to see the web app be secondary to a desktop app directly connecting to the database in most environments. On Sun, Nov 20, 2011 at 9:33 PM, John Locke <[email protected]> wrote: > Hi, > > On 11/20/2011 04:24 PM, Chris Travers wrote: >> I think John's points here raise some important questions I'd like to >> raise here for further discussion: >> >> On Sun, Nov 20, 2011 at 9:30 AM, John Locke <[email protected]> wrote: >>> On authentication, yes we can use http auth headers, but do we want to >>> explicitly require a session token, too? We're starting to delve into >>> OAuth -- which adds a layer of complexity but also can take away the >>> need for the remote system to collect the password at all. This seems >>> like a good option to support. >> A couple questions: >> 1) For an API aimed at other applications, why have a session token? >> What does it buy us? In the main application we use a session token >> to enforce a full round trip in some cases (XSRF prevention), and to >> handle discretionary locking in long workflows (actually more >> properly, timing out of such locks). Neither of those general >> requirements apply to a job from an OS Commerce installation which is >> feeding sales orders into LedgerSMB. > > Well, think mobile app for a minute, or desktop app. There may well be > many cases where you want round trip, transaction handling, and > anti-XSRF (or at least anti-replay) prevention. Ok, so we have layers of API's here, and in a lot of ways, a web services API is going to almost always be the wrong choice for a desktop app for reasons I have explained above. I could see mobile applications (in the sense of mobile wifi devices) using the db interface as well, though ones connecting over the public internet would probably not. One thing to keep in mind is that the db-level interface has taken a lot of inspiration from things I think SOAP and friends do well (discoverability, etc). Additionally you can do things which are (IMNSHO opinion) insane to try to do over HTTP, like control over database transactions. > > That is what I hate most about SOAP -- having to do multiple calls and > manage state. But to a certain extent, it seems unavoidable. Ever looked at SOAP over XMPP? For that matter although it would no longer be RESTful, I don't see any reason why RESTful approaches couldn't get encapsulated in XML stanzas and sent over XMPP. XMPP could then handle state and you'd no longer have to worry about it. Also if you are willing to to pass in HTTP to your accounting server through your firewall, not sure XMPP would be out. > > It's probably not a big deal to make a remote application pass the > company in the URL (instead of in a login session), but slightly easier > I would think to omit (in client implementation), to simply pass in with > authentication. > > In many ways, the web application front end is a model for other > applications that might call the web service -- ideally everything in > the web application should be reflected in the web service. Given the headaches dealing with application state we already have to worry about (including providing an interface for clearing discretionary locks when someone gets called away when running a selection for payment), I would highly recommend utilizing some discretion in what we push out to web services. I would instead focus on making more performance-friendly ways of closing that gap, even when that means ruling out using HTTP as a transfer protocol. Fortunately the areas where this is most likely to be an issue are also the areas where the database API is likely to be available. That doesn't mean, however, there can't be a common framework that couldn't be run stateless over HTTP and statefully over something like XMPP where there are questionable cases, however. > > In my experience, most web services still do make use of session > handling, it's not at all an uncommon approach. I guess I am questioning the need for it. We don't really have forms to submit, and requiring server-side tokens is going to mean more API calls rather than fewer (i.e. you'd have to get a token for each resource you intend to post). This means additional latency. >> >> 3) Does the added complexity make sense with general use cases? I am >> assuming we are primarily interested in a web services API for >> server->server integration since a piece of software used primarily by >> the end user would be more likely to just call the db-level API (which >> would provide greater control over db transactions, and the like than >> one would get from a web services interface)? > > Well, server-to-server is certainly the first step. And easiest to adapt > to just about any interface we develop. But today we're doing most web > services for iOS or Android apps. Think about the POS or an inventory > module being available as an app for an Android phone. I think we'd need more details to see what the relevant costs and benefits of using a web service in such an app would be. The questions in my mind become: 1) Is this an environment where the db-level API is appropriate and likely to be available? 2) If not, is this an environment where the document/resource metaphor of HTTP makes sense and where the systems can be loosely coupled? If so, web services are a good choice. 3) If not, then are there other approaches to encapsulating one of the above API's in another protocol that does make sense? > > The recent thread by a Google engineer praising Amazon for making > everything an API applies here. If you haven't read it: > https://plus.google.com/112678702228711889851/posts/eVeouesvaVX > I agree that everything should be an API. I am just less convinced that everything in LedgerSMB should be an API over HTTP. > >>>>> 2. Since companies are separate databases, where do we put the name of >>>>> the company in the URL? <prefix>/store.pl/<company>/<etc...>? >>>>> >>>> What do you think of the above proposal? >>> I suggest we include the company in the body of the login, and then it's >>> represented in the session cookie. If an external application needs to >>> work with multiple companies, it can manage multiple session ids. >> This is contrary to REST ideals, correct? Not that departures from >> that are necessarily out, but I would prefer to see some justification >> for the added complexity of requiring state handling and cookies. > > Well, yes, it is contrary to REST ideals -- but there's definitely room > in REST for actions as well as resources. And I was thinking while > writing this up about what might be an effective way of supporting > transactions -- complete with begin transaction, commit, and rollback posts. I don't see any sane way of handling database transaction controls over HTTP. I think any attempt to do so would significantly reduce the robustness of the controls on the server for anyone accessing the database. However if the same API can be encapsulated over XMPP, then the problem goes away entirely and now you can be sure of the state enough to expose transaction controls safely. > > I'm not entirely opposed to putting the company in the URL -- it's > certainly a viable approach. However, given the complex structure of > entity/eca/customer objects alone, having the ability to wrap that in a > transaction might be desirable... I dont see how you can possibly have a database transaction safely exist across multiple HTTP requests, hence my suggestion to explore XMPP for these areas. > > And I think leveraging the current session handling in the app can > reduce opening up new security vulnerabilities. Not suggesting we build > anything new for this, just use what we've already got. >> >> Also I suspect (though I will defer to others here) that debugging an >> incorrect company name may be easier if that shows up in the url in >> the access logs. > > This could easily be printed in a debug log. Not seeing how this is any > more complex than debugging issues in the current app... >> Fair enough. >>> So one thing is identifying supported formats for the data -- I suggest >>> we support JSON, multi-part form (e.g. URL encoded like a regular form >>> post) that returns HTML, and some relatively simple XML. Type can then >>> be specified via "Accept" header, and also by adding a suffix to the >>> URL. For example: >>> >>> http://myhost/ledgersmb/api/1.3/customer/224.json >>> http://myhost/ledgersmb/api/1.3/customer/224.xml >> But those formats are not all entirely equivalent are they? I JSON >> and XML are close and could be easily supported together, but they >> allow nested data structures while form submissions are flat, right? >> If we support form type submissions as a full API, then this means we >> have to choose between added maintenance of two very different data >> representations and forcing the xml and json to the least common >> denominator, correct? This becomes a bigger deal as time goes on and >> more stored procedures expect some form of nesting in argument lists. >> >> This being said, I like the use of extensions here, and I think the >> overall idea is sound. Now, if we are to do this, I would suggest we >> go with a plugin model for file types, i.e. require a parser which >> converts the incoming file into a Perl hashref, so that we can add >> other file types if we ever have to. That way if someone really needs >> plain form submission handling we have an avenue to support that in >> the future, even if the API might be more complex in order to handle >> lit. > > Totally agree. Plugin model for handling the format is the way to go. > That way if somebody wanted a particular XML dialect, it could be added > on as well. > > JSON is all I'm interested in actually using, though XML is handier for > testing... > > Form posts can be built much like the current web app -- using indexes > for fields with repeating values (e.g. on the invoice forms, qty_1, > qty_2, description_1, description_2, price_1, price_2, etc). Cumbersome > but not that difficult to support. If we go with a plugin model we can also decide to set it aside for now until someone needs it. > > Main point of abstracting this out though is that the plugin just needs > to convert it to an appropriate object with the defined properties set That is true. > >>> POST/PUT type should get specified by Content-Type header. >>> >>> I suggest we start with the base entity structure that mostly maps to >>> the database structure, then add "sugar" path shortcuts to make this >>> easier to use. e.g. all of the below might map to the same item: >>> >>> http://myhost/ledgersmb/api/1.3/entity?eca=224 >>> http://myhost/ledgersmb/api/1.3/entity/eca/224 >>> http://myhost/ledgersmb/api/1.3/eca/224 >>> http://myhost/ledgersmb/api/1.3/entity/eca/customer/224 >>> http://myhost/ledgersmb/api/1.3/customer/224 >>> http://myhost/ledgersmb/api/1.3/customer?meta=557 (which might redirect >>> to the actual eca id) >> Ok, if you mean the structure of the db api then I would entirely >> agree (I think the physical structure of the DB is beside the point). >> So let's do this: >> 1) A flat namespace for primary types/identifiers below the API base URL >> 2) Open discussion for what other shortcuts should be available for >> these, and whether they provide additional checks (I am assuming >> customer would check the entity_class but eca would not? Which entity >> class would it check? If grabbing from entity/... should we require >> an entity_id we can check? That sort of thing) > > Yes, exactly. This can start with the raw types/identifiers that map > relatively straightforward to the db schema. By providing versions, we > can add or change functionality as identified, without having to resolve > all these issues up front. Agreed entirely. >> >> >>> >>> ... then add more "sugar" methods to get related items: >>> >>> http://myhost/ledgersmb/api/1.3/customer/224/invoice?poststartdate=2011-01-01&poststartoper=gte >>> >>> ... might return a collection of invoice objects for customer 224 with a >>> post date greater than/equal to January 1, 2011. >> These can map to db-based search routines, correct? > > Yes, exactly... when I think of REST, I'm thinking of these methods: > > Index - GET with resource name but no id -- can pass various search > parameters > Create - POST to resource name with no id, data in body > Read - GET with resource id > Update - PUT with resource id, data in body > Delete - DELETE with resource id > > ... and then whatever actions to support, using POST with specific > resource paths associated with the action, and varying data necessary to > process the action in the body. > > The "sugar" methods are most commonly Indexes of related objects. > > With JSON, one standard for identifying related objects is passing a > $ref property with the resource URL so you can load the entire related > object with another GET. > > e.g. > > {"id":"234", > "url":"http://mycompany/api/1.3/customer/234","name":"Sample company", > ... "contacts":[{"id":"567","type":"email","value":"[email protected]", > "$ref":"http://mycompany/api/1.3/contact/567"},{"id":"568","type":"phone","value":"800-555-1234"}]} > > >> >>> http://myhost/ledgersmb/api/1.3/customer/224/invoice?status=open >>> >>> ... might return all open invoices for the customer. >> That becomes syntactic sugar above the above? > > ... Yes, that might be a Sugar method that's equivalent to: > > http://myhost/ledgersmb/api/1.3/ar/invoice?customer_id=234&status=open > > >>> ... and it would come back with the ECA id set, and a Location: header >>> set with the resource URL for that item. >> The ECA id would be in the location header, however, right? I guess >> what I am wondering is if we are going to return the ECA id as well, >> shouldn't we return the whole object? Or wouldn't it be better to >> just issue a redirect to the new object so that default values can be >> pulled? > > Yes, I suggest doing both -- returning the whole object as rewritten by > the server, as well as adding a header to the final URL. I like the idea of returning the whole object. I am thinking through the security, and I think we'll have to have some way of authenticating clients as well as users. I don't see another way around xsrf issues that doesn't break a web services model. Amazon Web Services does this with a preshared key approach. I would personally prefer client-side certificates with a configurable CN root, and ban use of any of these same certs in the web app. Best Wishes, Chris Travers ------------------------------------------------------------------------------ All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-novd2d _______________________________________________ Ledger-smb-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/ledger-smb-devel
