Re: Reporting system and potentially Python
On Thu, 07 Jul 2011 20:48:27 -0500, Tim M wrote: What I'm looking for is this: 1. Create the 'new' reporting system alongside the existing one so that the reports do not suffer until the existing functionality can be fully replaced by the new system. After all reports are replaced and working, the old scheme code could be pulled. 2. Create a set of libraries and/or use the existing gnucash libraries for querying information from the database. Based on the discussion, I presume C or Objective C would be best and just avoid Python/Scheme altogether (I am not sure how the scheme system actually performs the data queries right now). If all the necessary code is already in place, then this does not require any work. As far as I've been able to figure out so far, the python and scheme bindings use the existing gnucash libraries to access the data base. So if the current reports are what you need, why rewrite in C? But it does make sense to allow the Scheme reporting system to generate XML. And if it's essential for some reason to have them compiled, the Gambit C implementation of Scheme (as in Debian package gambc) is capable of compiling Scheme into (rather unreadable) C. Of course there are subtle differences between Gambit and guile, and including another tool into gnucash would be a big step. 3. Using these libraries, create the code for aggregating the data (also in C or Objective C) for each report. A single prototype report would be OK to start. Output the report data as structured XML. The XML data should adhere to an XML schema. All in favour of allowing reports to be in XML. It probably won't take too much effort to allow this. It's what I'm already doing with my personal reporting tool that reads the gnucash XML file. It's written in C++ and has to be changed every year or so to adapt to changes in teh gnucash file. All I needed for neat output was a .css file. It's quite effective. 4. For each report, create an XSLT file which will transform the data into HTML/XHTML for display. 5. A small amount of javascript would be needed to perform the actual XSLT transformation but it would be pretty small. 6. Style the page elements using CSS. Also, I think the jqplot patch that has been made could be migrated in to the new reporting system, as those graphs look really nice. If we're brainstorming about report-describing formalisms, has anyone looked at RPG, a commercial workhorse from back in the 1960's? It wouldn't be directly applicable unless it's changed drastically since then, but some of its ideas may be. I think there are several benefits to this approach: 1. Currently reports can only be exported as HTML. By making the reporting code export an XML data structure, reports could be easily exported from GnuCash as XML (pre-transformation) or as HTML (post-transformation). Much desired. But could probably be achieved with only minor changes in the present system. 6. No reliance on Scheme or Python. Minimal Javascript, but that should be handled easily by webkit. To achieve advantage 6 would require major changes, though. -- hendrik ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Reporting system and potentially Python
On Fri, 08 Jul 2011 23:33:16 -0400, John Ralls wrote: On Jul 8, 2011, at 8:15 PM, Yawar Amin wrote: If we stick with Scheme, we can take advantage of all the low-level functions that already exist for data extraction and report layout. But we can also move to a declarative model where we can have convention (re-use the report definitions as options) over configuration (build an options dialog box). Also, is it still true that we have to restart GnuCash every time we change a Scheme report, to see the changes? In any case, we need to make it dead easy for users to import and run and custom reports. Best, Yawar * I find that I’m saying ‘declarative’ a lot nowadays–I think it has to do with the fact that I’m learning Haskell :-) Fun. Two questions: Can that be easily converted into a string parser so that normal users aren't put off by the extra parentheses, and is there anything about that that works in Scheme but not in C? One of the hallmarks of Scheme is its metaprogrammability, for applications just like this. And its simple syntax promotes this. Not that it isn't possible to write string parsers and the like, and many Scheme systems come with packages for this. But once you go this route, coding tends to become inflexible, like in C. But as I've said elsewhere, the greatest barrier users encounter in trying to use the existing reporting tools isn't that they're written in Scheme. It's that the API they use is undocumented. That's something I hope to do something about. -- hendrik ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Reporting system and potentially Python
On Tue, 15 Nov 2011 13:30:37 + (UTC) Hendrik Boom hend...@topoi.pooq.com wrote: On Fri, 08 Jul 2011 23:33:16 -0400, John Ralls wrote: On Jul 8, 2011, at 8:15 PM, Yawar Amin wrote: If we stick with Scheme, we can take advantage of all the low-level functions that already exist for data extraction and report layout. But we can also move to a declarative model where we can have convention (re-use the report definitions as options) over configuration (build an options dialog box). Also, is it still true that we have to restart GnuCash every time we change a Scheme report, to see the changes? In any case, we need to make it dead easy for users to import and run and custom reports. Best, Yawar * I find that I’m saying ‘declarative’ a lot nowadays–I think it has to do with the fact that I’m learning Haskell :-) Fun. Two questions: Can that be easily converted into a string parser so that normal users aren't put off by the extra parentheses, and is there anything about that that works in Scheme but not in C? One of the hallmarks of Scheme is its metaprogrammability, for applications just like this. And its simple syntax promotes this. Not that it isn't possible to write string parsers and the like, and many Scheme systems come with packages for this. But once you go this route, coding tends to become inflexible, like in C. But as I've said elsewhere, the greatest barrier users encounter in trying to use the existing reporting tools isn't that they're written in Scheme. It's that the API they use is undocumented. That's something I hope to do something about. -- hendrik I believe Yawar has made a start on documenting the API at http://wiki.gnucash.org/wiki/User:Yawaramin ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Reporting system and potentially Python
Hi Hendrik, On Tue, Nov 15, 2011 at 8:30 AM, Hendrik Boom hend...@topoi.pooq.comwrote: [...] One of the hallmarks of Scheme is its metaprogrammability, for applications just like this. And its simple syntax promotes this. I've started writing a proof-of-concept declarative report ( https://github.com/yawaramin/gc-decl-reports/blob/master/hello-world.scm ). It doesn't really do anything other than populate a few options into the report options dialog box, and render a few lines of HTML. I'm trying to come up with a flexible format for pushing the accounting data around internally, so that it can filtered, sorted, and grouped, and then shown in either a tabular or graphical layout. If you want to try it out, just copy over hello-world.scm and replace your version of it (in your GnuCash install's shared reports directory) with my one. [...] But as I've said elsewhere, the greatest barrier users encounter in trying to use the existing reporting tools isn't that they're written in Scheme. It's that the API they use is undocumented. That's something I hope to do something about. Very true. Are you planning to start up a wiki page? Maybe we can pool efforts. Regards, Yawar ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Reporting system and potentially Python
I note that Yawar's page (http://wiki.gnucash.org/wiki/User:Yawaramin) looks like a great start to documenting the reports. It is currently a ghost on the site, though; nothing apparently links to it. I believe it would be useful to link to it directly from the main wiki page (wiki.gnucash.org/wiki/GnuCash), either under Development or Documentation. What would be the preference? David - Original Message - From: Mike Evans mi...@saxicola.idps.co.uk To: Hendrik Boom hend...@topoi.pooq.com Cc: gnucash-de...@lists.gnucash.org Sent: Tuesday, November 15, 2011 5:56 AM Subject: Re: Reporting system and potentially Python On Tue, 15 Nov 2011 13:30:37 + (UTC) Hendrik Boom hend...@topoi.pooq.com wrote: On Fri, 08 Jul 2011 23:33:16 -0400, John Ralls wrote: On Jul 8, 2011, at 8:15 PM, Yawar Amin wrote: If we stick with Scheme, we can take advantage of all the low-level functions that already exist for data extraction and report layout. But we can also move to a declarative model where we can have convention (re-use the report definitions as options) over configuration (build an options dialog box). Also, is it still true that we have to restart GnuCash every time we change a Scheme report, to see the changes? In any case, we need to make it dead easy for users to import and run and custom reports. Best, Yawar * I find that I’m saying ‘declarative’ a lot nowadays–I think it has to do with the fact that I’m learning Haskell :-) Fun. Two questions: Can that be easily converted into a string parser so that normal users aren't put off by the extra parentheses, and is there anything about that that works in Scheme but not in C? One of the hallmarks of Scheme is its metaprogrammability, for applications just like this. And its simple syntax promotes this. Not that it isn't possible to write string parsers and the like, and many Scheme systems come with packages for this. But once you go this route, coding tends to become inflexible, like in C. But as I've said elsewhere, the greatest barrier users encounter in trying to use the existing reporting tools isn't that they're written in Scheme. It's that the API they use is undocumented. That's something I hope to do something about. -- hendrik I believe Yawar has made a start on documenting the API at http://wiki.gnucash.org/wiki/User:Yawaramin ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Reporting system and potentially Python
Hi Derek, On 2011-11-15, at 20:43, Derek Atkins wrote: […] My preference would be to move it out of Yawar's user page onto a real page and then we could link from there. Or better yet, let's get the changes into the sources themselves so we could use sometihng like doxygen to generate the docs. I’d be happy to see my reference merged into the doxygen reference. What I was trying to do on my page was compile an API reference from the report developer’s point of view–maybe something of a broad target, but I felt that the doxygen reference kind of drops you in at the deep end and lets you figure everything out for yourself. Maybe I should see if I can point towards specific parts of the doxygen API docs for specific goals, like if you want to get an account balance, look at this, that and that function … . Another thing I’ve been wondering: do all the C APIs documented in doxygen have a one-to-one correspondence with Scheme functions, and if so what exactly is the mapping for the function names? Is it something like: xaccSomethingDoSomething ( C ) - xaccSomethingDoSomething ( Scheme ) gnc_account_find_split ( C ) - gnc-account-find-split ( Scheme ) Can I assume the Account structure ( http://svn.gnucash.org/docs/HEAD/structAccount.html ) is also transparently wrapped in Scheme? Appreciate any pointers/advice. Regards, Yawar PGP.sig Description: This is a digitally signed message part ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Reporting system and potentially Python
Hi, On Tue, November 15, 2011 10:16 pm, Yawar Amin wrote: Hi Derek, On 2011-11-15, at 20:43, Derek Atkins wrote: [ ] My preference would be to move it out of Yawar's user page onto a real page and then we could link from there. Or better yet, let's get the changes into the sources themselves so we could use sometihng like doxygen to generate the docs. Id be happy to see my reference merged into the doxygen reference. What I was trying to do on my page was compile an API reference from the report developers point of viewmaybe something of a broad target, but I felt that the doxygen reference kind of drops you in at the deep end and lets you figure everything out for yourself. Maybe I should see if I can point towards specific parts of the doxygen API docs for specific goals, like if you want to get an account balance, look at this, that and that function . Another thing Ive been wondering: do all the C APIs documented in doxygen have a one-to-one correspondence with Scheme functions, and if so what exactly is the mapping for the function names? Is it something like: xaccSomethingDoSomething ( C ) - xaccSomethingDoSomething ( Scheme ) gnc_account_find_split ( C ) - gnc-account-find-split ( Scheme ) With a few exceptions yes, this is the case. Can I assume the Account structure ( http://svn.gnucash.org/docs/HEAD/structAccount.html ) is also transparently wrapped in Scheme? This depends on your definition of 'wrapped.' In general, C structures are mapped to Scheme as opaque pointer objects, which requires using getters/setters from guile to manipulate the object. For example, you can't just do Account-desc in scheme, you need to use (xaccAccountGetDescription acc). Appreciate any pointers/advice. Regards, Yawar -derek -- Derek Atkins 617-623-3745 de...@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Reporting system and potentially Python
On Nov 15, 2011, at 7:24 PM, Derek Atkins wrote: Hi, This depends on your definition of 'wrapped.' In general, C structures are mapped to Scheme as opaque pointer objects, which requires using getters/setters from guile to manipulate the object. For example, you can't just do Account-desc in scheme, you need to use (xaccAccountGetDescription acc). Has anyone ever tried using the gnome 2 Guile bindings and whatever is the g_object_get mechanism from Guile? Regards, John Ralls ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Reporting system and potentially Python
Hi, On Tue, November 15, 2011 11:17 pm, John Ralls wrote: On Nov 15, 2011, at 7:24 PM, Derek Atkins wrote: Hi, This depends on your definition of 'wrapped.' In general, C structures are mapped to Scheme as opaque pointer objects, which requires using getters/setters from guile to manipulate the object. For example, you can't just do Account-desc in scheme, you need to use (xaccAccountGetDescription acc). Has anyone ever tried using the gnome 2 Guile bindings and whatever is the g_object_get mechanism from Guile? Probably not, because I don't think it existed in a sufficient maturity the last time we did a scheme generator migration (g-wrap - swig). What is the current maturity of these bindings? How supported are they? Are they being actively developed and maintained? What does it buy us that we don't already have? Regards, John Ralls -derek -- Derek Atkins 617-623-3745 de...@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Reporting system and potentially Python
On Nov 15, 2011, at 9:16 PM, Derek Atkins wrote: Hi, On Tue, November 15, 2011 11:17 pm, John Ralls wrote: On Nov 15, 2011, at 7:24 PM, Derek Atkins wrote: Hi, This depends on your definition of 'wrapped.' In general, C structures are mapped to Scheme as opaque pointer objects, which requires using getters/setters from guile to manipulate the object. For example, you can't just do Account-desc in scheme, you need to use (xaccAccountGetDescription acc). Has anyone ever tried using the gnome 2 Guile bindings and whatever is the g_object_get mechanism from Guile? Probably not, because I don't think it existed in a sufficient maturity the last time we did a scheme generator migration (g-wrap - swig). What is the current maturity of these bindings? How supported are they? Are they being actively developed and maintained? What does it buy us that we don't already have? It's different from gwrap or swig (that would be gobject-introspection, which doesn't appear to be well supported, unfortunately). Rather, it is a guile library like slib or ice-9 that permits calling GObject functions without explicitly wrapping subclasses. It would let us simplify the API by e.g., not having to export explicit getters/setters, directly using GObject memory management, and so on. (FWIW, Python integration with both GObject and GObject-Introspection is both extensive and very actively developed. Were we starting now we'd have to be nuts to pick Guile over Python.) Regards, John Ralls ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Reporting system and potentially Python
On Wed, November 16, 2011 12:40 am, John Ralls wrote: On Nov 15, 2011, at 9:16 PM, Derek Atkins wrote: Hi, On Tue, November 15, 2011 11:17 pm, John Ralls wrote: On Nov 15, 2011, at 7:24 PM, Derek Atkins wrote: Hi, This depends on your definition of 'wrapped.' In general, C structures are mapped to Scheme as opaque pointer objects, which requires using getters/setters from guile to manipulate the object. For example, you can't just do Account-desc in scheme, you need to use (xaccAccountGetDescription acc). Has anyone ever tried using the gnome 2 Guile bindings and whatever is the g_object_get mechanism from Guile? Probably not, because I don't think it existed in a sufficient maturity the last time we did a scheme generator migration (g-wrap - swig). What is the current maturity of these bindings? How supported are they? Are they being actively developed and maintained? What does it buy us that we don't already have? It's different from gwrap or swig (that would be gobject-introspection, which doesn't appear to be well supported, unfortunately). Rather, it is a guile library like slib or ice-9 that permits calling GObject functions without explicitly wrapping subclasses. It would let us simplify the API by e.g., not having to export explicit getters/setters, directly using GObject memory management, and so on. (FWIW, Python integration with both GObject and GObject-Introspection is both extensive and very actively developed. Were we starting now we'd have to be nuts to pick Guile over Python.) If we were starting now we'd probably not be using C at all (and I'd argue whether or not we should be using gnome/gtk/glib, either). But alas we have over a decade of legacy code that's had thousands of man-hours of work to get us where we are, and it would be a shame to give all that up. We can play the what if games or think about how we could have done things better if we could start over, but I feel it would be a better use of our time to look forward instead of looking back. Regards, John Ralls -derek -- Derek Atkins 617-623-3745 de...@ihtfp.com www.ihtfp.com Computer and Internet Security Consultant ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Reporting system and potentially Python
On vrijdag 8 juli 2011, Tim M wrote: What I'm looking for is this: Separate the act of actually gathering the data from the GnuCash database from transforming it for display, so that the output can be made much more interactive and easy to style and make 'pretty' :). Use a well known programming language to aggregate the data to make it easier to create new reports which output different sets of data. I've thought about this a little bit, and here is what I am thinking. Let me know if this sounds un/reasonable or just plain incorrect. (Note that I am basing this off of not being familiar yet with how the current scheme architecture actually manages pulling the data out of gnucash, so please correct me where I am wrong or point out the significant flaws) 1. Create the 'new' reporting system alongside the existing one so that the reports do not suffer until the existing functionality can be fully replaced by the new system. After all reports are replaced and working, the old scheme code could be pulled. 2. Create a set of libraries and/or use the existing gnucash libraries for querying information from the database. Based on the discussion, I presume C or Objective C would be best and just avoid Python/Scheme altogether (I am not sure how the scheme system actually performs the data queries right now). If all the necessary code is already in place, then this does not require any work. 3. Using these libraries, create the code for aggregating the data (also in C or Objective C) for each report. A single prototype report would be OK to start. Output the report data as structured XML. The XML data should adhere to an XML schema. 4. For each report, create an XSLT file which will transform the data into HTML/XHTML for display. 5. A small amount of javascript would be needed to perform the actual XSLT transformation but it would be pretty small. 6. Style the page elements using CSS. Also, I think the jqplot patch that has been made could be migrated in to the new reporting system, as those graphs look really nice. I think there are several benefits to this approach: 1. Currently reports can only be exported as HTML. By making the reporting code export an XML data structure, reports could be easily exported from GnuCash as XML (pre-transformation) or as HTML (post-transformation). 2. Anyone could write their own XSLT transformation file to display the data differently. Like pdf, using the fop transformer. An improvement that would also be greatly appreciated by users of the business features. We have already some experience with this transform in the gnucash-docs, so it would not be too difficult to apply that experience on the reports. So me too I like your ideas of starting with xml and the separation into XSLT, CSS and friends. 3. The actual XSLT, HTML, and CSS code would be very easy to read and modify by anyone familiar with it. The backend reports would still be in C/Objective C for compiling the data and as such require some coding knowledge, but it would be much easier to start using than the existing scheme code. This part is not clear to me. Do you mean that if Joe user wants to add a new report, he will have to write some C code, like for example to add a new data query ? I guess that would limit the extendability of the reports equally to having them in scheme. John's suggestion to find something sql-query like to gather data might be nicer, as would Yawar's proposal to go for a completely declarative report. Both mechanisms work generically with the raw data and would adapt easily to data format changes. By the way, GnuCash comes with a query framework for its data: qof-query. It mayor may not be suitable as a basis for the first suggestion. I'm tempted to say here that such a generic and easy to use interface could be implemented as a later improvement, but at the same time I have a feeling that such a decision should be taken early on, because it would greatly influence how everything gets implemented. 4. The XSLT and CSS could be very easily modified and applied not only inside of GnuCash, but a user could also take the XSLT and CSS scripts, modify and apply them to the exported XML reports to give it their own look without touching the GnuCash code base at all. For example they might want to style a report with a company logo or different colors or other graphical elements. 5a. In the future, it would be possible to provide multiple different XSLT and/or CSS styles for the reports so users could select the appearance of the reports. (If someone writes additional styles) 5b. In the future, it would be possible to allow users to import their own style sheets into GnuCash and apply them to the reports. (if someone writes the code to manage the import and style selection) I already some kind of a theme foundry as part of the GnuCash website, where users can add stylesheets for others to use. Something
Re: Reporting system and potentially Python
On Jul 8, 2011, at 8:20 AM, Tim M wrote: Yawar's suggestion is quite interesting, but the declarative XML you suggest is bordering the functionality of a web service. I like the idea of having the XML declarations but there still then must be some way for the user to select individual accounts included in the report, which will require some communication between the reporting system and gnucash. I think a web service would actually be perfect for this, as this would then totally externalize the reporting infrastructure. I'm not sure, however, exactly what implications this would have. I would have to see how this can be handled - if a port can simply be opened and listened on for requests, or if it would require bundling some type of web service such as tomcat. I would think it could be done just using a listening port but I haven't actually written a web service before so I'm not sure. There are probably libraries already available which could handle the requests, otherwise we would have to roll our own. Also, I wonder if we can make the actual requests with Javascript - the WSDL would eliminate the C - Javascript access issues mentioned earlier. What this would require then would be to create a WSDL for the available methods and then similar to Yawar's suggestion, the report would make a request to the GnuCash web service for the information needed to generate the report, and the web service would return the results as XML. At that point the XSLT would transform the XML data into HTML for display and voila, you have a report which has been generated entirely using code and methods externally accessible from GnuCash. Having a web service could make for some very interesting use cases such as generating reports directly inside a web page (such as if a business has an intranet where they want to be able to gather and view their financial data). Also, it opens up the possibility of creating additional web service methods which not only read but also write data. For example, a user could enter transactions from a web form and on submitting the form, the web service request is sent to GnuCash which adds the txn to the database for the appropriate account. Another use case for a read/write web service would be having a GnuCash Server which would mitigate the problems associated with having multiple clients reading and writing to gnucash at one time. For example, GnuCash would sit on a dedicated system listening for web service requests. Client machines running copies of GnuCash would then connect to the server (I think it could be implemented as just another interface to the data such as we have XML and DBI currently) to access the GnuCash data. Finally, when a user action causes a insert/update/delete request, all user's requests would go through the central GnuCash server so they would not be able to access the file concurrently, thus preventing user collisions when editing the file. This is of course very ambitious and wouldn't happen unless someone saw the need and implemented all of the functionality that would be needed in the web service, but you get the point. This is in general a good plan, but you need to know that there's a reason for the engine code to be between the storage backends and the GUI and reporting system: It's the part that knows how to manage currencies and lots and invoices and such. In database applications that's called the business logic layer. Now it's true that a lot of what is in our engine belongs in the database layer (the backend), but there's also a lot that can't go there because databases don't know how to do it. Fixing that is another long-term project. Reports, the GUI, and even the command-line are called the presentation layer. The presentation layer has to pass all of its requests through the business logic layer to ensure that all of the accounting rules are applied correctly; nothing but the business logic layer should talk to the database layer. That's the theory. The reality of Gnucash is that the engine API is so confusing and inflexible that the new DBI backend has tempted people to try querying the backend directly with SQL so that they can get the reports they want without having to spend the time to learn the engine's API. Geert mentioned the QofQuery class. I'm concerned that it's both too limited in function and too low-level (it's used by the engine to query data structures, so it may not apply business logic to its results) to work as a direct interface for a report service. It's worth studying, though, because modifying it might be the shortest path to getting a workable query interface. Regards, John Ralls ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Reporting system and potentially Python
On Fri, Jul 8, 2011 at 8:44 AM, John Ralls jra...@ceridwen.us wrote: This is in general a good plan, but you need to know that there's a reason for the engine code to be between the storage backends and the GUI and reporting system: It's the part that knows how to manage currencies and lots and invoices and such. In database applications that's called the business logic layer. Now it's true that a lot of what is in our engine belongs in the database layer (the backend), but there's also a lot that can't go there because databases don't know how to do it. Fixing that is another long-term project. Reports, the GUI, and even the command-line are called the presentation layer. The presentation layer has to pass all of its requests through the business logic layer to ensure that all of the accounting rules are applied correctly; nothing but the business logic layer should talk to the database layer. That's the theory. The reality of Gnucash is that the engine API is so confusing and inflexible that the new DBI backend has tempted people to try querying the backend directly with SQL so that they can get the reports they want without having to spend the time to learn the engine's API. Geert mentioned the QofQuery class. I'm concerned that it's both too limited in function and too low-level (it's used by the engine to query data structures, so it may not apply business logic to its results) to work as a direct interface for a report service. It's worth studying, though, because modifying it might be the shortest path to getting a workable query interface. Regards, John Ralls John, Thank you for the clarification. I think we are on the same page. Any sort of web service would act just as an interface to the outside (the Report system, or other third-party requests to the GnuCash web service) and then internally it would perform queries against the GnuCash DB via the engine code. I never had any intention of querying the DB directly - I only intended to wrap the engine code (whether it be QofQuery or whatever else is appropriate). The web service methods would only call the necessary engine code to query the data, and then wrap the results into the XML output and send it back to the client. Are you saying that it may be a problem to implement a web service to query the database using the engine code to generate the reports this way? Isn't this what the scheme reports do now? Otherwise how do they access the data to generate the reports? I presume they do not access the DB directly but use the engine code to do it. As I see it, the web service should work more or less as follows: 1. A Report or other service issues a SOAP/REST request to the GnuCash web service. The request must adhere to the GnuCash WSDL (anything not adhering to it will be ignored). 2. The GnuCash web service receives the request. 3. The web service queries the database using the GnuCash engine code. 4. The web service wraps the results from the engine code into an XML wrapper. 5. The web service sends the XML output back to the client. 6. The XSLT translation occurs on the client to display the data. Thanks, -Tim ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Reporting system and potentially Python
On Jul 8, 2011, at 10:24 AM, Tim M wrote: On Fri, Jul 8, 2011 at 8:44 AM, John Ralls jra...@ceridwen.us wrote: This is in general a good plan, but you need to know that there's a reason for the engine code to be between the storage backends and the GUI and reporting system: It's the part that knows how to manage currencies and lots and invoices and such. In database applications that's called the business logic layer. Now it's true that a lot of what is in our engine belongs in the database layer (the backend), but there's also a lot that can't go there because databases don't know how to do it. Fixing that is another long-term project. Reports, the GUI, and even the command-line are called the presentation layer. The presentation layer has to pass all of its requests through the business logic layer to ensure that all of the accounting rules are applied correctly; nothing but the business logic layer should talk to the database layer. That's the theory. The reality of Gnucash is that the engine API is so confusing and inflexible that the new DBI backend has tempted people to try querying the backend directly with SQL so that they can get the reports they want without having to spend the time to learn the engine's API. Geert mentioned the QofQuery class. I'm concerned that it's both too limited in function and too low-level (it's used by the engine to query data structures, so it may not apply business logic to its results) to work as a direct interface for a report service. It's worth studying, though, because modifying it might be the shortest path to getting a workable query interface. Regards, John Ralls John, Thank you for the clarification. I think we are on the same page. Any sort of web service would act just as an interface to the outside (the Report system, or other third-party requests to the GnuCash web service) and then internally it would perform queries against the GnuCash DB via the engine code. I never had any intention of querying the DB directly - I only intended to wrap the engine code (whether it be QofQuery or whatever else is appropriate). The web service methods would only call the necessary engine code to query the data, and then wrap the results into the XML output and send it back to the client. Are you saying that it may be a problem to implement a web service to query the database using the engine code to generate the reports this way? Isn't this what the scheme reports do now? Otherwise how do they access the data to generate the reports? I presume they do not access the DB directly but use the engine code to do it. As I see it, the web service should work more or less as follows: 1. A Report or other service issues a SOAP/REST request to the GnuCash web service. The request must adhere to the GnuCash WSDL (anything not adhering to it will be ignored). 2. The GnuCash web service receives the request. 3. The web service queries the database using the GnuCash engine code. 4. The web service wraps the results from the engine code into an XML wrapper. 5. The web service sends the XML output back to the client. 6. The XSLT translation occurs on the client to display the data. Tim, You confused me with the line I think it could be implemented as just another interface to the data such as we have XML and DBI currently Those are the database or storage layer interfaces that engine uses for persistence, not interfaces that presentation layer code should use. My concern with WSDL is that each Scheme report is a function which calls engine functions. There's no generalized query language or query interface AFAICT. I don't know anything about building a WSDL, but if it's like other query languages (e.g. SQL) it seems likely to me that there will be a pretty bad impedance mismatch in some areas. Also, as you've laid out the flow, it seems a bit heavyweight for the internal report generator. Rather than having to format an http request and setting up an IP listener in Gnucash (and consequent (x)inetd setup), wouldn't a socket interface that both the Report subsystem and the web server connect to? Simplifying the report side further, once we have the code to handle general query strings and return XML, that interface can pretty easily be provided both as an API (returning a DOM tree to save writing/parsing overhead) and a socket interface for a SOAP/REST server -- or anything else we dream up later -- to connect to. Regards, John Ralls ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Reporting system and potentially Python
On Fri, Jul 8, 2011 at 10:28 AM, John Ralls jra...@ceridwen.us wrote: I think it could be implemented as just another interface to the data such as we have XML and DBI currently Those are the database or storage layer interfaces that engine uses for persistence, not interfaces that presentation layer code should use. That was in relation to if someone were to (in the long distant future) add to the web service so that it provides more or less the entire GnuCash functionality via the web service. Then you could have a client-server setup where one instance of GnuCash runs on a central machine acting as the GnuCash server, and other remote machines connect to the server as clients. The clients would need to read and write data from the central server, hence why I suggested the requests to the central server could be implemented as just another data interface. Currently XML and databases are allowed, another interface could be added for SOAP/REST, which would pull the data from the central GnuCash server. This is just a potential use case of a greatly enhanced web service which could theoretically be provided by GnuCash, and goes well beyond the needs for the basic reporting infrastructure. I do not have any plans to implement it, I was just making the point that it would be theoretically possible to enhance the basic web service functionality to achieve a server/client access method, as well as for users to write their own third-party applications which could read/write data into the GnuCash database via the web service, such as a web page where users enter their expense transactions. The benefit of this is multiuser access - and also a business could provide their own web interface or standalone application for users to enter expenses or other txns, while restricting the user's access from any other functionality of GnuCash because they do not have direct access to the GnuCash interface and/or GnuCash data files. My concern with WSDL is that each Scheme report is a function which calls engine functions. There's no generalized query language or query interface AFAICT. I don't know anything about building a WSDL, but if it's like other query languages (e.g. SQL) it seems likely to me that there will be a pretty bad impedance mismatch in some areas. WSDL is not a query language. WSDL is Web Service Definition Language. Basically a WSDL is an XML file which defines functions that a web service exposes. The WSDL describes how to call the web service's functions, what variables it accepts (or requires) and if the web service returns any information, the WSDL describes what information the client can expect the web service to return. It is basically a web-based API definition. So the WSDL does not provide any actual code - it simply describes the web service. The actual code which executes any incoming requests sits inside the web service application. Also, as you've laid out the flow, it seems a bit heavyweight for the internal report generator. Indeed, it is heavyweight but on the other hand it would add a lot of flexibility to the reporting system. Rather than having to format an http request and setting up an IP listener in Gnucash (and consequent (x)inetd setup), wouldn't a socket interface that both the Report subsystem and the web server connect to? I am not familiar enough with what the actual networking implementation would require, so it will probably be a discovery process while writing the new reporting code to see what is the best option is, unless someone else steps in to code the network/socket interface for me (any takers? :oD ). As long as the interface meets the basic requirement of being able to communicate messages between the reporting system and GnuCash, then whatever communication method is used should be sufficient. Simplifying the report side further, once we have the code to handle general query strings and return XML, that interface can pretty easily be provided both as an API (returning a DOM tree to save writing/parsing overhead) and a socket interface for a SOAP/REST server -- or anything else we dream up later -- to connect to. Regards, John Ralls I think the best approach might be to work backwards by defining the WSDL based on the existing reports and determining what functions would need to be exposed to retrieve the data required to generate those reports. Once a draft of the WSDL is accepted, the web service can be written to provide the functionality defined by the WSDL. After or concurrently with that, the individual reports could be written and tested against the web services. As the web services are completed, the reports should generate successfully. Later, if new functionality is needed for the reporting system, then it can be defined in a new version of the WSDL and exposed by the web service. Does this make sense? Thanks, -Tim ___ gnucash-devel mailing list
Re: Reporting system and potentially Python
On Jul 8, 2011, at 1:25 PM, Tim M wrote: On Fri, Jul 8, 2011 at 10:28 AM, John Ralls jra...@ceridwen.us wrote: I think it could be implemented as just another interface to the data such as we have XML and DBI currently Those are the database or storage layer interfaces that engine uses for persistence, not interfaces that presentation layer code should use. That was in relation to if someone were to (in the long distant future) add to the web service so that it provides more or less the entire GnuCash functionality via the web service. Then you could have a client-server setup where one instance of GnuCash runs on a central machine acting as the GnuCash server, and other remote machines connect to the server as clients. The clients would need to read and write data from the central server, hence why I suggested the requests to the central server could be implemented as just another data interface. Currently XML and databases are allowed, another interface could be added for SOAP/REST, which would pull the data from the central GnuCash server. This is just a potential use case of a greatly enhanced web service which could theoretically be provided by GnuCash, and goes well beyond the needs for the basic reporting infrastructure. I do not have any plans to implement it, I was just making the point that it would be theoretically possible to enhance the basic web service functionality to achieve a server/client access method, as well as for users to write their own third-party applications which could read/write data into the GnuCash database via the web service, such as a web page where users enter their expense transactions. The benefit of this is multiuser access - and also a business could provide their own web interface or standalone application for users to enter expenses or other txns, while restricting the user's access from any other functionality of GnuCash because they do not have direct access to the GnuCash interface and/or GnuCash data files. My concern with WSDL is that each Scheme report is a function which calls engine functions. There's no generalized query language or query interface AFAICT. I don't know anything about building a WSDL, but if it's like other query languages (e.g. SQL) it seems likely to me that there will be a pretty bad impedance mismatch in some areas. WSDL is not a query language. WSDL is Web Service Definition Language. Basically a WSDL is an XML file which defines functions that a web service exposes. The WSDL describes how to call the web service's functions, what variables it accepts (or requires) and if the web service returns any information, the WSDL describes what information the client can expect the web service to return. It is basically a web-based API definition. So the WSDL does not provide any actual code - it simply describes the web service. The actual code which executes any incoming requests sits inside the web service application. Also, as you've laid out the flow, it seems a bit heavyweight for the internal report generator. Indeed, it is heavyweight but on the other hand it would add a lot of flexibility to the reporting system. Rather than having to format an http request and setting up an IP listener in Gnucash (and consequent (x)inetd setup), wouldn't a socket interface that both the Report subsystem and the web server connect to? I am not familiar enough with what the actual networking implementation would require, so it will probably be a discovery process while writing the new reporting code to see what is the best option is, unless someone else steps in to code the network/socket interface for me (any takers? :oD ). As long as the interface meets the basic requirement of being able to communicate messages between the reporting system and GnuCash, then whatever communication method is used should be sufficient. Simplifying the report side further, once we have the code to handle general query strings and return XML, that interface can pretty easily be provided both as an API (returning a DOM tree to save writing/parsing overhead) and a socket interface for a SOAP/REST server -- or anything else we dream up later -- to connect to. I think the best approach might be to work backwards by defining the WSDL based on the existing reports and determining what functions would need to be exposed to retrieve the data required to generate those reports. Once a draft of the WSDL is accepted, the web service can be written to provide the functionality defined by the WSDL. After or concurrently with that, the individual reports could be written and tested against the web services. As the web services are completed, the reports should generate successfully. Later, if new functionality is needed for the reporting system, then it can be defined in a new version of the WSDL and exposed
Re: Reporting system and potentially Python
Agreed. That sounds like a good plan. -Tim On Fri, Jul 8, 2011 at 4:54 PM, John Ralls jra...@ceridwen.us wrote: On Jul 8, 2011, at 1:25 PM, Tim M wrote: On Fri, Jul 8, 2011 at 10:28 AM, John Ralls jra...@ceridwen.us wrote: I think it could be implemented as just another interface to the data such as we have XML and DBI currently Those are the database or storage layer interfaces that engine uses for persistence, not interfaces that presentation layer code should use. That was in relation to if someone were to (in the long distant future) add to the web service so that it provides more or less the entire GnuCash functionality via the web service. Then you could have a client-server setup where one instance of GnuCash runs on a central machine acting as the GnuCash server, and other remote machines connect to the server as clients. The clients would need to read and write data from the central server, hence why I suggested the requests to the central server could be implemented as just another data interface. Currently XML and databases are allowed, another interface could be added for SOAP/REST, which would pull the data from the central GnuCash server. This is just a potential use case of a greatly enhanced web service which could theoretically be provided by GnuCash, and goes well beyond the needs for the basic reporting infrastructure. I do not have any plans to implement it, I was just making the point that it would be theoretically possible to enhance the basic web service functionality to achieve a server/client access method, as well as for users to write their own third-party applications which could read/write data into the GnuCash database via the web service, such as a web page where users enter their expense transactions. The benefit of this is multiuser access - and also a business could provide their own web interface or standalone application for users to enter expenses or other txns, while restricting the user's access from any other functionality of GnuCash because they do not have direct access to the GnuCash interface and/or GnuCash data files. My concern with WSDL is that each Scheme report is a function which calls engine functions. There's no generalized query language or query interface AFAICT. I don't know anything about building a WSDL, but if it's like other query languages (e.g. SQL) it seems likely to me that there will be a pretty bad impedance mismatch in some areas. WSDL is not a query language. WSDL is Web Service Definition Language. Basically a WSDL is an XML file which defines functions that a web service exposes. The WSDL describes how to call the web service's functions, what variables it accepts (or requires) and if the web service returns any information, the WSDL describes what information the client can expect the web service to return. It is basically a web-based API definition. So the WSDL does not provide any actual code - it simply describes the web service. The actual code which executes any incoming requests sits inside the web service application. Also, as you've laid out the flow, it seems a bit heavyweight for the internal report generator. Indeed, it is heavyweight but on the other hand it would add a lot of flexibility to the reporting system. Rather than having to format an http request and setting up an IP listener in Gnucash (and consequent (x)inetd setup), wouldn't a socket interface that both the Report subsystem and the web server connect to? I am not familiar enough with what the actual networking implementation would require, so it will probably be a discovery process while writing the new reporting code to see what is the best option is, unless someone else steps in to code the network/socket interface for me (any takers? :oD ). As long as the interface meets the basic requirement of being able to communicate messages between the reporting system and GnuCash, then whatever communication method is used should be sufficient. Simplifying the report side further, once we have the code to handle general query strings and return XML, that interface can pretty easily be provided both as an API (returning a DOM tree to save writing/parsing overhead) and a socket interface for a SOAP/REST server -- or anything else we dream up later -- to connect to. I think the best approach might be to work backwards by defining the WSDL based on the existing reports and determining what functions would need to be exposed to retrieve the data required to generate those reports. Once a draft of the WSDL is accepted, the web service can be written to provide the functionality defined by the WSDL. After or concurrently with that, the individual reports could be written and tested against the web services. As the web services are completed, the reports should generate successfully. Later, if new functionality is needed for the reporting system, then it can be
Re: Reporting system and potentially Python
Folks, On 2011-07-08, at 01:15, Yawar Amin wrote: […] I like the idea of XML - HTML + CSS + JavaScript plots. How about moving to a completely declarative XML report, where we use XML tags to ask for things like account names and balances, and using the Scheme (or […] I got to thinking that Scheme itself can be used as a declarative,* domain-specific language for defining GnuCash reports. We could have a declarative API that sits on top of the currently-existing report API, and abstracts away the complexities of writing out an HTML page and building a report options dialog box. Sort of like LaTeX hides the raw TeX complexities of laying out a page. Here’s my income statement converted over from XML into (declarative) Scheme: (report income-statement (defs (def-date start-date General/Start date 2011-01-01) (def-date end-date General/End date 2011-07-31) (def-label title General/Report title Income Statement) (def-label subtitle General/Report subtitle '((string (date start-date)) to (string (date end-date (def-account-group income Accounts/Income accounts '(3288757aa761c7bba993766bfb433466 e49c976843a1dba40570ed6295e89a33)) (def-account-group expenses Accounts/Expense accounts '(df72fb29f46e90e12c2ba10d582a6148 e518845492b7b9cfa614cc405a8cf715)) (def-account net-profit-loss Net Profit (Loss) (subtract (balance (account-group income)) (balance (account-group expenses) (title (label title)) (subtitle (label subtitle)) (filter-date (date start-date) (date end-date”) (group-date 'month (tabular (account-group income) (account-group expenses) (account net-profit-loss) And here’s the report structure in plain language: report definitions (can double as options for the report options dialog box) definition of date (start date) definition of date (end date) definition of label (report title) definition of label (report subtitle) definition of account group (income) definition of account group (expenses) definition of a calculated account (net profit or loss = income - expenses) report title report subtitle filter by date (only include transactions within date range) group by date (monthly–this should show balances in the table as one column per month) tabular display (we could also provide a graphical display–i.e. column chart) account group (income) account group (expenses) account (net profit or loss) What I’m trying to express here is that a tabular layout should know how to display accounts and groups of accounts: put the account name on the left and the balance on the right. And that the report engine should know how to re-use the definitions of account groups, labels, and dates to build up the options dialog box. We can help it along a little, e.g. in the call to (def-date start-date General/Start date 2011-01-01) you see that the second argument is a string that provides the tab the widget should be placed in (“General”) and widget’s label for the dialog box (“Start date”), separated by a slash. Oh, and since it’s a def-date, the option widget should be a date picker. This functionality can all be behind the scenes, leaving the report developer to just declare what his/her report should contain. Note that in the call to (def-account net-profit-loss Net Profit (Loss) …) the second argument is an empty string, meaning don’t place this as a widget in the options dialog box. If we stick with Scheme, we can take advantage of all the low-level functions that already exist for data extraction and report layout. But we can also move to a declarative model where we can have convention (re-use the report definitions as options) over configuration (build an options dialog box). Also, is it still true that we have to restart GnuCash every time we change a Scheme report, to see the changes? In any case, we need to make it dead easy for users to import and run and custom reports. Best, Yawar * I find that I’m saying ‘declarative’ a lot nowadays–I think it has to do with the fact that I’m learning Haskell :-) PGP.sig Description: This is a digitally signed message part ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Reporting system and potentially Python
On Jul 8, 2011, at 8:15 PM, Yawar Amin wrote: Folks, On 2011-07-08, at 01:15, Yawar Amin wrote: […] I like the idea of XML - HTML + CSS + JavaScript plots. How about moving to a completely declarative XML report, where we use XML tags to ask for things like account names and balances, and using the Scheme (or […] I got to thinking that Scheme itself can be used as a declarative,* domain-specific language for defining GnuCash reports. We could have a declarative API that sits on top of the currently-existing report API, and abstracts away the complexities of writing out an HTML page and building a report options dialog box. Sort of like LaTeX hides the raw TeX complexities of laying out a page. Here’s my income statement converted over from XML into (declarative) Scheme: (report income-statement (defs (def-date start-date General/Start date 2011-01-01) (def-date end-date General/End date 2011-07-31) (def-label title General/Report title Income Statement) (def-label subtitle General/Report subtitle '((string (date start-date)) to (string (date end-date (def-account-group income Accounts/Income accounts '(3288757aa761c7bba993766bfb433466 e49c976843a1dba40570ed6295e89a33)) (def-account-group expenses Accounts/Expense accounts '(df72fb29f46e90e12c2ba10d582a6148 e518845492b7b9cfa614cc405a8cf715)) (def-account net-profit-loss Net Profit (Loss) (subtract (balance (account-group income)) (balance (account-group expenses) (title (label title)) (subtitle (label subtitle)) (filter-date (date start-date) (date end-date”) (group-date 'month (tabular (account-group income) (account-group expenses) (account net-profit-loss) And here’s the report structure in plain language: report definitions (can double as options for the report options dialog box) definition of date (start date) definition of date (end date) definition of label (report title) definition of label (report subtitle) definition of account group (income) definition of account group (expenses) definition of a calculated account (net profit or loss = income - expenses) report title report subtitle filter by date (only include transactions within date range) group by date (monthly–this should show balances in the table as one column per month) tabular display (we could also provide a graphical display–i.e. column chart) account group (income) account group (expenses) account (net profit or loss) What I’m trying to express here is that a tabular layout should know how to display accounts and groups of accounts: put the account name on the left and the balance on the right. And that the report engine should know how to re-use the definitions of account groups, labels, and dates to build up the options dialog box. We can help it along a little, e.g. in the call to (def-date start-date General/Start date 2011-01-01) you see that the second argument is a string that provides the tab the widget should be placed in (“General”) and widget’s label for the dialog box (“Start date”), separated by a slash. Oh, and since it’s a def-date, the option widget should be a date picker. This functionality can all be behind the scenes, leaving the report developer to just declare what his/her report should contain. Note that in the call to (def-account net-profit-loss Net Profit (Loss) …) the second argument is an empty string, meaning don’t place this as a widget in the options dialog box. If we stick with Scheme, we can take advantage of all the low-level functions that already exist for data extraction and report layout. But we can also move to a declarative model where we can have convention (re-use the report definitions as options) over configuration (build an options dialog box). Also, is it still true that we have to restart GnuCash every time we change a Scheme report, to see the changes? In any case, we need to make it dead easy for users to import and run and custom reports. Best, Yawar * I find that I’m saying ‘declarative’ a lot nowadays–I think it has to do with the fact that I’m learning Haskell :-) Fun. Two questions: Can that be easily converted into a string parser so that normal users aren't put off by the extra parentheses, and is there anything about that that works in Scheme but not in C? Regards, John Ralls ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Reporting system and potentially Python
Am Donnerstag, 7. Juli 2011 schrieb John Ralls: So I am OK with needing to get more familiar with something such as Python, C, or Java to do the work, but I don't want to get knee deep in a major feature patch to find out something else is preferred. If you are able to insert some python code that will display some report, I will fully agree this to be a worthwhile goal and I will happily apply your patches to SVN trunk. Christian: Please don't accept Python code into Gnucash. Python has version-to-version compatibility problems, and I've found with pure-python applications like Gramps that it requires bundling a Python interpreter with the application so that everyone gets the same version. You do realize that gnucash already comes with a lot of python code for quite some time being, don't you? Namely we already have the python wrappers for the core engine object being generated by SWIG for approx. 2-3 years by now. More recently, there has been an optional python module added so that the framework is already there to do some real new features with that. To me, issues with version incompatibilities in python are indeed a drawback of that language, but all languages have drawbacks and advantages. The clear advantages to allow python code in gnucash is that * the python wrappers for all engine objects are already there, so using them is a rather efficient way of using your time to implement something new * the python language is significantly more easy to learn and suits the normal programmer's thinking much better than other languages As for JavaScript, you have the disadvantage that we (so far) don't have wrappers for all of our engine objects, so this is a very boring task that needs to be begun and somewhat completed first. Also, the JavaScript language is significantly difficult to work with in a serious way. It's much more like Scheme than one would initially expect, and it suffers the same problem than Scheme: Only a smaller set of programmers really understand the concepts behind it well enough to create good architectures in that language. With this being said, we had Tim's question here: I'd like to improve something with the reports; which language should I use. With the above advantages and disadvantages, I still think the most efficient way for Tim to spend his time is creating a report in python. And yes, I still think I will happily add related python code into the optional src/python/ module, even though we know about the limitations of python w.r.t. version issues. But we also know about its benefits. My cost-benefit comparison so far still turns out python as the suggested way to proceed for now. Regards, Christian ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Reporting system and potentially Python
On Jul 7, 2011, at 4:24 PM, Christian Stimming wrote: Am Donnerstag, 7. Juli 2011 schrieb John Ralls: So I am OK with needing to get more familiar with something such as Python, C, or Java to do the work, but I don't want to get knee deep in a major feature patch to find out something else is preferred. If you are able to insert some python code that will display some report, I will fully agree this to be a worthwhile goal and I will happily apply your patches to SVN trunk. Christian: Please don't accept Python code into Gnucash. Python has version-to-version compatibility problems, and I've found with pure-python applications like Gramps that it requires bundling a Python interpreter with the application so that everyone gets the same version. You do realize that gnucash already comes with a lot of python code for quite some time being, don't you? Namely we already have the python wrappers for the core engine object being generated by SWIG for approx. 2-3 years by now. More recently, there has been an optional python module added so that the framework is already there to do some real new features with that. To me, issues with version incompatibilities in python are indeed a drawback of that language, but all languages have drawbacks and advantages. The clear advantages to allow python code in gnucash is that * the python wrappers for all engine objects are already there, so using them is a rather efficient way of using your time to implement something new * the python language is significantly more easy to learn and suits the normal programmer's thinking much better than other languages As for JavaScript, you have the disadvantage that we (so far) don't have wrappers for all of our engine objects, so this is a very boring task that needs to be begun and somewhat completed first. Also, the JavaScript language is significantly difficult to work with in a serious way. It's much more like Scheme than one would initially expect, and it suffers the same problem than Scheme: Only a smaller set of programmers really understand the concepts behind it well enough to create good architectures in that language. With this being said, we had Tim's question here: I'd like to improve something with the reports; which language should I use. With the above advantages and disadvantages, I still think the most efficient way for Tim to spend his time is creating a report in python. And yes, I still think I will happily add related python code into the optional src/python/ module, even though we know about the limitations of python w.r.t. version issues. But we also know about its benefits. My cost-benefit comparison so far still turns out python as the suggested way to proceed for now. There's a big difference between a toy Python console for hackers and reports at the core of user interaction with Gnucash. Like src/gnc, src/optional.gtkmm, and src/cmakefilemodules, src/optional/python isn't appropriate to include in a general-distrbution binary. The standard reports *must* be included in such a binary. You're right, though, that Javascript is a non-starter. I hadn't adequately investigated what it would take to access the data from javascript. (Details in my reply to Phil.) I think Tim is looking for something different than you think, but I'll let him speak for himself. Regards, John Ralls ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Reporting system and potentially Python
What I'm looking for is this: Separate the act of actually gathering the data from the GnuCash database from transforming it for display, so that the output can be made much more interactive and easy to style and make 'pretty' :). Use a well known programming language to aggregate the data to make it easier to create new reports which output different sets of data. I've thought about this a little bit, and here is what I am thinking. Let me know if this sounds un/reasonable or just plain incorrect. (Note that I am basing this off of not being familiar yet with how the current scheme architecture actually manages pulling the data out of gnucash, so please correct me where I am wrong or point out the significant flaws) 1. Create the 'new' reporting system alongside the existing one so that the reports do not suffer until the existing functionality can be fully replaced by the new system. After all reports are replaced and working, the old scheme code could be pulled. 2. Create a set of libraries and/or use the existing gnucash libraries for querying information from the database. Based on the discussion, I presume C or Objective C would be best and just avoid Python/Scheme altogether (I am not sure how the scheme system actually performs the data queries right now). If all the necessary code is already in place, then this does not require any work. 3. Using these libraries, create the code for aggregating the data (also in C or Objective C) for each report. A single prototype report would be OK to start. Output the report data as structured XML. The XML data should adhere to an XML schema. 4. For each report, create an XSLT file which will transform the data into HTML/XHTML for display. 5. A small amount of javascript would be needed to perform the actual XSLT transformation but it would be pretty small. 6. Style the page elements using CSS. Also, I think the jqplot patch that has been made could be migrated in to the new reporting system, as those graphs look really nice. I think there are several benefits to this approach: 1. Currently reports can only be exported as HTML. By making the reporting code export an XML data structure, reports could be easily exported from GnuCash as XML (pre-transformation) or as HTML (post-transformation). 2. Anyone could write their own XSLT transformation file to display the data differently. 3. The actual XSLT, HTML, and CSS code would be very easy to read and modify by anyone familiar with it. The backend reports would still be in C/Objective C for compiling the data and as such require some coding knowledge, but it would be much easier to start using than the existing scheme code. 4. The XSLT and CSS could be very easily modified and applied not only inside of GnuCash, but a user could also take the XSLT and CSS scripts, modify and apply them to the exported XML reports to give it their own look without touching the GnuCash code base at all. For example they might want to style a report with a company logo or different colors or other graphical elements. 5a. In the future, it would be possible to provide multiple different XSLT and/or CSS styles for the reports so users could select the appearance of the reports. (If someone writes additional styles) 5b. In the future, it would be possible to allow users to import their own style sheets into GnuCash and apply them to the reports. (if someone writes the code to manage the import and style selection) 6. No reliance on Scheme or Python. Minimal Javascript, but that should be handled easily by webkit. Thoughts? -Tim On Thu, Jul 7, 2011 at 4:58 PM, John Ralls jra...@ceridwen.us wrote: On Jul 7, 2011, at 4:24 PM, Christian Stimming wrote: Am Donnerstag, 7. Juli 2011 schrieb John Ralls: So I am OK with needing to get more familiar with something such as Python, C, or Java to do the work, but I don't want to get knee deep in a major feature patch to find out something else is preferred. If you are able to insert some python code that will display some report, I will fully agree this to be a worthwhile goal and I will happily apply your patches to SVN trunk. Christian: Please don't accept Python code into Gnucash. Python has version-to-version compatibility problems, and I've found with pure-python applications like Gramps that it requires bundling a Python interpreter with the application so that everyone gets the same version. You do realize that gnucash already comes with a lot of python code for quite some time being, don't you? Namely we already have the python wrappers for the core engine object being generated by SWIG for approx. 2-3 years by now. More recently, there has been an optional python module added so that the framework is already there to do some real new features with that. To me, issues with version incompatibilities in python are indeed a drawback of that language, but all languages have drawbacks and advantages. The clear
Re: Reporting system and potentially Python
On Jul 7, 2011, at 9:48 PM, Tim M wrote: What I'm looking for is this: Separate the act of actually gathering the data from the GnuCash database from transforming it for display, so that the output can be made much more interactive and easy to style and make 'pretty' :). Use a well known programming language to aggregate the data to make it easier to create new reports which output different sets of data. I've thought about this a little bit, and here is what I am thinking. Let me know if this sounds un/reasonable or just plain incorrect. (Note that I am basing this off of not being familiar yet with how the current scheme architecture actually manages pulling the data out of gnucash, so please correct me where I am wrong or point out the significant flaws) 1. Create the 'new' reporting system alongside the existing one so that the reports do not suffer until the existing functionality can be fully replaced by the new system. After all reports are replaced and working, the old scheme code could be pulled. 2. Create a set of libraries and/or use the existing gnucash libraries for querying information from the database. Based on the discussion, I presume C or Objective C would be best and just avoid Python/Scheme altogether (I am not sure how the scheme system actually performs the data queries right now). If all the necessary code is already in place, then this does not require any work. 3. Using these libraries, create the code for aggregating the data (also in C or Objective C) for each report. A single prototype report would be OK to start. Output the report data as structured XML. The XML data should adhere to an XML schema. 4. For each report, create an XSLT file which will transform the data into HTML/XHTML for display. 5. A small amount of javascript would be needed to perform the actual XSLT transformation but it would be pretty small. 6. Style the page elements using CSS. Also, I think the jqplot patch that has been made could be migrated in to the new reporting system, as those graphs look really nice. I think there are several benefits to this approach: 1. Currently reports can only be exported as HTML. By making the reporting code export an XML data structure, reports could be easily exported from GnuCash as XML (pre-transformation) or as HTML (post-transformation). 2. Anyone could write their own XSLT transformation file to display the data differently. 3. The actual XSLT, HTML, and CSS code would be very easy to read and modify by anyone familiar with it. The backend reports would still be in C/Objective C for compiling the data and as such require some coding knowledge, but it would be much easier to start using than the existing scheme code. 4. The XSLT and CSS could be very easily modified and applied not only inside of GnuCash, but a user could also take the XSLT and CSS scripts, modify and apply them to the exported XML reports to give it their own look without touching the GnuCash code base at all. For example they might want to style a report with a company logo or different colors or other graphical elements. 5a. In the future, it would be possible to provide multiple different XSLT and/or CSS styles for the reports so users could select the appearance of the reports. (If someone writes additional styles) 5b. In the future, it would be possible to allow users to import their own style sheets into GnuCash and apply them to the reports. (if someone writes the code to manage the import and style selection) 6. No reliance on Scheme or Python. Minimal Javascript, but that should be handled easily by webkit. I like the idea of XML output with XSLT to transform it to HTML. That would be a great enhancement. Moving away from a heavy-weight general language like Scheme or Python is the right directin, but switching to C is unfortunately a step backward for custom reports. The current Scheme query interface makes it difficult to write a custom query; a C (or any other compiled language) query would make it impossible for anyone who isn't able to recompile Gnucash. Besides, C isn't really very expressive as a query language. Something SQL-like would be better. A query-by-example GUI would be best. I suppose that could be split off as a separate project, though, so that the Scheme query capability would remain for custom reports while the standard reports have their queries written in C. Regards, John Ralls ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: Reporting system and potentially Python
On 2011-07-07, at 21:48, Tim M wrote: […] Thoughts? I like the idea of XML - HTML + CSS + JavaScript plots. How about moving to a completely declarative XML report, where we use XML tags to ask for things like account names and balances, and using the Scheme (or Python) reporting engine to replace the XML tags with actual values? That way, you can have something like: report titleIncome Statement/title subtitle start_date/ to end_date/ /subtitle filter type=“date” fromstart_date//from toend_date//to tabular account_group name=“income”/ account_group name=“expenses”/ account title=“Net Profit (Loss)” add account_group_balance name=“income”/ account_group_balance name=“expenses”/ /add /account /tabular /filter type=“date” /report So people can be given a set of XML reporting ‘primitives’, and combine them in different ways to write different reports. My 2c, Yawar PGP.sig Description: This is a digitally signed message part ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
