Re: Getting start with Ledger and Python
Tim Crews t...@code-affinity.com writes: With that said, here are the tasks that I was planning to start chipping away at. None of this has been coordinated with John. I would welcome any feedback if my work is misguided. 1. I created a github fork at https://github.com/code-affinity/ledger 2. I want to produce a Python API document. I don't think the majority of Python developers would be comfortable crawling through the C++ code to discover the capabilities. Since the exposed Python functions don't have any docstrings (yet), the usual Pythonic approach to exploration from the Python REPL isn't very productive either. 3. I created a python_api_reference topic branch for this purpose. 4. I've started by editing docs/ledger3.texi to add documentation for each Python class. This is quickly getting large, and the section nesting levels are getting out of hand. I think it might need to be a separate document. Opinions? 5. So far, I'm only listing classes, methods, and properties by name, and listing parameters and return types. There is no descriptive documentation yet. 6. For each method/property, if I see any problems, I'm adding a @comment right in the texinfo document to describe the problem. I was planning to update Bugzilla bug 306 with my findings once I'm done. Summarizing what I've found so far: 1. Lots of methods that would be more Pythonic if they were represented as read-only properties (for example the xacts() and posts() methods that have been discussed so much in this thread). 2. Several native C++ types need converting to native Python types to be useful to a Python developer. For example, datetime and date, boost::filesystem::path, and perhaps FileInfo. 3. A few C++ containers are returned directly without giving them iterator wrappers first. 4. The C++ code uses boost::optional, and I don't think Boost::Python knows how to wrap that without help, so I think a conversion might be needed to produce the Python None type as appropriate. I could be wrong about that. 7. This process takes a lot of time! I've probably put 20 hours into it so far, and even with this incomplete documentation, I've only finished Account, AccountXData, AccountXDataDetails, Amount, FileInfo, Journal, PostCollectorWapper, and PostHandler. I guess this is about 25% of the overall work required. Ouch! 8. Depending on any feedback from John Wiegley or other interested parties, update the Python bridge implementation, and modify the documentation to match. 9. Fill out the documentation with user-friendly descriptions, perhaps stealing comments from the C++ code where they exist. 10. Modify the Python bridge code to include Python docstrings for each class, method, and property. (This is not hard with boost::python.) Python developers really like their docstrings! 11. Time permitting, I really want to figure out why ledger python segfaults so often. Even if it were feature-complete, the frequent segfaults make it very difficult for me to use and explore. Also, after ledger python exits, my shell is usually in some weird state that doesn't show subsequent typed input. That is also bad for productivity. And none of this involves writing any Python! I don't enjoy this kind of work, but I see it as preparatory for the more-fun stuff. This is fantastic, Tim. I hope you can get to the Python part sooner rather than later! I'll help you with the segfaults as you go, though my time may be a little limited this summer (several work contracts ongoing). John
Re: Getting start with Ledger and Python
Stefan Tunsch stun...@gmail.com writes: Hi! I am a Python developer, also interested in developing a good interface from Python to ledger. I have also taken a look at projects like beancount, who are reimplementations of ledger in Python, but I would prefer to stick to the original project. That said, I am willing to invest time developing this. So Tim and John, if you need help, let's team up and see if we can get this going. Stefan, I would love to have your help on that. Maybe you can coordinate with Tim and Simon here on the mailing list as the Python work progresses? John
Re: Getting start with Ledger and Python
Tim Crews tim-QJ3Fn9itRil4wfmgmkLWVgC/g2k4z...@public.gmane.org writes: I've been reviewing the entire Python API with an eye towards putting together an equivalent RESTful web API (see other post), and in the process I think I see a few other functions that might be more Pythonic if they were exposed as read-only properties. I was planning to publish a summary soon of the entire Python API, and I will note these. If you think we can get to where we want with Python + Javascript, then I can be talked away from trying to develop a full RESTful API. John
Re: Getting start with Ledger and Python
Hi! I am a Python developer, also interested in developing a good interface from Python to ledger. I have also taken a look at projects like beancount, who are reimplementations of ledger in Python, but I would prefer to stick to the original project. That said, I am willing to invest time developing this. So Tim and John, if you need help, let's team up and see if we can get this going. Regards, Stefan Tunsch El 30/05/2012 10:34, John Wiegley escribió: Tim Crewstim-QJ3Fn9itRil4wfmgmkLWVgC/g2k4z...@public.gmane.org writes: I've been reviewing the entire Python API with an eye towards putting together an equivalent RESTful web API (see other post), and in the process I think I see a few other functions that might be more Pythonic if they were exposed as read-only properties. I was planning to publish a summary soon of the entire Python API, and I will note these. If you think we can get to where we want with Python + Javascript, then I can be talked away from trying to develop a full RESTful API. John
Re: Getting start with Ledger and Python
On Wednesday, May 30, 2012 1:46:49 AM UTC-7, stunsch wrote: I am willing to invest time developing this. So Tim and John, if you need help, let's team up and see if we can get this going. Regards, Stefan Tunsch I have some down-time, I love to write Python, I am good at C++, and I have received a lot of value from Ledger, so I see this as an opportunity to contribute. I do not want to give the impression that I'm trying to take charge of anything! With that said, here are the tasks that I was planning to start chipping away at. None of this has been coordinated with John. I would welcome any feedback if my work is misguided. 1. I created a github fork at https://github.com/code-affinity/ledger 2. I want to produce a Python API document. I don't think the majority of Python developers would be comfortable crawling through the C++ code to discover the capabilities. Since the exposed Python functions don't have any docstrings (yet), the usual Pythonic approach to exploration from the Python REPL isn't very productive either. 3. I created a python_api_reference topic branch for this purpose. 4. I've started by editing docs/ledger3.texi to add documentation for each Python class. This is quickly getting large, and the section nesting levels are getting out of hand. I think it might need to be a separate document. Opinions? 5. So far, I'm only listing classes, methods, and properties by name, and listing parameters and return types. There is no descriptive documentation yet. 6. For each method/property, if I see any problems, I'm adding a @comment right in the texinfo document to describe the problem. I was planning to update Bugzilla bug 306http://bugs.ledger-cli.org/show_bug.cgi?id=306with my findings once I'm done. Summarizing what I've found so far: 1. Lots of methods that would be more Pythonic if they were represented as read-only properties (for example the xacts() and posts() methods that have been discussed so much in this thread). 2. Several native C++ types need converting to native Python types to be useful to a Python developer. For example, datetime and date, boost::filesystem::path, and perhaps FileInfo. 3. A few C++ containers are returned directly without giving them iterator wrappers first. 4. The C++ code uses boost::optional, and I don't think Boost::Python knows how to wrap that without help, so I think a conversion might be needed to produce the Python None type as appropriate. I could be wrong about that. 7. This process takes a lot of time! I've probably put 20 hours into it so far, and even with this incomplete documentation, I've only finished Account, AccountXData, AccountXDataDetails, Amount, FileInfo, Journal, PostCollectorWapper, and PostHandler. I guess this is about 25% of the overall work required. Ouch! 8. Depending on any feedback from John Wiegley or other interested parties, update the Python bridge implementation, and modify the documentation to match. 9. Fill out the documentation with user-friendly descriptions, perhaps stealing comments from the C++ code where they exist. 10. Modify the Python bridge code to include Python docstrings for each class, method, and property. (This is not hard with boost::python.) Python developers really like their docstrings! 11. Time permitting, I really want to figure out why ledger python segfaults so often. Even if it were feature-complete, the frequent segfaults make it very difficult for me to use and explore. Also, after ledger python exits, my shell is usually in some weird state that doesn't show subsequent typed input. That is also bad for productivity. And none of this involves writing any Python! I don't enjoy this kind of work, but I see it as preparatory for the more-fun stuff. Any feedback? Tim Crews
Re: Getting start with Ledger and Python
Tim Crews t...@code-affinity.com writes: On Wednesday, May 30, 2012 1:34:07 AM UTC-7, JohnW wrote: If you think we can get to where we want with Python + Javascript, then I can be talked away from trying to develop a full RESTful API. John I hope I'm not displaying my ignorance, but I think I might be missing something. I'm in favor of a RESTful API. Are you referring to client-side JavaScript or server-side? I can imagine a client-side HTML application that interfaced to a ledger server, and this would involve plenty of client-side JavaScript, but that code would still need to communicate with the server to access Ledger functions. I think a REST API would be a great way to model that. I don't have a strong opinion that this is the best possible system architecture; this is just the model that I assumed. Did you have a different model in mind? I noticed in another thread that you mentioned a pure JavaScript implementation, and I didn't know what that meant. Tim Crews There's no reason a RESTful API couldn't be done also with python on the backend.
Re: Getting start with Ledger and Python
On 5/30/12 9:17 AM, Tim Crews wrote: Any feedback? This is excellent work Tim. Until more help shows up maybe you could pick just a small subset of the API to polish, so you can get to the segfault/terminal issues and then some fun and motivating demo ?
Re: Getting start with Ledger and Python
On the subject of getting the example Python API code to run: On Sunday, May 27, 2012 4:39:38 PM UTC-7, Tim Crews wrote: for xact in ledger.read_journal(ledger.dat).xacts(): # Note the () after xacts for post in xact.posts(): # Note the () after posts print Transferring %s to/from %s % (post.amount, post.account) Reading my original post a day later, I see that it probably came across as abrupt. I also posted without providing any context. I'm sorry about that. Anyway, I read the Boost::Python documents to figure out how to make an iterator wrapper that is also a property. The example given at http://www.boost.org/doc/libs/1_49_0/libs/python/doc/tutorial/doc/html/python/iterators.htmluses a .property function for this purpose, but that doesn't compile. So I tried .add_property instead, and this works. If you change py_journal.cc to use .add_property for xacts, and change py_xact.cc to use .add_property for posts, then the original Python sample code will run as written. I've been reviewing the entire Python API with an eye towards putting together an equivalent RESTful web API (see other post), and in the process I think I see a few other functions that might be more Pythonic if they were exposed as read-only properties. I was planning to publish a summary soon of the entire Python API, and I will note these. Does anyone have an opinion about what form a Python API summary document ought to take? I would be willing to add it to the docs or the wiki, or just post it on the list. Or am I duplicating someone else's effort? Tim Crews
Re: Getting start with Ledger and Python
henry atting me.henryatting-re5jqeeqqe8avxtiumw...@public.gmane.org writes: Traceback (most recent call last): File scr.py, line 4, in module for xact in ledger.read_journal(ledger.dat).xacts: AttributeError: 'module' object has no attribute 'read_journal' Ah, try this instead: for xact in read_journal(ledger.dat).xacts: ... The Python bridge is still evolving. John
Re: Getting start with Ledger and Python
Yes, this is exactly what I do. On May 12, 12:17 am, John Wiegley jwieg...@gmail.com wrote: henry atting me.henryatting-re5jqeeqqe8avxtiumw...@public.gmane.org writes: Just out of curiousity... I never managed to get any small python script running, ledger built with/or without python or whatsoever. My current version is a build from today: Ledger 3.0.0-20110325; with --python dir(ledger) displays what it should. No script provided in this thread does work though. E.g. this one: import ledger for xact in ledger.read_journal(sample.dat).xacts: for post in xact.posts: print Transferring %s to/from %s % (post.amount, post.account) Leads to this error message: Traceback (most recent call last): File /home/me/script.py, line 3, in module for xact in ledger.read_journal(sample.dat).xacts: AttributeError: 'module' object has no attribute 'read_journal' How are you invoking the script? Please try ledger python foo.py. John
Re: Getting start with Ledger and Python
Just out of curiousity... I never managed to get any small python script running, ledger built with/or without python or whatsoever. My current version is a build from today: Ledger 3.0.0-20110325; with --python dir(ledger) displays what it should. No script provided in this thread does work though. E.g. this one: import ledger for xact in ledger.read_journal(sample.dat).xacts: for post in xact.posts: print Transferring %s to/from %s % (post.amount, post.account) Leads to this error message: Traceback (most recent call last): File /home/me/script.py, line 3, in module for xact in ledger.read_journal(sample.dat).xacts: AttributeError: 'module' object has no attribute 'read_journal' Any hint appreciated. henry
Re: Getting start with Ledger and Python
henry atting me.henryatting-re5jqeeqqe8avxtiumw...@public.gmane.org writes: Just out of curiousity... I never managed to get any small python script running, ledger built with/or without python or whatsoever. My current version is a build from today: Ledger 3.0.0-20110325; with --python dir(ledger) displays what it should. No script provided in this thread does work though. E.g. this one: import ledger for xact in ledger.read_journal(sample.dat).xacts: for post in xact.posts: print Transferring %s to/from %s % (post.amount, post.account) Leads to this error message: Traceback (most recent call last): File /home/me/script.py, line 3, in module for xact in ledger.read_journal(sample.dat).xacts: AttributeError: 'module' object has no attribute 'read_journal' How are you invoking the script? Please try ledger python foo.py. John
Re: Getting start with Ledger and Python
Chris Rossi chris-nf+wzpsdgwelmaghn9gi760brtrcj...@public.gmane.org writes: The result is: Error: Unrecognized command 'python' I don't notice any errors during the build process. Any idea what I'm doing wrong? v2.6.3 never had a Python command. Could you try building the 'master' branch? Thanks, John
Re: Getting start with Ledger and Python
On Sunday, March 11, 2012 6:37:32 AM UTC-4, JohnW wrote: Raphael Lorenzeto writes: Long story short, I haven't found a way to access this Python interface, my intent is to produce a series of scripts to output HMTL5 files with my reports on a regular basis (probably a cron-job). How can I do this? Is there any way to communicate with Ledger without having to put it in REPL mode and creating the interface myself? Make sure to pass --python to acprep when you build (you'll have to make distclean if you're just adding that flag now). Then you use 'ledger python foo.py' to run a Python script with the ability to 'import ledger' in that script. See earlier postings in this mailing list for some examples of what you can do then... I've tried the following under Ubuntu 12.04: $ git clone https://github.com/jwiegley/ledger.git $ cd ledger $ git checkout v2.6.3 $ ./acprep --python $ make $ ./ledger python -f path/to/ledgerfile The result is: Error: Unrecognized command 'python' I don't notice any errors during the build process. Any idea what I'm doing wrong? Chris
Re: Getting start with Ledger and Python
Added to docs. On Thu, Mar 1, 2012 at 20:06, John Wiegley jwieg...@gmail.com wrote: Hello all, I think it's time for a primer on using Python to extend your Ledger experience. But first, a word must be said about Ledger's data model, so that other things make sense later. -- # Basic data traversal Every interaction with Ledger happens in the context of a Session. Even if you don't create a session manually, one is created for you by the top-level interface functions. The Session is where objects live like the Commodity's that Amount's refer to. The make a Session useful, you must read a Journal into it, using the function `read_journal`. This reads Ledger data from the given file, populates a Journal object within the current Session, and returns a reference to that Journal object. Within the Journal live all the Transaction's, Posting's, and other objects related to your data. There are also AutomatedTransaction's and PeriodicTransaction's, etc. Here is how you would traverse all the postings in your data file: import ledger for xact in ledger.read_journal(sample.dat).xacts: for post in xact.posts: print Transferring %s to/from %s % (post.amount, post.account) -- # Raw vs. Cooked Ledger data exists in one of two forms: raw and cooked. Raw objects are what you get from a traversal like the above, and represent exactly what was seen in the data file. Consider this journal: = true (Assets:Cash)$100 2012-03-01 KFC Expenses:Food$100 Assets:Credit In this case, the *raw* regular transaction in this file is: 2012-03-01 KFC Expenses:Food$100 Assets:Credit While the cooked form is: 2012-03-01 KFC Expenses:Food$100 Assets:Credit $-100 (Assets:Cash)$100 So the easy way to think about raw vs. cooked is that raw is the unprocessed data, and cooked has had all considerations applied. When you traverse a Journal by iterating its transactions, you are generally looking at raw data. In order to look at cooked data, you must generate a report of some kind by querying the journal: for post in ledger.read_journal(sample.dat).query(food): print Transferring %s to/from %s % (post.amount, post.account) The reason why queries iterate over postings instead of transactions is that queries often return only a slice of the transactions they apply to. You can always get at a matching posting's transaction by looking at its xact member: last_xact = None for post in ledger.read_journal(sample.dat).query(): if post.xact != last_xact: for post in post.xact.posts: print Transferring %s to/from %s % (post.amount, post.account) last_xact = post.xact This query ends up reporting every cooked posting in the Journal, but does it transaction-wise. It relies on the fact that an unsorted report returns postings in the exact order they were parsed from the journal file. -- # Queries The Journal.query() method accepts every argument you can specify on the command-line, including --options. Since a query cooks the journal it applies to, only one query may be active for that journal at a given time. Once the query object is gone (after the for loop), then the data reverts back to its raw state. -- # Embedded Python Can you embed Python into your data files using the 'python' directive: python import so def check_path(path_value): print %s = %s % (str(path_value), os.path.isfile(str(path_value))) return os.path.isfile(str(path_value)) tag PATH assert check_path(value) 2012-02-29 KFC ; PATH: somebogusfile.dat Expenses:Food$20 Assets:Cash Any Python functions you define this way become immediately available as valexpr functions. -- # Amounts When numbers come from Ledger, like post.amount, the type of the value is Amount. It can be used just like an ordinary number, except that addition and subtraction are restricted to amounts with the same commodity. If you need to create sums of multiple commodities, use a Balance. For example: total = Balance() for post in ledger.read_journal(sample.dat).query(): total += post.amount print total John -- Craig, Corona De Tucson, AZ enderw88.wordpress.com
Re: Getting start with Ledger and Python
Raphael Lorenzeto raphael.lorenzeto-re5jqeeqqe8avxtiumw...@public.gmane.org writes: Long story short, I haven't found a way to access this Python interface, my intent is to produce a series of scripts to output HMTL5 files with my reports on a regular basis (probably a cron-job). How can I do this? Is there any way to communicate with Ledger without having to put it in REPL mode and creating the interface myself? Make sure to pass --python to acprep when you build (you'll have to make distclean if you're just adding that flag now). Then you use 'ledger python foo.py' to run a Python script with the ability to 'import ledger' in that script. See earlier postings in this mailing list for some examples of what you can do then... John
Re: Getting start with Ledger and Python
Raphael Lorenzeto raphael.lorenzeto-re5jqeeqqe8avxtiumw...@public.gmane.org writes: Thanks! I managed to compile with the --python passed to acprep but I get the following error while trying to execute the examples shown on this post: ./ledger/ledger python ledger-script.py Traceback (most recent call last): File ledger-script.py, line 6, in module for post in post.xact.posts: TypeError: 'instancemethod' object is not iterable Abort trap: 6 It should be post.xacts.posts, I believe. John
Re: Getting start with Ledger and Python
Hello to everyone! This is my first post here. I have to confess that I was very impressed by the features of ledger. As an avid GNUCash user I was a little disappointed with the absence of GNUCash reading capabilities. I managed to write a perl script to parse and output a ledger-formatted file and than I started looking for ways to present my data. Long story short, I haven't found a way to access this Python interface, my intent is to produce a series of scripts to output HMTL5 files with my reports on a regular basis (probably a cron-job). How can I do this? Is there any way to communicate with Ledger without having to put it in REPL mode and creating the interface myself? Thanks! On Friday, March 2, 2012 12:06:48 AM UTC-3, John Wiegley wrote: Hello all, I think it's time for a primer on using Python to extend your Ledger experience. But first, a word must be said about Ledger's data model, so that other things make sense later. -- # Basic data traversal Every interaction with Ledger happens in the context of a Session. Even if you don't create a session manually, one is created for you by the top-level interface functions. The Session is where objects live like the Commodity's that Amount's refer to. The make a Session useful, you must read a Journal into it, using the function `read_journal`. This reads Ledger data from the given file, populates a Journal object within the current Session, and returns a reference to that Journal object. Within the Journal live all the Transaction's, Posting's, and other objects related to your data. There are also AutomatedTransaction's and PeriodicTransaction's, etc. Here is how you would traverse all the postings in your data file: import ledger for xact in ledger.read_journal(sample.dat).xacts: for post in xact.posts: print Transferring %s to/from %s % (post.amount, post.account) -- # Raw vs. Cooked Ledger data exists in one of two forms: raw and cooked. Raw objects are what you get from a traversal like the above, and represent exactly what was seen in the data file. Consider this journal: = true (Assets:Cash)$100 2012-03-01 KFC Expenses:Food$100 Assets:Credit In this case, the *raw* regular transaction in this file is: 2012-03-01 KFC Expenses:Food$100 Assets:Credit While the cooked form is: 2012-03-01 KFC Expenses:Food$100 Assets:Credit $-100 (Assets:Cash)$100 So the easy way to think about raw vs. cooked is that raw is the unprocessed data, and cooked has had all considerations applied. When you traverse a Journal by iterating its transactions, you are generally looking at raw data. In order to look at cooked data, you must generate a report of some kind by querying the journal: for post in ledger.read_journal(sample.dat).query(food): print Transferring %s to/from %s % (post.amount, post.account) The reason why queries iterate over postings instead of transactions is that queries often return only a slice of the transactions they apply to. You can always get at a matching posting's transaction by looking at its xact member: last_xact = None for post in ledger.read_journal(sample.dat).query(): if post.xact != last_xact: for post in post.xact.posts: print Transferring %s to/from %s % (post.amount, post.account) last_xact = post.xact This query ends up reporting every cooked posting in the Journal, but does it transaction-wise. It relies on the fact that an unsorted report returns postings in the exact order they were parsed from the journal file. -- # Queries The Journal.query() method accepts every argument you can specify on the command-line, including --options. Since a query cooks the journal it applies to, only one query may be active for that journal at a given time. Once the query object is gone (after the for loop), then the data reverts back to its raw state. -- # Embedded Python Can you embed Python into your data files using the 'python' directive: python import so def check_path(path_value): print %s = %s % (str(path_value), os.path.isfile(str(path_value))) return os.path.isfile(str(path_value)) tag PATH assert check_path(value) 2012-02-29 KFC ; PATH: somebogusfile.dat Expenses:Food$20 Assets:Cash Any Python functions you define this way become immediately available as
