Re: Please help me start with a CSV importer

[Martin Blais]

I don't understand your question.
If you can post your example files, we can debug it.
If the regexp you doesn't match anything in your files, if won't match the
importer.
The matching code is here, add some printfs there:
https://bitbucket.org/blais/beancount/src/26f7bb21007f1fb49fdd23829470fc4df88a6d4c/beancount/ingest/importers/regexp.py?at=default=file-view-default#regexp.py-51




On Sun, Apr 22, 2018 at 3:46 PM, viq  wrote:

> On 18-04-08 17:01:48, Martin Blais wrote:
> > The CSV importer that is provided puts together a regexp from the list of
> > fields you provide.
> > This regexp is run against the contents of the files it is scanning, in
> > order to identify whether a particular importer should be able to
> process a
> > file.
> >
> > The regexp is built from the fields you provide, I think, for this
> > importer, it's in the source code.
> > You'll have to open up the source code to debug this, but I suspect your
> > list of fields may not exactly match that from your files.
> > (There ought to be better tools for debugging this, but because setting
> up
> > the import config usually requires coding, people just override methods
> and
> > put prints and debug it that way.)
>
> Sorry for long delay, I finally tried looking into this again.
> It seems like I'm severely misunderstanding something here. My importer
> init now looks like this:
>
> def __init__(self, account):
> csv.Importer.__init__(
> self, self.config,
> account, 'Currency',
> ('Transaction Date,Posted Date,Description,Payee,'
>  'Payee account,Amount,Balance'),
> 1)
> print("REGEXPS: {}".format(self.regexps))
>
> but what I'm seeing, makes no sense, I don't expect it to match anything
> ever, unless I'm misunderstanding how this works:
> $ PYTHONPATH=`pwd` bean-identify test.config.py bank_history_sample.csv
>
> REGEXPS: [('Transaction Date,Posted Date,Description,Payee,Payee
> account,Amount,Balance', re.compile('Transaction Date,Posted
> Date,Description,Payee,Payee account,Amount,Balance',
> re.IGNORECASE|re.MULTILINE|re.DOTALL))]
>  /home/viq/Work/Own/beancount/bank_history_sample.csv
>
> --
> You received this message because you are subscribed to the Google Groups
> "Beancount" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to beancount+unsubscr...@googlegroups.com.
> To post to this group, send email to beancount@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/beancount/20180422194611.z32tlg24bxpf27xk%40hirauchi.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to beancount+unsubscr...@googlegroups.com.
To post to this group, send email to beancount@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/beancount/CAK21%2BhMjLJW51rkirAafbKLKA5-CZYY0bRczt1TfZmO9%2BtC93w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: When to use tags and when to create separate account?

[Martin Blais]

On Sun, Apr 22, 2018 at 5:43 PM, Martin Michlmayr  wrote:

> * Martin Blais  [2018-04-22 17:31]:
> > You can achieve the same thing already with a group-by on the (account,
> > meta) pair:
>
> Doh, I should have thought of that.  That's a good example of why BQL
> is a great idea.
>

The most relevant insight in this whole thing is IMHO the fact that you can
reduce all these accounting tasks to a single query on a single table (a
join of transaction fields and postings fields). I started with the SQL
thing not really knowing how far it would take us, but it turns out it
pretty much can do it all, and you can factor out the reporting rendering
code into two functions: one to render journals (generic, just a table),
and one to render trees of accounts (implies a hierarchy from one
particular column). Actual pivot tables (e.g., expenses by month) can be a
post-processing of a group-by query with two columns (e.g., account,
month-of-date, rendering sum(postings) in the cells).

-- 
You received this message because you are subscribed to the Google Groups 
"Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to beancount+unsubscr...@googlegroups.com.
To post to this group, send email to beancount@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/beancount/CAK21%2BhMpurrXtsgqDXR-F7AWXkRwgjJdFAXG36YGiikeUdiOGQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: When to use tags and when to create separate account?

[mployby]

Thank you for all your replies guys! I will stick to separate accounts for 
now, but will sure play with group-by feature.

-- 
You received this message because you are subscribed to the Google Groups 
"Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to beancount+unsubscr...@googlegroups.com.
To post to this group, send email to beancount@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/beancount/214d2afe-2c9e-4d64-9de3-f01213409eeb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: When to use tags and when to create separate account?

[Martin Michlmayr]

* Martin Blais  [2018-04-22 17:31]:
> You can achieve the same thing already with a group-by on the (account,
> meta) pair:

Doh, I should have thought of that.  That's a good example of why BQL
is a great idea.
-- 
Martin Michlmayr
http://www.cyrius.com/

-- 
You received this message because you are subscribed to the Google Groups 
"Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to beancount+unsubscr...@googlegroups.com.
To post to this group, send email to beancount@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/beancount/20180422214357.zol27ue65w573omm%40jirafa.cyrius.com.
For more options, visit https://groups.google.com/d/optout.

Re: When to use tags and when to create separate account?

[Martin Blais]

On Sun, Apr 22, 2018 at 8:30 AM, Martin Michlmayr  wrote:

> * mplo...@gmail.com  [2018-04-21 05:37]:
> > As the title says i wonder when to use tags and when to create
> > separate account?
>
> There is no right or wrong.  Some people like to put a lot of
> information into their account names while others prefer simple account
> names.  My general impression is that professional accounts prefer a
> simple chart of accouts, though.
>
> > Or should i use same accounts
> >
> > Income:Business:
> > Expenses:Business:Ads
> > Expenses:Business:Hosting
> >
> > and just use tags #ECOM1 and #ECOM2 ?
>

It's an arbitrary choice.
The true question is about how you want to query it.
Consider this: Beancount essentially produces a single table of postings
whose fields are always joined with the fields of the transaction which
contains them.
e.g.

   

It all boils down to how you want to aggregate them.
You can aggregate them by account name.
You can aggregate them by tag.
You can aggregate them by combinations of those (and other fields as well).

The default reports (balance sheet, income statement) use the account names
to produce aggregations, so that's one consideration, but it doesn't matter
much, as in a future version of Beancount, the reports will be produced by
queries as well (more easily customizable in that way).



Personally, I like simple account names so I'd go with that plus tags.
>
> Ledger has an interesting concept called pivot which allows you to do
> both.  It allows you to make metadata information to become part of
> the account name.
>
> e.g.:
>
> 2018-04-22 * Income from ABC
> Assets:Bank100.00 EUR
> Income:Business:Hosting   -100.00 EUR
> ; Customer: ABC
>
> 2018-04-22 * Income from CDE
> Assets:Bank150.00 EUR
> Income:Business:Hosting   -150.00 EUR
> ; Customer: CDE
>
> Normal balance:
>
> ledger -f a bal income
>  -250.00 EUR  Income:Business:Hosting
>
> Balance with --pivot:
>
> ledger -f a bal income --pivot Customer
>  -250.00 EUR  Customer
>  -100.00 EURABC:Income:Business:Hosting
>  -150.00 EURCDE:Income:Business:Hosting
> 
>  -250.00 EUR
>
> The same would be easy to do with a beancount plugin.  I don't know if
> anyone has done it already.  If not, I intend to create one in the
> near future.
>

You can achieve the same thing already with a group-by on the (account,
meta) pair:


plugin "beancount.plugins.auto"

2018-04-22 * "Income from ABC"
customer: "ABC"
Assets:Bank100.00 EUR
Income:Business:Hosting   -100.00 EUR

2018-04-22 * "Income from CDE"
customer: "CDE"
Assets:Bank150.00 EUR
Income:Business:Hosting   -150.00 EUR


;; bean-query
/home/blais/r/q/beancount-data/user/michlmayr/groupby.beancount
;; "select account, entry_meta('customer'), sum(position) where account ~
'Income' group by 1, 2"
;; account ent sum_positio
;; --- --- ---
;; Income:Business:Hosting ABC -100.00 EUR
;; Income:Business:Hosting CDE -150.00 EUR

-- 
You received this message because you are subscribed to the Google Groups 
"Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to beancount+unsubscr...@googlegroups.com.
To post to this group, send email to beancount@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/beancount/CAK21%2BhOMk-pCvD7B45%2Bmj5q_G28EbOxx7cJFJ3X-1Y97p5MXzg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Importing data from vanguard and other institutions

[Martin Blais]

Ledgerhub is a retired project.

I have 3 importers that support Vanguard for OFX, CSV and PDF (for filing
only).
I don't maintain these externally by default, but if you want I can share a
copy of them.
(Warning: it's not great code, and it depends a little bit on the structure
of how accounts.)




On Sun, Apr 22, 2018 at 11:02 AM,  wrote:

> Martin,
>
> Currently, where is the "ledgerhub.importers.generic.ofx_invest"
> importer, or anything that substitutes it for Vanguard?
>
> Thanks,
> Lluis
>
> On Friday, May 22, 2015 at 4:57:35 PM UTC-7, redst...@gmail.com wrote:
>>
>> Hi Martin or anyone who has experience with this: I'm trying to figure
>> out how others are getting the data from their brokerage/investment
>> institutions into Beancount. I'll use Vanguard as an example. Specifically:
>>
>> 1. The import process: Do you use .csv or ofx? With ofx, Vanguard only
>> allows the past 18 months of transactions. So I have screenscraped .csv
>> data, which doesn't have the 18-month limit. I see an incompletely
>> ledgerhub importer. I'm trying to not reinvent the wheel here. Also, to get
>> into the gorier details, are there things to watch out for when importing?
>> I have my own importer cobbled together when I used ledger, but that falls
>> far short of what I'm hoping beancount will help me accomplish.
>>
>> 2. I've used Specific ID of shares. So I'm mainly interested in automatic
>> lot matching. Is this supported already? The documentation says to watch
>> out for "upcoming changes." Even if it is not supported, what's the best
>> practice on getting my data in, in such a way that a) when support arrives,
>> everything will work, and b) so I don't leave out any data from the source,
>> and get everything into Beancount so I don't have to redo this?
>>
>> Thanks in advance!
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Beancount" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to beancount+unsubscr...@googlegroups.com.
> To post to this group, send email to beancount@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/beancount/65f33a6c-2589-4d3e-8237-8b0e73d560d2%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to beancount+unsubscr...@googlegroups.com.
To post to this group, send email to beancount@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/beancount/CAK21%2BhO7rai2XAZuWBAXvbEp69xGfaQN%3DCLQxGWJfiLQ4C5Ypg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Please help me start with a CSV importer

[viq]

On 18-04-08 17:01:48, Martin Blais wrote:
> The CSV importer that is provided puts together a regexp from the list of
> fields you provide.
> This regexp is run against the contents of the files it is scanning, in
> order to identify whether a particular importer should be able to process a
> file.
> 
> The regexp is built from the fields you provide, I think, for this
> importer, it's in the source code.
> You'll have to open up the source code to debug this, but I suspect your
> list of fields may not exactly match that from your files.
> (There ought to be better tools for debugging this, but because setting up
> the import config usually requires coding, people just override methods and
> put prints and debug it that way.)

Sorry for long delay, I finally tried looking into this again.
It seems like I'm severely misunderstanding something here. My importer
init now looks like this:

def __init__(self, account):
csv.Importer.__init__(
self, self.config,
account, 'Currency',
('Transaction Date,Posted Date,Description,Payee,'
 'Payee account,Amount,Balance'),
1)
print("REGEXPS: {}".format(self.regexps))

but what I'm seeing, makes no sense, I don't expect it to match anything
ever, unless I'm misunderstanding how this works:
$ PYTHONPATH=`pwd` bean-identify test.config.py bank_history_sample.csv
REGEXPS: [('Transaction Date,Posted Date,Description,Payee,Payee 
account,Amount,Balance', re.compile('Transaction Date,Posted 
Date,Description,Payee,Payee account,Amount,Balance', 
re.IGNORECASE|re.MULTILINE|re.DOTALL))]
 /home/viq/Work/Own/beancount/bank_history_sample.csv

-- 
You received this message because you are subscribed to the Google Groups 
"Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to beancount+unsubscr...@googlegroups.com.
To post to this group, send email to beancount@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/beancount/20180422194611.z32tlg24bxpf27xk%40hirauchi.
For more options, visit https://groups.google.com/d/optout.

Re: Importing data from vanguard and other institutions

[llpamies]

Martin,

Currently, where is the "ledgerhub.importers.generic.ofx_invest" importer, 
or anything that substitutes it for Vanguard?

Thanks,
Lluis

On Friday, May 22, 2015 at 4:57:35 PM UTC-7, redst...@gmail.com wrote:
>
> Hi Martin or anyone who has experience with this: I'm trying to figure out 
> how others are getting the data from their brokerage/investment 
> institutions into Beancount. I'll use Vanguard as an example. Specifically:
>
> 1. The import process: Do you use .csv or ofx? With ofx, Vanguard only 
> allows the past 18 months of transactions. So I have screenscraped .csv 
> data, which doesn't have the 18-month limit. I see an incompletely 
> ledgerhub importer. I'm trying to not reinvent the wheel here. Also, to get 
> into the gorier details, are there things to watch out for when importing? 
> I have my own importer cobbled together when I used ledger, but that falls 
> far short of what I'm hoping beancount will help me accomplish.
>
> 2. I've used Specific ID of shares. So I'm mainly interested in automatic 
> lot matching. Is this supported already? The documentation says to watch 
> out for "upcoming changes." Even if it is not supported, what's the best 
> practice on getting my data in, in such a way that a) when support arrives, 
> everything will work, and b) so I don't leave out any data from the source, 
> and get everything into Beancount so I don't have to redo this?
>
> Thanks in advance!
>

-- 
You received this message because you are subscribed to the Google Groups 
"Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to beancount+unsubscr...@googlegroups.com.
To post to this group, send email to beancount@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/beancount/65f33a6c-2589-4d3e-8237-8b0e73d560d2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.