Re: Importing transactions from bank.

[mployby]

Martin, thank you for your detailed reply! Now it's super clear :)

On Tuesday, April 17, 2018 at 1:45:23 AM UTC-4, Martin Blais wrote:
>
> On Sun, Apr 15, 2018 at 8:35 PM,  wrote:
>
>> I've started playing with import scripts,the sample files provided in 
>> repo are very useful, just have couple of additional questions i was not 
>> able to figure out/not 100% sure about:
>>
>
> Really glad the example files are useful.
>
>
> 1 bean-extract does not write transactions in main beancount file 
>> (personal.beancount), it generates output into command line or 
>> tmp.beancount and we enter them manually, right?
>>
>
> Correct. You redirect its output to a temp file and using your editor, 
> cut/paste them to your main file wherever makes most sense for you (and 
> complete them, e.g., categorize the expenses).
>
>  
>
>>
>> 2 when running  "bean-extract -e BEANCOUNT_FILE" what is the algorithm 
>> used to check duplication? Does it check only for same amounts? Or also 
>> takes into consideration additional clues (payee, date range, etc)?
>>
>
> Not so great, definitely could use some improvement:
>
> https://bitbucket.org/blais/beancount/src/cd5fe599fdc80d47caa8a2618f5ee489ce88d101/beancount/ingest/similar.py
>
> https://bitbucket.org/blais/beancount/issues/185/make-duplicate-detection-configureable
>
>
> 3 When filing files bean-file example.import ../Downloads/ -o documents/ 
>> how are name and date generated?
>> Seems it is picks up date from original file name (example 
>> UTrade20160215.csv), but when it's not in file name how ofxdownload.ofx 
>> turns to 2013-12-18.bofa.ofx
>>
>
> From the file_name() and file_date() methods of your importer 
> implementation:
>
> https://bitbucket.org/blais/beancount/src/cd5fe599fdc80d47caa8a2618f5ee489ce88d101/beancount/ingest/importer.py
> If you use the example CSV importer it does what it can.
> Typically you'd write your own and choose what you want it to be.
>
>  
>
>>
>> 4 Main value of regression testing is to insure that new 
>> changes/additions to importers still work with old input files,
>> It's not so useful to help importers work with new formats if format was 
>> changed by bank, right?
>>
>
> Both.
> Those importer codes are typically the last thing you want to do on a 
> Saturday night, so they tend to be kludgey.
> - When you need to support something new, you want to make sure you don't 
> break old functionality.
> - When the Java programmers at your favorite bank change the output format 
> without warning you (or realize 10 years after that escaping the commas in 
> the CSV fields might have been a good idea), there's no faster way to test 
> a new / updated importer than to plot a newer file in the directory and run 
> regression tests
>
>
> On Mon, Apr 16, 2018 at 8:30 AM,  wrote:
>
>> 5 Also what's the right way to calculate balance 
>> ?
>>
>
> Ideally you don't calculate it. Ideally you find it somewhere and enter it.
>
>  
>
>>
>> Chase have the following CSV structure, 
>> 
>>
>> Balance fields are included in checking CSV, so i guess i can use balance 
>> of the last transactions for balance.
>> They are not included in credit card CSV however. Is there a way to 
>> calculate them?
>>
>> And how important "balance" in the first place? Cause for example this 
>> import script 
>> https://gist.github.com/mterwill/7fdcc573dc1aa158648aacd4e33786e8#file-importers-chase-py
>>  does not have "balance".
>>
>
> They're optional.
>
> The reason for Balance directives is to assert (and I mean that in the 
> computer programming sense as well) that your ledger's calculated balance 
> matches the institution's idea of what it should be. The picture perfect 
> idea of a Balance directive is a fuddy duddy accountant type wearing brown 
> socks staring at the bottom line of his paper statement carefully through 
> bifocals and manually copying the number to a Balance directive in 
> Beancount.
>
> My understanding is that most systems work on the basis of 
> "reconciliation", which is a fancy way of saying that you're meant to 
> eyeball what you've entered and confirm it perfect, and after that it's 
> frozen. I preferred granting my users the superpowers of changing the past 
> at will, so I decided to encode that as allowing you to assert balances at 
> particular points in time - Beancount will always automatically complain is 
> a computed balance fails to match your bank's expectation. All the 
> assertions are always checked. Adding Balance directives often also allows 
> you to zero in on errors faster when you're debugging things (e.g. some 
> missing transaction which for whatever reason you failed to enter - it 
> happens).
>
> If your downloaded file contains a parseable balance field, I encourage 
> you to make that importer automatically generate a corresponding Balance 
> directive.
> Otherwise, you can manually insert one 

Re: Importing transactions from bank.

[Martin Blais]

On Sun, Apr 15, 2018 at 8:35 PM,  wrote:

> I've started playing with import scripts,the sample files provided in repo
> are very useful, just have couple of additional questions i was not able to
> figure out/not 100% sure about:
>

Really glad the example files are useful.


1 bean-extract does not write transactions in main beancount file
> (personal.beancount), it generates output into command line or
> tmp.beancount and we enter them manually, right?
>

Correct. You redirect its output to a temp file and using your editor,
cut/paste them to your main file wherever makes most sense for you (and
complete them, e.g., categorize the expenses).



>
> 2 when running  "bean-extract -e BEANCOUNT_FILE" what is the algorithm
> used to check duplication? Does it check only for same amounts? Or also
> takes into consideration additional clues (payee, date range, etc)?
>

Not so great, definitely could use some improvement:
https://bitbucket.org/blais/beancount/src/cd5fe599fdc80d47caa8a2618f5ee489ce88d101/beancount/ingest/similar.py
https://bitbucket.org/blais/beancount/issues/185/make-duplicate-detection-configureable


3 When filing files bean-file example.import ../Downloads/ -o documents/
> how are name and date generated?
> Seems it is picks up date from original file name (example
> UTrade20160215.csv), but when it's not in file name how ofxdownload.ofx
> turns to 2013-12-18.bofa.ofx
>

>From the file_name() and file_date() methods of your importer
implementation:
https://bitbucket.org/blais/beancount/src/cd5fe599fdc80d47caa8a2618f5ee489ce88d101/beancount/ingest/importer.py
If you use the example CSV importer it does what it can.
Typically you'd write your own and choose what you want it to be.



>
> 4 Main value of regression testing is to insure that new changes/additions
> to importers still work with old input files,
> It's not so useful to help importers work with new formats if format was
> changed by bank, right?
>

Both.
Those importer codes are typically the last thing you want to do on a
Saturday night, so they tend to be kludgey.
- When you need to support something new, you want to make sure you don't
break old functionality.
- When the Java programmers at your favorite bank change the output format
without warning you (or realize 10 years after that escaping the commas in
the CSV fields might have been a good idea), there's no faster way to test
a new / updated importer than to plot a newer file in the directory and run
regression tests


On Mon, Apr 16, 2018 at 8:30 AM,  wrote:

> 5 Also what's the right way to calculate balance
> ?
>

Ideally you don't calculate it. Ideally you find it somewhere and enter it.



>
> Chase have the following CSV structure,
> 
>
> Balance fields are included in checking CSV, so i guess i can use balance
> of the last transactions for balance.
> They are not included in credit card CSV however. Is there a way to
> calculate them?
>
> And how important "balance" in the first place? Cause for example this
> import script https://gist.github.com/mterwill/7fdcc573dc1aa158648aacd4e33
> 786e8#file-importers-chase-py does not have "balance".
>

They're optional.

The reason for Balance directives is to assert (and I mean that in the
computer programming sense as well) that your ledger's calculated balance
matches the institution's idea of what it should be. The picture perfect
idea of a Balance directive is a fuddy duddy accountant type wearing brown
socks staring at the bottom line of his paper statement carefully through
bifocals and manually copying the number to a Balance directive in
Beancount.

My understanding is that most systems work on the basis of
"reconciliation", which is a fancy way of saying that you're meant to
eyeball what you've entered and confirm it perfect, and after that it's
frozen. I preferred granting my users the superpowers of changing the past
at will, so I decided to encode that as allowing you to assert balances at
particular points in time - Beancount will always automatically complain is
a computed balance fails to match your bank's expectation. All the
assertions are always checked. Adding Balance directives often also allows
you to zero in on errors faster when you're debugging things (e.g. some
missing transaction which for whatever reason you failed to enter - it
happens).

If your downloaded file contains a parseable balance field, I encourage you
to make that importer automatically generate a corresponding Balance
directive.
Otherwise, you can manually insert one manually every now and then when you
log into that website (e.g. to make a payment) and copy the number on the
screen to the current date. The idea is to enter there a number which you
know (or assume) is correct.







>
>
> On Monday, April 9, 2018 at 4:28:30 PM UTC-4, Michael Droogleever wrote:
>>
>> You'll need to use something like this, 

Re: Importing transactions from bank.

[mployby]

5 Also what's the right way to calculate balance 
?

Chase have the following CSV structure, 


Balance fields are included in checking CSV, so i guess i can use balance 
of the last transactions for balance.
They are not included in credit card CSV however. Is there a way to 
calculate them?

And how important "balance" in the first place? Cause for example this 
import script 
https://gist.github.com/mterwill/7fdcc573dc1aa158648aacd4e33786e8#file-importers-chase-py
 
does not have "balance".

-- 
You received this message because you are subscribed to the Google Groups 
"Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to beancount+unsubscr...@googlegroups.com.
To post to this group, send email to beancount@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/beancount/a4d296ff-10a0-4a0b-9949-d3aac04fd7eb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Importing transactions from bank.

[mployby]

I've started playing with import scripts,the sample files provided in repo 
are very useful, just have couple of additional questions i was not able to 
figure out/not 100% sure about:

1 bean-extract does not write transactions in main beancount file 
(personal.beancount), it generates output into command line or 
tmp.beancount and we enter them manually, right?

2 when running  "bean-extract -e BEANCOUNT_FILE" what is the algorithm used 
to check duplication? Does it check only for same amounts? Or also takes 
into consideration additional clues (payee, date range, etc)?

3 When filing files bean-file example.import ../Downloads/ -o documents/ 
how are name and date generated?
Seems it is picks up date from original file name (example 
UTrade20160215.csv), but when it's not in file name how ofxdownload.ofx 
turns to 2013-12-18.bofa.ofx

4 Main value of regression testing is to insure that new changes/additions 
to importers still work with old input files,
It's not so useful to help importers work with new formats if format was 
changed by bank, right?

On Monday, April 9, 2018 at 4:28:30 PM UTC-4, Michael Droogleever wrote:
>
> You'll need to use something like this, 
> https://gist.github.com/mterwill/7fdcc573dc1aa158648aacd4e33786e8#file-importers-chase-py,
>  
> or make your own: 
> https://docs.google.com/document/d/11EwQdujzEo2cxqaF5PgxCEZXWfKKQCYSMfdJowp_1S8/edit
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to beancount+unsubscr...@googlegroups.com.
To post to this group, send email to beancount@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/beancount/57580546-5574-4004-9794-4c3dd6f99812%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Importing transactions from bank.

[Daniele Nicolodi]

On 4/9/18 2:28 PM, Michael Droogleever wrote:
> You'll need to use something like
> this, 
> https://gist.github.com/mterwill/7fdcc573dc1aa158648aacd4e33786e8#file-importers-chase-py,
> or make your
> own: 
> https://docs.google.com/document/d/11EwQdujzEo2cxqaF5PgxCEZXWfKKQCYSMfdJowp_1S8/edit

That's very nice!  I think it is time I rewrite some of the import
scripts I wrote for ledger to take advantage of the beancount facilities.

Cheers,
Dan

-- 
You received this message because you are subscribed to the Google Groups 
"Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to beancount+unsubscr...@googlegroups.com.
To post to this group, send email to beancount@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/beancount/a7144b77-d8bb-c435-40f4-5bed0d4d4293%40grinta.net.
For more options, visit https://groups.google.com/d/optout.

Re: Importing transactions from bank.

[mployby]

Thank you Michael, that's exactly what i was looking for!

On Monday, April 9, 2018 at 4:28:30 PM UTC-4, Michael Droogleever wrote:
>
> You'll need to use something like this, 
> https://gist.github.com/mterwill/7fdcc573dc1aa158648aacd4e33786e8#file-importers-chase-py,
>  
> or make your own: 
> https://docs.google.com/document/d/11EwQdujzEo2cxqaF5PgxCEZXWfKKQCYSMfdJowp_1S8/edit
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to beancount+unsubscr...@googlegroups.com.
To post to this group, send email to beancount@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/beancount/c5669998-fdb8-44d3-8681-e19a12b54407%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Importing transactions from bank.

[Michael Droogleever]

You'll need to use something like 
this, 
https://gist.github.com/mterwill/7fdcc573dc1aa158648aacd4e33786e8#file-importers-chase-py,
 
or make your 
own: 
https://docs.google.com/document/d/11EwQdujzEo2cxqaF5PgxCEZXWfKKQCYSMfdJowp_1S8/edit

-- 
You received this message because you are subscribed to the Google Groups 
"Beancount" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to beancount+unsubscr...@googlegroups.com.
To post to this group, send email to beancount@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/beancount/41f2c7ca-6aae-42d7-9e87-45080536e393%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.