Thanks Red. bean-query works fine on my input file which now has >1000 transactions .
Ready with 1344 directives (2266 postings in 1133 transactions). beancount> I still get the error. I'm not sure what is causing and not sure how to debug it. The only other issue I recall seeing was some error with fund_info or something in getting prices but I thought it was an unrelated issue. Do you or does anyone have some suggestions on where/how to debug. E.g. I should print some variables to STDOUT at such and such point inside smart_importer code or inside bean-extract. thanks, Jonathan On Mon, May 17, 2021 at 9:34 PM [email protected] <[email protected]> wrote: > A minimum of two transactions should suffice for smart_importer. More will > increase prediction quality, but two should suffice. I can't tell what's > happening at your end, but you're likely ending up with zero transactions > for some reason. Run bean-query on the file you pass to "-f" of > bean-extract. > > beancount-reds-importers supports smart_importer out of the box for > banking, that shouldn't be an issue AFAICT. > > > > On Wednesday, May 12, 2021 at 10:23:14 PM UTC-7 [email protected] wrote: > >> Thanks for suggestions @Patrick and Alan. My beancount file has about 64 >> Asset accounts. It has about 41 expense accounts. I have only 2 months of >> labelled banking transactions (about 42 transactions) all associated with >> one bank account and various expense accounts. >> >> I had thought that some transactions were relatively deterministic (same >> $ amount and same description like rent/mortgage) and I was under the >> impression that only a few months of data are needed to get going. >> >> Perhaps I'll just go back to manually labelling data for now and trying >> again later or after I see more posts/explanation of smart_importer. I'm >> not well-versed enough with smart_importer to debug what is happening. >> >> On Thu, May 13, 2021 at 3:04 AM Alan H <[email protected]> wrote: >> >>> I get this error when there are insufficient entries in the journal to >>> teach the smart_importer how to file new transactions. Specifically there >>> are no matches for payees or narrations. >>> >>> Is that the case? Try adding a dummy transaction that matches the >>> narration in the import file. >>> >>> Alan >>> >>> >>> On Wednesday, May 12, 2021 at 12:24:55 PM UTC+1 [email protected] >>> wrote: >>> >>>> Hm, actually that looks ok, it has the existing_entries on the >>>> interface. But to be honest I'm not super familiar with how the apply hook >>>> is hooking this in, so there might be an issue. >>>> >>>> Maybe someone more familiar with this can respond on that. >>>> >>>> >>>> Otherwise if you could install smart_importer from git and then maybe >>>> add a bit more debug output in >>>> >>>> hooks.py and predictor.py to make sure that the existing entries >>>> arrive, this would give a better idea how to progress. >>>> >>>> >>>> On 12.05.2021 13:17, [email protected] wrote: >>>> >>>> Thank you. I think that is it. >>>> >>>> I'm using reds-importers and I see >>>> site-packages/beancount_reds_importers/libimport/banking.py and it has this >>>> entry: >>>> >>>> def extract(self, file, existing_entries=None): >>>> >>>> I think this importer tool needs to be updated to support the >>>> smart_importer. >>>> >>>> On Wednesday, May 12, 2021 at 11:11:37 PM UTC+12 [email protected] >>>> wrote: >>>> >>>>> I just remembered something. The issue could be that the importer >>>>> you're trying to use does not have the new interface and instead still >>>>> uses >>>>> the old (legacy) interface. >>>>> >>>>> the new one looks like this >>>>> >>>>> >>>>> def extract(self, file, existing_entries): >>>>> >>>>> the old one looks like this >>>>> >>>>> def extract(self, file): >>>>> >>>>> >>>>> Smart importer uses the existing_entries for training its model. >>>>> >>>>> >>>>> Regards, >>>>> >>>>> Patrick >>>>> >>>>> >>>>> >>>>> >>>>> On 12.05.2021 12:20, [email protected] wrote: >>>>> >>>>> Just checked and I got the same result. I can add some debugging code >>>>> in the config file perhaps. I'm not very experienced with beancount or >>>>> smart_importer so not sure what to look for. >>>>> >>>>> bean-extract -e journal/accounts.beancount jonathan_smart.import >>>>> ~/staging/mydata.qfx > ~/staging/dud.txt >>>>> >>>>> gives 2 printouts of >>>>> >>>>> Cannot train the machine learning model because the training data is >>>>> empty. >>>>> >>>>> Cannot train the machine learning model because the training data is >>>>> empty. >>>>> On Wednesday, May 12, 2021 at 7:15:19 PM UTC+12 [email protected] >>>>> wrote: >>>>> >>>>>> Can you try -e instead of -f that's what I use >>>>>> >>>>>> >>>>>> On May 12, 2021 8:31:36 AM GMT+02:00, "[email protected]" < >>>>>> [email protected]> wrote: >>>>>>> >>>>>>> Thanks for the suggestion @Patrick. I just tried changing that but >>>>>>> still doesn't work. I get the exact same behavior if I call it with an >>>>>>> empty file....seems the -f option doesn't make bean-extract behave as >>>>>>> expected for me. Here is my call: >>>>>>> >>>>>>> bean-extract -f journal/myledger.beancount jonathan_smart.import >>>>>>> ~/staging/62090_818496_1013051ofxdl.qfx > ~/staging/dud.txt >>>>>>> I get these messages: >>>>>>> >>>>>>> Cannot train the machine learning model because the training data is >>>>>>> empty. >>>>>>> >>>>>>> Cannot train the machine learning model because the training data is >>>>>>> empty. >>>>>>> >>>>>>> On Wednesday, May 12, 2021 at 5:31:25 PM UTC+12 [email protected] >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I think your setup looks good, the smart importer hook is in there >>>>>>>> as otherwise you would not get the errors about not able to train. >>>>>>>> >>>>>>>> I think the issue is on your call >>>>>>>> >>>>>>>> >>>>>>>> bean-extract jonathan_smart.import ~/staging/new_bank_data.qfx -f >>>>>>>> journal/myledger.beancount > ~/staging/dud.txt >>>>>>>> >>>>>>>> >>>>>>>> My guess is that the -f argument needs to come before you specify >>>>>>>> the importconfig and the location, so >>>>>>>> >>>>>>>> >>>>>>>> bean-extract -f journal/myledger.beancount jonathan_smart.import >>>>>>>> ~/staging/new_bank_data.qfx > ~/staging/dud.txt >>>>>>>> >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Patrick >>>>>>>> >>>>>>>> On 12.05.2021 01:58, [email protected] wrote: >>>>>>>> >>>>>>>> Thanks for looking at this module even though you aren't using it! >>>>>>>> >>>>>>>> I followed the code that was further down on the readme page >>>>>>>> <https://github.com/beancount/smart_importer> that describes how >>>>>>>> to convert an existing importer. >>>>>>>> >> >>>>>>>> from your_custom_importer import MyBankImporter >>>>>>>> from smart_importer import apply_hooks, PredictPayees, >>>>>>>> PredictPostings >>>>>>>> >>>>>>>> my_bank_importer = MyBankImporter('whatever', 'config', 'is', >>>>>>>> 'needed') >>>>>>>> apply_hooks(my_bank_importer, [PredictPostings(), PredictPayees()]) >>>>>>>> CONFIG = [ my_bank_importer, ] >>>>>>>> >> >>>>>>>> (my code looks just like this example) >>>>>>>> >>>>>>>> I had thought apply_hooks would operate on the importer so when I >>>>>>>> call it in config I can just then call the hookified bank_importer. Is >>>>>>>> this >>>>>>>> note the case? >>>>>>>> >>>>>>>> On Wednesday, May 12, 2021 at 1:26:27 AM UTC+12 [email protected] >>>>>>>> wrote: >>>>>>>> >>>>>>>>> * Disclaimer * I have never actually run smart importer. >>>>>>>>> >>>>>>>>> Looking at the README on GitHub for smart importer it looks like >>>>>>>>> you need to use the return object of apply_hooks in your CONFIG list. >>>>>>>>> >>>>>>>>> CONFIG = [ >>>>>>>>> apply_hooks(MyBankImporter(account='Assets:MyBank:MyAccount'), >>>>>>>>> [PredictPostings()]) ] >>>>>>>>> >>>>>>>>> In your config you apply the hooks but are not using the returned >>>>>>>>> object. >>>>>>>>> >>>>>>>>> Hope that helps. >>>>>>>>> >>>>>>>>> On Tuesday, 11 May 2021 at 04:06:33 UTC+1 [email protected] wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I'm trying to get smart_importer to work and not sure what I'm >>>>>>>>>> doing wrong. >>>>>>>>>> >>>>>>>>>> *1*. I successfully have done all the required beancount setup >>>>>>>>>> and created by own bank importer and ran it on two months of data. >>>>>>>>>> *2.* I then manually labelled about 2 months of data from one of >>>>>>>>>> my banks. >>>>>>>>>> *3.* I installed smart_importer using "pip install >>>>>>>>>> smart_importer" >>>>>>>>>> >>>>>>>>>> (base) MacBook-Air:beandata jonathan$ pip show smart_importer >>>>>>>>>> >>>>>>>>>> Name: smart-importer >>>>>>>>>> >>>>>>>>>> Version: 0.3 >>>>>>>>>> >>>>>>>>>> Summary: Augment Beancount importers with machine learning >>>>>>>>>> functionality. >>>>>>>>>> >>>>>>>>>> Home-page: https://github.com/beancount/smart_importer >>>>>>>>>> >>>>>>>>>> Author: Johannes Harms >>>>>>>>>> >>>>>>>>>> Author-email: UNKNOWN >>>>>>>>>> >>>>>>>>>> License: MIT >>>>>>>>>> >>>>>>>>>> Location: >>>>>>>>>> /Users/jonathan/opt/miniconda3/lib/python3.8/site-packages >>>>>>>>>> >>>>>>>>>> Requires: scikit-learn, beancount, numpy, scipy >>>>>>>>>> >>>>>>>>>> *4.* I created a new config file I called Jonathan_smart.import >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> base) MacBook-Air:beandata jonathan$ more jonathan_smart.import >>>>>>>>>> >>>>>>>>>> #!/usr/bin/env python3 >>>>>>>>>> >>>>>>>>>> """Import configuration.""" >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> import sys >>>>>>>>>> >>>>>>>>>> from os import path >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> sys.path.insert(0, path.join(path.dirname(__file__))) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> from beancount_reds_importers import vanguard >>>>>>>>>> >>>>>>>>>> from myimporters.bfsfcu import bfsfcu_bank >>>>>>>>>> >>>>>>>>>> from myimporters.anz import anz_bank >>>>>>>>>> >>>>>>>>>> from fund_info import * >>>>>>>>>> >>>>>>>>>> from smart_importer import apply_hooks, PredictPayees, >>>>>>>>>> PredictPostings >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> myBank_smart_importer =my_bank.Importer({ >>>>>>>>>> >>>>>>>>>> 'main_account' : 'Assets:US:Banks:Checking:myBank', >>>>>>>>>> >>>>>>>>>> 'account_number' : ''xxx'', >>>>>>>>>> >>>>>>>>>> 'transfer' : >>>>>>>>>> 'Assets:US:Zero-Sum-Accounts:Transfers:Bank-Account', >>>>>>>>>> >>>>>>>>>> 'income' : 'Income:US:Interest:myBank', >>>>>>>>>> >>>>>>>>>> 'fees' : 'Expenses:US:Bank-Fees:myBank', >>>>>>>>>> >>>>>>>>>> 'rounding_error' : 'Equity:US:Rounding-Errors:Imports', >>>>>>>>>> >>>>>>>>>> }) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> apply_hooks(myBank_smart_importer, [PredictPayees(), >>>>>>>>>> PredictPostings()]) >>>>>>>>>> >>>>>>>>>> CONFIG = [myBank_smart_importer, ...(other importers)] >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> *5*. I was following the README documentation that said write >>>>>>>>>> bean-extract -f to invoke it on existing data. So I tried the >>>>>>>>>> following.* >>>>>>>>>> Is this right?* >>>>>>>>>> >>>>>>>>>> bean-extract jonathan_smart.import ~/staging/new_bank_data.qfx -f >>>>>>>>>> journal/myledger.beancount > ~/staging/dud.txt >>>>>>>>>> >>>>>>>>>> Cannot train the machine learning model because the training data >>>>>>>>>> is empty. >>>>>>>>>> >>>>>>>>>> Cannot train the machine learning model because the training data >>>>>>>>>> is empty. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> The output is just like the normal output without all the >>>>>>>>>> smart_importer stuff. Seems I'm doing something wrong as the >>>>>>>>>> staging/dud.txt doesn't have any predictions. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Appreciate any assistance on this! >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> thanks, >>>>>>>>>> >>>>>>>>>> Jonathan >>>>>>>>>> >>>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Beancount" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/beancount/820ef641-8178-47d1-9e97-afbc709e6a83n%40googlegroups.com >>>>>>>> <https://groups.google.com/d/msgid/beancount/820ef641-8178-47d1-9e97-afbc709e6a83n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Beancount" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/beancount/fe28577c-8220-49cd-b976-40ef9f0b6a91n%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/beancount/fe28577c-8220-49cd-b976-40ef9f0b6a91n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Beancount" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/beancount/6248ca60-16fa-4ad0-88b5-1c4bb91f9feen%40googlegroups.com >>>> <https://groups.google.com/d/msgid/beancount/6248ca60-16fa-4ad0-88b5-1c4bb91f9feen%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Beancount" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/beancount/2b800e6d-fb0c-4b78-bde3-477eee6f9e7en%40googlegroups.com >>> <https://groups.google.com/d/msgid/beancount/2b800e6d-fb0c-4b78-bde3-477eee6f9e7en%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "Beancount" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/beancount/f1e3ce25-e842-45b4-bb28-4f3737a3cb9en%40googlegroups.com > <https://groups.google.com/d/msgid/beancount/f1e3ce25-e842-45b4-bb28-4f3737a3cb9en%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Beancount" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CANUAcYdz12pG%2BPyxiBdn5-L14TtSztkJ8A%2BQ8Fwfd753vN0-tg%40mail.gmail.com.
