A minimum of two transactions should suffice for smart_importer. More will increase prediction quality, but two should suffice. I can't tell what's happening at your end, but you're likely ending up with zero transactions for some reason. Run bean-query on the file you pass to "-f" of bean-extract.
beancount-reds-importers supports smart_importer out of the box for banking, that shouldn't be an issue AFAICT. On Wednesday, May 12, 2021 at 10:23:14 PM UTC-7 [email protected] wrote: > Thanks for suggestions @Patrick and Alan. My beancount file has about 64 > Asset accounts. It has about 41 expense accounts. I have only 2 months of > labelled banking transactions (about 42 transactions) all associated with > one bank account and various expense accounts. > > I had thought that some transactions were relatively deterministic (same $ > amount and same description like rent/mortgage) and I was under the > impression that only a few months of data are needed to get going. > > Perhaps I'll just go back to manually labelling data for now and trying > again later or after I see more posts/explanation of smart_importer. I'm > not well-versed enough with smart_importer to debug what is happening. > > On Thu, May 13, 2021 at 3:04 AM Alan H <[email protected]> wrote: > >> I get this error when there are insufficient entries in the journal to >> teach the smart_importer how to file new transactions. Specifically there >> are no matches for payees or narrations. >> >> Is that the case? Try adding a dummy transaction that matches the >> narration in the import file. >> >> Alan >> >> >> On Wednesday, May 12, 2021 at 12:24:55 PM UTC+1 [email protected] >> wrote: >> >>> Hm, actually that looks ok, it has the existing_entries on the >>> interface. But to be honest I'm not super familiar with how the apply hook >>> is hooking this in, so there might be an issue. >>> >>> Maybe someone more familiar with this can respond on that. >>> >>> >>> Otherwise if you could install smart_importer from git and then maybe >>> add a bit more debug output in >>> >>> hooks.py and predictor.py to make sure that the existing entries arrive, >>> this would give a better idea how to progress. >>> >>> >>> On 12.05.2021 13:17, [email protected] wrote: >>> >>> Thank you. I think that is it. >>> >>> I'm using reds-importers and I see >>> site-packages/beancount_reds_importers/libimport/banking.py and it has this >>> entry: >>> >>> def extract(self, file, existing_entries=None): >>> >>> I think this importer tool needs to be updated to support the >>> smart_importer. >>> >>> On Wednesday, May 12, 2021 at 11:11:37 PM UTC+12 [email protected] >>> wrote: >>> >>>> I just remembered something. The issue could be that the importer >>>> you're trying to use does not have the new interface and instead still >>>> uses >>>> the old (legacy) interface. >>>> >>>> the new one looks like this >>>> >>>> >>>> def extract(self, file, existing_entries): >>>> >>>> the old one looks like this >>>> >>>> def extract(self, file): >>>> >>>> >>>> Smart importer uses the existing_entries for training its model. >>>> >>>> >>>> Regards, >>>> >>>> Patrick >>>> >>>> >>>> >>>> >>>> On 12.05.2021 12:20, [email protected] wrote: >>>> >>>> Just checked and I got the same result. I can add some debugging code >>>> in the config file perhaps. I'm not very experienced with beancount or >>>> smart_importer so not sure what to look for. >>>> >>>> bean-extract -e journal/accounts.beancount jonathan_smart.import >>>> ~/staging/mydata.qfx > ~/staging/dud.txt >>>> >>>> gives 2 printouts of >>>> >>>> Cannot train the machine learning model because the training data is >>>> empty. >>>> >>>> Cannot train the machine learning model because the training data is >>>> empty. >>>> On Wednesday, May 12, 2021 at 7:15:19 PM UTC+12 [email protected] >>>> wrote: >>>> >>>>> Can you try -e instead of -f that's what I use >>>>> >>>>> >>>>> On May 12, 2021 8:31:36 AM GMT+02:00, "[email protected]" < >>>>> [email protected]> wrote: >>>>>> >>>>>> Thanks for the suggestion @Patrick. I just tried changing that but >>>>>> still doesn't work. I get the exact same behavior if I call it with an >>>>>> empty file....seems the -f option doesn't make bean-extract behave as >>>>>> expected for me. Here is my call: >>>>>> >>>>>> bean-extract -f journal/myledger.beancount jonathan_smart.import >>>>>> ~/staging/62090_818496_1013051ofxdl.qfx > ~/staging/dud.txt >>>>>> I get these messages: >>>>>> >>>>>> Cannot train the machine learning model because the training data is >>>>>> empty. >>>>>> >>>>>> Cannot train the machine learning model because the training data is >>>>>> empty. >>>>>> >>>>>> On Wednesday, May 12, 2021 at 5:31:25 PM UTC+12 [email protected] >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I think your setup looks good, the smart importer hook is in there >>>>>>> as otherwise you would not get the errors about not able to train. >>>>>>> >>>>>>> I think the issue is on your call >>>>>>> >>>>>>> >>>>>>> bean-extract jonathan_smart.import ~/staging/new_bank_data.qfx -f >>>>>>> journal/myledger.beancount > ~/staging/dud.txt >>>>>>> >>>>>>> >>>>>>> My guess is that the -f argument needs to come before you specify >>>>>>> the importconfig and the location, so >>>>>>> >>>>>>> >>>>>>> bean-extract -f journal/myledger.beancount jonathan_smart.import >>>>>>> ~/staging/new_bank_data.qfx > ~/staging/dud.txt >>>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Patrick >>>>>>> >>>>>>> On 12.05.2021 01:58, [email protected] wrote: >>>>>>> >>>>>>> Thanks for looking at this module even though you aren't using it! >>>>>>> >>>>>>> I followed the code that was further down on the readme page >>>>>>> <https://github.com/beancount/smart_importer> that describes how to >>>>>>> convert an existing importer. >>>>>>> >> >>>>>>> from your_custom_importer import MyBankImporter >>>>>>> from smart_importer import apply_hooks, PredictPayees, >>>>>>> PredictPostings >>>>>>> >>>>>>> my_bank_importer = MyBankImporter('whatever', 'config', 'is', >>>>>>> 'needed') >>>>>>> apply_hooks(my_bank_importer, [PredictPostings(), PredictPayees()]) >>>>>>> CONFIG = [ my_bank_importer, ] >>>>>>> >> >>>>>>> (my code looks just like this example) >>>>>>> >>>>>>> I had thought apply_hooks would operate on the importer so when I >>>>>>> call it in config I can just then call the hookified bank_importer. Is >>>>>>> this >>>>>>> note the case? >>>>>>> >>>>>>> On Wednesday, May 12, 2021 at 1:26:27 AM UTC+12 [email protected] >>>>>>> wrote: >>>>>>> >>>>>>>> * Disclaimer * I have never actually run smart importer. >>>>>>>> >>>>>>>> Looking at the README on GitHub for smart importer it looks like >>>>>>>> you need to use the return object of apply_hooks in your CONFIG list. >>>>>>>> >>>>>>>> CONFIG = [ >>>>>>>> apply_hooks(MyBankImporter(account='Assets:MyBank:MyAccount'), >>>>>>>> [PredictPostings()]) ] >>>>>>>> >>>>>>>> In your config you apply the hooks but are not using the returned >>>>>>>> object. >>>>>>>> >>>>>>>> Hope that helps. >>>>>>>> >>>>>>>> On Tuesday, 11 May 2021 at 04:06:33 UTC+1 [email protected] wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I'm trying to get smart_importer to work and not sure what I'm >>>>>>>>> doing wrong. >>>>>>>>> >>>>>>>>> *1*. I successfully have done all the required beancount setup >>>>>>>>> and created by own bank importer and ran it on two months of data. >>>>>>>>> *2.* I then manually labelled about 2 months of data from one of >>>>>>>>> my banks. >>>>>>>>> *3.* I installed smart_importer using "pip install smart_importer" >>>>>>>>> >>>>>>>>> (base) MacBook-Air:beandata jonathan$ pip show smart_importer >>>>>>>>> >>>>>>>>> Name: smart-importer >>>>>>>>> >>>>>>>>> Version: 0.3 >>>>>>>>> >>>>>>>>> Summary: Augment Beancount importers with machine learning >>>>>>>>> functionality. >>>>>>>>> >>>>>>>>> Home-page: https://github.com/beancount/smart_importer >>>>>>>>> >>>>>>>>> Author: Johannes Harms >>>>>>>>> >>>>>>>>> Author-email: UNKNOWN >>>>>>>>> >>>>>>>>> License: MIT >>>>>>>>> >>>>>>>>> Location: >>>>>>>>> /Users/jonathan/opt/miniconda3/lib/python3.8/site-packages >>>>>>>>> >>>>>>>>> Requires: scikit-learn, beancount, numpy, scipy >>>>>>>>> >>>>>>>>> *4.* I created a new config file I called Jonathan_smart.import >>>>>>>>> >>>>>>>>> >>>>>>>>> base) MacBook-Air:beandata jonathan$ more jonathan_smart.import >>>>>>>>> >>>>>>>>> #!/usr/bin/env python3 >>>>>>>>> >>>>>>>>> """Import configuration.""" >>>>>>>>> >>>>>>>>> >>>>>>>>> import sys >>>>>>>>> >>>>>>>>> from os import path >>>>>>>>> >>>>>>>>> >>>>>>>>> sys.path.insert(0, path.join(path.dirname(__file__))) >>>>>>>>> >>>>>>>>> >>>>>>>>> from beancount_reds_importers import vanguard >>>>>>>>> >>>>>>>>> from myimporters.bfsfcu import bfsfcu_bank >>>>>>>>> >>>>>>>>> from myimporters.anz import anz_bank >>>>>>>>> >>>>>>>>> from fund_info import * >>>>>>>>> >>>>>>>>> from smart_importer import apply_hooks, PredictPayees, >>>>>>>>> PredictPostings >>>>>>>>> >>>>>>>>> >>>>>>>>> myBank_smart_importer =my_bank.Importer({ >>>>>>>>> >>>>>>>>> 'main_account' : 'Assets:US:Banks:Checking:myBank', >>>>>>>>> >>>>>>>>> 'account_number' : ''xxx'', >>>>>>>>> >>>>>>>>> 'transfer' : >>>>>>>>> 'Assets:US:Zero-Sum-Accounts:Transfers:Bank-Account', >>>>>>>>> >>>>>>>>> 'income' : 'Income:US:Interest:myBank', >>>>>>>>> >>>>>>>>> 'fees' : 'Expenses:US:Bank-Fees:myBank', >>>>>>>>> >>>>>>>>> 'rounding_error' : 'Equity:US:Rounding-Errors:Imports', >>>>>>>>> >>>>>>>>> }) >>>>>>>>> >>>>>>>>> >>>>>>>>> apply_hooks(myBank_smart_importer, [PredictPayees(), >>>>>>>>> PredictPostings()]) >>>>>>>>> >>>>>>>>> CONFIG = [myBank_smart_importer, ...(other importers)] >>>>>>>>> >>>>>>>>> >>>>>>>>> *5*. I was following the README documentation that said write >>>>>>>>> bean-extract -f to invoke it on existing data. So I tried the >>>>>>>>> following.* >>>>>>>>> Is this right?* >>>>>>>>> >>>>>>>>> bean-extract jonathan_smart.import ~/staging/new_bank_data.qfx -f >>>>>>>>> journal/myledger.beancount > ~/staging/dud.txt >>>>>>>>> >>>>>>>>> Cannot train the machine learning model because the training data >>>>>>>>> is empty. >>>>>>>>> >>>>>>>>> Cannot train the machine learning model because the training data >>>>>>>>> is empty. >>>>>>>>> >>>>>>>>> >>>>>>>>> The output is just like the normal output without all the >>>>>>>>> smart_importer stuff. Seems I'm doing something wrong as the >>>>>>>>> staging/dud.txt doesn't have any predictions. >>>>>>>>> >>>>>>>>> >>>>>>>>> Appreciate any assistance on this! >>>>>>>>> >>>>>>>>> >>>>>>>>> thanks, >>>>>>>>> >>>>>>>>> Jonathan >>>>>>>>> >>>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Beancount" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/beancount/820ef641-8178-47d1-9e97-afbc709e6a83n%40googlegroups.com >>>>>>> >>>>>>> <https://groups.google.com/d/msgid/beancount/820ef641-8178-47d1-9e97-afbc709e6a83n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Beancount" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/beancount/fe28577c-8220-49cd-b976-40ef9f0b6a91n%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/beancount/fe28577c-8220-49cd-b976-40ef9f0b6a91n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Beancount" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/beancount/6248ca60-16fa-4ad0-88b5-1c4bb91f9feen%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/beancount/6248ca60-16fa-4ad0-88b5-1c4bb91f9feen%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Beancount" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/beancount/2b800e6d-fb0c-4b78-bde3-477eee6f9e7en%40googlegroups.com >> >> <https://groups.google.com/d/msgid/beancount/2b800e6d-fb0c-4b78-bde3-477eee6f9e7en%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Beancount" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/f1e3ce25-e842-45b4-bb28-4f3737a3cb9en%40googlegroups.com.
