Thanks for suggestions @Patrick and Alan. My beancount file has about 64 Asset accounts. It has about 41 expense accounts. I have only 2 months of labelled banking transactions (about 42 transactions) all associated with one bank account and various expense accounts.
I had thought that some transactions were relatively deterministic (same $ amount and same description like rent/mortgage) and I was under the impression that only a few months of data are needed to get going. Perhaps I'll just go back to manually labelling data for now and trying again later or after I see more posts/explanation of smart_importer. I'm not well-versed enough with smart_importer to debug what is happening. On Thu, May 13, 2021 at 3:04 AM Alan H <[email protected]> wrote: > I get this error when there are insufficient entries in the journal to > teach the smart_importer how to file new transactions. Specifically there > are no matches for payees or narrations. > > Is that the case? Try adding a dummy transaction that matches the > narration in the import file. > > Alan > > > On Wednesday, May 12, 2021 at 12:24:55 PM UTC+1 [email protected] wrote: > >> Hm, actually that looks ok, it has the existing_entries on the interface. >> But to be honest I'm not super familiar with how the apply hook is hooking >> this in, so there might be an issue. >> >> Maybe someone more familiar with this can respond on that. >> >> >> Otherwise if you could install smart_importer from git and then maybe add >> a bit more debug output in >> >> hooks.py and predictor.py to make sure that the existing entries arrive, >> this would give a better idea how to progress. >> >> >> On 12.05.2021 13:17, [email protected] wrote: >> >> Thank you. I think that is it. >> >> I'm using reds-importers and I see >> site-packages/beancount_reds_importers/libimport/banking.py and it has this >> entry: >> >> def extract(self, file, existing_entries=None): >> >> I think this importer tool needs to be updated to support the >> smart_importer. >> >> On Wednesday, May 12, 2021 at 11:11:37 PM UTC+12 [email protected] >> wrote: >> >>> I just remembered something. The issue could be that the importer you're >>> trying to use does not have the new interface and instead still uses the >>> old (legacy) interface. >>> >>> the new one looks like this >>> >>> >>> def extract(self, file, existing_entries): >>> >>> the old one looks like this >>> >>> def extract(self, file): >>> >>> >>> Smart importer uses the existing_entries for training its model. >>> >>> >>> Regards, >>> >>> Patrick >>> >>> >>> >>> >>> On 12.05.2021 12:20, [email protected] wrote: >>> >>> Just checked and I got the same result. I can add some debugging code in >>> the config file perhaps. I'm not very experienced with beancount or >>> smart_importer so not sure what to look for. >>> >>> bean-extract -e journal/accounts.beancount jonathan_smart.import >>> ~/staging/mydata.qfx > ~/staging/dud.txt >>> >>> gives 2 printouts of >>> >>> Cannot train the machine learning model because the training data is >>> empty. >>> >>> Cannot train the machine learning model because the training data is >>> empty. >>> On Wednesday, May 12, 2021 at 7:15:19 PM UTC+12 [email protected] >>> wrote: >>> >>>> Can you try -e instead of -f that's what I use >>>> >>>> >>>> On May 12, 2021 8:31:36 AM GMT+02:00, "[email protected]" < >>>> [email protected]> wrote: >>>>> >>>>> Thanks for the suggestion @Patrick. I just tried changing that but >>>>> still doesn't work. I get the exact same behavior if I call it with an >>>>> empty file....seems the -f option doesn't make bean-extract behave as >>>>> expected for me. Here is my call: >>>>> >>>>> bean-extract -f journal/myledger.beancount jonathan_smart.import >>>>> ~/staging/62090_818496_1013051ofxdl.qfx > ~/staging/dud.txt >>>>> I get these messages: >>>>> >>>>> Cannot train the machine learning model because the training data is >>>>> empty. >>>>> >>>>> Cannot train the machine learning model because the training data is >>>>> empty. >>>>> >>>>> On Wednesday, May 12, 2021 at 5:31:25 PM UTC+12 [email protected] >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I think your setup looks good, the smart importer hook is in there as >>>>>> otherwise you would not get the errors about not able to train. >>>>>> >>>>>> I think the issue is on your call >>>>>> >>>>>> >>>>>> bean-extract jonathan_smart.import ~/staging/new_bank_data.qfx -f >>>>>> journal/myledger.beancount > ~/staging/dud.txt >>>>>> >>>>>> >>>>>> My guess is that the -f argument needs to come before you specify the >>>>>> importconfig and the location, so >>>>>> >>>>>> >>>>>> bean-extract -f journal/myledger.beancount jonathan_smart.import >>>>>> ~/staging/new_bank_data.qfx > ~/staging/dud.txt >>>>>> >>>>>> >>>>>> Regards, >>>>>> >>>>>> Patrick >>>>>> >>>>>> On 12.05.2021 01:58, [email protected] wrote: >>>>>> >>>>>> Thanks for looking at this module even though you aren't using it! >>>>>> >>>>>> I followed the code that was further down on the readme page >>>>>> <https://github.com/beancount/smart_importer> that describes how to >>>>>> convert an existing importer. >>>>>> >> >>>>>> from your_custom_importer import MyBankImporter >>>>>> from smart_importer import apply_hooks, PredictPayees, >>>>>> PredictPostings >>>>>> >>>>>> my_bank_importer = MyBankImporter('whatever', 'config', 'is', >>>>>> 'needed') >>>>>> apply_hooks(my_bank_importer, [PredictPostings(), PredictPayees()]) >>>>>> CONFIG = [ my_bank_importer, ] >>>>>> >> >>>>>> (my code looks just like this example) >>>>>> >>>>>> I had thought apply_hooks would operate on the importer so when I >>>>>> call it in config I can just then call the hookified bank_importer. Is >>>>>> this >>>>>> note the case? >>>>>> >>>>>> On Wednesday, May 12, 2021 at 1:26:27 AM UTC+12 [email protected] >>>>>> wrote: >>>>>> >>>>>>> * Disclaimer * I have never actually run smart importer. >>>>>>> >>>>>>> Looking at the README on GitHub for smart importer it looks like you >>>>>>> need to use the return object of apply_hooks in your CONFIG list. >>>>>>> >>>>>>> CONFIG = [ >>>>>>> apply_hooks(MyBankImporter(account='Assets:MyBank:MyAccount'), >>>>>>> [PredictPostings()]) ] >>>>>>> >>>>>>> In your config you apply the hooks but are not using the returned >>>>>>> object. >>>>>>> >>>>>>> Hope that helps. >>>>>>> >>>>>>> On Tuesday, 11 May 2021 at 04:06:33 UTC+1 [email protected] wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I'm trying to get smart_importer to work and not sure what I'm >>>>>>>> doing wrong. >>>>>>>> >>>>>>>> *1*. I successfully have done all the required beancount setup and >>>>>>>> created by own bank importer and ran it on two months of data. >>>>>>>> *2.* I then manually labelled about 2 months of data from one of >>>>>>>> my banks. >>>>>>>> *3.* I installed smart_importer using "pip install smart_importer" >>>>>>>> >>>>>>>> (base) MacBook-Air:beandata jonathan$ pip show smart_importer >>>>>>>> >>>>>>>> Name: smart-importer >>>>>>>> >>>>>>>> Version: 0.3 >>>>>>>> >>>>>>>> Summary: Augment Beancount importers with machine learning >>>>>>>> functionality. >>>>>>>> >>>>>>>> Home-page: https://github.com/beancount/smart_importer >>>>>>>> >>>>>>>> Author: Johannes Harms >>>>>>>> >>>>>>>> Author-email: UNKNOWN >>>>>>>> >>>>>>>> License: MIT >>>>>>>> >>>>>>>> Location: /Users/jonathan/opt/miniconda3/lib/python3.8/site-packages >>>>>>>> >>>>>>>> Requires: scikit-learn, beancount, numpy, scipy >>>>>>>> >>>>>>>> *4.* I created a new config file I called Jonathan_smart.import >>>>>>>> >>>>>>>> >>>>>>>> base) MacBook-Air:beandata jonathan$ more jonathan_smart.import >>>>>>>> >>>>>>>> #!/usr/bin/env python3 >>>>>>>> >>>>>>>> """Import configuration.""" >>>>>>>> >>>>>>>> >>>>>>>> import sys >>>>>>>> >>>>>>>> from os import path >>>>>>>> >>>>>>>> >>>>>>>> sys.path.insert(0, path.join(path.dirname(__file__))) >>>>>>>> >>>>>>>> >>>>>>>> from beancount_reds_importers import vanguard >>>>>>>> >>>>>>>> from myimporters.bfsfcu import bfsfcu_bank >>>>>>>> >>>>>>>> from myimporters.anz import anz_bank >>>>>>>> >>>>>>>> from fund_info import * >>>>>>>> >>>>>>>> from smart_importer import apply_hooks, PredictPayees, >>>>>>>> PredictPostings >>>>>>>> >>>>>>>> >>>>>>>> myBank_smart_importer =my_bank.Importer({ >>>>>>>> >>>>>>>> 'main_account' : 'Assets:US:Banks:Checking:myBank', >>>>>>>> >>>>>>>> 'account_number' : ''xxx'', >>>>>>>> >>>>>>>> 'transfer' : >>>>>>>> 'Assets:US:Zero-Sum-Accounts:Transfers:Bank-Account', >>>>>>>> >>>>>>>> 'income' : 'Income:US:Interest:myBank', >>>>>>>> >>>>>>>> 'fees' : 'Expenses:US:Bank-Fees:myBank', >>>>>>>> >>>>>>>> 'rounding_error' : 'Equity:US:Rounding-Errors:Imports', >>>>>>>> >>>>>>>> }) >>>>>>>> >>>>>>>> >>>>>>>> apply_hooks(myBank_smart_importer, [PredictPayees(), >>>>>>>> PredictPostings()]) >>>>>>>> >>>>>>>> CONFIG = [myBank_smart_importer, ...(other importers)] >>>>>>>> >>>>>>>> >>>>>>>> *5*. I was following the README documentation that said write >>>>>>>> bean-extract -f to invoke it on existing data. So I tried the >>>>>>>> following.* >>>>>>>> Is this right?* >>>>>>>> >>>>>>>> bean-extract jonathan_smart.import ~/staging/new_bank_data.qfx -f >>>>>>>> journal/myledger.beancount > ~/staging/dud.txt >>>>>>>> >>>>>>>> Cannot train the machine learning model because the training data >>>>>>>> is empty. >>>>>>>> >>>>>>>> Cannot train the machine learning model because the training data >>>>>>>> is empty. >>>>>>>> >>>>>>>> >>>>>>>> The output is just like the normal output without all the >>>>>>>> smart_importer stuff. Seems I'm doing something wrong as the >>>>>>>> staging/dud.txt doesn't have any predictions. >>>>>>>> >>>>>>>> >>>>>>>> Appreciate any assistance on this! >>>>>>>> >>>>>>>> >>>>>>>> thanks, >>>>>>>> >>>>>>>> Jonathan >>>>>>>> >>>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Beancount" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/beancount/820ef641-8178-47d1-9e97-afbc709e6a83n%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/beancount/820ef641-8178-47d1-9e97-afbc709e6a83n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Beancount" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/beancount/fe28577c-8220-49cd-b976-40ef9f0b6a91n%40googlegroups.com >>> <https://groups.google.com/d/msgid/beancount/fe28577c-8220-49cd-b976-40ef9f0b6a91n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Beancount" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/beancount/6248ca60-16fa-4ad0-88b5-1c4bb91f9feen%40googlegroups.com >> <https://groups.google.com/d/msgid/beancount/6248ca60-16fa-4ad0-88b5-1c4bb91f9feen%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> -- > You received this message because you are subscribed to the Google Groups > "Beancount" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/beancount/2b800e6d-fb0c-4b78-bde3-477eee6f9e7en%40googlegroups.com > <https://groups.google.com/d/msgid/beancount/2b800e6d-fb0c-4b78-bde3-477eee6f9e7en%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Beancount" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CANUAcYf9NwixkDJ8jnMpOTwKMS-Ws%3DUQDLqMaGr1rU97ZweOTg%40mail.gmail.com.
