Yes, it does! Turns out I wasn't paying close enough attention to the terminal type.
Using the VS Code Beancount extension, I have it set to start fava when opening my beancount file. That automated process uses a PowerShell terminal, not Command Prompt, and I was just using that window, instead of a new terminal. Making sure to use a Command Prompt terminal inside of VS Code, running the stmt command resulted in UTF-8 encoding. On Saturday, September 21, 2024 at 10:50:31 AM UTC-5 Red S wrote: > You’re welcome, and glad to hear! > > Found this > <https://stackoverflow.com/questions/63723478/default-output-file-in-vs-code-being-stored-in-utf-16-le-whereas-default-is-set> > > if it helps. > > > > On Saturday, September 21, 2024 at 8:06:20 AM UTC-7 Joshua Cabrera wrote: > >> Thank you so much! Everything seems to be working now, so I just have to >> tweak my.import and catch up on recording in my ledger. >> >> Once I'm caught up, I'll come back to automating the download and extract >> workflow. >> >> Regarding the encoding issue, my first guess was it's because I'm doing >> all this on Windows and using the Command Prompt shell, instead of bash. >> That theory appeared to be incorrect after creating some test files from >> the shell, which resulted in utf-8 encoding. >> >> Next, I tried creating test files using the integrated terminal in VS >> Code since technically that's where I've been running my commands, and it >> turns out that was the culprit. ¯\_(ツ)_/¯ >> >> On Friday, September 20, 2024 at 9:42:32 PM UTC-5 Red S wrote: >> >>> Thanks for the file! I can’t tell whether it was an artifact of how the >>> ofx file you attached got created or whether the original ofx file suffers >>> from this problem, but your ofx file is encoded in UTF-16 even though the >>> header claims it’s UTF-8. ofxparse fails because of this. Running this >>> allows it to work normally: >>> iconv -f UTF-16 -t UTF-8 fidelity_X12345678.ofx -o a.ofx ofx-summarize >>> a.ofx Total number of accounts: 1 >>> ---------------- >>> Account info: X12345678 fidelity.com >>> Statement info: 2024-07-01 00:00:00 -- 2024-07-31 00:00:00. Bal: [] >>> Types: {'cash', 'check', 'dep', 'payment'} >>> >>> 2024-07-30 04:00:00 cash THIS IS FUN Note -15.0000 >>> 2024-07-30 04:00:00 dep THIS IS FUN Note -15.0000 >>> 2024-07-29 04:00:00 check THIS IS FUN Note -15.0000 >>> 2024-07-26 04:00:00 cash THIS IS FUN Note -15.0000 >>> 2024-07-26 04:00:00 cash THIS IS FUN Note -15.0000 >>> >>> It's not the same stack trace as you got originally though. Either way, >>> try this? >>> >>> On Friday, September 20, 2024 at 6:38:29 PM UTC-7 Joshua Cabrera wrote: >>> >>>> Thanks, Red. The file itself looks okay after another look. >>>> >>>> I tried the new line trick, but it didn't change the results. >>>> >>>> The simple parse threw the same unpacking value error on my ofx while >>>> working on the sample ofx. >>>> >>>> I'm sharing a sanitized version of my ofx (replaced account number, >>>> transaction details, etc.) and my import file. >>>> >>>> In the meantime, I'll try a different Fidelity account and smaller >>>> date/transaction ranges. >>>> On Friday, September 20, 2024 at 5:09:18 PM UTC-5 Red S wrote: >>>> >>>>> This is uncommon, but typically because there was an issue downloading >>>>> the ofx. Usually because of authentication problems. Have you examined >>>>> the >>>>> ofx to see if it’s valid? >>>>> >>>>> Try this to make it more readable: sed 's/>/>\n/g' xyz.ofx >>>>> >>>>> >>>>> You should see something like this on the top: >>>>> <STATUS> <CODE> 0 </CODE> <SEVERITY> INFO </SEVERITY> <MESSAGE> >>>>> Success </MESSAGE> </STATUS> >>>>> >>>>> >>>>> If you see that, then try a really simple parse: >>>>> #!/usr/bin/env python3 import ofxparse ofx_file_path = >>>>> "your_ofx_file.ofx" with open(ofx_file_path, 'r') as ofx_file: ofx = >>>>> ofxparse.OfxParser.parse(ofx_file) print("Account type:", >>>>> ofx.account.account_type) print("Account ID:", >>>>> ofx.account.account_id) >>>>> >>>>> >>>>> On Friday, September 20, 2024 at 2:54:15 PM UTC-7 Joshua Cabrera wrote: >>>>> >>>>>> So, I've got my .ofx file and am not worried about the download task >>>>>> anymore. That's the good news. >>>>>> >>>>>> The bad news is now I'm stuck on the import task. >>>>>> bean-extract my.import fidelity_1234_july2024.ofx >>>>>> >>>>>> >>>>>> Results >>>>>> ERROR:root:Importer >>>>>> beancount_reds_importers.importers.fidelity.Importer.identify() raised >>>>>> an >>>>>> unexpected error: not enough values to unpack (expected 2, got 1) >>>>>> Traceback (most recent call last): >>>>>> File >>>>>> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\beancount\ingest\identify.py" >>>>>> , line 63, in find_imports >>>>>> matched = importer.identify(file) >>>>>> ^^^^^^^^^^^^^^^^^^^^^^^ >>>>>> File >>>>>> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\beancount_reds_importers\libreader\reader.py" >>>>>> , line 26, in identify >>>>>> self.initialize_reader(file) >>>>>> File >>>>>> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\beancount_reds_importers\libreader\ofxreader.py" >>>>>> , line 26, in initialize_reader >>>>>> self.ofx = self.read_file(file) >>>>>> ^^^^^^^^^^^^^^^^^^^^ >>>>>> File >>>>>> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\beancount_reds_importers\libreader\ofxreader.py" >>>>>> , line 57, in read_file >>>>>> return ofxparse.OfxParser.parse(fh) >>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>>> File >>>>>> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\ofxparse\ofxparse.py" >>>>>> , line 396, in parse >>>>>> ofx_file = OfxPreprocessedFile(file_handle) >>>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >>>>>> File >>>>>> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\ofxparse\ofxparse.py" >>>>>> , line 155, in __init__ >>>>>> super(OfxPreprocessedFile, self).__init__(fh) >>>>>> File >>>>>> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\ofxparse\ofxparse.py" >>>>>> , line 82, in __init__ >>>>>> self.read_headers() >>>>>> File >>>>>> "C:\Users\joshd\AppData\Local\Programs\Python\Python312\Lib\site-packages\ofxparse\ofxparse.py" >>>>>> , line 95, in read_headers >>>>>> header, value = line.split(six.b(":")) >>>>>> ^^^^^^^^^^^^^ >>>>>> ValueError: not enough values to unpack (expected 2, got 1) >>>>>> ;; -*- mode: beancount -*- >>>>>> >>>>>> >>>>>> If I run ofx-summarize, it throws the same error, so I found a sample >>>>>> .ofx online and tested ofx-summarize with it. >>>>>> >>>>>> And of course, the example works, so it looks like Fidelity is >>>>>> providing an incorrectly formatted file? I noticed it doesn't have a >>>>>> single >>>>>> line break until the end of the file. >>>>>> >>>>>> On Monday, September 16, 2024 at 11:01:14 PM UTC-5 Red S wrote: >>>>>> >>>>>>> Glad you got it working with ofxget. bean-download is just a >>>>>>> wrapper around ofxget or whatever command you have to parallelize >>>>>>> all your downloads. If you don’t have several accounts you’re using it >>>>>>> with, it’s of limited use. The built in template for fidelity is >>>>>>> here, and lets you store your password securely in pass so it’s not >>>>>>> stored in plaintext. You could simply paste your working ofxget command >>>>>>> into the bean-download config file to get it to work if the >>>>>>> parallelization is useful for you. Either way, that’s the last thing >>>>>>> I’d do >>>>>>> after I got everything else in the import workflow working first. >>>>>>> >>>>>>> And oh wow, I’d forgotten that Fidelity stopped >>>>>>> <https://www.reddit.com/r/fidelityinvestments/comments/17tpgfr/why_doesnt_fidelity_support_ofx_downloads_anymore/> >>>>>>> letting >>>>>>> you download ofx via their website, and only let you do it via direct >>>>>>> download. >>>>>>> >>>>>>> >>>>>>> On Monday, September 16, 2024 at 7:31:34 PM UTC-7 Joshua Cabrera >>>>>>> wrote: >>>>>>> >>>>>>>> Quick update: >>>>>>>> >>>>>>>> Installed ofxget and manage to get my ofxget.cfg updated using >>>>>>>> ofxget acctinfo. Not sure if this required for bean-download. >>>>>>>> >>>>>>>> Also downloaded a .ofx file using: >>>>>>>> ofxget stmt fidelity > c:\users\joshd\fidelity.ofx >>>>>>>> >>>>>>>> Struggling to my download.cfg and bean-download download working - >>>>>>>> keep getting: >>>>>>>> 1/1 sites failed: fidelity >>>>>>>> >>>>>>>> Will keep trying whenever I have time this week. >>>>>>>> On Sunday, September 15, 2024 at 9:24:33 PM UTC-5 Red S wrote: >>>>>>>> >>>>>>>>> My reason for using .csv over .ofx is simply familiarity and ease >>>>>>>>> of starting up. >>>>>>>>> >>>>>>>>> I have no experience with ofx and thought the learning curve would >>>>>>>>> be much steeper, and I already feel like I'm drowning a bit as a >>>>>>>>> layperson. >>>>>>>>> >>>>>>>>> I know it may seem like that, especially given ofx is not a human >>>>>>>>> readable format. But might I suggest giving that a shot first, and >>>>>>>>> you >>>>>>>>> might be pleasantly surprised. Because there is nothing to do, no >>>>>>>>> importer >>>>>>>>> to write, since it is a machine format with an official spec, unlike >>>>>>>>> csv. >>>>>>>>> The learning curve should be close to zero. There are other >>>>>>>>> benefits >>>>>>>>> <https://reds-rants.netlify.app/personal-finance/a-word-about-input-formats-use-ofx-when-you-can/> >>>>>>>>> >>>>>>>>> too. >>>>>>>>> >>>>>>>>> I'll look into the csvreader changes you mentioned to try making a >>>>>>>>> replacement for prepare_raw_columns() and also look further into ofx. >>>>>>>>> >>>>>>>>> If I were you, I’d definitely spend five minutes with ofx first. >>>>>>>>> And reds-importers ships with ofx-summarize which you can use to >>>>>>>>> peek inside and poke around an ofx file, which makes it a lot less >>>>>>>>> opaque >>>>>>>>> and easy on a layperson. >>>>>>>>> >>>>>>>>> Thanks for the help here and confirming I'm doing alright, and >>>>>>>>> also thank you for maintaining it! >>>>>>>>> >>>>>>>>> Happy to, and glad its helpful! >>>>>>>>> >>>>>>>>> >>>>>>>> -- You received this message because you are subscribed to the Google Groups "Beancount" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/9d51722d-04e9-4f9e-ae2e-e3a51d1620e9n%40googlegroups.com.
