Re: [GNC] OFX import sometimes loses transaction memos
On Thu, Jan 17, 2019, 1:15 AM Geert Janssens Op donderdag 17 januari 2019 06:33:25 CET schreef David Carlson: > > I think that I need to spend more time examining the transactions that > had > > their memos dropped during import to see if I can find a threshold string > > length, then file a bug report asking for a short term fix truncating > > strings rather than dropping them, and a long term fix accepting up to > the > > 256 character max and truncating after that. > > > > David C > > I don't think there should be one. Some time back I submitted a number of > patches to handle long strings in the ofx library. IIRC it internally uses > a > fixed-length buffer to read parts of the ofx file. And I believe that > buffer > itself was already 1024 characters long. So at least for lines (including > tags) of that size there should not be a problem. Next if the buffer gets > full > before a full tag contents could be read, it simply fills that buffer a > second > time and appends that contents to the same first string. So in principle > you > can have text messages of unlimited size. > > Now there were bugs in that area, and my patches were partly to resolve > those. > IIRC one issue was if the ofx file didn't have newlines between tags that > buffer handling would not always work as it should. > > Note those changes have gone into a recent libofx release. As it's been a > while I don't know whether gnucash 2.6.x was ever packaged for Windows > with > this more recent release though I'm sure gnucash 3.x is. > > And of course there may be other bugs that weren't reported yet. I'm not > using > ofx myself. > > Lastly it's probably still interesting to run ofxdump as it can help > determine > whether the issue already starts in the libofx library or rather that > gnucash > messes up the data it received from libofx. > > Regards, > > Geert > > > I was not aware that release 3.4 might behave better. I have not tried it yet for other reasons. This is good news. I think that i mentioned that my test files do not have carriage returns, line feeds or newlines, instead preceeding the less than symbols with 15 space characters. I did not mention that they also contain transactions belonging to several different bank accounts within the same OFX file. I shall be sure to include a sanitized test file when I submit my bug report. Aside: I have discovered quite by accident that the keyboard on this tablet has a Ctrl key that actually works to help manipulate a buffer. It is still tricky to insert text after or in the middle of the previous reply. I think it worked! Now if I can learn how to override the spell checker when it is wrong David C > ___ gnucash-user mailing list gnucash-user@gnucash.org To update your subscription preferences or to unsubscribe: https://lists.gnucash.org/mailman/listinfo/gnucash-user If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information. - Please remember to CC this list on all your replies. You can do this by using Reply-To-List or Reply-All.
Re: [GNC] OFX import sometimes loses transaction memos
Op donderdag 17 januari 2019 06:33:25 CET schreef David Carlson: > I think that I need to spend more time examining the transactions that had > their memos dropped during import to see if I can find a threshold string > length, then file a bug report asking for a short term fix truncating > strings rather than dropping them, and a long term fix accepting up to the > 256 character max and truncating after that. > > David C I don't think there should be one. Some time back I submitted a number of patches to handle long strings in the ofx library. IIRC it internally uses a fixed-length buffer to read parts of the ofx file. And I believe that buffer itself was already 1024 characters long. So at least for lines (including tags) of that size there should not be a problem. Next if the buffer gets full before a full tag contents could be read, it simply fills that buffer a second time and appends that contents to the same first string. So in principle you can have text messages of unlimited size. Now there were bugs in that area, and my patches were partly to resolve those. IIRC one issue was if the ofx file didn't have newlines between tags that buffer handling would not always work as it should. Note those changes have gone into a recent libofx release. As it's been a while I don't know whether gnucash 2.6.x was ever packaged for Windows with this more recent release though I'm sure gnucash 3.x is. And of course there may be other bugs that weren't reported yet. I'm not using ofx myself. Lastly it's probably still interesting to run ofxdump as it can help determine whether the issue already starts in the libofx library or rather that gnucash messes up the data it received from libofx. Regards, Geert ___ gnucash-user mailing list gnucash-user@gnucash.org To update your subscription preferences or to unsubscribe: https://lists.gnucash.org/mailman/listinfo/gnucash-user If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information. - Please remember to CC this list on all your replies. You can do this by using Reply-To-List or Reply-All.
Re: [GNC] OFX import sometimes loses transaction memos
I think that I need to spend more time examining the transactions that had their memos dropped during import to see if I can find a threshold string length, then file a bug report asking for a short term fix truncating strings rather than dropping them, and a long term fix accepting up to the 256 character max and truncating after that. David C On Wed, Jan 16, 2019 at 10:44 PM David Reiser wrote: > The OFX 2.1.1 standard limits elements to a maximum of 255 > alphanumeric characters. Today’s principles are different from those of a > couple (or more) decades ago when these standards were started and both > memory and storage were orders of magnitude more expensive than they are > now. > > > -- > Dave Reiser > dbrei...@icloud.com > > > > > > > On Jan 16, 2019, at 11:09 PM, David Cousens > wrote: > > > > David > > > > Agreed. The OFX format should in principle have no limitation on a > string length as it should be anything between > > opening and closing tags, but there may be on importation in GnuCash. > Not familiar enough with the OFX importer to be > > sure whether it imposes any restrictions or not at this stage. > > > > David Cousens > > On Wed, 2019-01-16 at 18:19 -0600, David Carlson wrote: > >> > >> > >> On Wed, Jan 16, 2019, 4:36 PM David Cousens wrote: > >>> David > >>> > >>> > >>> > >>> Have you tried examining the file with ofxdump? If you are on Linux, > it can > >>> > >>> be installed with "sudo apt install ofx". Not sure about how to get > it for > >>> > >>> WIndows or MacOS. It will dump to stdout whatever ofxdump can determine > >>> > >>> about an ofx response file including error messages. It tests > conformance > >>> > >>> with the libofx librarryofxdump. You may need to use it with the debug > >>> > >>> switch and possibly the parsing messages if using it without the > optional > >>> > >>> switches doesn't give enough info. I.e. > >>> > >>> > >>> > >>> $ofxdump --msg_debug --msg_parser > >>> > >>> > >>> > >>> David Cousens > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> - > >>> > >>> David Cousens > >>> > >>> -- > >>> > >>> Sent from: > http://gnucash.1415818.n4.nabble.com/GnuCash-User-f1415819.html > >>> > >>> ___ > >>> > >>> gnucash-user mailing list > >>> > >>> gnucash-user@gnucash.org > >>> > >>> To update your subscription preferences or to unsubscribe: > >>> > >>> https://lists.gnucash.org/mailman/listinfo/gnucash-user > >>> > >>> If you are using Nabble or Gmane, please see > https://wiki.gnucash.org/wiki/Mailing_Lists for more information. > >>> > >>> - > >>> > >>> Please remember to CC this list on all your replies. > >>> > >>> You can do this by using Reply-To-List or Reply-All. > >> > >> David, > >> > >> I have already examined one of the OFX files that has a mix of some > trans actions with memos that were and we're not > >> imported with a binary editor and found no difference except possibly > number of characters in the string. > >> > >> Right now I am away from home and cannot try OFX DUMP to see if it > knows whether there is a maximum string length, but > >> if there is and it is less than 200 characters I would be surprised. > >> > >> If that is true, I would hope that it would truncate strings rather > than drop them. > >> > >> David C arlson > >> > >> > > ___ > > gnucash-user mailing list > > gnucash-user@gnucash.org > > To update your subscription preferences or to unsubscribe: > > https://lists.gnucash.org/mailman/listinfo/gnucash-user > > If you are using Nabble or Gmane, please see > https://wiki.gnucash.org/wiki/Mailing_Lists for more information. > > - > > Please remember to CC this list on all your replies. > > You can do this by using Reply-To-List or Reply-All. > > ___ gnucash-user mailing list gnucash-user@gnucash.org To update your subscription preferences or to unsubscribe: https://lists.gnucash.org/mailman/listinfo/gnucash-user If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information. - Please remember to CC this list on all your replies. You can do this by using Reply-To-List or Reply-All.
Re: [GNC] OFX import sometimes loses transaction memos
The OFX 2.1.1 standard limits elements to a maximum of 255 alphanumeric characters. Today’s principles are different from those of a couple (or more) decades ago when these standards were started and both memory and storage were orders of magnitude more expensive than they are now. -- Dave Reiser dbrei...@icloud.com > On Jan 16, 2019, at 11:09 PM, David Cousens wrote: > > David > > Agreed. The OFX format should in principle have no limitation on a string > length as it should be anything between > opening and closing tags, but there may be on importation in GnuCash. Not > familiar enough with the OFX importer to be > sure whether it imposes any restrictions or not at this stage. > > David Cousens > On Wed, 2019-01-16 at 18:19 -0600, David Carlson wrote: >> >> >> On Wed, Jan 16, 2019, 4:36 PM David Cousens >> David >>> >>> >>> >>> Have you tried examining the file with ofxdump? If you are on Linux, it can >>> >>> be installed with "sudo apt install ofx". Not sure about how to get it for >>> >>> WIndows or MacOS. It will dump to stdout whatever ofxdump can determine >>> >>> about an ofx response file including error messages. It tests conformance >>> >>> with the libofx librarryofxdump. You may need to use it with the debug >>> >>> switch and possibly the parsing messages if using it without the optional >>> >>> switches doesn't give enough info. I.e. >>> >>> >>> >>> $ofxdump --msg_debug --msg_parser >>> >>> >>> >>> David Cousens >>> >>> >>> >>> >>> >>> >>> >>> - >>> >>> David Cousens >>> >>> -- >>> >>> Sent from: http://gnucash.1415818.n4.nabble.com/GnuCash-User-f1415819.html >>> >>> ___ >>> >>> gnucash-user mailing list >>> >>> gnucash-user@gnucash.org >>> >>> To update your subscription preferences or to unsubscribe: >>> >>> https://lists.gnucash.org/mailman/listinfo/gnucash-user >>> >>> If you are using Nabble or Gmane, please see >>> https://wiki.gnucash.org/wiki/Mailing_Lists for more information. >>> >>> - >>> >>> Please remember to CC this list on all your replies. >>> >>> You can do this by using Reply-To-List or Reply-All. >> >> David, >> >> I have already examined one of the OFX files that has a mix of some trans >> actions with memos that were and we're not >> imported with a binary editor and found no difference except possibly number >> of characters in the string. >> >> Right now I am away from home and cannot try OFX DUMP to see if it knows >> whether there is a maximum string length, but >> if there is and it is less than 200 characters I would be surprised. >> >> If that is true, I would hope that it would truncate strings rather than >> drop them. >> >> David C arlson >> >> > ___ > gnucash-user mailing list > gnucash-user@gnucash.org > To update your subscription preferences or to unsubscribe: > https://lists.gnucash.org/mailman/listinfo/gnucash-user > If you are using Nabble or Gmane, please see > https://wiki.gnucash.org/wiki/Mailing_Lists for more information. > - > Please remember to CC this list on all your replies. > You can do this by using Reply-To-List or Reply-All. ___ gnucash-user mailing list gnucash-user@gnucash.org To update your subscription preferences or to unsubscribe: https://lists.gnucash.org/mailman/listinfo/gnucash-user If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information. - Please remember to CC this list on all your replies. You can do this by using Reply-To-List or Reply-All.
Re: [GNC] OFX import sometimes loses transaction memos
David Agreed. The OFX format should in principle have no limitation on a string length as it should be anything between opening and closing tags, but there may be on importation in GnuCash. Not familiar enough with the OFX importer to be sure whether it imposes any restrictions or not at this stage. David Cousens On Wed, 2019-01-16 at 18:19 -0600, David Carlson wrote: > > > On Wed, Jan 16, 2019, 4:36 PM David Cousens > David > > > > > > > > Have you tried examining the file with ofxdump? If you are on Linux, it can > > > > be installed with "sudo apt install ofx". Not sure about how to get it for > > > > WIndows or MacOS. It will dump to stdout whatever ofxdump can determine > > > > about an ofx response file including error messages. It tests conformance > > > > with the libofx librarryofxdump. You may need to use it with the debug > > > > switch and possibly the parsing messages if using it without the optional > > > > switches doesn't give enough info. I.e. > > > > > > > > $ofxdump --msg_debug --msg_parser > > > > > > > > David Cousens > > > > > > > > > > > > > > > > - > > > > David Cousens > > > > -- > > > > Sent from: http://gnucash.1415818.n4.nabble.com/GnuCash-User-f1415819.html > > > > ___ > > > > gnucash-user mailing list > > > > gnucash-user@gnucash.org > > > > To update your subscription preferences or to unsubscribe: > > > > https://lists.gnucash.org/mailman/listinfo/gnucash-user > > > > If you are using Nabble or Gmane, please see > > https://wiki.gnucash.org/wiki/Mailing_Lists for more information. > > > > - > > > > Please remember to CC this list on all your replies. > > > > You can do this by using Reply-To-List or Reply-All. > > David, > > I have already examined one of the OFX files that has a mix of some trans > actions with memos that were and we're not > imported with a binary editor and found no difference except possibly number > of characters in the string. > > Right now I am away from home and cannot try OFX DUMP to see if it knows > whether there is a maximum string length, but > if there is and it is less than 200 characters I would be surprised. > > If that is true, I would hope that it would truncate strings rather than drop > them. > > David C arlson > > ___ gnucash-user mailing list gnucash-user@gnucash.org To update your subscription preferences or to unsubscribe: https://lists.gnucash.org/mailman/listinfo/gnucash-user If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information. - Please remember to CC this list on all your replies. You can do this by using Reply-To-List or Reply-All.
Re: [GNC] OFX import sometimes loses transaction memos
On Wed, Jan 16, 2019, 4:36 PM David Cousens David > > Have you tried examining the file with ofxdump? If you are on Linux, it can > be installed with "sudo apt install ofx". Not sure about how to get it for > WIndows or MacOS. It will dump to stdout whatever ofxdump can determine > about an ofx response file including error messages. It tests conformance > with the libofx librarryofxdump. You may need to use it with the debug > switch and possibly the parsing messages if using it without the optional > switches doesn't give enough info. I.e. > > $ofxdump --msg_debug --msg_parser > > David Cousens > > > > - > David Cousens > -- > Sent from: http://gnucash.1415818.n4.nabble.com/GnuCash-User-f1415819.html > ___ > gnucash-user mailing list > gnucash-user@gnucash.org > To update your subscription preferences or to unsubscribe: > https://lists.gnucash.org/mailman/listinfo/gnucash-user > If you are using Nabble or Gmane, please see > https://wiki.gnucash.org/wiki/Mailing_Lists for more information. > - > Please remember to CC this list on all your replies. > You can do this by using Reply-To-List or Reply-All. > David, I have already examined one of the OFX files that has a mix of some trans actions with memos that were and we're not imported with a binary editor and found no difference except possibly number of characters in the string. Right now I am away from home and cannot try OFX DUMP to see if it knows whether there is a maximum string length, but if there is and it is less than 200 characters I would be surprised. If that is true, I would hope that it would truncate strings rather than drop them. David C arlson > ___ gnucash-user mailing list gnucash-user@gnucash.org To update your subscription preferences or to unsubscribe: https://lists.gnucash.org/mailman/listinfo/gnucash-user If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information. - Please remember to CC this list on all your replies. You can do this by using Reply-To-List or Reply-All.
Re: [GNC] OFX import sometimes loses transaction memos
David Have you tried examining the file with ofxdump? If you are on Linux, it can be installed with "sudo apt install ofx". Not sure about how to get it for WIndows or MacOS. It will dump to stdout whatever ofxdump can determine about an ofx response file including error messages. It tests conformance with the libofx librarryofxdump. You may need to use it with the debug switch and possibly the parsing messages if using it without the optional switches doesn't give enough info. I.e. $ofxdump --msg_debug --msg_parser David Cousens - David Cousens -- Sent from: http://gnucash.1415818.n4.nabble.com/GnuCash-User-f1415819.html ___ gnucash-user mailing list gnucash-user@gnucash.org To update your subscription preferences or to unsubscribe: https://lists.gnucash.org/mailman/listinfo/gnucash-user If you are using Nabble or Gmane, please see https://wiki.gnucash.org/wiki/Mailing_Lists for more information. - Please remember to CC this list on all your replies. You can do this by using Reply-To-List or Reply-All.
