Re: [GNC] Strange prices update issue (possibly with TSP)
Appreciate bringing up the updated F::Q to my attention. I'll re-approach in
that direction.
I'll also make note of addressing it via GitHub issue tracker.
-Original Message-
From: John Ralls
Sent: Monday, August 14, 2023 8:30 PM
To: Kalpesh Patel
Cc: Gnucash Users ; Bruce Schuck
Subject: Re: [GNC] Strange prices update issue (possibly with TSP)
Kalpesh,
Bruce posted at
https://github.com/finance-quote/finance-quote/issues/335#issuecomment-16764
43182:
"While the fix did not make it into v1.58, you can download and install
BPSCHUCK/Finance-Quote-1.58_01.tar.gz from CPAN for testing/verification. It
also has what I hope is a fix for TSP. It was not returning the hash if the
GET failed or if the body of the response produced a CSV file."
That's the BSEIndia issue;
https://github.com/finance-quote/finance-quote/issues/338 is the TSP one.
If you find more problems with TSP (or anything else in F::Q) and want to
help debug it the best way to communicate with Bruce and stay up to date
would be to open an issue at
https://github.com/finance-quote/finance-quote/issues/. You can post here
too if you want, but a Github issue is a better place to put debugging
results.
Regards,
John Ralls
> On Aug 14, 2023, at 1:54 PM, Kalpesh Patel wrote:
>
> Some good find in debugging ... Strawberry Perl cannot resolve the FQDN
name one out of three times for www.tsp.gov on my system (this might be
obfuscating the client behavior in response to real backend issue). It might
be that I am reaching servers that are in error state (rc = 500 which is
Internal Server Error; my observation on F::Q could be cause of concern on
amount of data being returned). Below is the output from the failed run:
>
> C:\Users\kalpesh\OneDrive\QuickenStuff\HELPERS>set DEBUG=1
>
> C:\Users\kalpesh\OneDrive\QuickenStuff\HELPERS>test2.pl
>
> ### [Mon Aug 14 16:26:56 2023] YIND_URL_HEAD :
'https://query1.finance.yahoo.com/v8/finance/chart/'
> ### [Mon Aug 14 16:26:56 2023] YIND_URL_TAIL :
'?interval=1d&period1=1691440016&period2=1692044816'
>
> ### AlphaVantage->new args : {}
>
> ### COUNT_URL: 'http://www.panix.com/~hd-fxsts/finance-quote.html?tsp'
> ### Code: '200'
>
> ### [Mon Aug 14 16:26:56 2023] url :
'https://www.tsp.gov/data/fund-price-history.csv?startdate=2023-08-07&enddat
e=2023-08-14&Lfunds=1&InvFunds=1&download=1'
> ### [Mon Aug 14 16:26:56 2023] reply: bless( {
> ###_content => 'Can\'t connect
to www.tsp.gov:443 (nodename nor servname provided, or not known)
>
> nodename nor servname provided, or not known at
C:/Strawberry/perl/site/lib/LWP/Protocol/http.pm line 50.
> ',
> ###_headers => bless( {
> ###
'::std_case' => {
> ###
'client-date' => 'Client-Date',
> ###
'client-warning' => 'Client-Warning'
> ###
},
> ###
'client-date' => 'Mon, 14 Aug 2023 20:26:56 GMT',
> ###
'client-warning' => 'Internal response',
> ###
'content-type' => 'text/plain'
> ### },
'HTTP::Headers' ),
> ###_msg => 'Can\'t connect to
www.tsp.gov:443 (nodename nor servname provided, or not known)',
> ###_rc => 500,
> ###_request => bless( {
> ###
_content => '',
> ###
_headers => bless( {
> ###
'user-agent' => 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Safari/537.36'
> ###
}, 'HTTP::Headers' ),
> ###
_method => 'GET',
> ### _uri
=> bless( do{\(my $o =
'https://www.tsp.gov/data/fund-price-history.csv?startdate=2023-08-07&enddat
e=2023-08-14&Lfunds=1&InvFunds=1&download=1')}, 'URI::https' )
> ### },
'HTTP::Request' )
> ### }, 'HTTP::Response' )
>
> C:\Users\kalpesh\OneDrive\QuickenStuff\HELPERS>
>
> My test file (test2.pl) is as follows:
>
> #test2.pl - start
>
>use Finance::Quote;
>use Data::Dumper;
>
>$q = Finance::Quote->new;
>
>%info = $q->fetch('tsp', "L2030");
>
>print Dumper(%info);
>
> # test2.pl - end
>
>
> Interesting side notes for GNC (I am on Windows 11 with GNC v4.14 and now
F::Q
Re: [GNC] Strange prices update issue (possibly with TSP)
6653,70.6948,72.5871,38.9458 > 2023-07-25,24.2163,12.3359,43.5775,13.0365,49.1733,13.4189,29.3220,14.4395,14.4379,14.4361,17.6133,18.6144,70.7047,72.2457,38.9313 > 2023-07-24,24.1974,12.3223,43.4989,13.0110,49.0687,13.3885,29.2512,14.3963,14.3947,14.3929,17.6114,18.6334,70.5044,72.1292,38.7806 > 2023-07-21,24.1841,12.3139,43.4569,12.9977,49.0157,13.3735,29.2167,14.3747,14.3732,14.3714,17.6055,18.6571,70.2201,72.1244,38.8425 > 2023-07-20,24.1808,12.3123,43.4515,12.9961,49.0096,13.3719,29.2133,14.3737,14.3722,14.3704,17.6036,18.6455,70.1974,72.2722,38.8218 > 2023-07-19,24.2266,12.3452,43.6382,13.0578,49.2628,13.4460,29.3849,14.4692,14.4677,14.4659,17.6017,18.7521,70.6685,73.1168,39.0026 > 2023-07-18,24.2119,12.3358,43.5911,13.0423,49.2001,13.4277,29.3433,14.4485,14.4469,14.4451,17.5998,18.6938,70.5015,72.8049,39.0442 > 2023-07-17,24.1639,12.3012,43.3926,12.9773,48.9337,13.3500,29.1627,14.3422,14.3407,14.3389,17.5978,18.6769,70.0036,72.0267,38.7854 > 2023-07-14,24.1430,12.2877,43.3244,12.9552,48.8445,13.3242,29.1032,14.3088,14.3072,14.3055,17.5920,18.6543,69.7345,71.3406,38.8917 > 2023-07-13,24.1672,12.3051,43.4224,12.9878,48.9782,13.3634,29.1941,14.3581,14.3566,14.3548,17.5901,18.7318,69.8035,72.0356,39.0693 > 2023-07-12,24.0841,12.2462,43.0912,12.8792,48.5343,13.2341,28.8948,14.1877,14.1862,14.1844,17.5881,18.6216,69.2107,71.2102,38.4014 > 2023-07-11,24.0007,12.1877,42.7678,12.7731,48.1017,13.1082,28.6041,14.0255,14.0239,14.0220,17.5862,18.4682,68.7006,70.6988,37.6648 > 2023-07-10,23.9468,12.1490,42.5478,12.7011,47.8070,13.0223,28.4048,13.9094,13.9078,13.9059,17.5843,18.4385,68.2399,69.6977,37.3488 > 2023-07-07,23.9111,12.1253,42.4239,12.6607,47.6429,12.9745,28.2947,13.8488,13.8471,13.8453,17.5785,18.3742,68.0764,68.6900,37.2301 > 2023-07-06,23.8994,12.1170,42.3776,12.6455,47.5804,12.9561,28.2516,13.8230,13.8214,13.8195,17.5765,18.3788,68.2614,68.0180,37.0282 > [snip] > > > Three observation on F::Q: > > 1) This request downloads CSVs that goes back to 2003-05-31 (20+ years!) even > though request is only for 7 days in the URL. It is like enddate= parameter > is ignored or incorrect parameter passed. Possibly a bug in TSP.pm module? > 2) Is the anonymous data collection of modules utilized still being > collected? Code execution is not harming anything if not being collected but > this is more curiosity on my part for v1.58 which was released few days back. > 3) Looks like Smart::Comment when activated emits messages for all modules > rather than the single one that is being instantiated for the source and > called to action. > > On to debugging the Perl LWP module... > > > > > -Original Message- > From: Bruce Schuck > Sent: Friday, August 11, 2023 9:40 PM > To: gnucash-user@gnucash.org > Cc: Kalpesh Patel ; john > Subject: Re: [GNC] Strange prices update issue (possibly with TSP) > > On Aug 10, 2023, at 12:13, Kalpesh Patel <mailto:kalpesh.pa...@usa.net> > wrote: > >> The difficult part in debugging this is that when I do query three >> times from the command line, it succeeds without any issue for me as >> well, but when I update prices from GNC, it fails two times and then >> succeeds third time consistently. So this is not an issue with >> throttling from TSP source which you validated as well. Is there more >> in terms of debugging this by any chance? Can I use STDERR and STDOUT >> from Perl to emanate messages to console but not interfere with what >> JSON comes back or butcher the response back to gnc-fq-helper so it >> doesn't bomb to run a full cycles from GNC to test? Any help in debugging is >> appreciated. > > Kalpesh, > > Many, but not all, of the F::Q modules make use of the Smart::Comments > module. TSP.pm is one of them. The environment variable DEBUG will need to be > set, and with no changes to the Perl code you should see helpful information > printed to STDERR. In the code, these are lines that begin with "###" (note, > just 3 pound signs) in those modules using Smart::Comments. > > A quick look at the TSP module and I see two where the "tsp" method will > return *without* setting "success" and "errormsg" for any of the symbols > passed to it. This will happen if the http get call fails, or whatever is > downloaded is not 2 or more lines. > > my $reply = $ua->get($url, @HEADERS); > ### [] url : $url > ### [] reply: $reply > > return unless ($reply->is_success); > > my @line = split(/\n/, $reply->content); > > return unless (@line > 1); > > In the code snippet, there are two examples Smart::Comments use which could > be useful in your case. With DEBUG set the code would print the url to > STDERR, and entire $reply object (status, headers, cookies, body). > > Whether or not this helps you, the omission of not returning the proper > failure data is an issue. > > Hope this helps. > > Bruce S > ___ gnucash-user mailing list gnucash-user@gnucash.org To update your subscription preferences or to unsubscribe: https://lists.gnucash.org/mailman/listinfo/gnucash-user - Please remember to CC this list on all your replies. You can do this by using Reply-To-List or Reply-All.
Re: [GNC] Strange prices update issue (possibly with TSP)
7-06,23.8994,12.1170,42.3776,12.6455,47.5804,12.9561,28.2516,13.8230,13.8214,13.8195,17.5765,18.3788,68.2614,68.0180,37.0282 [snip] Three observation on F::Q: 1) This request downloads CSVs that goes back to 2003-05-31 (20+ years!) even though request is only for 7 days in the URL. It is like enddate= parameter is ignored or incorrect parameter passed. Possibly a bug in TSP.pm module? 2) Is the anonymous data collection of modules utilized still being collected? Code execution is not harming anything if not being collected but this is more curiosity on my part for v1.58 which was released few days back. 3) Looks like Smart::Comment when activated emits messages for all modules rather than the single one that is being instantiated for the source and called to action. On to debugging the Perl LWP module... -Original Message- From: Bruce Schuck Sent: Friday, August 11, 2023 9:40 PM To: gnucash-user@gnucash.org Cc: Kalpesh Patel ; john Subject: Re: [GNC] Strange prices update issue (possibly with TSP) On Aug 10, 2023, at 12:13, Kalpesh Patel mailto:kalpesh.pa...@usa.net> > wrote: > The difficult part in debugging this is that when I do query three > times from the command line, it succeeds without any issue for me as > well, but when I update prices from GNC, it fails two times and then > succeeds third time consistently. So this is not an issue with > throttling from TSP source which you validated as well. Is there more > in terms of debugging this by any chance? Can I use STDERR and STDOUT > from Perl to emanate messages to console but not interfere with what > JSON comes back or butcher the response back to gnc-fq-helper so it > doesn't bomb to run a full cycles from GNC to test? Any help in debugging is > appreciated. Kalpesh, Many, but not all, of the F::Q modules make use of the Smart::Comments module. TSP.pm is one of them. The environment variable DEBUG will need to be set, and with no changes to the Perl code you should see helpful information printed to STDERR. In the code, these are lines that begin with "###" (note, just 3 pound signs) in those modules using Smart::Comments. A quick look at the TSP module and I see two where the "tsp" method will return *without* setting "success" and "errormsg" for any of the symbols passed to it. This will happen if the http get call fails, or whatever is downloaded is not 2 or more lines. my $reply = $ua->get($url, @HEADERS); ### [] url : $url ### [] reply: $reply return unless ($reply->is_success); my @line = split(/\n/, $reply->content); return unless (@line > 1); In the code snippet, there are two examples Smart::Comments use which could be useful in your case. With DEBUG set the code would print the url to STDERR, and entire $reply object (status, headers, cookies, body). Whether or not this helps you, the omission of not returning the proper failure data is an issue. Hope this helps. Bruce S ___ gnucash-user mailing list gnucash-user@gnucash.org To update your subscription preferences or to unsubscribe: https://lists.gnucash.org/mailman/listinfo/gnucash-user - Please remember to CC this list on all your replies. You can do this by using Reply-To-List or Reply-All.
Re: [GNC] Strange prices update issue (possibly with TSP)
I was reading up on the Hacker's guide at https://github.com/finance-quote/finance-quote/blob/master/Documentation/Hackers-Guide to see if that facility was available or not (and to write my own module) in the TSP.pm module. Appreciate confirming it and now debugging will become much more easier. I'll follow up with what I find. Thank you, to both, for all these valuable pointers. -Original Message- From: Bruce Schuck Sent: Friday, August 11, 2023 9:40 PM To: gnucash-user@gnucash.org Cc: Kalpesh Patel ; john Subject: Re: [GNC] Strange prices update issue (possibly with TSP) On Aug 10, 2023, at 12:13, Kalpesh Patel mailto:kalpesh.pa...@usa.net> > wrote: > The difficult part in debugging this is that when I do query three > times from the command line, it succeeds without any issue for me as > well, but when I update prices from GNC, it fails two times and then > succeeds third time consistently. So this is not an issue with > throttling from TSP source which you validated as well. Is there more > in terms of debugging this by any chance? Can I use STDERR and STDOUT > from Perl to emanate messages to console but not interfere with what > JSON comes back or butcher the response back to gnc-fq-helper so it > doesn't bomb to run a full cycles from GNC to test? Any help in debugging is > appreciated. Kalpesh, Many, but not all, of the F::Q modules make use of the Smart::Comments module. TSP.pm is one of them. The environment variable DEBUG will need to be set, and with no changes to the Perl code you should see helpful information printed to STDERR. In the code, these are lines that begin with "###" (note, just 3 pound signs) in those modules using Smart::Comments. A quick look at the TSP module and I see two where the "tsp" method will return *without* setting "success" and "errormsg" for any of the symbols passed to it. This will happen if the http get call fails, or whatever is downloaded is not 2 or more lines. my $reply = $ua->get($url, @HEADERS); ### [] url : $url ### [] reply: $reply return unless ($reply->is_success); my @line = split(/\n/, $reply->content); return unless (@line > 1); In the code snippet, there are two examples Smart::Comments use which could be useful in your case. With DEBUG set the code would print the url to STDERR, and entire $reply object (status, headers, cookies, body). Whether or not this helps you, the omission of not returning the proper failure data is an issue. Hope this helps. Bruce S ___ gnucash-user mailing list gnucash-user@gnucash.org To update your subscription preferences or to unsubscribe: https://lists.gnucash.org/mailman/listinfo/gnucash-user - Please remember to CC this list on all your replies. You can do this by using Reply-To-List or Reply-All.
Re: [GNC] Strange prices update issue (possibly with TSP)
On Aug 10, 2023, at 12:13, Kalpesh Patel mailto:kalpesh.pa...@usa.net> > wrote: The difficult part in debugging this is that when I do query three times from the command line, it succeeds without any issue for me as well, but when I update prices from GNC, it fails two times and then succeeds third time consistently. So this is not an issue with throttling from TSP source which you validated as well. Is there more in terms of debugging this by any chance? Can I use STDERR and STDOUT from Perl to emanate messages to console but not interfere with what JSON comes back or butcher the response back to gnc-fq-helper so it doesn't bomb to run a full cycles from GNC to test? Any help in debugging is appreciated. Kalpesh, Many, but not all, of the F::Q modules make use of the Smart::Comments module. TSP.pm is one of them. The environment variable DEBUG will need to be set, and with no changes to the Perl code you should see helpful information printed to STDERR. In the code, these are lines that begin with "###" (note, just 3 pound signs) in those modules using Smart::Comments. A quick look at the TSP module and I see two where the "tsp" method will return *without* setting "success" and "errormsg" for any of the symbols passed to it. This will happen if the http get call fails, or whatever is downloaded is not 2 or more lines. my $reply = $ua->get($url, @HEADERS); ### [] url : $url ### [] reply: $reply return unless ($reply->is_success); my @line = split(/\n/, $reply->content); return unless (@line > 1); In the code snippet, there are two examples Smart::Comments use which could be useful in your case. With DEBUG set the code would print the url to STDERR, and entire $reply object (status, headers, cookies, body). Whether or not this helps you, the omission of not returning the proper failure data is an issue. Hope this helps. Bruce S ___ gnucash-user mailing list gnucash-user@gnucash.org To update your subscription preferences or to unsubscribe: https://lists.gnucash.org/mailman/listinfo/gnucash-user - Please remember to CC this list on all your replies. You can do this by using Reply-To-List or Reply-All.
Re: [GNC] Strange prices update issue (possibly with TSP)
John, Since I juggle through many flavors of platforms, I am in habit of creating c:\temp or /tmp to force writing temporary junk to known location. In anyways I’ll listen to your advice and just provide the file name rather than absolute path-&-file combo. From: john Sent: Friday, August 11, 2023 1:22 PM To: Kalpesh Patel Cc: gnucash-u...@lists.gnucash.org Subject: Re: [GNC] Strange prices update issue (possibly with TSP) Kalpesh, c:\temp doesn't usually exist on Windows. The default path for tracefiles is C:\Users\myname\AppData\Local\Temp (see https://wiki.gnucash.org/wiki/Tracefile#Windows). That location is set by a system call that's been unchanged forever, though (as noted on that wiki page) *Microsoft* changed the location between XP and Vista. Unless you first create c:\temp trying to write there will just raise an error. The current working directory (just give it a filename, no path) is usually the most convenient. Regards, John Ralls On Aug 11, 2023, at 09:30, Kalpesh Patel mailto:kalpesh.pa...@usa.net> > wrote: There might be a bug on Windows platform for specification to --logto=\path\to\file option to GNC v4.14 binary. I see that the actual file created is c:\temp\gnc_debug.txt..log when I specify c:\temp\gnc_debug.txt as the parameter. I’ll check on the latest v5 release and log a bug if that also behaves the same way. From: john mailto:jra...@ceridwen.us> > Sent: Thursday, August 10, 2023 11:05 PM To: Kalpesh Patel mailto:kalpesh.pa...@usa.net> > Cc: gnucash-u...@lists.gnucash.org <mailto:gnucash-u...@lists.gnucash.org> Subject: Re: [GNC] Strange prices update issue (possibly with TSP) Kalpesh, You should be able to print diagnostics using `print STDERR ...`. Part of the problem with price-quotes.scm's error handling was that it didn't even look at gnc-fq-helper's stderr. The order in the tracefile you posted showed that tsp returned prices on the *first* try and failed on the subsequent two: * 10:02:36 DEBUG handling-request: (tsp C S L2030 I) * 10:02:42 DEBUG results: ((C (symbol . C) (gnc:time-no-zone . 2023-08-09 12:00:00) (last . 691951/1) (currency . USD)) (S (symbol . S) (gnc:time-no-zone . 2023-08-09 12:00:00) (last . 704343/1) (currency . USD)) (L2030 (symbol . L2030) (gnc:time-no-zone . 2023-08-09 12:00:00) (last . 53747/1250) (currency . USD)) (I (symbol . I) (gnc:time-no-zone . 2023-08-09 12:00:00) (last . 380083/1) (currency . USD))) [snip] * 10:03:11 DEBUG handling-request: (tsp C S L2030 I) * 10:03:17 DEBUG results: #f [snip] * 10:03:37 DEBUG handling-request: (tsp C S L2030 I) * 10:03:43 DEBUG results: #f Regards, John Ralls On Aug 10, 2023, at 12:13, Kalpesh Patel < <mailto:kalpesh.pa...@usa.net> kalpesh.pa...@usa.net> wrote: John, Appreciate that details. That is going to be quite helpful. The difficult part in debugging this is that when I do query three times from the command line, it succeeds without any issue for me as well, but when I update prices from GNC, it fails two times and then succeeds third time consistently. So this is not an issue with throttling from TSP source which you validated as well. Is there more in terms of debugging this by any chance? Can I use STDERR and STDOUT from Perl to emanate messages to console but not interfere with what JSON comes back or butcher the response back to gnc-fq-helper so it doesn't bomb to run a full cycles from GNC to test? Any help in debugging is appreciated. Looks like I may need to debug the TSP.pm module under the hood to figure out what might be going on and most likely with the response coming back from the request to <https://www.tsp.gov/data/fund-price-history.csv> https://www.tsp.gov/data/fund-price-history.csv URI. -Original Message- From: john < <mailto:jra...@ceridwen.us> jra...@ceridwen.us> Sent: Thursday, August 10, 2023 12:54 PM To: Kalpesh Patel < <mailto:kalpesh.pa...@usa.net> kalpesh.pa...@usa.net> Cc: <mailto:gnucash-u...@lists.gnucash.org> gnucash-u...@lists.gnucash.org Subject: Re: [GNC] Strange prices update issue (possibly with TSP) Kalpesh, It does indeed look like the tsp module failed to return anything in the second and third tries. You're correct that gnc:scm is the right logger for GnuCash 4. gnc:price-quotes works only in GnuCash 5. For GnuCash to process a partial result from F::Q, F::Q needs to produce a valid response for each symbol; the response includes a Success variable; GnuCash will process symbols with Success=1 and present a message box containing the symbols for which F::Q returned Success=0. In the two cases where the TSE module failed it returned something that gnc-fq-helper wasn't able to parse so it posted a Scheme #f (meaning false, the equivalent of Python's None type) and logged an "unknown" error.
Re: [GNC] Strange prices update issue (possibly with TSP)
Kalpesh, c:\temp doesn't usually exist on Windows. The default path for tracefiles is C:\Users\myname\AppData\Local\Temp (see https://wiki.gnucash.org/wiki/Tracefile#Windows). That location is set by a system call that's been unchanged forever, though (as noted on that wiki page) *Microsoft* changed the location between XP and Vista. Unless you first create c:\temp trying to write there will just raise an error. The current working directory (just give it a filename, no path) is usually the most convenient. Regards, John Ralls > On Aug 11, 2023, at 09:30, Kalpesh Patel wrote: > > There might be a bug on Windows platform for specification to > --logto=\path\to\file option to GNC v4.14 binary. > > I see that the actual file created is > c:\temp\gnc_debug.txt..log when I specify > c:\temp\gnc_debug.txt as the parameter. I’ll check on the latest v5 release > and log a bug if that also behaves the same way. > > From: john > Sent: Thursday, August 10, 2023 11:05 PM > To: Kalpesh Patel > Cc: gnucash-u...@lists.gnucash.org > Subject: Re: [GNC] Strange prices update issue (possibly with TSP) > > Kalpesh, > > You should be able to print diagnostics using `print STDERR ...`. Part of the > problem with price-quotes.scm's error handling was that it didn't even look > at gnc-fq-helper's stderr. > > The order in the tracefile you posted showed that tsp returned prices on the > *first* try and failed on the subsequent two: >> * 10:02:36 DEBUG handling-request: (tsp C S L2030 I) >> > >> * 10:02:42 DEBUG results: ((C (symbol . C) (gnc:time-no-zone . >> 2023-08-09 12:00:00) (last . 691951/1) (currency . USD)) (S (symbol . S) >> (gnc:time-no-zone . 2023-08-09 12:00:00) (last . 704343/1) (currency . >> USD)) (L2030 (symbol . L2030) (gnc:time-no-zone . 2023-08-09 12:00:00) (last >> . 53747/1250) (currency . USD)) (I (symbol . I) (gnc:time-no-zone . >> 2023-08-09 12:00:00) (last . 380083/1) (currency . USD))) > [snip] >> >> * 10:03:11 DEBUG handling-request: (tsp C S L2030 I) >> >> * 10:03:17 DEBUG results: #f > [snip] >> * 10:03:37 DEBUG handling-request: (tsp C S L2030 I) >> >> * 10:03:43 DEBUG results: #f > > > Regards, > John Ralls > > > >> On Aug 10, 2023, at 12:13, Kalpesh Patel > <mailto:kalpesh.pa...@usa.net>> wrote: >> >> John, >> >> Appreciate that details. That is going to be quite helpful. >> >> The difficult part in debugging this is that when I do query three times >> from the command line, it succeeds without any issue for me as well, but >> when I update prices from GNC, it fails two times and then succeeds third >> time consistently. So this is not an issue with throttling from TSP source >> which you validated as well. Is there more in terms of debugging this by any >> chance? Can I use STDERR and STDOUT from Perl to emanate messages to console >> but not interfere with what JSON comes back or butcher the response back to >> gnc-fq-helper so it doesn't bomb to run a full cycles from GNC to test? Any >> help in debugging is appreciated. >> >> Looks like I may need to debug the TSP.pm module under the hood to figure >> out what might be going on and most likely with the response coming back >> from the request to https://www.tsp.gov/data/fund-price-history.csv URI. >> >> -Original Message- >> From: john mailto:jra...@ceridwen.us>> >> Sent: Thursday, August 10, 2023 12:54 PM >> To: Kalpesh Patel mailto:kalpesh.pa...@usa.net>> >> Cc: gnucash-u...@lists.gnucash.org <mailto:gnucash-u...@lists.gnucash.org> >> Subject: Re: [GNC] Strange prices update issue (possibly with TSP) >> >> Kalpesh, >> >> It does indeed look like the tsp module failed to return anything in the >> second and third tries. >> >> You're correct that gnc:scm is the right logger for GnuCash 4. >> gnc:price-quotes works only in GnuCash 5. >> >> For GnuCash to process a partial result from F::Q, F::Q needs to produce a >> valid response for each symbol; the response includes a Success variable; >> GnuCash will process symbols with Success=1 and present a message box >> containing the symbols for which F::Q returned Success=0. In the two cases >> where the TSE module failed it returned something that gnc-fq-helper wasn't >> able to parse so it posted a Scheme #f (meaning false, the equivalent of >> Python's None type) and logged an "unknown" error. GnuCash 5 would do a >> better job passing on the error but whether it could process
Re: [GNC] Strange prices update issue (possibly with TSP)
There might be a bug on Windows platform for specification to --logto=\path\to\file option to GNC v4.14 binary. I see that the actual file created is c:\temp\gnc_debug.txt..log when I specify c:\temp\gnc_debug.txt as the parameter. I'll check on the latest v5 release and log a bug if that also behaves the same way. From: john Sent: Thursday, August 10, 2023 11:05 PM To: Kalpesh Patel Cc: gnucash-u...@lists.gnucash.org Subject: Re: [GNC] Strange prices update issue (possibly with TSP) Kalpesh, You should be able to print diagnostics using `print STDERR ...`. Part of the problem with price-quotes.scm's error handling was that it didn't even look at gnc-fq-helper's stderr. The order in the tracefile you posted showed that tsp returned prices on the *first* try and failed on the subsequent two: * 10:02:36 DEBUG handling-request: (tsp C S L2030 I) * 10:02:42 DEBUG results: ((C (symbol . C) (gnc:time-no-zone . 2023-08-09 12:00:00) (last . 691951/1) (currency . USD)) (S (symbol . S) (gnc:time-no-zone . 2023-08-09 12:00:00) (last . 704343/1) (currency . USD)) (L2030 (symbol . L2030) (gnc:time-no-zone . 2023-08-09 12:00:00) (last . 53747/1250) (currency . USD)) (I (symbol . I) (gnc:time-no-zone . 2023-08-09 12:00:00) (last . 380083/1) (currency . USD))) [snip] * 10:03:11 DEBUG handling-request: (tsp C S L2030 I) * 10:03:17 DEBUG results: #f [snip] * 10:03:37 DEBUG handling-request: (tsp C S L2030 I) * 10:03:43 DEBUG results: #f Regards, John Ralls On Aug 10, 2023, at 12:13, Kalpesh Patel mailto:kalpesh.pa...@usa.net> > wrote: John, Appreciate that details. That is going to be quite helpful. The difficult part in debugging this is that when I do query three times from the command line, it succeeds without any issue for me as well, but when I update prices from GNC, it fails two times and then succeeds third time consistently. So this is not an issue with throttling from TSP source which you validated as well. Is there more in terms of debugging this by any chance? Can I use STDERR and STDOUT from Perl to emanate messages to console but not interfere with what JSON comes back or butcher the response back to gnc-fq-helper so it doesn't bomb to run a full cycles from GNC to test? Any help in debugging is appreciated. Looks like I may need to debug the TSP.pm module under the hood to figure out what might be going on and most likely with the response coming back from the request to https://www.tsp.gov/data/fund-price-history.csv URI. -Original Message- From: john mailto:jra...@ceridwen.us> > Sent: Thursday, August 10, 2023 12:54 PM To: Kalpesh Patel mailto:kalpesh.pa...@usa.net> > Cc: gnucash-u...@lists.gnucash.org <mailto:gnucash-u...@lists.gnucash.org> Subject: Re: [GNC] Strange prices update issue (possibly with TSP) Kalpesh, It does indeed look like the tsp module failed to return anything in the second and third tries. You're correct that gnc:scm is the right logger for GnuCash 4. gnc:price-quotes works only in GnuCash 5. For GnuCash to process a partial result from F::Q, F::Q needs to produce a valid response for each symbol; the response includes a Success variable; GnuCash will process symbols with Success=1 and present a message box containing the symbols for which F::Q returned Success=0. In the two cases where the TSE module failed it returned something that gnc-fq-helper wasn't able to parse so it posted a Scheme #f (meaning false, the equivalent of Python's None type) and logged an "unknown" error. GnuCash 5 would do a better job passing on the error but whether it could process the quotes returned from the other sources would depend on the JSON emitted by F::Q. FWIW I tried your "tsp C S L2030 I" query three times from the command line just now and got three valid responses. You can tell GnuCash to write a specific file with the --logto=/path/to/file option. It will overwrite if you pass an existing path and the only way to prevent that would be to write a wrapper script in your favorite scripting language that does the filename substitution. Regards, John Ralls On Aug 10, 2023, at 07:50, Kalpesh Patel mailto:kalpesh.pa...@usa.net> > wrote: So I am seeing a strange issue updating prices on my system. For each try, every third one succeeds but other two fails and this is very consistent behavior all the time on my system. I am on Windows 11 and using GNC v4.14 (time to upgrade? There is a good reason why I need to stay there) with F::Q v1.57_02. I turned on debugging (via --log=gnc.price-quotes=debug --log=gnc.scm=debug; looks like latter one is applicable to GNC v4.14) and below is the output from the trace file. Am I correct to interpret that TSP source is failing two out of three times? I see result as being * 10:03:17 DEBUG results: #f which indicates it failed to retrieve prices for that source but oth
Re: [GNC] Strange prices update issue (possibly with TSP)
That is a start. Thank you for confirming STDERR pointer. Correct, GNC obtains prices on the first request then fail on subsequent two, and then pattern repeats . From: john Sent: Thursday, August 10, 2023 11:05 PM To: Kalpesh Patel Cc: gnucash-u...@lists.gnucash.org Subject: Re: [GNC] Strange prices update issue (possibly with TSP) Kalpesh, You should be able to print diagnostics using `print STDERR ...`. Part of the problem with price-quotes.scm's error handling was that it didn't even look at gnc-fq-helper's stderr. The order in the tracefile you posted showed that tsp returned prices on the *first* try and failed on the subsequent two: * 10:02:36 DEBUG handling-request: (tsp C S L2030 I) * 10:02:42 DEBUG results: ((C (symbol . C) (gnc:time-no-zone . 2023-08-09 12:00:00) (last . 691951/1) (currency . USD)) (S (symbol . S) (gnc:time-no-zone . 2023-08-09 12:00:00) (last . 704343/1) (currency . USD)) (L2030 (symbol . L2030) (gnc:time-no-zone . 2023-08-09 12:00:00) (last . 53747/1250) (currency . USD)) (I (symbol . I) (gnc:time-no-zone . 2023-08-09 12:00:00) (last . 380083/1) (currency . USD))) [snip] * 10:03:11 DEBUG handling-request: (tsp C S L2030 I) * 10:03:17 DEBUG results: #f [snip] * 10:03:37 DEBUG handling-request: (tsp C S L2030 I) * 10:03:43 DEBUG results: #f Regards, John Ralls On Aug 10, 2023, at 12:13, Kalpesh Patel mailto:kalpesh.pa...@usa.net> > wrote: John, Appreciate that details. That is going to be quite helpful. The difficult part in debugging this is that when I do query three times from the command line, it succeeds without any issue for me as well, but when I update prices from GNC, it fails two times and then succeeds third time consistently. So this is not an issue with throttling from TSP source which you validated as well. Is there more in terms of debugging this by any chance? Can I use STDERR and STDOUT from Perl to emanate messages to console but not interfere with what JSON comes back or butcher the response back to gnc-fq-helper so it doesn't bomb to run a full cycles from GNC to test? Any help in debugging is appreciated. Looks like I may need to debug the TSP.pm module under the hood to figure out what might be going on and most likely with the response coming back from the request to https://www.tsp.gov/data/fund-price-history.csv URI. -Original Message- From: john mailto:jra...@ceridwen.us> > Sent: Thursday, August 10, 2023 12:54 PM To: Kalpesh Patel mailto:kalpesh.pa...@usa.net> > Cc: gnucash-u...@lists.gnucash.org <mailto:gnucash-u...@lists.gnucash.org> Subject: Re: [GNC] Strange prices update issue (possibly with TSP) Kalpesh, It does indeed look like the tsp module failed to return anything in the second and third tries. You're correct that gnc:scm is the right logger for GnuCash 4. gnc:price-quotes works only in GnuCash 5. For GnuCash to process a partial result from F::Q, F::Q needs to produce a valid response for each symbol; the response includes a Success variable; GnuCash will process symbols with Success=1 and present a message box containing the symbols for which F::Q returned Success=0. In the two cases where the TSE module failed it returned something that gnc-fq-helper wasn't able to parse so it posted a Scheme #f (meaning false, the equivalent of Python's None type) and logged an "unknown" error. GnuCash 5 would do a better job passing on the error but whether it could process the quotes returned from the other sources would depend on the JSON emitted by F::Q. FWIW I tried your "tsp C S L2030 I" query three times from the command line just now and got three valid responses. You can tell GnuCash to write a specific file with the --logto=/path/to/file option. It will overwrite if you pass an existing path and the only way to prevent that would be to write a wrapper script in your favorite scripting language that does the filename substitution. Regards, John Ralls On Aug 10, 2023, at 07:50, Kalpesh Patel mailto:kalpesh.pa...@usa.net> > wrote: So I am seeing a strange issue updating prices on my system. For each try, every third one succeeds but other two fails and this is very consistent behavior all the time on my system. I am on Windows 11 and using GNC v4.14 (time to upgrade? There is a good reason why I need to stay there) with F::Q v1.57_02. I turned on debugging (via --log=gnc.price-quotes=debug --log=gnc.scm=debug; looks like latter one is applicable to GNC v4.14) and below is the output from the trace file. Am I correct to interpret that TSP source is failing two out of three times? I see result as being * 10:03:17 DEBUG results: #f which indicates it failed to retrieve prices for that source but other sources (currency, alphavantage, and yahoo json) are fine. That raises two questions: 1) Is prices update all or nothing operation, i.e., if t
Re: [GNC] Strange prices update issue (possibly with TSP)
Kalpesh, You should be able to print diagnostics using `print STDERR ...`. Part of the problem with price-quotes.scm's error handling was that it didn't even look at gnc-fq-helper's stderr. The order in the tracefile you posted showed that tsp returned prices on the *first* try and failed on the subsequent two: > * 10:02:36 DEBUG handling-request: (tsp C S L2030 I) > > * 10:02:42 DEBUG results: ((C (symbol . C) (gnc:time-no-zone . > 2023-08-09 12:00:00) (last . 691951/1) (currency . USD)) (S (symbol . S) > (gnc:time-no-zone . 2023-08-09 12:00:00) (last . 704343/1) (currency . > USD)) (L2030 (symbol . L2030) (gnc:time-no-zone . 2023-08-09 12:00:00) (last > . 53747/1250) (currency . USD)) (I (symbol . I) (gnc:time-no-zone . > 2023-08-09 12:00:00) (last . 380083/1) (currency . USD))) [snip] > > * 10:03:11 DEBUG handling-request: (tsp C S L2030 I) > > * 10:03:17 DEBUG results: #f [snip] > * 10:03:37 DEBUG handling-request: (tsp C S L2030 I) > > * 10:03:43 DEBUG results: #f Regards, John Ralls > On Aug 10, 2023, at 12:13, Kalpesh Patel wrote: > > John, > > Appreciate that details. That is going to be quite helpful. > > The difficult part in debugging this is that when I do query three times > from the command line, it succeeds without any issue for me as well, but > when I update prices from GNC, it fails two times and then succeeds third > time consistently. So this is not an issue with throttling from TSP source > which you validated as well. Is there more in terms of debugging this by any > chance? Can I use STDERR and STDOUT from Perl to emanate messages to console > but not interfere with what JSON comes back or butcher the response back to > gnc-fq-helper so it doesn't bomb to run a full cycles from GNC to test? Any > help in debugging is appreciated. > > Looks like I may need to debug the TSP.pm module under the hood to figure > out what might be going on and most likely with the response coming back > from the request to https://www.tsp.gov/data/fund-price-history.csv URI. > > -Original Message- > From: john > Sent: Thursday, August 10, 2023 12:54 PM > To: Kalpesh Patel > Cc: gnucash-u...@lists.gnucash.org > Subject: Re: [GNC] Strange prices update issue (possibly with TSP) > > Kalpesh, > > It does indeed look like the tsp module failed to return anything in the > second and third tries. > > You're correct that gnc:scm is the right logger for GnuCash 4. > gnc:price-quotes works only in GnuCash 5. > > For GnuCash to process a partial result from F::Q, F::Q needs to produce a > valid response for each symbol; the response includes a Success variable; > GnuCash will process symbols with Success=1 and present a message box > containing the symbols for which F::Q returned Success=0. In the two cases > where the TSE module failed it returned something that gnc-fq-helper wasn't > able to parse so it posted a Scheme #f (meaning false, the equivalent of > Python's None type) and logged an "unknown" error. GnuCash 5 would do a > better job passing on the error but whether it could process the quotes > returned from the other sources would depend on the JSON emitted by F::Q. > > FWIW I tried your "tsp C S L2030 I" query three times from the command line > just now and got three valid responses. > > You can tell GnuCash to write a specific file with the --logto=/path/to/file > option. It will overwrite if you pass an existing path and the only way to > prevent that would be to write a wrapper script in your favorite scripting > language that does the filename substitution. > > Regards, > John Ralls > > >> On Aug 10, 2023, at 07:50, Kalpesh Patel wrote: >> >> So I am seeing a strange issue updating prices on my system. For each >> try, every third one succeeds but other two fails and this is very >> consistent behavior all the time on my system. I am on Windows 11 and >> using GNC v4.14 (time to upgrade? There is a good reason why I need to >> stay there) with F::Q v1.57_02. I turned on debugging (via >> --log=gnc.price-quotes=debug --log=gnc.scm=debug; looks like latter >> one is applicable to GNC v4.14) and below is the output from the trace >> file. Am I correct to interpret that TSP source is failing two out of >> three times? I see result as being * 10:03:17 DEBUG results: >> #f which indicates it failed to retrieve prices for that source but >> other sources (currency, alphavantage, and yahoo json) are fine. >> >> >> >> That raises two questions: >> >> 1) Is prices update all or nothing operation, i.e., if there are >> multiple sources and f
Re: [GNC] Strange prices update issue (possibly with TSP)
John, Appreciate that details. That is going to be quite helpful. The difficult part in debugging this is that when I do query three times from the command line, it succeeds without any issue for me as well, but when I update prices from GNC, it fails two times and then succeeds third time consistently. So this is not an issue with throttling from TSP source which you validated as well. Is there more in terms of debugging this by any chance? Can I use STDERR and STDOUT from Perl to emanate messages to console but not interfere with what JSON comes back or butcher the response back to gnc-fq-helper so it doesn't bomb to run a full cycles from GNC to test? Any help in debugging is appreciated. Looks like I may need to debug the TSP.pm module under the hood to figure out what might be going on and most likely with the response coming back from the request to https://www.tsp.gov/data/fund-price-history.csv URI. -Original Message- From: john Sent: Thursday, August 10, 2023 12:54 PM To: Kalpesh Patel Cc: gnucash-u...@lists.gnucash.org Subject: Re: [GNC] Strange prices update issue (possibly with TSP) Kalpesh, It does indeed look like the tsp module failed to return anything in the second and third tries. You're correct that gnc:scm is the right logger for GnuCash 4. gnc:price-quotes works only in GnuCash 5. For GnuCash to process a partial result from F::Q, F::Q needs to produce a valid response for each symbol; the response includes a Success variable; GnuCash will process symbols with Success=1 and present a message box containing the symbols for which F::Q returned Success=0. In the two cases where the TSE module failed it returned something that gnc-fq-helper wasn't able to parse so it posted a Scheme #f (meaning false, the equivalent of Python's None type) and logged an "unknown" error. GnuCash 5 would do a better job passing on the error but whether it could process the quotes returned from the other sources would depend on the JSON emitted by F::Q. FWIW I tried your "tsp C S L2030 I" query three times from the command line just now and got three valid responses. You can tell GnuCash to write a specific file with the --logto=/path/to/file option. It will overwrite if you pass an existing path and the only way to prevent that would be to write a wrapper script in your favorite scripting language that does the filename substitution. Regards, John Ralls > On Aug 10, 2023, at 07:50, Kalpesh Patel wrote: > > So I am seeing a strange issue updating prices on my system. For each > try, every third one succeeds but other two fails and this is very > consistent behavior all the time on my system. I am on Windows 11 and > using GNC v4.14 (time to upgrade? There is a good reason why I need to > stay there) with F::Q v1.57_02. I turned on debugging (via > --log=gnc.price-quotes=debug --log=gnc.scm=debug; looks like latter > one is applicable to GNC v4.14) and below is the output from the trace > file. Am I correct to interpret that TSP source is failing two out of > three times? I see result as being * 10:03:17 DEBUG results: > #f which indicates it failed to retrieve prices for that source but > other sources (currency, alphavantage, and yahoo json) are fine. > > > > That raises two questions: > > 1) Is prices update all or nothing operation, i.e., if there are > multiple sources and for those ones that succeed, does GNC still > update its prices database from sources that succeeded? > > 2) Is there a way to peg the debug file name, possibly with a fully > qualified path, and whether to tell it to overwrite or append if > pegged name one exist? > > > > > > > > * 10:01:46 DEBUG rpt-subdir=gnucash/reports/standard > > * 10:01:46 DEBUG mod-dir=c:\Program Files > (x86)\gnucash\share/guile/site/2.2\gnucash/reports/standard > > * 10:01:46 DEBUG dir-files=(view-column trial-balance > transaction taxinvoice register reconcile-report receivables receipt > price-scatter portfolio payables owner-report new-owner-report > new-aging net-charts lot-viewer job-report invoice income-statement > income-gst-statement ifrs-cost-basis general-ledger general-journal > equity-statement dashboard customer-summary category-barchart > cashflow-barchart cash-flow budget budget-income-statement budget-flow > budget-barchart budget-balance-sheet balsheet-pnl balsheet-eg > balance-sheet balance-forecast advanced-portfolio account-summary > account-piecharts) > > * 10:01:46 DEBUG rpt-subdir=gnucash/reports/example > > * 10:01:46 DEBUG mod-dir=c:\Program Files > (x86)\gnucash\share/guile/site/2.2\gnucash/reports/example > > * 10:01:46 DEBUG dir-files=(welcome-to-gnucash sample-graphs > hello-world daily-reports average-balance) > > * 10:01:46 DEBUG
Re: [GNC] Strange prices update issue (possibly with TSP)
Kalpesh, It does indeed look like the tsp module failed to return anything in the second and third tries. You're correct that gnc:scm is the right logger for GnuCash 4. gnc:price-quotes works only in GnuCash 5. For GnuCash to process a partial result from F::Q, F::Q needs to produce a valid response for each symbol; the response includes a Success variable; GnuCash will process symbols with Success=1 and present a message box containing the symbols for which F::Q returned Success=0. In the two cases where the TSE module failed it returned something that gnc-fq-helper wasn't able to parse so it posted a Scheme #f (meaning false, the equivalent of Python's None type) and logged an "unknown" error. GnuCash 5 would do a better job passing on the error but whether it could process the quotes returned from the other sources would depend on the JSON emitted by F::Q. FWIW I tried your "tsp C S L2030 I" query three times from the command line just now and got three valid responses. You can tell GnuCash to write a specific file with the --logto=/path/to/file option. It will overwrite if you pass an existing path and the only way to prevent that would be to write a wrapper script in your favorite scripting language that does the filename substitution. Regards, John Ralls > On Aug 10, 2023, at 07:50, Kalpesh Patel wrote: > > So I am seeing a strange issue updating prices on my system. For each try, > every third one succeeds but other two fails and this is very consistent > behavior all the time on my system. I am on Windows 11 and using GNC v4.14 > (time to upgrade? There is a good reason why I need to stay there) with F::Q > v1.57_02. I turned on debugging (via --log=gnc.price-quotes=debug > --log=gnc.scm=debug; looks like latter one is applicable to GNC v4.14) and > below is the output from the trace file. Am I correct to interpret that TSP > source is failing two out of three times? I see result as being * 10:03:17 > DEBUG results: #f which indicates it failed to retrieve prices for > that source but other sources (currency, alphavantage, and yahoo json) are > fine. > > > > That raises two questions: > > 1) Is prices update all or nothing operation, i.e., if there are > multiple sources and for those ones that succeed, does GNC still update its > prices database from sources that succeeded? > > 2) Is there a way to peg the debug file name, possibly with a fully > qualified path, and whether to tell it to overwrite or append if pegged name > one exist? > > > > > > > > * 10:01:46 DEBUG rpt-subdir=gnucash/reports/standard > > * 10:01:46 DEBUG mod-dir=c:\Program Files > (x86)\gnucash\share/guile/site/2.2\gnucash/reports/standard > > * 10:01:46 DEBUG dir-files=(view-column trial-balance transaction > taxinvoice register reconcile-report receivables receipt price-scatter > portfolio payables owner-report new-owner-report new-aging net-charts > lot-viewer job-report invoice income-statement income-gst-statement > ifrs-cost-basis general-ledger general-journal equity-statement dashboard > customer-summary category-barchart cashflow-barchart cash-flow budget > budget-income-statement budget-flow budget-barchart budget-balance-sheet > balsheet-pnl balsheet-eg balance-sheet balance-forecast advanced-portfolio > account-summary account-piecharts) > > * 10:01:46 DEBUG rpt-subdir=gnucash/reports/example > > * 10:01:46 DEBUG mod-dir=c:\Program Files > (x86)\gnucash\share/guile/site/2.2\gnucash/reports/example > > * 10:01:46 DEBUG dir-files=(welcome-to-gnucash sample-graphs > hello-world daily-reports average-balance) > > * 10:01:46 DEBUG rpt-subdir=gnucash/reports/locale-specific/us > > * 10:01:46 DEBUG mod-dir=c:\Program Files > (x86)\gnucash\share/guile/site/2.2\gnucash/reports/locale-specific/us > > * 10:01:46 DEBUG dir-files=(taxtxf) > > * 10:01:46 DEBUG rpt-subdir=gnucash/report/stylesheets > > * 10:01:46 DEBUG mod-dir=c:\Program Files > (x86)\gnucash\share/guile/site/2.2\gnucash/report/stylesheets > > * 10:01:46 DEBUG dir-files=(plain head-or-tail footer css) > > * 10:01:53 DEBUG gnc:fq-check-sources results: (1.57_02 aex > alphavantage amfiindia asegr asx aufunds australia bamosz bet bloomberg > bourso bse bseindia bvb canada canadamutual comdirect cse deka dutch > dwsfunds europe fetch_live_currencies fidelity fidelity_direct finanzpartner > fondsweb fool france ftfunds fundata fundlibrary goldmoney googleweb greece > hu hufund hungary hustock iexcloud india indiamutual known_currencies > marketwatch morningstarau morningstarch morningstarjp morningstaruk mstaruk > nasdaq nseindia nyse nzx onvista oslobors romania seb_funds sinvestor six > tesouro_direto tiaacref tmx tradegate tradeville treasurydirect troweprice > troweprice_direct tsp tsx twelvedata ukfunds unionfunds usa xetra > yahoo_chart yahoo_json yahooweb za) > > * 10:01:53 MESSG Found Finance::Quote version 1.57_02 > > * 10:02:36 DEBUG ALPHAVANTAGE_API_KEY=NOTAREALAPIKEY > > * 1
[GNC] Strange prices update issue (possibly with TSP)
So I am seeing a strange issue updating prices on my system. For each try, every third one succeeds but other two fails and this is very consistent behavior all the time on my system. I am on Windows 11 and using GNC v4.14 (time to upgrade? There is a good reason why I need to stay there) with F::Q v1.57_02. I turned on debugging (via --log=gnc.price-quotes=debug --log=gnc.scm=debug; looks like latter one is applicable to GNC v4.14) and below is the output from the trace file. Am I correct to interpret that TSP source is failing two out of three times? I see result as being * 10:03:17 DEBUG results: #f which indicates it failed to retrieve prices for that source but other sources (currency, alphavantage, and yahoo json) are fine. That raises two questions: 1) Is prices update all or nothing operation, i.e., if there are multiple sources and for those ones that succeed, does GNC still update its prices database from sources that succeeded? 2) Is there a way to peg the debug file name, possibly with a fully qualified path, and whether to tell it to overwrite or append if pegged name one exist? * 10:01:46 DEBUG rpt-subdir=gnucash/reports/standard * 10:01:46 DEBUG mod-dir=c:\Program Files (x86)\gnucash\share/guile/site/2.2\gnucash/reports/standard * 10:01:46 DEBUG dir-files=(view-column trial-balance transaction taxinvoice register reconcile-report receivables receipt price-scatter portfolio payables owner-report new-owner-report new-aging net-charts lot-viewer job-report invoice income-statement income-gst-statement ifrs-cost-basis general-ledger general-journal equity-statement dashboard customer-summary category-barchart cashflow-barchart cash-flow budget budget-income-statement budget-flow budget-barchart budget-balance-sheet balsheet-pnl balsheet-eg balance-sheet balance-forecast advanced-portfolio account-summary account-piecharts) * 10:01:46 DEBUG rpt-subdir=gnucash/reports/example * 10:01:46 DEBUG mod-dir=c:\Program Files (x86)\gnucash\share/guile/site/2.2\gnucash/reports/example * 10:01:46 DEBUG dir-files=(welcome-to-gnucash sample-graphs hello-world daily-reports average-balance) * 10:01:46 DEBUG rpt-subdir=gnucash/reports/locale-specific/us * 10:01:46 DEBUG mod-dir=c:\Program Files (x86)\gnucash\share/guile/site/2.2\gnucash/reports/locale-specific/us * 10:01:46 DEBUG dir-files=(taxtxf) * 10:01:46 DEBUG rpt-subdir=gnucash/report/stylesheets * 10:01:46 DEBUG mod-dir=c:\Program Files (x86)\gnucash\share/guile/site/2.2\gnucash/report/stylesheets * 10:01:46 DEBUG dir-files=(plain head-or-tail footer css) * 10:01:53 DEBUG gnc:fq-check-sources results: (1.57_02 aex alphavantage amfiindia asegr asx aufunds australia bamosz bet bloomberg bourso bse bseindia bvb canada canadamutual comdirect cse deka dutch dwsfunds europe fetch_live_currencies fidelity fidelity_direct finanzpartner fondsweb fool france ftfunds fundata fundlibrary goldmoney googleweb greece hu hufund hungary hustock iexcloud india indiamutual known_currencies marketwatch morningstarau morningstarch morningstarjp morningstaruk mstaruk nasdaq nseindia nyse nzx onvista oslobors romania seb_funds sinvestor six tesouro_direto tiaacref tmx tradegate tradeville treasurydirect troweprice troweprice_direct tsp tsx twelvedata ukfunds unionfunds usa xetra yahoo_chart yahoo_json yahooweb za) * 10:01:53 MESSG Found Finance::Quote version 1.57_02 * 10:02:36 DEBUG ALPHAVANTAGE_API_KEY=NOTAREALAPIKEY * 10:02:36 DEBUG handling-request: (tsp C S L2030 I) * 10:02:42 DEBUG results: ((C (symbol . C) (gnc:time-no-zone . 2023-08-09 12:00:00) (last . 691951/1) (currency . USD)) (S (symbol . S) (gnc:time-no-zone . 2023-08-09 12:00:00) (last . 704343/1) (currency . USD)) (L2030 (symbol . L2030) (gnc:time-no-zone . 2023-08-09 12:00:00) (last . 53747/1250) (currency . USD)) (I (symbol . I) (gnc:time-no-zone . 2023-08-09 12:00:00) (last . 380083/1) (currency . USD))) * 10:02:42 DEBUG handling-request: (alphavantage BRK-B) * 10:02:43 DEBUG results: ((BRK-B (symbol . BRK-B) (gnc:time-no-zone . 2023-08-09 12:00:00) (last . 17901/50) (currency . USD))) * 10:02:43 DEBUG handling-request: (yahoo_json ^RUT IDCC ^VIX ^GDAXI NDAQ ^FTSE ^DJI ^N225 ARKF ^CMC200 ^FCHI ^GSPC ^IXIC ^TNX ^HSI ^RUT SLV ^XAX MET GLD ^NYA BHF FSPSX VFINX FSEVX VTIAX VIVAX MFEKX VEXAX VGPMX JNGLX VFSAX VISVX VWIGX JDSNX OIERX VFSVX VGSLX VISGX DODFX NAESX ASVDX VBTLX VMFXX VIAAX VIMAX OIEJX CXXRX FZILX FSIVX FPADX VVIAX VEXMX VSIAX SMVSX VFWAX VGENX VPMCX ODIIX FNILX VEMAX VGTSX VTSAX VEUSX VTSMX VDADX FSMAX VGSIX FPMAX VEIEX VSMAX VEURX VSGAX HACAX FZIPX FZROX ABNIX VBMFX FXAIX FUSVX SPAXX FSTVX VFIAX CSDIX VLCAX VWNFX MDIZX JSVUX PRNHX VEXPX FSKAX FTIHX VGHCX VLACX VGXRX VGRLX BK) * 10:02:54 DEBUG results: ((^RUT (symbol . ^RUT) (gnc:time-no-zone . 2023-08-10 12:00:00) (last . 9713861/5000) (currency . USD)) (IDCC (symbol . IDCC) (gnc:time-no-zone . 2023-08-10 12:00:00) (last . 1713/20) (currency . USD)) (^VIX (symbol . ^V
