Appreciate bringing up the updated F::Q to my attention. I'll re-approach in that direction.
I'll also make note of addressing it via GitHub issue tracker. -----Original Message----- From: John Ralls <jra...@ceridwen.us> Sent: Monday, August 14, 2023 8:30 PM To: Kalpesh Patel <kalpesh.pa...@usa.net> Cc: Gnucash Users <gnucash-user@gnucash.org>; Bruce Schuck <bsch...@asgard-systems.com> Subject: Re: [GNC] Strange prices update issue (possibly with TSP) Kalpesh, Bruce posted at https://github.com/finance-quote/finance-quote/issues/335#issuecomment-16764 43182: "While the fix did not make it into v1.58, you can download and install BPSCHUCK/Finance-Quote-1.58_01.tar.gz from CPAN for testing/verification. It also has what I hope is a fix for TSP. It was not returning the hash if the GET failed or if the body of the response produced a CSV file." That's the BSEIndia issue; https://github.com/finance-quote/finance-quote/issues/338 is the TSP one. If you find more problems with TSP (or anything else in F::Q) and want to help debug it the best way to communicate with Bruce and stay up to date would be to open an issue at https://github.com/finance-quote/finance-quote/issues/. You can post here too if you want, but a Github issue is a better place to put debugging results. Regards, John Ralls > On Aug 14, 2023, at 1:54 PM, Kalpesh Patel <kalpesh.pa...@usa.net> wrote: > > Some good find in debugging ... Strawberry Perl cannot resolve the FQDN name one out of three times for www.tsp.gov on my system (this might be obfuscating the client behavior in response to real backend issue). It might be that I am reaching servers that are in error state (rc = 500 which is Internal Server Error; my observation on F::Q could be cause of concern on amount of data being returned). Below is the output from the failed run: > > C:\Users\kalpesh\OneDrive\QuickenStuff\HELPERS>set DEBUG=1 > > C:\Users\kalpesh\OneDrive\QuickenStuff\HELPERS>test2.pl > > ### [Mon Aug 14 16:26:56 2023] YIND_URL_HEAD : 'https://query1.finance.yahoo.com/v8/finance/chart/' > ### [Mon Aug 14 16:26:56 2023] YIND_URL_TAIL : '?interval=1d&period1=1691440016&period2=1692044816' > > ### AlphaVantage->new args : {} > > ### COUNT_URL: 'http://www.panix.com/~hd-fxsts/finance-quote.html?tsp' > ### Code: '200' > > ### [Mon Aug 14 16:26:56 2023] url : 'https://www.tsp.gov/data/fund-price-history.csv?startdate=2023-08-07&enddat e=2023-08-14&Lfunds=1&InvFunds=1&download=1' > ### [Mon Aug 14 16:26:56 2023] reply: bless( { > ### _content => 'Can\'t connect to www.tsp.gov:443 (nodename nor servname provided, or not known) > > nodename nor servname provided, or not known at C:/Strawberry/perl/site/lib/LWP/Protocol/http.pm line 50. > ', > ### _headers => bless( { > ### '::std_case' => { > ### 'client-date' => 'Client-Date', > ### 'client-warning' => 'Client-Warning' > ### }, > ### 'client-date' => 'Mon, 14 Aug 2023 20:26:56 GMT', > ### 'client-warning' => 'Internal response', > ### 'content-type' => 'text/plain' > ### }, 'HTTP::Headers' ), > ### _msg => 'Can\'t connect to www.tsp.gov:443 (nodename nor servname provided, or not known)', > ### _rc => 500, > ### _request => bless( { > ### _content => '', > ### _headers => bless( { > ### 'user-agent' => 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Safari/537.36' > ### }, 'HTTP::Headers' ), > ### _method => 'GET', > ### _uri => bless( do{\(my $o = 'https://www.tsp.gov/data/fund-price-history.csv?startdate=2023-08-07&enddat e=2023-08-14&Lfunds=1&InvFunds=1&download=1')}, 'URI::https' ) > ### }, 'HTTP::Request' ) > ### }, 'HTTP::Response' ) > > C:\Users\kalpesh\OneDrive\QuickenStuff\HELPERS> > > My test file (test2.pl) is as follows: > > #test2.pl - start > > use Finance::Quote; > use Data::Dumper; > > $q = Finance::Quote->new; > > %info = $q->fetch('tsp', "L2030"); > > print Dumper(%info); > > # test2.pl - end > > > Interesting side notes for GNC (I am on Windows 11 with GNC v4.14 and now F::Q v1.58 so might have some subtle from it): > > I was able to put the module in debug mode (set DEBUG=1 in cmd.exe; of course install the Smart::Comment Perl module first) and have them write out the smart comment messages to cmd console as you can see from above when I run a small test file that I've created. However, when I lunch gnc in normal fashion from the same console where debugging is enabled, the screen does not display anything, not even the smart comment; it is almost like it is ignoring all env settings or not being passed to sub-shell. Windows is notorious for this so there may not be a "clean" way of doing so. There is a new window that pops up and then disappears at the start up (this process seems to be part normal start up - check if F::Q is there or not) and does not display anything also. I attempted to start gnc with debug (with --log=gnc.scm=debug option) as well but that also does not emit anything either. On the note of specifying a file name for logging (--logto= parameter), it does not create nor write any file anywhere on the file system when only a fixed file name is given. I took away that option and it was happy to write a file to %LOCALAPPDATA% path with gnucash.trace.<random_six_alphanumeric_characters>.log. > > > Interesting side notes for F::Q TSP module: > > When it does work properly, the return payload from HTTP seem to be hundreds of lines if not thousands ... Here is the snippet of successful run: > > C:\Users\kalpesh\OneDrive\QuickenStuff\HELPERS>test2.pl > > ### [Mon Aug 14 15:42:00 2023] YIND_URL_HEAD : 'https://query1.finance.yahoo.com/v8/finance/chart/' > ### [Mon Aug 14 15:42:00 2023] YIND_URL_TAIL : '?interval=1d&period1=1691437320&period2=1692042120' > > ### AlphaVantage->new args : {} > > ### COUNT_URL: 'http://www.panix.com/~hd-fxsts/finance-quote.html?tsp' > ### Code: '200' > > ### [Mon Aug 14 15:42:01 2023] url : 'https://www.tsp.gov/data/fund-price-history.csv?startdate=2023-08-07&enddat e=2023-08-14&Lfunds=1&InvFunds=1&download=1' > ### [Mon Aug 14 15:42:01 2023] reply: bless( { > ### _content => 'Date,L Income,L 2025,L 2030,L 2035,L 2040,L 2045,L 2050,L 2055,L 2060,L 2065,G Fund,F Fund,C Fund,S Fund,I Fund > 2023-08-11,24.0849,12.2327,42.9435,12.8255,48.3009,13.1625,28.7245,14. > 0971,14.0954,14.0934,17.6469,18.3534,69.1565,70.2391,37.9419 > 2023-08-10,24.1082,12.2494,43.0380,12.8567,48.4288,13.1999,28.8108,14. > 1442,14.1425,14.1406,17.6449,18.4205,69.2207,70.3124,38.2391 > 2023-08-09,24.1016,12.2437,42.9976,12.8439,48.3758,13.1845,28.7742,14. > 1155,14.1139,14.1120,17.6429,18.5294,69.1951,70.4343,38.0083 > 2023-08-08,24.1272,12.2634,43.1181,12.8835,48.5391,13.2323,28.8859,14. > 1834,14.1818,14.1799,17.6409,18.5100,69.6795,71.0740,38.0127 > 2023-08-07,24.1541,12.2847,43.2515,12.9273,48.7198,13.2853,29.0100,14. > 2616,14.2599,14.2580,17.6389,18.4477,69.9736,71.7269,38.2463 > 2023-08-04,24.1053,12.2504,43.0581,12.8645,48.4638,13.2110,28.8379,14. > 1583,14.1566,14.1547,17.6328,18.4709,69.3457,71.4617,38.0032 > 2023-08-03,24.1034,12.2517,43.0826,12.8723,48.4974,13.2209,28.8626,14. > 1833,14.1815,14.1796,17.6308,18.3188,69.7102,71.6307,37.8756 > 2023-08-02,24.1277,12.2685,43.1728,12.9023,48.6200,13.2568,28.9449,14. > 2243,14.2226,14.2207,17.6289,18.4426,69.8850,71.8330,38.0070 > 2023-08-01,24.2277,12.3420,43.6036,13.0440,49.2020,13.4269,29.3407,14. > 4570,14.4553,14.4534,17.6269,18.4987,70.8648,73.1095,38.7558 > 2023-07-31,24.2660,12.3694,43.7577,13.0949,49.4108,13.4880,29.4819,14. > 5344,14.5328,14.5310,17.6249,18.6043,71.0523,73.4061,39.1413 > 2023-07-28,24.2467,12.3571,43.6964,13.0751,49.3307,13.4647,29.4283,14. > 5045,14.5029,14.5010,17.6191,18.5820,70.9459,72.6930,39.1484 > 2023-07-27,24.1833,12.3118,43.4387,12.9905,48.9842,13.3635,29.1936,14. > 3691,14.3675,14.3656,17.6172,18.5164,70.2457,71.6543,38.8820 > 2023-07-26,24.2258,12.3417,43.6042,13.0453,49.2086,13.4293,29.3454,14. > 4499,14.4484,14.4465,17.6152,18.6653,70.6948,72.5871,38.9458 > 2023-07-25,24.2163,12.3359,43.5775,13.0365,49.1733,13.4189,29.3220,14. > 4395,14.4379,14.4361,17.6133,18.6144,70.7047,72.2457,38.9313 > 2023-07-24,24.1974,12.3223,43.4989,13.0110,49.0687,13.3885,29.2512,14. > 3963,14.3947,14.3929,17.6114,18.6334,70.5044,72.1292,38.7806 > 2023-07-21,24.1841,12.3139,43.4569,12.9977,49.0157,13.3735,29.2167,14. > 3747,14.3732,14.3714,17.6055,18.6571,70.2201,72.1244,38.8425 > 2023-07-20,24.1808,12.3123,43.4515,12.9961,49.0096,13.3719,29.2133,14. > 3737,14.3722,14.3704,17.6036,18.6455,70.1974,72.2722,38.8218 > 2023-07-19,24.2266,12.3452,43.6382,13.0578,49.2628,13.4460,29.3849,14. > 4692,14.4677,14.4659,17.6017,18.7521,70.6685,73.1168,39.0026 > 2023-07-18,24.2119,12.3358,43.5911,13.0423,49.2001,13.4277,29.3433,14. > 4485,14.4469,14.4451,17.5998,18.6938,70.5015,72.8049,39.0442 > 2023-07-17,24.1639,12.3012,43.3926,12.9773,48.9337,13.3500,29.1627,14. > 3422,14.3407,14.3389,17.5978,18.6769,70.0036,72.0267,38.7854 > 2023-07-14,24.1430,12.2877,43.3244,12.9552,48.8445,13.3242,29.1032,14. > 3088,14.3072,14.3055,17.5920,18.6543,69.7345,71.3406,38.8917 > 2023-07-13,24.1672,12.3051,43.4224,12.9878,48.9782,13.3634,29.1941,14. > 3581,14.3566,14.3548,17.5901,18.7318,69.8035,72.0356,39.0693 > 2023-07-12,24.0841,12.2462,43.0912,12.8792,48.5343,13.2341,28.8948,14. > 1877,14.1862,14.1844,17.5881,18.6216,69.2107,71.2102,38.4014 > 2023-07-11,24.0007,12.1877,42.7678,12.7731,48.1017,13.1082,28.6041,14. > 0255,14.0239,14.0220,17.5862,18.4682,68.7006,70.6988,37.6648 > 2023-07-10,23.9468,12.1490,42.5478,12.7011,47.8070,13.0223,28.4048,13. > 9094,13.9078,13.9059,17.5843,18.4385,68.2399,69.6977,37.3488 > 2023-07-07,23.9111,12.1253,42.4239,12.6607,47.6429,12.9745,28.2947,13. > 8488,13.8471,13.8453,17.5785,18.3742,68.0764,68.6900,37.2301 > 2023-07-06,23.8994,12.1170,42.3776,12.6455,47.5804,12.9561,28.2516,13. > 8230,13.8214,13.8195,17.5765,18.3788,68.2614,68.0180,37.0282 > [snip] > > > Three observation on F::Q: > > 1) This request downloads CSVs that goes back to 2003-05-31 (20+ years!) even though request is only for 7 days in the URL. It is like enddate= parameter is ignored or incorrect parameter passed. Possibly a bug in TSP.pm module? > 2) Is the anonymous data collection of modules utilized still being collected? Code execution is not harming anything if not being collected but this is more curiosity on my part for v1.58 which was released few days back. > 3) Looks like Smart::Comment when activated emits messages for all modules rather than the single one that is being instantiated for the source and called to action. > > On to debugging the Perl LWP module... > > > > > -----Original Message----- > From: Bruce Schuck <bsch...@asgard-systems.com> > Sent: Friday, August 11, 2023 9:40 PM > To: gnucash-user@gnucash.org > Cc: Kalpesh Patel <kalpesh.pa...@usa.net>; john <jra...@ceridwen.us> > Subject: Re: [GNC] Strange prices update issue (possibly with TSP) > > On Aug 10, 2023, at 12:13, Kalpesh Patel <kalpesh.pa...@usa.net <mailto:kalpesh.pa...@usa.net> > wrote: > >> The difficult part in debugging this is that when I do query three >> times from the command line, it succeeds without any issue for me as >> well, but when I update prices from GNC, it fails two times and then >> succeeds third time consistently. So this is not an issue with >> throttling from TSP source which you validated as well. Is there more >> in terms of debugging this by any chance? Can I use STDERR and STDOUT >> from Perl to emanate messages to console but not interfere with what >> JSON comes back or butcher the response back to gnc-fq-helper so it >> doesn't bomb to run a full cycles from GNC to test? Any help in debugging is appreciated. > > Kalpesh, > > Many, but not all, of the F::Q modules make use of the Smart::Comments module. TSP.pm is one of them. The environment variable DEBUG will need to be set, and with no changes to the Perl code you should see helpful information printed to STDERR. In the code, these are lines that begin with "###" (note, just 3 pound signs) in those modules using Smart::Comments. > > A quick look at the TSP module and I see two where the "tsp" method will return *without* setting "success" and "errormsg" for any of the symbols passed to it. This will happen if the http get call fails, or whatever is downloaded is not 2 or more lines. > > my $reply = $ua->get($url, @HEADERS); > ### [<now>] url : $url > ### [<now>] reply: $reply > > return unless ($reply->is_success); > > my @line = split(/\n/, $reply->content); > > return unless (@line > 1); > > In the code snippet, there are two examples Smart::Comments use which could be useful in your case. With DEBUG set the code would print the url to STDERR, and entire $reply object (status, headers, cookies, body). > > Whether or not this helps you, the omission of not returning the proper failure data is an issue. > > Hope this helps. > > Bruce S > _______________________________________________ gnucash-user mailing list gnucash-user@gnucash.org To update your subscription preferences or to unsubscribe: https://lists.gnucash.org/mailman/listinfo/gnucash-user ----- Please remember to CC this list on all your replies. You can do this by using Reply-To-List or Reply-All.
