Re: [GNC-dev] Robust error handling and respecting the Apha Vantage API limits in gnc-fq-helper.
On Wed, 12 Feb 2020 at 07:44, Mike Alexander wrote:
> On 11 Feb 2020, at 3:59, Edward d'Auvergne wrote:
> > F::Q uses this URL for all queries:
> >
> > https://www.alphavantage.co/query?function=GLOBAL_QUOTE=json=IDRUSD=$MY_KEY
> >
> > The result I see is:
> >
> > {
> > "Global Quote": {
> > "01. symbol": "IDRUSD",
> > "02. open": "0.0001",
> > "03. high": "0.0001",
> > "04. low": "0.0001",
> > "05. price": "0.0001",
> > "06. volume": "0",
> > "07. latest trading day": "2020-02-10",
> > "08. previous close": "0.0001",
> > "09. change": "0.",
> > "10. change percent": "0.%"
> > }
> > }
>
> It doesn't use that URL for currency exchange rates. It uses
>
> https://www.alphavantage.co/query?function=CURRENCY_EXCHANGE_RATE_currency=IDR_currency=USD=xxx
>
> which returns
>
> {
> "Realtime Currency Exchange Rate": {
> "1. From_Currency Code": "IDR",
> "2. From_Currency Name": "Indonesian Rupiah",
> "3. To_Currency Code": "USD",
> "4. To_Currency Name": "United States Dollar",
> "5. Exchange Rate": "0.7297",
> "6. Last Refreshed": "2020-02-12 06:30:18",
> "7. Time Zone": "UTC",
> "8. Bid Price": "-",
> "9. Ask Price": "-"
> }
> }
I see. The duplication of code in the 'currency' subroutine in
lib/Finance/Quote.pm and the 'alphavantage' sobroutine in
lib/Finance/Quote/AlphaVantage.pm is confusing. The first uses
CURRENCY_EXCHANGE_RATE and the second uses GLOBAL_QUOTE. I wonder why
the 'alphavantage' subroutine was not modified, or split to add a new
'alphavantage_currency' subroutine, and called from 'currency'? The
'sleep_before_query' function which uses the variable:
"""
my $maxQueries = { quantity =>5 , seconds => 60}; # no more than x
# queries per y
# seconds, based on
#
https://www.alphavantage.co/support/#support
"""
seems to be more robust than how 'currency' handles things (this seems
to be the major algorithmic difference between the two). This could
even shift into searching for an ALPHAVANTAGE_QUERIES_PER_MIN
environmental variable to allow those with an Alpha Vantage
subscription to fetch quotes faster. In the 'alphavantage' function,
the sleep if the query returns a 'Note' logic seems to be a final
fallback mechanism if the 'sleep_before_query' function fails, whereas
in 'currency' this is the main mechanism for dealing with the API
fetching limits.
It looks like some but not all parts of the 'alphavantage' function
were copied into 'currency'. But if multiple currency sources are
made available, it would make more sense to make the 'currency'
subrountine provider independent and defer the fetching to subroutines
in the lib/Finance/Quote/*.pm modules.
> > It looks like this code then kicks in, from the "currency" subroutine
> > in lib/Finance/Quote.pm:
> >
> > """
> > if ( $exchange_rate < 0.001 ) {
> > # exchange_rate is too little. we'll get more accuracy by using
> > # the inverse rate and inverse it
> > my $inverse_rate = $this->currency( $to, $from );
> > {
> > local $^W = 0;
> > return undef unless ( $exchange_rate + 0 );
> > }
> > $exchange_rate = int( 1 / $inverse_rate + .5 ) /
> > 1;
> > }
> > """
> >
>
> That code should be removed. It actually gets the same accuracy and it
> gets a different answer. The reverse quote is 13683.5. This inverts to
> a rate of 0.7308 which is not the same as 0.7297. That's a
> trivial difference, but F::Q should really return the quote it was asked
> for if possible.
Maybe it would be needed if other quote providers are added in the
future? Maybe it should be converted into a function that checks the
number of significant figures and asks for the inverse if <= 2?
> I noticed that there was some action on my pull request after you
> rattled their cage. Thanks for doing that. That may make this
> discussion largely irrelevant.
No problems. Hopefully this will be fixed and Finance:Quote v1.50
released soon. Then GnuCash will finally be able to fetch quotes
correctly after 2 long years!
Regards,
Edward
___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: [GNC-dev] Robust error handling and respecting the Apha Vantage API limits in gnc-fq-helper.
On Tue, 11 Feb 2020 at 09:59, Edward d'Auvergne wrote:
> On Mon, 10 Feb 2020 at 22:28, Mike Alexander wrote:
> > By the way, I didn't mean to suggest that we should encourage all users of
> > GnuCash to patch F::Q. I still remain hopeful that the maintainers will
> > merge my pull request that fixes this bug. In the meantime switching to the
> > quote API might be a good idea even if it does have issues. The code would
> > have to be smart enough to work around the problems. For example if the
> > quote API fails for a given currency it could try the currency API. If the
> > returned value appears to have lost all precision then it could try the
> > reverse quote and compute the reciprocal itself.
>
> I finally found your pull request:
> https://github.com/finance-quote/finance-quote/pull/130. Your
> changes, that were rejected, allow my previous script [1] to
> successfully fetch the quotes in serial [2]. I'll try to create a
> pure perl script to demonstrate that your bug fix is a proper bug fix,
> and that they should reopen your pull request, merge it, and then
> release F::Q v1.50! If that works, then there wouldn't be any changes
> required on the GnuCash side. Well, apart from the error handling to
> prevent GnuCash from segfaulting.
I've now created a script to demonstrate your fix in the pull request
[1]. On the master branch of F::Q, this shows the failure [2]. After
cherry-picking your commit 877c87bd3ea1cbd403c59dfcd4a58b97c14d232a,
most runs pass [3]. This does not include the "USD" "XAU" pair which
needs to be inverted. Maybe the inversion should be directly
performed in F::Q rather than GnuCash? Anyway, I'll post this to the
pull request and ask for it to be reopened.
Although most runs pass, I have noticed that some do not [4]. Have
you seen that behaviour or know what might be happening?
Regards,
Edward
[1] Script to demonstrate the F::Q currency fetching failures:
"""
#!/usr/bin/env perl
use Finance::Quote;
use Time::HiRes qw(time);
print("Using Perl $^V\n");
print("Using Finance::Quote $Finance::Quote::VERSION\n\n");
# The currencies to fetch.
my @symbols = (
"AUD",
"BRL",
"CAD",
"CHF",
"CNY",
"CZK",
"DKK",
"EUR",
"GBP",
"HKD",
"HRK",
"ILS",
"JPY",
"KRW",
"MYR",
"NOK",
"NZD",
"RUB",
"SGD",
"TWD",
"XAU"
);
# Fetch the currencies.
my $begin = time();
my $quoter = Finance::Quote->new();
foreach (@symbols) {
$price = $quoter->currency("USD", $_);
if (!defined $price) {
$price = "Failed.";
}
print("USD$_: $price\n");
}
my $end = time();
printf("Fetching time %0.02f s\n", $end-$begin);
"""
[2] Failure of F::Q at bcfc947ee0173e059573236c2cf4a9bcb9a09b65:
"""
Using Perl v5.30.1
Using Finance::Quote 1.50
USDAUD: 1.4909
USDBRL: 4.325
USDCAD: 1.3294
USDCHF: 0.9778
USDCNY: 6.9745
USDCZK: Failed.
USDDKK: Failed.
USDEUR: Failed.
USDGBP: Failed.
USDHKD: Failed.
USDHRK: Failed.
USDILS: Failed.
USDJPY: Failed.
USDKRW: Failed.
USDMYR: Failed.
USDNOK: Failed.
USDNZD: Failed.
USDRUB: Failed.
USDSGD: Failed.
USDTWD: Failed.
USDXAU: Failed.
Fetching time 12.44 s
"""
[3] Passing run after cherry-picking 877c87bd3ea1cbd403c59dfcd4a58b97c14d232a:
"""
Using Perl v5.30.1
Using Finance::Quote 1.50
USDAUD: 1.4907
USDBRL: 4.3242
USDCAD: 1.3293
USDCHF: 0.9781
USDCNY: 6.9745
USDCZK: 22.833
USDDKK: 6.8465
USDEUR: 0.9162
USDGBP: 0.7734
USDHKD: 7.7646
USDHRK: 6.8309
USDILS: 3.4163
USDJPY: 109.85
USDKRW: 1183.9
USDMYR: 4.134
USDNOK: 9.2539
USDNZD: 1.5643
USDRUB: 63.644
USDSGD: 1.3872
USDTWD: 30.034
USDXAU: Failed.
Fetching time 219.69 s
""'
[4] Failing run after cherry-picking 877c87bd3ea1cbd403c59dfcd4a58b97c14d232a:
"""
Using Perl v5.30.1
Using Finance::Quote 1.50
USDAUD: 1.4907
USDBRL: 4.3229
USDCAD: 1.3293
USDCHF: 0.9778
USDCNY: 6.9745
USDCZK: 22.836
USDDKK: 6.8453
USDEUR: 0.9161
USDGBP: 0.7735
USDHKD: 7.7646
USDHRK: 6.8309
USDILS: Failed.
USDJPY: 109.84
USDKRW: 1184
USDMYR: 4.134
USDNOK: 9.2512
USDNZD: 1.5639
USDRUB: 63.716
USDSGD: 1.3872
USDTWD: 30.03
USDXAU: Failed.
Fetching time 167.91 s
"""
___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: [GNC-dev] Robust error handling and respecting the Apha Vantage API limits in gnc-fq-helper.
On Mon, 10 Feb 2020 at 22:28, Mike Alexander wrote:
> On 10 Feb 2020, at 6:27, Edward d'Auvergne wrote:
>> I realise that F::Q is broken in some places. As I said, I reported
>> an issue. However, as you have seen yourself, the F::Q developers are
>> simply non-responsive. I believe that asking all GnuCash users to
>> manually patch their own Finance::Quote perl modules to get around
>> this 2 year old issue is too much to ask of users. My reasoning is
>> that a little more logic on the GnuCash side to handle F::Q
>> misbehaving is easier for all GnuCash users. And, from the behaviour
>> of the scripts below, I now wonder if GnuCash could benefit by
>> switching from piping '(currency "USD" "EUR")' into gnc-fc-helper to
>> instead piping in '(alphavantage "USDEUR")'? The only disadvantage is
>> that gold and silver quotes do not work via the fetch() interface with
>> Alpha Vantage.
>
> I think I understand now why we've been talking past each other. There isn't
> any API in F::Q to fetch multiple currency quotes in one call. Instead it's a
> feature of AlphaVantage to treat exchange rates as if they were stock quotes
> and use the stock quote API to fetch them. I wasn't aware that this existed.
I thought this was also the case with the old Yahoo API. Maybe I am
confusing this with the original Finance::YahooQuote perl module that
does not separate currencies from other symbols?
> I played with it for a few minutes and see a couple of problems with it. Some
> currencies are not available using the stock quote API but are available
> using the currency API. For example three in my list of currencies which
> don't work with the quote API are MNT, SBD, and VUV. These all work fine with
> the currency API.
>
> The other problem is that for very small exchange rates the stock quote
> interface loses precision. For example, compare
>
> ./gnc-fq-dump alphavantage IDRUSD
> Finance::Quote fields Gnucash uses:
> symbol: IDRUSD <=== required
> date: 02/10/2020 <=== recommended
> currency: USD <=== required
> last: 0.0001 <=\
> nav: <=== one of these
> price: <=/
> timezone: <=== optional
>
> and
>
> ./gnc-fq-dump currency IDR USD
> 1 IDR = 7.295e-05 USD
>
> These problems would make the quote API for currencies less useful.
F::Q uses this URL for all queries:
https://www.alphavantage.co/query?function=GLOBAL_QUOTE=json=IDRUSD=$MY_KEY
The result I see is:
{
"Global Quote": {
"01. symbol": "IDRUSD",
"02. open": "0.0001",
"03. high": "0.0001",
"04. low": "0.0001",
"05. price": "0.0001",
"06. volume": "0",
"07. latest trading day": "2020-02-10",
"08. previous close": "0.0001",
"09. change": "0.",
"10. change percent": "0.%"
}
}
It looks like this code then kicks in, from the "currency" subroutine
in lib/Finance/Quote.pm:
"""
if ( $exchange_rate < 0.001 ) {
# exchange_rate is too little. we'll get more accuracy by using
# the inverse rate and inverse it
my $inverse_rate = $this->currency( $to, $from );
{
local $^W = 0;
return undef unless ( $exchange_rate + 0 );
}
$exchange_rate = int( 1 / $inverse_rate + .5 ) / 1;
}
"""
The "fetch" subroutine simply does not do that. I guess you would
first have to identify that the symbol is a currency pair, then invert
it. Not too difficult. But then it comes back to the issue of
fixing/modifying F::Q itself.
> By the way, I didn't mean to suggest that we should encourage all users of
> GnuCash to patch F::Q. I still remain hopeful that the maintainers will merge
> my pull request that fixes this bug. In the meantime switching to the quote
> API might be a good idea even if it does have issues. The code would have to
> be smart enough to work around the problems. For example if the quote API
> fails for a given currency it could try the currency API. If the returned
> value appears to have lost all precision then it could try the reverse quote
> and compute the reciprocal itself.
I finally found your pull request:
https://github.com/finance-quote/finance-quote/pull/130. Your
changes, that were rejected, allow my previous script [1] to
successfully fetch the quotes in serial [2]. I'll try to create a
pure perl script to demonstrate that your bug fix is a proper bug fix,
and that they should reopen your pull request, merge it, and then
release F::Q v1.50! If that works, then there wouldn't be any changes
re
Re: [GNC-dev] Robust error handling and respecting the Apha Vantage API limits in gnc-fq-helper.
On Sun, 9 Feb 2020 at 23:41, Mike Alexander wrote:
> You don't seem to be following what I'm saying.
I thought I was ;)
> First, the delay when currency quotes exceed the AlphaVantage rate limit is
> in Finance::Quote itself, not in any code distributed as part of GnuCash.
> You're right that GnuCash asks for currency exchange rates as fast as
> possible, but this is not relevant to what I'm saying. As of F::Q 1.49 the
> code in F::Q to rate limit currency exchange rate calls is broken. Earlier I
> gave a patch to fix this. If you haven't applied this patch to F::Q, GnuCash
> won't be able to retrieve more than a few currency exchange rates. Have you
> applied this patch?
>
> Second, you keep saying that GnuCash should ask for multiple currency quotes
> per call to F::Q. I don't know of any way to do this. I just read the F::Q
> documentation and code again and can't see any method that returns more than
> one currency exchange rate per call. How do you do this in F::Q? Is there
> some undocumented method that I am missing?
Firstly, I have posted a Perl script that demonstrates parallel vs.
serial F::Q calls [1]. The output of this script is at [2]. I now
realise that the F::Q currency() call that is currently used for
currencies in gnc-fc-helper does not have the same flexibility as the
fetch() call, so that only serial operation is possible. From the
output in [2], it also seems like the rate limiting bugs are only in
the currency() call and not in fetch(). That output shows that both
parallel and serial calls respect the Alpha Vantage API limits and
have about the same running time.
As a 2nd test, I wrote a shell script to directly call gnc-fc-helper
and trigger the fetch() rather than currency() calls [3]. With this,
I see that the fetch() call can be used successfully, both in parallel
and serial [4]. This causes F::Q to be called in exactly the same way
as in the perl script [1], with pretty much the same timings. GnuCash
is not affected by the current F::Q API limit bugs in this mode of
operation.
As a 3rd test, I have mimicked the GnuCash behaviour in a script
calling gnc-fc-helper to fetch via the F::Q currency() call [5]. This
clearly shows the F::Q bug that GnuCash has tripping up on for two
years now [6].
I realise that F::Q is broken in some places. As I said, I reported
an issue. However, as you have seen yourself, the F::Q developers are
simply non-responsive. I believe that asking all GnuCash users to
manually patch their own Finance::Quote perl modules to get around
this 2 year old issue is too much to ask of users. My reasoning is
that a little more logic on the GnuCash side to handle F::Q
misbehaving is easier for all GnuCash users. And, from the behaviour
of the scripts below, I now wonder if GnuCash could benefit by
switching from piping '(currency "USD" "EUR")' into gnc-fc-helper to
instead piping in '(alphavantage "USDEUR")'? The only disadvantage is
that gold and silver quotes do not work via the fetch() interface with
Alpha Vantage.
Regards,
Edward
P. S. For other wanting to test this, note that you need to have a
free key from Alpha Vantage and set the environmental variable
ALPHAVANTAGE_API_KEY to that key. As a side note, I chased down the
current currency fetching logic to this commit
(https://github.com/Gnucash/gnucash/commit/4fae9b):
"""
commit 4fae9be45ed5cc7faa4a3190b118aa47900e6e20 (HEAD)
Author: Dave Peticolas
Date: Tue May 8 09:36:40 2001 +
2001-05-08 Dave Peticolas
* src/scm/price-quotes.scm: add currency quote support
* src/gnome/dialog-account.c: add currency quote support
* src/engine/Account.c: allow CURRENCY accounts to have price
source set.
* src/quotes/finance-quote-helper.in: add currency quote support
git-svn-id: svn+ssh://svn.gnucash.org/repo/gnucash/trunk@4128
57a11ea4-9604-0410-9ed3-97b8803252fd
"""
The logic has not changed significantly since then.
[1] The 'parallel_vs_serial.pl' Perl script demonstrating parallel vs.
serial calls to F::Q:
"""
#!/usr/bin/perl -w
use Finance::Quote;
use Time::HiRes qw(time);
print("Using Finance::Quote $Finance::Quote::VERSION\n");
# The currencies to fetch.
my @symbols = (
"USDAUD",
"USDBRL",
"USDCAD",
"USDCHF",
"USDCNY",
"USDCZK",
"USDDKK",
"USDEUR",
"USDGBP",
"USDHKD",
"USDHRK",
"USDILS",
"USDJPY",
"USDKRW",
"USDMYR",
"USDNOK",
"USDNZD",
"USDRUB",
"USDSGD",
"USDTWD",
"USDXAU"
);
# The labels to print out.
my @labels = (
"last",
"success",
"errormsg"
);
# Parallel execution.
my $begin_par = time();
my $quoter_par = Finance::Quote->new();
%data_par = $quoter_par->fetch("alphavantage", @symbols);
foreach my $symbol (@symbols) {
foreach my $label (@labels) {
if (exists $data_par{$symbol, $label}) {
print($symbol, " ", $label, ": ");
print($data_par{$symbol,
Re: [GNC-dev] Robust error handling and respecting the Apha Vantage API limits in gnc-fq-helper.
On Fri, 7 Feb 2020 at 05:50, Mike Alexander wrote: > > On 4 Feb 2020, at 6:21, Edward d'Auvergne wrote: > > That's a good point. In the future, F::Q might also be able to > respect the API fetching limits across multiple calls, and then this > would add an additional delay. That could be fixed by properly timing > this while loop. As for sleeping for different times in the loop for > those with an AlphaVantage subscription, how could that be done? Do > you know how F::Q knows about this? I can see that you can currently > buy 30, 120, 300, 600, or 1200 API requests per minute. Should there > be a GnuCash user setting for this, that is then used by the while > loop in this perl script? > > I don't think this should be a GNuCash configuration option. Right now > AlphaVantage tells the caller if too many requests have been made in too > short a time, but it doesn't return information about how many are too many. > F::Q recognizes this error return and delays the next call. It is pretty > stupid about how long to delay. If AlphaVantage returned more information > about how fast is too fast it could be smarter about it. Unless this is > changed in AlphaVantage I don't think thee is much that could usefully be > done about this. That sounds like it would work, however I see quite a different behaviour with the debugging (in my first post) and from the code (gnucash/price-quotes.scm and libgnucash/quotes/gnc-fq-helper.in). What I see in both is that for each iteration of the while loop, F::Q is called once per currency. If the fetch fails, due to the Alphavantage API limits, the first result is #. This seems to cause the guile 'fdes' file descriptor to close. Hence all subsequent currencies are looped through without calling F::Q. There is no sleeping in this. F::Q 1.49 does not recognise serial calls. F::Q's delay is only invoked when a list of currencies is feed into it in a single call. But GnuCash clearly does not do this. In any case whether or not the pipe is closed, the loop runs at full Perl execution speed, limited solely by the IO. F::Q currently does not insert a sleep period into this loop. The result is that only the first few currencies that are within the Alphavantage API limit will be fetched, the rest will be skipped. From the debugging output, the currency skipping is very fast. This is exactly the behaviour I have always seen in GnuCash since the Yahoo API shutdown. Regards, Edward ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: [GNC-dev] Robust error handling and respecting the Apha Vantage API limits in gnc-fq-helper.
On Sun, 19 Jan 2020 at 06:24, David H wrote: > > Mike/Edward, I'm not sure if either of you have seen Vince's posting in > Finance-quote-devel which seems > like he's working on a similar thing? > > Copy below > > Cheers Dave H > > > Hi, > > In June, Erik sent a message about his idea for cleaning up the FQ API to > have consistent ways to set/get instance variables and expand new to allow > for named parameters and module specific data. > > One benefit is a sustainable way to enable users to register API keys for > specific modules. > > I just pushed the branch "new-new" to explore these ideas. > > See the file > https://raw.githubusercontent.com/finance-quote/finance-quote/new-new/new-API-notes.txt > for details about the set/get methods. > > > I used the new API to thread an API key to the IEXCloud module from the test > suite as an example: > https://raw.githubusercontent.com/finance-quote/finance-quote/new-new/t/iexcloud.t > > > If there is agreement to proceed with the API update there are a few things > to do and I can help: > > - updating the POD documentation for Quote.pm > - enabling AlphaVantage to use the module specific data as another source for > its API key > - renew discussion with gnucash developers on enabling users to specify API > keys for more modules I could be wrong, but looking at the 'new-API-notes.txt' file (link above) I have the impression that these changes do not fix the issue of multiple sequential calls to F::Q respecting the API request limits. As far as I can tell, it would allow the GnuCash perl script to be rewritten to eliminate the while loop and instead call F::Q with a long list of all symbols. But the GnuCash side will need to know the exact API request limit that the user has access to, and then pass that into F::Q. If this is the case, it would nevertheless require some UI feedback/warning on the GnuCash side as fetching many quotes via the GUI is going to be very, very slow. Regards, Edward ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: [GNC-dev] Robust error handling and respecting the Apha Vantage API limits in gnc-fq-helper.
On Sun, 19 Jan 2020 at 01:51, Mike Alexander wrote: > > I'm glad you're looking at this since it does need some work. However I think > there are a couple of things you're not aware of. I, too, have quite a ffew > currencies in my file (around 30) and by coincidence I was running a price > fetch in the background when I first saw your message. It completed > successfully after fetching all currency quotes as usual. It does not surprise me that there are multiple ways to fix F::Q. > You're right that GnuCash fetches one currency per call, but this is the only > API for currencies that F::Q provides. It fetches all the other quotes for a > given source in one call to F::Q. The F::Q currency quote code has a test for > the rate limiting message that AlphaVantage returns when currency quote > fetches come too fast. With my debugging code, this internal F::Q rate limit is visibly not being triggered. I would assume that is because the current GnuCach code is calling F::Q once per currency pair, rather than the usual one F::Q call with a list of currency pairs. > However at some time in the past AlphaVantage changed this message. I updated > F::Q to look for the new message and sent a pull request upstream. This pul > request has never been applied which means that the current F::Q does not > recognize when AlphaVantage is trying to tell it to slow down. My copy of > F::Q has this change which is probably why it works for me and not for you. As I said, the F::Q developers do not seem to be responsive. So I chose to fix things on the GnuCash side instead. > The patch is simple, just change it to look for a "Note" field in the > returned data instead of an "Information" field. > > Try this and see if it helps. I have my own F::Q fixes and patches, and it looks like your changes would work quite well. But again I am tackling this from the GnuCash side. > Your change to make GnuCash handle F::Q failures better seems like a good > idea. It's better than GnuCash crashing ;) That change should really be considered for 'master'. > Putting the delay on GnuCash's side of the API might also be a good idea, but > really shouldn't be necessary. It also would be a problem for someone who has > actually paid for an AlphaVantage subscription and hence can get currency > quotes faster. Some day I hope that F::Q will find a currency quote source > that works better and faster at which time you wouldn't want the delay in > GnuCash. That's a good point. In the future, F::Q might also be able to respect the API fetching limits across multiple calls, and then this would add an additional delay. That could be fixed by properly timing this while loop. As for sleeping for different times in the loop for those with an AlphaVantage subscription, how could that be done? Do you know how F::Q knows about this? I can see that you can currently buy 30, 120, 300, 600, or 1200 API requests per minute. Should there be a GnuCash user setting for this, that is then used by the while loop in this perl script? Regards, Edward ___ gnucash-devel mailing list gnucash-devel@gnucash.org https://lists.gnucash.org/mailman/listinfo/gnucash-devel
[GNC-dev] Robust error handling and respecting the Apha Vantage API limits in gnc-fq-helper.
Hello, I've decided to tackle an issue that has been bugging me for a couple of years now [1]. I travel a lot so I have accounts currently set up for 45 currencies (and 5 other commodities). Ever since the shift from the Yahoo finance APIs to Alpha Vantage in Finance::Quote, I have had to manually add exchange rates in Gnucash. So I have now tried to debug this myself [2]. The result of adding these scheme printouts [3] clearly, to me, shows what the problem is. There is an unhandled exception that sometimes occurs in Finance::Quote. This is because the Alpha Vantage API is not very reliable and does not always return an exchange rate. Running gnc-fg-helper manually shows that exception [4] (the same problem I reported as a bug upsteam to Finance::Quote [1]). This causes the guile 'fdes' file descriptor to close, hence all subsequent attempts at fetching a quote with gnc-fg-helper fails. The result is that only the top few commodities in the Gnucash database are updated - ever. For me that is between 3-6 currencies max. Those on the bottom of my list are absolutely never updated. There was zero movement on the part of the Finance::Quote developers, so I decided instead to try to make Gnucash more robust against F::Q issues. The F::Q exception was causing gnc-fg-helper to exit. In Gnucash 3.6 and earlier, this would simply cause the bottom of the currency list to never be updated. From Gnucash 3.7 to the current master commit from today, instead the Gnucash GUI would die. But another issue I noticed in the way that quotes are retrieved is that Gnucash is not respecting the Alpha Vantage API limits. Instead of calling F::Q once with all currency pairs - which would trigger F::Q's sleeping mechanism to only fetch max 5 commodities per minute (introduced in version 1.48) - Gnucash would rather fetch quotes individually from F::Q at the maximum rate within the gnc-fq-helper while loop. I.e. Gnucash would not sleep. Therefore I made two minor changes to gnc-fq-helper [5]. Firstly the while loop waits for 15 seconds at the end of each iteration (60 s / 5 = 12 s might be enough for the API limits though, as the fetching itself takes time). Secondly I added error handling for the F::Q call so that the loop will simply continue. There might be better ways to do this, but now fetching quotes finally works for me. Due to the unreliability of the Alpha Vantage API, not all currencies are updated. But fetching quotes multiple times, i.e. via a cron job and the --add-price-quotes option, results in almost all currencies being updated. I hope this would be of interest to the Gnucash developers. This does have the effect of making quote fetching from the GUI "Price Database" window quite slow, so a warning dialog would probably have to be introduced explaining that only 5 currency pairs can be fetched per minute, so the GUI will freeze for a while (a rough time could even be calculated). Regards, Edward [1] My bug reports: https://bugs.gnucash.org/show_bug.cgi?id=795564 https://rt.cpan.org/Public/Bug/Display.html?id=125310 [2] My terrible scheme debugging: """ diff --git a/gnucash/price-quotes.scm b/gnucash/price-quotes.scm index d905d9739..dd667ae5d 100644 --- a/gnucash/price-quotes.scm +++ b/gnucash/price-quotes.scm @@ -127,6 +127,13 @@ (lambda (request) (catch #t (lambda () + (display "handling-request: ") (display request) (newline) + (display #\() + (display (car request)) + (display " ") + (for-each write (cdr request)) + (display #\)) + (newline) (gnc:debug "handling-request: " request) ;; we need to display the first element (the method, ;; so it won't be quoted) and then write the rest @@ -141,6 +148,7 @@ (force-output))) (let ((results (read (fdes->inport (gnc-process-get-fd quoter 1) + (display "results: " ) (display results) (newline) (gnc:debug "results: " results) results)) (lambda (key . args) key))) @@ -448,7 +456,11 @@ Run 'gnc-fq-update' as root to install them."))) ((memq 'system-error fq-results) (set! keep-going? #f) - (show-error (N_ "There was a system error while retrieving the price quotes."))) + (display "fq-calls: ") (display fq-calls) (newline) + (display "fq-call-data: ") (display fq-call-data) (newline) + (display "fq-results: ") (display fq-results) (newline) + (display "problem-syms: ") (display (string-join problem-syms ", ")) (newline) + (display "commod-tz-quote-triples: ") (display commod-tz-quote-triples) (newline)) ((not (list? (car fq-results))) (set! keep-going? #f) """ [3] My debugging output: """ [edward@localhost ~]$ gnucash3 --debug --add-price-quotes /data/money/accounts This is a
