Re: [GNC-dev] Robust error handling and respecting the Apha Vantage API limits in gnc-fq-helper.
On 11 Feb 2020, at 3:59, Edward d'Auvergne wrote:
F::Q uses this URL for all queries:
https://www.alphavantage.co/query?function=GLOBAL_QUOTE=json=IDRUSD=$MY_KEY
The result I see is:
{
"Global Quote": {
"01. symbol": "IDRUSD",
"02. open": "0.0001",
"03. high": "0.0001",
"04. low": "0.0001",
"05. price": "0.0001",
"06. volume": "0",
"07. latest trading day": "2020-02-10",
"08. previous close": "0.0001",
"09. change": "0.",
"10. change percent": "0.%"
}
}
It doesn't use that URL for currency exchange rates. It uses
https://www.alphavantage.co/query?function=CURRENCY_EXCHANGE_RATE_currency=IDR_currency=USD=xxx
which returns
{
"Realtime Currency Exchange Rate": {
"1. From_Currency Code": "IDR",
"2. From_Currency Name": "Indonesian Rupiah",
"3. To_Currency Code": "USD",
"4. To_Currency Name": "United States Dollar",
"5. Exchange Rate": "0.7297",
"6. Last Refreshed": "2020-02-12 06:30:18",
"7. Time Zone": "UTC",
"8. Bid Price": "-",
"9. Ask Price": "-"
}
}
It looks like this code then kicks in, from the "currency" subroutine
in lib/Finance/Quote.pm:
"""
if ( $exchange_rate < 0.001 ) {
# exchange_rate is too little. we'll get more accuracy by using
# the inverse rate and inverse it
my $inverse_rate = $this->currency( $to, $from );
{
local $^W = 0;
return undef unless ( $exchange_rate + 0 );
}
$exchange_rate = int( 1 / $inverse_rate + .5 ) /
1;
}
"""
That code should be removed. It actually gets the same accuracy and it
gets a different answer. The reverse quote is 13683.5. This inverts to
a rate of 0.7308 which is not the same as 0.7297. That's a
trivial difference, but F::Q should really return the quote it was asked
for if possible.
I noticed that there was some action on my pull request after you
rattled their cage. Thanks for doing that. That may make this
discussion largely irrelevant.
Mike
___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: [GNC-dev] Robust error handling and respecting the Apha Vantage API limits in gnc-fq-helper.
On 11 Feb 2020, at 8:14, Edward d'Auvergne wrote:
Although most runs pass, I have noticed that some do not [4]. Have
you seen that behaviour or know what might be happening?
Sometimes AlphaVantage refuses to return a quote. Right now
https://www.alphavantage.co/query?function=CURRENCY_EXCHANGE_RATE_currency=USD_currency=XAU=xxx
returns
{}
The ILS quote works now, but must not have been working when you tried
it. I don't think this is an F::Q problem. GnuCash deals with this
failure and uses the inverse quote. For example
mta@bayswater /<4>bin>./gnc-fq-dump currency XAU USD
1 XAU = 1565.2 USD
mta@bayswater /<4>bin>./gnc-fq-dump currency USD XAU
1 XAU = 1565.2 USD
Note that they both return the same thing since the other quote doesn't
work. GnuCash is smart enough to use an inverset currency rate if the
one it wants isn't stored.
Mike
___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel
Re: [GNC-dev] Robust error handling and respecting the Apha Vantage API limits in gnc-fq-helper.
On Mon, 10 Feb 2020 at 22:28, Mike Alexander wrote:
> On 10 Feb 2020, at 6:27, Edward d'Auvergne wrote:
>> I realise that F::Q is broken in some places. As I said, I reported
>> an issue. However, as you have seen yourself, the F::Q developers are
>> simply non-responsive. I believe that asking all GnuCash users to
>> manually patch their own Finance::Quote perl modules to get around
>> this 2 year old issue is too much to ask of users. My reasoning is
>> that a little more logic on the GnuCash side to handle F::Q
>> misbehaving is easier for all GnuCash users. And, from the behaviour
>> of the scripts below, I now wonder if GnuCash could benefit by
>> switching from piping '(currency "USD" "EUR")' into gnc-fc-helper to
>> instead piping in '(alphavantage "USDEUR")'? The only disadvantage is
>> that gold and silver quotes do not work via the fetch() interface with
>> Alpha Vantage.
>
> I think I understand now why we've been talking past each other. There isn't
> any API in F::Q to fetch multiple currency quotes in one call. Instead it's a
> feature of AlphaVantage to treat exchange rates as if they were stock quotes
> and use the stock quote API to fetch them. I wasn't aware that this existed.
I thought this was also the case with the old Yahoo API. Maybe I am
confusing this with the original Finance::YahooQuote perl module that
does not separate currencies from other symbols?
> I played with it for a few minutes and see a couple of problems with it. Some
> currencies are not available using the stock quote API but are available
> using the currency API. For example three in my list of currencies which
> don't work with the quote API are MNT, SBD, and VUV. These all work fine with
> the currency API.
>
> The other problem is that for very small exchange rates the stock quote
> interface loses precision. For example, compare
>
> ./gnc-fq-dump alphavantage IDRUSD
> Finance::Quote fields Gnucash uses:
> symbol: IDRUSD <=== required
> date: 02/10/2020 <=== recommended
> currency: USD <=== required
> last: 0.0001 <=\
> nav: <=== one of these
> price: <=/
> timezone: <=== optional
>
> and
>
> ./gnc-fq-dump currency IDR USD
> 1 IDR = 7.295e-05 USD
>
> These problems would make the quote API for currencies less useful.
F::Q uses this URL for all queries:
https://www.alphavantage.co/query?function=GLOBAL_QUOTE=json=IDRUSD=$MY_KEY
The result I see is:
{
"Global Quote": {
"01. symbol": "IDRUSD",
"02. open": "0.0001",
"03. high": "0.0001",
"04. low": "0.0001",
"05. price": "0.0001",
"06. volume": "0",
"07. latest trading day": "2020-02-10",
"08. previous close": "0.0001",
"09. change": "0.",
"10. change percent": "0.%"
}
}
It looks like this code then kicks in, from the "currency" subroutine
in lib/Finance/Quote.pm:
"""
if ( $exchange_rate < 0.001 ) {
# exchange_rate is too little. we'll get more accuracy by using
# the inverse rate and inverse it
my $inverse_rate = $this->currency( $to, $from );
{
local $^W = 0;
return undef unless ( $exchange_rate + 0 );
}
$exchange_rate = int( 1 / $inverse_rate + .5 ) / 1;
}
"""
The "fetch" subroutine simply does not do that. I guess you would
first have to identify that the symbol is a currency pair, then invert
it. Not too difficult. But then it comes back to the issue of
fixing/modifying F::Q itself.
> By the way, I didn't mean to suggest that we should encourage all users of
> GnuCash to patch F::Q. I still remain hopeful that the maintainers will merge
> my pull request that fixes this bug. In the meantime switching to the quote
> API might be a good idea even if it does have issues. The code would have to
> be smart enough to work around the problems. For example if the quote API
> fails for a given currency it could try the currency API. If the returned
> value appears to have lost all precision then it could try the reverse quote
> and compute the reciprocal itself.
I finally found your pull request:
https://github.com/finance-quote/finance-quote/pull/130. Your
changes, that were rejected, allow my previous script [1] to
successfully fetch the quotes in serial [2]. I'll try to create a
pure perl script to demonstrate that your bug fix is a proper bug fix,
and that they should reopen your pull request, merge it, and then
release F::Q v1.50! If that works, then there wouldn't be any changes
required on the GnuCash side. Well, apart from the error handling to
prevent GnuCash from segfaulting.
> Another possibility would be to abandon AlphaVantage entirely for currency
> exchange rates. For example https://currencylayer.com/ looks promising. They
> allow up to 250 queries per month for free and it seems very fast. Of course
> this would require changes to F::Q which gets us right back into those
> problems.
Maybe the user should be allowed to choose their favourite quote
provider via
Re: [GNC-dev] Robust error handling and respecting the Apha Vantage API limits in gnc-fq-helper.
On Tue, 11 Feb 2020 at 09:59, Edward d'Auvergne wrote:
> On Mon, 10 Feb 2020 at 22:28, Mike Alexander wrote:
> > By the way, I didn't mean to suggest that we should encourage all users of
> > GnuCash to patch F::Q. I still remain hopeful that the maintainers will
> > merge my pull request that fixes this bug. In the meantime switching to the
> > quote API might be a good idea even if it does have issues. The code would
> > have to be smart enough to work around the problems. For example if the
> > quote API fails for a given currency it could try the currency API. If the
> > returned value appears to have lost all precision then it could try the
> > reverse quote and compute the reciprocal itself.
>
> I finally found your pull request:
> https://github.com/finance-quote/finance-quote/pull/130. Your
> changes, that were rejected, allow my previous script [1] to
> successfully fetch the quotes in serial [2]. I'll try to create a
> pure perl script to demonstrate that your bug fix is a proper bug fix,
> and that they should reopen your pull request, merge it, and then
> release F::Q v1.50! If that works, then there wouldn't be any changes
> required on the GnuCash side. Well, apart from the error handling to
> prevent GnuCash from segfaulting.
I've now created a script to demonstrate your fix in the pull request
[1]. On the master branch of F::Q, this shows the failure [2]. After
cherry-picking your commit 877c87bd3ea1cbd403c59dfcd4a58b97c14d232a,
most runs pass [3]. This does not include the "USD" "XAU" pair which
needs to be inverted. Maybe the inversion should be directly
performed in F::Q rather than GnuCash? Anyway, I'll post this to the
pull request and ask for it to be reopened.
Although most runs pass, I have noticed that some do not [4]. Have
you seen that behaviour or know what might be happening?
Regards,
Edward
[1] Script to demonstrate the F::Q currency fetching failures:
"""
#!/usr/bin/env perl
use Finance::Quote;
use Time::HiRes qw(time);
print("Using Perl $^V\n");
print("Using Finance::Quote $Finance::Quote::VERSION\n\n");
# The currencies to fetch.
my @symbols = (
"AUD",
"BRL",
"CAD",
"CHF",
"CNY",
"CZK",
"DKK",
"EUR",
"GBP",
"HKD",
"HRK",
"ILS",
"JPY",
"KRW",
"MYR",
"NOK",
"NZD",
"RUB",
"SGD",
"TWD",
"XAU"
);
# Fetch the currencies.
my $begin = time();
my $quoter = Finance::Quote->new();
foreach (@symbols) {
$price = $quoter->currency("USD", $_);
if (!defined $price) {
$price = "Failed.";
}
print("USD$_: $price\n");
}
my $end = time();
printf("Fetching time %0.02f s\n", $end-$begin);
"""
[2] Failure of F::Q at bcfc947ee0173e059573236c2cf4a9bcb9a09b65:
"""
Using Perl v5.30.1
Using Finance::Quote 1.50
USDAUD: 1.4909
USDBRL: 4.325
USDCAD: 1.3294
USDCHF: 0.9778
USDCNY: 6.9745
USDCZK: Failed.
USDDKK: Failed.
USDEUR: Failed.
USDGBP: Failed.
USDHKD: Failed.
USDHRK: Failed.
USDILS: Failed.
USDJPY: Failed.
USDKRW: Failed.
USDMYR: Failed.
USDNOK: Failed.
USDNZD: Failed.
USDRUB: Failed.
USDSGD: Failed.
USDTWD: Failed.
USDXAU: Failed.
Fetching time 12.44 s
"""
[3] Passing run after cherry-picking 877c87bd3ea1cbd403c59dfcd4a58b97c14d232a:
"""
Using Perl v5.30.1
Using Finance::Quote 1.50
USDAUD: 1.4907
USDBRL: 4.3242
USDCAD: 1.3293
USDCHF: 0.9781
USDCNY: 6.9745
USDCZK: 22.833
USDDKK: 6.8465
USDEUR: 0.9162
USDGBP: 0.7734
USDHKD: 7.7646
USDHRK: 6.8309
USDILS: 3.4163
USDJPY: 109.85
USDKRW: 1183.9
USDMYR: 4.134
USDNOK: 9.2539
USDNZD: 1.5643
USDRUB: 63.644
USDSGD: 1.3872
USDTWD: 30.034
USDXAU: Failed.
Fetching time 219.69 s
""'
[4] Failing run after cherry-picking 877c87bd3ea1cbd403c59dfcd4a58b97c14d232a:
"""
Using Perl v5.30.1
Using Finance::Quote 1.50
USDAUD: 1.4907
USDBRL: 4.3229
USDCAD: 1.3293
USDCHF: 0.9778
USDCNY: 6.9745
USDCZK: 22.836
USDDKK: 6.8453
USDEUR: 0.9161
USDGBP: 0.7735
USDHKD: 7.7646
USDHRK: 6.8309
USDILS: Failed.
USDJPY: 109.84
USDKRW: 1184
USDMYR: 4.134
USDNOK: 9.2512
USDNZD: 1.5639
USDRUB: 63.716
USDSGD: 1.3872
USDTWD: 30.03
USDXAU: Failed.
Fetching time 167.91 s
"""
___
gnucash-devel mailing list
gnucash-devel@gnucash.org
https://lists.gnucash.org/mailman/listinfo/gnucash-devel
