Hi all,
sorry, this is going to be long and a bit involved. While working on a
package, I encountered some weirdness when creating new gretl list
objects off of a $model coming from a GUI model window after doing OLS.
I do have a stripped-down package test.gfn which I'm attaching for
illustration.
* Technical aim here: Inside the GUI function (here: GUI_uniFCboot)
which is run from the Analysis menu of the OLS model window, I want to
look at the regressors $model.xlist and I want to focus only on those
terms that are exogenous (not lagged endogenous, and also not the constant).
* Example:
1) Install somewhere the test.gfn package, attach it to the menu, load
the denmark dataset, and in the GUI regress LRM on IBO, const, and two
own lags.
2) Go to the new menu entry Analysis/testtesttest, choose horizon=2 (but
that shouldn't matter) and hit OK.
3) I get my debug printout: "We shouldn't have any lags remaining in
here: IBO,LRM_1,LRM_2", which obviously *does* contain the LRM lags.
* Background: In the function, I'm trying to create a sub-list by doing
(where 'mod' is a copy of $model):
list Lexo = mod.xlist - lags(mod.maxlag, mod.ylist) - const # all but
endo-lags and const
and then I'm printing out varname(Lexo), with the result above. Now I
can theoretically imagine some reasons why my approach doesn't work,
related to the list ID numbers inside bundles and functions and so on.
* HOWEVER, the real weirdness comes when I add/uncomment the following
lines right before the Lexo creation, namely:
bundle br = rhslags($model) # especially this line seems to make a
difference # although it shouldn't!
print br.ylagnames
(The package version with these lines active is enclosed as test2.gfn.)
-- And then everything works, i.e., there aren't any lagged endogenous
terms left in the subsequent Lexo creation; result: "We shouldn't have
any lags remaining in here: IBO" !
The point here is that the call to the private function rhslags() should
be completely redundant and without any effect, shouldn't it? The return
bundle br isn't used here, and there are no pointers involved. Instead
it magically makes it so that everything falls into place, at least in
this example.
What's behind all this? Some gretl bug? Or just further mystery with
bundled series and lists? And concretely, how can I successfully remove
from $model.xlist the lag terms in this context without doing black magic?
This is with gretl 2025a on Windows 11.
thanks a lot for reading until the end
sven
<?xml version="1.0" encoding="UTF-8"?>
<gretl-functions>
<gretl-function-package name="test" needs-time-series-data="true" model-requirement="ols" minver="2021b">
<author email="sven.schrei...@fu-berlin.de">Sven Schreiber</author>
<version>1.2</version>
<date>2025-04-05</date>
<description>Extensions for univariate forecast models</description>
<tags>C22 C53</tags>
<label>testtesttest</label>
<menu-attachment>MODELWIN/Analysis</menu-attachment>
<help>
test
</help>
<gretl-function name="uniFCboot" type="matrix">
<params count="5">
<param name="mod" type="bundle" const="true"/>
<param name="H" type="int" min="1" default="1">
<description>horizon</description>
</param>
<param name="reps" type="int" min="1" default="2000">
<description>bootstrap draws</description>
</param>
<param name="quants" type="matrix" optional="true">
<description>evaluated quantiles</description>
</param>
<param name="exo" type="matrix" optional="true" const="true">
<description>future exog. influences</description>
</param>
</params>
<code>/* Returns a H x (1 + 2*Q) matrix of bootstrapped quantiles of
forecast (FC) densities;
'H' is the forecast horizon up to which the bootstrap is done.
The first column always refers to the median (0.5-quantile).
Q is the number of elements in 'quants', which should have values between 0 and 0.5.
The default for 'quants' is {0.005, 0.025, 0.05, 0.1}.
(Corresponding to two-sided 1%, 5%, 10%, 20%. Each value
is mirrored to the right: e.g. 0.1 together with 0.9.)
'exo' (Hx1) provides the future values of any exogenous terms
(other than the constant!) in the underlying equation (X'b, i.e.
already with the coefficients factored in). Otherwise
-- can pass null or {} as arg -- they are set to zero.
The bootstrap is based on resampling.
(Future possibilities:
- wild bootstrap, with flag in the bundle mod,)
*/
## pre-process the quantiles
if !exists(quants)
matrix quants = {} # unified treatment
endif
quants = procquants(quants)
## pre-process the exogenous terms
if !exists(exo)
matrix exo = {}
endif
if nelem(exo)
matrix Xb = vec(exo)
errorif( rows(Xb) != H, "num of exo values must match horizon H")
else
matrix Xb = zeros(H,1)
endif
## constant?
wconst = inlist(mod.xlist, const)
if wconst
Xb += mod.coeff[1] # const always comes first in $coeff
endif
# process the lag information
bundle mod2 = rhslags(mod)
# lag coefficients
lagorder = max(mod2.ylags)
matrix comptop = zeros(1, lagorder)
loop i = 1..rows(mod2.ylags)
whichcf = mod2.ylagspos[i]
comptop[1, mod2.ylags[i]] = mod.coeff[whichcf]
endloop
# initial values for the FC (last 'lagorder' values in-sample)
matrix init = { mod.ylist[1] } # the dep var as vector
init = init[rows(init) - lagorder + 1 : ] # final lagorder values
## the bootstrap
matrix res = { mod.uhat }
matrix FCboot = zeros(H, 0)
loop d = 1..reps
# resampling (non-wild so far...)
matrix rndix = mrandgen(i, 1, rows(res), H, 1)
matrix innov = res[rndix]
FCboot ~= varsimul(comptop, Xb + innov, init)[lagorder + 1: ,]
endloop
# evaluate the distribution
matrix out = transp(quantile(FCboot', quants))
cnameset(out, sprintf("% g", 100*quants))
rnameset(out, sprintf("% d", seq(1,H)))
return out
</code>
</gretl-function>
<gretl-function name="GUI_uniFCboot" type="bundle" menu-only="1" pkg-role="gui-main">
<params count="5">
<param name="H" type="int" min="1" default="1">
<description>horizon</description>
</param>
<param name="reps" type="int" min="1" default="2000">
<description>bootstrap draws</description>
</param>
<param name="quants" type="matrix" optional="true">
<description>non-standard quantiles</description>
</param>
<param name="alpha" type="scalar" min="0" max="1" default="0.5">
<description>Linlin only: asymmetry param</description>
</param>
<param name="exo" type="matrix" optional="true">
<description>future exog. influences</description>
</param>
</params>
<code>bundle mod = $model # copy the active model (from GUI model context)
# check and warn if exogenous terms
/*
list endolags = lags(mod.maxlag, mod.ylist)
list endolags print
matrix elnumbers = lags(mod.maxlag, mod.ylist)
print elnumbers
matrix xlistnumbers = mod.xlist
print xlistnumbers
bundle br = rhslags($model) # especially this line seems to make a difference
# although it shouldn't!
print br.ylagnames
list checko = mod.xlist && lags(mod.maxlag, mod.ylist)
matrix checkonums = checko
print checkonums
*/
print "------"
list Lexo = mod.xlist - lags(mod.maxlag, mod.ylist) - const # all but endo-lags and const
if nelem(Lexo)
printf "We shouldn't have any lags remaining in here: %s\n", varname(Lexo)
endif
# pre-process optional args
if !exists(quants)
matrix quants = {}
endif
if !exists(exo)
matrix exo = {}
endif
if alpha != 0.5 # we do lin-lin point FC
print "test; do nothing"
else # intervals for quantiles
matrix temp = uniFCboot(mod,H,reps,quants,exo)
# add result to bundle
matrix mod.uniFCboot = temp
printf "Bootstrapped forecasts for %s, up to step %d,\n", varnames(mod.ylist)[1], H
printf " with %d replications,\n", reps
printf " starting from sample endpoint %s.\n", obslabel($t2)
printf "Chosen quantiles (plus right-tail analogues): % g\n", procquants(quants)
print temp
endif
return mod
</code>
</gretl-function>
<gretl-function name="rhslags" type="bundle" private="1">
<params count="1">
<param name="mod" type="bundle" const="true"/>
</params>
<code># (private function)
/* This function expects an input bundle of the type
that is provided by gretl's $model accessor.
It puts the information about the names of the
appearing lagged terms (ylagnames, strings array), which lags are used (ylags, matrix), and which is
their respective position in the regressor list
(ylagspos, matrix) into the returned bundle.
This only works if (a) the lagged terms are exactly
named as gretl's lags() function does it, (b) if
the ordering is strictly ascending (e.g. -1 first).
There can be gaps in the lag structure.
*/
bundle out = mod # copy
# create the max potential number of lags
list templags = lags(nelem(mod.xlist), mod.ylist)
# keep those lags that were actually used in $xlist
list sec = mod.xlist && templags
# and record their names
strings out.ylagnames = varnames(sec)
# determine which lags and where they appear
matrix out.ylags = {}
matrix out.ylagspos = {}
loop foreach s sec
out.ylags |= inlist(templags, sec.$s) # i-th pos in templags is lag
out.ylagspos |= inlist(mod.xlist, sec.$s) # where in $xlist/$coef ?
endloop
return out
</code>
</gretl-function>
<gretl-function name="standardquants" type="matrix" private="1">
<code>return {0.005, 0.025, 0.05, 0.1}
</code>
</gretl-function>
<gretl-function name="procquants" type="matrix" private="1">
<params count="1">
<param name="quants" type="matrix" optional="true"/>
</params>
<code>if !exists(quants)
matrix quants = {} # for unified treatment
endif
if nelem(quants)
matrix q = sort(vec(quants))'
errorif( min(q) <= 0 || max(q) >= 1, "invalid input for quants" )
else
matrix q = standardquants()
endif
# add the median
q = 0.5 ~ q ~ mreverse(1 - q')'
return q
</code>
</gretl-function>
<sample-script>
# Sample script for the uniFCextensions package for gretl
clear
include test.gfn --force
###################
#
# Initial example, pure AR equation
open denmark
H = 3
### test test
</sample-script>
</gretl-function-package>
</gretl-functions>
<?xml version="1.0" encoding="UTF-8"?>
<gretl-functions>
<gretl-function-package name="test" needs-time-series-data="true" model-requirement="ols" minver="2021b">
<author email="sven.schrei...@fu-berlin.de">Sven Schreiber</author>
<version>1.2</version>
<date>2025-04-05</date>
<description>Extensions for univariate forecast models</description>
<tags>C22 C53</tags>
<label>testtesttest</label>
<menu-attachment>MODELWIN/Analysis</menu-attachment>
<help>
test
</help>
<gretl-function name="uniFCboot" type="matrix">
<params count="5">
<param name="mod" type="bundle" const="true"/>
<param name="H" type="int" min="1" default="1">
<description>horizon</description>
</param>
<param name="reps" type="int" min="1" default="2000">
<description>bootstrap draws</description>
</param>
<param name="quants" type="matrix" optional="true">
<description>evaluated quantiles</description>
</param>
<param name="exo" type="matrix" optional="true" const="true">
<description>future exog. influences</description>
</param>
</params>
<code>/* Returns a H x (1 + 2*Q) matrix of bootstrapped quantiles of
forecast (FC) densities;
'H' is the forecast horizon up to which the bootstrap is done.
The first column always refers to the median (0.5-quantile).
Q is the number of elements in 'quants', which should have values between 0 and 0.5.
The default for 'quants' is {0.005, 0.025, 0.05, 0.1}.
(Corresponding to two-sided 1%, 5%, 10%, 20%. Each value
is mirrored to the right: e.g. 0.1 together with 0.9.)
'exo' (Hx1) provides the future values of any exogenous terms
(other than the constant!) in the underlying equation (X'b, i.e.
already with the coefficients factored in). Otherwise
-- can pass null or {} as arg -- they are set to zero.
The bootstrap is based on resampling.
(Future possibilities:
- wild bootstrap, with flag in the bundle mod,)
*/
## pre-process the quantiles
if !exists(quants)
matrix quants = {} # unified treatment
endif
quants = procquants(quants)
## pre-process the exogenous terms
if !exists(exo)
matrix exo = {}
endif
if nelem(exo)
matrix Xb = vec(exo)
errorif( rows(Xb) != H, "num of exo values must match horizon H")
else
matrix Xb = zeros(H,1)
endif
## constant?
wconst = inlist(mod.xlist, const)
if wconst
Xb += mod.coeff[1] # const always comes first in $coeff
endif
# process the lag information
bundle mod2 = rhslags(mod)
# lag coefficients
lagorder = max(mod2.ylags)
matrix comptop = zeros(1, lagorder)
loop i = 1..rows(mod2.ylags)
whichcf = mod2.ylagspos[i]
comptop[1, mod2.ylags[i]] = mod.coeff[whichcf]
endloop
# initial values for the FC (last 'lagorder' values in-sample)
matrix init = { mod.ylist[1] } # the dep var as vector
init = init[rows(init) - lagorder + 1 : ] # final lagorder values
## the bootstrap
matrix res = { mod.uhat }
matrix FCboot = zeros(H, 0)
loop d = 1..reps
# resampling (non-wild so far...)
matrix rndix = mrandgen(i, 1, rows(res), H, 1)
matrix innov = res[rndix]
FCboot ~= varsimul(comptop, Xb + innov, init)[lagorder + 1: ,]
endloop
# evaluate the distribution
matrix out = transp(quantile(FCboot', quants))
cnameset(out, sprintf("% g", 100*quants))
rnameset(out, sprintf("% d", seq(1,H)))
return out
</code>
</gretl-function>
<gretl-function name="GUI_uniFCboot" type="bundle" menu-only="1" pkg-role="gui-main">
<params count="5">
<param name="H" type="int" min="1" default="1">
<description>horizon</description>
</param>
<param name="reps" type="int" min="1" default="2000">
<description>bootstrap draws</description>
</param>
<param name="quants" type="matrix" optional="true">
<description>non-standard quantiles</description>
</param>
<param name="alpha" type="scalar" min="0" max="1" default="0.5">
<description>Linlin only: asymmetry param</description>
</param>
<param name="exo" type="matrix" optional="true">
<description>future exog. influences</description>
</param>
</params>
<code>bundle mod = $model # copy the active model (from GUI model context)
# check and warn if exogenous terms
/*
list endolags = lags(mod.maxlag, mod.ylist)
list endolags print
matrix elnumbers = lags(mod.maxlag, mod.ylist)
print elnumbers
matrix xlistnumbers = mod.xlist
print xlistnumbers
*/
bundle br = rhslags($model) # especially this line seems to make a difference
# although it shouldn't!
print br.ylagnames
/*
list checko = mod.xlist && lags(mod.maxlag, mod.ylist)
matrix checkonums = checko
print checkonums
*/
print "------"
list Lexo = mod.xlist - lags(mod.maxlag, mod.ylist) - const # all but endo-lags and const
if nelem(Lexo)
printf "We shouldn't have any lags remaining in here: %s\n", varname(Lexo)
endif
# pre-process optional args
if !exists(quants)
matrix quants = {}
endif
if !exists(exo)
matrix exo = {}
endif
if alpha != 0.5 # we do lin-lin point FC
print "test; do nothing"
else # intervals for quantiles
matrix temp = uniFCboot(mod,H,reps,quants,exo)
# add result to bundle
matrix mod.uniFCboot = temp
printf "Bootstrapped forecasts for %s, up to step %d,\n", varnames(mod.ylist)[1], H
printf " with %d replications,\n", reps
printf " starting from sample endpoint %s.\n", obslabel($t2)
printf "Chosen quantiles (plus right-tail analogues): % g\n", procquants(quants)
print temp
endif
return mod
</code>
</gretl-function>
<gretl-function name="rhslags" type="bundle" private="1">
<params count="1">
<param name="mod" type="bundle" const="true"/>
</params>
<code># (private function)
/* This function expects an input bundle of the type
that is provided by gretl's $model accessor.
It puts the information about the names of the
appearing lagged terms (ylagnames, strings array), which lags are used (ylags, matrix), and which is
their respective position in the regressor list
(ylagspos, matrix) into the returned bundle.
This only works if (a) the lagged terms are exactly
named as gretl's lags() function does it, (b) if
the ordering is strictly ascending (e.g. -1 first).
There can be gaps in the lag structure.
*/
bundle out = mod # copy
# create the max potential number of lags
list templags = lags(nelem(mod.xlist), mod.ylist)
# keep those lags that were actually used in $xlist
list sec = mod.xlist && templags
# and record their names
strings out.ylagnames = varnames(sec)
# determine which lags and where they appear
matrix out.ylags = {}
matrix out.ylagspos = {}
loop foreach s sec
out.ylags |= inlist(templags, sec.$s) # i-th pos in templags is lag
out.ylagspos |= inlist(mod.xlist, sec.$s) # where in $xlist/$coef ?
endloop
return out
</code>
</gretl-function>
<gretl-function name="standardquants" type="matrix" private="1">
<code>return {0.005, 0.025, 0.05, 0.1}
</code>
</gretl-function>
<gretl-function name="procquants" type="matrix" private="1">
<params count="1">
<param name="quants" type="matrix" optional="true"/>
</params>
<code>if !exists(quants)
matrix quants = {} # for unified treatment
endif
if nelem(quants)
matrix q = sort(vec(quants))'
errorif( min(q) <= 0 || max(q) >= 1, "invalid input for quants" )
else
matrix q = standardquants()
endif
# add the median
q = 0.5 ~ q ~ mreverse(1 - q')'
return q
</code>
</gretl-function>
<sample-script>
# Sample script for the uniFCextensions package for gretl
clear
include test.gfn --force
###################
#
# Initial example, pure AR equation
open denmark
H = 3
### test test
</sample-script>
</gretl-function-package>
</gretl-functions>
_______________________________________________
Gretl-devel mailing list -- gretl-devel@gretlml.univpm.it
To unsubscribe send an email to gretl-devel-le...@gretlml.univpm.it
Website:
https://gretlml.univpm.it/postorius/lists/gretl-devel.gretlml.univpm.it/
