[Gretl-devel] Re: panel sample restriction weirdness with dummy
Am 24.01.2024 um 21:17 schrieb Allin Cottrell: On Mon, 22 Jan 2024, Sven Schreiber wrote: I've stumbled over a certain behavior with panel data handling, which could perhaps be a bug. Yes, there was a bug. After going over all that Sven reported I was able to replicate the problem with these steps: ... then I got the correct result with the --dummy option. So it seemed the error must lie with the shrinkage of the original T=24 dataset at a result of step 4) above. Indeed, some "debris" was not being cleared out. That's now fixed in git. Wow, that's very nice bug-hunting ! - I didn't really expect it to be reproducible thanks, Allin -Sven ___ Gretl-devel mailing list -- gretl-devel@gretlml.univpm.it To unsubscribe send an email to gretl-devel-le...@gretlml.univpm.it Website: https://gretlml.univpm.it/postorius/lists/gretl-devel.gretlml.univpm.it/
[Gretl-devel] Re: panel sample restriction weirdness with dummy
On Mon, 22 Jan 2024, Sven Schreiber wrote: I've stumbled over a certain behavior with panel data handling, which could perhaps be a bug. Yes, there was a bug. After going over all that Sven reported I was able to replicate the problem with these steps: 1) launch GUI and run pansmpl.inp (see below), giving N=8, T=24 2) Do '/File/Save data' with name foo.gdt 3) Use '/Sample/Set range' and reduce T to 12 (2023 only) 4) Do '/File/Save data' again, saying No to restoring full data 5) Then in the console: a smpl -> Full data range: 1:01 - 8:12 (n = 96) # OK b smpl full -> same output as above # OK c smpl ichk --dummy -> Current sample: 1:01 - 4:12 (n = 48) # Bad! d smpl full -> Full data range: 1:01 - 8:12 (n = 96)# OK e smpl ichk==1 --restrict -> Current sample: 1:01 - 7:12 (n = 84) # OK Switching the order of console commands c and e didn't alter the discrepancy, I still got the bad result from the --dummy option but the correct result from --restrict. And this result was invariant wrt using '/Sample/Set range' to cut out the second year's data versus using 'smpl time < 13 --restrict'. But if I did, after step 4); smpl full store pantest.gdt open pantest.gdt smpl ichk --dummy then I got the correct result with the --dummy option. So it seemed the error must lie with the shrinkage of the original T=24 dataset at a result of step 4) above. Indeed, some "debris" was not being cleared out. That's now fixed in git. N = 8 T = 24 NT = N*T nulldata NT --preserve setobs 24 1:01 --stacked-time-series setobs 12 2023:01 --panel-time series u = $unit genr time series ichk = u != 6 Allin ___ Gretl-devel mailing list -- gretl-devel@gretlml.univpm.it To unsubscribe send an email to gretl-devel-le...@gretlml.univpm.it Website: https://gretlml.univpm.it/postorius/lists/gretl-devel.gretlml.univpm.it/
[Gretl-devel] Re: panel sample restriction weirdness with dummy
Am 22.01.2024 um 22:01 schrieb Allin Cottrell: On Mon, 22 Jan 2024, Sven Schreiber wrote: Well, I couldn't remember exactly, so I repeated this part of the steps, and I remembered correctly, namely that I most definitely did not get a second dialog window. After saying No to the first choice the data was simply saved (or so it seemed) and nothing else happened. OK, thanks. Now one more thing: if you answered "No" to restoring the full dataset but didn't get the second query that would seem to imply that the reduced dataset was being saved under its original name (not a "Save as..."). Does that sound right? Yes, exactly. I had done the save-as-step previously to create a copy of the full dataset. So after halving the range I didn't have to do save-as again. -s ___ Gretl-devel mailing list -- gretl-devel@gretlml.univpm.it To unsubscribe send an email to gretl-devel-le...@gretlml.univpm.it Website: https://gretlml.univpm.it/postorius/lists/gretl-devel.gretlml.univpm.it/
[Gretl-devel] Re: panel sample restriction weirdness with dummy
On Mon, 22 Jan 2024, Sven Schreiber wrote: Am 22.01.2024 um 19:26 schrieb Allin Cottrell: When you do a GUI data-save and the current dataset is sub-sampled, you get up to two option dialogs. First you're asked whether you want to restore the full dataset before saving. If you answer No to that you're then asked if you want to shrink the in-memory dataset to match what was saved. I take it that you said No to the first prompt but then I'm not 100% clear on how you responded to the second one. If you can remember, that might be helpful! Well, I couldn't remember exactly, so I repeated this part of the steps, and I remembered correctly, namely that I most definitely did not get a second dialog window. After saying No to the first choice the data was simply saved (or so it seemed) and nothing else happened. OK, thanks. Now one more thing: if you answered "No" to restoring the full dataset but didn't get the second query that would seem to imply that the reduced dataset was being saved under its original name (not a "Save as..."). Does that sound right? Allin ___ Gretl-devel mailing list -- gretl-devel@gretlml.univpm.it To unsubscribe send an email to gretl-devel-le...@gretlml.univpm.it Website: https://gretlml.univpm.it/postorius/lists/gretl-devel.gretlml.univpm.it/
[Gretl-devel] Re: panel sample restriction weirdness with dummy
Am 22.01.2024 um 19:26 schrieb Allin Cottrell: On Mon, 22 Jan 2024, Sven Schreiber wrote: * When you saved the restricted dataset, did you use the "store" command or the GUI? And if the latter, did you take the option (it's offered in a dialog) of reducing the dataset in memory to that saved to disk? I used the GUI for that, BTW, I used the very nice relatively recent dialog in the panel context "choose range", where you can manipulate the time dimension. So I did _not_ go to "Restrict by condition". and gretl asked whether I want to stick with the reduced dataset, and I answered yes. Or actually, I think it asked "do you want to revert to the full range", and I answered no. When you do a GUI data-save and the current dataset is sub-sampled, you get up to two option dialogs. First you're asked whether you want to restore the full dataset before saving. If you answer No to that you're then asked if you want to shrink the in-memory dataset to match what was saved. I take it that you said No to the first prompt but then I'm not 100% clear on how you responded to the second one. If you can remember, that might be helpful! Well, I couldn't remember exactly, so I repeated this part of the steps, and I remembered correctly, namely that I most definitely did not get a second dialog window. After saying No to the first choice the data was simply saved (or so it seemed) and nothing else happened. cheers sven ___ Gretl-devel mailing list -- gretl-devel@gretlml.univpm.it To unsubscribe send an email to gretl-devel-le...@gretlml.univpm.it Website: https://gretlml.univpm.it/postorius/lists/gretl-devel.gretlml.univpm.it/
[Gretl-devel] Re: panel sample restriction weirdness with dummy
On Mon, 22 Jan 2024, Sven Schreiber wrote:
Am 22.01.2024 um 15:03 schrieb Allin Cottrell:
On Mon, 22 Jan 2024, Sven Schreiber wrote:
The dataset is monthly in the time dimension and recognized as such by
gretl, n=1..435 and t=2023:01..2023:12. (N*T = 5220) Missings are present
somewhere in most of the included variables.
[...]
Sven, I tried replicating what you describe but failed to replicate the
problem (not very surprising). Could you give us a little more detail,
please?
* What sort of manipulations preceded your saving of the panel dataset to
disk?
I don't remember exactly in what order I took the steps. What I can say is
that I started with two panel datasets for the years 2022 and 2023 (so T=24).
Then I "reduced" both datasets to the year 2023, discarding the first (in a
time sense) halves of each.
Then (I think) I "joined" two variables of the second dataset into the first,
using "--ikey=stationids,iso" as the panel index variables. ("stationids" are
non-consecutive but unique integers for the units, "iso" is the ISO-format
date number of the month.)
The "ichk" series in my example was formed by running the pnobs() function
separately on two or three variables of the dataset, and then combining those
results with the appropriate boolean pattern to obtain an indicator dummy for
the obs of those cross-sectional units which don't have any valid obs (in
that year).
OK, thanks, I'll think about how that sort of sequence might work.
* When you saved the restricted dataset, did you use the "store" command or
the GUI? And if the latter, did you take the option (it's offered in a
dialog) of reducing the dataset in memory to that saved to disk?
I used the GUI for that, and gretl asked whether I want to stick with the
reduced dataset, and I answered yes. Or actually, I think it asked "do you
want to revert to the full range", and I answered no.
When you do a GUI data-save and the current dataset is sub-sampled,
you get up to two option dialogs. First you're asked whether you
want to restore the full dataset before saving. If you answer No to
that you're then asked if you want to shrink the in-memory dataset
to match what was saved. I take it that you said No to the first
prompt but then I'm not 100% clear on how you responded to the
second one. If you can remember, that might be helpful!
Allin
___
Gretl-devel mailing list -- gretl-devel@gretlml.univpm.it
To unsubscribe send an email to gretl-devel-le...@gretlml.univpm.it
Website:
https://gretlml.univpm.it/postorius/lists/gretl-devel.gretlml.univpm.it/
[Gretl-devel] Re: panel sample restriction weirdness with dummy
Am 22.01.2024 um 15:03 schrieb Allin Cottrell:
On Mon, 22 Jan 2024, Sven Schreiber wrote:
The dataset is monthly in the time dimension and recognized as such
by gretl, n=1..435 and t=2023:01..2023:12. (N*T = 5220) Missings are
present somewhere in most of the included variables.
[...]
Sven, I tried replicating what you describe but failed to replicate
the problem (not very surprising). Could you give us a little more
detail, please?
* What sort of manipulations preceded your saving of the panel dataset
to disk?
I don't remember exactly in what order I took the steps. What I can say
is that I started with two panel datasets for the years 2022 and 2023
(so T=24).
Then I "reduced" both datasets to the year 2023, discarding the first
(in a time sense) halves of each.
Then (I think) I "joined" two variables of the second dataset into the
first, using "--ikey=stationids,iso" as the panel index variables.
("stationids" are non-consecutive but unique integers for the units,
"iso" is the ISO-format date number of the month.)
The "ichk" series in my example was formed by running the pnobs()
function separately on two or three variables of the dataset, and then
combining those results with the appropriate boolean pattern to obtain
an indicator dummy for the obs of those cross-sectional units which
don't have any valid obs (in that year).
* When you saved the restricted dataset, did you use the "store"
command or the GUI? And if the latter, did you take the option (it's
offered in a dialog) of reducing the dataset in memory to that saved
to disk?
I used the GUI for that, and gretl asked whether I want to stick with
the reduced dataset, and I answered yes. Or actually, I think it asked
"do you want to revert to the full range", and I answered no.
thanks
sven
___
Gretl-devel mailing list -- gretl-devel@gretlml.univpm.it
To unsubscribe send an email to gretl-devel-le...@gretlml.univpm.it
Website:
https://gretlml.univpm.it/postorius/lists/gretl-devel.gretlml.univpm.it/
[Gretl-devel] Re: panel sample restriction weirdness with dummy
On Mon, 22 Jan 2024, Sven Schreiber wrote: I've stumbled over a certain behavior with panel data handling, which could perhaps be a bug. However, it turned out that when I closed and restarted gretl, the problem went away. It looks as if it could be relevant that the panel datafile is a subset that was saved as a gdt file as the result of earlier manipulation, and continuing to work with the reduced dataset without starting a new gretl session. My suspicion is that gretl wasn't able to fully and cleanly switch over to the reduced dataset that was created and retained. I know that the problem is therefore quite subtle and probably not directly reproducible. Nevertheless, let me show below what happened in the console. Maybe it's still useful for something. The dataset is monthly in the time dimension and recognized as such by gretl, n=1..435 and t=2023:01..2023:12. (N*T = 5220) Missings are present somewhere in most of the included variables. [...] Sven, I tried replicating what you describe but failed to replicate the problem (not very surprising). Could you give us a little more detail, please? * What sort of manipulations preceded your saving of the panel dataset to disk? * When you saved the restricted dataset, did you use the "store" command or the GUI? And if the latter, did you take the option (it's offered in a dialog) of reducing the dataset in memory to that saved to disk? Allin ___ Gretl-devel mailing list -- gretl-devel@gretlml.univpm.it To unsubscribe send an email to gretl-devel-le...@gretlml.univpm.it Website: https://gretlml.univpm.it/postorius/lists/gretl-devel.gretlml.univpm.it/
