If you do want to Fourier transform your data and you *do* have missing data points, I can highly recommend the Lomb-Scargle periodogram. I put it through its paces a while back. I took 3 sin waves of different periods and amplitude of 1 and added them together. I used 10000 data points. Then I added Gaussian noise of various standard deviations. And I also removed various amounts of data up to 90%.
The LSP still found the periods with noise of sd=5 and 90% of the data gone. With sd=10 it could still find signal with 50% of the points removed. I have some nice plots of all this if anyone is interested. The main disadvantage of LSP is speed. It performs as O(n^2). This was a huge disadvantage back in the 70s when it was published, but with today's computing power it's not a problem. Unless you have 1000000 data points, then patience is required. Jim Palfreyman On 26 May 2017 at 07:12, Michael Wouters <[email protected]> wrote: > There are 'better' ways of handling gaps when calculating ADEV and > siblings. Patrizia Tavela has a nice method: you pad out the time series, > tagging missing points with NaNs say, and then if a difference contains a > missing data point, you drop it. It works very well. I expect this is in > Stable32. I think it's implemented in allantools. It's definitely > implemented in the Matlab functions I wrote (tftools on GitHub). > > Cheers > Michael > > On Fri, 26 May 2017 at 12:00 am, Tom Van Baak <[email protected]> wrote: > > > Only Stable32 handles data gaps seamlessly. Give it a try (read the > manual > > for details). > > > > But also ask yourself how much gaps matter. Yes, they affect the accuracy > > of your y-axis sigma scale and your x-axis tau scale. A few seconds every > > 30 minutes is, what, a 0.1% error? That's like one pixel in a ADEV plot; > > not significant. > > > > What I've done when I need a perfectly seamless data set is just > > interpolate for rare and obviously missing phase data points. That keeps > > the timescale intact. This is especially important if you plan to Fourier > > transform the data: under no circumstances do you want to slip a sample > in > > that case. > > > > /tvb > > > > ----- Original Message ----- > > From: "jimlux" <[email protected]> > > To: "Discussion of precise time and frequency measurement" < > > [email protected]> > > Sent: Thursday, May 25, 2017 6:11 AM > > Subject: [time-nuts] calculating stats with gaps in the data > > > > > > > I'm looking at the python AllanTools package.. does it deal with gaps > in > > > the data series (e.g. I've got a series of phase and/or frequency > > > measurements, 1 per second, but there's gaps of a few seconds every 30 > > > minutes or so) > > > > _______________________________________________ > > time-nuts mailing list -- [email protected] > > To unsubscribe, go to > > https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts > > and follow the instructions there. > > > _______________________________________________ > time-nuts mailing list -- [email protected] > To unsubscribe, go to https://www.febo.com/cgi-bin/ > mailman/listinfo/time-nuts > and follow the instructions there. > _______________________________________________ time-nuts mailing list -- [email protected] To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts and follow the instructions there.
