Sowiks commented on PR #154:
URL: https://github.com/apache/otava/pull/154#issuecomment-4640617563
For the sake of discussion, I will assume that we are comparing the
following methods:
- Original e-divisive: permutation test + no sliding window <-- the only
non-deterministic approach
- Deterministic e-divisive: t-test + no sliding window
- Hunter approach (aka split e-divisive): t-test + sliding window
@henrikingo mentioned hunter vs two deterministic methods, which seems to be
a typo. Let me know if I'm mistaken.
The items, I feel worth discussing / pointing out:
- It's unclear to me why @henrikingo and @Gerrrr are getting contradictory
results on the same dataset: @henrikingo found that deterministic e-divisive
performs better, while @Gerrrr fails to find any substantial difference between
original and deterministic e-divisive methods. Given they used different
datasets - are the result data-dependent or something else if going on?
- Regarding the similar p-values producing different "sensitivity" level
across tests (seen on @Gerrrr graphs as well) - it's exactly what @henrikingo
said: "Who's to say we should compare them with constant p-values". In a sense,
p-values in the original e-divisive are "true" p-values if the test run long
enough (number of permutations is large), while t-test working is an
"empirical" find, not supported by theory. This is because the conditions of
t-test are not satisfied. So the p-values produced by t-test strictly speaking
are p-values in name only. With that being said, based on the observation, they
are still very useful in identifying the change points and they appear to
maintain the relation of "t-test pseudo p-values" ~ O("true p-values"). In that
sense, if we really want to compare the methods, I would suggest to find this
scale factor/linear transformation between these p-values. I don't want to go
too deep on this here, but the general idea is to use a portion of a dat
aset, match the number of change points across these tests, and derive the
scaling constants from there. It would be also interesting to see if the
constants are the same across datasets or not (I expect them to vary across
datasets, since t-test should depend on the distribution, while permutation
test is not).
- it's worth considering to change the format of flags, instead of using the
method name like right now, e.g., `--deterministic-edivisive`,
`--orig-edivisive`, we should consider two flags: one for statistical test,
another for window sliding/skipping tweak. This would even better if we decide
to support more statistical tests, avoiding O(n^2) method naming.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
