Hi Jonathan,
What things have you already tried to help narrow the problem down?
Have you tried cooling the boards to see if that has an effect? (Or do
you have more issues in summer than winter?)
Have you measured the power supply voltages going to the FPGA and to the
QDR? Perhaps some power supply components are getting weak and the
voltages are low.
Have you tried replacing the capacitors? A very common issue with a lot
of old electronics is that the filter capacitors degrade, especially if
they are electrolytic. This can cause noise on the power rails, leading
to problems like this.
Since you say that replacing the QDRs is sometimes effective, these
other problems become less likely. But if timing is marginal it's
possible multiple things can affect it. it doesn't hurt to check these off.
Regards,
Ross
On 9/22/26 04:23, 'Weintroub, Jonathan' via [email protected] wrote:
Hi Dale, hi Jack, and relevant CASPERites,
Dale, while I am sorry you are having trouble with EOVSA, I am
simultaneously happy you raised the topic of QDR on the CASPER list.
At SMA we have 68 ROACH2s installed in the operational SWARM (16 of
which are "aspirational" but still relevant), and between lab system
and spares, own close to 100 units. Recognizing that ROACH2 is a
decade old, and many have moved on, as they should, to newer tech (to
which we also aspire), for better or worse, we still depend on it for
our primary facility back end.
We have more failures than we'd like, and most of them are of QDR.
The the failures are in two broad categories: Either a ROACH2 doesn't
close the QDR calibration during start up, in which case part of the
system doesn't initialize; OR during operation data corruption with
characteristics we've come to recognize (narrow spikes in the
visibilities) is traceable to QDR data corruption.
We have dealt with this in two primary ways, either swapping in spares
(but inevitably we are running out); and for a time we had a contract
with Digicom to swap QDRs (which required dis-assembly of the ROACH2
box with 5 GS/s ADCs, mezzanines, cables etc), re-assemble and
retest. This had partial success, but since Mo sold Digicom the
arrangement has come to an end for various reasons, including that
they no longer have a test server, the QDR chips being far past
end-of-life, etc.
To no longer bury the lede: it is my view that what is really
underlying these issues is calibration and application of QDRs which,
in our case at least, works "by and large" but is on the margin and
goes outside for some reason, including aging. I base this on
recollection of back and forth when ROACH2 wss first being brought up,
and there is some record of this in the mailing list archive.
Our application includes a 286 MHz fabric clock, -2 speed grade
FPGAs. And I vaguely remember that at build time we can pick the
separate QDR clock (100 or 200 MHz?).
It is my hope that those who were at the center of the QDR
development might be able to contribute some technical context to my
handwaving history so that we at SMA can dig into what might be going
on. (We recently assigned a FPGA engineer to do so.)
Jonathan
On Tue, Sep 22, 2026 at 6:32 AM Jack Hickish <[email protected]>
wrote:
Hi Dale,
As far as I'm aware there are no special settings needed for QDR -
everything is baked into the firmware. Does the failure give any
useful information?
Has the new board passed its factory QDR tests at digicom?
Cheers
Jack
On Mon, 21 Sept 2026 at 23:38, Dale Gary <[email protected]>
wrote:
Hi All,
We've been running our EOVSA correlator for nearly a decade
now, but one of our ROACH2 boards has finally failed. We
replaced it with another ROACH2 and installed the same KatADCs
moved from the failed ROACH2. However, the new ROACH2 board
fails its QDR test on all three QDRs. Are there some settings
we have forgotten? Any other ideas?
Thanks,
Dale Gary
--
You received this message because you are subscribed to the
Google Groups "[email protected]" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to [email protected].
To view this discussion visit
https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAE1wuhZ%3DKb2n%2BDJ573t%2BaBj2yZb0ExKfGLn_BAvi4Mt4_u7uoA%40mail.gmail.com
<https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAE1wuhZ%3DKb2n%2BDJ573t%2BaBj2yZb0ExKfGLn_BAvi4Mt4_u7uoA%40mail.gmail.com?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google
Groups "[email protected]" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to [email protected].
To view this discussion visit
https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAG1GKSktCVbaUbUSnKAP%2BWfoFwzgzE45GuwUGu55dtR4OW%3DZMQ%40mail.gmail.com
<https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAG1GKSktCVbaUbUSnKAP%2BWfoFwzgzE45GuwUGu55dtR4OW%3DZMQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google
Groups "[email protected]" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [email protected].
To view this discussion visit
https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAPDrt26tig%3DEV7WMJS4G6X0GuOT4hD-JueC37mRHXbRcnHag3g%40mail.gmail.com
<https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAPDrt26tig%3DEV7WMJS4G6X0GuOT4hD-JueC37mRHXbRcnHag3g%40mail.gmail.com?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google Groups
"[email protected]" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion visit
https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/9db7b019-b808-4a47-bbb2-046e6cae09d8%40ieee.org.
