Hi Eamon,
I hope Jack is getting some well deserved sleep.
To the best of my knowledge there is no PMBus support in the FPGA.
Sorry if you've already seen the Power and Schematic sections of the SNAP's wiki
https://casper.ssl.berkeley.edu/wiki/SNAP
but they have links to files with:
PDF schematics with markup notes:
https://casper.berkeley.edu/wiki/images/e/e3/SNAPv2_schematics_with_markup_notes.pdf
"The file above has extra documentation notes added to clarify how the
hierarchical schematics map to the finished board. In particular, there
are many additional notes about the different DC-DC converters."
SNAPv2 power supply block diagram and programming
https://casper.berkeley.edu/wiki/images/c/c3/SNAPv2_TI_power.pdf
TI power controller detailed configuration
https://casper.berkeley.edu/wiki/images/c/c6/ADDR_52d_and_53d_sequence_config.pdf
I hope you won't have to deal with any of that sort of stuff but in case you do
I
wanted to make sure you knew of those files.
good luck,
Matt
On Fri, 23 Jul 2021, Eamon Egan, Mr wrote:
Date: Fri, 23 Jul 2021 02:33:13 +0000
From: "Eamon Egan, Mr" <[email protected]>
Reply-To: [email protected]
To: "[email protected]" <[email protected]>
Subject: RE: [casper] SNAP error: no programming informs yet. Odd?
Hi Jack,
thanks… so just to be sure: if the nets connecting to the 40-pin header pins
are simply labelled GPIOxx,
you’re saying you don’t think they are used for anything (even though they are
all connected to the FPGA
(U25-5) on page 14)?
While power cycling is not a first choice … and of course it won’t help an
underlying heat problem if that’s
what it is, it would be good to have it as a backup option.
Is there any SNAP PMBus support in any of the casperfpga or related code? If
there were examples of
addressing these chips (UCD9248PFC, U40 and U42) over PMBus, it would make it a
much simpler task to write
code to power cycle the loads. I’m asking because I didn’t seem to find any,
but I’m very unfamiliar with the
code.
Otherwise, a wire would be needed to connect a GPIO line to the reset line…
which certainly has the advantage
of simplicity.
Eamon
From: Jack Hickish <[email protected]>
Sent: July 22, 2021 2:12 PM
To: casper <[email protected]>
Subject: Re: [casper] SNAP error: no programming informs yet. Odd?
Hi Eamon,
Good question....
The schematics are here --
https://casper.ssl.berkeley.edu/wiki/images/8/8e/DAB-HERALD_Schematic_revC.pdf
--
and the relavant header is HDR40
[IMAGE]
It looks like the PMBUS signals are on there already, should you want to use
them. I think you'll find that
in the event of overheating, PMBUS_ALERT is asserted shortly before everything
shuts down - maybe you can use
this to deprogram the SNAP (you could do this with FPGA_PROG_B if you don't
want to do it through
borphserver) before the lock up even occurs.
Otherwise, any pins which don't go to nets DONE33V, FPGA_PROG_B, PMBUS*, JTAG*,
or SPI* are (I think) fair
game.
Cheers
Jack
On Thu, 22 Jul 2021 at 18:53, Eamon Egan, Mr <[email protected]> wrote:
@Jack, is there a list of which RPi GPIO pins are (un/)used in the
standard SNAP board setup?
They all seem to connect to the FPGA, but it’s not evident from the
schematic whether they’re all
used for something.
Eamon
From: Jack Hickish <[email protected]>
Sent: July 22, 2021 1:51 PM
To: casper <[email protected]>
Subject: Re: [casper] SNAP error: no programming informs yet. Odd?
In my experience the first thing to go in the event of overheating will be the
power regulators, which
will trigger a shutdown of one (or more) voltage rails. I don't think there is
an easy way to recover
from this short of a hard power cycle. You might be able to wire one of your
RPi GPIOs to a power
controller reset line, if such a thing exists somewhere and is able to reset
the FPGA without killing
the RPi power. Alternatively, you should (in theory, at least) be able to
interact with the power
controllers via their programming header. You'd have to figure out how to get
the RPi talking PMBus,
and what commands you'd need to send.
Of course, if you're overheating the regulator chips regularly, who knows what
else on the board might
be getting injured.....
Cheers
Jack
On Thu, 22 Jul 2021 at 18:12, Tristan Ménard <[email protected]>
wrote:
Hi,
After further testing, it seems that the problem is likely a symptom of
the SNAP
overheating inside the case. We were able to reproduce the error with the
SNAP inside a
thermally insulated cooler. Unfortunately, restarting the borphserver
after cooling down
the SNAP did not clear the error. We tried a few times, but the only fix
that seems to work
(at least some of the time) still is to power cycle the SNAP and RPi. I
have checked the
FPGA temperature logs but they never seem to exceed 65C. It may be the
power controllers
that are overheating? Is there a way to reset the power controllers on
the SNAP board
without a hard power cycle perhaps?
Thanks,
Tristan
From: Tristan Ménard
Sent: July 22, 2021 11:24 AM
To: Jack Hickish; casper
Subject: RE: [casper] SNAP error: no programming informs yet. Odd?
Hi Jack,
We’re using a RPi 4 model B with 4GB of RAM and the length of ribbon cable
between the SNAP and
the RPi is about 6 inches. The ribbon cable is 28 AWG. The error never occurred
while testing the
system with the exact same hardware in the lab prior to installing the SNAP+RPi
inside the case
with the power unit.
Thankfully, the RPi remains completely responsive so, next time we encounter
this error, we’ll
definitely try your suggestion of restarting the borphserver.
Kind regards,
Tristan
From: Jack Hickish
Sent: July 22, 2021 8:32 AM
To: casper
Cc: Tristan Ménard
Subject: Re: [casper] SNAP error: no programming informs yet. Odd?
Hi Cynthia, Tristan,
What model RPi are you using, and what type/length of ribbon cable is used to
connect the Pi to
the SNAP. Were both of these known-good hardware prior to installation in the
case?
Re. error recovery -- at the very least I would have thought restarting
borphserver (probably
`/etc/init.d/tcpborphserver3 restart` should work) if the error doesn't
completely lock up the Pi
forever.
Cheers
Jack
On Thu, 22 Jul 2021 at 12:58, Hsin Cynthia Chiang, Prof <[email protected]>
wrote:
Hi all,
I'm sending the email below on behalf of a colleague who encountered some
snags while
trying to sign up for the CASPER mailing list. In addition to the
information in
Tristan's email, I'll also add that fully power cycling the SNAP+RPi is
the only way
that we've found to clear the error, but even that doesn't work 100% of
the time.
Apart from helping us identify the root cause, does anyone know of a
gentler way to
clear the error without power cycling?
thanks!
Cynthia
--------------------------
Hello,
I am diagnosing a SNAP board error that has popped up while conducting
field tests.
We are using a 14-bit SNAP board along with a Raspberry Pi on a system
that has
successfully been used in the field before. The error message is as
follows: “ERROR
127.0.0.1 transport_katcp.py:594 - 127.0.0.1: no programming informs yet.
Odd?”,
which is repeatedly printed to the Raspberry Pi’s terminal screen. It
occurs while
initializing the SNAP board with casperfpga.upload_to_ram_and_program().
I’ve
attached a picture of the error message for reference.
The main difference between previous successful tests of this system and
the current
iteration is that the enclosure containing the SNAP board and the
Raspberry Pi is
placed inside a large pelican case that also houses our methanol fuel
cell power
unit. Some of the potential causes we’ve explored are overheating and RFI
from the
power unit. I’d greatly appreciate any input that folks on this mailing
list might
have regarding what might trigger this error.
Thanks,
Tristan Ménard <[email protected]>
--------------------------
--
You received this message because you are subscribed to the Google Groups
"[email protected]" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
[email protected].
To view this discussion on the web
visithttps://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/31104cf0-b9aa-a517-92b5-43b78dd3511e%40mcgill.
ca.
--
You received this message because you are subscribed to the Google Groups
"[email protected]" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
[email protected].
To view this discussion on the web
visithttps://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/YT1PR01MB46946AD67C933CC6C0D7154DA9E49%40YT1PR
01MB4694.CANPRD01.PROD.OUTLOOK.COM.
--
You received this message because you are subscribed to the Google Groups
"[email protected]" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
[email protected].
To view this discussion on the web
visithttps://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/YT1PR01MB46942A08C15F7562F4EB42F3A9E49%40YT1PR
01MB4694.CANPRD01.PROD.OUTLOOK.COM.
--
You received this message because you are subscribed to the Google Groups
"[email protected]"
group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
[email protected].
To view this discussion on the web
visithttps://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAG1GKSkmRCfxDAfCwODq7LcGPUHqEpszd9Ggh0_jhL_3r
as7AA%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups
"[email protected]"
group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
[email protected].
To view this discussion on the web
visithttps://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/YQBPR0101MB41636C800567FBDBFC6E99009DE49%40YQB
PR0101MB4163.CANPRD01.PROD.OUTLOOK.COM.
--
You received this message because you are subscribed to the Google Groups
"[email protected]" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
[email protected].
To view this discussion on the web
visithttps://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAG1GKSkmqQSAfEQOa7Hw%3DEaf5aP_9RU%2BHV%2BiMnB
A5ZvBQOzyyQ%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups
"[email protected]" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
[email protected].
To view this discussion on the web
visithttps://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/YQBPR0101MB4163831FBFA582EA9D08B01D9DE59%40YQB
PR0101MB4163.CANPRD01.PROD.OUTLOOK.COM.
--
You received this message because you are subscribed to the Google Groups
"[email protected]" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/alpine.WNT.2.20.2107221948190.3152%40DESKTOP-LM78PCL.
