Re: [ARTIQ] sayma gateware updates

[Sébastien Bourdeauducq]

On Tuesday, July 26, 2016 06:20 AM, j arl wrote:

Kintex UltraScale SYSMONE1 interface supports digitization of 4 power
supply voltages and up to 17 external analog signals. The resulting
data can be readout by FPGA (DRP), exposed to MMC using I2C and
readout using JTAG.

My read of the docs indicates that readout by I2C could be done by MMC
without involving FPGA gateware.


But then this requires the MMC to be used. I thought that the plan was 
to program the power supply to be on at all times, and then ignore the MMC?


Sébastien

___
ARTIQ mailing list
https://ssl.serverraum.org/lists/listinfo/artiq

Re: [ARTIQ] sayma gateware updates

[j arl]

Sebastien, Please correct me if I fail to recall truce we reached last
summer with respect to microTCA gadetry. Here's a recap.

* +12V is always-on for ARTIQ AMC cards
* WUT is responsible for implementing MMC and ensuring uTCA.4 compliance
* m-labs will not be responsible for MMC functionality
* m-labs will work with WUT to the extent required so that WUT can do this

The aim is for ARTIQ MCH and AMC devices to "play well" with AMCs from
other vendors. I very much value your opinion on these matters and
acknowledge your disinterest. The path I've selected includes MMC support.
I'd like to avoid rehashing this debate.

##
### Regarding: (a) serial JTAG

>SB For this technique, one could also build the JTAG adapter
>into the board - use a FT2232H chip and have a small USB
>connector on the front panel.

As I understand it m-labs requires both interfaces on the FT2232H chip for
configuration of the FPGA. One interface for JTAG and one interface for
UART. The FPGA will have dedicated flash chip.

The MMC uP requires a UART (no JTAG). This could be supplied by a second
independent FT2232H. The MMC uP will have its own dedicated flash chip.

Or, use a quad interface FT4232.

There is limited real-estate on the front-panel of sayma_motherboard -- 17
SMA are required! I don't expect there's room for USB.

As I understand it the (a) FPGA .bit update process is
1. JTAG write of SPI-master-core .bit to FPGA
2. FPGA reboots
3. JTAG write of ARTIQ-runtime .bit to flash (via FPGA-SPI)
4. FPGA reboots (using ICAP)
5. FPGA loads ARTIQ-runtime .bit from flash

Are there any additional considerations to keep in mind regarding (a)?


### Regarding: (b)
>>JWB: What implementation of (b) does m-labs have in mind for sayma? Since
>>it's DRTIO it involves the Port4 GTH from metlino_motherboard.
>
>SB: The PC sends bitfiles to the root Metlino over the Metlino's
>front panel Ethernet connector (the same used in regular operation
>of the system). The Metlino receives its own bitfile from there,
>and distributes the other bitfiles to the whole DRTIO system
>using the existing communications links (fiber/backplane).

OK. Let's scrap (c) to use Ethernet to update FPGA .bit.

>I don't think we really need the complexity of a fail-safe flash.
>A board that fails to boot will be clearly shown (or rather, not
>shown) during DRTIO device discovery, and it can then be manually
>serviced by reflashing over USB. And I expect this situation to be rare.

I'm hopeful this is rare as well. The front-face of sayma_motherboard will
be very dense and extracting the AMC card may be disruptive to the rats
nest of coaxial cables we commonly have in the lab. I propose the following
compromise.


m-labs supports automatic fallback to Golden Master .bit using MultiBoot.
At factory, (a) would be used to load Golden Master .bit and user-defined
.bit. Subsequent updates using (b) would modify only user-defined .bit. MMC
will not touch FPGA .bit. This uses built-in Xilinx methods.

Likewise for MMC. At factory, WUT uses (a) to load Golden Master .elf and
user-defined .elf. Subsequent updates using (b) would modify only
user-defined .elf. FPGA will not touch MMC .elf.


### Regarding: (c)

Ethernet will not be used to update FPGA .bit.

WUT would like to use (c) for interacting with MMC. In a chat with Greg we
came up with the following approach that shares the Port0 Ethernet
interface between FPGA and MMC. Presently, m-labs is not being asked to
provide FPGA gateware to support (c). The following would apply to both
metlino_motherboard and sayma_motherboard.

Port0 <-> MAX24287 <-> MUX <-> {FPGA-RGMII or MMC-MII}
* MUX-select power-on default chooses FPGA
* MMC can toggle MUX-select to choose MMC-MII in response to IPMI message
(Tongue 3 has has I2C interface and SPI interface)

This enables:
* WUT can update MMC .elf using Ethernet interface
* WUT can debug MMC using using Ethernet interface
* future option for interaction with FPGA via Ethernet interface

>So can the existing DRTIO links. What does in-crate Ethernet
>bring to the table? And if the system has multiple crates,
>we'd need to connect an Ethernet cable between them as opposed
>to just the fiber, or invent another gadget to somehow tunnel
>the in-crate Ethernet into the fiber.

m-labs is welcome to ignore in-crate Ethernet and use DRTIO.

###
### Regarding sensor readout

Greg:
>In proven AFC/AFCK designs the I2C interface of sensors
>is connected to both MMC and FPGA so you can read the
>sensors from any of them.

Do we want to use I2C or SPI for sensors? Or both? What agreement is needed
between m-labs and WUT so both FPGA and MMC can read from sensors?

If I2C do ARTIQ and MMC both support MultiMaster?
http://www.i2c-bus.org/multimaster/

If SPI is there something like MultiMaster?

Greg:
>>Well, we already have Ethernet switch in the crate, delvered to all
>>interesting places.
SB:
>But we do not 

Re: [ARTIQ] sayma gateware updates

[Sébastien Bourdeauducq]

On Sunday, July 24, 2016 10:04 PM, Grzegorz Kasprowicz wrote:

This can be also done using single FLASH and multi boot. Xilinx has
such option but never tested it.


I have used some of the multiboot features on Spartan-6, and it worked 
fine. Never tried on other FPGAs.



Well, we already have Ethernet switch in the crate, delvered to all
interesting places.


But we do not have it on any of our boards. And if we don't depend on 
Ethernet, it means we can remove it completely from the crate and/or 
backplane, should we decide to design our own. And Ethernet does not 
work in multi-crate systems, and Kasli, unless a) a separate cable is 
connected and b) Ethernet cruft is added to Kasli.



Since diagnostic data are not timing critical, you could devote main
fibre channel to real time traffic.


Diagnostic data is also low-bandwidth, and dedicating some fixed small
amount of the fiber traffic to such functions is also a solution.

Sébastien
___
ARTIQ mailing list
https://ssl.serverraum.org/lists/listinfo/artiq

Re: [ARTIQ] sayma gateware updates

[Grzegorz Kasprowicz]

On 24 July 2016 at 14:19, Sébastien Bourdeauducq  wrote:

> Hi,
>
> On Sunday, July 24, 2016 07:33 PM, Grzegorz Kasprowicz wrote:
>
>> > When receiving a bitfile, a board (Metlino/Sayma) writes it to its
>> > flash memory (via a SPI master core inside the FPGA) and then reboots
>> > (using ICAP) to load the new bitfile.
>> We used this approach some
>> time ago in AFC/AFCK designs. But it had no failover protection. For
>> this reason I added second FLASH that MMC can switch and do failsave
>> recovery of communication link.
>>
>
> In the rare case of broken flash contents, one can reflash using the USB
> link. And for development I guess the USB link will be used anyway.
>
Fully agree. I'm talking about future systems with many AMC boards..


>
> MTCA supports remote upgrade via IPMB, but it takes ~20h to write to
>> config FLASH.
>>
>
> This is another reason we should ignore IPMB.
>
That's what we do. I mentioned it just to give list of all possible ways.

>
> In our case, main remote update channel is over Ethernet, where FPGA
>> programs its own FLASH by custom gateware while keeping fail-safe
>> FLASH intact.
>>
>
> I don't think we really need the complexity of a fail-safe flash. A board
> that fails to boot will be clearly shown (or rather, not shown) during
> DRTIO device discovery, and it can then be manually serviced by reflashing
> over USB. And I expect this situation to be rare.

OK, we can ignore it or add option to mount second FLASH.
This can be also done using single FLASH and multi boot. Xilinx has such
option but never tested it.

>
> In case of AMC boards, we consider adding Ethernet PHY
>> to keep compatibility of AMC standard (PORT0). Together with simple
>> open source MAC we gain communication channel over PORT0 and MCH
>> switch. This can be used for setup, diagnistics and update.
>>
>
> So can the existing DRTIO links. What does in-crate Ethernet bring to the
> table? And if the system has multiple crates, we'd need to connect an
> Ethernet cable between them as opposed to just the fiber, or invent another
> gadget to somehow tunnel the in-crate Ethernet into the fiber.
>
Well, we already have Ethernet switch in the crate, delvered to all
interesting places.
By having separate config/update/diagnostic channel, you wouldn't have to
implement it in DRTIO.
Since diagnostic data are not timing critical, you could devote main fibre
channel to real time traffic.

>
> Since the PHY chip supports RGMII and MII standards and on the other
>> side has 1000BASE-X, we can use its MII port to connect to MMC chip
>> and in this way gain for free additional update channel where MMC can
>> do FLASH programming, direct FPGA ocnfig, diagnistic and fail
>> recovery functions.
>>
>
> It's not free: it needs to be developed, tested and maintained. And it
> bloats the design.
>
I'm taking now about HW implementation. I'd like to test it and add support
to the open source MMC, maintained by my team.
So you won't have to worry about it.

>
> So the procedure would be simple: MMC diables
>> FPGA by PROGRAM_B line, than takes control over MII interface of PHY
>> chip. The chip does translation between 100mbit and 1Gbit link speed
>> and in this way we can have quick access to both FLASH and all
>> resources of the AMC board (supply currents and voltages,
>> thermometers, ID, etc)
>>
>
> The FPGA can handle all this already.
>
Yes, except for failure, but now it is not real issue.


>
> But in final production system where several AMC boards are present,
>> it may make sense to boot all FPGAs over Ethernet at the startup
>> from single file location. In this way it's easier to keep control
>> over file versions. FLASH chips won't be used at all in this
>> scenario.
>>
>
> I agree that this approach to bitstream management nice. But it does
> require a lot of gadgetry (microcontroller, in-crate Ethernet, Ethernet
> links between crates, the tricks you mentioned to switch Ethernet between
> FPGA and uC), which is way beyond the complexity of a flash chip. We can
> embed version numbers in bitstreams and have the software automatically
> check them and reflash as needed, which would give similar behaviour.
>
That's true and this will satisfy our needs now.
As I said, Ethernet in crate is already there.
I'm talking about future, where we may want to have more advanced and
independent management system.
Greg

>
> Sébastien
>
___
ARTIQ mailing list
https://ssl.serverraum.org/lists/listinfo/artiq

Re: [ARTIQ] sayma gateware updates

[Sébastien Bourdeauducq]

Hi,

On Sunday, July 24, 2016 07:33 PM, Grzegorz Kasprowicz wrote:

> When receiving a bitfile, a board (Metlino/Sayma) writes it to its
> flash memory (via a SPI master core inside the FPGA) and then reboots
> (using ICAP) to load the new bitfile.
We used this approach some
time ago in AFC/AFCK designs. But it had no failover protection. For
this reason I added second FLASH that MMC can switch and do failsave
recovery of communication link.


In the rare case of broken flash contents, one can reflash using the USB
link. And for development I guess the USB link will be used anyway.


MTCA supports remote upgrade via IPMB, but it takes ~20h to write to
config FLASH.


This is another reason we should ignore IPMB.


In our case, main remote update channel is over Ethernet, where FPGA
programs its own FLASH by custom gateware while keeping fail-safe
FLASH intact.


I don't think we really need the complexity of a fail-safe flash. A 
board that fails to boot will be clearly shown (or rather, not shown) 
during DRTIO device discovery, and it can then be manually serviced by 
reflashing over USB. And I expect this situation to be rare.



In case of AMC boards, we consider adding Ethernet PHY
to keep compatibility of AMC standard (PORT0). Together with simple
open source MAC we gain communication channel over PORT0 and MCH
switch. This can be used for setup, diagnistics and update.


So can the existing DRTIO links. What does in-crate Ethernet bring to 
the table? And if the system has multiple crates, we'd need to connect 
an Ethernet cable between them as opposed to just the fiber, or invent 
another gadget to somehow tunnel the in-crate Ethernet into the fiber.



Since the PHY chip supports RGMII and MII standards and on the other
side has 1000BASE-X, we can use its MII port to connect to MMC chip
and in this way gain for free additional update channel where MMC can
do FLASH programming, direct FPGA ocnfig, diagnistic and fail
recovery functions.


It's not free: it needs to be developed, tested and maintained. And it 
bloats the design.



So the procedure would be simple: MMC diables
FPGA by PROGRAM_B line, than takes control over MII interface of PHY
chip. The chip does translation between 100mbit and 1Gbit link speed
and in this way we can have quick access to both FLASH and all
resources of the AMC board (supply currents and voltages,
thermometers, ID, etc)


The FPGA can handle all this already.


But in final production system where several AMC boards are present,
it may make sense to boot all FPGAs over Ethernet at the startup
from single file location. In this way it's easier to keep control
over file versions. FLASH chips won't be used at all in this
scenario.


I agree that this approach to bitstream management nice. But it does 
require a lot of gadgetry (microcontroller, in-crate Ethernet, Ethernet 
links between crates, the tricks you mentioned to switch Ethernet 
between FPGA and uC), which is way beyond the complexity of a flash 
chip. We can embed version numbers in bitstreams and have the software 
automatically check them and reflash as needed, which would give similar 
behaviour.


Sébastien
___
ARTIQ mailing list
https://ssl.serverraum.org/lists/listinfo/artiq

Re: [ARTIQ] sayma gateware updates

[Sébastien Bourdeauducq]

On Sunday, July 24, 2016 11:27 AM, Sébastien Bourdeauducq wrote:



It is reasonable to use the same approach to (a) and (c) on
metlino_motherboard?


I suggest (a) and (b) on all boards. (c) involves gadgetry.


And by "all boards" I also mean Kasli, which (c) would not support.

Sébastien
___
ARTIQ mailing list
https://ssl.serverraum.org/lists/listinfo/artiq

Re: [ARTIQ] sayma gateware updates

[Sébastien Bourdeauducq]

Hi,

On Sunday, July 24, 2016 07:59 AM, j arl wrote:

The bitfiles for the sayma_motherboard FPGA will be stored in flash.
We've discussed several ways of updating the bitfiles. a) serial JTAG
b) DRTIO over microTCA Port4 c) Ethernet microTCA Port0.

Update method (a) is implemented by a directly connecting a serial
cable to the board. This is well suited to factory debug and loading
the initial bitfiles. However, it's not convenient for in-field
upgrades.


For this technique, one could also build the JTAG adapter into the board 
- use a FT2232H chip and have a small USB connector on the front panel.



(b) and (c) permit in-field upgrades without external connectors.

What implementation of (b) does m-labs have in mind for sayma? Since
it's DRTIO it involves the Port4 GTH from metlino_motherboard.


The PC sends bitfiles to the root Metlino over the Metlino's front panel 
Ethernet connector (the same used in regular operation of the system). 
The Metlino receives its own bitfile from there, and distributes the 
other bitfiles to the whole DRTIO system using the existing 
communications links (fiber/backplane).


When receiving a bitfile, a board (Metlino/Sayma) writes it to its flash 
memory (via a SPI master core inside the FPGA) and then reboots (using 
ICAP) to load the new bitfile.



Greg advocates for use of ARM Cortex M3 (LPC1700 family from NXP) to
implement the microTCA MMC functionality (IPMI).


This is gadgetry that solves no real problem. And how do you program the 
microcontroller? It's not easier than loading a FPGA bitfile.


If you want temperature sensors (the only arguably useful monitoring 
feature IMO) I suggest connecting them to the FPGA and reading them out 
over DRTIO. The FPGA already contains one built-in temperature sensor, 
and an ADC with additional external channels.



Greg points out that
the M3 has a built-in MAC. If this were connected to Port0 the MMC
could be reconfigured over Ethernet. The M3 could also be connected to
the FGPA to permit loading of bitfiles.


I would not bother with this and also with Ethernet inside the crate. 
The built-in FPGA configuration mechanisms are good enough and do not 
require additional hardware.



This would enable (c).
Xilinx's recommended implementation is discussed in "Figure 3-2: Slave
Serial Mode Configuration Interface Example" of the
UltraScale Architecture Configuration guide.

It seems (b) and (c) are not mutually exclusive.


The microcontroller would have to be connected and programmed to drive 
the Mx pins of the FPGA to switch between the two modes - serial flash 
for (b) and slave serial for (c).



It is reasonable to use the same approach to (a) and (c) on
metlino_motherboard?


I suggest (a) and (b) on all boards. (c) involves gadgetry.

Sébastien
___
ARTIQ mailing list
https://ssl.serverraum.org/lists/listinfo/artiq