Re: [ARTIQ] FW: initial specification of the project

[Thomas Harty]

Dear Greg,

Thank you for the detailed Feedback.

The key question here seems to be: will the cross-talk between the clock line 
and the digital communications lines be sufficiently bad to prevent one from 
recovering a low-noise analog clock from a reference signal sent over the 
backplane? If it is then we will want an external clock, otherwise we won’t.

It seems to me that the answer is likely to be that, while there will be a 
significant level of cross-talk -- particularly at higher frequencies--, it 
shouldn’t be sufficiently bad to prevent one from recovering a clean clock. Let 
me make the following naïve argument, and maybe you can point out where I am 
wrong

In short, the argument will be this: we will model the digital communication as 
a constant stream of random data, with a white noise power spectral density 
(PSD). As a result, the effect of cross-talk will be to increase the phase 
noise of the 10MHz analog clock. We will estimate this phase noise contribution 
and compare it to the phase noise of the DAC chip to see if it’s significant. 
In detail:


1)Let’s assume that our reference clock frequency is 10MHz

2)Assume that on the AMC/FMC board we lock a 100MHz VCXO to the 10MHz with 
a 1kHz loop bandwidth

3)We are only sensitive to cross-talk-induced noise at offset frequencies 
of <~1kHz from the 10MHz reference, since higher offset frequencies are 
attenuated by the loop.

4)To avoid degrading the DAC’s performance, we would thus like the noise on 
the 10MHz signal to be lower than -155dBc/Hz, corresponding to the DAC’s noise 
@ 1kHz offset frequency, scaled to a 10MHz carrier.

5)Let’s assume that we do all our backplane communication at 2GSPS and that 
we send a constant stream of data.

6)Assume that all communication data is properly scrambled, so that its 
power spectrum is approximately white.

7)Further, make the approximation that all power from the digital 
communication lines is contained within a bandwidth between DC and 1GHz (the 
Nyquist frequency for the digital communications clock).

8)From (6) and (7) the power spectral density (PSD) for the backplane data 
is roughly -90dBc/Hz, taking our reference (0dBc) to be the power in the 10MHz 
reference clock signal (i.e. we assume the same power in both signals, but that 
the digital communication power is spread over a wide bandwidth due to 
scrambling)

9)Assuming a DC-balanced line encoding, the PSD for the communication data 
will actually roll-off at low frequencies. A quick-and-dirty MatLab simulation 
based on a PRBS of data @2GHz, encoded using 8b10b, suggests that the roll-off 
should be >30dB @ 10MHz

10)  As a result, so long as the cross-talk between the clock line and the 
digital communication lines (i.e. the S12 between the clock line and the data 
lines) for a 10MHz signal  is <~35dB (which would be extremely bad for a 10MHz 
signal!) no degradation of the DAC’s performance is expected as a result of 
cross-talk-related noise. In a well-designed backplane, the cross-talk should 
be far less than this @10MHz (see, e.g. 
http://www.harting-usa.com/fileadmin/harting/documents/public/40GBWhitePaper_01.pdf
 where the cross-talk was measured to be <60dB at 10MHz). Note that we are only 
sensitive to the cross-talk at frequencies close to 10MHz, since higher 
frequencies (e.g. 1GHz) can easily be filtered off. Thus, the fact that the 
cross-talk will be worse at higher frequencies isn’t a problem for us.

11)  Note that, in practice, most of the time the signal would actually just be 
a repeated ‘,’ (idle) as the volume of data is expected to be relatively low. 
The spectrum for the idle signal has no power at 10MHz at all (it’s periodic), 
so point (8) is actually probably very pessimistic

This is obviously just a quick order-of-magnitude calculation, but it does 
suggest that we should be able to get away with distributing the analog clock 
over the backplane if we’re careful with how we do our digital communication. 
As I said, we’re currently building hardware to test how well this works in 
practice though…

Note that to minimize EMI/cross-talk, it’s likely to be important to use a 
high-frequency communication clock. This could be a strong reason to consider 
using GTX for backplane communication...
Best,

Tom



From: kaspr...@gmail.com [mailto:kaspr...@gmail.com] On Behalf Of Grzegorz 
Kasprowicz
Sent: 08 April 2016 11:34
To: Thomas Harty
Cc: Slichter, Daniel H. (Fed); artiq@lists.m-labs.hk
Subject: Re: [ARTIQ] FW: initial specification of the project

Re: [ARTIQ] FW: initial specification of the project

[Grzegorz Kasprowicz]

One more thing
In case of ZynQ, instead of ZU11, we have smaller GTX count version - ZU7 in
the same package.
Its cost is about 700-750$.
So we can install ZU7 by default, and when necessary, upgrade it to ZU11.
Greg

-Original Message-
From: Grzegorz Kasprowicz [mailto:gkasp...@elka.pw.edu.pl] 
Sent: Friday, April 08, 2016 2:00 PM
To: 'Sébastien Bourdeauducq' 
Cc: 'Grzegorz Kasprowicz' ; 'Slichter, Daniel H. (Fed)'
; artiq@lists.m-labs.hk
Subject: RE: [ARTIQ] FW: initial specification of the project

Well, I compared highest speed grades (-3) in large packages :)
XC7K325T-3FFG900E  costs more than ZU11 in similar speed grade.
Personally, I like this disgusting ZynQ stuff. I didn't have any special
troubles to make it running. And it gives very handy feature of FPGA upgrade
over Ethernet :) In this way you can keep all FPGA bit streams on single nfs
server and load them at the startup.
THE CPU and FPGA are independent, so you can boot CPU without the bitstream,
connect to nfs, grab the bitteam and load it. So you can treat it as CPU
which simply loads the FPGA and from programmable logic you treat it as
ordinary Kintex device.
And you get bunch of IOs for free which can be controlled over Ethernet.
In case of complex system it is very desirable because you keep all the
settings in single location and to upgrade it don't have to walk with
programmer from board to board.
We have a system which consists of 22 FPGAs in single box and all FPGA files
are kept on single USB drive. The system is installed in UK and upgraded
once a month and it is real pleasure to work in such manner.
Another system is CMS muon trigger which consists of thousands of FPGAs and
they all are loaded at the startup. It's much easier to keep control on
bitstream versions of individual chips.
The only alternative update channel in MTCA is either JTAG or I2C.
With I2C you need about 30hours to programm the config FLASH. In case of
JTAG, not every MTCA crate has JSM.
Greg


-Original Message-
From: Sébastien Bourdeauducq [mailto:s...@m-labs.hk] 
Sent: Friday, April 08, 2016 12:13 PM
To: Grzegorz Kasprowicz 
Cc: 'Grzegorz Kasprowicz' ; 'Slichter, Daniel H. (Fed)'
; artiq@lists.m-labs.hk
Subject: Re: [ARTIQ] FW: initial specification of the project

On Friday, 8 April 2016 11:53:25 AM HKT Grzegorz Kasprowicz wrote:
> Btw,
> ZU11 FPGA costs more or less the same as 7K325 and offers almost twice 
> more logic resources. The price of ZU11 is $1,376.00 at 100pcs.
> Since we will buy such quantity for our CBM project (Fair facility in 
> GSI), we can use that step pricing in this case as well.

Even on Digikey in single quantities, 7K325T starts at $898.75 (and the
cheapest one in FF which seems to be recommended for the full transceiver
bandwidth is $1122.50) and we don't have to deal with any of this disgusting
Zynq stuff.

Sébastien



___
ARTIQ mailing list
https://ssl.serverraum.org/lists/listinfo/artiq

Re: [ARTIQ] FW: initial specification of the project

[Grzegorz Kasprowicz]

Anyway, we have already plenty of things to do so I'd relay on existing
backplane at the moment:)

On 8 April 2016 at 12:51, Grzegorz Kasprowicz  wrote:

> Well, I don't have such template so all must be done from scratch.
> It's really big piece of PCB, we need at least 6..8 layers so just tooling
> is a few k$.
> The AMC backplane offered by Schroff has at least 20 layers.
> Greg
>
> On 8 April 2016 at 12:39, Sébastien Bourdeauducq  wrote:
>
>> On Friday, 8 April 2016 12:37:02 PM HKT Grzegorz Kasprowicz wrote:
>> > Modification of the backplane is quite difficult and expensive.
>>
>> If we have a minimalistic backplane with just power, maybe IPMI, 4x
>> differential pairs between MCH and each AMCs, and clock - why is that
>> particularly expensive?
>>
>> Sébastien
>>
>>
>
___
ARTIQ mailing list
https://ssl.serverraum.org/lists/listinfo/artiq

Re: [ARTIQ] FW: initial specification of the project

[Sébastien Bourdeauducq]

On Friday, 8 April 2016 12:37:02 PM HKT Grzegorz Kasprowicz wrote:
> Modification of the backplane is quite difficult and expensive.

If we have a minimalistic backplane with just power, maybe IPMI, 4x 
differential pairs between MCH and each AMCs, and clock - why is that 
particularly expensive?

Sébastien

___
ARTIQ mailing list
https://ssl.serverraum.org/lists/listinfo/artiq

Re: [ARTIQ] FW: initial specification of the project

[Grzegorz Kasprowicz]

Modification of the backplane is quite difficult and expensive. You have to
have really good reason and money to pay to NAT, ELMA or other company.
But the good news is that together with NAT we are working on AMC backplane
optimized for timing distribution.
So additional layers of PCB are dedicated to the clock distribution.
Greg

On 8 April 2016 at 12:25, Sébastien Bourdeauducq  wrote:

> On Friday, 8 April 2016 12:02:43 PM HKT Grzegorz Kasprowicz wrote:
> > The MTCA backplane is not designed to distribute RF or analog clocks.
> >
> > Definitely they cannot be directly used to feed DACs or RF modules.
>
> Maybe we can modify the backplane, and get rid of what we do not need
> (Ethernet, AMC-to-AMC pairs, ...) and improve the clock distribution?
>
> Sébastien
>
>
___
ARTIQ mailing list
https://ssl.serverraum.org/lists/listinfo/artiq

Re: [ARTIQ] FW: initial specification of the project

[Sébastien Bourdeauducq]

On Friday, 8 April 2016 11:53:25 AM HKT Grzegorz Kasprowicz wrote:
> Btw,
> ZU11 FPGA costs more or less the same as 7K325 and offers almost twice more
> logic resources. The price of ZU11 is $1,376.00 at 100pcs.
> Since we will buy such quantity for our CBM project (Fair facility in GSI),
> we can use that step pricing in this case as well.

Even on Digikey in single quantities, 7K325T starts at $898.75 (and the 
cheapest one in FF which seems to be recommended for the full transceiver 
bandwidth is $1122.50) and we don't have to deal with any of this disgusting 
Zynq stuff.

Sébastien

___
ARTIQ mailing list
https://ssl.serverraum.org/lists/listinfo/artiq

Re: [ARTIQ] FW: initial specification of the project

[Sébastien Bourdeauducq]

Just to clarify this: We do need a pretty large FPGA (7K325T) on the AMC cards 
due to the resource requirements of signal processing. Then we have 16 GTX 
12.5Gbps transceivers, but if we want to make the two FMCs identical and 
capable of receiving fast 4-channel DACs with 8 lanes, then we have no more 
transceivers. So the IOSERDES option is still interesting.

Greg - How many data lanes can we easily have between the MCH and each AMC 
with a star topology backplane?

Sébastien


On Wednesday, 30 March 2016 9:33:51 PM HKT Grzegorz Kasprowicz wrote:
> Hi all
> I like that idea - in this way we can do it really affordable with lower end
> FPGAs and still compatible with MTCA.
> If we place all RF stuff on the modules, with RF connectors mounted directly
> on the front panel, this could work.
> How we would deliver the clocks? Using external SMA connectors?
> The only think I worry about is performance of the DACs.
> ZynQ US+ can run IOserdes with almost 2Gbit/s performance - one can make
> 1000base-x directly on IO pins using oversampling technique.
> Greg
> 
> -Original Message-
> From: Sébastien Bourdeauducq [mailto:s...@m-labs.hk]
> Sent: Wednesday, March 30, 2016 1:50 PM
> To: Grzegorz Kasprowicz 
> Cc: 'Slichter, Daniel H. (Fed)' ; 'Grzegorz
> Kasprowicz' ; artiq@lists.m-labs.hk
> Subject: Re: [ARTIQ] FW: initial specification of the project
> 
> On Tuesday, 29 March 2016 11:55:19 PM HKT Grzegorz Kasprowicz wrote:
> > - they can be used immediately with existing OSHW carriers like AFC/AFCK.
> 
> AFCK may be overkill. We are still working on evaluating the FPGA resource
> requirements.
> 
> > - it could be hard to fit FPGA, supply, DACs and several RF modules,
> > all on single dual width AMC, especially when shielding is required.
> > RTM relaxes these constraints
> > - on AMC+RTM you can place 8 ADC channels + 8 DAC channels + 8 RF
> > modules. In case of single AMC board it would be hard to achieve such
> > channels density.
> 
> How about this:
> * we reduce the number of channels per AMC to 4 DACs + 4 ADCs
> * we can therefore use a smaller FPGA. Communication lanes to the master
> board are relatively cheap if we put them on IOSERDES.
> * the power density and cooling requirements are also reduced.
> * for the RF daughter cards, we use a custom form factor that can use at
> least
> 2/3 of the AMC front panel
> * connectors between the DSP card and the RF daughter card are 2mm header
> and 8x SMP
> * the rest of the AMC front panel used for (optional, runtime selectable)
> clock input and some TTLs.
> 
> Sébastien


___
ARTIQ mailing list
https://ssl.serverraum.org/lists/listinfo/artiq