Hi Matt, See the commit trail at https://github.com/jack-h/mlib_devel/commits/ami-devel/xps_base/XPS_ROACH2_base/pcores/qdr_controller_v1_00_a/hdl/verilog/qdrc_infrastructure.v and in particular commits 06fa83a and 73cd65b.
In the first case, I basically played with the balancing of registering in the various pipeline chains and the phase of the IDDR sampling of data returning to the FPGA from the QDR. Here I believe the net result was to reduce the total fixed qdr latency by half (of perhaps a whole) clock cycle (obviously take the commit details over my memory on this). This got me running at 312.5 MHz, but introduced (predictably) troubles calibrating at lower clock speeds. I then added output IODELAY blocks on all the qdr output control signals (there were some on the clock and data, but I added some to the rd/wr_en signals) so that I could step the latency of the signals going to the qdr without skewing the relative arrival of the controls and data/addr. According to the git commit message, the net effect after all my commits was the ability to insert 4.8 ns delay dynamically, which seemed adequate for most use cases, and covered the range of clocks I was interested in. As a bonus, moving the data IDDR from clk180 to clk0 is a bit more timing friendly. I modified the calibration script I stole from SKA-SA a little bit -- the calibration method I use is (the creatively named) qdr_cal2 at https://github.com/jack-h/ami_correlator_sw/blob/master/ami/src/qdr.py . The general mechanism I used is to step the output delays of all qdr data/addr/control bits until any of the readback data bits are correct (or the maximum output delay is reached). Then step the input delays for each bit separately to find the centre of the eye. This certainly works at 312 MHz, and I think also works <200 (if I remember rightly). I assumed that relative trace lengths of signals going from the FPGA to the QDR did not need calibrating, which seems sound when I've tested the hardware. Note that I calibrate from a remote server running a katcp client, so it's fairly slow (I recall it takes me about ~60s to calibrate 3 qdrs on 10 boards, with rudimentary python multiprocessing over different roaches). Hope that helps, let me know if I can shed any light on anything, Cheers, Jack On Tue, 17 Mar 2015 at 17:51 Matt Strader <[email protected]> wrote: > Jack, > > When you were having problems calibrating the qdr at high clock rate, how > did you go about removing half a clock cycle latency from the interface? > > Thanks, > Matt > > On Wed, Mar 11, 2015 at 12:27 PM, Matt Strader <[email protected]> > wrote: > >> I have my own fork of mlib_devel (https://github.com/mstrader/mlib_devel), >> which is nearly identical to the current ska-sa repo. >> >> Thanks, >> Matt >> >> On Wed, Mar 11, 2015 at 12:04 AM, Andrew Martens <[email protected]> >> wrote: >> >>> I played around with ROACH2 QDR calibration a few months ago and the >>> following must be noted; >>> >>> The physical trace length between FPGA and QDR SRAM is a fixed length. >>> This fixed length equates to a different number of clock cycles, >>> depending on clock speed. >>> The interfacing firmware assumes a fixed number of clock cycles latency. >>> The calibration routine thus only works for a certain range of clock >>> speeds. >>> >>> In your case it looks like (at least) one of the interfaces is writing >>> (or reading) incorrectly, but is compensating by inserting or removing a >>> multiple of half a clock cycle of delay when reading (or writing). When >>> reading/writing via the other interface, a different number of compensating >>> half clock cycles is inserted. >>> >>> Which repo are you using? >>> >>> Regards >>> Andrew >>> >>> On Wed, Mar 11, 2015 at 8:15 AM, David George <[email protected]> >>> wrote: >>> >>>> A small chip-in - QDR doesn't work below 120 MHz: >>>> http://www.cypress.com/?id=4&rID=31542 >>>> >>>> Regards >>>> >>>> On 10 March 2015 at 22:17, Matt Strader <[email protected]> >>>> wrote: >>>> > Hi Jack, >>>> > >>>> > Thanks for your reply. I think I understand better what's going on. >>>> I was >>>> > thinking in terms of 64-bit words that get written or read each >>>> cycle, but >>>> > looking in qdr_cal(), I should probably think of two 32-bit words >>>> read or >>>> > written each cycle. Then, perhaps I shouldn't be too surprised if >>>> the order >>>> > of the two words read out doesn't match my intuition. >>>> > >>>> > Initially, I was compiling at 100 MHz, but the qdr_cal always >>>> failed. Then >>>> > I tried 200 and 250 MHz, and the qdr_cal succeeds. >>>> > >>>> > My tests go like this. For the 200 MHz design, >>>> > I call qdr_cal, and it returns True. >>>> > >>>> > Then I write this sequence of 64-bit values via the CPU: >>>> > [0, 1, 2, 3, 4, 5, ..., , 1048574, 1048575] >>>> > >>>> > I read back the sequence from the CPU, and it agrees. >>>> > [0, 1, 2, 3, 4, 5, ..., , 1048574, 1048575] >>>> > >>>> > I read it in fabric and get >>>> > [1048575<<32, 0<<32, 1<<32, 2<<32, 3<<32, 4<<32, ..., 1048574<<32, ] >>>> > >>>> > >>>> > Then I try the other direction, by writing a sequence of 64-bit >>>> values in >>>> > fabric: >>>> > [0,1,2,3,4,5, ... , 1048574, 1048575] >>>> > >>>> > I read back the sequence in fabric and it agrees: >>>> > [0,1,2,3,4,5, ... , 1048574, 1048575] >>>> > >>>> > I read it via the CPU: >>>> > nBytesPerSample=8 >>>> > struct.unpack('>{}Q'.format(nSamples), >>>> > roach.read('qdr0_memory',nSamples*nBytesPerSample)) >>>> > and get: >>>> > [1<<32, 2<<32, 3<<32, 4<<32, 5<<32, ..., 1048575<<32, 0<<32] >>>> > >>>> > So reading and writing only by the CPU is consistent, and reading and >>>> > writing only in fabric is consistent, but when I do one by the CPU >>>> and the >>>> > other in fabric, the two 32 bit words get swapped, and the sequence is >>>> > offset by one value. I actually see the same offset in a Roach1 >>>> version of >>>> > the design as well. It all seems consistent, not intermittent. >>>> > >>>> > Thanks, >>>> > Matt >>>> > >>>> > >>>> > >>>> > On Mon, Mar 9, 2015 at 10:42 AM, Jack Hickish <[email protected]> >>>> wrote: >>>> >> >>>> >> Hi Matt, >>>> >> >>>> >> The qdr_cal routine calibrates the relationship of clock and data >>>> signals >>>> >> in the link between the FPGA and QDR signals. Basically it writes >>>> test >>>> >> patterns and reads them back looking for glitches. It then changes >>>> the delay >>>> >> of IODELAY blocks so that data read from the QDR is captured >>>> reliably. >>>> >> If this calibration script runs successfully, I'm surprised you >>>> still see >>>> >> glitches with 32 bits of data swapped between read and write (both >>>> from >>>> >> PPC). This would (surely?) count as a glitch and cause the >>>> calibration >>>> >> routine to fail(?). Do you only see strange behaviour when you read >>>> from the >>>> >> FPGA fabric, or do straight write/reads from the CPU also misbehave? >>>> Is the >>>> >> behaviour the same on every read / write (i.e., not intermittent)? >>>> >> >>>> >> My experience with the QDR is that at high clock speeds (not sure how >>>> >> high, but I see this at 312 MHz) the read latency of the QDR >>>> interface isn't >>>> >> 10 clock cycles as it should be even with the minimum IODELAY >>>> configuration, >>>> >> and there are no reliable qdr_cal solutions. I solved this problem >>>> for my >>>> >> design by removing half a clock cycle latency from the interface, at >>>> which >>>> >> point the QDR would calibrate ok. What speed are you running your >>>> design at? >>>> >> To me, misordered 32-bit chunks of data sounds indicative of a >>>> latency >>>> >> problem in the qdr_cal solution, since the interface is 32 bit DDR, >>>> so half >>>> >> a clock cycle mangles 32-bit chunks of data. >>>> >> >>>> >> Good luck! >>>> >> >>>> >> Jack >>>> >> >>>> >> >>>> >> >>>> >> On Fri, 6 Mar 2015 at 23:54 Matt Strader <[email protected]> >>>> >> wrote: >>>> >>> >>>> >>> Hi all, >>>> >>> >>>> >>> I'm trying to use QDR on Roach2. I have a test design where I >>>> write to >>>> >>> QDR either in firmware or with katcp, then read it out in the >>>> firmware and >>>> >>> check the results with snapshot blocks. It mostly works, but with >>>> some >>>> >>> interesting quirks. >>>> >>> >>>> >>> I know that katcp can only write 64 bits of the 72 bit QDR word, so >>>> I >>>> >>> changed the bitbasher blocks in qdr_mask.m to move the inaccessible >>>> bits out >>>> >>> of the way to the most significant bits. Now everything seems to >>>> work well, >>>> >>> except that in some builds (seems to depend on clock rate?), the >>>> first 32 >>>> >>> bits (of the 64 bits I'm writing in a cycle) get swapped with the >>>> last 32 >>>> >>> bits sometime between writing the word and reading it back. And >>>> there is a >>>> >>> sometimes a difference in whether I am writing in firmware or by >>>> katcp to >>>> >>> whether this swap happens. >>>> >>> >>>> >>> Also, when I write with katcp starting at address zero, I find that >>>> when >>>> >>> reading out, the write started at what seems to be address 1. Is >>>> the >>>> >>> latency of QDR set to 10 cycles? >>>> >>> >>>> >>> I can live with all this, as long as it is repeatable for a given >>>> design, >>>> >>> but I thought I'd ask if anyone knows what is causing these quirks. >>>> >>> >>>> >>> Also, I understand that we should run a software calibration for >>>> qdr, >>>> >>> which seems to be qdr_cal() in qdr.py in the ska-sa:casperfpga >>>> repo. So, I >>>> >>> run this first, which seems to work for my latest compiles, but I >>>> haven't >>>> >>> noticed a difference in the results. What exactly does this >>>> calibration do? >>>> >>> >>>> >>> Thanks, >>>> >>> Matt Strader >>>> >>> UC Santa Barbara >>>> > >>>> > >>>> >>>> >>> >> >
