Re: Re: [casper] ROACH2 QDR testing address problem

[zhang laiyu]

  Thanks Jack.

  That is what I need. 

  What I want to do is using CPU interface to write data into QDR and using 
firmware interface to read data out from QDR.

  The two interface of QDR make me confused.





--

Cheers!

>
ZHANG Laiyu   
Phone(China)   010-88233078   
Cellphone(China)   13681385567
E-mail:zhan...@ihep.ac.cn
Address:   19B Yuquan Road,Shijingshan District,Beijing，China
Department:Center for Particle Astrophysics 
Office:Astrophysics Building 205
Institute of High Energy Physics, CAS  
web: http://www.ihep.cas.cn
>





-原始邮件-
发件人:"Jack Hickish" 
发送时间:2020-12-21 19:29:19 (星期一)
收件人: casper 
抄送: sebastian.jorqu...@ug.uchile.cl
主题: Re: [casper] ROACH2 QDR testing address problem


I'm not sure how that particular script / firmware works, but in general, a 
single read/write action in the ROACH2 QDR transfers 144 bits. The granularity 
of the python read writes is 32-bits, so what actually happens is that a read 
from casperfpga grabs 144 bytes, carves it up into 4x36 bit words, throws away 
the most significant 4 bits of each chunk, and then returns the appropriate 32 
bits.

Note that this makes writes from software particularly tricky -- if you don't 
write in bursts of 16 bytes, (aligned to 16-byte addresses) there is a very 
high probability that things will not work the way you want.


These lines ( this and this ) show how the 32-bit selection is implemented. The 
commit message associated with the second line linked says "MSb 32 bits are 
read first, followed by bits 32 to 63". I'm not sure how this relates to the 
convention assumed in the firmware interface. I would imagine the intention is 
that for a QDR write (which takes 2 clock cycles and writes 64 bits on each) 
the logic is (with addressing in 32-bit words):


First write cycle: most significant bits = word 0; least significant bits = 
word 1
Second write cycle: most significant bits = word 2; least significant bits = 
word 3;


I don't know of many occasions when both the CPU and firmware QDR interfaces 
are both used, so it's very possible that the relationship between the firmware 
and software interfaces is either not as intended, or is as intended but with 
an odd, undocumented mapping.


Hopefully the above helps point you in the right direction(?) Hopefully it 
doesn't just add to the confusion(?)


Cheers
Jack


On Mon, 21 Dec 2020 at 10:12, zhang laiyu  wrote:


Hi

  Thanks for your help.

  The firmware and python script had worked (qdr_basic and qdr_counter).I can 
get the plot. But I can not understand the PPC readout script in  
qdr_counter.py thoroughly .The address is really weird. After studying the wiki 
of the CASPER QDR yellow block and datasheet of the QDR chip(CY7C2565KV18), I 
still can not understand the python script very well. Why the 'PPC data' was 
picked like that? Could you give me some suggestion about where can I find more 
clear information about the addree of QDR interface to PowerPC?



 

Cheers!

>
ZHANG Laiyu   
Phone(China)   010-88233078   
Cellphone(China)   13681385567
E-mail: zhan...@ihep.ac.cn Address:   19B Yuquan Road,Shijingshan 
District,Beijing，China
Department:Center for Particle Astrophysics 
Office:Astrophysics Building 205
Institute of High Energy Physics, CAS  
web: http://www.ihep.cas.cn
>







--
You received this message because you are subscribed to the Google Groups 
"casper@lists.berkeley.edu" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to casper+unsubscr...@lists.berkeley.edu.
To view this discussion on the web visit 
https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/606c0c11.e7c3.17684c842a5.Coremail.zhangly%40ihep.ac.cn.





--
You received this message because you are subscribed to the Google Groups 
"casper@lists.berkeley.edu" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to casper+unsubscr...@lists.berkeley.edu.
To view this discussion on the web visit 
https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAG1GKSkVM8Y2Odu%2BW%2BjOMSLbZw8kaOq5jhzUEc_hkZtS2Jdmcg%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"casper@lists.berkeley.edu" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to casper+unsubscr...@lists.berkeley.edu.
To view this discussion on the web visit 
https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/58e39772.eeb0.176189e.Coremail.zhangly%40ihep.ac.cn.

Re: [casper] ROACH2 QDR testing address problem

[Jack Hickish]

I'm not sure how that particular script / firmware works, but in general, a
single read/write action in the ROACH2 QDR transfers 144 bits. The
granularity of the python read writes is 32-bits, so what actually happens
is that a read from casperfpga grabs 144 bytes, carves it up into 4x36 bit
words, throws away the most significant 4 bits of each chunk, and then
returns the appropriate 32 bits.
Note that this makes writes from software particularly tricky -- if you
don't write in bursts of 16 bytes, (aligned to 16-byte addresses) there is
a very high probability that things will not work the way you want.

These lines ( this

and
this

)
show how the 32-bit selection is implemented. The commit message associated
with the second line linked says "MSb 32 bits are read first, followed by
bits 32 to 63". I'm not sure how this relates to the convention assumed in
the firmware interface. I would imagine the intention is that for a QDR
write (which takes 2 clock cycles and writes 64 bits on each) the logic is
(with addressing in 32-bit words):

First write cycle: most significant bits = word 0; least significant bits =
word 1
Second write cycle: most significant bits = word 2; least significant bits
= word 3;

I don't know of many occasions when both the CPU and firmware QDR
interfaces are both used, so it's very possible that the
relationship between the firmware and software interfaces is either not as
intended, or is as intended but with an odd, undocumented mapping.

Hopefully the above helps point you in the right direction(?) Hopefully it
doesn't just add to the confusion(?)

Cheers
Jack

On Mon, 21 Dec 2020 at 10:12, zhang laiyu  wrote:

> Hi
>
>   Thanks for your help.
>
>   The firmware and python script had worked (qdr_basic and qdr_counter).I
> can get the plot. But I can not understand the PPC readout script in
> qdr_counter.py thoroughly .The address is really weird. After studying
> the wiki of the CASPER QDR yellow block and datasheet of the QDR
> chip(CY7C2565KV18), I still can not understand the python script very well.
> Why the 'PPC data' was picked like that? Could you give me some suggestion
> about where can I find more clear information about the addree of QDR
> interface to PowerPC?
>
>
>
>
>  Cheers!
>
> >
> ZHANG Laiyu
> Phone(China)   010-88233078
> Cellphone(China)   13681385567
> E-mail:zhan...@ihep.ac.cn
> Address:   19B Yuquan Road,Shijingshan District,Beijing，China
> Department:Center for Particle Astrophysics
> Office:Astrophysics Building 205
>
> Institute of High Energy Physics, CAS
> web: http://www.ihep.cas.cn 
>
> >
>
>
>   
>
>   
>
>   
>
>   
>
>   
>
>   
>
>   
>
>   
>
>
>
>   * *
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups "
> casper@lists.berkeley.edu" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to casper+unsubscr...@lists.berkeley.edu.
> To view this discussion on the web visit
> https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/606c0c11.e7c3.17684c842a5.Coremail.zhangly%40ihep.ac.cn
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"casper@lists.berkeley.edu" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to casper+unsubscr...@lists.berkeley.edu.
To view this discussion on the web visit 
https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAG1GKSkVM8Y2Odu%2BW%2BjOMSLbZw8kaOq5jhzUEc_hkZtS2Jdmcg%40mail.gmail.com.

[casper] ROACH2 QDR testing address problem

[zhang laiyu]

Hi

  Thanks for your help.

  The firmware and python script had worked (qdr_basic and qdr_counter).I can 
get the plot. But I can not understand the PPC readout script in  
qdr_counter.py thoroughly .The address is really weird. After studying the wiki 
of the CASPER QDR yellow block and datasheet of the QDR chip(CY7C2565KV18), I 
still can not understand the python script very well. Why the 'PPC data' was 
picked like that? Could you give me some suggestion about where can I find more 
clear information about the addree of QDR interface to PowerPC?



 

Cheers!

>
ZHANG Laiyu   
Phone(China)   010-88233078   
Cellphone(China)   13681385567
E-mail:zhan...@ihep.ac.cn
Address:   19B Yuquan Road,Shijingshan District,Beijing，China
Department:Center for Particle Astrophysics 
Office:Astrophysics Building 205
Institute of High Energy Physics, CAS  
web: http://www.ihep.cas.cn
>


-- 
You received this message because you are subscribed to the Google Groups 
"casper@lists.berkeley.edu" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to casper+unsubscr...@lists.berkeley.edu.
To view this discussion on the web visit 
https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/606c0c11.e7c3.17684c842a5.Coremail.zhangly%40ihep.ac.cn.
import calandigital as calan
import numpy as np
import time, struct
import matplotlib.pyplot as plt


IP = '192.168.0.40'
bof = 'qdr_counter.bof.gz'

fpga = calan.initialize_roach(IP, boffile=bof, upload=1)
time.sleep(1)

my_qdr = calan.Qdr(fpga, 'qdr0')
my_qdr.qdr_cal(fail_hard=1, verbosity=1)

time.sleep(1)

fpga.write_int('sel',0) ##just single cycle of data at input qdr
##(diferent from what says the wiki)
fpga.write_int('rst',1)
fpga.write_int('rst',0)
fpga.write_int('wen',1)
time.sleep(4)
##finish writting
fpga.write_int('wen',0)
fpga.write_int('rst',1)
fpga.write_int('rst',0)
fpga.write_int('ren',1)
time.sleep(4)
fpga.write_int('ren',0)
##read the data
qdr_data = struct.unpack('>1024I',fpga.read('dout',1024*4))

plt.plot(qdr_data,label='qdr_data')
plt.plot(np.arange(1024), label='gold data')
plt.grid()

##there is a 2 delay in the cast that i didnt see when compiling
##either way, the data is consistent :P

##read the data with the ppc interface
##the addressing is really weird.. study this!!
fpga.write_int('qdr0_ctrl', 0)
dat = fpga.read('qdr0_memory', 4*1024*4)
vals = np.array(struct.unpack('>4096I', dat))
vals = vals.reshape([-1,4])
vals = vals[::2,:].flatten()
vals = vals[1::2]

plt.plot(vals, label='PPC data')
plt.legend()
plt.show()

Re: [casper] Roach2 QDR

[Jack Hickish]

Hi Matt,

See the commit trail at
https://github.com/jack-h/mlib_devel/commits/ami-devel/xps_base/XPS_ROACH2_base/pcores/qdr_controller_v1_00_a/hdl/verilog/qdrc_infrastructure.v
 and in particular commits 06fa83a and 73cd65b.

In the first case, I basically played with the balancing of registering in
the various pipeline chains and the phase of the IDDR sampling of data
returning to the FPGA from the QDR. Here I believe the net result was to
reduce the total fixed qdr latency by half (of perhaps a whole) clock cycle
(obviously take the commit details over my memory on this).
This got me running at 312.5 MHz, but introduced (predictably) troubles
calibrating at lower clock speeds. I then added output IODELAY blocks on
all the qdr output control signals (there were some on the clock and data,
but I added some to the rd/wr_en signals) so that I could step the latency
of the signals going to the qdr without skewing the relative arrival of the
controls and data/addr. According to the git commit message, the net effect
after all my commits was the ability to insert 4.8 ns delay dynamically,
which seemed adequate for most use cases, and covered the range of clocks I
was interested in.
As a bonus, moving the data IDDR from clk180 to clk0 is a bit more timing
friendly.

I modified the calibration script I stole from SKA-SA a little bit -- the
calibration method I use is (the creatively named) qdr_cal2 at
https://github.com/jack-h/ami_correlator_sw/blob/master/ami/src/qdr.py .
The general mechanism I used is to step the output delays of all qdr
data/addr/control bits until any of the readback data bits are correct (or
the maximum output delay is reached). Then step the input delays for each
bit separately to find the centre of the eye. This certainly works at 312
MHz, and I think also works 200 (if I remember rightly). I assumed that
relative trace lengths of signals going from the FPGA to the QDR did not
need calibrating, which seems sound when I've tested the hardware.
Note that I calibrate from a remote server running a katcp client, so it's
fairly slow (I recall it takes me about ~60s to calibrate 3 qdrs on 10
boards, with rudimentary python multiprocessing over different roaches).

Hope that helps, let me know if I can shed any light on anything,
Cheers,
Jack


On Tue, 17 Mar 2015 at 17:51 Matt Strader mstra...@physics.ucsb.edu wrote:

 Jack,

 When you were having problems calibrating the qdr at high clock rate, how
 did you go about removing half a clock cycle latency from the interface?

 Thanks,
 Matt

 On Wed, Mar 11, 2015 at 12:27 PM, Matt Strader mstra...@physics.ucsb.edu
 wrote:

 I have my own fork of mlib_devel (https://github.com/mstrader/mlib_devel),
 which is nearly identical to the current ska-sa repo.

 Thanks,
 Matt

 On Wed, Mar 11, 2015 at 12:04 AM, Andrew Martens and...@ska.ac.za
 wrote:

 I played around with ROACH2 QDR calibration a few months ago and the
 following must be noted;

 The physical trace length between FPGA and QDR SRAM is a fixed length.
 This fixed length equates to a different number of clock cycles,
 depending on clock speed.
 The interfacing firmware assumes a fixed number of clock cycles latency.
 The calibration routine thus only works for a certain range of clock
 speeds.

 In your case it looks like (at least) one of the interfaces is writing
 (or reading) incorrectly, but is compensating by inserting or removing a
 multiple of half a clock cycle of delay when reading (or writing). When
 reading/writing via the other interface, a different number of compensating
 half clock cycles is inserted.

 Which repo are you using?

 Regards
 Andrew

 On Wed, Mar 11, 2015 at 8:15 AM, David George dgeorges...@gmail.com
 wrote:

 A small chip-in - QDR doesn't work below 120 MHz:
 http://www.cypress.com/?id=4rID=31542

 Regards

 On 10 March 2015 at 22:17, Matt Strader mstra...@physics.ucsb.edu
 wrote:
  Hi Jack,
 
  Thanks for your reply.  I think I understand better what's going on.
 I was
  thinking in terms of 64-bit words that get written or read each
 cycle, but
  looking in qdr_cal(), I should probably think of two 32-bit words
 read or
  written each cycle.  Then, perhaps I shouldn't be too surprised if
 the order
  of the two words read out doesn't match my intuition.
 
  Initially, I was compiling at 100 MHz, but the qdr_cal always
 failed.  Then
  I tried 200 and 250 MHz, and the qdr_cal succeeds.
 
  My tests go like this.  For the 200 MHz design,
  I call qdr_cal, and it returns True.
 
  Then I write this sequence of 64-bit values via the CPU:
  [0, 1, 2, 3, 4, 5, ..., , 1048574, 1048575]
 
  I read back the sequence from the CPU, and it agrees.
  [0, 1, 2, 3, 4, 5, ..., , 1048574, 1048575]
 
  I read it in fabric and get
  [104857532, 032, 132, 232, 332, 432, ..., 104857432, ]
 
 
  Then I try the other direction, by writing a sequence of 64-bit
 values in
  fabric:
  [0,1,2,3,4,5, ... , 1048574, 1048575]
 
  I read back the sequence in fabric and it agrees:
  

Re: [casper] Roach2 QDR

[Matt Strader]

Jack,

When you were having problems calibrating the qdr at high clock rate, how
did you go about removing half a clock cycle latency from the interface?

Thanks,
Matt

On Wed, Mar 11, 2015 at 12:27 PM, Matt Strader mstra...@physics.ucsb.edu
wrote:

 I have my own fork of mlib_devel (https://github.com/mstrader/mlib_devel),
 which is nearly identical to the current ska-sa repo.

 Thanks,
 Matt

 On Wed, Mar 11, 2015 at 12:04 AM, Andrew Martens and...@ska.ac.za wrote:

 I played around with ROACH2 QDR calibration a few months ago and the
 following must be noted;

 The physical trace length between FPGA and QDR SRAM is a fixed length.
 This fixed length equates to a different number of clock cycles,
 depending on clock speed.
 The interfacing firmware assumes a fixed number of clock cycles latency.
 The calibration routine thus only works for a certain range of clock
 speeds.

 In your case it looks like (at least) one of the interfaces is writing
 (or reading) incorrectly, but is compensating by inserting or removing a
 multiple of half a clock cycle of delay when reading (or writing). When
 reading/writing via the other interface, a different number of compensating
 half clock cycles is inserted.

 Which repo are you using?

 Regards
 Andrew

 On Wed, Mar 11, 2015 at 8:15 AM, David George dgeorges...@gmail.com
 wrote:

 A small chip-in - QDR doesn't work below 120 MHz:
 http://www.cypress.com/?id=4rID=31542

 Regards

 On 10 March 2015 at 22:17, Matt Strader mstra...@physics.ucsb.edu
 wrote:
  Hi Jack,
 
  Thanks for your reply.  I think I understand better what's going on.
 I was
  thinking in terms of 64-bit words that get written or read each cycle,
 but
  looking in qdr_cal(), I should probably think of two 32-bit words read
 or
  written each cycle.  Then, perhaps I shouldn't be too surprised if the
 order
  of the two words read out doesn't match my intuition.
 
  Initially, I was compiling at 100 MHz, but the qdr_cal always failed.
 Then
  I tried 200 and 250 MHz, and the qdr_cal succeeds.
 
  My tests go like this.  For the 200 MHz design,
  I call qdr_cal, and it returns True.
 
  Then I write this sequence of 64-bit values via the CPU:
  [0, 1, 2, 3, 4, 5, ..., , 1048574, 1048575]
 
  I read back the sequence from the CPU, and it agrees.
  [0, 1, 2, 3, 4, 5, ..., , 1048574, 1048575]
 
  I read it in fabric and get
  [104857532, 032, 132, 232, 332, 432, ..., 104857432, ]
 
 
  Then I try the other direction, by writing a sequence of 64-bit values
 in
  fabric:
  [0,1,2,3,4,5, ... , 1048574, 1048575]
 
  I read back the sequence in fabric and it agrees:
  [0,1,2,3,4,5, ... , 1048574, 1048575]
 
  I read it via the CPU:
  nBytesPerSample=8
  struct.unpack('{}Q'.format(nSamples),
  roach.read('qdr0_memory',nSamples*nBytesPerSample))
  and get:
  [132, 232, 332, 432, 532, ..., 104857532, 032]
 
  So reading and writing only by the CPU is consistent, and reading and
  writing only in fabric is consistent, but when I do one by the CPU and
 the
  other in fabric, the two 32 bit words get swapped, and the sequence is
  offset by one value.  I actually see the same offset in a Roach1
 version of
  the design as well.  It all seems consistent, not intermittent.
 
  Thanks,
  Matt
 
 
 
  On Mon, Mar 9, 2015 at 10:42 AM, Jack Hickish jackhick...@gmail.com
 wrote:
 
  Hi Matt,
 
  The qdr_cal routine calibrates the relationship of clock and data
 signals
  in the link between the FPGA and QDR signals. Basically it writes test
  patterns and reads them back looking for glitches. It then changes
 the delay
  of IODELAY blocks so that data read from the QDR is captured reliably.
  If this calibration script runs successfully, I'm surprised you still
 see
  glitches with 32 bits of data swapped between read and write (both
 from
  PPC). This would (surely?) count as a glitch and cause the calibration
  routine to fail(?). Do you only see strange behaviour when you read
 from the
  FPGA fabric, or do straight write/reads from the CPU also misbehave?
 Is the
  behaviour the same on every read / write (i.e., not intermittent)?
 
  My experience with the QDR is that at high clock speeds (not sure how
  high, but I see this at 312 MHz) the read latency of the QDR
 interface isn't
  10 clock cycles as it should be even with the minimum IODELAY
 configuration,
  and there are no reliable qdr_cal solutions. I solved this problem
 for my
  design by removing half a clock cycle latency from the interface, at
 which
  point the QDR would calibrate ok. What speed are you running your
 design at?
  To me, misordered 32-bit chunks of data sounds indicative of a latency
  problem in the qdr_cal solution, since the interface is 32 bit DDR,
 so half
  a clock cycle mangles 32-bit chunks of data.
 
  Good luck!
 
  Jack
 
 
 
  On Fri, 6 Mar 2015 at 23:54 Matt Strader mstra...@physics.ucsb.edu
  wrote:
 
  Hi all,
 
  I'm trying to use QDR on Roach2.  I have a test design where I write
 to
  QDR either in firmware or 

Re: [casper] Roach2 QDR

[Matt Strader]

Hi Jack,

Thanks for your reply.  I think I understand better what's going on.  I was
thinking in terms of 64-bit words that get written or read each cycle, but
looking in qdr_cal(), I should probably think of two 32-bit words read or
written each cycle.  Then, perhaps I shouldn't be too surprised if the
order of the two words read out doesn't match my intuition.

Initially, I was compiling at 100 MHz, but the qdr_cal always failed.  Then
I tried 200 and 250 MHz, and the qdr_cal succeeds.

My tests go like this.  For the 200 MHz design,
I call qdr_cal, and it returns True.

Then I write this sequence of 64-bit values via the CPU:
[0, 1, 2, 3, 4, 5, ..., , 1048574, 1048575]

I read back the sequence from the CPU, and it agrees.
[0, 1, 2, 3, 4, 5, ..., , 1048574, 1048575]

I read it in fabric and get
[104857532, 032, 132, 232, 332, 432, ..., 104857432, ]


Then I try the other direction, by writing a sequence of 64-bit values in
fabric:
[0,1,2,3,4,5, ... , 1048574, 1048575]

I read back the sequence in fabric and it agrees:
[0,1,2,3,4,5, ... , 1048574, 1048575]

I read it via the CPU:
nBytesPerSample=8
struct.unpack('{}Q'.format(nSamples),
roach.read('qdr0_memory',nSamples*nBytesPerSample))
and get:
[132, 232, 332, 432, 532, ..., 104857532, 032]

So reading and writing only by the CPU is consistent, and reading and
writing only in fabric is consistent, but when I do one by the CPU and the
other in fabric, the two 32 bit words get swapped, and the sequence is
offset by one value.  I actually see the same offset in a Roach1 version of
the design as well.  It all seems consistent, not intermittent.

Thanks,
Matt



On Mon, Mar 9, 2015 at 10:42 AM, Jack Hickish jackhick...@gmail.com wrote:

 Hi Matt,

 The qdr_cal routine calibrates the relationship of clock and data signals
 in the link between the FPGA and QDR signals. Basically it writes test
 patterns and reads them back looking for glitches. It then changes the
 delay of IODELAY blocks so that data read from the QDR is captured reliably.
 If this calibration script runs successfully, I'm surprised you still see
 glitches with 32 bits of data swapped between read and write (both from
 PPC). This would (surely?) count as a glitch and cause the calibration
 routine to fail(?). Do you only see strange behaviour when you read from
 the FPGA fabric, or do straight write/reads from the CPU also misbehave? Is
 the behaviour the same on every read / write (i.e., not intermittent)?

 My experience with the QDR is that at high clock speeds (not sure how
 high, but I see this at 312 MHz) the read latency of the QDR interface
 isn't 10 clock cycles as it should be even with the minimum IODELAY
 configuration, and there are no reliable qdr_cal solutions. I solved this
 problem for my design by removing half a clock cycle latency from the
 interface, at which point the QDR would calibrate ok. What speed are you
 running your design at? To me, misordered 32-bit chunks of data sounds
 indicative of a latency problem in the qdr_cal solution, since the
 interface is 32 bit DDR, so half a clock cycle mangles 32-bit chunks of
 data.

 Good luck!

 Jack



 On Fri, 6 Mar 2015 at 23:54 Matt Strader mstra...@physics.ucsb.edu
 wrote:

 Hi all,

 I'm trying to use QDR on Roach2.  I have a test design where I write to
 QDR either in firmware or with katcp, then read it out in the firmware and
 check the results with snapshot blocks.  It mostly works, but with some
 interesting quirks.

 I know that katcp can only write 64 bits of the 72 bit QDR word, so I
 changed the bitbasher blocks in qdr_mask.m to move the inaccessible bits
 out of the way to the most significant bits.  Now everything seems to work
 well, except that in some builds (seems to depend on clock rate?), the
 first 32 bits (of the 64 bits I'm writing in a cycle) get swapped with the
 last 32 bits sometime between writing the word and reading it back.  And
 there is a sometimes a difference in whether I am writing in firmware or by
 katcp to whether this swap happens.

 Also, when I write with katcp starting at address zero, I find that when
 reading out, the write started at what seems to be address 1.  Is the
 latency of QDR set to 10 cycles?

 I can live with all this, as long as it is repeatable for a given design,
 but I thought I'd ask if anyone knows what is causing these quirks.

 Also, I understand that we should run a software calibration for qdr,
 which seems to be qdr_cal() in qdr.py in the ska-sa:casperfpga repo.  So, I
 run this first, which seems to work for my latest compiles, but I haven't
 noticed a difference in the results.  What exactly does this calibration
 do?

 Thanks,
 Matt Strader
 UC Santa Barbara

Re: [casper] Roach2 QDR

[Matt Strader]

I have my own fork of mlib_devel (https://github.com/mstrader/mlib_devel),
which is nearly identical to the current ska-sa repo.

Thanks,
Matt

On Wed, Mar 11, 2015 at 12:04 AM, Andrew Martens and...@ska.ac.za wrote:

 I played around with ROACH2 QDR calibration a few months ago and the
 following must be noted;

 The physical trace length between FPGA and QDR SRAM is a fixed length.
 This fixed length equates to a different number of clock cycles, depending
 on clock speed.
 The interfacing firmware assumes a fixed number of clock cycles latency.
 The calibration routine thus only works for a certain range of clock
 speeds.

 In your case it looks like (at least) one of the interfaces is writing (or
 reading) incorrectly, but is compensating by inserting or removing a
 multiple of half a clock cycle of delay when reading (or writing). When
 reading/writing via the other interface, a different number of compensating
 half clock cycles is inserted.

 Which repo are you using?

 Regards
 Andrew

 On Wed, Mar 11, 2015 at 8:15 AM, David George dgeorges...@gmail.com
 wrote:

 A small chip-in - QDR doesn't work below 120 MHz:
 http://www.cypress.com/?id=4rID=31542

 Regards

 On 10 March 2015 at 22:17, Matt Strader mstra...@physics.ucsb.edu
 wrote:
  Hi Jack,
 
  Thanks for your reply.  I think I understand better what's going on.  I
 was
  thinking in terms of 64-bit words that get written or read each cycle,
 but
  looking in qdr_cal(), I should probably think of two 32-bit words read
 or
  written each cycle.  Then, perhaps I shouldn't be too surprised if the
 order
  of the two words read out doesn't match my intuition.
 
  Initially, I was compiling at 100 MHz, but the qdr_cal always failed.
 Then
  I tried 200 and 250 MHz, and the qdr_cal succeeds.
 
  My tests go like this.  For the 200 MHz design,
  I call qdr_cal, and it returns True.
 
  Then I write this sequence of 64-bit values via the CPU:
  [0, 1, 2, 3, 4, 5, ..., , 1048574, 1048575]
 
  I read back the sequence from the CPU, and it agrees.
  [0, 1, 2, 3, 4, 5, ..., , 1048574, 1048575]
 
  I read it in fabric and get
  [104857532, 032, 132, 232, 332, 432, ..., 104857432, ]
 
 
  Then I try the other direction, by writing a sequence of 64-bit values
 in
  fabric:
  [0,1,2,3,4,5, ... , 1048574, 1048575]
 
  I read back the sequence in fabric and it agrees:
  [0,1,2,3,4,5, ... , 1048574, 1048575]
 
  I read it via the CPU:
  nBytesPerSample=8
  struct.unpack('{}Q'.format(nSamples),
  roach.read('qdr0_memory',nSamples*nBytesPerSample))
  and get:
  [132, 232, 332, 432, 532, ..., 104857532, 032]
 
  So reading and writing only by the CPU is consistent, and reading and
  writing only in fabric is consistent, but when I do one by the CPU and
 the
  other in fabric, the two 32 bit words get swapped, and the sequence is
  offset by one value.  I actually see the same offset in a Roach1
 version of
  the design as well.  It all seems consistent, not intermittent.
 
  Thanks,
  Matt
 
 
 
  On Mon, Mar 9, 2015 at 10:42 AM, Jack Hickish jackhick...@gmail.com
 wrote:
 
  Hi Matt,
 
  The qdr_cal routine calibrates the relationship of clock and data
 signals
  in the link between the FPGA and QDR signals. Basically it writes test
  patterns and reads them back looking for glitches. It then changes the
 delay
  of IODELAY blocks so that data read from the QDR is captured reliably.
  If this calibration script runs successfully, I'm surprised you still
 see
  glitches with 32 bits of data swapped between read and write (both from
  PPC). This would (surely?) count as a glitch and cause the calibration
  routine to fail(?). Do you only see strange behaviour when you read
 from the
  FPGA fabric, or do straight write/reads from the CPU also misbehave?
 Is the
  behaviour the same on every read / write (i.e., not intermittent)?
 
  My experience with the QDR is that at high clock speeds (not sure how
  high, but I see this at 312 MHz) the read latency of the QDR interface
 isn't
  10 clock cycles as it should be even with the minimum IODELAY
 configuration,
  and there are no reliable qdr_cal solutions. I solved this problem for
 my
  design by removing half a clock cycle latency from the interface, at
 which
  point the QDR would calibrate ok. What speed are you running your
 design at?
  To me, misordered 32-bit chunks of data sounds indicative of a latency
  problem in the qdr_cal solution, since the interface is 32 bit DDR, so
 half
  a clock cycle mangles 32-bit chunks of data.
 
  Good luck!
 
  Jack
 
 
 
  On Fri, 6 Mar 2015 at 23:54 Matt Strader mstra...@physics.ucsb.edu
  wrote:
 
  Hi all,
 
  I'm trying to use QDR on Roach2.  I have a test design where I write
 to
  QDR either in firmware or with katcp, then read it out in the
 firmware and
  check the results with snapshot blocks.  It mostly works, but with
 some
  interesting quirks.
 
  I know that katcp can only write 64 bits of the 72 bit QDR word, so I
  changed the bitbasher blocks in 

Re: [casper] Roach2 QDR

[David George]

A small chip-in - QDR doesn't work below 120 MHz:
http://www.cypress.com/?id=4rID=31542

Regards

On 10 March 2015 at 22:17, Matt Strader mstra...@physics.ucsb.edu wrote:
 Hi Jack,

 Thanks for your reply.  I think I understand better what's going on.  I was
 thinking in terms of 64-bit words that get written or read each cycle, but
 looking in qdr_cal(), I should probably think of two 32-bit words read or
 written each cycle.  Then, perhaps I shouldn't be too surprised if the order
 of the two words read out doesn't match my intuition.

 Initially, I was compiling at 100 MHz, but the qdr_cal always failed.  Then
 I tried 200 and 250 MHz, and the qdr_cal succeeds.

 My tests go like this.  For the 200 MHz design,
 I call qdr_cal, and it returns True.

 Then I write this sequence of 64-bit values via the CPU:
 [0, 1, 2, 3, 4, 5, ..., , 1048574, 1048575]

 I read back the sequence from the CPU, and it agrees.
 [0, 1, 2, 3, 4, 5, ..., , 1048574, 1048575]

 I read it in fabric and get
 [104857532, 032, 132, 232, 332, 432, ..., 104857432, ]


 Then I try the other direction, by writing a sequence of 64-bit values in
 fabric:
 [0,1,2,3,4,5, ... , 1048574, 1048575]

 I read back the sequence in fabric and it agrees:
 [0,1,2,3,4,5, ... , 1048574, 1048575]

 I read it via the CPU:
 nBytesPerSample=8
 struct.unpack('{}Q'.format(nSamples),
 roach.read('qdr0_memory',nSamples*nBytesPerSample))
 and get:
 [132, 232, 332, 432, 532, ..., 104857532, 032]

 So reading and writing only by the CPU is consistent, and reading and
 writing only in fabric is consistent, but when I do one by the CPU and the
 other in fabric, the two 32 bit words get swapped, and the sequence is
 offset by one value.  I actually see the same offset in a Roach1 version of
 the design as well.  It all seems consistent, not intermittent.

 Thanks,
 Matt



 On Mon, Mar 9, 2015 at 10:42 AM, Jack Hickish jackhick...@gmail.com wrote:

 Hi Matt,

 The qdr_cal routine calibrates the relationship of clock and data signals
 in the link between the FPGA and QDR signals. Basically it writes test
 patterns and reads them back looking for glitches. It then changes the delay
 of IODELAY blocks so that data read from the QDR is captured reliably.
 If this calibration script runs successfully, I'm surprised you still see
 glitches with 32 bits of data swapped between read and write (both from
 PPC). This would (surely?) count as a glitch and cause the calibration
 routine to fail(?). Do you only see strange behaviour when you read from the
 FPGA fabric, or do straight write/reads from the CPU also misbehave? Is the
 behaviour the same on every read / write (i.e., not intermittent)?

 My experience with the QDR is that at high clock speeds (not sure how
 high, but I see this at 312 MHz) the read latency of the QDR interface isn't
 10 clock cycles as it should be even with the minimum IODELAY configuration,
 and there are no reliable qdr_cal solutions. I solved this problem for my
 design by removing half a clock cycle latency from the interface, at which
 point the QDR would calibrate ok. What speed are you running your design at?
 To me, misordered 32-bit chunks of data sounds indicative of a latency
 problem in the qdr_cal solution, since the interface is 32 bit DDR, so half
 a clock cycle mangles 32-bit chunks of data.

 Good luck!

 Jack



 On Fri, 6 Mar 2015 at 23:54 Matt Strader mstra...@physics.ucsb.edu
 wrote:

 Hi all,

 I'm trying to use QDR on Roach2.  I have a test design where I write to
 QDR either in firmware or with katcp, then read it out in the firmware and
 check the results with snapshot blocks.  It mostly works, but with some
 interesting quirks.

 I know that katcp can only write 64 bits of the 72 bit QDR word, so I
 changed the bitbasher blocks in qdr_mask.m to move the inaccessible bits out
 of the way to the most significant bits.  Now everything seems to work well,
 except that in some builds (seems to depend on clock rate?), the first 32
 bits (of the 64 bits I'm writing in a cycle) get swapped with the last 32
 bits sometime between writing the word and reading it back.  And there is a
 sometimes a difference in whether I am writing in firmware or by katcp to
 whether this swap happens.

 Also, when I write with katcp starting at address zero, I find that when
 reading out, the write started at what seems to be address 1.  Is the
 latency of QDR set to 10 cycles?

 I can live with all this, as long as it is repeatable for a given design,
 but I thought I'd ask if anyone knows what is causing these quirks.

 Also, I understand that we should run a software calibration for qdr,
 which seems to be qdr_cal() in qdr.py in the ska-sa:casperfpga repo.  So, I
 run this first, which seems to work for my latest compiles, but I haven't
 noticed a difference in the results.  What exactly does this calibration do?

 Thanks,
 Matt Strader
 UC Santa Barbara

Re: [casper] Roach2 QDR

[Andrew Martens]

I played around with ROACH2 QDR calibration a few months ago and the
following must be noted;

The physical trace length between FPGA and QDR SRAM is a fixed length.
This fixed length equates to a different number of clock cycles, depending
on clock speed.
The interfacing firmware assumes a fixed number of clock cycles latency.
The calibration routine thus only works for a certain range of clock speeds.

In your case it looks like (at least) one of the interfaces is writing (or
reading) incorrectly, but is compensating by inserting or removing a
multiple of half a clock cycle of delay when reading (or writing). When
reading/writing via the other interface, a different number of compensating
half clock cycles is inserted.

Which repo are you using?

Regards
Andrew

On Wed, Mar 11, 2015 at 8:15 AM, David George dgeorges...@gmail.com wrote:

 A small chip-in - QDR doesn't work below 120 MHz:
 http://www.cypress.com/?id=4rID=31542

 Regards

 On 10 March 2015 at 22:17, Matt Strader mstra...@physics.ucsb.edu wrote:
  Hi Jack,
 
  Thanks for your reply.  I think I understand better what's going on.  I
 was
  thinking in terms of 64-bit words that get written or read each cycle,
 but
  looking in qdr_cal(), I should probably think of two 32-bit words read or
  written each cycle.  Then, perhaps I shouldn't be too surprised if the
 order
  of the two words read out doesn't match my intuition.
 
  Initially, I was compiling at 100 MHz, but the qdr_cal always failed.
 Then
  I tried 200 and 250 MHz, and the qdr_cal succeeds.
 
  My tests go like this.  For the 200 MHz design,
  I call qdr_cal, and it returns True.
 
  Then I write this sequence of 64-bit values via the CPU:
  [0, 1, 2, 3, 4, 5, ..., , 1048574, 1048575]
 
  I read back the sequence from the CPU, and it agrees.
  [0, 1, 2, 3, 4, 5, ..., , 1048574, 1048575]
 
  I read it in fabric and get
  [104857532, 032, 132, 232, 332, 432, ..., 104857432, ]
 
 
  Then I try the other direction, by writing a sequence of 64-bit values in
  fabric:
  [0,1,2,3,4,5, ... , 1048574, 1048575]
 
  I read back the sequence in fabric and it agrees:
  [0,1,2,3,4,5, ... , 1048574, 1048575]
 
  I read it via the CPU:
  nBytesPerSample=8
  struct.unpack('{}Q'.format(nSamples),
  roach.read('qdr0_memory',nSamples*nBytesPerSample))
  and get:
  [132, 232, 332, 432, 532, ..., 104857532, 032]
 
  So reading and writing only by the CPU is consistent, and reading and
  writing only in fabric is consistent, but when I do one by the CPU and
 the
  other in fabric, the two 32 bit words get swapped, and the sequence is
  offset by one value.  I actually see the same offset in a Roach1 version
 of
  the design as well.  It all seems consistent, not intermittent.
 
  Thanks,
  Matt
 
 
 
  On Mon, Mar 9, 2015 at 10:42 AM, Jack Hickish jackhick...@gmail.com
 wrote:
 
  Hi Matt,
 
  The qdr_cal routine calibrates the relationship of clock and data
 signals
  in the link between the FPGA and QDR signals. Basically it writes test
  patterns and reads them back looking for glitches. It then changes the
 delay
  of IODELAY blocks so that data read from the QDR is captured reliably.
  If this calibration script runs successfully, I'm surprised you still
 see
  glitches with 32 bits of data swapped between read and write (both from
  PPC). This would (surely?) count as a glitch and cause the calibration
  routine to fail(?). Do you only see strange behaviour when you read
 from the
  FPGA fabric, or do straight write/reads from the CPU also misbehave? Is
 the
  behaviour the same on every read / write (i.e., not intermittent)?
 
  My experience with the QDR is that at high clock speeds (not sure how
  high, but I see this at 312 MHz) the read latency of the QDR interface
 isn't
  10 clock cycles as it should be even with the minimum IODELAY
 configuration,
  and there are no reliable qdr_cal solutions. I solved this problem for
 my
  design by removing half a clock cycle latency from the interface, at
 which
  point the QDR would calibrate ok. What speed are you running your
 design at?
  To me, misordered 32-bit chunks of data sounds indicative of a latency
  problem in the qdr_cal solution, since the interface is 32 bit DDR, so
 half
  a clock cycle mangles 32-bit chunks of data.
 
  Good luck!
 
  Jack
 
 
 
  On Fri, 6 Mar 2015 at 23:54 Matt Strader mstra...@physics.ucsb.edu
  wrote:
 
  Hi all,
 
  I'm trying to use QDR on Roach2.  I have a test design where I write to
  QDR either in firmware or with katcp, then read it out in the firmware
 and
  check the results with snapshot blocks.  It mostly works, but with some
  interesting quirks.
 
  I know that katcp can only write 64 bits of the 72 bit QDR word, so I
  changed the bitbasher blocks in qdr_mask.m to move the inaccessible
 bits out
  of the way to the most significant bits.  Now everything seems to work
 well,
  except that in some builds (seems to depend on clock rate?), the first
 32
  bits (of the 64 bits I'm writing in a 

Re: [casper] Roach2 QDR

[Jack Hickish]

Hi Matt,

The qdr_cal routine calibrates the relationship of clock and data signals
in the link between the FPGA and QDR signals. Basically it writes test
patterns and reads them back looking for glitches. It then changes the
delay of IODELAY blocks so that data read from the QDR is captured reliably.
If this calibration script runs successfully, I'm surprised you still see
glitches with 32 bits of data swapped between read and write (both from
PPC). This would (surely?) count as a glitch and cause the calibration
routine to fail(?). Do you only see strange behaviour when you read from
the FPGA fabric, or do straight write/reads from the CPU also misbehave? Is
the behaviour the same on every read / write (i.e., not intermittent)?

My experience with the QDR is that at high clock speeds (not sure how high,
but I see this at 312 MHz) the read latency of the QDR interface isn't 10
clock cycles as it should be even with the minimum IODELAY configuration,
and there are no reliable qdr_cal solutions. I solved this problem for my
design by removing half a clock cycle latency from the interface, at which
point the QDR would calibrate ok. What speed are you running your design
at? To me, misordered 32-bit chunks of data sounds indicative of a latency
problem in the qdr_cal solution, since the interface is 32 bit DDR, so half
a clock cycle mangles 32-bit chunks of data.

Good luck!

Jack



On Fri, 6 Mar 2015 at 23:54 Matt Strader mstra...@physics.ucsb.edu wrote:

 Hi all,

 I'm trying to use QDR on Roach2.  I have a test design where I write to
 QDR either in firmware or with katcp, then read it out in the firmware and
 check the results with snapshot blocks.  It mostly works, but with some
 interesting quirks.

 I know that katcp can only write 64 bits of the 72 bit QDR word, so I
 changed the bitbasher blocks in qdr_mask.m to move the inaccessible bits
 out of the way to the most significant bits.  Now everything seems to work
 well, except that in some builds (seems to depend on clock rate?), the
 first 32 bits (of the 64 bits I'm writing in a cycle) get swapped with the
 last 32 bits sometime between writing the word and reading it back.  And
 there is a sometimes a difference in whether I am writing in firmware or by
 katcp to whether this swap happens.

 Also, when I write with katcp starting at address zero, I find that when
 reading out, the write started at what seems to be address 1.  Is the
 latency of QDR set to 10 cycles?

 I can live with all this, as long as it is repeatable for a given design,
 but I thought I'd ask if anyone knows what is causing these quirks.

 Also, I understand that we should run a software calibration for qdr,
 which seems to be qdr_cal() in qdr.py in the ska-sa:casperfpga repo.  So, I
 run this first, which seems to work for my latest compiles, but I haven't
 noticed a difference in the results.  What exactly does this calibration
 do?

 Thanks,
 Matt Strader
 UC Santa Barbara

[casper] Roach2 QDR

[Matt Strader]

Hi all,

I'm trying to use QDR on Roach2.  I have a test design where I write to QDR
either in firmware or with katcp, then read it out in the firmware and
check the results with snapshot blocks.  It mostly works, but with some
interesting quirks.

I know that katcp can only write 64 bits of the 72 bit QDR word, so I
changed the bitbasher blocks in qdr_mask.m to move the inaccessible bits
out of the way to the most significant bits.  Now everything seems to work
well, except that in some builds (seems to depend on clock rate?), the
first 32 bits (of the 64 bits I'm writing in a cycle) get swapped with the
last 32 bits sometime between writing the word and reading it back.  And
there is a sometimes a difference in whether I am writing in firmware or by
katcp to whether this swap happens.

Also, when I write with katcp starting at address zero, I find that when
reading out, the write started at what seems to be address 1.  Is the
latency of QDR set to 10 cycles?

I can live with all this, as long as it is repeatable for a given design,
but I thought I'd ask if anyone knows what is causing these quirks.

Also, I understand that we should run a software calibration for qdr, which
seems to be qdr_cal() in qdr.py in the ska-sa:casperfpga repo.  So, I run
this first, which seems to work for my latest compiles, but I haven't
noticed a difference in the results.  What exactly does this calibration
do?

Thanks,
Matt Strader
UC Santa Barbara

[casper] ROACH2 QDR test/demo model

[David MacMahon]

Does anyone have a test and/or demo model for ROACH2 that exercises the QDR 
chips?  (Preferably one saved in Simulink version 7 format.)

Thanks,
Dave

Re: [casper] ROACH2 QDR test/demo model

[David MacMahon]

Thanks, Henno!

And thanks, Jason, for confirming the QDR dimensions!

Dave

On Feb 5, 2013, at 9:07 PM, Henno Kriel wrote:

 Hi Dave
 
 Attached is the model file + python script I used to test the QDR on Roach2 
 (Matlab 2008B  ISE 11.5)
 
 Regards
 Henno
 
 On Wed, Feb 6, 2013 at 12:48 AM, David MacMahon dav...@astro.berkeley.edu 
 wrote:
 Does anyone have a test and/or demo model for ROACH2 that exercises the QDR 
 chips?  (Preferably one saved in Simulink version 7 format.)
 
 Thanks,
 Dave
 
 
 
 
 
 -- 
 Henno Kriel
 
 DSP Engineer
 Digital Back End
 meerKAT
 
 SKA South Africa
 Third Floor
 The Park
 Park Road (off Alexandra Road)
 Pinelands
 7405
 Western Cape
 South Africa
 
 Latitude: -33.94329 (South); Longitude: 18.48945 (East).
 
 (p) +27 (0)21 506 7300
 (p) +27 (0)21 506 7365 (direct)
 (f) +27 (0)21 506 7375
 (m) +27 (0)84 504 5050
 qdr.tarroach2_qdr_tst.py