My thanks to SA SKA, Peralex, and CASPER community for the support. Combined with Mark’s diligent efforts, we are now in a position to build, load, and run bitcodes, and to move data on and off the SKARAB with 40GigE. This means we have all the key infrastructure in place to start to develop custom logic for our specialized real time interpolation, Fourier inversion and VDIF formatting application.
There has been a great deal detailed correspondence, which a lot to wade through in the CASPER email archive. I believe Adam, Clifford and others are going to capture this in SKARAB manuals. In the mean time, we thought it useful to at least list, in bullet form, the various issues we hit along, and a real quick summary of the solution. This may prove useful to someone bringing up a SKARAB in the wild in the future; of course some issues have been fixed in the various tools and won’t come up again. There is much detail in various email threads, abstracted to hell here. Summary SKARAB issues list follows, again, again the details are in the email archive. Further corrections or amendments are welcome. Thanks again for the support, and have a great weekend. Jonathan Building JASPER bitcodes ==================== We didn’t get anywhere without the right tools software rev levels: Matlab 2016b and Vivado 2016.2 Our Mathworks and Xilinx support had expired due to my own lazyness. We were informed that older versions should work, as there are no dependencies critical to these versions, but there are rev level cross checks in various config files, and we could not in practice work around these. In the end we asked for 30 day trials to get by while processing paid renewals. A documentation error, now corrected, directed us to the wrong branch of the Git Repo which obstructed builds. Use the master branch *not* jasper_vivado_2016_2 branch. After changing branches it was also necessary to update the vivado_config.sh script. Loading and running bitcodes on SKARAB ================================= One needs proper rev levels of the SKARAB Spartan (1.5) and Virtex 7 (2.2) FPGA firmware (The latter was known as the “SOC” firmware, now deprecated, and includes the MicroBlaze processor IP.) One also needs the correct version of casperfpga utility compatible with firmware versions Our SKARABs shipped with older firmware versions. In attempting to update them as a first step one of our SKARABs went into a non-responsive state, which was referred to as “bricked” on the CASPER list. The SKARAB was not in fact strictly bricked, rather it was found that pacing hostname with the DHCP response (via /etc/hosts and dnsmasq) then it would continuously keep sending DHCP requests and never actually boot up with an IP. We have only best guesses for how we got into this state: namely used a very old and apparently incompatible version of the casperfpga utility to first attempt the firmware upgrade via Ethernet. Recovery required updating the Virtex 7 firmware via JTAG following unpublished procedures (it is planned to have to take the lids off SKARABs in the field). For reasons we don’t fully understand the non-responsive unit only recovered on the second JTAG firmware upgrade attempt—a pleasant surprise at that point, we didn’t look that gift horse in the mouth. Other smaller issues: There is a need to reprogram the bitcode onto the SKARAB after exiting and restarting ipython in order to access software registers and snapshot blocks through casperfpga. One needs to disconnect the 40 GbE cables from the SKARABs when uploading bitcodes over 1 GbE, otherwise communication will be lost and the SKARAB needs to be reset. One needs to enable jumbo packets on 1 GigE networks (Server NIC MTU size 1500 to 9000); when not enabled caused checksum errors on loading. In our case the server NIC was not properly set up. 40GigE communications =================== To send data from one SKARAB to another, the 40 GbE ports must be routed through a switch in order for the SKARABs to be assigned an IP address via DHCP. A possible alternative is to use a static IP assignment, but this is not transparently supported in current JASPER tools and libraries. Covered in detail by Wes and Clifford in this very thread. The Jumbo packets issue also arose on the 40 GIgE network. When trying to send data over 40 GbE from a SKARAB, through the switch, and into a server, the MTU size had to be set to 9000. > On Jul 6, 2017, at 9:20 AM, Wesley New <wes...@ska.ac.za> wrote: > > Thanks Clifford for clearing that up. Yes, the hardware does support a static > IP but this is not currently supported in the CASPER tools. We only require > DHCP support and hence have not taken the time to implement support for > static IPs. But if that is something that you require, you are welcome to add > the functionality and we will support you as best we can. > > Regards > > Wesley > > Wesley New > South African SKA Project > +2721 506 7300 > www.ska.ac.za <http://www.ska.ac.za/> > > > > On Thu, Jul 6, 2017 at 2:27 PM, Clifford van Dyk <cliffordvan...@gmail.com > <mailto:cliffordvan...@gmail.com>> wrote: > Hi Mark (and Wes) > > For clarity (and at risk of poking my nose in where I shouldn't), I just > wanted to point out that the Skarab HW platform itself is quite flexible: > > It is possible to set up Skarabs to communicate directly with each other over > the 40 GbE links (we do this all the time in production testing of the > platforms). The typical setup involves using the 1 GbE port as the primary > platform management port (hooking this up to a network, typically with DHCP > support), and assigning IP addresses, ARP entries etc to the 40 GbE ports > through commands sent to the platform over this management interface. > Alternatively, it is also possible to drop the 40 GbE firmware stack entirely > and implement any serial standard over the 8 (4TX+4RX) SERDES lines, if > point-to-point links were all that were required. The advantage of this is > that a significant amount of FPGA resource could be saved by moving to a > lightweight interface (but routing would be lost). > > However, since this is not the way SKA-SA is currently using the platform, > your support/compatibility with SKA-SA toolflow probably won't be as > "out-the-box" as it is when sticking with the mainstream SKA-SA > toolflow/firmware implementations. > > Kind regards, > Clifford > > > On 7/6/2017 11:08 AM, Wesley New wrote: >> Hi Mark. >> >> The SKARABs currently only work with a DHCP server. So all bets are off >> while attempting to send/receive data without the interface receiving an IP, >> Netmask and Gateway. My guess would be that you are sending data, but I am >> not sure what IP/MAC it would be sent to or from. Because the ARP table >> wouldn't be set up correctly prior to a DHCP response being received and >> processed. I would recommend using the SKARABs on a network or connected >> directly to a server. I don't see much of a use-case for point-to-point >> connections either (How would you get the data off the SKARABs). >> >> Also be careful of the promiscuous mode. It allows all data received on the >> interface through to the fabric. It is only meant for debugging network >> problems, kind of a tcpdump for the SKARAB. You should be able to send and >> receive data between 2 SKARABs without setting promiscuous mode. >> >> If you would like I can pull up some test model files where we send and >> receive data between 2 networked SKARABs. With functionality to configure >> data rates and packet sizes etc. >> >> Regards >> >> Wes >> >> >> Wesley New >> South African SKA Project >> +2721 506 7300 <tel:+27%2021%20506%207300> >> www.ska.ac.za <http://www.ska.ac.za/> >> >> >> >> On Wed, Jul 5, 2017 at 10:46 PM, Peryer, Mark A. >> <mark.per...@cfa.harvard.edu <mailto:mark.per...@cfa.harvard.edu>> wrote: >> Hello, >> >> I have solved my issue of not being able to receive data. Instead of >> directly connecting the SKARABs, I routed them through a 40 GbE switch where >> the DHCP server can assign both of the SKARABs an IP address. I am still not >> sure why directly connecting them does not work, however that appears to be >> what was causing the issue all along. >> >> Thanks for your help, >> >> Mark >> >> On Wed, Jul 5, 2017 at 4:20 PM, Peryer, Mark A. <mark.per...@cfa.harvard.edu >> <mailto:mark.per...@cfa.harvard.edu>> wrote: >> Hello, >> >> I am still having issues receiving data over 40 GbE. I currently have a >> direct connection between the transmitting and receiving SKARABs, both on >> the leftmost 40 GbE port of Mezzanine slot 3 . Using tcpdump on a server >> with a 40GbE interface, I was able to confirm that the SKARAB which >> transmits data is successfully doing so. Also, both SKARABs can successfully >> be pinged over 40GbE. Despite this, I am still unable to receive data on the >> other SKARAB, even when Promiscuous Mode is enabled. I did observe that the >> orange and green LEDs are turned on for both of the SKARAB's 40GbE ports. >> The green light blinks on the SKARAB that is transmitting data and the >> orange light blinks on the SKARAB that is receiving data, therefore it >> appears they are communicating with each other. >> >> Any other suggestions on how to debug the reception data over 40 GbE would >> be much appreciated, as I am still unsure what could be causing the issue. >> >> Thanks, >> >> Mark >> >> On Fri, Jun 30, 2017 at 3:57 AM, Paul Prozesky <paul.proze...@gmail.com >> <mailto:paul.proze...@gmail.com>> wrote: >> Morning Mark >> >> Please have a look here for SKARAB 40gbe TX and RX models with demo software. >> >> https://github.com/ska-sa/mkat_fpga/tree/devel/source/skarab_dev/tut2 >> <https://github.com/ska-sa/mkat_fpga/tree/devel/source/skarab_dev/tut2> >> >> Cheers >> Paul >> >> >> On 30 June 2017 at 07:46, Jason Manley <jman...@ska.ac.za >> <mailto:jman...@ska.ac.za>> wrote: >> Hi Mark >> >> The current SKARAB 40G yellowblock is a bit of a hack. >> >> 1) It is currently not parameterised, and is hard-coded for the first port >> on Mezzanine slot 3. >> >> 2) The tx_valid and rx_valid lines are 4-bits wide, not 1 bit. This will >> be rectified soon, but in the meanwhile, just feed the value "3" to tx_valid >> to send packets. >> >> 3) Note that there was a recent change to the TX and RX 64-bit word >> ordering (was previously incorrectly swapped). Make you built your >> bitstreams with a recent git checkout. >> >> It's on our todo list to get it properly parameterised and to fix the 4-bit >> weirdness, but it's not clear when we'll have that done. In the meanwhile, >> the 40G does work, and we are successfully using this first port actively at >> SKA-SA. >> >> Some things you should try to help debug: >> >> 1) The microblaze will attempt to DHCP on the 40G interface. Did it obtain >> a lease? Check your DHCP server logs. >> >> 2) Check (using casperfpga software) that the cores are actually >> configured like you think. There was a version of microblaze that overwrote >> things. And if it's DHCP'ing (this is how we use the ports), then the IP >> will have changed from what you expect. >> myfpgaobject.gbes['my_forty_gbe_core'].print_core_details() (or >> g.get_core_details()). >> >> 3) What does tcpdump/wireshark show is coming out of the TX board? >> >> 4) Did you select different MAC addresses for the "hardcoded" yellowblocks >> on the TX and RX side? Else a switch might not forward the packets. >> >> 5) Can you ping both 40G interfaces from a computer? >> >> Jason Manley >> CBF Manager >> SKA-SA >> >> Cell: +27 82 662 7726 <tel:+27%2082%20662%207726> >> Work: +27 21 506 7300 <tel:+27%2021%20506%207300> >> >> On 29 Jun 2017, at 20:37, Peryer, Mark A. <mark.per...@cfa.harvard.edu >> <mailto:mark.per...@cfa.harvard.edu>> wrote: >> >> > Hello, >> > >> > I am trying to send data from one SKARAB to another SKARAB over 40GbE. I >> > have created two separate JASPER files, one for receiving and one for >> > transmitting. Each design has one forty_gbe yellow block and are >> > configured similar to Tutorial 2 on the casper website. In the JASPER file >> > used for transmitting, I have placed a snapshot block on the tx_data >> > signal for the forty_gbe block and am able to read the correct data from >> > the snapshot block using casperfpga. While it appears that the correct >> > data is being transmitted, nothing is received by the second SKARAB. In >> > the JASPER file used for receiving data, I have a snapshot block on the >> > rx_data signal, which should trigger once a valid frame is received. >> > However, when I run fpga01.snapshots.snapshot2.print_snap(50) it just >> > hangs, indicating nothing has been received. >> > >> > I did notice that when I right click on the forty_gbe yellow block and go >> > to Mask >> Edit Mask >> Parameters&Dialog, there is a menu that allows for >> > the card slot location of the 40GbE card to be selected. Our SKARAB units >> > have the 40GbE card populated on Mezzanine Slot 3, however only Card Slot >> > 0 and 1 are available to be selected. Should this option be set to slot 3 >> > in order to properly use the 40GbE ports? >> > >> > I should also mention that I have configured the tx_dest_ip block input to >> > be 192.168.5.20 and the tx_dest_port to be 10000, which are the default >> > values for the forty_gbe block. >> > >> > If there is anything else I should look at in order to properly configure >> > the forty_gbe yellow blocks please advise. >> > >> > Thanks, >> > >> > Mark >> > -- You received this message because you are subscribed to the Google Groups "casper@lists.berkeley.edu" group. To unsubscribe from this group and stop receiving emails from it, send an email to casper+unsubscr...@lists.berkeley.edu. To post to this group, send email to casper@lists.berkeley.edu.
