Hi Mark (and Wes) For clarity (and at risk of poking my nose in where I shouldn't), I just wanted to point out that the Skarab HW platform itself is quite flexible:
It is possible to set up Skarabs to communicate directly with each other over the 40 GbE links (we do this all the time in production testing of the platforms). The typical setup involves using the 1 GbE port as the primary platform management port (hooking this up to a network, typically with DHCP support), and assigning IP addresses, ARP entries etc to the 40 GbE ports through commands sent to the platform over this management interface. Alternatively, it is also possible to drop the 40 GbE firmware stack entirely and implement any serial standard over the 8 (4TX+4RX) SERDES lines, if point-to-point links were all that were required. The advantage of this is that a significant amount of FPGA resource could be saved by moving to a lightweight interface (but routing would be lost). However, since this is not the way SKA-SA is currently using the platform, your support/compatibility with SKA-SA toolflow probably won't be as "out-the-box" as it is when sticking with the mainstream SKA-SA toolflow/firmware implementations. Kind regards, Clifford On 7/6/2017 11:08 AM, Wesley New wrote: > Hi Mark. > > The SKARABs currently only work with a DHCP server. So all bets are > off while attempting to send/receive data without the interface > receiving an IP, Netmask and Gateway. My guess would be that you are > sending data, but I am not sure what IP/MAC it would be sent to or > from. Because the ARP table wouldn't be set up correctly prior to a > DHCP response being received and processed. I would recommend using > the SKARABs on a network or connected directly to a server. I don't > see much of a use-case for point-to-point connections either (How > would you get the data off the SKARABs). > > Also be careful of the promiscuous mode. It allows all data received > on the interface through to the fabric. It is only meant for debugging > network problems, kind of a tcpdump for the SKARAB. You should be able > to send and receive data between 2 SKARABs without setting promiscuous > mode. > > If you would like I can pull up some test model files where we send > and receive data between 2 networked SKARABs. With functionality to > configure data rates and packet sizes etc. > > Regards > > Wes > > > Wesley New > South African SKA Project > +2721 506 7300 > www.ska.ac.za <http://www.ska.ac.za> > > > > On Wed, Jul 5, 2017 at 10:46 PM, Peryer, Mark A. > <mark.per...@cfa.harvard.edu <mailto:mark.per...@cfa.harvard.edu>> wrote: > > Hello, > > I have solved my issue of not being able to receive data. Instead > of directly connecting the SKARABs, I routed them through a 40 GbE > switch where the DHCP server can assign both of the SKARABs an IP > address. I am still not sure why directly connecting them does not > work, however that appears to be what was causing the issue all along. > > Thanks for your help, > > Mark > > On Wed, Jul 5, 2017 at 4:20 PM, Peryer, Mark A. > <mark.per...@cfa.harvard.edu <mailto:mark.per...@cfa.harvard.edu>> > wrote: > > Hello, > > I am still having issues receiving data over 40 GbE. I > currently have a direct connection between the transmitting > and receiving SKARABs, both on the leftmost 40 GbE port of > Mezzanine slot 3 . Using tcpdump on a server with a 40GbE > interface, I was able to confirm that the SKARAB which > transmits data is successfully doing so. Also, both SKARABs > can successfully be pinged over 40GbE. Despite this, I am > still unable to receive data on the other SKARAB, even when > Promiscuous Mode is enabled. I did observe that the orange and > green LEDs are turned on for both of the SKARAB's 40GbE ports. > The green light blinks on the SKARAB that is transmitting data > and the orange light blinks on the SKARAB that is receiving > data, therefore it appears they are communicating with each > other. > > Any other suggestions on how to debug the reception data over > 40 GbE would be much appreciated, as I am still unsure what > could be causing the issue. > > Thanks, > > Mark > > On Fri, Jun 30, 2017 at 3:57 AM, Paul Prozesky > <paul.proze...@gmail.com <mailto:paul.proze...@gmail.com>> wrote: > > Morning Mark > > Please have a look here for SKARAB 40gbe TX and RX models > with demo software. > > > https://github.com/ska-sa/mkat_fpga/tree/devel/source/skarab_dev/tut2 > > <https://github.com/ska-sa/mkat_fpga/tree/devel/source/skarab_dev/tut2> > > Cheers > Paul > > > On 30 June 2017 at 07:46, Jason Manley <jman...@ska.ac.za > <mailto:jman...@ska.ac.za>> wrote: > > Hi Mark > > The current SKARAB 40G yellowblock is a bit of a hack. > > 1) It is currently not parameterised, and is > hard-coded for the first port on Mezzanine slot 3. > > 2) The tx_valid and rx_valid lines are 4-bits wide, > not 1 bit. This will be rectified soon, but in the > meanwhile, just feed the value "3" to tx_valid to send > packets. > > 3) Note that there was a recent change to the TX and > RX 64-bit word ordering (was previously incorrectly > swapped). Make you built your bitstreams with a recent > git checkout. > > It's on our todo list to get it properly parameterised > and to fix the 4-bit weirdness, but it's not clear > when we'll have that done. In the meanwhile, the 40G > does work, and we are successfully using this first > port actively at SKA-SA. > > Some things you should try to help debug: > > 1) The microblaze will attempt to DHCP on the 40G > interface. Did it obtain a lease? Check your DHCP > server logs. > > 2) Check (using casperfpga software) that the cores > are actually configured like you think. There was a > version of microblaze that overwrote things. And if > it's DHCP'ing (this is how we use the ports), then the > IP will have changed from what you expect. > myfpgaobject.gbes['my_forty_gbe_core'].print_core_details() > (or g.get_core_details()). > > 3) What does tcpdump/wireshark show is coming out of > the TX board? > > 4) Did you select different MAC addresses for the > "hardcoded" yellowblocks on the TX and RX side? Else a > switch might not forward the packets. > > 5) Can you ping both 40G interfaces from a computer? > > Jason Manley > CBF Manager > SKA-SA > > Cell: +27 82 662 7726 <tel:+27%2082%20662%207726> > Work: +27 21 506 7300 <tel:+27%2021%20506%207300> > > On 29 Jun 2017, at 20:37, Peryer, Mark A. > <mark.per...@cfa.harvard.edu > <mailto:mark.per...@cfa.harvard.edu>> wrote: > > > Hello, > > > > I am trying to send data from one SKARAB to another > SKARAB over 40GbE. I have created two separate JASPER > files, one for receiving and one for transmitting. > Each design has one forty_gbe yellow block and are > configured similar to Tutorial 2 on the casper > website. In the JASPER file used for transmitting, I > have placed a snapshot block on the tx_data signal for > the forty_gbe block and am able to read the correct > data from the snapshot block using casperfpga. While > it appears that the correct data is being transmitted, > nothing is received by the second SKARAB. In the > JASPER file used for receiving data, I have a snapshot > block on the rx_data signal, which should trigger once > a valid frame is received. However, when I run > fpga01.snapshots.snapshot2.print_snap(50) it just > hangs, indicating nothing has been received. > > > > I did notice that when I right click on the > forty_gbe yellow block and go to Mask >> Edit Mask >> > Parameters&Dialog, there is a menu that allows for the > card slot location of the 40GbE card to be selected. > Our SKARAB units have the 40GbE card populated on > Mezzanine Slot 3, however only Card Slot 0 and 1 are > available to be selected. Should this option be set to > slot 3 in order to properly use the 40GbE ports? > > > > I should also mention that I have configured the > tx_dest_ip block input to be 192.168.5.20 and the > tx_dest_port to be 10000, which are the default values > for the forty_gbe block. > > > > If there is anything else I should look at in order > to properly configure the forty_gbe yellow blocks > please advise. > > > > Thanks, > > > > Mark > > > > > > > > -- > > You received this message because you are subscribed > to the Google Groups "casper@lists.berkeley.edu > <mailto:casper@lists.berkeley.edu>" group. > > To unsubscribe from this group and stop receiving > emails from it, send an email to > casper+unsubscr...@lists.berkeley.edu > <mailto:casper%2bunsubscr...@lists.berkeley.edu>. > > To post to this group, send email to > casper@lists.berkeley.edu > <mailto:casper@lists.berkeley.edu>. > > -- > You received this message because you are subscribed > to the Google Groups "casper@lists.berkeley.edu > <mailto:casper@lists.berkeley.edu>" group. > To unsubscribe from this group and stop receiving > emails from it, send an email to > casper+unsubscr...@lists.berkeley.edu > <mailto:casper%2bunsubscr...@lists.berkeley.edu>. > To post to this group, send email to > casper@lists.berkeley.edu > <mailto:casper@lists.berkeley.edu>. > > > > > -- > You received this message because you are subscribed to the Google > Groups "casper@lists.berkeley.edu > <mailto:casper@lists.berkeley.edu>" group. > To unsubscribe from this group and stop receiving emails from it, > send an email to casper+unsubscr...@lists.berkeley.edu > <mailto:casper+unsubscr...@lists.berkeley.edu>. > To post to this group, send email to casper@lists.berkeley.edu > <mailto:casper@lists.berkeley.edu>. > > > -- > You received this message because you are subscribed to the Google > Groups "casper@lists.berkeley.edu" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to casper+unsubscr...@lists.berkeley.edu > <mailto:casper+unsubscr...@lists.berkeley.edu>. > To post to this group, send email to casper@lists.berkeley.edu > <mailto:casper@lists.berkeley.edu>. -- You received this message because you are subscribed to the Google Groups "casper@lists.berkeley.edu" group. To unsubscribe from this group and stop receiving emails from it, send an email to casper+unsubscr...@lists.berkeley.edu. To post to this group, send email to casper@lists.berkeley.edu.
