Re: [casper] Re: Help with 10GBE network

[James Smith]

Thanks for a well-thought-out response, Jack. Just to add my 2c -

Depending on your install, there's sometimes a firewall which blocks
high-range UDP ports which we typically use by default. So if you can see
packets coming in with Wireshark / tcpdump, but for some reason your Python
sockets are hanging around with nothing happening, check that your firewall
isn't blocking the packets.

Ubuntu *shouldn't* have this issue but I have seen it on CentOS.


On Wed, Oct 9, 2019 at 1:55 AM Jack Hickish  wrote:

> Hi Heystek,
>
> Gonna throw this one to the maillist because some other folk might find
> the conversation useful.
>
> Am I right in thinking that you have a ROACH2 design which is outputting
> data over the 10GbE port (similar to the transmission side of tutorial 2)
> but want to know how to receive the data from a server, rather than just
> the loopback test performed in the tutorial?
> If so, the answer is, you'll need some code similar to
> https://wiki.python.org/moin/UdpCommunication#Receiving
> This will work fine for relatively low speed - for higher performance
> you'll probably want to look at using the C socket library directly, or one
> of the various higher-level libraries that other astronomy folk use which
> include (e.g. spead2 (for reading SPEAD packets) and/od full piplineing
> engines like HASHPIPE / Bifrost). Needless to say, everyone likes their
> software to be slightly different to everyone else's, so there are a bunch
> of libraries around. But I'd recommend starting with python to get the hang
> of things.
>
> In the example, you instantiate a UDP/IP socket (the traffic from a CASPER
> 10GbE core is always UDP), and bind it to the IP address and port you want
> to receive on (i.e, this should be the same as the destination address the
> ROACH is using).
> Then you simply call recvfrom (or recv) against the socket and get your
> data back as a raw binary string (which you'll need to decode with
> struct.unpack calls (or numpy.fromstring, or similar).
>
> Things you might want to try when this inevitably doesn't work out the box.
> 1. Verify your ROACH really is sending packets by putting some registers
> which count the number of valid eof pulses going into the ethernet core.
> (some of this kind of debugging is automatically built in to casperfpga --
> see tut2 for details)
> 2. run tcpdump (`sudo tcpdump -i `) to verify
> that packets are showing up at your machine. Use tcpdump's -e flag to check
> that the MAC addresses / IP addresses in this packet are what you expect.
> 3. Write a simple python TX script (
> https://wiki.python.org/moin/UdpCommunication#Sending) to send packets
> from one server to another, to verify your receive code is working without
> the bazillion unknowns a ROACH brings to the table.
> 4. Be sure to check that the packets you are sending are smaller than the
> MTU (
> https://www.cyberciti.biz/faq/how-can-i-setup-the-mtu-for-my-network-interface/)
> that your server will accept
>
>
> Cheers
> Jack
>
> On Tue, 8 Oct 2019 at 03:08, Heystek Grobler 
> wrote:
>
>> Hey Jack
>>
>> I hope that you are still doing well?
>>
>> I am struggling with the 10GBE SFP+ block. I have it in my design, the
>> design complies and is working, but how do I actually get the data from
>> the 10 GBE block?
>>
>> To use a python script with a BRAM block over normal ethernet is easy, but
>> how do I go to work with the 10 GBE block? The tutorials do not really
>> explain it or provide an example.
>>
>> Have a great day
>>
>> Heystek
>>
>>
>> --
>> Heystek Grobler
>> 0832721009
>> 012 3013202
>> heys...@hartrao.ac.za
>>
>> Spectroscopy Research Group
>> Department of Fundamental Astronomy
>> South African Radio Astronomy Observatory (SARAO) Hartebeesthoek Site
>>
>>
>> --
> You received this message because you are subscribed to the Google Groups "
> casper@lists.berkeley.edu" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to casper+unsubscr...@lists.berkeley.edu.
> To view this discussion on the web visit
> https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAG1GKSnwU6og5vDkWJOa%2BOkpM5bN4Rr7W6NYTSLN%3DyzG6--6zw%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"casper@lists.berkeley.edu" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to casper+unsubscr...@lists.berkeley.edu.
To view this discussion on the web visit 
https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAG67D36ZkXUZVYw0rjAqw0tNMY97L-RNsiKW98RVP%3DYYvSfxwA%40mail.gmail.com.

[casper] Re: Help with 10GBE network

[Jack Hickish]

Hi Heystek,

Gonna throw this one to the maillist because some other folk might find the
conversation useful.

Am I right in thinking that you have a ROACH2 design which is outputting
data over the 10GbE port (similar to the transmission side of tutorial 2)
but want to know how to receive the data from a server, rather than just
the loopback test performed in the tutorial?
If so, the answer is, you'll need some code similar to
https://wiki.python.org/moin/UdpCommunication#Receiving
This will work fine for relatively low speed - for higher performance
you'll probably want to look at using the C socket library directly, or one
of the various higher-level libraries that other astronomy folk use which
include (e.g. spead2 (for reading SPEAD packets) and/od full piplineing
engines like HASHPIPE / Bifrost). Needless to say, everyone likes their
software to be slightly different to everyone else's, so there are a bunch
of libraries around. But I'd recommend starting with python to get the hang
of things.

In the example, you instantiate a UDP/IP socket (the traffic from a CASPER
10GbE core is always UDP), and bind it to the IP address and port you want
to receive on (i.e, this should be the same as the destination address the
ROACH is using).
Then you simply call recvfrom (or recv) against the socket and get your
data back as a raw binary string (which you'll need to decode with
struct.unpack calls (or numpy.fromstring, or similar).

Things you might want to try when this inevitably doesn't work out the box.
1. Verify your ROACH really is sending packets by putting some registers
which count the number of valid eof pulses going into the ethernet core.
(some of this kind of debugging is automatically built in to casperfpga --
see tut2 for details)
2. run tcpdump (`sudo tcpdump -i `) to verify
that packets are showing up at your machine. Use tcpdump's -e flag to check
that the MAC addresses / IP addresses in this packet are what you expect.
3. Write a simple python TX script (
https://wiki.python.org/moin/UdpCommunication#Sending) to send packets from
one server to another, to verify your receive code is working without the
bazillion unknowns a ROACH brings to the table.
4. Be sure to check that the packets you are sending are smaller than the
MTU (
https://www.cyberciti.biz/faq/how-can-i-setup-the-mtu-for-my-network-interface/)
that your server will accept


Cheers
Jack

On Tue, 8 Oct 2019 at 03:08, Heystek Grobler  wrote:

> Hey Jack
>
> I hope that you are still doing well?
>
> I am struggling with the 10GBE SFP+ block. I have it in my design, the
> design complies and is working, but how do I actually get the data from
> the 10 GBE block?
>
> To use a python script with a BRAM block over normal ethernet is easy, but
> how do I go to work with the 10 GBE block? The tutorials do not really
> explain it or provide an example.
>
> Have a great day
>
> Heystek
>
>
> --
> Heystek Grobler
> 0832721009
> 012 3013202
> heys...@hartrao.ac.za
>
> Spectroscopy Research Group
> Department of Fundamental Astronomy
> South African Radio Astronomy Observatory (SARAO) Hartebeesthoek Site
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"casper@lists.berkeley.edu" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to casper+unsubscr...@lists.berkeley.edu.
To view this discussion on the web visit 
https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAG1GKSnwU6og5vDkWJOa%2BOkpM5bN4Rr7W6NYTSLN%3DyzG6--6zw%40mail.gmail.com.