Hi all,
For reference I've attached a summary of our problems below, and a few
things I have attempted to do to isolate it. The short of it is that we
are unable to transfer large amounts of data across the ethernet
reliably regardless of;
--kernel version
--whether we are usb mount or nfs mount root file system.
--network protocol used for transfer
The way the crash happens varies, and is not repeatable. Sometime it
seems to be a userspace crash, sometimes it is a kernel panic. I have
been unable to see any real pattern in the crash reports. This to me
seems to indicate that the root cause of the problem may be common, and
either an obscure kernel problem or possibly something in the interface
between the kernel and the hardware or in the hardware itself.
It wouldn't be a big effort to re implement our software to run on a
remote machine and talk to the ROACH over KATCP, rather than run locally
on the ppc. But since it would require a complete rewrite of the
software, we haven't tested this yet. Perhaps it is worth trying.
The catch is that I am still really unsure whether we are dealing with
many symptoms of the same problem; or many different problems.
Anyway, I would like to thank you for all your input, and will let you
know if and how we find a satisfactory solution.
cheers
Kjetil
Here is the summary:
*The problem*
The system crashes when downloading large files. There appears to be
varying causes for this crash that may or may not have a common
underlying reason.
I have attempted to isolate the problem by
• Downloading using different protocols and software; ssh and two
different ftp servers.
• Mounting the filesystem over NFS as opposed to USB
• Installing well-known and used kernels, and comparing to custom kernels.
SSH
SSH always crashes with “Invalid MAC on input” or related error
messages. This appears to be a problem with SSH.
*FTP*
System instabilities were observed using two different ftp servers;
proftpd and pure-ftpd.
In the best case, with pure-ftpd was able to download 2-3 files, each of
size about 2GB before system crashing. Looking through the call stack
seemed to indicate that the crash happened in EMAC interface functions.
(ie ethernet).
However, we have no way of knowing whether these crashes are in fact
rather side-effects of the USB subsystem misbehaving. Jason from the
Casper mailing list has once again reconfirmed that USB on powerpcs is
"notoriously unreliable".
*DIFFERENT KERNELS - DIFFERENT PROBLEMS*
Using some kernels (the latest) saw the link unable to come up at all,
while both a custom compiled older kernel (a couple of months ago) and a
downloaded image, "uImage-20091006-mmcfix" both saw the link come up,
but with all the crashes described.
*ELIMINATING USB AS A CAUSE*
To eliminate the effects of USB, I mounted the root filesystem remotely
using NFS. I make a few observations;
*SSH*
Still dies from time to time with the "Invalid MAC" error message. This
was expected as we have already pretty much determined that this error
is ssh-specific and not related to our other worries.
*ETHERNET*
Comes up nicely. System mounts remotely and file access has not caused
any obvious problems. In fact I have not really had any problems that I
can trace directly back to the Ethernet.
That being said, the systems seems to crash after a little while with
this setup also. The error messages have been varying. Only once has it
been a kernel crash, and then, looking at the call stack it no longer
appears to crash inside EMAC access functions.
The download speeds seem quite variable; but this is probably more
likely due to the network since the operating system is over NFS than
the ROACH board itself.
Jason Manley wrote:
Marc Welz or David George built that kernel. They are the best people
to ask about this. I've cc'd them, though I'm not sure either would
have the config file from that release. It might be easiest to
checkout an older svn version.
Might I suggest that instead of recording data to a USB HDD, that you
rather record it across the network to another computer? If you don't
want to use KATCP for dumping the data directly from your FPGA, you
can always mount an NFS network share on your ROACH and record the
data there. The USB on the PPC platforms are notoriously unreliable.
Jason
On 03 Nov 2009, at 03:05, Kjetil Wormnes wrote:
Hi Jason,
Thank you again for your reply. I can use FTP or even write my own
little raw socket transfer routine, and it seems to work, I can
transfer a few gigabyte-size files.
However, at the end of this, the other problem kicks in; causing a
system crash. I believe this is a kernel problem, as it exhibits
itself differently with different kernels I have tried.
So, putting the ssh problem aside as something that we can work
around and returning to the other request I made;
I am compiling my own kernel because I seem to need to in order to
get EHCI and EXT3 to work properly.
However, when I do, EMAC can't autonegotiate a link, and even
forcing it to something doesn't work. The link comes up, then drops
out again... repeatedly.
The interesting thing is this problem *does not* occur when I
compile my kernel using an svn checkout from a couple of months ago.
Even with the exact same .config file.
At least this is the case as far as I can tell.
Now, in order to be 100% sure that it is in fact a difference in the
source that is causing this problem, rather than just the .config. I
would love it if you could send me the .config file used to compile
the uImage-20091006-mmcfix kernel.
The ethernet interface does appear to be more stable with that
kernel, but unfortunately I can't use it as it doesn't allow USB 2.0
speeds, so if you please, the .config file would be very useful.
Thanks again for all your help
Kjetil
Jason Manley wrote:
There appears to be some issue with ssh on ROACH with large
transfers. It is definitely not a hardware problem as other
network transfers work fine. Both Andrew Martens and myself
regularly transfer large amounts of data (>1GB) using KATCP. This
ssh bug has become a low priority for us as we concentrate on
other things. If you do not want to try'n debug it yourself, I
recommend you try an FTP server.
Kjetil, you are correct; at present, KATCP does not support
transfer of arbitrary files from filesystem.
Jason
On 02 Nov 2009, at 00:51, Kjetil Wormnes wrote:
Hi Jason,
thank you for your reply. The SUN link was very descriptive.
Firstly, it appears the problem is still there with the kernel
build you suggested/ After a few megabytes, the connection closes
telling me; "Corrupted MAC on input".
But interestingly it seems to have solved another problem that I
was having with one of our ROACH boards. It would be great if you
could send me the .config file for that build so I can compare it
with mine. I have a custom kernel as I like ext3 support and a
few other bits and pieces, but have been having some issues
getting the network to establish a stable link.
Now, back to the problem; We have a locally attached harddrive
that we are writing our data to over USB. Occasionally we want to
connect and download these. That's why I am using ssh. I can't
really use KATCP for this, can I?
Thanks again,
Kjetil
Jason Manley wrote:
Um, no, this is probably a different problem. You are getting
these errors while using SSH/SCP, right? The hardware problem
with faulty PHY manifests as one or more of the PHY LEDs
flashing on/ off (there are three red ones next to the PHY
chip). If your link is stable, then I believe the hardware is
fine.
The "MAC" problem appears to be software related, and comes and
goes depending on the kernel build. It does not refer to the
MAC address, but rather ssh's Machine Authentication Code.
Check out http://blogs.sun.com/janp/entry/ssh_messages_code_bad_packet
for some info.
Dave's made various changes to try'n fix it, and increasing
some software buffer has solved it for me. I no longer see
this problem, but it's probably been masked rather than solved.
Also, you never see it using KATCP, which is one more reason to
use that method for larger transfers.
WRT large (>1GB) transfers, remember that it will take a long
time to pull that much data off the FPGA. It does so in pages
of ~4000Bytes at a time. Also make sure you're using the
latest kernel. We discovered a bug in this paging system during
the workshop. http://casper.berkeley.edu/svn/trunk/roach/sw/binaries/linux/uImage-20091006-mmcfix
should be good. I have never tried pulling such volumes over
the SSH shell, but it works fine with KATCP.
I will ask him to comment further.
Jason
On 30 Oct 2009, at 01:25, John Ford wrote:
casper collaborators,
appended below is further info on roach ethernet problems seen
at CSIRO:
any ideas?
If I recall correctly, Alan mentioned this problem at the
workshop, and
the problem was that some of the PHY chips were faulty at one
point. This
may be what's going on. Hopefully someone knows for sure!
John
thanks,
dan
-------- Original Message --------
Subject: Re: SPDO ROACH spectrometer
Date: Fri, 30 Oct 2009 09:19:01 +1100
From: Kjetil Wormnes <kjetil.worm...@csiro.au>
To: Dan Werthimer <d...@ssl.berkeley.edu>
Hi Dan and Wan
I can confirm that we are seeing at least some of the problems
with
another ROACH board as well. This time it is connected
directly to a
computer with a short CATY5 cable.
So maybe this indicates that it is less likely to be a
hardware problem?
Incidentally, the error message that happens when attempting
to download
a large file over sftp is "Corrupted MAC on input".
cheers
Kjetil
Dan Werthimer wrote:
hi wan,
i don't know of anyone who has roach ethernet
problems at 100 Mbit/sec.
i'm cc'ing casper community to see if anyone has any ideas.
in general, it's good to post questions to cas...@lists,
so that everyone can help answer, and everyone can see the
answers,
and the info will be captured in the wiki/email archive.
if you want you can buy or ask digicom if they can send you
another national PHY chip and see if this helps.
also you might want to try using short cable, and/or a cat6
cable.
is your roach connected directly to a computer, or going
through a switch? might be interesting to try a different NIC
or different switch or different computer.
best,
dan
On 10/29/2009 02:47 PM, wan.ch...@csiro.au wrote:
Hi Dan:
I believe you have done a very nice job.
My problem is Ethernet port is not very reliable. Even
running at
100MHz, the Ethernet port will be disconnected at some
times. Normally,
it can resume after reboot whole system.
And I could not transfer big file through ethernet. Small
files like a
few MB are all right. But I could not download 1GB file
from Roach at
all.
So Dan, could this problem be solved by replacing the on
board PHY?
Thanks
Wan