I've looked at this issue some more and it appears to me that sched_pickcpu() is
never called for CPU-bound threads that use up their slices completely and never
do any voluntary switches. A result of this is that the threads stay on their
initial CPU until re-balance mechanism kicks in.
There
, at least when we talk about performance
problems. There can be issues with the scheduler making sub-optimal scheduling
decisions.
-- The bug might be some case where the spin lock isn't being released
when it should be, and that could result in Heavy I/O blocks
FreeBSD box for several
in Heavy I/O blocks
FreeBSD box for several seconds. Note that the server was under
heavy I/O load when this panic occurred.
A panic is a panic.
spin lock 0x80cb52c0 (sched lock 1) held by
0xff0012c7f8c0 (tid 100317) too long
panic: spin lock held too long
I
on 31/07/2011 22:03 Rick Macklem said the following:
Ok, so if the scheduler spin lock is being held too long, what could
cause this, if it isn't a scheduler bug?
I can't think of how NFS would affect this beyond causing a heavy I/O
load, but if there is some way it does, I suppose I need to
Abdriy Gapon wrote:
on 31/07/2011 22:03 Rick Macklem said the following:
Ok, so if the scheduler spin lock is being held too long, what could
cause this, if it isn't a scheduler bug?
I can't think of how NFS would affect this beyond causing a heavy
I/O
load, but if there is some way
on 26/07/2011 00:44 Rick Macklem said the following:
hrs sent me this panic. I'm wondering if it might be relevant to this?
To 'this' what?
A panic is a panic.
spin lock 0x80cb52c0 (sched lock 1) held by 0xff0012c7f8c0 (tid
100317) too long
panic: spin lock held too long
I
lock too long,
that it might be another symptom of the same problem as when the scheduler
seems to lock up for several seconds.
-- The bug might be some case where the spin lock isn't being released
when it should be, and that could result in Heavy I/O blocks
FreeBSD box for several seconds
on 12/07/2011 11:05 Andriy Gapon said the following:
I think that the best thing you can further provide (as objective evidence for
the problem at hand) is ktr(4) traces for at least KTR_SCHED mask. Perhaps
you
even already have them from your previous sessions with Jeff.
P.S. This is not
On Mon, Jul 25, 2011 at 01:00:27PM +0300, Andriy Gapon wrote:
on 12/07/2011 11:05 Andriy Gapon said the following:
I think that the best thing you can further provide (as objective evidence
for
the problem at hand) is ktr(4) traces for at least KTR_SCHED mask. Perhaps
you
even
Steve Kargl wrote:
On Mon, Jul 25, 2011 at 01:00:27PM +0300, Andriy Gapon wrote:
on 12/07/2011 11:05 Andriy Gapon said the following:
I think that the best thing you can further provide (as objective
evidence for
the problem at hand) is ktr(4) traces for at least KTR_SCHED mask.
Do you have a corefile for this panic?
Attilio
2011/7/25 Rick Macklem rmack...@uoguelph.ca:
Steve Kargl wrote:
On Mon, Jul 25, 2011 at 01:00:27PM +0300, Andriy Gapon wrote:
on 12/07/2011 11:05 Andriy Gapon said the following:
I think that the best thing you can further provide (as
On Wed, 13.07.2011 at 00:40:49 +0300, Gleb Kurtsou wrote:
On (11/07/2011 16:36), m...@freebsd.org wrote:
On Mon, Jul 11, 2011 at 4:00 PM, Ali Mashtizadeh mashtiza...@gmail.com
wrote:
Maybe someone can setup something like reviewboard [1] for developers
to use. This may also help folks
On Sun, Jul 17, 2011 at 4:54 AM, Ulrich Spörlein u...@spoerlein.net wrote:
...
Having a project adopted way of sharing work in progress will be a step
forward. Yes, I'm aware of perforce, it's to hard to use and wasn't
designed to share and test ideas. I think guthub can be a very good
on 12/07/2011 00:48 Arnaud Lacombe said the following:
Hi,
On Mon, Jul 11, 2011 at 5:14 PM, Andriy Gapon a...@freebsd.org wrote:
on 11/07/2011 23:33 Arnaud Lacombe said the following:
For the record, I would like to see enforced public review for _every_
patch *before* it is checked in, as
on 11/07/2011 19:16 Steve Kargl said the following:
On Mon, Jul 11, 2011 at 06:07:04PM +0300, Andriy Gapon wrote:
But it's not clear which of the processes are slaves and which is master.
It's also not clear why the master takes so much CPU (on par with the
slaves) -
from my reading of its
Hi,
On Mon, Jul 11, 2011 at 7:36 PM, m...@freebsd.org wrote:
On Mon, Jul 11, 2011 at 4:00 PM, Ali Mashtizadeh mashtiza...@gmail.com
wrote:
Maybe someone can setup something like reviewboard [1] for developers
to use. This may also help folks who want to keep abreast of the
current work in
Sic... If you allow me the comparison, FreeBSD development is as open
as are the US (and, to some extend, most western country) borders
nowadays open to aliens, and believe me, this is not a compliment.
- Arnaud
___
freebsd-current@freebsd.org
On 07/12/11 20:10, Matt wrote:
Sic... If you allow me the comparison, FreeBSD development is as open
as are the US (and, to some extend, most western country) borders
nowadays open to aliens, and believe me, this is not a compliment.
- Arnaud
I like the comment, although I disagree. In
On (11/07/2011 16:36), m...@freebsd.org wrote:
On Mon, Jul 11, 2011 at 4:00 PM, Ali Mashtizadeh mashtiza...@gmail.com
wrote:
Maybe someone can setup something like reviewboard [1] for developers
to use. This may also help folks who want to keep abreast of the
current work in a particular
On 07/07/2011 22:08, Steve Kargl wrote:
4BSD kernel gives for N = Ncpu + 1.
34 processes: 6 running, 28 sleeping
PID USERNAME THR PRI NICE SIZERES STATE C TIMECPU COMMAND
1417 kargl 1 710 370M 294M RUN 0 1:30 79.39% sasmp
1416 kargl 1 710
on 11/07/2011 17:41 Ivan Voras said the following:
On 07/07/2011 22:08, Steve Kargl wrote:
4BSD kernel gives for N = Ncpu + 1.
34 processes: 6 running, 28 sleeping
PID USERNAME THR PRI NICE SIZERES STATE C TIMECPU COMMAND
1417 kargl 1 710 370M 294M RUN
That top output is averaged and slow to adjust.
Using top as an indication as to what's really going on is likely
not a good idea.
2c,
Adrian
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To
On Mon, Jul 11, 2011 at 06:07:04PM +0300, Andriy Gapon wrote:
on 11/07/2011 17:41 Ivan Voras said the following:
On 07/07/2011 22:08, Steve Kargl wrote:
4BSD kernel gives for N = Ncpu + 1.
34 processes: 6 running, 28 sleeping
PID USERNAME THR PRI NICE SIZERES STATE C
On Mon, Jul 11, 2011 at 11:42:02PM +0800, Adrian Chadd wrote:
That top output is averaged and slow to adjust.
Using top as an indication as to what's really going on is likely
not a good idea.
Restoring top output here:
PID USERNAME THR PRI NICE SIZERES STATE C TIMECPU
On 11 July 2011 17:07, Andriy Gapon a...@freebsd.org wrote:
Yeah, but what problem is demonstrated here?
Are we confident that non-even workload is inherently bad?
E.g.:
79.39 + .. + 77.59 5 * 80 = 400
100.00 + ... + 55.18 ~~ 402 which is more than theoretically possible :-)
So it would
Hi,
On Thu, Jul 7, 2011 at 7:45 PM, Adrian Chadd adr...@freebsd.org wrote:
(OT, yes, but I'd like to take a stab at explaining why these things
fall to the wayside..)
On 7 July 2011 12:08, Arnaud Lacombe lacom...@gmail.com wrote:
What would be the point to even start looking at an issue?
On Mon, Jul 11, 2011 at 04:33:44PM -0400, Arnaud Lacombe wrote:
For the record, I would like to see enforced public review for _every_
patch *before* it is checked in, as a strong rule. gcc system is
particularly interesting. But it is not likely to happen in FreeBSD
where FreeBSD committers
on 11/07/2011 23:33 Arnaud Lacombe said the following:
For the record, I would like to see enforced public review for _every_
patch *before* it is checked in, as a strong rule. gcc system is
particularly interesting. But it is not likely to happen in FreeBSD
where FreeBSD committers are
Hi,
On Mon, Jul 11, 2011 at 4:40 PM, Steve Kargl
s...@troutmask.apl.washington.edu wrote:
On Mon, Jul 11, 2011 at 04:33:44PM -0400, Arnaud Lacombe wrote:
For the record, I would like to see enforced public review for _every_
patch *before* it is checked in, as a strong rule. gcc system is
Hi,
On Mon, Jul 11, 2011 at 5:14 PM, Andriy Gapon a...@freebsd.org wrote:
on 11/07/2011 23:33 Arnaud Lacombe said the following:
For the record, I would like to see enforced public review for _every_
patch *before* it is checked in, as a strong rule. gcc system is
particularly interesting.
Hi,
[re-sent publicly, I did not Replied-to-all:)]
On Mon, Jul 11, 2011 at 5:38 PM, Arnaud Lacombe lacom...@gmail.com wrote:
Hi,
On Mon, Jul 11, 2011 at 4:40 PM, Steve Kargl
s...@troutmask.apl.washington.edu wrote:
On Mon, Jul 11, 2011 at 04:33:44PM -0400, Arnaud Lacombe wrote:
For the
On Mon, Jul 11, 2011 at 05:50:44PM -0400, Arnaud Lacombe wrote:
On Mon, Jul 11, 2011 at 5:38 PM, Arnaud Lacombe lacom...@gmail.com wrote:
Hi,
On Mon, Jul 11, 2011 at 4:40 PM, Steve Kargl
s...@troutmask.apl.washington.edu wrote:
On Mon, Jul 11, 2011 at 04:33:44PM -0400, Arnaud Lacombe
Maybe someone can setup something like reviewboard [1] for developers
to use. This may also help folks who want to keep abreast of the
current work in a particular subsystem or get involved into the
development process more. At my company we use reviews and it seems to
help the catch some bugs and
On Mon, Jul 11, 2011 at 4:00 PM, Ali Mashtizadeh mashtiza...@gmail.com wrote:
Maybe someone can setup something like reviewboard [1] for developers
to use. This may also help folks who want to keep abreast of the
current work in a particular subsystem or get involved into the
development
m...@freebsd.org wrote:
On Mon, Jul 11, 2011 at 4:00 PM, Ali Mashtizadeh
mashtiza...@gmail.com wrote:
Maybe someone can setup something like reviewboard [1] for
developers
to use. This may also help folks who want to keep abreast of the
current work in a particular subsystem or get
on 07/07/2011 06:11 Steve Kargl said the following:
Unfortunately, I have neither the brain capacity and time nor
the money to fix the issue. To solve OP's problem in the
short, the simplest solution may be to switch to 4BSD. Let's
face, ULE is not a silver bullet.
I think that piling up
on 06/07/2011 21:00 Steve Kargl said the following:
On Wed, Jul 06, 2011 at 05:05:41PM +, Poul-Henning Kamp wrote:
In message 20110706170132.ga68...@troutmask.apl.washington.edu, Steve
Kargl w
rites:
I periodically ran the same type test in the 2008 post over the
last three years.
on 06/07/2011 21:11 Nathan Whitehorn said the following:
On 07/06/11 13:00, Steve Kargl wrote:
AFAICT, it is a cpu affinity issue. If I launch n+1 MPI images
on a system with n cpus/cores, then 2 (and sometimes 3) images
are stuck on a cpu and those 2 (or 3) images ping-pong on that
cpu. I
Steve Kargl s...@troutmask.apl.washington.edu wrote:
Let's face, ULE is not a silver bullet.
Or perhaps it is, but this particular problem is so heavily armored
as to demand depleted uranium :)
___
freebsd-current@freebsd.org mailing list
On 06/07/2011 19:05, Poul-Henning Kamp wrote:
In message20110706170132.ga68...@troutmask.apl.washington.edu, Steve Kargl w
rites:
I periodically ran the same type test in the 2008 post over the
last three years. Nothing has changed. I even set up an account
on one node in my cluster for
On 06/07/2011 20:11, Nathan Whitehorn wrote:
I've seen exactly this problem with multi-threaded math libraries, as
well. Using parallel GotoBLAS on FreeBSD gives terrible performance
because the threads keep migrating between CPUs, causing frequent cache
misses.
On both schedulers?
On Thu, Jul 07, 2011 at 10:27:53AM +0300, Andriy Gapon wrote:
on 06/07/2011 21:11 Nathan Whitehorn said the following:
On 07/06/11 13:00, Steve Kargl wrote:
AFAICT, it is a cpu affinity issue. If I launch n+1 MPI images
on a system with n cpus/cores, then 2 (and sometimes 3) images
are
On Thu, Jul 7, 2011 at 5:14 PM, Steve Kargl
s...@troutmask.apl.washington.edu wrote:
On Thu, Jul 07, 2011 at 10:27:53AM +0300, Andriy Gapon wrote:
on 06/07/2011 21:11 Nathan Whitehorn said the following:
On 07/06/11 13:00, Steve Kargl wrote:
AFAICT, it is a cpu affinity issue. If I
on 07/07/2011 18:14 Steve Kargl said the following:
On Thu, Jul 07, 2011 at 10:27:53AM +0300, Andriy Gapon wrote:
on 06/07/2011 21:11 Nathan Whitehorn said the following:
On 07/06/11 13:00, Steve Kargl wrote:
AFAICT, it is a cpu affinity issue. If I launch n+1 MPI images
on a system with n
On Thu, Jul 07, 2011 at 10:42:39PM +0300, Andriy Gapon wrote:
on 07/07/2011 18:14 Steve Kargl said the following:
I'm using OpenMPI. These are N Ncpu processes not threads,
I used 'thread' in a sense of a kernel thread. It shouldn't
actually matter if it's a process or a thread in
On 07/07/11 09:04, Andriy Gapon wrote:
on 07/07/2011 06:11 Steve Kargl said the following:
Unfortunately, I have neither the brain capacity and time nor
the money to fix the issue. To solve OP's problem in the
short, the simplest solution may be to switch to 4BSD. Let's
face, ULE is not a
On 07/07/11 09:27, Andriy Gapon wrote:
on 06/07/2011 21:11 Nathan Whitehorn said the following:
On 07/06/11 13:00, Steve Kargl wrote:
AFAICT, it is a cpu affinity issue. If I launch n+1 MPI images
on a system with n cpus/cores, then 2 (and sometimes 3) images
are stuck on a cpu and those 2
On Jul 7, 2011, at 3:51 PM, Hartmann, O. wrote:
This is quibbling. On heavy loads on networ, disk et cetera, isn't there
always and also a CPU bound load?
No. Properly written software blocks when waiting on network or disk I/O, and
doesn't sit there spinning in a busy-wait consuming CPU
(OT, yes, but I'd like to take a stab at explaining why these things
fall to the wayside..)
On 7 July 2011 12:08, Arnaud Lacombe lacom...@gmail.com wrote:
What would be the point to even start looking at an issue? You guys
(by you, I mean official committers on public list) don't care
When
When performing an update on the ports tree via portsnap fetch update
or when checking out (or) large Subversion repositories or when copying
large data files (~ 50 to 250 GB in size, results from numerical
modelings) or when compiling world, FreeBD 9.0 and FreeBSD 8.2-STABLE
tend to freeze
2011/7/6 O. Hartmann ohart...@zedat.fu-berlin.de:
having performance issues
Could you post /etc/sysctl.conf and /boot/loader.conf? Also, the
output of uname -a on all machines would be nice.
And since you don't use GENERIC, could you also tell us what
difference your setup is from a GENERIC
2011/7/6 O. Hartmann ohart...@zedat.fu-berlin.de
When performing an update on the ports tree via portsnap fetch update or
when checking out (or) large Subversion repositories or when copying large
data files (~ 50 to 250 GB in size, results from numerical modelings) or
when compiling world,
on 06/07/2011 13:37 arrowdodger said the following:
2011/7/6 O. Hartmann ohart...@zedat.fu-berlin.de
When performing an update on the ports tree via portsnap fetch update or
when checking out (or) large Subversion repositories or when copying large
data files (~ 50 to 250 GB in size, results
On Wed, Jul 6, 2011 at 3:37 PM, Andriy Gapon a...@freebsd.org wrote:
Just curious what your current value of sysctl kern.sched.preempt_thresh
is.
And if it's not 224 and if you haven't tried 224 yet, then could you please
try
it and see if there is any improvement?
This assumes that you
On Wed Jul 6 11, arrowdodger wrote:
2011/7/6 O. Hartmann ohart...@zedat.fu-berlin.de
When performing an update on the ports tree via portsnap fetch update or
when checking out (or) large Subversion repositories or when copying large
data files (~ 50 to 250 GB in size, results from
on 06/07/2011 15:35 arrowdodger said the following:
On Wed, Jul 6, 2011 at 3:37 PM, Andriy Gapon a...@freebsd.org
mailto:a...@freebsd.org wrote:
Just curious what your current value of sysctl kern.sched.preempt_thresh
is.
And if it's not 224 and if you haven't tried 224 yet, then
on 06/07/2011 16:33 Alexander Best said the following:
you might also want to try enabling options IPI_PREEMPTION. no idea, if this
improves your situation, though.
Just in case, this option has effect for 4BSD scheduler only.
--
Andriy Gapon
___
On Wed Jul 6 11, Andriy Gapon wrote:
on 06/07/2011 16:33 Alexander Best said the following:
you might also want to try enabling options IPI_PREEMPTION. no idea, if this
improves your situation, though.
Just in case, this option has effect for 4BSD scheduler only.
thanks. i did not know
On 07/06/11 12:37, arrowdodger wrote:
2011/7/6 O. Hartmannohart...@zedat.fu-berlin.de
When performing an update on the ports tree via portsnap fetch update or
when checking out (or) large Subversion repositories or when copying large
data files (~ 50 to 250 GB in size, results from numerical
On 07/06/11 14:35, arrowdodger wrote:
On Wed, Jul 6, 2011 at 3:37 PM, Andriy Gapona...@freebsd.org wrote:
Just curious what your current value of sysctl kern.sched.preempt_thresh
is.
And if it's not 224 and if you haven't tried 224 yet, then could you please
try
it and see if there is any
on 06/07/2011 18:33 O. Hartmann said the following:
On 07/06/11 14:35, arrowdodger wrote:
On Wed, Jul 6, 2011 at 3:37 PM, Andriy Gapona...@freebsd.org wrote:
Just curious what your current value of sysctl kern.sched.preempt_thresh
is.
And if it's not 224 and if you haven't tried 224 yet,
Has anyone re-run those IO benchmarks?
Something smells fishy there.. (with the benchmarking.)
adrian
2011/7/6 O. Hartmann ohart...@zedat.fu-berlin.de:
On 07/06/11 12:37, arrowdodger wrote:
2011/7/6 O. Hartmannohart...@zedat.fu-berlin.de
When performing an update on the ports tree via
On Wed, Jul 06, 2011 at 05:29:24PM +0200, O. Hartmann wrote:
I use SCHED_ULE on all machines, since it is supposed to be performing
better on multicore boxes, but there are lots of suggestions switching
back to the old SCHED_4BSD scheduler.
If you are using MPI in numerical codes, then
On 07/06/11 18:28, Steve Kargl wrote:
On Wed, Jul 06, 2011 at 05:29:24PM +0200, O. Hartmann wrote:
I use SCHED_ULE on all machines, since it is supposed to be performing
better on multicore boxes, but there are lots of suggestions switching
back to the old SCHED_4BSD scheduler.
If you are
On Wed, Jul 06, 2011 at 06:38:23PM +0200, Hartmann, O. wrote:
On 07/06/11 18:28, Steve Kargl wrote:
On Wed, Jul 06, 2011 at 05:29:24PM +0200, O. Hartmann wrote:
I use SCHED_ULE on all machines, since it is supposed to be performing
better on multicore boxes, but there are lots of suggestions
In message 20110706170132.ga68...@troutmask.apl.washington.edu, Steve Kargl w
rites:
I periodically ran the same type test in the 2008 post over the
last three years. Nothing has changed. I even set up an account
on one node in my cluster for jeffr to use. He was too busy to
investigate at
On Wed, Jul 06, 2011 at 05:05:41PM +, Poul-Henning Kamp wrote:
In message 20110706170132.ga68...@troutmask.apl.washington.edu, Steve Kargl
w
rites:
I periodically ran the same type test in the 2008 post over the
last three years. Nothing has changed. I even set up an account
on one
On 07/06/11 13:00, Steve Kargl wrote:
On Wed, Jul 06, 2011 at 05:05:41PM +, Poul-Henning Kamp wrote:
In message20110706170132.ga68...@troutmask.apl.washington.edu, Steve Kargl w
rites:
I periodically ran the same type test in the 2008 post over the
last three years. Nothing has changed.
Hi,
On Wed, Jul 6, 2011 at 12:28 PM, Steve Kargl
s...@troutmask.apl.washington.edu wrote:
On Wed, Jul 06, 2011 at 05:29:24PM +0200, O. Hartmann wrote:
I use SCHED_ULE on all machines, since it is supposed to be performing
better on multicore boxes, but there are lots of suggestions switching
On Wed, Jul 06, 2011 at 03:18:35PM -0400, Arnaud Lacombe wrote:
Hi,
On Wed, Jul 6, 2011 at 12:28 PM, Steve Kargl
s...@troutmask.apl.washington.edu wrote:
On Wed, Jul 06, 2011 at 05:29:24PM +0200, O. Hartmann wrote:
I use SCHED_ULE on all machines, since it is supposed to be performing
On 07/06/11 21:36, Steve Kargl wrote:
On Wed, Jul 06, 2011 at 03:18:35PM -0400, Arnaud Lacombe wrote:
Hi,
On Wed, Jul 6, 2011 at 12:28 PM, Steve Kargl
s...@troutmask.apl.washington.edu wrote:
On Wed, Jul 06, 2011 at 05:29:24PM +0200, O. Hartmann wrote:
I use SCHED_ULE on all machines, since
On 7/6/11, Hartmann, O. ohart...@zedat.fu-berlin.de wrote:
On 07/06/11 21:36, Steve Kargl wrote:
On Wed, Jul 06, 2011 at 03:18:35PM -0400, Arnaud Lacombe wrote:
Hi,
On Wed, Jul 6, 2011 at 12:28 PM, Steve Kargl
s...@troutmask.apl.washington.edu wrote:
On Wed, Jul 06, 2011 at 05:29:24PM
Offer a bounty for getting it fixed?
thanks,
Adrian
On 7 July 2011 05:00, Hartmann, O. ohart...@zedat.fu-berlin.de wrote:
On 07/06/11 21:36, Steve Kargl wrote:
On Wed, Jul 06, 2011 at 03:18:35PM -0400, Arnaud Lacombe wrote:
Hi,
On Wed, Jul 6, 2011 at 12:28 PM, Steve Kargl
On Thu, Jul 07, 2011 at 09:17:51AM +0800, Adrian Chadd wrote:
Offer a bounty for getting it fixed?
steve == ENOMONEY jeffr == ENOTIME
And, 4BSD works.
--
Steve
___
freebsd-current@freebsd.org mailing list
On 7 July 2011 09:51, Steve Kargl s...@troutmask.apl.washington.edu wrote:
On Thu, Jul 07, 2011 at 09:17:51AM +0800, Adrian Chadd wrote:
Offer a bounty for getting it fixed?
steve == ENOMONEY jeffr == ENOTIME
And, 4BSD works.
I meant it as a more general observation.
If something doesn't
On Thu, Jul 07, 2011 at 10:39:00AM +0800, Adrian Chadd wrote:
On 7 July 2011 09:51, Steve Kargl s...@troutmask.apl.washington.edu wrote:
On Thu, Jul 07, 2011 at 09:17:51AM +0800, Adrian Chadd wrote:
Offer a bounty for getting it fixed?
steve == ENOMONEY jeffr == ENOTIME
And, 4BSD
Hi,
On Wed, Jul 6, 2011 at 10:39 PM, Adrian Chadd adr...@freebsd.org wrote:
On 7 July 2011 09:51, Steve Kargl s...@troutmask.apl.washington.edu wrote:
On Thu, Jul 07, 2011 at 09:17:51AM +0800, Adrian Chadd wrote:
Offer a bounty for getting it fixed?
steve == ENOMONEY jeffr == ENOTIME
On 07/06/11 23:49, Oliver Pinter wrote:
On 7/6/11, Hartmann, O.ohart...@zedat.fu-berlin.de wrote:
On 07/06/11 21:36, Steve Kargl wrote:
On Wed, Jul 06, 2011 at 03:18:35PM -0400, Arnaud Lacombe wrote:
Hi,
On Wed, Jul 6, 2011 at 12:28 PM, Steve Kargl
s...@troutmask.apl.washington.edu wrote:
On 07/07/11 06:29, Arnaud Lacombe wrote:
Hi,
On Wed, Jul 6, 2011 at 5:00 PM, Hartmann, O.
ohart...@zedat.fu-berlin.de wrote:
On 07/06/11 21:36, Steve Kargl wrote:
On Wed, Jul 06, 2011 at 03:18:35PM -0400, Arnaud Lacombe wrote:
Hi,
On Wed, Jul 6, 2011 at 12:28 PM, Steve Kargl
79 matches
Mail list logo
