Send LSST-data mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
http://www.lsstmail.org/mailman/listinfo/lsst-data
or, via email, send a message with subject or body 'help' to
[EMAIL PROTECTED]
You can reach the person managing the list at
[EMAIL PROTECTED]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of LSST-data digest..."
Today's Topics:
1. Re: Google file system (Rob 'Commander' Pike)
2. Re: Google file system (Steffen Luitz)
3. Re: Google file system (Rob 'Commander' Pike)
4. Upcoming Meetings (Jeffrey Kantor)
5. Pipeline papers (Tim Axelrod)
----------------------------------------------------------------------
Message: 1
Date: Tue, 18 Jan 2005 12:55:01 -0800
From: "Rob 'Commander' Pike" <[EMAIL PROTECTED]>
Subject: Re: [LSST-data] Google file system
To: [email protected]
Message-ID: <[EMAIL PROTECTED]>
Content-Type: text/plain; charset=US-ASCII; format=flowed
The code for GFS is unlikely to be available soon, although
at some point it may well see the light of day. Sorry about that.
I wish it were not so.
GFS isn't much like a commodity file system, either in the
performance or usability. For one thing, it's not compiled
into the kernel; it's a user-level interface only. I prefer to
call it a storage system rather than a file system to avoid
this confusion. On the other hand, it operates at scales that
are mind-boggling - petabytes and up. And since it's
distributed over many machines, the aggregate bandwidth
for real computation when used with a tool like MapReduce
(see http://labs.google.com/papers/mapreduce.html) can
be tens of gigabytes per second. These are significant
factors to consider at LSST, and at least the ideas from
GFS should be kept in mind.
I say this because some of the messages I saw read as
though the writer thought of GFS as some sort of ext2
replacement; it's nothing of the kind.
GFS is reliable but not available. Those are jargon terms that
are important to understand. Reliable means that your data
is safe and guaranteed to be protected in the face of failure.
Available means that you can always get at your data. These
are distinct goals and GFS sides with reliability, a decision
I personally think is right. (Solving both at the same time
is truly daunting.) In short, GFS safeguards your data but there
may be periods when your data is unavailable. (Power goes
down, networks disconnect, and other disasters happen
that are unavoidable, so 100% availability requires a huge
amount of infrastructure and duplication; GFS takes a
different stance.)
There's one quote I want to argue against:
You can always run such a
thing on top of "gfs" if you want to, but it's very hard to change a
system designed with "gfs" in mind into a non-gfs system.
This is just wrong. We routinely run applications with GFS
and without GFS depending on the run. We have a file wrap
layer that calls GFS code if the file name begins /gfs/ and otherwise
calls the OS. For the most part, GFS is just like a file system.
Because it's not in the kernel, you can't chdir to it and there
are other related issues. As a result, the reverse statement
is closer to true: to convert from a real FS to GFS you may need
to make some significant changes, but going the other way is
dead easy.
-rob
------------------------------
Message: 2
Date: Tue, 18 Jan 2005 14:04:40 -0800 (PST)
From: Steffen Luitz <[EMAIL PROTECTED]>
Subject: Re: [LSST-data] Google file system
To: "Rob 'Commander' Pike" <[EMAIL PROTECTED]>
Cc: [email protected]
Message-ID:
<[EMAIL PROTECTED]>
Content-Type: TEXT/PLAIN; charset=US-ASCII
On Tue, 18 Jan 2005, Rob 'Commander' Pike wrote:
[...]
There's one quote I want to argue against:
You can always run such a
thing on top of "gfs" if you want to, but it's very hard to change a
system designed with "gfs" in mind into a non-gfs system.
This is just wrong. We routinely run applications with GFS
and without GFS depending on the run. We have a file wrap
layer that calls GFS code if the file name begins /gfs/ and otherwise
calls the OS. For the most part, GFS is just like a file system.
Because it's not in the kernel, you can't chdir to it and there
are other related issues. As a result, the reverse statement
is closer to true: to convert from a real FS to GFS you may need
to make some significant changes, but going the other way is
dead easy.
-rob
Hi Rob,
When I read the paper I missed that GFS would be used as library call
intercept or with an explicit API (I understood that it would work
at the
VFS level). Much better and much more maintainable. That mostly
invalidates my concerns about kernel versions, etc.
However, I still maintain my statement about the overall system
design and
should probably clarify it: A system that is designed to require
"uniform"
global high-performance file system space could and probably would
need
major changes in order to run (with the same performance) on a compute
farm that does not provide such a file system. When you remove GFS you
don't have to deal with the GFS API anymore, but instead you have to
design (or "find") some alternative mechanism with adequate
performance to
get your data to your compute nodes.
For the actual implementation of an individual application I agree
that it
would probably be more work to code it to use GFS than to not use GFS.
Steffen
----------------------------------------------------------------------
---
- Steffen Luitz - e-mail:
[EMAIL PROTECTED] -
- SLAC Mail Stop 41 - 2575 Sand Hill Rd - Menlo Park, CA 94025 -
U.S.A. -
- phone +1 650 926 2822 - cell phone +1 650 796
2266 -
----------------------------------------------------------------------
---
------------------------------
Message: 3
Date: Tue, 18 Jan 2005 14:38:39 -0800
From: "Rob 'Commander' Pike" <[EMAIL PROTECTED]>
Subject: Re: [LSST-data] Google file system
To: Steffen Luitz <[EMAIL PROTECTED]>
Cc: [email protected]
Message-ID: <[EMAIL PROTECTED]>
Content-Type: text/plain; charset=US-ASCII; format=flowed
For the actual implementation of an individual application I agree
that it
would probably be more work to code it to use GFS than to not use
GFS.
I don't understand this statement. Agree with whom? Surely it's
easier to
use a storage system that does automatic replication, load balancing,
distribution, and so on than to implement these in the application.
Of course, it's mostly moot unless you can get the code. Google is
talking
about an open source initiative (gradually, and GFS is unlikely to be
part of
it soon), so maybe one day...
-rob
------------------------------
Message: 4
Date: Tue, 18 Jan 2005 16:19:34 -0700
From: Jeffrey Kantor <[EMAIL PROTECTED]>
Subject: [LSST-data] Upcoming Meetings
To: <[email protected]>
Message-ID: <[EMAIL PROTECTED]>
Content-Type: text/plain; charset="US-ASCII"
Hello all,
It is past time to schedule our next face to face meeting. My current
thinking is to go for the first part of March, in Tucson.
Several agenda items for this F2F are:
PM Meeting - FY05 project plans and staffing progress (1/2 day post-
meeting)
Working Group Meetings
DOE Proposal
Pre-Cursor Data Server
Tour of LSST facility (optional)
So one full day for everyone, and a 1/2 day following for team
leads only
sounds right to me.
Other meetings that are occurring in that time frame are:
LSSTC Board Meeting 2/28/05 at LSSTC
SRD Baseline meeting, 3/18/05 at SLAC
Also, we are tentatively planning on hosting the next NVO workshop
in Tucson
in April.
Please reply with your availability on those dates and any other
suggested
agenda items.
Also, a reminder to the team leads, our next PM telecon is Friday,
Jan 21,
at 12:00pm EST.
Jeff
------------------------------
Message: 5
Date: Wed, 19 Jan 2005 11:15:07 -0700 (MST)
From: "Tim Axelrod" <[EMAIL PROTECTED]>
Subject: [LSST-data] Pipeline papers
To: "LSST Data list" <[email protected]>
Message-ID:
<[EMAIL PROTECTED]>
Content-Type: text/plain; charset="iso-8859-1"
Hi All,
Many of us are familiar with the SuperMacho mosaic image processing
pipeline, which is forming the basis of some our D&D work, the SDSS
pipelines, and tools such as mscred. I'd like to call your
attention to a
small set of papers on mosaic image processing pipelines that may
not be
so familiar:
Elixir: (attached)
EIS: http://www.eso.org/science/eis/publications/vandame.ps.gz
GaBoDS: http://arxiv.org/astro-ph/0501144
Each of these pipelines seeks to perform automated image processing of
data from wide field ccd mosaics, with functionality close to that
identified for the LSST Image Processing Pipeline. Each also
attempts to
be flexible, so that a variety of mosaics can be supported. They
differ
significantly in the algorithms they employ, and even more in the
way they
are implemented. For example, GaBoDS (who thought up that one?!)
strings
together a bunch of disparate software packages using shell scripts,
Elixir uses a home-grown work-flow package, and EIS implements a
uniform
environment that uses XML to define interfaces between pipeline
components.
I would like some discussion on the following topics, at least:
o How do the capabilities of these pipelines compare with what
we need
for LSST? Areas for comparison could include photometric and
astrometric accuracy; need for human intervention; configurability;
supportability of the software; speed.
o Does it make sense to try out one or more of these pipelines on
precursor data, as we are already planning to do with SuperMacho?
Actually using a pipeline is a good way to bring our requirements into
sharp focus.
o What other, similar, pipelines should we be looking at?
Thanks,
Tim.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Elixir.pdf
Type: application/pdf
Size: 464763 bytes
Desc: not available
Url : http://www.noao.edu/pipermail/lsst-data/attachments/
20050119/08c97a96/Elixir.pdf
------------------------------
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
End of LSST-data Digest, Vol 5, Issue 5
***************************************
