On Feb 23, 2007, at 2:38 PM, Rob Ross wrote:
Wow, I didn't think that this stuff would come up again soon :).
The current implementation (a) tracks what is free and (b) what is
recently used, (c) lets the server choose the handle to return, and
(d) keeps a global handle space (a handle is unique across all
servers).
the point of (a) was to avoid having to hit storage to find a free
handle. i agree that this is perhaps not that big a deal now that
we have a better handle on how to efficiently use berkeley db.
(a) and (c) together mean that clients never have to retry to get a
handle. i agree that in itself isn't all that valuable.
(b) ensures that clients caching metadata on some file don't end up
accessing some newly created file's data, or deleting some new
file's object, or some similar thing. this is an important part of
allowing clients to cache file metadata (specifically datafile
handles) without coordination.
(c) also allows us to precreate objects if we wanted to, although
we don't do that right now. this would be less important if/when
server-to-server communication is in place and we move file
creation over to the server side.
(d) eventually allows us to move objects around without updating
the file's metadata, assuming that we come up with a different
mechanism for determining where a file resides. A bloom filter sort
of approach might work, as an example. Or if server-to-server were
working the servers could just figure out where things are with
some aggregate comm.
walt's idea seems to allow us to map a collection of objects (a
"segment") to a given server, then a client could pick values in
that segment. my feeling is that this hamstrings our ability to
move objects around, because we would then need to move entire
segments around, as at the very least it could take a very long
time to reach a consistent state again (think of many large objects
needing to be moved; how do clients know where to contact?). this
idea is a generalization of pete's idea to have a server id be part
of the object handle; pete's approach makes it impossible to
migrate without changing file metadata. more on this below.
pete's idea of speeding up creates by guessing at free handles, is
ok, but the right way to speed up creates is to precreate. then the
latency can be hidden in the mix of other operations. lustre
already does this, and i believe it is every effective for them.
Can I try to clarify what precreate means for PVFS? There are
different pieces that can might benefit separately from pre-creation.
1. create metadata handle. requires:
a. client message to md server
b. new dspace db entry
c. new keyval db entry
2. create datafile handles. requires:
a. client message to each IO server
b. new db entry in IO server
3. setattr of datafile handles array to metadata
a. client message to md server
b. modify keyval db entry
4. crdirent to 2nd metadata server (potentially the same)
a. client message to md server 2
b. create keyval db entry
5. create bstream file (this happens on first write)
With PVFS right now (no server-to-server, etc) it should be easy to
get rid of 2a, and move 2b to the first write (with the bstream
create). All we need to do is partition the datafile handles of each
IO server to the metadata servers in the fs.conf (a sort of pre-
create). This allows the md server to pick datafile handles and do
1,2, and 3 all in one message.
If its still necessary to maintain ledgers for the datafile handles,
they can be kept on the md servers (one for each IO server), and at
initialization, we can pull the handles out from the keyval entries
to populate the ledger.
Usually when I hear pre-create I think of trying to get rid of all
those client->IOserver messages. But there's still the cost of
adding the db entries and bstream files. My guess would be that this
cost is negligible compared to the latency of sending all those
messages, but maybe on fast networks its not. We could certainly
populate the db with a bunch of entries, and keep a ledger of unused
handles, but that requires making the ledger persistent. I don't
think keeping placeholder db entries that have to be filled-in for
setattr and crdirent will be much faster than creating new ones.
There's also still the cost of the crdirent, which is a keyval db
entry create. That can't really be precreated, and its got to be
synchronous with the other creation bits.
As a first pass, partitioning the datafile handles for each IO server
up amongst the metadata servers seems like an obvious improvement,
and then when server-to-server is in place, making that partitioning
process more dynamic would be an easy add-on. My gut feeling is that
placeholder entries in the db would be significant effort to code and
isn't worth the benefit.
he is correct that randomly picking values would lead towards nasty
data structures in the ledger.
I don't see the point of keeping the ledger around (in its range
style data structure form) if we're picking values randomly.
i'd be happy to see the "free" list part of the leder disappear if
that helps. i do think that the "recently freed" list has to stay
for the reason listed above, although it could be implemented
differently perhaps -- maybe just leave an entry in the DB noting
when the object was freed, and if it is referenced again after the
appropriate time we consider it up for grabs? this has a nice side-
effect of keeping the object "off limits" even if a server is
restarted.
That sounds nice!
-sam
i don't understand why it is difficult to get a value in a
particular range in the OSD work. can you clarify this pete? can't
you just "guess" a value in the range until you get one?
one thing that we could discuss is the relative merit of migration
using this sort of approach. maybe in fact this idea that i have
that we want to keep a FS-wide object handle space is flawed, that
changing file metadata can be addressed in a reasonable way that
simplifies the overall system, allows for migration, and doesn't
have a negative impact on our caching of metadata.
overall i think that changing how we reference objects, with the
exception of perhaps redoing how we keep up with free/recently-
freed objects, is something that should perhaps wait until we have
server-to-server working. we're likely to want to make some changes
at that point anyway, once the system has more control over the
construction of files and directories. maybe we can discuss how
we'd like things to work in that context and concentrate on getting
there, rather than torquing things now and then perhaps messing
with things again?
thanks everyone! it's fun to get to sit and think about this stuff,
especially after many days of travel and meetings :).
regards,
rob
Walter B. Ligon III wrote:
I don't understand this. Is there a scheme whereby there is no
mapping of the handle ID to a server? If not, then what we are
talking about, I think, is whether the server mapping is fixed or
not. The idea behind the current scheme was to make the mapping
of servers to handles flexible. That said, the specific
implementation might could be better. For example, using 128 bits
we could have a 64 bit segment tag and a 64 bit handle ID. The
segment tag would map the handle to a server via the tables, and
the ID would be unique within that segment. This might simplify
some things without losing the flexibility we have.
As it is, the server can still randomly pick an ID, or a client
could randomly pick an ID, they just have to do it within a range,
which isn't particularly hard. With this suggested modification
we could "eliminate" the range by giving all "handle ranges" a
built-in extent of 64 bits, which I think is the same as what you
were suggesting.
If I'm not being clear, let me know and I'll try again. Or, if I
don't understand the problem, let me know that.
Walt
Pete Wyckoff wrote:
For create scalability, you may want the client to pick handle IDs
and offer those to the server, so that you can optimistically create
a metafile assuming there are no collisions on the server. These
guessed handle IDs can be random though. We did not implement this
as it would be quite expensive if implemented in terms of the
existing extent/extentlist/ledger data structures.
In the OSD work, we have to do painful things to return a handle ID
in a particular range. I would much rather have the server pick a
random ID and give it to the client. Or for the client to try to
pick a particular ID and hope there is no collision at the server.
So I'd like to discard the idea of pre-assigned per-server handle
ranges and augment our notion of PVFS_handle to include some sort of
"server identifier" as well as the 64-bit ID that is private to the
particular device on which the object sits.
Various distributed FS implementations for wide-area use seem to be
happy with 128-bit handles and assume collisions will never happen.
This always struck me as wildly reckless, but maybe it is time to
accept the fact that these number spaces are really big.
-- Pete
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
