Wow, I didn't think that this stuff would come up again soon :).
The current implementation (a) tracks what is free and (b) what is
recently used, (c) lets the server choose the handle to return, and (d)
keeps a global handle space (a handle is unique across all servers).
the point of (a) was to avoid having to hit storage to find a free
handle. i agree that this is perhaps not that big a deal now that we
have a better handle on how to efficiently use berkeley db.
(a) and (c) together mean that clients never have to retry to get a
handle. i agree that in itself isn't all that valuable.
(b) ensures that clients caching metadata on some file don't end up
accessing some newly created file's data, or deleting some new file's
object, or some similar thing. this is an important part of allowing
clients to cache file metadata (specifically datafile handles) without
coordination.
(c) also allows us to precreate objects if we wanted to, although we
don't do that right now. this would be less important if/when
server-to-server communication is in place and we move file creation
over to the server side.
(d) eventually allows us to move objects around without updating the
file's metadata, assuming that we come up with a different mechanism for
determining where a file resides. A bloom filter sort of approach might
work, as an example. Or if server-to-server were working the servers
could just figure out where things are with some aggregate comm.
walt's idea seems to allow us to map a collection of objects (a
"segment") to a given server, then a client could pick values in that
segment. my feeling is that this hamstrings our ability to move objects
around, because we would then need to move entire segments around, as at
the very least it could take a very long time to reach a consistent
state again (think of many large objects needing to be moved; how do
clients know where to contact?). this idea is a generalization of pete's
idea to have a server id be part of the object handle; pete's approach
makes it impossible to migrate without changing file metadata. more on
this below.
pete's idea of speeding up creates by guessing at free handles, is ok,
but the right way to speed up creates is to precreate. then the latency
can be hidden in the mix of other operations. lustre already does this,
and i believe it is every effective for them.
he is correct that randomly picking values would lead towards nasty data
structures in the ledger. i'd be happy to see the "free" list part of
the leder disappear if that helps. i do think that the "recently freed"
list has to stay for the reason listed above, although it could be
implemented differently perhaps -- maybe just leave an entry in the DB
noting when the object was freed, and if it is referenced again after
the appropriate time we consider it up for grabs? this has a nice
side-effect of keeping the object "off limits" even if a server is
restarted.
i don't understand why it is difficult to get a value in a particular
range in the OSD work. can you clarify this pete? can't you just "guess"
a value in the range until you get one?
one thing that we could discuss is the relative merit of migration using
this sort of approach. maybe in fact this idea that i have that we want
to keep a FS-wide object handle space is flawed, that changing file
metadata can be addressed in a reasonable way that simplifies the
overall system, allows for migration, and doesn't have a negative impact
on our caching of metadata.
overall i think that changing how we reference objects, with the
exception of perhaps redoing how we keep up with free/recently-freed
objects, is something that should perhaps wait until we have
server-to-server working. we're likely to want to make some changes at
that point anyway, once the system has more control over the
construction of files and directories. maybe we can discuss how we'd
like things to work in that context and concentrate on getting there,
rather than torquing things now and then perhaps messing with things again?
thanks everyone! it's fun to get to sit and think about this stuff,
especially after many days of travel and meetings :).
regards,
rob
Walter B. Ligon III wrote:
I don't understand this. Is there a scheme whereby there is no mapping
of the handle ID to a server? If not, then what we are talking about, I
think, is whether the server mapping is fixed or not. The idea behind
the current scheme was to make the mapping of servers to handles
flexible. That said, the specific implementation might could be better.
For example, using 128 bits we could have a 64 bit segment tag and a 64
bit handle ID. The segment tag would map the handle to a server via the
tables, and the ID would be unique within that segment. This might
simplify some things without losing the flexibility we have.
As it is, the server can still randomly pick an ID, or a client could
randomly pick an ID, they just have to do it within a range, which isn't
particularly hard. With this suggested modification we could
"eliminate" the range by giving all "handle ranges" a built-in extent of
64 bits, which I think is the same as what you were suggesting.
If I'm not being clear, let me know and I'll try again. Or, if I don't
understand the problem, let me know that.
Walt
Pete Wyckoff wrote:
For create scalability, you may want the client to pick handle IDs
and offer those to the server, so that you can optimistically create
a metafile assuming there are no collisions on the server. These
guessed handle IDs can be random though. We did not implement this
as it would be quite expensive if implemented in terms of the
existing extent/extentlist/ledger data structures.
In the OSD work, we have to do painful things to return a handle ID
in a particular range. I would much rather have the server pick a
random ID and give it to the client. Or for the client to try to
pick a particular ID and hope there is no collision at the server.
So I'd like to discard the idea of pre-assigned per-server handle
ranges and augment our notion of PVFS_handle to include some sort of
"server identifier" as well as the 64-bit ID that is private to the
particular device on which the object sits.
Various distributed FS implementations for wide-area use seem to be
happy with 128-bit handles and assume collisions will never happen.
This always struck me as wildly reckless, but maybe it is time to
accept the fact that these number spaces are really big.
-- Pete
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
_______________________________________________
Pvfs2-developers mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
