100 is kind of low. That will mean 10,000 directories for 1 million
loaded users. I don't like it when we see 10,000 entries on ls. Gets
unwieldy again.
I'd limit to 1,000. Seems reasonable for directories to handle. But the
limit here is really a million users with two level directories. If we
increase the stored:concurrent user ratio to 100:1, we'll be stuck at
10,000 concurrent users. I think Olio has the potential to scale much
higher.
Coming down, I really think we should do a 3 layer directory (or 2 layer
subdirectory). With a limit of 1000 files/users per directory you can
actually go up to 10^9 stored users and 10^7 concurrent users - and even
if we at some point decide the user ratio to be 1000:1, we are still up
to 10^6 concurrent users. That's probably as big a rig as anybody will
want to test.
In summary, I'd go for 2 layer subdirectory limiting each directory to
have no more than 1000 entries.
-Akara
Shanti Subramanyam wrote:
Currently, the filestore is a single flat layer with all files both
pre-loaded and those that are generated during a run going into the same
directory. As the number of users are scaled, this directory is unwieldy
- I'm sure some of you may have tried to do a 'ls /filestore' and then
cursed yourself for doing it !
How about using one directory for every 100 users or so ? Something
similar for events ? Not sure how we'll deal with the added events and
files during a run though.
Thoughts ?
Shanti
