>>> Each NAS_000 ~ N storages have approximately 300,000 >>> files, the average size of file is few MB (not over GB). >>> The broker servers (with SQLite library) are on the >>> NAS > > It's not clear how many broker servers there are. > One per NAS?
80 ~ 100 servers. and generally one per nas, but it can be connected to any NAS. (so it's a number of combination) >>> and The front-end web servers (more than 200 servers) >>> communicate with living broker servers after request >>> location from location addressing system. > > , which is implemented in MySQL, right? Yes. >>> There are high frequency read/write/delete operations. > > Let's say a few MB is 50 MB, so 300,000 files on one NAS > would contain 5E7 * 3E5 = 15E12 = 15 TB > > There would have to be 20E6 / 3E5 = 67 NAS installations, > all connected to 200 webservers via broker servers. > > I'm afraid the chosen architecture isn't scalable, and code > tweaking in sqlite will not help much. > > Opening and closing one of 20,000,000 files for every > logical transaction is not suitable for such a scale. An > operation of that size should be able to construct a better > solution. > Exactly. may be not suitable. At this time, We have focused on the short-term efforts. If I solve them, I'll report my experiences to this community. All of your advices are really helpful to me. Thanks, Edward On Thu, Jan 8, 2009 at 6:04 AM, Kees Nuyt <k.n...@zonnet.nl> wrote: > On Wed, 7 Jan 2009 10:17:06 -0800, "Jim Dodgen" > <j...@dodgen.us> wrote in General Discussion of SQLite > Database <sqlite-users@sqlite.org>: > > >> I'm a little worried about how long it takes to open one >> of 20,000,000 files in a directory on the NAS? > > I agree. It would require a very cleverly contructed > directory tree, and very short (sub)dir names to reduce the > effort to locate a file. > > "Edward J. Yoon" wrote: > >>> Each NAS_000 ~ N storages have approximately 300,000 >>> files, the average size of file is few MB (not over GB). >>> The broker servers (with SQLite library) are on the >>> NAS > > It's not clear how many broker servers there are. > One per NAS? > >>> and The front-end web servers (more than 200 servers) >>> communicate with living broker servers after request >>> location from location addressing system. > > , which is implemented in MySQL, right? > >>> There are high frequency read/write/delete operations. > > Let's say a few MB is 50 MB, so 300,000 files on one NAS > would contain 5E7 * 3E5 = 15E12 = 15 TB > > There would have to be 20E6 / 3E5 = 67 NAS installations, > all connected to 200 webservers via broker servers. > > I'm afraid the chosen architecture isn't scalable, and code > tweaking in sqlite will not help much. > > Opening and closing one of 20,000,000 files for every > logical transaction is not suitable for such a scale. An > operation of that size should be able to construct a better > solution. > > Or we still don't understand what's really going on. > -- > ( Kees Nuyt > ) > c[_] > _______________________________________________ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > -- Best Regards, Edward J. Yoon @ NHN, corp. edwardy...@apache.org http://blog.udanax.org _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
