Re: [PHP] Re: [BULK] Re: [PHP] OK to have many files in one folder?
On Mon, 2007-06-18 at 08:41 -0700, Jim Lucas wrote: > Daniel Brown wrote: > > On 6/18/07, Tijnema <[EMAIL PROTECTED]> wrote: > >> On 6/18/07, Daniel Brown <[EMAIL PROTECTED]> wrote: > >> > On 6/18/07, Tijnema <[EMAIL PROTECTED]> wrote: > >> > > On 6/18/07, Daniel Brown <[EMAIL PROTECTED]> wrote: > >> > > > On 6/18/07, Colin Guthrie <[EMAIL PROTECTED]> wrote: > >> > > > > clive wrote: > >> > > > > >> > >> > > > > >> I have no clue how big the files are, but you might want to > >> store them > >> > > > > >> in a database. That can speed up things, but don't ask me > >> how much ;) > >> > > > > >> > >> > > > > >> Tijnema > >> > > > > >> > >> > > > > > no dude, while database are convenient, files systems are > >> faster, I mean > >> > > > > > thats what they were designed for, serving files. > >> > > > > > > >> > > > > > For lots of files I would store them in directories and sub > >> directories. > >> > > > > > >> > > > > Yeah amen! > >> > > > > > >> > > > > I generally use a two character hex hash in my main folder to > >> give 255 > >> > > > > sub folders each containing files. > >> > > > > > >> > > > > In my case the hash is calculated from some known info - e.g. > >> a database > >> > > > > ID and then stored in a meta data table. > >> > > > > > >> > > > > Col > >> > > > > > >> > > > > -- > >> > > > > PHP General Mailing List (http://www.php.net/) > >> > > > > To unsubscribe, visit: http://www.php.net/unsub.php > >> > > > > > >> > > > > > >> > > > > >> > > >Tij, > >> > > > > >> > > >Referring to one of my earlier posts in this thread, as a > >> > > > refresher, database information is stored in files. So to store > >> files > >> > > > in a database means the following has to take place: > >> > > > > >> > > >1.) Data is uploaded to the server. > >> > > >2.) Data is processed by the database server. > >> > > >3.) Data is compressed and encrypted. > >> > > >4.) Data is written to disk. > >> > > >(4b. - optional) Data file is checked for integrity and > >> > > > correctness of write. > >> > > > > >> > > >Then, to serve the content, a similar reverse occurs: > >> > > > > >> > > >1.) Request is sent to the server via a script. > >> > > >2.) The script interfaces with the database to locate the > >> > > > desired data row. > >> > > >3.) Once located, the data is pulled in raw form from the > >> database file. > >> > > >4.) The data is then decompressed and decrypted. > >> > > >5.) The data is streamed back to the script. > >> > > >6.) The script decides how to handle it, based upon > >> hard-coded options. > >> > > > > >> > > >For a filesystem, it's simpler. Uploading: > >> > > > > >> > > >1.) Data is uploaded. > >> > > >2.) Data is written to disk. > >> > > > > >> > > >Serving: > >> > > > > >> > > >1.) Request is sent via script, FTP, socket connection, or > >> > > > local request. > >> > > >2.) File is served through via the method requested. > >> > > > > >> > > >So there are several pros and cons for each, but just to list > >> > > > three of each with regard to database storage: > >> > > > > >> > > >Pros: > >> > > >Can be stored centralized and portable, without needing > >> to archive. > >> > > >Automatically encrypted and compressed upon storage. > >> > > >Easier and faster to search. > >> > > >Cons: > >> > > >Slower storage and retrieval > >> > > >Much greater risk of corruption > >> > > >Not always cross-compatible across platforms, versions, > >> > > > installations, etc. > >> > > > > >> > > > -- > >> > > > Daniel P. Brown > >> > > > >> > > How sure are you about all this? > >> > > Benchmarks? > >> > > > >> > > What about search time for the harddrives etc? It needs to scan the > >> > > inode table to find the file, etc.., then it needs to locate that > >> file > >> > > on the harddrive, while the second file might be on the other side of > >> > > the disk > >> > > > >> > > Storing all files in one single filesystem means you have only 1 > >> inode > >> > > :) All data is stored at same place (atleast, it should be) > >> > > > >> > > Tijnema > >> > > > >> > > >> >Do I have benchmarks? No. Do I have the time to do them now? No. > >> > > >> >Do I feel confident enough that if you were to run those benchmark > >> > tests that it would prove me right? Damn straight. ;-P > >> > > >> > -- > >> > Daniel P. Brown > >> > >> I think I have some time to write a simple script in a few hours, I'd > >> like to see what the performance difference is :) > >> > >> Tijnema > >> > > > >Well, keep in mind that I don't expect that you'd see much > > difference at all with a single file, but try doing multiple files of > > ASCII and binary, various sizes, and also several simultaneous > > connections. > > > >Two other things I forgot to mention with database storage are a > > pro an
Re: [PHP] Re: [BULK] Re: [PHP] OK to have many files in one folder?
Daniel Brown wrote: On 6/18/07, Tijnema <[EMAIL PROTECTED]> wrote: On 6/18/07, Daniel Brown <[EMAIL PROTECTED]> wrote: > On 6/18/07, Tijnema <[EMAIL PROTECTED]> wrote: > > On 6/18/07, Daniel Brown <[EMAIL PROTECTED]> wrote: > > > On 6/18/07, Colin Guthrie <[EMAIL PROTECTED]> wrote: > > > > clive wrote: > > > > >> > > > > >> I have no clue how big the files are, but you might want to store them > > > > >> in a database. That can speed up things, but don't ask me how much ;) > > > > >> > > > > >> Tijnema > > > > >> > > > > > no dude, while database are convenient, files systems are faster, I mean > > > > > thats what they were designed for, serving files. > > > > > > > > > > For lots of files I would store them in directories and sub directories. > > > > > > > > Yeah amen! > > > > > > > > I generally use a two character hex hash in my main folder to give 255 > > > > sub folders each containing files. > > > > > > > > In my case the hash is calculated from some known info - e.g. a database > > > > ID and then stored in a meta data table. > > > > > > > > Col > > > > > > > > -- > > > > PHP General Mailing List (http://www.php.net/) > > > > To unsubscribe, visit: http://www.php.net/unsub.php > > > > > > > > > > > > > >Tij, > > > > > >Referring to one of my earlier posts in this thread, as a > > > refresher, database information is stored in files. So to store files > > > in a database means the following has to take place: > > > > > >1.) Data is uploaded to the server. > > >2.) Data is processed by the database server. > > >3.) Data is compressed and encrypted. > > >4.) Data is written to disk. > > >(4b. - optional) Data file is checked for integrity and > > > correctness of write. > > > > > >Then, to serve the content, a similar reverse occurs: > > > > > >1.) Request is sent to the server via a script. > > >2.) The script interfaces with the database to locate the > > > desired data row. > > >3.) Once located, the data is pulled in raw form from the database file. > > >4.) The data is then decompressed and decrypted. > > >5.) The data is streamed back to the script. > > >6.) The script decides how to handle it, based upon hard-coded options. > > > > > >For a filesystem, it's simpler. Uploading: > > > > > >1.) Data is uploaded. > > >2.) Data is written to disk. > > > > > >Serving: > > > > > >1.) Request is sent via script, FTP, socket connection, or > > > local request. > > >2.) File is served through via the method requested. > > > > > >So there are several pros and cons for each, but just to list > > > three of each with regard to database storage: > > > > > >Pros: > > >Can be stored centralized and portable, without needing to archive. > > >Automatically encrypted and compressed upon storage. > > >Easier and faster to search. > > >Cons: > > >Slower storage and retrieval > > >Much greater risk of corruption > > >Not always cross-compatible across platforms, versions, > > > installations, etc. > > > > > > -- > > > Daniel P. Brown > > > > How sure are you about all this? > > Benchmarks? > > > > What about search time for the harddrives etc? It needs to scan the > > inode table to find the file, etc.., then it needs to locate that file > > on the harddrive, while the second file might be on the other side of > > the disk > > > > Storing all files in one single filesystem means you have only 1 inode > > :) All data is stored at same place (atleast, it should be) > > > > Tijnema > > > >Do I have benchmarks? No. Do I have the time to do them now? No. > >Do I feel confident enough that if you were to run those benchmark > tests that it would prove me right? Damn straight. ;-P > > -- > Daniel P. Brown I think I have some time to write a simple script in a few hours, I'd like to see what the performance difference is :) Tijnema Well, keep in mind that I don't expect that you'd see much difference at all with a single file, but try doing multiple files of ASCII and binary, various sizes, and also several simultaneous connections. Two other things I forgot to mention with database storage are a pro and con, respectively - caching and accessibility. and the fact that a lot of hosting environments have a DB on a different machine. This would make for a much worse performance then the local filesystem. -- Jim Lucas "Some men are born to greatness, some achieve greatness, and some have greatness thrust upon them." Twelfth Night, Act II, Scene V by William Shakespeare -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Re: [BULK] Re: [PHP] OK to have many files in one folder?
On Mon, 2007-06-18 at 11:25 -0400, Daniel Brown wrote: > On 6/18/07, Tijnema <[EMAIL PROTECTED]> wrote: > > On 6/18/07, Daniel Brown <[EMAIL PROTECTED]> wrote: > > > > >Referring to one of my earlier posts in this thread, as a > > > > > refresher, database information is stored in files. So to store files > > > > > in a database means the following has to take place: > > > > > > > > > >1.) Data is uploaded to the server. > > > > >2.) Data is processed by the database server. > > > > >3.) Data is compressed and encrypted. > > > > >4.) Data is written to disk. > > > > >(4b. - optional) Data file is checked for integrity and > > > > > correctness of write. > > > > > > > > > >Then, to serve the content, a similar reverse occurs: > > > > > > > > > >1.) Request is sent to the server via a script. > > > > >2.) The script interfaces with the database to locate the > > > > > desired data row. > > > > >3.) Once located, the data is pulled in raw form from the > > > > > database file. > > > > >4.) The data is then decompressed and decrypted. > > > > >5.) The data is streamed back to the script. > > > > >6.) The script decides how to handle it, based upon hard-coded > > > > > options. > > > > > > > > > >For a filesystem, it's simpler. Uploading: > > > > > > > > > >1.) Data is uploaded. > > > > >2.) Data is written to disk. This isn't quite correct. There are numerous databases that allow you to assign a raw partition and they manage the filesystem themselves thus the intermediate OS dependence is eliminated. As such, the database probably wins in this situation since you also get rapid searching on any meta data fields and since you probably have to issue a query anyways (for either approach), this approach is superior since you no longer need to interact with the filesystem at all. The only negative is that writing to the database will be less efficient due to escaping of the data. But reading will be much faster. And in practice, I'd wager, it's rare that images are created more often than read. Cheers, Rob. -- .. | InterJinn Application Framework - http://www.interjinn.com | :: | An application and templating framework for PHP. Boasting | | a powerful, scalable system for accessing system services | | such as forms, properties, sessions, and caches. InterJinn | | also provides an extremely flexible architecture for | | creating re-usable components quickly and easily. | `' -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Re: [BULK] Re: [PHP] OK to have many files in one folder?
On 6/18/07, Tijnema <[EMAIL PROTECTED]> wrote: On 6/18/07, Daniel Brown <[EMAIL PROTECTED]> wrote: > On 6/18/07, Tijnema <[EMAIL PROTECTED]> wrote: > > On 6/18/07, Daniel Brown <[EMAIL PROTECTED]> wrote: > > > On 6/18/07, Colin Guthrie <[EMAIL PROTECTED]> wrote: > > > > clive wrote: > > > > >> > > > > >> I have no clue how big the files are, but you might want to store them > > > > >> in a database. That can speed up things, but don't ask me how much ;) > > > > >> > > > > >> Tijnema > > > > >> > > > > > no dude, while database are convenient, files systems are faster, I mean > > > > > thats what they were designed for, serving files. > > > > > > > > > > For lots of files I would store them in directories and sub directories. > > > > > > > > Yeah amen! > > > > > > > > I generally use a two character hex hash in my main folder to give 255 > > > > sub folders each containing files. > > > > > > > > In my case the hash is calculated from some known info - e.g. a database > > > > ID and then stored in a meta data table. > > > > > > > > Col > > > > > > > > -- > > > > PHP General Mailing List (http://www.php.net/) > > > > To unsubscribe, visit: http://www.php.net/unsub.php > > > > > > > > > > > > > >Tij, > > > > > >Referring to one of my earlier posts in this thread, as a > > > refresher, database information is stored in files. So to store files > > > in a database means the following has to take place: > > > > > >1.) Data is uploaded to the server. > > >2.) Data is processed by the database server. > > >3.) Data is compressed and encrypted. > > >4.) Data is written to disk. > > >(4b. - optional) Data file is checked for integrity and > > > correctness of write. > > > > > >Then, to serve the content, a similar reverse occurs: > > > > > >1.) Request is sent to the server via a script. > > >2.) The script interfaces with the database to locate the > > > desired data row. > > >3.) Once located, the data is pulled in raw form from the database file. > > >4.) The data is then decompressed and decrypted. > > >5.) The data is streamed back to the script. > > >6.) The script decides how to handle it, based upon hard-coded options. > > > > > >For a filesystem, it's simpler. Uploading: > > > > > >1.) Data is uploaded. > > >2.) Data is written to disk. > > > > > >Serving: > > > > > >1.) Request is sent via script, FTP, socket connection, or > > > local request. > > >2.) File is served through via the method requested. > > > > > >So there are several pros and cons for each, but just to list > > > three of each with regard to database storage: > > > > > >Pros: > > >Can be stored centralized and portable, without needing to archive. > > >Automatically encrypted and compressed upon storage. > > >Easier and faster to search. > > >Cons: > > >Slower storage and retrieval > > >Much greater risk of corruption > > >Not always cross-compatible across platforms, versions, > > > installations, etc. > > > > > > -- > > > Daniel P. Brown > > > > How sure are you about all this? > > Benchmarks? > > > > What about search time for the harddrives etc? It needs to scan the > > inode table to find the file, etc.., then it needs to locate that file > > on the harddrive, while the second file might be on the other side of > > the disk > > > > Storing all files in one single filesystem means you have only 1 inode > > :) All data is stored at same place (atleast, it should be) > > > > Tijnema > > > >Do I have benchmarks? No. Do I have the time to do them now? No. > >Do I feel confident enough that if you were to run those benchmark > tests that it would prove me right? Damn straight. ;-P > > -- > Daniel P. Brown I think I have some time to write a simple script in a few hours, I'd like to see what the performance difference is :) Tijnema Well, keep in mind that I don't expect that you'd see much difference at all with a single file, but try doing multiple files of ASCII and binary, various sizes, and also several simultaneous connections. Two other things I forgot to mention with database storage are a pro and con, respectively - caching and accessibility. -- Daniel P. Brown [office] (570-) 587-7080 Ext. 272 [mobile] (570-) 766-8107 -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Re: [BULK] Re: [PHP] OK to have many files in one folder?
On 6/18/07, Daniel Brown <[EMAIL PROTECTED]> wrote: On 6/18/07, Tijnema <[EMAIL PROTECTED]> wrote: > On 6/18/07, Daniel Brown <[EMAIL PROTECTED]> wrote: > > On 6/18/07, Colin Guthrie <[EMAIL PROTECTED]> wrote: > > > clive wrote: > > > >> > > > >> I have no clue how big the files are, but you might want to store them > > > >> in a database. That can speed up things, but don't ask me how much ;) > > > >> > > > >> Tijnema > > > >> > > > > no dude, while database are convenient, files systems are faster, I mean > > > > thats what they were designed for, serving files. > > > > > > > > For lots of files I would store them in directories and sub directories. > > > > > > Yeah amen! > > > > > > I generally use a two character hex hash in my main folder to give 255 > > > sub folders each containing files. > > > > > > In my case the hash is calculated from some known info - e.g. a database > > > ID and then stored in a meta data table. > > > > > > Col > > > > > > -- > > > PHP General Mailing List (http://www.php.net/) > > > To unsubscribe, visit: http://www.php.net/unsub.php > > > > > > > > > >Tij, > > > >Referring to one of my earlier posts in this thread, as a > > refresher, database information is stored in files. So to store files > > in a database means the following has to take place: > > > >1.) Data is uploaded to the server. > >2.) Data is processed by the database server. > >3.) Data is compressed and encrypted. > >4.) Data is written to disk. > >(4b. - optional) Data file is checked for integrity and > > correctness of write. > > > >Then, to serve the content, a similar reverse occurs: > > > >1.) Request is sent to the server via a script. > >2.) The script interfaces with the database to locate the > > desired data row. > >3.) Once located, the data is pulled in raw form from the database file. > >4.) The data is then decompressed and decrypted. > >5.) The data is streamed back to the script. > >6.) The script decides how to handle it, based upon hard-coded options. > > > >For a filesystem, it's simpler. Uploading: > > > >1.) Data is uploaded. > >2.) Data is written to disk. > > > >Serving: > > > >1.) Request is sent via script, FTP, socket connection, or > > local request. > >2.) File is served through via the method requested. > > > >So there are several pros and cons for each, but just to list > > three of each with regard to database storage: > > > >Pros: > >Can be stored centralized and portable, without needing to archive. > >Automatically encrypted and compressed upon storage. > >Easier and faster to search. > >Cons: > >Slower storage and retrieval > >Much greater risk of corruption > >Not always cross-compatible across platforms, versions, > > installations, etc. > > > > -- > > Daniel P. Brown > > How sure are you about all this? > Benchmarks? > > What about search time for the harddrives etc? It needs to scan the > inode table to find the file, etc.., then it needs to locate that file > on the harddrive, while the second file might be on the other side of > the disk > > Storing all files in one single filesystem means you have only 1 inode > :) All data is stored at same place (atleast, it should be) > > Tijnema > Do I have benchmarks? No. Do I have the time to do them now? No. Do I feel confident enough that if you were to run those benchmark tests that it would prove me right? Damn straight. ;-P -- Daniel P. Brown I think I have some time to write a simple script in a few hours, I'd like to see what the performance difference is :) Tijnema -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Re: [BULK] Re: [PHP] OK to have many files in one folder?
On 6/18/07, Tijnema <[EMAIL PROTECTED]> wrote: On 6/18/07, Daniel Brown <[EMAIL PROTECTED]> wrote: > On 6/18/07, Colin Guthrie <[EMAIL PROTECTED]> wrote: > > clive wrote: > > >> > > >> I have no clue how big the files are, but you might want to store them > > >> in a database. That can speed up things, but don't ask me how much ;) > > >> > > >> Tijnema > > >> > > > no dude, while database are convenient, files systems are faster, I mean > > > thats what they were designed for, serving files. > > > > > > For lots of files I would store them in directories and sub directories. > > > > Yeah amen! > > > > I generally use a two character hex hash in my main folder to give 255 > > sub folders each containing files. > > > > In my case the hash is calculated from some known info - e.g. a database > > ID and then stored in a meta data table. > > > > Col > > > > -- > > PHP General Mailing List (http://www.php.net/) > > To unsubscribe, visit: http://www.php.net/unsub.php > > > > > >Tij, > >Referring to one of my earlier posts in this thread, as a > refresher, database information is stored in files. So to store files > in a database means the following has to take place: > >1.) Data is uploaded to the server. >2.) Data is processed by the database server. >3.) Data is compressed and encrypted. >4.) Data is written to disk. >(4b. - optional) Data file is checked for integrity and > correctness of write. > >Then, to serve the content, a similar reverse occurs: > >1.) Request is sent to the server via a script. >2.) The script interfaces with the database to locate the > desired data row. >3.) Once located, the data is pulled in raw form from the database file. >4.) The data is then decompressed and decrypted. >5.) The data is streamed back to the script. >6.) The script decides how to handle it, based upon hard-coded options. > >For a filesystem, it's simpler. Uploading: > >1.) Data is uploaded. >2.) Data is written to disk. > >Serving: > >1.) Request is sent via script, FTP, socket connection, or > local request. >2.) File is served through via the method requested. > >So there are several pros and cons for each, but just to list > three of each with regard to database storage: > >Pros: >Can be stored centralized and portable, without needing to archive. >Automatically encrypted and compressed upon storage. >Easier and faster to search. >Cons: >Slower storage and retrieval >Much greater risk of corruption >Not always cross-compatible across platforms, versions, > installations, etc. > > -- > Daniel P. Brown How sure are you about all this? Benchmarks? What about search time for the harddrives etc? It needs to scan the inode table to find the file, etc.., then it needs to locate that file on the harddrive, while the second file might be on the other side of the disk Storing all files in one single filesystem means you have only 1 inode :) All data is stored at same place (atleast, it should be) Tijnema Do I have benchmarks? No. Do I have the time to do them now? No. Do I feel confident enough that if you were to run those benchmark tests that it would prove me right? Damn straight. ;-P -- Daniel P. Brown [office] (570-) 587-7080 Ext. 272 [mobile] (570-) 766-8107 -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Re: [BULK] Re: [PHP] OK to have many files in one folder?
On 6/18/07, Daniel Brown <[EMAIL PROTECTED]> wrote: On 6/18/07, Colin Guthrie <[EMAIL PROTECTED]> wrote: > clive wrote: > >> > >> I have no clue how big the files are, but you might want to store them > >> in a database. That can speed up things, but don't ask me how much ;) > >> > >> Tijnema > >> > > no dude, while database are convenient, files systems are faster, I mean > > thats what they were designed for, serving files. > > > > For lots of files I would store them in directories and sub directories. > > Yeah amen! > > I generally use a two character hex hash in my main folder to give 255 > sub folders each containing files. > > In my case the hash is calculated from some known info - e.g. a database > ID and then stored in a meta data table. > > Col > > -- > PHP General Mailing List (http://www.php.net/) > To unsubscribe, visit: http://www.php.net/unsub.php > > Tij, Referring to one of my earlier posts in this thread, as a refresher, database information is stored in files. So to store files in a database means the following has to take place: 1.) Data is uploaded to the server. 2.) Data is processed by the database server. 3.) Data is compressed and encrypted. 4.) Data is written to disk. (4b. - optional) Data file is checked for integrity and correctness of write. Then, to serve the content, a similar reverse occurs: 1.) Request is sent to the server via a script. 2.) The script interfaces with the database to locate the desired data row. 3.) Once located, the data is pulled in raw form from the database file. 4.) The data is then decompressed and decrypted. 5.) The data is streamed back to the script. 6.) The script decides how to handle it, based upon hard-coded options. For a filesystem, it's simpler. Uploading: 1.) Data is uploaded. 2.) Data is written to disk. Serving: 1.) Request is sent via script, FTP, socket connection, or local request. 2.) File is served through via the method requested. So there are several pros and cons for each, but just to list three of each with regard to database storage: Pros: Can be stored centralized and portable, without needing to archive. Automatically encrypted and compressed upon storage. Easier and faster to search. Cons: Slower storage and retrieval Much greater risk of corruption Not always cross-compatible across platforms, versions, installations, etc. -- Daniel P. Brown How sure are you about all this? Benchmarks? What about search time for the harddrives etc? It needs to scan the inode table to find the file, etc.., then it needs to locate that file on the harddrive, while the second file might be on the other side of the disk Storing all files in one single filesystem means you have only 1 inode :) All data is stored at same place (atleast, it should be) Tijnema -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Re: [BULK] Re: [PHP] OK to have many files in one folder?
On 6/18/07, Colin Guthrie <[EMAIL PROTECTED]> wrote: clive wrote: >> >> I have no clue how big the files are, but you might want to store them >> in a database. That can speed up things, but don't ask me how much ;) >> >> Tijnema >> > no dude, while database are convenient, files systems are faster, I mean > thats what they were designed for, serving files. > > For lots of files I would store them in directories and sub directories. Yeah amen! I generally use a two character hex hash in my main folder to give 255 sub folders each containing files. In my case the hash is calculated from some known info - e.g. a database ID and then stored in a meta data table. Col -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php Tij, Referring to one of my earlier posts in this thread, as a refresher, database information is stored in files. So to store files in a database means the following has to take place: 1.) Data is uploaded to the server. 2.) Data is processed by the database server. 3.) Data is compressed and encrypted. 4.) Data is written to disk. (4b. - optional) Data file is checked for integrity and correctness of write. Then, to serve the content, a similar reverse occurs: 1.) Request is sent to the server via a script. 2.) The script interfaces with the database to locate the desired data row. 3.) Once located, the data is pulled in raw form from the database file. 4.) The data is then decompressed and decrypted. 5.) The data is streamed back to the script. 6.) The script decides how to handle it, based upon hard-coded options. For a filesystem, it's simpler. Uploading: 1.) Data is uploaded. 2.) Data is written to disk. Serving: 1.) Request is sent via script, FTP, socket connection, or local request. 2.) File is served through via the method requested. So there are several pros and cons for each, but just to list three of each with regard to database storage: Pros: Can be stored centralized and portable, without needing to archive. Automatically encrypted and compressed upon storage. Easier and faster to search. Cons: Slower storage and retrieval Much greater risk of corruption Not always cross-compatible across platforms, versions, installations, etc. -- Daniel P. Brown [office] (570-) 587-7080 Ext. 272 [mobile] (570-) 766-8107 -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] Re: [BULK] Re: [PHP] OK to have many files in one folder?
clive wrote: >> >> I have no clue how big the files are, but you might want to store them >> in a database. That can speed up things, but don't ask me how much ;) >> >> Tijnema >> > no dude, while database are convenient, files systems are faster, I mean > thats what they were designed for, serving files. > > For lots of files I would store them in directories and sub directories. Yeah amen! I generally use a two character hex hash in my main folder to give 255 sub folders each containing files. In my case the hash is calculated from some known info - e.g. a database ID and then stored in a meta data table. Col -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] Re: [BULK] Re: [PHP] OK to have many files in one folder?
I have no clue how big the files are, but you might want to store them
in a database. That can speed up things, but don't ask me how much ;)
Tijnema
no dude, while database are convenient, files systems are faster, I mean
thats what they were designed for, serving files.
For lots of files I would store them in directories and sub directories.
-
--
Regards,
Clive.
Real Time Travel Connections
{No electrons were harmed in the creation, transmission or reading of
this email. However, many were excited and some may well have enjoyed
the experience.}
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] OK to have many files in one folder?
On 6/17/07, Brian Dunning <[EMAIL PROTECTED]> wrote: For everyone who advised me to "beware the inode", allow me to forward what the Rackspace admins told me. This is Greek to me, and I'm hoping one of you can translate. All I understood was where he said "I appear to have more than enough." Yes? ---snip--- All of your slices on the disk are ext3. As this is the only file system that the Red Hat kernel provides support for. [EMAIL PROTECTED] ~]# df -hT Filesystem Type Size Used Avail Use% Mounted on /dev/hda5 ext3 227G 93G 123G 43% / /dev/hda1 ext3 99M 12M 83M 12% /boot none tmpfs 1.9G 0 1.9G 0% /dev/shm /dev/hda2 ext3 2.0G 36M 1.9G 2% /tmp As far as I am able to recall, there are only 2 limitations that you must be concerned with. There might be others, but these are the only ones I am aware of, which also appears to be true from the research I have done. 1. The number if i-nodes. You appear to have more than enough. [EMAIL PROTECTED] ~]# df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/hda5 30130176 3729871 26400305 13% / /dev/hda1 26104 46 26058 1% /boot none 220613 1 220612 1% /dev/shm /dev/hda2 262144 18 262126 1% /tmp 2. The subdirectory limitation for the ext3 file system. You can only have 32000 subdirectories with in any directory. Enough Inodes, and 32000 subdirectory's is quiet a lot, and definitly not a problem if you place all files in one directory ;) But, speed is a different story, so you might want to test how it works when you store the files in a database, it might be faster... Tijnema -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] OK to have many files in one folder?
For everyone who advised me to "beware the inode", allow me to forward what the Rackspace admins told me. This is Greek to me, and I'm hoping one of you can translate. All I understood was where he said "I appear to have more than enough." Yes? ---snip--- All of your slices on the disk are ext3. As this is the only file system that the Red Hat kernel provides support for. [EMAIL PROTECTED] ~]# df -hT Filesystem Type Size Used Avail Use% Mounted on /dev/hda5 ext3 227G 93G 123G 43% / /dev/hda1 ext3 99M 12M 83M 12% /boot none tmpfs 1.9G 0 1.9G 0% /dev/shm /dev/hda2 ext3 2.0G 36M 1.9G 2% /tmp As far as I am able to recall, there are only 2 limitations that you must be concerned with. There might be others, but these are the only ones I am aware of, which also appears to be true from the research I have done. 1. The number if i-nodes. You appear to have more than enough. [EMAIL PROTECTED] ~]# df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/hda5 30130176 3729871 26400305 13% / /dev/hda1 26104 46 26058 1% /boot none 220613 1 220612 1% /dev/shm /dev/hda2 262144 18 262126 1% /tmp 2. The subdirectory limitation for the ext3 file system. You can only have 32000 subdirectories with in any directory. ---/snip--- -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] OK to have many files in one folder?
[snip] > Two words. Beware the inode. ^^ ^^^ ^ 1 23 Here endeth the lesson. [/snip] Can I get an Admin brotha'!? -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] OK to have many files in one folder?
Jay Blanchard wrote: [snip] Server is running Linux, and PHP is constantly creating and modifying images in a directory. Apache is constantly serving these same images. There are about 250,000 of them in the same directory. Seems to be running OK, no problematic CPU load, but I'm wondering if anyone knows whether I'm living dangerously having that many docs in one directory with that much activity. It is an extremely busy server. Sorry is this seems like more of a linux sysad question than a PHP question. Thanks... :) [/snip] Two words. Beware the inode. ^^ ^^^ ^ 1 23 Here endeth the lesson. -Stut -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] OK to have many files in one folder?
On Saturday 16 June 2007 03:47, Daniel Brown wrote: > Once again, this doesn't matter so much for per-directory (though > listing will take longer, as I think I mentioned) as it does the > filesystem mount. Several years ago, having say 3000+ files in single directory on ext2 would mean that a simple 'ls -al' takes several orders of magnitude longer to perform than on a directory with just several hundred files. As I haven't used ext2/3 for ages I don't know whether the same is still true today. > The ext2/ext3 filesystems were made for these > reasons, especially with journaling like ReiserFS, XFS, et cetera > (which is a completely different bag of nuts). Not sure what point you're trying to make here, but, of the common filesystems for linux, ext2/3 is the absolute slowest (by far) when you have a large number of files in a directory. -- Crayon -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] OK to have many files in one folder?
Since it depends on many factors, some out of your control, some others that might vary over the lifetime of the application, one thing you might do, to be on the safe side is try to put files in different directories, for example, by breaking up their filenames every three characters and making each segment a folder but no more than three levels deep so that the file ./abcdefghijkl.jpg would be located at ./abc/def/ghijkl.jpg. This breakup would allow even to mount extra disk volumes for some of the folders, should they exceed certain capacity or to split the risk. Whether to break filenames every two, three or more characters might be a matter of optimization, but I would guess that around three (or four) for the first two segments and whatever is left for the filename would be a reasonable number. I wouldn't go more than three levels deep either, because jumping from one directory level to the next also takes some time, thus it is a compromise in between searching sequentially in a directory for a filename (for those filesystems that do so) and going deep into the directory tree. Satyam - Original Message - From: "Brian Dunning" <[EMAIL PROTECTED]> To: Sent: Friday, June 15, 2007 9:38 PM Subject: Re: [PHP] OK to have many files in one folder? Thanks for the replies but I'm not sure I know what to do with them. Is the problem with the number of files, or is the problem with the activity? Can you dumb down your answers at all for me? :) On Jun 15, 2007, at 11:51 AM, Daniel Brown wrote: 6/15/07, Jay Blanchard <[EMAIL PROTECTED]> wrote: [snip] Server is running Linux, and PHP is constantly creating and modifying images in a directory. Apache is constantly serving these same images. There are about 250,000 of them in the same directory. Seems to be running OK, no problematic CPU load, but I'm wondering if anyone knows whether I'm living dangerously having that many docs in one directory with that much activity. It is an extremely busy server. Sorry is this seems like more of a linux sysad question than a PHP question. Thanks... :) [/snip] Two words. Beware the inode. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php If that directory is all on one single partition, God help you if there's any corruption. Check into RAID 0+1/RAID 5 disk striping if you've got that many images. And remember, the fact that they're all in one directory doesn't matter at all to the system, as directories, folders, et cetera, are just representations for human readability and organization. In fact, those files reside on several sectors throughout the drive, and each file itself is probably fragmented many times. Just for fun, though, if you want to crash your server, you could always do: `while [ 1 = 1 ]; do ls -l;done` -- Daniel P. Brown [office] (570-) 587-7080 Ext. 272 [mobile] (570-) 766-8107 -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php -- No virus found in this incoming message. Checked by AVG Free Edition. Version: 7.5.472 / Virus Database: 269.8.16/849 - Release Date: 14/06/2007 12:44 -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] OK to have many files in one folder?
On 6/15/07, Jay Blanchard <[EMAIL PROTECTED]> wrote: [snip] I can easily break it up into 100 subdirectories, 2500 files in each, would that be good insurance against problems? [/snip] As someone mentioned, directories are just a human convenience. Each file will have an inode and is identified by an inode number in the file system where it resides. Inodes store information on files such as user and group ownership, access mode (read, write, execute permissions) and type of file. There is a fixed number of inodes, which indicates the maximum number of files each filesystem can hold. Hence, beware the inode. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php Perfect, Jay. That's something I forgot to mention. Also, keep in mind that, while there are file byte size limits (4GB, 8GB, etc.), directories are NOT limited in byte size. -- Daniel P. Brown [office] (570-) 587-7080 Ext. 272 [mobile] (570-) 766-8107 -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] OK to have many files in one folder?
On 6/15/07, Brian Dunning <[EMAIL PROTECTED]> wrote: Thanks for the replies but I'm not sure I know what to do with them. Is the problem with the number of files, or is the problem with the activity? Can you dumb down your answers at all for me? :) On Jun 15, 2007, at 11:51 AM, Daniel Brown wrote: > 6/15/07, Jay Blanchard <[EMAIL PROTECTED]> wrote: >> [snip] >> Server is running Linux, and PHP is constantly creating and modifying >> images in a directory. Apache is constantly serving these same >> images. There are about 250,000 of them in the same directory. Seems >> to be running OK, no problematic CPU load, but I'm wondering if >> anyone knows whether I'm living dangerously having that many docs in >> one directory with that much activity. It is an extremely busy >> server. >> >> Sorry is this seems like more of a linux sysad question than a PHP >> question. Thanks... :) >> [/snip] >> >> Two words. Beware the inode. >> >> -- >> PHP General Mailing List (http://www.php.net/) >> To unsubscribe, visit: http://www.php.net/unsub.php >> >> > >If that directory is all on one single partition, God help you if > there's any corruption. Check into RAID 0+1/RAID 5 disk striping if > you've got that many images. And remember, the fact that they're all > in one directory doesn't matter at all to the system, as directories, > folders, et cetera, are just representations for human readability and > organization. In fact, those files reside on several sectors > throughout the drive, and each file itself is probably fragmented many > times. > >Just for fun, though, if you want to crash your server, you > could always do: >`while [ 1 = 1 ]; do ls -l;done` > > -- > Daniel P. Brown > [office] (570-) 587-7080 Ext. 272 > [mobile] (570-) 766-8107 > > -- > PHP General Mailing List (http://www.php.net/) > To unsubscribe, visit: http://www.php.net/unsub.php > -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php Brian, I stand behind my suggestion, though I also suggest you take whatever everyone else says into consideration just as much as me. I have well over a decade of experience, but I'll be the first to admit that sometimes a teenage computer geek is more correct than I am. Yes, list, believe it or not, I'm not always correct. I'm just going to let that sink in a little bit. Okay, now that everyone is done laughing at me (well, mostly everyone), let's get on with it. I still think it's best to split the files across a RAID. Ask your datacenter to discuss the pros and cons of both RAID 0+1 and RAID 5 with you, and research it on the web. It would be better to split the files into different directories, as well, but *probably* is not necessary. Take that as you will. That said, one way of doing it that is very common is to create a minimum of 36 directories, from 0 - 9 and a - z. Each directory should contain only files where the first character is the same as the directory within which it resides. However, if you have a specific character that comes up much more frequently than others, or even all the time, this won't work, of course. The point is, in my experience, holding even a terabyte of data in smaller files within one directory is not a Bad Thing [tm], per se. I've administered MySQL databases on servers that were in the two-to-three terabyte range, and one of which had well over 3,000 separate tables. Each of those tables is three separate files (minimum) on disk, mind you: table.frm, table.MYD, table.MYI. Also, keep in mind that I didn't create the database structure, but was instead hired to ensure that it ran smoothly. In the four months and n-million queries that were run on it while it was in use, the server responded as if it were a standard 50MB database in under a dozen files (if memory serves, it was a Dual Xeon 2.6GHz with 4GB RAM) and only used for MySQL). After the project was completed, however, we did upgrade the systems to a distributed network for further expansion, but I think you get the point there. Bottom line, however, is that just because you're not experiencing problems now doesn't mean you won't soon in the future. Seriously consider, with that many separate files and that much data, how much it's worth to you in the long run, and if a RAID setup and directory-splitting is a Good Thing [tm] for your situation. -- Daniel P. Brown [office] (570-) 587-7080 Ext. 272 [mobile] (570-) 766-8107 -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] OK to have many files in one folder?
On 6/15/07, Brian Dunning <[EMAIL PROTECTED]> wrote: I can easily break it up into 100 subdirectories, 2500 files in each, would that be good insurance against problems? I have no clue how big the files are, but you might want to store them in a database. That can speed up things, but don't ask me how much ;) Tijnema -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] OK to have many files in one folder?
[snip] I can easily break it up into 100 subdirectories, 2500 files in each, would that be good insurance against problems? [/snip] As someone mentioned, directories are just a human convenience. Each file will have an inode and is identified by an inode number in the file system where it resides. Inodes store information on files such as user and group ownership, access mode (read, write, execute permissions) and type of file. There is a fixed number of inodes, which indicates the maximum number of files each filesystem can hold. Hence, beware the inode. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] OK to have many files in one folder?
I can easily break it up into 100 subdirectories, 2500 files in each, would that be good insurance against problems? -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] OK to have many files in one folder?
On 6/15/07, Crayon Shin Chan <[EMAIL PROTECTED]> wrote: On Saturday 16 June 2007 02:51, Daniel Brown wrote: > And remember, the fact that they're all > in one directory doesn't matter at all to the system, as directories, > folders, et cetera, are just representations for human readability and > organization. In fact, those files reside on several sectors > throughout the drive, and each file itself is probably fragmented many > times. Actually it does matter depending on the filesystem you use. If you're using ext2/ext3 then having several thousand files in a directory seriously slows things down. -- Crayon -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php Once again, this doesn't matter so much for per-directory (though listing will take longer, as I think I mentioned) as it does the filesystem mount. The ext2/ext3 filesystems were made for these reasons, especially with journaling like ReiserFS, XFS, et cetera (which is a completely different bag of nuts). -- Daniel P. Brown [office] (570-) 587-7080 Ext. 272 [mobile] (570-) 766-8107 -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] OK to have many files in one folder?
Thanks for the replies but I'm not sure I know what to do with them. Is the problem with the number of files, or is the problem with the activity? Can you dumb down your answers at all for me? :) On Jun 15, 2007, at 11:51 AM, Daniel Brown wrote: 6/15/07, Jay Blanchard <[EMAIL PROTECTED]> wrote: [snip] Server is running Linux, and PHP is constantly creating and modifying images in a directory. Apache is constantly serving these same images. There are about 250,000 of them in the same directory. Seems to be running OK, no problematic CPU load, but I'm wondering if anyone knows whether I'm living dangerously having that many docs in one directory with that much activity. It is an extremely busy server. Sorry is this seems like more of a linux sysad question than a PHP question. Thanks... :) [/snip] Two words. Beware the inode. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php If that directory is all on one single partition, God help you if there's any corruption. Check into RAID 0+1/RAID 5 disk striping if you've got that many images. And remember, the fact that they're all in one directory doesn't matter at all to the system, as directories, folders, et cetera, are just representations for human readability and organization. In fact, those files reside on several sectors throughout the drive, and each file itself is probably fragmented many times. Just for fun, though, if you want to crash your server, you could always do: `while [ 1 = 1 ]; do ls -l;done` -- Daniel P. Brown [office] (570-) 587-7080 Ext. 272 [mobile] (570-) 766-8107 -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] OK to have many files in one folder?
On Saturday 16 June 2007 02:51, Daniel Brown wrote: > And remember, the fact that they're all > in one directory doesn't matter at all to the system, as directories, > folders, et cetera, are just representations for human readability and > organization. In fact, those files reside on several sectors > throughout the drive, and each file itself is probably fragmented many > times. Actually it does matter depending on the filesystem you use. If you're using ext2/ext3 then having several thousand files in a directory seriously slows things down. -- Crayon -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] OK to have many files in one folder?
6/15/07, Jay Blanchard <[EMAIL PROTECTED]> wrote: [snip] Server is running Linux, and PHP is constantly creating and modifying images in a directory. Apache is constantly serving these same images. There are about 250,000 of them in the same directory. Seems to be running OK, no problematic CPU load, but I'm wondering if anyone knows whether I'm living dangerously having that many docs in one directory with that much activity. It is an extremely busy server. Sorry is this seems like more of a linux sysad question than a PHP question. Thanks... :) [/snip] Two words. Beware the inode. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php If that directory is all on one single partition, God help you if there's any corruption. Check into RAID 0+1/RAID 5 disk striping if you've got that many images. And remember, the fact that they're all in one directory doesn't matter at all to the system, as directories, folders, et cetera, are just representations for human readability and organization. In fact, those files reside on several sectors throughout the drive, and each file itself is probably fragmented many times. Just for fun, though, if you want to crash your server, you could always do: `while [ 1 = 1 ]; do ls -l;done` -- Daniel P. Brown [office] (570-) 587-7080 Ext. 272 [mobile] (570-) 766-8107 -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
RE: [PHP] OK to have many files in one folder?
[snip] Server is running Linux, and PHP is constantly creating and modifying images in a directory. Apache is constantly serving these same images. There are about 250,000 of them in the same directory. Seems to be running OK, no problematic CPU load, but I'm wondering if anyone knows whether I'm living dangerously having that many docs in one directory with that much activity. It is an extremely busy server. Sorry is this seems like more of a linux sysad question than a PHP question. Thanks... :) [/snip] Two words. Beware the inode. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
