If I understand how your loop works, you are reading one record from the hard disk at a time, then processing that record, then going to fetch the next record from the hard disk. If that is what you are doing, then this a VERY slow way to process 300,000 records.
Each hit to the hard drive is going to incur a seek and a latency to find the record before you read it. As an example, if you have a 10ms average seek time, that is an average .01 secs to find each record. For 300,000 records you have added 3000 seconds, or 50 minutes, onto your processing time. (Actual performance may vary. Hard drive seeks vary dramatically with 8-12 MS being a good estimate for most. And I haven't even begun to factor in latency speeds, which will also hit you for every read from the hard drive.) You will get MUCH better performance if you read blocks of records into memory, then process all the record in that block. This results in far fewer seeks and better data throughput. For example, if you read 1000 records at a time, for 300,000 records you only hit the hard drive 300 times. 300 *.01 avg seek time = 3 seconds instead of 3000 seconds. Depending upon the size of the file and available RAM, you might well want to read the entire file before you start processing records. Doing this might make your overall process so fast you won't need to perform it on a separate thread. -- R.B. Davidson On Feb 8, 1:02 am, "Joe C." <[email protected]> wrote: > thank you for reading. > > i have a windows application, a tool built to take large text files > (containing data) to import into a sql server table. > > the text files contain more than 300,000 lines of data at a time. > > vb.net code in the windows application reads each line, parses, then > inserts into the database table. > > on the form i have set a label to show the progress of the data > migration, and a progress bar reflecting percentage finished. > > problem is, during the loop, if the form is moved or focus taken away > from the form, the form itself freezes. the progress bar no longer > moves forward, and label also stops showing any updates. > > but the background process, of reading the text line, then inserting > into sql, still runs and eventually the entire process will finish. > > so my problem is a cosmetic one, i would like to form to continue > updating the information while the long running loop is completed. > > any help would be greatly appreciated. > > - Joe
