Regarding the loop. you could do a
put the detailed files to get the file size of that csv file. With that size you would now how often you have to run the loop - Matthias Rebbe Life Is Too Short For Boring Code > Am 03.11.2020 um 15:14 schrieb Keith Clarke via use-livecode > <use-livecode@lists.runrev.com>: > > Thank you Matthias, I wasn’t aware of that ability to open the file and read > its contents straight from the disk. > > A quick experiment looks positive - no hanging and fast access - so I just > need to work on the loop. > > Much obliged. > > Best, > Keith > >> On 3 Nov 2020, at 13:37, matthias rebbe via use-livecode >> <use-livecode@lists.runrev.com> wrote: >> >> Couldn't you read the file partially >> >> you would open the file for read >> >> open file tFile for read >> >> then in a repeat loop you could read e.g. 10000 lines and process the data >> >> read from file tFile for 10000 lines >> >> >> Just a thought. >> >> >> - >> Matthias Rebbe >> Life Is Too Short For Boring Code >> >>> Am 03.11.2020 um 13:14 schrieb Keith Clarke via use-livecode >>> <use-livecode@lists.runrev.com>: >>> >>> Hi folks, >>> I’ve a 3GB CSV file with 18M rows from which I need to create a subset CSV >>> file for the rows containing tString. The following script results in >>> LiveCode becoming unresponsive, so I’m wondering if I’m missing an obvious >>> trick or what alternative approach I might take... >>> on mouseUp pButtonNumber >>> >>> -- Select CSV File >>> >>> answer file prompt as sheet >>> >>> put it into tFile >>> >>> put tFile into field "File" >>> >>> -- Process CSV extract >>> >>> put URL ("file:" & tFile) into tText >>> >>> put line 1 of tText into field "List" >>> >>> filter lines of tText with "*tString*" >>> >>> put tText into field tList >>> >>> end mouseUp >>> >>> I don’t normally deal with large data sets, so maybe LiveCode isn’t the >>> best tool for this but Excel 365 seems to only offer CSV file import, not >>> query. >>> >>> Any thoughts greatly appreciated. >>> >>> Thanks & regards, >>> Keith >>> _______________________________________________ >>> use-livecode mailing list >>> use-livecode@lists.runrev.com >>> Please visit this url to subscribe, unsubscribe and manage your >>> subscription preferences: >>> http://lists.runrev.com/mailman/listinfo/use-livecode >> >> >> _______________________________________________ >> use-livecode mailing list >> use-livecode@lists.runrev.com >> Please visit this url to subscribe, unsubscribe and manage your subscription >> preferences: >> http://lists.runrev.com/mailman/listinfo/use-livecode > > > _______________________________________________ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your subscription > preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode _______________________________________________ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode