Couldn't you read the file partially
you would open the file for read
open file tFile for read
then in a repeat loop you could read e.g. 10000 lines and process the data
read from file tFile for 10000 lines
Just a thought.
-
Matthias Rebbe
Life Is Too Short For Boring Code
> Am 03.11.2020 um 13:14 schrieb Keith Clarke via use-livecode
> <use-livecode@lists.runrev.com>:
>
> Hi folks,
> I’ve a 3GB CSV file with 18M rows from which I need to create a subset CSV
> file for the rows containing tString. The following script results in
> LiveCode becoming unresponsive, so I’m wondering if I’m missing an obvious
> trick or what alternative approach I might take...
> on mouseUp pButtonNumber
>
> -- Select CSV File
>
> answer file prompt as sheet
>
> put it into tFile
>
> put tFile into field "File"
>
> -- Process CSV extract
>
> put URL ("file:" & tFile) into tText
>
> put line 1 of tText into field "List"
>
> filter lines of tText with "*tString*"
>
> put tText into field tList
>
> end mouseUp
>
> I don’t normally deal with large data sets, so maybe LiveCode isn’t the best
> tool for this but Excel 365 seems to only offer CSV file import, not query.
>
> Any thoughts greatly appreciated.
>
> Thanks & regards,
> Keith
> _______________________________________________
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
_______________________________________________
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
