Re: Query large CSV file as a data source?
> Am 03.11.2020 um 23:02 schrieb Bob Sneidar via use-livecode
> :
>
> Just read for 10,000 lines and check for empty each time.
>
To be honest, i've never used the read file for ... lines.
I was expecting that there might be an error if one tries to read more lines
than there are left for read
Good to know that one can just read in a loop without calculating the loop
count first.
-
Matthias Rebbe
Life Is Too Short For Boring Code
> Bob S
>
>
>> On Nov 3, 2020, at 7:09 AM, matthias rebbe via use-livecode
>> wrote:
>>
>> Regarding the loop.
>>
>> you could do a
>>
>> put the detailed files
>>
>> to get the file size of that csv file.
>>
>> With that size you would now how often you have to run the loop
>>
>>
>>
>>
>>
>> -
>> Matthias Rebbe
>> Life Is Too Short For Boring Code
>>
>>> Am 03.11.2020 um 15:14 schrieb Keith Clarke via use-livecode
>>> :
>>>
>>> Thank you Matthias, I wasn’t aware of that ability to open the file and
>>> read its contents straight from the disk.
>>>
>>> A quick experiment looks positive - no hanging and fast access - so I just
>>> need to work on the loop.
>>>
>>> Much obliged.
>>>
>>> Best,
>>> Keith
>>>
On 3 Nov 2020, at 13:37, matthias rebbe via use-livecode
wrote:
Couldn't you read the file partially
you would open the file for read
open file tFile for read
then in a repeat loop you could read e.g. 1 lines and process the data
read from file tFile for 1 lines
Just a thought.
-
Matthias Rebbe
Life Is Too Short For Boring Code
> Am 03.11.2020 um 13:14 schrieb Keith Clarke via use-livecode
> :
>
> Hi folks,
> I’ve a 3GB CSV file with 18M rows from which I need to create a subset
> CSV file for the rows containing tString. The following script results in
> LiveCode becoming unresponsive, so I’m wondering if I’m missing an
> obvious trick or what alternative approach I might take...
> on mouseUp pButtonNumber
>
> -- Select CSV File
>
> answer file prompt as sheet
>
> put it into tFile
>
> put tFile into field "File"
>
> -- Process CSV extract
>
> put URL ("file:" & tFile) into tText
>
> put line 1 of tText into field "List"
>
> filter lines of tText with "*tString*"
>
> put tText into field tList
>
> end mouseUp
>
> I don’t normally deal with large data sets, so maybe LiveCode isn’t the
> best tool for this but Excel 365 seems to only offer CSV file import, not
> query.
>
> Any thoughts greatly appreciated.
>
> Thanks & regards,
> Keith
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your
subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
>>>
>>>
>>> ___
>>> use-livecode mailing list
>>> use-livecode@lists.runrev.com
>>> Please visit this url to subscribe, unsubscribe and manage your
>>> subscription preferences:
>>> http://lists.runrev.com/mailman/listinfo/use-livecode
>>
>>
>> ___
>> use-livecode mailing list
>> use-livecode@lists.runrev.com
>> Please visit this url to subscribe, unsubscribe and manage your subscription
>> preferences:
>> http://lists.runrev.com/mailman/listinfo/use-livecode
>
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Query large CSV file as a data source?
Just read for 10,000 lines and check for empty each time.
Bob S
> On Nov 3, 2020, at 7:09 AM, matthias rebbe via use-livecode
> wrote:
>
> Regarding the loop.
>
> you could do a
>
> put the detailed files
>
> to get the file size of that csv file.
>
> With that size you would now how often you have to run the loop
>
>
>
>
>
> -
> Matthias Rebbe
> Life Is Too Short For Boring Code
>
>> Am 03.11.2020 um 15:14 schrieb Keith Clarke via use-livecode
>> :
>>
>> Thank you Matthias, I wasn’t aware of that ability to open the file and read
>> its contents straight from the disk.
>>
>> A quick experiment looks positive - no hanging and fast access - so I just
>> need to work on the loop.
>>
>> Much obliged.
>>
>> Best,
>> Keith
>>
>>> On 3 Nov 2020, at 13:37, matthias rebbe via use-livecode
>>> wrote:
>>>
>>> Couldn't you read the file partially
>>>
>>> you would open the file for read
>>>
>>> open file tFile for read
>>>
>>> then in a repeat loop you could read e.g. 1 lines and process the data
>>>
>>> read from file tFile for 1 lines
>>>
>>>
>>> Just a thought.
>>>
>>>
>>> -
>>> Matthias Rebbe
>>> Life Is Too Short For Boring Code
>>>
Am 03.11.2020 um 13:14 schrieb Keith Clarke via use-livecode
:
Hi folks,
I’ve a 3GB CSV file with 18M rows from which I need to create a subset CSV
file for the rows containing tString. The following script results in
LiveCode becoming unresponsive, so I’m wondering if I’m missing an obvious
trick or what alternative approach I might take...
on mouseUp pButtonNumber
-- Select CSV File
answer file prompt as sheet
put it into tFile
put tFile into field "File"
-- Process CSV extract
put URL ("file:" & tFile) into tText
put line 1 of tText into field "List"
filter lines of tText with "*tString*"
put tText into field tList
end mouseUp
I don’t normally deal with large data sets, so maybe LiveCode isn’t the
best tool for this but Excel 365 seems to only offer CSV file import, not
query.
Any thoughts greatly appreciated.
Thanks & regards,
Keith
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your
subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
>>>
>>>
>>> ___
>>> use-livecode mailing list
>>> use-livecode@lists.runrev.com
>>> Please visit this url to subscribe, unsubscribe and manage your
>>> subscription preferences:
>>> http://lists.runrev.com/mailman/listinfo/use-livecode
>>
>>
>> ___
>> use-livecode mailing list
>> use-livecode@lists.runrev.com
>> Please visit this url to subscribe, unsubscribe and manage your subscription
>> preferences:
>> http://lists.runrev.com/mailman/listinfo/use-livecode
>
>
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Query large CSV file as a data source?
Regarding the loop.
you could do a
put the detailed files
to get the file size of that csv file.
With that size you would now how often you have to run the loop
-
Matthias Rebbe
Life Is Too Short For Boring Code
> Am 03.11.2020 um 15:14 schrieb Keith Clarke via use-livecode
> :
>
> Thank you Matthias, I wasn’t aware of that ability to open the file and read
> its contents straight from the disk.
>
> A quick experiment looks positive - no hanging and fast access - so I just
> need to work on the loop.
>
> Much obliged.
>
> Best,
> Keith
>
>> On 3 Nov 2020, at 13:37, matthias rebbe via use-livecode
>> wrote:
>>
>> Couldn't you read the file partially
>>
>> you would open the file for read
>>
>> open file tFile for read
>>
>> then in a repeat loop you could read e.g. 1 lines and process the data
>>
>> read from file tFile for 1 lines
>>
>>
>> Just a thought.
>>
>>
>> -
>> Matthias Rebbe
>> Life Is Too Short For Boring Code
>>
>>> Am 03.11.2020 um 13:14 schrieb Keith Clarke via use-livecode
>>> :
>>>
>>> Hi folks,
>>> I’ve a 3GB CSV file with 18M rows from which I need to create a subset CSV
>>> file for the rows containing tString. The following script results in
>>> LiveCode becoming unresponsive, so I’m wondering if I’m missing an obvious
>>> trick or what alternative approach I might take...
>>> on mouseUp pButtonNumber
>>>
>>> -- Select CSV File
>>>
>>> answer file prompt as sheet
>>>
>>> put it into tFile
>>>
>>> put tFile into field "File"
>>>
>>> -- Process CSV extract
>>>
>>> put URL ("file:" & tFile) into tText
>>>
>>> put line 1 of tText into field "List"
>>>
>>> filter lines of tText with "*tString*"
>>>
>>> put tText into field tList
>>>
>>> end mouseUp
>>>
>>> I don’t normally deal with large data sets, so maybe LiveCode isn’t the
>>> best tool for this but Excel 365 seems to only offer CSV file import, not
>>> query.
>>>
>>> Any thoughts greatly appreciated.
>>>
>>> Thanks & regards,
>>> Keith
>>> ___
>>> use-livecode mailing list
>>> use-livecode@lists.runrev.com
>>> Please visit this url to subscribe, unsubscribe and manage your
>>> subscription preferences:
>>> http://lists.runrev.com/mailman/listinfo/use-livecode
>>
>>
>> ___
>> use-livecode mailing list
>> use-livecode@lists.runrev.com
>> Please visit this url to subscribe, unsubscribe and manage your subscription
>> preferences:
>> http://lists.runrev.com/mailman/listinfo/use-livecode
>
>
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Query large CSV file as a data source?
Thank you Matthias, I wasn’t aware of that ability to open the file and read
its contents straight from the disk.
A quick experiment looks positive - no hanging and fast access - so I just need
to work on the loop.
Much obliged.
Best,
Keith
> On 3 Nov 2020, at 13:37, matthias rebbe via use-livecode
> wrote:
>
> Couldn't you read the file partially
>
> you would open the file for read
>
> open file tFile for read
>
> then in a repeat loop you could read e.g. 1 lines and process the data
>
> read from file tFile for 1 lines
>
>
> Just a thought.
>
>
> -
> Matthias Rebbe
> Life Is Too Short For Boring Code
>
>> Am 03.11.2020 um 13:14 schrieb Keith Clarke via use-livecode
>> :
>>
>> Hi folks,
>> I’ve a 3GB CSV file with 18M rows from which I need to create a subset CSV
>> file for the rows containing tString. The following script results in
>> LiveCode becoming unresponsive, so I’m wondering if I’m missing an obvious
>> trick or what alternative approach I might take...
>> on mouseUp pButtonNumber
>>
>> -- Select CSV File
>>
>> answer file prompt as sheet
>>
>> put it into tFile
>>
>> put tFile into field "File"
>>
>> -- Process CSV extract
>>
>> put URL ("file:" & tFile) into tText
>>
>> put line 1 of tText into field "List"
>>
>> filter lines of tText with "*tString*"
>>
>> put tText into field tList
>>
>> end mouseUp
>>
>> I don’t normally deal with large data sets, so maybe LiveCode isn’t the best
>> tool for this but Excel 365 seems to only offer CSV file import, not query.
>>
>> Any thoughts greatly appreciated.
>>
>> Thanks & regards,
>> Keith
>> ___
>> use-livecode mailing list
>> use-livecode@lists.runrev.com
>> Please visit this url to subscribe, unsubscribe and manage your subscription
>> preferences:
>> http://lists.runrev.com/mailman/listinfo/use-livecode
>
>
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Query large CSV file as a data source?
Couldn't you read the file partially
you would open the file for read
open file tFile for read
then in a repeat loop you could read e.g. 1 lines and process the data
read from file tFile for 1 lines
Just a thought.
-
Matthias Rebbe
Life Is Too Short For Boring Code
> Am 03.11.2020 um 13:14 schrieb Keith Clarke via use-livecode
> :
>
> Hi folks,
> I’ve a 3GB CSV file with 18M rows from which I need to create a subset CSV
> file for the rows containing tString. The following script results in
> LiveCode becoming unresponsive, so I’m wondering if I’m missing an obvious
> trick or what alternative approach I might take...
> on mouseUp pButtonNumber
>
> -- Select CSV File
>
> answer file prompt as sheet
>
> put it into tFile
>
> put tFile into field "File"
>
> -- Process CSV extract
>
> put URL ("file:" & tFile) into tText
>
> put line 1 of tText into field "List"
>
> filter lines of tText with "*tString*"
>
> put tText into field tList
>
> end mouseUp
>
> I don’t normally deal with large data sets, so maybe LiveCode isn’t the best
> tool for this but Excel 365 seems to only offer CSV file import, not query.
>
> Any thoughts greatly appreciated.
>
> Thanks & regards,
> Keith
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Query large CSV file as a data source?
Hi folks,
I’ve a 3GB CSV file with 18M rows from which I need to create a subset CSV file
for the rows containing tString. The following script results in LiveCode
becoming unresponsive, so I’m wondering if I’m missing an obvious trick or what
alternative approach I might take...
on mouseUp pButtonNumber
-- Select CSV File
answer file prompt as sheet
put it into tFile
put tFile into field "File"
-- Process CSV extract
put URL ("file:" & tFile) into tText
put line 1 of tText into field "List"
filter lines of tText with "*tString*"
put tText into field tList
end mouseUp
I don’t normally deal with large data sets, so maybe LiveCode isn’t the best
tool for this but Excel 365 seems to only offer CSV file import, not query.
Any thoughts greatly appreciated.
Thanks & regards,
Keith
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
