Mitch,
If the speed is the issue, you may try to write/read array elements to disk
using SEND PACKET / RECEIVE PACKET. If you want to speed up reading, insert
number of elements into first line, like
Create document
SEND PACKET(file;Size of array(arrayElement)+"\n")
loop for all elements in array
SEND PACKET(file;arrayElement{}+"\n")
end loop
Close Document
and vice versa
Open document
RECEIVE PACKET(file;$t;"\n")
arraySize:=Num($t)
array text(arrayElement;arraySize)
for 1..arraySize
RECEIVE PACKET(file;$t;"\n")
arrayElement{}:=$t
end for
Close Document
The above should avoid a part, where you (or 4D) is resizing a big memory
block to insert additional array elements.
--
Peter Bozek
On Mon, Jan 6, 2020 at 5:09 AM Mitchell Shiller via 4D_Tech <
4d_tech@lists.4d.com> wrote:
> Hi,
>
> I have large string arrays (about 200 k elements).
> I need to write and read them to disk.
> Speed is the most important criteria.
>
> Options
> 1) Create a text variable (loop with CR delimiter) and then TEXT TO
> DOCUMENT.
> 2) VARIABLE TO BLOB, COMPRESS BLOB, BLOB TO DOCUMENT
> 3) OB SET ARRAY, JSON STRINGIFY, TEXT TO DOCUMENT
>
> Obviously using the reverse commands to read from disk.
> Option one seems by far the slowest.
>
> Anyone know if 2 or 3 is faster?
> Does the COMPRESS BLOB help? The resultant file is about 2 MB in size
> without compression.
> If yes to COMPRESS BLOB , Any différence from native 4D vs. GZIP
> compression?
>
> Thanks as always.
>
> Mitch
>
>
>
> Sent from my iPad
> **********************************************************************
> 4D Internet Users Group (4D iNUG)
> Archive: http://lists.4d.com/archives.html
> Options: https://lists.4d.com/mailman/options/4d_tech
> Unsub: mailto:4d_tech-unsubscr...@lists.4d.com
> **********************************************************************
--
--
Peter Bozek
**********************************************************************
4D Internet Users Group (4D iNUG)
Archive: http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4d_tech-unsubscr...@lists.4d.com
**********************************************************************
