So I tested my setup. V17R5, MacOS Mojave. My results were slightly different. Your mileage may vary.
Compared BLOB, JSON Stringify Array, and OB SET ARRAY / JSON Stringify. For writing an array that exists in memory, BLOB was fastest with average times in 6-9 milliseconds. JSA was in the 30-34 milliseconds. OB / JS in the 36-40 milliseconds. Compressing the BLOB prior to writing slowed the process with times of about 20-21 milliseconds. However, the file size was 1/3 as big. Shrinking from 2.3MB to 0.74 MB. Note that the files written with JSA and OB/ JS were 1.1- 1.3 MB in size. Reading showed similar results . BLOB 10-16 milliseconds. JSA 36-43 milliseconds. OB / JS 38-47 milliseconds. Reading a compressed BLOB took 16-18 milliseconds. In the end, any of these methods is fast enough for my needs. The lesson being, that while a text loop looks simple and intuitive, it incurs huge overhead and cannot be used. I guess your choice would depend on your need / preference for smaller file size, readability etc. Thanks for everyone’s input. Mitch >> >> Writing > I believe disk access is now heavily cached by both disk controller, system > and (probably) 4D, with asynchronous access. I would be surprised if this is > bottleneck, but it need to be tested. > > So I tested various approaches, code is down. I skipped first option - it is > painstakingly slow - and added another possibility - similar to option 3, but > instead of OB SET ARRAY / JSON Stringify, I used JSON Stringify Array. This > turned up to be the fastest method. > > Here are results as a screenshot: sorry, too lazy to write it down. > > <Screenshot 2020-01-06 18.52.08.png> > > > My conclusion: direct writing to disk is pretty fast, and grows linearly with > size of the array and does not depend weakly on size of elements, so it seems > to be a good choice for large arrays and solutions that need to be scalable. > > VARIABLE TO BLOB is too slow. Did not tried compression, but that would not > improve performance, IMHO. > > JSON Stringify Array is the fastest solution, it seems to grow slightly > faster, depending on overall size of data so may be slower for large arrays / > element length, but I am surprised how well are these libraries optimised. As > the code is really simple, I would go with this. > ********************************************************************** 4D Internet Users Group (4D iNUG) Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:[email protected] **********************************************************************
